<chapt id="bugs">
      <heading>Reporting and handling bugs</heading>

      <sect>
	Bug handling policy for the kernel team

	<sect1>
	  Required information
	  <p>
	    Submitters are expected to run <prgn>reportbug</prgn> or
	    other tool that runs our <package>bug</package> script
	    under the kernel version in question.  The response to
	    reports without this information should be a request to
	    follow-up using <prgn>reportbug</prgn>.  If we do not
	    receive this information within a month of the request,
	    the bug may be closed.
	  </p>
	  <p>
	    Exceptions:
	    <list>
	      <item>
		If the kernel does not boot or is very unstable, instead
		of the usual system information we need the console
		messages via
		<url id="http://www.kernel.org/doc/Documentation/networking/netconsole.txt" name="netconsole">,
		<url id="http://www.kernel.org/doc/Documentation/serial-console.txt" name="serial console">,
		or a photograph.
	      </item>
	      <item>
		If the report is relaying information about a bug
		acknowledged upstream, we do not need system
		information but we do need specific references
		(<httpsite>bugzilla.kernel.org</httpsite>
		or <prgn>git</prgn> commit id).
	      </item>
	      <item>
		If the bug is clearly not hardware-specific
		(e.g. packaging error), we do not need system
		information.
	      </item>
	      <item>
		If the bug is reported against a well-defined model, we
		may not need device listings.
	      </item>
	    </list>
	  </p>
	</sect1>

	<sect1>
	  Severities
	  <p>
	    Many submitters report bugs with the wrong severity.  We
	    interpret the criteria as follows and will adjust severity
	    as appropriate:
	    <taglist>
	      <tag>
		critical: makes unrelated software on the system (or the
		whole system) break...
	      </tag>
	      <item>
		The bug must make the kernel unbootable or unstable on
		common hardware or all systems that a specific flavour
		is supposed to support.  There is no 'unrelated
		software' since everything depends on the kernel.
	      </item>
	      <tag>
		grave: makes the package in question unusable or mostly
		so...
	      </tag>
	      <item>
		If the kernel is unusable, this already qualifies as
		critical.
	      </item>
	      <tag>
		grave: ...or causes data loss...
	      </tag>
	      <item>
		We exclude loss of data in memory due to a crash.  Only
		corruption of data in storage or communication, or
		silent failure to write data, qualifies.
	      </item>
	      <tag>
		important
	      </tag>
	      <item>
		We include lack of support for new hardware that is
		generally available.
	      </item>
	    </taglist>
	  </p>
	</sect1>

	<sect1>
	  Tagging
	  <p>
	    We do not use user-tags.  In order to aid bug triage we
	    should make use of the standard tags
	    and <tt>forwarded</tt> field defined by the BTS.  In
	    particular:
	    <list>
	      <item>
		Add <tt>moreinfo</tt> whenever we are waiting for a
		response from the submitter and remove it when we are
		not
	      </item>
	      <item>
		Do not add <tt>unreproducible</tt> to bugs that may be
		hardware-dependent
	      </item>
	    </list>
	  </p>
	</sect1>

	<sect1>
	  Analysis by maintainers
	  <p>
	    Generally we should not expect to be able to reproduce
	    bugs without having similar hardware.  We should consider:
	    <list>
	      <item>
		Searching <url id="http://bugzilla.kernel.org" name="bugzilla.kernel.org">
		(including closed bugs) or other relevant bug tracker
	      </item>
	      <item>
		Searching kernel mailing lists
		<list>
		  <item>
		    Of the many
		    archives, <url id="http://news.gmane.org" name="news.gmane.org"> seems
		    to suck least
		  </item>
		  <item>
		    Patches submitted to some lists are archived at
		    <url id="http://patchwork.kernel.org" name="patchwork.kernel.org">
		  </item>
		</list>
	      </item>
	      <item>
		Viewing git commit logs for relevant source files
		<list>
		  <item>
		    In case of a regression, from the known good to the
		    bad version
		  </item>
		  <item>
		    In other cases, from the bad version forwards, in
		    case the bug has been fixed since
		  </item>
		</list>
	      </item>
	      <item>
		Searching kerneloops.org for similar oopses
	      </item>
	      <item>
		Matching the machine code and registers in an 'oops'
		against the source and deducing how the impossible
		happened (this doesn't work that often but when it does
		you look like a genius ;-)
	      </item>
	    </list>
	  </p>
	</sect1>

	<sect1>
	  Testing by submitter
	  <p>
	    Depending on the technical sophistication of the submitter
	    and the service requirements of the system in question
	    (e.g. whether it's a production server) we can request one
	    or more of the following:
	    <list>
	      <item>
		Gathering more information passively (e.g. further
		logging, reporting contents of files in procfs or sysfs)
	      </item>
	      <item>
		Upgrading to the current
		stable/stable-proposed-updates/stable-security version,
		if it includes a fix for a similar bug
	      </item>
	      <item>
		Adding debug or fallback options to the kernel command
		line or module parameters
	      </item>
	      <item>
		Installing the unstable or backports version temporarily
	      </item>
	      <item>
		Rebuilding and installing the kernel with a specific
		patch added (the
		script <prgn>debian/bin/test-patches</prgn> should make
		this easy)
	      </item>
	      <item>
		Using <prgn>git bisect</prgn> to find a specific
		upstream change that introduced the bug
	      </item>
	    </list>
	  </p>
	  <p>
	    When a bug occurs in what upstream considers the current
	    or previous stable release, and we cannot fix it, we ask
	    the submitter to report it upstream
	    at <httpsite>bugzilla.kernel.org</httpsite> under a
	    specific Product and Component, and to tell us the
	    upstream bug number.  We do not report bugs directly
	    because follow-up questions from upstream need to go to
	    the submitter, not to us.  Given the upstream bug number,
	    we mark the bug as forwarded.
	    <prgn>bts-link</prgn> then updates its status.
	  </p>
	</sect1>

	<sect1>
	  Keeping bugs separate
	  <p>
	    Many submitters search for a characteristic error message
	    and treat this as indicating a specific bug.  This can
	    lead to many 'me too' follow-ups where, for example, the
	    message indicates a driver bug and the second submitter is
	    using a different driver from the original submitter.
	  </p>
	  <p>
	    In order to avoid the report turning into a mess of
	    conflicting information about two or more different bugs:
	    <list>
	      <item>
		We should try to respond to such a follow-up quickly,
		requesting a separate bug report
	      </item>
	      <item>
		We can use the BTS <tt>summary</tt> command to improve
		the description of the bug
	      </item>
	      <item>
		As a last resort, it may be necessary to open new bugs
		with the relevant information, set their submitters
		accordingly, and close the original report
	      </item>
	    </list>
	  </p>
	  <p>
	    Where the original report describes more than one bug
	    ('...and other thing...'), we should clone it and deal
	    with each separately.
	  </p>
	</sect1>

	<sect1>
	  Applying patches
	  <p>
	    Patches should normally be reviewed and accepted by the
	    relevant upstream maintainer (aside from necessary
	    adjustments for an older kernel version) before being
	    applied.
	  </p>
	</sect1>

	<sect1>
	  Talking to submitters
	  <p>
	    We should always be polite to submitters.  Not only is
	    this implied by the
	    <url id="http://www.debian.org/social_contract"
	    name="Social Contract">, but it is likely to lead to a
	    faster resolution of the bug.  If a submitter overrated
	    the severity, quietly downgrade it.  If a submitter has
	    done something stupid, request that they undo that and
	    report back.  'Sorry' and 'please' make a big difference
	    in tone.
	  </p>
	  <p>
	    We will maintain general advice to submitters at
	    <url id="http://wiki.debian.org/DebianKernelReportingBugs"
		 name="http://wiki.debian.org/DebianKernelReportingBugs">.
	  </p>
	</sect1>

      </sect>

      <sect>
	Filing a bug against a kernel package

      <p>
        Debian kernel team keeps track of the kernel package bugs in
	the Debian Bug Tracking System (BTS). For information on how
	to use the system see <url id="http://bugs.debian.org"
	name="http://bugs.debian.org">.  You can also submit the bugs
	by using the <tt>reportbug</tt> command from the package with
	the same name. Please note that kernel bugs found in
	distributions derived from Debian (such as Knoppix, Mepis,
	Progeny, Ubuntu, Xandros, etc.) should <em>not</em> be
	reported to the Debian BTS (unless they can be also reproduced
	on a Debian system using official Debian kernel
	packages). Derived distributions have their own policies and
	procedures regarding kernel packaging, so the bugs found in
	them should be reported directly to their bug tracking systems
	or mailing lists.
      </p>
      <p>
        Nothing in this chapter is intended to keep you from filing a
	bug against one of the Debian kernel packages.  However, you should
	recognize that the resources of the Debian kernel team are
	limited, and efficient reaction to a bug is largely determined
	by the amount and quality of the information included in the
	bug report. Please help us to do a better job by using the
	following guidelines when preparing to file the bug against
	kernel packages:
      <list>
	<item>
	  <em>Do the research.</em> Before filing the bug search the
	  web for the particular error message or symptom you are
	  getting. As it is highly unlikely that you are the only
	  person experiencing a particular problem, there is always a
	  chance that it has been discussed elsewhere, and a possible
	  solution, patch, or workaround has been proposed. If such
	  information exists, always include the references to it in
	  your report. Check the <url id="http://bugs.debian.org/cgi-bin/pkgreport.cgi?src=linux&#38;amp;src=linux-2.6" name="current bug list">
	  to see whether something similar has been reported already.
	</item>
	<item>
	  <em>Collect the information.</em> Please provide enough
	  information with your report. At a minimum, it should
	  contain the exact version of the official Debian kernel
	  package, where the bug is encountered, and steps to
	  reproduce it. Depending on the nature of the bug you are
	  reporting, you might also want to include the output of
	  <tt>dmesg</tt> (or portions thereof), output of the
	  <tt>lspci -vn</tt>. <prgn>reportbug</prgn> will do this
	  automatically. If applicable,
	  include the information about the latest known kernel
	  version where the bug is not present, and output of the
	  above commands for the working kernel as well. Use
	  common sense and include other relevant information,
	  if you think that it might help in solving the problem.
	</item>
	<item>
	  <em>Try to reproduce the problem with "vanilla" kernel.</em>
	  If you have a chance, try to reproduce the problem by
	  building the binary kernel image from the "vanilla" kernel
	  source, available from <url id="http://www.kernel.org"
	  name="http://www.kernel.org"> or its mirrors, using the same
	  configuration as the Debian stock kernels. For more
	  information on how to do this, look at <ref
	  id="common-building">. If there is convincing evidence that
	  the buggy behavior is caused by the Debian-specific changes
	  to the kernel, the bug will usually be assigned higher
	  priority by the kernel team. If the bug is not specific for
	  Debian, check out the upstream <url
	  id="http://bugzilla.kernel.org" name="kernel bug database">
	  to see if it has been reported there. If you are sure that
	  it is an upstream problem, you can also report your bug
	  there (but submit it to Debian BTS anyway, so that we can
	  track it properly).
	</item>
        <item>
          <em>Use the correct package to report the bug against.</em>
	  Please file bugs against the package containing the kernel
	  version where the problem occurs
	  (e.g. <package>linux-image-3.2.0-2-686-pae</package>), not
	  a metapackage (e.g. <package>linux-image-686-pae</package>).
        </item>
	<item>
	  <em>Bugs involving tainted kernels.</em> If a kernel
	  crashes, it normally prints out some debugging
	  information, indicating, among other things, whether the
	  running kernel has been tainted. The kernel is referred to
	  as tainted if at the time of the crash it had some binary
	  third-party modules loaded. As kernel developers do not
	  have access to the source code for such modules, problems
	  involving them are notoriously difficult to debug. It is
	  therefore strongly recommended to try and reproduce the
	  problem with an untainted kernel (by preventing the loading
	  of binary modules, for example). If the problem is due to
	  the presence of such modules, there is not much the kernel
	  community can do about it and it should be reported directly
	  to their authors.
	</item>
      </list>
      </p>

	<sect1>
	  Bisecting (finding the upstream version that introduced a bug)

	<p>
	  When a bug is easy to reproduce locally but hard to get developers
	  to reproduce (as is often true of workflow- or hardware-dependent
	  bugs), it can be useful to compile and test a few versions to narrow
	  down what changes introduced the regression.
	</p>

	<p>
	  To start, recreate the problem with a vanilla kernel:
	  <example>
# apt-get install git build-essential
$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
$ cd linux
	  </example>
	  The above commands acquire a vanilla kernel.
	  Configure, build and test a binary package as explained in
	  <ref id="common-building">:
	  <example>
$ make localmodconfig; # minimal configuration
$ scripts/config --disable DEBUG_INFO; # to keep the build reasonably small
$ make deb-pkg
# dpkg -i ../linux-image-3.5.0_3.5.0-1_i386.deb; # substitute package name from the previous command
# reboot
	  </example>
	  If the bug doesn't show up, try again with the official
	  configuration file from /boot.  (If it still doesn't show up
	  after that, declare victory and celebrate.)
	</p>

	<p>
	  Initialize the bisection process by declaring which versions worked
	  and did not work:
	  <example>
$ cd linux
$ git bisect start
$ git bisect good v3.0; # or whichever was known to be good
$ git bisect bad; # current version is bad
	  </example>
	  Now git checks out a version half-way in between to test.
	  Build it, reusing the prepared configuration.
	  <example>
$ make silentoldconfig
$ make deb-pkg
	  </example>
	</p>

	<p>
	  Install the package, reboot, and test.
	  <example>
$ git bisect good; # if this version doesn't exhibit the bug
$ git bisect bad; # if it does
$ git bisect skip; # if some other bug makes it hard to test
	  </example>
	  And on to the next iteration:
	  <example>
$ make silentoldconfig
$ make deb-pkg
	  </example>
	</p>

	<p>
	  At the end of the process, the name of the "first bad commit" is
	  printed, which is very useful for tracking down the bug. Narrowing
	  down the regression range with a few rounds is useful even if you
	  don't get that far; in that case, run <tt>git bisect log</tt> to
	  produce a log. If you are the visual sort of person, <tt>git bisect
	  visualize</tt> with the <tt>gitk</tt> package installed can show
	  what is happening between steps.
	</p>

	<p>
	  See Christian Couder's article "Fighting regressions with git bisect"
	  from <url id="http://www.kernel.org/pub/software/scm/git/docs/git-bisect-lk2009.html"
	  name="kernel.org"> or <url id="file:///usr/share/doc/git-doc/git-bisect-lk2009.html"
	  name="the git-doc package"> for details.
	</p>
	</sect1>
      </sect>
</chapt>