<!-- CVS revision of this document "$Revision: 1.18 $" -->

<chapt id="after-compromise">After the compromise (incident response)

<sect>General behavior

<p>If you are physically present when an attack is happening, your
first response should be to remove the machine from the network by
unplugging the network card (if this will not adversely affect any
business transactions). Disabling the network at layer 1 is the only
true way to keep the attacker out of the compromised box (Phillip
Hofmeister's wise advice).

<p>However, some tools installed by rootkits, trojans and, even,
a rogue user connected through a back door, might be capable of detecting this
event and react to it. Seeing a <tt>rm -rf /</tt> executed when you unplug
the network from the system is not really much fun. If you are
unwilling to take the risk, and you are sure that the system is
compromised, you should <em>unplug the power cable</em> (all of them
if more than one) and cross your fingers. This may be extreme but, in
fact, will avoid any logic-bomb that the intruder might have
programmed. In this case, the compromised system <em>should not be
re-booted</em>. Either the hard disks should be moved to another
system for analysis, or you should use other media (a CD-ROM) to boot
the system and analyze it. You should <em>not</em> use Debian's rescue
disks to boot the system, but you <em>can</em> use the shell provided
by the installation disks (remember, Alt+F2 will take you to it) to
analyze <footnote>If you are adventurous, you can login to the system and save
information on all running processes (you'll get a lot from
/proc/nnn/). It is possible to get the whole executable code from
memory, even if the attacker has deleted the executable files from
disk. Then pull the power cord.</footnote>
the system.

<p>The most recommended method for recovering a compromised system is
to use a live-filesystem on CD-ROM with all the tools (and kernel
modules) you might need to access the compromised system. You can use
the <package>mkinitrd-cd</package> package to build such a
CD-ROM<footnote>In fact, this is the tool used to build the CD-ROMs for
the <url id="http://www.gibraltar.at/" name="Gibraltar"> project (a
firewall on a live CD-ROM based on the Debian
distribution).</footnote>. You might find the <url
id="http://biatchux.dmzs.com/" name="FIRE"> (previously called Biatchux)
CD-ROM useful here too,
since it's also a live CD-ROM with forensic tools useful in these
situations. There is not (yet) a Debian-based tool such as this, nor
an easy way to build the CD-ROM using your own selection of Debian
packages and <package>mkinitrd-cd</package> (so you'll have to read
the documentation provided with it to make your own CD-ROMs).

<p>If you really want to fix the compromise quickly, you should remove
the compromised host from your network and re-install the operating
system from scratch. Of course, this may not be effective because you
will not learn how the intruder got root in the first place. For that
case, you must check everything: firewall, file integrity, log host,
log files and so on. For more information on what to do following a
break-in, see 
<url
id="http://www.cert.org/tech_tips/root_compromise.html" name="CERT's
Steps for Recovering from a UNIX or NT System Compromise"> or
SANS's
<url name="Incident Handling whitepapers"
id="http://www.sans.org/reading_room/whitepapers/incident/">.

<p>Some common questions on how to handle a compromised Debian
GNU/Linux system are also available in <ref id="vulnerable-system">.

<sect>Backing up the system

<p>Remember that if you are sure the system has been compromised you
cannot trust the installed software or any information that it gives
back to you. Applications might have been trojanized, kernel modules
might be installed, etc.

<p>The best thing to do is a complete file system backup copy (using
<prgn>dd</prgn>) after booting from a safe medium. Debian GNU/Linux
CD-ROMs can be handy for this since they provide a shell in console 2
when the installation is started (jump to it using Alt+2 and pressing
Enter). From this shell, backup the information to another host if
possible (maybe a network file server through NFS/FTP). Then any
analysis of the compromise or re-installation can be performed while
the affected system is offline.

<p>If you are sure that the only compromise is a Trojan kernel module,
you can try to run the kernel image from the Debian CD-ROM in
<em>rescue</em> mode. Make sure to startup in <em>single user</em>
mode, so no other Trojan processes run after the kernel.

<sect>Contact your local CERT
<p>The CERT (Computer and Emergency Response Team) is an organization
that can help you recover from a system compromise. There are CERTs
worldwide
<footnote>
This is a list of some CERTs, for a full list look at the 
<url id="http://www.first.org/about/organization/teams/index.html"
name="FIRST Member Team information">
(FIRST is the Forum of Incident Response and Security Teams):
<url id="http://www.auscert.org.au" name="AusCERT"> (Australia),
<url id="http://www.unam-cert.unam.mx/" name="UNAM-CERT"> (Mexico)
<url id="http://www.cert.funet.fi" name="CERT-Funet"> (Finland),
<url id="http://www.dfn-cert.de" name="DFN-CERT"> (Germany), 
<url id="http://cert.uni-stuttgart.de/" name="RUS-CERT"> (Germany),
<!--FIXME URL -->
<url id="http://security.dico.unimi.it/" name="CERT-IT"> (Italy),
<url id="http://www.jpcert.or.jp/" name="JPCERT/CC"> (Japan),
<url id="http://cert.uninett.no" name="UNINETT CERT"> (Norway),
<url id="http://www.cert.hr" name="HR-CERT"> (Croatia)
<url id="http://www.cert.pl" name="CERT Polskay"> (Poland),
<url id="http://www.cert.ru" name="RU-CERT"> (Russia),
<url id="http://www.arnes.si/si-cert/" name="SI-CERT"> (Slovenia)
<url id="http://www.rediris.es/cert/" name="IRIS-CERT"> (Spain),
<url id="http://www.switch.ch/cert/" name="SWITCH-CERT"> (Switzerland),
<url id="http://www.cert.org.tw" name="TWCERT/CC"> (Taiwan), 
and
<url id="http://www.cert.org" name="CERT/CC"> (US).
</footnote>
and you should contact your local CERT in the event of a security incident
which has lead to a system compromise. The people at your local CERT
can help you recover from it.

<p>Providing your local CERT (or the CERT coordination center) with information
on the compromise even if you do not seek assistance can also help
others since the aggregate information of reported incidents is used
in order to determine if a given vulnerability is in wide spread use,
if there is a new worm aloft, which new attack tools are being used.
This information is used in order to provide the Internet community
with information on the <url id="http://www.cert.org/current/" name="current 
security incidents activity">, and to publish 
<url id="http://www.cert.org/incident_notes/" name="incident notes"> and
even <url id="http://www.cert.org/advisories/" name="advisories">. 
For more detailed information read on how (and why) to report an
incident read <url id="http://www.cert.org/tech_tips/incident_reporting.html"
name="CERT's Incident Reporting Guidelines">.

<p>You can also use less formal mechanisms if you need help for recovering
from a compromise or want to discuss incident information. This includes
the
<url id="http://marc.theaimsgroup.com/?l=incidents" name="incidents
mailing list"> and the 
<url id="http://marc.theaimsgroup.com/?l=intrusions" name="Intrusions
mailing list">.


<sect>Forensic analysis

<p>If you wish to gather more information, the <package>tct</package>
(The Coroner's Toolkit from Dan Farmer and Wietse Venema) package
contains utilities which perform a <em>post mortem</em> analysis of a system.
<package>tct</package> allows the user to collect information about
deleted files, running processes and more. See the included
documentation for more information. These same utilities and some others
can be found in <url name="Sleuthkit and Autopsy" id="http://www.sleuthkit.org/">
by Brian Carrier, which provides a web front-end for forensic analysis 
of disk images. In Debian you can find both <package>sleuthkit</package> (the
tools) and <package>autopsy</package> (the graphical front-end).

<p>Remember that forensics analysis should be done always on the
backup copy of the data, <em>never</em> on the data itself, in case
the data is altered during analysis and the evidence is lost.

<p>You will find more information on forensic analysis in 
Dan Farmer's and Wietse Venema's 
<url id="http://www.porcupine.org/forensics/forensic-discovery/" name="Forensic
Discovery"> book (available online), as well as in their
<url id="http://www.porcupine.org/forensics/column.html" name="Computer Forensics
Column"> and their <url
id="http://www.porcupine.org/forensics/handouts.html" name="Computer Forensic
Analysis Class handouts">. Brian Carrier's newsletter
<url id="http://www.sleuthkit.org/informer/index.php" name="The Sleuth Kit Informer">
is also a very good resource on forensic analysis tips. 
Finally, the <url id="http://www.honeynet.org/misc/chall.html" name="Honeynet
Challenges"> are an excellent way to hone your forensic analysis skills
as they include real attacks against honeypot systems and provide challenges
that vary from forensic analysis of disks to firewall logs and packet captures.

<p>FIXME: This paragraph will hopefully provide more information about
forensics in a Debian system in the coming future.

<p>FIXME: Talk on how to do a debsums on a stable system with the
MD5sums on CD and with the recovered file system restored on a
separate partition.

<p>FIXME: Add pointers to forensic analysis papers (like the Honeynet's
reverse challenge or <url id="http://staff.washington.edu/dittrich/"
name="David Dittrich's papers">).

<sect1>Analysis of malware
<p>Some other tools that can be used for forensic analysis provided
in the Debian distribution are:

<list>
<item><package>strace</package>.
<item><package>ltrace</package>.
</list>

<p>Any of these packages can be used to analyze rogue binaries (such
as back doors), in order to determine how they work and what they do
to the system. Some other common tools include <prgn>ldd</prgn> (in
<package>libc6</package>), <prgn>strings</prgn> and
<prgn>objdump</prgn> (both in <package>binutils</package>).

<p>If you try to do forensic analysis with back doors or suspected
binaries retrieved from compromised systems, you should do so in a
secure environment (for example in a <package>bochs</package> or
<package>xen</package> image or a
<prgn>chroot</prgn>'ed environment using a user with low
privileges<footnote>Be <em>very</em> careful if using chroots, since if the 
binary uses a kernel-level exploit to increase its privileges it might
still be able to infect your system</footnote>). Otherwise your own system can be back doored/r00ted too!

<p>If you are interested in malware analysis then you should read the <url
id="http://www.porcupine.org/forensics/forensic-discovery/chapter6.html"
name="Malware Analysis Basics"> chapter of Dan Farmer's and Wietse Venema's
forensics book.