<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<title>Finding Evidence of Your Cracker LG #36</title>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000" LINK="#0000FF" VLINK="#A000A0"
"Linux Gazette...<I>making Linux just a little more fun!</I>"
<P> <HR> <P>
<H1><font color="maroon">Through the Looking Glass: Finding Evidence of Your Cracker</font></H1>
<H4>By <a href="mailto:firstname.lastname@example.org">Chris Kuethe</a></H4>
<P> <HR> <P>
You've subscribed to Bugtraq and The Happy Hacker list, bought
yourself a copy of The Happy Hacker, and read The Cuckoo's Egg a
few times. It's been a very merry Christmas, with the arrival of a cable
modem and a load of cash for you, so you run out and go shopping to
start your own hacker lab. A week later, you notice that one of your
machines is being an especially slow slug and you've got no disk
space. Guess what - you got cracked, and now it's time to clean up the
mess. The only way to be sure you get it right is to restore from a
clean backup - usually install media and canonical source - but let's
see what the "h4x0r" left for us to study.
In late October of this year, we experienced a rash of attacks on some
workstations here at the University of Alberta's Department of
Mathematical Sciences. Many of our faculty machines run RedHat 5.1
(there's a good platform to learn how to try to secure...) since it's
cheap and easy to install. Workstations are often dual-boot with
Windows 95, but we'll be phasing that out as we get Citrix WinFrame
installed. This paper is an analysis of the compromise of one
One fine day I was informed that we'd just had another break-in, and
it was time for me to show my bosses some magic. But like a skilled
cardshark who's forced to use an unmarked deck, my advantage of
being at the console had been tainted. Our cracker had used a decent
rootkit and almost covered her tracks.
In general, a rootkit is a collection of utilities a cracker will install in
order to keep her root access. Things like versions of ps, ls, passwd,
sh, and other fairly essential utilities will be replaced with versions
containing back doors. In this way, the cracker can control how much
evidence she leaves behind. Ls gets replaced so that the cracker's files
don't show up, and ps is done so that her processes are not displayed
either. Commonly a cracker will leave a sniffer and a backdoor hidden
somwhere on your machine. Packet sniffers - programs that record
network traffic which can be configured to filter for login names and
passwords - are not part of a rootkit per se, but they are nearly as
loved by hackers as a buggered copy of ls. What wouldn't want to try
intercept other legitimate user passwords?
In nearly all cases, you can trust the copy of ls on the cracked box to
lie like a rug. Don't bet on finding any suspicious files with it, and
don't trust the filesizes or dates it reports; there's a reason why a
rootkit binary is generally bigger than the real one, but we'll get there
in a moment. In order to find anything interesting, you'll have to use
find. Find is a clever version of 'ls -RalF <w> | grep <x> | grep <y> |
... | grep <z> '. It has a powerful matching syntax to allow precise
specification of where to look and what to look for. I wasn't being
picky - anything whose name began with a dot was worth looking at.
The command: find / -name ".*" -ls
Sandwiched in the middle of a ton of useless temporary files and the
usual '.thingrc' files (settings like MS-DOS's .ini) we found
'/etc/rc.d/init.d/...'. Yes, with 3 dots. One dot by itself isn't suspicious,
nor are two. Play around with DOS for about two seconds and you'll
see why: '.' means "this directory" and '..' means "one directory up."
They exist in every directory and are necessary for the proper
operation of the file system. But '...' ? That has no special reason to
Well, it was getting late, and I was fried after a day of class and my
contacts were drying up, so I listed /etc/rc.d/init.d/ to check for this
object. Nada. Just the usual SysV / RH5.1 init files. To see who was
lying, changed my directory into /tmp/foo, the echoed the current date
into a file called '...' and tried ls on it. '...' was not found. I'd found
the first rootkit binary: a copy of ls written to not show the name '...' .
I will admit that find is another target to be compromised; in this case
it was still clean and gave me some useful information.
Now that we knew that '...' was not part of a canonical distribution, I
moved into to it and had a look. There were only two files; linsniffer
and tcp.log. I viewed tcp.log with more and made a list of the staff
who would get some unhappy news. Ps didn't show the sniffer
running, but ps should not be trusted in this case, so I had to check
We were running in tcsh, an enhanced C-syntax shell which supports
asychronous (background) job execution. I typed './linsniffer &'
which told tcsh to run the program called linsniffer in this directory,
and background it. Tcsh said that was job #1, with process ID 2640.
Time for another ps - and no linsniffer. Well, that wasn't too
shocking. Either ps was hacked or linsniffer changed its name to
something else. The kicker: 'ps 2640' reported that there were no
processes available. Good enough. Ps got cracked. This was the
second rootkit binary. Kill the currently running sniffer.
Now we check the obvious: /etc/passwd. There were no strange
entries and all the logins worked. That is, the passwords were
unchanged. In fact the only wierd thing was that the file had been
modified earlier in the day. An invocation of last showed us that
'bomb' had logged in for a short time around 235am. That time would
prove to be significant. Ain't nobody here but us chickens, and none
of us is called bomb...
I went and got my crack-detection disk - a locked floppy with binaries
I trust - and mounted the RedHat CD. I used my clean ls and found
that the real ls was about 28K, while the rootkit one was over 130K!
Would anyone like to explain to me what all those extra bytes are
supposed to be? The 'file' program has our answer: ELF 32-bit LSB
executable, Intel 80386, version 1, dynamically linked, not stripped.
Aha! So when she compiled it, our scriptkiddie forgot to strip the file.
That means that gcc left all its debugging info in the file. Indeed,
stripping the program brings it down to 36K, which is about
reasonable for the extra functionality (hiding certain files) that was
Remember how I mentioned that the increased filesize is important?
This is where we find out why. First, new "features" have been added.
Second, the binaries have verbose symbol tables, to aid debugging
without having to include full debug code. And third, many
scriptkiddies like to compile things with debugging enabled, thinking
that it'll give them more debug-mode backdoors. Certainly our 'kiddie
was naive enough to think so. Her copy of ls had a full symbol table,
and source and was compiled from /home/users/c/chlorine/fileutils-
3.13/ls.c - which is useful info. We can fetch canonical distributions
and compare those against what's installed to get another clue into
what she may have damaged.
I naively headed for the log files, which were, of course, nearly as
pure as the driven snow. In fact the only evidence of a crack they held
was a four day gap. Still, I did find out something useful: this box
seemed to have TCP wrappers installed. OK, those must have failed
somehow since she got in to our system. On RH51, the TCP wrappers
live in /usr/sbin/in.* so what's this in.sockd doing in /sbin? Being
Naughty, that's what. I munged in.sockd through strings, and found
some very interesting strings indeed. I quote: You are being logged ,
FUCK OFF , /bin/sh , Password: , backon . I doubt that this is part of
an official RedHat release.
I quickly checked the other TCP wrappers, and found that RedHat's
in.rshd is 11K, and the one on the HD was 200K. OK, 2 bogus
wrappers. It seems that, looking at the file dates, this cracked wrapper
came out the day after RH51 was released. Spooky, huh?
I noticed that these binaries, though dynamicically linked, used libc5,
not libc6 which we have. Sure, libc5 exists, but nothing, and I mean
nothing at all uses it. Pure background compatiblity code. After
checking the other suspect binaries, they too used libc5. Thats where
strings and grep (or a pager) gets used.
Now I'm getting bored of looking by hand, so lets narrow our search a
little using find. Try everything in October of this year... I doubt our
cracker was the patient sort - look at her mistakes so far - so she
probably didn't get on before the beginning of the month. I don't
claim to be a master of the find syntax, so I did this:
find / -xdev -ls | grep "Oct" | grep -v "19[0-7]" > octfiles.txt
In english: start from the root, and don't check on other drives, print
out all the file names. Pass this through a grep which filters everything
except for "Oct" and then another grep to filter out years that I don't
care about. Sure, the 80's produced some good music (Depeche
Mode) and good code (UN*X / BSD) but this is not the time to study
One of the files reported by the find was /sbin/in.sockd. Interestingly
enough, ps said that there was one unnamed process with a low (76)
process id owned by uid=0, gid=26904. That group is unknown on
campus here - whose is it? And how did this file get run so early so as
to get that low a PID? In.sockd has that uid/gid pair... funky. It has to
get called from the init scripts since this process appears on startup,
with a consistently low PID. Grepping the rc.sysinit file for in.sockd,
the last 2 lines of the file are this:
#Start Socket Deamon
Yeah, sure... That's not part of the normal install. And Deamon is
spelled wrong. Should a spellchecker be included as an crack-
detector? Well, RedHat isn't famous for poor docs and tons of typos,
but it is possible to add words to a dictionary. So our cracker tried to
install a backdoor and tried to disguise it by stuffing it in with some
related programs. This adds credibility to my theory that our cracker
has so far confined her skills to net searches for premade exploits.
The second daemon that was contaminated was rshd. About 10 times
as big as the standard copy, it can't be up to anything but trouble.
What does rsh mean here? RemoteSHell or RootShell? Your guess is
as good as mine.
So far what we've found are compromised versions of ls, ps, rshd,
in.sockd, and the party's just beginning. I suggest that once you're
finished reading this, you do a web search for rootkit and see how
many you can scrounge up and defeat. You have to know what to look
for in order to be able to remove it.
While the log files had been all but wiped clean, the console still had
some errors printed on it, quite a few after 0235h. One of these was a
refusal to serve root access to / via nfs at 0246h. That coincided
perfectly with the last access time to the NFS manpage. So our
scriptkiddie found something neat, and she tried to mount this
computer via NFS, but she didn't set it up properly. All crackers, I'd
say, make mistakes. If they did everything perfectly we'd never notice
them and there would be no problems. But it's the problems that arise
from their flaws that cause us any amount of grief. So read your
manuals. The more thorougly you know your system, the more likely
you are to notice abnormalities.
One of the useful things (for stopping a cracker) about NFS is that if
the server goes down, all the NFS clients with directories still
mounted will hang. You'll have to 120-cycle the machine to get it
back. Hmmm. This presents an interesting tool opportunity: write a
script to detect an NFS hack, and if a remote machine gets in, ifconfig
that interface off. Granted, that presents a possible denial-of-service if
authorized users get cut off. But it's useful if you don't want your
workstation getting compromised.
At this point I gave up. I learned what I'd set out to do - how to find
the crap left behind by a cracker. Since the owner of this system had
all her files on (removed) removable media there was no danger of
them being in any way compromised. The ~janedoe directory was
mounted off a Jaz disk which she took home at night, so I just dropped
the CD into her drive and reinstalled. This is why you always keep
user files on a separate partition, why you always keep backups and
why it's a good plan to write down where to get the sources for things
you downloaded, if you can't keep the original archives.
Now that we've accumulated enough evidence and we're merely
spirited sluggers pulverizing an equine cadaver, it's time to consider
the appropriate response. Similar to Meinel's you-can-get-punched
and you-can-go-to-jail warnings in The Happy Hacker, I would
suggest that a vicious retaliatory hack is not appropriate. In Canada,
the RCMP does actually have their collective head out of the sand. I
am not a lawyer, so don't do anything based on these words except
find a lawyer of your own. With that out of the way, suffice it to say
that we're big on property protection here. Aside from finding a
lawyer of your own, my advice here is for you to call the national
police, whoever they are. People like the RCMP, FBI, BKA, MI-5 and
KGB probably don't mind a friendly phone call, especially if you're
calling to see how you can become a better law-abiding citizen.
Chances are, you'll get some really good tips, or at least some handy
references. And of course you'll know someone who'll help you
My communication with RCMP's Commercial Crimes unit (that
includes theft of computing and/or network services) can be
summarized as follows: E-mail has no expectation of privacy. You
wish email was a secret, but wake up and realize that it's riskier than a
postcard. As systems administrator, you can do anything you want
with your computer - since it's your responsibility either because you
own it or because you are its appointed custodian - so long as you
warn the users first. So I can monitor each and every byte all of my
users send or receive, since they've been warned verbally,
electronically and in writing, of my intent to do so. My browse of the
FBI's website shows similar things. But that was only browsing.
Don't run afoul of provincial or state laws regulating the interception
of electronic communication either.
While I have attempted to make this reconstruction of events
as accurate as possible, there's always a chance I might have
misread a log entry, or have misinterpreted something. Further,
this article is solely my opinion, and should not be read as
the official position of my employer.
Appendix A: Programs you want in a crack-detection kit
<li>find, ps, ls, cp, rm, mv
<li>gdb, nm, strings, file, strip
<li>(GNU)tar, gzip, grep
<li>less / more
<li>vi / pico
<li>tcsh / bash / csh / sh
For security reasons these should all be statically linked.
<B>Appendix B: References</B>
About the filesystem:<BR>
McKusik, M.K., Joy, W.N., Leffler, S.J., Fabry, R.S.,
"A Fast File System for UNIX" Unix System Manager's Manual,
Computer Systems Reseach Group, Berkeley. SMM-14 April 1986
LEA and Computer Crime:<BR>
<P> <hr> <P>
<center><H5>Copyright © 1999, Chris Kuethe <BR>
Published in Issue 36 of <i>Linux Gazette</i>, January 1999</H5></center>
<P> <hr> <P>
<A HREF="./lg_toc36.html"><IMG ALIGN=BOTTOM SRC="../gx/indexnew.gif"
ALT="[ TABLE OF CONTENTS ]"></A>
<A HREF="../lg_frontpage.html"><IMG ALIGN=BOTTOM SRC="../gx/homenew.gif"
ALT="[ FRONT PAGE ]"></A>
<A HREF="./bennetjan.html"><IMG SRC="../gx/back2.gif"
ALT=" Back "></A>
<A HREF="./lussier.html"><IMG SRC="../gx/fwd.gif" ALT=" Next "></A>
<P> <hr> <P>