"Linux Gazette...making Linux just a little more fun!"

It started out pretty simple. I was on a my client's webserver, simply paging through routine entries, when I came on the following entries (IP addresses have been changed to protect the guilty):

May  2 23:25:09 bankweb ps[12613]: connect from 128.128.128.128
May  2 23:25:09 bankweb netstat[12614]: connect from 128.128.128.128
May  2 23:25:10 bankweb wu.ftpd[12616]: connect from 128.128.128.128
May  2 23:25:10 bankweb in.telnetd[12617]: connect from 128.128.128.128
May  2 23:25:15 bankweb in.fingerd[12619]: connect from 128.128.128.128
May  2 23:25:16 bankweb in.pop3d[12620]: connect from 128.128.128.128
May  2 23:25:17 bankweb in.nntpd[12622]: connect from 128.128.128.128
May  2 23:25:17 bankweb nntpd[12622]: 128.128.128.128 connect
May  2 23:25:17 bankweb nntpd[12622]: 128.128.128.128 refused connection
May  2 23:26:55 bankweb wu.ftpd[12624]: connect from 128.128.128.128
May  2 23:28:03 bankweb ftpd[12624]: FTP session closed
May  2 23:28:19 bankweb in.telnetd[12632]: connect from 128.128.128.128
May  2 23:28:44 bankweb login: 2 LOGIN FAILURES FROM 128.128.128.128, guest
May  2 23:29:12 bankweb ps[12634]: connect from 128.128.128.128
May  2 23:31:20 bankweb ps[12637]: connect from 128.128.128.128
May  2 23:32:25 bankweb netstat[12638]: connect from 128.128.128.128
May  2 23:34:21 bankweb in.fingerd[12641]: connect from 128.128.128.128
May  2 23:35:54 bankweb in.rlogind[12644]: connect from 128.128.128.128
May  2 23:35:54 bankweb in.rshd[12645]: connect from 128.128.128.128
May  2 23:35:54 bankweb rshd[12645]: Connection from 128.128.128.128 on illegal port
May  2 23:36:56 bankweb in.telnetd[12647]: connect from 128.128.128.128
May  2 23:37:11 bankweb login: 2 LOGIN FAILURES FROM 128.128.128.128, root

Uh-oh. For a couple of minutes, someone was definitely rattling my client's locks, looking for a way in. As a system administrator, what do you do now?? How do you find out if the intruder managed to actually get in? Did he *do* anything to your system? How do you make sure he doesn't get back in?

In this article, I'll make a few suggestions about a few basic steps that can be taken and some of the specialized tools that can help keep your system secure.

The first thing you want to check for is the possibility that the intruder is still logged on. A quick way to check this is 'w' or 'who' commands--look for someone logged from a remote machine. The thing to remember about these commands is that they work by reading a file ('utmp', typically found in /var/adm) that keeps track of who is logged in. If the intruder has broken into the root account, then he can change that file to make it look like he's not there.

Two good ways of finding such phantom users are to use the ps and netstat programs. Since these query kernel data structures rather than files, they are harder to spoof. Using ps, look for shells and programs that aren't associated with a legitimate user. Netstat is a lesser-used utility used to display the network status. If it is not in the normal system directories, look in /sbin or /usr/sbin. By default netstat displays active Internet connections. Again, look for connections to suspicious sites.

If he's on your system

In the worst case, the intruder is lurking around in your system. If the intruder has managed to break into the root account, he will be able to remove all your files with a quick rm -rf /. Fortunately, such toe-to-toe combat with intruders is rare.

The best solution to an intruder on your system is to immediately disconnect the Ethernet cable. Without giving him any warning, this puts a stop to whatever he is doing and isolates your computer, preventing further damage. Furthermore, it will appear to him that the network has failed-- which is in fact what has happened.

Unfortunately, you (or no one you can contact) may not have physical access to the machine when this happens. The second best thing to do about an intruder involves a judgment call. You can A) let him alone and hope he doesn't destroy the system and assess the damage later, B) talk to him using 'talk' or C) kick him off and hope he can't get back in.

If you decide to kick him off, you of course need to be root. Simply rebooting the system isn't a good idea, since the system will come back up and the intruder will probably be able to re-enter and will know that someone is onto him. The usual way of kicking someone off is to run a kill -9 command on their telnetd or rshd processes. These processes act as glue connecting the network to the intruder's shell. An equally valid method is to kill their shells. Either way, the intruder will see the message "Connection closed by foreign host" and will know that something is up.

The right way to do this is to remember that kill -9 will accept multiple process ids, and you want to blindside him. After you've used 'ps' to find *all* of his process ids, include the process id of the 'inetd' process. Inetd is sometimes referred to as the "Internet super-server"-- all it does is watch for incoming network connections and makes sure they are connected to the right handler. By killing inetd, you prevent new connections from being accepted, be they telnet, ftp, finger or whatever. Of course, if he's root, he can do this to you.

Assessing the damage

The real danger posed by an intruder is that once in, he can make it easy for himself to get back in, though you may close the hole he originally came through. The way this has to be done is by modifying the filesystem in some manner (with Linux, he could easily compile you a new kernel, but the kernel is ultimately stored in the filesystem).

The freeware program tripwire is used to detect modified system files. The idea is that tripwire records the size, date and a quantity called a "one way hash function". The idea is to take the data of the file and compute a fairly large random number. If the file changes, a different hash value results. The "one way" part means that it is "difficult" to make a small change to file and still come up with the same hash value. Of course, if the database of hash values is stored on the hard disk and the cracker finds it, he can just update the database... which is why you want to keep the database on a floppy.

The find program is extremely useful for finding suspicious looking files that the intruder has left laying around. Use find to look for recently-modified files in the /lib, /usr and /etc hierarchies, keeping in mind that it is possible to change the timestamps.

An easy situation occurs when you have installed a system via CDROM. Since the CDROM cannot be modified, you can compare what is on the CD with what is on the hard disk. Something like:

find /bin -exec cmp {} /CDROM/dist/{}

Another thing to check for is:

find / -perm +6000 -print

Note that the -perm option is specific to GNU find-- other systems may have different syntax. What you're looking for are suspicious files. A great way to learn Unix is to simply go through the system files and figure out what each one does.

Perhaps the best way to sleep easier at night is to simply reinstall the operating system and all of the utilities after a breakin. This operation is much easier under Linux than most other Unixes and goes a long way toward giving you peace of mind about any time-bombs left behind. Besides, you were meaning to upgrade anyway, weren't you?

War Story #1

By default, most Linux systems come with tcp wrappers automatically installed. This program intercepts the initial service requests from remote machines and logs them in the system logs. The wrappers can be configured to reject or allow access from listed sites.

In the attempted bankweb breakin, the wrappers let me know that there had been an attempt in the first place. From the listings, you can see that by default several services were enable during the installation that really shouldn't be running. The ps service let the intruder see processes running on our machine-- and gain account names. netstat let him see the machine's active network connections. The first step was to disable those two "services" by commenting out their lines in /etc/inetd.conf and resetting inetd.

Step number two was to track the cracker back to his network provider. Fingering and telnetting to the IP number produced a refused connection, implying that the machine in question was probably not a Unix machine. Telnetting to port 137, the windows name service port was accepted, implying that the machine was a windows box. It was quite possible that the machine I was looking at was not the intruder's machine-- if the intruder was dialed in via ppp, then the IP number could have simply been reassigned to the machine that I was probing.

A lookup using whois with the first three parts of the IP number produced the provider's name but not an email address to send a complaint to. Using the traceroute program gave some intermediate addresses that I used to find the intruder's provider. The next-to-last address in the route to the intruder refused connections, but trying to telnet into the second-to-last produced a shocking result-- the address turned out to be a completely unprotected router. Not only were current network statistics displayed and updated, but "Configure" was a prominent menu item...

War Story #2

I logged in to a workstation cluster at school late one Saturday night to check the progress of one of my jobs. I was quite surprised to find 'root' logged in running a couple of shells and a chess demo from the local X-windows console. I chose to leave the intruder alone since I was unable to do much-- the recent installation of NIS had been botched and left me unable to change the cluster's root password.

I called the police after he shut down the system, since we've had a few computer thefts. I did have to do some fancy explaining to the dispatcher on why I thought a theft was taking place from several miles away. It turned out that the intruder was an idiot who didn't know the difference between shutting the system down and logging out.

He had acquired the password by watching one of the faculty miss the "enter" key while logging in as root-- the password was echoed right after "root". Coincidentally, I had acquired the cluster's root password in the same way, only I found it by seeing the log entry login failed for rootrs314m. The moral is change the damn password if someone sees it, or if it has accidentally gone into the system logs.

War Story #3

One day, on a machine I used as well as administered, I received a very strange letter that had been originally addressed to root-- I had forwarded root's mail to myself. The letter appeared to be (and in fact was) a command that was supposed to mail our password file to an address at an ivy-league university on the east coast. Old versions of the 'sendmail' program had a mode that allowed commands to be sent in letters to facilitate debugging. When the program was distributed, this "feature" was not disabled. Fortunately, the vendor for the workstation (not a Linux box) had closed that hole.

The next step was to contact the source of the attacks. I have found that the proper attitude is to be polite, and inform the administrators that you are having a problem with one of their users, then show them everything you have... and hope that the person you've contacted isn't the one who is launching the attacks. The address turned out to be on a completely insecure mail server, ending the hunt, but we at least made the right people aware of the problem.

That machine suffered several additional attacks over the next couple of months. The reason was that one of my users, who happened to be Russian, had a bunch of less-than-reputable "friends" back home who wanted impress him by breaking into his machine. At a group meeting, I mentioned these attacks, and half-suggested we all kick in twenty dollars, send the total to Moscow, and have a few legs broken. The other Russian, in our group, a very mild-mannered man, said "Break their legs?? Break their heads!". Watch out for those Russians...

War Story #4

I received an email notifying me that the machine in War Story #3 was being used as a base for attacking other machines. I forwarded root's mail to myself by putting my email in a file named .forward in root's home directory. If you administer a workstation, you want to do something like this, because the root account is typically rarely used and you want to know about this sort of thing the moment it happens.

As it turned out, the people complaining had waited too long for us to figure out who had been on the machine when the attacks took place. The logs on that machine were rotated every two weeks. Since the prime suspect had graduated, we chose to close his account along with all the other accounts that had never been deleted. Examining the suspect's files, we did find tools for breaking in to a variety of systems as well as a utmp editor for hiding his tracks. The root password was changed at the same time.

So, in conclusion, if you find out that *your* machine is under attack, stay calm, do it quick, do it first and keep your backups handy.

Further reading:

Check out Bill Cheswick's classic "Evening with Berferd" paper. ftp://cert.org/pub/papers/Bill_Cheswick_berferd.ps Andy Vaught andy@maxwell.la.asu.edu