Question: Has My Box Been compromised?

Posted by ajt on Wed 20 Jul 2005 at 15:47

Today my Debian/Sarge box at home took a very sustained SSH attack. After the attack, I got an email saying that "rootkit004w" and "LKM" have been detected. I'm quite aware that the various automated security tools do generate false positives, however it's quite a coincidence. I had restarted some services that generate false positives inadvertently during the attack, so it really could just be coincidence

The box is set to allow only certificate based SSH logins only, so I'm quite confident that they could not have got in via a simple dictionary attack, but now I'm not convinced that the box is as safe as it should be.

I've taken the safe step of shutting the box down, as I don't trust it at the moment.

I'm quite prepared to rebuild the box from scratch, everything of importance is backed up, but I'd rather not if it's actually safe.

What's the best live-CD to check out a Debian system out with?

 

 


Posted by Steve (82.41.xx.xx) on Wed 20 Jul 2005 at 16:58
[ Send Message | View Steve's Scratchpad | View Weblogs ]

I think most of the LiveCDs are much the same for this purpose. Whilst I could be mistaken most of the LiveCDs which I've previously seen which are targetted at "security use" have been more focussed upon scanning remote machines via nessus, etc, rather than detecting security issues upon their host machine.

Whilst there are variation between the software packages included on various CDs mostly this will include software which you don't care about; browsers, office programs, etc.

If you believe that you have been compromised the obvious things to check are the system processes and the system logs - although the former might have been wiped, and the latter will be no good if you've rebooted.

Using a LiveCD the most important tool's will be the filesystem scanners, but if you don't have a known good collection of checksums/hashes they're not going to be terribly effective.

You might be able to detect "strangely named" files, which rootkit detectors frequently flag such as "...", or ".. " - but detecting new kernel modules, or changed binaries will be a little bit more challenging.

There are checksums you can use in /var/lib/dpkg/info/*.md5sums, however checking those will involve some simple scripting - and you might not trust those either.

Steve
-- Steve.org.uk

[ Parent | Reply to this comment ]

Posted by ajt (82.133.xx.xx) on Wed 20 Jul 2005 at 19:11
[ Send Message | View Weblogs ]

I'm downloading INSERT and Knoppix-STD as I type onto a clean machine, and I'll see what they turn up. I think the root-kit email is a false positive - I've seen it before. Chkrootkit said the box was okay, and I'll run that again from a clean boot.

I've had AIDE installed on the box, but when Sarge was "testing" the packages changed too often for it to practical. Now Sarge is stable, it's probably wise to put it back on again.

I think my sound SSH policy: root login disabled, and user login allowed by certificate only, should see the box okay, I'm just paranoid.

--
"It's Not Magic, It's Work"
Adam

[ Parent | Reply to this comment ]

Posted by ugob (216.113.xx.xx) on Wed 20 Jul 2005 at 20:09
[ Send Message ]
If root ssh login is disabled and only ssh keys access, not much chance of having a compromise through ssh login. Your packages are up to date?

One thing you could check is the headers of the e-mail you received... May give a lot of info... Like, if it comes from your server, it is bad news...

[ Parent | Reply to this comment ]

Posted by ajt (82.133.xx.xx) on Wed 20 Jul 2005 at 20:20
[ Send Message | View Weblogs ]
Packages are always kept upto date, I check daily.

What probably happened was my better half complained about the noise the PC was making. I logged in via SSH and decided to restart some of the demons in case one had got stuck - the box was busy (rejecting SSH logins apparently). One of the standard demons does give a false positive I know that, so it was probably just a fluke.

I downloaded a current INSERT livedisk and ran chkrootkit and rootkit hunter on the suspect mnachine from a clean boot and everything is peachy.

I'm just paranoid, I'm not yet confident enough a sysadmin to be comfortable, but I know too much to be naively confident...

--
"It's Not Magic, It's Work"
Adam

[ Parent | Reply to this comment ]

Posted by ugob (216.113.xx.xx) on Wed 20 Jul 2005 at 20:23
[ Send Message ]
The PC was making noise because it was rejecting ssh logins? I doubt so.


I'm just paranoid, I'm not yet confident enough a sysadmin to be comfortable, but I know too much to be naively confident...


That is ok. Better to be paranoid than sorry ;).

[ Parent | Reply to this comment ]

Posted by ajt (82.133.xx.xx) on Wed 20 Jul 2005 at 21:17
[ Send Message | View Weblogs ]
Well logging the attempts, it's a small box, with tiny old and noisy disks.

--
"It's Not Magic, It's Work"
Adam

[ Parent | Reply to this comment ]

Posted by Chris00 (193.190.xx.xx) on Thu 28 Jul 2005 at 12:35
[ Send Message ]
You can also restrict ssh access to certain machines by putting them in hosts.allow

sshd: my.machine.org : ALLOW

I also put at the bottom a line that disable all but specifically listed accesses (and keep a simple log of denied requests -- can be better done):

ALL: ALL: spawn (echo "[`date +%%c`] %c -> %d" >> /tmp/tcpd.DENY) & : DENY

[ Parent | Reply to this comment ]

Posted by Anonymous (208.29.xx.xx) on Wed 20 Jul 2005 at 19:22
You could also try using RootKit Hunter http://www.rootkit.nl. This software has its own database of known good checksums. It can also serve as a sanity check for false positives that you believe you may have.

It is not a debianised package, but is is simple to install.

[ Parent | Reply to this comment ]

Posted by ajt (204.193.xx.xx) on Wed 23 May 2007 at 14:08
[ Send Message | View Weblogs ]
RootKit Hunter has now been debianised and can be installed in the current stable (Etch), Testing (Lenny) and unstable (Sid) systems.

--
"It's Not Magic, It's Work"
Adam

[ Parent | Reply to this comment ]

Posted by Anonymous (216.111.xx.xx) on Thu 21 Jul 2005 at 19:00
http://denyhosts.sourceforge.net/ is a good program that will scan the login log and look for multiple failed logins from the same IP and add them to denyhost, that way the IP can never connect again and cuts down on traffic.

To test your system, Auditor is a good live CD just for that purpose and more upto date then Knoppix-STD.
http://new.remote-exploit.org/index.php/Auditor_main

[ Parent | Reply to this comment ]

Posted by Anonymous (213.164.xx.xx) on Fri 22 Jul 2005 at 09:21
> If you believe that you have been compromised the obvious
> things to check are the system processes and the system
> logs - although the former

But if he's been compromised, and the attacker is half-competent, the logs will have been altered or removed.

[ Parent | Reply to this comment ]

Posted by ugob (216.113.xx.xx) on Wed 20 Jul 2005 at 18:42
[ Send Message ]

[ Parent | Reply to this comment ]

Posted by ajt (82.133.xx.xx) on Wed 20 Jul 2005 at 20:05
[ Send Message | View Weblogs ]
Many thanks for the suggestions, downloading them as I type.

--
"It's Not Magic, It's Work"
Adam

[ Parent | Reply to this comment ]

Posted by Kellen (68.15.xx.xx) on Thu 21 Jul 2005 at 04:16
[ Send Message | View Weblogs ]
That's output from tiger?

I've had a box compromised by a weak password on a user account, then rooted with the suckit kit. I believe I got the same messages.

tiger doesn't seem to send the whole chkrootkit output, so run it independently.

[ Parent | Reply to this comment ]

Posted by ajt (204.193.xx.xx) on Thu 21 Jul 2005 at 07:50
[ Send Message | View Weblogs ]
Yes, Tiger is running on the box, and that was an email from Tiger/chkrootkit.

Running chkrootkit and rootkit hunter from a clean boot with the INSERT CD found nothing. I grovelled over the logs, and it looks like a basic dictionary attack - which should be ineffective as you can't login without a SSH key anyway.

--
"It's Not Magic, It's Work"
Adam

[ Parent | Reply to this comment ]

Posted by Anonymous (62.254.xx.xx) on Thu 21 Jul 2005 at 17:52
my small 2p, all these programs have their uses but if you dont trust the system then low level format the hd and reinstall everything, only way you can really be sure, or can you...

sno

[ Parent | Reply to this comment ]

Posted by peterhoeg (62.242.xx.xx) on Fri 29 Jul 2005 at 12:55
[ Send Message ]

Low level formatting is a very radical step to take - especially considering most (IDE) disks these days are no longer low-level formattable by the user.

A regular "formatting" (on linux this would be creation of a new filesystem) is sufficient, followed by full re-installation and restoration of data from backups.

/peter

[ Parent | Reply to this comment ]

Posted by Anonymous (207.161.xx.xx) on Wed 10 Aug 2005 at 03:19
Yes, low level formatting is a bad idea, and pretty much useless. If you're worried about residual data on the drive, you could always just wipe it. A quick "dd if=/dev/zero of=/dev/hdd bs=512" will zero out all the data on the drive. Even this, is mostly pointless. Clear out the MBR (dd if=/dev/zero of=/dev/hdd bs=512 count=1), recreate your partitions, and mkfs your new partitions, and then there's no way any sort of hostile code could regain control (aside from being compromised again).

Replace hdd in those dd commands with hda, sda, or whatever your hard drive device node is.

[ Parent | Reply to this comment ]

Posted by Kellen (132.239.xx.xx) on Fri 22 Jul 2005 at 18:18
[ Send Message | View Weblogs ]
One thing I'm not sure about is whether chkrootkit will detect a LKM if it's run from a livecd -- seems like not. So if there's no other sniffers or hidden stuff but there is a LKM installed chkrootkit might not give you anything unless it's run on the normal system.

Your box could have been compromised via some route other than ssh and you didn't specify what other public services you had running or if you had any suspect users.

That said, on my machine that got compromised, I cleaned out the rootkit and a sniffer, but I occasionally get the same message you're describing (from tiger, but not chkrootkit when run independently) and I've chalked it up to a false positive.

[ Parent | Reply to this comment ]

Posted by ajt (82.133.xx.xx) on Fri 22 Jul 2005 at 19:01
[ Send Message | View Weblogs ]
tiger/chkrootkit warns about a possible LKM, but when run directly on the machine or via the INSERT disk came up clean. I'm pretty sure it's just a false positive from one of the daemons, and nothing serious.

The only exposed services are SSH and HTTP/Apache2. The attack was a plain old SSH dictionary attack, which I'm confident was repulsed. I've now implemented Steve's iptable throttle suggestion, to reduce the load on the logging system.

--
"It's Not Magic, It's Work"
Adam

[ Parent | Reply to this comment ]

Sign In

Username:

Password:

[Register|Advanced]

 

Flattr

 

Current Poll

Which init system are you using in Debian?






( 1017 votes ~ 6 comments )