Question: Has My Box Been compromised?
Posted by ajt on Wed 20 Jul 2005 at 15:47
Today my Debian/Sarge box at home took a very sustained SSH attack. After the attack, I got an email saying that "rootkit004w" and "LKM" have been detected. I'm quite aware that the various automated security tools do generate false positives, however it's quite a coincidence. I had restarted some services that generate false positives inadvertently during the attack, so it really could just be coincidence
The box is set to allow only certificate based SSH logins only, so I'm quite confident that they could not have got in via a simple dictionary attack, but now I'm not convinced that the box is as safe as it should be.
I've taken the safe step of shutting the box down, as I don't trust it at the moment.
I'm quite prepared to rebuild the box from scratch, everything of importance is backed up, but I'd rather not if it's actually safe.
What's the best live-CD to check out a Debian system out with?
[ Send Message | View Steve's Scratchpad | View Weblogs ]
I think most of the LiveCDs are much the same for this purpose. Whilst I could be mistaken most of the LiveCDs which I've previously seen which are targetted at "security use" have been more focussed upon scanning remote machines via nessus, etc, rather than detecting security issues upon their host machine.
Whilst there are variation between the software packages included on various CDs mostly this will include software which you don't care about; browsers, office programs, etc.
If you believe that you have been compromised the obvious things to check are the system processes and the system logs - although the former might have been wiped, and the latter will be no good if you've rebooted.
Using a LiveCD the most important tool's will be the filesystem scanners, but if you don't have a known good collection of checksums/hashes they're not going to be terribly effective.
You might be able to detect "strangely named" files, which rootkit detectors frequently flag such as "...", or ".. " - but detecting new kernel modules, or changed binaries will be a little bit more challenging.
There are checksums you can use in /var/lib/dpkg/info/*.md5sums, however checking those will involve some simple scripting - and you might not trust those either.
Steve
-- Steve.org.uk
[ Parent | Reply to this comment ]
I'm downloading INSERT and Knoppix-STD as I type onto a clean machine, and I'll see what they turn up. I think the root-kit email is a false positive - I've seen it before. Chkrootkit said the box was okay, and I'll run that again from a clean boot.
I've had AIDE installed on the box, but when Sarge was "testing" the packages changed too often for it to practical. Now Sarge is stable, it's probably wise to put it back on again.
I think my sound SSH policy: root login disabled, and user login allowed by certificate only, should see the box okay, I'm just paranoid.
--
"It's Not Magic, It's Work"
Adam
[ Parent | Reply to this comment ]
One thing you could check is the headers of the e-mail you received... May give a lot of info... Like, if it comes from your server, it is bad news...
[ Parent | Reply to this comment ]
What probably happened was my better half complained about the noise the PC was making. I logged in via SSH and decided to restart some of the demons in case one had got stuck - the box was busy (rejecting SSH logins apparently). One of the standard demons does give a false positive I know that, so it was probably just a fluke.
I downloaded a current INSERT livedisk and ran chkrootkit and rootkit hunter on the suspect mnachine from a clean boot and everything is peachy.
I'm just paranoid, I'm not yet confident enough a sysadmin to be comfortable, but I know too much to be naively confident...
--
"It's Not Magic, It's Work"
Adam
[ Parent | Reply to this comment ]
I'm just paranoid, I'm not yet confident enough a sysadmin to be comfortable, but I know too much to be naively confident...
That is ok. Better to be paranoid than sorry ;).
[ Parent | Reply to this comment ]
--
"It's Not Magic, It's Work"
Adam
[ Parent | Reply to this comment ]
sshd: my.machine.org : ALLOW
I also put at the bottom a line that disable all but specifically listed accesses (and keep a simple log of denied requests -- can be better done):
ALL: ALL: spawn (echo "[`date +%%c`] %c -> %d" >> /tmp/tcpd.DENY) & : DENY
[ Parent | Reply to this comment ]
It is not a debianised package, but is is simple to install.
[ Parent | Reply to this comment ]
http://packages.debian.org/stable/misc/chkrootkit
Huh : at least for 'old' rootkit : http://packages.debian.org/changelogs/pool/main/c/chkrootkit/chkr ootkit_0.44-2/changelog
;-/
[ Parent | Reply to this comment ]
--
"It's Not Magic, It's Work"
Adam
[ Parent | Reply to this comment ]
To test your system, Auditor is a good live CD just for that purpose and more upto date then Knoppix-STD.
http://new.remote-exploit.org/index.php/Auditor_main
[ Parent | Reply to this comment ]
> things to check are the system processes and the system
> logs - although the former
But if he's been compromised, and the attacker is half-competent, the logs will have been altered or removed.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
--
"It's Not Magic, It's Work"
Adam
[ Parent | Reply to this comment ]
I've had a box compromised by a weak password on a user account, then rooted with the suckit kit. I believe I got the same messages.
tiger doesn't seem to send the whole chkrootkit output, so run it independently.
[ Parent | Reply to this comment ]
Running chkrootkit and rootkit hunter from a clean boot with the INSERT CD found nothing. I grovelled over the logs, and it looks like a basic dictionary attack - which should be ineffective as you can't login without a SSH key anyway.
--
"It's Not Magic, It's Work"
Adam
[ Parent | Reply to this comment ]
sno
[ Parent | Reply to this comment ]
Low level formatting is a very radical step to take - especially considering most (IDE) disks these days are no longer low-level formattable by the user.
A regular "formatting" (on linux this would be creation of a new filesystem) is sufficient, followed by full re-installation and restoration of data from backups.
/peter
[ Parent | Reply to this comment ]
Replace hdd in those dd commands with hda, sda, or whatever your hard drive device node is.
[ Parent | Reply to this comment ]
Your box could have been compromised via some route other than ssh and you didn't specify what other public services you had running or if you had any suspect users.
That said, on my machine that got compromised, I cleaned out the rootkit and a sniffer, but I occasionally get the same message you're describing (from tiger, but not chkrootkit when run independently) and I've chalked it up to a false positive.
[ Parent | Reply to this comment ]
The only exposed services are SSH and HTTP/Apache2. The attack was a plain old SSH dictionary attack, which I'm confident was repulsed. I've now implemented Steve's iptable throttle suggestion, to reduce the load on the logging system.
--
"It's Not Magic, It's Work"
Adam
[ Parent | Reply to this comment ]