Trying to figure out how server was rooted

Hi all,

One of my servers was rooted recently. I'm trying to figure out how, to prevent it happening again.

It's an OpenVZ virtual machine, running debian 5.0.4. Services running are:

postfix 2.5.5
vsftp 2.0.7
SSH-2.0-OpenSSH_5.1p1 Debian-5
Apache 2.2.9 with PHP/5.2.6-1 (SSL not enabled)

There's not much running on the website, other than phpmyadmin.

I discovered the compromise because some system binaries weren't working (ps, ls etc). Looks as if the attacker had installed a root kit with trojaned binaries, but they hadn't worked because it's a 64 bit machine.

After restoring these binaries, I was able to use 'find' to find all the rootkit files (by searching in uid and modified times). As well as the trojaned binaries, there was a backdoored SSH server and psybnc (irc proxy). None of these were running.

My guess is that the attacker wasn't too savvy and gave up when he found that none of this pre-compiled binaries would run on the system.

Two root-level accounts had been added to /etc/passwd. My guess is he added these when he couldn't get the rootkit SSH server to run. So not very sophisticated, plus he hadn't removed his logs from wtmp, so they were visible with 'last'.

My main worry is how he got in. If the Apache user had been compromised (say via phpmyadmin), I wouldn't be too surprised; but if he did come in via Apache, he had also found a way to gain root. Unfortunately many of the logs from the original incident have been rotated out. There was no trace of the www-data user having been the source though: no files owned by www-data in /tmp (or other locations), no sign of this user having logged in. Also, no sign of say a local root exploit script.

Having Googled around, there was rumour of a openssh exploit last summer, but can't find anything more recent. I've tried fetching the latest version of openssh, vsftp, apache via apt, but am already up to date.

well if they did log in via ssh AND didn't scrub the logs you should be able to see some activity via:

Code:

---

last

---

of course its only as good as your wtmp file is, if you don't rotate that file regularly it should have something. also check file MAC times to give you an idea of when this happened.

they may have cleaned up after themselves as far as the /tmp exploit. Since it probably phoned home.

Unfortunately when you get rooted, there really is no telling how far the damage goes. My suggestion is to salvage what data you got and scrub it. Just because it looks like a userland rootkit doesn't mean there are not traces of it left or that the kernel itself isn't compromised, in which case you are now merely a puppet in the attacker's reality. I would no longer be able to trust the system knowing it has been rooted.

1. I've no idea since I am more or less a noob at linux

There's something I don't get ...
You write:
His programs didn't work since you use a 64bit processor.

Why?! ... Most 64bit possessors are able to use all 32bit commands.
So there are two options ... you didn't mention that it's something like a 64bit sun-system Server ... or the hacker expected a special hardware.

I remember there was a security bug recently within php, something with the command too convert content into html code ... guess it was htmlspecialchars ... don't really remember the whole thing ... just read about it in any magazine ...

zomby: perhaps that wasn't the reason then. Maybe they'd been built against an older version of glibc. Either way, they were seg faulting when run.

I agree that a fresh install is the best option. Trouble is, this was a pretty much fresh installation. I'd apt-got a few packages, but that was it.

Looks like my best option is to hope it happens again, and collect some more evidence before the logs are rotated out (think I'll send the log events to another machine too)

Quote:

---

Originally Posted by kerm1t

Unfortunately many of the logs from the original incident have been rotated out.

---

Since most successful attacks are preceded by lots of reconnaissance that's a shame... Nothing in backups (you do make backups, right?) that can be used?

Quote:

---

Originally Posted by kerm1t

There was no trace of the www-data user having been the source though: no files owned by www-data in /tmp (or other locations), no sign of this user having logged in. Also, no sign of say a local root exploit script.

---

So what UID were the files you found by modification date owned by?


Wrt app versions:
Apache 2.2.9: CVE - CVE-2008-2939 (under review)
OpenSSH (4.3, 4.8, RH/F only): CVE - CVE-2009-2904 (under review)
Postfix: CVE - CVE-2009-2939 (under review)
phpMyAdmin: CVE - CVE-2009-4605 (under review)

Most likely attack vector still is a web stack compromise: phpMyAdmin and anything PHP-based running are likely culprits. So what do or did you run other than phpMyAdmin? Some bulletin board? A CMS?

If ssh is running on port 22, rest assured you will be attacked by every tool who can download a tool....
I left a server facing the net overnight once and found brute-force attacks from 3 different continents.

There are numerous exploits for PHPMyAdmin, you were probably cracked through that. When you are running free PHP scripts you have to keep up with security patches. One example of a crack for PHPMyAdmin can be found at milw0rm.com.

There are many others available if you do a google search for phpmyadmin exploit. I have noticed most script kiddies come in through CMSs and stuff right over port 80. If your logs haven't been modified by the hacker I would search for POST data in your access logs. This particular crack requires the hacker to know the absolute path of the vulnerable script. I'd start by grepping for that, and of course POST. You only want to look at 200s, 301, and 302s. A 200 is most likely the hack, 200 is the http code for a successful question. I am quite adept at grep, I do security at a hosting company. If you would like some help with the grep let me know.

Not sure I agree about 'most successful attacks are preceded by lots of reconnaissance". In my experience most of them are just kids with exploit scanners, scanning blocks for services which are vulnerable to whichever exploit they've downloaded.

Anyway, the uid of the trojaned system binaries was 122, so I ran 'find' for all files owned by that user. Since the machine has been pretty idle, I then ran a find for all files modified since the intrusion. Yes, he may have changed the timestamps on some of his files, but judging from the things he didn't do (clean the logs, left obvious footprints in /etc/passwd etc) I'm thinking probably not.

Its a custom PHP site, no well known stuff (other than phpmyadmin). If this was means of entry, I'm still worried as to how he managed to get root.

No backups of the logs - just the database and php code

Apart from the OP already telling us his logs are gone...

Quote:

---

Originally Posted by microlaser

I'd start by grepping for that, and of course POST. You only want to look at 200s, 301, and 302s. (..) If you would like some help with the grep let me know.

---

sure grep is cool but why not let automation do the hard work for you? Use Logwatch, maybe reconfigure it a bit, run it on the logs to weed out most noise and only show clues you're looking for.

Quote:

---

Originally Posted by kerm1t

Not sure I agree about 'most successful attacks are preceded by lots of reconnaissance".

---

If agreement is based on your extensive knowledge of server compromises then I'm probably ready to learn more from you.

Quote:

---

Originally Posted by kerm1t

the uid of the trojaned system binaries was 122

---

Anyway, being able to add an UID to the machine requires (delegated) root account rights and in absence of logs there's not much you're going to find to trace back the attack to any point of entry. So I can only suggest you cut your losses and move on, wipe the complete setup, reinstall a supported distribution version from scratch and properly harden the machine before exposing it to the 'net.