Very strange auth logs, possibly hacked?

[genlee]

Ok when I woke up today, one the servers I admin was down. So I called and had the datacenter reboot it. Anyway after looking through the logs for the possible cause, I came across this:

May 18 02:12:44 vortex sshd[1833]: SELECT password FROM users WHERE username='root'
May 18 02:12:44 vortex sshd[1833]: ***DEBUG: MySQL:$1$FPuHiYlG$f6ObPojLF57ZtxOgAMlg2. given:$1$FPuHiYlG$mcOJz8odgTyJTTA.vu97T1
May 18 02:12:44 vortex sshd[1833]: returning 7 .
May 18 02:12:44 vortex sshd[1833]: returning 7 after db_checkpasswd.
May 18 02:12:44 vortex sshd(pam_unix)[1833]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=62.231.98.116 user=root
May 18 02:12:44 vortex sshd[1833]: pam_mysql: acct_mgmt called but not implemented. Dont panic though
May 18 02:12:44 vortex sshd[1833]: Accepted password for root from 62.231.98.116 port 1881 ssh2
May 18 02:12:45 vortex sshd(pam_unix)[1833]: session opened for user root by (uid=0)

Now I am using pam_mysql, and the person didn't enter the correct root pw but it still logged them in. I can't figure out why, plus they also show up in wmtp log as well. Has anyone seen that before, and yes sshd is up to date.

[Dolda2000]

I find it strange that it is pam_unix that logged the authentication failure event, and not pam_mysql, don't you agree? Are your sshd's PAM file correct? Or does the MySQL authentication work via pam_unix with an nsswitch module?
Couldn't this be a valid login, just that whoever logged in accidently mistyped the password the first time. Of course, the Failed password line from sshd is missing.
It does of course look a bit like you were cracked, but it seems so strange that sshd could be cracked so seemingly easily. Did whoever did this leave some trace in root's .bash_history?

[genlee]

I have auth set to use pam_unix as well as pam_mysql incase the database were to ever go down. Would make it a little difficult to login if that would happen. But I agree that pam_mysql should of logged the failure unless it attempted to auth by mysql and then pam_unix and pam_unix logged the attempt since that was the last method. I looked throough .bash_history and nothing was in there. We log all commands which are run by root to syslog and that was empty as well. I ran chkrootkit and that didn't find anything. I also ran netstat -antwup to see if there was any backdoors listening and that came up clear as well. This person was also able to log into our reseller account as well. Also the connection appeared to be from an internet cafe in romania.

[Dolda2000]

An internet cafe in romania, you say? Now that is a bit suspicious, I guess. Did you contact them? Maybe they keep logs about who use the machines.
It's a bit strange that he didn't leave any traces whatsoever, though, isn't it? He should at least have run exit, right? You haven't considered logging root commands to remote system as well? Can't you run a search of all files that have been changed around that time?
Btw., you haven't considered public key authentication for SSH?

[genlee]

Might of not been someone actually at the cafe, the ip address that the person connected from has a proxy running on it which is probably comprimised. Yes we are switching to public keys now for ssh. I already used them for my user but the other users on the box didn't know how to use them. Well they are going to learn quickly now. I also ran find to get a list of files accessed in the last 3 days and nothing useful came up. I just can't figure out why it let them login with an incorrect root pw. Even the md5 on sshd was correct. None of this is adding up at all.

[Dolda2000]

Indeed, it seems exceedingly strange. Maybe there's some bug in pam_mysql?

[andutt]

Are you allowing direct root access through ssh?? And how was it possible to comprimize the system by getting a mysql-password from a mysqldatabase??

Regards

[genlee]

Yes you can login through ssh with root. They didn't get the pass from the database. Even if they did get the pass from the db, it would be useless since we use md5. Ssh logged them in after it said incorrect password. I have no been able to replicate that at all.

[vous]

Wooow...I'm learning how to set up a firewall at the moment, and then I saw this thread!!! Scary stuff people....!

How can you find out where the hacker came from??? How did you figure that out?

[genlee]

wtmp log keeps track of everyone who logged in and from what ip address. You can access that log from the command "last"