Been hacked - need help...

[Angersum]

I've been running this box for about 3 years now with no issues....until about a month ago. I am running CentOS on a VM hosted by a web company. I administrate the box by myself.

It hosts a forum website (SMF).

The website started going down hard and was completely unresponsive 3-4 times a day. The only solution was to have the host hard boot it. The web hosting company just kept telling me it was running out of memory and I needed to pay to upgrade to a larger package which had more memory. I refused to accept this. My site isnt that big and no changes had been made. Why had it been running ok for 3 years and now its a problem?

In the secure log there was evidence of someone brute forcing the SSH port. I installed DenyHosts and that took care of that problem, but the site still went down regularly.

Then they (the VM host) found a script in the /tmp directory called 'back'. It appears to be a mass mailing script of some sort. i removed it and CHMOD'd the dir. Also, i disabled SMTP because I do not use it.

Now, when I do a netstat, I notice the following connection:

tcp 0 0 my.server.com:34286 61.194.4.65:62224 ESTABLISHED
(obviously edited to protect my server)

I traced the foreign address to somewhere in Japan. I had seen this same type of connection last week and didnt like it so I added that ip to be blocked by iptables. Now its back.

So can I find out what this IP is connecting to on my server? or can I only just see that its connected?

Nothing is detected by rkhunter and chkrootkit

what else can I do?

I assume I got hacked by a vulnerability in SMF as I was running an old version. I upgraded it today.

[scathefire]

are they connected in via ssh? are your last logs still intact (run last from the terminal)? in your ssh logs does it show them logging in? what other ports besides ssh are they hitting?

[Angersum]

are they connected in via ssh? are your last logs still intact (run last from the terminal)? in your ssh logs does it show them logging in?

nope, they arent connecting via ssh. they never successfully logged in via ssh during the BFA's.

[scathefire]

so what makes you think you got hacked? is your web site defaced? is your database hosed? or is it that file you are talking about? did you get rid of that file?

[Angersum]

so what makes you think you got hacked? is your web site defaced? is your database hosed? or is it that file you are talking about? did you get rid of that file?

nothing defaced. just BFA's and that file (I tried to post a link to a site with more info on that exploit, but i cant until i have a post count of 15 - google "/tmp/back" )

what concerns me is this connection from japan on my box. TheMy site's demographics is nitched to a very small geographic part of the US. I dont have any Asain users on my site. This connection is maintained for long periods of time on odd ports (currently 34286)

Also, in a deep directory structure, I found a few gigs of movies someone uploaded. The upload dates were back in dec '09 and my bandwidth usage for that month reflects this. I was concerned that the above mentioned port was maybe a torrent or something, but my daily bandwidth has been normal for the past few weeks.

is there anything i can check? any way i can trace to see exactly what that connection is from japan?

[scathefire]

Actually your server might be connected to them. So if you do:

netstat -pantu

It should give a process ID, which you can find out more with:

ps -aux | grep rogue_process_id

I'd be willing to bet you got some weak file upload permissions, which is probably how all this came about. Probably a bit of perl got uploaded and built the files, or maybe phoned home to the them.

[Angersum]

netstat gives this:

tcp 0 0 xxx.xxx.xxx.xxx:34286 61.194.4.65:62224 ESTABLISHED 11447/httpd

Then the ps gives this:

# ps -aux | grep 11447
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.3/FAQ
nobody 11447 6.3 1.1 22724 11996 ? S 03:00 45:20 /usr/local/apache/bin/httpd -k start -DSSL
root 16382 0.0 0.0 1596 476 pts/0 S+ 14:54 0:00 grep 11447

[scathefire]

you may check your apache configs, make sure there are no weird Includes inside your config that would cause there program to run. Have there been any new users created on the server? Maybe where user nobody (which is what apache is loaded as: nobody) would be able to log in to maybe?

Have you try to connect to the port it claims your web server is running on? Are there any more httpd services running?

[coopstah13]

did you install your own version of httpd? because usually only custom installed software outside of package manager goes in /usr/local

I would kill that process, because that shouldn't be the web server hosting your site. you can confirm this by running

ps -eaf | grep httpd

[Angersum]

did you install your own version of httpd? because usually only custom installed software outside of package manager goes in /usr/local

I would kill that process, because that shouldn't be the web server hosting your site. you can confirm this by running

ps -eaf | grep httpd

that returns this...

root@server [~]# ps -eaf | grep httpd
root 22335 1 0 Jan31 ? 00:00:12 /usr/local/apache/bin/httpd -k start -DSSL
nobody 11447 22335 5 03:00 ? 00:45:20 /usr/local/apache/bin/httpd -k start -DSSL
root 7774 22335 0 13:24 ? 00:00:00 /usr/local/apache/bin/httpd -k start -DSSL
nobody 14114 22335 1 15:25 ? 00:00:10 /usr/local/apache/bin/httpd -k start -DSSL
nobody 14178 22335 0 15:25 ? 00:00:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody 20019 22335 0 15:29 ? 00:00:03 /usr/local/apache/bin/httpd -k start -DSSL
nobody 21989 22335 0 15:30 ? 00:00:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody 28031 22335 0 15:33 ? 00:00:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody 30344 22335 0 15:34 ? 00:00:00 /usr/local/apache/bin/httpd -k start -DSSL
nobody 30363 22335 0 15:34 ? 00:00:00 /usr/local/apache/bin/httpd -k start -DSSL
nobody 30389 22335 0 15:34 ? 00:00:00 /usr/local/apache/bin/httpd -k start -DSSL
nobody 1425 22335 0 15:36 ? 00:00:00 /usr/local/apache/bin/httpd -k start -DSSL
nobody 1609 22335 0 15:36 ? 00:00:00 /usr/local/apache/bin/httpd -k start -DSSL
root 2017 15695 0 15:37 pts/0 00:00:00 grep httpd