I think my Linux Server has been hacked- Now what!?!?

[RobSki]

Hi,

I'm running a web server on Linux FC-5 with apache, php, and mysql.

I realized something was wrong when was trying to get vsftp to work on my server and found a phpinfo.php file in my html root directory. After reviewing some of the logs, i also made the following obervations: (It's possible that they may not have anything to do with the breakin.)

i found that the "operator" user belongs to the root group and "/root" is the home directory (is this normal?)
i found in my /var/log/audit/audit log (see attached file) that a user called "dave" tried to execute /usr/sbin/vsftpd (I don't have a user named "Dave" on my system)
i found in my /var/log/httpd/error_log the following notice: " [date/time] [notice] suEXEC mechanism enabled (wrapper:/usr/sbin/suexec)" (see attached file)

my network printing has stopped working
vsftpd fails to load at startup (I had it running ever since in installed the server without any error messages.)

Can someone tell me what hapened based on the information in the log files?
How can i pinpoint the exact medthod and time of the breakin?
What other files should i look at and where are they located?

I've already changed the root and operator passwords and disconnected my server from the network, but what else should do?

What intrusion detection tool works with Fedora core 5? I saw something about SNARE but I couldn't get it to work on my server

Can anyone recommend articles and/or links on how to lockdown a LAMP server?

[bigtomrodney]

Well for one sec I'll sidestep your questions - If you think your box has been compromised you really have no choice but to format and reinstall. You may have compromised binaries, rootkits, back doors or any other number of security breaches that may be undetectable to you. As extreme as that may seem, I thinkk a lot of others here will agree with me.

I generally check my /var directory for a few keywords. A search like

cd /var && grep -ir warning
grep -ir ssh
grep -ir attempt

Along with another few keywords usually are a good indicator. I don't use fedora so I won't comment on the operator user, or its config - but I would be concerned with the 'dave' user. I would bare in mind that it was an attempt. I find if I open port 22 on my router in a few hours I can have thousands of attempts at ssh bruteforcing. I would also look at chrootkit to look for any signs of rootkit exploits.

*nix boxes are more secure in general, but they are not invulnerable. If you have a reasonable suspicion that you are compromised I would go for that reinstall...

[RobSki]

Thanks for the reply. I had a feeling that the feedback would be to re-install the server.
I tried to keep security in mind when installing and configuring network services the first time, but I'm still learning LINUX. Do you have any advice on how I can pinpoint the method and time of the break-in? Also, what can i do to prevent this from happening again?
Regarding your statement: "I would also look at chrootkit to look for any signs of rootkit exploits." How do i do that?

[bigtomrodney]

http://www.chkrootkit.org/

That's the badboy you're looking for. The logs in /var are your best bet. They really hold a lot of information in there, but bare in mind - If I was a cracker I would be deleting you logs or at least editing them to remove traces so it's hit or miss as to whether you'll find anything. You might try searching your system for files modified since the time of the attack.

...Then again you can force a file to show a different modified time/date. Unfortunately the power of *nix can occasionally be pointed back at it. This is why a reinstall is usually recommended.

[RobSki]

So after restarting my server, I've found that the problem is worse than I thought. Although I can browse the filesystem from the Xwin desktop, I can't seem to open or edit any of the log files anymore. worse yet, i can't open a terminal window to see what's happening underneath.

I have a usb drive that i want to use to copy the files that have changed since the last backup, but my system no longer recognizes it. What should i do to save what I can before re-formatting and re-installing?

[RobSki]

Thanks for the link for chkrootkit. I took a quick look at the site and there seems to be a lot of good information there. As far as the logs go, I don't think the hacker was smart enough to do remove or edit them. I copied off about 20megs of logs that were generated over the last couple of days. (I just wish i had the time to track down this !*#&*!#)

[bigtomrodney]

If you can, the best thing is to use a live disc to access the system offline. That way you won't be letting any compromised elements do their work. You will be able to extract files you need to get, and the system shouldn't get any worse while it's offline.

[RobSki]

How do I access the damaged hard drive"offline?" Do I need to mount the hacked hard drive on another Linux system?

[bigtomrodney]

If you get a live disc like Knoppix, Ubuntu or Slax you can boot that, and they should automatically mount the drive and your USB stick for you. Easy as pie

[Thrillhouse]

What intrusion detection tool works with Fedora core 5?

I've installed Snort a few times on FC and a couple of times on RHEL. One of the best IDS's that just happens to be open source. Here is a manual for installing it (with a nice GUI) on FC5.

http://www.snort.org/docs/setup_guid...se_Minimal.pdf

While it won't lock the hacker out at all, you can at least see what he's up to. PM me if you have any questions about Snort. Good Luck.