Hacked - Linux Redhat

[BananaBob]

Hey folks,

I am running two Redhat Linux boxes. One sits outside my router and has an external IP address. This box is a Webserver and DNS server. Runs Apache and mysql.

The other box is a Mail Server and Webserver. This box sits behind a port forwarding router.

Both my servers have apparently been compromised. I started getting errors when attampting to start and stop certain services. In diagnosing this behavior I came across entries in my "root" history that shows someone gaining access to the root account and downloading and installing applications. These application appear to be geared towards attacking other machines. Slowly more and more of my system is getting crunched as now netstat no longer functions on this machine.

I tried locking down the machines by killing all unfamiliar services, checking all accounts for cron jobs, changing root passwords, and restricting what ports were being listened to on each machine. I thought this would at least keep the culprits out until I could determine the extent of the damage. But I have evidence that they got right back into the machine. Not sure how yet.

So my question is, All I want this Linux box for is to run my Mail Server with a Squirrelmail front-end, a webserver, and to handle DNS services for my domains. What should I do at this point? Are these boxes salvageable, or should I back up as much as I can and re-install?

If I reinstall, what dist should I use? I'm currently looking at Fedora, or Trustix.

Any help VERY appreciated.

Jz.

[retired1af]

About the only way you're going to be totally sure your boxes are OK is to take them both off line, reformat and reload the OS, making sure you have all patches and necessary security routines loaded.

[lakerdonald]

They might have installed a rootkit, just an FYI, so that might be what foiled your efforts. As to what to reinstall, I'd suggest Slackware, which is a very stable/secure distro right out of the box. If you're willing to spend some time working on locking things down, I'd also suggest Gentoo, which is what my server runs (FTP, Rsync, SSH and not a break-in yet! [Note: That's not an invitation to try ] ) but it can be a pain for some users new to Gentoo to configure.

[BananaBob]

That stinks. But is what I expected to hear.

Any suggestions for a Distro. Just am looking to run the services I mentioned above (mail server [7 Users - pop - imap - squirrelmail], web server [7 websites - mysql - php], DNS).

Don't really need much else. Maybe SAMBA and FTP on occasion.

Just asking in case there is a reason I should choose one over another. I download Fedora and Trustix last night. The Fedora iso file was 2.5 Gigs while the Trustix iso was only 414 MB. While I don't want that many extras, I want my machine to be easily upgradeable.

Anyway. Never been hacked before... You know what... It kinda sux.

Jz.

[lakerdonald]

They might have installed a rootkit, just an FYI, so that might be what foiled your efforts. As to what to reinstall, I'd suggest Slackware, which is a very stable/secure distro right out of the box. If you're willing to spend some time working on locking things down, I'd also suggest Gentoo, which is what my server runs (FTP, Rsync, SSH and not a break-in yet! [Note: That's not an invitation to try ] ) but it can be a pain for some users new to Gentoo to configure.

Like I said, Gentoo or Slack would be my suggestions.

[anomie]

Well, I have had good experiences running a SuSE server, but I think what is more important than the distro you choose is taking steps to harden, secure and monitor the machine. For every service you run and make available to the world there is a lot to learn about hardening it.

I will also echo what has been said: Take those compromised machines off the network, backup important data (and potential evidence including user names, IPs, etc.) and do not plug them back into the network until you have reinstalled and secured them.

[BananaBob]

Yes, you are right. I thought I had done a decent job of it. I had log files emailed to me every night (which is where I had my first clues something was wrong - no messages log to rotate is not a good thing). But I am a relatively inexperienced admin just trying to run some simple net services for personal use.

I guess my big question is whther something like ipchains is necessary, or whether just tightening down my services via xinetd, and restricting access via hosts.allow/deny, and placing the whole thing behind a port forwarding router is enough. I had anawful time trying to understand and enable ipchains and eventually gave up.

As far as Distro suggestions (I'm trying to determine if this is something I should be concerned with or not):
1) Gentoo - This seems like a lot more work than I'm interested in. Not doing this soo much for the hobby anymore as I am just interested in having the Webmail Server that I can control.
2) SuSE and Slackware are the other suggestions. Are these better options than Fedora? Just want the Distro that's going to require the least amount of work and upkeep after install. As I said above, not interested in the hobby of it anymore, just want it to stay secure and up2date with min my part. Yeah I know... Dreaming...

Jz.

[anomie]

iptables is probably what you mean (which is the successor to ipchains). The project page is here: http://www.netfilter.org/

Many distros include a frontend program or script that makes it easier to manage.

[psic]

You might want to look into Bastille:

www.bastille-linux.org

It can be used with quite a few distros.

[Monkeh]

I'd suggest slackware, it's small (single CD), easy to set up and maintain, and pretty tough. Most important thing here is to find out exactly how they got into your machine so they don't do it again.