Probably got hacked
I have a machine running CentOS 5.3.
There are many bad things happening on it, and I hope that someone can suggest me a solution.
1) I have noticed many strange things in the Apache error log, like someone downloading strange tgz files like "MechBot.tgz", "Stealth.tgz", "cmd.tgz", or attempts to do commands like "cat" "chmod" "mkdir" or "rm".
2) I also noticed many strange files and directories inside the "tmp" and "var/tmp" directories owned by the apache user, with names like "." or ".img" or "config93DKDJ", and also some scripts with names like "mech" or "juno".
3) There was a cron job created by the apache user pointing to /tmp/.img/update that ran every minute, and I've deleted it.
4) Furthermore I've seen that there are continuous outgoing connections on ports like 6667 6669 or high ports like 54377.
So I really think that someone is trying to hack my system (and maybe he partially managed to do).
I think it all began after I have deleted the access and error log files of a website because they were becoming very big, even if I don't know how this could have caused the attack. I use Awstats, I don't know if it could be related.
How can I safely delete all those bad files in the tmp directory and stop the attack?
I know about tools like Chkrootkit and RKhunter. Are they useful? Is it better to run them on a live cd? And how can I do it?
hi and welcome
Given your description, this server is compromised.
Imho there is no alternative to a complete reinstall, because you cannot tell for sure what was modified.
- Detach the machine from the network
- Save your data, db dumps, config and logs
- Of course, you need to find out, how the hack happened and close the security holes.
- verify all data.
- change all passwords, including system, the websites, db grants, ssh keys, certificates
Then start from scratch with an up to date system, that means centos 6.3, not 5.3
I agree with everything but this. Generally speaking, RHEL (and therefor, CentOS), patch security wholes and bugs until EOL.
Originally Posted by Irithori
That said, keeping the system up to date for the latest security patches is an absolute priority for system administrators.
You were likely compromised by out-dated web content, such as an old wordpress installation, or something of that nature. CentOS 5.x was not the problem, unless you failed to keep up with the latest security patches.
Since it bares repeating, you should 100% re-install the system from uninfected backups, changes passwords, etc.
You are right, centos 5 is still supported and updating to 6 is not strictly neccessary.
But rhes 5 is a) five years old and b) nearing the "End of Production 1" phase.
This happens in Q4 2012 and then redhat will slowly desert this version.
Rebuilding the machine manually is a major task anyway.
So why not use that effort to migrate to version 6, which is the current focus of redhat.
Some people have custom apps, etc, that for whatever reason just won't run on version 6. While I think upgrading distros is a generally good idea, while in the midst of restoring from backups to get your environment back online is not the best time to try to migrate. Ideally, a second system (or VM, etc) should be stood up and thoroughly tested prior to going live.
Originally Posted by Irithori
Well, at my workplace these custom apps would be considered long overdue for migration.
Especially for internet facing servers.
But ok, whether or not staying with 5 or migrating to 6 is a good idea does depend on a few mainly resource related contraints.
In the end robycentos needs to decide this.
I think it's a cinch that the attacker has compromised Apache or some child if it, because all the questionable files are owned by Apache. No doubt he's not working to escalate privs. Probably outdated software with vulns.