Compromised Linux servers

What is the best way to investigate a compromised Linux Server? I have downloaded Knoppix 6.2.1 on my laptop which is booting off my USB drive. What are the next steps I should take?

If the server is still online, try to get an overview of
- the running processes
- their open files
- and network connections.
However, do not rely on the installed tools to do so, as they may have been compromised.
Instead copy known good binaries and their libs to the server, and try to run these in a chroot. That wont help against injected kernel modules, but worth a try to get some data.
If succesfull, safe the data.


Then disconnect it before powering it down.
Start it from a CD/USBstick/etc and create an image of all disks with dd.
That requires a lot of space of course.

After that is done, loopmount the images readonly and start investigating.
The tools here The Sleuth Kit (TSK) & Autopsy: Open Source Digital Investigation Tools might help.

Irithori gives good advice.

1. Shut down systems affected
2. Disconnect from network
3. Boot from recovery or live CD/DVD
4. Make backups of all discs to external (clean) media.
5. Manually mount discs and scan for infections and root kits - there are a number of good tools (some free, some commercial) to do this.

Depending upon what you find, you may be able to clean the compromised data from the systems, or you may have to scrub them clean, reinstall/update the operating systems and programs, and restore your data (after cleaning any infected files). If you have recent known-good backups (without infections) of your system discs, that can help speed up the process.

BTW, what I mean by "scrub them clean" is this:

1. Zero out all discs, including boot sectors and partition tables.
2. Reinstall all BIOS's with factory-provided images.
3. Reset all onboard flash devices, often used to re-infect systems after cleaning.

Also, check your network routers and firewalls to make sure they have not been compromised as well, becoming a source of reinfection.