Detecting hidden Linux processes

[TRaef06]

I work in security and we're seeing a new webserver attack that injects a process into a server that runs as the same user as Apache. Sysadmins can never find the program that's starting the process so I'm assuming that either the attacker is running the proc in memory only and then deleting the binary (is this possible?) or they're using something like Metasploit to inject directly into a process, use it's user to start another process.

I assume that it's a vulnerability with Apache, but I'd like to create a C program that lists all processes and shows all processes, show which ones are hidden, their start time and their parent process if forked, and the user the process is running under.

I'm not looking for anyone to write this for me, if you could just show me what libraries to use, examples of partial code, or any other ideas, gotchas, thoughts on my thinking pattern regarding Apache, etc.

If you think I should split this into more postings here in the forums, please tell me that as well. But keep in mind I'm not looking for specific answers, just guidance to my end goal - to see everything running on a 'nix box.

Thank you in advance.

[Cabhan]

All processes running on any decently recent Linux have an entry in the proc filesystem, located at /proc. This is a special directory maintained by the kernel that gives you information on every process.

Under /proc you will find a collection of files, and a collection of directories that are named with only a number. Each number is a PID, and the directory gives you information about the process. For your concern about a deleted binary, for instance, there is a symlink in the directory called "exe" that points to the executable from which the process was launched. If this link is broken, then presumably the binary has been deleted.

If you know that the process is running as the same user as the server, you can use "ps" to list all processes run by that user, find the questionable one, and use /proc to get more information on it.

Does this help at all?

[TRaef06]

That helps my understanding of the whole process.

However, I'm dealing with hackers (the cybercriminal type) so I need to be sure that I have a full list of processes. I've been reading about hidden processes and how some people compare the command line "ps" output to the files in the /proc/somenumber folders to detect any differences.

I'll need to list all of the hidden processes as well.

[Cabhan]

I guess I'm confused about what a "hidden process" is. Assuming that it is simply your process that has been hijacked, it must fork a new process in order to do anything (it could, for instance, use ptrace() to change the executed code), but this still requires a new process that will be detectable via ps (you would have two apaches instead of one).

The only way I can think of for a process to be hidden would be if the kernel was somehow compromised, and that is an attack that I have no experience with, and can't offer any advice for. But I can't imagine how the kernel could be compromised via a simple process hijacking, especially of an unprivileged user.

[TRaef06]

Well I guess if you're confused about what a hidden process is, then I must be way off base. I'm not being sarcastic - just honest.

I thought that Linux could have hidden processes. A hidden process being a process that doesn't show up under "ps" command.

Maybe I'm wrong. I thought I remember reading about a utility that would compare the output from a "ps" command to the listings in the /proc directory and things that were in /proc but not in the "ps" results was considered a hidden process. Maybe that's not possible?

That's why I was hoping to get some insight into how to do that same utility in C without using shell or exec functions.

I don't know Linux well enough to know if I'm on the right track or not.

[gerard4143]

The only hidden process, for Linux, that I've heard of is "free the fish" easter egg....Gerard4143

[TRaef06]

What about like, rootkits? Aren't they hidden processes? I'm not arguing, I'm just curious.

[gerard4143]

The thing is, if you can accesses(modify) the kernel then you can do anything you want. Its just a matter of checks and balances then, making sure you don't clobber the system in the act of hiding a process....Gerard4143

If this is an Apache problem, you really should check the Apache site for possible problems/solutions...

[Cabhan]

Linux is pretty good on making sure that you have all the facts, so I don't know of any way to have a hidden process under a non-compromised Linux.

Now, if someone was able to load a kernel module, that could theoretically redefine everything to hide certain processes. However, any well-written rootkit that would do this would also hide the process form /proc as well, I would expect.

In any event, by Googling for [linux hidden process], I came across a number of utilities that, among other things, compare the output of ps and /proc (they also ignore /proc and go straight to system calls, and some other techniques). Maybe check out some of these?

I admit that I don't know a huge amount about Linux security, so I could, of course, be entirely off base on this stuff. Maybe search / post in our security forum to see if anyone there knows more about hidden processes?