[BrianCarpio]

Ok,

I am investigating this problem for a friend of mine. His dedicated Linux server was experiencing high CPU load so I checked out the box and noticed that Perl was killing the server however none of his applications where running perl.

Anyway I killed the perl process and the server went back to 99% idle, then I started to check the apache logs to see what I could find.

All of a sudden the box started to come to a crawl and I noticed that the perl process was back so I ran lsof +p <pid_here> nothing was out of the ordinary but I noticed that when perl was using 99% CPU that it was spawning these processes [httpsd] which thought was odd as well since he isn't running any https websites on this server.

Anyway I killed the perl process again and was looking at the logs and then I saw this:

------------------------- start ---------------

[Sun Mar 16 05:10:04 2008] [notice] Digest: generating secret for digest authentication ...
[Sun Mar 16 05:10:04 2008] [notice] Digest: done
[Sun Mar 16 05:10:05 2008] [notice] Apache/2.2.6 (Unix) DAV/2 PHP/5.1.6 mod_ssl/2.2.6 OpenSSL/0.9.8b mod_perl/2.0.2 Perl/v5.8.8 configured --
resuming normal operations
error: "kern.ostype" is an unknown key
error: "kern.osrelease" is an unknown key
error: "kern.ostype" is an unknown key
error: "kern.osrelease" is an unknown key
error: "kern.ostype" is an unknown key
error: "kern.osrelease" is an unknown key
error: "kern.ostype" is an unknown key
error: "kern.osrelease" is an unknown key
error: "kern.ostype" is an unknown key
error: "kern.osrelease" is an unknown key
[Sun Mar 16 07:56:08 2008] [error] [client 127.0.0.1] Directory index forbidden by Options directive: /var/www/html/
[Sun Mar 16 07:56:08 2008] [error] [client 127.0.0.1] Directory index forbidden by Options directive: /var/www/html/
error: "kern.ostype" is an unknown key
error: "kern.osrelease" is an unknown key
sh: wgte: command not found
error: "kern.ostype" is an unknown key
error: "kern.osrelease" is an unknown key
error: "kern.ostype" is an unknown key
error: "kern.osrelease" is an unknown key
sh: wgte: command not found
error: "kern.ostype" is an unknown key
error: "kern.osrelease" is an unknown key
error: "kern.ostype" is an unknown key
error: "kern.osrelease" is an unknown key
sh: wgte: command not found
error: "kern.ostype" is an unknown key
error: "kern.osrelease" is an unknown key
--07:59:16-- http://rhama.110mb.com/pk.txt
Resolving rhama.110mb.com... 195.242.99.89
Connecting to rhama.110mb.com|195.242.99.89|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 84125 (82K) [text/plain]
Saving to: `pk.txt'

0K .......... .......... .......... .......... .......... 60% 35.3K 1s
50K .......... .......... .......... .. 100% 72.3M=1.4s

07:59:18 (57.9 KB/s) - `pk.txt' saved [84125/84125]


-------------------------- end ----------------

So I visited this site and checked out this pk.txt file which low and behold is an IRC bot.

I ran the find command for this file and found it here

/tmp/.ICE-unix/pk.txt

I removed the file and killed the perl process again.


I checked the /var/log/httpd/error_log and I can see that this pk.txt file was downloaded again and it owned by the user and group apache:apache.

I assume one of his websites is compromised and somehow apache is calling for this file to be downloaded every time I remove the file in about 5 minutes I can see from the error_log that its being downloaded again to the same exact spot.

So created an empty file here: /tmp/.ICE-unix/pk.txt and then chmod 000 chown root:root the file.

Now when I check the error_log I can see some process is attempting to download the file again but now I see this:

------- start ---------

error: "kern.ostype" is an unknown key
error: "kern.osrelease" is an unknown key
error: "kern.ostype" is an unknown key
error: "kern.osrelease" is an unknown key
error: "kern.ostype" is an unknown key
error: "kern.osrelease" is an unknown key
error: "kern.ostype" is an unknown key
error: "kern.osrelease" is an unknown key
error: "kern.ostype" is an unknown key
error: "kern.osrelease" is an unknown key
error: "kern.ostype" is an unknown key
error: "kern.osrelease" is an unknown key
Can't open perl script "pk.txt": Permission denied
error: "kern.ostype" is an unknown key
error: "kern.osrelease" is an unknown key
Can't open perl script "pk.txt": Permission denied
error: "kern.ostype" is an unknown key
error: "kern.osrelease" is an unknown key
Can't open perl script "pk.txt": Permission denied
error: "kern.ostype" is an unknown key
error: "kern.osrelease" is an unknown key
error: "kern.ostype" is an unknown key
error: "kern.osrelease" is an unknown key
Can't open perl script "pk.txt": Permission denied
error: "kern.ostype" is an unknown key
error: "kern.osrelease" is an unknown key
Can't open perl script "pk.txt": Permission denied

------- end ----------------

So as you can see I have temporarily resolved the issue, but I can't figure out how apache is being exploited to download this file any suggestions?


Thanks.

[dressed_in_black]

Hey, I have told the host that there is somebody using their service to partake in an exploit and they are investigating it currently.
As for sorting out the server; I would take a look removing apache completely (including ALL!! config files) then re-installing.
Check that everything is up to date as they will be the most secure.
Ie. is he on the latest version of his distro?
Is he using an up todate kernel?
apache?
etc

[BrianCarpio]

Thanks for sending the report for me, I was too busy investigating to think about that.

Anyway, I scanned his site for popular web exploits didn't find anything. I did however notice that he didn't have his awstats.pl secure, it was accessible to everyone in the world if they typed http://hisdomain.com/awstats/awstats.pl so I restricted access to that directory by IP address and user/pass in the httpd.conf.

I think that was the problem, I do know that awstats in the past has allowed for remote file uploads and execution it also happens to be the only perl application he is running (even though he said he didn't run any lol)

I did check the latest security advisories for awstats and apparently the latest version of awstats has resolved the vulnerability of remote file uploads however I still think awstats is the problem.

I also checked the md5 checksum for apache against Fedora's database of the same version and it all checks out. I have also checked that he is running the latest pre-compiled kernel available from Fedora.

I think I got it, but I will keep watching to see what happens. I removed my locked down version of

/tmp/.ICE-unix/pk.txt

So I can check if the file is uploaded again, I also threw a quick scrip together to check the box for the file every hour.

I guess we will see what happens.