Find the answer to your Linux question:
Page 1 of 2 1 2 LastLast
Results 1 to 10 of 11
Hi, I have a CentOS 5 server running OpenVZ that has been used in an attack for the second time in the last month. Today is the second time the ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Apr 2012
    Posts
    5

    Angry DDOS from my server, 2nd time


    Hi,
    I have a CentOS 5 server running OpenVZ that has been used in an attack for the second time in the last month.
    Today is the second time the provider has shut down the connection and I'm not sure where to look anymore.

    Both attacks originate from the same virtual machine. It hosts a blog with about 3k unique visitors per day; not huge, but large enough that the logs get big. The blog is using wordpress and the web server is lighttpd.
    In the first case, that VM was issuing a lot of 8k packets to the target through UDP port 80; the second attack is still UDP but on various ports instead.
    I found that the CPU usage will be maxed in the PHP-CGI, so they are somehow executing a PHP file.
    On the host, the contrack counter is to the max.

    After the first attack, here is what I did:
    - Update all files (CentOS + Wordpress, etc)
    - Close outbound UDP port 80, just because it couldn't hurt
    - Do a diff of wordpress against the original files, they all matched but the ones I modified manually.
    - there was a vulnerability on timthumb where it could be used to make thumbnails from outside sources and this could be used to bring a script into the cache folder. The version I had was the vulnerable one, so I did the update and wiped all the caches.

    but... it happened again.
    Someone, they're executing a php script from the blog and I can't find it! I even scanned non php files to see if they had a php string in them and nothing sticks out.
    The logs do not reveal anything out of the ordinary either, but I can't inspect them manually, so I ran out of idea about what to look for.

    Is there a way to know what script is being executed? I have thought about replacing the php executable with a script that would dump the command line and then call the real php.

    I also tried RKHunter just to see and it didn't find anything. I don't think the VM is compromised, nothing in the log indicates it is and you can't ssh to it without going through the root node; I think that somehow some external PHP gets in the machine and gets executed. The target IP has changed too, which means it is taking external commands probably through some WP vulnerability or some glaring hole I don't know

    Any suggestion?

  2. #2
    Linux Guru Lazydog's Avatar
    Join Date
    Jun 2004
    Location
    The Keystone State
    Posts
    2,677
    Once hacked the best you can do is wipe and reinstall. I do not know if the attacker has root access. {Please check your logs carefully to to figure this out. But if he has root then he is most likely smart enough to remove entries in the log files.

    Next after reinstall lock the system down hard.

    20 Linux Server Hardening Security Tips

    And lock down your firewall to only allow what is needed.

    Regards
    Robert

    Linux
    The adventure of a life time.

    Linux User #296285
    Get Counted

  3. #3
    Just Joined!
    Join Date
    Apr 2012
    Posts
    5
    I don't think the person could have root access because sshd is not even running on the VMs; it would take hacking the root node and then go in the VM to do that.

    I'm definitely going to do a re-install of WP and move the DB and files, but I still need to figure out what has happened in order to learn from it.
    I asked the provider to give me the parts of the switch's log that are relevant to my IP to find if there are any discrepancies between my logs and theirs, but they do not want to spend time on that.

    In order to find, the key to the solution is to know what php-cgi is executing when it is using 100% of the CPU; I have checked the files open by the process when it's at 100% but it doesn't indicate anything, so the file may be already closed.

  4. #4
    Penguin of trust elija's Avatar
    Join Date
    Jul 2004
    Location
    Either at home or at work or down the pub
    Posts
    3,482
    You don't need sshd to gain root access to a server! For example there is this famous exploit. If you have this or something like it then it can be a bugger to track down.
    What do we want?
    Time machines!

    When do we want 'em?
    Doesn't really matter does it!?


    Conkybots: Interactive plugins for your Conkys!

  5. #5
    Just Joined!
    Join Date
    Apr 2012
    Posts
    5
    good point!
    I also didn't do a virus scan on the machine, but I will do that tonight.
    Thanks!

  6. #6
    Just Joined!
    Join Date
    Apr 2012
    Posts
    5
    I did a virus scan with Kaspersky... nothing.

    I started to reinstall Wordpress, but I realized that I have to move over so many files, I may also be moving the problem.

    I'd rather trying to find the source of the problem and understand it rather than just flushing everything and hope I didn't copy the problem...

    Until I find out, I still need that server online, so to prevent it from being part of a DDOS attack, I was thinking about the following:
    - Allow outgoing UDP on port 53 with a rate limiter.
    - Deny all other outgoing UDP connections.

    Does that make sense?

  7. #7
    Linux Guru Lazydog's Avatar
    Join Date
    Jun 2004
    Location
    The Keystone State
    Posts
    2,677
    How about dropping all output except what you require?

    Regards
    Robert

    Linux
    The adventure of a life time.

    Linux User #296285
    Get Counted

  8. #8
    Just Joined!
    Join Date
    Apr 2012
    Posts
    5
    That would be ideal, but I do now know how; here's the setup:

    There is a host running OpenVZ with a variety of containers (all in the 10.0.x.x ip range).

    So the incoming connections go through the firewall then to the host, then pound (a reverse proxy that listens on port 80) decides where to forward the connections (to which container).
    A firewall rule then does post routing from all the VM's connectionx (10.0.x.x) toward the host's IP so that outbound connections can work.

    So, I do not really know if an outgoing connection is in response to an incoming connection but also there is the problem of DNS: the VMs are going to do UDP:53 outgoing requests with a random source port (for the return) and I do not know how to let that go through.

    I do not know also if it's better to filter at the outgoing or forward level but I also have options at the nat level too, so I'm a bit confused about what would be the right way.

    Any help regarding how I should set this up would be more than welcome.

  9. #9
    Linux Engineer
    Join Date
    Apr 2012
    Location
    Virginia, USA
    Posts
    881
    If the VPS was compromised, you need to wipe it, and install clean. Hopefully the blog has recent offline backups before the compromise took place.
    Any number of system files could have been tampered with, many back doors installed, etc.
    Often times, the source of exploit is a compromised FTP password. Most people use simple passwords and connect to FTP plaintext. The same can happen with Word Press.

  10. #10
    Linux Guru Lazydog's Avatar
    Join Date
    Jun 2004
    Location
    The Keystone State
    Posts
    2,677
    Quote Originally Posted by ThomasD View Post
    That would be ideal, but I do now know how;
    As has been said many times you really need to do a wipe and re-install.

    As for the firewall rules, you need to be running a STATEFUL firewall which is looking at the types of connections.

    Your INPUT rules should contain the ports/ip address that you want to allow in.
    Your OUTPUT rules could only contain already established connection to be allowed back out and anything new be dropped.

    Here is a basic firewall using the ideas above;
    Code:
    iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
    iptables -A INPUT -p tcp -m tcp --dport 53 -m state --state NEW -j ACCEPT
    iptables -A INPUT -p udp -m udp --dport 53 -m state --state NEW -j ACCEPT
    iptables -A INPUT -p tcp -m tcp --dport 80 -m state --state NEW -j ACCEPT
    iptables -A INPUT -j DROP
    iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
    iptables -A OUTPUT -j DROP
    The above example will allow dns and web in and drop everything else. It at the same time willl allow the server to answer to the new connection but drop everything else including any new connection the server tries to make.

    Please be aware that nothing new will be allowed out from this firewall.

    Regards
    Robert

    Linux
    The adventure of a life time.

    Linux User #296285
    Get Counted

Page 1 of 2 1 2 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •