Find the answer to your Linux question:
Results 1 to 7 of 7
Hi all, I am not a full time sys admin but I am given the partial responsibility of keeping two of the servers up and running. Recently one of machines ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Sep 2008
    Location
    Fairborn OH
    Posts
    4

    Server affected by botnet! keeps on sending mail


    Hi all,
    I am not a full time sys admin but I am given the partial responsibility of keeping two of the servers up and running. Recently one of machines has been affected (probably a botnet) and keeps on sending mail. Luckily the firewall is blocking outgoing SMTP and the issue is controlled to some extent. However since the server is part of an academic network the Campus IT people want us to completely reinstall the OS (which is going to be a pain since I'll have to do a ton of reconfigurations). I will mention the symptoms and any help to figure out the problem and resolve it without a reinstall (if possible) is appreciated.

    Here are the details

    1. The machine is behind a firewall though it has a public IP. Only web and SSH ports are open right now.

    2. The system is Redhat (cat /etc/redhat-release produces Red Hat Enterprise Linux ES release 4 (Nahant Update 6).
    Uname -a output is
    Linux xxx 2.6.9-5.ELsmp #1 SMP Wed Jan 5 19:30:39 EST 2005 i686 i686 i386 GNU/Linux

    3. The sendmail program gets called every hour and we see output similar to following
    Sep 9 11:08:39 xxx sendmail[21418]: m872cTtV015758: to=<xxx@xxx.org>, ctladdr=<apache@xxx.edu> (48/4, delay=2+12:30:10, xdelay=00:00:00, mailer=esmtp, pri=5610722, relay=aspmx5.googlemail.com., dsn=4.0.0, stat=Deferred: aspmx5.googlemail.com.: No route to host

    Note that the firewall is blocking the SMTP port right now.

    4. I suspected two drupal based sites hosted on the server to be the culprits and took them offline. However the mailing continues (thus they seem not to be the problem)

    For now I've stopped the sendmail daemon and restored the sites. Any help regarding this matter is appreciated

    Thanks

  2. #2
    Super Moderator Roxoff's Avatar
    Join Date
    Aug 2005
    Location
    Nottingham, England
    Posts
    3,882
    This seems to be a mail being sent out to, or via, googlemail. This would require a password/username if it were being used as a relay, so it could be a mail to a googlemail account holder.

    My thoughts are maybe that it's an outgoing mail stuck in the queue that cant send for some obscure reason (the DNS loopup is failing, the port is blocked, or summat like that).

    I also wondered if one of your system users has a googlemail email polling tool running? You could check that out, too.
    Linux user #126863 - see http://linuxcounter.net/

  3. #3
    Just Joined!
    Join Date
    Sep 2008
    Location
    Fairborn OH
    Posts
    4

    Still trying to find the cause!

    Thanks for the reply

    I suspect its a bit more than that. This is only a single entry that I picked out from the log. There are many other entries that use different relay servers.

    Since I've stopped sendmail I started seeing a different output now
    Here are a few example entries

    Sep 11 04:02:02 xxx sendmail[26863]: m8B822P3026863: from=root, size=4395, class=0, nrcpts=1, msgid=<200809110802.m8B822P3026863@xxx.xxx.edu>, relay=root@localhost
    Sep 11 04:02:02 xxx sendmail[26863]: m8B822P3026863: to=root, ctladdr=root (0/0), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=34395, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection refused by [127.0.0.1]

  4. #4
    Super Moderator Roxoff's Avatar
    Join Date
    Aug 2005
    Location
    Nottingham, England
    Posts
    3,882
    Surely these messages must be backing up in the mail queue, then? What do they say?
    Linux user #126863 - see http://linuxcounter.net/

  5. #5
    Just Joined!
    Join Date
    Sep 2008
    Location
    Fairborn OH
    Posts
    4
    Thanks for taking the time to look at this.
    Here is a bit more info as to why IT believes this machine is infected (Although I seem to notice the point in your logic and understand that these could be queued messages that has not been sent for a long time)

    1. This machine has been under firewall protection and we needed to enable mail sending via PHP for a few critical webapps (alsmost all these webapps are Drupal based. There is one webapp based on the openconf conference management system []Web & Technology Development & Consulting, OpenConf - Zakon Group LLC)

    2. The firewall has been configured to block smtp. We (webadmins) put in a request asking to open up the firewall and for it to go through the necessary channels and the actual thing to happen it took a few months. All webapps were active at this time.

    3. Once the firewall was opened IT noticed that there were many SMTP connections being made through this server (roughly around 350) and the rate kept on increasing. They immediately shut it down and concluded the machine is infected by a bot net.

    I've been monitoring the /var/log/maillog for a few days and noticed that there is a specific call at 4 am in the morning for the past 2 days. My understanding is that maillog is primarily written by sendmail and when the sendmail deamon is shutdown there would not be any other program that would put entries in the maillog.( I've looked at the ps output but nothing that I can immediately identify as a mailer). What baffles me is why am I seeing entries in the maillog even after shutting down sendmail.

  6. #6
    Super Moderator Roxoff's Avatar
    Join Date
    Aug 2005
    Location
    Nottingham, England
    Posts
    3,882
    Take a look in /var/spool/mail, or wherever the mail queue is on your computer, you'll see the emails queued up. You can take a look through and find out what sort of messages are being send, who to, and who from. This will be very helpful in finding out what process is generating these emails. Remember, as this is Linux solving it is probably just a matter of finding, terminating, and removing the process that's creating the mail.

    You may, of course, find that the emails are all mundane and your server generates these as a matter of course. You may also find that it's not a process that's doing this, but a misconfigured mailer, allowing some spam company to relay mail.
    Linux user #126863 - see http://linuxcounter.net/

  7. #7
    Just Joined!
    Join Date
    Sep 2008
    Location
    Fairborn OH
    Posts
    4
    Hi,
    Thanks for the info. I did look at the spool folder and there's a ton of messages there, primarily system messages and also some of the mails that we've seen on the maillog. I've moved the root mail spool to a different location and continuing to observe the logs. The root mail file is 43M and would take me a while to go through and see whose the culprit.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •