Find the answer to your Linux question:
Page 1 of 2 1 2 LastLast
Results 1 to 10 of 13
Like Tree2Likes
Hi All, I am having a Ubuntu 10.04 Server with Shorewall 4.4.6. For some days, i have been seeing that the logs shorewall.log, kernel.log and syslog getting to huge sizes ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Linux Newbie raghaven.kumar's Avatar
    Join Date
    Mar 2008
    Location
    Bangalore, India
    Posts
    209

    Exclamation Possible hack attack in Ubuntu Server?


    Hi All,

    I am having a Ubuntu 10.04 Server with Shorewall 4.4.6.

    For some days, i have been seeing that the logs shorewall.log, kernel.log and syslog getting to huge sizes above 20G per log.
    It occupied all the free disk space, Server become dread slow and all the sites hosted on it stopped working.
    I had to put a auto-empty of logs every 30 mins.
    While exploring these logs i found lot of packets being dropped, from a unknown program sending to a specific IP y.y.y.y, originating from the Ubuntu 10.04 server x.x.x.x
    (IPs masked for security reasons).
    Here is some of the log entries -
    Code:
    Oct 25 12:17:35 web-server kernel: [18401369.775248] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7851 PROTO=UDP SPT=46899 DPT=14000 LEN=8200
    Oct 25 12:17:35 web-server kernel: [18401369.775356] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7852 PROTO=UDP SPT=47578 DPT=16413 LEN=8200
    Oct 25 12:17:35 web-server kernel: [18401369.775464] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7853 PROTO=UDP SPT=60750 DPT=14557 LEN=8200
    Oct 25 12:17:35 web-server kernel: [18401369.775572] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7854 PROTO=UDP SPT=56465 DPT=22698 LEN=8200
    Oct 25 12:17:35 web-server kernel: [18401369.775680] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7855 PROTO=UDP SPT=39699 DPT=56776 LEN=8200
    Oct 25 12:17:35 web-server kernel: [18401369.775790] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7856 PROTO=UDP SPT=40388 DPT=49843 LEN=8200
    Oct 25 12:17:35 web-server kernel: [18401369.775897] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7857 PROTO=UDP SPT=40385 DPT=47112 LEN=8200
    Oct 25 12:17:35 web-server kernel: [18401369.776004] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7858 PROTO=UDP SPT=52745 DPT=21869 LEN=8200
    Oct 25 12:17:35 web-server kernel: [18401369.776112] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7859 PROTO=UDP SPT=48034 DPT=33058 LEN=8200
    Oct 25 12:17:35 web-server kernel: [18401369.776220] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7860 PROTO=UDP SPT=60825 DPT=33964 LEN=8200
    Oct 25 12:17:35 web-server kernel: [18401369.776331] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7861 PROTO=UDP SPT=56701 DPT=17518 LEN=8200
    Oct 25 12:17:35 web-server kernel: [18401369.776442] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7862 PROTO=UDP SPT=49237 DPT=21521 LEN=8200
    Oct 25 12:17:35 web-server kernel: [18401369.776551] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7863 PROTO=UDP SPT=47788 DPT=37887 LEN=8200
    Oct 25 12:17:35 web-server kernel: [18401369.776660] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7864 PROTO=UDP SPT=52436 DPT=12071 LEN=8200
    Oct 25 12:17:35 web-server kernel: [18401369.776770] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7865 PROTO=UDP SPT=52870 DPT=27053 LEN=8200
    Oct 25 12:17:35 web-server kernel: [18401369.776880] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7866 PROTO=UDP SPT=59962 DPT=14336 LEN=8200
    Oct 25 12:17:35 web-server kernel: [18401369.776992] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7867 PROTO=UDP SPT=56726 DPT=39180 LEN=8200
    Oct 25 12:17:35 web-server kernel: [18401369.777137] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7868 PROTO=UDP SPT=40108 DPT=14175 LEN=8200
    Oct 25 12:17:35 web-server kernel: [18401369.777249] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7869 PROTO=UDP SPT=55149 DPT=1090 LEN=8200
    Oct 25 12:17:35 web-server kernel: [18401369.777357] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7870 PROTO=UDP SPT=52778 DPT=58315 LEN=8200
    I was able to locate the program using 'netstat', something like this.
    Code:
    udp        0   9968 x.x.x.x:52911          y.y.y.y:53809     ESTABLISHED
    But every second it dies off and respawns on some other pid.
    So, i am not able to kill it also.

    Also, all the packets, cud be millions of them are being sent to the same IP y.y.y.y

    Can anyone help me find the program which triggers these packets?
    Last edited by raghaven.kumar; 10-25-2011 at 08:32 AM.

  2. #2
    Linux Engineer Kloschüssel's Avatar
    Join Date
    Oct 2005
    Location
    Italy
    Posts
    773
    If there's a application running on your server that causes this problem you should be able to identify it with something like:

    Code:
    $netstat -A --program
    See "$man netstat" for more information. Since something weird is going on on your server, it is generally a good idea to inform all customers of a possible security break and a planned downtime to remedy the issue. Once you took your server offline, you should run several sanity checks, tighten passwords, revoke certificates, etc. If the problem persists you should replace the server with a fresh installation.

    Good luck!

  3. #3
    Linux Newbie raghaven.kumar's Avatar
    Join Date
    Mar 2008
    Location
    Bangalore, India
    Posts
    209
    Quote Originally Posted by Kloschüssel View Post
    If there's a application running on your server that causes this problem you should be able to identify it with something like:

    Code:
    $netstat -A --program
    Good luck!
    Nope !!
    Code:
    root@web-server:~# netstat -A --program
    Unknown address family `--program'.
    Also, i had removed pass based auth for SSH, its only pub-key based auth enabled.

  4. $spacer_open
    $spacer_close
  5. #4
    Linux Newbie BoDiddley's Avatar
    Join Date
    Oct 2010
    Location
    Plainfield, New Jersey
    Posts
    137
    I agree with informing your customers, but I am curious - does Shorewall not have a method to disable logging of the messages while you continue to extrapolate the cause?

  6. #5
    Linux Newbie raghaven.kumar's Avatar
    Join Date
    Mar 2008
    Location
    Bangalore, India
    Posts
    209
    @BoDiddley, the issue is not shorewall logging, actually i do want shorewall to report all packet drops.
    I want to find the program which is the cause of this.

  7. #6
    Linux Engineer Kloschüssel's Avatar
    Join Date
    Oct 2005
    Location
    Italy
    Posts
    773
    Sorry, there is a typo in my message above.

    Code:
    $ netstat -p
    (No info could be read for "-p": geteuid()=1000 but you should be root.)
    Active Internet connections (w/o servers)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
    [...]
    Active UNIX domain sockets (w/o servers)
    Proto RefCnt Flags       Type       State         I-Node   PID/Program name    Path
    unix  2      [ ]         DGRAM                    6899     -                   /var/spool/postfix/dev/log
    unix  2      [ ]         DGRAM                    5988     -                   @/org/kernel/udev/udevd
    This should also work with the -a option, but won't with -A (note the upper-/lowercase difference).

    To quote the manual pages ("man netstat", as hinted in my previous message):

    --protocol=family , -A
    Specifies the address families (perhaps better described as low level protocols) for which connections are to be shown. family is a comma (',')
    separated list of address family keywords like inet, unix, ipx, ax25, netrom, and ddp. This has the same effect as using the --inet, --unix (-x),
    --ipx, --ax25, --netrom, and --ddp options.
    -a, --all
    Show both listening and non-listening sockets. With the --interfaces option, show interfaces that are not up
    Cheers
    raghaven.kumar likes this.

  8. #7
    Linux Newbie BoDiddley's Avatar
    Join Date
    Oct 2010
    Location
    Plainfield, New Jersey
    Posts
    137
    I am definitely a newbie, however I have learned a few tricks. Rootkit when it runs at the end will tell you if any PID's have invalid Uid's. Once I killed a PID based on the info provided, and then emptied my Java Cache to clear some inappropriate activity that was giving me trouble. Just a thought. TOP might catch-it if it is a process topping off resources. Just thoughts, sorry to be a bother.

  9. #8
    Linux Engineer Kloschüssel's Avatar
    Join Date
    Oct 2005
    Location
    Italy
    Posts
    773
    You're making assumptions and yet you don't know what causes the trouble. Take your server offline, do the mantainance and if needed do a fresh installation.

    Cheers

  10. #9
    Linux Newbie raghaven.kumar's Avatar
    Join Date
    Mar 2008
    Location
    Bangalore, India
    Posts
    209

    Lightbulb

    OK, i had previously used chkrootkit, but it didnt show me anything.

    So I did clamscan of /var/www, which led me to some PHP Shell rootkits.
    I deleted them.(took backup of them to investigate it further)
    Then, I downloaded rkhunter and ran it.

    It showed me hidden ports found
    Code:
    [13:08:33]   Checking for hidden ports                       [ Warning ]
    [13:08:33] Warning: Hidden ports found:
    [13:08:33]          Port number: 46692
    [13:08:34]          Port number: 52742
    I then used unhide-tcp
    Code:
    root@web-server:~/rkhunter-1.3.8# unhide-tcp
    Unhide 20080519 
    yjesus@security-projects.com
    
    Starting TCP checking
    
    Starting UDP checking
    
    Found Hidden port that not appears in netstat: 52763
    root@web-server:~/rkhunter-1.3.8# unhide-tcp -v
    Unhide 20080519 
    yjesus@security-projects.com
    
    Starting TCP checking
    
    Starting UDP checking
    
    Found Hidden port that not appears in netstat: 41936
    As said earlier each time the (worm) program runs on different port.
    Again, i am stuck at a point where i need to know the program that causes this

  11. #10
    Linux Newbie BoDiddley's Avatar
    Join Date
    Oct 2010
    Location
    Plainfield, New Jersey
    Posts
    137
    Ok, you have found hidden ports, not sure what that means.

    Try ports that are listening, they will give you program name. Theoretically, if it is a hidden port (communication) it is listening for an instruction.

    lsof -i -n -P

    This will list ports listening and programs.

    As I am sure you are aware, viruses are designed to multiply and fight for survival - medical, or computer. I would suggest a rebuild... However, if you do not find what caused it they will simply re-infect you.

    You are on the right track - you must accumulate enough info to identify the hack, by name - in order to cleanse and block future attacks.

    run chkrootkit at various stages to see if anything changes.

    Additionally,

    netstat -ntulp

    will list listening ports with foreign address. Maybe you can block the source system.
    raghaven.kumar likes this.

Page 1 of 2 1 2 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •