Possible hack attack in Ubuntu Server?

[raghaven.kumar]

Hi All,

I am having a Ubuntu 10.04 Server with Shorewall 4.4.6.

For some days, i have been seeing that the logs shorewall.log, kernel.log and syslog getting to huge sizes above 20G per log.
It occupied all the free disk space, Server become dread slow and all the sites hosted on it stopped working.
I had to put a auto-empty of logs every 30 mins.
While exploring these logs i found lot of packets being dropped, from a unknown program sending to a specific IP y.y.y.y, originating from the Ubuntu 10.04 server x.x.x.x
(IPs masked for security reasons).
Here is some of the log entries -

Oct 25 12:17:35 web-server kernel: [18401369.775248] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7851 PROTO=UDP SPT=46899 DPT=14000 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.775356] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7852 PROTO=UDP SPT=47578 DPT=16413 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.775464] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7853 PROTO=UDP SPT=60750 DPT=14557 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.775572] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7854 PROTO=UDP SPT=56465 DPT=22698 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.775680] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7855 PROTO=UDP SPT=39699 DPT=56776 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.775790] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7856 PROTO=UDP SPT=40388 DPT=49843 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.775897] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7857 PROTO=UDP SPT=40385 DPT=47112 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.776004] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7858 PROTO=UDP SPT=52745 DPT=21869 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.776112] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7859 PROTO=UDP SPT=48034 DPT=33058 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.776220] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7860 PROTO=UDP SPT=60825 DPT=33964 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.776331] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7861 PROTO=UDP SPT=56701 DPT=17518 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.776442] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7862 PROTO=UDP SPT=49237 DPT=21521 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.776551] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7863 PROTO=UDP SPT=47788 DPT=37887 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.776660] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7864 PROTO=UDP SPT=52436 DPT=12071 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.776770] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7865 PROTO=UDP SPT=52870 DPT=27053 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.776880] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7866 PROTO=UDP SPT=59962 DPT=14336 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.776992] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7867 PROTO=UDP SPT=56726 DPT=39180 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.777137] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7868 PROTO=UDP SPT=40108 DPT=14175 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.777249] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7869 PROTO=UDP SPT=55149 DPT=1090 LEN=8200
Oct 25 12:17:35 web-server kernel: [18401369.777357] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=x.x.x.x DST=y.y.y.y LEN=8220 TOS=0x00 PREC=0x00 TTL=64 ID=7870 PROTO=UDP SPT=52778 DPT=58315 LEN=8200

I was able to locate the program using 'netstat', something like this.

udp        0   9968 x.x.x.x:52911          y.y.y.y:53809     ESTABLISHED

But every second it dies off and respawns on some other pid.
So, i am not able to kill it also.

Also, all the packets, cud be millions of them are being sent to the same IP y.y.y.y

Can anyone help me find the program which triggers these packets?

[Kloschüssel]

If there's a application running on your server that causes this problem you should be able to identify it with something like:

$netstat -A --program

See "$man netstat" for more information. Since something weird is going on on your server, it is generally a good idea to inform all customers of a possible security break and a planned downtime to remedy the issue. Once you took your server offline, you should run several sanity checks, tighten passwords, revoke certificates, etc. If the problem persists you should replace the server with a fresh installation.

Good luck!

[raghaven.kumar]

If there's a application running on your server that causes this problem you should be able to identify it with something like:

$netstat -A --program

Good luck!

Nope !!

root@web-server:~# netstat -A --program
Unknown address family `--program'.

Also, i had removed pass based auth for SSH, its only pub-key based auth enabled.

[BoDiddley]

I agree with informing your customers, but I am curious - does Shorewall not have a method to disable logging of the messages while you continue to extrapolate the cause?

[raghaven.kumar]

@BoDiddley, the issue is not shorewall logging, actually i do want shorewall to report all packet drops.
I want to find the program which is the cause of this.

[Kloschüssel]

Sorry, there is a typo in my message above.

$ netstat -p
(No info could be read for "-p": geteuid()=1000 but you should be root.)
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
[...]
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name    Path
unix  2      [ ]         DGRAM                    6899     -                   /var/spool/postfix/dev/log
unix  2      [ ]         DGRAM                    5988     -                   @/org/kernel/udev/udevd

This should also work with the -a option, but won't with -A (note the upper-/lowercase difference).

To quote the manual pages ("man netstat", as hinted in my previous message):

--protocol=family , -A
Specifies the address families (perhaps better described as low level protocols) for which connections are to be shown. family is a comma (',')
separated list of address family keywords like inet, unix, ipx, ax25, netrom, and ddp. This has the same effect as using the --inet, --unix (-x),
--ipx, --ax25, --netrom, and --ddp options.

-a, --all
Show both listening and non-listening sockets. With the --interfaces option, show interfaces that are not up

Cheers

[BoDiddley]

I am definitely a newbie, however I have learned a few tricks. Rootkit when it runs at the end will tell you if any PID's have invalid Uid's. Once I killed a PID based on the info provided, and then emptied my Java Cache to clear some inappropriate activity that was giving me trouble. Just a thought. TOP might catch-it if it is a process topping off resources. Just thoughts, sorry to be a bother.

[Kloschüssel]

You're making assumptions and yet you don't know what causes the trouble. Take your server offline, do the mantainance and if needed do a fresh installation.

Cheers

[raghaven.kumar]

OK, i had previously used chkrootkit, but it didnt show me anything.

So I did clamscan of /var/www, which led me to some PHP Shell rootkits.
I deleted them.(took backup of them to investigate it further)
Then, I downloaded rkhunter and ran it.

It showed me hidden ports found

[13:08:33]   Checking for hidden ports                       [ Warning ]
[13:08:33] Warning: Hidden ports found:
[13:08:33]          Port number: 46692
[13:08:34]          Port number: 52742

I then used unhide-tcp

root@web-server:~/rkhunter-1.3.8# unhide-tcp
Unhide 20080519 
yjesus@security-projects.com

Starting TCP checking

Starting UDP checking

Found Hidden port that not appears in netstat: 52763
root@web-server:~/rkhunter-1.3.8# unhide-tcp -v
Unhide 20080519 
yjesus@security-projects.com

Starting TCP checking

Starting UDP checking

Found Hidden port that not appears in netstat: 41936

As said earlier each time the (worm) program runs on different port.
Again, i am stuck at a point where i need to know the program that causes this

[BoDiddley]

Ok, you have found hidden ports, not sure what that means.

Try ports that are listening, they will give you program name. Theoretically, if it is a hidden port (communication) it is listening for an instruction.

lsof -i -n -P

This will list ports listening and programs.

As I am sure you are aware, viruses are designed to multiply and fight for survival - medical, or computer. I would suggest a rebuild... However, if you do not find what caused it they will simply re-infect you.

You are on the right track - you must accumulate enough info to identify the hack, by name - in order to cleanse and block future attacks.

run chkrootkit at various stages to see if anything changes.

Additionally,

netstat -ntulp

will list listening ports with foreign address. Maybe you can block the source system.