ssh boots me out

[dougwo]

I have been successfully shelling into my server for several years, but today it started immediately booting me out after login. I have two other servers in the same rack and can still shell into those so I know it isn't my terminal program or firewall. I also tried to shell into one of my other servers and shell across from there and got the same result. I am afraid my server may have been compromised. Here are the results of ssh -v :

OpenSSH_5.2p1, OpenSSL 0.9.8l 5 Nov 2009
debug1: Reading configuration data /etc/ssh_config
debug1: Connecting to web [xx.xxx.xx.xxx] port 22.
debug1: Connection established.
debug1: identity file /Users/xxxx/.ssh/identity type -1
debug1: identity file /Users/xxxx/.ssh/id_rsa type 1
debug1: identity file /Users/xxxx/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
debug1: match: OpenSSH_4.3 pat OpenSSH_4*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.2
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host 'web' is known and matches the RSA host key.
debug1: Found key in /Users/xxxx/.ssh/known_hosts:2
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-with-mic,password
debug1: Next authentication method: publickey
debug1: Trying private key: /Users/xxxx/.ssh/identity
debug1: Offering public key: /Users/xxxx/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 149
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
Last login: Wed Jun 15 12:18:27 2011 from xxxxx
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
Connection to web closed.
Transferred: sent 2064, received 2168 bytes, in 0.2 seconds
Bytes per second: sent 13169.3, received 13832.9
debug1: Exit status 0


Everything is right except immediate expulsion. Any ideas on what I should check? I can go to the data center and log in locally (I hope!) but wanted to be prepared when I went over...

The server is still running and I was able to FTP in and delete some large files just in case the partition was full.

Perplexed on this one...

[Irithori]

If filesystems are readonly (for example because of a disk or raid error), then login will fail.

Do you have decent remote management and/or hardware/raid monitoring?

[dougwo]

There is no raid. I have seen similar things before when a partition was full and that is why I deleted files. But it could be another partition that is full. I really can't tell until I log in. The fact that you replied as such give me hope that it is a disk problem, rather than an intrusion. The machine is still running and is supporting several websites and a small mail server.

FTP is chrooted to my home directory so that doesn't allow me to see much. I'll definitely check the disk for errors when I can login locally.

[zombykillah]

I've experienced a simmilar problem in following situation:
user1 is normal unix user id 1001
user1 is also the name of an user from a connected ldap database (for samba) ... bit id 10003

What happens.
ssh accepts user1 with bouth passwords and ids.

But when the console session is initialized by user id 10003 he gets thrown out immediately.

So I would start search in "/etc/pam.d/"

[dougwo]

Thanks zombykillah -- but what could happen overnight that would create this problem? I have done no upgrades or added any new users...

[atreyu]

can you just run a command remotely?, e.g.:

ssh <server_ipaddr> df

when you get there, check out the login profiles of the user you are trying to log in as (e.g., ~/.bashrc ~/.bash_profile ~/.profile /etc/bashrc, etc.) and make sure there is nothing in any of them causing your session to abort. Also check out the files in /etc/profile.d/*.sh as they are sourced by bash at login, too. Remember that you can put '-x' at the top of shell scripts to see what is going on when they run.

also, be sure to check out /var/log/secure for additional clues.

[dougwo]

Thanks atreyu. To be sure of this, I logged in as three (I only have three) different users and all have the same characteristics and one of the users is an email account only so he had never logged in via shell. I tried to run ssh <ip> df and got back nothing. Just a repeat of the prompt. I imagine it logged me in. I will try sendmail as well to see if I can get it to send me a message. Thanks for the tip!

[dougwo]

I got this in the server logs that were emailed to me last night...

--------------------- Disk Space Begin ------------------------

Filesystem Size Used Avail Use% Mounted on
df: `;': No such file or directory
df: `=': No such file or directory
df: `=': No such file or directory
df: `=': No such file or directory
df: `=': No such file or directory
df: `=': No such file or directory
df: `=': No such file or directory
df: `=': No such file or directory
df: `this': No such file or directory
df: `=': No such file or directory
df: `=': No such file or directory
df: `chose': No such file or directory
df: `=': No such file or directory


---------------------- Disk Space End -------------------------

Doesn't look good. To I have to rebuild fstab or something?

[Irithori]

My 2c:
The disks or the controller are borked,
and the only commands working are the ones buffered in memory (aka: that dont need to be read from disk)

You can verify that by either
- remote management
- or by looking at the local console

That machine is in backup and/or redundant and/or not important, no?

[dougwo]

I just went to the data center and cannot login locally either. it is still running the email and web servers so it is OK for now, but I will have to re-install Linux it looks like. Turns out timing is good because I am running FC 5 on that machine and it is time to upgrade anyway. Good thing is I have regular backups and I have check them and they are tight. Just need to shut it down for an hour and rebuild.

I am now thankful I use the old-school method of creating separate partitions for var, home, tmp, root etc... If one gets borked it doesn't bring down the whole machine...

Thanks for all the help and advice everyone.