SSH Troubleshooting

[Lanmanagers]

Hello.
Running 2 Suse 8.1 boxes - one off-site. The two connect each night via SSH to do a backup from on-site to off-site, initiated by the off-site Suse box. This ran fine initially, but then "broke". I am trying to find where it is broken, and need some advice on troubleshooting the ssh connection.

Here's what I know so far.
1. From on-site, I can Putty SSH into the on-site Linux box. (local LAN connection)
2. From off-site, I can Putty SSH into the off-site Linux box (local LAN connection)
Conclusion: no problem with SSH server on either box. Turn attention to the connection.

3. From on-site, I can Putty SSH into the off-site Linux box.
Conclusion: No problem getting through the router & firewall at off-site location.

4. From off-site, I cannot PuttySSH into the on-site Linux box.
5. From off-site, I CAN PuttySSH into a foreign SSH server site.

Conclusion: From all of the above, it appears to me that I am getting blocked by either the DSL modem or the Firewall at the on-site location. HOWEVER, see following.

SOLUTIONS TRIED:
I switched to a different on-site Firewall brand, with same results. I am now using the same brand/model of Firewall at both sites. They are identically configured. Port 22 is open. The log shows receipt of port22 traffic, and marks it as "bypassed". Port 22 is forwarded to the inside Linux box IP.

It is important to remember that this connection did work when we started this project, and that I could PuttySSH into either Linux box through these firewalls. Nothing has changed on either network. Other ports forward fine on the on-site firewall. I have tried assigning port 2200 as the SSH port, with same results.

The connection error is: "Fatal Putty Error: The network connection timed out" when trying to connect from Off-site to On-site.

Here is the hardware layout: Suse --->TrendNet Firewall --->NetopiaDSL Modem ---->>>Internet<<<----QWestDSLModem<---TrendNet Firewall<---Suse On/site.

Do any of you know of any tools that I can use to see where SSH is getting blocked? I hate to say this in this forum, but I am a Linux neophyte, and would prefer to troubleshoot this using Windows. However, I can use Suse linux if you will give well-defined steps.

Thank you for any assistance anyone can offer.
Jeff.

[gogalthorp]

Hello.
Running 2 Suse 8.1 boxes - one off-site. The two connect each night via SSH to do a backup from on-site to off-site, initiated by the off-site Suse box. This ran fine initially, but then "broke". I am trying to find where it is broken, and need some advice on troubleshooting the ssh connection.

Here's what I know so far.
1. From on-site, I can Putty SSH into the on-site Linux box. (local LAN connection)
2. From off-site, I can Putty SSH into the off-site Linux box (local LAN connection)
Conclusion: no problem with SSH server on either box. Turn attention to the connection.

3. From on-site, I can Putty SSH into the off-site Linux box.
Conclusion: No problem getting through the router & firewall at off-site location.

4. From off-site, I cannot PuttySSH into the on-site Linux box.
5. From off-site, I CAN PuttySSH into a foreign SSH server site.

Conclusion: From all of the above, it appears to me that I am getting blocked by either the DSL modem or the Firewall at the on-site location. HOWEVER, see following.

SOLUTIONS TRIED:
I switched to a different on-site Firewall brand, with same results. I am now using the same brand/model of Firewall at both sites. They are identically configured. Port 22 is open. The log shows receipt of port22 traffic, and marks it as "bypassed". Port 22 is forwarded to the inside Linux box IP.

It is important to remember that this connection did work when we started this project, and that I could PuttySSH into either Linux box through these firewalls. Nothing has changed on either network. Other ports forward fine on the on-site firewall. I have tried assigning port 2200 as the SSH port, with same results.

The connection error is: "Fatal Putty Error: The network connection timed out" when trying to connect from Off-site to On-site.

Here is the hardware layout: Suse --->TrendNet Firewall --->NetopiaDSL Modem ---->>>Internet<<<----QWestDSLModem<---TrendNet Firewall<---Suse On/site.

Do any of you know of any tools that I can use to see where SSH is getting blocked? I hate to say this in this forum, but I am a Linux neophyte, and would prefer to troubleshoot this using Windows. However, I can use Suse linux if you will give well-defined steps.

Thank you for any assistance anyone can offer.
Jeff.

Since nothing has changed on either end could one of you ISPs changed something. Worth checking out any way.

[Lanmanagers]

Yes, that's possible. I was saving that as a last resort, but I'm pretty close to the end of the resort rope, unless anyone else out there can think of something I might have missed. Thank you for reading and replying.

[anomie]

From off-site, I cannot PuttySSH into the on-site Linux box.

Best thing would be to eliminate putty from the equation. It's not necessary, and the true test (if I'm understanding) will be for the offsite suse box to ssh into the onsite suse box.

From the offsite suse box:

First, try to see if the port is open using netcat:

nc onsitebox_here 22

Did you see any dialog at all?

Second, try to ssh in with verbose mode turned on:

ssh -vvv user_here@onsitebox_here

Did that work? If not, post the output. It should give a lot of clues about what is happening.

[anomie]

P.S. Those are very old versions of suse. You might start thinking about an upgrade path - soon. Are you even able to get security updates for 8.1 any more??

[Lanmanagers]

Anomie -
Thank you for such an informative post.
I will give this a test and report back.
This is cool!!
Jeff.

[Lanmanagers]

OK!, Well, obviously I don't know much about posting here - I posted results of tests 2 days ago using the "Quote" feature, but somehow I never got the post to "post". So, here we go again, this time using a Suse terminal....

Here are the results of the tests recommended by Anomie


Linux2:~ # nc <ip_address> 22
bash: nc: command not found


Linux2:" # ssh -vvv user@ip_address
OpenSSH_3.4p1, SSH protocols 1.5/2.0, OpenSSL 0x0090607f
10734: debug1: Reading configuration data /etc/ssh/ssh_config
10734: debug1: Applying options for *
10734: debug1: Rhosts Authentication disabled, originating port will not be trusted.
10734: debug1: ssh_connect: needpriv 0
10734: debug1: Connecting to <ip-address> [ip-address] port 22.
10734: ssh: connect to address <ip-address> port 22: Connection timed out
Linux2:~ #

So, that was cool, interesting, and I learned something. This is good!
Now, it would seem that the nc command is not in the expected search directory, so do I need to download a copy? Or switch to another directory?
What do I try next?
Thanks for the help!!
Jeff. (Hope this posts this time....)

[anomie]

Netcat must not be installed by default on the suse box. Not important.

Your verbose ssh client session did give some useful feedback, though. That the connection timed out probably means that a packet filtering firewall is preventing the offsite suse box from ssh'ing to the onsite suse box.

My guess is you're allowing all or most outbound access from the offsite suse box. So the likely culprit is the trendnet firewall in front of the onsite suse box. Can you confirm that it's accepting inbound connections to port 22?

[Lanmanagers]

Thank you for your time and reply.

The only confirmation I have of the firewall receipt on port 22 is 1) the firewall log, and 2) the use of 2 different firewalls to correct this problem. As I said in the original post, the firewall log shows 2 packets "bypassed" by the TrendNet router, each time I attempt an SSH connection. That terminology is the opposite of "Rejected", which is what the log states when it blocks packets, so I presume "bypassed" means "forwarded" (I wish firewall manufacturers would sync on their terminology).

The other firewall used previously, did not make any difference either.

The fact that the TrendNet is "bypassing" port 22 packets implies that the outside modem is passing them on also, or they would not make it to the Trendnet.

Also as said previously, this whole thing worked initially, albeit only for a day or two, which really baffles me.

Any other tools/tests we can try to see where this is failing?
jeff.

[anomie]

Sure, you can try

# netstat -atnp | grep sshd

on the onsite suse box. (Run it as root.) The output of that will tell us whether sshd is listening on the appropriate interfaces.

And you can watch /var/log/secure on the onsite suse box to see if sshd is ever receiving any authentication attempt. (I can almost guarantee you it isn't.)

Are you running susefirewall2 on the onsite suse box? If so, did you poke a hole for port 22?