SSH connects to wrong IP when being routed
I have 3 interfaces on my Linux router:
172.16.4.0/23 (Called: eth0, GW: 172.16.4.1)
172.16.2.0/23 (Called: lan0, GW: 172.16.2.1)
192.168.10.0/24 (Called: wan0, masqs to lan0 and eth0, is an internet connection)
All hosts can ping each other correctly and routing seems to be working fine (ie, if a host is down, ping obviously fails).
Now here's where it gets weird, if I SSH from any host in 172.16.2.0 into and try to connect to a host in 172.16.4.0, then my router that's routing traffic will always reply. Likewise if I try to from 172.16.4.0 into 172.16.2.0, my router always answers.
I can SSH into any machine perfectly fine on the same network, and from the internet inwards. The router itself can SSH into any host on any network.
It only does this for SSH. VNC remote desktop for example will work fine between the networks.
I've checked IP Tables, and there seems to be no SSH rule that would cause this (It's quite long), I made doubly sure by commenting some out to see if it had any changes.