I've got two gateways as I'm switching network providers for faster lines, and I have observed some strange behaviour.

Basically, I have one server on another subnet going via the old gateway, one going via the new gateway. I connect via ssh from my workstation to either through the new gateway, connection works. This is source routing from what I understand, the packets are sent back the way they came (I also know this can be disabled in the kernel)

Now, I keep both ssh windows open and active, then the server going through the old gateway disconnects me but the one set to the new gateway is fine and stays connected.

Anyone care to explain why this works for ~20 secs and then breaks?

If source routing was to work, then surely it should just work? Not work for a short time and then disconnect. I'm trying to understand this behaviour.
Yes, I know I should set the gateway to be the right one, but I'm trying to understand this, not just change the gateway.

Anyone care to explain this?