routing packets based on modem

Guys I'm wondering if somehow i can route a packet based on the modem it came from...

so if my default route is 10.0.0.2

and a packet comes from the external address 60.20.30.40 to 10.0.0.2(router) then to linux box it will work

however their next packet comes from 134.45.66.3 to 10.0.0.4(routerB) to linux box, it will fail, because my linux box is responding back via 10.0.0.2 (default gw)

You might be able to use iptables to do this. You can associate a particular IP address with a specific gateway, so any replies to that IP address should use the associated gateway. I'm not too familiar with iptables as yet, but it should (I would think) be possible to associate a range of addresses (netmask?) with a particular gateway as well.