Results 1 to 2 of 2
Enjoy an ad free experience by logging in. Not a member yet? Register.
[SOLVED] problem with Routing for multiple uplinks/providers on RH9 18.104.22.168
Also note that I am on a redhat 9 with an updated kernel 22.214.171.124 compiled with everything needed for advanced routing and packet filtering.
using the notations from that document, my setup is as follows:
- $IP1 is my main ISP and everything is configured through it and working fine
- $IP2 is my secondary ISP and it is configured and apparently is working (accessing $IP2 from a browser on a remote PC brings up the web page. ping also works. DNS doesn't. I'll get back to it later)
- my internal network is SNAT-ed through $IP1 and it works fine because showmyip-like sites report $IP1
however, if I do a DIG from another external machine, like "dig example.com A @$IP2" I get
reply from unexpected source: $IP1#53, expected $IP2#53
I only need this working for DNS, however it would be nice that ALL packets that come through an interface go out the same interface.
During my testings I found out why this happens:
- the packets comming in on $IP2 are being routed to the default gateway which is $P1
- the SNAT is done on all packets that go out on $IF1 for obvious reasons
- so obviously the response packet got SNATed
iptables -t nat -I POSTROUTING 1 -o $IF1 -p udp --sport 53 -j DROP
and the external dig call timesout with no other error.
Also, doing the same in OUTPUT chain has same effect.
from my RTFM-ing, I learned that the route is decided after the packet is generated and before it hits iptables. I tested it with
iptables -t mangle -A OUTPUT -o $IF1 -p udp --sport 53 -j DROP
it's the red "output routing" here: docum.org/docum.org/kptd
So it is obvious that the packet gets on $IF1 instead of $IF2, based on the routing rules and routes.
I also asked a friend to do that dig and checked the chace:
[root@Impuls root]# ip route show table cache | grep friends_ip local $IP2 from friends_ip dev lo src $IP2 friends_ip via $P1 dev $IF1 src $IP1
I have tried all sort of tehniques, starting with routing rules and finishing with iptables marking and connection tracking. nothing seems to work. I started from the above mentioend document/how-to which appears to be mentioned lot on the internet, so it must be good but also it is noted that it lacks some scenarios, specifically related to NATing of some kind.
I can mark an incomming packet in PREROUTING and still have it marked in INPUT (again tested with -j DROP by selecting the packed based on mark). but after that, in OUTPUT it's no longer there, which I think makes sense since in OUTPUT I have a new packet which is a response to the incomming one (at least that's how I understand this) so that packet is not marked. I'm not even sure if it should be.
I was marking packets with:
iptables -t mangle -A PREROUTING -i $IF2 -d $IP2 -p udp --dport 53 -j MARK --set-mark 6
All the above being said, I'd like to start out fresh. I cleaned up the routing rules and tables and my iptables rules so all is left is whatever was there before this ordeal started, so I'm back to square 1 as explained initially.
and I am out of ideas. The main question is, how can I get a packet comming in on an interface go out the same interface?
exact routing tables and rules:
[root@Impuls /]# ip rule 0: from all lookup local 32766: from all lookup main 32767: from all lookup 253 [root@Impuls /]# [root@Impuls /]# ip route show table local broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1 broadcast $P0_NET dev $IF0 proto kernel scope link src $IP0 broadcast $P1_NET dev $IF1 proto kernel scope link src $IP1 local $IP2 dev $IF2 proto kernel scope host src $IP2 broadcast $P2_BCAST dev $IF2 proto kernel scope link src $IP2 broadcast $P2_NET dev $IF2 proto kernel scope link src $IP2 local $IP1 dev $IF1 proto kernel scope host src $IP1 broadcast $P0_BCAST dev $IF0 proto kernel scope link src $IP0 broadcast $P1_BCAST dev $IF1 proto kernel scope link src $IP1 broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1 local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1 local $IP0 dev $IF0 proto kernel scope host src $IP0 local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1 [root@Impuls /]# [root@Impuls /]# ip route show table main $P2_NET/30 dev $IF2 scope link $P1_NET/24 dev $IF1 scope link $P0_NET/24 dev $IF0 scope link 169.254.0.0/16 dev $IF1 scope link 127.0.0.0/8 dev lo scope link default via $P1 dev $IF1 [root@Impuls /]# [root@Impuls /]# ip route show table 253 [root@Impuls /]# relevant iptables rules [root@Impuls /]# iptables -t nat -L -n Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination SNAT all -- 0.0.0.0/0 0.0.0.0/0 to:$IP1 Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain linkup (1 references) target prot opt source destination [root@Impuls /]# [root@Impuls /]# iptables -t mangle -L -n Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination [root@Impuls /]#
I also read through the PolicyRoutingBook from policyrouting.org but it's kind of missing on examples.
bottom line: in order to successfully test, I am thinking that the best solution is to "block" the outgoing (response) packets by IP (not iptables) directly with policy routing.
the following did not work for me
[root@Impuls root]# iptables -t mangle -A PREROUTING -i eth0 -d $IP2 -p udp --dport 53 -j MARK --set-mark 2 [root@Impuls root]# ip rule 0: from all lookup local 32761: from all to $P2_NET lookup 2 32762: from all to $IP2 lookup 2 32763: from $IP2 lookup 2 32764: from $P2_net lookup 2 32765: from all fwmark 0x2 lookup 2 32766: from all lookup main 32767: from all lookup 253 [root@Impuls root]# ip route show table 2 blackhole 0.0.0.0 [root@Impuls root]#
as usually, I end up answering my own (painfull, not google-able) problems.
so, I was too concentrated on DNS (that being the only service I actually needed to work right) that I did not bother testing any other services. After doing a lot of rtfm-ing on the linux routing and packet filtering and trying out all sorts of things I decided that until I find a solution, to make the DNS work. long stroy short, I configured my DNS to bind to each interface and it was than when I saw that power dsn can actually change the source address in the packet. So I got DNS working and then I figured I should try another service, http. I did the test without lartc implementation, it behaved as described, packet went out the wrong interface. however, when I implemented the lartc split, it went on the right interface.
So the actual problem was not the lartc but the powerdns. maybe I should log a bug with them.
so for whoever searches for a similar problem: make sure your service is not messing up the packets to begin with. test your setup with a nice playing apache for example and if that works, it's definitelly your service to blaim.