The first part of this article can be found here
Two networks = twice as much fun
Part 2. Routing two for DSL lines
A recap: I am responding this letter:
Dear Dr. UN*X,
I am currently using my Linux box as a server on the Internet, and
it's all working fine. The installer program configured the network
interface for me. I am currently using a single DSL line but now I
have added a second line and I want remote sites to be able to connect
to my system using either DSL line.
I have already installed a second network card, but the system
does not see it. I checked the card maker's site and there are no
Linux drivers for it!
I don't need to do anything fancy (yet) like load balancing, I just want
to be able to support two outside IP addresses.
Where do I go from here?
--Adrian
Hopefully you were able to use part one of this article. If so, you
now have a functioning network connection on each of your two DSL
lines.
Adrian, you are setting up two lines so that your web server can be
reached from two different DSL connections. You didn't say why in your
email, but I am guessing you want redundancy and possibly more
bandwidth to your server. Round
robin DNS would be a simple approach to take once routing over both
lines is working. Using multiple DSL lines is not as effective for
increasing bandwidth for a single desktop system; I will talk about
that later.
For the impatient
It has taken me a lot of work to get to to the point where I am comfortable
working with Linux advanced routing. Let's see if I can sum things up
for those of you who already understand basic routing. Then you can skip
ahead to the section "Adding the second line"
With basic packet routing, there is only one routing table. The
kernel choses the path for each packet by examining the destination
address of that packet.
With advanced routing, you can create as many routing tables as
you want using 'ip route' commands. By using 'ip rule' commands, you
teach the kernel which routing table to use based on the source
address of each packet. 'Advanced routing' is also called 'iproute2' or
just 'iproute' in the Linux world.
That is it in a nutshell.
To get started with advanced routing you don't need to know
anything about 'netfilter' (aka iptables). Netfilter can be used to
build firewalls and to re-write the addresses in packets (NAT and
MASQUERADE). Netfilter is a topic for another day.
Before we begin
Your system probably already has advanced routing built into the
kernel; all the distributions that I use come with it. To configure it
though, requires installation of a package of utilities so that you
can configure the kernel.
Should you be building your own kernel, it needs to have the 'IP:
Advanced router' and 'Network packet filtering' options turned on.
Whether the required utility package is installed or not depends on your
distribution. The package is usually called 'iproute'. If you have
the 'ip' command (look for the file '/sbin/ip') you already have
it. Otherwise install the iproute package using the command appropriate
for your distribution.
{mospagebreak title=Just what is routing?}
Just what is routing?
I like to compare network packet routing with the post office. Let's
say that you and a friend play chess by sending postcards to each
other. Each postcard has a source and a destination address (From:
and To: addresses) and a small amount of information, the next move in
your game. You send a postcard with your move on it, then you wait for
a reply to come back with your friend's next move. The game progresses
as a series of transmissions and replies; the whole thing routed
through any number of post offices without your knowledge. Under
ordinary circumstances, to tap into this vast network, all you have to
need to know are the addresses and how to find your mailbox.
Each postcard is like a network packet. With the postal system, you
could send a letter if your information doesn't fit on a card. With a
network, you just send more packets instead because you can't
(normally) change the packet size; your operating system handles the
task of breaking down large messages into small packets for
transmission; likewise it reassembles incoming packets for you. This
is what happens when you request a web page; you enter one address
(the URL) and the network coordinates the exchange of hundreds of
packets unbenownst to you. To learn about routing though, you can
ignore this fact and concentrate only on the routing of a single
packet.
After you drop your postcard into a mailbox, a system for routing it
to its destination takes over; the reply comes back the same way. You
only care about this routing mechanism when something changes, for
example, if you move to a new address and want things
forwarded. Otherwise it all remains invisible to you.
With the Internet, you normally only deal with one connection
to this invisible network; the router provided to you by your
Internet Service Provider (ISP) is your connection.
If you have one computer and one network connection, this is just
about all you need to know about routing. Any packet leaving your
computer heads out to the Internet, and any reply coming back from the
Internet is destined for your one computer.
More complexity
Continuing with the postal analogy, if you work in the mailroom for a
large company, you still generally only have one connection to the
outside world, but there is an internal mailbox for each employee. As
the clerk in the mailroom, you decide based on the To: address to
either route a letter internally (delivering it to its local mailbox)
or as before, you throw it into an outgoing pile for your connection
to the world.
When you add a second computer in your office, each of your
computers will have addresses indicating they are connected to the
same LAN (local area network). The routing table in each of your
computers has to know the correct LAN address range so that they can
talk to each other.
With additional equipment, network complexity grows and things become
progressively more complicated and much more interesting (to me
anyway); there can be many routes in your computer's routing table. We
are not getting into that here.
In the simple set up we are doing here, you only have to recognize two
types of address, internal (LAN) or external (Internet).
{mospagebreak title=Where do the IP addresses come from?}
Where do the IP addresses come from?
You used to have to set up ip addresses, netmasks, gateways, and name
resolvers all manually. Refer to any basic networking text to learn
more about this aspect.
These days DHCP (Dynamic Host Control Protocol) is very popular. Let's
assume that a DHCP server (typically inside your Cable/DSL modem)
and issues all the network information required. The network engineer
at your upstream provider has configured the modem for you so that it will
work with their network.
With a typical home setup, you are assigned just one IP address and if
you have more than one computer at home, you have to play games (NAT
or MASQUERADE) to share the one IP address. A typical small business
connection might have 5 IP addresses assigned statically so that you
can publish them; a sixth address in the range lets you communicate
with the router in your DSL modem.
For a 6 address block, you'd be assigned a netmask with 29 bits
(255.255.255.248 in "dotted decimal" format). You don't need to know
what a netmask is right now, but it divides the address into two
parts, network and local. In the postcard analogy, a p.o. box number
would be the local part of your address.
Let's say your first DSL provider gives you the address range from
63.63.63.1 to 63.63.63.6; your computer is at 63.63.63.1; your DSL
router is at 63.63.63.6; and the remaining 4 addresses are currently
unused.
{mospagebreak title=Revealing your routing tables}
Revealing your routing tables
From a command line on any Linux system, you can see the existing
routing table by simply typing 'route' at the prompt. (Type
'/sbin/route' if /sbin is not in your path.)
Your routing table will be similar to this:
# route
destination gateway interface
63.63.63.0 * eth0
default 63.63.63.6 eth
Advanced routing commands are issued as arguments to the 'ip'
command. To see the routing table with iproute2, you can use the long
version "ip route show table main" or the shortcut version "ip route"
like this:
# ip route
63.63.63.0/29 dev eth0 proto kernel scope link src 63.63.63.1
default via 63.63.63.6 dev eth0
Recall that in our simple example, all ip addresses are considered to
be either 'local' or 'not local'. For this network, a local address
will be in the range from 63.63.63.1 to 63.63.63.6. A non-local
address is just anything else.* It's traditional to assign the router the
highest address in the range (ending in 6 for this network).
The first line in the routing table says that if the address is
local, your computer puts the packet right out onto the ethernet wire
using interface /dev/eth0. The only other device on your local
network right now is the DSL router at 63.63.63.6. You can probably
use a browser to bring up the router's built-in web-based
configuration interface by connecting to http://63.63.63.6/
When you want to surf over to linuxforums.org, the packets need to go
to the Internet. You computer will use DNS to translate the name
linuxforums.org into its IP address (67.15.52.42). Your computer sees
that address is not local, so it uses teh 'default' entry in the routing
table and sends the packet to your DSL router.
In turn, the DSL router looks at the destination address, and
sees that the data is destined for somewhere on the
Internet. The DSL router then pushes the packet out its DSL connection to
the next router upstream which is at your ISP. Just as with the
postcard analogy, it's now out of your hands; more upstream routers
send it on its way to the linuxforums.org server.
When the response comes back from linuxforums.org, there is a
destination address of 63.63.63.1 in the packet, so when it hits your
DSL router it will pass the packet onto your LAN. Your desktop
computer sees the packet on the LAN and picks it up. The round trip
connection is now completed. It takes many such exchanges of packets
to pass all the data required for just a single page.
If you have a web server running on your local computer, everything works
pretty much the same way, but the traffic flows the other
direction. Requests for pages come in from the Internet and your web
server responds by sending pages back out via the DSL router. Usually
you arrange to get your ip address into a DNS server somewhere so that
you can publish a URL for your server with a friendly name like
"http://myveryownserver.org" but you could just as easily not bother
with that and just tell all your friends that your URL is
http://63.63.63.1/.
When a friend surfs to http://63.63.63.1/, your web server will
receive the request with a source address pointing back to your
friend's computer. Thus the reply will be routed back out via your one
and only default router.
* Yes, I know there is another internal network on the address
127.0.0.1. It's called the loopback interface /dev/lo. But let's not
worry about it here, okay?
{mospagebreak title=Adding the second line}
Adding the second line
Still with me? It should have been straightforward so far; adding the
second line is when things get difficult.
After installing the new network card and getting it connected to DSL
#2, you find that it's been assigned the address of 30.31.32.1 and
the new DSL router is at 30.31.32.6.
Unless you made other changes, your system will probably continue to
use DSL #1 as the default route and your routing table will now look
like this. (I have also seen systems use the second line as the default router,
I think it might be dependent on the distro.)
# ip route
63.63.63.0/29 dev eth0 proto kernel scope link src 63.63.63.1
30.31.32.0/29 dev eth1 proto kernel scope link src 30.31.32.1
default via 63.63.63.6 dev eth0
If you are sitting at the console of your machine and your personal
web server is configured to listen to all available interfaces, you
can bring up a web browser on the console and surf to either
http://63.63.63.1/ or http://30.31.32.1/; you will see your home page
both places.
But what happens when your friend on the Internet tries to hit your page
at its new address http://30.31.32.1/ ?
The request comes in via DSL router #2, which passes it onto
your new ethernet line (/dev/eth1).
The problem is that the system does not know how to route packets back
to your friend's computer. Because no connection can be established, no
page gets sent back.
On your computer, you can use the ping command to test connections.
ping 63.63.63.1 test your own network interface on eth0
ping 30.31.32.1 test the interface on eth1
ping 63.63.63.6 test the router on dsl 1
ping 30.31.32.6 test the router on dsl 2
ping yahoo.com see if you can reach the Internet
In each case you will see something similar to this:
# ping 63.63.63.6
PING 63.63.63.254 (63.63.63.254) 56(84) bytes of data.
64 bytes from 63.63.63.254: icmp_seq=1 ttl=150 time=0.480 ms
64 bytes from 63.63.63.254: icmp_seq=2 ttl=150 time=0.481 ms
64 bytes from 63.63.63.254: icmp_seq=3 ttl=150 time=0.471 ms
.
.
.
(Hit control-c to end the test)
--- 63.63.63.254 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.471/0.477/0.481/0.018 ms
All these tests should still work without changing routing, including
the one that hits yahoo.com out on the Internet. Traffic bound for
the Internet continues to use the default route on DSL #1.
{mospagebreak title=How can I have two default routes?}
How can I have two default routes?
This is where advanced routing comes into play (finally!). With
advanced routing, you can have as many routing tables as you need; in
this case you need to add just one for the new DSL line.
First, add a name for the new routing table to the file
/etc/iproute2/rt_tables. You can append it to the file with command
"echo 2 dsl2 >> /etc/iproute2/rt_tables".
# echo 2 dsl2 >> /etc/iproute2/rt_tables
# cat /etc/iproute2/rt_tables list the file contents
#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
#1 inr.ruhep
2 dsl2 the line you just added
Earlier I mentioned that typing "ip route" is a shortcut for the
longer command "ip route show table main". Well, to list the new routing table
you have to use the long form: "ip route show table dsl2" If you enter this
command right now, you will see the new table is empty.
You really only need to add the new default route to the new table;
the old "main" table will continue to handle everything else. You will
see why in a minute. Once again, here is the existing "main" table.
# ip route show table main
63.63.63.0/29 dev eth0 proto kernel scope link src 63.63.63.1
30.31.32.0/29 dev eth1 proto kernel scope link src 30.31.32.1
default via 63.63.63.6 dev eth0
Add the new default route to table dsl2 and then look at the (short) table.
# ip route add default via 30.31.32.6 dev eth1 table dsl2
# ip route show table dsl2
default via 30.31.32.6 dev eth1 the whole table is just one line
But the new table is not used yet!
You need to learn one more command, "ip rule"
A routing table tells where packets should go (its destination).
You need to be able to tell the kernel to use a different table,
based on where a packet is from (its source address).
The existing ip ruleset is very simple, look at it now.
# ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
You need to add just one new rule:
# ip rule add from 30.31.32.1 lookup dsl2 prio 1000
This command says "add a rule" to handle the case when a packet has a "from"
pattern of "30.31.32.1"; use the routing table called "dsl2", and
assign the rule a priority level of "1000". Now relist the rules. In
this example, the "pattern" only needs to match one address but if you
build a Linux router, you could set patterns that would match
different sets of addresses.
# ip rule
0: from all lookup local
1000: from 30.31.32.1 lookup dsl2
32766: from all lookup main
32767: from all lookup default
The kernel searches the ip rules in order, starting with the lowest
priority and continuing through each rule and routing table until the
packet has been routed successfully.
Your default ruleset will always have a 'local' table with 'all'
as the match pattern. The local table handles traffic that is
supposed to stay on the local machine, and broadcast traffic.
Our new rule comes next, with a priority of 1000. I picked this number
to make adding other rules before and after ours easy later on.
After our rule comes the 'main' table, which is the one that is
manipulated by the old 'route' command. Finally comes the 'default'
table. I don't know the official purpose of default, it's empty on all
the systems I have set up. There is a 'default' route in the table
'main', so no traffic ever gets to the table 'detault'.
Caveats
When you are playing with multiple routing tables, you have to
remember to add the 'table' portion to the command. I have only
forgotten about 1000 times now. It can be mystifying when rules change
in the wrong table (main). And of course, you are sure to confuse
things when learning and lock yourself out if you are remotely logged
in. The changes happen FAST. Use a console connection.
Another tip. Routes are cached. This means that if you update a
routing table and nothing seems to happen, don't get frustrated, just
flush the cache. You can make several changes at once and then flush
the cache at the end so that the changes effectively all happen
simultaneously. This is handy when working on a running router.
The flush command I use is "ip route flush table cache". Be very careful
with the flush command!! Enter it wrong and you will remove all routing
rules, instantly cutting off your networks.
{mospagebreak title=Making the new commands sticky}
Making the new commands sticky
Once the new routing table and the new rule are in place, packets
coming in on the new DSL line will be routed back out the same line
and all is well. But a reboot now will reset things to the previous
state.
On Trustix, I put the commands in a script in
/etc/sysconfig/iptables-ipv4.d/ called "routing.sh" and make it
executable; any executable script in this directory is run at boot
time. I think this will work on other relatives of Redhat including
Fedora and SUSE.
On Ubuntu and its Debian-based cousins, you can put networking scripts
in the /etc/network/ifup.d directory, but you create a separate script
in /etc/init.d that can set up routing and firewalling and set it to
run after /etc/init.d/networking. I haven't done much work with
complex routing setups on Debian, as I prefer to use Trustix on
servers. Refer to the update-rc.d man page on Debian systems.
Load balancing
When I originally embarked on learning advanced routing, it was to
balance traffic from a LAN with around 300 nodes over several DSL
lines. I am still working on it. If you want more information on load
balancing see the iproute manual, especially section 4.2. (See
"resources" section below.)
If you are thinking you will get twice the bandwidth on your desktop
system if you add a second line, think again. Load balancing won't
work very well. You can do a form of round-robin load balancing; each
new outbound connection is routed to the next available line. The
problem with this approach is that the route for each connection
becomes associated with that one line (because the routes are cached).
If you start a download from a site, all traffic will come over
that one line; your download speed will be exactly the same whether
you have one connection or 50. If you start a second simultaneous
download from the same site, it too will use the same DSL line.
The only way this form of balancing can help is if you have many users
all accessing many different sites through a central router. This is
the approach that I am working on now.
I hope this article helps you to better understand the relationship between
the ip rules and the ip routing tables. It seems simple to me now
but it took Dr. Un*x many hours to figure it out.
{mospagebreak title=Further resources}
Further resources
In learning about routing for Linux I found this book to be very useful:
Linux Routers, A Primer for Network Administrators
by Tony Mancill
Publisher: Prentice Hall
ISBN 0-13-009026-3
Linux Advanced Routing and Traffic Control (iproute) manual:
http://lartc.org/
Linux documentation project networking howto:
http://www.tldp.org/HOWTO/NET3-4-HOWTO.html
The Debian reference manual has a chapter on networking:
http://www.debian.org/doc/manuals/reference/ch-gateway.en.html
Commands
Tcpdump
The 'tcpdump' command is your best friend when it comes to setting up
and debugging network problems.
ping
Ping is always there for you, on any Linux or Windows system.
It won't always work, sometimes firewall rules block it. But using
it with tcpdump is invaluable.
Suggested use: Set up one system to ping the destination, and run
'tcpdump -n -i interface icmp' on the destination system and/or
a router system in between. The interface should be something
like eth0 or eth1, the '-n' turns off DNS name lookups, and 'icmp'
limits the traffic dumped to icmp protocol, which is what ping uses.
It's very satisfying to see ping responses start coming back when you
finally get the routing set correctly.
Traceroute
The 'traceroute' command will give you a blow by blow account of the
path between your machine and a remote machine. Most Linux distros
don't install it by default; install the package named 'traceroute'.
Just type 'traceroute' followed by a remote hostname. You will see a
listing of the route packets have to take to get to the remote
location. (If you only see lines with * * * there is probably a
firewall in the way.)
MS Windows has the same command renamed 'tracert'. |
Submit Your Article
Latest Articles
Editor's Choice
Article Categories
Join Linux Forums Today
Win Great Prizes
User Score
Editor Score
Tools »
Bookmark
Share
Comment