Routing for multiple uplinks
Posted by sphaero on Mon 27 Mar 2006 at 08:25
Debian suits perfectly for use as a gateway for computers on your LAN. However once bandwidth usage grows it could be handy to just add another internet uplink to your gateway. Debian does not cater for this out of the box so this document describes how to setup your debian gateway for multiple uplinks.
The whole setup is based on the Linux Advanced Routing & Traffic Control HOWTO http://lartc.org/howto/
The most important and only package we need is `iproute`
apt-get install iproute
The iproute package is a tool to talk to more advanced routing capabilities of the linux kernel. I suggest you read the man page and the howto mentioned above. Some basic theory of iproute is that it does it routing through tables just like iptables does. It determines what table to route through based on rules you define.
As an example I have 2 uplinks to the internet. The first uplink device eth1 has ipaddress 1.0.0.1/24 and gateway 1.0.0.2. The second uplink device eth2 has ip 2.0.0.1/24 and gateway 2.0.0.2.
First we will define the tables. We do this by editing /etc/iproute2/rt_tables which looks like this:
# # reserved values # 255 local 254 main 253 default 0 unspec # # local # #1 inr.ruhepIn this file we append a new table name with an unique number for every uplink.
# # reserved values # 255 local 254 main 253 default 0 unspec # # local # #1 inr.ruhep 200 uplink1 201 uplink2We can now focus on setting up the interfaces file. Setup /etc/network/interfaces like this.
auto lo
iface lo inet loopback
auto eth0
#LAN interface
iface eth0 inet static
address 10.0.0.1
netmask 255.255.255.0
#Uplink1
auto eth1
iface eth1 inet static
address 1.0.0.1
netmask 255.255.255.0
post-up ip route add 1.0.0.2/32 dev eth1 src 1.0.0.1 table uplink1
post-up ip route add default via 1.0.0.2 table uplink1
post-up ip rule add from 1.0.0.1 table uplink1
post-down ip rule del from 1.0.0.1 table uplink1
#Uplink2
auto eth2
iface eth2 inet static
address 2.0.0.1
netmask 255.255.255.0
post-up ip route add 2.0.0.2/32 dev eth2 src 2.0.0.1 table uplink2
post-up ip route add default via 2.0.0.2 table uplink2
post-up ip rule add from 2.0.0.1 table uplink2
post-down ip rule del from 2.0.0.1 table uplink2
You are now ready to bring the interfaces up.
ifup -aThere is no default gateway at this moment. We want to balance traffic over both uplinks. The following command will set this up.
ip route add default scope global nexthop via 1.0.0.2 dev eth1 weight 1 nexthop via 2.0.0.2 eth2 weight 1As stated in the howto mentioned above "The weight parameters can be tweaked to favor one provider over the other".
We have now setup our gateway for multiple uplinks. An other benefit of this setup is that the gateway responds through the same uplink as external traffic originated from.
Remember the balancing of the uplinks is route based and routes are cached. You will not get double bandwidth when downloading a file. You can flush the route cache with ‘ip route flush cache'. The uplink balancing is not perfect but to get a better solution you need to patch and recompile the kernel. Have a look at the patches provided by Julian Anastatov. http://www.ssi.bg/~ja/#routes
[ Parent | Reply to this comment ]
[ Send Message | View Utumno's Scratchpad | View Weblogs ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
m0n0wall - http://www.m0n0.ch/wall - geared toward smaller, embedded systems
pfsense - http://www.pfsense.com -new project in the idea of m0n0wall, aims to be enterprise level
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
If you define the interfaces in /etc/interfaces/network, they're started automatically as needed, but AFAIK, that unusual ip route command won't be done automatically...
[ Parent | Reply to this comment ]
[ Send Message | View Steve's Scratchpad | View Weblogs ]
A small script inside /etc/network/if-up.d ?
[ Parent | Reply to this comment ]
It could be added to either, or both, interfaces I assume.
If it doesn't play nice when only one is up, write a simple script.
I'm a big fan of doing all your basic networking config in the one file, makes figuring it out later, or by someone else, much easier.
[ Parent | Reply to this comment ]
turned off the CACHED_ROUTING flag & compiled/installed kernel.
first thing I found was to change the order of the ethernet cards.
eth0 and eth1 are the wan ports,
eth2 is the lan port.
this lets me put the last 'route add' command in a post-up on eth2, and eth2 comes up on boot, as opposed to failing, cause the post up fails.
to test it I am pulling the cable out of eth0 and eth1, and I'm finding that most of the time the routing wont switch across. particulary it switches from eth1 to eth0 ok, but not the other way, and even doing 'ip route flush cache' doesnt do it for me.
going to try the patches http://www.ssi.bg/~ja/#routes on my 2.6.20.whatever it was kernel.
[ Parent | Reply to this comment ]
applied said patches, now the 'pull the cable' test results in about 3 minutes of no connection to established route, then it cuts across. 3 minutes of outage is acceptable for what I'm doing, its heaps better than say 3 days of no internet 8^(.
if anyone knows how to reduce the 3 minutes setting, it'd be good to know
[ Parent | Reply to this comment ]
We have been using a similar system for a couple of years now and have very good results with 5 ADSL lines connected to 1 Debian router, also as we are the ISP we can do this at both ends of the connection which enables us to balance the traffic in both directions and have fail over :-)
For dead gateway detection we are using ping (working on using SNMP) on the customer CPE and PPP/radius on the LNS. We are running Soekris boxes at the customer end which support upto 7 interfaces.
Shorewall has a feature which will start scripts after it has setup the firewall, it's ideal for running iproute scripts.
Regards
Wayne
[ Parent | Reply to this comment ]
Thanks.
[ Parent | Reply to this comment ]
We have a script placed in /etc/ppp/ip-down.d/ which checks to see which interfaces are up or down and removes the routes to suit. We also use this script on ppp-up.
Wayne
[ Parent | Reply to this comment ]
To see if that option is set, try the "egrep MULTIPATH /boot/config-`uname -r`" command.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
ip route del default ...
And add your multipath route with several nexthops sfter that.
[ Parent | Reply to this comment ]
I've followed these instructions along with the ones at : http://lartc.org/howto/lartc.rpdb.multiple-links.html and I haven't been able to get dual wan out going to work.
If I set a single wan it'll work fine without any issues, but with dual I can't get it to deal with multiple default gateways, it will always only pick the second on specified.
Any ideas?
[ Parent | Reply to this comment ]
This statement...
ip route add default scope global nexthop via 1.0.0.2 dev eth1 weight 1 nexthop via 2.0.0.2 eth2 weight 1
Should read...
ip route add default scope global nexthop via 1.0.0.2 dev eth1 weight 1 nexthop via 2.0.0.2 dev eth2 weight 1
add "dev" in front of eth2, this may help.
Also, it might be good to note, if you are using other ip appresses than the one in this how to both the ip and the gateway need to be consecutive. ie: 192.168.0.1 and 192.168.0.2 not 192.168.0.1 and 192.168.0.254.
-- Just my .02.
[ Parent | Reply to this comment ]
I am running in teh same problem. Did you find a solution???
[ Parent | Reply to this comment ]
"ip route add default scope global nexthop via 1.0.0.2 dev eth1 weight 1 nexthop via 2.0.0.2 eth2 weight 1"
at the end is supposed to be a part of network file with "post-up" in front on eth1/eth2 definition or in a separate rc.local file?
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
See, for example,
http://mailman.ds9a.nl/pipermail/lartc/2006q4/019898.html
http://www.debianhelp.org/node/2900#comment-12647
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
This one doesn't work:
ip route del default >/dev/null 2>&1
ip route add default scope global equalize \
nexthop via $GW_EXT2 dev $IF_EXT2 weight 20 \
nexthop via $GW_EXT1 dev $IF_EXT1 weight 80
That is all traffic goes through $IF_EXT1.
This one also doesn't work:
ip route del default >/dev/null 2>&1
ip route add default scope global equalize \
nexthop via $GW_EXT1 dev $IF_EXT1 weight 80 \
nexthop via $GW_EXT2 dev $IF_EXT2 weight 20
That is all traffic goes through $IF_EXT2.
When I change nexthops order all traffic is gradually switched to the last route in the list. I didn't try this with more than two routes. Now I'm trying to add first route twice and see what happens. I need some time.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
If someone needs, I can post a complete diff -u file for Debian Etch's and ASP's kernel conf files.
In particular, I have noticed (diff -u config-2.6.18-1.2257.0.112asp config-2.6.18-3-686):
Can it be an SMP issue?
-CONFIG_BROKEN_ON_SMP=y
+CONFIG_LOCK_KERNEL=y
-# CONFIG_SMP is not set
+CONFIG_SMP=y
-CONFIG_X86_GENERIC=y
+# CONFIG_X86_GENERIC is not set
-CONFIG_X86_L1_CACHE_SHIFT=7
+CONFIG_X86_L1_CACHE_SHIFT=5
-# CONFIG_HZ_250 is not set
-CONFIG_HZ_1000=y
-CONFIG_HZ=1000
+CONFIG_HZ_250=y
+# CONFIG_HZ_1000 is not set
+CONFIG_HZ=250
# Networking
-CONFIG_XFRM_USER=y
+CONFIG_XFRM_USER=m
-# CONFIG_IP_ROUTE_MULTIPATH_CACHED is not set
+CONFIG_IP_ROUTE_MULTIPATH_CACHED=y
+CONFIG_IP_ROUTE_MULTIPATH_RR=m
+CONFIG_IP_ROUTE_MULTIPATH_RANDOM=m
+CONFIG_IP_ROUTE_MULTIPATH_WRANDOM=m
+CONFIG_IP_ROUTE_MULTIPATH_DRR=m
-CONFIG_IPV6_ROUTER_PREF=y
-CONFIG_IPV6_ROUTE_INFO=y
+# CONFIG_IPV6_ROUTER_PREF is not set
# QoS and/or fair queueing
-# CONFIG_NET_SCH_CLK_JIFFIES is not set
-CONFIG_NET_SCH_CLK_GETTIMEOFDAY=y
+CONFIG_NET_SCH_CLK_JIFFIES=y
+# CONFIG_NET_SCH_CLK_GETTIMEOFDAY is not set
-CONFIG_NET_SCH_ESFQ=m
-CONFIG_NET_SCH_LOG=m
Can someone point Debian developers to this discussion, please?
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
kernel: Bug in ip_route_input_slow(). Please, report
I think it somehow connected to this problem. And the problem is definitely in Linux Kernel, not Etch itself. Please, someone report this to kernel maintainers.
[ Parent | Reply to this comment ]
Thanks!
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
wget ftp://ftp.xs4all.nl/pub/test/100mb.bin first uplink
wget http://ftp.easynet.nl/testfile100 second uplink
ip route show:
default
nexthop via 172.16.30.2 dev eth1 weight 1
nexthop via 193.172.165.33 dev eth2 weight 1
[ Parent | Reply to this comment ]
auto dsl-provider
iface dsl-provider inet ppp
pre-up /sbin /ifconfig eth1 up # line maintained ;by pppoeconf
provider dsl -provider
Tnks and best regards
[ Parent | Reply to this comment ]
Those should be helpful
[ Parent | Reply to this comment ]
iface eth2 inet dhcp
Is there a clever way to obtain the IP once it is set, or do I need to write a script in /etc/network/if-up.d to set the routes?
[ Parent | Reply to this comment ]
Thanks for good article, our primary gateway with two optical uplinks works on Sarge, load balancing for default gateway was set as you have suggested.
But now there is some problem - all connections that have permanent outcoming streams (SSH, ICQ, Internet radio) broke several times during a hour, because server starts sending data through second interface
For example, with SSH connections, after reconnect with new session, I see:
dmitry@servername:~$ w
17:01:54 up 14 days, 5:49, 5 users, load average: 0,19, 0,14, 0,10
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
dmitry pts/0 A.A.A.A 10:28 1:19 0.15s 0.04s sshd: dmitry [priv]
dmitry pts/1 A.A.A.A 10:30 2:34 0.51s 0.05s sshd: dmitry [priv]
dmitry pts/2 B.B.B.B 15:46 1:15 0.18s 0.01s sshd: dmitry [priv]
Where A.A.A.A - primary interface of our server, and B.B.B.B - secondary interface.
[ Parent | Reply to this comment ]
Our vpn connections drop without warning, and messenger logs out/in over and over.
using routes-2.6.22-15.diff
[ Parent | Reply to this comment ]
I had a similar problem when using IMs like MSN: every time that my server flushed the routes, my users had to reauthenticate their MSN clients because the new connection used a different source ip from the previous one.
In my case, the addition of static routes was enough to solve this problem.
[ Parent | Reply to this comment ]
Hi! Very useful article, but I have one small question.
Here you describe how to setup a kind of load balancing with string
ip route add default scope global nexthop via 1.0.0.2 dev eth1 weight 1 nexthop via 2.0.0.2 eth2 weight 1
using weight 1 for both interfaces. But if I want to use second link only when my first link is down? Even I type 100 for second weight ( and what is the maximum number to weight here? ), some traffic will go via second link although first link is up.
What modifications must I do to use second link only as failover, without load balancing?
[ Parent | Reply to this comment ]
Also, if I change my external interface settings like you wrote, I cant ping my external interface from my internal network.
For example, my external ip is 1.2.3.4 (eth0) and my internal network is 192.168.0.0/24 (eth1). I use shorewall to setup masquerading and hosts from 192.168.0.x cant ping 1.2.3.4. Why?
[ Parent | Reply to this comment ]