Routing for multiple uplinks

Posted by sphaero on Mon 27 Mar 2006 at 08:25

Debian suits perfectly for use as a gateway for computers on your LAN. However once bandwidth usage grows it could be handy to just add another internet uplink to your gateway. Debian does not cater for this out of the box so this document describes how to setup your debian gateway for multiple uplinks.

The whole setup is based on the Linux Advanced Routing & Traffic Control HOWTO http://lartc.org/howto/

Required packages

The most important and only package we need is `iproute`

apt-get install iproute

The iproute package is a tool to talk to more advanced routing capabilities of the linux kernel. I suggest you read the man page and the howto mentioned above. Some basic theory of iproute is that it does it routing through tables just like iptables does. It determines what table to route through based on rules you define.

Setup

As an example I have 2 uplinks to the internet. The first uplink device eth1 has ipaddress 1.0.0.1/24 and gateway 1.0.0.2. The second uplink device eth2 has ip 2.0.0.1/24 and gateway 2.0.0.2.

First we will define the tables. We do this by editing /etc/iproute2/rt_tables which looks like this:

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
In this file we append a new table name with an unique number for every uplink.
#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
200     uplink1
201     uplink2
We can now focus on setting up the interfaces file. Setup /etc/network/interfaces like this.
auto lo
iface lo inet loopback
auto eth0
#LAN interface
iface eth0 inet static
        address 10.0.0.1
        netmask 255.255.255.0
#Uplink1
auto eth1
iface eth1 inet static
        address 1.0.0.1
        netmask 255.255.255.0
        post-up ip route add 1.0.0.2/32 dev eth1 src 1.0.0.1 table uplink1
        post-up ip route add default via 1.0.0.2 table uplink1
        post-up ip rule add from 1.0.0.1 table uplink1
        post-down ip rule del from 1.0.0.1 table uplink1
#Uplink2
auto eth2
iface eth2 inet static
        address 2.0.0.1
        netmask 255.255.255.0
        post-up ip route add 2.0.0.2/32 dev eth2 src 2.0.0.1 table uplink2
        post-up ip route add default via 2.0.0.2 table uplink2
        post-up ip rule add from 2.0.0.1 table uplink2
        post-down ip rule del from 2.0.0.1 table uplink2
You are now ready to bring the interfaces up.
ifup -a
There is no default gateway at this moment. We want to balance traffic over both uplinks. The following command will set this up.
ip route add default scope global nexthop via 1.0.0.2 dev eth1 weight 1 nexthop via 2.0.0.2 eth2 weight 1
As stated in the howto mentioned above "The weight parameters can be tweaked to favor one provider over the other".

We have now setup our gateway for multiple uplinks. An other benefit of this setup is that the gateway responds through the same uplink as external traffic originated from.

Final thoughts

Remember the balancing of the uplinks is route based and routes are cached. You will not get double bandwidth when downloading a file. You can flush the route cache with ‘ip route flush cache'. The uplink balancing is not perfect but to get a better solution you need to patch and recompile the kernel. Have a look at the patches provided by Julian Anastatov. http://www.ssi.bg/~ja/#routes

 

 


Posted by Anonymous (194.185.xx.xx) on Mon 27 Mar 2006 at 08:33
Yes, but when a failure occour on one line, whats happen? Is there a solution?

[ Parent | Reply to this comment ]

Posted by Utumno (211.22.xx.xx) on Mon 27 Mar 2006 at 09:24
[ Send Message | View Utumno's Scratchpad | View Weblogs ]
All connections carried by the failed line will perish, but new ones will automatically use the other line.

[ Parent | Reply to this comment ]

Posted by nix (217.174.xx.xx) on Mon 27 Mar 2006 at 11:21
[ Send Message ]
Yes, work fine for me, you can try it too;)
http://www.ssi.bg/~ja/#routes

[ Parent | Reply to this comment ]

Posted by Stash (87.116.xx.xx) on Fri 29 Dec 2006 at 14:36
[ Send Message ]
OK, it works but when one line is down the change it is too slow , and the cache for the route is still there and when I want this Host again the old route is through from the "down" line, How can i clear the cache?

[ Parent | Reply to this comment ]

Posted by Anonymous (85.159.xx.xx) on Mon 13 Oct 2008 at 17:22
i resolved this problem with a small bash-script. look at http://smok.zoxt.net/blog/#13 (russian, but all you need is script code, written in bash, not russian)

[ Parent | Reply to this comment ]

Posted by Anonymous (193.138.xx.xx) on Mon 27 Mar 2006 at 12:25
If you wan't a nice and straightforward but still advanced firewall solution i recommend shorewall, it supports multiple uplinks and QoS out-of-the-box (and much much more). http://www.shorewall.net

[ Parent | Reply to this comment ]

Posted by bezell (70.88.xx.xx) on Mon 27 Mar 2006 at 21:57
[ Send Message ]
Not Linux, but my favorites if you have a box lying around. Both come in live CDs if you don't want to install to a drive/cf card. Slick web-based administration along with great feature sets:

m0n0wall - http://www.m0n0.ch/wall - geared toward smaller, embedded systems
pfsense - http://www.pfsense.com -new project in the idea of m0n0wall, aims to be enterprise level

[ Parent | Reply to this comment ]

Posted by Anonymous (201.40.xx.xx) on Sat 15 Apr 2006 at 04:57
Where can I find the rule to do it with Shorewall? I could not find in the Shorewall site.

[ Parent | Reply to this comment ]

Posted by meeas (64.213.xx.xx) on Sat 22 Apr 2006 at 06:34
[ Send Message ]
Shorewall is great, but its multiple links aren't dynamic, meaning if one link fails, all traffic that is assigned to that port won't transfer to the other port. This solution fixes that.

[ Parent | Reply to this comment ]

Posted by Anonymous (68.165.xx.xx) on Tue 28 Mar 2006 at 20:28
Where would you put the last ip route add default gateway command so that Debian starts it up automatically?

If you define the interfaces in /etc/interfaces/network, they're started automatically as needed, but AFAIK, that unusual ip route command won't be done automatically...

[ Parent | Reply to this comment ]

Posted by Steve (82.41.xx.xx) on Tue 28 Mar 2006 at 22:36
[ Send Message | View Steve's Scratchpad | View Weblogs ]

A small script inside /etc/network/if-up.d ?

Steve

[ Parent | Reply to this comment ]

Posted by Anonymous (84.45.xx.xx) on Wed 29 Mar 2006 at 20:55
Why not a "post-up" like the others in the example.

It could be added to either, or both, interfaces I assume.

If it doesn't play nice when only one is up, write a simple script.

I'm a big fan of doing all your basic networking config in the one file, makes figuring it out later, or by someone else, much easier.

[ Parent | Reply to this comment ]

Posted by Anonymous (220.245.xx.xx) on Tue 17 Apr 2007 at 13:08
I've just tried with the latest 2.6.20 series kernel, the result is intermittent.

turned off the CACHED_ROUTING flag & compiled/installed kernel.

first thing I found was to change the order of the ethernet cards.
eth0 and eth1 are the wan ports,
eth2 is the lan port.

this lets me put the last 'route add' command in a post-up on eth2, and eth2 comes up on boot, as opposed to failing, cause the post up fails.

to test it I am pulling the cable out of eth0 and eth1, and I'm finding that most of the time the routing wont switch across. particulary it switches from eth1 to eth0 ok, but not the other way, and even doing 'ip route flush cache' doesnt do it for me.

going to try the patches http://www.ssi.bg/~ja/#routes on my 2.6.20.whatever it was kernel.




[ Parent | Reply to this comment ]

Posted by Anonymous (203.217.xx.xx) on Wed 18 Apr 2007 at 01:37
its 2.6.20.7 kernel, fwiw.

applied said patches, now the 'pull the cable' test results in about 3 minutes of no connection to established route, then it cuts across. 3 minutes of outage is acceptable for what I'm doing, its heaps better than say 3 days of no internet 8^(.

if anyone knows how to reduce the 3 minutes setting, it'd be good to know


[ Parent | Reply to this comment ]

Posted by Wayne (82.144.xx.xx) on Tue 28 Mar 2006 at 22:36
[ Send Message | View Weblogs ]
Hi All

We have been using a similar system for a couple of years now and have very good results with 5 ADSL lines connected to 1 Debian router, also as we are the ISP we can do this at both ends of the connection which enables us to balance the traffic in both directions and have fail over :-)

For dead gateway detection we are using ping (working on using SNMP) on the customer CPE and PPP/radius on the LNS. We are running Soekris boxes at the customer end which support upto 7 interfaces.

Shorewall has a feature which will start scripts after it has setup the firewall, it's ideal for running iproute scripts.

Regards

Wayne

[ Parent | Reply to this comment ]

Posted by Anonymous (85.187.xx.xx) on Sun 17 Sep 2006 at 14:03
When a ppp device disappears from the network devices' list the routing table infromation changes because of the lack of this device. For example if I had a nexthop via XXX.XXX.XXX.XXX dev ppp0 when this device goes down the entire row in the routing table disappears? Any ideas how to handle this?

Thanks.

[ Parent | Reply to this comment ]

Posted by Wayne (82.144.xx.xx) on Mon 18 Sep 2006 at 21:45
[ Send Message | View Weblogs ]
Hi,

We have a script placed in /etc/ppp/ip-down.d/ which checks to see which interfaces are up or down and removes the routes to suit. We also use this script on ppp-up.


Wayne











[ Parent | Reply to this comment ]

Posted by shavenwarthog (24.130.xx.xx) on Tue 18 Apr 2006 at 06:42
[ Send Message ]
If, at the last command, you get "RTNETLINK answers: File exists", that means you need IP_ROUTE_MULTIPATH in your kernel. Generally the kernel lets you have only one default route. This option lets you route across two devices.

To see if that option is set, try the "egrep MULTIPATH /boot/config-`uname -r`" command.

[ Parent | Reply to this comment ]

Posted by Anonymous (200.232.xx.xx) on Wed 8 Nov 2006 at 10:27
MY LINUX KERNEL IS SETUP FOR IP_ROUTE_MULTIPATH.. THE OPTION IS YES..BUT I CONTINUED RECEAVED THE MSG "RTNETLINK answers: File exists". WHAT´S WRONG? I HAVE THE UBUNTU DAPPER DRAKE KERNEL 2.6.15.26-server.Please i need any help

[ Parent | Reply to this comment ]

Posted by prepaid (68.229.xx.xx) on Sat 27 May 2006 at 04:25
[ Send Message ]
So I'm trying to set this up on a Debian Etch box running 2.6.15

I've followed these instructions along with the ones at : http://lartc.org/howto/lartc.rpdb.multiple-links.html and I haven't been able to get dual wan out going to work.

If I set a single wan it'll work fine without any issues, but with dual I can't get it to deal with multiple default gateways, it will always only pick the second on specified.

Any ideas?

[ Parent | Reply to this comment ]

Posted by Anonymous (208.186.xx.xx) on Mon 17 Jul 2006 at 05:13
I think there is an error in the how to.

This statement...

ip route add default scope global nexthop via 1.0.0.2 dev eth1 weight 1 nexthop via 2.0.0.2 eth2 weight 1

Should read...

ip route add default scope global nexthop via 1.0.0.2 dev eth1 weight 1 nexthop via 2.0.0.2 dev eth2 weight 1

add "dev" in front of eth2, this may help.

Also, it might be good to note, if you are using other ip appresses than the one in this how to both the ip and the gateway need to be consecutive. ie: 192.168.0.1 and 192.168.0.2 not 192.168.0.1 and 192.168.0.254.

-- Just my .02.

[ Parent | Reply to this comment ]

Posted by alanrr (189.2.xx.xx) on Wed 3 Jan 2007 at 12:00
[ Send Message ]
Hi

I am running in teh same problem. Did you find a solution???

[ Parent | Reply to this comment ]

Posted by Anonymous (195.204.xx.xx) on Thu 31 Aug 2006 at 11:38
That

"ip route add default scope global nexthop via 1.0.0.2 dev eth1 weight 1 nexthop via 2.0.0.2 eth2 weight 1"

at the end is supposed to be a part of network file with "post-up" in front on eth1/eth2 definition or in a separate rc.local file?

[ Parent | Reply to this comment ]

Posted by sphaero (194.121.xx.xx) on Thu 31 Aug 2006 at 12:50
[ Send Message | View Weblogs ]
Whatever suits your needs but if your needs are just static interfaces you could add the command to a if-up.d script.

[ Parent | Reply to this comment ]

Posted by Anonymous (195.204.xx.xx) on Thu 31 Aug 2006 at 14:05
Any chance you could check out my problem (http://hosting-irc.net/iproute.txt) and give me a comment or two? Would be really appreciated :-) I can also be reached through iprouter@hosting-irc.net.

[ Parent | Reply to this comment ]

Posted by Anonymous (200.139.xx.xx) on Wed 18 Apr 2007 at 13:43
Hello, Nice and helpful article! But I have a question... in my case, I have 1 ethernet (HDSL fixed ip...dns..gw) and 1 pppoe (ppp0 ADSL dinamic ip...dns...gw..) links, the first link goes like you post... but the second I dont know what to do... The "interfaces" part for ADSL is:
auto dsl-provider
iface dsl-provider inet ppp
        pre-up /sbin /ifconfig eth1 up # line maintained  ;by pppoeconf
        provider dsl -provider
Tnks and best regards

[ Parent | Reply to this comment ]

Posted by sphaero (83.160.xx.xx) on Wed 18 Apr 2007 at 13:52
[ Send Message | View Weblogs ]
Lookat the man pages for ppp and the example scripts in /etc/ppp/ip-(up|down).d/
Those should be helpful

[ Parent | Reply to this comment ]

Posted by Anonymous (80.218.xx.xx) on Thu 20 Sep 2007 at 13:35
I've got the same setup, except mine gets its IP by DHCP:

iface eth2 inet dhcp

Is there a clever way to obtain the IP once it is set, or do I need to write a script in /etc/network/if-up.d to set the routes?

[ Parent | Reply to this comment ]

Posted by biffant (194.106.xx.xx) on Thu 5 Jul 2007 at 14:07
[ Send Message ]
Hi!

Thanks for good article, our primary gateway with two optical uplinks works on Sarge, load balancing for default gateway was set as you have suggested.

But now there is some problem - all connections that have permanent outcoming streams (SSH, ICQ, Internet radio) broke several times during a hour, because server starts sending data through second interface

For example, with SSH connections, after reconnect with new session, I see:

dmitry@servername:~$ w
17:01:54 up 14 days, 5:49, 5 users, load average: 0,19, 0,14, 0,10
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
dmitry pts/0 A.A.A.A 10:28 1:19 0.15s 0.04s sshd: dmitry [priv]
dmitry pts/1 A.A.A.A 10:30 2:34 0.51s 0.05s sshd: dmitry [priv]
dmitry pts/2 B.B.B.B 15:46 1:15 0.18s 0.01s sshd: dmitry [priv]

Where A.A.A.A - primary interface of our server, and B.B.B.B - secondary interface.

[ Parent | Reply to this comment ]

Posted by Anonymous (220.245.xx.xx) on Wed 3 Oct 2007 at 05:46
yes I am having the same result.
Our vpn connections drop without warning, and messenger logs out/in over and over.
using routes-2.6.22-15.diff

[ Parent | Reply to this comment ]

Posted by alvesfonseca (164.41.xx.xx) on Mon 22 Oct 2007 at 14:27
[ Send Message ]
Sorry about my poor english.

I had a similar problem when using IMs like MSN: every time that my server flushed the routes, my users had to reauthenticate their MSN clients because the new connection used a different source ip from the previous one.

In my case, the addition of static routes was enough to solve this problem.

[ Parent | Reply to this comment ]

Posted by Anonymous (194.50.xx.xx) on Tue 14 Oct 2008 at 12:51
can you show in details solution for IMs?

[ Parent | Reply to this comment ]

Posted by Anonymous (164.41.xx.xx) on Tue 14 Oct 2008 at 14:17
I added a set of lines like this
; post-up ip route add 207.68.128.0/18 via MYGW1IP
after these
;iface eth0 inet static
; address MYIP1
; netmask MYNM1
; post-up ip route add MYGW1IP dev eth0 src MYIP1 table uplink1
; post-up ip route add default via MYGW1 table uplink1
and before
; post-up ip rule add from MYIP1 table uplink1
; post-down ip rule del from MYIP1 table uplink1
in my /etc/network/interfaces.

The static route presented above points to a network owned by M$ Corp. I suggest you to look for another ip ranges used by your IMs and set static routes to them in other to avoid connection drops every time the system flushes the routes. This can also be useful when accessing sites when authentication depends also on client IP.

[ Parent | Reply to this comment ]

Posted by amadis (195.19.xx.xx) on Sat 13 Oct 2007 at 12:05
[ Send Message ]

Hi! Very useful article, but I have one small question.
Here you describe how to setup a kind of load balancing with string

ip route add default scope global nexthop via 1.0.0.2 dev eth1 weight 1 nexthop via 2.0.0.2 eth2 weight 1

using weight 1 for both interfaces. But if I want to use second link only when my first link is down? Even I type 100 for second weight ( and what is the maximum number to weight here? ), some traffic will go via second link although first link is up.

What modifications must I do to use second link only as failover, without load balancing?

[ Parent | Reply to this comment ]

Posted by Anonymous (81.88.xx.xx) on Fri 8 Aug 2008 at 09:25
Maybe the traffic, which goes via second link belongs to the subnet, (auto-)configured on a second network?

[ Parent | Reply to this comment ]

Posted by amadis (195.19.xx.xx) on Mon 15 Oct 2007 at 12:11
[ Send Message ]

Also, if I change my external interface settings like you wrote, I cant ping my external interface from my internal network.

For example, my external ip is 1.2.3.4 (eth0) and my internal network is 192.168.0.0/24 (eth1). I use shorewall to setup masquerading and hosts from 192.168.0.x cant ping 1.2.3.4. Why?

[ Parent | Reply to this comment ]

Posted by Anonymous (194.50.xx.xx) on Tue 14 Oct 2008 at 12:43
You need to add some routes as described in LARTC

[ Parent | Reply to this comment ]

Posted by Anonymous (200.186.xx.xx) on Wed 16 Jul 2008 at 17:04
I have a load balance with four interfaces, and when stat a download of size relatively big, this
file is interrupted any moment. I believe that the load balance change this is connection for other
interface, however this is not correct ok.

what´s happening.

[ Parent | Reply to this comment ]

Posted by Anonymous (66.166.xx.xx) on Thu 29 Jul 2010 at 00:17
I have seen the exact same problem when using kernel 2.6.27.45 on an i386 platform. On the LAN side of the computer is an FTP server, and PCs try to get files from it on the WAN side which is load-balanced. The FTP server was given a public IP on the WAN side using masquerading rules in 'iptables'.

Transferring a large file from the FTP server revealed that after some data has been transmitted (roughly 14-15k), the packets for the data transmission will start coming out of the other load balanced interface. This is when the FTP transfer fails.

Using 'tcpdump' on both WAN interfaces (named wan0 and wan1) revealed that when traffic stops coming from wan0 and starts coming from wan1, the source address is still that of wan0.

I think the source address is wrong because the load-balanced routing decision is made before the POSTROUTING chain is hit in the NAT table.

How to simultaneously use masquerading and load-balancing remains a mystery to me.

[ Parent | Reply to this comment ]

Posted by Anonymous (213.240.xx.xx) on Tue 12 Aug 2008 at 13:52

Here is a small script that monitors the WAN interfaces and adjusts the default route accordingly. If any interface is down, it is removed from the default route. When it comes back, it is added again. I run it each minute via cron.

Example:

multir --iface eth1:1.0.0.1:5 --iface eth2:2.0.0.1:6 -- iface eth3:3.0.0.1

--iface's argument is interface name:gateway:weight, weight defaulting to 1.

There are also --debug and --dry-run options.

ftp://ftp.modsoftsys.net/public/multir

[ Parent | Reply to this comment ]

Posted by Anonymous (85.159.xx.xx) on Tue 14 Oct 2008 at 12:48
this script pings gateways, but this is not reliable. i use the same script, but i ping host in the internet.

[ Parent | Reply to this comment ]

Posted by Anonymous (88.73.xx.xx) on Sat 11 Oct 2008 at 19:31
I couldn't make this setup work at all for computers on the local interface without additional masquerading. I did it with Iptables. I have firewall routers that connect to the internet, so I don't need firwalling on that router.

Just using Iproute the computer itself was using the connection, but computers behind it could not. I also did not forget to turn on routing in /etc/sysctl.conf But it still didn't work without iptables.

[ Parent | Reply to this comment ]

Posted by Anonymous (156.34.xx.xx) on Sat 28 Feb 2009 at 05:21
NO DNS connect: Network is unreachable

---rt_tables file
#added these two
200 uplink1
201 uplink2
---resolv.conf file
nameserver 192.168.1.1
nameserver 192.168.0.1
---interfaces file
auto eth0
iface eth0 inet static

address 192.168.1.2
netmask 255.255.255.0
post-up ip route add 192.168.1.0/24 dev eth0 src 192.168.1.2 table uplink1
post-up ip route add default via 192.168.1.1 table uplink1
post-up ip rule add from 192.168.1.2 table uplink1
post-down ip rule del from 192.168.1.2 table uplink1

auto eth1
iface eth1 inet static

address 192.168.0.3
netmask 255.255.255.0
post-up ip route add 192.168.0.0/24 dev eth1 src 192.168.0.3 table uplink2
post-up ip route add default via 192.168.0.1 table uplink2
post-up ip rule add from 192.168.0.3 table uplink2
post-down ip rule del from 192.168.0.3 table uplink2

ip route add default scope global nexthop via 192.168.1.1 dev eth0 weight 1 nexthop via 192.168.0.1 dev eth1 weight 1

[ Parent | Reply to this comment ]

Posted by Anonymous (85.145.xx.xx) on Thu 21 Jan 2010 at 12:59
muggles (http://code.google.com/p/muggles/) for lenny and squeeze works well as a base on which to build things.

[ Parent | Reply to this comment ]

Sign In

Username:

Password:

[Register|Advanced]

 

Flattr

 

Current Poll

Which init system are you using in Debian?






( 1599 votes ~ 7 comments )