Routing for multiple uplinks
Posted by sphaero on Mon 27 Mar 2006 at 08:25
Debian suits perfectly for use as a gateway for computers on your LAN. However once bandwidth usage grows it could be handy to just add another internet uplink to your gateway. Debian does not cater for this out of the box so this document describes how to setup your debian gateway for multiple uplinks.
The whole setup is based on the Linux Advanced Routing & Traffic Control HOWTO http://lartc.org/howto/
The most important and only package we need is `iproute`
apt-get install iproute
The iproute package is a tool to talk to more advanced routing capabilities of the linux kernel. I suggest you read the man page and the howto mentioned above. Some basic theory of iproute is that it does it routing through tables just like iptables does. It determines what table to route through based on rules you define.
As an example I have 2 uplinks to the internet. The first uplink device eth1 has ipaddress 1.0.0.1/24 and gateway 1.0.0.2. The second uplink device eth2 has ip 2.0.0.1/24 and gateway 2.0.0.2.
First we will define the tables. We do this by editing /etc/iproute2/rt_tables which looks like this:
# # reserved values # 255 local 254 main 253 default 0 unspec # # local # #1 inr.ruhepIn this file we append a new table name with an unique number for every uplink.
# # reserved values # 255 local 254 main 253 default 0 unspec # # local # #1 inr.ruhep 200 uplink1 201 uplink2We can now focus on setting up the interfaces file. Setup /etc/network/interfaces like this.
auto lo
iface lo inet loopback
auto eth0
#LAN interface
iface eth0 inet static
address 10.0.0.1
netmask 255.255.255.0
#Uplink1
auto eth1
iface eth1 inet static
address 1.0.0.1
netmask 255.255.255.0
post-up ip route add 1.0.0.2/32 dev eth1 src 1.0.0.1 table uplink1
post-up ip route add default via 1.0.0.2 table uplink1
post-up ip rule add from 1.0.0.1 table uplink1
post-down ip rule del from 1.0.0.1 table uplink1
#Uplink2
auto eth2
iface eth2 inet static
address 2.0.0.1
netmask 255.255.255.0
post-up ip route add 2.0.0.2/32 dev eth2 src 2.0.0.1 table uplink2
post-up ip route add default via 2.0.0.2 table uplink2
post-up ip rule add from 2.0.0.1 table uplink2
post-down ip rule del from 2.0.0.1 table uplink2
You are now ready to bring the interfaces up.
ifup -aThere is no default gateway at this moment. We want to balance traffic over both uplinks. The following command will set this up.
ip route add default scope global nexthop via 1.0.0.2 dev eth1 weight 1 nexthop via 2.0.0.2 eth2 weight 1As stated in the howto mentioned above "The weight parameters can be tweaked to favor one provider over the other".
We have now setup our gateway for multiple uplinks. An other benefit of this setup is that the gateway responds through the same uplink as external traffic originated from.
Remember the balancing of the uplinks is route based and routes are cached. You will not get double bandwidth when downloading a file. You can flush the route cache with ‘ip route flush cache'. The uplink balancing is not perfect but to get a better solution you need to patch and recompile the kernel. Have a look at the patches provided by Julian Anastatov. http://www.ssi.bg/~ja/#routes
[ Send Message | View Utumno's Scratchpad | View Weblogs ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
m0n0wall - http://www.m0n0.ch/wall - geared toward smaller, embedded systems
pfsense - http://www.pfsense.com -new project in the idea of m0n0wall, aims to be enterprise level
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
If you define the interfaces in /etc/interfaces/network, they're started automatically as needed, but AFAIK, that unusual ip route command won't be done automatically...
[ Parent | Reply to this comment ]
[ Send Message | View Steve's Scratchpad | View Weblogs ]
A small script inside /etc/network/if-up.d ?
[ Parent | Reply to this comment ]
It could be added to either, or both, interfaces I assume.
If it doesn't play nice when only one is up, write a simple script.
I'm a big fan of doing all your basic networking config in the one file, makes figuring it out later, or by someone else, much easier.
[ Parent | Reply to this comment ]
turned off the CACHED_ROUTING flag & compiled/installed kernel.
first thing I found was to change the order of the ethernet cards.
eth0 and eth1 are the wan ports,
eth2 is the lan port.
this lets me put the last 'route add' command in a post-up on eth2, and eth2 comes up on boot, as opposed to failing, cause the post up fails.
to test it I am pulling the cable out of eth0 and eth1, and I'm finding that most of the time the routing wont switch across. particulary it switches from eth1 to eth0 ok, but not the other way, and even doing 'ip route flush cache' doesnt do it for me.
going to try the patches http://www.ssi.bg/~ja/#routes on my 2.6.20.whatever it was kernel.
[ Parent | Reply to this comment ]
applied said patches, now the 'pull the cable' test results in about 3 minutes of no connection to established route, then it cuts across. 3 minutes of outage is acceptable for what I'm doing, its heaps better than say 3 days of no internet 8^(.
if anyone knows how to reduce the 3 minutes setting, it'd be good to know
[ Parent | Reply to this comment ]
We have been using a similar system for a couple of years now and have very good results with 5 ADSL lines connected to 1 Debian router, also as we are the ISP we can do this at both ends of the connection which enables us to balance the traffic in both directions and have fail over :-)
For dead gateway detection we are using ping (working on using SNMP) on the customer CPE and PPP/radius on the LNS. We are running Soekris boxes at the customer end which support upto 7 interfaces.
Shorewall has a feature which will start scripts after it has setup the firewall, it's ideal for running iproute scripts.
Regards
Wayne
[ Parent | Reply to this comment ]
Thanks.
[ Parent | Reply to this comment ]
We have a script placed in /etc/ppp/ip-down.d/ which checks to see which interfaces are up or down and removes the routes to suit. We also use this script on ppp-up.
Wayne
[ Parent | Reply to this comment ]
To see if that option is set, try the "egrep MULTIPATH /boot/config-`uname -r`" command.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
I've followed these instructions along with the ones at : http://lartc.org/howto/lartc.rpdb.multiple-links.html and I haven't been able to get dual wan out going to work.
If I set a single wan it'll work fine without any issues, but with dual I can't get it to deal with multiple default gateways, it will always only pick the second on specified.
Any ideas?
[ Parent | Reply to this comment ]
This statement...
ip route add default scope global nexthop via 1.0.0.2 dev eth1 weight 1 nexthop via 2.0.0.2 eth2 weight 1
Should read...
ip route add default scope global nexthop via 1.0.0.2 dev eth1 weight 1 nexthop via 2.0.0.2 dev eth2 weight 1
add "dev" in front of eth2, this may help.
Also, it might be good to note, if you are using other ip appresses than the one in this how to both the ip and the gateway need to be consecutive. ie: 192.168.0.1 and 192.168.0.2 not 192.168.0.1 and 192.168.0.254.
-- Just my .02.
[ Parent | Reply to this comment ]
I am running in teh same problem. Did you find a solution???
[ Parent | Reply to this comment ]
"ip route add default scope global nexthop via 1.0.0.2 dev eth1 weight 1 nexthop via 2.0.0.2 eth2 weight 1"
at the end is supposed to be a part of network file with "post-up" in front on eth1/eth2 definition or in a separate rc.local file?
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
auto dsl-provider
iface dsl-provider inet ppp
pre-up /sbin /ifconfig eth1 up # line maintained ;by pppoeconf
provider dsl -provider
[ Parent | Reply to this comment ]
Those should be helpful
[ Parent | Reply to this comment ]
iface eth2 inet dhcp
Is there a clever way to obtain the IP once it is set, or do I need to write a script in /etc/network/if-up.d to set the routes?
[ Parent | Reply to this comment ]
Thanks for good article, our primary gateway with two optical uplinks works on Sarge, load balancing for default gateway was set as you have suggested.
But now there is some problem - all connections that have permanent outcoming streams (SSH, ICQ, Internet radio) broke several times during a hour, because server starts sending data through second interface
For example, with SSH connections, after reconnect with new session, I see:
dmitry@servername:~$ w
17:01:54 up 14 days, 5:49, 5 users, load average: 0,19, 0,14, 0,10
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
dmitry pts/0 A.A.A.A 10:28 1:19 0.15s 0.04s sshd: dmitry [priv]
dmitry pts/1 A.A.A.A 10:30 2:34 0.51s 0.05s sshd: dmitry [priv]
dmitry pts/2 B.B.B.B 15:46 1:15 0.18s 0.01s sshd: dmitry [priv]
Where A.A.A.A - primary interface of our server, and B.B.B.B - secondary interface.
[ Parent | Reply to this comment ]
Our vpn connections drop without warning, and messenger logs out/in over and over.
using routes-2.6.22-15.diff
[ Parent | Reply to this comment ]
I had a similar problem when using IMs like MSN: every time that my server flushed the routes, my users had to reauthenticate their MSN clients because the new connection used a different source ip from the previous one.
In my case, the addition of static routes was enough to solve this problem.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
; post-up ip route add 207.68.128.0/18 via MYGW1IP
after these
;iface eth0 inet static
; address MYIP1
; netmask MYNM1
; post-up ip route add MYGW1IP dev eth0 src MYIP1 table uplink1
; post-up ip route add default via MYGW1 table uplink1
and before
; post-up ip rule add from MYIP1 table uplink1
; post-down ip rule del from MYIP1 table uplink1
in my /etc/network/interfaces.
The static route presented above points to a network owned by M$ Corp. I suggest you to look for another ip ranges used by your IMs and set static routes to them in other to avoid connection drops every time the system flushes the routes. This can also be useful when accessing sites when authentication depends also on client IP.
[ Parent | Reply to this comment ]
Hi! Very useful article, but I have one small question.
Here you describe how to setup a kind of load balancing with string
ip route add default scope global nexthop via 1.0.0.2 dev eth1 weight 1 nexthop via 2.0.0.2 eth2 weight 1
using weight 1 for both interfaces. But if I want to use second link only when my first link is down? Even I type 100 for second weight ( and what is the maximum number to weight here? ), some traffic will go via second link although first link is up.
What modifications must I do to use second link only as failover, without load balancing?
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
Also, if I change my external interface settings like you wrote, I cant ping my external interface from my internal network.
For example, my external ip is 1.2.3.4 (eth0) and my internal network is 192.168.0.0/24 (eth1). I use shorewall to setup masquerading and hosts from 192.168.0.x cant ping 1.2.3.4. Why?
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
file is interrupted any moment. I believe that the load balance change this is connection for other
interface, however this is not correct ok.
what´s happening.
[ Parent | Reply to this comment ]
Transferring a large file from the FTP server revealed that after some data has been transmitted (roughly 14-15k), the packets for the data transmission will start coming out of the other load balanced interface. This is when the FTP transfer fails.
Using 'tcpdump' on both WAN interfaces (named wan0 and wan1) revealed that when traffic stops coming from wan0 and starts coming from wan1, the source address is still that of wan0.
I think the source address is wrong because the load-balanced routing decision is made before the POSTROUTING chain is hit in the NAT table.
How to simultaneously use masquerading and load-balancing remains a mystery to me.
[ Parent | Reply to this comment ]
Here is a small script that monitors the WAN interfaces and adjusts the default route accordingly. If any interface is down, it is removed from the default route. When it comes back, it is added again. I run it each minute via cron.
Example:
multir --iface eth1:1.0.0.1:5 --iface eth2:2.0.0.1:6 -- iface eth3:3.0.0.1
--iface's argument is interface name:gateway:weight, weight defaulting to 1.
There are also --debug and --dry-run options.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
Just using Iproute the computer itself was using the connection, but computers behind it could not. I also did not forget to turn on routing in /etc/sysctl.conf But it still didn't work without iptables.
[ Parent | Reply to this comment ]
---rt_tables file
#added these two
200 uplink1
201 uplink2
---resolv.conf file
nameserver 192.168.1.1
nameserver 192.168.0.1
---interfaces file
auto eth0
iface eth0 inet static
address 192.168.1.2
netmask 255.255.255.0
post-up ip route add 192.168.1.0/24 dev eth0 src 192.168.1.2 table uplink1
post-up ip route add default via 192.168.1.1 table uplink1
post-up ip rule add from 192.168.1.2 table uplink1
post-down ip rule del from 192.168.1.2 table uplink1
auto eth1
iface eth1 inet static
address 192.168.0.3
netmask 255.255.255.0
post-up ip route add 192.168.0.0/24 dev eth1 src 192.168.0.3 table uplink2
post-up ip route add default via 192.168.0.1 table uplink2
post-up ip rule add from 192.168.0.3 table uplink2
post-down ip rule del from 192.168.0.3 table uplink2
ip route add default scope global nexthop via 192.168.1.1 dev eth0 weight 1 nexthop via 192.168.0.1 dev eth1 weight 1
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]