Virtual IP addresses with ucarp, for high-availability

Posted by Steve on Tue 18 Dec 2012 at 09:53

If you're interested in high-availability you typically want to use some kind of load-balancer, to divide traffic between a pool of machines. A simpler approach can be to have a pair of hosts each of which is prepared to take over a single virtual IP address - this is what ucarp allows you to do.

The idea behind ucarp is that a small number of hosts will have their own IP address, but each of them is potentially able to grab a "floating" or "virtual" IP address. This address is the one that is then highly-available.

For example you might wish to host a website, and so you'd configure apache upon two hosts, each of which would have identical contents. Using ucarp you'd then configure an IP address which would be live on only one host at the same time. This shared IP address is then the address that Apache would listen upon.

Now having two hosts, and moving one address between them like this, will allow you to survive the outage of a single host, but you must bear in mind that this will only allow you to be more available and not more scalable. (Because obviously the host which isn't the owner of the floating IP will not receive any traffic and will be sat idle.)

As an example we have two hosts here da-db1 and da-db2, we'll configure the floating IP address 1.2.3.4 to be shared between these hosts. The first step is installing the software on both hosts:

~# apt-get install ucarp
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  libpcap0.8
The following NEW packages will be installed:
  libpcap0.8 ucarp
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 165 kB of archives.
After this operation, 504 kB of additional disk space will be used.

Installing the package will allow you to update the /etc/network/interfaces file on both hosts. I've added identical contents to both files. Some guides will tell you that you should set one to be explicitly the master, and one to be the fall-back, but for the use of ucarp to be slightly useful the servers must have identical content. On that basis I'd rather let them fight it out between themselves!

# /etc/network/interfaces
# ..

# The primary network interface
auto eth0
iface eth0 inet static
address  ..
netmask  ..
gateway  ..
  #######################
  # ucarp configuration
  #######################
  ucarp-vid      1
  ucarp-vip      1.2.3.4
  ucarp-password mypass32
  ucarp-advskew  1
  ucarp-advbase  1
  ucarp-master   no

# The carp network interface, on top of eth0
iface eth0:ucarp inet static
        address 1.2.3.4
        netmask 255.255.255.255

There are several variables used here, and in brief they have the following meaning:

ucarp-vid

The identifier for this virtual interface. Increment for each additional ucarp shared IP you have.

ucarp-vip

The shared IP address we're maintaining.

ucarp-password

The password with which the two hosts will communicate (not in plain-text) to determine which is the master.

ucarp-advskew

All things being equal the host with the lowest value here will become the master.

ucarp-advbase

How often, in seconds, the hosts should try to communicate with each other.

Once these settings are in place you can run "/etc/init.d/networking restart" and you should find that one host will receive the new IP address, as an alias.

Bringing the IP address up/down is the result of executing a simple shell script which is located beneath the directory /usr/share/ucarp. You can specify an alternative script like so:

  ..
  ucarp-advbase 1
  ucarp-master no
  ucarp-downscript  /usr/local/bin/vip-down
  ucarp-upscript    /usr/local/bin/vip-up
  ..

The script will be called with the IP to bring up, or down, as the first argument. You can then stop/start daemons as appropriate.

 

 


Posted by rjc (131.111.xx.xx) on Wed 19 Dec 2012 at 13:50
[ Send Message ]
Some guides will tell you that you should set one to be explicitly the master, and one to be the fall-back, but for the use of ucarp to be slightly useful the servers must have identical content. On that basis I'd rather let them fight it out between themselves!

Hi Steve,

I don't think I'm following.

By "the content" doesn't it simply mean that the servers need to be configured exactly the same in every other manner, i.e. web, DB, etc. apart from ucarp?

They won't be identical anyway, i.e. eth0 IP address will be different ;^)

P.S. Good to see your post here, I'm glad d-a.o is still alive.

Cheers,

rjc

[ Parent | Reply to this comment ]

Posted by Steve (46.43.xx.xx) on Wed 19 Dec 2012 at 14:15
[ Send Message | View Steve's Scratchpad | View Weblogs ]

I guess I wasn't clear - but in the example listed where you'd want to have a webserver deployed with a virtual IP the only content that needs to be identical would be the htdocs/ folder.

The rest of the system could be completely different (and in some cases will be, as you suggest the hostname, IP details, etc are going to vary). The reason for keeping the systems "identical" is so that if either host is the master the visitor will have the same content.

There are other things that could be balanced though - for example you might want to have a shared IP as the destination for an SMTP server, so you could rotate one live and reboot the other for kernel upgrades, etc. SMTP is a bad example because clients will always retry sending mail "a bit" before bouncing. But in that situation the only identical thing would need to be the email setup, the list of valid accounts , etc.

I hope that clarifies things!

PS. Yes the site has been stalled for a while, but I'm intending to change that. The next few posts are laying the groundwork for some interesting changes... </tease>

Steve

[ Parent | Reply to this comment ]

Posted by linulin (188.168.xx.xx) on Sun 23 Dec 2012 at 11:15
[ Send Message ]

heartbeat is another utility for creating highly-available services. I have not used ucarp but from the first glance heartbeat looks more flexible.

BTW, if you host several services, (each on its own virtual IP), then some sort of scalability could be achieved by binding them to different physical servers by default. Only in case of failures or scheduled maintenance periods they will end up on the same node.

--
...Bye..Dmitry.

[ Parent | Reply to this comment ]

Posted by Steve (90.220.xx.xx) on Sun 23 Dec 2012 at 14:26
[ Send Message | View Steve's Scratchpad | View Weblogs ]

Yes heartbeat is more capable, and allows you to do more interesting things - ucarp is solely contained with moving a virtual IP around, rather than monitoring services ore doing anything more complex.

That makes ucarp useful for low-overhead use where you're decoupling the service from the IP, and doing other adhoc deployments.

Steve

[ Parent | Reply to this comment ]

Posted by Anonymous (212.120.xx.xx) on Tue 5 Feb 2013 at 11:08

In the last section about the up/down script is this by intention:

ucarp-downscript  /usr/local/bin/vip-up
ucarp-upscript    /usr/local/bin/vip-down

Shouldn't it be the other way round?

ucarp-downscript  /usr/local/bin/vip-down
ucarp-upscript    /usr/local/bin/vip-up

This confuses me a little bit at the moment.

[ Parent | Reply to this comment ]

Posted by Steve (46.43.xx.xx) on Tue 5 Feb 2013 at 11:12
[ Send Message | View Steve's Scratchpad | View Weblogs ]

You're right; those are the wrong way round and I'm surprised nobody else noticed!

I'll fix the article text now.

Steve

[ Parent | Reply to this comment ]

Posted by Larry2 (109.22.xx.xx) on Wed 24 Apr 2013 at 10:44
[ Send Message ]
Hello,

May I ask you if two servers belonging to separate public networks, say 111.222.333 and 222.111.333 could make good use of ucarp ?

Is there any requirement for a third public ip, say 333.222.111 to be shared ?

Config on every nodes :


# /etc/network/interfaces
# ..

# The primary network interface
auto eth0
iface eth0 inet static
address ..
netmask ..
gateway ..
#######################
# ucarp configuration
#######################
ucarp-vid 1
ucarp-vip 333.222.111
ucarp-password mypass32
ucarp-advskew 1
ucarp-advbase 1
ucarp-master no

# The carp network interface, on top of eth0
iface eth0:ucarp inet static
address 333.222.111
netmask 255.255.255.255



Many thanks,

[ Parent | Reply to this comment ]

Posted by Steve (90.193.xx.xx) on Wed 24 Apr 2013 at 10:46
[ Send Message | View Steve's Scratchpad | View Weblogs ]

I'm pretty sure that's not going to work, because ucarp will only broadcast to the local LAN - i.e. it won't get past routers so you can't do it arbitrarily over the internet.You'll want a virtual LAN (VLAN) for this to work.

Steve

[ Parent | Reply to this comment ]

Posted by Larry2 (109.22.xx.xx) on Wed 24 Apr 2013 at 11:41
[ Send Message ]
So do you think a vpn should do fine, connecting the two ip's over the wan ?

[ Parent | Reply to this comment ]

Posted by Steve (90.193.xx.xx) on Wed 24 Apr 2013 at 11:42
[ Send Message | View Steve's Scratchpad | View Weblogs ]

If you have a VPN so you have, say 192.168.0.0/24 routing across two locations AND the VPN transfers multicast/broadcast packets that ucarp will use, then yes that would work.

However you'd then have issues because your floating IP would be in the private VPN-range and you'd have to handle making the service(s) running upon it be visible to the outside network.

Steve

[ Parent | Reply to this comment ]

Posted by Larry2 (109.22.xx.xx) on Wed 24 Apr 2013 at 12:02
[ Send Message ]
yes, because of the different actual ip of the two servers.. right ?
The problem remains the same for a real private network, no ?

Private network :
node A fails, node B failovers but node B has to answer the public request too. So it will answer back to the shared ip.

VPN :
node A fails. node B failovers and answers the client from another ip. Where is the problem ?
It should last too long because of the cpu overhead but...

Any enlightenment ? Did I miss something?

[ Parent | Reply to this comment ]

Posted by Steve (90.193.xx.xx) on Wed 24 Apr 2013 at 12:15
[ Send Message | View Steve's Scratchpad | View Weblogs ]

Yes, because your floating address, 192.168.0.10 for example, won't be routable.

For real failover we assume a private VLAN so there might be:

100.200.200.1 - server 1
100.200.200.2 - server 2
100.200.200.3 - floating IP

In that case if the floating IP moves there's no problem because the routing of 100.200.200.0/24 to the VLAN works. (Obviously if you have a private VLAN it is going to be something like a /29, or a /27, rather than a /24, but the principle is the same. Random visitors from the internet will know how to route to you via the global router table.)

What you're missing is that if you put your service on the private floating IP how will people outside the VPN reach it?

Steve

[ Parent | Reply to this comment ]

Posted by Anonymous (201.209.xx.xx) on Sat 11 Jan 2014 at 22:32
I try and I try but the content in interfaces is ignored. ucarp wants the variables in the command line. Also, in daemon mode it exits when I do "ip link set eth0 down", so I have to restart it again.

[ Parent | Reply to this comment ]

Posted by Steve (90.220.xx.xx) on Sat 11 Jan 2014 at 23:25
[ Send Message | View Steve's Scratchpad | View Weblogs ]

I could never make it work with more than one IP address, unless I used the command-line rather than the interfaces file.

But that quirk aside it seemed to be reliable to me. Have you considered reporting a bug, or asking for assistance on the debian-users mailing list?

Steve

[ Parent | Reply to this comment ]

Posted by Anonymous (201.209.xx.xx) on Sun 12 Jan 2014 at 06:03
I might, thank you. I'm using lubuntu and debian stable for the tests, I'll switch to stable and testing, and to stable - stable just in case. Another thing, the primary (by skew value) doesn't return to be primary after I turn it on again. I'm using VMs.

[ Parent | Reply to this comment ]

Posted by Anonymous (201.209.xx.xx) on Sun 12 Jan 2014 at 08:49
It works as expected if I turn off the primary VM (the secondary answers the queries). If I turn on the primary VM it becomes the primary server answering the queries. If I stop the networking or eth0 ucarp stops working. If I start ucarp again it doesn't work as expected, as if it can't find the other ucarp instance in the other server, even though both have the same vhid. I'll try with two real machines, maybe it's the VMs. Bye.

[ Parent | Reply to this comment ]

Sign In

Username:

Password:

[Register|Advanced]

 

Flattr

 

Current Poll

Which init system are you using in Debian?






( 1068 votes ~ 7 comments )