Preparing a machine to instantly deploy in case of a failure?
Posted by bacardi on Tue 22 May 2007 at 09:58
This must be a common problem, but I can't find the canonical solution. We have two identical machines running a few medium size websites. One is a production machine, the other is an identically setup machine used for development. We want to be able to deploy the development machine quickly in case of problems with the primary server - how should we go about that?
Currently, when features on the development server are adequately tested, they are pushed into production.
Due to a tight budget, we do not have a live redundant backup to our production server (other than its RAID 1). One thing we do have is hard drive space - so we've been trying to setup a process where live production data (mysql, apache) is backed up nightly onto the development machine, so if the production server goes down, we can flip the development server into production mode.
This seems to be an extremely dirty method, requiring manually changing a lot of configuration files, questions about whether the development server can actually acquire the production IP, and many other quirks.
I have not completed this process, but there must be a more efficient (standard?) way. I've only been involved with Debian for a little over a year, but have done enough Linux to pick things up quickly.. and after some browsing, haven't run into many cases of easily converting a development machine into production mode.
Any suggestions would be appreciated!
Use rsync to syncronize the following directory on the development server
/etc2/apache2
/var/www2
with the following directory on the production server:
/etc/apache2
/var/www or whatever your Document root is
in case of disaster:
copy from /etc2 to /etc the needed directory
rename /var/www in /var/www.dev and /var/www2 in /var/www
assign the ip of the production server to an alias of eth0
[ Parent | Reply to this comment ]
You can also limit the bandwith to e.g. 120kb/s with the command line option --bwlimit=120
[ Parent | Reply to this comment ]
but when you have thousands of files, rsync needs to compare them all... and that can take some minutes...
[ Parent | Reply to this comment ]
This tool may help sync only the changed files, when they change. You may even get some e-mail triggers to important files, such as configuration files. It is on unstable, but I believe it can be compiled under Etch.
I had similar demands at my work. At that moment, bringing up some high level solution would not suit the time / resources available. A couple of rsyncs and cron scripts to sync MySQL did the job and proved stable and functional.
[ Parent | Reply to this comment ]
However, you need to make a few adjustments on your development server to make this work. For instance, the night database backup has to be restored every night on the development database server (preferable under the same name as your live database) and in apache you need to make an extra VirtualHost defintion for the production website. We (I) use Ultra Monkey (http://www.ultramonkey) as the director.
This is how it works: Ultra Monkey periodically checks your server on IP bases and a special configured webpage which needs to return a predefined response. Using this page you can check depended services such as your database connection, SOAP services or whatever you need. As long the IP is connectable and the webpage returns the predefined response it is considered 'alive'. When a check fails the director will remove the particular (real) server from the pool. When all real servers are removed from the pool, the director will use the sorry (your development server) until (one of) the real server(s) is 'up' again.
This technique is called High Availability.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
I'm not sure if they have Etch packages, but this lines come from my sources.list:
deb http://www.ultramonkey.org/download/3/ sarge main
deb-src http://www.ultramonkey.org/download/3/ sarge main
To prevent a Emacs vs VI discussion, just choose the director which suits your environment the most. Both are very good.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
This does not tackle synchronization problems; but for the connectivity it might be a less involved approach than LinuxVirtualServer (which redirects network traffic on a lower level?) which probably also needs a front node(?). An additional benefit could be to automatically route requests from your development client to your development server.
[ Parent | Reply to this comment ]
Perhaps you need to finish of with a network change or two, but then you're done.
Other than that, there are quite a few teorethical solutions like distributed/network block devices, but none of them proves practical.
(those who disagree: have you really tried this in production?)
One of the promising options is to use a distributed filesystem; but afaik only commercial ones are "usable" on linux right now.
[ Parent | Reply to this comment ]
Cheers.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
Surely somebody's already done this.... Ah, Linux-HA.org recommends DRBD.
While you're at it don't forget to use lvm2.
[ Parent | Reply to this comment ]
What I have done :
- heartbeat to monitor the server availability
- rsync script in a cronjob to sync system files, since you can't use drbd on a root part.
- drbd partition(s) to sync data in real-time. Works great with firewire network ;)
What I have not done but would be cool :
- monitor services with something like mon, since heartbeat basically monitors only network connections.
Hope this help,
cheers,
Luc
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
Mondo will create a bootable DVD that will reinstall your system anywhere you want.
In fact i use this solution in two stage:
* build nightly daily changing data backup using simple genisoimage + wodim on DVD
* build from time to time a system backup without daily changing data using mondo
This quite efficient, if you have 2 DVD+RW and a DVD writer.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
http://oldwiki.linux-vserver.org/Vserver+DRBD
ucarp is another toolkit no one has mentioned
http://xn.pinkhamster.net/blog/tech/mail/high-availability_smtp_w ith_ucarp_on_debian.html
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]