Migrating from one machine to another

Posted by Steve on Sat 25 Jun 2005 at 19:30

When it comes to migrating a server machine, which handles webserving for several domains along with email, DNS, and several other services there is a lot to remember. Since I've just migrated the machine handling this domain I'm going to describe how I did it.

Whilst there's nothing intrinsically difficult about migrating a group of services to a new IP address, and location, there are often a lot of very small steps to take which need concentration.

Writing a plan is a reasonable thing to do - especially if you're planning on upgrading services as part of the migration.

My general plan for the migration was:

1. Copy all data across to the new host, web trees, database contents, etc.
2. Update all DNS entries such that www.* point correctly to the new host.
3. Wait until this propogated, making sure that things were working correctly, and there were no missing Perl modules, etc.
4. Migrate all the MX records to point to the new host - so that mail deliveries go to the new machine, instead of the old.
5. Migrate user accounts. (Passwords + groups, rather than the data).
6. Disable all services on the old host, and take a full backup of the system state.

If you're in a hurry you could migrate both the www, and MX records to the new host all in one step. This would save waiting for two sets of DNS changes to propogate. But the attraction of doing it in two stages is that you get to catch any potential problems without risking mail, which is often more important than anything else.

It's also means that any existing users don't need to be bothered by email changes until the migration is half-done.

In my case the services being migrated were:

• Apache2 virtual hosting for a number of domains.
• Exim3 handling hosting for the virtual domains, which I also planned to upgrade to Exim4.
• DNS services for a couple of domains.

None of these services are terribly difficult to move individually, or even en masse, but there are some potential issues which can cause problems. Almost all the problems are caused by two things:

• Waiting for registrars to update nameserver entries.
• Delays in DNS propogation.
• Hardcoded IP addresses in suprising locations.

The single most important thing in planning a migration is to account for DNS propogation delays.

DNS really is the backbone of the internet without working DNS a lot of things become more difficult to use. If you're not familiar with DNS it's very straightforward in purpose: When you wish to connect to a webserver, or mailserver, you tend to do so by name, eg "www.debian.org". For your computer to connect it needs to know the IP address though, and DNS is nothing more than a mapping of hostnames to IP addresses, and vice-versa.

Because the DNS system is designed to scale and provide reduncency entries aren't updated and available immediately globally. There are caches along the way. (Simplifying things here ....)

This means making updates potentially difficult, when you make a change some clients will resolve your hostname to the old address and some will get the new address. Generally it takes a couple of days for the new IP address to be resolved by all hosts.

For static hosts this doesn't matter too much. You can simply have the two machines, old and new, serving the website.

However when you're running a database-backed dynamic server you can soon run into problems, as updates will be made to two distinct databases.

There are a couple of solutions to this problem:

• If you're worried about data inconsistancy you can switch the old server to connecting to a database on the new host, so there's two front-ends, but only one back-end database.
• If you're worried about there being multiple disparate servers around you can redirect all connections from the old host to the new one via a firewall rule, proxy server, or other means.
• If you have access to the DNS server(s) which are responsible for the domain name in question you can plan for the migration by dropping the TTL field before updating the address.

Migrating the mail handling is similarly problematic, it's easy to end up in a sitation where different hosts are trying to deliver mail to your new and old machine.

Similarly when people connect to your server to receive their mail they may connect to the old one, and not see freshly delivered mail which was addressed to the new host.

My solution to this problem was to use netcat to redirect traffic from the old host to the new one.

The simple introduction to netcat describes how this is done. It's a fairly simple process:

• On the old host disable apache, your mailserver, etc.
• Setup inetd to forward connections.

As a sample the following snippet from my /etc/xinetd.d/smtp file shows how I forwarded incoming SMTP connections to the new host:

service smtp
{
        disable = no
        socket_type     = stream
        protocol        = tcp
        user            = nobody
        wait            = no
        server          = /bin/nc
        server_args     = -w 2 xx.xx.xx.xx 25
}                                               

This causes all incoming connections on port 25 to be seemlessly redirected to the new host - making sure I didn't have to worry about the delay in DNS propogation.

Once you're sure that everything has been migrated properly you can relax for a day or two to make sure things are stable and you've not forgotten anything.

(Note: Don't forget to check that you've migrated cronjobs for all users too - I forgot this for a couple of days when moving things to this server.)

One important thing that you must check is that you don't have configuration files copied across from the old host which have hardcoded references to the old IP address.

A simple way to test for this is a recursive grep:

steve@skx:~$ rgrep xx.xx.xx.xx /etc

For my server I discovered that some of the Apache setup had rules in place which allowed access to particular directories from only the old address - updating this was simple once I'd noticed it.

Similarly sometimes services allow you to specify which addresses they should listen upon. In my case I'd setup a caching proxy server, and it's configuration file /etc/squid/squid.conf included this entry:

http_port xx.xx.xx.xx:8080

Updating this to match the new IP address was a simple job, and allowed the proxy to continue working.

This article can be found online at the Debian Administration website at the following bookmarkable URL:

• http://www.debian-administration.org/articles/169

This article is copyright 2005 Steve - please ask for permission to republish or translate.