Cloning a Debian system - identical packages and versions

Posted by telent on Wed 15 Jun 2011 at 19:14

I have long wished for the ability to install a Debian system specifying both the package list and the exact versions of the packages installed. dpkg --get-selections | ssh newhost dpkg --set-selections is useful but always chooses the latest version, and especially when using testing or unstable, it is sometimes necessary to temporarily downgrade a package from the latest version available.
Plus, it's just good software engineering practice to know the exact components that went into your build and to be able to replicate it.

So when I recently had to set up a new test server for $WORK, I finally knuckled down and figured out the necessary incantations to make it work. The first part of the puzzle is getting the versions right:

$ cat save-package-versions
aptitude -q -F "%?p=%?V %M" --disable-columns search \~i  
$ cat restore-package-versions
aptitude -q -R --schedule-only install $(awk < $list '{print $1}')
aptitude -q -R --schedule-only markauto $(awk < $list '$2=="A" {split($1,A,"=");print A[1]}')

The first script produces a list of installed packages and their versions, followed by an A if the package was installed automatically: the second parses that list to reinstall the desired packages and mark the dependencies as automatic. After running the restore script you will want to call aptitude -y -o Dpkg::Options::="--force-confdef" install

The second part of the puzzle is to save the debconf questions: debconf-get-selections and debconf-set-selections are your friends here

After that you might get carried away: I integrated these into a Makefile-based system which uses debootstrap and sfdisk and dd and loopback mounts and extlinux to automate the production of bootable kvm/qemu images which contain the specified versions of the specified packages

[... actually, I simplify - for "contain", please read "contain scripts which install these packages when they are first booted, to accommodate the probability that they are running somewhere with better bandwidth than the host on which the image was originally created" ...]

along with any other files I wanted to put in there like /etc/passwd, /etc/shadow or /home/appuser/.ssh/authorized_keys. And then I put it on Github - but this is a sysadmin forum not a software announcement forum so that's all I'll say about that

Except this: if you've ever wanted to boot an ext{2,3,4} system in a way that the wrapper scripts around Grub won't accommodate and then thrown your hands in the air at the prospect of configuring it by hand, go and look at Extlinux. It's simple and it's documented and it deserves to be better known.



Posted by sebrem (92.201.xx.xx) on Wed 15 Jun 2011 at 21:50
Hi, there's a new package in sid, apt-clone. From the description: "This package can be used to clone/restore the packages on a apt based system. It will save/restore the packages, sources.list, keyring and automatic-installed states. It can also save/restore no longer downloadable packages using dpkg-repack." Homepage is - haven't tried it yet, but will soon.

[ Parent | Reply to this comment ]

Posted by Anonymous (75.18.xx.xx) on Thu 16 Jun 2011 at 09:06
But if you're running anything else than stable and you need a different
version than the one currently in the pool, you're still SOL, aren't
you? That happened to me all the time until I set up my own little
package repo.

[ Parent | Reply to this comment ]

Posted by Anonymous (151.71.xx.xx) on Thu 16 Jun 2011 at 09:27
Another solution to this problem would be to use snapshot.d.o and build a apt sourcelist that has a specific entry for every package version you wish to install on your system. The problem here is that you need to create this list. You might want to to use the snapshot api to do so, that afaik allows you to query a package by version and retrieve the url of the package ... 30 lines of python should solve the problem... :)

[ Parent | Reply to this comment ]

Posted by telent (82.68.xx.xx) on Thu 16 Jun 2011 at 10:13

Yes, good point. In that scenario I'd think about using an apt-cacher proxy and setting clean_cache=0 in its config.

[ Parent | Reply to this comment ]

Posted by mcortese (20.142.xx.xx) on Thu 16 Jun 2011 at 16:14
[ View Weblogs ]
So what's the value of your original approach? If you're running stable, --get-selection / --set-selection are enough; if you're running testing or unstable, you have to set up something different anyway...

[ Parent | Reply to this comment ]

Posted by telent (82.68.xx.xx) on Sat 18 Jun 2011 at 16:43

Thanks for the question - obviously I wasn't clear in my comment. To build repeatable testing/unstable images you will probably need a local package repository (either a manually maintained one or a cache which is never flushed), but you will still additionally need a way to specify which versions of which packages to install from it. Like this.

And my experience has been that even on stable, --get-selection / --set-selection are not not always enough. Consider: a security hole in package A is found, and a new version is released. You upgrade your test system and make sure it works. While you are doing this, a new version of B is released. When you are happy with testing A you upgrade your live box to match, but now you unexpectedly have dragged in the new B too. If B is something like exim4 and you have a complicated custom configuration that the new version doesn't like, this is not a desirable outcome. Yes, you would probably have to upgrade it anyway, but better to find out about the problem in testing than in production ...

[ Parent | Reply to this comment ]

Posted by Anonymous (88.3.xx.xx) on Fri 22 Jul 2011 at 10:29

[ Parent | Reply to this comment ]

Posted by ezequielandrush (190.188.xx.xx) on Wed 27 Jul 2011 at 13:57
Very usefull... thanks very much


[ Parent | Reply to this comment ]

Posted by Anonymous (200.255.xx.xx) on Wed 19 Oct 2011 at 12:56

[ Parent | Reply to this comment ]

Posted by Anonymous (217.153.xx.xx) on Wed 4 Jul 2012 at 15:59
Unfortunately aptitude will not report problem with not found package in return code...

I think that the better way would be to use "apt-get install":

apt-get -q --no-install-recommends install $(awk < $list '{print $1}') &&
aptitude -q -R markauto $(awk < $list '$2=="A" {split($1,A,"=");print A[1]}');
exit $?;

[ Parent | Reply to this comment ]

Posted by Anonymous (78.105.xx.xx) on Thu 9 Aug 2012 at 02:17
The %?V should rather be %?v as V refers to the installation candidate for a package while v refers to the current version. In most cases the two will be the same unless updates have been held back for whatever reason.

[ Parent | Reply to this comment ]

Posted by Anonymous (108.250.xx.xx) on Sat 31 Jan 2015 at 18:01
I believe the original script was correct and this comment has them reversed. The aptitude man page states:

"passing %p %V %v for <format> will display a package's name, followed by
its currently installed version and its available version"

The script should mark what is installed so it can be recovered later.

[ Parent | Reply to this comment ]

Sign In







Current Poll

Will you stick to systemd as the default in Debian?

( 5 votes ~ 0 comments )