Weblog entry #195 for ajt

How to make a qemu image sparse again?
Posted by ajt on Thu 21 Apr 2011 at 16:21
Tags: , ,
Some time ago I added zeros to the end of Qemu/KVM disk images on a Bytemark hosted server to make the images larger. One of the files is still sparse two of them are not.

All of them are about 2G of used space in the VM, but only one of them is a 2G sparse file on the outside, the others don't appear to be sparse any more. I tried copying one of them with "cp --sparse=always" but that didn't help, and fsck of the VM's file system find anything, it really is only 2G in the VM and the remaining 18G is empty space.

Not sure what to do next. If it were not a hosted machine I'd just move the images to a local server and copy the contents into a new disk image and not worry, but as it's hosted I only have so much RAM/disk to play with on the box and moving 200G files over the public internet is impossible.

It's something to keep me occupied anyway.

 

Comments on this Entry

Posted by Steve (82.41.xx.xx) on Sat 23 Apr 2011 at 10:33
[ Send Message | View Steve's Scratchpad | View Weblogs ]

I don't believe this is possible, which means your only option is the one you suggest - create a new sparse file and rsync/cp into it.

I'll look forward to finding out that I'm wrong though!

Steve

[ Parent | Reply to this comment ]

Posted by ajt (195.112.xx.xx) on Sun 24 Apr 2011 at 08:41
[ Send Message | View Weblogs ]

It's boring and a pain to do on a remote machine. But it is doable!

--
"It's Not Magic, It's Work"
Adam

[ Parent | Reply to this comment ]

Posted by ajt (204.193.xx.xx) on Thu 28 Apr 2011 at 11:18
[ Send Message | View Weblogs ]

I've not done it yet with a live VM guest but it looks like filling the VM up with /dev/zero then deleting the zero files and running sparsify on the VM file from the host system generates a new VM guest file that is the right size again. I've just got to test that it will boot from it and everything runs okay.

--
"It's Not Magic, It's Work"
Adam

[ Parent | Reply to this comment ]

Posted by Anonymous (70.224.xx.xx) on Mon 25 Apr 2011 at 15:49
Since cp --sparse=always didn't work I expect that the VM has written then deleted to/from that extra space, thus the bytes are no longer all nulls.

A couple options I can think of (all untested):

* Force nulls in the VM: "dd if=/dev/null of=/tmp/padder" until you run out of space then remove /tmp/padder then retry the cp --sparse=always. This of sort of a poor-man's "discard"; I have heard talk/rumor of changes that will eventually cause discard in VM's to permit the host filesystem to automatically re-sparse the image. I do not think that such a feature is currently implemented and I am unsure whether it qualifies as a "wishlist" or "planned" feature (or even where to look for more information).

* Shrink the filesystem in the VM, then truncate the disk image (then re-grow if you want).

Of course the problem will come back as the VM scribbles over the newly nulled space as it adds and removes files.

Good Luck!

[ Parent | Reply to this comment ]

Posted by Anonymous (83.232.xx.xx) on Wed 27 Apr 2011 at 09:31
https://github.com/masover/sparsify might work. Haven't used it though. Also, this will only work if the unused portions of the image are filled with zeroes, which might not be the case.

[ Parent | Reply to this comment ]

Posted by ajt (195.112.xx.xx) on Wed 27 Apr 2011 at 21:14
[ Send Message | View Weblogs ]

It's the same problem. With a lot of zero padding within the VM it may work. It's worth considering if nothing else.

--
"It's Not Magic, It's Work"
Adam

[ Parent | Reply to this comment ]