Migrating a live system from ext3 to ext4 filesystem

Posted by Linas on Mon 31 Aug 2009 at 13:01

This article is meant to serve as a guide for migrating a live system from ext3 to an ext4 filesystem, including migration of files to use extents, a major feature in ext4. It describes the entire migration procedure, including common pitfalls involving a migration of a live system, as opposed to doing a fresh install.

Reasons for conversion

The explanation of advantages and disadvantages of ext4 is beyond the scope of this article. If you are not affected by the limitations of ext3, and not willing to take risks, it may not be worth it. On the other hand, on successful completion of the migration procedure your system may perform faster, experience shortened file system checks, and have increased reliability with no ill effects.

Trying ext4 without conversion

An interesting property worth noting is that you can mount ext3 filesystems as ext4 without doing the conversion first. All you need to do is to modify your /etc/fstab to say "ext4" instead of "ext3" for all your filesystems, and reboot. This way you will be able to go back to ext3 at any time.

This may be a nice test before attempting full conversion, but this will only enable ext4 features that are compatible with ext3. And that means that no major feature of ext4, like extents, will be enabled.

Migrating to ext4

Word of warning

Do a backup before attempting this procedure. It may render your system unbootable, and may destroy your data.

Your filesystems will become incompatible with ext3, so you need to make sure that you have a complete toolkit available with ext4 support before doing a conversion. This includes a bootloader, e2fsprogs, mount, and a recent kernel.

Prerequisites

  • e2fsprogs 1.41.6
  • mount 2.16
  • linux-image 2.6.30
  • grub 1.96+20090808

All of these packages are available in Debian unstable or experimental. Lower versions may work, except for e2fsprogs - this is indeed the lowest version.

Converting a non-root filesystem to ext4

As long as you are converting a filesystem that can be unmounted, it is fairly simple procedure. In this example we will be converting a /dev/sdc1 partition mounted as /home directory.

First, unmount the partition.

umount /dev/sdc1

Next, run a filesystem check on it to make sure it is in sane condition. We are still on ext3.

fsck.ext3 -pf /dev/sdc1

Enable new features of ext4 on the filesystem.

tune2fs -O extents,uninit_bg,dir_index /dev/sdc1

Option "extents" enables the filesystem to use extents instead of bitmap mapping for files, "uninit_bg" reduces file system check times by only checking used portions of the disk, and "dir_index" allows storing the contents of large directories in a htree for faster access. Option "dir_index" is also supported by ext3, so you may already be using it, but it makes no harm to specify it here.

Run a filesystem check. It will find errors. It is normal. Let it fix them. You may want to run the check twice to make sure that the filesystem is now clean.

fsck.ext4 -yfD /dev/sdc1

The "-D" parameter will actually enable the "dir_index" option by rebuilding directory index. It can be rebuilt (optimized) at any later time by running the check with the parameter.

Now edit your /etc/fstab file to say "ext4" instead of "ext3" for /home. Other options may differ for your system.

/dev/sdc1 /home ext4 defaults 0 2

Try to mount your new ext4 filesystem.

mount /home

If it succeeds, congratulations. If not, do not panic. You have not lost your data. And you have a backup after all, right? Make sure you have all the latest tools listed in prerequisites. Get them form Debian unstable or experimental if needed. Upgrade and try again.

The /boot partition

If your /boot is a separate partition, all is good and great. Just leave it as ext3. Latest development grub versions do have support for ext4, but it still may and will fail for some given snapshot of grub.

As ext3 can be mounted ext4 without conversion, you can just edit your /etc/fstab to say "ext4" instead of "ext3" for boot partition.

/dev/sdb1 /boot ext4 defaults 0 1

Most features of ext4 will not be used, but that makes little difference for /boot, as it is only used early at boot time. And since this is essentially still an ext3 partition, grub will have no problem booting it.

If, on the other hand, you do not have a separate /boot partition, you should consider creating one. Otherwise you must be really careful, and not enable features not supported by grub, or make sure that you are using a version of grub that supports all of them.

Converting a root filesystem to ext4

Converting a root filesystem is a bit more tricky because you cannot unmount it, as your system is running on it. Nevertheless it is still possible to do it without using an external bootable media. You should do this in a single-user mode.

First step is to modify your /etc/fstab file to say "ext4" instead of "ext3" for root partition. This is important because you will be operating on a read-only filesystem later, and will not be able to make the change, and this would result in your system unable to mount a root filesystem on next boot.

Let us assume that root partition is /dev/sda1, so your /etc/fstab should look something like this.

/dev/sda1 / ext4 defaults 0 1

Now remount the root filesystem read-only.

mount -o remount,ro /

Then run a filesystem check on the root filesystem.

fsck.ext3 -pf /dev/sda1

It will tell you to reboot the system. That may be a good idea, so simply boot into single-user mode and remount it read-only again. It is fine even though we have already modified /etc/fstab, because ext3 can be mounted as ext4 without conversion.

Next, enable all the ext4 features on the root filesystem.

tune2fs -O extents,uninit_bg,dir_index /dev/sda1

And run run a filesystem check on the root filesystem again. It will find and fix errors. This is normal.

fsck.ext4 -yfD /dev/sda1

You can now reboot to your new ext4 system, and enjoy faster filesystem check times, better performance, and all the improvements of ext4. Well, almost; read the next section.

Migrating files to extents

It may seem that the migration from ext3 to ext4 is now complete, and it is almost true. Except that any old files created before the conversion will continue using the bitmap mapping of ext3 instead of extents of ext4.

Files will eventually migrate to the new format as they are updated during normal system operation, because on next write they will be saved using extents. Unfortunately many frequently used files (like application binaries) are often read and rarely written to. The outcome is that the files will remain using the old format for a long time, and you will not be able to experience full potential of ext4.

A utility called e4defrag, which would be able to migrate files, is being developed. But unfortunately it is far complete, and is not suitable for use on real data as of time of writing.

Fortunately it is possible to migrate separate files to extents by using chattr utility, which comes with e2fsprogs package. It allows you to set an attribute on a file which causes the kernel to rewrite the file using extents. It even possible to do this on a mounted and working file system. In fact, that is the only way to do it.

Please not that this feature is still experimental and has not been tested thoroughly. Performing such operation may be dangerous. It may also flood your system log with warnings and errors. You should first test this on small number of insignificant files.

You can check if a file (or a directory) is using extents with lsattr.

lsattr /home/user/foo/bar

If it not using extents, the output will be something like this.

------------------- /home/user/foo/bar

The dashed line here is simply a placeholder for various attributes a file can have. This means that it has no attributes.

Now set the attribute to use extents on the file.

chattr +e /home/user/foo/bar

Now list the attributes again, and you will notice that the output looks like this.

-----------------e- /home/user/foo/bar

Note the "e", which means that it is now using extents.

Check your system log and look for scary things. Not finding any is a good sign. That means you can continue.

Modifying attributes with chattr can be done on multiple files. Although digging trough the entire directory system is not really feasible, so you can use some of the shell magic to accomplish the task.

find /home -xdev -type f -print0 | xargs -0 chattr +e
find /home -xdev -type d -print0 | xargs -0 chattr +e

The first command will run "chattr +e" on all files in /home, and the second will do the same for directories.

It is possible to run this on the root directory and convert everything at once. But running this on one file system at a time with "-xdev" parameter will prevent it from diving into file systems that do not support extents. It may also be useful to run filesystem check on a partition after conversion.

It may make sense to perform this in single-user mode to minimize the chances of something interfering with a task. You may also want to shutdown syslog, as it may generate a lot of warnings.

Final word

Author takes no responsibility for bad things that may or may not happen. Comments, corrections, and discussions most welcome.

 

 


Posted by pauldoo (194.247.xx.xx) on Mon 31 Aug 2009 at 15:25
[ Send Message ]
Using "e2fsprogs 1.41.6" (or better) is only necessary for the "chattr" part AFAICT. I merrily used 1.41.3 (on Debian lenny + lenny-backports) and have managed to convert my (non-boot and non-root) partitions to ext4 just fine using the steps you outline here. The only package I needed from backports was the kernel package itself.

[ Parent | Reply to this comment ]

Posted by pauldoo (194.247.xx.xx) on Wed 2 Sep 2009 at 15:33
[ Send Message ]
Compiling e2fsprogs from source in order to get a more up to date "chattr" worked fine (I didn't "install" this compiled version). I have now been able to complete the "chattr" steps without installing non-lenny packages.

[ Parent | Reply to this comment ]

Posted by dmn (213.240.xx.xx) on Tue 1 Sep 2009 at 10:19
[ Send Message ]
I get the following warning on each chattr invocation:
[  345.949988] ------------[ cut here ]------------
[  345.950006] WARNING: at 
/build/buildd-linux-2.6_2.6.30-6-amd64-s9DPiZ/linux-2.6-2.6.30/de bian/build
/source_amd64_none/fs/inode.c:1177 generic_delete_inode+0x64/0x168()
[  345.950012] Hardware name: System Product Name
[  345.950015] Modules linked in: isofs udf nls_base crc_itu_t hid_a4tech ppdev
lp parport sco bnep rfcomm l2cap bluetooth xt_multiport binfmt_misc powernow_k8
tun autofs4 cpufreq_powersave cpufreq_conservative cpufreq_stats fuse
xt_tcpudp iptable_filter ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4
nf_conntrack nf_defrag_ipv4 ip_tables x_tables nfsd exportfs nfs lockd fscache
nfs_acl auth_rpcgss sunrpc bridge stp ext3 jbd radeon drm it87 hwmon_vid
cpufreq_userspace loop snd_hda_codec_atihdmi snd_hda_codec_realtek
snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm
snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device
i2c_piix4 snd soundcore i2c_core snd_page_alloc evdev processor button
asus_atk0110 ext4 mbcache jbd2 crc16 sha256_generic aes_x86_64 aes_generic cbc
dm_crypt dm_mod usbhid hid ide_cd_mod cdrom ata_generic ide_pci_generic sd_mod
crc_t10dif atiixp ide_core ohci_hcd ahci libata ehci_hcd scsi_mod atl1 mii
floppy thermal fan thermal_sys [last unloaded: scsi_wait_scan]
[  345.950131] Pid: 5975, comm: chattr Not tainted 2.6.30-1-amd64 #1
[  345.950134] Call Trace:
[  345.950143]  [] ? generic_delete_inode+0x64/0x168
[  345.950149]  [] ? generic_delete_inode+0x64/0x168
[  345.950156]  [] ? warn_slowpath_common+0x77/0xa3
[  345.950166]  [] ? generic_delete_inode+0x64/0x168
[  345.950199]  [] ? ext4_ext_migrate+0x637/0x6b4 [ext4]
[  345.950226]  [] ? ext4_ioctl+0x2cf/0x621 [ext4]
[  345.950233]  [] ? vfs_ioctl+0x21/0x6c
[  345.950238]  [] ? do_vfs_ioctl+0x42b/0x464
[  345.950243]  [] ? sys_ioctl+0x51/0x70
[  345.950250]  [] ? system_call_fastpath+0x16/0x1b
[  345.950254] ---[ end trace c449b9f75be0d6b0 ]---
Am I safe ignoring it?

[ Parent | Reply to this comment ]

Posted by Anonymous (87.234.xx.xx) on Fri 4 Sep 2009 at 15:26
I've been running ext4 for ages and never manually converted files to use extents, so I just tried and got the same warning. Turns out that chattr +e might not yet be the safest thing to do, see this very recent commit: http://tr.im/xSwb
The converted files seem to be fine though, so maybe it's really just a paranoia warning..

[ Parent | Reply to this comment ]

Posted by tweek (80.63.xx.xx) on Tue 8 Sep 2009 at 07:40
[ Send Message ]

Thanks ... :-)

It's only a one-liner, so it's easy to apply — even without git.

[ Parent | Reply to this comment ]

Posted by Anonymous (220.253.xx.xx) on Sat 5 Sep 2009 at 05:39
Just a quick note of caution for other antedeluvians. I migrated to ext4 and migrated most of my filesystems back again when I found that dump/restore didn't (yet) support ext4.

[ Parent | Reply to this comment ]

Posted by Anonymous (81.184.xx.xx) on Sat 5 Sep 2009 at 08:42
If you are using a stock 2.6.30 kernel from debian repositories, be careful when you are converting you root filesystem. It is possible you need to update initrd (update_initramfs -u) to add ext4 driver. If the driver is not in initrd image, your system will not boot after the conversion.

[ Parent | Reply to this comment ]

Posted by amitdarpan (182.68.xx.xx) on Fri 23 Mar 2012 at 05:46
[ Send Message ]
This looks absolutely perfect. All these tinny details are made with lot of background knowledge. I like it a lot. This was a useful post and I think it is rather easy to see from the other comments as well that this post is well written and useful. Jodi Breakers songs , Jannat 2 Songs

[ Parent | Reply to this comment ]

Posted by Anonymous (77.119.xx.xx) on Mon 7 Sep 2009 at 09:34
What does "live" mean in the title here? I read it as "online", but this article unmounts the filesystems. A confusing title.

[ Parent | Reply to this comment ]

Posted by Anonymous (193.48.xx.xx) on Mon 5 Oct 2009 at 14:28
Right: confusing. "It will tell you to reboot the system. That may be a good idea, so simply boot into single-user mode and remount it read-only again."
Do you mean "reboot", or "init 1"...?
Whatever, jumping to runlevel 1 is only righteous to preserve your uptime! ;-)

[ Parent | Reply to this comment ]

Posted by Anonymous (82.233.xx.xx) on Tue 4 May 2010 at 21:41
IMPORTANT: you say "boot in single user mode", but this won't work since the file system is then busy. It does work with an "init=/bin/bash" kernel option.

many thanks for the tutorial, by the way...

[ Parent | Reply to this comment ]

Sign In

Username:

Password:

[Register|Advanced]

 

Flattr

 

Current Poll

Which init system are you using in Debian?






( 1029 votes ~ 6 comments )