growing ext3 partition on RAID1 without rebooting

Posted by joostje on Tue 25 Jul 2006 at 09:56

Although rather straightforward, I couldn't find an easy step-by-step guide, so here I'll describe how I ended up growing my ext3 partion on a RAID-1 array.

For me, the complicating factors are 1) using commands I don't use daily, and 2) that the kernel doesn't re-read the partition table if a disk still has a mounted partition.

First, the commands if you don't mind rebooting to grow /dev/md1 on /dev/sda3 and /dev/sdb3:

  #Alter the partition tables, to make /dev/sda3 and /dev/sdb3 have new size:
  fdisk /dev/sda
  fdisk /dev/sdb

  #reboot (make kernel read new partition table)

  #then, grow the raid1 /dev/md1
  mdadm --grow /dev/md1 --size=max

  #and finaly, grow the ext2 (or ext3) fs on /dev/md1
  resize2fs /dev/md1

The trick to avoiding the reboot is to make sure a hd is not used (no partition at all) when we alter the partition table with fdisk, this is possible as when all partitions on the disk are raid1 (or can be unmounted). So, before changing the partition sizes with fdisk, we'll mdadm /dev/mdX --fail /dev/sdXX --remove /dev/sdXX each partition on that disk from every raid device on the disk, alter the partition tables, then add the partitions again to the raid devices, and repeat for the other disk.

First, here's the layout of my disks:

/dev/sda:
  sda1    part of /dev/md0
  sda2    SWAP (used)
  sda3    part of /dev/md1
  sda4    UNUSED

/dev/sdb:
  sdb1    part of /dev/md0
  sdb2    SWAP (unused)
  sdb3    part of /dev/md1
  sdb4    UNUSED

Next, the commands I issued:

#Freeing /dev/sda:
#First, removing all RAID1 partitions on /dev/sda:
mdadm /dev/md1 --fail /dev/sda3 --remove /dev/sda3
mdadm /dev/md0 --fail /dev/sda1 --remove /dev/sda1

#Then, stopping the swapspace on /dev/sda:
swapoff /dev/sda2

#Then, altering the partition tables:
fdisk /dev/sda
  #entering 'w' at the end to write. This should go OK now.

#Start using the partitions again:
swapon /dev/sda2
mdadm /dev/md0 --add /dev/sda1
mdadm /dev/md1 --add /dev/sda3

# wait for both md devices to be fully synced
# (check /proc/mdstat)

#same with /dev/sdb
mdadm /dev/md1 --fail /dev/sdb3 --remove /dev/sdb3
mdadm /dev/md0 --fail /dev/sdb1 --remove /dev/sdb1
fdisk /dev/sdb
mdadm /dev/md0 --add /dev/sdb1
#don't add /dev/sdb3 yet, to have a spare copy while resize2fs-ing

mdadm --grow /dev/md1 --size=max

resize2fs /dev/md1

#If the resize2fs-ing went OK, we can now add /dev/sdb3:
mdadm /dev/md1 --add /dev/sdb3

Anyway, that's the theory. In my case, resize2fs reported resize2fs: Operation not permitted While trying to add group #384, and that made me worry enough to unmount the partition to run fsck.ext3 on it. fsck.ext3 BTW found Directories count wrong for group #299 (65535, counted=0) errors, for groups 299 through 383 (but not 384).

So, in the end I still ended up with some effective down-time (unmounted /dev/md1), but not as much as I would have had had I done it with a reboot. Also, the error message from resize2fs is unknown to google, so it seems quite rare.

For the 'live' growing of ext3 partitions to work, a kernel newer than 2.6.10 is needed (IIRC). Packages I'm using:

ii  e2fsprogs      1.39-1         ext2 file system utilities and libraries
ii  mdadm          2.4.1-6        tool to administer Linux MD device arrays (s
ii  util-linux     2.12r-10       Miscellaneous system utilities
linux-kernel 2.6.16.18

This article can be found online at the Debian Administration website at the following bookmarkable URL:

http://www.debian-administration.org/articles/424