Debian Administration

Working with network block devices

There are times when you have a machine, or two, which is short of disk space and yet you have spare capacity elsewhere upon your LAN. For these times using a Network Block Device could come in handy. This allows you literally export files as block devices remotely.

As another useful tip because the disk activity is going over the network you can filter or rate-limit IO in a simple fashion.

For example imagine you were running a lot of Xen guests upon a single host and you didn't want a greedy client performing a lot of I/O operations to adversely affect the other users of the machine. This is hard to manage by traditional methods, but using iptables you could easily manage it.

There are drawbacks to using network block devices, and I'm sure they are obvious:

• You're limited by the speed of your network - which is probably lower than the speed of your disks.
• You must care about security a lot.

In the system we're going to describe we'll have two machines mine.my.flat and yours.my.flat. The first will export a single file, which will be mounted upon the second. We'll use port 1234 for this, and we must firewall this port such that only the client system which is supposed to connect to it can.

Server Setup

Setting up the server is pretty simple. We'll need to install the software, prepare a device, and then export it.

To install the server run:

root@mine:/# apt-get install nbd-server

Now we'll create a file to export:

root@mine:/# modprobe nbd
root@mine:/# mkdir -p /home/exported
root@mine:/# dd if=/dev/zero of=/home/exported/trial.img count=256 bs=1024k
root@mine:/# mkfs.ext3 /home/exported/trial.img

Starting the server should be simple, but I found that it immediately segfaulted on my machine - so we'll demonstrate two commands for this:

root@mine:/# nbd-server 1234 /home/exported/trial.img

If this errors like mine did run this instead:

root@mine:/# touch /root/empty
root@mine:/# nbd-server 1234 /home/exported/trial.img -C /root/empty

(There is a global configuration which can be used to list exports /etc/nbd-server/config - however everytime I tried to use this file I received a segfault from the server process so I can't tell you anything useful about it.)

Setup The Client

Setting up the client is very similar to setting up the server, we need to install the relevant software then mount the remote image.

root@yours:/# apt-get install nbd-client

To mount the system we'll run:

root@yours:~# nbd-client mine.my.flat 1234 /dev/nbd0
Negotiation: ..size = 262144KB
bs=1024, sz=262144

root@yours:~# mkdir /mnt/remote
root@yours:~#  mount /dev/nbd0 /mnt/remote

Now we can play:

for i in $(seq 1 100) ; do echo $i > /mnt/remote/$i; done

Unmount the volume:

root@yours:/# umount /mnt/remote 

Now try mounting it back upon the server, to make sure that those files have persisted and been created as we expect:

root@vain:~#  nbd-client 127.0.0.1 1234 /dev/nbd0
root@vain:~# mkdir /tmp/foo
root@vain:~#  mount /dev/nbd0 /tmp/foo
root@vain:~#  ls /tmp/foo/
1    14  2   25  30  36  41  47  52  58  63  69  74  8   85  90  96
10   15  20  26  31  37  42  48  53  59  64  7   75  80  86  91  97
100  16  21  27  32  38  43  49  54  6   65  70  76  81  87  92  98
11   17  22  28  33  39  44  5   55  60  66  71  77  82  88  93  99
12   18  23  29  34  4   45  50  56  61  67  72  78  83  89  94  
13   19  24  3   35  40  46  51  57  62  68  73  79  84  9   95  lost+found

Fun, huh?

I spent a while trying to get this working with Xen, but only found success when using the /dev/nbdN devices.

For this to work I had to use the following Xen configuration, which is less than ideal:

disk        = [ 'phy:vain-vol/etch-builder.my.flat-disk,sda1,w', 
                'phy:vain-vol/etch-builder.my.flat-swap,sda2,w',
                'phy:/dev/nbd0,sda3,w' ]

I had hoped to be able to specify the host + port inside the disk stanza, so that I could be sure of device ordering - but I couldn't make that work.

Any suggestions welcome..

Add Comment Printable version

 

<<< Spam filtering with qpsmtpd Bash eternal history >>>