Posted by Steve on Thu 2 Aug 2007 at 09:43
There are times when you have a machine, or two, which is short of disk space and yet you have spare capacity elsewhere upon your LAN. For these times using a Network Block Device could come in handy. This allows you literally export files as block devices remotely.
As another useful tip because the disk activity is going over the network you can filter or rate-limit IO in a simple fashion.
For example imagine you were running a lot of Xen guests upon a single host and you didn't want a greedy client performing a lot of I/O operations to adversely affect the other users of the machine. This is hard to manage by traditional methods, but using iptables you could easily manage it.
There are drawbacks to using network block devices, and I'm sure they are obvious:
In the system we're going to describe we'll have two machines mine.my.flat and yours.my.flat. The first will export a single file, which will be mounted upon the second. We'll use port 1234 for this, and we must firewall this port such that only the client system which is supposed to connect to it can.
Server Setup
Setting up the server is pretty simple. We'll need to install the software, prepare a device, and then export it.
To install the server run:
root@mine:/# apt-get install nbd-server
Now we'll create a file to export:
root@mine:/# modprobe nbd root@mine:/# mkdir -p /home/exported root@mine:/# dd if=/dev/zero of=/home/exported/trial.img count=256 bs=1024k root@mine:/# mkfs.ext3 /home/exported/trial.imgStarting the server should be simple, but I found that it immediately segfaulted on my machine - so we'll demonstrate two commands for this:
root@mine:/# nbd-server 1234 /home/exported/trial.imgIf this errors like mine did run this instead:
root@mine:/# touch /root/empty root@mine:/# nbd-server 1234 /home/exported/trial.img -C /root/empty(There is a global configuration which can be used to list exports /etc/nbd-server/config - however everytime I tried to use this file I received a segfault from the server process so I can't tell you anything useful about it.)
Setup The Client
Working With XenSetting up the client is very similar to setting up the server, we need to install the relevant software then mount the remote image.
root@yours:/# apt-get install nbd-clientTo mount the system we'll run:
root@yours:~# nbd-client mine.my.flat 1234 /dev/nbd0 Negotiation: ..size = 262144KB bs=1024, sz=262144 root@yours:~# mkdir /mnt/remote root@yours:~# mount /dev/nbd0 /mnt/remoteNow we can play:
for i in $(seq 1 100) ; do echo $i > /mnt/remote/$i; doneUnmount the volume:
root@yours:/# umount /mnt/remoteNow try mounting it back upon the server, to make sure that those files have persisted and been created as we expect:
root@vain:~# nbd-client 127.0.0.1 1234 /dev/nbd0 root@vain:~# mkdir /tmp/foo root@vain:~# mount /dev/nbd0 /tmp/foo root@vain:~# ls /tmp/foo/ 1 14 2 25 30 36 41 47 52 58 63 69 74 8 85 90 96 10 15 20 26 31 37 42 48 53 59 64 7 75 80 86 91 97 100 16 21 27 32 38 43 49 54 6 65 70 76 81 87 92 98 11 17 22 28 33 39 44 5 55 60 66 71 77 82 88 93 99 12 18 23 29 34 4 45 50 56 61 67 72 78 83 89 94 13 19 24 3 35 40 46 51 57 62 68 73 79 84 9 95 lost+foundFun, huh?
I spent a while trying to get this working with Xen, but only found success when using the /dev/nbdN devices.
For this to work I had to use the following Xen configuration, which is less than ideal:
disk = [ 'phy:vain-vol/etch-builder.my.flat-disk,sda1,w', 'phy:vain-vol/etch-builder.my.flat-swap,sda2,w', 'phy:/dev/nbd0,sda3,w' ]I had hoped to be able to specify the host + port inside the disk stanza, so that I could be sure of device ordering - but I couldn't make that work.
Any suggestions welcome..
This article can be found online at the Debian Administration website at the following bookmarkable URL:
This article is copyright 2007 Steve - please ask for permission to republish or translate.