Weblog entry #3 for endecotp
See all weblog entries made by endecotp.
The idea is that they share a single root but have their own /tmp and /var. /home is also an NFS mount.
But there are files in /var/lib/dpkg, and elsewhere in /var, that presumably need to be kept in-sync with the contents of /bin, /usr and so on. So some parts of /var need to be shared too.
So, what I need is a list of the subdirectories of /var that should be shared (like /var/lib/dpkg) and the subdirectories that should be per-machine (like /var/tmp and /var/run). Is it safer to default to "shared" or "per-machine"?
I'm particularly worried about the dpkg and apt-related stuff, since its operation is "magic" to me and getting it wrong could leave everything horribly screwed up...
Here's the editted output of "tree -L 2" in /var :
|-- backups
| |-- dpkg.status.0
| |-- group.bak
| |-- gshadow.bak
| |-- infodir.bak
| |-- passwd.bak
| `-- shadow.bak
|-- cache
| |-- apt
| |-- debconf
| |-- locate
| `-- man
|-- lib
| |-- apt
| |-- aptitude
| |-- dhcp3
| |-- dpkg
| |-- initramfs-tools
| |-- initscripts
| |-- logrotate
| |-- misc
| |-- urandom
| |-- usbutils
| `-- vim
|-- local
|-- lock
|-- log
| [snip]
|-- opt
|-- run
| |-- crond.pid
| |-- crond.reboot
| |-- klogd.pid
| |-- motd
| |-- network
| |-- sshd
| |-- sshd.pid
| |-- syslogd.pid
| `-- utmp
|-- spool
| |-- cron
| `-- mail -> ../mail
`-- tmp
[snip]
Comments on this Entry
http://www.pathname.com/fhs/
--
"It's Not Magic, It's Work"
Adam
[ Parent | Reply to this comment ]
Unfortunately the FHS says that /usr is shareable while /etc is not shareable. Hmm. So if I apt-get install something on one machine that puts files in /usr/bin and /etc, then it will be broken on the other machines - they will see the binaries but not the configuration files.
I presume that people doing virtual machines with shared filesystems encounter the same problems. Or maybe no-one actually tries to share anything...
[ Parent | Reply to this comment ]
[ Send Message | View Steve's Scratchpad | View Weblogs ]
When I share root filesystems over NFS I give every machine a particular export ... Which means I duplicate space on the NFS server, but don't have to worry about other side-effects. (I'm sure with bind mounts, or union I could cut down on the disk space required; but I've never needed to do that yet.)
The only thing I share would be /home. That way I avoid these kind of issues.
[ Parent | Reply to this comment ]
Yes, I think that most people just replicate everything. Disk is cheap. Then they discover that they want to do the same install or upgrade on all their machines, and they look for tools like the multiple-ssh things described in a recent article here. It's a trade-off.
I am now running two machines sharing everything except /var/backups,cache,lock,log,run,tmp and /tmp, and I'm waiting to see how well it works.
One thing that I've noticed is that an NFS-mounted /var is treated specially by the init scripts - /etc/init.d/mountnfs will wait for NFS mounts of /usr, /usr/*, /var and /var/* to complete before continuing, since subsequent scripts may want to access /var/run, /var/log etc. But other NFS mounts continue in the background. This breaks if your per-client directories are done using symlinks pointing out of /var; my first attempt had a /per_client in the root directory. The solution is to have a structure like this on the server:
var
|---lib [shared]
|---opt [shared]
|---run ---> /var/per_client/run [symlink]
|---tmp ---> /var/per_client/tmp [symlink]
`---per_client [empty dir as mount point]
Then mount server:/path/to/`hostname`/var on /var/per_client. Because the mount point is in /var, mountnfs.sh will wait for it.
Phil.
[ Parent | Reply to this comment ]
There's one thing that is not clear to me: do you want to do the maintenance work from the small systems (let's call them clients), or from the central machine that exports the shares (let's call it the server)?
In the first case, what is your expectation when you apt-get install something on one client? The same package (with all its config files) should be propagated to the other clients? This suggests that /etc should be shared, but what about mtab, hostname, etc.?
On the other hand, if all the administrative tasks like installing packages are done centrally on the server, why do you bother about clients not being able to see /var/lib/dpkg? You should not run apt-get on the clients in the first place!
[ Parent | Reply to this comment ]
> maintenance work from the small systems (let's call them clients),
> or from the central machine that exports the shares (let's call it
> the server)?
Preferably from one of the clients, with the others shut down if necessary, but doing from the server (e.g. with a chroot) could be an option if it made something simpler.
> In the first case, what is your expectation when you apt-get install
> something on one client? The same package (with all its config files)
> should be propagated to the other clients?
Yes.
> This suggests that /etc should be shared, but what about mtab,
> hostname, etc.?
Right, well that's another list of things to worry about....
If you just delete /etc/hostname (or make it empty?) then the init scripts will trust that the kernel alrady knows its hostname; in my case, it got it already from DHCP.
mtab can be made a symlink to /proc/mounts for most purposes. (There is some obscure case where /proc/mounts has less info than a real mtab would provide, but I can't recall the details.)
I haven't found anything else to worry about, yet. Let me know if you think of any more examples.
Phil.
[ Parent | Reply to this comment ]