Automatic host installation using Viper
Posted by docelic on Mon 10 Aug 2009 at 15:16
The purpose of this article is to give you a straight-forward, Debian-friendly way of installing and configuring Viper, a system for completely automated installation and configuration of Debian GNU based systems.
Introduction
Viper is a system for completely automated installation and configuration of Debian GNU based systems.It is a newly developed system that had the first public release in August 2009. This article should serve as an early introduction, focusing on the results. For more on the underlying logic and in-depth treatment of various topics, refer to official Viper documentation.
Viper uses a modern approach (debian-installer, Puppet etc.), works with existing Debian packages, and does not require any custom patching of either install media, client or server.
The basis of the whole system is a LDAP server, containing all configuration data.
If you are not familiar with LDAP and related terminology, glance over my other article, OpenLDAP installation on Debian, found here or here.
LDAP has the potential to be enormously successful in configuration management, but its flat data structure, absence of any kind of dynamic behavior, and heaps of duplicated data make it a mediocre tool.
To build on LDAP's great strengths while solving its deficiencies, a custom OpenLDAP backend has been implemented.
The setup explained here will get you to a point that you can perform an automatic installation of a Debian host. The configuration part is done using Puppet, but is not covered here; expect it soon as another article on debian-administration.org and the official website.
Viper download and setup
The instructions shown should quickly set you up with a Viper install server, using fully-working example configuration shipped with Viper, that you can test with and without actual client hosts. Then, after that, you should be able to start with writing your own site and hosts configuration.
It would be best if you can install Viper on a clean system without important data. However, the install script will only modify OpenLDAP, DHCP and Puppet server config so you should be able to install it on any system that does not already use one of those.
That said, let's move onto the setup.
Download Viper:
apt-get install git-core mkdir -p /etc/ldap cd /etc/ldap git clone git://github.com/docelic/Viper.git viper
Now, to set things up, you should just have to run scripts/viper-setup.sh.
It should do everything
automatically.
However, for the first time, I suggest you read the script (it contains
comments) and execute commands manually.
Before running the script, you should do just one thing related to the DHCP server -- you should create a network interface with IP address 10.0.1.1, to fit in the example subnet used by our default config. Then, you can run the install script with the name of the physical interface. Here's the session transcript:
ifconfig eth0:1 inet 10.0.1.1 netmask 255.255.255.0 sh scripts/viper-setup.sh eth0To configure eth0:1 on every boot, add it to /etc/network/interfaces with a stanza like this:
allow-hotplug eth0:1 iface eth0:1 inet static address 10.0.1.1 netmask 255.255.255.0
Net::LDAP::FilterMatch fix
Net::LDAP's FilterMatch module contains a bug that you have to patch manually, at least until it is fixed in the official distribution.
The patch file is included in downloaded files, so you just need to execute:
patch -p0 < support/FilterMatch.pm.patch
And then re-start slapd with invoke-rc.d slapd restart.
Testing the installation
The above simple procedure should have set you up with a working Viper-enabled LDAP server, DHCP server, and default configuration data you can query from the command line.
Tests you can run:
With ldapsearch
ldapsearch -x -b ou=dhcpldapsearch -x -b ou=defaults
ldapsearch -x -b ou=clients
ldapsearch -x -b cn=h2,ou=hosts,o=c1.com,ou=clients
ldapsearch -x -b cn=popularity-contest/participate,ou=hosts,ou=defaults
ldapsearch -x -b cn=debian-installer/locale,cn=h2,ou=hosts,o=c1.com,ou=clients
ldapsearch -x -b cn=ntp/servers,cn=h2,ou=hosts,o=c1.com,ou=clients
Test results
Ldapsearch query for cn=h2,ou=hosts,o=c1.com,ou=clients is a pretty good way of determining if everything is working alright. Here's how the output from the command should look like:$ ldapsearch -x -b cn=h2,ou=hosts,o=c1.com,ou=clients # extended LDIF # # LDAPv3 # basewith scope subtree # filter: (objectclass=*) # requesting: ALL # # h2, hosts, c1.com, clients dn: cn=h2,ou=hosts,o=c1.com,ou=clients objectClass: top objectClass: device objectClass: dhcpHost objectClass: ipHost objectClass: ieee802Device objectClass: puppetClient cn: h2 ipHostNumber: 10.0.1.8 macAddress: 00:11:6b:34:ae:8d puppetclass: test puppetclass: ntp::server dhcpHWAddress: ethernet 00:11:6b:34:ae:8d dhcpOption: host-name "h2" dhcpOption: routers 10.0.1.1 dhcpOption: domain-name-servers 192.168.1.254 dhcpOption: nis-domain "c1.com" dhcpOption: domain-name "c1.com" dhcpOption: subnet-mask 255.255.255.0 dhcpOption: broadcast-address 10.0.1.255 dhcpStatements: fixed-address 10.0.1.8 hostName: h2 ipNetmaskNumber: 255.255.255.0 clientName: c1.com ipNetworkNumber: 10.0.1.0 ipBroadcastNumber: 10.0.1.255 domainName: c1.com # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1
With scripts/node_data
perl scripts/node_data h2.c1.comWith scripts/preseed
QUERY_STRING=ip=10.0.1.8 perl scripts/preseedInstalling a client host
Adjust defaults and host's macAddress
After you've got the Viper server running and the tests are successful, you can attempt installing a host.
To do so, go to /etc/ldap/viper/ldifs/ and edit two LDIF files, 0-defaults.ldif and c1.com.ldif.
In the defaults file, you should go over all the values (search for occurences of "value:") and adjust them if any need change in your setup.
In the c1.com.ldif file, you should basically just modify the macAddress of host h1, and set it to the MAC address of the host you want to install.
Then, you should run 'make' in the current directory to load all .ldif files into LDAP.
Set up Apache to handle requests for preseed.cgi
Apache setup will already be done correctly if your cgi-bin directory is the standard /usr/lib/cgi-bin/ and accessible through default vhost at location /cgi-bin/. If not, adjust accordingly.
On the client host
To install the client, boot from official boot media, and simply select Advanced --> Automatic installation.
If your host has multiple network interfaces, you might also add interface=ethX on the boot line, or answer that question manually when asked by the installer. (The d-i installer should supposedly auto-detect the interface, but it seems this isn't working 100% well yet).
The results
The client host should install automatically and reboot into the newly installed system.Comments, questions and requests for clarification welcome here as the article followup, or on the Viper mailing list.
For more about Viper and further documentation, refer to Viper resources:
Viper homepage
Viper mailing list
Viper @ GitHub
Davor Ocelic
http://www.spinlocksolutions.com/
Copyright (C) 2007,2008 Davor Ocelic, <docelic@spinlocksolutions.com>
Spinlock Solutions, http://www.spinlocksolutions.com/
[ Parent | Reply to this comment ]
Viper tends to be a transparent central information and interoperability point, where other tools that make up the complete picture can search for information they need to do their job (DHCP for IPs, d-i for installation, Puppet for configuration etc.).
So it basically does the following things:
1) Provides central information store with support for dynamic values (unlike regular LDAP which only stores "flat" data)
2) Provides glue code or scripts used by other components to work with LDAP (such as preseed.cgi script for d-i's HTTP preseed method, or node_data script for Puppet's external_nodes)
3) Provides recipes on how to combine everything (Viper, dhcp, d-i, puppet) into a complete system
4) Provides a collection of Puppet modules to do typical configuration tasks (as of now, modules yet have to be written)
[ Parent | Reply to this comment ]
I tried Puppet and, yeah, it was infinitely nicer to handle than the venerable CfEngine (v2). But this is a ressources blackhole. As of today, all of my services are separated on different virtualized ("semi-virtualized", more particularly, thanks to the godly OpenVZ) instances. Hence, if I don't care that much for some services that eat a lot of memory, I will certainly not make them part of the skeleton of all my containers. There is not any use in having a scripted crap that ends up eating hundreds of MB on dozens and dozens of machines (which run on the same hardware). Such a thing is a big no-no: 24-32MB of RAM is more than enough for such a skeleton (the rest of what I allocate is for the daemon[s] the machine is for); no Perl+Ruby juggernauted daemons in each and every corner: what I earn for not having them allows me to have more memory for useful things, which allows me to segregate more efficiently. Less-is-more.
So I'll stay with FAI and CFengine v2. Definately. Well, up until CFEngine 3 comes to Debian, of course.
[ Parent | Reply to this comment ]
But then I realized that the dollar cost of the time saved by using puppet was worth the dollar cost of the RAM (even including the power to run it).
If you're trying to make do on a limited hardware budget, and you already have a CFengine setup, then ok, puppet may not be for you.
But if you are running 40 VMs on a 32gig box (which is a reasonable size, depending on your workloads you might be lower or higher in population), the
RAM spent on puppet is about 5% of your total. Or about $50.
RAM is cheap enough that most people don't benefit from optimizing for it.
[ Parent | Reply to this comment ]
I know RAM is always getting cheaper, but still: when the skeletons of my VMs eat a lot more of memory than most of the daemons they run (I said it: I also got juggernauts, like Slapd and such, but they are a minority in my case; most of my daemons are as lightweight as possible), I'm convinced there is something really wrong.
Well, sure, as I said: I recognize CFEngine is very old, necessitates a lot of DIY, and Puppet syntax is very nice. But still: I hadn't tested scripted daemons for a long time, and Puppet reminded me this was a thing I did not want. I'll wait for CFEngine v3, which I've tested and been impressed with, to come on Debian. It'll probably still be to young when Squeeze comes out, but I am not so stressed.
[ Parent | Reply to this comment ]
I only disagree with you in that puppet is nicer to handle than cfengine 2. You may like it better (lots of people do) but lots of people do not want to touch pupet with a 3 meter long stick. Cfengine is a proven technology, it works really well and it will stay here for a very long time. I have yet to start with cfengine 3 because 2 works for me.
Besides that, I think that misleading customers in your corporate website about what puppet does that cfengine doesn't is not a nice way of running a busines ...
[ Parent | Reply to this comment ]
> Viper uses a modern approach (debian-installer,...
Mmm, d-i is not a modern approach, it's more like a hack to fit into small RAM.
Using debconf preseeding for partitioning is totally ugly IMO.
IMO Vipers only seems to be a LDAP backend for d-i and puppet.
[ Parent | Reply to this comment ]
Using debconf preseeding for partitioning is a legitimate feature of the d-i, carried out by partman. There are patches available for partman (should be on its way into the distribution) that add RAID support in combination with other methods (LVM, crypto), which was the only important missing piece.
Viper is not "only a LDAP backend for d-i and puppet". But I do consider it a plus that it serves a specific purpose and uses successful existing software where possible, instead of reinventing the wheel.
Regards,
-doc
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]