A simple introduction to fabric

Posted by Steve on Mon 1 Aug 2011 at 18:48

fabric is described as a simple library and command-line tool for performing application deployment and system administration tasks. Here we'll take a look at using it to deploy simple applications remotely.

Recently I wanted to streamline the deployment of several services, applications, and utilities I maintain and develop. There are several deployment systems out there which promised to be useful, but ultimately I decided to standardise upon fabric which is packaged for Debian GNU/Linux.

Some people have suggested that I look at CFEngine, Puppet, Chef, or similar. On the face of it I understand that advice completely as I do use CFEngine already to maintain 150+ systems.

However I largely regard the use of such complex tools as being responsible for "systems", not "projects". To me a project can be deployed upon many many hosts, and on that basis I don't want the installation and configuration to be tied into any system/company-wide infrastructure.

(Obviously there are exceptions and the use of CFEngine, Puppet, or Chef can and does often involve the manipulation of software and the installation of packages.)

Although there are many deployment tools out there several of the commonly used systems tend to suffer from a lack of documentation. (For example Capistrano is an example of a system I've struggled with in the past; historically documentation was both scant and unreliable.)

Happily fabric is wonderfully documented, at least in part because it is available as both a library and a simple driver tool. The expectation of the authors is presumably that you will interface with the library, and the driver is merely a slightly useful example. That said I've not needed to integrate with it so far, because all the obvious things I wish to do are easily supported right out of the box.

When it comes to managing remote machines there are several things that you want to do, and these basic primitives are supported nicely. Although there are many other things you can do we're going to primarily concentrate upon these three options:

Connecting to hosts

Connecting to remote hosts is carried out via SSH, and you can specify usernames, hostnames, and appropriate SSH keys.

Uploading files

Using the put() primitive files may be uploaded from your local system to the remote host(s).

Running commands

Running commands, easily, is one of the most important things that fabric gives you. It allows you to run commands locally, remotely, and using sudo to gain additional privileges.

To get started you'll need to install the package:

apt-get update
apt-get install fabric

Then we can create our first file. Since fabric is designed around the idea of connecting to a remote host we will need one. Let us start by assuming you can ssh to the host test.example.org as user bob.

The fabric driver code will read and execute the contents of a file named fabfile.py in the current directory. This file is actually python which gives you a lot of room for "cleverness" but lets start by saving the following contents there:

# ~/fabfile.py

from fabric.api import *

env.hosts = ['test.example.org']
env.user  = 'bob'

def remote_info():
    run('uname -a')

def local_info():
    local('uname -a')

Given this file we can now run:

fab remote_info

This will:

  • Load the file ~/fabfile.py
    • Compiling the file into ~/fabfile.pyc - A standard pythonism
  • Connect via SSH to the host test.example.org
    • Using the username bob
  • Show the output of running the command "uname -a" upon that remote host.

If you preferred to run this command upon another host you could manually specify that upon the command line:

skx@birthday:~$ fab --hosts=localhost,birthday.my.flat --user=skx remote_info
[birthday.my.flat] run: uname -s
[birthday.my.flat] out: Linux
[localhost] run: uname -s
[localhost] out: Linux

Done.
Disconnecting from birthday.my.flat... done.
Disconnecting from localhost... done.

Note: In the second sample above two things: Explicitly changed the username, and specified two hosts to connect to separated by commas.

Also note that we ran the code in the function "remote_info", we could have similarly executed the code in the local_info subroutine by executing:

skx@birthday:~$ fab local_info

Now that we've demonstrated a simple example of connecting and running a command we can do something more useful. Let us pretend we have a remote cluster of machines, each of which is running Varnish. Further suppose that we have a configuration file that we wish to deploy upon these hosts. We can do this via the following fabfile.py:

# ~/fabfile.py
# deploy a varnish configuration file upon the cluster.
#

from fabric.api import *

env.user = "steve"
env.hosts = [ 'cluster-1.example.org', 'cluster-2.example.org', 'cluster-3.example.org.' ]

def update():
  """ Upload varnish configuration file and restart it. """"
   put( "/tmp/varnish.vcl", "~steve/varnish.vcl" )
   sudo( "mv ~steve/varnish.vcl /etc/varnish/", pty=true )
   sudo( "/etc/init.d/varnish restart", pty=true )

With this in place we can then run "fab update", which will upload the file from the local filesystem, using SSH as the transport, and then restart varnish with that updated configuration file. We've had to break the upload into two steps as it isn't possible to play with multiple usernames in one block.

The sudo() primitive is exactly like the run() primitive we previously demonstrated - with the exception that it runs a command as root. Obviously we need to be root to restart the varnish server, perhaps less obviously we uploaded the file initially into a location we had permission to do so, then used sudo to move it into the final location - that is because we can only upload files as "ourself".

A More Complex Example

As fabric is a mere wrapper around a python library we can avoid using it entirely; and instead use the library directly. This is what the following library does:

#!/usr/bin/python
#
#  save this as ~/project/deploy
#
from __future__ import with_statement
from fabric.api import *
from fabric.contrib.console import confirm
import sys
import os
import subprocess


env.user = 'steve'
env.host = [ 'example.org' ]


def test():
  """
  Run our test suite.  If it fails prompt the user for action.
  """
  with settings(warn_only=True):
     result = local('make test', capture=True)
     if result.failed and not confirm("Tests failed. Continue anyway?"):
        abort("Aborting at user request.")


def deploy():
  """
  Archive our current code and upload to the remote host.

  NOTE: Insecure since /tmp/project.tar.gz might be a symlink ..!
  """

  local( "tar czf /tmp/project.tar.gz .")
  put( "/tmp/project.tar.gz /tmp/project.tar.gz" )
  local( "rm /tmp/project.tar.gz" )


def install():
   """
   Install remotely if the test suite passes.
   """
   test()
   deploy()
   with cd('/home/steve/project/'):
        run("tar xzf /tmp/project.tar.gz")
        run('make install')

#
#  This is our entry point.
#
if __name__ == '__main__':

    if len(sys.argv) > 1:
        #
        #  If we got an argument then invoke fabric with it.
        #
        subprocess.call(['fab', '-f', __file__] + sys.argv[1:])
    else:
        #
        #  Otherwise list our targets.
        #
        subprocess.call(['fab', '-f', __file__, '--list'])


The example above has three targets which you can see if you run it with no arguments:

skx@birthday:~$ ./deploy
Available commands:

    deploy   Archive our current code and upload to the remote host.
    install  Install remotely if the test suite passes.
    test     Run our test suite.  If it fails prompt the user for action.

This output could also be found by running "fab --list", which lists the top-level functions along with any associated documentation strings that have been supplied. (That's what the """ Text """ is for.)

The command line handling is pretty basic, but adequate for our demonstration purposes. The more interesting part of this code is that it runs a local test suite prior to deployment, and if that fails it will prompt you to decide whether to abort or complete.

In the case of the code above we're obviously only failing back to a Makefile to do the real work, as we can see that I've invoked:

  • "make test"
    • To run a local test suite.
  • "make install"
    • To install system-wide.

However with the freedom to run commands locally, remotely, and remotely as root, along with the ability to upload files securely it should be possible for you to script almost any deployment scenario.

 

 


Posted by maxk (85.64.xx.xx) on Tue 2 Aug 2011 at 20:28
[ Send Message | View Weblogs ]
There are some limitations to the standard fabric. One of them is the execution model. The main is the assumption "each fab task is run over 1 single node, and is independent of others"
Usage scenarios like source sync over multiple tree-structured node sets, when a task traverses more than 1 host, go beyond that model, and require some mess with ssh connection related details of env dictionary, which may be delicate.
It is doable, but requires more care than just write simplistic fab tasks.

Also, worth mentioning is a nice fork off the official (bitprophet's) fabric called "Tav's fabric". It adds some nice bells and whistles to fabric - @task decorator, etc.
Very recommended to consider.


I know this site is about Debian, but some Debian servers help managing Windows networks, so windows compatibility:
Theoretically, python is relatively platform agnostic, but the way fabric runs programs requires good process communication, like i/o redirection, signals, etc., those are not very stable under Windows, so not purely pythonic code (especially sudo() and run()) are not always stable.


thumbs up, Steve, as usual.

[ Parent | Reply to this comment ]

Posted by Steve (90.193.xx.xx) on Sun 7 Aug 2011 at 16:34
[ Send Message | View Steve's Scratchpad | View Weblogs ]

Thank you for that comment - the fork you mention seems to be exceptionally useful. Just running things in parallel is a great time-saver when you're installing code on 50+ hosts. The other changes are all neat sounding too.

Thankfully (?) I only have to deal with *BSD, Solaris, & Debian on the whole. So windows compatibility isn't a concern for me right now. Though it is often useful for others.

Steve

[ Parent | Reply to this comment ]

Posted by maxk (85.64.xx.xx) on Wed 18 Jan 2012 at 19:50
[ Send Message | View Weblogs ]
long time :)
since then many of the Tav's fabric fork got into the official Fabric package installable from pip/easy_install.. so life is good :)

[ Parent | Reply to this comment ]

Sign In

Username:

Password:

[Register|Advanced]

 

Flattr

 

Current Poll

Which init system are you using in Debian?






( 1604 votes ~ 7 comments )