Simple rotated backups with rsnapshot

Posted by Steve on Wed 17 Aug 2005 at 21:34

rsnapshot is a lightweight backup solution which creates rotated backups of local or remote directories. One of the key benefits of rsnapshot is its extreme simplicity. As a bonus backups are created using hardlinks to reduce the space used upon the backup host.

Like many backup solutions rsnapshot is a script which is built upon a foundation of OpenSSH and Rsync - the latter being used to synchronise directory contents without using excessive bandwidth, and the former to ensure the communication is encrypted and secure.

The script takes care of rotation automatically allowing you to setup the directories you wish backed up and then forget it.

You will most likely wish to run it in one of two ways:

  • Give it a list of local directories to be backed up.
  • Give it a list of remote directories to be backed up, and the host from which they should be taken.

The installation is straightforward, as you would expect:

apt-get install rsnapshot

Once installed you'll need to make several changes to the configuration file /etc/rsnapshot.conf:

  • Setup the rotation periods.
  • Setup the list of directories/files to backup.
  • Optionally exclude files from the backups.

The only thing to remember is that when setting up keys and values in the configuration file you must specify them as Tab-separated. Why this choice was made I have no idea - syntactically significant whitespace should be regarded as evil IMHO.

Moving on the first thing you'll need to do is uncomment the various rotation periods:

interval        hourly  6
interval        daily   7
interval        weekly  4
interval        monthly 6

If these are left commented out (Comments are lines beginning with the '#' character) then nothing will happen.

The numbers listed against the names will control how often the backups are rotated, so by default you'll receive:

  • 6 hourly backups.
  • 7 daily backups.
  • 4 weekly backups.
  • 6 monthly backups.

Once these have been uncommented you'll need to setup the directories to backup. How this is done depends on the nature of the directories, local vs. remote.

A local directory will look like this:

# LOCALHOST
backup /home/          localhost/
backup /etc/           localhost/
backup /usr/local/     localhost/

(Remember that tabs are used to separate these values).

A remote system will instead look like this:

backup  root@hostname:/home/		hostname/
backup  root@hostname:/etc/		hostname/
backup  root@hostname:/usr/local/bin/   hostname/

The difference between these two samples is the second section of the lines; for local systems it is just a directory name with a trailing "/" character. For remote hosts you should use "login@full.hostname.com:/path".

The last field in the samples is the backup directory to use upon the localhost - by default the backups will be stored beneath "/var/cache/rsnapshot". You can change this by setting:

#
# Change the location where backups are stored.
#
snapshot_root	/mnt/nfs/backupstore

If you are setting up remote backups the system assumes the there is a password-less login setup between the local machine and the remote machine specified (root@hostname in this example).

If you've not setup key-based OpenSSH logins then now is a good time to do it.

Note that you don't need to use remote root logins unless the files you wish to backup are unreadable to ordinary users upon the machine you're backing up. (Although it's fair to say if you're backing up /etc you'll almost certainly wish to do so as root - as otherwise important files such as /etc/shadow will not be read).

Once you've configured your logins for remote machines, and setup the filepaths you wish to backup you only need to enable the cronjob used to run the backups.

To do that remove the "#" comment characters from the relevant lines in /etc/cron.d/rsnapshot file leaving you with :

# This is a sample cron file for rsnapshot. 
# The values used correspond to the examples in /etc/rsnapshot.conf.
# There you can also set the backup points and many other things.
#
# To activate this cron file you have to uncomment the lines below.
# Feel free to adapt it to your needs.

0 */4   * * *           root    /usr/bin/rsnapshot hourly
30 3    * * *           root    /usr/bin/rsnapshot daily
0  3    * * 1           root    /usr/bin/rsnapshot weekly
30 2    1 * *           root    /usr/bin/rsnapshot monthly

The names used here "hourly", "daily", etc, match those used in the rsnapshot.conf file - which is important.

Once you've enabled the cronjobs you should be ready to make your first backup - which will take a while.

To do so run:

rsnapshot hourly

If all goes well you'll receive your backup files beneath "backup_root"/hourly.0/hostname, and any errors will be reported in the file /var/log/rsnapshot.log.

After a few days you'll notice that you have several new files such as:

root@scratchy:/mnt/backup# ls -l
total 68
drwxr-xr-x  4 root root  4096 2005-08-16 04:00 daily.0
drwxr-xr-x  4 root root  4096 2005-08-15 04:00 daily.1
drwxr-xr-x  4 root root  4096 2005-08-14 04:00 daily.2
drwxr-xr-x  4 root root  4096 2005-08-13 04:01 daily.3
drwxr-xr-x  4 root root  4096 2005-08-12 04:00 daily.4
drwxr-xr-x  4 root root  4096 2005-08-11 04:00 daily.5
drwxr-xr-x  4 root root  4096 2005-08-10 04:01 daily.6
drwxr-xr-x  4 root root  4096 2005-08-17 20:01 hourly.0
drwxr-xr-x  4 root root  4096 2005-08-17 16:00 hourly.1
drwxr-xr-x  4 root root  4096 2005-08-17 12:00 hourly.2
drwxr-xr-x  4 root root  4096 2005-08-17 08:01 hourly.3
drwxr-xr-x  4 root root  4096 2005-08-17 04:00 hourly.4
drwxr-xr-x  4 root root  4096 2005-08-17 00:01 hourly.5

These are the rotated backups which we've saved, the initial backups will be the largest - the later ones will only contain changes made since the previous run.

To see this you can view the amount of disk space taken up by each backup with "rsnapshot du":

root@scratchy:/mnt/backup# rsnapshot du
8.1G    /mnt/backup/hourly.0/
124M    /mnt/backup/hourly.1/
164M    /mnt/backup/hourly.2/
90M     /mnt/backup/hourly.3/
90M     /mnt/backup/hourly.4/
79M     /mnt/backup/hourly.5/
194M    /mnt/backup/daily.0/
199M    /mnt/backup/daily.1/
216M    /mnt/backup/daily.2/
189M    /mnt/backup/daily.3/
229M    /mnt/backup/daily.4/
224M    /mnt/backup/daily.5/
324M    /mnt/backup/daily.6/
11G     total

As the output shows the initial backup consumes a significant amount of disk space, and later ones only show the small amount of changes since then.

To reduce the space of the backups you can exclude files you don't care about, such as Apache logfiles. Simply add the filenames, or patterns, to the configuration file:

exclude access.log
exclude access.log.*
exclude error.log
exclude error.log.*
exclude referer.log
exclude referer.log.*
exclude agent.log
exclude agent.log.*
exclude Mail/backup/*
exclude /home/qemu
exclude Cache
exclude history.dat
exclude .bash_history
exclude viminfo

Note that any filename which matches the patterns will be excluded - so don't be too permissive or you might accidentally include a filename you did care about.

 

 


Posted by sno (62.254.xx.xx) on Thu 18 Aug 2005 at 17:24
[ Send Message | View Weblogs ]
After experimenting with a tar/rsync script i gave up and installed 'backup-manager' and after answering a few questions it set a schedule and was ready to go. I do have to do ls -lh in the folder to see sizes thou.

*space available for advertising*

[ Parent | Reply to this comment ]

Posted by Anonymous (66.82.xx.xx) on Tue 20 Sep 2005 at 01:35
How would you do a restore across ssh with it? A bare metal restore??

[ Parent | Reply to this comment ]

Posted by DaveV (24.8.xx.xx) on Sun 25 Sep 2005 at 18:29
[ Send Message ]

Ok to do a bare metal restore takes a little more preparation.

First, lets assume that we are backing up the following directories on the target system:

/boot/
/etc/
/root/
/usr/local/
/home/
/lib/modules

Ok that covers any data that changes, for the core system files, we'll need a list of what is installed so we can reinstall later. To create this list automatically every night, create a simple cron job in /etc/cron.d/dpkg-listfiles:

#
# Create list of all distro installed packages for restore
#
00 1   * * *   root     dpkg -l | awk '{print $2}' > /root/.dpkg-filelist

Boot the corrupted machine to a knoppix or other bootable media that includes support for ssh.

Repartition and Reformat the drive as necessary. Mount the partitions with temporary mount points:

...
mkfs.ext2 /dev/hda1
mkfs.ext2 -j /dev/hda2
mkswap /dev/hda3
mkfs.ext2 -j /dev/hda5
mkdir /mnt/temproot
mount /dev/hda2 /mnt/temproot
mkdir /mnt/temproot/boot
mount /dev/hda1 /mnt/temproot/boot
mkdir /mnt/temproot/home
mount /dev/hda5 /mnt/temproot/home

Install base system:

debootstrap --arch i386 sid /mnt/temproot/

Copy .dpkg-listfiles to the new system.

scp [user]@[backupserver]:/path/to/backups/daily.0/root/.dpkg-listfil es /mnt/temproot/dpkg-listfiles

Chroot into the new environment and install all of the applications that we had.

chroot /mnt/temproot /bin/su -
cat /dpkg-listfiles | xargs apt-get install
...

Warning: You may have to run the install step a couple times to get past a few errors and/or package changes.

Exit the chroot and copy back our data from our backups:

scp -r [user]@[backupserver]:/path/to/backups/daily.0/* /mnt/temproot/

Install grub (or lilo) to the boot sector of the hard drive:

grub-install /dev/hda
mkdir /boot/grub
grub
 root (hd1,0)
 setup (hd1)
 quit

If I didn't forget anything, you can reboot to the "new" system and it should be just like the old one.

[ Parent | Reply to this comment ]

Posted by DaveV (24.8.xx.xx) on Sun 25 Sep 2005 at 18:33
[ Send Message ]

Opps small error in the Grub install step. Remove the line:

mkdir /boot/grub

It will be created by the restore files step.

[ Parent | Reply to this comment ]

Posted by jrvz (129.83.xx.xx) on Wed 17 Jan 2007 at 19:12
[ Send Message ]
Unfortunately dpkg will truncate some package names. To get a list of installed packages, I suggest something like this instead:

dpkg --get-selections | awk '/\<install\>/ {print $1}'

- Jim Van Zandt

[ Parent | Reply to this comment ]

Posted by Anonymous (209.66.xx.xx) on Thu 12 Jul 2012 at 22:06
Even easier is just using dpkg --get-selections > output_file to export software state then use dpkg --set-selections with the same file to re-install.

Not sure what would happen if you have a purged package in your output file but the same package is installed on your new system, but this shouldn't be an issue if you're recreating the same system.

[ Parent | Reply to this comment ]

Posted by Anonymous (81.130.xx.xx) on Sat 24 Sep 2005 at 20:57
I have installed rsnapshot on Debian Sarge. When I run rsnapshot hourly I experience loads of error messages::

rsync: chown "/mnt/backup/snapshots/hourly.0/localhost/home/compass/posh-promd resses.co.uk/htdocs/stats" failed: Operation not permitted (1)
rsync: chown "/mnt/backup/snapshots/hourly.0/localhost/home/compass/posh-promd resses.co.uk/logs" failed: Operation not permitted (1)
rsync: chown "/mnt/backup/snapshots/hourly.0/localhost/home/compass/sf-limited .com" failed: Operation not permitted (1)
rsync: chown "/mnt/backup/snapshots/hourly.0/localhost/home/compass/sf-limited .com/htdocs" failed: Operation not permitted (1)
rsync error: some files could not be transferred (code 23) at main.c(789)


Anyone have any idea what the problem is??

Pete

[ Parent | Reply to this comment ]

Posted by Anonymous (84.9.xx.xx) on Sat 16 Sep 2006 at 11:42
Are you trying to write to a drive formatted for dos/windows (VFAT)? These can't cope with all the file properties of a unix file. Try writing to a linux (ext2, ext3) disk and see if the problem goes away. It worked for me!

[ Parent | Reply to this comment ]

Posted by eric (82.64.xx.xx) on Fri 11 Nov 2005 at 18:38
[ Send Message | View Weblogs ]
Great article !!

I was currently trying to use the magnifical properties of rsync and ext2/3 fs, following the Easy Automated Snapshot-Style Backups with Linux and Rsync guide when i saw your article.
I'm currently trying rsnapshot, but it seems to do exactly what I want, and I can backup /etc, /home, /var/log and others (nearly 6Go) and each snapshot (except the first/last : hourly.0) is nearly 300-500Mo............ so good !
I think with a volume double (12Go) I should have a good way of backup important files regularly....
my own backup scripts will terminate in /dev/null...

thanks.

[ Parent | Reply to this comment ]

Posted by Anonymous (70.240.xx.xx) on Mon 13 Feb 2006 at 01:50
"Give it a list of remote directories to be backed up, and the host from which they should be taken."

Did you really try to back up remote directories? I believed rsnapshot does not support this feature? At least, that's what their FAQ suggests (http://rsnapshot.org/faq.html).

[ Parent | Reply to this comment ]

Posted by Anonymous (70.240.xx.xx) on Mon 13 Feb 2006 at 01:59
Please forget this stupid comment. I did not read your sentence carefully enough.

[ Parent | Reply to this comment ]

Posted by Anonymous (213.207.xx.xx) on Wed 29 Mar 2006 at 10:17
Really nice article... Read it, installed it and working beautifully!!

Although my backups take up 18G it's still the most suitable for my needs! Easy install, clear manual and ready to go!

Compliments!

[ Parent | Reply to this comment ]

Posted by Anonymous (62.254.xx.xx) on Thu 30 Mar 2006 at 18:45
Nice article. I useing an old pc in one location to backup another server in another location so I wanted a bit of reporting. I'm using the script below - I'm sure it could be done a lot better but it works for me. I just run in in cron every morning.

#!/bin/bash

#send report to:
MAILTO="me@domain.com"
MAIL="/usr/bin/mail"
logfile="/var/log/rsnapshot.log"

# tmp files
mailfile="/tmp/rsmailfile"
tmplog="/tmp/tmprslog"

# extract todays log ( format 30/Mar/2006 )
fgrep `date +"%d/%b/%Y"` $logfile >$tmplog

echo "Whatever's Backup Report from on `date`" >$mailfile
echo >>$mailfile

# pick out lines with completed or error.
egrep "completed|ERROR" $tmplog >>$mailfile
echo >>$mailfile
echo "Disk Usage:" >>$mailfile
echo >>$mailfile
/usr/bin/rsnapshot du >>$mailfile
echo >>$mailfile
/bin/df -h >>$mailfile
echo >>$mailfile
echo "Info:" >>$mailfile
/usr/bin/w >>$mailfile
echo >>$mailfile
echo >>$mailfile
echo >>$mailfile
echo >>$mailfile
# Uncomment below to send full days log
#echo "Appended full days log:" >>$mailfile
#echo >>$mailfile
#cat $tmplog >>$mailfile

#Mail report file
$MAIL -s "Whatever's Backup report" $MAILTO < $mailfile

#remove tmp stuff
rm -f $tmplog
rm -f $mailfile

[ Parent | Reply to this comment ]

Posted by Utumno (60.248.xx.xx) on Wed 3 Jan 2007 at 09:38
[ Send Message | View Utumno's Scratchpad | View Weblogs ]
technical note: rather than the tedious

command1 >> file
command2 >> file
...
commandN >> file

simply write

{
command1
command2
...
commandN
} > file

[ Parent | Reply to this comment ]

Posted by Anonymous (62.77.xx.xx) on Thu 28 May 2009 at 19:06
Thanks for the report script, i try it!

[ Parent | Reply to this comment ]

Posted by Anonymous (87.72.xx.xx) on Thu 26 Jul 2007 at 23:24
If running rsync via ssh from nslu2-box(32mb ram), consider enhancing the data transfer by lowering down encryption with:

$ grep ssh /etc/rsnapshot.conf
ssh_args        -c blowfish


or make the crypto-tests with:

$ openssl speed aes rc4 blowfish 
...
OpenSSL 0.9.8c 05 Sep 2006
built on: Fri Dec  1 00:24:29 UTC 2006
options:bn(64,32) md2(int) rc4(ptr,int) des(idx,risc1,4,long) aes(partial) blowfish(idx)
compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -DTERMIO -O2 -g -Wall
available timing options: TIMES TIMEB HZ=100 [sysconf value]
timing function used: times
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
rc4               9168.20k    10685.83k    11152.32k    11268.47k    11309.36k
blowfish cbc      4344.99k     4863.81k     5006.95k     5047.15k     5060.90k
aes-128 cbc       2082.01k     2210.08k     2248.68k     2258.30k     2262.42k
aes-192 cbc       1802.92k     1902.07k     1929.79k     1933.57k     1940.79k
aes-256 cbc       1593.36k     1668.96k     1690.97k     1694.75k     1696.13k


--
BR,Admir Trakic,Copenhagen,DK

[ Parent | Reply to this comment ]

Sign In

Username:

Password:

[Register|Advanced]

 

Flattr

 

Current Poll

Which init system are you using in Debian?






( 1037 votes ~ 6 comments )

 

 

Related Links