Introduction BackupPC part 1

Posted by martijnvanb on Thu 27 Mar 2008 at 10:07

This HOWTO will describe how to install BackupPC and how to create a simple backup using backuppc. BackupPC can offer a nice solution for both simple and complex backups.

I will describe how to backup one client with rsync and ssh with key-based authentication. We will create a special user which uses sudo powers to read the whole filesystem.

By using a combination of rsync/ssh and BackupPC we will create a fast secure and reliable backup system.

Some information about the computers I used

Both machines have Debian Lenny installed, but I think most flavours will do.

One machine is called aikido with ip address 192.168.1.1 we will use this machine to install the BackupPC server upon.

The other machine is called hapkido, we will use this as a backup client, this machine has ip address 192.168.1.2. On this client we will need to place our public key and create a special user.

Both names are resolvable in my own DNS, if you can't resolve your own hostnames, you can add them to your /etc/hosts file or use IP adresses.

Server Installation

We are going to install BackupPC with the following command:

aikido:~#  aptitude install backuppc

You will have to answer some question, one about which webserver you will use, most likely this will be apache or apache2. Backuppc will create a default user "backuppc" with a random password write it down, you will need it later!

Changing the backuppc password

If you forget the password or you want a other password you can change it with:

aikido:~# htpasswd /etc/backuppc/htpasswd backuppc
New password:
Re-type new password:
Updating password for user backuppc

We need to create a ssh-key pair for the backuppc user which we will use to connect to our client.

We will create a key pair without a passphrase.

aikido:~# su - backuppc
backuppc@aikido:~$ ssh-keygen -t rsa -C "backuppc"
Generating public/private rsa key pair.
Enter file in which to save the key (/var/lib/backuppc/.ssh/id_rsa):
Created directory '/var/lib/backuppc/.ssh'.
Enter passphrase (empty for no passphrase):  Leave this empty!
Enter same passphrase again:  Leave this empty!
Your identification has been saved in /var/lib/backuppc/.ssh/id_rsa.
Your public key has been saved in /var/lib/backuppc/.ssh/id_rsa.pub.
The key fingerprint is:
96:6e:b0:ee:09:06:e9:82:e4:f9:68:21:06:f3:85:a5 backuppc@aikido

The server is now setup.

Client part

Now we will need to install rsync (and sudo if it wasn't installed):

hapkido:~# aptitude install sudo rsync

We will also need to add a special user with rsync super powers.

First we will add the user backuppc:

hapkido:~# adduser backuppc

Lets give backkuppc some superpowers!

hapkido:~# echo "backuppc  ALL=NOPASSWD: /usr/bin/rsync" >> /etc/sudoers

This will add a line to the sudoers file which will allow the user backuppc to run rsync with root privileges without entering a password, this is needed for a full backup.

Back to the server

We need to copy the public key to the client hapkido.

backuppc@aikido:~$ ssh-copy-id backuppc@hapkido
The authenticity of host 'hapkido (192.168.1.2)' can't be established.
RSA key fingerprint is 15:c0:63:cf:af:74:c4:23:a6:f5:b4:80:ce:3b:0f:30.
Are you sure you want to continue connecting ( yes /no)
backuppc@hapkido's password:

Now try logging into the machine, with "ssh backuppc@hapkido", and check in ~/.ssh/authorized_keys to make sure we haven't added extra keys that you weren't expecting.

This would be a good time to see if our key is being accepted. We should be able to make a connect without a password/passphrase:

backuppc@aikido:~$ ssh backuppc@hapkido

We will add an extra line to ~/.ssh/authorized_keys. This option makes it possible to connect from only one host(aikido). This will improve security a bit:

backuppc@aikido:~$ vi ~/.ssh/authorized_keys

Add "from='aikido'" before ssh-rsa, this should make the start of the file look like this:

from="aikido" ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAvWh3e

Browsertime

Now we are almost done, let's browse to http://aikido/backuppc, and login with the username backuppc and the given password

Let's add our client to our backup:

  • click on "edit" host
  • Click on "delete" localhost
  • Click on "add" host
  • Fill in the hostname hapkido
  • Click "xfer"
  • Change "xfermethod" from "smb" to "rsync"
  • Change the value of "RsyncClientCmd" from $sshPath -q -x -l root $host $rsyncPath $argList+ to $sshPath -q -x -l backuppc $host sudo $rsyncPath $argList+
  • Change the value of "RsyncClientRestoreCmd" from $sshPath -q -x -l root $host $rsyncPath $argList+ to $sshPath -q -x -l backuppc $host sudo $rsyncPath $argList+
  • Click "save"

Congratulations you are now ready to start your first backup:

  • Click on "host summary"
  • Click on "hapkido"
  • Click on "start backup"
  • Click on "Start full backup"

That wasn't so hard right!

Have fun.
Martijn

 

 


Posted by ajt (204.193.xx.xx) on Thu 27 Mar 2008 at 12:30
[ Send Message | View Weblogs ]
That's very interesting and helpful, I'm looking forward to more parts.

A couple of minor points:

1. We should be using aptitude not apt-get these days. It's a small point, but for most purposes aptitude is a drop in replacement and it does offer genuine advantages over apt-get

2. You should not edit the sudoers file directly with an editor or echoing things to it. The sudo man page makes it quite clear you should use the included visudo tool. I know it's something we all do, but we shouldn't propagate it.

--
"It's Not Magic, It's Work"
Adam

[ Parent | Reply to this comment ]

Posted by Anonymous (128.97.xx.xx) on Tue 24 Feb 2009 at 16:46
" You should not edit the sudoers file directly with an editor or echoing things to it"

The purpose of the visudo command is to lock the file while writing to prevent race condition type mishaps. Though simply echo'ing the output to /etc/sudoers may not be correct, you can safely edit the file in scripts by also locking the file access while it is open and not freeing it till you are done.

This is what I do for my automatic client configuration scripts.

[ Parent | Reply to this comment ]

Posted by Anonymous (121.72.xx.xx) on Fri 25 Nov 2011 at 23:49
If you have a sudoers.d directory, you can create a file in there with the same contents as are appended to sudoers. (eg echo "backuppc ALL=NOPASSWD: /usr/bin/rsync" > /etc/sudoers.d/backuppc)

But, make sure to get the permissions right on the new file afterwards:
chmod 0440 /etc/sudoers.d/backuppc

Otherwise (in my recent painful experience), the next time sudo is used, it will crash!

This is particularly bad on ubuntu systems that don't have a true root password. I had to use a recovery CD to fix this.

[ Parent | Reply to this comment ]

Posted by Anonymous (220.233.xx.xx) on Thu 27 Mar 2008 at 12:59
The biggest advantage of BackupPC is its pooling mechanism. If you have the same file across several systems, only one copy is stored on the backup machine (via hard-links in the pool). Future backups will not store another copy of each file, but rather link to the pool. If you have several like systems this is a very space efficient method for backups as the pool is common to all clients.

Also worth mentioning is that general BackupPC tuning is to store your pool on reiserfs or xfs, mounted with noatime as you will get significant improvements. Other tips here: http://backuppc.wiki.sourceforge.net/speedupbackups

[ Parent | Reply to this comment ]

Posted by Anonymous (212.238.xx.xx) on Thu 27 Mar 2008 at 15:08
Mooi verhaal, Brum!

gr. DB

[ Parent | Reply to this comment ]

Posted by Anonymous (136.173.xx.xx) on Thu 27 Mar 2008 at 15:24
Hi ! Thanks for the article !

I have to insist on major BackupPC features, which makes it unique:
- BackupPC is disk based backup and not tape based.
- It can use smb, ssh, rsync, etc to backup clients, so no client agent has to be installed.
- It has a nice web interface.
- It has a clever pooling scheme which minimizes disk storage and disk I/O.
Identical files across multiple backups of the same or different PC are
stored only once resulting in substantial savings in disk storage and disk
writes. Also known as "data deduplication".

This "data deduplication" is very interesting and only possible on disk based backups: If you backup 200 Windows PCs, only a single copy of the system will be on the backup disk (e.g. not 2x200GB=400GB but only 2GB + compression). It's as if BackupPC only stored the differences, so that you can also have a very long backup retention policy without needing lots of disks.

A must try !

Ludovic (BackupPC Debian package maintainer :-)

[ Parent | Reply to this comment ]

Posted by Anonymous (67.88.xx.xx) on Thu 27 Mar 2008 at 16:10
Thanks Ludovic for maintaining the package! You expose great software and really make our lives so much easier!

[ Parent | Reply to this comment ]

Posted by Anonymous (172.188.xx.xx) on Thu 27 Mar 2008 at 22:44
Do you not also need to change the "RsyncClientRestoreCmd" setting too?

[ Parent | Reply to this comment ]

Posted by martijnvanb (213.84.xx.xx) on Fri 28 Mar 2008 at 07:03
[ Send Message | View Weblogs ]
Hi,

You are right, I fixed it.
Thanks.

regards,
Martijn

[ Parent | Reply to this comment ]

Posted by Anonymous (217.149.xx.xx) on Fri 28 Mar 2008 at 12:53
hello
I use backuppc for about 1.5 year, and i.m happy witch it.

Why use backuppc and sudo, u can easy use root acount to ssh.
If you have acces to run rsync witch id=0, you can simply overwrite any file, so u can change /etc/passwd.

My idea of use backuppc was to create ssh tunell to rsync demon, this idea have some benefits:
- nobady from backuppc can log to backupded host
- nobady from backuped host cant log to backuppc host
- NAT is no more a problem, connection direction is from backuped host to backuppc.

Hire you have my help script, to create cut/paste:

cat create_backuppc_paste.sh
#!/bin/bash

# free port to forward ssh connectin
# each host has own port
PORT=6054
# rsync speed ( 128 = 1Mb/s )
SPEED=512

# backuped host
REMOTE_HOST_NAME="my_hostname"

# rsuncd password (from generator)
RSYND_PASSWORD="Mu6Kacxx"

# command to do ssh on backuppp
SSH_NA_BACKUPPC="ssh backuppc"


###
# Rsync config
###
#in debian
RSYNC_LOCATION="/usr/bin/rsync"
# in OpenBSD
# RSYNC_LOCATION="/usr/local/bin/rsync"

BACKUP_DIR="/"
# exclude from backup
# if you use LVN snapshot put /backup/ prefix
EXCLUDE="/backup/dev /backup/vz/sharedvz/2027/root/old_nagios /backup/vz/sharedvz/2027/root/dev "

###
# LVN SNAPSHOT
###

# YES OR NO
USE_LVN_SNAPSHOT="YES"

# VOLUME GROUP
VG="/dev/main"
# LOGICAL VOLUME
LV="root"

# default mount point = /mnt/rootmirror
if [[ "$USE_LVN_SNAPSHOT" == "YES" ]] ; then BACKUP_DIR="/mnt/rootmirror$BACKUP_DIR" ; fi

palza(){
echo ;echo;echo
echo "# press enter to continue ... "
read
clear
}


clear
echo "#####################################################"
echo "# paste this to backuppc "
echo "#####################################################"
echo
echo "##################"
echo "# create tunel usre "
echo "#################"
echo "useradd -m $REMOTE_HOST_NAME -s /bin/true"
echo "# create ssh key for tunel user "
echo "su $REMOTE_HOST_NAME -s /bin/bash -c \"mkdir /home/$REMOTE_HOST_NAME/.ssh\" "
echo "su $REMOTE_HOST_NAME -s /bin/bash -c \"ssh-keygen -N '' -f /home/$REMOTE_HOST_NAME/.ssh/id_rsa\" "
echo "su $REMOTE_HOST_NAME -s /bin/bash -c \"cp /home/$REMOTE_HOST_NAME/.ssh/id_rsa.pub /home/$REMOTE_HOST_NAME/.ssh/authorized_keys\" "
palza

echo "##################"
echo "# create backuppc konfiguration "
echo "#################"

echo "cat >> /etc/backuppc/hosts << _EOF_"
echo "lh-$PORT-$REMOTE_HOST_NAME 0 backuppc
_EOF_
"

echo "cat >> /etc/backuppc/lh-$PORT-$REMOTE_HOST_NAME.pl << _EOF_"
echo "
#
# Rsync Backup (with rsync daemon on the client)
#

\\\$Conf{XferMethod} = 'rsyncd';

\\\$Conf{RsyncShareName} = ['backup'];

# if needed set a user name and password to access the remote shares
\\\$Conf{RsyncdUserName} = 'backuppc';
\\\$Conf{RsyncdPasswd} = '$RSYND_PASSWORD';
\\\$Conf{RsyncdClientPort} = $PORT;
_EOF_
"

echo "cp /etc/hosts /etc/hosts-`date -I`.backup"
echo "sed -i -e \"s/localhost/localhost lh-$PORT-$REMOTE_HOST_NAME/\" /etc/hosts"

echo "# reload backuppc "
echo "/etc/init.d/backuppc reload"
palza


echo "#####################################################"
echo "# paste this to remote host $REMOTE_HOST_NAME "
echo "#####################################################"
echo
echo "##################"
echo "# create a private ssh key on $REMOTE_HOST_NAME"
echo "#################"
echo "mkdir -p /root/.ssh"
echo "cat > /root/.ssh/id_rsa_backuppc << _EOF_"
$SSH_NA_BACKUPPC "cat /home/$REMOTE_HOST_NAME/.ssh/id_rsa"
echo "_EOF_"
echo "chmod og-rw /root/.ssh/id_rsa_backuppc"
echo
palza


echo "##################"
echo "# create and run tunel on $REMOTE_HOST_NAME"
echo "#################"
SCRYPT="/root/bin/redirect_port_from_backuppc.sh"
echo "mkdir -p /root/bin"
echo "cat > $SCRYPT << _EOF_ "
echo "#!/usr/bin/env bash
exec &> >(while read line; do logger -t \"\\\$0\" -i \"\\\$line\"; done)

while true
do
/usr/bin/ssh -NR 127.0.0.1:$PORT:127.0.0.1:873 -i /root/.ssh/id_rsa_backuppc -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ServerAliveInterval=300 -p 6667 -l $REMOTE_HOST_NAME 217.17.45.189
sleep 60
done
"
echo "_EOF_"
echo "chmod a+x $SCRYPT"
echo "####"
echo "# tunell start "
echo "####"
echo "# to crontab "
echo "( crontab -l ; echo \"#Start tunell ssh to backuppc \"; echo \"@reboot /root/bin/redirect_port_from_backuppc.sh &\" ) | crontab -"
echo "# first run: "
echo "nohup /root/bin/redirect_port_from_backuppc.sh &"
palza


echo "##################"
echo "# rsync konfiguration on $REMOTE_HOST_NAME "
echo "#################"
PRE_BACKUP="/root/bin/pre_backup.sh"
POST_BACKUP="/root/bin/post_backup.sh"
echo "cat > /etc/rsyncd.conf << _EOF_"
echo "timeout = 600
[backup]
path = $BACKUP_DIR
uid = 0
gid = 0
max connections = 1
comment = backap
auth users = backuppc
secrets file = /etc/rsyncd.secrets
exclude = $EXCLUDE
read only = yes
pre-xfer exec = $PRE_BACKUP
post-xfer exec = $POST_BACKUP
_EOF_
"
echo "cat > /etc/rsyncd.secrets << _EOF_"
echo "backuppc:$RSYND_PASSWORD
_EOF_
"
echo "chmod og-rw /etc/rsyncd.secrets "
palza


echo "##################"
echo "# pre and post backup on $REMOTE_HOST_NAME "
echo "#################"
echo "####"
echo "# file $PRE_BACKUP"
echo "####"
echo "cat > $PRE_BACKUP << _EOF_"
echo "#!/usr/bin/env bash
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin :/usr/bin/X11
exec &> >(while read line; do logger -t \"\\\$0\" -i \"\\\$line\"; done)
set -x
"

if [ "$USE_LVN_SNAPSHOT" == "YES" ] ;then
echo "
VG='$VG'
LV='$LV'
MIRROR='rootmirror'

mkdir -p /mnt/\\\$MIRROR/backup

lvcreate -n \\\$MIRROR -s -L 3G \\\$VG/\\\$LV
mount \\\$VG/\\\$MIRROR /mnt/\\\$MIRROR/backup

cp -a /boot/* /mnt/\\\$MIRROR/backup/boot/
"
fi
echo "_EOF_

"
echo "chmod a+x $PRE_BACKUP"

echo "####"
echo "# file $POST_BACKUP "
echo "####"
echo "cat > $POST_BACKUP << _EOF_"
echo "#!/usr/bin/env bash
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin :/usr/bin/X11
exec &> >(while read line; do logger -t \"\\\$0\" -i \"\\\$line\"; done)
set -x
"

if [ "$USE_LVN_SNAPSHOT" == "YES" ] ;then
echo "
VG='$VG'
LV='$LV'
MIRROR='rootmirror'

umount /mnt/\\\$MIRROR/backup
lvremove -f \\\$VG/\\\$MIRROR
"
fi
echo "
_EOF_

"

echo "chmod a+x $POST_BACKUP"

if [ "$USE_LVN_SNAPSHOT" == "YES" ] ;then
echo "mkdir -p /mnt/rootmirror"
fi
palza

echo "####"
echo "# start rsyncd on $REMOTE_HOST_NAME "
echo "####"
RSYNC="$RSYNC_LOCATION --daemon --config /etc/rsyncd.conf --bwlimit=$SPEED --address=127.0.0.1"
echo "#to cronab"
echo "( crontab -l ; echo \"#Odpalenie rsyncd przys starcie \"; echo \"@reboot $RSYNC \" ) | crontab -"
echo "# start rsyncd "
echo "$RSYNC"
palza

echo "####"
echo "# rsync test on backuppc - list all files that will be backuped. "
echo "####"
echo "RSYNC_PASSWORD=$RSYND_PASSWORD rsync -a -v -n --port=$PORT backuppc@127.0.0.1::backup /tmp/jakistest"





Wolodyj

[ Parent | Reply to this comment ]

Posted by Anonymous (203.208.xx.xx) on Mon 31 Mar 2008 at 06:25
Been using this software for 8 months already..

really loved it..

Saved my ass on quite a few occasion and most of the time i just need to check that the backups are done through the web administration page..

But recently i am thinking maybe we also can combine this with amanda so that we can backup the backuppc files to tapes and store it offsite to made this even a more complete backup solution.

[ Parent | Reply to this comment ]

Posted by Grimnar (85.221.xx.xx) on Tue 1 Apr 2008 at 02:03
[ Send Message | View Weblogs ]
I must be really stupid. Followed the guide, but ended up with this:

Error: Unable to read config.pl or language strings!!


Even tried +777 on /etc/backuppc and checking that backuppc was running BackupPC.

Checked the logs and got this:

Couldn't execute /etc/backuppc/config.pl: Insecure directory in $ENV{PATH} while running setuid at /etc/backuppc/config.pl line 46.
Debian/lenny.

[ Parent | Reply to this comment ]

Posted by coops (130.88.xx.xx) on Tue 1 Apr 2008 at 13:13
[ Send Message ]
I've followed this guide, and it's working great.

But I have one question.

I presumed that an incremental backup (via rsync) would only copy over the changed files. However it seems to copy over all files, then only store the changed ones.

There is one machine we want to backup which has to be done via the net, so copying all files nightly takes a large chunk of our bandwidth.

Is there a solution to this, as I'd love to keep using this program rather than going back to my custom rsync script.

Coops.

[ Parent | Reply to this comment ]

Posted by Grimnar (85.221.xx.xx) on Thu 3 Apr 2008 at 00:46
[ Send Message | View Weblogs ]
Still having problems with backuppc:/

[ Parent | Reply to this comment ]

Posted by martijnvanb (213.84.xx.xx) on Thu 3 Apr 2008 at 06:52
[ Send Message | View Weblogs ]
Hi,

Try this for the correct permissions:
chmod 755 /etc/backuppc
chmod 644 /etc/backuppc/*
chown -R backuppc /etc/backuppc
/etc/init.d/backuppc restart

[ Parent | Reply to this comment ]

Posted by Grimnar (153.110.xx.xx) on Thu 3 Apr 2008 at 15:39
[ Send Message | View Weblogs ]
Yeah, I belive I just tried that one. But I will try once again later tonight.

[ Parent | Reply to this comment ]

Posted by Grimnar (85.221.xx.xx) on Thu 3 Apr 2008 at 23:56
[ Send Message | View Weblogs ]
Okay
Got home and checked out some stuff.
gwdebian:/home/andreas# /etc/init.d/backuppc start
Restarting backuppc...No process in pidfile `/var/run/backuppc/BackupPC.pid' found running; none killed.
No language setting
BackupPC::Lib->new failed
gwdebian:/home/andreas# cd /etc/backuppc/
gwdebian:/etc/backuppc# perl -wc config.pl 
config.pl syntax OK
So, still not working for me:/ Google turned up some stuff but nothing helpful.

[ Parent | Reply to this comment ]

Posted by Anonymous (217.216.xx.xx) on Thu 10 Apr 2008 at 02:38
Have you checked is there's some bug for backuppc in Lenny?

http://bugs.debian.org/cgi-bin/pkgreport.cgi?pkg=backuppc;dist=un stable

[ Parent | Reply to this comment ]

Posted by Grimnar (85.221.xx.xx) on Thu 10 Apr 2008 at 12:36
[ Send Message | View Weblogs ]
Ah, well look at that!:D Thanks alot.

[ Parent | Reply to this comment ]

Posted by Grimnar (85.221.xx.xx) on Mon 14 Apr 2008 at 13:53
[ Send Message | View Weblogs ]
Still having problems, removed apache 1.3 then installed apache2.2 to see if that helped. But still the same issues with config.pl.

[ Parent | Reply to this comment ]

Posted by Anonymous (67.165.xx.xx) on Fri 18 Apr 2008 at 02:12
Thanks for a nice article.

If you are looking for a backup solution then check out dar. http://dar.sourceforge.net/

[ Parent | Reply to this comment ]

Posted by Anonymous (195.241.xx.xx) on Mon 30 Jun 2008 at 22:01
Is there still a chance on part 2?

[ Parent | Reply to this comment ]

Posted by Anonymous (149.99.xx.xx) on Wed 25 Feb 2009 at 15:18
If you're running into samba problems after upgrading to Lenny, remove the "-N" from the incremental and full backup smbclient lines in the config files.
(Found in the Fedora forums... seems to be an incompatibility with Samba 3.2)

[ Parent | Reply to this comment ]

Posted by Anonymous (77.74.xx.xx) on Mon 18 May 2009 at 16:17
Great howto !!! thanks

I follow this howto, but i've this error :

Read EOF: Connection reset by peer
Tried again: got 0 bytes
Done: 0 files, 0 bytes
Got fatal error during xfer (Unable to read 4 bytes)
Backup aborted (Unable to read 4 bytes)
Not saving this as a partial backup since it has fewer files than the prior one (got 0 and 0 files versus 0)

My backuppc server is an Ubtuntu and the workstation a lenny, the ssl works, i can connect on the lenny without password.

I you have a idea

Thanks

[ Parent | Reply to this comment ]

Posted by Anonymous (66.68.xx.xx) on Thu 21 May 2009 at 23:33
Is that last command to edit the authorized_keys file intended to be editing it on the CLIENT authorized_keys file since the previous command was to ssh to the client (command prompt suggest it's being edited on the server copy of authorized_keys) ?

[ Parent | Reply to this comment ]

Posted by Anonymous (84.246.xx.xx) on Tue 18 Aug 2009 at 22:13
The backup will be stored in /var/lib/backuppc/pc
acording to the doc's this can be set with
$topDir top-level BackupPC data directory
Where can i change this location ?

[ Parent | Reply to this comment ]

Posted by martijnvanb (194.13.xx.xx) on Tue 9 Mar 2010 at 11:37
[ Send Message | View Weblogs ]
Hi,

You can fix this by changing this value in /etc/backuppc/config.pl

Regards,
martijn

[ Parent | Reply to this comment ]

Posted by PatE (115.64.xx.xx) on Tue 23 Feb 2010 at 05:32
[ Send Message ]
Just a tip for others reading this great howto, I found I needed to change "from='aikido'" into "from='aikido.mydomain.com'". Apparently SSH does a reverse dns lookup on the client IP address and matches against the returned value. The returned name can be checked with "dig -x ip.of.client"

[ Parent | Reply to this comment ]

Sign In

Username:

Password:

[Register|Advanced]

 

Flattr

 

Current Poll

Which init system are you using in Debian?






( 1593 votes ~ 7 comments )