Debian Administration

Setting up your own APT repository with upload support

We've previously covered setting up your own repository for the Debian's apt-get system, but we didn't cover managing automatic uploads. Thankfully this is a simple task with the reprepro, and dupload tools and a small amount of scripting.

The reprepro package is tool for creating an APT repository with a pool structure, the same type of structure the official Debian mirrors use.

The repository may:

• Contain packages for multiple distributions:
  • Stable, Unstable, Testing, etc.
• Contain packages for multiple architectures:
  • x86, sparc, all, etc.
• Be managed quickly and easily.

Installing the package is straightforward if you're using unstable, or etch, simply install it as you would install any other package:

apt-get install reprepro

Unfortunately the package contained within Debian's stable release, sarge, is a little outdated. For that reason I'd recommend that you install a back ported version. Thankfully this is readily available either from the projects homepage or elsewhere:

• From the download page of Alioth project.
• From my personal repository.

Once you've installed the software then we're ready to begin creating a repository. A new repository can be created in just a couple of steps:

• Decide where you wish the archive to be located.
• Create the configuration file.
• Import your first package

Almost certainly you will wish to serve your repository via a webserver, so the location should be beneath your webservers root directory at a simple path.

I use http://steve.org.uk/apt as my repository root location so my archive will be located beneath /home/www/www.steve.org.uk/htdocs/apt - your location will obviously differ.

First of all create the directory, a subdirectory conf/ to contain the configuration file, and a directory incoming/ to which we'll later setup automatic upload processing:

mkdir -p /home/www/www.steve.org.uk/htdocs/apt
mkdir -p /home/www/www.steve.org.uk/htdocs/apt/conf
mkdir -p /home/www/www.steve.org.uk/htdocs/apt/incoming

Now that we have a directory to contain our repository we can look at creating the configuration file. The configuration file will specify which releases the repository will contain (sid, stable, etc) as well as the architectures. A sample configuration file will look like this:

Origin: Your Name
Label: Your own label
Suite: stable
Codename: sarge
Version: 3.1
Architectures: i386 all source
Components: main non-free contrib
Description: Your description

Here we've defined a repository which only contains packages for stable/sarge, which contains packages targeted to x86, or all. If you wish to contain both stable and unstable packages your configuration file will look like this:

Origin: Your Name
Label: Your own label
Suite: stable
Codename: sarge
Version: 3.1
Architectures: i386 all source
Components: main non-free contrib
Description: Your description

Origin: Your Name
Label: Your own label
Suite: unstable
Codename: sid
Architectures: i386 all source
Components: main non-free contrib
Description: Your description

Save your configuration to the file conf/distributions and you should now be ready to import a package.

You can either import a .deb file into the repository, or a .changes file which is produced by building a package from source.

From the main directory run:

reprepro -Vb . include sarge name_of_file

For example:

skx@lappy:~/apt$ reprepro -Vb . include sarge \ 
    /home/skx/debian/sarge/reprepro/reprepro_0.6-1sarge0_i386.changes 
Created directory "./db"
Created directory "./pool"
Created directory "./pool/main"
Created directory "./pool/main/r"
Created directory "./pool/main/r/reprepro"
db: 'reprepro' added to 'sarge|main|i386'.
db: 'reprepro' added to 'sarge|main|source'.
Created directory "./dists"
Created directory "./dists/sarge"
Created directory "./dists/sarge/main"
Created directory "./dists/sarge/main/binary-i386"
  writing to './dists/sarge/main/binary-i386/Packages.new'...
  writing to './dists/sarge/main/binary-i386/Packages.gz.new'...
Created directory "./dists/sarge/main/binary-all"
  writing to './dists/sarge/main/binary-all/Packages.new'...
  writing to './dists/sarge/main/binary-all/Packages.gz.new'...
Created directory "./dists/sarge/main/source"
  writing to './dists/sarge/main/source/Sources.gz.new'...
Created directory "./dists/sarge/non-free"
Created directory "./dists/sarge/non-free/binary-i386"
  writing to './dists/sarge/non-free/binary-i386/Packages.new'...
  writing to './dists/sarge/non-free/binary-i386/Packages.gz.new'...
Created directory "./dists/sarge/non-free/binary-all"
  writing to './dists/sarge/non-free/binary-all/Packages.new'...
  writing to './dists/sarge/non-free/binary-all/Packages.gz.new'...
Created directory "./dists/sarge/non-free/source"
  writing to './dists/sarge/non-free/source/Sources.gz.new'...
Created directory "./dists/sarge/contrib"
Created directory "./dists/sarge/contrib/binary-i386"
  writing to './dists/sarge/contrib/binary-i386/Packages.new'...
  writing to './dists/sarge/contrib/binary-i386/Packages.gz.new'...
Created directory "./dists/sarge/contrib/binary-all"
  writing to './dists/sarge/contrib/binary-all/Packages.new'...
  writing to './dists/sarge/contrib/binary-all/Packages.gz.new'...
Created directory "./dists/sarge/contrib/source"
  writing to './dists/sarge/contrib/source/Sources.gz.new'...

This has imported the package described in the file reprepro_0.6-1sarge0_i386.changes to the archive, creating the appropriate directories as required.

To have a less verbose output simply omit the -V flag, thusly:

skx@lappy:~/apt$ reprepro -b . include sarge \
    /home/skx/debian/sarge/reprepro/reprepro_0.6-1sarge0_i386.changes 
skx@lappy:~/apt$

If you wish to remove a package from your repository you can do so with the remove command:

skx@lappy:~/apt$ reprepro -b . remove sarge reprepro
Deleting files no longer referenced...
deleting and forgetting pool/main/r/reprepro/reprepro_0.6-1sarge0_i386.deb
deleting and forgetting pool/main/r/reprepro/reprepro_0.6-1sarge0.dsc
deleting and forgetting pool/main/r/reprepro/reprepro_0.6.orig.tar.gz
deleting and forgetting pool/main/r/reprepro/reprepro_0.6-1sarge0.diff.gz

Note: you don't need to remove a package if you simply wish to replace an existing package with a newer version - this will be handled for you. When you import a newer version of a package contained in the archive already the older version will be removed.

Once your packages has been added to the archive they may be downloaded via apt-get, or aptitude, with the appropriate lines in your /etc/apt/sources.list file:

deb     http://example.com/apt  sarge main contrib non-free
deb-src http://example.com/apt  sarge main contrib non-free

If you're using two distributions simply repeat for each:

deb     http://example.com/apt  sarge main contrib non-free
deb-src http://example.com/apt  sarge main contrib non-free

deb     http://example.com/apt  sid main contrib non-free
deb-src http://example.com/apt  sid main contrib non-free

You should find that packages can be downloaded and installed as expected.

dupload is a tool which is designed to allow you to upload packages to different repositories.

Once installed (via "apt-get install dupload") it may be configured either via the file /etc/dupload.conf or ~/.dupload.conf.

To configure uploads for your remote host then simply add a section such as this:

$cfg{'example'} = {
        fqdn => "example.com",
        login => "steve",
        method => "scpb",
        incoming => "/incoming/",
        # The dinstall on ftp-master sends emails itself
        dinstall_runs => 1,
};

This configuration :

• Names the upload target 'example'.
• Specifies that file uploads should be conducted using scp.
• With the login name steve.
• That packages should be uploaded into the directory /incoming.

You will certainly need to change the hostname, login name, and incoming directory. You might also wish to specify an alternative means of uploading, such as anonymous FTP. For more details of the available options please see "man dupload.conf".

Once you've configured the upload settings you should be able to upload a Debian package by executing:

dupload --to example \
   /home/skx/debian/sarge/reprepro/reprepro_0.6-1sarge0_i386.changes 

(Note that you should upload the .changes file, not the .deb file)

Once this command completes you'll find that you've successfully transferred the package to your incoming directory, beneath your apt root.

The next step is to allow reprepro to automatically add that package to the repository.

As a result of any dupload commands we'll expect to process the incoming files from the incoming/ directory. We've already seen that reprepro can handle the importing of packages via the .changes file.

So we simply need to write a small shell script which will locate each .changes file, import the package, then clean the directory afterwards.

(We must remove, or move, the files from the incoming directory to make sure we don't repeatedly try to re-add the same package.)

This is a sample script which does the job. It is based upon the one I use myself and should be easily usable elsewhere - simply change the incoming directory at the top of the script:

#!/bin/sh

INCOMING=/home/www/www.steve.org.uk/htdocs/apt/incoming

#
# Make sure we're in the apt/ directory
#
cd $INCOMING
cd ..

#
#  See if we found any new packages
#
found=0
for i in $INCOMING/*.changes; do
  if [ -e $i ]; then
    found=`expr $found + 1`
  fi
done


#
#  If we found none then exit
#
if [ "$found" -lt 1 ]; then
   exit
fi


#
#  Now import each new package that we *did* find
#
for i in $INCOMING/*.changes; do

  # Import package to 'sarge' distribution.
  reprepro -Vb . include sarge $i

  # Delete the referenced files
  sed '1,/Files:/d' $i | sed '/BEGIN PGP SIGNATURE/,$d' \
       | while read MD SIZE SECTION PRIORITY NAME; do
        
      if [ -z "$NAME" ]; then
           continue
      fi

      #
      #  Delete the referenced file
      #
      if [ -f "$INCOMING/$NAME" ]; then
          rm "$INCOMING/$NAME"  || exit 1
      fi
  done

  # Finally delete the .changes file itself.
  rm  $i
done

To use this simply setup a cronjob running every few minutes to process any new files in the incoming/ directory - something like this:

*/5 * *  *  * /usr/local/bin/import-new-packages.sh

Add Comment Printable version

 

<<< WebDAV on Apache2 Question: Setting up printing in Sarge >>>