Offline Package Management for APT
Posted by riteshsarraf on Sat 24 Oct 2009 at 14:43
This article is about Offline Package Management in Debian. Debian is a pretty well known project. One of the things that makes Debian very popular is APT (a.k.a Advanced Packaging Tool) which allows remote package downloads, upgrades and dependency resolution. Unfortunately it does require a network connection - unless you use apt-offline.
In Debian, when you need to install a package, you usually would fire up the apt-get command and the software would just install without any hand holding.
While APT is really very cool one of the main reasons for its success is the Debian Policy. The Debian Policy is like the brain of the project that controls the entire project ensuring that all the bits and pieces fit well together upto the Debian Standards. APT is just a result of the fantastic Debian Policy work.
In Debian, every package is very well self-contained and is tightly related to each other using APT. APT does a very good job of integrating and resolving dependencies for Package Management and takes off all the Dependency Hell problems from the user.
This is where the problem starts - for a machine which has network access it works very well because APT generates the list of packages and their dependencies and is able to download and install them successfully.
But when it comes to downloading a package individually on a different machine, along wih resolving any dependencies this can be a big problem.
Consider this real world example: I have a Debian box at home. At home, I have no (or very slow/expensive dial-up) internet connection. At work, I (or my friend) do have a very fast connection but (as part of IT policy) am required to use Windows.
I would still like to be able to painlessly update/upgrade my Debian box at home, with all the power and flexibility of APT.
This is where apt-offline is useful. apt-offline is an offline APT Package Manager.
Using apt-offline:
- You generate a signature on your Debian box at home and carry the signature file on a removable medium (Probably a USB Stick).(e.g. "apt-offline set /tmp/apt-offline.txt")
- Now you take the USB Stick (with the apt-offline.txt signature file) to the office machine which could be running any linux version, or as I mentioned above, even Windows.
- There, you could run apt-offline giving it the signature file. (e.g. "apt-offline get C:\apt-offline.txt")
- apt-offline would generate you an archive file or a folder with all the data. That data can be copied on a removable media. The removable media can be attached back to the disconnected Debian box at home and installed. (e.g. "apt-offline install /tmp/apt-offline.zip")
Let's start with a 3 step example
Step 1
Generate a signature file on the Disconnected Debian box at home
apt-offline set /tmp/apt-offline.sig
The above command will generate all information required from apt about updating its database.
By default, with no additional arguments passed, apt-offline will extract information about APT Package Database Update i.e. the --update option as well as the list of Packages to be upgraded i.e. the --upgrade option.
These options can also be individually passed if you want only one of those.
Step 2
Download data based on the signature file generated earlier
apt-offline get C:\apt-offline.sig --threads 5
The above command will download data as mentioned in the signature file. To speed up downloads (that can be from multiple apt repositories), in this example we spawn 5 download threads.
Note: It would be good to also download the bug reports for the packages that you are downloading. So that example now becomes:
apt-offline get C:\apt-offline.sig --bug-reports --threads 5
There are many more options that you can pass to apt-offline, like the --bundle option which would generate for you, an archive file with all the data.
Once completed, you could just copy the data (an archive file, if you used the --bundle option) back to the removable medium and copy it back onto your offline host.
Step 3
Once you're back upon the home Debian machine, you feed the data from the removable medium to apt-offline:
apt-offline install /media/USB/apt-offline.zip
This will update the APT database on your disconnected machine seamlessly.
If there were packages that needed to be upgraded, now they would all be available (with dependencies) in the APT database. So if you do an apt-get upgrade now, APT won't prompt you mentioning even a single bye download. APT would find that all required packages are already present in the APT cache.
If you had used the --bug-reports switch that I mentioned earlier, during install apt-offline would prompt you with the list of bug reports related to the packages on your machine that need be upgraded/installed - not just the list but the full bug report will be available for you to look at and evaluate the severity involved.
As you can see from the article above, apt-offline helps you achieve the power of APT, in just 3 steps. apt-offline is part of Debian and is developed at Alioth.
thanks
[ Parent | Reply to this comment ]
My plans are to directly use python-apt if available. Support for aptitude is also planned, I just need to find out a way to generate the list of URIs in aptitude.
[ Parent | Reply to this comment ]
thanks
[ Parent | Reply to this comment ]
That's exactly what I wanted a few years ago. As it happens the PC I need it for now has fast network and I've migrated to using aptitude for all my upgrades now.
I can still see a need for this tool though, there is nothing worse than installing a box knowing that you have a nasty wait for the upgrades to come down a slow pipe - and a local cache of updated packages would always be useful.
--
"It's Not Magic, It's Work"
Adam
[ Parent | Reply to this comment ]
The signature file is called apt-offline.sig on the first step and then on the next steps it's called apt-offline.txt
Thanks
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
Step 2 is the get operation, whose sole purpose is to only fetch the required data (debs, bug reports, apt package database updates) from the internet.
You will need python installed on the windows box. Then just download the apt-offline zip archive from the alioth project page, unzip it and execute it.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
At work (non-tech related business), users can not install applications on the PC and we're given a minimal amount of software of which Python is not included.
You can put Python on the same stick you plan to download to, and just add X:\<Python Directory>\bin to PATH and continue as normal.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
View this image to see what message i take if i try to build it.
http://f.imagehost.org/0646/image.jpg
Sorry for my english :/.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
c:\apt-offline> python apt-offline set C:\set.uris
There were some minor issues with the setup.py file. They have been fixed and should be part of the next (0.9.7) release.
[ Parent | Reply to this comment ]
Please see this image http://i.imagehost.org/view/0940/Screenshot
[ Parent | Reply to this comment ]
Are you installing it through the deb package that is available in Debian at:
http://packages.debian.org/search?keywords=apt-offline
From the screenshot it looks like, it is installed under /usr/lib/python2.5/site-packages/ but you seem to also be in ~/Desktop/apt-offline/ folder, which gives the impression that it might be an external installation.
Please file a bug report at http://apt-offline.alioth.debian.org in case you still see the problem.
[ Parent | Reply to this comment ]
I do not know why there was a problem before, maybe I had something wrong
[ Parent | Reply to this comment ]
apt-offline set --install-packages rcconf -- /tmp/apt-offline.sig
and I get the following error: Couldn't find package rcconf
[ Parent | Reply to this comment ]
For that, run:
apt-offline set /tmp/apt-offline.sig --update
It will generate the .sig file. Then run the get operation on the network machine. Then run the install operation on the disconnected Linux box with the data file.
Once the apt database is available, apt-offline will be able to generate the dependency for rcconf and its dependency packages.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
Actually I have several problems with apt-offline.
When I start apt-offline get apt-offline.sig
the following errors occured:
debian:/usr# apt-offline get apt-offline.sig
/usr/lib/python2.5/site-packages/apt/__init__.py:18: FutureWarning: apt API not stable yet
warnings.warn("apt API not stable yet", FutureWarning)
Fetching APT Data
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.5/threading.py", line 486, in __bootstrap_inner
self.run()
File "/usr/lib/python2.5/threading.py", line 446, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/pymodules/python2.5/apt_offline_core/AptOfflineLib .py", line 578, in run
self.responseQueue.put( self.WorkerFunction( item, thread_name ) )
File "/usr/lib/pymodules/python2.5/apt_offline_core/AptOfflineCor eLib.py", line 540, in DataFetcher
(url, file, download_size, checksum) = stripper(item)
File "/usr/lib/pymodules/python2.5/apt_offline_core/AptOfflineCor eLib.py", line 304, in stripper
size = int(string.rstrip(string.lstrip(''.join(item[2]), chars = "'"), chars="'"))
ValueError: invalid literal for int() with base 10: '5.0.4'
Downloaded data to /tmp/apt-offline-downloads-3467
Can anyone help me?
[ Parent | Reply to this comment ]
And the content of the apt-offline.sig file, can you post it ?
[ Parent | Reply to this comment ]
I installed apt-offline_0.9.7_all.deb with python-argparse_1.1-1_all.deb and python-support_1.0.7_all.deb
Here the content of the .sig-File:
'http://volatile.debian.org/debian-volatile/dists/lenny/volatile/m ain/binary-i386/Packages.bz2'; volatile.debian.org_debian-volatile_dists_lenny_volatile_main_bin ary-i386_Packages 0 :
'http://volatile.debian.org/debian-volatile/dists/lenny/volatile/c ontrib/binary-i386/Packages.bz2'; volatile.debian.org_debian-volatile_dists_lenny_volatile_contrib_ binary-i386_Packages 0 :
'http://volatile.debian.org/debian-volatile/dists/lenny/volatile/m ain/source/Sources.bz2'; volatile.debian.org_debian-volatile_dists_lenny_volatile_main_sou rce_Sources 0 :
'http://volatile.debian.org/debian-volatile/dists/lenny/volatile/c ontrib/source/Sources.bz2'; volatile.debian.org_debian-volatile_dists_lenny_volatile_contrib_ source_Sources 0 :
'http://volatile.debian.org/debian-volatile/dists/lenny/volatile/R elease'; volatile.debian.org_debian-volatile_dists_lenny_volatile_Release 0
'http://volatile.debian.org/debian-volatile/dists/lenny/volatile/R elease.gpg'; volatile.debian.org_debian-volatile_dists_lenny_volatile_Release. gpg 0
'cdrom:[Debian GNU/Linux 5.0.4 _Lenny_ - Official i386 DVD Binary-1 20100131-19:16]/dists/lenny/contrib/binary-i386/Packages.bz2' Debian%20GNU_Linux%205.0.4%20%5fLenny%5f%20-%20Official%20i386%20 DVD%20Binary-1%2020100131-19:16_dists_lenny_contrib_binary-i386_P ackages 0 :
@
[ Parent | Reply to this comment ]
Problem is, you have the cdrom repository enabled in your apt database. For now, if you want to use apt-offline, you will have to disable the cdrom repository in your apt sources.list and rerun the set command.
In the next release (0.9.8), apt-offline will check and skip the cdrom repository items. Thanks again for uncovering this.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
C:\Python31\Lib\site-packages\apt-offline>python apt-offl ine get .\tmp\apt-offline.sig --threads 5
Traceback (most recent call last):
File "apt-offline", line 25, in&n bsp;<module>
from apt_offline_core.AptOfflineCore Lib import main
File "C:\Python31\Lib\site-packages\apt-offline\ apt_offline_core\AptOfflineCor
eLib.py", line 242
except socket.timeout, timeout:
; &nbs p; ^
SyntaxError: invalid syntax
What to do?
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
Is this a problem of apt-offline or it is other problem?
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
I can manually get the files one by one by using the url's in IE.
But, haven't a clue what to do with them once I get them.
Any pointers on how to assemble the files so that apt-offline can use them on the Linux side would be most helpful.
Thanks!
BTW: Supporting Linux in a non-net-connected system. So, we really need a way to get the updates on a locked down Windows Machine.
[ Parent | Reply to this comment ]
And if so, what kind of proxy it is?
[ Parent | Reply to this comment ]
I cann't get the apt-offline work for me,I want install emacs,samba.But when I follow the steps,I cann,t install them too!
this is the way I did:
on ubuntu(no internet):
apt-offline set /tmp/apt-off-emacs.sig --release emacs
on windows(fast internet):
apt-offline get C:\\apt-off-emacs.sig --threads 2 (I got a folder)
back ubuntu:
apt-offline install /home/...(the folder)
and after a while when the apt-offline install command finished,nothing have been changed! After that I tried apt-get install emacs,nothing happened too!
so can you give me an example of how I get specific soft like emacs!
because I cann't find the example!!
thank you!
[ Parent | Reply to this comment ]
from what you have mentioned, --release option requires a valid release, something like testing, unstable or stable.
So your correct command would be: "apt-offline set /tmp/apt-off-emacs.sig --release unstable --install-packages emacs samba"
More details available with: apt-offline set -h
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]