Booting Debian in 14 seconds

Posted by endecotp on Mon 10 Nov 2008 at 22:42

Many readers will have heard about Arjan van de Ven and Auke Kok's work to boot an ASUS Eee 901 in 5 seconds. Inspired by this work, and because I have the same laptop, I decided to try to reproduce their results. So far I have not come very close to their 5 seconds, but I have made some significant improvements compared to the default boot time for Debian on that machine; this article describes what I've done.

Although some of what follows is specific to the Eee 901, most of it isn't and could be applied to other laptops and PCs in general.

This article assumes that you're already familiar with things like building kernels, applying patches and so on. The target audience is the "advanced end user", and also the Debian developers responsible for the packages concerned who I hope will be motivated to incorporate some of this work.

It's worth noting that many of the things that are described here are already making their way into the upstream sources, so the lazy reader might decide simply to wait for all this fast-booting goodness to arrive in its own good time.

Instrumenting the boot process

Your first step should be to measure how the time is currently being spent while your machine boots. Then optimise the slow bits, and don't worry about the bits that are already fast.

A couple of tools are available for measuring the time taken during boot and visualising the results. I suggest that you install these tools first and save their results somewhere safe: I have not done so, and so I can no longer show you how slowly my machine booted before I started fixing it, which is a shame The total time was, IIRC, 33 seconds from the end of Grub to the xdm login dialog being visible; I've knocked 19 seconds off that.

bootchart

bootchart is available as a Debian package. Install it and boot with "init=/sbin/bootchartd" added to the kernel command line. (In Grub, select the kernel using the cursor keys, press e, select the line with the kernel command line, press e, edit, press return, and then press b.) Then run the bootchart utility which reads the log written during boot and creates an SVG graph. You can view the resulting file using most web browsers, or you can try "see" which will probably launch inkscape.

The bootchart will show you which processes took the most time, and you can also see how much time was spent waiting for I/O and how much time was CPU-limited. If the results don't seem to make much sense, try running bootchart with its -n option; this makes the results more verbose.

bootgraph

This similarly-named utility plots a graph showing how the kernel spent its time during initialisation, i.e. the blank period at the beginning of the bootchart. The script is included in the scripts/ directory of the kernel source, but I believe it is only in Linus' tree since 2.6.28-rc1. If you have an earlier kernel you can probably download the script alone; there is one kernel patch (to init/main.c) but I don't think it's vital unless you're also using asynchronous init calls, as described below.

To use bootgraph, boot with "initcall_debug" added to the kernel command line and then run "dmesg|perl scripts/bootgraph.pl > bootgraph.svg".

Fix the really obvious things

Before spending time on the hard stuff, fix these easy and obvious things:

  • Minimise the time that Grub waits before booting its default kernel by adjusting the timeout paramter in /boot/grub/menu.lst. I believe that the Debian default is 5 seconds.
  • Remove anything that takes time at boot that you're not using. (Personally I find it's easier to not install such things in the first place...)
  • If you're using a cpufreq governor, make sure that boot runs at full speed. (I load the powersave governor mainly because it makes it unlikely that the fan will ever come on - I don't like fans. However, when booting from cold it's unlikely that the fan will be needed even at full speed. So I load the cpufreq governor at S99.)

Now on to the more complex stuff.

Building a fast-booting kernel

There are a number of things that you can do to the kernel to make it boot faster:

  • You can eliminate the initrd or initramfs. These features make it possible for Debian to ship a kernel that will boot on a lot of different hardware without the bloat of building-in drivers for everyone's root disks. But it results in slower boot. If you build in the essential drivers for your root filesystem an initrd is not needed.
  • You can build in drivers for all of your hardware, rather than having udev load modules for them afterwards. Again this conflicts with a distribution's desire to provide a kernel package that works with all hardware, but by avoiding all the work that udev does loading modules this can make boot faster.
  • There are a few patches that reduce unnecessary delays during boot, described below.

Configuring a kernel with built-in drivers

I have been thinking about how a distribution like Debian could make it easier for users to create custom kernels that build in all of the drivers needed for their hardware. What I've come up with is the following:

  • The user boots a conventional Debian all-modular kernel, checking first that they don't have any extraneous USB devices or similar hardware attached.
  • The conventional udev startup will load all of the modules needed to drive their hardware.
  • lsmod will report which modules were loaded By some means we map from the module names to the kernel config settings that enable them, and change them from "m" to "y" so that they will be built in.
  • They then build and install a kernel with this new config.

The hard bit is the third step above. Luckily I found a script by Steven Rostedt that did almost what was needed - it did the hard part of mapping from module names to config settings - and I adapted it to buildin_used_mods.pl (local copy). Run this at the root of your kernel tree; it will write the new .config to stdout.

This script seems to do a good job, but it's not perfect. The particular problem that I found was that although it determines the correct config setting for the IDE hardware and sets it to "y", it doesn't know that it must also set the higher-level setting CONFIG_IDE to "y". Furthermore, when you "make menuconfig" it will detect this inconsistency and fix it in the wrong way by changing the IDE driver back to a module. The solution to this is to "make menuconfig" before running the script and to change CONFIG_IDE to "y". There may be other such problems; is there a way to automatically resolve them correctly?

A further useful but non-essential step, since it makes the kernel build more quickly, would be to disable all of those modules that are for internal hardware that we don't have, so that we only build modular drivers for things like USB devices.

So, could we have a Debian kernel package that did all of that automagically?

Kernel patches for faster booting

I have applied the following patches to improve boot time:

  • This patch, which I believe is in 2.6.28-rc1, eliminates some unnecessary locking in the driver-to-device matching code. Believe it or not, without this patch the pc speaker driver will wait until the mouse has been initialised (which may take several seconds) in order to check whether it is actually a speaker. Now, it still does the check but it doesn't take the lock before doing so. Of course it's not only that particular pair of devices but rather every pair of devices on every bus; it just happened to be that pair that wasted the most time in my bootgraph.
  • The Eee 901 uses PCI Express hotplug (pciehp) to toggle the Wifi power. This driver had a number of 1-second pauses which slow boot and also suspend/resume; all of them have now been eliminated for this hardware thanks to a couple of patches, this one which has made it into Linus' tree and I believe 2.6.28-rc2, and this one which hasn't.
  • One of Arjan's main innovations to achieve his fast boot time was to introduce more concurrency during the kernel startup: specifically, some drivers that are not on the critical path to getting the root filesystem mounted are initialised on an asynchronous thread. In particular, USB seems to take a while to initialise, as does the Eee's ACPI battery monitor. This work can be found in its own git tree. I'm not sure when we can expect to see this merged; for example, someone will have to decide which drivers should be on the async thread and which not, and the answer might be "it depends" in a lot of cases. Anyway, Arjan's choices are good for the Eee 901 and I have saved a bit of time by using it.

Eliminating coldplugging

In most modern Linux systems, whether or not they have modular kernels, soon after the kernel has booted the udev daemon performs "coldplugging". This enumerates all of the devices present at boot time and loads kernel modules, creates /dev entries, and does anything else necessary to get the device working. It's called coldplugging because these are the same operations that are done for hotplugged devices, except that they're not in response to hotplugging events.

Looking at bootcharts it's clear that this takes quite some time. Building all of the drivers in to the kernel, rather than having modules, makes some difference but that is not where all the time goes: even when the drivers are built in, the udev daemon will still run modprobe which wastes some time before realising that it's a no-op.

It may be possible to speed this up by making the udev system smarter in some way. But I've followed Arjan's approach and used a pre-populated /dev. For this to work, you need to be sure that:

  • The only action that udev would do for the devices is to create /dev entries. Often udev would load modules, but we don't have to worry about that as everything is built in. In principle, udev rules can carry out arbitrary actions though this is rare.
  • The device major/minor numbers aren't going to change from one boot to the next. I'm unclear about this and would welcome advice! For example, if the order in which disks appear is non-deterministic (as it is with USB devices) then this is broken.

I've also been told that HAL relies on udev and that X version 1.5 relies on HAL; since I use neither of these I don't know the whole story and it may be that the touchpad is the only affected device. Can anyone shed any light on this?

It's important to note that pre-populating /dev and not doing coldplugging does not mean that you have to give up hotplugging. The approach that I describe here still starts the udev daemon to handle hotplugged devices, and also removeable devices that are attached at boot.

It is relatively simple to use a fixed /dev on a "locked down" system, but it's more of a challenge to do it on a system like Debian which can run on different hardware. I have therefore used the following method:

  • Initially the system is booted with an unmodified udev system which does conventional coldplugging to populate /dev.
  • Immediately that coldplugging is finished, tar is used to record the contents of /dev.
  • On subsequent boots, the tar file is detected and coldplugging is not done but instead the tar file is extracted to create the contents of /dev. udevd is still used to handle hotplugging and coldplugging of removeable devices.
  • If at any time it's necessary to update the contents of /dev, perhaps because new hardware has been added to a desktop machine or if a new kernel has been installed, the tar file can be removed and the process is repeated.

I've implemented this by modifying the standard Debian /etc/initd/udev script; my modified version can be downloaded here (local copy). As you'll see if you diff that against your regular script, my changes are quite limited in scope and more than a bit hacky. No doubt the implementation could be improved, but first we need to decide whether this is the right strategy.

Disk read-ahead

Bootchart shows that the system spends quite a lot of its time at well below 100% CPU utilisation, waiting for the disk. A technique that Arjan and Auke used to alleviate this is read-ahead, i.e. to prefetch from the disk those files (or parts of files?) that it's known will be needed later in the boot. Debian already packages another readahead program, but Arjan and Auke have invented Super ReadAhead. I'm not aware of how it differs and it seems to lack documentation; however, I was able to get it to work by following the instructions posted on the download page by John Lamb.

The improvement resulting from read-ahead is worth having, but is not spectacular. It's a technique that's worth applying as well as everything else described here, but by itself I think you're unlikely to notice the improvement unless you use a stopwatch.

Setting the clock

Setting the clock, i.e. reading the hardware battery-backed clock into the kernel, seemed to be taking an inordinate amount of time. There turned out to be about 3 factors involved in this:

  • Debian sets the clock twice, via the hwclock.sh and hwclockfirst.sh init scripts. I'm still unsure why this is; see Debian bug 327584. I've removed one of the scripts and nothing seems to have broken.
  • On some systems, including the Eee until a recent kernel fix, hwclock's --directisa option was used. This option causes hwclock to use more CPU, so you should not enable it unless you believe that your combination of hardware and kernel needs it.
  • Most seriously, hwclock waits until the seconds in the hardware clock tick over; this will take on average half a second, except that in the case where hwclock is run twice (see above) the second invocation will take nearer a second. Fix this and the other problems don't matter any more.

The underlying issue with the last point is that the hardware doesn't tell us fractional seconds. So if we want our clock to be accurate we need to wait for the hardware to tick over. But do we actually need our clock to be that accurate? (And if we later run ntp, the inaccuracy will only be temporary.) If you're happy with your clock being wrong by up to plus or minus half a second, this patch that I knocked together adds a --notickwait option to hwclock. This makes hwclock almost instantaneous.

An alternative might be to run hwclock in parallel with other initialisation. The problem with this is that it can't start until /dev/rtc has been created and it needs to be done by the time fsck runs, and this is a fairly small window.

NFS

If you don't run NFS you can ignore this section - though you might like to double-check that you don't have any unused NFS packages installed that are slowing down your boot.

In my case, I use NFS with autofs on my Eee to access filesystems on other local machines. But this is something that I use only rarely, and certainly only when I'm at home. It turns out that there's a significant boot delay that can be avoided unless NFS was in use when the machine was last shut down.

The process to look out for is sm-notify, and it took up a big chunk of my bootchart with a very large associated peak in disk activity. It seems that the purpose of sm-notify is to send a message to those NFS servers that the machine was using before shutdown to tell them that it is now back up. But before starting to send these messages, it does something which has the side-effect of invoking sync() and causing all pending writes to be flushed out to disk. That takes ages.

This is especially wasteful in the case where you didn't use NFS at all during the last session, so there are no servers to notify. For me this is the common case. So I have written this patch against nfs-utils version 1.13 which detects the case when there are no servers to communicate with and terminates early, before the sync(). This patch has now been applied upstream and is included in nfs-utils 1.14 - however, there is a some doubt about whether it is really safe in all cases. You might want to review this thread to see if this has been resolved.

Starting X sooner

X takes a long time to start. At some point there should be a significant improvement to this when "kernel modesetting" is introduced - perhaps in 2.6.29. If you're keen you could try to use this now - you'll need kernel patches and a new X server - but I'm going to wait.

Some of the X startup time can be hidden by running it in parallel with other activity. At present, Debian starts xdm as the very last thing (at S99). gdm starts earlier at S30, but that's still quite late in the boot process. I now start X at S04.

Quite how early you're prepared to start it depends on what other services X depends on. In particular, does X need that the network is up? In some cases it makes sense to wait; an example would be when home directories are on NFS. However even in that case it would still be possible to start xdm and let the user type their username and password; if necessary it could wait for the network at that point. On a laptop, however, it's very unlikely that X (or anything much) will depend on the network being up. Perhaps something in the X packages could automatically detect or ask the user about these dependencies and start X at the earliest safe opportunity.

Note that if you start X early you may not want to shut it down late. Typically, startup and shutdown scripts are symetrical but you might want to make an exception in this case. The example that was pointed out to me was taking away networked filesystems before the programs that are using them have terminated. I've left xdm at K99.

Starting networking later

As noted above, on a laptop in particular it's unlikely that very much depends on networking being up during boot. And startng networking can be slow, especially if DHCP is involved. So I postpone starting the network until late in the boot where it will run in parallel with X starting up.

There are a couple of subtleties:

  • The driver for the Eee 901's wifi is an out-of-tree module that can't be built in to the kernel.
  • Network devices are a case where udevd does do more than just load modules and create /dev nodes.

I have therefore adopted the following scheme:

  • During initial coldplugging I skip network devices. When I'm using the pre-populated /dev I skip them anyway because I only coldplug USB devices, but when I'm not using the pre-populated /dev for some reason I still skip network devices. I have to match the wifi device by its PCI id since at that point the kernel hasn't recognised that it is a network device. This is in my modified udev script linked above
  • I have a coldplug_networking script that runs at S09, i.e. after xdm. This coldplugs the wifi device and the other network devices

Conclusions and future work

Using the methods described above, the boot time for my Eee 901 from the end of Grub to the xdm login dialog being visible has been reduced from about 33 seconds to about 14 seconds. Here are the bootgraph and bootchart for the system as it is now. Perhaps also of interest to Eee 901 users is my kernel config (local copies).

The "sore thumb" that still stands out in those 14 seconds is the startup time for X. (However, it doesn't stand out in the bootchart as that stops when the rc scripts have finished, which is several seconds before the login dialog appears.) But there is hope there, and I'm happy to wait for a few months and see how the kernel modesetting stuff pans out.

In addition to those 14 seconds, there's also the time taken by the BIOS before Grub runs; that seems to vary a bit, maybe 4 seconds when rebooting up to 10 seconds when powering on. It would be great to reduce that; maybe Intel are secretly working on this, or if not perhaps we could use Coreboot (AKA LinuxBIOS). I note that CoreBoot has recently announced support for some of the chips in the '901. This isn't something I'm planning to work on myself, but if someone would like to post a recipe for how to put CoreBoot on an Eee without bricking it, I'd love to see it!

I hope that this article inspires some other users to see what can be done on their own machines. Also, I hope that the Debian developers responsible for some of the affected packages can think about what they can do. So, over to you...

 

 


Posted by Anonymous (71.163.xx.xx) on Tue 11 Nov 2008 at 00:48
I made a custom kernel for my laptop so I could eliminate the initrd; it does save some boot time. (I don't bother with my desktop; it's currently been up 13 days so it's not worth the bother.) But I didn't build the drivers into the kernel because some of the power management scripts, like suspend and resume, will unload these drivers before suspending or hibernating. I could see if suspend and resume still work if I build all those drivers in, but I'm just lazy.

[ Parent | Reply to this comment ]

Posted by Anonymous (84.58.xx.xx) on Tue 11 Nov 2008 at 09:00
great inspiringly article - many thanks dude!
/thorsten

[ Parent | Reply to this comment ]

Posted by dmn (212.36.xx.xx) on Tue 11 Nov 2008 at 09:13

I thought a bit about the pseudo-static /dev and the main issue with it is that you need to re-create it when you add some hardware. Could this be automated somehow?

  • when creating the tarball, store the list of hardware (pci ids?, only storage-related devices?) along it
  • when checking for tarball existence also check if the hardware list matches. If it doesn't - recreate the tarball.

Caveats:

I am not sure about USB storage devices - they can change often but may well be needed for locating the root file system.

Also, creating/comparing device list would take some time.

Just an idea.

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Tue 11 Nov 2008 at 09:42
[ View Weblogs ]
Laptops have the advantage, compared to desktops, that adding internal hardware is unusual. As you say it would be possible to check the PCI IDs: maybe generate a checksum. It shouldn't take long to do if done right. You could even name the /dev tarball using the hardware checksums.

USB storage devices certainly can and do get different device numbers each time they're used, but I don't generate static /dev entries for them.

Thanks for the comments everyone.

Phil

[ Parent | Reply to this comment ]

Posted by Anonymous (141.84.xx.xx) on Tue 11 Nov 2008 at 09:15
Thanks, very nice, informative read. I especially liked the way you described how to edit the kernel line in GRUB - while it may be nothing new to most readers it probably will help first-timers.

[ Parent | Reply to this comment ]

Posted by Anonymous (62.40.xx.xx) on Tue 11 Nov 2008 at 10:56
I think all those changes made in init.d scripts and stuff around will be overwritten with every upgrade of any particular package. So it looks for me, that it's more usable for stable release, but... who uses stable on desktop?
And who want's to bother changing stuff after every upgrade?

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Tue 11 Nov 2008 at 11:27
[ View Weblogs ]
Part of my motivation for writing this article is to encourage these changes to be adopted by the package maintainers, avoiding the problem that you describe.

As for who want to bother, well this depends on your personal priorities obviously. Some people are happy to wait for a minute while their machines boot, but others are more impatient.

[ Parent | Reply to this comment ]

Posted by Anonymous (83.12.xx.xx) on Tue 11 Nov 2008 at 22:40
You seem to be using something different than Debian: on my system every upgrade is paused by prompt that says:
/etc/init.d/script has been modified,
do you want:
1) install new maintainer script,
2) keep current,
3) diff current-new,
4) stop upgrade to resolve...
and after install there are always files like /etc/init.d/script.dpkg-{old,dist}

[ Parent | Reply to this comment ]

Posted by Anonymous (57.66.xx.xx) on Tue 11 Nov 2008 at 11:00
Hi,

how much time did you spent with the analysis of your services, configuring, compiling the kernel, ...? If you boot your Debian each day and you saved 50 seconds of boot time, you will save in a year theoretically more than 5 hours... Theoretically. You spent the time for the analysis,... And you will spent your time again by configuring/compiling the kernel, or solving some issues with new hardware,... In real world, you will loose about 20 hours in a year. ;) BTW I don't boot daily.

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Tue 11 Nov 2008 at 11:22
[ View Weblogs ]
Your analysis is "selfish" in the sense that it assumes that I would be the only one ever to make use of the improvements. If you instead multiply the time saved by the number of people who would benefit - and my motivation for writing the article is to ensure that others will benefit - then there is a clear time saving. Open source development is about making contributions that will benefit others and in exchange receiving the benefits of the contributions that they make.

[ Parent | Reply to this comment ]

Posted by Anonymous (57.66.xx.xx) on Tue 11 Nov 2008 at 15:20
I am rather spending my time with my family, or with contribution to open source development, but this looks for me like - we have a nice desktop, but we want to speed up the computer by killing the desktop and using only the shell... Yeah, really innovative solution... If a new kernel is out, you will do probably more or less the same analysis. If you buy a new hardware, you will do analysis again (because you don't remember on your 3 computers which modules you have loaded...). Pretty selfish in the meaning I want to be the master of my time. ;) BTW - when I am booting my computer, I am doing some other activity, so there is no real saved time for me. If you want to contribute, solve bugs, make translations, develop new code.. What ever. By changing the order of the services you will speed up boot time of your computer, true, but the real world is out there.

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Tue 11 Nov 2008 at 15:36
[ View Weblogs ]
> we want to speed up the computer by killing the desktop and using
> only the shell...

I'm not sure where you got that idea from. It's not whay my article describes.

> If you buy a new hardware, you will do analysis again

No. My article links to a script that I have developed to automate the process of creating the correct kernel configuration, so no manual analysis should be required in that case.

> when I am booting my computer, I am doing some other activity

A pretty common scenario for me is, "Hey, look at this cool thing ... just a minute while I boot my laptop ... er, how are you? read any good books? planning a holiday? how are the kids? ...pause... OK, it's finally booted (or resumed)". Now look at my friend with his Mac that resumes from sleep in literally TWO SECONDS: "Look at this cool thing... da-da!".

> If you want to contribute [...]

I don't know who you are, but I have no need for your advice. I am quite happy with the extent and direction of my contributions.

[ Parent | Reply to this comment ]

Posted by Anonymous (83.60.xx.xx) on Tue 11 Nov 2008 at 22:42
Please, ignore the stupid trolling of the previous comment. Your article is great, and it will be pretty helpful to make maintainers more aware of possible optimizations to their packages.

Thanks!

[ Parent | Reply to this comment ]

Posted by Anonymous (198.24.xx.xx) on Tue 11 Nov 2008 at 19:57
By changing the order of the services you will speed up boot time of your computer, true, but the real world is out there.

In your "real world" you waste your time berating other's use of their time (and reading about what others do instead of doing yourself). I prefer the author's real world to yours.

[ Parent | Reply to this comment ]

Posted by Anonymous (194.114.xx.xx) on Tue 11 Nov 2008 at 13:44
Well, I don't mind about the boot time actually. These few seconds are not very important to me. But, what makes my life much easier and saves me much more time is the usage of hibernation.
Instead of shutting down my computer I send it do the S4 state (save on disk). When starting it up, it skips some of the boot time and (much more important) all applications are in exactly the same state as before. _That_ saves time.

Actually since I use hiberation, I shut down my computer at home several times a day. When going away from the computer for more than a few minutes I hibernate it. And additionally to saving time, it saves energy. ;-)

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Tue 11 Nov 2008 at 13:58
[ View Weblogs ]
Agreed, though some laptops still consume battery fast enough when asleep that shutting down is needed for longer breaks. And when I've finished using it I may not know if the forthcoming break will be a short one or a long one.

If you have an Eee or other machine that uses pciehp, I think you'll find that the two kernel patches related to that described in the article will also speed up resuming from hibernation.

[ Parent | Reply to this comment ]

Posted by Anonymous (122.104.xx.xx) on Wed 12 Nov 2008 at 03:03
Hibernation does not use power when active (it saves the memory to disk, then shuts down), suspend goes to a low power state instead so it does use power.

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Wed 12 Nov 2008 at 08:57
[ View Weblogs ]
Yes, sorry, I wasn't thinking straight when I wrote that.

The complaint I hear about hibernation (-to-disk) is that it takes a long time to read the data back off the disk, and if all that you're loading is e.g. your web browser's image cache you would have been better off rebooting, and using an application-level feature to restore your state. That's only hearsay though, I haven't used hibernation at all with my Eee.

[ Parent | Reply to this comment ]

Posted by Anonymous (93.195.xx.xx) on Wed 12 Nov 2008 at 21:54
Incidentally, I can confirm what you have heard. A year back, I've played around with the various kinds of suspend available for my regular desktop to achieve both: save boot time and get my work environment back.

Actually, a third point I had in mind was saving energy, of course, and that's why I first went for hibernation. But with that, you really end up with about the same shutdown/startup times as you have with a regular shut down and boot, thanks to memory image compression and disk storage.

So at this point, it was trading shutdown/startup time versus energy savings, and I went for the first ending up with suspend to ram. Since it's a desktop, I don't have to worry about batteries, but I'd still like to reduce as much standby energy waste as possible. Maybe your initiative opens up another door further down the road for ppl like me to go. So thanks for your work and valuable pointers.

[ Parent | Reply to this comment ]

Posted by Anonymous (84.249.xx.xx) on Tue 11 Nov 2008 at 16:42
Great work, hopefully it will find its way into Debian proper. However, I'd like to point out that you have omitted the startup of the desktop, which was included in the Arjan's 5 seconds. For testing/timing purposes it requires automatic logging in to be enabled.

The desktop will, in the default Debian configuration, take another 15-30 seconds for eg. GNOME or KDE to startup.

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Tue 11 Nov 2008 at 16:56
[ View Weblogs ]
Yes. Arjan booted xfce I think. I'm not sure how much faster that is than Gnome and KDE. My "14 seconds" number does include the time for X to start; personally I use xdm and then blackbox (on the desktop) or ratpoison (on the laptop).

I recommend the following pages - which were pointed out to me by Tomaz Solc on the Debian Eee list - for more about speeding up the Gnome start time:

http://mces.blogspot.com/2008/10/improving-login-time-part-1-gnom e.html http://mces.blogspot.com/2008/11/improving-login-time-part-2-gnom e.html

[ Parent | Reply to this comment ]

Posted by Anonymous (216.232.xx.xx) on Wed 12 Nov 2008 at 00:37
I have a fully booting system (X included) in about seven seconds, and I am still working to improve it even more. I use a v9fs-based device manager called Dev9, however, I am merely testing it. Notably, with a few glitches it populates the /dev directory in about 19ms. Compared to Udev's five seconds or more, it's a significant difference. With it's caveats, it still aims to replace quite a bit, and with it's (freestanding) counterpart initialization daemon, it may overcome slow boot up speeds by many orders of magnitude.

http://www.kyuba.org/

~syntropy

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Wed 12 Nov 2008 at 09:04
[ View Weblogs ]
Sounds very interesting. What hardware are you seeing 7 seconds on?

Perhaps you'd like to write an article about it ???

[ Parent | Reply to this comment ]

Posted by peterhoeg (222.68.xx.xx) on Wed 12 Nov 2008 at 01:45

Just a quick hint if you don't want to add the init=/sbin/bootchart part manually. You can instead add a new bootentry.

  1. Edit /boot/grub/menu.lst
  2. Find the line that says:
    # altoptions=(recovery) single
  3. Add another line after (do not delete the line above):
    # altoptions=(boot chart) init=/sbin/bootchartd
  4. Run: sudo update-grub

When the PC boots, hit Esc to enter the grub menu and select the entry bootchart entry to with bootcharting enabled.

[ Parent | Reply to this comment ]

Posted by mcortese (213.70.xx.xx) on Wed 12 Nov 2008 at 18:16
[ View Weblogs ]

Good job, endecotp. I was trying the same path, and came to similar conclusions.

I like very much your approach to selective coldplugging using the --subsystem-match option when synthesizing udev events. I hadn't worked this out myself: I just commented out the udevadm trigger line, which left me with nonfunctional USB devices if plugged before boot.

One question about the static contents of /dev: why don't you simply create the needed device files in /lib/udev/devices? The udev init script copies the contents of this directory to /dev before starting udev, so it serves exactly our purpose.

Regarding the real time clock, I find your solution of patching the hwclock program too intrusive. I solved the 1-second delay issue running hwclock in background: it is as easy as adding an "&" to the hwclock invocation in the *hwclockfirst init script (I, like you, got rid of the other hwclock init script right away).

I too discovered that DHCP was another big delayer, and decided that I don't need the LAN to be brought up until after X is started, so I simply removed the "auto eth0" in /etc/network/interfaces. I explicitly configure it at S99 with "ifup eth0".

I also have some further optimization ideas related to networking: why servers like sshd have to be started at init time? They should come into play as soon as the first network is configured, so the sshd script had better belong to the /etc/network/if-up.d domain. I'm still experimenting with this, more to come...

Finally, there is one very common trick I cannot find mentioned here: use dash instead of bash for running init scripts:

# aptitude install dash
# dpkg-reconfigure dash
and select that you want /bin/sh to be a link to dash. Since dash has a lot of commands hard-coded in it, and misses the featurism of bash, it is way faster.

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Wed 12 Nov 2008 at 18:27
[ View Weblogs ]
Thanks for posting, lots of interesting points...

/lib/udev/devices - simple answer, I didn't discover it until later! Post-hoc justification: I don't want to interfere with whatever other uses there are for that directory, and I think it could also be argued that tar is quicker.

hwclock& - the danger is that the clock is not yet set when some script needs the time to be correct; specifically, fsck apparently needs that the clock is set.

dash - very interesting; is there a safe way to change over to bash later? Have you measured how much time is saved?

Cheers, Phil.

[ Parent | Reply to this comment ]

Posted by mcortese (213.70.xx.xx) on Thu 13 Nov 2008 at 11:19
[ View Weblogs ]
dash - very interesting; is there a safe way to change over to bash later?
You don't need to. Your default interactive shell remains bash, and the scripts that begin with #!/bin/bash will be unaffected as well. Only scripts that begin with #!/bin/sh will be redirected to dash, but these must not rely on bash-specific features by design.
Have you measured how much time is saved?
I have a huge stack of "progressive" bootchart plots to search. If I can isolate the impact of dash over bash, I'll post the charts here.

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Sat 15 Nov 2008 at 11:58
[ View Weblogs ]
I've just tried dash, and the improvement is too small to measure :-(

[ Parent | Reply to this comment ]

Posted by Anonymous (80.70.xx.xx) on Thu 13 Nov 2008 at 19:59
It's interesting, but as you point out, this is only possible when you have a defined set of hardware and can't be easily be a general solution.
Last but not least :

1. "The device major/minor numbers aren't going to change from one boot to the next".
This will be wrong in the future : http://lwn.net/Articles/290141/ ( Block layer -> Extended partitions ).

2. "Starting networking later" : can't be done for a workstation in an enterprise where you have centralised user auth. But for laptop users, this is interesting and can be done indeed.

3. You may want to use parralel initscripts startup too. This may improve boot speed.

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Thu 13 Nov 2008 at 20:37
[ View Weblogs ]
Thanks for posting. Some quick comments:

> can't be easily be a general solution

By generating the static /dev by archiving a udevd-generated one, and by building a kernel based on the modules that are actually loaded for a modular one, I think it should be possible to approach a general solution.

> parralel initscripts

Arjan's analysis was that parallel init doesn't gain much; the important thing is to determine the critical path and to prioritise it.

Cheers, Phil.

[ Parent | Reply to this comment ]

Posted by TRS-80 (203.59.xx.xx) on Sun 7 Dec 2008 at 12:29
pam_ccreds and nss_updatedb can cache user information for use when the network isn't up.

[ Parent | Reply to this comment ]

Posted by Anonymous (80.70.xx.xx) on Thu 13 Nov 2008 at 20:02

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Thu 13 Nov 2008 at 20:52
[ View Weblogs ]
Thanks for the link; it describes efforts to improve boot time at Mandriva. Some highlights:

- They saved 2 seconds by not generating 256 ptys during coldplugging.
- They saved 2 seconds by not loading the floppy driver.
- They have attempted a couple of things to make modprobe faster, but without any improvement unfortunately.

They also seem to be making the assumption that building modules in to the kernel and using a pre-populated /dev is not practical with a general-purpose distribution, which is a view that I'm trying to challenge. Still no feedback from Debian kernel & udev people as far as I've seen - time to give them a poke...

[ Parent | Reply to this comment ]

Posted by mcortese (213.140.xx.xx) on Fri 14 Nov 2008 at 01:37
[ View Weblogs ]
Ehm... in your init.d-udev script, at row 292, I think you mean
tar cf /etc/init-devs.tar /dev
and not
tar cf /dev/init-devs.tar /dev

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Fri 14 Nov 2008 at 10:28
[ View Weblogs ]
I was uncertain about which filesystems were writeable when this ran, so I dumped it in /dev on the basis that that must be writeable as udevd was writing to it....

[ Parent | Reply to this comment ]

Posted by Anonymous (130.157.xx.xx) on Thu 5 Nov 2009 at 23:56
Here is my solution:

/etc/init.d/udev
http://pastebin.com/f37fcc246

/etc/init.d/udev-tar-move.sh #put this in your boot order after mountall
http://pastebin.com/f648db02e

[ Parent | Reply to this comment ]

Posted by Anonymous (77.65.xx.xx) on Sat 15 Nov 2008 at 16:54
Where is your fsck/mount? Each of these take about 3sec on my system.

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Sat 15 Nov 2008 at 17:35
[ View Weblogs ]
For me, mount takes very little time; the '901 has two SSDs of 4G and 16G and mounting the 16G drive takes 13ms.

I'm confused by fsck. Once every N mounts it does a proper check with an ASCII-art progress bar, taking a few seconds. My "14 seconds" number isn't this case - but fsck does still do a "light" check, does it? In this case, I think it's the bit marked "logsave" in the bootchart and it takes a couple of seconds. But I'm confused because looking at checkfs.sh it seems that it doesn't do any check at all when the laptop is running on battery power [why?] - yet I still see the same logsave in my bootchart with or without the mains power, and grep can't find anything else that might be running logsave.

Any ideas anyone?

[ Parent | Reply to this comment ]

Posted by mcortese (213.140.xx.xx) on Tue 18 Nov 2008 at 23:59
[ View Weblogs ]

I think that logsave doesn't actually use any CPU: it's just a wrapper that executes a command and keeps its output in memory until it is safe to write it down to a log file. The only reason it shows up in the bootchart is that it's launched at S10 when the file system is read-only, and can terminate only after all the file systems are mounted read/write.

[ Parent | Reply to this comment ]

Posted by Anonymous (69.226.xx.xx) on Sun 16 Nov 2008 at 00:54

You know, systems where the hardware clock uses UTC shouldn't need to run hwclock at all. Statically link your RTC driver, then CONFIG_RTC_HCTOSYS=y covers it on most platforms ... x86 doesn't even need you to configure in the RTC framework and driver, it will do the equivalent automagically (using arch-specific knowledge about the PC/AT RTC existing).

That is, if you don't dual boot, you can shave not just one second off boot time, but both of them... I'm kind of surprised that Debian still calls hwclock on boot at all, much less thinking it needs to do that twice.

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Sun 16 Nov 2008 at 12:13
[ View Weblogs ]
Thanks, I didn't know about that config option. But does the kernel implementation also wait until the clock ticks to the next second? And does it block everything while waiting or does it let other initialisation to run concurrently?

[ Parent | Reply to this comment ]

Posted by mcortese (213.140.xx.xx) on Sat 22 Nov 2008 at 22:51
[ View Weblogs ]

To answer your question, I looked at include/asm-generic/rtc.h and it explicitly says:

There is no need to to poll-wait (up to 1s - eeccch)
for the falling edge of RTC_UIP.
and the following code is just a series of CMOS_READ(). So I guess that this solution should be faster than the vanilla hwclock and on par with your patched version.

By the way, you must enable CONFIG_RTC_CLASS in order to see this option.

[ Parent | Reply to this comment ]

Posted by Anonymous (194.231.xx.xx) on Tue 18 Nov 2008 at 12:08
Have you considered using InitNG (http://www.initng.org/ )? It's stable, though the init files will propably need some tweaking. It starts processes asynchronous, so this might bring some more seconds.

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Tue 18 Nov 2008 at 12:17
[ View Weblogs ]
Several people have suggested init alternatives, but although I haven't investigated it myself Arjan's analysis was that this was not the right approach. I do suggest reading the LWN story linked at the start of the article if you haven't already done so.

If anyone has quantified the benefits of an init alternative, please post your results!

[ Parent | Reply to this comment ]

Posted by Anonymous (83.146.xx.xx) on Tue 18 Nov 2008 at 19:29
You can shorten the time taken by BIOS considerably by creating a 16 MB partition on hda with type 0xEF. After that there will be an option to enable "BootBooster" in the BIOS. This apparently caches this state of the BIOS to the disk which accelerates subsequent boots. See http://nathancoulson.com/proj_eee.shtml.

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Tue 18 Nov 2008 at 19:49
[ View Weblogs ]
Thanks for reminding me of that. I read about it a while ago but I have a problem: I filled my /dev/sda with a single root partition which I'll need to shrink. As far as I can tell, resize2fs supports offline shrinking and oneline growing, but not online shrinking. So the only way I can do this is to boot from something else. Hmm, I still have the Debian PXE installer on my tftp server and I could boot from that; can it shrink an ext3 partition? I'll report back.....

How common is this "Boot Booster" feature? If it could be detected (e.g. via ACPI tables or similar) it would be good if the Debian installer suggested creating the partition.

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Tue 18 Nov 2008 at 20:59
[ View Weblogs ]
Well I have managed to shrink my /dev/sda1 by 16 MB and created the Boot Booster partition, and as you say it does significantly improve the boot time. From pressing the power button to GRUB used to take about 10 seconds, and it's now down to about 7 seconds. Thanks!

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Wed 19 Nov 2008 at 14:53
[ View Weblogs ]
> it would be good if the Debian installer suggested creating the partition.

It seems that this is already filed, but with no response yet from the installer people:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=501517

[ Parent | Reply to this comment ]

Posted by Anonymous (128.139.xx.xx) on Wed 19 Nov 2008 at 18:18
I have a few suggestions, but first a correction:

That's incorrect. Udev doen't run modprobe on aliases for devices that already have modules associated with them (so udev running time should be shorter).
This also means that even with the modules _not_ built in, you might be able to speed things up by creating a list of all the modules loaded and then have a script that loads them run before udev is started, probably something like:
/bin/sh

MODULE_LIST=/etc/bla/module-list

function try_load_module(){
modprobe $@
}

while read module
do try_load_module $module &
done<$MODULE_LIST

wait

(the function is intentional... you can probably even use insmod to speed things even more.)

Some suggestions that might help speed things up:
- The Mandriva folk saved some time by getting rid of the legacy ttys, but there still are heaps of ptys/ttys... much more than needed, it seems to me (at least on a laptop). You can certainly live without most of them:
sh-3.00# ls /dev/ [pt] ty*
/dev/tty   /dev/tty1  /dev/tty3 &nb sp;/dev/tty5  /dev/tty7  /dev/tty9
/dev/tty0  /dev/tty2  /dev/tty4  /d ev/tty6  /dev/tty8

The simplest way to achieve this is to just copy the ones you want to /lib/udev/devices and change the lines with the pty[^a-z]/tty[^a-z] globs in the udev rules from "last_rule" to "ignore_device".

- You mentioned having to wait for udev to finish before setting the clock: just copy /dev/rtc to /lib/udev/devices.

- If the directory /dev/.udev exists on your root partition (i.e. in the static /dev), then the debian udev init script will _not_ mount a tempfs on top of dev,
so rather than play with the tarball, you can just create that directory and udevd will work with the static /dev (whether you do coldplugging or not).

- If you're running X with dri, then modprobing the drm module in advance (in the background) will help X start up faster (X runs a modprobe when it starts).

There's also a pretty simple way to speed up the modprobe alias lookup, but it's a little more than can be explained here (and I couldn't find an email address anywhere...).

Dougal

[ Parent | Reply to this comment ]

Posted by Anonymous (128.139.xx.xx) on Wed 19 Nov 2008 at 18:30
The quote seems to have got lost in my previous post:
"Building all of the drivers in to the kernel, rather than having modules, makes some difference but that is not where all the time goes: even when the drivers are built in, the udev daemon will still run modprobe which wastes some time before realising that it's a no-op."

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Wed 19 Nov 2008 at 20:00
[ View Weblogs ]
Hi Dougal,

Thanks for all of that very interesting information. Some brief comments:

> - Udev doen't run modprobe on aliases for devices that already have
> modules associated with them

Really? That's very interesting. After building a kernel with everything built in, I was still seeing very many modprobe calls from udevd in my bootchart -n. Perhaps there is some other explanation for where they come from. Perhaps my system was broken and the optimisation that you describe wasn't happening. Unfortunately I can't easily go back and look at this again.

It would be very interesting to see a bootchart for a machine that loads modules explicitly as you describe. Ideally another Eee 901 so that we can compare them. Can anyone help?

> just copy /dev/rtc to /lib/udev/devices

But that still happens while init.d/udev is running, so the earliest opportunity to run hwclock is still the same. Anyway, I'm happy for my clock to be half a second out until ntp runs, and with that change it's all moot.

> /dev/.udev ...
> rather than play with the tarball

Yes; lots of possibilities. However, if you want to do any sort of hardware change detection (i.e. fall back to the conventional system if something has changed) then having a tarball or directory that's keyed to the hardware configuration (PCI ID checksum or similar) is an advantage.

> I couldn't find an email address

http://chezphil.org/email/genemail.cgi

If you email something, I'll post a summary here.


Cheers, Phil.

[ Parent | Reply to this comment ]

Posted by Anonymous (128.139.xx.xx) on Fri 21 Nov 2008 at 09:55
>Really? That's very interesting. After building a kernel with everything built
>in, I was still seeing very many modprobe calls from udevd in my bootchart -n.
>Perhaps there is some other explanation for where they come from.

The reason for that is that there are many devices that are never supported by
any module.
To get an idea of how many, try the following:
/bin/sh
find /sys -name modalias | while re ad FILE
do
  if  [ -d ${FILE%/modalias}/d river ]  ; then
    echo "${FILE%/modalias}/driver  exists, skipping"
    continue
  fi
  read ALIAS < $FILE
  modprobe -b "$ALIAS"
done

find /sys/devices/pnp* -name id | while& nbsp;read FILE
do
  if  [ -d ${FILE%/id}/driver& nbsp;]  ; then
    echo "${FILE%/id}/driver exists , skipping"
    continue
  fi
  # multiple IDs in a file...
  while read ID
  do modprobe -b "pnp:d$ID"
  done <$FILE
done

...and see how many modprobe errors you get.
(I replaced the modprobes above with echos, then run the script into "wc -l" and
into "grep exists | wc -l" and got 89 and 50 -- so we're talking about a
sizeable portion of the devices...at least on my laptop.)
The majority of the unmatched aliases will probably be acpi/input (especially
from /sys/devices/LNXSYSTM:00).

All that said, I seem to have discovered something... it seems like the default
Debian udev rules do run modprobe always!
Rules that come with udev (in rules.d/80-drivers.rules):
DRIVER!="?*", ENV{MODALIAS}=="?*", RUN{ignore_error}+=" /sbin/modprobe $env{MODALIAS}"
Debian rules (hotplug.rules):
# load the drivers
ENV{MODALIAS}=="?*", \
RUN+="/sbin/modprobe --use-blacklist $env{MODALIAS}"

I suggest someone take the Debian udev maintainer out the back and spank him.

>> just copy /dev/rtc to /lib/udev/devices
>But that still happens while init.d/udev is running, so the earliest opportunity
>to run hwclock is still the same. Anyway, I'm happy for my clock to be half a
>second out until ntp runs, and with that change it's all moot.

That depends on if you wait for udev or not... I run udev in the background.
To do this, you can do something like this:
In the udev init script, before udevsettle is run, add "touch /tmp/.udevsettle".
Then move that init script somewhere where it will not be run at startup,
creating instead an alternate init script that looks something like:
/bin/sh
REAL_INIT_SCRIPT="/etc/where/you/put/it/udev"

case "$1" in
start) 
$REAL_INIT_SCRIPT start &
# Need to wait a bit, so basic  things like /dev/zero are available
while  [ ! -e /tmp/.udevsettle  ;]  
do usleep 200000
done
    ;;

    *) exec "$REAL_INIT_SCRIPT"&nbs p;"$1"
    ;;
esac

This way you just wait for the essentials to be available in /dev and you can
move on to start up X (I include the most basic devices, like /dev/zero,
/dev/null and /dev/console in /lib/udev/devices, anyway).

The reason I did this is that using bootchartd and plotting the cpu usage, I
found out that for most of the time Xorg takes to start up (between running
xinit and when xinitrc starts executing: 4-4.5 seconds with my Xorg-7.0), the
cpu is 80% idle! Seems to make sense to do it while udev is also using the cpu.
(In my case it only saves about a second -- Xorg startup has gone up to 6.5-7
seconds -- but that's with a 900MHz Celeron...)

Dougal

[ Parent | Reply to this comment ]

Posted by mcortese (213.70.xx.xx) on Wed 26 Nov 2008 at 18:47
[ View Weblogs ]

The reason for that is that there are many devices that are never supported by any module.
...And this is ok? I mean, this is the intended behaviour?

All that said, I seem to have discovered something... it seems like the default Debian udev rules do run modprobe always!
Let's suppose I replace the evil Debian-specific rule with the original one: what I get is that it does not modprobe for devices for which I already have a (static) device file in /dev. But I still get the modprobes for the acpi/input devices, right?

In that case, I would prefer endecotp's solution of selective trigger (via --subsystem-match).

What's your feeling?

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Thu 27 Nov 2008 at 20:39
[ View Weblogs ]
> what I get is that it does not modprobe for devices for which I already
> have a (static) device file in /dev.

No, it should do a modprobe only for the devices which haven't yet been claimed by any driver. Maybe someone could count how many modprobes this saves.

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Thu 27 Nov 2008 at 21:44
[ View Weblogs ]
> Debian rules (hotplug.rules):
> # load the drivers
> ENV{MODALIAS}=="?*", \
> RUN+="/sbin/modprobe --use-blacklist $env{MODALIAS}"

Apparently there is also this:
ENV{DRIVER}=="?*", SUBSYSTEM!="input", GOTO="hotplug_driver_loaded"

[ Parent | Reply to this comment ]

Posted by Anonymous (128.139.xx.xx) on Tue 2 Dec 2008 at 10:24
>Apparently there is also this:
>ENV{DRIVER}=="?*", SUBSYSTEM!="input", GOTO="hotplug_driver_loaded"

Oh, so I gather it skips them. I intended to look for something like that, but I guess I must have forgotten.
My apologies to the Debian udev maintainer...

Note that according to a recent udev announcement, there is no more need for the pnp hack (the rule that runs a subshell), since ACPI devices have now got proper modaliases.
The only problem is that it doesn't say since which kernel it has been that way... (and distro maintainers probably need to keep that in for backwards compatibility).

Dougal

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Thu 27 Nov 2008 at 21:12
[ View Weblogs ]
> If you email something, I'll post a summary here.

Following was sent by Dougal by email. There are some associated scripts which I have put at http://chezphil.org/tmp/speeding-modprobe.tar.gz


The guy from Mandriva mentioned in his blog how modprobe is slow, since it re-reads config files every time you run it.

I had noticed this, too, by observing that even running something like
time modprobe -b "pci:blabla"
would give approx 100 ms (with a 900MHz cpu) -- when we load nothing.

The reason for this is that if there's no module name matching what we pass to it (i.e. the alias), modprobe will do a regex comparison of the alias against every line of modules.alias -- several thousand lines.

Now, I noticed that most of the aliases in the system are of the types acpi:, pnp:, input: etc. -- the same ones that have very few entries in modules.alias. So what happens is that modprobe looks through lots of pci: and usb: alias lines for all types of devices. Further, most pci/usb devices tend to have modules already associated with them if we build a somewhat specialized kernel -- so most of the times modprobe is run are in fact for acpi/pnp/input etc.! (To get a rough idea of the types of aliases on your system, try "find /sys -name modalias -exec cat {} +" for most -- the rest are pnp:)

So I got the idea of only going through the acpi/input etc. aliases if the alias we have is of that type -- only running modprobe proper for pci/usb/pcmcia. Since the expressions in modules.alias that match acpi:/pnp: etc. look like globs, I figured I'd try them in a script with a case structure and it seems to work... much faster than modprobe (and since most aliases that have no matching modules at all are acpi: or pnp:, this can save a lot of time).

All that is required to use this is:

- The udev startup script should make sure the scripts exist for our kernel:

KERNVER=$(uname -r)
if [ ! -f /lib/modules/$KERNVER/findmods.sh ] ; then
echo "creating /lib/modules/$KERNVER/findmods.sh" &
make-loadmods.sh
fi
if [ ! -f /lib/modules/$KERNVER/loadpnpmods.sh ] ; then
echo "creating /lib/modules/$KERNVER/loadpnpmods.sh" &
make-loadpnpmods.sh
fi

- The udev rules for loading modules based on aliases should be changed to use modprobe.udev/loadpnpmods.sh:

DRIVER!="?*", ENV{MODALIAS}=="?*", RUN{ignore_error}+="/sbin/modprobe.udev $env{MODALIAS}"

SUBSYSTEM=="pnp", DRIVER!="?*", ENV{MODALIAS}!="?*", \
RUN+="/bin/sh -c 'while read id; do /lib/modules/$$(/bin/uname -r)/loadpnpmods.sh pnp:d$$id ; done < /sys$devpath/id'"

That's it... a little tacky-looking, but it works.

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Thu 27 Nov 2008 at 22:26
[ View Weblogs ]
> modprobe will do a regex comparison of the alias against every line
> of modules.alias

3258 lines for me, and it takes 83 ms. That could certainly be improved. How about (pseudo-code):

cat modules.alias | awk '
BEGIN { print "%%"; }
$1=="alias" { printf("%s\tprintf(\"%s\\n\");\n", gensub("\\*",".*","g",$2), $3); }
END { print ".*\t{}\n"; }
' > modules.l

flex modules.l
gcc lex.yy.c -ll

echo "tty-ldisc-2" | ./a.out
serport

echo "jkgberguerhg" | ./a.out


A quick experiment finds that flex barfs on the complete input, unfortnately. And the flex/gcc runtime is too long to run this in a "if modules.alias is newer , regenerate" method. But I would guess that an efficient scheme along these lines could be built. Would anyone like to try? (Who remembers most from their undergraduate grammar lectures? Who still has the textbooks?)

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Fri 28 Nov 2008 at 18:32
[ View Weblogs ]
I've just tried module-init-tools version 3.5 and it seems to be somewhat faster than 3.4. (3.5 hasn't made it into Debian yet.) Time for one no-op modprobe has gone down from about 70 ms to about 28 ms on my '901. There doesn't seem to be any explanation in the changelog, but a quick diff of modprobe.c finds mention of "binary indexes". It looks like it tries to open a modules.alias.bin file before trying modules.alias - of course I don't have a modules.alias.bin, yet it still gets this speedup. Perhaps you'll get even more if you know how to generate the .bin version. If anyone works this out, please post something.

modprobe seems to spend most of its time in its getline_wrapped() function, reading the modules.dep and modules.alias files. This processes the files byte at a time using getc_unlocked(). It typically calls malloc once per line but that's not the bottleneck. These days I typically use mmap() for this sort of thing. The difficulty is that you have to postpone any munging (e.g. in this case un-escaping \s) to the point of use, rather than doing it as the data is read.

Summary: you could make this faster, but not with incremental changes; you'd have to re-architect the whole thing. But maybe it's all fine if you have modules.alias.bin.

[ Parent | Reply to this comment ]

Posted by mcortese (213.70.xx.xx) on Mon 1 Dec 2008 at 18:46
[ View Weblogs ]

I think that modprobe was not designed to be fast in the first place: it traded speed with flexibility...

However, I'm a bit lost here. You originally proposed a solution that got rid of udev trigger (for unchanging, static devices) and now you're discussing about speeding up modprobes issued by udev trigger. IMHO your first idea was superior: why do we need to dynamically search for drivers when the HW configuration is static?

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Mon 1 Dec 2008 at 19:48
[ View Weblogs ]
My reason for posting this article specifically on this site was to encourage Debian to adopt some of these techniques. I now have the impression that getting any support for a static /dev into Debian is a non-starter; I've asked the Debian udev maintainer Marco d'Itri to look at this article and his very brief reply was along the lines of "there's a lot of stuff in /lib/udev and it's there for a reason". An approach that speeds up dynamic population might have more chance of adoption; it is at least worth investigating.

[ Parent | Reply to this comment ]

Posted by mcortese (213.70.xx.xx) on Tue 2 Dec 2008 at 13:02
[ View Weblogs ]
I've asked the Debian udev maintainer Marco d'Itri to look at this article and his very brief reply was along the lines of "there's a lot of stuff in /lib/udev and it's there for a reason".

Yes, he's not known for being very collaborative!

[ Parent | Reply to this comment ]

Posted by Anonymous (128.139.xx.xx) on Wed 3 Dec 2008 at 15:35
I just think it's something that's worth raising and looking into.
Looking into old code you can sometimes find absolute atrocities and, considering the fact that module-init-tools is one of that packages that are related to the kernel, yet not a part of it, I tend to assume it is neither developed as actively as the kernel, nor as actively as a completely independent project. And it's always good to have someone unfamiliar with the code have a look at it.

Dougal

[ Parent | Reply to this comment ]

Posted by Anonymous (128.139.xx.xx) on Sat 13 Dec 2008 at 11:32
Hey, I just found out why the new modprobe is faster: it creates the .bin files!
Look in /lib/modules/$(uname -r) and you'll see that modules.alias.bin, moduels.dep.bin and modules.symbols.bin are there..

Dougal

[ Parent | Reply to this comment ]

Posted by Anonymous (68.100.xx.xx) on Wed 3 Dec 2008 at 06:53
So with your recent progress, how much more time have you managed to shave off?

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Wed 3 Dec 2008 at 11:33
[ View Weblogs ]
The only thing here that's incremental is Boot Booster which saves about 3 seconds - however, that time wasn't included in the original measure. Everything else discussed is either ineffective (e.g. dash) or an alternative to what I've done (e.g. making modprobe faster).

[ Parent | Reply to this comment ]

Posted by Anonymous (85.221.xx.xx) on Mon 8 Dec 2008 at 11:11
I installed bootchart and everything is working great. But the bootchart image is FULL of sleep-sleep-sleep-sleep messages?

[ Parent | Reply to this comment ]

Posted by Anonymous (86.7.xx.xx) on Mon 8 Dec 2008 at 11:20
Bootchart itself seems to consist of a loop that sleeps and then does something. If I run bootchart with the -n option I see all of these. Is this what you're seeing? Scroll down to see the real activity.

[ Parent | Reply to this comment ]

Posted by Anonymous (85.221.xx.xx) on Mon 8 Dec 2008 at 11:51
Yeah, I was "extracting" the image with -n option. Thanks for the fast reply!

[ Parent | Reply to this comment ]

Posted by Anonymous (68.100.xx.xx) on Sat 13 Dec 2008 at 22:13
I've applied most of your suggested changes, but I've run into a bit of a problem. My /home is mounted separately on an sdhc card which my kernel takes at least 5 seconds to detect (the usb-stor-scan thread, according to my bootchart, lasts about that long).

This is problematic for two reasons:
-the boot process itself is so fast that fsck attempts to check that partition before the kernel can tell it exists, leading to a failed fsck and starting a shell (prolonging boot time)
-I'd have to put a sleep somewhere in my boot process in order to accomodate for the slow detection.

Any suggestions on how I might deal with this? Maybe there's a usb-stor-scan related kernel patch I missed? Right now, the only other resolution I can think of is to put /home back into /.

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Sat 13 Dec 2008 at 22:48
[ View Weblogs ]
Have you considered automounting it?

Otherwise, I think it's the same problem that I had here:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=482273
I don't think there's a good solution in the framework of the Debian boot scripts at present. Hacks are possible though.

Phil.

[ Parent | Reply to this comment ]

Posted by Anonymous (68.100.xx.xx) on Sat 13 Dec 2008 at 23:48
For me it's the positioning of the automount in rcS.d that was problematic; I've worked around it now by (rather unreliably) moving it just before GDM starts - about 8 seconds into my boot. The kernel has detected the block device holding my /home at around 7.2 seconds so it's really cutting it close. Bootchart is here:
http://xs134.xs.to/xs134/08506/bootchart539.png

Very hacky and ultimately a sleep is more reliable, but I'd like to avoid artificially prolonging my boot time if possible (the only other solution being put /home back on /).

[ Parent | Reply to this comment ]

Posted by endecotp (86.7.xx.xx) on Sun 14 Dec 2008 at 00:05
[ View Weblogs ]
I don't think you mean the same as me by "automount"; if you use autofs then it shouldn't need the device to be mounted until you have logged in and it looks for your home directory. If you don't want that, just hack in a loop that waits until the device has appeared (e.g. using test -f on the /dev node) in your rcS script.

[ Parent | Reply to this comment ]

Posted by Anonymous (68.100.xx.xx) on Sun 14 Dec 2008 at 01:04
Oh, okay. Yeah, I see what you mean. I'll have a look into autofs later today.

For what it's worth, what I ended up doing was changing a kernel parameter. usb_storage.delay_use defaults to 5 seconds - safe but long, tried changing it to 2. Worked like a charm and shaved a second off.

[ Parent | Reply to this comment ]

Posted by Anonymous (128.139.xx.xx) on Wed 17 Dec 2008 at 10:52
>My /home is mounted separately on an sdhc card which my kernel
>takes at least 5 seconds to detect (the usb-stor-scan thread,
>according to my bootchart, lasts about that long).
That is a default 5-second delay in the usb-storage module, which can be changed:
1) If usb-storage is built into your kernel, add the following to your kernel command line:
usb-storage.delay_use=0
(You should obviously make a separate Grub menu entry for it, so if it doesn't work you will still be able to boot...)
2) If usb-storage is a module (lsmod | grep usb[-_]storage), then you should add an option line to the modprobe config file (most likely in /etc/modprobe.d/). I think it will look like this:
option usb-storage delay_use=0

I boot off a usb-connected internal flash drive and it works fine without the usb-storage delay (connected thumb-drives are also detected faster and without a problem).

Dougal

[ Parent | Reply to this comment ]

Posted by pmate (212.14.xx.xx) on Mon 23 Feb 2009 at 15:13
[ View Weblogs ]
Hi, nice job.

I tried to speed the /dev part:
1) touch /etc/create-init.devs
2) rebooted
3) mv /dev/init-devs.tar /etc
4) rm /etc/create-init.devs
5) rebooted

The system boot speed it's 6 seconds faster and it's good, but X doesn't start anymore.
I have eeepc 701 - Debian Lenny

Regards

[ Parent | Reply to this comment ]

Posted by Anonymous (90.13.xx.xx) on Tue 10 Mar 2009 at 12:44
Please could you explain a bit more how you do the cold plugging, because it doesn't seem to work on my laptop. first tar the /dev? but after start-up?

[ Parent | Reply to this comment ]

Posted by endecotp (86.6.xx.xx) on Tue 10 Mar 2009 at 13:29
[ View Weblogs ]
The article links to my modified udev script, though it is hacky; I suggest using it only to get the general idea.

To be honest, if you can't work out what needs to be done with the tar file on the subsequent boot you're going to be lost by this.

[ Parent | Reply to this comment ]

Posted by Anonymous (90.13.xx.xx) on Tue 10 Mar 2009 at 16:23
Lol, Yeah i'm already lost, but i don't care, i just wanna try things ^^.

[ Parent | Reply to this comment ]

Posted by Anonymous (209.129.xx.xx) on Thu 7 May 2009 at 20:03
HAH! 13 seconds to XDM! You and your ponderous 14 seconds have officially LOST!
(Kidding.)


Details here:
http://ubuntuforums.org/showpost.php?p=7233608&postcount=523

[ Parent | Reply to this comment ]

Posted by Anonymous (86.6.xx.xx) on Thu 7 May 2009 at 20:38
Congratulations!

My Eee is currently back to slow-boot mode having upgraded the kernel and not yet re-run the magic script to determine the config settings to build in. It's on my to-do list.

I've had a quick look at your Ubuntu Forums message and I see it is number five hundred and something in the thread. I hope that's a sign that faster booting is getting some attention there. Despite the positive responses here to this article, I have seen zero interest from the relevant Debian package maintainers. Even LKML seemed uninterested in the config-munging script. I think it would be unfortunate if Moblin ended up as the only platform with decent boot speed, since they're only really interested in platform-specific solutions.

Do please keep me posted with any further improvements, especially if you try to do the kernel mode setting stuff.


Phil.

[ Parent | Reply to this comment ]

Posted by Stian1979 (114.44.xx.xx) on Sat 25 Jul 2009 at 12:08
I want to try your script, but don't know how to use it.
I think like you that this could be automated at instalation.
Let the install cd contain a booting kernel that write a config file to the memory and use this to make a new kernel from a kernel source located in a folder on the install cd.
Maybe the script also could add kernel upgrades to the blacklist. I gues it would be posible to make the script download, build and install a source kernel when it see that aptitude see a newer verson in the repo.

It would increase install and upgrade time, but a install can be done while I'm asleep and upgrading kernel don't hapend everyday.

[ Parent | Reply to this comment ]

Posted by Stian1979 (114.44.xx.xx) on Sat 25 Jul 2009 at 16:04
I want to try your script, but don't know how to use it.
I think like you that this could be automated at instalation.
Let the install cd contain a booting kernel that write a config file to the memory and use this to make a new kernel from a kernel source located in a folder on the install cd.
Maybe the script also could add kernel upgrades to the blacklist. I gues it would be posible to make the script download, build and install a source kernel when it see that aptitude see a newer verson in the repo.

It would increase install and upgrade time, but a install can be done while I'm asleep and upgrading kernel don't hapend everyday.

[ Parent | Reply to this comment ]

Sign In

Username:

Password:

[Register|Advanced]

 

Flattr

 

Current Poll

What do you use for configuration management?








( 523 votes ~ 7 comments )