Creating dynamic swap space

Posted by Steve on Thu 27 Sep 2007 at 06:32

When a GNU/Linux machine runs out of physical memory it will start to use any configured swap-space. This is usually a sign of trouble as swap files and partitions are significantly slower to access than physical memory, however having some swap is generally better than having none at all. The size of swap allocated to files, or partitions, is usually chosen arbitrarily with many people adopting the "double the memory size" rule of thumb. Using a dynamic system can ease the maintainance of this size.

The relatively unknown dphys-swapfile package contains a simple script which will create and activate a swapfile at boottime which is sized appropriately for your system.

The advantage of this dynamic creation is that the swap will be resized automatically if you upgrade your memory and don't remember to do it.

Upon recent kernels there doesn't appear to be a significant penalty to using swap files as opposed to swap partitions. With this in mind I'd recommend files rather than partitions, to give yourself more flexibility.

To get started first remove any existing swap you have allocated. You can view any swap space which is in use by running:

skx@vain:~$ /sbin/swapon  -s
Filename                                Type            Size    Used    Priority
/dev/md1                                partition       2931768 557428  -1

Here we see that there is swap allocated to the physical raid volume /dev/md1. We can disable that by running:

root@vain:~# /sbin/swapoff /dev/md1

Once it is gone we can now install the package upon installation the system will create and activate then new swap:

root@vain:~# apt-get install dphys-swapfile
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed
  dphys-swapfile
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 9572B of archives.
After unpacking 111kB of additional disk space will be used.
Get: 1 http://apt-cache sid/main dphys-swapfile 20061020-1 [9572B]
Fetched 9572B in 0s (60.5kB/s)       
Selecting previously deselected package dphys-swapfile.
(Reading database ... 116206 files and directories currently installed.)
Unpacking dphys-swapfile (from .../dphys-swapfile_20061020-1_all.deb) ...
Setting up dphys-swapfile (20061020-1) ...
Starting dphys-swapfile swapfile setup ...
computing size, want /var/swap=1876MByte, generating swapfile ... of 1876MBytes
done.

Now whenever you boot you'll have /var/swap created at a size of twice your amount of physical memory, automatically.

You can verify this yourself with the swapon command we demonstrated earlier:

skx@vain:~$ /sbin/swapon  -s
Filename                                Type            Size    Used    Priority
/var/swap                               file            1921016 0       -3

If you wish to change the location, or size, of the generated swapfile please create the file /etc/dphys-swapfile and give it contents such as this:

# /etc/dphys-swapfile - user settings for dphys-swapfile package
# author Neil Franklin, last modification 2006.09.15
# copyright ETH Zuerich Physics Departement
#   use under either modified/non-advertising BSD or GPL license
# this file is sourced with . so full normal sh syntax applies

# where we want the swapfile to be, this is the default
CONF_SWAPFILE=/swap.file

# size we want to force it to be, default (empty) gives 2*RAM
CONF_SWAPSIZE=2048

##
#  Give yourself three times the memory size of swap?
#
# mem=$(grep MemTotal /proc/meminfo |awk '{print $2}')
# CONF_SWAPSIZE=$(expr $mem \* 3)
#

 

 


Posted by GhostR (217.237.xx.xx) on Thu 27 Sep 2007 at 07:47
[ Send Message | View Weblogs ]
OK, I never heard of this, sounds interessting tough. I ll try it on a test machine. SO whats the cons in doing this vs. a real swap partition? I guess there has to be some otherwise it would be more known and maybe even state of the art.

[ Parent | Reply to this comment ]

Posted by ajt (204.193.xx.xx) on Thu 27 Sep 2007 at 09:20
[ Send Message | View Weblogs ]
Interesting.

Are swap files really as fast as swap partitions? Cool.

I know people with lots of RAM particularly on desktop or laptop systems don't want any swap at all, as they claim it slows the system down too much. Even the LPI doesn't advise double RAM as swap once you get over the 1Gig mark on a desktop system.

Our SAP systems have silly amounts of swap because SAP demands it, 8 Gig of swap compared with 4 Gig of RAM. 99% of the time the RAM isn't even half used and the swap is unused almost all the time.

--
"It's Not Magic, It's Work"
Adam

[ Parent | Reply to this comment ]

Posted by paulgear (124.171.xx.xx) on Thu 27 Sep 2007 at 21:51
[ Send Message ]
Think about the relative cost of the following: - Using 8 GB of a 160 GB disk for swap - Arguing with SAP over support for your installation when you admit to them that it has less than the recommended swap space. I know which one i'm going to choose... ;-)

[ Parent | Reply to this comment ]

Posted by ajt (84.12.xx.xx) on Thu 27 Sep 2007 at 23:06
[ Send Message | View Weblogs ]
We don't have anything like 160GB of disk, SCSI disk space is/was expensive, our fairly recent boxes have only 40 GB disks. 8GB of unused swap space on those boxes is a fair sized chunk of space.

In this case there is actually space (the real data is on the AIX box and a SAN), the point is that they require 8GB of swap irrelevant of the actual RAM and CPUs you have. Too many of SAP's guidelines are too crude and inflexible for reality.

You are correct, you don't argue with SAP no matter how many times they are wrong and you are right. SAP couldn't write a decent bit of code, documentation or instruction if their life depended on it. I've never seen as much rubbish in my life before, but it's what we have and their word (however hard it is to interpret) is LAW...

--
"It's Not Magic, It's Work"
Adam

[ Parent | Reply to this comment ]

Posted by paulgear (124.171.xx.xx) on Fri 28 Sep 2007 at 00:40
[ Send Message ]
It doesn't matter how expensive SCSI disk is, it's still cheaper than arguing with SAP... ;-)

[ Parent | Reply to this comment ]

Posted by Anonymous (194.86.xx.xx) on Thu 27 Sep 2007 at 10:05
So does anyone know what the actual recommended size for swap is nowdays when computers are running with 4-16G memory?

i'm having trouble seeing 8-32G swap space being utilized, or am i wrong?

[ Parent | Reply to this comment ]

Posted by Steve (82.32.xx.xx) on Thu 27 Sep 2007 at 22:11
[ Send Message | View Steve's Scratchpad | View Weblogs ]

Depends on your kernel, your load, and your configuration. Like I said the general rule of thumb has been double-your-memory. By the time you're swapping IO will kill performance, so in some cases it might be best to have none - and just add real memory..

Steve

[ Parent | Reply to this comment ]

Posted by simonw (84.45.xx.xx) on Mon 1 Oct 2007 at 02:41
[ Send Message | View Weblogs ]
I pondered this a while back.

I think (but I'm not 100% sure) that it is best to have some swap for most systems, as it can be used to page out allocated memory that is rarely used, clearing physical memory to be used for caching I/O more efficiently, thus allowing a slight performance gain over the same system with no swap at all. The idea is swap has the stuff that should never have been allocated in the first place, or should have been freed.

In practice this is rarely more than a few percent of memory, and the gains are marginal, but one might be able to tweak it for more real world gains.

I guess for some work loads actively using swap might make sense, but it is pretty rare these days, now memory is cheap.

The other factor to consider is where the kernel dump utilities will put the crash dump, some folk opt to put this in swap space, so the size of memory would seem a sensible size for them to make their swap, but there is no need to put it there these days, but if you really want space for such a dump pre-allocated, it is a plausible way to reserve it, without having to keep that much disk space free somewhere "just in case".

Time for article on avoiding run away processes causing these issues in the first place?


[ Parent | Reply to this comment ]

Posted by shadone (213.132.xx.xx) on Thu 27 Sep 2007 at 15:48
[ Send Message ]
Weird package. Don't you know that you can easily create swap file by yourself with:

dd if=/dev/zero of=/var/swapfile bs=1024 count=1024000
mkswap /var/swapfile
swapon /var/swapfile

?

[ Parent | Reply to this comment ]

Posted by jimbren (199.43.xx.xx) on Thu 27 Sep 2007 at 16:57
[ Send Message ]
I'm not seeing what the advantages are here. Say I've got a 2GB swap partition. If I eliminate it using the swap file package and it creates a 2GB swap file, what's the gain? I'm still using 2GB of swap that I will rarely, if ever, access.

The only benefit I see is in the instance where I upgrade the RAM, as you mentioned in the article. This method would eliminate the need to increase the size of my swap partition. On the other hand, as the article and comments mention, the price and size availability of RAM have more or less eliminated the need for swap space at all.

What am i missing?

jim.

[ Parent | Reply to this comment ]

Posted by eric (82.254.xx.xx) on Thu 27 Sep 2007 at 18:01
[ Send Message | View Weblogs ]

In fact, I got the same idea: now my /var partition must have room for 2*my RAM to host my swapfile... and what happens if it get bigger? it'll fill my partition and i'll run in trouble.
I really don't see the interest, but nice tip.

:eric:
http://blog.sietch-tabr.com

[ Parent | Reply to this comment ]

Posted by Anonymous (67.78.xx.xx) on Thu 27 Sep 2007 at 18:06
jimbren: You're probably not missing anything. It's kinda simple, really; if you have (more than) enough RAM for your apps, you don't need swap at all. I haven't had a swap file or partition enabled on my desktop machine for almost a year now and I don't miss it; I do have some swap files created "just in case" (since you want to create swap files on brand new partitions to avoid fragmentation), but they're not enabled. I do think it's a good idea to have them ready, though, in case your memory needs change drastically. Then you have swap if you need it, and since you created them early they're not fragmented (which makes them even slower than they already are).

GhostR, ajt: Yes, as of Linux 2.6 swap in files and partitions are interchangeable (assuming un-fragmented swap files, of course). Here it is straight from the horse's mouth, for the curious:

http://lkml.org/lkml/2005/7/7/326

Quite frankly, I'm a little surprised that this information isn't reflected in the modern distro installers, considering how much more flexible it is to use swap file(s) instead of partition(s). Just create three or four 512-MB swap files, and you can dynamically grow/shrink/remove swap size as needed (hmm...perhaps some kind of swap daemon that monitors usage and "swapon"s additional swap files on-the-fly?).

Anonymous (about how much swap space is necessary): I would just audit your own needs. Make a giant (4GB?) swap file, so you have all you could possibly use. Install the "sysstat" and "isag" packages, enable the sysstat service, then do your thing for a couple of weeks. After that, view the aggregate swap and "real memory" usage statistics (with isag) and see how much swap you actually use. You may find that you don't even need swap. :)

[ Parent | Reply to this comment ]

Posted by Steve (82.32.xx.xx) on Thu 27 Sep 2007 at 22:13
[ Send Message | View Steve's Scratchpad | View Weblogs ]

You're not missing much, but having the resize be done automatically if you ever increase physical memory isn't a bad thing.

Really swapping is to be avoided if at all possible, and if you wanted to do it manually via files/LVM/Whatever it is just as good.

I just was suprised I'd never heard of the package and figured a brief introduction might be useful to somebody ..

Steve

[ Parent | Reply to this comment ]

Posted by kaerast (82.47.xx.xx) on Fri 28 Sep 2007 at 13:17
[ Send Message | View Weblogs ]
Can you submit this, and articles like it, to Debian Package of The Day? They're really struggling for submissions at the moment.

[ Parent | Reply to this comment ]

Posted by Steve (80.68.xx.xx) on Fri 28 Sep 2007 at 14:32
[ Send Message | View Steve's Scratchpad | View Weblogs ]

I have a hard enough time writing for this site, let alone another one.

I'm happy for people to resubmit things elsewhere if they leave credit in tact, but submitting to multiple places is probably beyond me..

Steve

[ Parent | Reply to this comment ]

Posted by Utumno (211.72.xx.xx) on Sun 30 Sep 2007 at 14:42
[ Send Message | View Utumno's Scratchpad | View Weblogs ]

Russell Coker made an insightful comment about swap space in http://planet.debian.net.

Basically he says nowadays the 'double-your-ram' rule of thumb is entirely wrong, and it was only right for some early systems derived from 4.3 BSD, such as Ultrix.

Let me paste his comments here:

There is a wide-spread myth that swap space should be twice the size of RAM. This might have provided some benefit when 16M of RAM was a lot and disks had average access times of 20ms. Now disks can have average access times less than 10ms but RAM has increased to 1G for small machines and 8G or more for large machines. Multiplying the seek performance of disks by a factor of two to five while increasing the amount of data stored by a factor of close to 1000 is obviously not going to work well for performance.

A Linux machine with 16M of RAM and 32M of swap MIGHT work acceptably for some applications (although when I was running Linux machines with 16M of RAM I found that if swap use exceeded about 16M then the machine became so slow that a reboot was often needed). But a Linux machine with 8G of RAM and 16G of swap is almost certain to be unusable long before the swap space is exhausted. Therefore giving the machine less swap space and having processes be killed (or malloc() calls fail - depending on the configuration and some other factors) is probably going to be a better situation.

There are factors that can alleviate the problems such as RAID controllers that implement write-back caching in hardware, but this only has a small impact on the performance requirements of paging. The 512M of cache RAM that you might find on a RAID controller won't make that much impact on the IO requirements of 8G or 16G of swap.

I often make the swap space on a Linux machine equal the size of RAM (when RAM is less than 1G) and be half the size of RAM for RAM sizes from 2G to 4G. For machines with more than 4G of RAM I will probably stick to a maximum of 2G of swap. I am not convinced that any mass storage system that I have used can handle the load from more than 2G of swap space in active use.

The reason for the myths about swap space size are due to some old versions of Unix that used to allocate a page of disk space for every page of virtual memory. Therefore having swap space less than or equal to the size of RAM was impossible and having swap space less than twice the size of RAM was probably a waste of effort (see this reference [1]). However Linux has never worked this way, in Linux the virtual memory size is the size of RAM plus the size of the swap space. So while the 'double the size of RAM' rule of thumb gave virtual memory twice the size of physical RAM on some older versions of Unix it gave three times the size of RAM on Linux! Also swap spaces smaller than RAM have always worked well on Linux (I once ran a Linux machine with 8M of RAM and used a floppy disk as a swap device).

As far as I recall some time ago (I can't remember how long) the Linux kernel would by default permit overcommitting of memory. For example if a program tried to malloc() 1G of memory on a machine that had 64M of RAM and 128M of swap then the system call would succeed. However if the program actually tried to use that memory then it would end up getting killed.

The current policy is that /proc/sys/vm/overcommit_memory determines what happens when memory is overcommitted, the default value 0 means that the kernel will estimate how much RAM and swap is available and reject memory allocation requests that exceed that value. A value of 1 means that all memory allocation requests will succeed (you could have dozens of processes each malloc 2G of RAM on a machine with 128M of RAM and 128M of swap). A value of 2 means that a different policy will be followed, incidentally my test results don't match the documentation for value 2.

Now if you run a machine with /proc/sys/vm/overcommit_memory set to 0 then you have an incentive to use a moderately large amount of swap, safe in the knowledge that many applications will allocate memory that they don't use, so the fact that the machine would deliver unacceptably low performance if all the swap was used might not be a problem. In this case the ideal size for swap might be the amount that is usable (based on the storage speed) plus a percentage of the RAM size to cater for programs that allocate memory and never use it. By 'moderately large' I mean something significantly less than twice the size of RAM for all machines less than 7 years old.

If you run a machine with /proc/sys/vm/overcommit_memory set to 1 then the requirements for swap space should decrease, but the potential for the kernel to run out of memory and kill some processes is increased (not that it's impossible to have this happen when /proc/sys/vm/overcommit_memory is set to 0).

The debian-administration.org site has an article about a package to create a swap file at boot [2] with the aim of making it always be twice the size of RAM. I believe that this is a bad idea, the amount of swap which can be used with decent performance is a small fraction of the storage size on modern systems and often less than the size of RAM. Increasing the amount of RAM will not increase the swap performance, so increasing the swap space is not going to do any good.

* [1] http://wombat.san-francisco.ca.us/faqomatic/cache/53.html

* [2] http://www.debian-administration.org/articles/550

[ Parent | Reply to this comment ]

Posted by mcortese (213.70.xx.xx) on Mon 1 Oct 2007 at 11:40
[ Send Message | View Weblogs ]

Very interesting!

I wonder if the choice of size and the partition-vs-file argument should be revised if we also want suspend-to-disk.

IIRC, many (laptop) systems copy the whole RAM content to swap before going to sleep. In that case, a swap smaller than RAM would not be appropriate. And at wake-up time, I do not know if resuming from file instead of partition is possible at all.

Can someone advise?

[ Parent | Reply to this comment ]

Posted by Utumno (60.248.xx.xx) on Mon 1 Oct 2007 at 11:59
[ Send Message | View Utumno's Scratchpad | View Weblogs ]

Well, I dont know the theory, but I can give some empirical evidence.

For 2 years, I was using a laptop with

swap size = size of RAM + size of video RAM

and suspend-to-disk worked with no problems ( IIRC, suspend-to-disk also copies the video RAM to swap )

[ Parent | Reply to this comment ]

Posted by mcortese (213.70.xx.xx) on Tue 2 Oct 2007 at 17:31
[ Send Message | View Weblogs ]

With swap partition or swap file?

[ Parent | Reply to this comment ]

Posted by Utumno (211.72.xx.xx) on Fri 5 Oct 2007 at 19:18
[ Send Message | View Utumno's Scratchpad | View Weblogs ]

With a swap partition, but I can see no reason why the same wouldn't work with a swap file.

[ Parent | Reply to this comment ]

Posted by mcortese (213.70.xx.xx) on Tue 9 Oct 2007 at 17:23
[ Send Message | View Weblogs ]

Actually, it is possible to use a swap file for suspend-to-disk, at least with the Suspend2 implementation by Nigel Cunningham.

It is tricky, because you have to tell the kernel where to resume from. If you use a swap partition, it is as easy as adding

resume=/dev/hda2
to your grub's menu.lst file.

But with a swap file, you have to specify something like:

resume=/dev/hda2:0x4f7810
where the hex number is the disk sector where the swap file begins.

[ Parent | Reply to this comment ]

Posted by Anonymous (195.71.xx.xx) on Fri 5 Oct 2007 at 04:00

I wrote an article advocating the use of swap files over swap partitions and utilized some of the information in this article to build my case. I hope it answers some of the questions raised in the comments here.

Reasons to choose swap files over swap partitions

[ Parent | Reply to this comment ]

Posted by ido50 (89.1.xx.xx) on Fri 5 Oct 2007 at 20:20
[ Send Message | View Weblogs ]
Just installed it, after two "bad experiences" with swapd and swapspace, which never created any swap. Looks good, didn't have to reboot, swap file is already usable.

[ Parent | Reply to this comment ]

Posted by Anonymous (79.131.xx.xx) on Wed 9 Mar 2011 at 12:08
I did that but I have the following

Starting dphys-swapfile swapfile setup ...
computing size, want /var/swap=18434MByte, limiting to kernel limit: 2048MBytes, generating swapfile ... of 2048MBytes
done.

Now I have 2 swap files

Filename Type Size Used Priority
/dev/md0 partition 2102456 0 -1
/var/swap file 2097144 0 -2

Any idea?

[ Parent | Reply to this comment ]

Posted by Anonymous (210.18.xx.xx) on Fri 16 Nov 2012 at 06:59
Swapoff the first swap partition. Then delete the first swap from your fstab. Then it should work.

[ Parent | Reply to this comment ]

Sign In

Username:

Password:

[Register|Advanced]

 

Flattr

 

Current Poll

Which init system are you using in Debian?






( 1068 votes ~ 7 comments )

 

 

Related Links