Article submitted by Kim Merley

This article describes the required steps to convert a portion of RAM into swap space. We examine some of the reasons you might want to do this, including that using RAM as swap can be many times less expensive than using most fast solid state storage solutions. Additionally, it can be a lot easier to purchase and implement. Read on for the details.

Say you have a Linux system, and you are noticing latency. Then you find that the slowness is from the time taken to swap programs back into main memory. Say you have 512 MB of RAM and a 1 GB swap partition on hard disk. Now, what if you could afford to remedy this with a solid state storage device of GB size, and you could just plug it into your computer in an already available slot or connector? This would be used to replace the swap partition. If it were in a PCI connector it could have a maximum transfer rate of perhaps 133 MBytes/sec, which is a lot faster than an IDE drive, for sustained data transfers. Plus, the seek time would be a lot less than a millisecond. If this weren't too expensive, you might go for it? There are some multi-GB rack mount units available for tens of thousands of dollars that fulfill this, and they would reduce the swap in (and out) latency to almost the point where it would not be noticeable.

However, if almost the same thing could be accomplished for a lot less money, and have a much higher data transfer rate, that would be even better. So, if you happened to have 512 MB of memory using a single stick of DDR in a motherboard that has 4 slots, you could get another two 512 MB sticks (fulfilling the 2x RAM guideline for swap) and plug them in, and use the procedure in this article to use them as swap space. Then you would have the same memory situation you have now (if you were using 1 GB swap partition), but you will hardly notice when something needs to be swapped back in. Looking at it this way, you can have an approximately $200
(circa 2004) silicon hard drive with a very fast interface connection (PC2100 DDR has a 2100 MBytes/sec transfer rate, a lot faster than the PCI bus). $200 is a lot less than $10,000, and 2100 MBytes/sec is a lot faster than the PCI bus. The only negatives are that it sounds like a crazy idea to use valuable RAM as swap, and this has not been fully tested.

This information is provided so that people can easily (hopefully) set their computers to use a portion of their RAM, real physical memory, as swap space, thus making it possible to have about the fastest swap drive available at the least cost. These steps have been tried on my Red Hat (9 and FC1) and Debian computers. My results were 30 seconds to swap in a large application with hard disk swap verses 0.5 seconds using either swap in RAM or 0.75 seconds using no swap.

If you are using Linux as a workstation, and are experiencing its characteristic slowness in switching applications (sometimes making the computer seem to have a video adapter problem due to the slowness of painting the screen) then you might seriously consider using a portion of your RAM as swap, and perhaps turning off the swap partition on the hard drive.

It does seem, in my testing so far, that there should be at least 512 MB of RAM installed for this to help in the 2.4 and later kernels. From what I see in my limited testing of 2.6 kernels, this might not be as relevant in that case, because you can use the swappiness parameter. I have not tested this yet on 2.6.

Now, you could just shut off the swap partition (which is a lot easier):

(only do this if you have a fair amount of RAM, like at least 512 MB in RH9. I first tried this in September 2003.)

Bring up a terminal/console and log in as root: Use the command:

swapon -s

to see on which partition the swap currently resides, and how much of it is in use. Then use:

free -mt

to see how much memory is in caches and buffers. If the sum of the memory in caches and buffers is at least 10% more than that in the swap, you should be able to successfully execute the command

swapoff /dev/hdxx

Where xx is the designation of the swap partition(s) obtained above in the swapon -s results. This might take a minute, and actually it is best done just after boot up, before anything is in the swap partition, but it can be done later, when heavy swapping is slowing your system down. But the condition that the swap has to be able to fit back into the memory used by the cache and buffers is a solid requirement. If there is not enough space, the system crashes. If you have important data, don't try this until it is saved.

Now that swap is off, things should speed up considerably. When I ran with this configuration (no swap and 512 MB RAM) it seemed to flicker and be a little erratic, but it didn't crash. I suspect it was experiencing the paging storms I have seen discussed in the kerneltrap.org site. But this is the simplest test. If you have a GB of physical memory you will have to be using a real memory hog application, or hundreds of processes up at the same time (perhaps more than 300) to run into a problem. But if you had 256 MB of physical memory and 512 MB of swap you would run out of memory sooner and have memory problems too.

If a machine had 256 MB of RAM and a 256 MB swap hard drive partition, and later this was converted to 512 MB of RAM, with 256 MB of that as a swap file, and the hard drive swap turned off, will there be more application kills in the swap ram case or the hard drive swap case, all other things being equal? It would seem that they both would run out of memory, requiring a user mode process to be killed, at the same point. Again, this is something that can be tested, and tested more easily using these steps to make swap RAM.

However, using part of memory as swap space seems to allow the computer to run more smoothly, and since the virtual memory system is written expecting swap, and is optimised for it, why not just try it and see what happens.

Here is the thinking. If you have a system with 128 MB of RAM (like an old laptop) and a swap partition of 256 MB, with the more current Linux versions, the memory and swap get fairly full after a couple days or hours depending on your use. Then when the computer is used interactively, it will be spending a lot of time swapping and may take 30 seconds or more to get a swapped out application back on the screen.

I have seen this swap thrashing phenomenon on my 400 MHz laptop with 128 MB and FC1. And with only that much RAM, I can't do much about it. I did notice that changing the kswapd parameters from 512 32 8 to 512 32 64 seemed to make the swapping somewhat faster, and I haven't had any trouble with overloading the request que yet, but a server might, or might not.

To use RAM as swap, quite a few steps are necessary.

Here is a procedure that works for me.

1) Determine the size of your default ramdisks in your distribution. This has been 4 MB in the past, but I have seen 96 MB as the default. If 96 MB is the default you may not need to change it.

For example, if your RAM is 1 GB, but your default ramdisk size is 4 MB, and the default number of ramdisks is 20 (as in RH FC1), even if you used all your 4 MB ramdisks as swap that would only be 80 MB. If we are going to blindy follow the "one to two times the physical RAM as swap" rule, we need to use 512 MB for a 1 GB physical RAM where we are using only half of that as our "physical RAM". So we would have to increase the default ramdisk_size to at least 25.6 MB each. But they should be larger so you have to use less of them to aviod possible problems if some of the ramdisks are already linked and used for something else. Say we decide to use 4 ramdisks to contain the 512 MB swap. Each one will have to be 128 MB, so use the command:

ramdisk_size = 131072

(for example, this doesn't seem to have to be exact)

This command is added to the kernel line in grub, and in to the kernel subsection(s) of LILO. Make sure it works by testing the size of your ramdisks as described later.

Now that you (hopefully) have an appropriately sized default ramdisk, it has to be prepared.

2) Each ramdisk needs a directory on which to be mounted. I used /swapram/rdxx (where xx is the number of the drive). For this example, using 4 ramdisks, they would be:

mkdir /swapram
mkdir /swapram/rd10
mkdir /swapram/rd11
mkdir /swapram/rd12
mkdir /swapram/rd13

Now you have four mount points for the swap ram at a place where you can probably find them later. This step needs to be done only once. All commands past this need to be used whenever you want to use swapram. You can make a script file for this purpose.

The ramdisks I chose to use were ram10, ram11, ram12, and ram13. These already exist in the /dev directory.

3) Next comes making the filesystems on the ramdisks we are going to use.

mke2fs /dev/ram10
mke2fs /dev/ram11
mke2fs /dev/ram12
mke2fs /dev/ram13

4) Now they need to be mounted:

mount -t ext2 /dev/ram10 /swapram/rd10
mount -t ext2 /dev/ram11 /swapram/rd11
mount -t ext2 /dev/ram12 /swapram/rd12
mount -t ext2 /dev/ram13 /swapram/rd13

5) During this step you can test what these ramdisks will really hold. Just try the command:

dd if=/dev/zero of=/swapram/rd10/sw bs=1024 count=500000

It will tell the maximum size it attained before it ran out of space. I would use a number smaller than that maybe 4 to be safe. Then, for example if it said it wrote 129034 I would use 129030, thus:

dd if=/dev/zero of=/swapram/rd10/sw bs=1024 count=129030
dd if=/dev/zero of=/swapram/rd11/sw bs=1024 count=129030
dd if=/dev/zero of=/swapram/rd12/sw bs=1024 count=129030
dd if=/dev/zero of=/swapram/rd13/sw bs=1024 count=129030

6) Now the swap has been initialized. The ramdisks still need to be made into swap: This requires making the swapfiles. In this example I just named them all sw as they are in different directories.

mkswap /swapram/rd10/sw 129030
mkswap /swapram/rd11/sw 129030
mkswap /swapram/rd12/sw 129030
mkswap /swapram/rd13/sw 129030

7) Now Change Permissions

chmod 0600 /swapram/rd10/sw
chmod 0600 /swapram/rd11/sw
chmod 0600 /swapram/rd12/sw
chmod 0600 /swapram/rd13/sw

8) sync

9) Now activate these prepared ramdisk swap files

swapon /swapram/rd10/sw
swapon /swapram/rd11/sw
swapon /swapram/rd12/sw
swapon /swapram/rd13/sw

Now you can issue

swapon -s

and see them. It is now time to (if there is enough memory to swap everything back in)

swapoff /dev/hdxx

Where again xx is repladec by the partition number of the swap partition on the disk drive. You can also assign priorities to the drives (in the swapon command) if you want different ones than they are given by default.

Now it is set so that all swap is swap ram, and you can try your performance.

Other Comments

I did experience a slight problem on one computer that was set up to have quota warnings on all drives. Every day there would be an email, from the system to root, that said a swap drive or two were over XX% full. If this happens, take these partitions out of the quota list.

At first glance, this concept of using RAM as swap sounds a little strange. To get more memory, and then not use it for memory but for swap just doesn't sound right at first. But Linux server administrators sometimes have used fast solid state drives of 4GB (for $10k or more) for this purpose. It seem to me that if you have 1 GB of memory now, even if it requires upgrading the motherboard to one that takes at least 4 GB RAM, and then using 1 GB original RAM and putting in 2GB more to use for swap is going to be less expensive than $10k. A lot less. If the motherboard can already take 2 GB, just add another GB to the original GB (making 2 GB total) and make the new GB swap. When you are done, you have a silicon disk to use as lightning fast swap for way less than a thousand dollars.

There are those that think there might be some bug that would make this swap ram dangerous to use. (There seem to be other bugs that make the kernel dangerous sometimes). My purpose is to make it easy to try this on many (hopefully not too critical, in case of a discovered bug, which then could be fixed) computers and see if it works well over time or not. My experience on 3 computers is that it works well.

Consider this about normal swap setups. Having more memory (and correspondingly more swap space on the hard drive to fulfill "the swap space should be one to two times the size of RAM" 'requirement') might actually make the swap thrashing time expand further. We used to have 64 MB of RAM and a 128 MB swap partition. 192 MB total. The absolute most you could ever swap out and in at once would be something less than 64 MB, the total size of RAM. If hard drives were at 5400 RPM, it took some X amount of time. Now, if we have 1GB RAM and 2 GB swap space, and 7200 RPM drives, to do the same thing, swap out almost all of memory, would take 12X time. This is because, even though the computer processor/memory may be 20 x faster than before, the drives are not even 2x faster.

So, larger memory capacity coupled with little change in hard drive speed has probably this problem easier to notice. Processor speeds and FSB speeds don't matter much when you have to swap something back in from the hard drive. The applications are larger and more complex now, and take more RAM (and more swap when swapped out). So since the hard drive speed is almost the same, but the size of what is swapped out and in is ten times (or more) larger, things get sometimes 10x (or more) slower when swap is heavily in use. Where will we be when we are using 16 GB of memory and 32 GB swap space, and drives are still at 7200 or 10,000 RPM? That will be 16 times longer than what we have now, which would then be about 200 times slower than on the 64MB main memory 128 MB swap example above. I can hardly wait for all the waiting that will involve.

In the O'Reilly book "Understanding the Linux Kernel", by Bovet and Cesati, in Reclaiming Page Frame in chapter 16, it is stated that many programs request program memory and then request more as cache and buffers, and the operating system never releases them until it has to. Add to this that programs many times request much more memory than they will need, just in case, and the reason why Linux memory is full at almost all times is clear.

When swap ram was used on a 500 MHz FC1 computer with only 256 MB of RAM, the results were not that good. First, it was not enough memory, and it didn't speed things up much.

Second, the program 'free' is not made to handle swap RAM. I can see that from the beginning. If I turn off a 500 MB swap partition on the hard drive, then run the command

free -mt

It shows 0 bytes in swap or available as swap. If I then make half of the 256 MB of memory into swap, then it faithfully shows 128 MB in swap, but many times still has more than 128 MB in cache and buffers, so it must not actually be allocating the swap files in RAM until they fill up. Also, free -mt shows the memory total (and this is the interesting part) as the 128 MB of swap plus the 256 MB of physical RAM, which is about 384 MB. The free command actually shows total memory of 384 MB! So talk about your virtual memory. There are only 256 MB of RAM on that system, and the physical hard drive swap partition is off, but the total memory with swap in RAM is now 384 MB! Almost sounds like an accounting trick.

So it looks like nobody expected swap to be placed in RAM. But even though this accounting error exists, I have had no problem with the swap in RAM, unless the amount of RAM is too small in the first place. The recommended RAM for these later versions of Red Hat is more than 128 MB, so it was good to test this on the 256 MB RAM system, but it provided no performance gain. With at least 512 MB of RAM it provides definite gains. No more long waits for the next workspace to appear.

This idea of using swap RAM is just the result of trying to deal with the way Linux handles the virtual memory subsystem. The only reason there is virtual memory in the first place is that there wasn't enough RAM for the programs, but there was a lot of cheaper disk space. This is why this complex superstructure called Virtual Memory exists. If we just had enough memory in the first place, there wouldn't be swap. Everything could be in memory. You would still have to use the hard drive to load programs and store data, so there would still be caches and buffers to lessen the impact of having to actually write to or read from the (about 50,000 times slower than RAM according to Riel) hard disk. But caches and buffers would not be allowed to take up all free RAM like they are now. They would be controlled, if swap was not available. That would save a lot of code and complexity, making the kernel more reliable, with less places for bugs to exist.

But we do have swap tightly integrated into the kernel. So, it seems that there should be some swap, even if it is swap RAM. (All I can say is my computers run more smoothly with ram swap than with no swap at all). And it could be that the kernel code will be satisfied (run well) with perhaps only 200 MB swap in a 1 GB physical memory system. 800 MB main memory, and 200 MB swap in RAM. That can be a part of this test.

So, please use this to make and use swap in RAM, and report on the results to discussion groups. There is nothing like many people testing this to see how well it works. No more speculation and a pirori knowledge of what might happen, but observation of what does happen.

If you like how this works for you, you can make a script to set it up so you can either turn it on when you want, and not if you don't want. If you want it to persist after a reboot, you will have to put the script in the initialization scripts area. I did this, and it is nice to just forget about it after that. No swap thrashing is nice, and sometimes I don't even remember it is set that way. Perhaps most people reading this have no need of help with scripting or init.

In the script, you have to put in all the steps except the making of the mountpoint directories. Again the steps are:

use mke2fs to put the filesystems on the /ram* drives on your system

mount the ram* 's to the already existing mount points

fill them with zeros to initialize them.

use mkswap to make them into swap.

change their permissions to 0600

sync the filesystems

turn them on as swap with swapon.

turn off the swap partition on the hard drive (optional)

Sample Script

This script is just a sample. For your particular situation you may be using more or less ramdisks, of different size. The procedure above should be done manually at least once so that the count number is known accurately. This was written when the mountpoints were in the root directory at /rd1 through /rd9, different from the example procedure above. /dev/ram1 was not used as it was already linked to something else, so I used /dev/ram10 instead. All the steps are shown in the first 8 lines (for the first ramdisk). The rest is a repeat to get all the others ready and working. The priorities in swapon are set to be higher than the hard drive swap partition priority. If you set your ramdisk_size large enough you would only need one ramdisk, and thus would only need the first 8 lines of this example.

#!/bin/bash
mke2fs /dev/ram10
mount -t ext2 /dev/ram10 /rd1
dd if=/dev/zero of=/rd1/sw bs=1024 count=56000
mkswap /rd1/sw 56000
chmod 0600 /rd1/sw
sync
swapon -p9 /rd1/sw
#
mke2fs /dev/ram2
mke2fs /dev/ram3
mke2fs /dev/ram4
mke2fs /dev/ram5
mke2fs /dev/ram6
mke2fs /dev/ram7
mke2fs /dev/ram8
mke2fs /dev/ram9
#
mount -t ext2 /dev/ram2 /rd2
mount -t ext2 /dev/ram3 /rd3
mount -t ext2 /dev/ram4 /rd4
mount -t ext2 /dev/ram5 /rd5
mount -t ext2 /dev/ram6 /rd6
mount -t ext2 /dev/ram7 /rd7
mount -t ext2 /dev/ram8 /rd8
mount -t ext2 /dev/ram9 /rd9
#
dd if=/dev/zero of=/rd2/sw bs=1024 count=56000
dd if=/dev/zero of=/rd3/sw bs=1024 count=56000
dd if=/dev/zero of=/rd4/sw bs=1024 count=56000
dd if=/dev/zero of=/rd5/sw bs=1024 count=56000
dd if=/dev/zero of=/rd6/sw bs=1024 count=56000
dd if=/dev/zero of=/rd7/sw bs=1024 count=56000
dd if=/dev/zero of=/rd8/sw bs=1024 count=56000
dd if=/dev/zero of=/rd9/sw bs=1024 count=56000
#
mkswap /rd2/sw 56000
mkswap /rd3/sw 56000
mkswap /rd4/sw 56000
mkswap /rd5/sw 56000
mkswap /rd6/sw 56000
mkswap /rd7/sw 56000
mkswap /rd8/sw 56000
mkswap /rd9/sw 56000
#
chmod 0600 /rd2/sw
chmod 0600 /rd3/sw
chmod 0600 /rd4/sw
chmod 0600 /rd5/sw
chmod 0600 /rd6/sw
chmod 0600 /rd7/sw
chmod 0600 /rd8/sw
chmod 0600 /rd9/sw
sync
#
swapon -p8 /rd2/sw
swapon -p7 /rd3/sw
swapon -p6 /rd4/sw
swapon -p5 /rd5/sw
swapon -p4 /rd6/sw
swapon -p3 /rd7/sw
swapon -p2 /rd8/sw
swapon -p1 /rd9/sw