Maximum swappage

The original version of this article was first published on IBM developerWorks, and is property of Westtech Information Services. This document is an updated version of the original article, and contains various improvements made by the Gentoo Linux Documentation team.

When you set up a brand new Linux server, do you create a single 128 MB swap partition? If so, did you know that you are severely limiting swap performance? Would you like to increase swap performance by several orders of magnitude, and to create swap partitions larger than 1 GB? It's possible, requiring no kernel patches or special hardware, just pure geek know-how!

Some of you may not really care about swap. After all, Linux systems are typically very memory efficient, and swap is often barely touched. While often true on desktop systems, servers are another story. Because servers may handle unexpected stresses, such as runaway processes, denial of service attacks, or even the Slashdot effect, they need to have adequate high-speed swap so that they do not grind to a halt and possibly crash when all physical memory (and then some) is exhausted.

Still not convinced that this is a big deal? I'll show you how easy it is to bring down a server by launching a massive amount of new processes.

Please, if you try this, do it only on a non-production server that you actually administer!

Let's say you have two customized grep commands in /usr/bin, called bobgrep and jimgrep. Now, let's assume that bobgrep is simply a shell script that calls the ELF executable jimgrep, as follows:

#!/bin/bash
jimgrep -r $*

Everything looks good so far, but what happens if jimgrep gets accidentally replaced with a symbolic link to bobgrep? Well, in that case, calling bobgrep or jimgrep will cause an infinite loop, causing hundreds of bash processes to be spawned in mere seconds. This actually happened to me once, and believe me, it hurt!

If a server doesn't have adequate swap, a situation like this can cause the machine to lock up in much less than a minute. How do we fix the problem? One way is to increase the swap size beyond 128 MB. Fortunately for us, there is absolutely no 128 MB limit on swap under Linux 2.2.x+ and later kernels, as there was in the past. The current limits are approximately 2 GB on x86, PowerPC, and MC680x0 systems, 512 MB on MIPS systems, 128 GB on Alpha, and a whopping 3 terabytes on UltraSparc platforms!

x86                 2 Gigabytes
PowerPC             2 Gigabytes
Motorola 680x0      2 Gigabytes
Sparc               1 Gigabyte
MIPS                512 Megabytes
Alpha               128 Gigabytes
UltraSparc          3 Terabytes

While it's nice to be able to increase swap partition size to beyond 128 MB, how about increasing performance? Ideally, it would be nice if we could set up swap partitions in a RAID 0 stripe, so that reads and writes are equally distributed between all partitions. If these partitions are on separate drives and/or controllers, this will multiply swap file performance, allowing your servers to handle temporary memory usage "spikes" without getting dramatically bogged down.

Amazingly, all modern Linux kernels, by default (with no special kernel options or patches) allow you to parallelize swap, just like a RAID 0 stripe. By using the pri option in /etc/fstab to set multiple swap partitions to the same priority, we tell Linux to use them in parallel:

/dev/sda2       none    swap    sw,pri=3        0       0
/dev/sdb2       none    swap    sw,pri=3        0       0
/dev/sdc2       none    swap    sw,pri=3        0       0
/dev/sdd2       none    swap    sw,pri=1        0       0

In the above example, Linux will use swap partitions sda2, sdb2, and sdc2 in parallel. Since these partitions are on different drives, and possibly even different SCSI controllers, read and write throughput will nearly triple. The fourth swap partition, sdd2, will be used only after the first three partitions have been exhausted.

The pri option is really easy to use. The priority must be a number between 0 and 32767, with 32767 being the highest priority. The swap partitions will be used from highest priority to lowest priority, meaning that a partition with a priority of x will only be used only if all partitions with a priority greater than x are already full. If several partitions have the same priority, Linux will automatically parallelize access between them. This allows you to not only parallelize swap, but also prioritize access so that the partitions on the fastest drives (or regions of the drives) are used first. So, you can set up an emergency swap partition on an old, slower drive that will be used only if all high-speed swap is exhausted first.

Now it's time to put some of this swapping knowledge into action. To loosely quote Mr. Miyagi of Karate Kid fame: "Swap on, swap off, geek-san!"

Resources

In Partitioning in action: Moving /home, I show you step-by-step how to move /home to a new partition.
In Partitioning in action, I show you step-by-step how to move /var and /tmp to a new partition.
In my second tip, "Partition planning tips", I share several ways of effectively organizing your partition layout.
The Linux system administrator's guide mirrored on Red Hat's home page.
A good high-level overview of Linux system administration.
For more information on optimizing file system performance, take a look at Linas Vepstas' Linux Software RAID HOWTO. Since the most recent software RAID code has been merged into the 2.3 series kernels, it is a viable option for further enhancing Linux disk performance.

1.1 xml/htdocs/doc/en/articles/partition-planning-tips.xml file : http://www.gentoo.org/cgi-bin/viewcvs.cgi/xml/htdocs/doc/en/articles/partition-planning-tips.xml?rev=1.1&content-type=text/x-cvsweb-markup&cvsroot=gentoo plain: http://www.gentoo.org/cgi-bin/viewcvs.cgi/xml/htdocs/doc/en/articles/partition-planning-tips.xml?rev=1.1&content-type=text/plain&cvsroot=gentoo Index: partition-planning-tips.xml =================================================================== Partition planning tips Daniel Robbins Joshua Saddler Organizing your partitions correctly can be fun and rewarding. This collection of tips will help you to use those sectors wisely. 1.0 2005-08-15 How to keep things organized on disk

Have you noticed how many new filesystems are now available for Linux these days? Not only ReiserFS, but also SGI's XFS and IBM's own JFS (in source form) are now available for Linux. What an incredible selection! And, of course, we also have the Coda filesystem as well as good old ext2. As usual, choice is an excellent thing. In this collection of tips, we'll be looking at another area where we have lots of freedom -- how to organize partitions on disk. Regardless of which new-fangled filesystem you are using, good partition layout is essential. Here are three tips that will help to keep things organized.

Don't partition the entire drive

These days, new machines often have 20 or 34 gigabytes of storage on a single drive. These drives are great, but they create a new problem: how exactly do you partition these things? Many of us can only conceive of filling up 12 GB, and that's if we actually tried to fill things up.

Instead of partitioning the entire drive during installation, you may want to consider partitioning only the portion of the drive that you know you will actually use. Later, when you've figured out how to use the 22 GB of free space sitting at the end of your drive, you'll have a better idea of how to partition it. Then, you can fire up fdisk or cfdisk and create a partition that meets your exact needs. For now, create partitions that are large enough to hold your current files, plus allow for future growth. Don't worry, the free space at the end of your drive will still be there when you need it.

Partition for backups

My second organizational tip will help your backups go more smoothly. If you are going to regularly back up a portion of your file system tree, it's a very good idea to place this data on a separate partition. Why? Because many copying and archiving tools have a special option for not crossing partition boundaries (such as cp's -x option), which will make it easy to select only the files you want. Also, you can select a size for this partition so that it is close to capacity of your backup media. That way, you'll never have to worry about filling up a tape, and you can monitor how close you are to filling up your backup media by using the df command to see how much space is left on that particular partition. Then, you can track your storage and backup needs and plan accordingly.

Partition to reduce fragmentation

At the beginning of this article, I mentioned all the fancy new journaled filesystems now available for Linux. Amid all the excitement, it's easy to forget our old friend, the ext2 filesystem. ext2 does have some drawbacks; it is not extremely robust, for example. However, one thing it does excel at (especially when compared to Microsoft's NTFS) is keeping fragmentation to a minimum. If you don't know, file fragmentation occurs when there is not enough contiguous space on a filesystem to store an entire file. The result? The file is broken into non-contiguous "fragments," which are stored on disk. When this file is accessed, the hard drive's head needs to jump all over the place to read in the entire file. This slows down I/O, adding additional seeks to the reading process.

OK, you may ask, what's the big deal? After all, ext2 filesystems take a long time to exhibit signs of fragmentation. However, I would argue that fragmentation is still a big problem, because although ext2 does not get fragmented easily, fragmentation is a one-way, cumulative process. That is, while ext2 fragments slowly, it cannot defragment itself. In other words, any often-modified ext2 filesystem will gradually get more and more fragmented, and thus slower. Even worse, there are no production-quality ext2 filesystem defragmenting programs currently available. This means that fragmentation is guaranteed to get worse over time, and the only way to fix it is to wipe the filesystem clean, and restore all the original files from a backup. That's a pretty involved and time-consuming process. So, how can the problem of fragmentation be dramatically reduced and contained? I'll cover all the juicy details in my next tip. See you then!

Resources

In Partitioning in action: Moving /home, I show you step-by-step how to move /home to a new partition.
In Partitioning in action, I show you step-by-step how to move /var and /tmp to a new partition.
See Maximum swappage, another tip by Daniel, where he shows how to dramatically increase swap performance under Linux.
Check out the howtos at the Linux Documentation Project.

About the author

Daniel Robbins lives in Albuquerque, New Mexico. He was the President/CEO of Gentoo Technologies Inc., the Chief Architect of the Gentoo Project and is a contributing author of several books published by MacMillan: Caldera OpenLinux Unleashed, SuSE Linux Unleashed, and Samba Unleashed. Daniel has been involved with computers in some fashion since the second grade when he was first exposed to the Logo programming language and a potentially lethal dose of Pac Man. This probably explains why he has since served as a Lead Graphic Artist at SONY Electronic Publishing/Psygnosis. Daniel enjoys spending time with his wife Mary and his new baby daughter, Hadassah. You can contact Daniel at drobbins@gentoo.org.

1.1 xml/htdocs/doc/en/articles/partitioning-p1.xml file : http://www.gentoo.org/cgi-bin/viewcvs.cgi/xml/htdocs/doc/en/articles/partitioning-p1.xml?rev=1.1&content-type=text/x-cvsweb-markup&cvsroot=gentoo plain: http://www.gentoo.org/cgi-bin/viewcvs.cgi/xml/htdocs/doc/en/articles/partitioning-p1.xml?rev=1.1&content-type=text/plain&cvsroot=gentoo Index: partitioning-p1.xml =================================================================== Partitioning in action, Part 1 Daniel Robbins Joshua Saddler In this new series of tips, Daniel Robbins shows you how to change partition layout on a running system. He'll also cover several tricks of the trade to minimize downtime and avoid making costly mistakes. In this particular tip, he'll show you how to move /home to another partition. 1.0 2005-08-15 Moving /home

Introduction The original version of this article was first published on IBM developerWorks, and is property of Westtech Information Services. This document is an updated version of the original article, and contains various improvements made by the Gentoo Linux Documentation team.

The partition /home is one of the most-often-moved partitions. Sometimes, all the space in /home becomes exhausted, and an additional hard drive is required. Other times, /home is set up as part of the root partition, and it may need to be moved to improve performance or facilitate backup. Whatever the case, I'll show you how to move /home safely and efficiently.

The following technique describes how to move a partition(s). Although this technique is designed so that you can "back out" of a failed partition move, it doesn't protect against user error. In other words, any time that you format partitions or copy large numbers of files, there's a possibility that you will type the wrong thing, causing lots of data to be destroyed. For this reason, it's highly recommended that you take appropriate steps to back up all critical files before proceeding.

Now that you're prepared, we're ready to start moving /home. The exact steps you will take depend on whether /home currently resides on its own separate partition, or whether it is located on the root partition. Keep this in mind as we go through the steps (I'll make notes where necessary). Right now, if you are moving /home to a new hard drive, it should be physically installed in your system.

If you are moving /home to an existing partition (it doesn't need to be ext2, as long as the target primary or extended partition exists), you can proceed to step 2.

Create a new partition, if necessary

If the new partition doesn't exist yet, you'll need to create it using cfdisk (preferred) or fdisk. If the partition doesn't reside on your first drive, remember to specify the name of the device as the first argument to cfdisk or fdisk. After creating the appropriate primary or extended partition, you should reboot so that the partition table can be reread correctly. This is the only time you will need to reboot the system.

Create a filesystem on the new partition

To create a filesystem on the new partition, first make sure you know the exact device name for the new partition (for example, /dev/sda5). If you're not sure of the exact device name, stop now and double-check! Then type the following, as root:

# mkfs.ext2 /dev/???

In the above and following code samples, ??? should be replaced with the target partition name. After executing this command, the target partition will contain an empty ext2 filesystem.

Mount the new filesystem in /mnt

Create a directory called /mnt/newpart, and then mount the new partition there:

# mount /dev/??? /mnt/newpart

Drop to single-user mode

I delayed this step as long as possible to maximize system availability, but we now must drop into single-user mode, and copy files from /home to /mnt/newpart. You shouldn't have any files open in /home, and entering single-user mode eliminates this possibility:

# init 1

If prompted, enter the root password to perform system maintenance. You should now have a root shell.

Change directories to /home and copy files

Type the following:

# cd /home
# cp -ax * /mnt/newpart

The cp -ax command recursively copies the contents of /home to /mnt/newpart, preserving all file attributes, and not crossing any mount points. After this command finishes, /mnt/newpart will contain an exact copy of all the files and directories currently in /home. If the old /home was on its own separate partition (listed on a separate line in /etc/fstab), go to step 6a. Otherwise, proceed to step 6b.

Use the new partition

6a: /home on its own partition 1.1 xml/htdocs/doc/en/articles/partitioning-p2.xml file : http://www.gentoo.org/cgi-bin/viewcvs.cgi/xml/htdocs/doc/en/articles/partitioning-p2.xml?rev=1.1&content-type=text/x-cvsweb-markup&cvsroot=gentoo plain: http://www.gentoo.org/cgi-bin/viewcvs.cgi/xml/htdocs/doc/en/articles/partitioning-p2.xml?rev=1.1&content-type=text/plain&cvsroot=gentoo Index: partitioning-p2.xml =================================================================== Partitioning in action, Part 2 Daniel Robbins Joshua Saddler In this second tip on changing partition layout on a running system, Daniel Robbins shows you how to move /tmp and /var to their own shared partition. He also covers several tricks of the trade to minimize downtime and avoid making costly mistakes. 1.0 2005-08-15 Consolidating data

In my last tip, we successfully moved /home to a new partition. Now, it's time to learn how to consolidate often-modified files onto a new partition. Why would we want to do this? Here's an example. Often-modified files contribute heavily to fragmentation. One of the best ways to contain fragmentation is to store often-modified files on their own partition. That way, the other partitions are unaffected by the fragmentation caused by the heavily modified files. In concept, this is easy to understand, but how do you actually go about doing it?

First, you must create a new partition for the specific purpose of storing frequently modified files. You might want to locate this partition on a separate disk to enhance performance. Then, I'll walk you through the steps required to move both /tmp and /var to this new partition.

Create a filesystem on the new partition

The first step of this process is to create a new partition that's big enough to hold /var and /tmp, with a little extra space left over. You'll need either an additional drive, or a spare (unused) partition that will house the often-modified files. If you do need to use fdisk or cfdisk to create the partition, you'll need to reboot once. Then, it's time to format the new partition as follows (it's OK to be in multiuser mode while you do this; I'll let you know when to switch to single-user):

# mkfs.ext2 /dev/???

Mount it to /mnt/rwstorage

As in my previous tip, ??? should be replaced with the device name for the new, empty partition that you are creating. Accidentally typing the wrong name will destroy data on an existing partition, so be careful! After typing this command, you'll have a brand-new ext2 filesystem on the new partition. We're almost ready to mount it, but first, let's make a new mount point.

# mkdir /mnt/rwstorage

I chose the name rwstorage to remind us that this particular partition is going to be specifically used to house files that are read from and written to frequently. To mount the partition, type:

# mount /dev/??? /mnt/rwstorage

Creating a new /tmp

The partition is now mounted and we're ready to create our new /tmp directory:

# cd /mnt/rwstorage
# mkdir tmp
# chmod 1777 tmp

Drop to single-user mode

Our new directory at /mnt/rwstorage/tmp has the right permissions for a temporary directory. Now, drop to single-user mode, since we must copy over /var. As usual, we've delayed our drop to single-user mode to the last possible moment. Right now, we don't want any programs reading or writing files in /var, so we have to stop all daemons, disconnect all users, and do some quick maintenance by typing:

# init 1

If you're prompted to enter a password to perform system maintanance, do so. You should now have a root shell, and all unnecessary daemons will be stopped. Create a new location for our /var files by typing:

# cd /mnt/rwstorage
# mkdir var

Copy /var

Default permissions on our new /mnt/newstorage/var directory should be correct, so now we're ready to copy all of our original /var data over to the new partition:

# cd /var 
# cp -ax * /mnt/rwstorage/var

-- gentoo-doc-cvs@gentoo.org mailing list