title: TPA-RFC-55: Swap file policy
approval: TPA
status: standard
discussion: tpo/tpa/team#41115This policy aims to define the use of swap space on TPA-administered systems.
Background
Currently, our machine creation procedures in the wiki recommend the creation of swap partitions of various sizes: 2GB for Ganeti instances and ">= 1GB" for physical machines.
In the case of Ganeti instances, because there is one such volume per instance, this leads to an unnecessary clutter of DRBD devices, LVM volumes and Ganeti disks.
Swap partitions have historically been recommended because swap files were not well supported in old Linux versions (pre-2.6), and because swap performance on rotational hard drives is best when the swap space is contiguous; disk partitions were a convenient way to ensure this contiguity.
Today, however, the abundance of solid-state disk space and improvements to the kernel have made this advantage obsolete, and swap files perform virtually identically to swap partitions, while being much more convenient to administer: operations such as resizing do not require any modifications to the system's partition or volume manager.
Metrics
This is a portrait of swap space usage for 102 systems for which we have gathered system metrics over the last 30 days:
- No swap usage at all: 40
- Maximum usage under 100M: 49
- Maximum usage between 100M and 1G: 10
- Maximum usage over 1G: 2
The two heaviest swap consumers are GitLab and Request Tracker. Some build machines (tb-build-01 and tb-build-05), the mail exchanger (eugeni), metrics team machines (corsicum, meronense and polyanthum) and the GitLab development instance (gitlab-dev-01) are among the moderate consumers of swap space.
Although these machines have the most swap space of all (tens of gigabytes),
almost all Ganeti nodes have no record of using any swap at all. Only
dal-node-02 has been using a whopping 1M of swap recently.
Proposal
In order to reduce this clutter and improve flexibility around swap space, we proposed adjusting our machine creation policies and tools to use file-backed swap instead of swap partitions.
In the absence of a partition named "swap", our Ganeti installer will automatically configure a 512MB swap file on the root filesystem, which is adequate for the majority of systems.
The fabric installer used for setting up physical nodes should be modified to create a 1GB swap file instead of a swap partition. A ticket will be created to track the progress on this work once the RFC is standard.
For systems with increased memory requirements such as database servers, our procedures should include documentation related to expanding the existing swap file, or adding an extra swap file. A separate ticket will be created to ensure this documentation is added once the RFC is standard.
Scope
All new systems created after this proposal is adopted, including virtual and physical machines.
Currently deployed systems are not to be automatically converted from swap partitions to swap files, although this may be done on a case-by-case basis in the future.
Alternatives considered
Swapspace is a system daemon, currently packaged in Debian, which monitors swap usage and dynamically provisions additional swap space when needed, and deallocates it when it's not.
Because deploying swapspace in our infrastructure is a more involved process which would require additional Puppet code and possibly tweaks to our monitoring, it is considered out of scope for this proposal. It may be brought up in a future proposal, however.