Backups

⚠️ This service will change during policy/tpa-rfc-73-tails-infra-merge-roadmap and this page should be updated when that happens.

We user borgbackups: see https://borgbackup.readthedocs.io/en/stable/ for elaborate documentation.

• Backups
  • General
  • Retrieving backups
    • File ownership
  • Garbage collecting backups
  • Adding new backups
  • Deleting backups of a decommissioned system

General

Backups are pushed to stone. Lizard uses LVM snapshots to backup both its own filesystem and the majority of the data on the virtual machines running on lizard (some temporary data is excluded). Buse and ecours simply push their root filesystem to stone. This means that lizard and its virtual machines have a good chance of database integrity as is on the backups (worst case, most databases are dumped daily to /var/backups/mysql by backupninja). For ecours, you will have to resort to the local database backups in /var/backups/mysql.

To be able to use the backups, install borgbackup locally:

sudo apt install borgbackup

Make sure you have the keys for all the repositories:

install -d -m 0700 ~/.config/borg
cp -r ./backups/keys ~/.config/borg/

Lizard, teels and ecours all use different passphrases, which can be found in their respective eyaml files in the git.tails.net:puppet-code repository.

Before attempting to access their backups, set the appropriate passphrase:

export BORG_PASSPHRASE=bladiblabidbla

Then you can check at which times a backup was made:

borg list borg@stone.tails.net:/srv/backups/reponame

In the above command, reponame is the name of the borg repository, which defaults to the title of the corresponding @tails::borgbackup::{fs,lv}@ Puppet resource. For example:

borg list borg@stone.tails.net:/srv/backups/dns-system

Retrieving backups

To retrieve data from the backups, start by looking inside the repository at a particular archive. Say the first column of the output of borg list tells you there was a archive at 1907170854. You can then view the data inside the archive by running:

borg list borg@stone.tails.net:/srv/backups/reponame::1907170854

You can retrieve a particular file by running:

borg extract borg@stone.tails.net:/srv/backups/reponame::1907170854 filename

You can retrieve the entire archive by running:

borg extract borg@stone.tails.net:/srv/backups/reponame::1907170854

For easier selection of files to retrieve, you can mount the archive locally:

mkdir ./mnt
borg mount borg@stone.tails.net:/srv/backups/reponame::1907170854 ./mnt

When you're done, unmount by running:

borg umount ./mnt

File ownership

If you wish to preserve the file ownership of files retrieved from backups, you will have to run the borg commands as root:

• be sure all the required key material is in /root/.config/borg
• be sure you've exported the BORG_PASSPHRASE
• be sure you have access to stone as root, by running:
  • eval ssh-agent $SHELL
  • ssh-add /home/amnesia/.ssh/id_rsa # replace with your tails sysadmin ssh key

Garbage collecting backups

Backups are written in append-only mode, meaning that lizard and ecours do not have the necessary rights to remove old backups. Eventually, our disks on stone will run full and we will need to manually prune old backups.

N.B.: although lizard and ecours have no rights to actually remove old backups, they are allowed to mark them for deletion! See this discussion for more details. Always be careful before removing old backups, especially if we suspect systems have been compromised!

If you want to remove old archives, after having verified that the integrity of the backups is in order, ssh into stone and edit the file /srv/backups/reponame/config, changing the value for append_only to 0.

Then to delete an old archive, for example archive 1812091627, run on your local machine:

borg delete borg@stone.tails.net:/srv/backups/reponame::1812091627

For easier mass deletion, use borg prune:

borg prune --keep-within 6m borg@stone.tails.net:/srv/backups/reponame

Will delete all archives older then 6 months.

After you are done, ssh into stone again and set the append_only value in the config file back to 1.

Adding new backups

Adding new backups is mostly a matter of adding a line in the manifests/nodes.pp file in puppet-code.git.

You can call tails::borgbackup::lv to back up virtual machines on lizard by snapshotting their logical volumes. Add the rawdisk => true parameter if the logical volume is directly mountable in lizard (and not a virtual disk with partitions).

You can call tails::borgbackup::fs to back up machines that are not on lizard and don't use or have access to LVM. Be sure to exclude proc, dev, tmp, and sys.

See tails/how-tos/install-base-systems for more detailed instructions.

Once the first backup has run, note that a key has been generated in /root/.config/borg/keys. Be sure to copy this key into the password store under tails-sysadmins/borg, without it we won't be able to access the backups!

Deleting backups of a decommissioned system

To delete all backups of a decommissioned system, for each borg archive ARCHIVE corresponding to that system:

1. SSH into borg@stone.tails.net and set append_only = 0 in ~/ARCHIVE/config.

2. On your own system:

   • Set the borg passphrase for the decommissioned system (see the "General" section above for details):

     export BORG_PASSPHRASE=bladiblabidbla

   • Delete the backups:

     borg delete borg@stone.tails.net:/srv/backups/ARCHIVE