From 0b416d64c4405a552e63726abc5a2f3c8bf18143 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= <anarcat@debian.org>
Date: Wed, 4 Dec 2024 21:57:09 -0500
Subject: [PATCH] expand pg backup docs with newer pgbackrest stuff
(tpo/tpa/team#40950)
---
howto/postgresql.md | 94 ++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 93 insertions(+), 1 deletion(-)
diff --git a/howto/postgresql.md b/howto/postgresql.md
index 391c71e5..563b5257 100644
--- a/howto/postgresql.md
+++ b/howto/postgresql.md
@@ -1880,6 +1880,12 @@ Note that there is a program called [pgstatsmon](https://github.com/joyent/pgsta
very detailed information about the state of a PostgreSQL database,
see [this blog post for details](https://www.davepacheco.net/blog/2019/visualizing-postgresql-vacuum-progress/).
+The backup system keeps logs of its periodic full/diff backups in
+systemd's journal files. To consult the logs for the full backups on
+rude, for example, see:
+
+ journalctl -b -u pgbackrest-backup-full@rude.service
+
## Backups
### DSA setup
@@ -1950,9 +1956,74 @@ The new backup system is based on [pgbackrest](https://pgbackrest.org/). It work
between the client and server and running `pgbackrest` commands, which
encapsulates all functionality including backup, and restore.
+#### Puppet setup
+
+PostgreSQL servers are automatically configured to use pgbackrest to
+backup to a central server (called `repository`), as soon as the
+`profile::postgresql` is included.
+
+By default, the `backup-storage-01.torproject.org` server is used as a
+repository, but this can be overriden in Hiera with the
+`profile::postgresql::pgbackrest_repository` parameter.
+
+While the transition from the legacy system to pgbackrest happens, you
+need to do manual configurations and changes for the new backup system
+to work. First, you need something like this in Hiera:
+
+```yaml
+# testing pgbackrest (tpo/tpa/team#40950)
+profile::postgresql::backups: false
+profile::postgresql::pgbackrest: true
+# yamllint disable rule:line-length
+profile::postgresql::config_entries:
+ wal_level: 'replica'
+ archive_command: 'pgbackrest --stanza=weather-01.torproject.org archive-push %p'
+# yamllint enable rule:line-length
+```
+
+This will eventually become the default, but for now that needs to be
+added by hand, either in `hiera/roles/$ROLE.yaml` or
+`hiera/nodes/$FQDN.yaml` if the node does not have a role.
+
+Then Puppet needs to run on the "client" (the PostgreSQL server) and
+on the "repository" (the backup server, currently
+`backup-storage-01`). The full backups will run in the coming week,
+but can be ran manually with:
+
+ systemctl start pgbackrest-backup-full@$HOSTNAME.service
+
+For example:
+
+ systemctl start pgbackrest-backup-full@materculae.service
+
+Finally, the old backup directory needs to be moved out to avoid
+warnings on the old storage server (currently `bungei`):
+
+ mv /srv/backup/pg/$HOSTNAME{,.old}
+
+For example:
+
+ mv /srv/backup/pg/materculae /srv/backup/pg/materculae.old
+
+The old backups should be scheduled for removal with:
+
+ echo rm -r /srv/backup/pg/$HOSTNAME.old | at now + 30 days
+
+The configuration of the new backup can be tested with this command on
+the client:
+
+ sudo -u postgres pgbackrest --stanza=$FQDN check
+
+For example, this was done to test `weather-01`:
+
+ root@weather-01:~# sudo -u postgres pgbackrest --stanza=weather-01.torproject.org check
+
#### Manual configuration
-We followed the [dedicated repository host installation
+Those instructions are kept for historical reference, and are all
+covered by the `profile::postgresql` class.
+
+Here, we followed the [dedicated repository host installation
instructions](https://pgbackrest.org/user-guide.html#repo-host). Below, we treat the "client" (`weather-01`) as the
server that's actually running PostgreSQL in production and the
"server" (`backup-storage-01`) as the backup server that's receiving
@@ -2069,6 +2140,14 @@ See also:
* [Debian DSA documentation](https://dsa.debian.org/howto/postgres-backup/)
* [postgresqlco.nf](https://postgresqlco.nf/): easily accessible parameter documentation
+### pgbackrest
+
+- [Debian setup guide](https://pgbackrest.org/user-guide.html#repo-host), built with SSH
+- [RHEL setup guide](https://pgbackrest.org/user-guide-rhel.html#repo-host/setup-tls), built with TLS, see also [this TLS guide](https://pgstef.github.io/2022/02/21/pgbackrest_tls_server.html),
+ TLS considered for performance improvements, but might be premature optimization
+- [Configuration reference](https://pgbackrest.org/configuration.html)
+- [FAQ](https://pgbackrest.org/faq.html)
+
# Discussion
## Overview
@@ -2261,6 +2340,19 @@ license](https://github.com/omniti-labs/omnipitr/blob/master/doc/LICENSE).
I do not believe it is a suitable replacement for our backup system.
+### pgbackrest TLS server
+
+pgbackrest has a [server command](https://pgbackrest.org/configuration.html#section-server) that runs a TLS-enabled server
+that runs on the PostgreSQL server and the repository. Then the server
+uses TLS instead of SSH pipes to push WAL files to the repository, and
+the repository pulls backups over TLS from the servers.
+
+We haven't picked that option because it requires running `pgbackrest
+server` *everywhere*. We prefer to rely on SSH instead.
+
+Using SSH also allows us to use multiple, distinct users for each
+backup server which reduces lateral movement between backed up hosts.
+
## Replication
We don't do high availability right now, but if we would, we might
--
GitLab