more pgbackrest docs

howto/postgresql.md

@@ -162,6 +162,24 @@ Consider tuning the whole database with [pgtune](https://pgtune.leopard.in.ua/).
 
 ## Running a full backup
 
+In pgBackRest, there is a systemd unit for each full or diff backup,
+so this is as simple as:
+
+    systemctl start pgbackrest-backup-full@materculae.service
+
+You can follow the logs with:
+
+    journalctl -u pgbackrest-backup-full@materculae -f
+
+And check progress with:
+
+    watch -d sudo -u pgbackrest-materculae pgbackrest --stanza=materculae.torproject.org info
+
+### Legacy
+
+NOTE: this section concerns the legacy backup system and should be
+retired when that system is retired.
+
 Backups are normally automatically ran on the backup server (currently
 `bungei`) but you can also run a backup by hand. This will run a
 backup of `meronense`, for example:
@@ -715,9 +733,95 @@ See [this
 page](https://www.cybertec-postgresql.com/en/why-does-my-pg_wal-keep-growing/)
 for information on other cases where the WAL can start growing.
 
-## Direct backup recovery
+## Checking backup health
+
+This will run the `check` command on all configured backups:
+
+```sh
+for stanza in $( ls /var/lib/pgbackrest/backup ); do
+    echo checking $stanza
+    hostname=$(basename $stanza .torproject.org)
+    sudo -u pgbackrest-$hostname pgbackrest  --stanza=$stanza check
+done
+```
+
+This can be used to check the status of all backups in batch:
+
+```sh
+for stanza in $( ls /var/lib/pgbackrest/backup ); do
+    echo checking $stanza
+    hostname=$(basename $stanza .torproject.org)
+    sudo -u pgbackrest-$hostname pgbackrest  --stanza=$stanza info
+done
+```
+
+It's essentially the same as the first, but with `info` instead of
+`check`.
+
+## pgBackRest backup recovery
+
+pgBackRest is our new PostgreSQL backup system. It features [restore
+procedure](https://pgbackrest.org/user-guide.html#quickstart/perform-restore) and [restore command](https://pgbackrest.org/configuration.html#section-restore), and [detailed restore
+procedures](https://pgbackrest.org/user-guide.html#restore), which includes instructions on how to restore a
+specific database in a cluster, do [point in time recovery](https://pgbackrest.org/user-guide.html#pitr), to go
+back to a specific time in the past.
+
+### Simple latest version restore
+
+The procedure here assumes you are restoring to the latest version in
+the backups, overwriting the current server.
+
+ 1. visit the right cluster version:
+ 
+        cd /var/lib/postgresql/15/
+
+ 1. stop the server, and keep it from starting:
+ 
+        service postgresql.service stop
+        sudo -u postgres rm main/global/pg_control
+
+ 2. move or remove all files from the old cluster, alternatively:
+ 
+        mv main main.old && sudo -u postgres mkdir --mode 700 main
+
+    or to remove all files:
+    
+        find main -mindepth 1 -delete
+
+    You should typically move files aside unless you don't have enough
+    room to restore while keeping the bad data in place.
+
+ 3. Run the restore command:
+ 
+        sudo -u postgres pgbackrest --stanza=`hostname -f` restore
+
+ 4. Start the restored server:
+ 
+        sudo service postgresql.service start
+
+This procedure also assumes that the `pgbackrest` command is
+functional. This should normally be the case on an existing server,
+but if everything is wrong with the universe, you might 
+
+### Restoring on a new server
+
+The above also assumes the server is already configured with the
+proper "stanza". If that's not the case, for example if you're
+recovering the database to a new server, you first need to do a proper
+[PostgreSQL installation](#installation) which should setup the [backups](#backups)
+properly.
+
+## Legacy direct backup recovery
 
 <a name="direct-restore-procedures" />
+<a name="direct-backup-recovery" />
+
+IMPORTANT: those instructions are *legacy*. They concern the old
+legacy backup system we are migrating away from, to pgBackRest. They
+are kept here for historical reference, and as a fallback in case
+pgBackRest fails, as it shows a bit more of the internals of the
+backup system. It's unlikely those procedures will work once the
+legacy system is properly retired, that said.
 
 Note: this procedure **streams** the files from the backup server to
 the database server and restore directly. See the [disaster
@@ -1043,9 +1147,18 @@ The correct way to recover here is to issue a `pg_promote` statement:
 
     sudo -u postgres psql -c 'SELECT pg_promote();'
 
-## Indirect backup recovery
+## Legacy indirect backup recovery
 
 <a name="indirect-restore-procedures" />
+<a name="indirect-backup-recovery" />
+
+IMPORTANT: those instructions are *legacy*. They concern the old
+legacy backup system we are migrating away from, to pgBackRest. They
+are kept here for historical reference, and as a fallback in case
+pgBackRest fails. Those instructions are generic enough that they
+*might* be possible to adapt to pgBackRest and should be kept for the
+forseeable future, perhaps adapted to pgBackRest when the legacy
+system is fully retired.
 
 Note: this procedure **copies** the backup files from the backup
 server to the database server and restores from those. It is a