Skip to content
Snippets Groups Projects
Verified Commit 39ca7877 authored by anarcat's avatar anarcat
Browse files

document bare bones restore procedure (team#40950) 

This was succesfully performed to restore weather-01 on a test-01
server, not even hooked in Puppet.
parent db777eb1
No related branches found
No related tags found
No related merge requests found
Pipeline #233381 failed
Stage: build
Stage: test
Hide whitespace changes
Inline Side-by-side
Showing
with 122 additions and 38 deletions
howto/postgresql.md
+ 122
− 38
View file @ 39ca7877
......@@ -808,7 +808,7 @@ installed, if not, see the [installation procedure](#installation).
It takes a couple of minutes to start, but eventually you should
see lines like:
2024-12-05 19:22:52.582 P01 DETAIL: restore file /var/lib/postgresql/15/main/base/16402/852859.4 (1GB, 11.39%) checksum 8a17b30a73a1d1ea9c8566bd264eb89d9ed3f35c
The percentage there (`11.39%` above) is how far in the restore
......@@ -821,7 +821,7 @@ installed, if not, see the [installation procedure](#installation).
It *will* finish eventually.
4. Start the restored server:
sudo service postgresql start
5. You're not done yet. This will replay log files from
......@@ -879,7 +879,69 @@ If that's not the case, for example if you're recovering the database
to a new server, you first need to do a proper [PostgreSQL
installation](#installation) which should setup the [backups](#backups) properly.
TODO: document how to pull from a *different* server.
The only twist is that you will need to tweak the stanza names to
match the server you are restoring *from* and will also likely need to
add extra SSH keys.
TODO: document exact procedure, see the bare bones restore below for
an example.
Worst case, follow the bare bones restore below.
### Bare bones restore
This assumes the host is configured with Puppet. If this is a real
catastrophe (e.g. the Puppet server is down!), you might not have that
luxury. In that case, you need to need to [manually configure
pgbackrest](#manual-configuration), except steps:
- 2.b: user and SSH keys are probably already present on server
- 4.b: server won't be able to connect to client
- 5: don't configure the pgbackrest server, it's already done
- stop at step seven:
- 7: don't create the stanza on the server, already present
- 8: no need to configure backups on the client, we're restoring
- 9: the check comamnd will fail if the server is stopped
- 10: server configuration talks to the old server
- 11: we're doing a restore, not a backup
Essentially, once you have a new machine to restore on, you will:
1. Install required software:
apt install sudo pgbackrest posgresql
2. Create SSH keys on the new VM:
sudo -u postgres ssh-keygen
3. Add that public to the repository server, in
`/etc/ssh/userkeys/pgbackrest-weather-01`:
echo 'no-agent-forwarding,no-X11-forwarding,no-port-forwarding,command="/usr/bin/pgbackrest ${SSH_ORIGINAL_COMMAND#* }" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJrOnnOpX0cyzQ/lqvNLQt2mcJUziiJ0MdubSf/c1+2g postgres@test-01' \
> /etc/ssh/userkeys/pgbackrest-weather-01
4. Configure the repository on the client, in `/etc/pgbackrest.conf`:
```
[weather-01.torproject.org]
lock-path = /var/lock/pgbackrest/weather-01.torproject.org
pg1-host = weather-01.torproject.org
pg1-path = /var/lib/postgresql/15/main
log-path = /var/log/pgbackrest/weather-01.torproject.org
repo1-path = /var/lib/pgbackrest
```
5. Restore with:
sudo -u postgres pgbackrest --stanza=weather-01.torproject.org restore
Once this is done, make sure to reconfigure the machine with Puppet
properly so that it's again hooked up with the backup system.
Note that if the machine has been gone long enough, it's possible the
user and configuration is gone from the server as well, in which case
you'll need to create those as well.
## Legacy direct backup recovery
......@@ -1934,6 +1996,10 @@ a new full backup.
If a PostgreSQL server is destroyed completely or in part, we need to
restore from backups, using the [backup recovery procedure](#backup-recovery).
This requires Puppet to be up and running. If the Puppet
infrastructure is damaged, a manual recovery procedure is required,
see [Bare bones restore](#bare-bones-restore).
# Reference
## Installation
......@@ -2391,66 +2457,84 @@ The old backups should be scheduled for removal with:
### Manual configuration
Those instructions are kept for historical reference, and are all
covered by the `profile::postgresql` class.
Those instructions are for disaster recovery scenarios, when a manual
configuration of pgBackRest is required. This typically happens when
Puppet is down, for example if the PuppetDB server was destroyed and
need to be recovered, it wouldn't be possible to deploy the backup
system with Puppet.
Otherwise those instructions should generally *not* be used, as they
are normally covered by the `profile::postgresql` class.
Here, we followed the [dedicated repository host installation
instructions](https://pgbackrest.org/user-guide.html#repo-host). Below, we treat the "client" (`weather-01`) as the
server that's actually running PostgreSQL in production and the
"server" (`backup-storage-01`) as the backup server that's receiving
the backups :
the backups.
1. Install package on both the client and the server:
apt install pgbackrest
Note: this creates a `postgresql` user instead of `pgbackrest`.
2. Keys were created on both the client and the server:
2. Create an SSH key on the client:
sudo -u postgres ssh-keygen
Create a user and SSH key on the server:
adduser --system pgbackrest-weather-01
sudo -u pgbackrest-weather-01 ssh-keygen
3. Those keys were exchanged to the other host by adding them in
`/etc/ssh/userkeys/postgres` with the prefix:
no-agent-forwarding,no-X11-forwarding,no-port-forwarding,command="/usr/bin/pgbackrest ${SSH_ORIGINAL_COMMAND#* }"
`/etc/ssh/userkeys/$HOSTNAME` with the prefix:
restrict,command="/usr/bin/pgbackrest ${SSH_ORIGINAL_COMMAND#* }"
For example, on the server:
echo 'restrict,command="/usr/bin/pgbackrest ${SSH_ORIGINAL_COMMAND#* }" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJrOnnOpX0cyzQ/lqvNLQt2mcJUziiJ0MdubSf/c1+2g postgres@test-01' \
> /etc/ssh/userkeys/pgbackrest-weather-01
On the client, the key should be in `/etc/ssh/userkeys/postgres`.
4. Test the cross-connect with:
root@backup-storage-01:~# sudo -u postgres ssh postgres@weather-01.torproject.org
root@weather-01:~# sudo -u postgres ssh postgres@backup-storage-01.torproject.org
root@weather-01:~# sudo -u postgres ssh pgbackrest-weather-01@backup-storage-01.torproject.org
This should display the `pgbackrest` usage.
This should display the `pgbackrest` usage. Also test from the
server to the client:
5. Configure the client on the server, in `/etc/pgbackrest.conf`:
root@backup-storage-01:~# sudo -u weather-01 ssh postgres@weather-01.torproject.org
```
[global]
repo1-path=/var/lib/pgbackrest
repo1-retention-full=2
start-fast=y
5. Configure the client on the server, in `/etc/pgbackrest/conf.d/weather-01.torproject.org.conf`:
[weather-01]
pg1-host=weather-01.torproject.org
pg1-path=/var/lib/postgresql/15/main/
```
[weather-01.torproject.org]
lock-path = /var/lock/pgbackrest/weather-01.torproject.org
pg1-host = weather-01.torproject.org
pg1-path = /var/lib/postgresql/15/main
log-path = /var/log/pgbackrest/weather-01.torproject.org
repo1-path = /var/lib/pgbackrest
```
6. Configure the server on the client, same location:
6. Configure the server on the client, in `/etc/pgbackrest/conf.d/server.conf`:
```
[global]
repo1-path=/var/lib/pgbackrest
log-level-file=detail
repo1-host=backup-storage-01.torproject.org
log-level-file = detail
repo1-path = /var/lib/pgbackrest
repo1-host = backup-storage-01.torproject.org
repo1-host-user = pgbackrest-weather-01
[weather-01]
pg1-path=/var/lib/postgresql/15/main
[weather-01.torproject.org]
pg1-path = /var/lib/postgresql/15/main
```
7. Create the "stanza" on the server:
sudo -u postgres pgbackrest --stanza=weather-01 stanza-create
sudo -u pgbackrest-weather-01 pgbackrest --stanza=weather-01 stanza-create
8. Modify the PostgreSQL configuration on the client to archive to
pgBackRest, in `/etc/postgresql/15/main/postgresql.conf`:
......@@ -2461,13 +2545,13 @@ wal_level = replica
```
9. Test the configuration, on the client:
root@weather-01:~# sudo -u postgres pgbackrest --stanza=weather-01 check
Note that this will wait for an archive to be successfully sent to
the server. It will wait a full minute before failing with a
helpful error message, like:
ERROR: [082]: WAL segment 000000010000001F00000004 was not archived before the 60000ms timeout
HINT: check the archive_command to ensure that all options are correct (especially --stanza).
HINT: check the PostgreSQL server log for errors.
......@@ -2477,11 +2561,11 @@ wal_level = replica
incorrect.
10. Test the configuration, on the server:
root@backup-storage-01:~# sudo -u postgres pgbackrest --stanza=weather-01 check
root@backup-storage-01:~# sudo -u pgbackrest-weather-01 pgbackrest --stanza=weather-01 check
11. Perform a first backup, from the server:
root@backup-storage-01:~# sudo -u postgres pgbackrest --stanza=weather-01 backup
The warning (`WARN: no prior backup exists, incr backup has been
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment