pgbackrest timers mess up when backup-storage-01 server reboots
there's something off with the timers on the pgbackrest backup server. normally (say) incrementals should run weekly. for example, here's weather-01's configuration:
# /etc/systemd/system/pgbackrest-backup-diff@.timer
[Unit]
Description=trigger diff backups on %i
[Timer]
OnCalendar=weekly
RandomizedDelaySec=7d
FixedRandomDelay=true
Persistent=true
[Install]
WantedBy=timers.target
yet right now, weather-01 is in alert because its weekly backups are too old:
21:34:03 -ALERTOR1:#tor-alerts- PgBackRestStaleBackups [firing] PostgreSQL backups are stale on weather-01.torproject.org
here's what it looks like, according to systemd:
root@backup-storage-01:~# uptime
19:00:33 up 4 days, 23:03, 1 user, load average: 0.00, 0.02, 0.06
root@backup-storage-01:~# systemctl list-timers | grep -e NEXT -e pgbackrest.*weather
NEXT LEFT LAST PASSED UNIT ACTIVATES
Tue 2025-02-18 20:16:36 UTC 1 day 1h left Sun 2025-02-09 00:18:55 UTC 1 week 1 day ago pgbackrest-backup-diff@weather-01.timer pgbackrest-backup-diff@weather-01.service
Sat 2025-03-01 14:36:00 UTC 1 week 4 days left Tue 2025-01-14 23:49:54 UTC 1 month 3 days ago pgbackrest-backup-full@weather-01.timer pgbackrest-backup-full@weather-01.service
root@backup-storage-01:~# systemctl status pgbackrest-backup-diff@weather-01.timer
● pgbackrest-backup-diff@weather-01.timer - trigger diff backups on weather-01
Loaded: loaded (/etc/systemd/system/pgbackrest-backup-diff@.timer; enabled; preset: enabled)
Active: active (waiting) since Wed 2025-02-12 19:57:45 UTC; 4 days ago
Trigger: Tue 2025-02-18 20:16:36 UTC; 1 day 1h left
Triggers: ● pgbackrest-backup-diff@weather-01.service
Feb 12 19:57:45 backup-storage-01 systemd[1]: Started pgbackrest-backup-diff@weather-01.timer - trigger diff backups on weather-01.
it seems like our "persist" bit either doesn't work, or i don't understand something fundamental about those timers.