warnings from the postgresql backups
we've been getting those for a while now:
From: root@bungei.torproject.org (Cron Daemon)
Subject: Cron <torbackup@bungei> chronic /usr/lib/nagios/plugins/dsa-check-backuppg -e
To: torproject-admin@torproject.org
Date: Mon, 30 May 2022 02:20:04 +0000
[rude, main] WAL-MISSING-AFTER: rude/main.WAL.00000001000000C700000002
[materculae, main] WAL-MISSING-AFTER: materculae/main.WAL.000000010000053E0000000C
[materculae, main] NOT-EXPIRING-DUE-TO-WARNINGS: have seen warnings, will not expire anything
[bacula-director-01, main] WAL-MISSING-AFTER: bacula-director-01/main.WAL.0000000100000884000000A8
[bacula-director-01, main] NOT-EXPIRING-DUE-TO-WARNINGS: have seen warnings, will not expire anything
We've received those once a week since May 8th. Here are the entire messages, in order:
From: root@bungei.torproject.org (Cron Daemon)
Subject: Cron <torbackup@bungei> chronic /usr/lib/nagios/plugins/dsa-check-backuppg -e
To: torproject-admin@torproject.org
Date: Mon, 09 May 2022 02:20:02 +0000
NOT-CONFIGURED: bacula-director-01-11
NOT-CONFIGURED: materculae-11
[materculae, main] WAL-MISSING-AFTER: materculae/main.WAL.000000010000053E0000000C
[materculae, main] WAL-IS-OLD: latest wal file is too old
[bacula-director-01, main] WAL-MISSING-AFTER: bacula-director-01/main.WAL.0000000100000884000000A8
From: root@bungei.torproject.org (Cron Daemon)
Subject: Cron <torbackup@bungei> chronic /usr/lib/nagios/plugins/dsa-check-backuppg -e
To: torproject-admin@torproject.org
Date: Mon, 16 May 2022 02:20:04 +0000
[materculae, main] WAL-MISSING-AFTER: materculae/main.WAL.000000010000053E0000000C
[bacula-director-01, main] WAL-MISSING-AFTER: bacula-director-01/main.WAL.0000000100000884000000A8
From: root@bungei.torproject.org (Cron Daemon)
Subject: Cron <torbackup@bungei> chronic /usr/lib/nagios/plugins/dsa-check-backuppg -e
To: torproject-admin@torproject.org
Date: Mon, 23 May 2022 02:20:04 +0000
[rude, main] WAL-MISSING-AFTER: rude/main.WAL.00000001000000C700000002
[materculae, main] WAL-MISSING-AFTER: materculae/main.WAL.000000010000053E0000000C
NOT-CONFIGURED: rude-11
[bacula-director-01, main] WAL-MISSING-AFTER: bacula-director-01/main.WAL.0000000100000884000000A8
From: root@bungei.torproject.org (Cron Daemon)
Subject: Cron <torbackup@bungei> chronic /usr/lib/nagios/plugins/dsa-check-backuppg -e
To: torproject-admin@torproject.org
Date: Mon, 30 May 2022 02:20:04 +0000
[rude, main] WAL-MISSING-AFTER: rude/main.WAL.00000001000000C700000002
[materculae, main] WAL-MISSING-AFTER: materculae/main.WAL.000000010000053E0000000C
[materculae, main] NOT-EXPIRING-DUE-TO-WARNINGS: have seen warnings, will not expire anything
[bacula-director-01, main] WAL-MISSING-AFTER: bacula-director-01/main.WAL.0000000100000884000000A8
[bacula-director-01, main] NOT-EXPIRING-DUE-TO-WARNINGS: have seen warnings, will not expire anything
summarizing this, one by one:
-
rude
:WAL-MISSING-AFTER
since 23 May 2022 -
materculae
andbacula-director-01:
WAL-MISSING-AFTERsince 09 May 2022, other warnings disappeared,
NOT-EXPIRING-DUE-TO-WARNINGS` added today
i suspect this matches their Debian bullseye upgrade dates, but that still need to be investigated.
those warnings were originally ignored because we thought they were related to some transient network error, but they persist, week after week, so we should audit backups and why this is happening.