warnings from the postgresql backups

we've been getting those for a while now:

From: root@bungei.torproject.org (Cron Daemon)
Subject: Cron <torbackup@bungei> chronic /usr/lib/nagios/plugins/dsa-check-backuppg  -e
To: torproject-admin@torproject.org
Date: Mon, 30 May 2022 02:20:04 +0000

[rude, main] WAL-MISSING-AFTER: rude/main.WAL.00000001000000C700000002
[materculae, main] WAL-MISSING-AFTER: materculae/main.WAL.000000010000053E0000000C
[materculae, main] NOT-EXPIRING-DUE-TO-WARNINGS: have seen warnings, will not expire anything
[bacula-director-01, main] WAL-MISSING-AFTER: bacula-director-01/main.WAL.0000000100000884000000A8
[bacula-director-01, main] NOT-EXPIRING-DUE-TO-WARNINGS: have seen warnings, will not expire anything

We've received those once a week since May 8th. Here are the entire messages, in order:

From: root@bungei.torproject.org (Cron Daemon)
Subject: Cron <torbackup@bungei> chronic /usr/lib/nagios/plugins/dsa-check-backuppg  -e
To: torproject-admin@torproject.org
Date: Mon, 09 May 2022 02:20:02 +0000

NOT-CONFIGURED: bacula-director-01-11
NOT-CONFIGURED: materculae-11
[materculae, main] WAL-MISSING-AFTER: materculae/main.WAL.000000010000053E0000000C
[materculae, main] WAL-IS-OLD: latest wal file is too old
[bacula-director-01, main] WAL-MISSING-AFTER: bacula-director-01/main.WAL.0000000100000884000000A8

From: root@bungei.torproject.org (Cron Daemon)
Subject: Cron <torbackup@bungei> chronic /usr/lib/nagios/plugins/dsa-check-backuppg  -e
To: torproject-admin@torproject.org
Date: Mon, 16 May 2022 02:20:04 +0000

[materculae, main] WAL-MISSING-AFTER: materculae/main.WAL.000000010000053E0000000C
[bacula-director-01, main] WAL-MISSING-AFTER: bacula-director-01/main.WAL.0000000100000884000000A8

From: root@bungei.torproject.org (Cron Daemon)
Subject: Cron <torbackup@bungei> chronic /usr/lib/nagios/plugins/dsa-check-backuppg  -e
To: torproject-admin@torproject.org
Date: Mon, 23 May 2022 02:20:04 +0000

[rude, main] WAL-MISSING-AFTER: rude/main.WAL.00000001000000C700000002
[materculae, main] WAL-MISSING-AFTER: materculae/main.WAL.000000010000053E0000000C
NOT-CONFIGURED: rude-11
[bacula-director-01, main] WAL-MISSING-AFTER: bacula-director-01/main.WAL.0000000100000884000000A8

From: root@bungei.torproject.org (Cron Daemon)
Subject: Cron <torbackup@bungei> chronic /usr/lib/nagios/plugins/dsa-check-backuppg  -e
To: torproject-admin@torproject.org
Date: Mon, 30 May 2022 02:20:04 +0000

[rude, main] WAL-MISSING-AFTER: rude/main.WAL.00000001000000C700000002
[materculae, main] WAL-MISSING-AFTER: materculae/main.WAL.000000010000053E0000000C
[materculae, main] NOT-EXPIRING-DUE-TO-WARNINGS: have seen warnings, will not expire anything
[bacula-director-01, main] WAL-MISSING-AFTER: bacula-director-01/main.WAL.0000000100000884000000A8
[bacula-director-01, main] NOT-EXPIRING-DUE-TO-WARNINGS: have seen warnings, will not expire anything

summarizing this, one by one:

  • rude: WAL-MISSING-AFTER since 23 May 2022
  • materculae and bacula-director-01: WAL-MISSING-AFTERsince 09 May 2022, other warnings disappeared,NOT-EXPIRING-DUE-TO-WARNINGS` added today

i suspect this matches their Debian bullseye upgrade dates, but that still need to be investigated.

those warnings were originally ignored because we thought they were related to some transient network error, but they persist, week after week, so we should audit backups and why this is happening.