pgbackrest failing with permission errors on multiple hosts
we currently have alerts for 3 hosts that fail their database backups: bacula-director-01.torproject.org, polyanthum.torproject.org and weather-01.torproject.org
The failing command showing up in the logs mentions a permission issue for creating a lock file:
root@bacula-director-01:/var/lib/postgresql/15/main# sudo -u postgres pgbackrest --stanza=bacula-director-01.torproject.org archive-push pg_wal/0000000100000AEB00000098
ERROR: [103]: unable to find a valid repository:
repo1: [FileOpenError] raised from remote-0 ssh protocol on 'backup-storage-01.torproject.org': unable to get info for path/file '/var/lock/pgbackrest/bacula-director-01.torproject.org/bacula-director-01.torproject.org.stop': [13] Permission denied
on bacula-director-01, the directory is not available, but the lowest point available on that path seems to be owned by one of the users for backup client hosts and contains only one directory for the host corresponding to that user:
root@backup-storage-01:~# ls -l /var/lock/pgbackrest/
total 0
drwxr-x--- 2 pgbackrest-rude pgbackrest-rude 40 Feb 25 21:28 rude.torproject.org
root@backup-storage-01:~# ls -ld /var/lock/pgbackrest/
drwxr-x--- 3 pgbackrest-rude pgbackrest-rude 60 Feb 25 20:35 /var/lock/pgbackrest/
I'm not sure who the owner of /var/lock/pgbackrest/ is supposed to be