Verified Commit cc2d91b2 authored by anarcat's avatar anarcat
Browse files

disable backups on Prometheus WAL files (team#41627) 

Normal chunks are okay to backup without any special snapshotting, but
WAL files are written on the fly by Prometheus, and doing a backup
with Bacula will lead to inconsistencies in the storage which can make
recovery more difficult. See also https://github.com/prometheus/prometheus/pull/14317
parent ecc7fd75

howto/prometheus.md

+5 −2
Original line number Diff line number Diff line
@@ -952,8 +952,11 @@ require little backups. The metrics themselves are kept in 
`/var/lib/prometheus2` and should be backed up along with our regular

[backup procedures](howto/backup).



This might be incorrect: upstream documentation says we need special

procedures for backups to be consistent, see [tpo/tpa/team#41627](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41627).

WAL (write-ahead log) files are ignored by the backups, which can lead

to an extra 2-3 hours of data loss since the last backup in the case

of a total failure, see [tpo/tpa/team#41627](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41627) for the

discussion. This should eventually be mitigated by a high availability

setup ([tpo/tpa/team#41643](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41643)).



## Other documentation