Prometheus backups might be inconsistent

thinking about the prometheus server merge (#40755 (closed)) I realized some metrics are actually precious now, something I hadn't realized. we have basic backups through bacula, but they might not be consistent!

the Prometheus storage documentation actually even says:

The use of RAID is suggested for storage availability, and snapshots are recommended for backups.

The upstream docs are rather poor for this, this guide is slightly better, but doesn't say what happens to snapshots after they're performed.. (do we need to delete them? are they garbage-collected?)

so this has to be investigated more thoroughly.