Add details about recovering alertmanager data authored by lelutin's avatar lelutin
As noted, this is not mandatory but we can do it and it's very fast.

This change also further untangles details about alertmanager and karma.
......@@ -2411,11 +2411,19 @@ transfer took somewhere between 2.5 and 3h.
If even backups are destroyed, history will be lost, but the server should still
recover and start tracking new metrics.
Note that Alertmanager holds information about the current alert silences in
place. If those are lost, we can recreate silences on a need-to basis. Karma
does not hold specific state data, so nothing needs to be taken out of backups
for it. Also, as long as prometheus is tracking metrics both services should
both be working as well.
As long as prometheus is tracking new metrics values Alertmanager and Karma
should both be working as well.
Alertmanager holds information about the current alert silences in place. This
information is held in `/var/lib/alertmanager` and can be restored from backups.
Restoring the Alertmanager directory from backups should only take a couple of
seconds since it contains two very small files.
If those are lost, we can recreate silences on a need-to basis.
Karma polls Alertmanager directly so it does not hold specific state data. Thus,
nothing needs to be taken out of backups for it.
# Reference
......
......