critical alerts should only fire if verified on secondary (or if secondary is down)

in #41643, we setup high availability in that two servers monitor things so that we can survive an outage and still have monitoring.

but some alerts should not be send unless we verify this is a real outage from another endpoint. float has some tricks to do that up its sleeve, and we should use them.

marked this issue as related to #41643

changed milestone to %TPA-RFC-33-C: Prometheus high availability, long term metrics, other exporters

added Icebox Prometheus labels

marked this issue as related to #41714

mentioned in issue #41714

but some alerts should not be send unless we verify this is a real outage from another endpoint

My first thought: what if there are routing problems again, and the host looks down for half the Internet but not the secondary monitor?

mentioned in issue prometheus-alerts#25 (closed)