document how to look at prometheus alerts better

have instead been following [this tutorial](https://ashish.one/blogs/setup-alertmanager/) which was quite

helpful.

### Looking at alerts

There are a couple of interfaces to see alerts in our setup. The

primary one is the [Karma dashboard](https://karma.torproject.org), which shows currently firing

alerts grouped by labels.

But it won't show you history: for this, you might want to take a look

at the [Grafana availability dashboard](https://grafana.torproject.org/d/adwbl8mxnaneoc/availability) which drills down into

alerts and, more importantly shows their past values.

The ultimate source of truth for alerts and the related alerting

rules, however, is Prometheus itself. The [Alerts dashboard](https://prometheus.torproject.org/classic/alerts) show

all alerting rules and which file they are from. Normally, all rules

are defined in the [prometheus-alerts repository](https://gitlab.torproject.org/tpo/tpa/prometheus-alerts). Another view of

this is the [rules configuration dump](https://prometheus.torproject.org/classic/rules) which also shows when the

rule was last evaluated and how long it took.

Each alert should have a link to a "runbook", typically a link to this

very wiki, in the "Pager playbook" section, which shows how to handle

any particular outage. If it's not present, it's a bug and can be

filed as such.

Note that Grafana also has its own [alerting system](https://grafana.torproject.org/alerting/) but we are not

using that, see the [Grafana for alerting section of the TPA-RFC-33

proposal](policy/tpa-rfc-33-monitoring#grafana-for-alerting).

### Adding alerts in Puppet

The Alertmanager can (but currently isn't, on the external server)