service/prometheus.md

@@ -61,6 +61,59 @@ any particular outage. If it's not present, it's a bug and can be filed as such.
 [`prometheus-alerts.git` repository]: https://gitlab.torproject.org/tpo/tpa/prometheus-alerts
 [rules configuration dump]: https://prometheus.torproject.org/classic/rules
 
+## Silencing an alert in advance
+
+Say you are planning some service maintenance and expect an alert to
+trigger, but you don't want things to be screaming everywhere.
+
+For this, you want to create a "[silence][]", which technically
+resides in the Alertmanager, but we manage them through the [Karma
+dashboard][].
+
+[silence]: https://prometheus.io/docs/alerting/latest/alertmanager/#silences
+
+Here is how to set an alert to silence notifications in the future:
+
+ 1. Head for the [Karma dashboard][]
+ 2. Click on the "bell" on the top right
+ 3. Enter a label name and value matching the expected alert,
+    typically you would pick `alertname` as a key and the name as the
+    value (e.g. `JobDown` for a reboot)
+    
+    You will also likely want to select an `alias` to match for a
+    specific host.
+ 4. Pick the duration: this can be done through duration (e.g. one
+    hour is the default) or start and end time
+ 5. Enter your name
+ 6. Enter a comment describing why this silence is there, preferably
+    pointing at an issue describing the work.
+ 7. Click `Preview`
+ 8. It will likely say "No alerts matched", ignore that and click
+    `Submit`
+
+When submitting an alert, Karma is quite terse: it only shows a green
+checkbox and a UUID, which is the unique identifier for this alert, as
+a link to the Alertmanager. Don't click that link, as it doesn't work
+and anyways we can do everything we do with alerts in Karma.
+
+## Silencing active alerts
+
+Silencing active alerts is slightly easier than planning one in
+advance. You can just:
+
+ 1. Head for the [Karma dashboard][]
+ 2. Click on the "hamburger menu"
+ 3. Select "Silence this group"
+ 4. Change the comment to link to the incident or who's working on
+    this
+ 5. Click `Preview`
+ 6. It will show which alerts are affected, click `Submit`
+
+When submitting an alert, Karma is quite terse: it only shows a green
+checkbox and a UUID, which is the unique identifier for this alert, as
+a link to the Alertmanager. Don't click that link, as it doesn't work
+and anyways we can do everything we do with alerts in Karma.
+
 ## Adding metrics to applications
 
 If you want your service to be monitored by Prometheus, you need to