Add Matrix alerts to Prometheus AlertManager

We currently send emails from the Prometheus AlertManager which is great as long as those emails are read by the right person in a timely manner. There are some issues though:

• mail may be send with unencrypted transport (containing sensitive log information)
• difficult to update list of recipients
• no easy place to see history of alerts
• if the mail server is down, you get no alerts

Some effort has recently been made to start using Matrix inside Tor, and this seems like an oppportunity to move with that momentum and solve some or all of the above issues.

The webhook receiver used in the other project I mentioned is: https://github.com/jaywink/matrix-alertmanager.

If desirable, I could write an Ansible role to run this as a systemd user service on a TPA machine as we have done for Metrics services in the past, or you could write some Puppet to do the same.

added Backlog label

assigned to @anarcat

unassigned @anarcat

added Icebox label and removed Backlog label

mentioned in issue #40274 (closed)

added Prometheus label

added First Contribution label

just to throw this out there somewhere, there's many more possible ways of doing notifications, so we might consider some sort of unified bridge.

one that came up recently on Hacker News is https://ntfy.sh/, which is basically like a patch panel for notification hooks.

but there's tons more implementations of this, from all over the stack, it's quite confusing. let me just throw them all out here and i'll sort through this at some future time:

• apprise: "Push Notifications that work with just about every platform", IRC support through https://n.tkte.ch/, matrix, sms, signal, email, desktop, etc, missing mastodon
• unifiedpush another push service thing?
• patchbay, similar to ntfy.sh, but more barebones
• ntfy, confusingly something else than ntfy.sh, see https://github.com/dschep/ntfy/pull/234

marked this issue as related to #41138 (closed)

The webhook receiver used in the other project I mentioned is: https://github.com/jaywink/matrix-alertmanager.

i just found out there's a maubot plugin called alertbot which bridges alertmanager and matrix. we could clearly use this to send notifications and, rather neatly i think, also do other notifications like GitLab web hooks and other nifty things...

@ahf anyone you know considered running a maubot for tor?

For my home setup, I have pondered having a signald with a group in it for alertmanager alerts - that may be worth considering as well. It's written by one of the old IRC hax0rs. See: https://github.com/dgl/alertmanager-webhook-signald

i'm pretty sure i don't want monitoring alerts on Signal, actually. but thanks for the reference! :)

mentioned in issue #40755 (closed)

i'm experimenting with https://github.com/google/alertmanager-irc-relay for now since it's packaged in debian, in my home lab. so far it's... not great.

configuration is okay, and i managed to authentify with nickserv (but it doesn't do registration on its own, i don't think). it also doesn't support SASL auth but there is a patch for it.

but the output is just awful, as it dumped a bunch of JSON in my channel instead of formatting it correctly, still doing some tests.

another webhook/irc endpoint is this thing called bot (ugh) which seems to have some interactivity as well, which is interesting. it seems it pages users on rotation, with private messages. not clear if it supports authenticating with the IRC server (which is definitely a problem). documentation poor. not in debian and not tested.

changed milestone to %TPA-RFC-33-C: Prometheus high availability, long term metrics, other exporters

added Roadmap::Future label and removed Icebox label

added time estimate of 24h

added Icebox label and removed Roadmap::Future label

marked this issue as related to #41745 (closed)

looking at this again because we're having trouble with the IRC relay. i've also found that upstream prometheus docs refer to another project which is not linked here (go-neb)...

so let's recap our options:

• go-neb, archived, suggests matrix-hookshot but that doesn't seem to know anything about the alertmanager...
• slxh/matrix/alertmanager_matrix, another active project that specifically targets alertmanager, seems to support silences, unclear documentation
• matrix-alertmanager, original suggestion from @irl here, Javascript, very little documentation, supports multiple rooms
• alertbot, mentioned above, a maubot plugin, also with very little documentation, but has a hosted instance you can invite, whee, has a good example on how to test alerts and test alerts
• mawby: rust, seems to have a better templating system, but too verbose by default as well
• matrix-alertbot: python, reactions to silence alerts, unclear how verbose it is
• alerter: another rust thing

it's not clear to me any of those would adress the issues we're getting in #41745 (closed)

mentioned in issue #41745 (closed)

Oh hey, we do this with Guardian Project's alert manager. Let me check how we do it and maybe we can both do the same thing.

https://gitlab.com/guardianproject-ops/matrix-ops-bot

whee... another one! does it have some significant differences from the ones above? is it interactive? how does it handle notification floods?

do you have any special group_by parameters for it?

thanks!