Add Matrix alerts to Prometheus AlertManager
We currently send emails from the Prometheus AlertManager which is great as long as those emails are read by the right person in a timely manner. There are some issues though:
- mail may be send with unencrypted transport (containing sensitive log information)
- difficult to update list of recipients
- no easy place to see history of alerts
- if the mail server is down, you get no alerts
Some effort has recently been made to start using Matrix inside Tor, and this seems like an oppportunity to move with that momentum and solve some or all of the above issues.
The webhook receiver used in the other project I mentioned is: https://github.com/jaywink/matrix-alertmanager.
If desirable, I could write an Ansible role to run this as a systemd user service on a TPA machine as we have done for Metrics services in the past, or you could write some Puppet to do the same.