email destination for prometheus alerts sent to TPA should be changed
There's currently an alert about mta-dal-01 that says there's files under /var/mail
From what I could see, there were mails in the local root mailbox that were alerts sent by prometheus. Somehow they got routed to root@localhost on mta-dal-01 instead of being distributed to admins.
root@mta-dal-01:~# mail
Mail version 8.1.2 01/15/2001. Type ? for help.
"/var/mail/root": 4 messages 4 new
>N 1 alertmanager@hetz Tue Jan 14 21:23 286/13370 HTTPSUnreachable Website https://btcpayserver-02.torproject.org/ is unreachable via HTTPS
N 2 alertmanager@hetz Tue Jan 14 21:29 281/13012 HostDown Host ssh-dal-01.torproject.org is not responding
N 3 alertmanager@hetz Tue Jan 14 21:43 346/17064 HTTPSUnreachable affecting 2 targets
N 4 alertmanager@hetz Tue Jan 14 23:25 336/16599 HostDown affecting 2 targets
the ones above were saved in /root/mbox when I exited, but there might be new ones because of today's reboots.
I wonder if hetzner-nbg1-01 (prometheus1) got changed to send all outbound mail through mta-dal-01 instead of directly with its own local MTA. if that's the case, then maybe we just need to change the recipient address. otherwise we'll need to figure out how the emails got routed there.
/cc @groente