Make a prometheus alert for abnormal NAT assignments from probetest
Related to #40071 (closed):
https://lists.torproject.org/pipermail/anti-censorship-team/2021-October/000197.html
...looking into the broker graphs there is something weird since 2 days. The number of proxies with 'unknown' type of nat has rised heavily at the same time the 'restricted' nat has gone down. There are long periods without idle proxies and many requests being denied of nat type uknown. It doesn't look like the proxy capacity has gone down, can it be something broken on the way we test the nat type?
We want to get an automated alert when something like this happens.
At the 2021-10-28 anti-censorship team meeting we discussed how to add new alerts:
<+meskio> who can do the alertmanager config? do we have access to that machine? or do we need to ask the metrics team?
<+cohosh> oh we can do it
<+cohosh> i set it up with anarcat during the last hackweek that all we need to do is make a MR
<+meskio> ahh, cool, so the config file is in a repo
<+meskio> I can do that, never touched alertmanager, but is in my list of things to learn
<+cohosh> https://gitlab.torproject.org/tpo/tpa/prometheus-alerts
Edited by David Fifield