A silence should be added for backups on newly-provisioned machines

With the new monitoring system, we have a value of 0 for the timestamp of the last backup until the first full backup of a machine is run.

To avoid useless alerts right after provisioning a new host, we should modify our fabric scripts to trigger the addition of a silence for the backup alerts of that host (e.g. alias=$fqdn and job=bacula).

The expiration delay that we need is not super clear to me. Is 2 days about right?

assigned to @anarcat

added Doing Prometheus labels

i have a prototype that can post alerts now:

anarcat@angela:fabric-tasks$ fab silence.create --comment="test from anarcat" --matchers foo=bar --starts-at 2024-10-16T15:10:11.484Z --ends-at 2024-10-16T20:10:11.484Z
INFO: sending silence {'matchers': [{'name': 'foo', 'value': 'bar', 'isRegex': True, 'isEqual': True}], 'startsAt': '2024-10-16T15:10:11.484Z', 'endsAt': '2024-10-16T20:10:11.484Z', 'createdBy': 'anarcat', 'comment': 'test from anarcat'}
INFO: posted silence ea5e517b-ac94-45a8-989c-dca1ce8f5f6b: https://alertmanager.torproject.org/#/silences/ea5e517b-ac94-45a8-989c-dca1ce8f5f6b

i think i'll need some date parsing to get this finished, but it's pretty much almost done, yay for documented APIs.

changed title from A silcence should be added for backups on newly-provisioned machines to A silence should be added for backups on newly-provisioned machines

mentioned in commit fabric-tasks@dd8453ab

mentioned in commit fabric-tasks@17cd6736

mentioned in commit fabric-tasks@32ed77f1

okay, so we now have natural date parsing too, i think the magic command is now:

fab silence.create --comment="machine waiting for first backup" --matchers job=bacula --matchers alias=idle-dal-02.torproject.org --ends-at "in 2 days"

@lelutin could you test that and add it to docs?

added Needs Review label and removed Doing label

mentioned in commit fabric-tasks@cf96b08b

mentioned in commit fabric-tasks@46949421

@anarcat nice, I was able to create and update a silence with the two new tasks.

Maybe the only thing that might become annoying in the future is the default author being named "tor". If we only create short-lived silences then it's not such a big issue but we'll have to keep in mind that if we want longer-lived silences we should find a way to get the author from an environment variable or some configuration file somewhere..

I'll add documentation now in the wiki

The username comes from os.getlogin(), which relies on /etc/passwd, i'll change it to getpass.getuser() which also relies on LOGNAME or USERNAME.

https://docs.python.org/3/library/os.html#os.getlogin

done in fabric-tasks@89b483d

...

On 2024-10-16 17:58:43, lelutin (@lelutin) wrote:

Maybe the only thing that might become annoying in the future is the default author being named "tor". If we only create short-lived silences then it's not such a big issue but we'll have to keep in mind that if we want longer-lived silences we should find a way to get the author from an environment variable or some configuration file somewhere..

Documentation sent in 7b11a8cbb0ec38cd41f162ec6bf8b71b2848a2cf in the wiki

closed

reopened

woops I closed this too fast.

now that we have the fabric task for creating a silence, we'll need change the host provisioning task(s) to create such a silence for the new host.

done

mentioned in commit fabric-tasks@89b483d6

mentioned in commit wiki-replica@5450ef3e

mentioned in commit fabric-tasks@4dda9db0

closed