no alerts when icinga2 is down
Originally created by @groente on #16126 (Redmine)
icinga2 on monitor was down for several days without us noticing. the web frontend showed no indication of the backend being down and there seem to be no other checks outside of icinga to keep an eye whether our monitoring is still actually functioning.
let’s set an hourly cron for a simple script called that attempts to connect to monitor on port 5665 and mails tails-sysadmins on failure. i’d propose running this script on ecours, what do you think?
Note: For S11, this fits in:
-
B.2 - Keep our infrastructure up-to-date and secure
: No redundancy in monitoring impacts Sysadmins' ability to have up-to-date information about the infra when the monitoring part is down for some reason.
Edited by groente-admin