prometheus reboot alerts are extremely noisy on IRC
reboot jobs are somewhat farcical on IRC:
10:46:32 -ALERTOR1:#tor-alerts- [warning:resolved] Host alberti.torproject.org needs to reboot
10:46:32 -ALERTOR1:#tor-alerts- [warning:firing] Host anonticket-01.torproject.org needs to reboot
10:46:32 -ALERTOR1:#tor-alerts- [warning:resolved] Host archive-01.torproject.org needs to reboot
10:46:32 -ALERTOR1:#tor-alerts- [warning:firing] Host backup-storage-01.torproject.org needs to reboot
10:46:35 -ALERTOR1:#tor-alerts- [warning:firing] Host bungei.torproject.org needs to reboot
10:46:38 -ALERTOR1:#tor-alerts- [warning:firing] Host cdn-backend-sunet-02.torproject.org needs to reboot
10:46:40 -ALERTOR1:#tor-alerts- [warning:firing] Host check-01.torproject.org needs to reboot
10:46:43 -ALERTOR1:#tor-alerts- [warning:firing] Host chives.torproject.org needs to reboot
10:46:46 -ALERTOR1:#tor-alerts- [warning:firing] Host ci-runner-x86-02.torproject.org needs to reboot
10:46:49 -ALERTOR1:#tor-alerts- [warning:firing] Host colchicifolium.torproject.org needs to reboot
10:46:51 -ALERTOR1:#tor-alerts- [warning:firing] Host collector-02.torproject.org needs to reboot
10:46:54 -ALERTOR1:#tor-alerts- [warning:firing] Host crm-ext-01.torproject.org needs to reboot
10:46:57 -ALERTOR1:#tor-alerts- [warning:firing] Host crm-int-01.torproject.org needs to reboot
10:47:00 -ALERTOR1:#tor-alerts- [warning:resolved] Host dal-rescue-02.torproject.org needs to reboot
10:47:02 -ALERTOR1:#tor-alerts- [warning:firing] Host dangerzone-01.torproject.org needs to reboot
10:47:05 -ALERTOR1:#tor-alerts- [warning:firing] Host donate-01.torproject.org needs to reboot
10:47:08 -ALERTOR1:#tor-alerts- [warning:firing] Host gayi.torproject.org needs to reboot
10:47:10 -ALERTOR1:#tor-alerts- [warning:firing] Host hetzner-hel1-02.torproject.org needs to reboot
10:47:13 -ALERTOR1:#tor-alerts- [warning:firing] Host hetzner-nbg1-01.torproject.org needs to reboot
10:47:16 -ALERTOR1:#tor-alerts- [warning:firing] Host mandos-01.torproject.org needs to reboot
10:47:19 -ALERTOR1:#tor-alerts- [warning:firing] Host meronense.torproject.org needs to reboot
10:47:21 -ALERTOR1:#tor-alerts- [warning:firing] Host metrics-store-01.torproject.org needs to reboot
10:47:24 -ALERTOR1:#tor-alerts- [warning:resolved] Host metricsdb-01.torproject.org needs to reboot
10:47:27 -ALERTOR1:#tor-alerts- [warning:firing] Host minio-01.torproject.org needs to reboot
10:47:30 -ALERTOR1:#tor-alerts- [warning:firing] Host neriniflorum.torproject.org needs to reboot
10:47:32 -ALERTOR1:#tor-alerts- [warning:firing] Host onionoo-frontend-01.torproject.org needs to reboot
10:47:36 -ALERTOR1:#tor-alerts- [warning:firing] Host palmeri.torproject.org needs to reboot
10:47:38 -ALERTOR1:#tor-alerts- [warning:firing] Host probetelemetry-01.torproject.org needs to reboot
10:47:41 -ALERTOR1:#tor-alerts- [warning:firing] Host puppetdb-01.torproject.org needs to reboot
10:47:43 -ALERTOR1:#tor-alerts- [warning:firing] Host rdsys-frontend-01.torproject.org needs to reboot
10:47:46 -ALERTOR1:#tor-alerts- [warning:firing] Host rdsys-test-01.torproject.org needs to reboot
10:47:49 -ALERTOR1:#tor-alerts- [warning:firing] Host rude.torproject.org needs to reboot
10:47:52 -ALERTOR1:#tor-alerts- [warning:firing] Host static-master-fsn.torproject.org needs to reboot
10:47:54 -ALERTOR1:#tor-alerts- [warning:firing] Host survey-01.torproject.org needs to reboot
10:47:57 -ALERTOR1:#tor-alerts- [warning:firing] Host tb-build-02.torproject.org needs to reboot
10:48:00 -ALERTOR1:#tor-alerts- [warning:firing] Host tb-build-03.torproject.org needs to reboot
10:48:02 -ALERTOR1:#tor-alerts- [warning:firing] Host tb-build-06.torproject.org needs to reboot
10:48:05 -ALERTOR1:#tor-alerts- [warning:firing] Host tb-pkgstage-01.torproject.org needs to reboot
10:48:08 -ALERTOR1:#tor-alerts- [warning:resolved] Host tbb-nightlies-master.torproject.org needs to reboot
10:48:11 -ALERTOR1:#tor-alerts- [warning:firing] Host telegram-bot-01.torproject.org needs to reboot
10:48:14 -ALERTOR1:#tor-alerts- [warning:firing] Host weather-01.torproject.org needs to reboot
10:48:16 -ALERTOR1:#tor-alerts- [warning:firing] Host web-dal-07.torproject.org needs to reboot
10:48:19 -ALERTOR1:#tor-alerts- [warning:firing] Host web-dal-08.torproject.org needs to reboot
10:48:22 -ALERTOR1:#tor-alerts- [warning:firing] Host web-fsn-01.torproject.org needs to reboot
nagios, which we found noisy before, is actually much better in that regard, possibly because it doesn't check that often:
10:47:14 <nsa> tor-nagios: [archive-01] system - all services running is OK: running
10:47:55 <nsa> tor-nagios: [materculae] apt - security updates is OK: OK: 657 ok, 5 rc
10:50:55 <nsa> tor-nagios: [alberti] uptime check is WARNING: Uptime WARNING: Uptime less than one hour (currently 20 minutes)