---
title: TPA-RFC-33: Monitoring
---

[[_TOC_]]

Summary: TODO. currently writing requirements for the new monitoring
system.

# Background

As part of the fleet-wide [Debian bullseye upgrades](howto/upgrades/bullseye), we are
evaluating whether or not it is worth upgrading from the [Debian Icinga
1 package](https://tracker.debian.org/pkg/icinga) to [Icinga 2](https://tracker.debian.org/pkg/icinga), or whether we should consider
[switching to Prometheus][tpo/tpa/team#29864] instead.

Icinga 1 is not available in Debian bullseye and this is therefore a
mandatory upgrade. Because of the design of the service, it cannot
just be converted over easily, so we are considering alternatives.

[tpo/tpa/team#29864]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/29864

## History

TPA's monitoring infrastructure has been originally setup with
[Nagios](https://en.wikipedia.org/wiki/Nagios) and [Munin][]. Nagios was eventually [removed from
Debian in 2016](https://tracker.debian.org/news/818363/removed-351dfsg-22-from-unstable/) and replaced with Icinga 1. Munin somehow "died in
a fire" some time before 2019, when anarcat joined TPA.

[Munin]: https://en.wikipedia.org/wiki/Munin_(software)

At that point, the lack of trending infrastructure was seen as a
serious problem, so [Prometheus](howto/prometheus) and [Grafana](howto/grafana) were [deployed in
2019](https://gitlab.torproject.org/tpo/tpa/team/-/issues/29681) as a stopgap measure.

A secondary Prometheus server (`prometheus2`) was setup for service
admins, with stronger authentication. The rationale was that those
services were more privacy-sensitive and that the primary TPA setup
(`prometheus1`) was too open to the public, which could allow for
side-channels attacks.

Those tools has been used for trending ever since, while keeping
Icinga for monitoring.

During the March 2021 hack week, Prometheus' [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) was
deployed on the secondary Prometheus server to provide alerting to the
Metrics and Anti-Censorship teams.

## Current configuration

The Prometheus configuration is almost fully Puppetized, using the
[Voxpupuli Prometheus module](https://github.com/voxpupuli/puppet-prometheus/), with rare exceptions: the PostgreSQL
exported, needs some manual configuration, and the secondary
Prometheus servers has a [Git repository](https://gitlab.torproject.org/tpo/tpa/prometheus-alerts) where teams can submit
alerts and target definitions, the latter which need to be retrofitted
in Puppet by TPA.

Prometheus is currently scraping 160 exporters, including 95 hosts and
"220772 series". It is using about 100GB of disk space, scrapes
metrics every minute, and keeps those metrics for a year. This implies
that it does about 160 checks per minute, although each check
generates much more than a single metric. We [previously estimated](https://gitlab.torproject.org/tpo/tpa/team/-/issues/31244#note_2541965)
an average of 2000 metrics per host.

The Icinga server's configuration is semi-automatic: configuration is
kept in a YAML file the [tor-nagios.git repository](https://gitweb.torproject.org/admin/tor-nagios.git/). That file, in
turn, gets turned into Nagios (note: not Icinga) configuration files
by a script maintained by TPA. That script was inherited by the Debian
System Administrator (DSA) team.

Nagios's NRPE probes configuration get generated by that same script
and then copied over to the Puppet server, which then distributes
those scripts to all nodes, regardless of where the script is supposed
to run.

Nagios NRPE checks often have many side effects. For example, the
DNSSEC checks automatically renew DNSSEC anchors. The extent of those
side effects has not been fully investigated.

Nagios is currently monitoring 106 hosts and 5658 services, it using
6GB of disk space. It scrapes less than 5% of services every minute,
takes 15 minutes to scrape 85% and less than an hour to scrape 96.6%
of services. The 100 hosts are typically tested for reachability
within 5 minutes. It processes about 200 checks per minute.

## Problem statement

The current Icinga deployment cannot be upgraded without Bullseye as
is. At the very least the post-receive hook in git needs to be
rewritten to support the Icinga 2 configuration files, since Icinga 2
has dropped support for Nagios configurations.

Note that weasel has started on rewriting the [DSA Puppet
configuration](https://salsa.debian.org/dsa-team/mirror/dsa-puppet) to automatically generate Icinga 2 configurations
using [a custom Puppet module](https://salsa.debian.org/dsa-team/mirror/dsa-puppet-weasel-mon), ditching the "push to git"
design. This has the limitation that service admins will not have
access to modifying the alerting configuration unless they somehow get
access to the Puppet repository.

The Nagios configuration is error-prone: because of the way the script
is deployed (post-receive), an error in the configuration can go
un-detected and not being deployed for extended periods of time, which
had lead some services to stay unmonitored.

Since new services must be manually configured in Nagios, this has
lead to new servers and services not being monitored at all, and in
fact many services do not have any form of monitoring.

The way the NRPE configuration is deployed is also problematic:
because the files get deployed asynchronously, it's common for
warnings to pop up in Nagios because the NRPE definitions are not
properly deployed everywhere.

Furthermore, there is some overlap between the Nagios and
Prometheus/Grafana services. In particular, both:

 * Nagios and Prometheus deploy remote probes (Prometheus
   "exporters" and Nagios NRPE)

 * Nagios and Grafana (and Prometheus) provide dashboards (although
   Prometheus' dashboard is minimal)

 * Nagios and Prometheus retain metrics about services

 * Nagios, Prometheus, and Grafana can all do alerting, both Nagios
   and Prometheus are currently used for alerting, TPA and service
   admins in the case of Nagios, only service admins for Prometheus
   right now

## Definitions

 * **"system" metrics**: directly under the responsibility of TPA, for
   example: memory, CPU, disk usage, TCP/IP reachability, TLS
   certificates expiration, DNS, etc

 * **"user" metrics**: under the responsability of service admins, for
   example: number of overloaded relays, `bridges.torproject.org`
   accessibility

 * **alerting**: notifying an operator of a fault related to some
   metric, for example: unreachable host, expired certificate, too
   many overloaded relays, unreachable sites

 * **trending**: long term storage and rendering of metrics and
   alerts, for example: Icinga's alert history, Prometheus TSDB,
   Grafana graphics based on Prometheus

 * **TSDB**: Time-Series Database, for example: Prometheus, RRD,
   Nagios log files, etc

 * **alert storms**: a flood of alerts, often duplicates or related to
   the same underlying root cause

## Requirements

This section establishes what constitutes a valid and sufficient
monitoring system, as provided by TPA.

### Must have

 * **trending**: it should be possible to look back in metrics history
   and analyse long term patterns (for example: "when did the disk
   last fill up, and what happened then?" or "what is the average
   latency of this service over the last year?")

 * **alerting**: the system should allow operators to set "normal"
   operational thresholds outside of which a service is considered in
   "fault" and an alert is raised (for example: "95 percentile latency
   above 500 ms", "disk full")

 * **user-defined**: user-defined metrics must be somehow configurable
   by the service admins with minimal intervention by TPA

 * **dashboard**: it MUST be possible for TPA operators to access an
   overview dashboard giving the global status of metrics and alerts
   service admins SHOULD also have access to their own
   service-specific dashboards

 * **automatic configuration**: monitoring MUST NOT require a manual
   intervention from TPA when a new server is provisioned

 * **reduced alert fatigue**: avoid sending many alerts for the same
   problem

 * **performance-level alerting**: alerts MUST focus on user-visible
   performance metrics instead of underlying assumptions about
   architecture (e.g. alert on "CI jobs waiting for more than X hours"
   not "load too high on runners"), which should help with alert
   fatigue and auto-configuration

### Nice to have

 * **alert notifications**: it SHOULD be possible for operators to
   receive notifications when a fault is found in the collected
   metrics (as opposed to having to consult a dashboard), the exact
   delivery mechanism is left as a "Nice to have" implementation
   detail

 * **notification groups**: service admins SHOULDN'T receive
   notification from system-level faults and TPA SHOULDN'T receive
   notifications from service-level faults, service A admin should
   *only* receive alerts for service A and not service B

 * **Email alerts**: alerts may be sent by email

 * **IRC alerts**: alerts may be transmitted in an IRC channel, for
   example the current `nsa` bot in `#tor-nagios`

 * **Matrix alerts**: alerts may be transmitted over Matrix instead of
   IRC, assuming this will not jeopardize the reliability of
   notifications compared to the current IRC notifications

 * **predictive alerting**: instead of raising an alert after a given
   threshold (e.g. "disk 90% full"), notify operators about planned
   outage date (e.g. "disk will be full in 5 days")

 * **actionable notifications**: alert dashboards or notifications
   should have a clear resolution path, preferably embedded in the
   notification or, alternatively, possible to lookup in a pager
   playbook (example: "expand this disk before 5 days", "renew the
   DNSSEC records by following this playbook"; counter-example: "disk
   80% full", "security delegations is WARNING")

 * **long term storage**: it should be possible to store metrics
   indefinitely, possibly with downsampling, to make long term
   (multi-year) analysis

 * **automatic service discovery**: it should be possible for *service
   admins* to automatically provide monitoring targets to the
   monitoring server without having to manually make changes to the
   monitoring system

 * **tool deduplication**: duplication of concern should be reduced so
   that only one tool is used for a specific tasks, for example only
   one tool should be collecting metrics, only one tool should be
   issuing alerts, and there should be a single, unified dashboard

 * **high availability**: it should be possibly for the monitoring
   system to survive the failure of one of the monitoring nodes and
   keep functioning, without alert storms, duplicated or missed alerts

### Non-Goals

 * **SLA**: we do not plan on providing any specific Service Level
   Agreement through this proposal, those are defined in [TPA-RFC-2:
   Support](policy/tpa-rfc-2-support)

 * **on-call rotation**: we do not provide 24/7 on-call services, nor
   do we ascribe to an on-call schedule - there is a "star of the
   weeks" that's responsible for checking the status of things and
   dealing with interruptions, but they do so during work hours, in
   their own time, in accordance with [TPA-RFC-2: Support](policy/tpa-rfc-2-support)

 * **escalation**: we do not need to call Y when X person fails to answer,
   mainly because we do not expect either X or Y to answer alerts immediately

 * **log analysis**: while logging might *eventually* be considered
   part of our monitoring systems, the questions of whether we use
   syslog-ng, rsyslog, journald, or loki are currently out of scope of
   this proposal

# Personas

## Jackie, the TPA admin

Jackie is a member of the TPA team. She has access to the Puppet
repository, and all other Git repositories managed by TPA. She has
access to everything and the kitchen sink, and is generally asked to
fix all of this on a regular basis.

She sometimes ends rotating as the "star of the week", which makes her
responsible for handling "interruptions", new tickets, and also
keeping an eye on the monitoring server. This involves responding to
alerts like, by order of frequency in the last year:

 * 2805 pending upgrades (packages blocked from unattended upgrades)
 * 2325 pending restarts (services blocked from needrestart) or reboots
 * 1818 load alerts
 * 1709 disk usage alerts
 * 1062 puppet catalog failures
 * 999 uptime alerts (after reboots)
 * 843 reachability alerts
 * 602 process count alerts
 * 585 swap usage alerts
 * 499 backup alerts
 * 484 systemd alerts e.g. systemd says "degraded" and you get to
   figure out what didn't start)
 * 383 zombie alerts
 * 199 missing process (e.g. "0 postgresql processes")
 * 168 unwanted processes or network services
 * numerous warnings about service admin specific things:
   * 129 mirror static sync alert storms (15 at a time), mostly host
     unreachability warnings
   * 69 bridgedb
   * 67 collector
   * 26 out of date chroots
   * 14 translation cron - stuck
   * 17 mail queue (polyanthum)
 * 96 RAID - DRBD warnings, mostly false alerts
 * 95 SSL cert warnings about db.torproject.org, all about the same
   problem
 * 94 DNS SOA synchronization alerts
 * 88 DNSSEC alerts (81 delegation and signature expiry, 4 DS expiry,
   2 security delegations)
 * 69 hardware RAID warnings
 * 69 Ganeti cluster verification warnings
 * numerous alerts about NRPE availability, often falsly flagged as an
   error in a specific service (e.g. "SSL cert - host")
 * 28 unbound trust alerts
 * 24 alerts about unexpected software RAID
 * 19 SAN health alerts
 * 5 false (?) alerts about mdadm resyncing
 * 3 expiring Let's Encrypt X509 certificates alerts
 * 3 redis liveness alerts
 * 4 onionoo backend reachability alerts

Jackie finds that is way too much noise. That list is actually an
interpretation of the actual alerts received to make them more human
readable.

The current Nagios dashboard, that said, is pretty useful in the sense
that she can ignore all of those emails and just look at the dashboard
to see what's *actually* going on right now. This sometimes causes her
to miss some problems, however.

TODO: what does she want out of monitoring?

### Note

The alert list was created with the following utterly horrible shell
pipeline:

    notmuch search --format=sexp  tag:nagios date:2021-06-20.. \
      | sed -n '/PROBLEM/{s/.*:subject "//;s/" :query .*//;s/.*Alert: [^\/ ]*[\/ ]//;p}' 
      | sed -e 's/ is UNKNOWN.*//' -e 's/ is WARNING.*//' -e 's/ is CRITICAL.*//' \
        -e 's/disk usage .*/disk usage/'\
        -e 's/mirror static sync.*/mirror static sync/' \
        -e 's/unwanted.*/unwanted/' \
        -e '/DNS/s/ - .*//' \
        -e 's/process - .*/process/' \
        -e 's/network service - .*/network service/' \
        -e 's/backup - .*/backup/' \
        -e 's/mirror sync - .*/mirror sync/' \
        | sort | uniq -c | sort -n 

Then the alerts were parsed by a TPA brain. Some alerts were redacted
because considered mostly noise.

## Ethan, the service admin

TODO: what do service admins want?

# Proposal

# Alternatives considered

# Costs

# Approval

# Deadline

# Status

This proposal is currently in the `draft` state.

# References

This proposal is discussed in [tpo/tpa/team#40755](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40755).
