TPA-RFC-33: document history, background, try some requirements

[[_TOC_]]

Summary: this template describes the basic field a TPA-RFC policy

should have. Refer to [policy/tpa-rfc-1-policy](policy/tpa-rfc-1-policy) for the actual policy requirements.

Summary: TODO. currently writing requirements for the new monitoring

system.

# Background

# Proposal

As part of the fleet-wide [Debian bullseye upgrades](howto/upgrades/bullseye), we are

evaluating whether or not it is worth upgrading from the [Debian Icinga

1 package](https://tracker.debian.org/pkg/icinga) to [Icinga 2](https://tracker.debian.org/pkg/icinga), or whether we should consider

[switching to Prometheus][tpo/tpa/team#29864] instead.

Icinga 1 is not available in Debian bullseye and this is therefore a

mandatory upgrade. Because of the design of the service, it cannot

just be converted over easily, so we are considering alternatives.

[tpo/tpa/team#29864]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/29864

## History

TPA's monitoring infrastructure has been originally setup with

[Nagios](https://en.wikipedia.org/wiki/Nagios) and [Munin][]. Nagios was eventually [removed from

Debian in 2016](https://tracker.debian.org/news/818363/removed-351dfsg-22-from-unstable/) and replaced with Icinga 1. Munin somehow "died in

a fire" some time before 2019, when anarcat joined TPA.

[Munin]: https://en.wikipedia.org/wiki/Munin_(software)

At that point, the lack of trending infrastructure was seen as a

serious problem, so [Prometheus](howto/prometheus) and [Grafana](howto/grafana) were deployed as

a stopgap measure.

A secondary Prometheus server (`prometheus2`) was setup for service

admins, with stronger authentication. The rationale was that those

services were more privacy-sensitive and that the primary TPA setup

(`prometheus1`) was too open to the public, which could allow for

side-channels attacks.

Those tools has been used for trending ever since, while keeping

Icinga for monitoring.

During the March 2021 hack week, Prometheus' [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) was

deployed on the secondary Prometheus server to provide alerting to the

Metrics and Anti-Censorship teams.

## Current configuration

The Prometheus configuration is almost fully Puppetized, using the

[Voxpupuli Prometheus module](https://github.com/voxpupuli/puppet-prometheus/), with rare exceptions: the PostgreSQL

exported, needs some manual configuration, and the secondary

Prometheus servers has a [Git repository](https://gitlab.torproject.org/tpo/tpa/prometheus-alerts) where teams can submit

alerts and target definitions, the latter which need to be retrofitted

in Puppet by TPA.

Prometheus is currently scraping 160 exporters, including 95 hosts and

"220772 series". It is using about 100GB of disk space, scrapes

metrics every minute, and keeps those metrics for a year. This implies

that it does about 160 checks per minute, although each check

generates much more than a single metric. We [previously estimated](https://gitlab.torproject.org/tpo/tpa/team/-/issues/31244#note_2541965)

an average of 2000 metrics per host.

The Icinga server's configuration is semi-automatic: configuration is

kept in a YAML file the [tor-nagios.git repository](https://gitweb.torproject.org/admin/tor-nagios.git/). That file, in

turn, gets turned into Nagios (note: not Icinga) configuration files

by a script maintained by TPA. That script was inherited by the Debian

System Administrator (DSA) team.

Nagios's NRPE probes configuration get generated by that same script

and then copied over to the Puppet server, which then distributes

those scripts to all nodes, regardless of where the script is supposed

to run.

Nagios NRPE checks often have many side effects. For example, the

DNSSEC checks automatically renew DNSSEC anchors. The extent of those

side effects has not been fully investigated.

Nagios is currently monitoring 106 hosts and 5658 services, it using

6GB of disk space. It scrapes less than 5% of services every minute,

takes 15 minutes to scrape 85% and less than an hour to scrape 96.6%

of services. The 100 hosts are typically tested for reachability

within 5 minutes. It processes about 200 checks per minute.

## Problem statement

The current Icinga deployment cannot be upgraded without Bullseye as

is. At the very least the post-receive hook in git needs to be

rewritten to support the Icinga 2 configuration files, since Icinga 2

has dropped support for Nagios configurations.

Note that weasel has started on rewriting the [DSA Puppet

configuration](https://salsa.debian.org/dsa-team/mirror/dsa-puppet) to automatically generate Icinga 2 configurations

using [a custom Puppet module](https://salsa.debian.org/dsa-team/mirror/dsa-puppet-weasel-mon), ditching the "push to git"

design. This has the limitation that service admins will not have

access to modifying the alerting configuration unless they somehow get

access to the Puppet repository.

The Nagios configuration is error-prone: because of the way the script

is deployed (post-receive), an error in the configuration can go

un-detected and not being deployed for extended periods of time, which

had lead some services to stay unmonitored.

Since new services must be manually configured in Nagios, this has

lead to new servers and services not being monitored at all, and in

fact many services do not have any form of monitoring.

The way the NRPE configuration is deployed is also problematic:

because the files get deployed asynchronously, it's common for

warnings to pop up in Nagios because the NRPE definitions are not

properly deployed everywhere.

Furthermore, there is some overlap between the Nagios and

Prometheus/Grafana services. In particular, both:

 * Nagios and Prometheus deploy remote probes (Prometheus

   "exporters" and Nagios NRPE)

 * Nagios and Grafana (and Prometheus) provide dashboards (although

   Prometheus' dashboard is minimal)

 * Nagios and Prometheus retain metrics about services

 * Nagios, Prometheus, and Grafana can all do alerting, both Nagios

   and Prometheus are currently used for alerting, TPA and service

   admins in the case of Nagios, only service admins for Prometheus

   right now

## Definitions

## Goals

 * **"system" metrics**: directly under the responsibility of TPA, for

   example: memory, CPU, disk usage, TCP/IP reachability, TLS

   certificates expiration, DNS, etc

<!-- include bugs to be fixed -->

 * **"user" metrics**: under the responsability of service admins, for

   example: number of overloaded relays, `bridges.torproject.org`

   accessibility

 * **alerting**: notifying an operator of a fault related to some

   metric, for example: unreachable host, expired certificate, too

   many overloaded relays, unreachable sites

 * **trending**: long term storage and rendering of metrics and

   alerts, for example: Icinga's alert history, Prometheus TSDB,

   Grafana graphics based on Prometheus

 * **TSDB**: Time-Series Database, for example: Prometheus, RRD,

   Nagios log files, etc

 * **alert storms**: a flood of alerts, often duplicates or related to

   the same underlying root cause

## Requirements

This section establishes what constitutes a valid and sufficient

monitoring system, as provided by TPA.

TODO: review https://gitlab.torproject.org/tpo/tpa/team/-/issues/29864

for requirements (but also everything else?)

### Must have

 * **trending**: it should be possible to look back in metrics history

   and analyse long term patterns (for example: "when did the disk

   last fill up, and what happened then?" or "what is the average

   latency of this service over the last year?")

 * **alerting**: the system should allow operators to set "normal"

   operational thresholds outside of which a service is considered in

   "fault" and an alert is raised (for example: "95 percentile latency

   above 500 ms", "disk full")

 * **user-defined**: user-defined metrics must be somehow configurable

   by the service admins with minimal intervention by TPA

 * **dashboard**: it MUST be possible for TPA operators to access an

   overview dashboard giving the global status of metrics and alerts

   service admins SHOULD also have access to their own

   service-specific dashboards

 * **automatic configuration**: monitoring MUST NOT require a manual

   intervention from TPA when a new server is provisioned

 * **reduced alert fatigue**: avoid sending many alerts for the same

   problem

 * **performance-level alerting**: alerts MUST focus on user-visible

   performance metrics instead of underlying assumptions about

   architecture (e.g. alert on "CI jobs waiting for more than X hours"

   not "load too high on runners"), which should help with alert

   fatigue and auto-configuration

### Nice to have

 * **alert notifications**: it SHOULD be possible for operators to

   receive notifications when a fault is found in the collected

   metrics (as opposed to having to consult a dashboard), the exact

   delivery mechanism is left as a "Nice to have" implementation

   detail

 * **notification groups**: service admins SHOULDN'T receive

   notification from system-level faults and TPA SHOULDN'T receive

   notifications from service-level faults, service A admin should

   *only* receive alerts for service A and not service B

 * **Email alerts**: alerts may be sent by email

 * **IRC alerts**: alerts may be transmitted in an IRC channel, for

   example the current `nsa` bot in `#tor-nagios`

 * **Matrix alerts**: alerts may be transmitted over Matrix instead of

   IRC, assuming this will not jeopardize the reliability of

   notifications compared to the current IRC notifications

 * **predictive alerting**: instead of raising an alert after a given

   threshold (e.g. "disk 90% full"), notify operators about planned

   outage date (e.g. "disk will be full in 5 days")

 * **actionable notifications**: alert dashboards or notifications

   should have a clear resolution path, preferably embedded in the

   notification or, alternatively, possible to lookup in a pager

   playbook (example: "expand this disk before 5 days", "renew the

   DNSSEC records by following this playbook"; counter-example: "disk

   80% full", "security delegations is WARNING")

 * **long term storage**: it should be possible to store metrics

   indefinitely, possibly with downsampling, to make long term

   (multi-year) analysis

 * **automatic service discovery**: it should be possible for *service

   admins* to automatically provide monitoring targets to the

   monitoring server without having to manually make changes to the

   monitoring system

 * **tool deduplication**: duplication of concern should be reduced so

   that only one tool is used for a specific tasks, for example only

   one tool should be collecting metrics, only one tool should be

   issuing alerts, and there should be a single, unified dashboard

 * **high availability**: it should be possibly for the monitoring

   system to survive the failure of one of the monitoring nodes and

   keep functioning, without alert storms, duplicated or missed alerts

### Non-Goals

## Scope

 * **SLA**: we do not plan on providing any specific Service Level

   Agreement through this proposal, those are defined in [TPA-RFC-2:

   Support](policy/tpa-rfc-2-support)

 * **on-call rotation**: we do not provide 24/7 on-call services, nor

   do we ascribe to an on-call schedule - there is a "star of the

   weeks" that's responsible for checking the status of things and

   dealing with interruptions, but they do so during work hours, in

   their own time, in accordance with [TPA-RFC-2: Support](policy/tpa-rfc-2-support)

## Affected users

 * **escalation**: we do not need to call Y when X person fails to answer,

   mainly because we do not expect either X or Y to answer alerts immediately

 * **log analysis**: while logging might *eventually* be considered

   part of our monitoring systems, the questions of whether we use

   syslog-ng, rsyslog, journald, or loki are currently out of scope of

   this proposal

# Examples or Personas

 * ...

# Proposal

# Alternatives considered

# Costs