move long-term metrics discussion to TPA-RFC-33 (team#40755)

generated by Prometheus, and use it to leverage side-channel attacks,

which is why the external Prometheus server access is restricted.

### Long term metrics storage

Metrics are held for about a year or less, depending on the server,

see [ticket 29388][] for storage requirements and possible

alternatives for data retention policies.

Note that extra long-term data retention might be possible [using

the remote read functionality](https://www.robustperception.io/looking-beyond-retention), which enables the primary server to

read metrics from a secondary, longer-term server transparently,

keeping graphs working without having to change data source, for

example.

That way you could have a short-term server which keeps lots of

metrics and polls every minute or even 15 seconds, but keeps (say)

only 30 days of data and a long-term server which would poll the

short-term server every (say) 5 minutes) but keep (say) 5 years of

metrics. But how much data would that be?

The [last time we made an estimate, in May 2020](https://gitlab.torproject.org/tpo/tpa/team/-/issues/31244#note_2541965), we had the

following calculation for 1 minute polling interval over a year:

```

> 365d×1.3byte/(1min)×2000×78 to Gibyte

99,271238 gibibytes

```

At the time of writing (August 2021), that is still the configured

interval, and the disk usage roughly matches that (98GB used). This

implies that we could store about 5 years of metrics with a 5 minute

polling interval, using the same disk usage, obviously:

```

> 5*365d×1.3byte/(5min)×2000×78 to Gibyte

99,271238 gibibytes

```

... or 15 years with 15 minutes, etc... As a rule of thumb, as long as

we multiple the scrape interval, we can multiply the retention period

as well.

On the other side, we might be able to increase granularity quite a

bit by lowering the retention to (say) 30 days and 5 seconds polling

interval, which would give us:

```

> 30d*1.3byte/(5 second)*2000*78 to Gibyte

97,911358 gibibytes

```

That might be a bit aggressive though: the default Prometheus

`scrape_interval` is 15 seconds, not 5 seconds... With the defaults

(15 seconds scrape interval, 30 days retention), we'd be at about

30GiB disk usage, which makes for a quite reasonable and easy to

replicate primary server.

A few more samples calculations:

| Frequency | Retention period | Storage used |

|-----------|------------------|--------------|

|   5 second|           30 days|       100 GiB|

|  15 second|           30 days|        33 GiB|

|  15 second|            1 year|       400 GiB|

|      1 min|            1 year|       100 GiB|

|      1 min|           10 year|      1000 GiB|

|      5 min|            1 year|        20 GiB|

|      5 min|            5 year|        60 GiB|

|      5 min|           10 year|       100 GiB|

|     15 min|            1 year|         7 GiB|

|     15 min|            5 year|        33 GiB|

|     15 min|           10 year|        66 GiB|

|     15 min|          100 year|       662 GiB|

|     1 hour|            1 year|         2 GiB|

|     1 hour|            5 year|         8 GiB|

|     1 hour|           10 year|        17 GiB|

|     1 hour|          100 year|       167 GiB|

Another option would be to use another backend for prometheus metrics,

something like [TimescaleDB](https://www.timescale.com), see [this blog post](https://blog.timescale.com/blog/sql-nosql-data-storage-for-prometheus-devops-monitoring-postgresql-timescaledb-time-series-3cde27fd1e07/) for more

information.

Note that [TPA-RFC-33][] discusses alternative metrics retention

policies.

See also the projects [Cortex](https://cortexmetrics.io/) (not in Debian) and [Thanos](https://thanos.io/)

([not in Debian](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1032842), [used at Wikimedia](https://wikitech.wikimedia.org/wiki/Thanos)).

[TPA-RFC-33]: policy/tpa-rfc-33-monitoring

## Backups

### Retention

In the highly available setup, different servers have different

retention policies and scrape interval. The primary server has a short

retention policy, similar to the current server (one year, 1 minute

scrape interval) while the other has a longer retention policy (10

years, 5 minutes) and a larger disk, for longer term queries.

We have been looking at [longer-term metrics retention][]. This could

be accomplished in a highly available setup, different servers have

different retention policies and scrape interval. The primary server

would have a short retention policy, similar or shorter to the current

server (one year, 1 minute scrape interval) while the other has a

longer retention policy (10 years, 5 minutes) and a larger disk, for

longer term queries.

The way to accomplish this is [using the remote read

functionality](https://www.robustperception.io/looking-beyond-retention), which enables the primary server to read metrics

from a secondary, longer-term server transparently, keeping graphs

working without having to change data source, for example.

The [last time we made an estimate, in May 2020](https://gitlab.torproject.org/tpo/tpa/team/-/issues/31244#note_2541965), we had the

following calculation for 1 minute polling interval over a year:

```

> 365d×1.3byte/(1min)×2000×78 to Gibyte

99,271238 gibibytes

```

When that analysis was last performed (August 2021), that was still

the configured interval, and the disk usage roughly matched that (98GB

used). This implies that we could store about 5 years of metrics with

a 5 minute polling interval, using the same disk usage, obviously:

```

> 5*365d×1.3byte/(5min)×2000×78 to Gibyte

99,271238 gibibytes

```

... or 15 years with 15 minutes, etc... As a rule of thumb, as long as

we multiple the scrape interval, we can multiply the retention period

as well.

On the other side, we might be able to increase granularity quite a

bit by lowering the retention to (say) 30 days and 5 seconds polling

interval, which would give us:

```

> 30d*1.3byte/(5 second)*2000*78 to Gibyte

97,911358 gibibytes

```

That might be a bit aggressive though: the default Prometheus

`scrape_interval` is 15 seconds, not 5 seconds... With the defaults

(15 seconds scrape interval, 30 days retention), we'd be at about

30GiB disk usage, which makes for a quite reasonable and easy to

replicate primary server.

A few more samples calculations:

| Interval  | Retention | Storage |

|-----------|-----------|---------|

|  5 second |   30 days |  100 GiB|

| 15 second |   30 days |   33 GiB|

| 15 second |    1 year |  400 GiB|

|     1 min |    1 year |  100 GiB|

|     1 min |   10 year | 1000 GiB|

|     5 min |    1 year |   20 GiB|

|     5 min |    5 year |   60 GiB|

|     5 min |   10 year |  100 GiB|

|     5 min |  100 year | 1000 GiB|

|    15 min |    1 year |    7 GiB|

|    15 min |    5 year |   33 GiB|

|    15 min |   10 year |   66 GiB|

|    15 min |  100 year |  662 GiB|

|    1 hour |    1 year |    2 GiB|

|    1 hour |    5 year |    8 GiB|

|    1 hour |   10 year |   17 GiB|

|    1 hour |  100 year |  167 GiB|

Naturally, those are going to scale up with service complexity and

fleet size, so they should be considered just to be an order of

magnitude.

For the primary server, a 30 day / 15 second retention policy seems

lean and mean, while for the secondary server, a 100 year retention

policy and 5 minute interval should server our needs for the

foreseeable future.

TODO: actually redo current calculations: current disk usage, metrics

counts, retention, etc

TODO: name each server according to retention? say mon-short-01 and

the other mon-long-02?

TODO: review https://gitlab.torproject.org/tpo/tpa/team/-/issues/40330

TODO: there's something about an upper limit to scrape interval, check

https://utcc.utoronto.ca/~cks/space/blog/sysadmin/MetricsHowFarBackDepends?showcomments

and source

TODO: there's something about an upper limit to scrape interval, check that.

TODO: double-check that remote read and pull from the other actually works

TODO: double-check that remote read and pull from the other actually

works, read https://www.robustperception.io/looking-beyond-retention/

### Self-monitoring

TODO: https://grafana.com/blog/2022/06/14/introducing-grafana-oncall-oss-open-source/

## Cortex, Thanos and TimescaleDB

TODO: WMF [use Thanos](https://wikitech.wikimedia.org/wiki/Thanos) for long-term metrics retention, see  https://thanos.io/ ([not in Debian](https://thanos.io/)).

Another option would be to use another backend for prometheus metrics,

something like [TimescaleDB](https://www.timescale.com), see [this blog post](https://blog.timescale.com/blog/sql-nosql-data-storage-for-prometheus-devops-monitoring-postgresql-timescaledb-time-series-3cde27fd1e07/) for more

information.

See also the projects [Cortex](https://cortexmetrics.io/) (not in Debian) and [Thanos](https://thanos.io/)

([not in Debian](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1032842), [used at Wikimedia](https://wikitech.wikimedia.org/wiki/Thanos)).

## InfluxDB

In [this random GitHub project][], a user reports using InfluxDB

instead of Prometheus for long term, "keep forever" metrics

storage. it's tricky though: in 2017, InfluxDB [added remote

read/write support][] but then promptly went ahead and [removed it

from InfluxDB 2.0 in 2021][]. That functionality still seems available

through [Telegraf][], which is [not packaged in Debian][793749] (but

is in Ubuntu).

 [Telegraf]: https://docs.influxdata.com/telegraf/

 [removed it from InfluxDB 2.0 in 2021]: https://www.influxdata.com/blog/prometheus-remote-write-support-with-influxdb-2-0/

 [added remote read/write support]: https://www.influxdata.com/blog/influxdb-now-supports-prometheus-remote-read-write-natively/

 [this random GitHub project]: https://github.com/djjudas21/ecowitt-exporter/issues/20#issuecomment-1925246463

After a quick chat with GPT-4, it appears that InfluxDB is somewhat of

an "open core" model, with the multi-server, high availability

features part of the closed-source software. This is based on a

[controversy documented on Wikipedia][] that [dates from

2016][]. There's [influxdb-relay][] now but it seems a tad more

complicated than Prometheus' high availability setups which are all

open source, as far as I know.

 [influxdb-relay]: https://github.com/influxdata/influxdb-relay

 [dates from 2016]: https://www.influxdata.com/blog/update-on-influxdb-clustering-high-availability-and-monetization/

 [controversy documented on Wikipedia]: https://en.wikipedia.org/wiki/InfluxDB#Closed_source_clustering_components

Also, InfluxDB is a fundamentally different architecture, with a

different querying system: it would be hard to keep the same alerts

and Grafana dashboards across the two systems.

We have therefore completely excluded InfluxDB at this point in time.

[793749]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=793749

## Nagios

https://gitlab.torproject.org/tpo/tpa/team/-/issues/29864#note_2801540

TODO: https://gitlab.torproject.org/tpo/tpa/team/-/issues/29864#note_2801540

## SLA and notifications improvements

 * [exporter policy][tpo/tpa/team#41280]

 * [improve incident response procedures][tpo/tpa/team#40421]

 * [better monitoring for webserver response times][]

 * [longer-term metrics retention][]

[better monitoring for webserver response times]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40568

 [longer-term metrics retention]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40330