multi-year prometheus metrics storage
in #31244 (closed), we discussed provisionning the prometheus server to store "long-term" prometheus metrics. what that meant wasn't exactly clear: we settled on "one year" (as opposed to the default "30 days"). but now I feel it's not enough. i'd like to have a permanent record of metrics: basically a multi-year account of all metrics (or maybe just critical metrics, but that's harder to figure out).
in the original ticket, we considered the idea of having a secondary server that would scrape metrics from the first, but ended up rejecting this idea. one of the reasons is that then we need Grafana and other tools to talk to multiple datasources and it makes things complicated.
but in this post, one of the prometheus authors mentions the possibility of the primary prometheus server using the secondary, long-term server as a datasource itself for metrics it doesn't have available.
that way we could have a much smaller (say 30 days!) server that would fetch metrics every N seconds like now), and a much longer term server that would fetch metrics much less frequently (say 5 minutes or an hour).
calculate what the requirements would be based on the current and projected metrics counts, retention period and scrape frequencies.