Skip to content
GitLab
  • Menu
Projects Groups Snippets
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
  • TPA team TPA team
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Issues 173
    • Issues 173
    • List
    • Boards
    • Service Desk
    • Milestones
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
  • Wiki
    • Wiki
  • Activity
  • Create a new issue
  • Issue Boards
Collapse sidebar
  • The Tor Project
  • TPA
  • TPA teamTPA team
  • Issues
  • #40330
Closed
Open
Created Jul 01, 2021 by anarcat@anarcatOwner

multi-year prometheus metrics storage

in #31244 (closed), we discussed provisionning the prometheus server to store "long-term" prometheus metrics. what that meant wasn't exactly clear: we settled on "one year" (as opposed to the default "30 days"). but now I feel it's not enough. i'd like to have a permanent record of metrics: basically a multi-year account of all metrics (or maybe just critical metrics, but that's harder to figure out).

in the original ticket, we considered the idea of having a secondary server that would scrape metrics from the first, but ended up rejecting this idea. one of the reasons is that then we need Grafana and other tools to talk to multiple datasources and it makes things complicated.

but in this post, one of the prometheus authors mentions the possibility of the primary prometheus server using the secondary, long-term server as a datasource itself for metrics it doesn't have available.

that way we could have a much smaller (say 30 days!) server that would fetch metrics every N seconds like now), and a much longer term server that would fetch metrics much less frequently (say 5 minutes or an hour).

calculate what the requirements would be based on the current and projected metrics counts, retention period and scrape frequencies.

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Assignee
Assign to
Time tracking