Skip to content

prometheus1's disk is close to being full

As we can see in grafana, the disk on prometheus1 is nearly full:

https://grafana.torproject.org/d/zbCoGRjnz/disk-usage?orgId=1&from=now-1y&to=now&timezone=utc&var-class=%24__all&var-instance=hetzner-nbg1-01.torproject.org&var-Filters&refresh=auto&viewPanel=panel-2-clone-0

There were recently some gitlab-related scrape-jobs that were added so we can expect disk usage to climb more.

We should:

  1. filter out all metrics from all gitlab-related scrape jobs that cause cardinal explosion and that we won't use
  2. verify what was added in early may that caused an increase in disk consumption. was there something other than the gitlab-related ones?
  3. [ ] consider growing the disk volume
    • growing the disk for this machine at hetzner incurs new recurring costs, so we'd need approval from ops for that
Edited by lelutin
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information