prometheus1's disk is close to being full
As we can see in grafana, the disk on prometheus1 is nearly full:
There were recently some gitlab-related scrape-jobs that were added so we can expect disk usage to climb more.
We should:
-
filter out all metrics from all gitlab-related scrape jobs that cause cardinal explosion and that we won't use -
verify what was added in early may that caused an increase in disk consumption. was there something other than the gitlab-related ones? -
[ ] consider growing the disk volume- growing the disk for this machine at hetzner incurs new recurring costs, so we'd need approval from ops for that
Edited by lelutin