metrics-store-01 disk space warning
Date: Thu, 26 Oct 2023 22:05:38 +0000
From: nagios@hetzner-hel1-01.torproject.org
To: anarcat+rapports@orangeseeds.org
Subject: ** PROBLEM Service Alert: metrics-store-01/disk usage - all is WARNING **
***** Icinga *****
Notification Type: PROBLEM
Service: disk usage - all
Host: metrics-store-01
Address: 116.202.120.183
State: WARNING
Date/Time: Thu Oct 26 22:05:38 UTC 2023
Additional Info:
DISK WARNING - free space: / 7020 MB (74% inode=89%): /dev 966 MB (100% inode=99%): /dev/shm 985 MB (99% inode=99%): /run 196 MB (99% inode=99%): /run/lock 5 MB (100% inode=99%): /tmp 985 MB (100% inode=99%): /srv 78438 MB (10% inode=99%): /home 78438 MB (10% inode=99%): /run/credentials 196 MB (99% inode=99%): /var/tmp 7020 MB (74% inode=89%):
that's not very readable, basically it's /srv
that passed 90% usage:
https://grafana.torproject.org/d/zbCoGRjnz/disk-usage?orgId=1&var-class=All&var-instance=metrics-store-01.torproject.org&from=now-1y&to=now
last time this happened, we doubled the disk space from 400 to 800G (#41322 (closed)) and before that it was from 300 to 400G (#41075 (closed)).
seems like we're running around in circles a bit here... how big is this thing going to be, @hiro? should we double again (to 1.6T)?? i mean we have the room on those HDDs...
/cc @hiro