prometheus1 running out of disk space

Date: Fri, 10 Feb 2023 00:03:34 +0000
From: nagios@hetzner-hel1-01.torproject.org
To: anarcat+rapports@orangeseeds.org
Subject: ** PROBLEM Service Alert: hetzner-nbg1-01/disk usage - all is CRITICAL **

***** Icinga *****

Notification Type: PROBLEM

Service: disk usage - all
Host: hetzner-nbg1-01
Address: 195.201.139.202
State: CRITICAL

Date/Time: Fri Feb 10 00:03:34 UTC 2023

Additional Info:

DISK CRITICAL - free space: / 9098 MB (5% inode=99%): /dev 3864 MB (100% inode=99%): /dev/shm 3883 MB (99% inode=99%): /run 776 MB (99% inode=99%): /run/lock 5 MB (100% inode=99%): /tmp 3883 MB (100% inode=99%): /run/credentials 776 MB (99% inode=99%): /var/tmp 9098 MB (5% inode=99%):

disk space was fairly constant until june, and then it started increasing semi-linearly. we gained 30G since then with ~8GB left. we lose about 2GB at each compaction run, and those run roughly every day, so it's unlikely we'll have room until monday.

last time this happened (#40840 (closed)) we just silenced the warning, we can't get away with that this time...