vineale out of disk space
vineale has been filling up its /srv
slowly over the past few months:
that turned into a warning recently:
2022-05-22 14:07:13 <nsa> tor-nagios: [vineale] disk usage on /srv is WARNING: DISK WARNING - free space: /srv 3227 MB (8% inode=93%):
2022-05-24 22:07:29 <nsa> tor-nagios: [vineale] disk usage on /srv is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 50 seconds.
2022-05-24 22:21:40 <nsa> tor-nagios: [vineale] disk usage on /srv is WARNING: DISK WARNING - free space: /srv 3218 MB (8% inode=93%):
.. and it's been flapping between WARNING and CRITICAL ever since.
but more seriously, the root partition has filled up last week, completely:
2022-05-29 10:34:59 <nsa> tor-nagios: [vineale] disk usage on / is WARNING: DISK WARNING - free space: / 3468 MB (18% inode=86%):
2022-05-29 12:05:00 <nsa> tor-nagios: [vineale] disk usage on / is CRITICAL: DISK CRITICAL - free space: / 1534 MB (8% inode=86%):
this also triggered warnings from dsa-update-apt-status
:
From: root@vineale.torproject.org (Cron Daemon)
Subject: Cron <root@vineale> [ -x /usr/sbin/dsa-update-apt-status ] && /usr/sbin/dsa-update-apt-status
To: root@vineale.torproject.org
Date: Tue, 31 May 2022 02:10:39 +0000
W: Failed to fetch https://mirror.hetzner.de/debian/packages/dists/buster-backports/InRelease Error writing to output file - write (28: No space left on device) [IP: 2a01:4f8:0:1::1:97 443]
W: Failed to fetch http://security.debian.org/dists/buster/updates/InRelease Error writing to output file - write (28: No space left on device) [IP: 2a04:4e42:3::644 80]
W: Failed to fetch https://mirror.hetzner.de/debian/packages/dists/buster-updates/InRelease Error writing to output file - write (28: No space left on device) [IP: 2a01:4f8:0:1::1:97 443]
W: Some index files failed to download. They have been ignored, or old ones used instead.
that graph is way more problematic:
https://grafana.torproject.org/d/xfpJB9FGz/1-node-exporter-for-prometheus-dashboard-en-v20201010?orgId=1&var-origin_prometheus=&var-job=node&var-hostname=All&var-node=vineale.torproject.org:9100&var-device=All&var-interval=2m&var-maxmount=%2Fhome&var-show_hostname=vineale&var-total=93&viewPanel=174&from=now-1y&to=now
ie. it's fluctuating up and down all week and, suddenly, completely fills up.