Object storage (MinIO) filesystem filled up to 100%

At 02:19 UTC we got this alert:

DiskWillFillSoon [firing] Disk /srv on minio-01.torproject.org is almost full

I became aware of this at 03:07 UTC at which time only 15GB of space was left available on the filesystem.

/dev/sdb        984G  919G   15G  99% /srv

Shortly thereafter, the filesystem hit 100%.

The current filesystem usage is this:

  • gitlab-ci-runner-cache 216.6 GiB
  • gitlab-dependency-proxy 59.7 MiB
  • gitlab-registry 442.8 GiB
  • network-health 255.0 GiB

I suspect there might have been a rapid increase in the network-health bucket usage, which was created in #41873 (closed)

/cc @hiro @anarcat

Edited by anarcat