lists-01 disk filling up abruptly

got this alert on IRC:

16:22:36 -ALERTOR1:#tor-alerts- DiskWillFillSoon [firing] Disk / on lists-01.torproject.org is almost full

(time UTC-5)

(two issues with that alert:

  1. it should say when. the long version says "less than 24h", we should do better, if possible
  2. we should have a direct link to the dashboard, in this case https://grafana.torproject.org/d/zbCoGRjnz/disk-usage?var-instance=lists-01.torproject.org&orgId=1&from=now-7d&to=now

image )

looking at grafana, disk use rate spiked in the last 24h. grafana ETA is 0.8 days, which is ~19h, but i suspect it could be worse than this.

could be related to perf issues (#41957 (closed))

will try to look later.

Edited by anarcat