gitlab-02 running out of disk space in gitlab-shared
GitLab is running out of disk space again.
something happened in the last month that made it eat 100GB in about 3 weeks:
https://grafana.torproject.org/d/zbCoGRjnz/disk-usage?orgId=1&var-class=All&var-instance=gitlab-02.torproject.org&from=now-90d&to=now
even more visible in the yearly:
https://grafana.torproject.org/d/zbCoGRjnz/disk-usage?orgId=1&var-class=All&var-instance=gitlab-02.torproject.org&from=now-1y&to=now
our daily rate is 6.65GB which means we have slightly more than 48h before running out of disk completely (15GB remaining)
battle plan:
-
@lavamind figure out a short term fix
-
erase artifacts in projects under
tpo/web/l10n
+ disable "Keep latest artifacts" -
erase artifacts in
gabi-205/arti
(22GB) + disable "Keep latest artifacts" -
deal with storage used by
tpo/core/debian/tor
(39GB)
-
erase artifacts in projects under
- @anarcat make a ticket to move artifacts to minio, possibly growing minio in the process (#41403)
-
emergency procedures
- vacuum pipelines
- donate backup drive to shared
- move to MinIO (#41403)