gitlab-02 running out of disk space in gitlab-shared

GitLab is running out of disk space again.

something happened in the last month that made it eat 100GB in about 3 weeks:

image https://grafana.torproject.org/d/zbCoGRjnz/disk-usage?orgId=1&var-class=All&var-instance=gitlab-02.torproject.org&from=now-90d&to=now

even more visible in the yearly:

image https://grafana.torproject.org/d/zbCoGRjnz/disk-usage?orgId=1&var-class=All&var-instance=gitlab-02.torproject.org&from=now-1y&to=now

our daily rate is 6.65GB which means we have slightly more than 48h before running out of disk completely (15GB remaining)

battle plan:

  • @lavamind figure out a short term fix
    • erase artifacts in projects under tpo/web/l10n + disable "Keep latest artifacts"
    • erase artifacts in gabi-205/arti (22GB) + disable "Keep latest artifacts"
    • deal with storage used by tpo/core/debian/tor (39GB)
  • @anarcat make a ticket to move artifacts to minio, possibly growing minio in the process (#41403)
  • emergency procedures
    • vacuum pipelines
    • donate backup drive to shared
    • move to MinIO (#41403)
Edited Nov 20, 2023 by anarcat
Assignee Loading
Time tracking Loading