gitlab-02 production_json.log exploded in size, filling up root partition
the /var/log/gitlab/gitlab-rails/production_json.log
is 10GB on gitlab-02 and has almost completely filled up the root partition:
we clearly see the jump on the last day... i suspect log rotation failed because it was lacking the free disk space, and there is a .2
file but not a .1
file.
note that mail started bouncing or at least being deferred:
Date: Wed, 10 Apr 2024 12:28:38 +0000
From: Mail Delivery System <MAILER-DAEMON@gitlab-02.torproject.org>
To: Postmaster <postmaster@gitlab-02.torproject.org>
Subject: Postfix SMTP server: errors from unknown[REDACTED]
Transcript of session follows.
Out: 220 gitlab-02.torproject.org ESMTP Postfix (Debian/GNU)
In: ehlo WIN-JOBGFU9RC1C
Out: 250-gitlab-02.torproject.org
Out: 250-PIPELINING
Out: 250-SIZE 10240000
Out: 250-ETRN
Out: 250-STARTTLS
Out: 250-ENHANCEDSTATUSCODES
Out: 250-8BITMIME
Out: 250-DSN
Out: 250-SMTPUTF8
Out: 250 CHUNKING
In: Rset
Out: 250 2.0.0 Ok
In: Mail from:<REDACTED>
Out: 452 4.3.1 Insufficient system storage
In: Quit
Out: 221 2.0.0 Bye
For other details, see the local mail logfile
Edited by anarcat