gitlab-02 production_json.log exploded in size, filling up root partition

the /var/log/gitlab/gitlab-rails/production_json.log is 10GB on gitlab-02 and has almost completely filled up the root partition:

https://grafana.torproject.org/d/zbCoGRjnz/disk-usage?orgId=1&var-class=All&var-instance=gitlab-02.torproject.org&from=now-30d&to=now&refresh=auto

we clearly see the jump on the last day... i suspect log rotation failed because it was lacking the free disk space, and there is a .2 file but not a .1 file.

note that mail started bouncing or at least being deferred:

Date: Wed, 10 Apr 2024 12:28:38 +0000
From: Mail Delivery System <MAILER-DAEMON@gitlab-02.torproject.org>
To: Postmaster <postmaster@gitlab-02.torproject.org>
Subject: Postfix SMTP server: errors from unknown[REDACTED]

Transcript of session follows.

 Out: 220 gitlab-02.torproject.org ESMTP Postfix (Debian/GNU)
 In:  ehlo WIN-JOBGFU9RC1C
 Out: 250-gitlab-02.torproject.org
 Out: 250-PIPELINING
 Out: 250-SIZE 10240000
 Out: 250-ETRN
 Out: 250-STARTTLS
 Out: 250-ENHANCEDSTATUSCODES
 Out: 250-8BITMIME
 Out: 250-DSN
 Out: 250-SMTPUTF8
 Out: 250 CHUNKING
 In:  Rset
 Out: 250 2.0.0 Ok
 In:  Mail from:<REDACTED>
 Out: 452 4.3.1 Insufficient system storage
 In:  Quit
 Out: 221 2.0.0 Bye

For other details, see the local mail logfile