gitlab backups running out of space (again)

i've had this situation when i came back from holiday where the main gitlab backups failed:

Subject: Cron <root@gitlab-02> /usr/bin/gitlab-rake gitlab:backup:create CRON=1
To: root@gitlab-02.torproject.org
Date: Mon, 11 Jan 2021 02:41:44 +0000

tar: -: Wrote only 2048 of 10240 bytes
tar: Error is not recoverable: exiting now
rake aborted!
Backup::Error: Backup failed
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:36:in `block in pack'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:27:in `chdir'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:27:in `pack'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:25:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:create
(See full trace by running task with --trace)
creating archive 1610331869_2021_01_11_13.7.3_gitlab_backup.tar failed

It's unclear to me why that happened, but I did several changes in the backup system to try to improve on the situation:

commit 046c523c7da7b69a585333193961e56bcc1a29e1
Author: Antoine Beaupré <anarcat@debian.org>
Date:   Mon Jan 11 11:38:44 2021 -0500

    deploy cron job with a cron resource
    
    The previous incantation didn't correctly have a trailing newline or
    "puppet warning" on top. This fixes both issues while being more
    idiomatic.

 modules/profile/manifests/gitlab/app.pp | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

commit ca34a4fe5545987c0edee91cb3be9b6582a58b73
Author: Antoine Beaupré <anarcat@debian.org>
Date:   Mon Jan 11 11:33:25 2021 -0500

    do not run the main and config gitlab backups simultaneously
    
    The previous configuration would run both at once, which meant that
    the config and builtin backups could and often would run
    together. This would create a situation where they would both be
    taking up too much space at once: the config would be backed up before
    the builtin would have time to purge older backups.
    
    This was particularly a problem when we were backing up the uploads
    and artifacts twice, but still seems like something we should fix.
    
    It also makes it more obvious where *all* the backups are: in that
    single script.
    
    It's too bad we're not using that part of the module anymore, maybe
    this is something we should send upstream to that Puppet module...

 modules/profile/manifests/gitlab/app.pp | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

commit e44af8bb6930e28089197ef4928a056fbb7b3230
Author: Antoine Beaupré <anarcat@debian.org>
Date:   Mon Jan 11 11:26:43 2021 -0500

    clarify GitLab rails config settings

 modules/profile/manifests/gitlab/app.pp | 6 ++++++
 1 file changed, 6 insertions(+)

commit e27e7666ae393effcc92fb8f903c36f04a9c3e01
Author: Antoine Beaupré <anarcat@debian.org>
Date:   Mon Jan 11 11:25:47 2021 -0500

    only prune our own (configuration) backups
    
    The builtin backup already has its own rotation policy: don't try to
    mess with that and instead just cleanup after ourselves.
    
    If the builtin has problems cleaning up, we should deal with that
    separately: either disable the builtin cleanup or fix it.

 modules/profile/manifests/gitlab/app.pp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

commit 0cb4855e033dbf6fa755a302e66af551e892aab0
Author: Antoine Beaupré <anarcat@debian.org>
Date:   Mon Jan 11 11:19:29 2021 -0500

    avoid duplicating builtin backups
    
    The builtin backup job covers all the repositories/ but it also
    includes the following:
    
    db/
    db/database.sql.gz
    uploads.tar.gz
    builds.tar.gz
    artifacts.tar.gz
    pages.tar.gz
    lfs.tar.gz
    backup_information.yml
    
    In `/var/opt/gitlab/gitlab-rails/`, we have the following:
    
    root@gitlab-02:/srv/gitlab-backup# find  /var/opt/gitlab/gitlab-rails/ -type d -maxdepth 2 | sort
    /var/opt/gitlab/gitlab-rails/
    /var/opt/gitlab/gitlab-rails/etc
    /var/opt/gitlab/gitlab-rails/shared
    /var/opt/gitlab/gitlab-rails/shared/artifacts
    /var/opt/gitlab/gitlab-rails/shared/cache
    /var/opt/gitlab/gitlab-rails/shared/dependency_proxy
    /var/opt/gitlab/gitlab-rails/shared/external-diffs
    /var/opt/gitlab/gitlab-rails/shared/lfs-objects
    /var/opt/gitlab/gitlab-rails/shared/packages
    /var/opt/gitlab/gitlab-rails/shared/pages
    /var/opt/gitlab/gitlab-rails/shared/terraform_state
    /var/opt/gitlab/gitlab-rails/shared/tmp
    /var/opt/gitlab/gitlab-rails/tmp
    /var/opt/gitlab/gitlab-rails/upgrade-status
    /var/opt/gitlab/gitlab-rails/uploads
    /var/opt/gitlab/gitlab-rails/uploads/-
    /var/opt/gitlab/gitlab-rails/uploads/@hashed
    /var/opt/gitlab/gitlab-rails/uploads/ahf-admin
    /var/opt/gitlab/gitlab-rails/uploads/pili
    /var/opt/gitlab/gitlab-rails/uploads/tmp
    /var/opt/gitlab/gitlab-rails/uploads/tpo
    /var/opt/gitlab/gitlab-rails/working
    
    I assume the following are already backed up:
    
    /var/opt/gitlab/gitlab-rails/shared/artifacts
    /var/opt/gitlab/gitlab-rails/shared/lfs-objects
    /var/opt/gitlab/gitlab-rails/shared/packages
    /var/opt/gitlab/gitlab-rails/shared/pages
    /var/opt/gitlab/gitlab-rails/uploads/*

    Which leaves us with:
    
    /var/opt/gitlab/gitlab-rails/etc
    
    ... which this commit keeps. Those are empty directories:
    
    /var/opt/gitlab/gitlab-rails/shared/cache/archive
    /var/opt/gitlab/gitlab-rails/shared/dependency_proxy
    /var/opt/gitlab/gitlab-rails/shared/external-diffs
    /var/opt/gitlab/gitlab-rails/shared/terraform_state
    /var/opt/gitlab/gitlab-rails/working
    
    I assume they would be part of the builtin backup if they were
    important. The following seems to be a temporary directory:
    
    /var/opt/gitlab/gitlab-rails/shared/tmp
    
    And this seems relevant only for upgrades, and I would assume would be
    part of the builtin backup otherwise.
    
    /var/opt/gitlab/gitlab-rails/upgrade-status
    
    So I think it's safe for our "configuration" backup to only include
    /etc. This should further reduce the risk of filling up the backup
    drive.

 modules/profile/manifests/gitlab/app.pp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

commit b1fceee012832f8dd2c119eeebea669b1a65a953
Author: Antoine Beaupré <anarcat@debian.org>
Date:   Mon Jan 11 11:15:13 2021 -0500

    rewrite the gitlab config backup job
    
    It was getting too hairy and convoluted to just sit in this one-line
    cron job. Now it is much more readable, and can be expanded to fix
    some bugs we still seem to have with the backup system.
    
    In particular, I noticed that we backup all of gitlab-rails/ in there,
    which overlaps with the builtin backups (uploads/ and artifacts/,
    namely, are already backed up by the builtin).
    
    So before piling up more stuff in this oneliner, split it in a shell
    script.

 modules/profile/manifests/gitlab/app.pp | 21 +++++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)

i was hoping this would fix the problem. it didn't. this ticket is trying to figure out why.