gitlab backups running out of space (again)
i've had this situation when i came back from holiday where the main gitlab backups failed:
Subject: Cron <root@gitlab-02> /usr/bin/gitlab-rake gitlab:backup:create CRON=1
To: root@gitlab-02.torproject.org
Date: Mon, 11 Jan 2021 02:41:44 +0000
tar: -: Wrote only 2048 of 10240 bytes
tar: Error is not recoverable: exiting now
rake aborted!
Backup::Error: Backup failed
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:36:in `block in pack'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:27:in `chdir'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:27:in `pack'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:25:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:create
(See full trace by running task with --trace)
creating archive 1610331869_2021_01_11_13.7.3_gitlab_backup.tar failedIt's unclear to me why that happened, but I did several changes in the backup system to try to improve on the situation:
commit 046c523c7da7b69a585333193961e56bcc1a29e1
Author: Antoine Beaupré <anarcat@debian.org>
Date: Mon Jan 11 11:38:44 2021 -0500
deploy cron job with a cron resource
The previous incantation didn't correctly have a trailing newline or
"puppet warning" on top. This fixes both issues while being more
idiomatic.
modules/profile/manifests/gitlab/app.pp | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
commit ca34a4fe5545987c0edee91cb3be9b6582a58b73
Author: Antoine Beaupré <anarcat@debian.org>
Date: Mon Jan 11 11:33:25 2021 -0500
do not run the main and config gitlab backups simultaneously
The previous configuration would run both at once, which meant that
the config and builtin backups could and often would run
together. This would create a situation where they would both be
taking up too much space at once: the config would be backed up before
the builtin would have time to purge older backups.
This was particularly a problem when we were backing up the uploads
and artifacts twice, but still seems like something we should fix.
It also makes it more obvious where *all* the backups are: in that
single script.
It's too bad we're not using that part of the module anymore, maybe
this is something we should send upstream to that Puppet module...
modules/profile/manifests/gitlab/app.pp | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
commit e44af8bb6930e28089197ef4928a056fbb7b3230
Author: Antoine Beaupré <anarcat@debian.org>
Date: Mon Jan 11 11:26:43 2021 -0500
clarify GitLab rails config settings
modules/profile/manifests/gitlab/app.pp | 6 ++++++
1 file changed, 6 insertions(+)
commit e27e7666ae393effcc92fb8f903c36f04a9c3e01
Author: Antoine Beaupré <anarcat@debian.org>
Date: Mon Jan 11 11:25:47 2021 -0500
only prune our own (configuration) backups
The builtin backup already has its own rotation policy: don't try to
mess with that and instead just cleanup after ourselves.
If the builtin has problems cleaning up, we should deal with that
separately: either disable the builtin cleanup or fix it.
modules/profile/manifests/gitlab/app.pp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
commit 0cb4855e033dbf6fa755a302e66af551e892aab0
Author: Antoine Beaupré <anarcat@debian.org>
Date: Mon Jan 11 11:19:29 2021 -0500
avoid duplicating builtin backups
The builtin backup job covers all the repositories/ but it also
includes the following:
db/
db/database.sql.gz
uploads.tar.gz
builds.tar.gz
artifacts.tar.gz
pages.tar.gz
lfs.tar.gz
backup_information.yml
In `/var/opt/gitlab/gitlab-rails/`, we have the following:
root@gitlab-02:/srv/gitlab-backup# find /var/opt/gitlab/gitlab-rails/ -type d -maxdepth 2 | sort
/var/opt/gitlab/gitlab-rails/
/var/opt/gitlab/gitlab-rails/etc
/var/opt/gitlab/gitlab-rails/shared
/var/opt/gitlab/gitlab-rails/shared/artifacts
/var/opt/gitlab/gitlab-rails/shared/cache
/var/opt/gitlab/gitlab-rails/shared/dependency_proxy
/var/opt/gitlab/gitlab-rails/shared/external-diffs
/var/opt/gitlab/gitlab-rails/shared/lfs-objects
/var/opt/gitlab/gitlab-rails/shared/packages
/var/opt/gitlab/gitlab-rails/shared/pages
/var/opt/gitlab/gitlab-rails/shared/terraform_state
/var/opt/gitlab/gitlab-rails/shared/tmp
/var/opt/gitlab/gitlab-rails/tmp
/var/opt/gitlab/gitlab-rails/upgrade-status
/var/opt/gitlab/gitlab-rails/uploads
/var/opt/gitlab/gitlab-rails/uploads/-
/var/opt/gitlab/gitlab-rails/uploads/@hashed
/var/opt/gitlab/gitlab-rails/uploads/ahf-admin
/var/opt/gitlab/gitlab-rails/uploads/pili
/var/opt/gitlab/gitlab-rails/uploads/tmp
/var/opt/gitlab/gitlab-rails/uploads/tpo
/var/opt/gitlab/gitlab-rails/working
I assume the following are already backed up:
/var/opt/gitlab/gitlab-rails/shared/artifacts
/var/opt/gitlab/gitlab-rails/shared/lfs-objects
/var/opt/gitlab/gitlab-rails/shared/packages
/var/opt/gitlab/gitlab-rails/shared/pages
/var/opt/gitlab/gitlab-rails/uploads/*
Which leaves us with:
/var/opt/gitlab/gitlab-rails/etc
... which this commit keeps. Those are empty directories:
/var/opt/gitlab/gitlab-rails/shared/cache/archive
/var/opt/gitlab/gitlab-rails/shared/dependency_proxy
/var/opt/gitlab/gitlab-rails/shared/external-diffs
/var/opt/gitlab/gitlab-rails/shared/terraform_state
/var/opt/gitlab/gitlab-rails/working
I assume they would be part of the builtin backup if they were
important. The following seems to be a temporary directory:
/var/opt/gitlab/gitlab-rails/shared/tmp
And this seems relevant only for upgrades, and I would assume would be
part of the builtin backup otherwise.
/var/opt/gitlab/gitlab-rails/upgrade-status
So I think it's safe for our "configuration" backup to only include
/etc. This should further reduce the risk of filling up the backup
drive.
modules/profile/manifests/gitlab/app.pp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
commit b1fceee012832f8dd2c119eeebea669b1a65a953
Author: Antoine Beaupré <anarcat@debian.org>
Date: Mon Jan 11 11:15:13 2021 -0500
rewrite the gitlab config backup job
It was getting too hairy and convoluted to just sit in this one-line
cron job. Now it is much more readable, and can be expanded to fix
some bugs we still seem to have with the backup system.
In particular, I noticed that we backup all of gitlab-rails/ in there,
which overlaps with the builtin backups (uploads/ and artifacts/,
namely, are already backed up by the builtin).
So before piling up more stuff in this oneliner, split it in a shell
script.
modules/profile/manifests/gitlab/app.pp | 21 +++++++++++++++------
1 file changed, 15 insertions(+), 6 deletions(-)
i was hoping this would fix the problem. it didn't. this ticket is trying to figure out why.