gitlab backups running out of space (again)
i've had this situation when i came back from holiday where the main gitlab backups failed:
Subject: Cron <root@gitlab-02> /usr/bin/gitlab-rake gitlab:backup:create CRON=1 To: firstname.lastname@example.org Date: Mon, 11 Jan 2021 02:41:44 +0000 tar: -: Wrote only 2048 of 10240 bytes tar: Error is not recoverable: exiting now rake aborted! Backup::Error: Backup failed /opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:36:in `block in pack' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:27:in `chdir' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:27:in `pack' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:25:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in `load' /opt/gitlab/embedded/bin/bundle:23:in `<main>' Tasks: TOP => gitlab:backup:create (See full trace by running task with --trace) creating archive 1610331869_2021_01_11_13.7.3_gitlab_backup.tar failed
It's unclear to me why that happened, but I did several changes in the backup system to try to improve on the situation:
commit 046c523c7da7b69a585333193961e56bcc1a29e1 Author: Antoine Beaupré <email@example.com> Date: Mon Jan 11 11:38:44 2021 -0500 deploy cron job with a cron resource The previous incantation didn't correctly have a trailing newline or "puppet warning" on top. This fixes both issues while being more idiomatic. modules/profile/manifests/gitlab/app.pp | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) commit ca34a4fe5545987c0edee91cb3be9b6582a58b73 Author: Antoine Beaupré <firstname.lastname@example.org> Date: Mon Jan 11 11:33:25 2021 -0500 do not run the main and config gitlab backups simultaneously The previous configuration would run both at once, which meant that the config and builtin backups could and often would run together. This would create a situation where they would both be taking up too much space at once: the config would be backed up before the builtin would have time to purge older backups. This was particularly a problem when we were backing up the uploads and artifacts twice, but still seems like something we should fix. It also makes it more obvious where *all* the backups are: in that single script. It's too bad we're not using that part of the module anymore, maybe this is something we should send upstream to that Puppet module... modules/profile/manifests/gitlab/app.pp | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) commit e44af8bb6930e28089197ef4928a056fbb7b3230 Author: Antoine Beaupré <email@example.com> Date: Mon Jan 11 11:26:43 2021 -0500 clarify GitLab rails config settings modules/profile/manifests/gitlab/app.pp | 6 ++++++ 1 file changed, 6 insertions(+) commit e27e7666ae393effcc92fb8f903c36f04a9c3e01 Author: Antoine Beaupré <firstname.lastname@example.org> Date: Mon Jan 11 11:25:47 2021 -0500 only prune our own (configuration) backups The builtin backup already has its own rotation policy: don't try to mess with that and instead just cleanup after ourselves. If the builtin has problems cleaning up, we should deal with that separately: either disable the builtin cleanup or fix it. modules/profile/manifests/gitlab/app.pp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) commit 0cb4855e033dbf6fa755a302e66af551e892aab0 Author: Antoine Beaupré <email@example.com> Date: Mon Jan 11 11:19:29 2021 -0500 avoid duplicating builtin backups The builtin backup job covers all the repositories/ but it also includes the following: db/ db/database.sql.gz uploads.tar.gz builds.tar.gz artifacts.tar.gz pages.tar.gz lfs.tar.gz backup_information.yml In `/var/opt/gitlab/gitlab-rails/`, we have the following: root@gitlab-02:/srv/gitlab-backup# find /var/opt/gitlab/gitlab-rails/ -type d -maxdepth 2 | sort /var/opt/gitlab/gitlab-rails/ /var/opt/gitlab/gitlab-rails/etc /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared/artifacts /var/opt/gitlab/gitlab-rails/shared/cache /var/opt/gitlab/gitlab-rails/shared/dependency_proxy /var/opt/gitlab/gitlab-rails/shared/external-diffs /var/opt/gitlab/gitlab-rails/shared/lfs-objects /var/opt/gitlab/gitlab-rails/shared/packages /var/opt/gitlab/gitlab-rails/shared/pages /var/opt/gitlab/gitlab-rails/shared/terraform_state /var/opt/gitlab/gitlab-rails/shared/tmp /var/opt/gitlab/gitlab-rails/tmp /var/opt/gitlab/gitlab-rails/upgrade-status /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads/- /var/opt/gitlab/gitlab-rails/uploads/@hashed /var/opt/gitlab/gitlab-rails/uploads/ahf-admin /var/opt/gitlab/gitlab-rails/uploads/pili /var/opt/gitlab/gitlab-rails/uploads/tmp /var/opt/gitlab/gitlab-rails/uploads/tpo /var/opt/gitlab/gitlab-rails/working I assume the following are already backed up: /var/opt/gitlab/gitlab-rails/shared/artifacts /var/opt/gitlab/gitlab-rails/shared/lfs-objects /var/opt/gitlab/gitlab-rails/shared/packages /var/opt/gitlab/gitlab-rails/shared/pages /var/opt/gitlab/gitlab-rails/uploads/* Which leaves us with: /var/opt/gitlab/gitlab-rails/etc ... which this commit keeps. Those are empty directories: /var/opt/gitlab/gitlab-rails/shared/cache/archive /var/opt/gitlab/gitlab-rails/shared/dependency_proxy /var/opt/gitlab/gitlab-rails/shared/external-diffs /var/opt/gitlab/gitlab-rails/shared/terraform_state /var/opt/gitlab/gitlab-rails/working I assume they would be part of the builtin backup if they were important. The following seems to be a temporary directory: /var/opt/gitlab/gitlab-rails/shared/tmp And this seems relevant only for upgrades, and I would assume would be part of the builtin backup otherwise. /var/opt/gitlab/gitlab-rails/upgrade-status So I think it's safe for our "configuration" backup to only include /etc. This should further reduce the risk of filling up the backup drive. modules/profile/manifests/gitlab/app.pp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) commit b1fceee012832f8dd2c119eeebea669b1a65a953 Author: Antoine Beaupré <firstname.lastname@example.org> Date: Mon Jan 11 11:15:13 2021 -0500 rewrite the gitlab config backup job It was getting too hairy and convoluted to just sit in this one-line cron job. Now it is much more readable, and can be expanded to fix some bugs we still seem to have with the backup system. In particular, I noticed that we backup all of gitlab-rails/ in there, which overlaps with the builtin backups (uploads/ and artifacts/, namely, are already backed up by the builtin). So before piling up more stuff in this oneliner, split it in a shell script. modules/profile/manifests/gitlab/app.pp | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-)
i was hoping this would fix the problem. it didn't. this ticket is trying to figure out why.