rethink gitlab backup strategy

added Icebox label

marked this issue as related to #40478 (closed)

marked this issue as related to gitlab#20 (moved)

marked this issue as related to #40517 (closed)

marked this issue as related to #40479

mentioned in issue #40479

mentioned in issue #40517 (closed)

mentioned in issue #40516 (closed)

marked this issue as related to #40615 (closed)

mentioned in issue #40615 (closed)

in #40615 (closed) we ran out of space again, but this time only in the "artifacts" filesystem. we've somewhat recovered and hopefully implemented some expiration policies that will keep us running for a bit (but that remains to be seen).

Not directly related to backups of course, but I should point out that backups still double space on the server, and therefore take up valuable space that we could otherwise dedicate to our users instead:

Filesystem                                    Size  Used Avail Use% Mounted on
/dev/mapper/vg_gitlab--02-var--opt            197G   67G  122G  36% /var/opt
/dev/mapper/vg_gitlab--02_hdd-gitlab--shared  275G  198G   63G  76% /srv/gitlab-shared
/dev/mapper/vg_gitlab--02_hdd-gitlab--backup  295G   92G  204G  32% /srv/gitlab-backup

in the above, /srv/gitlab-shared is only artifacts, which are not backed up on /srv/gitlab-backup. it's also unclear if we'd be able to actually restore this server from backups in an emergency, because of the possible inconsistencies between artifacts and backups...

i should also mention that we added another 30GB to that gitlab-shared partition, which brings us pretty close to not having enough space to rebuild a new filesystem on the side of the existing disk:

root@gitlab-02:~# pvs
  PV         VG               Fmt  Attr PSize     PFree  
  /dev/sdc   vg_gitlab-02     lvm2 a--   <200.00g      0 
  /dev/sdd   vg_gitlab-02_hdd lvm2 a--  <1000.00g 420.00g

if we look at the Used column above, that adds up to 357G, so it's still *doable, but we only have ~60G loose, and if we have different partitions, that would need to be split up too. So not a trivial switch.

It generally seems like artifacts storage size is unbounded and only expected to grow. It might be that we can only fix this problem by addressing the "large storage" problem (#40478 (closed)) as well.

Finally, I should mention that I've so far assume that we would reduce the above physical volume allocation (/dev/sdd) to free up 420G but there's probably a simpler solution for this: just create a new disk from the Ganeti node and assign it to the instance, creating a clean filesystem on it. In fact, we could just create a completely new VM (running bullseye, why not!) with a normal LVM root filesystem and GitLab's /opt and /srv on ZFS (on SSD and HDD vdevs, respectively).

So maybe we're being silly by keeping space on that PV after all, and we should just not block on this for normal maintenance operations.

added Gitlab label

marked this issue as related to #40744 (closed)

we stumbled upon this problem again (#40744 (closed)) and this time it was the repositories storage that exploded. somehow, in /srv/gitlab-backup/repositories there was 280GB of repositories even though the original is still less than 100GB (!?). This is possibly related with the incremental backup mechanism introduced in 14.10. the fix was to exclude repositories from the gitlab rake job, as they are already backed up by bacula anyways.

more broadly, taking a step back here, i think that filesystem snapshots would allow us to have consistency between the different components, but that backup script is mostly useful because it gives us a plain SQL copy of the postgresql database. everything else is just files, and while those can be inconsistent somehow, that shouldn't be that much of a deal in a disaster recovery situation. a corrupt PostgreSQL database, however, is another story, although I bet it could recover from a half-assed bacula backup somewhat correctly.

a guess, however, is not as good as a SQL dump, so i'm keeping that backup script for now, even though it still duplicates a lot of data.

marked this issue as related to #40820 (closed)

marked this issue as related to #41402 (closed)

assigned to @anarcat

in #41402 (closed), I've noticed the backup partition is virtually empty, so this has become more urgent: it seems this was rethought indeed, but accidentally, which is not "thinking" at all.

need to investigate.

added Next label and removed Icebox label

added Doing label and removed Next label

this is what partitions look like right now:

root@gitlab-02:~# df -h / /srv/gitlab-*
Filesystem                                    Size  Used Avail Use% Mounted on
/dev/sda1                                      30G   19G  8.9G  69% /
/dev/mapper/vg_gitlab--02_hdd-gitlab--backup  295G   50G  246G  17% /srv/gitlab-backup
/dev/mapper/vg_gitlab--02_hdd-gitlab--shared  275G  208G   54G  80% /srv/gitlab-shared

the actual content is:

root@gitlab-02:/srv/gitlab-backup# du -sch * | sort -h
4.0K	backup_information.yml
4.0K	ci_secure_files.tar.gz
4.0K	registry.tar.gz
4.0K	terraform_state.tar.gz
3.5M	builds.tar.gz
353M	lfs.tar.gz
1.1G	1700030933_2023_11_15_16.5.1_gitlab_backup.tar
1.1G	1700228908_2023_11_17_16.5.2_gitlab_backup.tar
1.1G	db
3.1G	uploads.tar.gz
5.1G	packages.tar.gz
38G	pages.tar.gz
50G	total

mysteriously, the .tar files only contain the database:

root@gitlab-02:/srv/gitlab-backup# ls -alhS
total 49G
-rw------- 1 git  git   38G Nov 22 00:39 pages.tar.gz
-rw------- 1 git  git  5.1G Nov 22 00:43 packages.tar.gz
-rw------- 1 git  git  3.1G Nov 22 00:07 uploads.tar.gz
-rw------- 1 git  git  1.1G Nov 17 13:51 1700228908_2023_11_17_16.5.2_gitlab_backup.tar
-rw------- 1 git  git  1.1G Nov 15 06:51 1700030933_2023_11_15_16.5.1_gitlab_backup.tar
-rw------- 1 git  git  353M Nov 22 00:39 lfs.tar.gz
-rw------- 1 git  git  3.5M Nov 22 00:07 builds.tar.gz
drwx------ 3 git  git  4.0K Nov 22 00:43 .
drwxr-xr-x 6 root root 4.0K Jun 28  2022 ..
drwx------ 2 git  git  4.0K Nov 22 00:02 db
-rw------- 1 git  git  3.5K Nov 22 00:43 ci_secure_files.tar.gz
-rw------- 1 git  git   466 Nov 22 00:43 backup_information.yml
-rw------- 1 git  git   153 Nov 22 00:39 registry.tar.gz
-rw------- 1 git  git   149 Nov 22 00:39 terraform_state.tar.gz

i have verified that those files in /srv/gitlab-backup are picked up by bacula, so that's good.

i guess that most of the space originally used in the backup partition there was due to artifacts and that has been out of backups for years at this point (since #40517 (closed)). at least disk usage has been stable at around 20GB for a year, rising recently probably because of #41402 (closed), although that would be strange precisely because we're not backing up artifacts in that partition.

i'm not sure to what to do about this ticket. maybe we could go through each one of those tar files above and revise whether or not they need to be individually backed up by gitlab or if the bacula backups are sufficient.

at the very least the database backups are required here, but that's covered by gitlab#20 (moved).

fundamentally, the question here is what needs to be backed up as per the upstream fine manual. upstream says:

PostgreSQL databases: covered by current backup script, could be externalized (gitlab#20 (moved))
Git repositories: not covered by current script, assumed bacula works, but actually this is problematic: gitaly needs to be stopped before backups can be performed consistently, so we actually need to either do that (!) or re-enable the script... there's actually a contradiction in documentation about this, i filed a ticket. it looks like the solution here is to use object storage to do server-side repository backups
Blobs: currently on disk, assumed to be safe to backup by bacula, but also covered by the rake task, could be moved to object storage and rely on that for backups
Container registry: same, currently in object storage without backups
Configuration files: backed up by bacula, assumed safe
Other data: mainly redis for the job queue and elastic search for the advanced search, we don't use the latter and the former we could probably live without

So, i think the immediate fix here is to switch Gitaly's backup system to object storage. this can be done even if we don't have backups in the object storage, i think, because the object storage itself is the backup.

marked this issue as related to #41415

mentioned in issue #41415

mentioned in issue #41416 (closed)

marked this issue as related to #41425 (closed)

mentioned in issue #41425 (closed)

mentioned in commit wiki-replica@65c07bc2

i opened #41425 (closed) to move backups to object storage, so that's covered. what remains here is the overlap between the rake backup task and our standard backups. we might be able to simply disable the backup job once postgresql backups are covered (#41426 (closed)).

so that is the next step: go through all components covered by the rake task and retire it when it is confirmed they are covered by another backup system. at this point, it is believed gitaly (#41425 (closed)) and postgresql (#41426 (closed)) need to be moved, but another check need to be done before the retirement.

i opened #41425 (closed) to move backups to object storage, so that's covered. what remains here is the overlap between the rake backup task and our standard backups. we might be able to simply disable the backup job once postgresql backups are covered (#41426 (closed)).

That kind of failed miserably. So we're back to square one on this one, hoping to get a response in https://gitlab.com/gitlab-org/gitlab/-/issues/432743 that will reassure us that bacula can just do the job for now.

mentioned in issue gitlab#20 (moved)

not going to do this before the new year, too big of a change, and not urgent.

added Backlog Backup labels and removed Doing label

mentioned in commit wiki-replica@1e695fd1

marked this issue as related to #41453 (closed)

mentioned in issue #41453 (closed)

changed the description

marked the checklist item PostgreSQL databases: moved to our normal backup system (#41426 (closed)) as incomplete

marked the checklist item PostgreSQL databases: moved to our normal backup system (#41426 (closed)) as completed

added Roadmap::Future label and removed Backlog label

added Backlog label and removed Roadmap::Future label

added Next label and removed Backlog label

added Backlog label and removed Next label

rethink gitlab backup strategy

Designs

Child items ...

Activity