Skip to content

move gitaly backups to object storage

In #40518, I've evaluated the situation with GitLab backups, and concluded we need a better backup system for Gitaly backups, as:

they are "not covered by current script, assumed bacula works, but actually this is problematic: gitaly needs to be stopped before backups can be performed consistently, so we actually need to either do that (!) or re-enable the script... there's actually a contradiction in documentation about this, i filed a ticket. it looks like the solution here is to use object storage to do server-side repository backups." (source)

The problem with object storage backups is that object storage itself is not backed up (#41415), so we need to figure out how to handle this problem.

This is higher priority to backing up minio itself, if there is another solution, that's possibly a better way forward.

Checklist:

  • configure a bucket and accesses
  • configure gitlab (gitaly?) to access it
  • test a backup (works!)
  • test a nightly (works, but takes +200GiB!! oops.)
  • test INCREMENTAL backups (works!)
  • purge previous extra 200GiB backup
  • figure out expiration policies (upstream)
  • document the design of server-side backups (upstream)
  • document and test restore procedures
  • document install procedures: how gitaly backups were setup, mainly from this issue
  • disable bacula backups once we're sure everything is in order
Edited by anarcat
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information