move gitaly backups to object storage
In #40518, I've evaluated the situation with GitLab backups, and concluded we need a better backup system for Gitaly backups, as:
they are "not covered by current script, assumed bacula works, but actually this is problematic: gitaly needs to be stopped before backups can be performed consistently, so we actually need to either do that (!) or re-enable the script... there's actually a contradiction in documentation about this, i filed a ticket. it looks like the solution here is to use object storage to do server-side repository backups." (source)
The problem with object storage backups is that object storage itself is not backed up (#41415), so we need to figure out how to handle this problem.
This is higher priority to backing up minio itself, if there is another solution, that's possibly a better way forward.
Checklist:
-
configure a bucket and accesses -
configure gitlab (gitaly?) to access it -
test a backup (works!) -
test a nightly (works, but takes +200GiB!! oops.) -
test INCREMENTAL backups (works!) -
purge previous extra 200GiB backup -
figure out expiration policies (upstream) -
document the design of server-side backups (upstream) -
document and test restore procedures -
document install procedures: how gitaly backups were setup, mainly from this issue -
disable bacula backups once we're sure everything is in order