... | ... | @@ -640,31 +640,201 @@ include secrets, which are backed up separately, for example with: |
|
|
See `/etc/cron.d/gitlab-config-backup`, and the `gitlab::backup` and
|
|
|
`profile::gitlab::app` classes for the actual jobs that runs nightly.
|
|
|
|
|
|
### baremetal recovery
|
|
|
### Recovering this wiki from backups
|
|
|
|
|
|
Untested procedure extracted from the [upstream docs](https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-for-omnibus-gitlab-installations):
|
|
|
If you need to immediately restore the wiki from backups, you can head
|
|
|
to the backup server and restore the directory:
|
|
|
|
|
|
1. reinstall the same version you are restoring from
|
|
|
/var/opt/gitlab/git-data/repositories/@hashed/11/f8/11f8e31ccbdbb7d91589ecf40713d3a8a5d17a7ec0cebf641f975af50a1eba8d.git
|
|
|
|
|
|
2. restore the secrets backup:
|
|
|
|
|
|
tar -C /opt/gitlab/etc/ -x -v -z -f config_backup20200627T0200.tar.gz
|
|
|
The hash above is the SHA256 checksum of the [wiki-replica](https://gitlab.torproject.org/tpo/tpa/wiki-replica/)
|
|
|
project id (695):
|
|
|
|
|
|
2. configure and start gitlab:
|
|
|
|
|
|
sudo gitlab-ctl reconfigure
|
|
|
sudo gitlab-ctl start
|
|
|
|
|
|
3. restore the file:
|
|
|
|
|
|
sudo cp 11493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar /var/opt/gitlab/backups
|
|
|
sudo chown git.git /var/opt/gitlab/backups/11493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar
|
|
|
sudo gitlab-ctl stop unicorn
|
|
|
sudo gitlab-ctl stop puma
|
|
|
sudo gitlab-ctl stop sidekiq
|
|
|
# Verify
|
|
|
sudo gitlab-ctl status
|
|
|
sudo gitlab-backup restore BACKUP=11493107454_2018_04_25_10.6.4-ce
|
|
|
$ printf 695 | sha256sum
|
|
|
11f8e31ccbdbb7d91589ecf40713d3a8a5d17a7ec0cebf641f975af50a1eba8d -
|
|
|
|
|
|
On the backup server, that would be something like:
|
|
|
|
|
|
bconsole
|
|
|
restore
|
|
|
5
|
|
|
46
|
|
|
cd /var/opt/gitlab/git-data/repositories/@hashed/11/f8
|
|
|
mark 11f8e31ccbdbb7d91589ecf40713d3a8a5d17a7ec0cebf641f975af50a1eba8d.git
|
|
|
done
|
|
|
yes
|
|
|
|
|
|
The files will end up in `/var/tmp/bacula-restore` on
|
|
|
`gitlab-02`. Note that the number `46`, above, will vary according to
|
|
|
other servers backed up on the backup server, of course.
|
|
|
|
|
|
This should give you a copy of the git repository, which you can then
|
|
|
use, presumably, to read this procedure and restore the rest of
|
|
|
GitLab.
|
|
|
|
|
|
(Although then, how did you read *this* part of the procedure?
|
|
|
Anyways, I thought this could save your future self one day. You'll
|
|
|
thank me later.)
|
|
|
|
|
|
### Restoring from backups
|
|
|
|
|
|
The [upstream documentation](https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-for-omnibus-gitlab-installations) has a fairly good restore procedure,
|
|
|
but because our backup procedure is non-standard -- we exclude
|
|
|
repositories and artifacts, for example -- you should follow this
|
|
|
procedure instead.
|
|
|
|
|
|
Note that the procedure assumes some familiarity with the general
|
|
|
[backup and restore procedures](howto/backup), particularly how to restore a
|
|
|
bunch of files from the backup server (see the [restore files
|
|
|
section](howto/backup#restore-files).
|
|
|
|
|
|
This entire procedure will take many hours to complete. In our tests,
|
|
|
it took:
|
|
|
|
|
|
1. an hour or two to setup a VM
|
|
|
2. less than an hour to do a basic GitLab install
|
|
|
3. 20 minutes to restore the basic system (database, tickets are
|
|
|
visible at this point)
|
|
|
4. an hour to restore repositories
|
|
|
5. more than an hour to restore artifacts
|
|
|
|
|
|
This gives a time to recovery of about 6 hours. Most of that time is
|
|
|
spent waiting for files to be copied, interspersed with a few manual
|
|
|
commands.
|
|
|
|
|
|
So here's the procedure that was followed to deploy a development
|
|
|
server, from backups, in [tpo/tpa/team#40820](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40820):
|
|
|
|
|
|
1. [install GitLab using Puppet](#installation): basically create a server large
|
|
|
enough for everything, apply the Puppet `role::gitlab`
|
|
|
|
|
|
That includes creating new certificates and DNS records, if not
|
|
|
already present (those may be different if you are created a dev
|
|
|
server from backups, for example, which was the case for the the
|
|
|
above ticket).
|
|
|
|
|
|
Also note that you need to install the *same* GitLab version as
|
|
|
the one from the backup. If you are unsure of the GitLab version
|
|
|
that's in the backup (bad day uh?), try to restore the
|
|
|
`/var/opt/gitlab/gitlab-rails/VERSION` file from the backup server
|
|
|
first.
|
|
|
|
|
|
2. at this point, a blank GitLab installation should be
|
|
|
running. verify that you can reach the login page, possibly trying
|
|
|
to login with the root account, because a working GitLab
|
|
|
installation is a pre-requisite for the rest of the restore
|
|
|
procedure.
|
|
|
|
|
|
(it might be technically possible to restore the entire server
|
|
|
from scratch using only the backup server, but that procedure has
|
|
|
not been established or tested.)
|
|
|
|
|
|
3. on the backup server (currently `bacula-director-01`), restore the
|
|
|
latest GitLab backup job from the `/srv/gitlab-backup` and the
|
|
|
secrets from `/etc/gitlab`:
|
|
|
|
|
|
# bconsole
|
|
|
*restore
|
|
|
To select the JobIds, you have the following choices:
|
|
|
[...]
|
|
|
5: Select the most recent backup for a client
|
|
|
[...]
|
|
|
Select item: (1-13): 5
|
|
|
Defined Clients:
|
|
|
[...]
|
|
|
47: gitlab-02.torproject.org-fd
|
|
|
[...]
|
|
|
Select the Client (1-98): 47
|
|
|
Automatically selected FileSet: Standard Set
|
|
|
[...]
|
|
|
Building directory tree for JobId(s) 199535,199637,199738,199847,199951 ... ++++++++++++++++++++++++++++++++
|
|
|
596,949 files inserted into the tree.
|
|
|
[...]
|
|
|
cwd is: /
|
|
|
$ cd /etc
|
|
|
cwd is: /etc/
|
|
|
$ mark gitlab
|
|
|
84 files marked.
|
|
|
$ cd /srv
|
|
|
cwd is: /srv/
|
|
|
$ mark gitlab-backup
|
|
|
12 files marked.
|
|
|
$ done
|
|
|
|
|
|
This took about 20 minutes in a simulation done in June 2022,
|
|
|
including 5 minutes to load the file list.
|
|
|
|
|
|
4. move the files in place and fix ownership, possibly moving
|
|
|
pre-existing backups out of place, if the new server has been
|
|
|
running for more than 24 hours:
|
|
|
|
|
|
mkdir /srv/gitlab-backup.blank
|
|
|
mv /srv/gitlab-backup/* /srv/gitlab-backup.blank
|
|
|
cd /var/tmp/bacula-restores/srv/gitlab-backup
|
|
|
mv *.tar.gz backup_information.yml db /srv/gitlab-backup/
|
|
|
cd /srv/gitlab-backup/
|
|
|
chown git:git *.tar.gz backup_information.yml db/ db/database.sql.gz
|
|
|
|
|
|
5. stop GitLab services that talk with the database (those might have
|
|
|
changed since the time of writing, [review upstream documentation
|
|
|
just in case](https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-for-omnibus-gitlab-installations):
|
|
|
|
|
|
gitlab-ctl stop puma
|
|
|
gitlab-ctl stop sidekiq
|
|
|
|
|
|
6. restore the files:
|
|
|
|
|
|
sudo gitlab-backup restore
|
|
|
|
|
|
The last step will ask you to confirm the restore, because it actually
|
|
|
destroys the existing install. It will also ask you to confirm the
|
|
|
rewrite of the `authorized_keys` file, which you want to accept
|
|
|
(unless you specifically want to restore that from backup as well).
|
|
|
|
|
|
At this point, basic functionality like logging-in and issues should
|
|
|
be working again, but not wikis.
|
|
|
|
|
|
That procedure only restores a *part* of the system, namely what is
|
|
|
covered by the nightly backup job. To restore the rest (at the time of
|
|
|
writing: artifacts and repositories, which includes wikis!), you also
|
|
|
need to specifically restore those files from the backup server.
|
|
|
|
|
|
For example, this procedure will restore the repositories from the
|
|
|
backup server:
|
|
|
|
|
|
$ cd /var/opt/gitlab/git-data
|
|
|
cwd is: /var/opt/gitlab
|
|
|
$ mark repositories
|
|
|
113,766 files marked.
|
|
|
$ done
|
|
|
|
|
|
The files will then end up in
|
|
|
`/var/tmp/bacula-restores/var/opt/gitlab/git-data`. They will need to
|
|
|
be given to the right users and moved into place:
|
|
|
|
|
|
chown -R git:root /var/tmp/bacula-restores/var/opt/gitlab/git-data/repositories
|
|
|
mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.orig
|
|
|
mv /var/tmp/bacula-restores/var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories/
|
|
|
|
|
|
During the last simulation, restoring repositories took an hour.
|
|
|
|
|
|
Restoring artifacts is similar:
|
|
|
|
|
|
$ cd /srv/gitlab-shared
|
|
|
cwd is: /srv/gitlab-shared/
|
|
|
$ mark artifacts
|
|
|
434,788 files marked.
|
|
|
$ done
|
|
|
|
|
|
Then the files need to be given and moved as well, notice the
|
|
|
`git:git` instead of `git:root`:
|
|
|
|
|
|
chown -R git:git /var/tmp/bacula-restores/srv/gitlab-shared/artifacts
|
|
|
mv /var/opt/gitlab/gitlab-rails/shared/artifacts/ /var/opt/gitlab/gitlab-rails/shared/artifacts.orig
|
|
|
mv /var/tmp/bacula-restores/srv/gitlab-shared/artifacts /var/opt/gitlab/gitlab-rails/shared/artifacts/
|
|
|
|
|
|
And that's it! Note that this procedure may vary if the subset of
|
|
|
files backed up by the GitLab backup job changes.
|
|
|
|
|
|
# Reference
|
|
|
|
... | ... | |