Skip to content
Snippets Groups Projects
Verified Commit ed873978 authored by anarcat's avatar anarcat
Browse files

document a working gitlab restore procedure (!) (team#40820)

parent 51d7e1ac
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
Showing
with 191 additions and 21 deletions
howto/gitlab.md
+ 191
− 21
View file @ ed873978
......@@ -640,31 +640,201 @@ include secrets, which are backed up separately, for example with:
See `/etc/cron.d/gitlab-config-backup`, and the `gitlab::backup` and
`profile::gitlab::app` classes for the actual jobs that runs nightly.
### baremetal recovery
### Recovering this wiki from backups
Untested procedure extracted from the [upstream docs](https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-for-omnibus-gitlab-installations):
If you need to immediately restore the wiki from backups, you can head
to the backup server and restore the directory:
1. reinstall the same version you are restoring from
/var/opt/gitlab/git-data/repositories/@hashed/11/f8/11f8e31ccbdbb7d91589ecf40713d3a8a5d17a7ec0cebf641f975af50a1eba8d.git
2. restore the secrets backup:
tar -C /opt/gitlab/etc/ -x -v -z -f config_backup20200627T0200.tar.gz
The hash above is the SHA256 checksum of the [wiki-replica](https://gitlab.torproject.org/tpo/tpa/wiki-replica/)
project id (695):
2. configure and start gitlab:
sudo gitlab-ctl reconfigure
sudo gitlab-ctl start
3. restore the file:
sudo cp 11493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar /var/opt/gitlab/backups
sudo chown git.git /var/opt/gitlab/backups/11493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar
sudo gitlab-ctl stop unicorn
sudo gitlab-ctl stop puma
sudo gitlab-ctl stop sidekiq
# Verify
sudo gitlab-ctl status
sudo gitlab-backup restore BACKUP=11493107454_2018_04_25_10.6.4-ce
$ printf 695 | sha256sum
11f8e31ccbdbb7d91589ecf40713d3a8a5d17a7ec0cebf641f975af50a1eba8d -
On the backup server, that would be something like:
bconsole
restore
5
46
cd /var/opt/gitlab/git-data/repositories/@hashed/11/f8
mark 11f8e31ccbdbb7d91589ecf40713d3a8a5d17a7ec0cebf641f975af50a1eba8d.git
done
yes
The files will end up in `/var/tmp/bacula-restore` on
`gitlab-02`. Note that the number `46`, above, will vary according to
other servers backed up on the backup server, of course.
This should give you a copy of the git repository, which you can then
use, presumably, to read this procedure and restore the rest of
GitLab.
(Although then, how did you read *this* part of the procedure?
Anyways, I thought this could save your future self one day. You'll
thank me later.)
### Restoring from backups
The [upstream documentation](https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-for-omnibus-gitlab-installations) has a fairly good restore procedure,
but because our backup procedure is non-standard -- we exclude
repositories and artifacts, for example -- you should follow this
procedure instead.
Note that the procedure assumes some familiarity with the general
[backup and restore procedures](howto/backup), particularly how to restore a
bunch of files from the backup server (see the [restore files
section](howto/backup#restore-files).
This entire procedure will take many hours to complete. In our tests,
it took:
1. an hour or two to setup a VM
2. less than an hour to do a basic GitLab install
3. 20 minutes to restore the basic system (database, tickets are
visible at this point)
4. an hour to restore repositories
5. more than an hour to restore artifacts
This gives a time to recovery of about 6 hours. Most of that time is
spent waiting for files to be copied, interspersed with a few manual
commands.
So here's the procedure that was followed to deploy a development
server, from backups, in [tpo/tpa/team#40820](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40820):
1. [install GitLab using Puppet](#installation): basically create a server large
enough for everything, apply the Puppet `role::gitlab`
That includes creating new certificates and DNS records, if not
already present (those may be different if you are created a dev
server from backups, for example, which was the case for the the
above ticket).
Also note that you need to install the *same* GitLab version as
the one from the backup. If you are unsure of the GitLab version
that's in the backup (bad day uh?), try to restore the
`/var/opt/gitlab/gitlab-rails/VERSION` file from the backup server
first.
2. at this point, a blank GitLab installation should be
running. verify that you can reach the login page, possibly trying
to login with the root account, because a working GitLab
installation is a pre-requisite for the rest of the restore
procedure.
(it might be technically possible to restore the entire server
from scratch using only the backup server, but that procedure has
not been established or tested.)
3. on the backup server (currently `bacula-director-01`), restore the
latest GitLab backup job from the `/srv/gitlab-backup` and the
secrets from `/etc/gitlab`:
# bconsole
*restore
To select the JobIds, you have the following choices:
[...]
5: Select the most recent backup for a client
[...]
Select item: (1-13): 5
Defined Clients:
[...]
47: gitlab-02.torproject.org-fd
[...]
Select the Client (1-98): 47
Automatically selected FileSet: Standard Set
[...]
Building directory tree for JobId(s) 199535,199637,199738,199847,199951 ... ++++++++++++++++++++++++++++++++
596,949 files inserted into the tree.
[...]
cwd is: /
$ cd /etc
cwd is: /etc/
$ mark gitlab
84 files marked.
$ cd /srv
cwd is: /srv/
$ mark gitlab-backup
12 files marked.
$ done
This took about 20 minutes in a simulation done in June 2022,
including 5 minutes to load the file list.
4. move the files in place and fix ownership, possibly moving
pre-existing backups out of place, if the new server has been
running for more than 24 hours:
mkdir /srv/gitlab-backup.blank
mv /srv/gitlab-backup/* /srv/gitlab-backup.blank
cd /var/tmp/bacula-restores/srv/gitlab-backup
mv *.tar.gz backup_information.yml db /srv/gitlab-backup/
cd /srv/gitlab-backup/
chown git:git *.tar.gz backup_information.yml db/ db/database.sql.gz
5. stop GitLab services that talk with the database (those might have
changed since the time of writing, [review upstream documentation
just in case](https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-for-omnibus-gitlab-installations):
gitlab-ctl stop puma
gitlab-ctl stop sidekiq
6. restore the files:
sudo gitlab-backup restore
The last step will ask you to confirm the restore, because it actually
destroys the existing install. It will also ask you to confirm the
rewrite of the `authorized_keys` file, which you want to accept
(unless you specifically want to restore that from backup as well).
At this point, basic functionality like logging-in and issues should
be working again, but not wikis.
That procedure only restores a *part* of the system, namely what is
covered by the nightly backup job. To restore the rest (at the time of
writing: artifacts and repositories, which includes wikis!), you also
need to specifically restore those files from the backup server.
For example, this procedure will restore the repositories from the
backup server:
$ cd /var/opt/gitlab/git-data
cwd is: /var/opt/gitlab
$ mark repositories
113,766 files marked.
$ done
The files will then end up in
`/var/tmp/bacula-restores/var/opt/gitlab/git-data`. They will need to
be given to the right users and moved into place:
chown -R git:root /var/tmp/bacula-restores/var/opt/gitlab/git-data/repositories
mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.orig
mv /var/tmp/bacula-restores/var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories/
During the last simulation, restoring repositories took an hour.
Restoring artifacts is similar:
$ cd /srv/gitlab-shared
cwd is: /srv/gitlab-shared/
$ mark artifacts
434,788 files marked.
$ done
Then the files need to be given and moved as well, notice the
`git:git` instead of `git:root`:
chown -R git:git /var/tmp/bacula-restores/srv/gitlab-shared/artifacts
mv /var/opt/gitlab/gitlab-rails/shared/artifacts/ /var/opt/gitlab/gitlab-rails/shared/artifacts.orig
mv /var/tmp/bacula-restores/srv/gitlab-shared/artifacts /var/opt/gitlab/gitlab-rails/shared/artifacts/
And that's it! Note that this procedure may vary if the subset of
files backed up by the GitLab backup job changes.
# Reference
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment