| ... | ... | @@ -696,14 +696,15 @@ it took: |
|
|
|
3. 20 minutes to restore the basic system (database, tickets are
|
|
|
|
visible at this point)
|
|
|
|
4. an hour to restore repositories
|
|
|
|
5. more than an hour to restore artifacts
|
|
|
|
5. another hour to restore artifacts
|
|
|
|
|
|
|
|
This gives a time to recovery of about 6 hours. Most of that time is
|
|
|
|
spent waiting for files to be copied, interspersed with a few manual
|
|
|
|
commands.
|
|
|
|
This gives a time to recovery of about 5 to 6 hours. Most of that time
|
|
|
|
is spent waiting for files to be copied, interspersed with a few
|
|
|
|
manual commands.
|
|
|
|
|
|
|
|
So here's the procedure that was followed to deploy a development
|
|
|
|
server, from backups, in [tpo/tpa/team#40820](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40820):
|
|
|
|
server, from backups, in [tpo/tpa/team#40820](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40820) (run everything as
|
|
|
|
root):
|
|
|
|
|
|
|
|
1. [install GitLab using Puppet](#installation): basically create a server large
|
|
|
|
enough for everything, apply the Puppet `role::gitlab`
|
| ... | ... | @@ -782,17 +783,56 @@ server, from backups, in [tpo/tpa/team#40820](https://gitlab.torproject.org/tpo/ |
|
|
|
gitlab-ctl stop puma
|
|
|
|
gitlab-ctl stop sidekiq
|
|
|
|
|
|
|
|
6. restore the secrets files (note: this wasn't actually tested, but
|
|
|
|
should work):
|
|
|
|
|
|
|
|
chown root:root /var/tmp/bacula-restores/etc/gitlab/*
|
|
|
|
mv /var/tmp/bacula-restores/etc/gitlab/{gitlab-secrets.json,gitlab.rb} /etc/gitlab/
|
|
|
|
|
|
|
|
Note that if you're setting up a development environment, you do
|
|
|
|
*not* want to perform that step, which means that CI/CD variables
|
|
|
|
and 2FA tokens will be lost, which means people will need to reset
|
|
|
|
those and login with their recovery codes. This is what you want
|
|
|
|
for a dev server, because you do not want a possible dev server
|
|
|
|
compromise to escalate to the production server, or the dev server
|
|
|
|
to have access to the prod deployments.
|
|
|
|
|
|
|
|
6. restore the files:
|
|
|
|
|
|
|
|
sudo gitlab-backup restore
|
|
|
|
gitlab-backup restore
|
|
|
|
|
|
|
|
This last step will ask you to confirm the restore, because it
|
|
|
|
actually destroys the existing install. It will also ask you to
|
|
|
|
confirm the rewrite of the `authorized_keys` file, which you want
|
|
|
|
to accept (unless you specifically want to restore that from
|
|
|
|
backup as well).
|
|
|
|
|
|
|
|
7. restart the services and check everything:
|
|
|
|
|
|
|
|
The last step will ask you to confirm the restore, because it actually
|
|
|
|
destroys the existing install. It will also ask you to confirm the
|
|
|
|
rewrite of the `authorized_keys` file, which you want to accept
|
|
|
|
(unless you specifically want to restore that from backup as well).
|
|
|
|
gitlab-ctl reconfigure
|
|
|
|
gitlab-ctl restart
|
|
|
|
gitlab-rake gitlab:check SANITIZE=true
|
|
|
|
gitlab-rake gitlab:doctor:secrets
|
|
|
|
gitlab-rake gitlab:lfs:check
|
|
|
|
gitlab-rake gitlab:uploads:check
|
|
|
|
gitlab-rake gitlab:artifacts:check
|
|
|
|
|
|
|
|
Note: in the simulation, GitLab was started like this instead,
|
|
|
|
which just worked as well:
|
|
|
|
|
|
|
|
gitlab-ctl start puma
|
|
|
|
gitlab-ctl start sidekiq
|
|
|
|
|
|
|
|
We did try the "verification" tasks above, but many of them
|
|
|
|
failed, especially in the `gitlab:doctor:secrets` job, possibly
|
|
|
|
because we didn't restore the secrets (deliberately).
|
|
|
|
|
|
|
|
At this point, basic functionality like logging-in and issues should
|
|
|
|
be working again, but not wikis.
|
|
|
|
be working again, but not wikis (because they are not restored
|
|
|
|
yet). Note that it's **normal** to see a 502 error message ("Whoops,
|
|
|
|
GitLab is taking too much time to respond.") when GitLab restarts: it
|
|
|
|
takes a *long* time to start (think minutes)... You can follow its
|
|
|
|
progress in `/var/log/gitlab/gitlab-rails/*.log`.
|
|
|
|
|
|
|
|
That procedure only restores a *part* of the system, namely what is
|
|
|
|
covered by the nightly backup job. To restore the rest (at the time of
|
| ... | ... | @@ -833,6 +873,8 @@ Then the files need to be given and moved as well, notice the |
|
|
|
mv /var/opt/gitlab/gitlab-rails/shared/artifacts/ /var/opt/gitlab/gitlab-rails/shared/artifacts.orig
|
|
|
|
mv /var/tmp/bacula-restores/srv/gitlab-shared/artifacts /var/opt/gitlab/gitlab-rails/shared/artifacts/
|
|
|
|
|
|
|
|
Restoring the artifacts took another hour of copying.
|
|
|
|
|
|
|
|
And that's it! Note that this procedure may vary if the subset of
|
|
|
|
files backed up by the GitLab backup job changes.
|
|
|
|
|
| ... | ... | |
