... | ... | @@ -608,24 +608,11 @@ project ID up to 2000: |
|
|
|
|
|
## Connect to the PostgreSQL server
|
|
|
|
|
|
The GitLab Omnibus setup is special: it ships its own embedded
|
|
|
PostgreSQL server (!), which means the regular `sudo -u postgres psql`
|
|
|
command doesn't work.
|
|
|
We previously had instructions on how to connect to the GitLab Omnibus
|
|
|
PostgreSQL server, with the [upstream instructions](https://docs.gitlab.com/omnibus/maintenance/#starting-a-postgresql-superuser-psql-session) but this is now
|
|
|
deprecated. Normal PostgreSQL procedures should just work, like:
|
|
|
|
|
|
To get access to the PostgreSQL server, you need to [follow the
|
|
|
upstream instructions](https://docs.gitlab.com/omnibus/maintenance/#starting-a-postgresql-superuser-psql-session) which are, at the time of writing:
|
|
|
|
|
|
sudo gitlab-psql -d gitlabhq_production
|
|
|
|
|
|
This actually expands to the following command:
|
|
|
|
|
|
sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -p 5432 -h /var/opt/gitlab/postgresql -d gitlabhq_production -d gitlabhq_production
|
|
|
|
|
|
An emergency dump, therefore, could be taken with:
|
|
|
|
|
|
cd /tmp ; sudo -u gitlab-psql /opt/gitlab/embedded/bin/pg_dump -p 5432 -h /var/opt/gitlab/postgresql -d gitlabhq_production -d gitlabhq_production | pv -s 2G > /srv/gitlab-backup/db/db-$(date '+%Y-%m-%d').sql
|
|
|
|
|
|
Yes, that is silly. See also [issue 20][].
|
|
|
sudo -u postgres psql
|
|
|
|
|
|
## Pager playbook
|
|
|
|
... | ... | @@ -673,9 +660,8 @@ problems. |
|
|
|
|
|
### PostgreSQL debugging
|
|
|
|
|
|
The PostgreSQL configuration in GitLab is [particular][issue 20]. See the
|
|
|
[connect to the PostgreSQL server](#connect-to-the-postgresql-server) section above on how to connect
|
|
|
to it.
|
|
|
The PostgreSQL configuration in GitLab was [particular][issue 20], but you
|
|
|
should now follow our [normal PostgreSQL procedures](howto/postgresql).
|
|
|
|
|
|
### Disk full on GitLab server
|
|
|
|
... | ... | @@ -1102,7 +1088,7 @@ root): |
|
|
cd /var/tmp/bacula-restores/srv/gitlab-backup
|
|
|
mv *.tar.gz backup_information.yml db /srv/gitlab-backup/
|
|
|
cd /srv/gitlab-backup/
|
|
|
chown git:git *.tar.gz backup_information.yml db/ db/database.sql.gz
|
|
|
chown git:git *.tar.gz backup_information.yml
|
|
|
|
|
|
5. stop GitLab services that talk with the database (those might have
|
|
|
changed since the time of writing, [review upstream documentation
|
... | ... | @@ -1140,6 +1126,9 @@ root): |
|
|
to accept (unless you specifically want to restore that from
|
|
|
backup as well).
|
|
|
|
|
|
7. restore the database: note that this was never tested. Now you
|
|
|
should follow the [direct backup recovery procedure](howto/postgresql#direct-backup-recovery).
|
|
|
|
|
|
7. restart the services and check everything:
|
|
|
|
|
|
gitlab-ctl reconfigure
|
... | ... | @@ -1241,29 +1230,26 @@ classes which configure: |
|
|
[puppet/nginx](https://forge.puppet.com/puppet/nginx) module from the Forge
|
|
|
* `profile::gitlab::app`: the core of the configuration of gitlab
|
|
|
itself, uses the [puppet/gitlab](https://forge.puppet.com/puppet/gitlab) module from the Forge, with
|
|
|
Prometheus, Grafana, and Nginx support disabled, but Redis,
|
|
|
PostgreSQL, and other exporters enabled
|
|
|
Prometheus, Grafana, PostgreSQL and Nginx support disabled, but Redis,
|
|
|
and some exporters enabled
|
|
|
* `profile::gitlab::db`: the PostgreSQL server
|
|
|
* `profile::dovecot::private`: a simple IMAP server to receive mails
|
|
|
destined to GitLab
|
|
|
|
|
|
[issue 20]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/20
|
|
|
|
|
|
This installs the [GitLab Omnibus](https://docs.gitlab.com/omnibus/) distribution which duplicates a
|
|
|
lot of resources we would otherwise manage elsewhere in Puppet,
|
|
|
including (but possibly not limited to):
|
|
|
|
|
|
* [howto/prometheus](howto/prometheus) exporters (see [issue 40077](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40077) for example)
|
|
|
* [howto/postgresql](howto/postgresql)
|
|
|
* redis
|
|
|
|
|
|
This therefore leads to a "particular" situation regarding monitoring
|
|
|
and PostgreSQL backups, in particular. See [issue 20](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/20) for details.
|
|
|
lot of resources we would otherwise manage elsewhere in Puppet, mostly
|
|
|
Redis now.
|
|
|
|
|
|
The install takes a *long* time to complete. It's going to take a few
|
|
|
minutes to download, unpack, and configure GitLab. There's no precise
|
|
|
timing of this procedure yet, but assume each of those steps takes
|
|
|
about 2 to 5 minutes.
|
|
|
|
|
|
Note that you'll need special steps to configure the database during
|
|
|
the install, see below.
|
|
|
|
|
|
After the install, the administrator account details are stored in
|
|
|
`/etc/gitlab/initial_root_password`. After logging in, you most likely
|
|
|
want to [disable new signups](https://docs.gitlab.com/ee/user/admin_area/settings/sign_up_restrictions.html#disable-new-sign-ups) as recommended, or possibly restore
|
... | ... | @@ -1278,6 +1264,115 @@ here and there, but some effort was done during the 2022 hackweek |
|
|
(2022-06-28) to clean that up in Puppet at least. See
|
|
|
[tpo/tpa/gitlab#127](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/127) for some of that cleanup work.
|
|
|
|
|
|
### PostgreSQL standalone transition
|
|
|
|
|
|
In early 2024, PostgreSQL was migrated to its own setup, outside of
|
|
|
GitLab Omnibus, to ease maintenance and backups (see [issue
|
|
|
41426](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41426)). This is how that was performed.
|
|
|
|
|
|
First, there are two different documents upstream explaining how to do
|
|
|
this, one is [Using a non-packaged PostgreSQL database management
|
|
|
server](https://docs.gitlab.com/omnibus/settings/database.html#using-a-non-packaged-postgresql-database-management-server), and the other is [Configure GitLab using an external
|
|
|
PostgreSQL service](https://docs.gitlab.com/ee/administration/postgresql/external.html). This discrepancy was [filed as a bug](https://gitlab.com/gitlab-org/gitlab/-/issues/439789).
|
|
|
|
|
|
In any case, the `profile::gitlab::db` Puppet class is designed to
|
|
|
create a database capable of hosting the GitLab service. It only
|
|
|
creates the database and doesn't actually populate it, which is
|
|
|
something the Omnibus package normally does.
|
|
|
|
|
|
In our case, we backed up the production "omnibus" cluster and
|
|
|
restored to the managed cluster using the following procedure:
|
|
|
|
|
|
1. deploy the `profile::gitlab::db` profile, make sure the port
|
|
|
doesn't conflict with the omnibus database (e.g. use port `5433`
|
|
|
instead of `5432`), note that the postgres export will fail to
|
|
|
start, that's normal because it conflicts with the omnibus one:
|
|
|
|
|
|
pat
|
|
|
|
|
|
2. backup the GitLab database a first time, note down the time it
|
|
|
takes:
|
|
|
|
|
|
gitlab-backup create SKIP=tar,artifacts,repositories,builds,ci_secure_files,lfs,packages,registry,uploads,terraform_state,pages
|
|
|
|
|
|
3. restore said database into the new database created, noting down
|
|
|
the time it took to restore:
|
|
|
|
|
|
date ; time pv /srv/gitlab-backup/db/database.sql.gz | gunzip -c | sudo -u postgres psql -q gitlabhq_production; date
|
|
|
|
|
|
Note that the last step (`CREATE INDEX`) can take a few minutes on
|
|
|
its own, even after the `pv` progress bar completed.
|
|
|
|
|
|
4. drop the database and recreate it:
|
|
|
|
|
|
sudo -u postgres psql -c 'DROP DATABASE gitlabhq_production';
|
|
|
pat
|
|
|
|
|
|
5. post an announcement of a 15-60 minute downtime (adjust according
|
|
|
to the above test)
|
|
|
|
|
|
6. change the parameters in `gitlab.rb` to point to the other
|
|
|
database cluster (in our case, this is done in
|
|
|
`profile::gitlab::app`), make sure you also turn off `postgres`
|
|
|
and `postgres_exporter`, with:
|
|
|
|
|
|
postgresql['enable'] = false
|
|
|
postgresql_exporter['enable'] = false
|
|
|
gitlab_rails['db_adapter'] = "postgresql"
|
|
|
gitlab_rails['db_encoding'] = "utf8"
|
|
|
gitlab_rails['db_host'] = "127.0.0.1"
|
|
|
gitlab_rails['db_password'] = "[REDACTED]"
|
|
|
gitlab_rails['db_port'] = 5433
|
|
|
gitlab_rails['db_user'] = "gitlab"
|
|
|
|
|
|
... or, in Puppet:
|
|
|
|
|
|
class { 'gitlab':
|
|
|
postgresql => {
|
|
|
enable => false,
|
|
|
},
|
|
|
postgres_exporter => {
|
|
|
enable => false,
|
|
|
},
|
|
|
|
|
|
gitlab_rails => {
|
|
|
db_adapter => 'postgresql',
|
|
|
db_encoding => 'utf8',
|
|
|
db_host => '127.0.0.1',
|
|
|
db_user => 'gitlab',
|
|
|
db_port => '5433',
|
|
|
db_password => trocla('profile::gitlab::db', 'plain'),
|
|
|
|
|
|
# [...]
|
|
|
}
|
|
|
}
|
|
|
|
|
|
That configuration is detailed in [this guide](https://docs.gitlab.com/omnibus/settings/database.html#using-a-non-packaged-postgresql-database-management-server).
|
|
|
|
|
|
7. stop GitLab, but keep postgres running:
|
|
|
|
|
|
gitlab-ctl stop
|
|
|
gitlab-ctl start postgresql
|
|
|
|
|
|
8. do one final backup and restore:
|
|
|
|
|
|
gitlab-backup create SKIP=tar,artifacts,repositories,builds,ci_secure_files,lfs,packages,registry,uploads,terraform_state,pages
|
|
|
date ; time pv /srv/gitlab-backup/db/database.sql.gz | gunzip -c | sudo -u postgres psql -q gitlabhq_production; date
|
|
|
|
|
|
9. apply the above changes to `gitlab.rb` (or just run Puppet):
|
|
|
|
|
|
pat
|
|
|
gitlab-ctl reconfigure
|
|
|
gitlab-ctl start
|
|
|
|
|
|
10. make sure only one database is running, this should be empty:
|
|
|
|
|
|
gitlab-ctl status | grep postgresql
|
|
|
|
|
|
And this should show only the Debian package cluster:
|
|
|
|
|
|
ps axfu | grep postgresql
|
|
|
|
|
|
### GitLab CI installation
|
|
|
|
|
|
See [the CI documentation](service/ci) for documentation specific to GitLab CI.
|
... | ... | @@ -1441,9 +1536,9 @@ The web frontend is Nginx (which we incidentally also use in our |
|
|
[howto/cache](howto/cache) system) but GitLab wrote their own reverse proxy called
|
|
|
[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse/) which in turn talks to the underlying GitLab
|
|
|
Rails application, served by the [Unicorn](https://yhbt.net/unicorn/) application
|
|
|
server. The Rails app stores its data in a [howto/postgresql](howto/postgresql) database
|
|
|
(although not our own deployment, for now, [should be fixed](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/20)). GitLab also offloads
|
|
|
long-term background tasks to a tool called [sidekiq](https://github.com/mperham/sidekiq).
|
|
|
server. The Rails app stores its data in a [howto/postgresql](howto/postgresql)
|
|
|
database. GitLab also offloads long-term background tasks to a tool
|
|
|
called [sidekiq](https://github.com/mperham/sidekiq).
|
|
|
|
|
|
Those all server HTTP(S) requests but GitLab is of course also
|
|
|
accessible over SSH to push/pull git repositories. This is handled by
|
... | ... | @@ -1739,16 +1834,14 @@ to dump some components of the GitLab installation in the backup |
|
|
directory (`/srv/gitlab-backup`).
|
|
|
|
|
|
The backup system is deployed by Puppet and (*at the time of
|
|
|
writing*!) **skips** **repositories** and **artifacts**. It contains:
|
|
|
writing*!) **skips** the **database**, **repositories** and **artifacts**. It contains:
|
|
|
|
|
|
* GitLab CI build logs (`builds.tar.gz`)
|
|
|
* a compressed database dump (`db/database.sql.gz`)
|
|
|
* Git Large Files (Git LFS, `lfs.tar.gz`)
|
|
|
* packages (`packages.tar.gz`)
|
|
|
* GitLab pages (`pages.tar.gz`)
|
|
|
* some terraform thing (`terraform_state.tar.gz`)
|
|
|
* uploaded files (`uploads.tar.gz`)
|
|
|
* a PostgreSQL database dump (`db/database.sql.gz`)
|
|
|
|
|
|
The backup job is ran nightly. GitLab also creates a backup on
|
|
|
upgrade. Jobs are purged daily, and are assumed to be covered by
|
... | ... | |