document the brave new separate postgresql world (team#41426)

howto/gitlab.md

@@ -608,24 +608,11 @@ project ID up to 2000:
 
 ## Connect to the PostgreSQL server
 
-The GitLab Omnibus setup is special: it ships its own embedded
-PostgreSQL server (!), which means the regular `sudo -u postgres psql`
-command doesn't work.
+We previously had instructions on how to connect to the GitLab Omnibus
+PostgreSQL server, with the [upstream instructions](https://docs.gitlab.com/omnibus/maintenance/#starting-a-postgresql-superuser-psql-session) but this is now
+deprecated. Normal PostgreSQL procedures should just work, like:
 
-To get access to the PostgreSQL server, you need to [follow the
-upstream instructions](https://docs.gitlab.com/omnibus/maintenance/#starting-a-postgresql-superuser-psql-session) which are, at the time of writing:
-
-    sudo gitlab-psql -d gitlabhq_production
-
-This actually expands to the following command:
-
-    sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -p 5432 -h /var/opt/gitlab/postgresql -d gitlabhq_production -d gitlabhq_production
-
-An emergency dump, therefore, could be taken with:
-
-    cd /tmp ; sudo -u gitlab-psql /opt/gitlab/embedded/bin/pg_dump -p 5432 -h /var/opt/gitlab/postgresql -d gitlabhq_production -d gitlabhq_production | pv -s 2G > /srv/gitlab-backup/db/db-$(date '+%Y-%m-%d').sql
-
-Yes, that is silly. See also [issue 20][].
+    sudo -u postgres psql
 
 ## Pager playbook
 
@@ -673,9 +660,8 @@ problems.
 
 ### PostgreSQL debugging
 
-The PostgreSQL configuration in GitLab is [particular][issue 20]. See the
-[connect to the PostgreSQL server](#connect-to-the-postgresql-server) section above on how to connect
-to it.
+The PostgreSQL configuration in GitLab was [particular][issue 20], but you
+should now follow our [normal PostgreSQL procedures](howto/postgresql).
 
 ### Disk full on GitLab server
 
@@ -1102,7 +1088,7 @@ root):
         cd /var/tmp/bacula-restores/srv/gitlab-backup
         mv *.tar.gz backup_information.yml  db /srv/gitlab-backup/
         cd /srv/gitlab-backup/
-        chown git:git *.tar.gz backup_information.yml db/ db/database.sql.gz 
+        chown git:git *.tar.gz backup_information.yml
 
  5. stop GitLab services that talk with the database (those might have
     changed since the time of writing, [review upstream documentation
@@ -1140,6 +1126,9 @@ root):
     to accept (unless you specifically want to restore that from
     backup as well).
 
+ 7. restore the database: note that this was never tested. Now you
+    should follow the [direct backup recovery procedure](howto/postgresql#direct-backup-recovery).
+
  7. restart the services and check everything:
  
         gitlab-ctl reconfigure
@@ -1241,29 +1230,26 @@ classes which configure:
    [puppet/nginx](https://forge.puppet.com/puppet/nginx) module from the Forge
  * `profile::gitlab::app`: the core of the configuration of gitlab
    itself, uses the [puppet/gitlab](https://forge.puppet.com/puppet/gitlab) module from the Forge, with
-   Prometheus, Grafana, and Nginx support disabled, but Redis,
-   PostgreSQL, and other exporters enabled
+   Prometheus, Grafana, PostgreSQL and Nginx support disabled, but Redis,
+   and some exporters enabled
+ * `profile::gitlab::db`: the PostgreSQL server
  * `profile::dovecot::private`: a simple IMAP server to receive mails
    destined to GitLab
 
 [issue 20]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/20
 
 This installs the [GitLab Omnibus](https://docs.gitlab.com/omnibus/) distribution which duplicates a
-lot of resources we would otherwise manage elsewhere in Puppet,
-including (but possibly not limited to):
-
- * [howto/prometheus](howto/prometheus) exporters (see [issue 40077](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40077) for example)
- * [howto/postgresql](howto/postgresql)
- * redis
-
-This therefore leads to a "particular" situation regarding monitoring
-and PostgreSQL backups, in particular. See [issue 20](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/20) for details.
+lot of resources we would otherwise manage elsewhere in Puppet, mostly
+Redis now.
 
 The install takes a *long* time to complete. It's going to take a few
 minutes to download, unpack, and configure GitLab. There's no precise
 timing of this procedure yet, but assume each of those steps takes
 about 2 to 5 minutes.
 
+Note that you'll need special steps to configure the database during
+the install, see below.
+
 After the install, the administrator account details are stored in
 `/etc/gitlab/initial_root_password`. After logging in, you most likely
 want to [disable new signups](https://docs.gitlab.com/ee/user/admin_area/settings/sign_up_restrictions.html#disable-new-sign-ups) as recommended, or possibly restore
@@ -1278,6 +1264,115 @@ here and there, but some effort was done during the 2022 hackweek
 (2022-06-28) to clean that up in Puppet at least. See
 [tpo/tpa/gitlab#127](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/127) for some of that cleanup work.
 
+### PostgreSQL standalone transition
+
+In early 2024, PostgreSQL was migrated to its own setup, outside of
+GitLab Omnibus, to ease maintenance and backups (see [issue
+41426](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41426)). This is how that was performed.
+
+First, there are two different documents upstream explaining how to do
+this, one is [Using a non-packaged PostgreSQL database management
+server](https://docs.gitlab.com/omnibus/settings/database.html#using-a-non-packaged-postgresql-database-management-server), and the other is [Configure GitLab using an external
+PostgreSQL service](https://docs.gitlab.com/ee/administration/postgresql/external.html). This discrepancy was [filed as a bug](https://gitlab.com/gitlab-org/gitlab/-/issues/439789). 
+
+In any case, the `profile::gitlab::db` Puppet class is designed to
+create a database capable of hosting the GitLab service. It only
+creates the database and doesn't actually populate it, which is
+something the Omnibus package normally does. 
+
+In our case, we backed up the production "omnibus" cluster and
+restored to the managed cluster using the following procedure:
+
+ 1. deploy the `profile::gitlab::db` profile, make sure the port
+    doesn't conflict with the omnibus database (e.g. use port `5433`
+    instead of `5432`), note that the postgres export will fail to
+    start, that's normal because it conflicts with the omnibus one:
+    
+        pat
+
+ 2. backup the GitLab database a first time, note down the time it
+    takes:
+    
+        gitlab-backup create SKIP=tar,artifacts,repositories,builds,ci_secure_files,lfs,packages,registry,uploads,terraform_state,pages
+
+ 3. restore said database into the new database created, noting down
+    the time it took to restore:
+ 
+        date ; time pv /srv/gitlab-backup/db/database.sql.gz | gunzip -c | sudo -u postgres psql -q gitlabhq_production; date
+
+    Note that the last step (`CREATE INDEX`) can take a few minutes on
+    its own, even after the `pv` progress bar completed.
+
+ 4. drop the database and recreate it:
+ 
+        sudo -u postgres psql -c 'DROP DATABASE gitlabhq_production';
+        pat
+
+ 5. post an announcement of a 15-60 minute downtime (adjust according
+    to the above test)
+
+ 6. change the parameters in `gitlab.rb` to point to the other
+    database cluster (in our case, this is done in
+    `profile::gitlab::app`), make sure you also turn off `postgres`
+    and `postgres_exporter`, with:
+    
+        postgresql['enable'] = false
+        postgresql_exporter['enable'] = false
+        gitlab_rails['db_adapter'] = "postgresql"
+        gitlab_rails['db_encoding'] = "utf8"
+        gitlab_rails['db_host'] = "127.0.0.1"
+        gitlab_rails['db_password'] = "[REDACTED]"
+        gitlab_rails['db_port'] = 5433
+        gitlab_rails['db_user'] = "gitlab"
+
+    ... or, in Puppet:
+    
+        class { 'gitlab':
+          postgresql               => {
+            enable => false,
+          },
+          postgres_exporter        => {
+            enable => false,
+          },
+      
+          gitlab_rails             => {
+            db_adapter                     => 'postgresql',
+            db_encoding                    => 'utf8',
+            db_host                        => '127.0.0.1',
+            db_user                        => 'gitlab',
+            db_port                        => '5433',
+            db_password                    => trocla('profile::gitlab::db', 'plain'),
+      
+            # [...]
+          }
+        }
+
+    That configuration is detailed in [this guide](https://docs.gitlab.com/omnibus/settings/database.html#using-a-non-packaged-postgresql-database-management-server).
+
+ 7. stop GitLab, but keep postgres running:
+ 
+        gitlab-ctl stop
+        gitlab-ctl start postgresql
+
+ 8. do one final backup and restore:
+
+        gitlab-backup create SKIP=tar,artifacts,repositories,builds,ci_secure_files,lfs,packages,registry,uploads,terraform_state,pages
+        date ; time pv /srv/gitlab-backup/db/database.sql.gz | gunzip -c | sudo -u postgres psql -q gitlabhq_production; date
+
+ 9. apply the above changes to `gitlab.rb` (or just run Puppet):
+ 
+        pat
+        gitlab-ctl reconfigure
+        gitlab-ctl start
+
+ 10. make sure only one database is running, this should be empty:
+ 
+         gitlab-ctl status | grep postgresql
+        
+     And this should show only the Debian package cluster:
+     
+         ps axfu | grep postgresql
+
 ### GitLab CI installation
 
 See [the CI documentation](service/ci) for documentation specific to GitLab CI.
@@ -1441,9 +1536,9 @@ The web frontend is Nginx (which we incidentally also use in our
 [howto/cache](howto/cache) system) but GitLab wrote their own reverse proxy called
 [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse/) which in turn talks to the underlying GitLab
 Rails application, served by the [Unicorn](https://yhbt.net/unicorn/) application
-server. The Rails app stores its data in a [howto/postgresql](howto/postgresql) database
-(although not our own deployment, for now, [should be fixed](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/20)). GitLab also offloads
-long-term background tasks to a tool called [sidekiq](https://github.com/mperham/sidekiq).
+server. The Rails app stores its data in a [howto/postgresql](howto/postgresql)
+database. GitLab also offloads long-term background tasks to a tool
+called [sidekiq](https://github.com/mperham/sidekiq).
 
 Those all server HTTP(S) requests but GitLab is of course also
 accessible over SSH to push/pull git repositories. This is handled by
@@ -1739,16 +1834,14 @@ to dump some components of the GitLab installation in the backup
 directory (`/srv/gitlab-backup`).
 
 The backup system is deployed by Puppet and (*at the time of
-writing*!) **skips** **repositories** and **artifacts**. It contains:
+writing*!) **skips** the **database**, **repositories** and **artifacts**. It contains:
 
  * GitLab CI build logs (`builds.tar.gz`)
- * a compressed database dump (`db/database.sql.gz`)
  * Git Large Files (Git LFS, `lfs.tar.gz`)
  * packages (`packages.tar.gz`)
  * GitLab pages (`pages.tar.gz`)
  * some terraform thing (`terraform_state.tar.gz`)
  * uploaded files (`uploads.tar.gz`)
- * a PostgreSQL database dump (`db/database.sql.gz`)
 
 The backup job is ran nightly.  GitLab also creates a backup on
 upgrade. Jobs are purged daily, and are assumed to be covered by