document the kind of horrible gitlab SNAFU we just survived

howto/upgrades/bullseye.md

@@ -598,6 +598,84 @@ patched locally to follow an upstream fix that wasn't shipped in
 bullseye. Specifically, he [uploaded](https://ftp-master.debian.org/new/onionbalance_0.2.2-1~bpo11+1.html) `onionbalance
 0.2.2-1~bpo11+1` to `bullseye-backports`.
 
+### GitLab upgrade failure
+
+During the upgrade of `gitlab-02`, we ran into problems in step 6
+"Actual upgrade run".
+
+The GitLab omnibus package was unexpectedly upgraded, and the upgrade
+failed at the "unpack" stage:
+
+    Preparing to unpack .../244-gitlab-ce_15.0.0-ce.0_amd64.deb ...
+    gitlab preinstall:
+    gitlab preinstall: This node does not appear to be running a database
+    gitlab preinstall: Skipping version check, if you think this is an error exit now
+    gitlab preinstall:
+    gitlab preinstall: Checking for unmigrated data on legacy storage
+    gitlab preinstall:
+    gitlab preinstall: Upgrade failed. Could not check for unmigrated data on legacy storage.
+    gitlab preinstall:
+    gitlab preinstall: Waiting until database is ready before continuing...
+    Failed to connect to the database...
+    Error: FATAL:  Peer authentication failed for user "gitlab"
+    gitlab preinstall:
+    gitlab preinstall: If you want to skip this check, run the following command and try again:
+    gitlab preinstall:
+    gitlab preinstall:  sudo touch /etc/gitlab/skip-unmigrated-data-check
+    gitlab preinstall:
+    dpkg: error processing archive /tmp/apt-dpkg-install-ODItgL/244-gitlab-ce_15.0.0-ce.0_amd64.deb (--unpack):
+     new gitlab-ce package pre-installation script subprocess returned error exit status 1
+    Errors were encountered while processing:
+     /tmp/apt-dpkg-install-ODItgL/244-gitlab-ce_15.0.0-ce.0_amd64.deb
+
+Then, any attempt to [connect to the Omnibux PostgreSQL instance](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/gitlab/#connect-to-the-postgresql-server)
+yielded the error:
+    
+    psql: FATAL: Peer authentication failed for user "gitlab-psql"
+
+We attempted the following workarounds, with no effect:
+
+ * restore the Debian `/etc/postgresql/` directory, which was purged
+   in step 4: no effect
+ * fix unbound/DNS resolution (restarting unbound, `dpkg --configure
+   -a`, adding `1.1.1.1` or `trust-ad` to `resolv.conf`): no effect
+ * run "gitlab-ctl reconfigure": also aborted with a pgsql connection
+   failure
+
+Note that the Postgresql configuration files were eventually
+re-removed, alongside `/var/lib/postgresql`, as the production
+database is vendored by gitlab-omnibus, in
+`/var/opt/gitlab/postgresql/`.
+
+This is what eventually fixed the problem: `gitlab-ctl restart
+postgresql`. Witness:
+
+    root@gitlab-02:/var/opt/gitlab/postgresql# gitlab-ctl restart postgresql
+    ok: run: postgresql: (pid 17501) 0s
+    root@gitlab-02:/var/opt/gitlab/postgresql# gitlab-psql 
+    psql (12.10)
+    Type "help" for help.
+
+    gitlabhq_production=# ^D\q
+
+Then when we attempted to resume the package upgrade:
+
+    Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/gitlab-02.torproject.org.json.
+    This usually happens when your last run of `gitlab-ctl reconfigure` didn't complete successfully.
+    This file is used to check if any of the unsupported configurations are enabled,
+    and hence require a working reconfigure before upgrading.
+    Please run `sudo gitlab-ctl reconfigure` to fix it and try again.
+    dpkg: error processing archive /var/cache/apt/archives/gitlab-ce_15.0.0-ce.0_amd64.deb (--unpack):
+     new gitlab-ce package pre-installation script subprocess returned error exit status 1
+    Errors were encountered while processing:
+     /var/cache/apt/archives/gitlab-ce_15.0.0-ce.0_amd64.deb
+    needrestart is being skipped since dpkg has failed
+
+After running `gitlab-ctl reconfigure` and `apt upgrade` once more,
+the package was upgraded successfully and the procedure was resumed.
+
+Go figure.
+
 # Troubleshooting
 
 ## Upgrade failures