document the kind of horrible gitlab SNAFU we just survived

bullseye. Specifically, he [uploaded](https://ftp-master.debian.org/new/onionbalance_0.2.2-1~bpo11+1.html) `onionbalance

0.2.2-1~bpo11+1` to `bullseye-backports`.

### GitLab upgrade failure

During the upgrade of `gitlab-02`, we ran into problems in step 6

"Actual upgrade run".

The GitLab omnibus package was unexpectedly upgraded, and the upgrade

failed at the "unpack" stage:

    Preparing to unpack .../244-gitlab-ce_15.0.0-ce.0_amd64.deb ...

    gitlab preinstall:

    gitlab preinstall: This node does not appear to be running a database

    gitlab preinstall: Skipping version check, if you think this is an error exit now

    gitlab preinstall:

    gitlab preinstall: Checking for unmigrated data on legacy storage

    gitlab preinstall:

    gitlab preinstall: Upgrade failed. Could not check for unmigrated data on legacy storage.

    gitlab preinstall:

    gitlab preinstall: Waiting until database is ready before continuing...

    Failed to connect to the database...

    Error: FATAL:  Peer authentication failed for user "gitlab"

    gitlab preinstall:

    gitlab preinstall: If you want to skip this check, run the following command and try again:

    gitlab preinstall:

    gitlab preinstall:  sudo touch /etc/gitlab/skip-unmigrated-data-check

    gitlab preinstall:

    dpkg: error processing archive /tmp/apt-dpkg-install-ODItgL/244-gitlab-ce_15.0.0-ce.0_amd64.deb (--unpack):

     new gitlab-ce package pre-installation script subprocess returned error exit status 1

    Errors were encountered while processing:

     /tmp/apt-dpkg-install-ODItgL/244-gitlab-ce_15.0.0-ce.0_amd64.deb

Then, any attempt to [connect to the Omnibux PostgreSQL instance](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/gitlab/#connect-to-the-postgresql-server)

yielded the error:

    psql: FATAL: Peer authentication failed for user "gitlab-psql"

We attempted the following workarounds, with no effect:

 * restore the Debian `/etc/postgresql/` directory, which was purged

   in step 4: no effect

 * fix unbound/DNS resolution (restarting unbound, `dpkg --configure

   -a`, adding `1.1.1.1` or `trust-ad` to `resolv.conf`): no effect

 * run "gitlab-ctl reconfigure": also aborted with a pgsql connection

   failure

Note that the Postgresql configuration files were eventually

re-removed, alongside `/var/lib/postgresql`, as the production

database is vendored by gitlab-omnibus, in

`/var/opt/gitlab/postgresql/`.

This is what eventually fixed the problem: `gitlab-ctl restart

postgresql`. Witness:

    root@gitlab-02:/var/opt/gitlab/postgresql# gitlab-ctl restart postgresql

    ok: run: postgresql: (pid 17501) 0s

    root@gitlab-02:/var/opt/gitlab/postgresql# gitlab-psql 

    psql (12.10)

    Type "help" for help.

    gitlabhq_production=# ^D\q

Then when we attempted to resume the package upgrade:

    Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/gitlab-02.torproject.org.json.

    This usually happens when your last run of `gitlab-ctl reconfigure` didn't complete successfully.

    This file is used to check if any of the unsupported configurations are enabled,

    and hence require a working reconfigure before upgrading.

    Please run `sudo gitlab-ctl reconfigure` to fix it and try again.

    dpkg: error processing archive /var/cache/apt/archives/gitlab-ce_15.0.0-ce.0_amd64.deb (--unpack):

     new gitlab-ce package pre-installation script subprocess returned error exit status 1

    Errors were encountered while processing:

     /var/cache/apt/archives/gitlab-ce_15.0.0-ce.0_amd64.deb

    needrestart is being skipped since dpkg has failed

After running `gitlab-ctl reconfigure` and `apt upgrade` once more,

the package was upgraded successfully and the procedure was resumed.

Go figure.

# Troubleshooting

## Upgrade failures