diff --git a/howto/upgrades/bullseye.md b/howto/upgrades/bullseye.md index 951be39fa3a5859dee98bdf0a32affa193b855e7..aef709a19aff7e18339cfaf83ea448a451486632 100644 --- a/howto/upgrades/bullseye.md +++ b/howto/upgrades/bullseye.md @@ -598,6 +598,84 @@ patched locally to follow an upstream fix that wasn't shipped in bullseye. Specifically, he [uploaded](https://ftp-master.debian.org/new/onionbalance_0.2.2-1~bpo11+1.html) `onionbalance 0.2.2-1~bpo11+1` to `bullseye-backports`. +### GitLab upgrade failure + +During the upgrade of `gitlab-02`, we ran into problems in step 6 +"Actual upgrade run". + +The GitLab omnibus package was unexpectedly upgraded, and the upgrade +failed at the "unpack" stage: + + Preparing to unpack .../244-gitlab-ce_15.0.0-ce.0_amd64.deb ... + gitlab preinstall: + gitlab preinstall: This node does not appear to be running a database + gitlab preinstall: Skipping version check, if you think this is an error exit now + gitlab preinstall: + gitlab preinstall: Checking for unmigrated data on legacy storage + gitlab preinstall: + gitlab preinstall: Upgrade failed. Could not check for unmigrated data on legacy storage. + gitlab preinstall: + gitlab preinstall: Waiting until database is ready before continuing... + Failed to connect to the database... + Error: FATAL: Peer authentication failed for user "gitlab" + gitlab preinstall: + gitlab preinstall: If you want to skip this check, run the following command and try again: + gitlab preinstall: + gitlab preinstall: sudo touch /etc/gitlab/skip-unmigrated-data-check + gitlab preinstall: + dpkg: error processing archive /tmp/apt-dpkg-install-ODItgL/244-gitlab-ce_15.0.0-ce.0_amd64.deb (--unpack): + new gitlab-ce package pre-installation script subprocess returned error exit status 1 + Errors were encountered while processing: + /tmp/apt-dpkg-install-ODItgL/244-gitlab-ce_15.0.0-ce.0_amd64.deb + +Then, any attempt to [connect to the Omnibux PostgreSQL instance](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/gitlab/#connect-to-the-postgresql-server) +yielded the error: + + psql: FATAL: Peer authentication failed for user "gitlab-psql" + +We attempted the following workarounds, with no effect: + + * restore the Debian `/etc/postgresql/` directory, which was purged + in step 4: no effect + * fix unbound/DNS resolution (restarting unbound, `dpkg --configure + -a`, adding `1.1.1.1` or `trust-ad` to `resolv.conf`): no effect + * run "gitlab-ctl reconfigure": also aborted with a pgsql connection + failure + +Note that the Postgresql configuration files were eventually +re-removed, alongside `/var/lib/postgresql`, as the production +database is vendored by gitlab-omnibus, in +`/var/opt/gitlab/postgresql/`. + +This is what eventually fixed the problem: `gitlab-ctl restart +postgresql`. Witness: + + root@gitlab-02:/var/opt/gitlab/postgresql# gitlab-ctl restart postgresql + ok: run: postgresql: (pid 17501) 0s + root@gitlab-02:/var/opt/gitlab/postgresql# gitlab-psql + psql (12.10) + Type "help" for help. + + gitlabhq_production=# ^D\q + +Then when we attempted to resume the package upgrade: + + Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/gitlab-02.torproject.org.json. + This usually happens when your last run of `gitlab-ctl reconfigure` didn't complete successfully. + This file is used to check if any of the unsupported configurations are enabled, + and hence require a working reconfigure before upgrading. + Please run `sudo gitlab-ctl reconfigure` to fix it and try again. + dpkg: error processing archive /var/cache/apt/archives/gitlab-ce_15.0.0-ce.0_amd64.deb (--unpack): + new gitlab-ce package pre-installation script subprocess returned error exit status 1 + Errors were encountered while processing: + /var/cache/apt/archives/gitlab-ce_15.0.0-ce.0_amd64.deb + needrestart is being skipped since dpkg has failed + +After running `gitlab-ctl reconfigure` and `apt upgrade` once more, +the package was upgraded successfully and the procedure was resumed. + +Go figure. + # Troubleshooting ## Upgrade failures