... | ... | @@ -598,6 +598,84 @@ patched locally to follow an upstream fix that wasn't shipped in |
|
|
bullseye. Specifically, he [uploaded](https://ftp-master.debian.org/new/onionbalance_0.2.2-1~bpo11+1.html) `onionbalance
|
|
|
0.2.2-1~bpo11+1` to `bullseye-backports`.
|
|
|
|
|
|
### GitLab upgrade failure
|
|
|
|
|
|
During the upgrade of `gitlab-02`, we ran into problems in step 6
|
|
|
"Actual upgrade run".
|
|
|
|
|
|
The GitLab omnibus package was unexpectedly upgraded, and the upgrade
|
|
|
failed at the "unpack" stage:
|
|
|
|
|
|
Preparing to unpack .../244-gitlab-ce_15.0.0-ce.0_amd64.deb ...
|
|
|
gitlab preinstall:
|
|
|
gitlab preinstall: This node does not appear to be running a database
|
|
|
gitlab preinstall: Skipping version check, if you think this is an error exit now
|
|
|
gitlab preinstall:
|
|
|
gitlab preinstall: Checking for unmigrated data on legacy storage
|
|
|
gitlab preinstall:
|
|
|
gitlab preinstall: Upgrade failed. Could not check for unmigrated data on legacy storage.
|
|
|
gitlab preinstall:
|
|
|
gitlab preinstall: Waiting until database is ready before continuing...
|
|
|
Failed to connect to the database...
|
|
|
Error: FATAL: Peer authentication failed for user "gitlab"
|
|
|
gitlab preinstall:
|
|
|
gitlab preinstall: If you want to skip this check, run the following command and try again:
|
|
|
gitlab preinstall:
|
|
|
gitlab preinstall: sudo touch /etc/gitlab/skip-unmigrated-data-check
|
|
|
gitlab preinstall:
|
|
|
dpkg: error processing archive /tmp/apt-dpkg-install-ODItgL/244-gitlab-ce_15.0.0-ce.0_amd64.deb (--unpack):
|
|
|
new gitlab-ce package pre-installation script subprocess returned error exit status 1
|
|
|
Errors were encountered while processing:
|
|
|
/tmp/apt-dpkg-install-ODItgL/244-gitlab-ce_15.0.0-ce.0_amd64.deb
|
|
|
|
|
|
Then, any attempt to [connect to the Omnibux PostgreSQL instance](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/gitlab/#connect-to-the-postgresql-server)
|
|
|
yielded the error:
|
|
|
|
|
|
psql: FATAL: Peer authentication failed for user "gitlab-psql"
|
|
|
|
|
|
We attempted the following workarounds, with no effect:
|
|
|
|
|
|
* restore the Debian `/etc/postgresql/` directory, which was purged
|
|
|
in step 4: no effect
|
|
|
* fix unbound/DNS resolution (restarting unbound, `dpkg --configure
|
|
|
-a`, adding `1.1.1.1` or `trust-ad` to `resolv.conf`): no effect
|
|
|
* run "gitlab-ctl reconfigure": also aborted with a pgsql connection
|
|
|
failure
|
|
|
|
|
|
Note that the Postgresql configuration files were eventually
|
|
|
re-removed, alongside `/var/lib/postgresql`, as the production
|
|
|
database is vendored by gitlab-omnibus, in
|
|
|
`/var/opt/gitlab/postgresql/`.
|
|
|
|
|
|
This is what eventually fixed the problem: `gitlab-ctl restart
|
|
|
postgresql`. Witness:
|
|
|
|
|
|
root@gitlab-02:/var/opt/gitlab/postgresql# gitlab-ctl restart postgresql
|
|
|
ok: run: postgresql: (pid 17501) 0s
|
|
|
root@gitlab-02:/var/opt/gitlab/postgresql# gitlab-psql
|
|
|
psql (12.10)
|
|
|
Type "help" for help.
|
|
|
|
|
|
gitlabhq_production=# ^D\q
|
|
|
|
|
|
Then when we attempted to resume the package upgrade:
|
|
|
|
|
|
Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/gitlab-02.torproject.org.json.
|
|
|
This usually happens when your last run of `gitlab-ctl reconfigure` didn't complete successfully.
|
|
|
This file is used to check if any of the unsupported configurations are enabled,
|
|
|
and hence require a working reconfigure before upgrading.
|
|
|
Please run `sudo gitlab-ctl reconfigure` to fix it and try again.
|
|
|
dpkg: error processing archive /var/cache/apt/archives/gitlab-ce_15.0.0-ce.0_amd64.deb (--unpack):
|
|
|
new gitlab-ce package pre-installation script subprocess returned error exit status 1
|
|
|
Errors were encountered while processing:
|
|
|
/var/cache/apt/archives/gitlab-ce_15.0.0-ce.0_amd64.deb
|
|
|
needrestart is being skipped since dpkg has failed
|
|
|
|
|
|
After running `gitlab-ctl reconfigure` and `apt upgrade` once more,
|
|
|
the package was upgraded successfully and the procedure was resumed.
|
|
|
|
|
|
Go figure.
|
|
|
|
|
|
# Troubleshooting
|
|
|
|
|
|
## Upgrade failures
|
... | ... | |