Skip to content
Snippets Groups Projects
Verified Commit a77baeca authored by anarcat's avatar anarcat
Browse files

import and expand bullseye upgrade procedures from anarcat

parent 5110ef8b
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
howto/upgrades/bullseye.mdwn
+ 227
− 7
View file @ a77baeca
......@@ -13,16 +13,236 @@ and upgrade progress of the fleet.
[[_TOC_]]
# TODO
# Procedure
This is missing the entire procedure, which needs to be copied and
updated from the buster procedure and anarcat's.
This procedure is designed to be applied, in batch, on multiple
servers. Do NOT follow this procedure unless you are familiar with the
command line and the Debian upgrade process. It has been crafted by
and for experienced system administrators that have dozens if not
hundreds of servers to upgrade.
Notable changes to do, especially on our infra:
In particular, it runs almost completely unattended: configuration
changes are not prompted during the upgrade, and just not applied at
all, which *will* break services in many cases. We use a
[clean-conflicts](https://gitlab.com/anarcat/koumbit-scripts/-/blob/master/vps/clean_conflicts) script to do this all in one shot to shorten the
upgrade process (without it, configuration file changes stop the
upgrade at more or less random times). Then those changes get applied
after a reboot. And yes, that's even more dangerous.
* [merged /usr will be the default in bullseye + 1](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=978636#178) so our
(bullseye) installers should be tweaked to remove the
`--no-merged-usr` everywhere
IMPORTANT: if you are doing this procedure over SSH (I had the
privilege of having a console), you may want to [upgrade SSH first](https://www.debian.org/releases/bullseye/amd64/release-notes/ch-information.en.html#ssh-not-available)
as it has a longer downtime period, especially if you are on a flaky
connection.
1. Preparation:
: reset to the default locale
export LC_ALL=C.UTF-8 &&
sudo apt install ttyrec screen debconf-utils apt-show-versions deborphan apt-forktracer &&
sudo ttyrec -e screen /var/log/upgrade-bullseye.ttyrec
2. Backups and checks:
(
umask 0077 &&
tar cfz /var/backups/pre-bullseye-backup.tgz /etc /var/lib/dpkg /var/lib/apt/extended_states /var/cache/debconf $( [ -e /var/lib/aptitude/pkgstates ] && echo /var/lib/aptitude/pkgstates ) &&
dpkg --get-selections "*" > /var/backups/dpkg-selections-pre-bullseye.txt &&
debconf-get-selections > /var/backups/debconf-selections-pre-bullseye.txt
) &&
apt-mark showhold &&
dpkg --audit &&
: look for dkms packages and make sure they are relevant, if not, purge. &&
dpkg -l '*dkms' || true &&
: look for leftover config files &&
find /etc -name '*.dpkg-*' -o -name '*.ucf-*' -o -name '*.merge-error' &&
: run backups &&
/home/anarcat/bin/backup-$(hostname) &&
printf "End of Step 2\a\n"
3. Perform any pending upgrade and clear out old pins:
puppet agent --disable "running major upgrade" &&
: Check for pinned, on hold, packages, and possibly disable &&
rm -f /etc/apt/preferences /etc/apt/preferences.d/* &&
rm -f /etc/apt/sources.list.d/backports.debian.org.list &&
rm -f /etc/apt/sources.list.d/backports.list &&
rm -f /etc/apt/sources.list.d/bullseye.list &&
rm -f /etc/apt/sources.list.d/buster-backports.list &&
rm -f /etc/apt/sources.list.d/experimental.list &&
rm -f /etc/apt/sources.list.d/incoming.list &&
rm -f /etc/apt/sources.list.d/proposed-updates.list &&
rm -f /etc/apt/sources.list.d/sid.list &&
rm -f /etc/apt/sources.list.d/testing.list &&
apt update && apt -y upgrade &&
: list kernel images and purge unused packages &&
dpkg -l 'linux-image-*' &&
: look for packages from backports, other suites or archives &&
: if possible, switch to official packages by disabling third-party repositories &&
apt-forktracer | sort &&
printf "End of Step 3\a\n"
4. Check free space, see [this guide to free up space][] and
download packages:
systemctl stop apt-daily.timer &&
sed -i 's#buster/updates#bullseye-security#' /etc/apt/sources.list $(ls /etc/apt/sources.list.d/*) &&
sed -i 's/buster/bullseye/g' /etc/apt/sources.list $(ls /etc/apt/sources.list.d/*) &&
apt update &&
( apt -o APT::Get::Trivial-Only=true dist-upgrade || true ) &&
df -h &&
apt -y -d full-upgrade &&
printf "End of Step 4\a\n"
[this guide to free up space]: http://www.debian.org/releases/bullseye/amd64/release-notes/ch-upgrading.en.html#sufficient-space
5. Actual upgrade run:
export DEBIAN_FRONTEND=noninteractive APT_LISTCHANGES_FRONTEND=none APT_LISTBUGS_FRONTEND=none &&
apt full-upgrade -y -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold' &&
printf "\a" &&
/home/anarcat/src/koumbit-scripts/vps/clean_conflicts &&
printf "End of Step 5\a\n"
6. Post-upgrade procedures:
apt-get update --allow-releaseinfo-change &&
puppet agent --enable &&
puppet agent -t --noop &&
(puppet agent -t || true) &&
: reinstall Python packages to follow Python upgrade &&
for package in rsendmail pubpaste ; do
cd ~/src/$package && pip3 install .
done &&
systemctl start apt-daily.timer &&
printf "End of Step 6\a\n" &&
shutdown -r +1 "rebooting to get rid of old kernel image..."
7. Post-upgrade checks:
apt purge $(dpkg -l | awk '/^rc/ { print $2 }') # purge removed packages
apt autoremove -y --purge
apt purge $(deborphan --guess-dummy)
while deborphan -n | grep -q . ; do apt purge $(deborphan -n); done
apt autoremove -y --purge
apt clean
# review and purge older kernel if the new one boots properly
dpkg -l 'linux-image*'
# review packages that are not in the new distribution
apt-forktracer | sort
printf "All procedures completed\a\n" &&
# Notable changes
Here is a list of notable changes from a system administration
perspective:
* new: [driverless scanning and printing](https://www.debian.org/releases/bullseye/amd64/release-notes/ch-whats-new.en.html#driverless-operation)
* persistent systemd journal, which might have some privacy issues
(`rm -rf /var/log/journal` to disable, see [journald.conf(5)](https://manpages.debian.org/bullseye/systemd/journald.conf.5.en.html))
* last release to support non-merged /usr
* security archive changed to `deb https://deb.debian.org/debian-security bullseye-security main contrib` (covered by script above)
* [password hashes have changed](https://www.debian.org/releases/bullseye/amd64/release-notes/ch-information.en.html#pam-default-password) to [yescrypt](https://www.openwall.com/yescrypt/) (recognizable
from its `$y$` prefix), a major change from the previous default,
SHA-512 (recognizable from its `$6$` prefix), see also
[crypt(5)](https://manpages.debian.org/crypt.5) (in bullseye), [crypt(3)](https://manpages.debian.org/crypt.3) (in buster), and
`mkpasswd -m help` for a list of supported hashes on whatever
There is a more [exhaustive review of server-level changes from
mikas](https://michael-prokop.at/blog/2021/05/27/what-to-expect-from-debian-bullseye-newinbullseye/) as well. Notable:
* `kernel.unprivileged_userns_clone` enabled by default ([bug
898446](https://bugs.debian.org/898446))
* Prometheus [hardering](https://salsa.debian.org/go-team/packages/prometheus/-/commit/62017e7de3f9e5ae02bc842cabd3b2da69fb354f), initiated by anarcat
* Ganeti has a major upgrade! there were concerns about the upgrade
path, not sure how that turned out
## New packages
* [podman](https://tracker.debian.org/pkg/libpod), a Docker replacement
## Updated packages
This table summarizes package version changes I find interesting.
| Package | Buster | Bullseye | Notes |
|-------------|--------|----------|-------------------------------------------------------------------------------------------------------------------------|
| Docker | 18 | 20 | Docker made it for a second release |
| Emacs | 26 | 27 | JSON parsing for LSP? ~/.config/emacs/? harfbuzz?? oh my! [details](https://emacsredux.com/blog/2020/08/13/emacs-27-1/) |
| Ganeti | 2.16.0 | 3.0.1 | breaking upgrade? |
| OpenSSH | 7.9 | 8.4 | [FIDO/U2F, Include][8.2], [signatures][8.1], [quantum-resistant key exchange, key fingerprint as confirmation][8.0] |
| Postgresql | 11 | 13 | |
| Python | 3.7 | 3.9 | walrus operator, importlib.metadata, dict unions, zoneinfo |
| Puppet | 5.5 | 5.5 | Missed the Puppet 6 (and 7!) releases |
[8.0]: http://www.openssh.com/txt/release-8.0
[8.1]: http://www.openssh.com/txt/release-8.1
[8.2]: http://www.openssh.com/txt/release-8.2
Note that this table may not be up to date with the current bullseye
release. See the [official release notes](https://www.debian.org/releases/bullseye/amd64/release-notes/ch-whats-new.en.html#newdistro) for a more up to date
list.
## Removed packages
* Python 2 support is removed! hopefully most of my stuff is already
Python 3, but I did lose monkeysign and gameclock, as mentioned above
* Mailman 2 is consequently removed
See also the [noteworthy obsolete packages](https://www.debian.org/releases/bullseye/amd64/release-notes/ch-information.en.html#noteworthy-obsolete-packages) list.
## Deprecation notices
### usrmerge
It might be important to install `usrmerge` package as well,
considering that [merged /usr will be the default in bullseye +
1](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=978636#178). This, however, can be done *after* the upgrade but needs to be
done *before* the next major upgrade (Debian 12, bookworm).
The (bullseye) installers should be tweaked to remove the
`--no-merged-usr` everywhere, in any case. See [ticket 40367](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40367).
### slapd
OpenLDAP dropped support for all backends but [slapd-mdb](https://manpages.debian.org//bullseye/slapd/slapd-mdb.5.html). This
will require a migration on the LDAP server.
### apt-key
The `apt-key` command is deprecated and should not be used. Files
should be dropped in ` /etc/apt/trusted.gpg.d` or (preferably) into an
outside directory (we typically use `/usr/share/keyrings`). It is
believed that we already do the correct thing here.
# Issues
See also the official list of [known issues](https://www.debian.org/releases/bullseye/amd64/release-notes/ch-information.en.html).
## Pending
## Resolved
# Troubleshooting
## Upgrade failures
Instructions on errors during upgrades can be found in [the release
notes troubleshooting section](https://www.debian.org/releases/bullseye/amd64/release-notes/ch-upgrading.en.html#trouble).
## Reboot failures
If there's any trouble during reboots, you should use some recovery
system. The [release notes actually have good documentation on
that](https://www.debian.org/releases/bullseye/amd64/release-notes/ch-upgrading.en.html#recovery), on top of "use a live filesystem".
# References
* [Official guide](https://www.debian.org/releases/bullseye/amd64/release-notes/ch-upgrading.en.html)
* [Release notes](https://www.debian.org/releases/bullseye/amd64/release-notes/ch-whats-new.en.html)
* [Koumbit guide](https://wiki.koumbit.net/BullseyeUpgrade) (N/A yet)
* [DSA guide](https://dsa.debian.org/howto/upgrade-to-bullseye/) (WIP, reviewed)
* [anarcat guide](https://anarc.at/services/upgrades/bullseye/)
* [Solution proposal to automate this](https://wiki.debian.org/AutomatedUpgrade)
# Fleet-wide changes
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment