Debian 11 [bullseye](https://wiki.debian.org/DebianBullseye) was [released on August 14 2021](https://www.debian.org/News/2021/20210814)). Tor
started the upgrade to bullseye shortly after and hopes to complete
the process before the [buster](howto/upgrades/buster) EOL, [one year after the stable
release](https://www.debian.org/security/faq#lifespan), so normally around August 2022.

It is an aggressive timeline, which might be missed.

Starting from now on however, no new Debian 10 buster machine will be
created: all new machines will run Debian 11 bullseye.

This page aims at documenting the upgrade procedure, known problems
and upgrade progress of the fleet.

[[_TOC_]]

# Procedure

This procedure is designed to be applied, in batch, on multiple
servers. Do NOT follow this procedure unless you are familiar with the
command line and the Debian upgrade process. It has been crafted by
and for experienced system administrators that have dozens if not
hundreds of servers to upgrade.

In particular, it runs almost completely unattended: configuration
changes are not prompted during the upgrade, and just not applied at
all, which *will* break services in many cases. We use a
[clean-conflicts](https://gitlab.com/anarcat/koumbit-scripts/-/blob/master/vps/clean_conflicts) script to do this all in one shot to shorten the
upgrade process (without it, configuration file changes stop the
upgrade at more or less random times). Then those changes get applied
after a reboot. And yes, that's even more dangerous.

IMPORTANT: if you are doing this procedure over SSH (I had the
privilege of having a console), you may want to [upgrade SSH first](https://www.debian.org/releases/bullseye/amd64/release-notes/ch-information.en.html#ssh-not-available)
as it has a longer downtime period, especially if you are on a flaky
connection.

WARNING: this procedure has not been tested yet on TPA
infrastructure. It's a merge of the TPA buster procedure and anarcat's
bullseye procedure.

 1. Preparation:

        : reset to the default locale
        export LC_ALL=C.UTF-8 &&
        sudo apt install ttyrec screen debconf-utils apt-show-versions deborphan apt-forktracer &&
        sudo ttyrec -e screen /var/log/upgrade-bullseye.ttyrec

 2. Backups and checks:

        ( 
          umask 0077 &&
          tar cfz /var/backups/pre-bullseye-backup.tgz /etc /var/lib/dpkg /var/lib/apt/extended_states /var/cache/debconf $( [ -e /var/lib/aptitude/pkgstates ] && echo /var/lib/aptitude/pkgstates ) &&
          dpkg --get-selections "*" > /var/backups/dpkg-selections-pre-bullseye.txt &&
          debconf-get-selections > /var/backups/debconf-selections-pre-bullseye.txt
        ) &&
        apt-mark showhold &&
        dpkg --audit &&
        : look for dkms packages and make sure they are relevant, if not, purge. &&
        ( dpkg -l '*dkms' || true ) &&
        : look for leftover config files &&
        find /etc -name '*.dpkg-*' -o -name '*.ucf-*' -o -name '*.merge-error' &&
        : make sure backups are up to date in Nagios &&
        printf "End of Step 2\a\n"

 3. Enable module loading (for ferm) and test reboots:

        systemctl disable modules_disabled.timer &&
        puppet agent --disable "running major upgrade" &&
        shutdown -r +1 "rebooting with module loading enabled"

        export LC_ALL=C.UTF-8 &&
        sudo ttyrec -a -e screen /var/log/upgrade-buster.ttyrec

 4. Perform any pending upgrade and clear out old pins:

        : Check for pinned, on hold, packages, and possibly disable &&
        rm -f /etc/apt/preferences /etc/apt/preferences.d/* &&
        rm -f /etc/apt/sources.list.d/backports.debian.org.list &&
        rm -f /etc/apt/sources.list.d/backports.list &&
        rm -f /etc/apt/sources.list.d/bullseye.list &&
        rm -f /etc/apt/sources.list.d/buster-backports.list &&
        rm -f /etc/apt/sources.list.d/experimental.list &&
        rm -f /etc/apt/sources.list.d/incoming.list &&
        rm -f /etc/apt/sources.list.d/proposed-updates.list &&
        rm -f /etc/apt/sources.list.d/sid.list &&
        rm -f /etc/apt/sources.list.d/testing.list &&
        apt update && apt -y upgrade &&
        : list kernel images and purge unused packages &&
        dpkg -l 'linux-image-*' &&
        : look for packages from backports, other suites or archives &&
        : if possible, switch to official packages by disabling third-party repositories &&
        apt-forktracer | sort &&
        printf "End of Step 4\a\n"

 5. Check free space (see [this guide to free up space][]), disable
    auto-upgrades, and download packages:

        systemctl stop apt-daily.timer &&
        sed -i 's#buster/updates#bullseye-security#' /etc/apt/sources.list $(ls /etc/apt/sources.list.d/*) &&
        sed -i 's/buster/bullseye/g' /etc/apt/sources.list $(ls /etc/apt/sources.list.d/*) &&
        apt update &&
        ( apt -o APT::Get::Trivial-Only=true dist-upgrade || true ) &&
        df -h &&
        apt -y -d full-upgrade &&
        apt -y -d upgrade &&
        apt -y -d dist-upgrade &&
        printf "End of Step 5\a\n"

 6. Actual upgrade run:

        DEBIAN_FRONTEND=noninteractive APT_LISTCHANGES_FRONTEND=none APT_LISTBUGS_FRONTEND=none && apt full-upgrade -y -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold' &&
        printf "\a" &&
        /home/anarcat/src/koumbit-scripts/vps/clean_conflicts &&
        printf "End of Step 6\a\n"

 7. Post-upgrade procedures:

        apt-get update --allow-releaseinfo-change &&
        puppet agent --enable &&
        puppet agent -t --noop &&
        (puppet agent -t || true) &&
        (puppet agent -t || true) &&
        systemctl start apt-daily.timer &&
        printf "End of Step 7\a\n" &&
        shutdown -r +1 "rebooting to get rid of old kernel image..."

 8. Post-upgrade checks:

        export LC_ALL=C.UTF-8 &&
        sudo ttyrec -a -e screen /var/log/upgrade-buster.ttyrec

        apt purge $(dpkg -l | awk '/^rc/ { print $2 }') # purge removed packages
        apt autoremove -y --purge
        apt purge $(deborphan --guess-dummy)
        dsa-check-packages | tr -d ,
        while deborphan -n | grep -q . ; do apt purge $(deborphan -n); done
        apt autoremove -y --purge
        apt clean
        # review and purge older kernel if the new one boots properly
        dpkg -l 'linux-image*'
        # review packages that are not in the new distribution
        apt-forktracer | sort
        printf "End of Step 8\a\n"
        shutdown -r +1 "testing reboots one final time"

 9. Change the hostgroup of the host to bullseye in Nagios (in
    `tor-nagios/config/nagios-master.cfg` on `git@git-rw.tpo`)

[this guide to free up space]: http://www.debian.org/releases/buster/amd64/release-notes/ch-upgrading.en.html#sufficient-space

# Service-specific upgrade procedures

## RT and PostgreSQL upgrades

Both of those required special handling in [buster](howto/buster), probably going
to be similar here.

## Ganeti upgrades

Ganeti has a major version change, from 2.16.0-5 in Debian 10 "buster"
to 3.0.1-2 in Debian 11 "bullseye". There's a backport of 3.x in
"buster-backports", so we can actually perform the upgrade to 3.0
*prior* to the bullseye upgrade, which allows cluster to add
*bullseye* nodes without first having to upgrade all nodes in the
cluster to bullseye.

Update: it might be *mandatory* to *first* upgrade to
`bullseye-backports`, then purge the old packages, before upgrading to
bullseye, see [bug 993559][].

[bug 993559]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=993559

### Release notes

We upgrade from 2.15 to 3.0.1, the [3.0.1 NEWS file](https://github.com/ganeti/ganeti/blob/v3.0.1/NEWS) has the
relevant release notes (including 2.16 changes). Notable changes:

 * Automatic [postcopy migration](http://grivon.apgrid.org/quick-kvm-migration) handling for KVM guests (AKA:
   execution switches over the other node immediately, memory is
   copied *after*!)
 * Improved support for DRBD >= 8.4
 * VLAN-aware bridging: support for multiple VLANs in a single bridge
   (previously possible only with Open vSwitch)
 * Updated X.509 certificate signing algorithm (SHA-256)
 * Python 3 support (see the [Ganeti 3.0 design document](https://docs.ganeti.org/docs/ganeti/3.0/html/design-3.0.html))
 * [2.16 design documents](https://docs.ganeti.org/docs/ganeti/3.0/html/design-2.16.html), specifically:
   * support for "location tags" in the cluster rebalancer (hbal), see
     the design document [Improving location awareness of Ganeti](https://docs.ganeti.org/docs/ganeti/3.0/html/design-location.html_)
   * [N+1 redundancy for shared storage](https://docs.ganeti.org/docs/ganeti/3.0/html/design-shared-storage-redundancy.html) and [Redundancy for the
     plain disk template](https://docs.ganeti.org/docs/ganeti/3.0/html/design-plain-redundancy.html)
   * partial implementation of the [Ganeti OS installation redesign](https://docs.ganeti.org/docs/ganeti/3.0/html/design-os.html)

### Procedure

This procedure ~~should (ideally)~~ MUST (see [bug 993559][]) be
performed *before* the upgrade to bullseye, but can also be performed
after:

 1. on all nodes, upgrade Ganeti to backports (obviously only
    necessary on buster):
    
        apt install -y ganeti/buster-backports

    On the `gnt-chi` cluster, this was done by hand on `chi-node-04`,
    and then automatically on the other nodes, with [clustershell](https://tracker.debian.org/pkg/clustershell):

        clush -w chi-node-01.torproject.org,chi-node-02.torproject.org,chi-node-03.torproject.org

    Then type the `apt install` command to interactively perform the
    upgrade.
    
    An alternative would have been to use `cumin`:
    
        cumin 'C:roles::ganeti::chi' "apt install -y ganeti/buster-backports"

    There was a change on the `/etc/default/ganeti` file. The diff was
    checked with:

        cumin 'C:roles::ganeti::chi' 'diff -u /etc/default/ganeti.dpkg-dist /etc/default/ganeti'

    And applied with:
    
        cumin 'C:roles::ganeti::chi' 'mv /etc/default/ganeti.dpkg-dist /etc/default/ganeti'

 2. then, on the master server, run the cluster upgrade program:

        gnt-cluster upgrade --to 3.0

 3. on the master, renew the node certificates to switch from SHA-1 to
    SHA-256 in certificate signatures:
    
        gnt-cluster renew-crypto --new-cluster-certificate

 4. on the master, verify the cluster

        gnt-cluster verify

That's it!

Important caveats:

 * as long as the *entire* cluster is not upgraded, live migrations
   will fail with a strange error message, for example:
   
       Could not pre-migrate instance static-gitlab-shim.torproject.org: Failed to accept instance: Failed to start instance static-gitlab-shim.torproject.org: exited with exit code 1 (qemu-system-x86_64: -enable-kvm: unsupported machine type
       Use -machine help to list supported machines
       )

   note that you can generally migrate *to* the *newer* nodes, just
   not back to the old ones. but in practice, it's safer to just avoid
   doing live migrations between Ganeti releases, state doesn't carry
   well across major Qemu and KVM versions, and you might also find
   that the entire VM *does* migrate, but is hung. For example, this
   is the console after a failed migration:
   
        root@chi-node-01:~# gnt-instance console static-gitlab-shim.torproject.org
        Instance static-gitlab-shim.torproject.org is paused, unpausing

   ie. it's hung. the `qemu` process had to be killed to recover from
   that failed migration, on the node.
   
   a workaround for this issue is to use `failover` instead of
   `migrate`, which involves a shutdown. another workaround might be
   to upgrade qemu to backports.

 * `gnt-cluster verify` might warn about incompatible DRBD
   versions. if it's a minor version, it shouldn't matter and the
   warning can be ignored.

### upgrade discussion

On the other hand, the [upgrade instructions](https://sources.debian.org/src/ganeti/3.0.1-2/UPGRADE/) seem pretty confident
that the upgrade should just go smoothly. The [koumbit upgrade
procedures](https://wiki.koumbit.net/GanetiConfiguration#Mise_.2BAOA_jour) (to 2.15, ie. to Debian buster) mention the following
steps:

 1. install the new packages on all nodes
 2. `service ganeti restart` on all nodes
 3. `gnt-cluster upgrade --to 2.15` on the master

I suspect we might be able to just do this instead:

 1. install the new packages on all nodes
 2. `gnt-cluster upgrade --to 3.0` on the master

The [official upgrade guide](https://docs.ganeti.org/docs/ganeti/3.0/html/upgrade.html) does say that we need to restart
ganeti on all nodes, but I suspect that might be taken care of by the
Debian package so the restart might be redundant. Still, it won't
hurt: that doesn't restart the VMs.

It used to be that live migration between different versions of QEMU
would fail, but apparently that hasn't been a problem since 2018
(according to `#ganeti` on OFTC).

# Notable changes

Here is a list of notable changes from a system administration
perspective:

 * new: [driverless scanning and printing](https://www.debian.org/releases/bullseye/amd64/release-notes/ch-whats-new.en.html#driverless-operation)
 * persistent systemd journal, which might have some privacy issues
   (`rm -rf /var/log/journal` to disable, see [journald.conf(5)](https://manpages.debian.org/bullseye/systemd/journald.conf.5.en.html))
 * last release to support non-merged /usr
 * security archive changed to `deb
   https://deb.debian.org/debian-security bullseye-security main
   contrib` (covered by script above, also requires a change in
   unattended-upgrades)
 * [password hashes have changed](https://www.debian.org/releases/bullseye/amd64/release-notes/ch-information.en.html#pam-default-password) to [yescrypt](https://www.openwall.com/yescrypt/) (recognizable
   from its `$y$` prefix), a major change from the previous default,
   SHA-512 (recognizable from its `$6$` prefix), see also
   [crypt(5)](https://manpages.debian.org/crypt.5) (in bullseye), [crypt(3)](https://manpages.debian.org/crypt.3) (in buster), and
   `mkpasswd -m help` for a list of supported hashes on whatever

There is a more [exhaustive review of server-level changes from
mikas](https://michael-prokop.at/blog/2021/05/27/what-to-expect-from-debian-bullseye-newinbullseye/) as well. Notable:

 * `kernel.unprivileged_userns_clone` enabled by default ([bug
   898446](https://bugs.debian.org/898446))
 * Prometheus [hardering](https://salsa.debian.org/go-team/packages/prometheus/-/commit/62017e7de3f9e5ae02bc842cabd3b2da69fb354f), initiated by anarcat
 * Ganeti has a major upgrade! there were concerns about the upgrade
   path, not sure how that turned out

## New packages

 * [podman](https://tracker.debian.org/pkg/libpod), a Docker replacement

## Updated packages

This table summarizes package version changes I find interesting.

| Package     | Buster | Bullseye | Notes                                                                                                                   |
|-------------|--------|----------|-------------------------------------------------------------------------------------------------------------------------|
| Docker      | 18     | 20       | Docker made it for a second release                                                                                     |
| Emacs       | 26     | 27       | JSON parsing for LSP? ~/.config/emacs/? harfbuzz?? oh my! [details](https://emacsredux.com/blog/2020/08/13/emacs-27-1/) |
| Ganeti      | 2.16.0 | 3.0.1    | breaking upgrade?                                                                                                       |
| OpenSSH     | 7.9    | 8.4      | [FIDO/U2F, Include][8.2], [signatures][8.1], [quantum-resistant key exchange, key fingerprint as confirmation][8.0]     |
| Postgresql  | 11     | 13       |                                                                                                                         |
| Python      | 3.7    | 3.9      | walrus operator, importlib.metadata, dict unions, zoneinfo                                                              |
| Puppet      | 5.5    | 5.5      | Missed the Puppet 6 (and 7!) releases                                                                                   |

[8.0]: http://www.openssh.com/txt/release-8.0
[8.1]: http://www.openssh.com/txt/release-8.1
[8.2]: http://www.openssh.com/txt/release-8.2

Note that this table may not be up to date with the current bullseye
release. See the [official release notes](https://www.debian.org/releases/bullseye/amd64/release-notes/ch-whats-new.en.html#newdistro) for a more up to date
list.

## Removed packages

 * Python 2 support is removed! hopefully most of my stuff is already
   Python 3, but I did lose monkeysign and gameclock, as mentioned above
 * Mailman 2 is consequently removed

See also the [noteworthy obsolete packages](https://www.debian.org/releases/bullseye/amd64/release-notes/ch-information.en.html#noteworthy-obsolete-packages) list.

## Deprecation notices

### usrmerge

It might be important to install `usrmerge` package as well,
considering that [merged /usr will be the default in bullseye +
1](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=978636#178). This, however, can be done *after* the upgrade but needs to be
done *before* the next major upgrade (Debian 12, bookworm).

In other words, in the `bookworm` upgrade instructions, we should
prepare the machines by doing:

    apt install usrmerge

This can also be done at any time after the bullseye upgrade (and can
even be done in buster, for what that's worth).

### slapd

OpenLDAP dropped support for all backends but [slapd-mdb](https://manpages.debian.org//bullseye/slapd/slapd-mdb.5.html). This
will require a migration on the LDAP server.

### apt-key

The `apt-key` command is deprecated and should not be used. Files
should be dropped in ` /etc/apt/trusted.gpg.d` or (preferably) into an
outside directory (we typically use `/usr/share/keyrings`). It is
believed that we already do the correct thing here.

# Issues

See also the official list of [known issues](https://www.debian.org/releases/bullseye/amd64/release-notes/ch-information.en.html).

## Pending

 * Ganeti packages fail to upgrade (see above and [bug 993559][])

 * (from buster:) upgrading restarts openvswitch will mean all guests lose network

 * (from buster:) Puppet might try to downgrade the `sources.list`
   files to `stretch` or `n/a` for some reason, just re-run Puppet
   after fixing the `sources.list` files, it will eventually figure it
   out.

 * (from buster:) a bunch of config files from Puppet had conflicts
   needing to be resolved and that we should really, really finish
   upgrading in Puppet now

 * (from buster:) ferm fails to reload during upgrade, with the following error:
 
        Couldn't load match `state':No such file or directory

 * The official list of [known issues][]

[known issues]: https://www.debian.org/releases/buster/amd64/release-notes/ch-information.en.html

## Resolved

### tor-nagios-checks tempfile

[this patch](https://gitweb.torproject.org/admin/tor-nagios.git/commit/?id=661d0dbb00a2876da21f3c66c18d8eb8f8cae790) was necessary to port from `tempfile` to `mktemp` in
that TPA-specific Debian package.

# Troubleshooting

## Upgrade failures

Instructions on errors during upgrades can be found in [the release
notes troubleshooting section](https://www.debian.org/releases/bullseye/amd64/release-notes/ch-upgrading.en.html#trouble).

## Reboot failures

If there's any trouble during reboots, you should use some recovery
system. The [release notes actually have good documentation on
that](https://www.debian.org/releases/bullseye/amd64/release-notes/ch-upgrading.en.html#recovery), on top of "use a live filesystem".

# References

 * [Official guide](https://www.debian.org/releases/bullseye/amd64/release-notes/ch-upgrading.en.html)
 * [Release notes](https://www.debian.org/releases/bullseye/amd64/release-notes/ch-whats-new.en.html)
 * [Koumbit guide](https://wiki.koumbit.net/BullseyeUpgrade) (WIP, reviewed 2021-08-26)
 * [DSA guide](https://dsa.debian.org/howto/upgrade-to-bullseye/) (WIP, reviewed 2021-08-26)
 * [anarcat guide](https://anarc.at/services/upgrades/bullseye/) (WIP, last sync 2021-08-26)
 * [Solution proposal to automate this](https://wiki.debian.org/AutomatedUpgrade)

# Fleet-wide changes

The following changes need to be performed *once* for the entire
fleet, generally at the beginning of the upgrade process.

## installer changes

The installer need to be changed to support the new release. This
includes:

 * the Ganeti installers (add a `gnt-instance-debootstrap` variant,
   `modules/profile/manifests/ganeti.pp` in `tor-puppet.git`, see
   commit 4d38be42 for an example)
 * the (deprecated) libvirt installer
   (`modules/roles/files/virt/tor-install-VM`, in `tor-puppet.git`)
 * the wiki documentation:
   * create a new page like this one documenting the process, linked
     from [howto/upgrades](howto/upgrades)
   * make an entry in the `data.csv` to start tracking progress (see
     below), copy the `Makefile` as well, changing the suite name
   * change the [Ganeti procedure](howto/ganeti#adding-a-new-instance) so that the new suite is used by
     default
   * change the [Hetzner robot](howto/new-machine-hetzner-robot) install procedure
 * `tsa-misc` and the fabric installer (TODO)

## Debian archive changes

The Debian archive on `db.torproject.org` (currently alberti) need to
have a new suite added. This can be (partly) done by editing files
`/srv/db.torproject.org/ftp-archive/`. Specifically, the two following
files need to be changed:

 * `apt-ftparchive.config`: a new stanza for the suite, basically
   copy-pasting from a previous entry and changing the suite
 * `Makefile`: add the new suite to the for loop

But it is not enough: the directory structure need to be crafted by
hand as well. A simple way to do so is to replicate a previous release
structure:

    cd /srv/db.torproject.org/ftp-archive
    rsync -a --include='*/' --exclude='*' archive/dists/buster/  archive/dists/bullseye/

# Per host progress

When a critical mass of servers have been upgraded and only "hard"
ones remain, they can be turned into tickets and tracked in GitLab. In
the meantime...

A list of servers to upgrade can be obtained with:

    curl -s -G http://localhost:8080/pdb/query/v4 --data-urlencode 'query=nodes { facts { name = "lsbdistcodename" and value != "bullseye" }}' | jq .[].certname | sort

Policy established in [howto/upgrades](howto/upgrades).

<figure>
<img alt="graph showing planned completion date, currently around September 2020" src="/howto/upgrades/bullseye/predict.png" />
<figcaption>

The above graphic shows the progress of the migration between major
releases. It can be regenerated with the [predict-os](https://gitlab.com/anarcat/predict-os) script. It
pulls information from [howto/puppet](howto/puppet) to update a [CSV file](data.csv) to
keep track of progress over time.

WARNING: the graph may be incorrect or missing as the upgrade
procedure ramps up.
</figcaption>
</figure>
