buster.mdwn

Debian 10 [buster](https://wiki.debian.org/DebianBuster) was released on [July 6th 2019](https://lists.debian.org/debian-devel-announce/2019/07/msg00002.html). Tor started
the upgrade to buster during the freeze and hopes to complete the
process before the stretch EOL, [one year after the stable
release](https://www.debian.org/security/faq#lifespan), so normally around July 2020.

[[!toc levels=3]]

Procedure
=========

Before upgrading a box, it might be preferable to coordinate with the
service admins to see if the box will survive the upgrade. See
[[upgrades]] for the list of teams and how they prefer to handle that
process.

 1. Preparation:

        : reset to the default locale
        export LC_ALL=C.UTF-8 &&
        sudo apt install ttyrec screen debconf-utils apt-show-versions deborphan &&
        sudo ttyrec -e screen /var/log/upgrade-buster.ttyrec

 2. Backups and checks:

        ( umask 0077 &&
          tar cfz /var/backups/pre-buster-backup.tgz /etc /var/lib/dpkg /var/lib/apt/extended_states /var/lib/aptitude/pkgstates /var/cache/debconf &&
          dpkg --get-selections "*" > /var/backups/dpkg-selections-pre-buster.txt &&
          debconf-get-selections > /var/backups/debconf-selections-pre-buster.txt
        ) &&
        apt-mark showhold &&
        dpkg --audit &&
        : look for dkms packages and make sure they are relevant, if not, purge. &&
        ( dpkg -l '*dkms' || true ) &&
        : make sure backups are up to date in Nagios &&
        printf "End of Step 2\a\n"

 3. Enable module loading (for ferm) and test reboots:

        systemctl disable modules_disabled.timer &&
        shutdown -r +1 "rebooting with module loading enabled"

        export LC_ALL=C.UTF-8 &&
        sudo ttyrec -a -e screen /var/log/upgrade-buster.ttyrec

 4. Perform any pending upgrade and clear out old pins:

        puppet agent --disable "running major upgrade" &&
        : Check for pinned, on hold, packages, and possibly disable &&
        rm -f /etc/apt/preferences /etc/apt/preferences.d/* &&
        rm -f /etc/apt/sources.list.d/testing.list &&
        rm -f /etc/apt/sources.list.d/stretch-backports.list &&
        rm -f /etc/apt/sources.list.d/backports.debian.org.list &&
        apt update && apt -y upgrade &&
        : list kernel images and purge unused packages &&
        dpkg -l 'linux-image-*' &&
        : look for packages from backports, other suites or archives &&
        : if possible, switch to official packages by disabling third-party repositories &&
        apt-show-versions | grep -v /stretch | grep -v 'not installed$' &&
        printf "End of Step 4\a\n"

 5. Check free space, see [this guide to free up space][] and
    download packages:

        sed -i 's/stretch/buster/g' /etc/apt/sources.list.d/* &&
        apt update && apt -o APT::Get::Trivial-Only=true dist-upgrade &&
        df -h &&
        apt -y -d upgrade &&
        apt -y -d dist-upgrade &&
        printf "End of Step 5\a\n"

 6. Actual upgrade run:

        apt install -y dpkg apt &&
        apt install ferm &&
        apt dist-upgrade &&
        printf "End of Step 6\a\n"

 7. Post-upgrade procedures:

        apt-get update --allow-releaseinfo-change &&
        apt-mark manual git &&
        apt --purge autoremove &&
        apt purge $(for i in apt-transport-https dh-python emacs24-nox gnupg-agent libbind9-140 libcryptsetup4 libdns-export162 libdns162 libevent-2.0-5 libevtlog0 libgdbm3 libicu57 libisc-export160 libisc160 libisccc140 libisccfg140 liblvm2app2.2 liblvm2cmd2.02 liblwres141 libmpfr4 libncurses5 libperl5.24 libprocps6 libpython3.5 libpython3.5-minimal libpython3.5-stdlib libruby2.3 libssl1.0.2 libunbound2 libunistring0 python3-distutils python3-lib2to3 python3.5 python3.5-minimal ruby-nokogiri ruby-pkg-config ruby-rgen ruby-safe-yaml ruby2.3 sgml-base xml-core git-core gcc-6-base:amd64 nagios-plugins-basic perl-modules-5.24 libsensors4:amd64 grub2 iproute libncursesw5 libustr-1.0-1; do dpkg -l "$i" 2>/dev/null | grep -q '^ii' && echo "$i"; done) &&
        dpkg --purge libsensors4:amd64 syslog-ng-mod-json || true &&
        puppet agent --enable &&
        (puppet agent -t || true) &&
        (puppet agent -t || true) &&
        printf "End of Step 7\a\n" &&
        shutdown -r +1 "rebooting to get rid of old kernel image..."

 8. Post-upgrade checks:

        export LC_ALL=C.UTF-8 &&
        sudo ttyrec -a -e screen /var/log/upgrade-buster.ttyrec

        # review and purge old packages, including kernels
        apt --purge autoremove
        dsa-check-packages | tr -d ,
        while deborphan -n | grep -q . ; do apt purge $(deborphan -n); done
        apt --purge autoremove
        dpkg -l '*-dbg' # look for dbg package and possibly replace with -dbgsym
        apt clean
        # review packages that are not in the new distribution
        apt-show-versions | grep -v /buster
        printf "End of Step 8\a\n"
        shutdown -r +1 "testing reboots one final time"

 9. Change the hostgroup of the host to buster in Nagios (in
    `tor-nagios/config/nagios-master.cfg` on `git@git-rw.tpo`)

[this guide to free up space]: http://www.debian.org/releases/buster/amd64/release-notes/ch-upgrading.en.html#sufficient-space

Service-specific upgrade procedures
===================================

PostgreSQL
----------

PostgreSQL is special and needs to be upgraded manually. 

 1. make a full backup of the old cluster:

        ssh bungei.torproject.org "sudo -u torbackup postgres-make-one-base-backup $(grep ^meronense.torproject.org $(which postgres-make-base-backups ))"

    The above assumes the host to backup is `meronense` and the backup
    server is `bungei`. See [[postgresql]] for details of that
    procedure.

 2. Once the backup completes, move the directory out of the way and
    recreate it:

        ssh bungei.torproject.org "mv /srv/backups/pg/meronense /srv/backups/pg/meronense-9.6 && sudo -u torbackup mkdir /srv/backups/pg/meronense"

 3. do the actual cluster upgrade, on the database server:

        export LC_ALL=C.UTF-8 &&
        printf "about to drop cluster main on postgresql-11, press enter to continue" &&
        read _ &&
        pg_dropcluster --stop 11 main &&
        pg_upgradecluster -m upgrade -k &&
        for cluster in `ls /etc/postgresql/9.6/`; do
            mv /etc/postgresql/9.6/$cluster/conf.d/* /etc/postgresql/11/$cluster/conf.d/
        done

 4. make sure the new cluster isn't backed up by bacula:

        touch /var/lib/postgresql/11/.nobackup

    TODO: put in Puppet.

 4. change the cluster target in the backup system, in `tor-puppet`,
    for example:

        --- a/modules/postgres/manifests/backup_source.pp
        +++ b/modules/postgres/manifests/backup_source.pp
        @@ -30,7 +30,7 @@ class postgres::backup_source {
                case $hostname {
                        'gitlab-01': {
                        }
        -               'subnotabile', 'bacula-director-01': {
        +               'meronense', 'subnotabile', 'bacula-director-01': {
                                postgres::backup_cluster { $::hostname:
                                        pg_version => '11',
                                }

 5. change the postgres version in `tor-nagios` as well:

        --- a/config/nagios-master.cfg
        +++ b/config/nagios-master.cfg
        @@ -354,7 +354,7 @@ servers:
           meronense:
             address: 94.130.28.195
             parents: kvm4
        -    hostgroups: computers, buster, syslog-ng-hosts, hassrvfs, apache2-hosts, apache-https-host, postgres96-hosts, hassrvfs90
        +    hostgroups: computers, buster, syslog-ng-hosts, hassrvfs, apache2-hosts, apache-https-host, postgres11-hosts, hassrvfs90
           # db.tpo
           alberti:
             address: 94.130.28.196

 6. once everything works okay, remove the old packages:

        apt purge postgresql-9.6 postgresql-client-9.6

 7. purge the old backups directory after a week:

        ssh bungei.torproject.org "echo 'rm -r /srv/backups/pg/meronense-9.6/' | at now + 7day"

Notable changes
===============

Here is a subset of the [notable changes in this release](https://www.debian.org/releases/buster/amd64/release-notes/ch-whats-new.en.html#newdistro), along
with our risk analysis and notes:

| Package      | Stretch | Buster | Notes |
| ------------ | ------- | ------ | ----- |
| Apache       | 2.4.25  | 2.4.38 | |
| Bind         | 9.10    | 9.11   | |
| Cryptsetup   | 1.7     | 2.1    | |
| Docker       | N/A     | 18     | Docker back in Debian? |
| GnuPG        | 2.1     | 2.2    | |
| Linux kernel | 4.9     | 4.19   | |
| MariaDB      | 10.1    | 10.3   | |
| OpenJDK      | 8       | 11     | major upgrade, TBD |
| OpenSSH      | 7.4     | 7.8    | |
| Perl         | 5.24    | 5.28   | |
| Postfix      | 3.1.8   | 3.3.2  | |
| PostgreSQL   | 9.6     | 11     | two major upgrades, release notes: [10](https://www.postgresql.org/docs/10/release-10.html) [11](https://www.postgresql.org/docs/11/release-11.html)|
| Rustc        | N/A     | 1.34   | Rust enters Debian |

Many packages were removed from Buster. Anarcat built an [exhaustive
list](https://paste.anarc.at/publish/debian-packages-removed-from-buster-without-libs/stdin.txt) on May 16th 2019, but it's probably changed since then. See
also the [noteworthy obsolete packages](https://www.debian.org/releases/buster/amd64/release-notes/ch-information.en.html#noteworthy-obsolete-packages) list.

Python 2 is unsupported upstream since January 1st 2020. We have a
significant number of Python scripts that will need to be upgraded. It
is unclear what will happen to Python 2 in Debian in terms of security
support for the buster lifetime.

Issues
======

Pending
-------

 * upgrading restarts openvswitch will mean all guests lose network

 * At least on `kvm5`, `brpub` was having issues.  Either ipv4 or ipv6
   address was missing, or the v6 route to the guests was missing.
   Probably because the ipv6 route setting failed since we set a
   prefsrc and that was only brought up later?

   Rewrote `/etc/network/interfaces` to set things up more manually.
   On your host, check if `brpub` has both ipv4 and ipv6 addresses after
   boot before launching VMs, and that is has an ipv6 route into `brpub`
   with the configured `prefsrc` address.  If not, fiddle
   likewise.
   
   See [ticket #31083](https://trac.torproject.org/projects/tor/ticket/31083) for followup on possible routing issues.

 * On physical hosts witch `/etc/sysfs.d/local-io-schedulers.conf`,
   note that `deadline` no longer existsts. Probably it is also not
   necessary as Linux might pick the right scheduler anyhow.

 * the following config files had conflicts but were managed by Puppet
   so those changes were ignored for now. eventually they should be
   upgraded in Puppet as well.

        /etc/bacula/bacula-fd.conf
        /etc/bind/named.conf.options
        /etc/default/stunnel4
        /etc/ferm/ferm.conf
        /etc/init.d/stunnel4
        /etc/nagios/nrpe.cfg
        /etc/ntp.conf
        /etc/syslog-ng/syslog-ng.conf

 * ferm fails to reload during upgrade, with the following error:
 
        Couldn't load match `state':No such file or directory

 * Puppet might try to downgrade the `sources.list` files to `stretch`
   or `n/a` for some reason, just re-run Puppet after fixing the
   `sources.list` files, it will eventually figure it out.

 * The official list of [known issues][]

[known issues]: https://www.debian.org/releases/buster/amd64/release-notes/ch-information.en.html

Resolved
--------

 * `apt-get` complains like this after upgrade ([bug #929248](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=929248)):

        E: Repository 'https://mirrors.wikimedia.org/debian buster InRelease' changed its 'Suite' value from 'testing' to 'stable'

   the following workaround was added to the upgrade instructions,
   above, but might be necessary on machines where this procedure was
   followed before the note was added:

        apt-get update --allow-releaseinfo-change

 * the following config files were updated to buster:

        /etc/logrotate.d/ulogd2
        /etc/ssh/sshd_config

 * Puppet was warning with the following when running against a
   master running stretch, harmlessly:

        Warning: Downgrading to PSON for future requests

References
==========

Note: the official upgrade guide and release notes not available at
the time of writing (2019-04-08) as the documentation is usually
written during the freeze and buster is not there yet.

 * [Official guide](https://www.debian.org/releases/buster/amd64/release-notes/ch-upgrading.en.html)
 * [Release notes](https://www.debian.org/releases/buster/amd64/release-notes/ch-whats-new.en.html)
 * [Koumbit guide](https://wiki.koumbit.net/BusterUpgrade)
 * [DSA guide](https://dsa.debian.org/howto/upgrade-to-buster/)
 * [Solution proposal to automate this](https://wiki.debian.org/AutomatedUpgrade)