[[!toc]]
Procedure
=========
Before upgrading a box, it might be preferable to coordinate with the
service admins to see if the box will survive the upgrade. See
[[upgrades]] for the list of teams and how they prefer to handle that
process.
1. Preparation:
: reset to the default locale
export LC_ALL=C.UTF-8 &&
sudo apt install ttyrec screen debconf-utils apt-show-versions deborphan &&
sudo ttyrec -e screen /var/log/upgrade-buster.ttyrec
2. Backups and checks:
( umask 0077 &&
tar cfz /var/backups/pre-buster-backup.tgz /etc /var/lib/dpkg /var/lib/apt/extended_states /var/lib/aptitude/pkgstates /var/cache/debconf &&
dpkg --get-selections "*" > /var/backups/dpkg-selections-pre-buster.txt &&
debconf-get-selections > /var/backups/debconf-selections-pre-buster.txt
) &&
apt-mark showhold &&
dpkg --audit &&
: look for dkms packages and make sure they are relevant, if not, purge. &&
( dpkg -l '*dkms' || true ) &&
: make sure backups are up to date in Nagios &&
echo End of Step 2
3. Perform any pending upgrade and clear out old pins:
: Check for pinned, on hold, packages, and possibly disable &&
rm -f /etc/apt/preferences /etc/apt/preferences.d/* &&
rm -f /etc/apt/sources.list.d/testing.list &&
rm -f /etc/apt/sources.list.d/stretch-backports.list &&
rm -f /etc/apt/sources.list.d/backports.debian.org.list &&
apt update && apt -y upgrade &&
: list kernel images and purge unused packages &&
dpkg -l 'linux-image-*' &&
: look for packages from backports, other suites or archives &&
: if possible, switch to official packages by disabling third-party repositories &&
apt-show-versions | grep -v /stretch | grep -v 'not installed$' &&
echo End of Step 3
4. Check free space, see [this guide to free up space][] and
download packages:
sed -i 's/stretch/buster/g' /etc/apt/sources.list.d/* &&
apt update && apt -o APT::Get::Trivial-Only=true dist-upgrade &&
df -h &&
apt -y -d upgrade &&
apt -y -d dist-upgrade &&
echo End of Step 4
5. Enable module loading (for ferm)
sed -i -e 's/.*modules_disabled/#&/' /etc/rc.local
reboot
export LC_ALL=C.UTF-8 &&
sudo ttyrec -e screen /var/log/upgrade-buster.ttyrec.2
6. Actual upgrade run:
apt install -y dpkg apt &&
apt install ferm &&
apt dist-upgrade &&
echo 'End of Step 6'
7. Post-upgrade procedures:
apt-get update --allow-releaseinfo-change &&
apt-mark manual git &&
apt --purge autoremove &&
apt purge $(for i in apt-transport-https dh-python emacs24-nox gnupg-agent libbind9-140 libcryptsetup4 libdns-export162 libdns162 libevent-2.0-5 libevtlog0 libgdbm3 libicu57 libisc-export160 libisc160 libisccc140 libisccfg140 liblvm2app2.2 liblvm2cmd2.02 liblwres141 libmpfr4 libncurses5 libperl5.24 libprocps6 libpython3.5 libpython3.5-minimal libpython3.5-stdlib libruby2.3 libssl1.0.2 libunbound2 libunistring0 python3-distutils python3-lib2to3 python3.5 python3.5-minimal ruby-nokogiri ruby-pkg-config ruby-rgen ruby-safe-yaml ruby2.3 sgml-base xml-core git-core gcc-6-base:amd64 nagios-plugins-basic perl-modules-5.24 libsensors4:amd64 grub2 iproute libncursesw5 libustr-1.0-1; do dpkg -l "$i" 2>/dev/null | grep -q '^ii' && echo "$i"; done) &&
dpkg --purge libsensors4:amd64 syslog-ng-mod-json
(puppet agent -t || true) &&
(puppet agent -t || true)
7. reboot once so we can then get rid of the old kernel image
reboot
8. Post-upgrade checks:
export LC_ALL=C.UTF-8 &&
sudo ttyrec -e screen /var/log/upgrade-buster.ttyrec.3
# review and purge old packages, including kernels
apt --purge autoremove
dsa-check-packages | tr -d ,
while deborphan -n | grep -q . ; do apt purge $(deborphan -n); done
apt --purge autoremove
dpkg -l '*-dbg' # look for dbg package and possibly replace with -dbgsym
apt clean
# review packages that are not in the new distribution
apt-show-versions | grep -v /buster
reboot
9. Change the hostgroup of the host to buster in Nagios (in
`tor-nagios/config/nagios-master.cfg` on `git@git-rw.tpo`)
[this guide to free up space]: http://www.debian.org/releases/buster/amd64/release-notes/ch-upgrading.en.html#sufficient-space
Service-specific upgrade procedures
===================================
PostgreSQL
----------
PostgreSQL is special and needs to be upgraded manually.
1. make a full backup of the old cluster:
ssh bungei.torproject.org "sudo -u torbackup postgres-make-one-base-backup $(grep ^meronense.torproject.org $(which postgres-make-base-backups ))"
The above assumes the host to backup is `meronense` and the backup
server is `bungei`. See [[postgresql]] for details of that
procedure.
2. Once the backup completes, move the directory out of the way and
recreate it:
ssh bungei.torproject.org "mv /srv/backups/pg/meronense /srv/backups/pg/meronense-9.6 && sudo -u torbackup mkdir /srv/backups/pg/meronense"
3. do the actual cluster upgrade, on the database server:
pg_upgradecluster -m upgrade -k &&
for cluster in `ls /etc/postgresql/9.6/`; do
mv /etc/postgresql/9.6/$cluster/conf.d/* /etc/postgresql/11/$cluster/conf.d/
done
Issues
======
Pending
-------
* upgrading restarts openvswitch will mean all guests lose network
* At least on `kvm5`, `brpub` was having issues. Either ipv4 or ipv6
address was missing, or the v6 route to the guests was missing.
Probably because the ipv6 route setting failed since we set a
prefsrc and that was only brought up later?
Rewrote `/etc/network/interfaces` to set things up more manually.
On your host, check if `brpub` has both ipv4 and ipv6 addresses after
boot before launching VMs, and that is has an ipv6 route into `brpub`
with the configured `prefsrc` address. If not, fiddle
likewise.
See [ticket #31083](https://trac.torproject.org/projects/tor/ticket/31083) for followup on possible routing issues.
* On physical hosts witch `/etc/sysfs.d/local-io-schedulers.conf`,
note that `deadline` no longer existsts. Probably it is also not
necessary as Linux might pick the right scheduler anyhow.
* the following config files had conflicts but were managed by Puppet
so those changes were ignored for now. eventually they should be
upgraded in Puppet as well.
/etc/bacula/bacula-fd.conf
/etc/bind/named.conf.options
/etc/default/stunnel4
/etc/ferm/ferm.conf
/etc/init.d/stunnel4
/etc/nagios/nrpe.cfg
/etc/ntp.conf
/etc/syslog-ng/syslog-ng.conf
* ferm fails to reload during upgrade, with the following error:
Couldn't load match `state':No such file or directory
* Puppet might try to downgrade the `sources.list` files to `stretch`
or `n/a` for some reason, just re-run Puppet after fixing the
`sources.list` files, it will eventually figure it out.
* The official list of [known issues][]
[known issues]: https://www.debian.org/releases/buster/amd64/release-notes/ch-information.en.html
Resolved
--------
* `apt-get` complains like this after upgrade ([bug #929248](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=929248)):
E: Repository 'https://mirrors.wikimedia.org/debian buster InRelease' changed its 'Suite' value from 'testing' to 'stable'
the following workaround was added to the upgrade instructions,
above, but might be necessary on machines where this procedure was
followed before the note was added:
apt-get update --allow-releaseinfo-change
* the following config files were updated to buster:
/etc/logrotate.d/ulogd2
/etc/ssh/sshd_config
* Puppet was warning with the following when running against a
master running stretch, harmlessly:
Warning: Downgrading to PSON for future requests
References
==========
Note: the official upgrade guide and release notes not available at
the time of writing (2019-04-08) as the documentation is usually
written during the freeze and buster is not there yet.
* [Official guide](https://www.debian.org/releases/buster/amd64/release-notes/ch-upgrading.en.html)
* [Release notes](https://www.debian.org/releases/buster/amd64/release-notes/ch-whats-new.en.html)
* [Koumbit guide](https://wiki.koumbit.net/BusterUpgrade)
* [DSA guide](https://dsa.debian.org/howto/upgrade-to-buster/)
* [Solution proposal to automate this](https://wiki.debian.org/AutomatedUpgrade)