[[_TOC_]]

# Debian upgrades

## Major upgrades

Major upgrades are done by hand, with a "cheat sheet" created for each
major release. Here are the currently documented ones:

 * Debian 11, [bullseye](howto/upgrades/bullseye)
 * Debian 10, [buster](howto/upgrades/buster)

### Team-specific upgrade policies

Before we perform a major upgrade, it might be advisable to consult
with the team working on the box to see if it will interfere for their
work. Some teams might block if they believe the major upgrade will
break their service. They are not allowed to indefinitely block the
upgrade, however.

Team policies:

 * anti-censorship: TBD
 * metrics: one or two work-day advance notice ([source](https://gitlab.torproject.org/legacy/trac/-/issues/32998#note_2345807))
 * funding: schedule a maintenance window
 * git: TBD
 * gitlab: TBD
 * translation: TBD

Some teams might be missing from the list.

## Minor upgrades

### Unattended upgrades

Most of the packages upgrades are handled by the unattended-upgrades package which
is configured via puppet.

Unattended-upgrades writes logs to `/var/log/unattended-upgrades/` but
also `/var/log/dpkg.log`.

The default configuration file for unattended-upgrades is at `/etc/apt/apt.conf.d/50unattended-upgrades`.

Pending upgrades are still noticed by Nagios which warns loudly about them in its
usual channels.

If a package origin isn't picked by unattended upgrades it will need to be upgraded
by hand or its origin added to `modules/profile/manifests/unattended_upgrades.pp` in
puppet.

### Manual upgrades with Cumin

It's also possible to do a manual mass-upgrade run with
[Cumin](howto/cumin):

    cumin -b 10  '*' 'apt update ; apt upgrade -yy ; dsa-update-apt-status'

### Restarting services

After upgrades, there's a Nagios check that might trigger and tell you
that some services are running with outdated libraries. For example,
after a Bacula upgrade:

    The following processes have libs linked that were upgraded: bacula: bacula-fd (1787)

While the entire host can be rebooted (using the procedure below) to
fix this problem, it's sometimes less disruptive to just restart that
one process.

For this purpose, `needrestart` is installed on all machines, and it makes sure to
restart services. It can also be useful to restart services manually, for
example with:

    ssh root@cupani.torproject.org needrestart -u NeedRestart::UI::stdio -r a

(Note that earlier versions of needrestart showed spurious warnings in
this mode, see [bug #859387](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859387), fixed in buster.)

If you cannot figure out why the warning happens, you might want to
run the check by hand:

    /usr/lib/nagios/plugins/dsa-check-libs

The `--verbose` flag also shows which file trigger the warning.

Some services will have `cron` as a parent, and will make
`needrestart` want to restart cron which is, of course,
ineffective. The only "proper" way to restart those services is to
reboot the host.

Services setup with the new systemd-based startup system documented in
[doc/services](doc/services) can be restarted with:

    systemctl restart user@1504.service

There's a feature request ([bug #843778](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=843778)) to implement support for
those services directly in needrestart.

### Packages blocked from automatic upgrades

Those packages are currently blocked from automatic upgrades in `unattended-upgrades`:

- **Open vSwitch** (`openvswitch-switch` and `openvswitch-common`, [bug
  34185](https://bugs.torproject.org/34185)): to upgrade manually, empty the server, restart, OVS,
  then migrate the machines back.
  
   1. on the Ganeti master, list the instances on the Ganeti node:
   
        INSTANCES=$(gnt-instance list -o name --no-headers --filter "pnode == \"$NODE\"")

   2. on the Ganeti master, empty the Ganeti node:
  
        gnt-node migrate -f $NODE

   2. on the Ganeti node where OVS needs to be upgraded:

        service openvswitch-nonetwork.service restart

   3. on the Ganeti master, migrate all the instances back:

        gnt-instance migrate -f $INSTANCES

      the instance list comes from the first step

  Note that this might be fixed in Debian bullseye, [bug 961746](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=961746) in
  Debian is marked as fixed, but will still need to be tested on our
  side first.

- **Grub** (`grub-pc`, [bug 40042](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40042)) has been known to have issues as
  well, so it is blocked. to upgrade, make sure the install device is
  defined, by running `dpkg-reconfigure grub-pc`. this issue might
  actually have been fixed in the package, see [issue 40185](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40185).

Packages are blocked from upgrades when they cause significant
breakage during an upgrade run, enough to cause an outage and/or
require significant recovery work. This is done through Puppet, in the
`profile::unattended_upgrades` class, in the `blacklist` setting.

Packages can be unblocked if and only if:

 * the bug is confirmed as fixed in Debian
 * the fix is deployed on all servers and confirmed as working
 * we have good confidence that future upgrades will not break the
   system again

### Kernel upgrades and reboots

Sometimes it is necessary to perform a reboot on the hosts, when the
kernel is updated. Nagios will warn about this, with something like
this:

    WARNING: Kernel needs upgrade [linux-image-4.9.0-9-amd64 != linux-image-4.9.0-8-amd64]

#### Rebooting guests

If this is only a virtual machine, and the only one affected, it can
be rebooted directly. This can be done with the `tsa-misc` script
called `reboot`:

    ./reboot -H test-01.torproject.org,test-02.torproject.org

By default, the script will wait 2 minutes before hosts: that should
be changed to *30 minutes* if the hosts are part of a mirror network
to give the monitoring systems (`mini-nag`) time to rotate the hosts
in and out of DNS:

    ./reboot -H mirror-01.torproject.org,mirror-02.torproject.org --delay-nodes 1800

If the host has an encrypted filesystem and is hooked up with Mandos, it
will return automatically. Otherwise it might need a password to be
entered at boot time, either through the initramfs (if it has the
`profile::fde` class in Puppet) or manually, after the boot. That is
the case for the `mandos-01` server itself, for example, as it
currently can't unlock itself, naturally.

This routine should be able to reboot all hosts with a `rebootPolicy`
defined to `justdoit` or `rotation`:

    echo "rebooting 'justdoit' hosts with a 10-minute delay, every 2 minutes...."
    ./reboot -H $(ssh db.torproject.org 'ldapsearch -h db.torproject.org -x -ZZ -b ou=hosts,dc=torproject,dc=org -LLL "(rebootPolicy=justdoit)" hostname | awk "\$1 == \"hostname:\" {print \$2}" | sort -R') --delay-shutdown=10 --delay-hosts=120 -v

    echo "rebooting 'rotation' hosts with a 10-minute delay, every 30 minutes...."
    ./reboot -H $(ssh db.torproject.org 'ldapsearch -h db.torproject.org -x -ZZ -b ou=hosts,dc=torproject,dc=org -LLL "(rebootPolicy=rotation)" hostname | awk "\$1 == \"hostname:\" {print \$2}" | sort -R') --delay-shutdown=10 --delay-hosts=1800 -v

### Rebooting KVM hosts

The remaining is the "manual" procedure, the KVM hosts:

    ./reboot-host moly.torproject.org

### Rebooting Ganeti nodes

The ganeti hosts, using Fabric:

    ./reboot -v --delay-shutdown 1 --delay-hosts 30 -H fsn-node-0{1,2,3,4,5,6,7}.torproject.org

This can be done in parallel across clusters:

    ./reboot -v --delay-shutdown 1 --delay-hosts 30 -H chi-node-0{1,2,3,4}.torproject.org

This is also documented in the [howto/ganeti](howto/ganeti) section. Do not
forget to rebalance the cluster after the reboot.

### Rebooting Ganeti guests

If you see this in Nagios:

    The following processes have libs linked that were upgraded: ganeti14: qemu-system-x86 (41509): ganeti15: qemu-system-x86 (41081): ganeti8: qemu-system-x86 (22106)

... and the Ganeti node itself doesn't need to be restarted, you can
see a stressful reboot by just migrating the instances between the
nodes. This will restart the `qemu` processes and complete the
upgrade, while imposing minimal (if any) downtime.

The process here is to do a `gnt-node migrate` on all nodes, which
will empty one node at a time. When that is complete, the cluster
needs to be rebalanced. This is not exactly an "idempotent" process:
you might not end up with exactly the same state as you had in the
beginning, even after rebalancing the cluster.

Make sure you run in a screen session, because this process takes
time:

    screen

Then, look at the current state of the cluster:

    hbal -L -C -v

Take note of the score and the proposed solution, but do not execute
it. This will give you an idea of how good or bad things are after the
migrate.

Then migrate all guests, for example:

    for node in chi-node-0{1,2,3,4}; do gnt-node migrate -f $node; done

Once that is done, all the warnings should be gone from Nagios.

Then rebalance the cluster:

    hbal -L -C -v --no-disk-moves

Note that we use `--no-disk-moves` to try to keep the solver from
moving actual disks. Since the `migrate` task above shouldn't have
moved any disk, it should be able to find a solution with a score
similar than the one we started with, without moving disks (which is
an even slower operation).

### Remaining nodes

The scaleway box needs special handholding, see [ticket 32920](https://bugs.torproject.org/32920). The
windows boxes should normally not need a reboot.

When all hosts are rebooted, see [Nagios unhandled problems](https://nagios.torproject.org/cgi-bin/icinga/status.cgi?allunhandledproblems) to
confirm.

#### Generic upgrade routines

LDAP hosts have information about how they can be rebooted, in the
`rebootPolicy` field. Here are what the various fields mean:

 * `justdoit` - can be rebooted any time, with a 10 minute delay,
   possibly in parallel
 * `rotation` - part of a cluster where each machine needs to be
   rebooted one at a time, with a 30 minute delay for DNS to update
 * `manual` - needs to be done by hand or with a special tool (fabric
   in case of ganeti, reboot-host in the case of KVM, nothing for
   windows boxes)

### Example runs

**This documenation is now deprecated as we are now using
unattended-upgrades and needrestart.**

Here's an example run of the upgrade tool:

    weasel@orinoco:~$ torproject-upgrade-prepare                      
    Agent pid 5384               
    Pass a valid window to KWallet::Wallet::openWallet().
    Identity added: /home/weasel/.ssh/id_rsa (/home/weasel/.ssh/id_rsa)
    build-arm-03.torproject.org: ControlSocket /home/weasel/.ssh/.pipes/orinoco/weasel@rouyi.torproject.org:22 already exists, disabling multiplexing
    rouyi.torproject.org: ControlSocket /home/weasel/.ssh/.pipes/orinoco/weasel@rouyi.torproject.org:22 already exists, disabling multiplexing
    build-arm-01.torproject.org: ControlSocket /home/weasel/.ssh/.pipes/orinoco/weasel@rouyi.torproject.org:22 already exists, disabling multiplexing
    build-arm-02.torproject.org: ControlSocket /home/weasel/.ssh/.pipes/orinoco/weasel@rouyi.torproject.org:22 already exists, disabling multiplexing
    gillii.torproject.org: ssh: connect to host gillii.torproject.org port 22: No route to host
    geyeri.torproject.org: ssh: connect to host geyeri.torproject.org port 22: No route to host
    chiwui.torproject.org: W: Size of file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_jessie-backports_InRelease is not what the server reported 166070 130112
    chiwui.torproject.org: W: Size of file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_jessie-updates_InRelease is not what the server reported 145060 16384
    ------------------------------------------------------------
    Upgrade available on alberti.torproject.org brulloi.torproject.org chamaemoly.torproject.org colchicifolium.torproject.org corsicum.torproject.org cupani.torproject.org gayi.torproject.org henryi.torproject.org iranicum.torproject.org materculae.torproject.org meronense.torproject.org nevii.torproject.org palmeri.torproject.org scw-arm-ams-01.torproject.org troodi.torproject.org vineale.torproject.org:

    Reading package lists...
    Building dependency tree...
    Reading state information...
    Calculating upgrade...
    The following packages will be upgraded:
      libssl1.0.2 linux-image-4.9.0-8--x- openssh-client openssh-server
      openssh-sftp-server ssh
    6 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    Inst openssh-sftp-server [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-]) []
    Inst libssl1.0.2 [1.0.2q-1~deb9u1] (1.0.2r-1~deb9u1 Debian-Security:9/stable [-x-]) []
    Inst openssh-server [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-]) []
    Inst openssh-client [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Inst ssh [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [all])
    Inst linux-image-4.9.0-8--x- [4.9.144-3] (4.9.144-3.1 Debian:stable-updates [-x-])
    Conf openssh-sftp-server (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Conf libssl1.0.2 (1.0.2r-1~deb9u1 Debian-Security:9/stable [-x-])
    Conf openssh-server (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Conf openssh-client (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Conf ssh (1:7.4p1-10+deb9u6 Debian-Security:9/stable [all])
    Conf linux-image-4.9.0-8--x- (4.9.144-3.1 Debian:stable-updates [-x-])

    Accept [y/N]? y
    ------------------------------------------------------------
    Upgrade available on orestis.torproject.org:

    Reading package lists...
    Building dependency tree...
    Reading state information...
    Calculating upgrade...
    The following packages will be upgraded:
      libssl1.0.2 linux-image-4.9.0-8--x- linux-libc-dev openssh-client
      openssh-server openssh-sftp-server
    6 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    Inst libssl1.0.2 [1.0.2q-1~deb9u1] (1.0.2r-1~deb9u1 Debian-Security:9/stable [-x-])
    Inst openssh-sftp-server [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-]) []
    Inst openssh-server [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-]) []
    Inst openssh-client [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Inst linux-image-4.9.0-8--x- [4.9.144-3] (4.9.144-3.1 Debian:stable-updates [-x-])
    Inst linux-libc-dev [4.9.144-3] (4.9.144-3.1 Debian:stable-updates [-x-])
    Conf libssl1.0.2 (1.0.2r-1~deb9u1 Debian-Security:9/stable [-x-])
    Conf openssh-sftp-server (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Conf openssh-server (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Conf openssh-client (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Conf linux-image-4.9.0-8--x- (4.9.144-3.1 Debian:stable-updates [-x-])
    Conf linux-libc-dev (4.9.144-3.1 Debian:stable-updates [-x-])

    Accept [y/N]? y
    ------------------------------------------------------------
    Upgrade available on cdn-backend-sunet-01.torproject.org hetzner-hel1-01.torproject.org hetzner-hel1-02.torproject.org hetzner-hel1-03.torproject.org kvm4.torproject.org kvm5.torproject.org listera.torproject.org macrum.torproject.org nutans.torproject.org textile.torproject.org unifolium.torproject.org:

    Reading package lists...
    Building dependency tree...
    Reading state information...
    Calculating upgrade...
    The following packages will be upgraded:
      libssl1.0.2 linux-image-4.9.0-8--x- openssh-client openssh-server
      openssh-sftp-server
    5 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    Inst libssl1.0.2 [1.0.2q-1~deb9u1] (1.0.2r-1~deb9u1 Debian-Security:9/stable [-x-])
    Inst openssh-sftp-server [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-]) []
    Inst openssh-server [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-]) []
    Inst openssh-client [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Inst linux-image-4.9.0-8--x- [4.9.144-3] (4.9.144-3.1 Debian:stable-updates [-x-])
    Conf libssl1.0.2 (1.0.2r-1~deb9u1 Debian-Security:9/stable [-x-])
    Conf openssh-sftp-server (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Conf openssh-server (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Conf openssh-client (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Conf linux-image-4.9.0-8--x- (4.9.144-3.1 Debian:stable-updates [-x-])

    Accept [y/N]? y
    ------------------------------------------------------------
    Upgrade available on eugeni.torproject.org omeiense.torproject.org pauli.torproject.org polyanthum.torproject.org rouyi.torproject.org rude.torproject.org:

    Reading package lists...
    Building dependency tree...
    Reading state information...
    Calculating upgrade...
    The following packages will be upgraded:
      libssl1.0.2 linux-image-4.9.0-8--x- linux-libc-dev openssh-client
      openssh-server openssh-sftp-server ssh
    7 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    Inst openssh-sftp-server [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-]) []
    Inst libssl1.0.2 [1.0.2q-1~deb9u1] (1.0.2r-1~deb9u1 Debian-Security:9/stable [-x-]) []
    Inst openssh-server [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-]) []
    Inst openssh-client [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Inst ssh [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [all])
    Inst linux-image-4.9.0-8--x- [4.9.144-3] (4.9.144-3.1 Debian:stable-updates [-x-])
    Inst linux-libc-dev [4.9.144-3] (4.9.144-3.1 Debian:stable-updates [-x-])
    Conf openssh-sftp-server (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Conf libssl1.0.2 (1.0.2r-1~deb9u1 Debian-Security:9/stable [-x-])
    Conf openssh-server (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Conf openssh-client (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Conf ssh (1:7.4p1-10+deb9u6 Debian-Security:9/stable [all])
    Conf linux-image-4.9.0-8--x- (4.9.144-3.1 Debian:stable-updates [-x-])
    Conf linux-libc-dev (4.9.144-3.1 Debian:stable-updates [-x-])

    Accept [y/N]? y
    ------------------------------------------------------------
    Upgrade available on arlgirdense.torproject.org bracteata.torproject.org build-x86-07.torproject.org build-x86-08.torproject.org build-x86-09.torproject.org carinatum.torproject.org crm-ext-01.torproject.org crm-int-01.torproject.org forrestii.torproject.org gitlab-01.torproject.org neriniflorum.torproject.org opacum.torproject.org perdulce.torproject.org savii.torproject.org saxatile.torproject.org staticiforme.torproject.org subnotabile.torproject.org togashii.torproject.org web-hetzner-01.torproject.org:

    Reading package lists...
    Building dependency tree...
    Reading state information...
    Calculating upgrade...
    The following packages will be upgraded:
      linux-image-4.9.0-8--x-
    1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    Inst linux-image-4.9.0-8--x- [4.9.144-3] (4.9.144-3.1 Debian:stable-updates [-x-])
    Conf linux-image-4.9.0-8--x- (4.9.144-3.1 Debian:stable-updates [-x-])

    Accept [y/N]? y
    ------------------------------------------------------------
    Upgrade available on build-arm-01.torproject.org build-arm-02.torproject.org build-arm-03.torproject.org scw-arm-par-01.torproject.org:

    Reading package lists...
    Building dependency tree...
    Reading state information...
    Calculating upgrade...
    The following packages will be upgraded:
      libssl1.0.2 openssh-client openssh-server openssh-sftp-server ssh
    5 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    Inst openssh-sftp-server [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-]) []
    Inst libssl1.0.2 [1.0.2q-1~deb9u1] (1.0.2r-1~deb9u1 Debian-Security:9/stable [-x-]) []
    Inst openssh-server [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-]) []
    Inst openssh-client [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Inst ssh [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [all])
    Conf openssh-sftp-server (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Conf libssl1.0.2 (1.0.2r-1~deb9u1 Debian-Security:9/stable [-x-])
    Conf openssh-server (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Conf openssh-client (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Conf ssh (1:7.4p1-10+deb9u6 Debian-Security:9/stable [all])

    Accept [y/N]? y
    ------------------------------------------------------------
    Upgrade available on chiwui.torproject.org:

    Reading package lists...
    Building dependency tree...
    Reading state information...
    The following packages will be upgraded:
      bind9-host dnsutils file libbind9-90 libdns-export100 libdns100
      libirs-export91 libisc-export95 libisc95 libisccc90 libisccfg-export90
      libisccfg90 liblwres90 libmagic1 libssl-dev libssl1.0.0 openssl
      qemu-guest-agent
    18 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    Inst libssl-dev [1.0.1t-1+deb8u10] (1.0.1t-1+deb8u11 Debian-Security:8/oldstable [-x-]) []
    Inst libssl1.0.0 [1.0.1t-1+deb8u10] (1.0.1t-1+deb8u11 Debian-Security:8/oldstable [-x-])
    Inst file [1:5.22+15-2+deb8u4] (1:5.22+15-2+deb8u5 Debian-Security:8/oldstable [-x-]) []
    Inst libmagic1 [1:5.22+15-2+deb8u4] (1:5.22+15-2+deb8u5 Debian-Security:8/oldstable [-x-])
    Inst libisc-export95 [1:9.9.5.dfsg-9+deb8u16] (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-])
    Inst libdns-export100 [1:9.9.5.dfsg-9+deb8u16] (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-])
    Inst libisccfg-export90 [1:9.9.5.dfsg-9+deb8u16] (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-])
    Inst libirs-export91 [1:9.9.5.dfsg-9+deb8u16] (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-])
    Inst dnsutils [1:9.9.5.dfsg-9+deb8u16] (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-]) []
    Inst bind9-host [1:9.9.5.dfsg-9+deb8u16] (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-]) []
    Inst libisc95 [1:9.9.5.dfsg-9+deb8u16] (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-]) []
    Inst libdns100 [1:9.9.5.dfsg-9+deb8u16] (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-]) []
    Inst libisccc90 [1:9.9.5.dfsg-9+deb8u16] (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-]) []
    Inst libisccfg90 [1:9.9.5.dfsg-9+deb8u16] (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-]) []
    Inst liblwres90 [1:9.9.5.dfsg-9+deb8u16] (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-]) []
    Inst libbind9-90 [1:9.9.5.dfsg-9+deb8u16] (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-])
    Inst openssl [1.0.1t-1+deb8u10] (1.0.1t-1+deb8u11 Debian-Security:8/oldstable [-x-])
    Inst qemu-guest-agent [1:2.1+dfsg-12+deb8u9] (1:2.1+dfsg-12+deb8u10 Debian-Security:8/oldstable [-x-])
    Conf libssl1.0.0 (1.0.1t-1+deb8u11 Debian-Security:8/oldstable [-x-])
    Conf libssl-dev (1.0.1t-1+deb8u11 Debian-Security:8/oldstable [-x-])
    Conf libmagic1 (1:5.22+15-2+deb8u5 Debian-Security:8/oldstable [-x-])
    Conf file (1:5.22+15-2+deb8u5 Debian-Security:8/oldstable [-x-])
    Conf libisc-export95 (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-])
    Conf libdns-export100 (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-])
    Conf libisccfg-export90 (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-])
    Conf libirs-export91 (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-])
    Conf libisc95 (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-])
    Conf libdns100 (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-])
    Conf libisccc90 (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-])
    Conf libisccfg90 (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-])
    Conf libbind9-90 (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-])
    Conf liblwres90 (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-])
    Conf bind9-host (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-])
    Conf dnsutils (1:9.9.5.dfsg-9+deb8u17 Debian-Security:8/oldstable [-x-])
    Conf openssl (1.0.1t-1+deb8u11 Debian-Security:8/oldstable [-x-])
    Conf qemu-guest-agent (1:2.1+dfsg-12+deb8u10 Debian-Security:8/oldstable [-x-])

    Accept [y/N]? y
    ------------------------------------------------------------
    Upgrade available on nova.torproject.org:

    Reading package lists...
    Building dependency tree...
    Reading state information...
    Calculating upgrade...
    The following packages will be upgraded:
      libssl1.0.2 linux-image-4.9.0-8-686-pae openssh-client openssh-server
      openssh-sftp-server ssh
    6 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    Inst openssh-sftp-server [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-]) []
    Inst libssl1.0.2 [1.0.2q-1~deb9u1] (1.0.2r-1~deb9u1 Debian-Security:9/stable [-x-]) []
    Inst openssh-server [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-]) []
    Inst openssh-client [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Inst ssh [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [all])
    Inst linux-image-4.9.0-8-686-pae [4.9.144-3] (4.9.144-3.1 Debian:stable-updates [-x-])
    Conf openssh-sftp-server (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Conf libssl1.0.2 (1.0.2r-1~deb9u1 Debian-Security:9/stable [-x-])
    Conf openssh-server (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Conf openssh-client (1:7.4p1-10+deb9u6 Debian-Security:9/stable [-x-])
    Conf ssh (1:7.4p1-10+deb9u6 Debian-Security:9/stable [all])
    Conf linux-image-4.9.0-8-686-pae (4.9.144-3.1 Debian:stable-updates [-x-])

    Accept [y/N]? y
    ------------------------------------------------------------
    Upgrade available on crispum.torproject.org oo-hetzner-03.torproject.org oschaninii.torproject.org:
    build-arm-01.torproject.org
    Hit:1 http://security.debian.org stretch/updates InRelease
    Hit:2 https://mirror.netcologne.de/debian stretch-backports InRelease
    Ign:3 https://mirror.netcologne.de/debian stretch InRelease
    Hit:4 https://mirror.netcologne.de/debian stretch-updates InRelease
    Hit:5 https://mirror.netcologne.de/debian stretch Release
    Ign:6 https://db.torproject.org/torproject-admin tpo-all InRelease
    Ign:7 https://db.torproject.org/torproject-admin stretch InRelease
    Hit:8 https://db.torproject.org/torproject-admin tpo-all Release
    Hit:9 https://db.torproject.org/torproject-admin stretch Release
    Hit:10 https://cdn-aws.deb.debian.org/debian stretch-backports InRelease
    Ign:11 https://cdn-aws.deb.debian.org/debian stretch InRelease
    Hit:12 https://cdn-aws.deb.debian.org/debian stretch-updates InRelease
    Hit:13 https://cdn-aws.deb.debian.org/debian stretch Release
    Reading package lists... Done
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    Calculating upgrade... Done
    The following packages will be upgraded:
      libssl1.0.2 openssh-client openssh-server openssh-sftp-server ssh
    5 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    Inst openssh-sftp-server [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [arm64]) []
    Inst libssl1.0.2 [1.0.2q-1~deb9u1] (1.0.2r-1~deb9u1 Debian-Security:9/stable [arm64]) []
    Inst openssh-server [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [arm64]) []
    Inst openssh-client [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [arm64])
    Inst ssh [1:7.4p1-10+deb9u5] (1:7.4p1-10+deb9u6 Debian-Security:9/stable [all])
    Conf openssh-sftp-server (1:7.4p1-10+deb9u6 Debian-Security:9/stable [arm64])
    Conf libssl1.0.2 (1.0.2r-1~deb9u1 Debian-Security:9/stable [arm64])
    Conf openssh-server (1:7.4p1-10+deb9u6 Debian-Security:9/stable [arm64])
    Conf openssh-client (1:7.4p1-10+deb9u6 Debian-Security:9/stable [arm64])
    Conf ssh (1:7.4p1-10+deb9u6 Debian-Security:9/stable [all])
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    Calculating upgrade... Done
    The following packages will be upgraded:
      libssl1.0.2 openssh-client openssh-server openssh-sftp-server ssh
    5 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    Need to get 2125 kB of archives.
    After this operation, 4096 B of additional disk space will be used.
    Get:1 http://security.debian.org stretch/updates/main arm64 openssh-sftp-server arm64 1:7.4p1-10+deb9u6 [34.1 kB]
    Get:2 http://security.debian.org stretch/updates/main arm64 libssl1.0.2 arm64 1.0.2r-1~deb9u1 [913 kB]
    Get:3 http://security.debian.org stretch/updates/main arm64 openssh-server arm64 1:7.4p1-10+deb9u6 [289 kB]
    Get:4 http://security.debian.org stretch/updates/main arm64 openssh-client arm64 1:7.4p1-10+deb9u6 [699 kB]
    Get:5 http://security.debian.org stretch/updates/main arm64 ssh all 1:7.4p1-10+deb9u6 [189 kB]
    Fetched 2125 kB in 1s (1464 kB/s)
    Preconfiguring packages ...
    (Reading database ... 52534 files and directories currently installed.)
    Preparing to unpack .../openssh-sftp-server_1%3a7.4p1-10+deb9u6_arm64.deb ...
    Unpacking openssh-sftp-server (1:7.4p1-10+deb9u6) over (1:7.4p1-10+deb9u5) ...
    Preparing to unpack .../libssl1.0.2_1.0.2r-1~deb9u1_arm64.deb ...
    Unpacking libssl1.0.2:arm64 (1.0.2r-1~deb9u1) over (1.0.2q-1~deb9u1) ...
    Preparing to unpack .../openssh-server_1%3a7.4p1-10+deb9u6_arm64.deb ...
    Unpacking openssh-server (1:7.4p1-10+deb9u6) over (1:7.4p1-10+deb9u5) ...
    Preparing to unpack .../openssh-client_1%3a7.4p1-10+deb9u6_arm64.deb ...
    Conf libssl1.0.2 (1.0.2r-1~deb9u1 Debian-Security:9/stable [amd64])
    Conf openssh-server (1:7.4p1-10+deb9u6 Debian-Security:9/stable [amd64])
    Conf openssh-client (1:7.4p1-10+deb9u6 Debian-Security:9/stable [amd64])
    Conf ssh (1:7.4p1-10+deb9u6 Debian-Security:9/stable [all])
    Conf linux-image-4.9.0-8-amd64 (4.9.144-3.1 Debian:stable-updates [amd64])
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    Calculating upgrade... Done
    The following packages will be upgraded:
      libssl1.0.2 linux-image-4.9.0-8-amd64 openssh-client openssh-server openssh-sftp-server ssh
    6 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    Need to get 41.8 MB of archives.
    After this operation, 4096 B of additional disk space will be used.
    Get:1 http://security.debian.org stretch/updates/main amd64 openssh-sftp-server amd64 1:7.4p1-10+deb9u6 [39.7 kB]
    Get:2 http://security.debian.org stretch/updates/main amd64 libssl1.0.2 amd64 1.0.2r-1~deb9u1 [1302 kB]
    Get:3 http://security.debian.org stretch/updates/main amd64 openssh-server amd64 1:7.4p1-10+deb9u6 [332 kB]
    Get:4 http://security.debian.org stretch/updates/main amd64 openssh-client amd64 1:7.4p1-10+deb9u6 [781 kB]
    Get:5 https://mirrors.wikimedia.org/debian stretch-updates/main amd64 linux-image-4.9.0-8-amd64 amd64 4.9.144-3.1 [39.1 MB]
    Get:6 http://security.debian.org stretch/updates/main amd64 ssh all 1:7.4p1-10+deb9u6 [189 kB]
    Fetched 41.8 MB in 10s (4111 kB/s)                                                                                                                                                                                                                                              
    Preconfiguring packages ...
    (Reading database ... 45757 files and directories currently installed.)
    Preparing to unpack .../0-openssh-sftp-server_1%3a7.4p1-10+deb9u6_amd64.deb ...
    Unpacking openssh-sftp-server (1:7.4p1-10+deb9u6) over (1:7.4p1-10+deb9u5) ...
    Preparing to unpack .../1-libssl1.0.2_1.0.2r-1~deb9u1_amd64.deb ...
    Unpacking libssl1.0.2:amd64 (1.0.2r-1~deb9u1) over (1.0.2q-1~deb9u1) ...
    Preparing to unpack .../2-openssh-server_1%3a7.4p1-10+deb9u6_amd64.deb ...
    Unpacking openssh-server (1:7.4p1-10+deb9u6) over (1:7.4p1-10+deb9u5) ...
    Preparing to unpack .../3-openssh-client_1%3a7.4p1-10+deb9u6_amd64.deb ...
    Unpacking openssh-client (1:7.4p1-10+deb9u6) over (1:7.4p1-10+deb9u5) ...
    Preparing to unpack .../4-ssh_1%3a7.4p1-10+deb9u6_all.deb ...
    Unpacking ssh (1:7.4p1-10+deb9u6) over (1:7.4p1-10+deb9u5) ...
    Preparing to unpack .../5-linux-image-4.9.0-8-amd64_4.9.144-3.1_amd64.deb ...
    Unpacking linux-image-4.9.0-8-amd64 (4.9.144-3.1) over (4.9.144-3) ...
    Setting up linux-image-4.9.0-8-amd64 (4.9.144-3.1) ...
    /etc/kernel/postinst.d/initramfs-tools:
    update-initramfs: Generating /boot/initrd.img-4.9.0-8-amd64
    I: The initramfs will attempt to resume from /dev/sdc
    I: (UUID=4e725edc-e3df-4122-89aa-19ce43ec9a0e)
    I: Set the RESUME variable to override this.
    Inst linux-image-4.9.0-8-amd64 [4.9.144-3] (4.9.144-3.1 Debian:stable-updates [amd64])
    Inst linux-libc-dev [4.9.144-3] (4.9.144-3.1 Debian:stable-updates [amd64])
    Conf openssh-sftp-server (1:7.4p1-10+deb9u6 Debian-Security:9/stable [amd64])
    Processing triggers for libc-bin (2.24-11+deb9u4) ...
    Processing triggers for systemd (232-25+deb9u9) ...
    Processing triggers for man-db (2.7.6.1-2) ...
    Setting up openssh-client (1:7.4p1-10+deb9u6) ...
    Setting up openssh-sftp-server (1:7.4p1-10+deb9u6) ...
    Setting up openssh-server (1:7.4p1-10+deb9u6) ...
    Setting up ssh (1:7.4p1-10+deb9u6) ...
    [master 4f397dc] committing changes in /etc after apt run
     Committer: root <root@peninsulare.torproject.org>
    Unpacking linux-image-4.9.0-8-amd64 (4.9.144-3.1) over (4.9.144-3) ...
    [master a58f4a8] committing changes in /etc after apt run

    Reading package lists...
    Building dependency tree...
    Reading state information...
    Calculating upgrade...
    The following packages will be upgraded:
      linux-image-4.9.0-8--x- linux-libc-dev
    2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    Inst linux-image-4.9.0-8--x- [4.9.144-3] (4.9.144-3.1 Debian:stable-updates [-x-])
    Inst linux-libc-dev [4.9.144-3] (4.9.144-3.1 Debian:stable-updates [-x-])
    Conf linux-image-4.9.0-8--x- (4.9.144-3.1 Debian:stable-updates [-x-])
    Conf linux-libc-dev (4.9.144-3.1 Debian:stable-updates [-x-])

    Accept [y/N]? y
    ============================================================
    Failed:  build-x86-05.torproject.org build-x86-06.torproject.org dictyotum.torproject.org fallax.torproject.org getulum.torproject.org geyeri.torproject.org gillii.torproject.org hedgei.torproject.org majus.torproject.org moly.torproject.org peninsulare.torproject.org web-cymru-01.torproject.org
    No updates on:
    Accepted changes:
      torproject-upgrade '180463a1d97794d04af05ba99ff0dabd2c5648c4|29af4f3d269e7a8ddc22fc2d2d970c70a373d9e8|36d9a3e4d8c4b58c8e8b325022afa78573ac2667|42bec5e8be9279422bf3b5dc704f4f077c597099|83b56903918edad25655a7c85d5dc12448e1619b|eb227197d16c5e1a613a00a5e5abcb817a0ec65d|ec52867d0041b23a27393ee5afa3b45df2e3b4b9|ed456dc5ed1c12d7024644af5aac54c9370b7466|fe78acc5ff2d634eabf7676c6bcd60f248e7b610'
    weasel@orinoco:~$   torproject-upgrade '180463a1d97794d04af05ba99ff0dabd2c5648c4|29af4f3d269e7a8ddc22fc2d2d970c70a373d9e8|36d9a3e4d8c4b58c8e8b325022afa78573ac2667|42bec5e8be9279422bf3b5dc704f4f077c597099|83b56903918edad25655a7c85d5dc12448e1619b|eb227197d16c5e1a613a00a5e5abcb817a0ec65d|ec52867d0041b23a27393ee5afa3b45df2e3b4b9|ed456dc5ed1c12d7024644af5aac54c9370b7466|fe78acc5ff2d634eabf7676c6bcd60f248e7b610'
    [exited]
