Changes

Page history
remove one useless heading level authored by anarcat's avatar anarcat
Show whitespace changes
Inline Side-by-side
howto/upgrades.md
View page @ b2bc790e
[[_TOC_]] [[_TOC_]]
# Debian upgrades
## Major upgrades # Major upgrades
Major upgrades are done by hand, with a "cheat sheet" created for each Major upgrades are done by hand, with a "cheat sheet" created for each
major release. Here are the currently documented ones: major release. Here are the currently documented ones:
...@@ -10,7 +9,7 @@ major release. Here are the currently documented ones: ...@@ -10,7 +9,7 @@ major release. Here are the currently documented ones:
* Debian 11, [bullseye](howto/upgrades/bullseye) * Debian 11, [bullseye](howto/upgrades/bullseye)
* Debian 10, [buster](howto/upgrades/buster) * Debian 10, [buster](howto/upgrades/buster)
### Team-specific upgrade policies ## Team-specific upgrade policies
Before we perform a major upgrade, it might be advisable to consult Before we perform a major upgrade, it might be advisable to consult
with the team working on the box to see if it will interfere for their with the team working on the box to see if it will interfere for their
...@@ -29,9 +28,9 @@ Team policies: ...@@ -29,9 +28,9 @@ Team policies:
Some teams might be missing from the list. Some teams might be missing from the list.
## Minor upgrades # Minor upgrades
### Unattended upgrades ## Unattended upgrades
Most of the packages upgrades are handled by the unattended-upgrades package which Most of the packages upgrades are handled by the unattended-upgrades package which
is configured via puppet. is configured via puppet.
...@@ -54,7 +53,7 @@ that new `sources.list` entries be paired with a "pin" (see ...@@ -54,7 +53,7 @@ that new `sources.list` entries be paired with a "pin" (see
[apt_preferences(5)](https://manpages.debian.org/apt_preferences.5)). See also [tpo/tpa/team#40771](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40771) for a [apt_preferences(5)](https://manpages.debian.org/apt_preferences.5)). See also [tpo/tpa/team#40771](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40771) for a
discussion and rationale of that change. discussion and rationale of that change.
### Manual upgrades with Cumin ## Manual upgrades with Cumin
It's also possible to do a manual mass-upgrade run with It's also possible to do a manual mass-upgrade run with
[Cumin](howto/cumin): [Cumin](howto/cumin):
...@@ -69,7 +68,7 @@ block certain upgrades. If you want to bypass that, use regular `apt`: ...@@ -69,7 +68,7 @@ block certain upgrades. If you want to bypass that, use regular `apt`:
cumin -b 10 '*' 'apt update ; apt upgrade -yy ; TERM=doit dsa-update-apt-status' cumin -b 10 '*' 'apt update ; apt upgrade -yy ; TERM=doit dsa-update-apt-status'
### GitLab runner upgrades ## GitLab runner upgrades
Every month or so GitLab publishes a update to the `gitlab-runner` apt Every month or so GitLab publishes a update to the `gitlab-runner` apt
package. The package is excluded from `unattended-upgrades` to avoid any package. The package is excluded from `unattended-upgrades` to avoid any
...@@ -81,7 +80,7 @@ shadow sims are being executed, and launch `apt upgrade`. If any regular ...@@ -81,7 +80,7 @@ shadow sims are being executed, and launch `apt upgrade`. If any regular
CI jobs are running, systemd will wait up to one hour for them to end, CI jobs are running, systemd will wait up to one hour for them to end,
then proceed with the package upgrade. then proceed with the package upgrade.
### Restarting services by hand ## Restarting services by hand
After upgrades, there's a Nagios check that might trigger and tell you After upgrades, there's a Nagios check that might trigger and tell you
that some services are running with outdated libraries. Normally, that some services are running with outdated libraries. Normally,
...@@ -157,7 +156,7 @@ Services setup with the new systemd-based startup system documented in ...@@ -157,7 +156,7 @@ Services setup with the new systemd-based startup system documented in
There's a feature request ([bug #843778](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=843778)) to implement support for There's a feature request ([bug #843778](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=843778)) to implement support for
those services directly in needrestart. those services directly in needrestart.
### Kernel upgrades and reboots ## Kernel upgrades and reboots
Sometimes it is necessary to perform a reboot on the hosts, when the Sometimes it is necessary to perform a reboot on the hosts, when the
kernel is updated. Nagios will warn about this, with something like kernel is updated. Nagios will warn about this, with something like
...@@ -165,7 +164,7 @@ this: ...@@ -165,7 +164,7 @@ this:
WARNING: Kernel needs upgrade [linux-image-4.9.0-9-amd64 != linux-image-4.9.0-8-amd64] WARNING: Kernel needs upgrade [linux-image-4.9.0-9-amd64 != linux-image-4.9.0-8-amd64]
#### Rebooting guests ### Rebooting guests
If this is only a virtual machine, and the only one affected, it can If this is only a virtual machine, and the only one affected, it can
be rebooted directly. This can be done with the `tsa-misc` script be rebooted directly. This can be done with the `tsa-misc` script
...@@ -196,23 +195,23 @@ defined to `justdoit` or `rotation`: ...@@ -196,23 +195,23 @@ defined to `justdoit` or `rotation`:
echo "rebooting 'rotation' hosts with a 10-minute delay, every 30 minutes...." echo "rebooting 'rotation' hosts with a 10-minute delay, every 30 minutes...."
./reboot -H $(ssh db.torproject.org 'ldapsearch -h db.torproject.org -x -ZZ -b ou=hosts,dc=torproject,dc=org -LLL "(rebootPolicy=rotation)" hostname | awk "\$1 == \"hostname:\" {print \$2}" | sort -R') --delay-shutdown=10 --delay-hosts=1800 -v ./reboot -H $(ssh db.torproject.org 'ldapsearch -h db.torproject.org -x -ZZ -b ou=hosts,dc=torproject,dc=org -LLL "(rebootPolicy=rotation)" hostname | awk "\$1 == \"hostname:\" {print \$2}" | sort -R') --delay-shutdown=10 --delay-hosts=1800 -v
### Rebooting KVM hosts ## Rebooting KVM hosts
The remaining is the "manual" procedure, the KVM hosts: The remaining is the "manual" procedure, the KVM hosts:
./reboot-host moly.torproject.org ./reboot-host moly.torproject.org
### Rebooting Ganeti nodes ## Rebooting Ganeti nodes
See the [Ganeti reboot procedures](howto/ganeti#rebooting) for this See the [Ganeti reboot procedures](howto/ganeti#rebooting) for this
procedure. procedure.
### Remaining nodes ## Remaining nodes
The [Nagios unhandled problems](https://nagios.torproject.org/cgi-bin/icinga/status.cgi?allunhandledproblems) will show remaining hosts that The [Nagios unhandled problems](https://nagios.torproject.org/cgi-bin/icinga/status.cgi?allunhandledproblems) will show remaining hosts that
might have been missed by the above procedure.. might have been missed by the above procedure..
#### Generic upgrade routines ### Generic upgrade routines
LDAP hosts have information about how they can be rebooted, in the LDAP hosts have information about how they can be rebooted, in the
`rebootPolicy` field. Here are what the various fields mean: `rebootPolicy` field. Here are what the various fields mean:
... ...
......