remove one useless heading level authored by anarcat's avatar anarcat
[[_TOC_]]
# Debian upgrades
## Major upgrades
# Major upgrades
Major upgrades are done by hand, with a "cheat sheet" created for each
major release. Here are the currently documented ones:
......@@ -10,7 +9,7 @@ major release. Here are the currently documented ones:
* Debian 11, [bullseye](howto/upgrades/bullseye)
* Debian 10, [buster](howto/upgrades/buster)
### Team-specific upgrade policies
## Team-specific upgrade policies
Before we perform a major upgrade, it might be advisable to consult
with the team working on the box to see if it will interfere for their
......@@ -29,9 +28,9 @@ Team policies:
Some teams might be missing from the list.
## Minor upgrades
# Minor upgrades
### Unattended upgrades
## Unattended upgrades
Most of the packages upgrades are handled by the unattended-upgrades package which
is configured via puppet.
......@@ -54,7 +53,7 @@ that new `sources.list` entries be paired with a "pin" (see
[apt_preferences(5)](https://manpages.debian.org/apt_preferences.5)). See also [tpo/tpa/team#40771](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40771) for a
discussion and rationale of that change.
### Manual upgrades with Cumin
## Manual upgrades with Cumin
It's also possible to do a manual mass-upgrade run with
[Cumin](howto/cumin):
......@@ -69,7 +68,7 @@ block certain upgrades. If you want to bypass that, use regular `apt`:
cumin -b 10 '*' 'apt update ; apt upgrade -yy ; TERM=doit dsa-update-apt-status'
### GitLab runner upgrades
## GitLab runner upgrades
Every month or so GitLab publishes a update to the `gitlab-runner` apt
package. The package is excluded from `unattended-upgrades` to avoid any
......@@ -81,7 +80,7 @@ shadow sims are being executed, and launch `apt upgrade`. If any regular
CI jobs are running, systemd will wait up to one hour for them to end,
then proceed with the package upgrade.
### Restarting services by hand
## Restarting services by hand
After upgrades, there's a Nagios check that might trigger and tell you
that some services are running with outdated libraries. Normally,
......@@ -157,7 +156,7 @@ Services setup with the new systemd-based startup system documented in
There's a feature request ([bug #843778](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=843778)) to implement support for
those services directly in needrestart.
### Kernel upgrades and reboots
## Kernel upgrades and reboots
Sometimes it is necessary to perform a reboot on the hosts, when the
kernel is updated. Nagios will warn about this, with something like
......@@ -165,7 +164,7 @@ this:
WARNING: Kernel needs upgrade [linux-image-4.9.0-9-amd64 != linux-image-4.9.0-8-amd64]
#### Rebooting guests
### Rebooting guests
If this is only a virtual machine, and the only one affected, it can
be rebooted directly. This can be done with the `tsa-misc` script
......@@ -196,23 +195,23 @@ defined to `justdoit` or `rotation`:
echo "rebooting 'rotation' hosts with a 10-minute delay, every 30 minutes...."
./reboot -H $(ssh db.torproject.org 'ldapsearch -h db.torproject.org -x -ZZ -b ou=hosts,dc=torproject,dc=org -LLL "(rebootPolicy=rotation)" hostname | awk "\$1 == \"hostname:\" {print \$2}" | sort -R') --delay-shutdown=10 --delay-hosts=1800 -v
### Rebooting KVM hosts
## Rebooting KVM hosts
The remaining is the "manual" procedure, the KVM hosts:
./reboot-host moly.torproject.org
### Rebooting Ganeti nodes
## Rebooting Ganeti nodes
See the [Ganeti reboot procedures](howto/ganeti#rebooting) for this
procedure.
### Remaining nodes
## Remaining nodes
The [Nagios unhandled problems](https://nagios.torproject.org/cgi-bin/icinga/status.cgi?allunhandledproblems) will show remaining hosts that
might have been missed by the above procedure..
#### Generic upgrade routines
### Generic upgrade routines
LDAP hosts have information about how they can be rebooted, in the
`rebootPolicy` field. Here are what the various fields mean:
......
......