|
[[_TOC_]]
|
|
[[_TOC_]]
|
|
|
|
|
|
# Debian upgrades
|
|
|
|
|
|
|
|
## Major upgrades
|
|
# Major upgrades
|
|
|
|
|
|
Major upgrades are done by hand, with a "cheat sheet" created for each
|
|
Major upgrades are done by hand, with a "cheat sheet" created for each
|
|
major release. Here are the currently documented ones:
|
|
major release. Here are the currently documented ones:
|
... | @@ -10,7 +9,7 @@ major release. Here are the currently documented ones: |
... | @@ -10,7 +9,7 @@ major release. Here are the currently documented ones: |
|
* Debian 11, [bullseye](howto/upgrades/bullseye)
|
|
* Debian 11, [bullseye](howto/upgrades/bullseye)
|
|
* Debian 10, [buster](howto/upgrades/buster)
|
|
* Debian 10, [buster](howto/upgrades/buster)
|
|
|
|
|
|
### Team-specific upgrade policies
|
|
## Team-specific upgrade policies
|
|
|
|
|
|
Before we perform a major upgrade, it might be advisable to consult
|
|
Before we perform a major upgrade, it might be advisable to consult
|
|
with the team working on the box to see if it will interfere for their
|
|
with the team working on the box to see if it will interfere for their
|
... | @@ -29,9 +28,9 @@ Team policies: |
... | @@ -29,9 +28,9 @@ Team policies: |
|
|
|
|
|
Some teams might be missing from the list.
|
|
Some teams might be missing from the list.
|
|
|
|
|
|
## Minor upgrades
|
|
# Minor upgrades
|
|
|
|
|
|
### Unattended upgrades
|
|
## Unattended upgrades
|
|
|
|
|
|
Most of the packages upgrades are handled by the unattended-upgrades package which
|
|
Most of the packages upgrades are handled by the unattended-upgrades package which
|
|
is configured via puppet.
|
|
is configured via puppet.
|
... | @@ -54,7 +53,7 @@ that new `sources.list` entries be paired with a "pin" (see |
... | @@ -54,7 +53,7 @@ that new `sources.list` entries be paired with a "pin" (see |
|
[apt_preferences(5)](https://manpages.debian.org/apt_preferences.5)). See also [tpo/tpa/team#40771](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40771) for a
|
|
[apt_preferences(5)](https://manpages.debian.org/apt_preferences.5)). See also [tpo/tpa/team#40771](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40771) for a
|
|
discussion and rationale of that change.
|
|
discussion and rationale of that change.
|
|
|
|
|
|
### Manual upgrades with Cumin
|
|
## Manual upgrades with Cumin
|
|
|
|
|
|
It's also possible to do a manual mass-upgrade run with
|
|
It's also possible to do a manual mass-upgrade run with
|
|
[Cumin](howto/cumin):
|
|
[Cumin](howto/cumin):
|
... | @@ -69,7 +68,7 @@ block certain upgrades. If you want to bypass that, use regular `apt`: |
... | @@ -69,7 +68,7 @@ block certain upgrades. If you want to bypass that, use regular `apt`: |
|
|
|
|
|
cumin -b 10 '*' 'apt update ; apt upgrade -yy ; TERM=doit dsa-update-apt-status'
|
|
cumin -b 10 '*' 'apt update ; apt upgrade -yy ; TERM=doit dsa-update-apt-status'
|
|
|
|
|
|
### GitLab runner upgrades
|
|
## GitLab runner upgrades
|
|
|
|
|
|
Every month or so GitLab publishes a update to the `gitlab-runner` apt
|
|
Every month or so GitLab publishes a update to the `gitlab-runner` apt
|
|
package. The package is excluded from `unattended-upgrades` to avoid any
|
|
package. The package is excluded from `unattended-upgrades` to avoid any
|
... | @@ -81,7 +80,7 @@ shadow sims are being executed, and launch `apt upgrade`. If any regular |
... | @@ -81,7 +80,7 @@ shadow sims are being executed, and launch `apt upgrade`. If any regular |
|
CI jobs are running, systemd will wait up to one hour for them to end,
|
|
CI jobs are running, systemd will wait up to one hour for them to end,
|
|
then proceed with the package upgrade.
|
|
then proceed with the package upgrade.
|
|
|
|
|
|
### Restarting services by hand
|
|
## Restarting services by hand
|
|
|
|
|
|
After upgrades, there's a Nagios check that might trigger and tell you
|
|
After upgrades, there's a Nagios check that might trigger and tell you
|
|
that some services are running with outdated libraries. Normally,
|
|
that some services are running with outdated libraries. Normally,
|
... | @@ -157,7 +156,7 @@ Services setup with the new systemd-based startup system documented in |
... | @@ -157,7 +156,7 @@ Services setup with the new systemd-based startup system documented in |
|
There's a feature request ([bug #843778](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=843778)) to implement support for
|
|
There's a feature request ([bug #843778](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=843778)) to implement support for
|
|
those services directly in needrestart.
|
|
those services directly in needrestart.
|
|
|
|
|
|
### Kernel upgrades and reboots
|
|
## Kernel upgrades and reboots
|
|
|
|
|
|
Sometimes it is necessary to perform a reboot on the hosts, when the
|
|
Sometimes it is necessary to perform a reboot on the hosts, when the
|
|
kernel is updated. Nagios will warn about this, with something like
|
|
kernel is updated. Nagios will warn about this, with something like
|
... | @@ -165,7 +164,7 @@ this: |
... | @@ -165,7 +164,7 @@ this: |
|
|
|
|
|
WARNING: Kernel needs upgrade [linux-image-4.9.0-9-amd64 != linux-image-4.9.0-8-amd64]
|
|
WARNING: Kernel needs upgrade [linux-image-4.9.0-9-amd64 != linux-image-4.9.0-8-amd64]
|
|
|
|
|
|
#### Rebooting guests
|
|
### Rebooting guests
|
|
|
|
|
|
If this is only a virtual machine, and the only one affected, it can
|
|
If this is only a virtual machine, and the only one affected, it can
|
|
be rebooted directly. This can be done with the `tsa-misc` script
|
|
be rebooted directly. This can be done with the `tsa-misc` script
|
... | @@ -196,23 +195,23 @@ defined to `justdoit` or `rotation`: |
... | @@ -196,23 +195,23 @@ defined to `justdoit` or `rotation`: |
|
echo "rebooting 'rotation' hosts with a 10-minute delay, every 30 minutes...."
|
|
echo "rebooting 'rotation' hosts with a 10-minute delay, every 30 minutes...."
|
|
./reboot -H $(ssh db.torproject.org 'ldapsearch -h db.torproject.org -x -ZZ -b ou=hosts,dc=torproject,dc=org -LLL "(rebootPolicy=rotation)" hostname | awk "\$1 == \"hostname:\" {print \$2}" | sort -R') --delay-shutdown=10 --delay-hosts=1800 -v
|
|
./reboot -H $(ssh db.torproject.org 'ldapsearch -h db.torproject.org -x -ZZ -b ou=hosts,dc=torproject,dc=org -LLL "(rebootPolicy=rotation)" hostname | awk "\$1 == \"hostname:\" {print \$2}" | sort -R') --delay-shutdown=10 --delay-hosts=1800 -v
|
|
|
|
|
|
### Rebooting KVM hosts
|
|
## Rebooting KVM hosts
|
|
|
|
|
|
The remaining is the "manual" procedure, the KVM hosts:
|
|
The remaining is the "manual" procedure, the KVM hosts:
|
|
|
|
|
|
./reboot-host moly.torproject.org
|
|
./reboot-host moly.torproject.org
|
|
|
|
|
|
### Rebooting Ganeti nodes
|
|
## Rebooting Ganeti nodes
|
|
|
|
|
|
See the [Ganeti reboot procedures](howto/ganeti#rebooting) for this
|
|
See the [Ganeti reboot procedures](howto/ganeti#rebooting) for this
|
|
procedure.
|
|
procedure.
|
|
|
|
|
|
### Remaining nodes
|
|
## Remaining nodes
|
|
|
|
|
|
The [Nagios unhandled problems](https://nagios.torproject.org/cgi-bin/icinga/status.cgi?allunhandledproblems) will show remaining hosts that
|
|
The [Nagios unhandled problems](https://nagios.torproject.org/cgi-bin/icinga/status.cgi?allunhandledproblems) will show remaining hosts that
|
|
might have been missed by the above procedure..
|
|
might have been missed by the above procedure..
|
|
|
|
|
|
#### Generic upgrade routines
|
|
### Generic upgrade routines
|
|
|
|
|
|
LDAP hosts have information about how they can be rebooted, in the
|
|
LDAP hosts have information about how they can be rebooted, in the
|
|
`rebootPolicy` field. Here are what the various fields mean:
|
|
`rebootPolicy` field. Here are what the various fields mean:
|
... | | ... | |