... | ... | @@ -857,3 +857,52 @@ procedure ramps up. The following graph will be converted into a |
|
|
Grafana dashboard to fix that, see [issue 40512](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40512).
|
|
|
</figcaption>
|
|
|
</figure>
|
|
|
|
|
|
# Post-mortem
|
|
|
|
|
|
Note that the approach taken for bullseye was to "do the right thing"
|
|
|
on many fronts, for example:
|
|
|
|
|
|
* for Icinga, we entered into a discussion about replacing it with
|
|
|
Prometheus
|
|
|
* for the Sunet cluster, we waited to rebuild the VMs in a new
|
|
|
location
|
|
|
* for Puppet, we actually updated the Debian packaging, even though
|
|
|
that was going to be only usable in bookworm
|
|
|
* for gitolite/gitweb, we proposed a retirement instead
|
|
|
|
|
|
This wasn't the case for all servers, for example we just upgraded
|
|
|
gayi and did not wait for the SVN retirement. But in general, this
|
|
|
upgrade dragged on longer than the previous jessie to buster
|
|
|
upgrade.
|
|
|
|
|
|
This can be seen in the following all-time upgrade graph:
|
|
|
|
|
|
<img alt="graph showing the number of hosts per Debian release over time" src="/howto/upgrades/data.png" />
|
|
|
|
|
|
Here we see the buster upgrades we performed over a little over 14
|
|
|
months with a very long tail of 3 machines upgraded over another 14
|
|
|
months or so.
|
|
|
|
|
|
In comparison, the bulk of the bullseye upgrades were faster (10
|
|
|
months!) but then stalled at 12 machines for 10 more months. In terms
|
|
|
of machines*time product, it's worse as we had 10 outdated machines
|
|
|
over 12 months as opposed to 3 over 14 months... And it's not over
|
|
|
yet.
|
|
|
|
|
|
That said, the time between the min and the max for bullseye was
|
|
|
*much* shorter than buster. Taken this way, we could count the upgrade
|
|
|
as:
|
|
|
|
|
|
| suite | start | end | diff |
|
|
|
|----------|------------|------------|-----------|
|
|
|
| buster | 2019-03-01 | 2020-11-01 | 20 months |
|
|
|
| bullseye | 2021-08-01 | 2022-07-01 | 12 months |
|
|
|
|
|
|
In both cases, machines from the previous release remained to be
|
|
|
upgraded, but the bulk of the machines was upgraded quickly, which is
|
|
|
a testament to the "batch" system that was adopted for the bullseye
|
|
|
upgrade.
|
|
|
|
|
|
In this upgrade phase, we also hope to have lesser time with three
|
|
|
suites to maintain at once, but that remains to be confirmed. |