playbook, pending upgrades: present the query that's more useful first authored by lelutin's avatar lelutin
when we reach this playbook section, we usually want to check what's up
with ongoing alerts. So the default query which shows a bunch of results
unrelated to the alerts is very distracting.
......@@ -154,14 +154,7 @@ automatically.
The list of affected hosts and packages can be inspected with the
following [fabric](howto/fabric) command:
fab -H pauli.torproject.org fleet.pending-upgrades
Note that this will *also* catch hosts that have pending upgrade that
*may* be upgraded automatically by unattended-upgrades, as it doesn't
check for alerts, but for the metric directly. You can use the
`--query` parameter to restrict to the alerting hosts instead:
fab -H pauli.torproject.org fleet.pending-upgrades --query='ALERTS{alertname="PackagesPendingTooLong",alertstate="firing"}'
fab fleet.pending-upgrades --query='ALERTS{alertname="PackagesPendingTooLong",alertstate="firing"}'
Look at the list of packages to be upgraded, and consider upgrading
them manually, with Cumin (see below), or individually, by logging
......@@ -172,6 +165,16 @@ the timer `prometheus-node-exporter-apt.timer` triggers. It currently runs every
15 minutes, so it's probably not necessary to trigger it by hand to speed things
up.
Alternatively, if you would like to list pending packages from all hosts, and
not just the ones that triggered an alert, you can use the `--query` parameter
to restrict to the alerting hosts instead:
fab fleet.pending-upgrades
Note that this will *also* catch hosts that have pending upgrade that
*may* be upgraded automatically by unattended-upgrades, as it doesn't
check for alerts, but for the metric directly.
## Obsolete packages
<!-- note that this section is cross-referenced from the -->
......
......