Skip to content

prometheus should monitor for obsolete packages (warning)

in #41639 (closed), we have all sorts of metrics deployed, but we somehow missed the important dsa-check-packages script that's hooked into nagios.

there's some of those metrics available thanks to the apt_info.py script in prometheus-node-exporter-collectors, but it doesn't have all of it. namely, it has:

# /usr/share/prometheus-node-exporter-collectors/apt_info.py
# HELP apt_upgrades_pending Apt packages pending updates by origin.
# TYPE apt_upgrades_pending gauge
apt_upgrades_pending{origin="",arch=""} 0
# HELP apt_upgrades_held Apt packages pending updates but held back.
# TYPE apt_upgrades_held gauge
apt_upgrades_held{origin="",arch=""} 0
# HELP apt_autoremove_pending Apt packages pending autoremoval.
# TYPE apt_autoremove_pending gauge
apt_autoremove_pending 21
# HELP node_reboot_required Node reboot is required for software updates.
# TYPE node_reboot_required gauge
node_reboot_required 0

dsa-check-packages has this:

# dsa-check-packages 
OK: 618 ok, 4 rc
618 packages current.
4 packages removed but not purged: linux-image-6.1.0-18-amd64, linux-image-6.1.0-17-amd64, linux-image-6.1.0-21-amd64, linux-image-6.1.0-20-amd64
|obs_loc=0;1;5;0 outdated=0;1;5;0 current=618;;;0 obs_ign=0;;;0 rm_unprg=4;;;0 hold=0;;;0 prg_conf=0;1;;0

that is:

  • held packages (apt_upgrades_held)
  • obsolete packages, including a way to exclude packages (not covered!)
  • outdated packages (apt_upgrades_pending)
  • current packages (not covered!)
  • removed packages with leftover configuration ("removed but not purged", not covered)
  • purged packages with leftove configuration (!?) (not covered)
  • pending upgrades but that apt upgrade won't upgrade (see for example Ubuntu phased upgrades or pinned packages?) (prometheus-alerts#19 (closed))
  • add an ignore list

the "obsolete packages" is the most important, as those are packages that might not be covered by security upgrade and a manual configuration...

all of those should be warnings, of course.

we should make this as a standalone script outside of apt_info.py at first considering the time it took to fix that thing, but see also #41355 (closed).

Edited by lelutin
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information