Skip to content

apt_upgrades_pending metric incorrect on gitlab-02

This is very minor, but the apt_upgrades_pending metric is actually incorrect on gitlab-02. The apt_info.py collector claims this:

root@gitlab-02:~# /usr/share/prometheus-node-exporter-collectors/apt_info.py
# HELP apt_upgrades_pending Apt packages pending updates by origin.
# TYPE apt_upgrades_pending gauge
apt_upgrades_pending{origin="",arch=""} 0
# HELP apt_upgrades_held Apt packages pending updates but held back.
# TYPE apt_upgrades_held gauge
apt_upgrades_held{origin="packages.gitlab.com/gitlab/gitlab-ce:bookworm/bookworm",arch="amd64"} 1
# HELP apt_autoremove_pending Apt packages pending autoremoval.
# TYPE apt_autoremove_pending gauge
apt_autoremove_pending 21
# HELP apt_package_cache_timestamp_seconds Apt update last run time.
# TYPE apt_package_cache_timestamp_seconds gauge
apt_package_cache_timestamp_seconds 1726761888.3600302
# HELP node_reboot_required Node reboot is required for software updates.
# TYPE node_reboot_required gauge
node_reboot_required 0

in other words, no pending package, one held package. But that's not entirely accurate. Here's what apt-cache policy thinks:

root@gitlab-02:~# apt-cache policy gitlab-ce| head
gitlab-ce:
  Installed: 17.3.3-ce.0
  Candidate: 17.4.0-ce.0
  Version table:
     17.4.0-ce.0 501
        500 https://packages.gitlab.com/gitlab/gitlab-ce/debian bookworm/main amd64 Packages
 *** 17.3.3-ce.0 501
        500 https://packages.gitlab.com/gitlab/gitlab-ce/debian bookworm/main amd64 Packages
        100 /var/lib/dpkg/status

that's not entirely accurate either: apt-cache policy notoriously ignores pinned packages, and this package is pinned. Except it's pinned by major version:

root@gitlab-02:~# cat /etc/apt/preferences.d/gitlab-ce.pref 
# This file is managed by Puppet. DO NOT EDIT.
Explanation: pin GitLab to major version '17.x', see tpo/tpa/team#40769
Package: gitlab-ce
Pin: version 17.*
Pin-Priority: 501

And indeed, unattended-upgrades will upgrade this package soon:

root@gitlab-02:~# unattended-upgrade --dry-run -v
Checking if system is running on battery is skipped. Please install powermgmt-base package to check power status and skip installing updates when the system is running on battery.
Checking if connection is metered is skipped. Please install python3-gi package to detect metered connections and skip downloading updates.
Starting unattended upgrades script
Allowed origins are: origin=*
Initial blacklist: gitlab-runner openvswitch-switch openvswitch-common
Initial whitelist (not strict): 
Option --dry-run given, *not* performing real actions
Packages that will be upgraded: gitlab-ce
Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
/usr/bin/dpkg --force-confdef --force-confold --force-confdef --force-confold --status-fd 10 --no-triggers --unpack --auto-deconfigure /var/cache/apt/archives/gitlab-ce_17.4.0-ce.0_amd64.deb 
/usr/bin/dpkg --force-confdef --force-confold --force-confdef --force-confold --status-fd 10 --configure --pending 
All upgrades installed
The list of kept packages can't be calculated in dry-run mode.

So i think this is yet another bug in apt_info.py... it's relatively minor because it does mark the package as held, at least, but it's definitely inaccurate.