Skip to content
Snippets Groups Projects
New look: Off

change the unattended-upgrades policy to delegate to apt_preferences

    • Closed Issue created by anarcat @anarcat

    in #40758 (closed), we need to tell unattended-upgrades to upgrade tor and tor-geoipdb to follow bullseye-backports.

    but the Unattended-Upgrade::Origins-Pattern setting we have forbids this explicitly. we don't necessarily want to allow upgrades from backports in all hosts, so a solution might be to start having that setting be customizable in the class. but then we need to remember to add that setting in Hiera, which is error prone.

    the solution i'm thinking of is, instead, to stop pretending the origins-pattern can save our backs, and embrace apt pinning instead. this involves setting Unattended-Upgrade::Origins-Pattern to * to allow any origin to upgrade packages in u-u. this doesn't actually mean it will upgrade everything to whatever: u-u still respects apt pinning. this policy is actually described in the u-u README file:

    If you already configure what to install via apt pinning, you can simply use "origin=*", e.g.:

     Unattended-Upgrade::Origins-Pattern {
        "origin=*";
 };

    also, relying on an unattended-upgrades specific hack to keep some upgrades from going through can backfire when we run apt upgrade manually, for example. furthermore, we don't want to have to remember to specify those hacks in two places. an example of this problem is @lavamind's attempt at pinning onionbalance to bullseye-backports in Puppet:

    commit 502e855a066230e0b02a4ff9fc040cf709dfa65f
Author: Jérôme Charaoui <jerome@riseup.net>
AuthorDate: Tue May 24 09:03:08 2022 -0400

    add onionbalance pin for bullseye-backports
---
 modules/onion/manifests/balancev3.pp | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/modules/onion/manifests/balancev3.pp b/modules/onion/manifests/balancev3.pp
index 0f56478f..dc38a801 100644
[ 0001-branch-master-updated-add-onionbalance-pin-for-bulls.patch: inline patch (as text/x-diff) ]
--- a/modules/onion/manifests/balancev3.pp
+++ b/modules/onion/manifests/balancev3.pp
@@ -7,6 +7,19 @@ class onion::balancev3(
   package { 'onionbalance':
     ensure => installed,
   }
+
+  apt::pin { 'onionbalance':
+    ensure      => $facts['os']['release']['major'] == '11' ? {
+      true  => 'present',
+      false => 'absent',
+    },
+    explanation => 'version >= 2.2 is needed',
+    packages    => ['onionbalance'],
+    priority    => 500,
+    codename    => 'bullseye-backports',
+    notify      => Package['onionbalance'],
+  }
+
   service { 'onionbalance':
     ensure  => running,
     require => Package['onionbalance'],

    that looks alright, but actually doesn't work: unattended-upgrades will not upgrade this package because while it respects pins, the origin is not in the allowed list, so it will skip it.

    i have a similar problem on polyanthum right now, and we had to install hacks for grafana and gitlab for those upgrades to automatically go through.

    @lavamind suggested an alternative solution to this which was to add a package-specific cron-job with a special Unattended-Upgrade::Origins-Pattern to upgrade just that package, alongside pinning, held together by a puppet define glue. i have those objections to this:

    1. we are likely to forget to put the glue on, which will mean out of date packages
    2. it's possible those u-u jobs will run in parallel which could, at best, lead to one of the job failing
    3. the apt code is already an ugly mess in puppet, and i don't want to add more glue

    So i'm going to go through with those steps

    • run puppet everywhere
    • lock puppet everywhere to keep it from re-starting the timer
    • disable the unattended-upgrades timer (systemctl stop apt-daily.timer) to keep u-u from running automatically
    • run unattended-upgrades everywhere to make sure everything is up to date already
    • push the patch to enable Unattended-Upgrade::Origins-Pattern=*
    • re-enable puppet
    • enable, run puppet, disable, everywhere
    • run unattended-upgrades in noop mode everywhere to make sure there is no change (unattended-upgrades --dry-run -v)
    • if there is an unwanted change, add it to pinning or the unattended-upgrades block list, go back to the previous step no unwanted change, did a pretty good audit
    • run unattended-upgrades in wet mode everywhere to make sure there is no change
    • re-enable the timer
    ✓ 11 of 11 checklist items completed · Edited by anarcat

    Designs

    Child items ...

    • Activity

    • anarcat added Doing lifecycle labels

      added Doing lifecycle labels

    • anarcat assigned to @anarcat

      assigned to @anarcat

    • anarcat marked this issue as related to #40758 (closed)

      marked this issue as related to #40758 (closed)

    • anarcat marked the checklist item run puppet everywhere as completed

      marked the checklist item run puppet everywhere as completed

    • anarcat
      anarcat @anarcat ·
      Author Owner

      i ran puppet everywhere with:

      cumin -b 10 -p 0 '*' 'patc'

      this takes about 4m30s each time, with 10 hosts failing because of the puppetdb bug each time. luckily, on the second run, only one host (colchicifolium) overlapped between the two batches, so it was fairly easy to resolve.

      i have to pause for today, next step is to do this again and go down the rest of the list.

    • anarcat
      anarcat @anarcat ·
      Author Owner

      reran puppet everywhere.

    • anarcat
      anarcat @anarcat ·
      Author Owner

      disabled puppet everywhere:

      cumin-all 'puppet agent --disable "changes to u-u tpo/tpa/team#40771"'
    • anarcat marked the checklist item lock puppet everywhere to keep it from re-starting the timer as completed

      marked the checklist item lock puppet everywhere to keep it from re-starting the timer as completed

    • anarcat marked the checklist item disable the unattended-upgrades timer (systemctl stop apt-daily.timer) to keep u-u from running automatically as completed

      marked the checklist item disable the unattended-upgrades timer (systemctl stop apt-daily.timer) to keep u-u from running automatically as completed

    • anarcat marked the checklist item run unattended-upgrades everywhere to make sure everything is up to date already as completed

      marked the checklist item run unattended-upgrades everywhere to make sure everything is up to date already as completed

    • anarcat marked the checklist item push the patch to enable Unattended-Upgrade::Origins-Pattern=* as completed

      marked the checklist item push the patch to enable Unattended-Upgrade::Origins-Pattern=* as completed

    • anarcat marked the checklist item enable, run puppet, disable, everywhere as completed

      marked the checklist item enable, run puppet, disable, everywhere as completed

    • anarcat
      anarcat @anarcat ·
      Author Owner

      puppet doesn't restart the apt-daily.timer (verified on perdulce), so i just re-enabled puppet everywhere and took it out of the equation.

    • anarcat marked the checklist item enable, run puppet, disable, everywhere as incomplete

      marked the checklist item enable, run puppet, disable, everywhere as incomplete

    • anarcat changed the description

      changed the description

    • anarcat
      anarcat @anarcat ·
      Author Owner

      ran puppet everywhere to deploy the new changes, all hosts are marked as "errors" in cumin, but that's normal because they all changed. rerunning puppet to make sure it ran on all hosts (to workaround the puppetdb bug):

      15.1% (14/93) of nodes failed to execute command 'patc': bacula-director-01.torproject.org,carinatum.torproject.org,chi-node-02.torproject.org,chives.torproject.org,ci-runner-x86-05.torproject.org,colchicifolium.torproject.org,corsicum.torproject.org,fallax.torproject.org,fsn-node-06.torproject.org,loghost01.torproject.org,onionoo-backend-02.torproject.org,pauli.torproject.org,perdulce.torproject.org,web-cymru-01.torproject.org

      rerunning on those hosts:

      35.7% (5/14) of nodes failed to execute command 'patc': chives.torproject.org,colchicifolium.torproject.org,corsicum.torproject.org,onionoo-backend-02.torproject.org,web-cymru-01.torproject.org

      rerunning:

      60.0% (3/5) of nodes failed to execute command 'patc': colchicifolium.torproject.org,corsicum.torproject.org,web-cymru-01.torproject.org

      then puppet ran everywhere.

      Edited by anarcat
    • anarcat marked the checklist item enable, run puppet, disable, everywhere as completed

      marked the checklist item enable, run puppet, disable, everywhere as completed

    • anarcat
      anarcat @anarcat ·
      Author Owner

      running cumin-all 'unattended-upgrades --dry-run -v'

    • anarcat
      anarcat @anarcat ·
      Author Owner

      it looks like dry-run is not quite going to cut it, because it says, on many hosts:

      Checking if system is running on battery is skipped. Please install powermgmt-base package to check power status and skip installing updates when the system is running on battery.                                                                                                                                                           
Starting unattended upgrades script
Allowed origins are: origin=*
Initial blacklist: gitlab-runner grub-pc openvswitch-switch openvswitch-common
Initial whitelist (not strict): 
No packages found that can be upgraded unattended and no pending auto-removals
The list of kept packages can't be calculated in dry-run mode.

      the last line is a little alarming, because it seems to say it won't tell us what it's actually going to do in "wet" mode. so i'll try with --download-only instead, and see if that gives a better result.

      i have also ran it in wet mode on build-x85-05 and perdulce, and restart the apt-daily.timer there, so those two are done at least.

      Edited by anarcat
    • anarcat
      anarcat @anarcat ·
      Author Owner

      it really does look like the change is a noop:

      anarcat@curie:~$ cumin-all 'unattended-upgrades --download-only -v'
93 hosts will be targeted:
alberti.torproject.org,archive-01.torproject.org,bacula-director-01.torproject.org,btcpayserver-02.torproject.org,build-x86-[05-06].torproject.org,bungei.torproject.org,carinatum.torproject.org,cdn-backend-sunet-01.torproject.org,check-01.torproject.org,chi-node-[01-14].torproject.org,chives.torproject.org,ci-runner-01.torproject.org,ci-runner-arm64-02.torproject.org,ci-runner-x86-05.torproject.org,colchicifolium.torproject.org,corsicum.torproject.org,crm-ext-01.torproject.org,crm-int-01.torproject.org,cupani.torproject.org,dangerzone-01.torproject.org,eugeni.torproject.org,fallax.torproject.org,fsn-node-[01-08].torproject.org,gayi.torproject.org,gettor-01.torproject.org,gitlab-02.torproject.org,henryi.torproject.org,hetzner-hel1-[01-03].torproject.org,hetzner-nbg1-[01-02].torproject.org,loghost01.torproject.org,majus.torproject.org,mandos-01.torproject.org,materculae.torproject.org,media-01.torproject.org,meronense.torproject.org,metrics-store-01.torproject.org,moly.torproject.org,neriniflorum.torproject.org,nevii.torproject.org,nutans.torproject.org,onionbalance-02.torproject.org,onionoo-backend-[01-02].torproject.org,onionoo-frontend-[01-02].torproject.org,palmeri.torproject.org,pauli.torproject.org,peninsulare.torproject.org,perdulce.torproject.org,polyanthum.torproject.org,probetelemetry-01.torproject.org,relay-01.torproject.org,rude.torproject.org,static-gitlab-shim.torproject.org,static-master-fsn.torproject.org,staticiforme.torproject.org,submit-01.torproject.org,subnotabile.torproject.org,tb-build-[01,03,05].torproject.org,tb-pkgstage-01.torproject.org,tb-tester-01.torproject.org,tbb-nightlies-master.torproject.org,vineale.torproject.org,web-chi-03.torproject.org,web-cymru-01.torproject.org,web-fsn-[01-02].torproject.org
Ok to proceed on 93 hosts? Enter the number of affected hosts to confirm or "q" to quit 93
===== NODE GROUP =====                                                                                                                                                 
(1) ci-runner-arm64-02.torproject.org                                                                                                                                  
----- OUTPUT of 'unattended-upgra...download-only -v' -----                                                                                                            
Starting unattended upgrades script                                                                                                                                    
Allowed origins are: origin=*                                                                                                                                          
Initial blacklist: gitlab-runner grub-pc openvswitch-switch openvswitch-common
Initial whitelist (not strict): 
===== NODE GROUP =====                                                                                                                                                 
(4) fsn-node-[05-08].torproject.org                                                                                                                                    
----- OUTPUT of 'unattended-upgra...download-only -v' -----                                                                                                            
Checking if system is running on battery is skipped. Please install powermgmt-base package to check power status and skip installing updates when the system is running on battery.                                                                                                                                                           
Initial blacklist : gitlab-runner grub-pc openvswitch-switch openvswitch-common
Initial whitelist: 
Starting unattended upgrades script
Allowed origins are: origin=*
===== NODE GROUP =====                                                                                                                                                 
(32) btcpayserver-02.torproject.org,chi-node-[01-14].torproject.org,ci-runner-01.torproject.org,ci-runner-x86-05.torproject.org,dangerzone-01.torproject.org,mandos-01.torproject.org,media-01.torproject.org,metrics-store-01.torproject.org,onionbalance-02.torproject.org,onionoo-frontend-02.torproject.org,probetelemetry-01.torproject.org,relay-01.torproject.org,static-gitlab-shim.torproject.org,tb-build-[01,03,05].torproject.org,tb-pkgstage-01.torproject.org,tb-tester-01.torproject.org,web-chi-03.torproject.org
----- OUTPUT of 'unattended-upgra...download-only -v' -----                                                                                                            
Checking if system is running on battery is skipped. Please install powermgmt-base package to check power status and skip installing updates when the system is running on battery.                                                                                                                                                           
Starting unattended upgrades script
Allowed origins are: origin=*
Initial blacklist: gitlab-runner grub-pc openvswitch-switch openvswitch-common
Initial whitelist (not strict): 
===== NODE GROUP =====                                                                                                                                                 
(23) alberti.torproject.org,cdn-backend-sunet-01.torproject.org,corsicum.torproject.org,crm-ext-01.torproject.org,crm-int-01.torproject.org,cupani.torproject.org,eugeni.torproject.org,fsn-node-[01-04].torproject.org,gayi.torproject.org,gettor-01.torproject.org,gitlab-02.torproject.org,henryi.torproject.org,hetzner-hel1-01.torproject.org,meronense.torproject.org,moly.torproject.org,nutans.torproject.org,pauli.torproject.org,peninsulare.torproject.org,subnotabile.torproject.org,vineale.torproject.org
----- OUTPUT of 'unattended-upgra...download-only -v' -----                                                                                                            
Checking if system is running on battery is skipped. Please install powermgmt-base package to check power status and skip installing updates when the system is running on battery.                                                                                                                                                           
Checking if connection is metered is skipped. Please install python3-gi package to detect metered connections and skip downloading updates.
Initial blacklist : gitlab-runner grub-pc openvswitch-switch openvswitch-common
Initial whitelist: 
Starting unattended upgrades script
Allowed origins are: origin=*
===== NODE GROUP =====                                                                                                                                                 
(33) archive-01.torproject.org,bacula-director-01.torproject.org,build-x86-[05-06].torproject.org,bungei.torproject.org,carinatum.torproject.org,check-01.torproject.org,chives.torproject.org,colchicifolium.torproject.org,fallax.torproject.org,hetzner-hel1-[02-03].torproject.org,hetzner-nbg1-[01-02].torproject.org,loghost01.torproject.org,majus.torproject.org,materculae.torproject.org,neriniflorum.torproject.org,nevii.torproject.org,onionoo-backend-[01-02].torproject.org,onionoo-frontend-01.torproject.org,palmeri.torproject.org,perdulce.torproject.org,polyanthum.torproject.org,rude.torproject.org,static-master-fsn.torproject.org,staticiforme.torproject.org,submit-01.torproject.org,tbb-nightlies-master.torproject.org,web-cymru-01.torproject.org,web-fsn-[01-02].torproject.org
----- OUTPUT of 'unattended-upgra...download-only -v' -----                                                                                                            
Checking if system is running on battery is skipped. Please install powermgmt-base package to check power status and skip installing updates when the system is running on battery.                                                                                                                                                           
Checking if connection is metered is skipped. Please install python3-gi package to detect metered connections and skip downloading updates.
Starting unattended upgrades script
Allowed origins are: origin=*
Initial blacklist: gitlab-runner grub-pc openvswitch-switch openvswitch-common
Initial whitelist (not strict): 
================                                                                                                                                                       
PASS |███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (93/93) [01:25<00:00,  1.09hosts/s]
FAIL |                                                                                                                                |   0% (0/93) [01:25<?, ?hosts/s]
100.0% (93/93) success ratio (>= 0.0% threshold) for command: 'unattended-upgra...download-only -v'.
100.0% (93/93) success ratio (>= 0.0% threshold) of nodes successfully executed all commands.

      ie. there are no pending changes at all.

      so i think i'll just enable this and be done with it. one thing i'm curious to look at, however, is the apt-cache policy output vs our old setting and see if i missed anything, so i'm going to run that first.

    • anarcat
      anarcat @anarcat ·
      Author Owner

      so things are a little more complicated than i hoped in apt-cache policy. here are the different mirrors we use, after s/buster/bullseye and (basically) s/wikipedia/debian.org and s/hetzner/debian.org) and sort -u:

       100 https://deb.debian.org/debian bullseye-backports/contrib amd64 Packages
 100 https://deb.debian.org/debian bullseye-backports/contrib arm64 Packages
 100 https://deb.debian.org/debian bullseye-backports/main amd64 Packages
 100 https://deb.debian.org/debian bullseye-backports/main arm64 Packages
 100 https://deb.debian.org/debian bullseye-backports/non-free amd64 Packages
 100 https://deb.debian.org/debian bullseye-backports/non-free arm64 Packages
 100 https://deb.debian.org/debian/packages bullseye-backports/contrib amd64 Packages
 100 https://deb.debian.org/debian/packages bullseye-backports/main amd64 Packages
 100 https://deb.debian.org/debian/packages bullseye-backports/non-free amd64 Packages
 100 /var/lib/dpkg/status
   1 https://deb.debian.org/debian/packages bullseye/main amd64 Packages
 500 https://db.torproject.org/torproject-admin bullseye/main amd64 Packages
 500 https://db.torproject.org/torproject-admin bullseye/main arm64 Packages
 500 https://db.torproject.org/torproject-admin packages-restricted/non-free amd64 Packages
 500 https://db.torproject.org/torproject-admin tpo-all/main amd64 Packages
 500 https://db.torproject.org/torproject-admin tpo-all/main arm64 Packages
 500 https://deb.debian.org/debian bullseye/contrib amd64 Packages
 500 https://deb.debian.org/debian bullseye/contrib arm64 Packages
 500 https://deb.debian.org/debian bullseye/main amd64 Packages
 500 https://deb.debian.org/debian bullseye/main arm64 Packages
 500 https://deb.debian.org/debian bullseye/non-free amd64 Packages
 500 https://deb.debian.org/debian bullseye/non-free arm64 Packages
 500 https://deb.debian.org/debian bullseye-updates/main amd64 Packages
 500 https://deb.debian.org/debian bullseye-updates/main arm64 Packages
 500 http://security.debian.org bullseye-security/main amd64 Packages
 500 http://security.debian.org bullseye-security/main arm64 Packages
 500 http://security.debian.org bullseye/updates/main amd64 Packages
 500 http://security.debian.org bullseye/updates/non-free amd64 Packages
 500 http://snapshot.debian.org/archive/debian/20190225T033235Z unstable/main amd64 Packages
 500 https://packages.gitlab.com/gitlab/gitlab-ce/debian bullseye/main amd64 Packages
 500 https://packages.gitlab.com/runner/gitlab-runner/debian bullseye/main amd64 Packages
 500 https://packages.gitlab.com/runner/gitlab-runner/debian bullseye/main arm64 Packages
 500 https://packages.grafana.com/oss/deb stable/main amd64 Packages

      so first off, great news, backports is lower priority (100) than installed packages (100). i think. i'm not actually sure how that logic works, because if i read the apt_preferences manpage, that would seem to say that the package should be upgraded. but I did a test on polyanthum and deployed this pin:

      # This file is managed by Puppet. DO NOT EDIT.
Explanation: newer version needed, see tpo/tpa/team#40758
Package: tor tor-geoipdb
Pin: release n=bullseye-backports
Pin-Priority: 500

      and that gives me this policy:

      root@polyanthum:~# apt-cache policy tor-geoipdb
tor-geoipdb:
  Installed: 0.4.5.10-1~deb11u1
  Candidate: 0.4.7.7-1~bpo11+1
  Version table:
     0.4.7.7-1~bpo11+1 500
        100 https://mirror.hetzner.de/debian/packages bullseye-backports/main amd64 Packages
 *** 0.4.5.10-1~deb11u1 500
        500 https://mirror.hetzner.de/debian/packages bullseye/main amd64 Packages
        500 http://security.debian.org bullseye-security/main amd64 Packages
        100 /var/lib/dpkg/status

      and unattended-upgrades then happily respects that:

      root@polyanthum:~# unattended-upgrade --dry-run -v
Checking if system is running on battery is skipped. Please install powermgmt-base package to check power status and skip installing updates when the system is running on battery.
Checking if connection is metered is skipped. Please install python3-gi package to detect metered connections and skip downloading updates.
Starting unattended upgrades script
Allowed origins are: origin=*
Initial blacklist: gitlab-runner grub-pc openvswitch-switch openvswitch-common
Initial whitelist (not strict): 
Option --dry-run given, *not* performing real actions
Packages that will be upgraded: tor-geoipdb
Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
[master b725fab2] saving uncommitted changes in /etc prior to apt run
 3 files changed, 6 insertions(+), 1 deletion(-)
 create mode 100644 apt/preferences.d/tor.pref
/usr/bin/dpkg --force-confdef --force-confold --force-confdef --force-confold --status-fd 10 --no-triggers --unpack --auto-deconfigure --force-remove-protected /var/cache/apt/archives/tor-geoipdb_0.4.7.7-1~bpo11+1_all.deb 
/usr/bin/dpkg --force-confdef --force-confold --force-confdef --force-confold --status-fd 10 --configure --pending 
All upgrades installed
The list of kept packages can't be calculated in dry-run mode.

      whereas it didn't do the upgrade without the pin:

      root@polyanthum:~# apt-cache policy tor-geoipdb
tor-geoipdb:
  Installed: 0.4.5.10-1~deb11u1
  Candidate: 0.4.5.10-1~deb11u1
  Version table:
     0.4.7.7-1~bpo11+1 100
        100 https://mirror.hetzner.de/debian/packages bullseye-backports/main amd64 Packages
 *** 0.4.5.10-1~deb11u1 500
        500 https://mirror.hetzner.de/debian/packages bullseye/main amd64 Packages
        500 http://security.debian.org bullseye-security/main amd64 Packages
        100 /var/lib/dpkg/status
root@polyanthum:~# unattended-upgrade -v
Checking if system is running on battery is skipped. Please install powermgmt-base package to check power status and skip installing updates when the system is running on battery.
Checking if connection is metered is skipped. Please install python3-gi package to detect metered connections and skip downloading updates.
Starting unattended upgrades script
Allowed origins are: origin=*
Initial blacklist: gitlab-runner grub-pc openvswitch-switch openvswitch-common
Initial whitelist (not strict): 
No packages found that can be upgraded unattended and no pending auto-removals

      ... so i think this works as designed, and the --dry-run actually does what i want.

      i have upgraded the tor-geoipdb package back right now, i had downgraded this to run this test.

      next i'll study the policy output a little more and do a real run everywhere.

    • anarcat
      anarcat @anarcat ·
      Author Owner

      those are the interesting ones, i think.

       500 https://packages.gitlab.com/gitlab/gitlab-ce/debian bullseye/main amd64 Packages
 500 https://packages.gitlab.com/runner/gitlab-runner/debian bullseye/main amd64 Packages
 500 https://packages.gitlab.com/runner/gitlab-runner/debian bullseye/main arm64 Packages
 500 https://packages.grafana.com/oss/deb stable/main amd64 Packages

      all those three were either in the allow list (gitlab and grafana) or are blocked in unattended-upgrades (gitlab-runner), so i think we're safe on all counts.

    • anarcat
      anarcat @anarcat ·
      Author Owner

      those are the pins found:

           libjetty9-extra-java -> 9.4.15-1 with priority 501
     libjetty9-java -> 9.4.15-1 with priority 501
     mtail -> 3.0.0~rc43-3+b2 with priority 500
     onionbalance -> 0.2.2-1~bpo11+1 with priority 500
     origin db.torproject.org
     origin deb.debian.org
     origin mirror.hetzner.de
     origin mirrors.wikimedia.org
     origin packages.gitlab.com
     origin packages.grafana.com
     origin security.debian.org
     origin snapshot.debian.org

      those also look okay: mtail is pinned to bullseye in buster, onionbalance to backports (which is what triggered this entire bug), and jetty to snapshots (although that one is actually an error, i think i'll have to look into that: it might be an ineffective pin, but we had that problem anyways).

      so that part looks good too.

    • anarcat
      anarcat @anarcat ·
      Author Owner

      another way to look at the pins:

      Pinned packages:
     release a=now
     release o=Debian,a=oldstable-updates,n=bullseye-updates,l=Debian,c=main,b=amd64
     release o=Debian,a=unstable,n=sid,l=Debian,c=main,b=amd64
     release o=Debian Backports,a=bullseye-backports,n=bullseye-backports,l=Debian Backports,c=contrib,b=amd64
     release o=Debian Backports,a=bullseye-backports,n=bullseye-backports,l=Debian Backports,c=contrib,b=arm64
     release o=Debian Backports,a=bullseye-backports,n=bullseye-backports,l=Debian Backports,c=main,b=amd64
     release o=Debian Backports,a=bullseye-backports,n=bullseye-backports,l=Debian Backports,c=main,b=arm64
     release o=Debian Backports,a=bullseye-backports,n=bullseye-backports,l=Debian Backports,c=non-free,b=amd64
     release o=Debian Backports,a=bullseye-backports,n=bullseye-backports,l=Debian Backports,c=non-free,b=arm64
     release o=grafana stable,a=stable,n=stable,l=grafana stable,c=main,b=amd64
     release o=torproject-admin@torproject.org,a=stable,n=bullseye,c=main,b=amd64
     release o=torproject-admin@torproject.org,a=stable,n=bullseye,c=main,b=arm64
     release o=torproject-admin@torproject.org,a=stable,n=packages-restricted,c=non-free,b=amd64
     release o=torproject-admin@torproject.org,a=stable,n=tpo-all,c=main,b=amd64
     release o=torproject-admin@torproject.org,a=stable,n=tpo-all,c=main,b=arm64
     release v=10.12,o=Debian,a=oldstable,n=bullseye,l=Debian,c=contrib,b=amd64
     release v=10.12,o=Debian,a=oldstable,n=bullseye,l=Debian,c=main,b=amd64
     release v=10.12,o=Debian,a=oldstable,n=bullseye,l=Debian,c=non-free,b=amd64
     release v=10,o=Debian,a=oldstable,n=bullseye,l=Debian-Security,c=main,b=amd64
     release v=10,o=Debian,a=oldstable,n=bullseye,l=Debian-Security,c=non-free,b=amd64
     release v=11.3,o=Debian,a=stable,n=bullseye,l=Debian,c=contrib,b=amd64
     release v=11.3,o=Debian,a=stable,n=bullseye,l=Debian,c=contrib,b=arm64
     release v=11.3,o=Debian,a=stable,n=bullseye,l=Debian,c=main,b=amd64
     release v=11.3,o=Debian,a=stable,n=bullseye,l=Debian,c=main,b=arm64
     release v=11.3,o=Debian,a=stable,n=bullseye,l=Debian,c=non-free,b=amd64
     release v=11.3,o=Debian,a=stable,n=bullseye,l=Debian,c=non-free,b=arm64
     release v=11,o=Debian,a=stable-security,n=bullseye-security,l=Debian-Security,c=main,b=amd64
     release v=11,o=Debian,a=stable-security,n=bullseye-security,l=Debian-Security,c=main,b=arm64
     release v=11-updates,o=Debian,a=stable-updates,n=bullseye-updates,l=Debian,c=main,b=amd64
     release v=11-updates,o=Debian,a=stable-updates,n=bullseye-updates,l=Debian,c=main,b=arm64
     release v=1,o=packages.gitlab.com/gitlab/gitlab-ce,a=bullseye,n=bullseye,l=gitlab-ce,c=main,b=amd64
     release v=1,o=packages.gitlab.com/runner/gitlab-runner,a=bullseye,n=bullseye,l=gitlab-runner,c=main,b=amd64
     release v=1,o=packages.gitlab.com/runner/gitlab-runner,a=bullseye,n=bullseye,l=gitlab-runner,c=main,b=arm64

      also looks legit. we have the TPA archive in there, which i think it's fine to allow upgrades on. the unstable thing in there is a snapshots.do pin.

      as a reminder, this was our previous "allow origins" configuration, as of commit 9e17805a4ad7b7902e35cdd5a569ca881ef209cb:

              'origin=Debian,codename=${distro_codename},label=Debian',
        'origin=Debian,codename=${distro_codename}-updates',
        'origin=Debian,codename=${distro_codename}-security',
        'origin=Debian,codename=${distro_codename},label=Debian-Security',
        'origin=grafana stable',
        'origin=torproject-admin@torproject.org,codename=tpo-all',
        'origin=*packages.gitlab.com/gitlab/gitlab-ce,codename=${distro_codename}'

      so i'm confident this change will not impact negatively our infrastructure. the worst thing that could happen would be an accidental upgrade to bullseye, i think. but if there was one pending, u-u dry-run would have caught it.

      so let's just do this.

    • anarcat marked the checklist item run unattended-upgrades in noop mode everywhere to make sure there is no change (unattended-upgrades --dry-run -v) as completed

      marked the checklist item run unattended-upgrades in noop mode everywhere to make sure there is no change (unattended-upgrades --dry-run -v) as completed

    • anarcat marked the checklist item if there is an unwanted change, add it to pinning or the unattended-upgrades block list, go back to the previous step as completed

      marked the checklist item if there is an unwanted change, add it to pinning or the unattended-upgrades block list, go back to the previous step as completed

    • anarcat changed the description

      changed the description

    • anarcat marked the checklist item run unattended-upgrades in wet mode everywhere to make sure there is no change as completed

      marked the checklist item run unattended-upgrades in wet mode everywhere to make sure there is no change as completed

    • anarcat
      anarcat @anarcat ·
      Author Owner

      final u-u run looks good:

      anarcat@curie:~$ cumin-all 'unattended-upgrades -v'
93 hosts will be targeted:
alberti.torproject.org,archive-01.torproject.org,bacula-director-01.torproject.org,btcpayserver-02.torproject.org,build-x86-[05-06].torproject.org,bungei.torproject.org,carinatum.torproject.org,cdn-backend-sunet-01.torproject.org,check-01.torproject.org,chi-node-[01-14].torproject.org,chives.torproject.org,ci-runner-01.torproject.org,ci-runner-arm64-02.torproject.org,ci-runner-x86-05.torproject.org,colchicifolium.torproject.org,corsicum.torproject.org,crm-ext-01.torproject.org,crm-int-01.torproject.org,cupani.torproject.org,dangerzone-01.torproject.org,eugeni.torproject.org,fallax.torproject.org,fsn-node-[01-08].torproject.org,gayi.torproject.org,gettor-01.torproject.org,gitlab-02.torproject.org,henryi.torproject.org,hetzner-hel1-[01-03].torproject.org,hetzner-nbg1-[01-02].torproject.org,loghost01.torproject.org,majus.torproject.org,mandos-01.torproject.org,materculae.torproject.org,media-01.torproject.org,meronense.torproject.org,metrics-store-01.torproject.org,moly.torproject.org,neriniflorum.torproject.org,nevii.torproject.org,nutans.torproject.org,onionbalance-02.torproject.org,onionoo-backend-[01-02].torproject.org,onionoo-frontend-[01-02].torproject.org,palmeri.torproject.org,pauli.torproject.org,peninsulare.torproject.org,perdulce.torproject.org,polyanthum.torproject.org,probetelemetry-01.torproject.org,relay-01.torproject.org,rude.torproject.org,static-gitlab-shim.torproject.org,static-master-fsn.torproject.org,staticiforme.torproject.org,submit-01.torproject.org,subnotabile.torproject.org,tb-build-[01,03,05].torproject.org,tb-pkgstage-01.torproject.org,tb-tester-01.torproject.org,tbb-nightlies-master.torproject.org,vineale.torproject.org,web-chi-03.torproject.org,web-cymru-01.torproject.org,web-fsn-[01-02].torproject.org
Ok to proceed on 93 hosts? Enter the number of affected hosts to confirm or "q" to quit 93
===== NODE GROUP =====                                                                                                                                                 
(1) ci-runner-arm64-02.torproject.org                                                                                                                                  
----- OUTPUT of 'unattended-upgrades -v' -----                                                                                                                         
Starting unattended upgrades script                                                                                                                                    
Allowed origins are: origin=*                                                                                                                                          
Initial blacklist: gitlab-runner grub-pc openvswitch-switch openvswitch-common
Initial whitelist (not strict): 
No packages found that can be upgraded unattended and no pending auto-removals
===== NODE GROUP =====                                                                                                                                                 
(4) fsn-node-[05-08].torproject.org                                                                                                                                    
----- OUTPUT of 'unattended-upgrades -v' -----                                                                                                                         
Checking if system is running on battery is skipped. Please install powermgmt-base package to check power status and skip installing updates when the system is running on battery.                                                                                                                                                           
Initial blacklist : gitlab-runner grub-pc openvswitch-switch openvswitch-common
Initial whitelist: 
Starting unattended upgrades script
Allowed origins are: origin=*
No packages found that can be upgraded unattended and no pending auto-removals
===== NODE GROUP =====                                                                                                                                                 
(32) btcpayserver-02.torproject.org,chi-node-[01-14].torproject.org,ci-runner-01.torproject.org,ci-runner-x86-05.torproject.org,dangerzone-01.torproject.org,mandos-01.torproject.org,media-01.torproject.org,metrics-store-01.torproject.org,onionbalance-02.torproject.org,onionoo-frontend-02.torproject.org,probetelemetry-01.torproject.org,relay-01.torproject.org,static-gitlab-shim.torproject.org,tb-build-[01,03,05].torproject.org,tb-pkgstage-01.torproject.org,tb-tester-01.torproject.org,web-chi-03.torproject.org
----- OUTPUT of 'unattended-upgrades -v' -----                                                                                                                         
Checking if system is running on battery is skipped. Please install powermgmt-base package to check power status and skip installing updates when the system is running on battery.                                                                                                                                                           
Starting unattended upgrades script
Allowed origins are: origin=*
Initial blacklist: gitlab-runner grub-pc openvswitch-switch openvswitch-common
Initial whitelist (not strict): 
No packages found that can be upgraded unattended and no pending auto-removals
===== NODE GROUP =====                                                                                                                                                 
(23) alberti.torproject.org,cdn-backend-sunet-01.torproject.org,corsicum.torproject.org,crm-ext-01.torproject.org,crm-int-01.torproject.org,cupani.torproject.org,eugeni.torproject.org,fsn-node-[01-04].torproject.org,gayi.torproject.org,gettor-01.torproject.org,gitlab-02.torproject.org,henryi.torproject.org,hetzner-hel1-01.torproject.org,meronense.torproject.org,moly.torproject.org,nutans.torproject.org,pauli.torproject.org,peninsulare.torproject.org,subnotabile.torproject.org,vineale.torproject.org
----- OUTPUT of 'unattended-upgrades -v' -----                                                                                                                         
Checking if system is running on battery is skipped. Please install powermgmt-base package to check power status and skip installing updates when the system is running on battery.                                                                                                                                                           
Checking if connection is metered is skipped. Please install python3-gi package to detect metered connections and skip downloading updates.
Initial blacklist : gitlab-runner grub-pc openvswitch-switch openvswitch-common
Initial whitelist: 
Starting unattended upgrades script
Allowed origins are: origin=*
No packages found that can be upgraded unattended and no pending auto-removals
===== NODE GROUP =====                                                                                                                                                 
(1) polyanthum.torproject.org                                                                                                                                          
----- OUTPUT of 'unattended-upgrades -v' -----                                                                                                                         
Checking if system is running on battery is skipped. Please install powermgmt-base package to check power status and skip installing updates when the system is running on battery.                                                                                                                                                           
Checking if connection is metered is skipped. Please install python3-gi package to detect metered connections and skip downloading updates.
Starting unattended upgrades script
Allowed origins are: origin=*
Initial blacklist: gitlab-runner grub-pc openvswitch-switch openvswitch-common
Initial whitelist (not strict): 
Packages that will be upgraded: tor-geoipdb
Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
[master f431e17e] saving uncommitted changes in /etc prior to apt run
 1 file changed, 0 insertions(+), 0 deletions(-)
(Reading database ... 67112 files and directories currently installed.)
Preparing to unpack .../tor-geoipdb_0.4.7.7-1~bpo11+1_all.deb ...
Unpacking tor-geoipdb (0.4.7.7-1~bpo11+1) over (0.4.5.10-1~deb11u1) ...
Setting up tor-geoipdb (0.4.7.7-1~bpo11+1) ...
(Reading database ... 100%
Running kernel seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.
All upgrades installed
===== NODE GROUP =====                                                                                                                                                 
(32) archive-01.torproject.org,bacula-director-01.torproject.org,build-x86-[05-06].torproject.org,bungei.torproject.org,carinatum.torproject.org,check-01.torproject.org,chives.torproject.org,colchicifolium.torproject.org,fallax.torproject.org,hetzner-hel1-[02-03].torproject.org,hetzner-nbg1-[01-02].torproject.org,loghost01.torproject.org,majus.torproject.org,materculae.torproject.org,neriniflorum.torproject.org,nevii.torproject.org,onionoo-backend-[01-02].torproject.org,onionoo-frontend-01.torproject.org,palmeri.torproject.org,perdulce.torproject.org,rude.torproject.org,static-master-fsn.torproject.org,staticiforme.torproject.org,submit-01.torproject.org,tbb-nightlies-master.torproject.org,web-cymru-01.torproject.org,web-fsn-[01-02].torproject.org
----- OUTPUT of 'unattended-upgrades -v' -----                                                                                                                         
Checking if system is running on battery is skipped. Please install powermgmt-base package to check power status and skip installing updates when the system is running on battery.                                                                                                                                                           
Checking if connection is metered is skipped. Please install python3-gi package to detect metered connections and skip downloading updates.
Starting unattended upgrades script
Allowed origins are: origin=*
Initial blacklist: gitlab-runner grub-pc openvswitch-switch openvswitch-common
Initial whitelist (not strict): 
No packages found that can be upgraded unattended and no pending auto-removals
================                                                                                                                                                       
PASS |███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (93/93) [01:21<00:00,  1.15hosts/s]
FAIL |                                                                                                                                |   0% (0/93) [01:21<?, ?hosts/s]
100.0% (93/93) success ratio (>= 0.0% threshold) for command: 'unattended-upgrades -v'.
100.0% (93/93) success ratio (>= 0.0% threshold) of nodes successfully executed all commands.

      notice how it upgraded tor-geoipdb to backports, as expected.

    • anarcat mentioned in issue #40758 (closed)

      mentioned in issue #40758 (closed)

    • anarcat marked the checklist item re-enable the timer as completed

      marked the checklist item re-enable the timer as completed

    • anarcat
      anarcat @anarcat ·
      Author Owner

      re-enabled the timer, we're all done here.

    • anarcat closed

      closed

    • anarcat mentioned in commit wiki-replica@248f53d4

      mentioned in commit wiki-replica@248f53d4

    • anarcat marked this issue as related to #40769 (closed)

      marked this issue as related to #40769 (closed)

    • anarcat mentioned in issue #40769 (closed)

      mentioned in issue #40769 (closed)

    Please register or sign in to reply