talkback is broken on isoworkers due to invalid GitLab token since 2025-04-09

The first occurrence was on Apr 9 01:37, and in the log it looks like this:

21:13:52 + /usr/local/bin/talkback.py
21:13:53 Traceback (most recent call last):
21:13:53   File "/usr/local/bin/talkback.py", line 31, in <module>
21:13:53     tails = gl.projects.get('tails/tails')
21:13:53             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
21:13:53   File "/usr/lib/python3/dist-packages/gitlab/v4/objects/projects.py", line 834, in get
21:13:53     return cast(Project, super().get(id=id, lazy=lazy, **kwargs))
21:13:53                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
21:13:53   File "/usr/lib/python3/dist-packages/gitlab/exceptions.py", line 337, in wrapped_f
21:13:53     return f(*args, **kwargs)
21:13:53            ^^^^^^^^^^^^^^^^^^
21:13:53   File "/usr/lib/python3/dist-packages/gitlab/mixins.py", line 122, in get
21:13:53     server_data = self.gitlab.http_get(path, **kwargs)
21:13:53                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
21:13:53   File "/usr/lib/python3/dist-packages/gitlab/client.py", line 834, in http_get
21:13:53     result = self.http_request(
21:13:53              ^^^^^^^^^^^^^^^^^^
21:13:53   File "/usr/lib/python3/dist-packages/gitlab/client.py", line 794, in http_request
21:13:53     raise gitlab.exceptions.GitlabAuthenticationError(
21:13:53 gitlab.exceptions.GitlabAuthenticationError: 401: invalid_token

I did some investigation myself which maybe is useful for you to debug this further:

• Collect the bad runs on jenkins:

  $ BAD_RUNS=$(grep -l -F 'gitlab.exceptions.GitlabAuthenticationError: 401: invalid_token' /var/lib/jenkins/jobs/test_Tails_ISO_*/builds/[0123456789]*/log | xargs dirname)

• It affects all isoworkers:

  $ for x in $BAD_RUNS; do grep ^NODE_NAME $x/injectedEnvVars.txt; done | sort -u
  NODE_NAME=isoworker1.dragon
  NODE_NAME=isoworker2.dragon
  NODE_NAME=isoworker3.dragon
  NODE_NAME=isoworker4.dragon
  NODE_NAME=isoworker5.dragon
  NODE_NAME=isoworker6.iguana
  NODE_NAME=isoworker7.iguana
  NODE_NAME=isoworker8.iguana

• Runs of test_Tails_ISO_stable and test_Tails_ISO_devel have never been affected so far, but all other (active) jobs are always affected (I compared the list below against all jobs listed in the jenkins web interface):

  $ for x in $BAD_RUNS; do echo $x | cut -d/ -f6; done | sort -u
  test_Tails_ISO_18215-update-chutney-force-all-tests
  test_Tails_ISO_20263-remote-shell-debug-logging
  test_Tails_ISO_20263-test-suite-disable-spice-vdagent
  test_Tails_ISO_20263-test-suite-without-spice
  test_Tails_ISO_20297-debug-proxy-refusing-connections
  test_Tails_ISO_20297-test-suite-reject-allow-internal-dns-resolution
  test_Tails_ISO_20816-block-paths-bwrap
  test_Tails_ISO_20861-allow-close-zenity-exit-code
  test_Tails_ISO_20867-document-filechooser-portal
  test_Tails_ISO_20872-more-gtk-launch
  test_Tails_ISO_feature-arti
  test_Tails_ISO_tca-misc-improvements
  test_Tails_ISO_update-acng-config-for-trixie