service/ci.md

@@ -217,6 +217,18 @@ default is 50, but you can set it to zero or empty to disable shallow
 clones. See also "[Limit the number of changes fetched during
 clone](https://gitlab.torproject.org/help/ci/pipelines/settings#limit-the-number-of-changes-fetched-during-clone)" in the upstream documentation.
 
+### gitlab-runner package upgrade
+
+Every month or so GitLab publishes a update to the `gitlab-runner` apt
+package. The package is excluded for `unattended-upgrades` to avoid any
+risk of interrupting long-running CI jobs (eg. large shadow sims).
+
+The recommended procedure is to go through each CI machine one at a time,
+pause all the runners on that single machine, ensure no long-running
+shadow sims are being executed, and launch `apt upgrade`. If any regular
+CI jobs are running, systemd will wait up to one hour for them to end,
+then proceed with the upgrade.
+
 ## Disaster recovery
 
 Runners should be disposable: if a runner is destroyed, at most the