service/ci.md

@@ -377,6 +377,11 @@ We also avoided using the [puppetlabs/docker](https://forge.puppet.com/modules/p
 containers, volumes and so on right now. All that is (currently)
 handled by GitLab runner.
 
+IMPORTANT: when installing a new runner, it is likely to run into rate
+limiting if it is put into the main rotation immediately. Either
+slowly add it to the pool by not allowing it to "run untagged jobs" or
+[pre-fetch them from a list generated on another runner](#pre-seeding-container-images).
+
 ### Podman on Debian
 
 A [Podman](https://podman.io/) runner was configured to see if we could workaround
@@ -441,6 +446,11 @@ unchecking the "run untagged jobs" checkbox in the UI.
 Note that this is currently in testing, see [issue 41296](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41296) and
 [TPA-RFC-58](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-58-podman-runner).
 
+IMPORTANT: when installing a new runner, it is likely to run into rate
+limiting if it is put into the main rotation immediately. Either
+slowly add it to the pool by not allowing it to "run untagged jobs" or
+[pre-fetch them from a list generated on another runner](#pre-seeding-container-images).
+
 ### MacOS/Windows
 
 A special machine (currently `chi-node-13`) was built to allow builds
@@ -462,6 +472,39 @@ documented, here, and eventually converted into a Puppet manifest, see
 and runners are setup. don't hesitate to create separate headings for
 Windows vs MacOS and for image creation vs runner setup.
 
+### Pre-seeding container images
+
+pre-seed the images by fetching them from a list generated from
+another runner.
+
+Here's how to generate a list of images from an existing runner:
+
+    docker images --format "{{.Repository}}:{{.Tag}}" | sort -u | grep -v -e '<none>' -e registry.gitlab.com > images
+
+Note that we skipped untagged images (`<none>`) and runner-specific
+images (from `registry.gitlab.com`). The latter might match more
+images than needed but it was just a quick hack. The actual image we
+are ignoring is
+`registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper`.
+
+Then that images file can be copied on another host and then read to
+pull all images at once:
+    
+    while read image ; do
+        if podman images --format "{{.Repository}}:{{.Tag}}" | grep "$image" ; then 
+            echo "$image already present"
+        else
+            while ! podman pull "$image"; do 
+                printf "failed to pull image, sleeping 240 seconds, now is: "; date
+                sleep 240
+            done
+        fi 
+    done < images
+
+This will probably run into rate limiting, but should gently retry
+once it hits it to match the 100 queries / 6h (one query every 216
+seconds, technically) [rate limit](https://www.docker.com/increase-rate-limits/).
+
 ## SLA
 
 The GitLab CI service is offered on a "best effort" basis and might