service/ci.md

@@ -377,6 +377,67 @@ We also avoided using the [puppetlabs/docker](https://forge.puppet.com/modules/p
 containers, volumes and so on right now. All that is (currently)
 handled by GitLab runner.
 
+### Podman on Debian
+
+A [Podman](https://podman.io/) runner was configured to see if we could workaround
+limitations in image building (currently requiring Kaniko) and avoid
+possible issues with Docker itself, specifically [those intermittent
+failures](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41295).
+
+The machine was built with less disk space than ci-runner-x86-01
+(above), but more or less the same specifications, see [this
+ticket](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41296) for details on the installation.
+
+After installation, the following steps were taken:
+
+ 1. setup the large partition in `/srv`, and bind-mount it to cover
+    for GitLab Runner's home which includes the Podman images:
+
+        mkfs -t ext4 -j /dev/sda
+        echo "/dev/sda	/srv	ext4	defaults	1	2" >> /etc/fstab
+        echo "/srv/gitlab-runner	/home/gitlab-runner	none	bind	0	0" >> /etc/fstab
+        mount /srv
+        mount /home/gitlab-runner
+
+ 2. disable module loading:
+
+        touch /etc/no_modules_disabled
+        reboot
+
+    ... otherwise Podman will fail it will try to load extra kernel
+    modules.
+
+ 2. add the `role::gitlab::runner` class to the node in Puppet
+
+ 3. add the following blob in `tor-puppet.git`'s
+    `hiera/nodes/ci-runner-x86-02.torproject.org.yaml`:
+
+        profile::gitlab::runner::docker::backend: "podman"
+        profile::gitlab::runner::defaults:
+          executor: 'docker'
+          run_untagged: false
+          docker_host: "unix:///run/user/999/podman/podman.sock"
+          docker_tlsverify: false
+          docker_image: "quay.io/podman/stable"
+        profile::sysctl:
+          unprivileged_userns_clone:
+            key: "kernel.unprivileged_userns_clone"
+            value: "1"
+
+ 4. run Puppet to deploy `gitlab-runner`, `podman`
+
+ 5. reboot to get the user session started correctly
+
+ 6. run a test job on the host
+
+The last step, specifically, was done by removing all tags from the
+runner (those were `tpa, linux, amd64, kvm, x86_64, x86-64, 16 CPU,
+94.30 GiB, debug-terminal, docker`), adding a `podman` tag, and
+unchecking the "run untagged jobs" checkbox in the UI.
+
+Note that this is currently in testing, see [issue 41296](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41296) and
+[TPA-RFC-58](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-58-podman-runner).
+
 ### MacOS/Windows
 
 A special machine (currently `chi-node-13`) was built to allow builds
@@ -476,7 +537,9 @@ We currently use the following tags:
    be added for other OS
  * **executor** type: `docker`, `KVM`, etc. `docker` are the typical
    runners, `KVM` runners are possibly more powerful and can, for
-   example, run Docker-inside-Docker (DinD)
+   example, run Docker-inside-Docker (DinD), note that `docker` can
+   also mean a `podman` runner, which is tagged on top of `docker`, as
+   a feature
  * **memory** size: `64GB`, `32GB`, `4GB`, etc.
  * **hosting** provider:
    * `tpa`: runners managed by the sysadmin team
@@ -491,6 +554,8 @@ We currently use the following tags:
      one such job may run at a time on a given runner
    * `verylarge`: same as `large`, with sysctl tweaks to allow high
      numbers of processes (runners with >1TB memory)
+   * `podman`: a `docker` executor which talks to the `podman` socket
+     instead of Docker, might be better suited to build container images
 
 Use tags in your configuration only if your job can be fulfilled by
 only some of those runners. For example, only specify a memory tag if
@@ -572,17 +637,15 @@ had to tweak this to deal with out of disk issues.
 
 ### rootless containers
 
-We are considering [podman](https://podman.io/) for running containers more securely:
+We are testing [podman](https://podman.io/) for running containers more securely:
 because they can run containers "rootless" (without running as root on
 the host), they are generally thought to be better immune against
-container escapes. See [those instructions](https://github.com/jonasbb/podman-gitlab-runner). Do note that custom
-executors have limitations that the default Docker executor do not,
-see for example the [lack of ENTRYPOINT support](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27301).
+container escapes.
 
 This could also possibly make it easier to build containers inside
 GitLab CI, which would otherwise require docker-in-docker (DinD),
-unsupported by upstream. This can be done with [buildah](https://buildah.io/) using, for
-example, [those instructions](https://medium.com/prgcont/using-buildah-in-gitlab-ci-9b529af19e42).
+unsupported by upstream. See [those GitLab instructions](https://docs.gitlab.com/runner/executors/docker.html#use-podman-to-build-container-images-from-a-dockerfile) for
+details.
 
 ### Current services