TPA-RFC-58: install a podman runner

we have two places where we feel we might be able to do more with podman as a runner backend instead of docker:

• gitlab#90 (closed) - building images required user namespaces (and, in general, privileged containers, see also https://gitlab.torproject.org/tpo/tpa/container-images/-/merge_requests/1)
• #41295 (closed) - intermittent docker failures

Obviously, introducing a new podman runner could raise new issues, but it's something we've been meaning to do for a while and if it has the chance of fixing things, it seems worth a try.

This, obviously, would need to run bookworm to get the latest and greatest podman that works with GitLab.

Update: VM and runner deployed, limited to jobs tagged with podman, testing requested on tor-project with https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-58-podman-runner

marked this issue as related to #41295 (closed)

marked this issue as related to gitlab#90 (closed)

added CI Gitlab Next lifecycle labels

assigned to @anarcat

mentioned in issue #41295 (closed)

marked this issue as related to #41044

added Doing label and removed Next label

after looking at usage graphs of the other runner, i installed a VM with:

• 300G disk (other VM has 500G but never reached beyond ~280G)
• 96G RAM (other has 300G ram, but again doesn't use it all)
• 16 cores (harder to tell, but similar to the other one)

i also set it up with a "plain" disk with the understanding that this is a disposable VM that doesn't need HA, especially since we reboot it in a peculiar way in the first place.

mentioned in issue #41297 (closed)

i managed to install podman in a new VM (ci-runner-x86-02), but it's failing to run jobs:

https://gitlab.torproject.org/tpo/tpa/ci-test/-/jobs/340685

it seems like my podman install is somewhat non-functional, even as the normal (gitlab-runner) user it fails:

root@ci-runner-x86-02:~# sudo -u gitlab-runner -i
gitlab-runner@ci-runner-x86-02:~$ podman ps
WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available 
WARN[0000] For using systemd, you may need to login using an user session 
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 999` (possibly as root) 
WARN[0000] Falling back to --cgroup-manager=cgroupfs    
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES
WARN[0000] Failed to add pause process to systemd sandbox cgroup: exec: "dbus-launch": executable file not found in $PATH 
gitlab-runner@ci-runner-x86-02:~$ podman run debian:latest
WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available 
WARN[0000] For using systemd, you may need to login using an user session 
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 999` (possibly as root) 
WARN[0000] Falling back to --cgroup-manager=cgroupfs    
WARN[0002] Failed to add pause process to systemd sandbox cgroup: exec: "dbus-launch": executable file not found in $PATH 

i knocked down quite a few hurdles to make this work so far, so maybe i'm missing some more. it's a bit frustrating but maybe i need to go through the install procedure formally:

https://podman.io/getting-started/installation

and follow the rootless tutorial more thoroughly:

https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md

grepping through gitlab issues might help here as well... ran out of time for today though. :(

failed job is https://gitlab.torproject.org/tpo/tpa/ci-test/-/jobs/340685 runner is paused for now

i rebooted the box, ran podman --rm -it debian:latest and it seems like it unlocked everything we need!

this job succeeded running on the -02 runner, whoohoo!

https://gitlab.torproject.org/tpo/tpa/ci-test/-/jobs/342010

but some jobs are failing, so i removed all the tags but podman:

those were: tpa, linux, amd64, kvm, x86_64, x86-64, 16 CPU, 94.30 GiB, debug-terminal, docker

okay, so the runner is now unpaused. i bind-mounted /home/gitlab-runner into /srv to avoid running out of disk space.

next step is to get people to test it, i pinged tickets #41295 (closed) and MR https://gitlab.torproject.org/tpo/tpa/container-images/-/merge_requests/1 to get their feedback, and will also post to #tor-project.

the basic message i copied in #41295 (closed) is:

hey folks, i've setup a podman runner. it doesn't run untagged jobs right now as a safety measure, in case it starts breaking people's CI.

i'd love if people could give it a go. see https://docs.gitlab.com/ee/ci/yaml/#tags for how to add tags to your configuration, but it basically requires a configuration change.

note that in our ci-test gitlab-ci.yaml file we added a TPA_TAG_VALUE variable to be able to pass arbitrary tags down into the jobs without having to constantly change the .yaml file...

added Needs Review label and removed Doing label

added RFC label

changed title from install a podman runner to TPA-RFC-58: install a podman runner

mentioned in commit wiki-replica@41b0b232

i asked for help in testing it by sending TPA-RFC-58 to tor-project, feedback welcome here!

see https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-58-podman-runner

changed the description