title: TPA-RFC-58: Podman CI runner deployment, help needed
status: standard
discussion: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41296Summary: I deployed a new GitLab CI runner backed by Podman instead of Docker, we hope it will improve the stability and our capacity at building images, but I need help testing it.
Background
We've been having stability issues with the Docker runners for a while now. We also started looking again at container image builds, which are currently failing without Kaniko.
Proposal
Testers needed
I need help testing the new runner. Right now it's marked as not running "untagged jobs", so it's unlikely to pick your CI jobs and run them. It would be great if people could test the new runner.
See the GitLab tag documentation for how to add tags to your
configuration. It's basically done by adding a tags field to the
.gitlab-ci.yml file.
Note that in TPA's ci-test gitlab-ci.yaml file, we use a
TPA_TAG_VALUE variable to be able to pass arbitrary tags down into
the jobs without having to constantly change the .yaml file, which
might be a useful addition to your workflow.
The tag to use is podman.
You can send any job you want to the podman runner, but we'd like to
test a broad variety of things before we put it in production, but
especially image buildings. Upstream even has a set of instructions
to build packages inside podman.
Long term plan
If this goes well, we'd like to converge towards using podman for
all workloads. It's better packaged in Debian, and better designed,
than Docker. It also allows us to run containers as non-root.
That, however, is not part of this proposal. We're already running
Podman for another service (MinIO) but we're not proposing to
convert all existing services to podman. If things work well
enough for a long enough period (say 30 days), we might turn off the
older Docker running instead.
Alternatives considered
To fix the stability issues in Docker, it might be possible to upgrade to the latest upstream package and abandon the packages from Debian.org. We're hoping that will not be necessary thanks to Podman.
To build images, we could create a "privileged" runner. For now, we're hoping Podman will make building container images easier. If we do create a privileged runner, it needs to take into account the long term tiered runner approach.
Deadline
The service is already available, and will be running untagged jobs in two weeks unless an objection is raised.