... | @@ -377,6 +377,67 @@ We also avoided using the [puppetlabs/docker](https://forge.puppet.com/modules/p |
... | @@ -377,6 +377,67 @@ We also avoided using the [puppetlabs/docker](https://forge.puppet.com/modules/p |
|
containers, volumes and so on right now. All that is (currently)
|
|
containers, volumes and so on right now. All that is (currently)
|
|
handled by GitLab runner.
|
|
handled by GitLab runner.
|
|
|
|
|
|
|
|
### Podman on Debian
|
|
|
|
|
|
|
|
A [Podman](https://podman.io/) runner was configured to see if we could workaround
|
|
|
|
limitations in image building (currently requiring Kaniko) and avoid
|
|
|
|
possible issues with Docker itself, specifically [those intermittent
|
|
|
|
failures](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41295).
|
|
|
|
|
|
|
|
The machine was built with less disk space than ci-runner-x86-01
|
|
|
|
(above), but more or less the same specifications, see [this
|
|
|
|
ticket](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41296) for details on the installation.
|
|
|
|
|
|
|
|
After installation, the following steps were taken:
|
|
|
|
|
|
|
|
1. setup the large partition in `/srv`, and bind-mount it to cover
|
|
|
|
for GitLab Runner's home which includes the Podman images:
|
|
|
|
|
|
|
|
mkfs -t ext4 -j /dev/sda
|
|
|
|
echo "/dev/sda /srv ext4 defaults 1 2" >> /etc/fstab
|
|
|
|
echo "/srv/gitlab-runner /home/gitlab-runner none bind 0 0" >> /etc/fstab
|
|
|
|
mount /srv
|
|
|
|
mount /home/gitlab-runner
|
|
|
|
|
|
|
|
2. disable module loading:
|
|
|
|
|
|
|
|
touch /etc/no_modules_disabled
|
|
|
|
reboot
|
|
|
|
|
|
|
|
... otherwise Podman will fail it will try to load extra kernel
|
|
|
|
modules.
|
|
|
|
|
|
|
|
2. add the `role::gitlab::runner` class to the node in Puppet
|
|
|
|
|
|
|
|
3. add the following blob in `tor-puppet.git`'s
|
|
|
|
`hiera/nodes/ci-runner-x86-02.torproject.org.yaml`:
|
|
|
|
|
|
|
|
profile::gitlab::runner::docker::backend: "podman"
|
|
|
|
profile::gitlab::runner::defaults:
|
|
|
|
executor: 'docker'
|
|
|
|
run_untagged: false
|
|
|
|
docker_host: "unix:///run/user/999/podman/podman.sock"
|
|
|
|
docker_tlsverify: false
|
|
|
|
docker_image: "quay.io/podman/stable"
|
|
|
|
profile::sysctl:
|
|
|
|
unprivileged_userns_clone:
|
|
|
|
key: "kernel.unprivileged_userns_clone"
|
|
|
|
value: "1"
|
|
|
|
|
|
|
|
4. run Puppet to deploy `gitlab-runner`, `podman`
|
|
|
|
|
|
|
|
5. reboot to get the user session started correctly
|
|
|
|
|
|
|
|
6. run a test job on the host
|
|
|
|
|
|
|
|
The last step, specifically, was done by removing all tags from the
|
|
|
|
runner (those were `tpa, linux, amd64, kvm, x86_64, x86-64, 16 CPU,
|
|
|
|
94.30 GiB, debug-terminal, docker`), adding a `podman` tag, and
|
|
|
|
unchecking the "run untagged jobs" checkbox in the UI.
|
|
|
|
|
|
|
|
Note that this is currently in testing, see [issue 41296](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41296) and
|
|
|
|
[TPA-RFC-58](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-58-podman-runner).
|
|
|
|
|
|
### MacOS/Windows
|
|
### MacOS/Windows
|
|
|
|
|
|
A special machine (currently `chi-node-13`) was built to allow builds
|
|
A special machine (currently `chi-node-13`) was built to allow builds
|
... | @@ -476,7 +537,9 @@ We currently use the following tags: |
... | @@ -476,7 +537,9 @@ We currently use the following tags: |
|
be added for other OS
|
|
be added for other OS
|
|
* **executor** type: `docker`, `KVM`, etc. `docker` are the typical
|
|
* **executor** type: `docker`, `KVM`, etc. `docker` are the typical
|
|
runners, `KVM` runners are possibly more powerful and can, for
|
|
runners, `KVM` runners are possibly more powerful and can, for
|
|
example, run Docker-inside-Docker (DinD)
|
|
example, run Docker-inside-Docker (DinD), note that `docker` can
|
|
|
|
also mean a `podman` runner, which is tagged on top of `docker`, as
|
|
|
|
a feature
|
|
* **memory** size: `64GB`, `32GB`, `4GB`, etc.
|
|
* **memory** size: `64GB`, `32GB`, `4GB`, etc.
|
|
* **hosting** provider:
|
|
* **hosting** provider:
|
|
* `tpa`: runners managed by the sysadmin team
|
|
* `tpa`: runners managed by the sysadmin team
|
... | @@ -491,6 +554,8 @@ We currently use the following tags: |
... | @@ -491,6 +554,8 @@ We currently use the following tags: |
|
one such job may run at a time on a given runner
|
|
one such job may run at a time on a given runner
|
|
* `verylarge`: same as `large`, with sysctl tweaks to allow high
|
|
* `verylarge`: same as `large`, with sysctl tweaks to allow high
|
|
numbers of processes (runners with >1TB memory)
|
|
numbers of processes (runners with >1TB memory)
|
|
|
|
* `podman`: a `docker` executor which talks to the `podman` socket
|
|
|
|
instead of Docker, might be better suited to build container images
|
|
|
|
|
|
Use tags in your configuration only if your job can be fulfilled by
|
|
Use tags in your configuration only if your job can be fulfilled by
|
|
only some of those runners. For example, only specify a memory tag if
|
|
only some of those runners. For example, only specify a memory tag if
|
... | @@ -572,17 +637,15 @@ had to tweak this to deal with out of disk issues. |
... | @@ -572,17 +637,15 @@ had to tweak this to deal with out of disk issues. |
|
|
|
|
|
### rootless containers
|
|
### rootless containers
|
|
|
|
|
|
We are considering [podman](https://podman.io/) for running containers more securely:
|
|
We are testing [podman](https://podman.io/) for running containers more securely:
|
|
because they can run containers "rootless" (without running as root on
|
|
because they can run containers "rootless" (without running as root on
|
|
the host), they are generally thought to be better immune against
|
|
the host), they are generally thought to be better immune against
|
|
container escapes. See [those instructions](https://github.com/jonasbb/podman-gitlab-runner). Do note that custom
|
|
container escapes.
|
|
executors have limitations that the default Docker executor do not,
|
|
|
|
see for example the [lack of ENTRYPOINT support](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27301).
|
|
|
|
|
|
|
|
This could also possibly make it easier to build containers inside
|
|
This could also possibly make it easier to build containers inside
|
|
GitLab CI, which would otherwise require docker-in-docker (DinD),
|
|
GitLab CI, which would otherwise require docker-in-docker (DinD),
|
|
unsupported by upstream. This can be done with [buildah](https://buildah.io/) using, for
|
|
unsupported by upstream. See [those GitLab instructions](https://docs.gitlab.com/runner/executors/docker.html#use-podman-to-build-container-images-from-a-dockerfile) for
|
|
example, [those instructions](https://medium.com/prgcont/using-buildah-in-gitlab-ci-9b529af19e42).
|
|
details.
|
|
|
|
|
|
### Current services
|
|
### Current services
|
|
|
|
|
... | | ... | |