Building Docker images with Gitlab CI may require User Namespace support
I tried to use buildah to build some Docker images for the network team to use for our CI with the Gitlab CI runner.
It looks like the build jobs fails because we don't have User Namespace support. See: https://gitlab.torproject.org/tpo/core/tor-ci-support/-/jobs/9686
Activity
The guide I looked for this was https://medium.com/prgcont/using-buildah-in-gitlab-ci-9b529af19e42 from Anarcat.
I just tried to pass
--isolation=chrootto buildah's build-from-dockerfiles option and it doesn't seem to resolve it
not sure we want to do this, but there's a
userns_modemode in there:https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnersdocker-section
and it might be necessary on the parent host as well...
added Icebox label
-
related: enabling the registry (#89 (closed))
-
i can confirm i cannot
unsharethings (with e.g. mmdebstrap) in CI right now: https://gitlab.torproject.org/tpo/tpa/container-images/-/jobs/43887 -
i tried to do some things to make builds work here, in general i can't even get mmdebstrap to work:
https://gitlab.torproject.org/tpo/tpa/container-images/-/jobs/43888#L787
it seems like we need to setup a special runner that would run as --privileged but not as root, so that privileged would not gain any extra privileges. this might be easier to do with podman as well. a simple way to reproduce outside gitlab CI is:
podman run -it --rm debian:latestthen:
apt update apt install mmdebstrapthen:
root@4775f7cc1056:/# mmdebstrap bullseye - > a.tar I: automatically chosen mode: root E: root mode requires mount which requires CAP_SYS_ADMIN root@4775f7cc1056:/# mmdebstrap --mode=unshare bullseye - > a.tar unshare: unshare failed: Operation not permitted E: unable to unshare the mount namespace root@4775f7cc1056:/#that's for bootstrapping an image of course, maybe buildah would be more lenient and might be able to build a tarball in some other way. but i was kind of happy about my reproducible, minimalist build script...
-
i did manage to build an image with mmdebootstrap, so that's encouraging. in other words, if you have a way of generating a tarball, you can feed it to
docker importand that works out of the box in our runners. the bootstrap job here, worked withmmdebstrap --mode=fakeroot, at the very least. so that's encouraging.but buildah does need namespace support as well, as this more involved job shows:
STEP 2: RUN apt-get update && apt-get dist-upgrade --yes --no-install-recommends --no-install-suggests && apt-get install --yes --no-install-recommends --no-install-suggests podman && apt-get clean && rm -rf /var/lib/apt/lists/* error running container: error creating new mount namespace for [/bin/sh -c apt-get update && apt-get dist-upgrade --yes --no-install-recommends --no-install-suggests && apt-get install --yes --no-install-recommends --no-install-suggests podman && apt-get clean && rm -rf /var/lib/apt/lists/*]: operation not permittedthis is probably the same "unshare" error as we were getting with
mmdebstrap --mode=unshareearlier (which we worked around withfakeroot). i did try to allow containers to do their own userns thing inside docker, but couldn't quite figure it out. i triedkernel.unprivileged_userns_clone=1(in tor-puppet.git's 08fd91af "try to enable userns mode to allow building containers inside CI") and that didn't work.i'm still unclear on the userns option in the runner configuration. i thought
--userns=hostwould enable the user namespaces inside the container, but according to this upstream doc, it's the opposite: that disables it.from what i understand, we might need a "privileged" runtime for this to work. this sounds scary to me, but i was told it might be safe if we run Docker rootless or use podman (rootless as well?), both of which require a special runner.
RedHat has an extensive blog post about the various ways podman can run inside a container which are key here. A quick read tells me the minimal "podman inside podman" option is:
podman run --security-opt label=disable --user podman --device /dev/fuse quay.io/podman/stable podman run alpine echo helloi.e.:
- no selinux
- pass-through FUSE
- rootless podman inside rootless podman
So yeah, that's about it for me right now. :)
-
RedHat has an extensive blog post about the various ways podman can run inside a container which are key here. A quick read tells me the minimal "podman inside podman" option is:
podman run --security-opt label=disable --user podman --device /dev/fuse quay.io/podman/stable podman run alpine echo helloand locally, this works.
but running
podman builddoes not work against our dockerfile. it fails so:anarcat@angela:container-images(main)$ podman run --security-opt label=disable --user podman --device /dev/fuse -v $PWD:/srv quay.io/podman/stable podman build /srv time="2021-10-22T01:06:34Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers" STEP 1/2: FROM debian:latest Resolved "debian" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf) Trying to pull docker.io/library/debian:latest... Getting image source signatures Copying blob sha256:bb7d5a84853b217ac05783963f12b034243070c1c9c8d2e60ada47444f3cce04 Copying blob sha256:bb7d5a84853b217ac05783963f12b034243070c1c9c8d2e60ada47444f3cce04 Copying config sha256:f776cfb21b5e06bb5b4883eb15c09ab928a411476b8275293c7f96d09b90f7f9 Writing manifest to image destination Storing signatures STEP 2/2: RUN apt-get update && apt-get dist-upgrade --yes --no-install-recommends --no-install-suggests && apt-get install --yes --no-install-recommends --no-install-suggests podman && apt-get clean && rm -rf /var/lib/apt/lists/* error running container: error from /usr/bin/crun creating container for [/bin/sh -c apt-get update && apt-get dist-upgrade --yes --no-install-recommends --no-install-suggests && apt-get install --yes --no-install-recommends --no-install-suggests podman && apt-get clean && rm -rf /var/lib/apt/lists/*]: mount `/proc` to `/proc`: Operation not permitted : exit status 1 Error: error building at STEP "RUN apt-get update && apt-get dist-upgrade --yes --no-install-recommends --no-install-suggests && apt-get install --yes --no-install-recommends --no-install-suggests podman && apt-get clean && rm -rf /var/lib/apt/lists/*": error while running runtime: exit status 1so even that hack doesn't work... it seems we do need
--privilegd... and this does work:podman run --privileged -v $PWD:/srv quay.io/podman/stable podman build /srvbut this doesn't:
podman run --privileged --user podman -v $PWD:/srv quay.io/podman/stable podman build /srvsame with
dockerinstead of podman, for what that's worth:anarcat@angela:container-images(main)$ docker run --privileged -v $PWD:/srv quay.io/podman/stable podman build /srv STEP 1/2: FROM debian:latest Resolved "debian" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf) Trying to pull docker.io/library/debian:latest... Getting image source signatures Copying blob sha256:bb7d5a84853b217ac05783963f12b034243070c1c9c8d2e60ada47444f3cce04 Copying blob sha256:bb7d5a84853b217ac05783963f12b034243070c1c9c8d2e60ada47444f3cce04 Copying config sha256:f776cfb21b5e06bb5b4883eb15c09ab928a411476b8275293c7f96d09b90f7f9 Writing manifest to image destination Storing signatures STEP 2/2: RUN apt-get update && apt-get dist-upgrade --yes --no-install-recommends --no-install-suggests && apt-get install --yes --no-install-recommends --no-install-suggests podman && apt-get clean && rm -rf /var/lib/apt/lists/* error running container: error from /usr/bin/crun creating container for [/bin/sh -c apt-get update && apt-get dist-upgrade --yes --no-install-recommends --no-install-suggests && apt-get install --yes --no-install-recommends --no-install-suggests podman && apt-get clean && rm -rf /var/lib/apt/lists/*]: writing file `/sys/fs/cgroup/buildah-buildah583247746/cgroup.procs`: Operation not supported : exit status 1 Error: error building at STEP "RUN apt-get update && apt-get dist-upgrade --yes --no-install-recommends --no-install-suggests && apt-get install --yes --no-install-recommends --no-install-suggests podman && apt-get clean && rm -rf /var/lib/apt/lists/*": error while running runtime: exit status 1 [125]anarcat@angela:container-images(main)$so that's ... not great.
-
The guide I looked for this was https://medium.com/prgcont/using-buildah-in-gitlab-ci-9b529af19e42 from Anarcat.
the key in that guide is that the runner has this configuration:
privileged = true... which is something we'd like to avoid because it makes container escapes easier.
it even mounts the docker socket inside the runner, which is a big no-no security-wise.
I just tried to pass
--isolation=chrootto buildah's build-from-dockerfiles option and it doesn't seem to resolve itthat said, i heard that that, combined with VFS as storage could make it work. i haven't quite figured out how to make that play either.
mentioned in issue team#40493 (closed)
added Gitlab label
mentioned in issue #89 (closed)
-
I don't think I did. I tried another tool that claimed it could do the same thing (avoid D-in-D builds without running Docker in privileged mode) but it didn't work.
Looks like this is officially supported too and have docs: https://docs.gitlab.com/ee/ci/docker/using_kaniko.html
@gk Do you need to use it for something? I'm very interested in getting this done as we could greatly speed up the netteam CI builds I think.
I am not sure what you mean with "multi-stage" builds. I checked a bit on the Internet and it seems what I found tells me we might be able to work around the problems with "multi-stage" builds. But as I said, not sure what you had in mind and I am fine trying to cross that bridge once we have to.
That said: I was able to build all images in tpo/core/tor-ci-support with kaniko in our Gitlab CI. That's pretty promising. @anarcat: I would like to test a bit more with an own Gitlab registry. Could we set something like that up for testing purposes? I'd then push the images to that registry and check whether they are usable for core Tor CI purposes. And is so, then this could be a reasonable way forward, tackling #89 (closed) as well.
Edited by Georg Koppen-
I am not sure what you mean with "multi-stage" builds. I checked a bit on the Internet and it seems what I found tells me we might be able to work around the problems with "multi-stage" builds. But as I said, not sure what you had in mind and I am fine trying to cross that bridge once we have to.
I am actually not absolutely certain about this, because it's based on something @hacim said, but in my book, multi-stage Docker builds are basically building a (or, multiple, to be accurate) Docker containers from a single Docker file which contains multiple
FROMstatements. The example in the above linked documentation is:# syntax=docker/dockerfile:1 FROM golang:1.16 WORKDIR /go/src/github.com/alexellis/href-counter/ RUN go get -d -v golang.org/x/net/html COPY app.go ./ RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o app . FROM alpine:latest RUN apk --no-cache add ca-certificates WORKDIR /root/ COPY --from=0 /go/src/github.com/alexellis/href-counter/app ./ CMD ["./app"]That said: I was able to build all images in tpo/core/tor-ci-support with kaniko in our Gitlab CI. That's pretty promising. @anarcat: I would like to test a bit more with an own Gitlab registry. Could we set something like that up for testing purposes? I'd then push the images to that registry and check whether they are usable for core Tor CI purposes. And is so, then this could be a reasonable way forward, tackling #89 (closed) as well.
That is super exciting! I'll check with the team before enabling the registry though. We already have a lot of disk issues and I just want to make sure this doesn't grow nuts either...
and yeah, maybe that part of the conversation would belong to #89 (closed)... could you use Docker hub in the short term?
-
It's been a bit of a hassle due to a kaniko bug but I managed to get that work and tested tor's CI with my kaniko-built oldstable image: all is still green. So, I think this is a viable way forward, in particular as I tried out multi-stage builds as well and they worked fine.
@anarcat: as you are a fan of closing tickets I believe we are done here. ;p
-
@gk Do you need to use it for something? I'm very interested in getting this done as we could greatly speed up the netteam CI builds I think.
I want to use it for getting us away from our YOLO-mode wrt to Docker images we are currently in. :) Like, using random images from wherever which do whatever they are doing seems not an ideal solution for us as a project. I don't have a direct use for that for may day-to-day work which is why I need to work on that topic in my spare time/on my weekend (with all the caveats that implies).
But if the side-effect of that effort is greatly speeding up the network team's CI builds then even better. :)
-
Georg Koppen commented:
@gk Do you need to use it for something? I'm very interested in getting this done as we could greatly speed up the netteam CI builds I think.
I want to use it for getting us away from our YOLO-mode wrt to Docker images we are currently in. :) Like, using random images from wherever which do whatever they are doing seems not an ideal solution for us as a project. I don't have a direct use for that for may day-to-day work which is why I need to work on that topic in my spare time/on my weekend (with all the caveats that implies).
If you have the capacity to work on this, I think you should, as part of your day-to-day work, to be honest. It might not immediately be obvious to you that it will be part of your work, but I am absolutely certain it eventually will. ;)
But if the side-effect of that effort is greatly speeding up the network team's CI builds then even better. :)
That too: if it helps other teams, i think it can pass as work. ;)
a.
...
On 2022-02-08 07:51:06, Georg Koppen (@gk) wrote:
-- Antoine Beaupré torproject.org system administration
# syntax=docker/dockerfile:1 FROM golang:1.16 WORKDIR /go/src/github.com/alexellis/href-counter/ RUN go get -d -v golang.org/x/net/html COPY app.go ./ RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o app . FROM alpine:latest RUN apk --no-cache add ca-certificates WORKDIR /root/ COPY --from=0 /go/src/github.com/alexellis/href-counter/app ./ CMD ["./app"]That does work, but I think its more legible to label your stages like this:
FROM golang:1.16 AS build ... COPY --from=build /go/src/github.com/alexellis/href-counter/app ./
The basic idea of these is that you can build in one layer, then take what you want from that layer, and then discard everything else. You end up with only the specific binary in your final layer. Although if your final one is all of
alpine:latestyou probably aren't saving a whole lot.Regarding multi-stage build problems with kaniko, I used kaniko for a long time without issues, but if you browse their issue tracker, just for the 'stage' keyword, you will see people have a lot of issues:
https://github.com/GoogleContainerTools/kaniko/issues?q=is%3Aissue+is%3Aopen+stage
for me it was just frustrating because it wasn't doing what I expected it to do, and it took a long time to finally peel back the layers to determine the problem wasn't with something I was doing, but a problem with kaniko itself. Once I got to that layer, I realized my only solution was to not use kaniko, which I've stuck to since.
-
[...]
for me it was just frustrating because it wasn't doing what I expected it to do, and it took a long time to finally peel back the layers to determine the problem wasn't with something I was doing, but a problem with kaniko itself. Once I got to that layer, I realized my only solution was to not use kaniko, which I've stuck to since.
... but that means you need a privileged runner to build images, which is what we're trying to avoid here. the point of using kaniko is that it seems to be the only solution that allows normal runners to build images.
maybe we could use kaniko for most containers and have exceptions for staged builds?
...
On 2022-03-08 01:47:37, Micah Anderson (@hacim) wrote:
-- Antoine Beaupré torproject.org system administration
-
... but that means you need a privileged runner to build images, which is what we're trying to avoid here.
You only need a privileged runner to build certain types of images (such as when you are using the Docker-in-Docker service).
Maybe for people who need privileged runners, they can just bring their own?
-
for me it was just frustrating because it wasn't doing what I expected it to do, and it took a long time to finally peel back the layers to determine the problem wasn't with something I was doing, but a problem with kaniko itself. Once I got to that layer, I realized my only solution was to not use kaniko, which I've stuck to since.
... but that means you need a privileged runner to build images, which is what we're trying to avoid here. the point of using kaniko is that it seems to be the only solution that allows normal runners to build images.
maybe we could use kaniko for most containers and have exceptions for staged builds?
Staged builds work as far as I can tell. I played with the example you linked to (https://docs.docker.com/develop/develop-images/multistage-build/) including @hacim's proposed legibility improvements and, while it was a bit of a PITA to get that going (which was mostly due to the example not being up-to-date in Docker's doc), eventually it worked fine in a kaniko context. So, there is no need to think about any potential exceptions at this point IMO.
That said what I envision is that we assemble our kaniko images we use as base images ourselves with a reproducible build process. So, we have the option as well to be a proper free software project and fix stuff that breaks for us and get those fixes upstreamed. (Yes, I know that's easily said but I'd be willing to take on that responsibility if the need would arise at some point).
-
That said what I envision is that we assemble our kaniko images we use as base images ourselves with a reproducible build process. So, we have the option as well to be a proper free software project and fix stuff that breaks for us and get those fixes upstreamed. (Yes, I know that's easily said but I'd be willing to take on that responsibility if the need would arise at some point).
that sounds like a nice ticket to open. ;)
otherwise, yes, this seems like it's a ticket that could be closed, but i think we should document this more clearly in our documentation.
@gk could you see if you can submit a MR on the wiki replica to share how you did your magic "hey i built a docker container in gitlab ci" sauce? probably as a patch to
service/ci.md...thanks!
assigned to @gk
mentioned in merge request tpo/core/tor!476 (closed)
added First Contribution label
mentioned in issue #123 (closed)
-
@gk could you see if you can submit a MR on the wiki replica to share how you did your magic "hey i built a docker container in gitlab ci" sauce? probably as a patch to
service/ci.md...thanks!
Done: wiki-replica!28 (merged).
I have no clue how to test that one though apart from making sure CI is reasonably happy....
mentioned in merge request wiki-replica!28 (merged)
mentioned in commit wiki-replica@bae52faa
closed with merge request wiki-replica!28 (merged)
mentioned in issue #129 (moved)
marked this issue as related to #89 (closed)
-
for those who failed to build images with buildah in the past, you might want to check out this blog post:
https://arnaudr.io/2023/01/18/build-container-images-in-gitlab-ci-iptables-legacy-at-the-rescue/
it looks like you need to install iptables and set iptables-legacy as the default to get things to work, basically...
build-container-image: stage: build image: debian:testing before_script: - apt-get update - apt-get install -y buildah ca-certificates - | # Switch to iptables legacy, as GitLab CI doesn't support nftables. apt-get install -y --no-install-recommends iptables update-alternatives --set iptables /usr/sbin/iptables-legacy script: - buildah build -t $CI_REGISTRY_IMAGE . - buildah login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY - buildah push $CI_REGISTRY_IMAGEthat applies to gitlab.com, but it may be the same situation in our runners as well.
marked this issue as related to team#41296 (closed)
mentioned in issue team#41296 (closed)
marked this issue as related to team#41295 (closed)
mentioned in issue team#41295 (closed)
mentioned in issue team#41327 (closed)
marked this issue as related to team#41328 (closed)
mentioned in issue team#41328 (closed)