Skip to content

retire Debian package builds from jenkins

As part of the Jenkins retirement work (#40218 (closed)), we want to retire the Debian package builds from Jenkins. Those are:

https://jenkins.torproject.org/view/packages%20-%20debian/

it's basically three jobs, with two also built on ARM, so 5 jobs total. we need to figure out whether we just stop building those packages automatically, or migrate them to GitLab CI, and if so, how.

Q&A with @weasel

  • the builds run on our infra? yes, ci-runner-01 right now. there's also a runner provided by f-droid and users can register their own runners as well, although projects need to opt in to the "shared runners", iirc
  • it is not trivial to build trust in the CI chain: we depend on the main GitLab app and while we can build our own hardened docker images, those get selected by the GitLab app, ultimately, since it is what feeds the .gitlab-ci.yml script into the runner
  • the runners are actually a fairly large attack surface themselves, 800,000 SLOC, but 90% is vendored golang code though, so it's actually more like 80k
  • i was hoping the runners could check openpgp signatures on the git repo, so that we wouldn't have to trust the main gitlab app stuff, but that is just an idea
  • we could have runners specifically for debian packages
  • the .gitlab-ci.yml could verify signatures on Git repositories but that's kind of silly, because the script is shipped from the gitlab app, so it doesn't protect against a compromise there
  • what are runners? gitlab runner is a golang project, separate from the main gitlab. you install it inside a machine, virtual or not (it can run on windows and mac and linux, i think). it can fire jobs into different "backends" which it calls "executors" (like Docker, kubernetes, shell, etc), we use Docker. it basically pipes a shell script (built from the gitlab-ci.yml) into that thing
  • do we build our own docker image? no, not yet. ideally, we would. ideally, we would never use images straight from hub.docker.com and build our own ecosystem of images, built from scratch or from debootstrap
  • the docker image is specified in the .gitlab-ci.yml file. but through Docker image policies, it might be possible for specific runners to be restricted to specific, controlled Docker images.
  • Debian packages could be built for different suites and (hopefully) different arches with different Docker images (instead of using schroot/sbuild), not sure if Docker can run on different arches, e.g. i386 on amd64 or armhf on arm64. you assume a clean environment, as the Docker image is basically a clean chroot.
  • can runners build from multiple git repositories? not directly: you get a shell with your project's git repository checked out, but you can clone other repositories (because you have network access). git submodules might work around possible future restrictions on networking.

Update: a subset of those was added as a FAQ in the CI docs.

Possible options:

  1. deb.torproject.org switches to manual uploads, no GitLab CI integration
  2. retire deb.tpo altogether, users go to Debian and Ubuntu for their packages, no GitLab CI integration
  3. retire deb.tpo like 2, except option to get .debs from GitLab CI artefacts (but no APT repository)
  4. add (web)hook to trigger a mirror sync from the GitLab artifacts to the deb.tpo archive
  5. add webhook to trigger build on a separate, non-GitLab CI system

I favor option 3, but I will leave the decision to the deb.tpo maintainer (@weasel) of course, although I would need a very strong reason to go the option 5 way, because it feels it would require a parallel infrastructure just for Debian packages...

One key problem is that Debian packages are built for multiple architectures and Debian derivatives/suites (Ubuntu/Debian releases, mostly), which has been abstracted away above. Right now we have:

  • GitLab runners:
    • only amd64
  • Jenkins builders:
    • amd64
    • i386 chroots on amd64
    • arm64
    • armhf/armel were retired

In tpo/core/tor#40347, the network team is looking at options to build their stuff on ARM again, in the GitLab CI world. So that's something to keep in mind, and that Debian package builds could reuse. We could already setup a gitlab-runner on build-arm-10 to have ARM builds on GitLab CI, although that feels a bit messy.

Options for distros:

  • Docker image per distro
  • ?

Options for arches:

  • amd64: native
  • i386: special Docker image? drop support?
  • arm64: native
  • armhf/armel: special hardware? same as i386?
Edited by anarcat