[Continuous Integration](https://en.wikipedia.org/wiki/Continuous_integration) is the system that allows tests to be ran
and packages to be built, automatically, when new code is pushed to
the version control system (currently [git](howto/git)).

Note that the CI system is implemented with [GitLab](howto/gitlab),
which has its own documentation. This page, however, documents the
GitLab CI things specific to TPA.

This service was setup as a replacement to the previous CI system,
[Jenkins][], which has its own documentation, for historical purposes.

[Jenkins]: service/jenkins

[[_TOC_]]

# Tutorial

[GitLab CI][GitLab CI splash] has [good documentation upstream][GitLab CI upstream]. This section
documents frequent questions we might get about the work.

<!-- simple, brainless step-by-step instructions requiring little or -->
<!-- no technical background -->

## Getting started

The [GitLab CI quickstart][] should get you started here. Note that
there are some "shared runners" you can already use, and which should
be available to all projects. So your main task here is basically to
[write a `.gitlab-ci.yml` file](https://docs.gitlab.com/ee/ci/quick_start/README.html#create-a-gitlab-ciyml-file).

# How-to

<!-- more in-depth procedure that may require interpretation -->

## Why is my CI job not running?

There might be too many jobs in the queue. You can monitor the queue
in our [Grafana dashboard](https://grafana.torproject.org/d/QrDJktiMz/gitlab-omnibus).

## Enabling/disabling runners

If a runner is misbehaving, it might be worth "pausing" it while we
investigate, so that jobs don't all fail on that runner. For this,
head for the [runner admin interface](https://gitlab.torproject.org/admin/runners) and hit the "pause" button on
the runner.

## Registering more runners

Anyone can run their own personal runner in their own infrastructure
and register them inside a project on our GitLab instance. For this
you need to first [install a runner](https://docs.gitlab.com/runner/install/) and [register it in
GitLab](https://docs.gitlab.com/runner/register/). But we already have shared runners, if they are not
sufficient, it might be best to request a new one from TPA.

## Converting a Jenkins job

See [static-shim](service/static-shim) for how to migrate jobs from
Jenkins.

## Finding largest volumes users

See [Runner disk fills up](#runner-disk-fills-up).

## Running a job locally

If you are having trouble reproducing a problem or just want to run a
job in your local, trusted environment, know that you can [install
your own GitLab runner](https://docs.gitlab.com/runner/install/index.html) and actually use it to run jobs locally,
without registering it with the central server. The magic command is
[`gitlab-runner exec`](https://docs.gitlab.com/runner/commands/#gitlab-runner-exec). For example, this will run a job named
`tests` with a `shell` executor:

    gitlab-runner exec shell tests

To see a list of available executors, run:

    gitlab-runner exec

To see a list of all available options for the shell executor, run:

    gitlab-runner exec shell

Note that this feature was deprecated for a while but re-enabled. It
is being [redesigned](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2797) to be more complete, as the above has
important limitations.

## FAQ

 * do runners have **network access**? **yes**, but that might
   eventually change
 * how to build from **multiple git repositories**? install `git` and
   clone the extra repositories. using git submodules might work
   around eventual network access restrictions
 * how do I **trust runners**? you can setup your own runner for your
   own project in the GitLab app, but in any case you need to trust
   the GitLab *app*. we are considering options for this, see
   [security](#security)
 * how do i control the image used by the runners? the docker image is
   specified in the `.gitlab-ci.yml` file. but through Docker image
   policies, it might be possible for specific runners to be
   restricted to specific, controlled, Docker images.
 * do we provide, build, or host our own **Docker images**? **not
   yet** (but see how to [build Docker images with kaniko](#build-docker-images-with-kaniko)
   below). ideally, we would never use images straight from
   hub.docker.com and build our own ecosystem of images, built `FROM
   scratch` or from `debootstrap`

## Pager playbook

<!-- information about common errors from the monitoring system and -->
<!-- how to deal with them. this should be easy to follow: think of -->
<!-- your future self, in a stressful situation, tired and hungry. -->

### A runner fails all jobs

[Pause the runner](#enabling/disabling-runners).

### Jobs pile up

If too many jobs pile up in the queue, consider inspecting which jobs
those are in the [job admin interface](https://gitlab.torproject.org/admin/jobs). Jobs can be canceled there
by GitLab admins. For really long jobs, consider talking with the
project maintainers and see how those jobs can be optimized.

### Runner disk fills up

If you see a warning like:

    DISK WARNING - free space: /srv 6483 MB (11% inode=82%):

It's because the runner is taking up all the disk space. This is
usually containers, images, or caches from the runner. Those are
normally [purged regularly](#image-volume-and-container-storage-and-caching) but some extra load on the CI system
might use up too much space all of a sudden.

To diagnose this issue better, you can see the running containers
with:

    docker ps

... and include stopped or dead containers with:

    docker ps -a

Images are visible with:

    docker images

And volumes with:

    docker volume ls

... although that output is often not very informative because GitLab
runner uses volumes to cache data and uses opaque volume names.

If there are any obvious offenders, they can be removed with `docker
rm` (for containers), `docker image rm` (for images) and `docker
volume rm` (for volumes). But usually, you should probably just run
the cleanup jobs by hand, in order:

    docker system prun --filter until=72h

The time frame can be lowered for a more aggressive cleanup.

Alternatively, this will also clean old containers:

    /usr/local/sbin/tpo-docker-clean-cache

The `tpa-du-gl-volumes` script can also be used to analyse which
project is using the most disk space. Then those pipelines can be
adjusted to cache less.

### Disk full on GitLab server

Similar to the above, but typically happens on the GitLab
server. Documented in the GitLab documentation, see [Disk full on
GitLab server](howto/gitlab#disk-full-on-gitlab-server).

### DNS resolution failures

Under certain circumstances (upgrades?) Docker loses DNS resolution
(and possibly all of networking?). A symptom is that it simply fails
to clone the repository at the start of the job, for example:

    fatal: unable to access 'https://gitlab-ci-token:[MASKED]@gitlab.torproject.org/tpo/network-health/sbws.git/': Could not resolve host: gitlab.torproject.org

A workaround is to reboot the runner's virtual machine. It might be
that we need to do some more configuration of Docker, see [upstream
issue 6644](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6644), although it's unclear why this problem is happening
right now. Still to be more fully investigated, see [tpo/tpa/gitlab#93](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/93).

### "unadvertised object" error

If a project's pipeline fails to clone submodules with this error:

```
Updating/initializing submodules recursively with git depth set to 1...
Submodule 'lego' (https://git.torproject.org/project/web/lego.git) registered for path 'lego'
Cloning into '/builds/tpo/web/tpo/lego'...
error: Server does not allow request for unadvertised object 0d9efebbaec064730fba8438dda2d666585247a0
Fetched in submodule path 'lego', but it did not contain 0d9efebbaec064730fba8438dda2d666585247a0. Direct fetching of that commit failed.
```

that is because the depth configuration is too shallow. In the above,
we see:

```
Updating/initializing submodules recursively with git depth set to 1...
```

In this case, the submodule is being cloned with only the latest
commit attached. If the project refers to a previous version of that
submodule, this will fail.

To fix this, change the `Git shallow clone` value to a higher one. The
default is 50, but you can set it to zero or empty to disable shallow
clones. See also "[Limit the number of changes fetched during
clone](https://gitlab.torproject.org/help/ci/pipelines/settings#limit-the-number-of-changes-fetched-during-clone)" in the upstream documentation.

## Disaster recovery

Runners should be disposable: if a runner is destroyed, at most the
jobs it is currently running will be lost. Otherwise artifacts should
be present on the GitLab server, so to recover a runner is as "simple"
as creating a new one.

## Build Docker images with kaniko

It is possible do build Docker images in our Gitlab CI without requiring user
namespace support using [kaniko](https://github.com/GoogleContainerTools/kaniko).
The Gitlab documentation [has examples](https://docs.gitlab.com/ee/ci/docker/using_kaniko.html)
to get started with that task. There are some caveats, though, at the moment:

1. One needs to pass `--force` to kaniko's executor or use a different
   workaround due to a [bug in kaniko](https://github.com/GoogleContainerTools/kaniko/issues/1542)
2. Pushing images to the Docker hub is not working out of the box. One rather
   needs to [use the v1 endpoint](https://github.com/GoogleContainerTools/kaniko/issues/1209) at the moment due to a bug. Right now passing something like

   ```
   --destination "index.docker.io/gktpo/${CI_REGISTRY_IMAGE}:oldstable"
   ```

   to kaniko's executor does the trick for me.

Additionally, as we want to build our images reproducibly, passing
`--reproducible` to the executor is recommended as well.

One final note: the Gitlab CI examples show that a debug image is used as a base
image in Gitlab CI. That is important as the non-debug flavor does not come with
a shell which is a requirement for Gitlab CI.

# Reference

## Installation

Since GitLab CI is basically GitLab with external runners hooked up to
it, this section documents how to install and register runners into
GitLab.

### Docker on Debian

A first runner (`ci-runner-01`) was setup by Puppet in the `gnt-chi`
cluster, using this command:

    gnt-instance add \
          -o debootstrap+buster \
          -t drbd --no-wait-for-sync \
          --net 0:ip=pool,network=gnt-chi-01 \
          --no-ip-check \
          --no-name-check \
          --disk 0:size=10G \
          --disk 1:size=2G,name=swap \
          --disk 2:size=60G \
          --backend-parameters memory=64g,vcpus=8 \
          ci-runner-01.torproject.org

The `role::gitlab::runner` Puppet class deploys the GitLab runner code
and hooks it into GitLab. It uses the
[gitlab_ci_runner](https://forge.puppet.com/modules/puppet/gitlab_ci_runner)
module from Voxpupuli to avoid reinventing the wheel. But before
enabling it on the instance, the following operations need to be
performed:

 1. setup the large partition in `/srv`, and bind-mount it to cover
    for Docker:

        mkfs -t ext4 -j /dev/sdc
        echo "/dev/sdc	/srv	ext4	defaults	1	2" >> /etc/fstab
        echo "/srv/docker	/var/lib/docker	none	bind	0	0" >> /etc/fstab
        mount /srv
        mount /var/lib/docker

 2. disable module loading:

        touch /etc/no_modules_disabled
        reboot

    ... otherwise the Docker package will fail to install because it
    will try to load extra kernel modules.

 3. the default `gitlab::runner` role deploys a single docker runner
    on the host. For group- or project-specific runners which need
    special parameters (eg. for Docker), then a new role may be created
    to pass those to the `profile::gitlab::runner` class using Hiera.
    See `hiera/roles/gitlab::runner::shadow.yaml` for an example.

 4. *ONLY THEN* the Puppet agent may run to configure the executor,
    install `gitlab-runner` and register it with GitLab.

NOTE: we originally used the Debian packages ([docker.io](https://tracker.debian.org/pkg/docker.io) and
[gitlab-runner](https://tracker.debian.org/gitlab-runner)) instead of the upstream official packages, because
those have a somewhat [messed up installer](https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh) and [weird key
deployment policies](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3897). In other words, we would rather avoid having
to trust the upstream packages for runners, even though we use them
for the GitLab omnibus install. The Debian packages are both somewhat
out of date, and the latter is not available in Debian buster (current
stable), so it had to be installed from bullseye.

UPDATE: the above turned out to fail during the bullseye freeze
(2021-04-27), as gitlab-runner was [removed from bullseye](https://tracker.debian.org/news/1239007/gitlab-ci-multi-runner-removed-from-testing/),
[because of an unpatched security issue](https://bugs.debian.org/985377). We have switched to the
upstream Debian packages, since they are used for GitLab itself
anyways, which is unfortunate, but will have to do for now.

We also avoided using the [puppetlabs/docker](https://forge.puppet.com/modules/puppetlabs/docker) module because we
"only" need to setup Docker, and not specifically deal with
containers, volumes and so on right now. All that is (currently)
handled by GitLab runner.

### MacOS/Windows

A special machine (currently `chi-node-13`) was built to allow builds
to run on MacOS and Windows virtual machines. The machine was
installed in the Cymru cluster (so following
[new-machine-cymru](howto/new-machine-cymru)). On top of that procedure, the following extra
steps were taken on the machine:

 1. a bridge (`br0`) was setup
 2. a basic libvirt configuration was built in Puppet (within
    `roles::gitlab::ci::foreign`)

The `gitlab-ci-admin` role user and group have access to the
machine.

TODO: The remaining procedure still needs to be implemented and
documented, here, and eventually converted into a Puppet manifest, see
[issue 40095](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40095). @ahf document how MacOS/Windows images are created
and runners are setup. don't hesitate to create separate headings for
Windows vs MacOS and for image creation vs runner setup.

## SLA

The GitLab CI service is offered on a "best effort" basis and might
not be fully available.

## Design

The CI service was served by [Jenkins][] until the end of the [2021
roadmap](roadmap/2021). This section documents how the new GitLab CI
service is built. See [Jenkins section](#jenkins) below for more
information about the old Jenkins service.

### GitLab CI architecture

GitLab CI sits somewhat outside of the main GitLab architecture, in
that it is not featured prominently in the [GitLab architecture
documentation](https://docs.gitlab.com/ee/development/architecture.html). In practice, it is a core component of GitLab in
that the continuous integration and deployment features of GitLab have
become a key feature and selling point for the project.

GitLab CI works by scheduling "pipelines" which are made of one or
many "jobs", defined in a project's git repository (the
[`.gitlab-ci.yml`](https://docs.gitlab.com/ee/ci/yaml/) file). Those jobs then get picked up by one of
many "runners". Those runners are separate processes, usually running
on a different host than the main GitLab server.

GitLab runner is a program written in Golong which clocks at about
800,000 SLOC, including vendored dependencies, 80,000 SLOC without.

Runners regularly poll the central GitLab for jobs and execute those
inside an "[executor](https://docs.gitlab.com/runner/executors/README.html)". We currently support only "Docker" as an
executor but are working on different ones, like a custom "podman"
(for more trusted runners, see below) or KVM executor (for foreign
platforms like MacOS or Windows).

What the runner effectively does is basically this:

 1. it fetches the git repository of the project
 2. it runs a sequence of shell commands on the project inside the
    executor (e.g. inside a Docker container) with [specific
    environment variables](https://docs.gitlab.com/ee/ci/variables/README.html#gitlab-cicd-environment-variables) populated from the project's settings
 3. it collects artifacts and logs and uploads those back to the main
    GitLab server

The jobs are therefore affected by the `.gitlab-ci.yml` file but also
the configuration of each project. It's a simple yet powerful design.

### Types of runners

There are three types of runners:

 * **shared**: "shared" across all projects, they will pick up any
   job from any project
 * **group**: those are restricted to run jobs only within a
   specific group
 * **project**: those will only run job within a specific project

In addition, jobs can be targeted at specific runners by assigning
them a "tag".

### Runner tags

Whether a runner will pick a job depends on a few things:

 * if it is a "shared", "project" or "group-"specific runner (above)
 * if it has a tag matching the [`tags` field in the configuration](https://docs.gitlab.com/ee/ci/yaml/#tags)

We currently use the following tags:

 * **architecture**:
   * `amd64`: popular [64-bit Intel/AMD architecture](https://en.wikipedia.org/wiki/X86-64) (equivalents: `x86_64` and `x86-64`)
   * `aarch64`: the [64-bit ARM extension](https://en.wikipedia.org/wiki/AArch64) (equivalents: `arm64` and `arm64-v8a`)
   * `ppc64le`: [IBM Power architecture](https://en.wikipedia.org/wiki/Ppc64)
   * `s390x`: [Linux on IBM Z architecture](https://en.wikipedia.org/wiki/Linux_on_IBM_Z)
 * **OS**: `linux` is usually implicit but other tags might eventually
   be added for other OS
 * **executor** type: `docker`, `KVM`, etc. `docker` are the typical
   runners, `KVM` runners are possibly more powerful and can, for
   example, run Docker-inside-Docker (DinD)
 * **memory** size: `64GB`, `32GB`, `4GB`, etc.
 * **hosting** provider:
   * `tpa`: runners managed by the sysadmin team
   * `fdroid`: provided as a courtesy by the [F-Droid project](https://f-droid.org/)
   * `osuosl`: runners provided by the [OSUOSL](https://osuosl.org/)
 * **features**:
   * `privileged`: those containers have actual root access and should
     explicitly be able to run `DinD`
   * `debug-terminal`: supports [interactively debugging
     jobs](https://docs.gitlab.com/ee/ci/interactive_web_terminal/)

Use tags in your configuration only if your job can be fulfilled by
only some of those runners. For example, only specify a memory tag if
your job requires a lot of memory.

### Upstream release schedules

GitLab CI is an integral part of GitLab itself and gets released along
with the core releases. GitLab runner is a [separate software
project](https://gitlab.com/gitlab-org/gitlab-runner) but usually gets released alongside GitLab.

### Security

We do not currently trust GitLab runners for security purposes: at
most we trust them to correctly report errors in test suite, but we do
not trust it with compiling and publishing artifacts, so they have a
low value in our trust chain.

This might eventually change: we may eventually want to build
artefacts (e.g. tarballs, binaries, Docker images!) through GitLab CI
and even deploy code, at which point GitLab runners could actually
become important "trust anchors" with a smaller attack surface than
the entire GitLab infrastructure.

The tag-, group-, and project- based allocation of runners is based on
a secret token handled on the GitLab server. It is technically
possible for an attacker to compromise the GitLab server and access a
runner, which makes those restrictions depend on the security of the
GitLab server as a whole. Thankfully, the [permission model](https://docs.gitlab.com/ee/user/project/new_ci_build_permissions_model.html) of
runners now actually reflects the permissions in GitLab itself, so
there are some constraints in place.

Inversely, if a runner's token is leaked, it could be used to
impersonate the runner and "steal" jobs from projects. Normally,
runners do not leak their own token, but this could happen through,
for example, a virtualization or container escape.

Runners currently have full network access: this could be abused by an
hostile contributor to use the runner as a start point for scanning or
attacking other entities on the network, and even without our
network. We might eventually want to firewall runners to prevent them
from accessing certain network resources, but that is currently not
implemented.

The [runner documentation](https://docs.gitlab.com/runner/) has a [section on security](https://docs.gitlab.com/runner/security/) which
this section is based on.

### Image, volume and container storage and caching

GitLab runner creates quite a few containers, volumes and images in
the course of its regular work. Those tend to pile up, unless they get
cleaned. [Upstream suggests](https://docs.gitlab.com/runner/executors/docker.html#clearing-docker-cache) a [fairly naive shell script](https://gitlab.com/gitlab-org/gitlab-runner/blob/master/packaging/root/usr/share/gitlab-runner/clear-docker-cache) to do
this cleanup, but it has a number of issues:

 1. it is noisy (tried to [patch this locally with this MR](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/2711), but
    was refused upstream)
 2. it might be too aggressive

Also note that documentation on this inside GitLab runner is
inconsistent at best, see [this other MR](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/2711) and [this issue](https://gitlab.com/gitlab-org/gitlab-runner-docker-cleanup/-/issues/21).

So we're not using the upstream cleanup script, and we suspect
upstream itself is not using it at all (i.e. on `gitlab.com`) because
it's fundamentally ineffective.

Instead, we have a set of cron jobs (in
`profile::gitlab::runner::docker`) which does the following:

 1. clear all volumes and dead containers, daily (equivalent of the
    upstream clear-docker-cache for volumes, basically)
 2. clear images older than 30 days, daily (unless used by a running
    container)
 3. clear all dangling (ie. untagged) images, daily
 4. clear all "nightly" images, daily

Note that this documentation might be out of date and the Puppet code
should be considered authoritative on this policy, as we've frequently
had to tweak this to deal with out of disk issues.

### rootless containers

We are considering [podman](https://podman.io/) for running containers more securely:
because they can run containers "rootless" (without running as root on
the host), they are generally thought to be better immune against
container escapes. See [those instructions](https://github.com/jonasbb/podman-gitlab-runner). Do note that custom
executors have limitations that the default Docker executor do not,
see for example the [lack of ENTRYPOINT support](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27301).

This could also possibly make it easier to build containers inside
GitLab CI, which would otherwise require docker-in-docker (DinD),
unsupported by upstream. This can be done with [buildah](https://buildah.io/) using, for
example, [those instructions](https://medium.com/prgcont/using-buildah-in-gitlab-ci-9b529af19e42).

### Current services

GitLab CI, at TPO, currently runs the following services:

 * continuous integration: mostly testing after commit

This is currently used by many teams and is quickly becoming a
critical service.

### Possible services

It could eventually also run those services:

 * web page hosting through GitLab pages or the existing static site
   system. this is a requirement to replace Jenkins
 * continuous deployment: applications and services could be deployed
   directly from GitLab CI/CD, for example through a Kubernetes
   cluster or just with plain Docker
 * artifact publication: tarballs, binaries and Docker images could be
   built by GitLab runners and published on the GitLab server (or
   elsewhere). this is a requirement to replace Jenkins

## Issues

[File][] or [search][] for issues in our [GitLab issue
tracker][search] with the ~CI label. Upstream has of course an [issue tracker for GitLab
runner](https://gitlab.com/gitlab-org/gitlab-runner/-/issues) and a [project page](https://gitlab.com/gitlab-org/gitlab-runner).

 [File]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/new
 [search]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues?label_name%5B%5D=CI

### Known upstream issues

 * job log files (`job.log`) do *not* get automatically purged, even
   if their related artifacts get purged (see [upstream feature
   request 17245](https://gitlab.com/gitlab-org/gitlab/-/issues/17245)).

 * the web interface might not correctly count disk usage of objects
   related to a project ([upstream issue 228681](https://gitlab.com/gitlab-org/gitlab/-/issues/228681)) and certainly
   doesn't count container images or volumes in disk usage

 * [kept artifacts cannot be unkept](https://gitlab.com/gitlab-org/gitlab/-/issues/289954)

## Monitoring and testing

To test a runner, it can be registered only with a project, to run
non-critical jobs against it. See the [installation section](#Installation) for
details on the setup.

Monitoring is otherwise done through Prometheus, on a need-to basis,
see the [log and metrics](#log-and-metrics) section below.

## Logs and metrics

GitLab runners send logs to `syslog` and `systemd`. They contain minimal
private information: the most I could find were Git repository and
Docker image URLs, which do contain usernames. Those end up in
`/var/log/daemon.log`, which gets rotated daily, with a one-week
retention.

The GitLab instance exports a set of metrics to monitor CI. For
example, `ci_pending_builds` shows the size of the queue,
`ci_running_builds` shows the number of currently running builds,
etc. Those are visible in the [GitLab grafana dashboard](https://grafana.torproject.org/d/QrDJktiMz/gitlab-omnibus),
particularly in [this view](https://grafana.torproject.org/d/QrDJktiMz/gitlab-omnibus?orgId=1&refresh=1m&var-node=gitlab-02.torproject.org).

Other metrics might become available in the future: for example,
runners can export their own Prometheus metrics, but currently do
not. They are, naturally, monitored through the `node-exporter` like
all other TPO servers, however.

We may eventually monitor GitLab runners directly; they can be
configured to expose metrics through a Prometheus exporter. The Puppet
module supports this through the `gitlab_ci_runner::metrics_server`
variable, but we would need to hook it into our server as well. See
also [the upstream documentation](https://docs.gitlab.com/runner/monitoring/README.html). Right now it feels the existing
"node"-level and the GitLab-level monitoring in Prometheus is
sufficient.

## Backups

This service requires no backups: all configuration should be
performed by Puppet and/or documented in this wiki page. A lost runner
should be rebuilt from scratch, as per [disaster recover](#disaster-recovery).

## Other documentation

 * [GitLab CI promotional page][GitLab CI splash]
 * [GitLab CI upstream documentation portal][GitLab CI upstream]
   * [GitLab CI quickstart][]

[GitLab CI upstream]: https://docs.gitlab.com/ee/ci/
[GitLab CI splash]: https://about.gitlab.com/stages-devops-lifecycle/continuous-integration/
[GitLab CI quickstart]: https://docs.gitlab.com/ee/ci/quick_start/README.html

# Discussion

Tor currently previously used [Jenkins][] to run tests, builds and
various automated jobs. This discussion was about if and how to
replace this with GitLab CI. This was done and GitLab CI is now the
preferred CI tool.

## Overview

<!-- describe the overall project. should include a link to a ticket -->
<!-- that has a launch checklist -->

Ever since the [GitLab migration](howto/gitlab), we have discussed the
possibility of replacing Jenkins with GitLab CI, or at least using
GitLab CI in some way.

Tor currently utilizes a mixture of different CI systems to ensure
some form of quality assurance as part of the software development
process:

- Jenkins (provided by TPA)
- Gitlab CI (currently Docker builders kindly provided by the FDroid
  project via Hans from The Guardian Project)
- Travis CI (used by some of our projects such as tpo/core/tor.git for
  Linux and MacOS builds)
- Appveyor (used by tpo/core/tor.git for Windows builds)

By the end of 2020 however, [pricing changes at Travis
CI](https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing) made it difficult for the network team to continue running the
Mac OS builds there. Furthermore, it was felt that Appveyor was too
slow to be useful for builds, so it was proposed ([issue 40095][]) to
create a pair of bare metal machines to run those builds, through a
`libvirt` architecture. This is an exception to [TPA-RFC 7: tools](policy/tpa-rfc-7-tools)
which was formally proposed in [TPA-RFC-8][].

[issue 40095]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40095
[TPA-RFC-8]: policy/tpa-rfc-8-gitlab-ci-libvirt

## Goals

In general, the idea here is to evaluate GitLab CI as a unified
platform to replace Travis, and Appveyor in the short term, but also,
in the longer term, Jenkins itself.

### Must have

 * automated configuration: setting up new builders should be done
   through Puppet
 * the above requires excellent documentation of the setup procedure
   in the development stages, so that TPA can transform that into a
   working Puppet manifest
 * Linux, Windows, Mac OS support
 * x86-64 architecture ("64-bit version of the x86 instruction set",
   AKA x64, AMD64, Intel 64, what most people use on their computers)
 * Travis replacement
 * autonomy: users should be able to setup new builds without
   intervention from the service (or system!) administrators
 * clean environments: each build should run in a clean VM

### Nice to have

 * fast: the runners should be fast (as in: powerful CPUs, good disks,
   lots of RAM to cache filesystems, CoW disks) and impose little
   overhead above running the code natively (as in: no emulation)
 * ARM64 architecture
 * Apple M-1 support
 * Jenkins replacement
 * Appveyor replacement
 * BSD support (FreeBSD, OpenBSD, and NetBSD in that order)

### Non-Goals

 * in the short term, we don't aim at doing "Continuous
   Deployment". this is one of the possible goal of the GitLab CI
   deployment, but it is considered out of scope for now. see also the
   [LDAP proposed solutions section][]

[LDAP proposed solutions section]: howto/ldap#Proposed-Solution

## Approvals required

TPA's approbation required for the libvirt exception, see
[TPA-RFC-8][].

## Proposed Solution

The [original proposal][issue 40095] from @ahf was as follows:

> \[...\] Reserve two (ideally) "fast" Debian-based machines on TPO infrastructure to build the following:
>
> * Run Gitlab CI runners via KVM (initially with focus on Windows
>   x86-64 and macOS x86-64). This will replace the need for Travis CI
>   and Appveyor. This should allow both the network team, application
>   team, and anti-censorship team to test software on these platforms
>   (either by building in the VMs or by fetching cross-compiled
>   binaries on the hosts via the Gitlab CI pipeline feature). Since
>   none(?) of our engineering staff are working full-time on MacOS
>   and Windows, we rely quite a bit on this for QA.
> * Run Gitlab CI runners via KVM for the BSD's. Same argument as
>   above, but is much less urgent.
> * Spare capacity (once we have measured it) can be used a generic
>   Gitlab CI Docker runner in addition to the FDroid builders.
> * The faster the CPU the faster the builds.
> * Lots of RAM allows us to do things such as having CoW filesystems
>   in memory for the ephemeral builders and should speed up builds
>   due to faster I/O.

All this would be implemented through a GitLab [custom executor][]
using [libvirt](https://libvirt.org/) (see [this example implementation](https://docs.gitlab.com/runner/executors/custom_examples/libvirt.html)).

This is an excerpt from the [proposal sent to TPA][TPA-RFC-8]:

> \[TPA would\] build two (bare metal) machines (in the Cymru cluster)
> to manage those runners. The machines would grant the GitLab runner
> (and also @ahf) access to the libvirt environment (through a role
> user).
>
> ahf would be responsible for creating the base image and deploying the
> first machine, documenting every step of the way in the TPA wiki. The
> second machine would be built with Puppet, using those instructions,
> so that the first machine can be rebuilt or replaced. Once the second
> machine is built, the first machine should be destroyed and rebuilt,
> unless we are absolutely confident the machines are identical.
>
> [custom executor]: https://docs.gitlab.com/runner/executors/custom.html

## Cost

The machines used were donated, but that is still an "hardware
opportunity cost" that is currently undefined.

Staff costs, naturally, should be counted. It is estimated the initial
runner setup should take less than two weeks.

## Alternatives considered

### Ganeti

Ganeti has been considered as an orchestration/deployment platform for
the runners, but there is no known integration between GitLab CI
runners and Ganeti.

If we find the time or an existing implementation, this would still be
a nice improvement.

### SSH/shell executors

This works by using an existing machine as a place to run the
jobs. Problem is it doesn't run with a clean environment, so it's not
a good fit.

### Parallels/VirtualBox

Note: couldn't figure out what the difference is between Parallels and
VirtualBox, nor if it matters.

Obviously, VirtualBox could be used to run Windows (and possibly
MacOS?) images (and maybe BSDs?) but unfortunately, Oracle has made of
mess of VirtualBox which [keeps it out of Debian](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=794466) so this could be
a problematic deployment as well.

### Docker

[Support in Debian](https://tracker.debian.org/pkg/docker.io) has improved, but is still hit-and-miss. no
support for Windows or MacOS, as far as I know, so not a complete
solution, but could be used for Linux runners.

### Docker machine

This was abandoned upstream and is considered irrelevant.

### Kubernetes

@anarcat has been thinking about setting up a Kubernetes cluster for
GitLab. There are high hopes that it will help us not only with GitLab
CI, but also the "CD" (Continuous Deployment) side of things. This
approach was briefly [discussed in the LDAP audit][LDAP proposed solutions section], but basically the
idea would be to replace the "SSH + role user" approach we currently
use for service with GitLab CI.

As explained in the [goals](#goals) section above, this is currently out of
scope, but could be considered instead of Docker for runners.

### Jenkins

See [the Jenkins replacement discussion](service/jenkins#discussion)
for more details about that alternative.
