expand ci docs

service/ci.md

@@ -34,6 +34,12 @@ the shared runners in a project?
 
 <!-- more in-depth procedure that may require interpretation -->
 
+## Image security
+
+TODO: document how to create and use more secure Docker images. For
+example, most images run as root: try to make images run as a regular
+user.
+
 ## Pager playbook
 
 <!-- information about common errors from the monitoring system and -->
@@ -111,10 +117,17 @@ the following operations need to be performed:
 
 NOTE: we used the Debian packages ([docker.io](https://tracker.debian.org/pkg/docker.io) and
 [gitlab-runner](https://tracker.debian.org/gitlab-runner)) instead of the upstream official packages, because
-those have a somewhat messed up installer and weird key deployment
-policies. They are both somewhat out of date, and the latter is not
-available in Debian buster (current stable), so it had to be installed
-from bullseye.
+those have a somewhat [messed up installer](https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh) and [weird key
+deployment policies](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3897). In other words, we would rather avoid having
+to trust the upstream packages for runners, even though we use them
+for the GitLab omnibus install. The Debian packages are both somewhat
+out of date, and the latter is not available in Debian buster (current
+stable), so it had to be installed from bullseye. 
+
+We also avoided using the [puppetlabs/docker](https://forge.puppet.com/modules/puppetlabs/docker) module because we
+"only" need to setup Docker, and not specifically deal with
+containers, volumes and so on right now. All that is (currently)
+handled by GitLab runner.
 
 ### F-Droid runners
 
@@ -153,6 +166,20 @@ TODO: expand on GitLab CI's design and architecture, following [this
 checklist](https://bluesock.org/~willkg/blog/dev/auditing_projects.html). See also the [Jenkins section](#jenkins) below for the same
 thing about Jenkins.
 
+Some things to look into:
+
+ * https://docs.gitlab.com/ee/user/project/new_ci_build_permissions_model.html
+ * https://docs.gitlab.com/runner/security/
+
+TODO: consider podman for running containers more securely, and
+possibly also to build container images inside GitLab CI, which would
+otherwise require docker-in-docker (DinD), unsupported by
+upstream. some ideas here:
+
+ * https://medium.com/prgcont/using-buildah-in-gitlab-ci-9b529af19e42
+ * https://github.com/containers/podman/issues/7982
+ * https://github.com/jonasbb/podman-gitlab-runner
+
 ## Issues
 
 [File][] or [search][] for issues in the [GitLab issue tracker][search].
@@ -189,7 +216,9 @@ not. They are, naturally, monitored through the `node-exporter` like
 all other TPO servers, however.
 
 TODO: monitor GitLab runners; they can be configured to expose metrics
-through a Prometheus exporter, we could hook this in our setup.
+through a Prometheus exporter. The Puppet module supports this through
+the `gitlab_ci_runner::metrics_server` variable, but we would need to
+hook it into our server as well. See also [the upstream documentation](https://docs.gitlab.com/runner/monitoring/README.html).
 
 ## Backups