Document the donate-review service

service/donate-review.md

+Donate-review is a gitlab runner and review-app deployment target for [donate-neo](gitlab.torproject.org/tpo/web/donate-neo).
+
+[[_TOC_]]
+
+# Tutorial
+
+## Starting a review app
+
+1. Push some changes to a branch (other than main) on tpo/web/donate-neo
+1. Create an MR
+1. Wait for the `lint`, `mypy`, and `test` jobs to finish
+1. Run the `deploy-review` job, and wait for it to finish
+1. Click the "View app" button
+1. When done reviewing, click "Stop environment" or run the `stop-review` job
+1. Merge changes
+
+# How-to
+
+<!-- more in-depth procedure that may require interpretation -->
+
+## Pager playbook
+
+<!-- information about common errors from the monitoring system and -->
+<!-- how to deal with them. this should be easy to follow: think of -->
+<!-- your future self, in a stressful situation, tired and hungry. -->
+
+## Disaster recovery
+
+In the event that the box is compromised, it should be rebuilt from scratch.
+See [Installation](#installation) below.
+
+# Reference
+
+## Installation
+
+1. bootstrap a new vm, and set up puppet
+1. install gitlab runner on the machine and configure a [shell executor][]
+1. add the `role: donate_review` parameter to the new machine in hiera
+1. run puppet
+1. add the runner to a gitlab project
+
+[shell executor][]: <https://docs.gitlab.com/runner/executors/shell.html>
+
+## Upgrades
+
+Upgrades are performed automatically through Debian packages. Gitlab runner is
+excluded from unattended-upgrades and must be upgraded manually.
+
+Packages used by the review-app instances (gunicorn, django, etc.) are
+installed into a virtualenv as part of the CI process. These packages will be
+updated as part of donate-neo maintenance.
+
+## SLA
+
+<!-- this describes an acceptable level of service for this service -->
+
+## Design and architecture
+
+There are two major pieces to donate-review. The donate-neo [gitlab-ci file][],
+and a `setup-apache-config.sh` script that lives in puppet. Here's what happens
+when the deploy-review job runs:
+
+1. check the access-level of the user who ran the job
+1. install dependencies, gunicorn, create and run migrations
+1. start gunicorn on port 0 (kernel chooses), and write the gunicorn pid to a file
+1. use the gunicorn pid to find what port the kernel chose
+1. run the setup-apache-config script as root (with a `NOPASSWD` sudoers rule)
+1. the script writes an apache site config file for this review-app instance
+1. reload apache
+
+[gitlab-ci file]: <https://gitlab.torproject.org/tpo/web/donate-neo/-/blob/main/.gitlab-ci.yml>
+
+### Subdomains and TLS Certs
+
+the apache configuration uses subdomain-based vhosts. the review-apps use
+domains like `<ci commit ref slug>.donate-review-01.torproject.org`.
+Letsencrypt won't give us a wildcard cert for
+`*.donate-review-01.torproject.org` because it's too many nested subdomains.
+Instead, we use apache's [mod\_md][] to dynamically request Letsencrypt
+certificates as-needed for new review-apps
+
+[mod\_md]: <https://httpd.apache.org/docs/trunk/mod/mod_md.html>
+
+## Services
+
+- gitlab-runner
+- apache
+- gunicorn (one instance per review app)
+
+## Storage
+
+Each donate-neo instance has its own sqlite database. The database resides
+on-disk (as opposed to in-memory).
+
+## Authentication
+
+The donate-review *runner* uses token authentication to pick up jobs fom
+Gitlab. Each donate-neo instance handles its own authentication.
+
+## Issues
+
+There is no issue tracker specifically for this the donate-review runner.
+[File][] or [search][] for issues in the donate-neo repository.
+
+ [File]: https://gitlab.torproject.org/tpo/web/donate-neo/-/issues/new
+ [search]: https://gitlab.torproject.org/tpo/web/donate-neo/-/issues
+
+## Maintainer
+
+[kez](https://gitlab.torproject.org/kez)
+
+## Users
+
+Anyone contributing to tpo/web/donate-neo
+
+## Logs
+
+The only relevant logs are from apache, logrotate rotates them every 14 days.
+
+## Backups
+
+donate-review has no special backup needs. All the donate-neo instances are
+ephemeral, and a new system can be bootstrapped solely from puppet.
+
+## Overview
+
+<!-- describe the overall project. should include a link to a ticket -->
+<!-- that has a launch checklist -->
+
+<!-- if this is an old project being documented, summarize the known -->
+<!-- issues with the project. -->
+
+## Security and risk assessment
+
+donate-review is a shell executor, which means each CI job is executed with no
+real sandboxing or containerization. There was an attempt to set up the runner
+using systemd-nspawn, but it was taking too long and we
+[eventually decided against it][].
+
+The current security measures include the basic things we get for free from
+debian (file permissions, apparmor), as well as some access control in the
+gitlab-ci file. The deploy-review job has to be run manually, and the job's
+script starts by [checking if the user who ran the job is a repo maintainer][].
+
+A proper security and risk assessment should be prioritized to find and
+mitigate potential security problems.
+
+[eventually decided against it]: <https://gitlab.torproject.org/tpo/tpa/team/-/issues/41108#note_2913481>
+
+[checking if the user who ran the job is a repo maintainer]: <https://gitlab.torproject.org/tpo/web/donate-neo/-/blob/2344c59a/.gitlab-ci.yml#L46>
+
+## Technical debt and next steps
+
+donate-neo instances (files, databases, and gunicorn instances) are not cleaned
+up automatically. Cleanup only happens when someone runs the `stop-review` job.
+If this job isn't run, These instances will not be cleaned up. Currently, there
+is no solution for this. There's an [open Gitlab ticket][] to implement running
+an `after_script` when a job is cancelled, but no news on when it might be
+implemented. A potential solution is to write a cron job that just cleans up
+old gunicorn processes, and removes old builds.
+
+[open Gitlab ticket]: <https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4843>