@@ -20,9 +20,24 @@ Donate-review is a gitlab runner and review-app deployment target for [donate-ne
## Pager playbook
<!-- information about common errors from the monitoring system and -->
<!-- how to deal with them. this should be easy to follow: think of -->
<!-- your future self, in a stressful situation, tired and hungry. -->
### deploy-review job fails with "The user who started this CI job does not have permission to deploy"
The donate-review runner is a shell runner, it doesn't use any kind of sandboxing or containerization. To try to mitigate potential security issues with created by letting any contributor run a shell script on the donate-review machine, a very simple "access control" was implemented in the gitlab CI script.
echo The user who started this CI job does not have permission to deploy
echo"Required level: 50. Actual level: $access_level"
exit 1
fi
```
This script checks if the user is an owner of the repository (access level 50). Anyone who is *not an owner* won't be allowed to run the deploy job. If a user's jobs are failing with the above error message, make sure they're an owner, not a maintainer, developer, reporter, or guest.
### Static assets don't load on deployed review apps
An issue we've seen several times is that static assets sometimes don't load. The issue was tracked back to donate-neo handling its own static assets. To fix this, the gitlab CI script was changed to run `manage.py collectstatic` and configure donate-review apache vhosts to serve the static files directly. This issue shouldn't rear its head again, but in the event that it does, start by ensuring that django's collectstatic is being run, that the static files are deployed to the correct location, and that the review app vhost is configured to serve the static files from the correct directory, with `Require all granted`.
## Disaster recovery
...
...
@@ -107,7 +122,7 @@ There is no issue tracker specifically for this the donate-review runner.
## Maintainer
[kez](https://gitlab.torproject.org/kez)
[TPA](https://gitlab.torproject.org/tpo/tpa/team)
## Users
...
...
@@ -124,11 +139,7 @@ ephemeral, and a new system can be bootstrapped solely from puppet.
## Overview
<!-- describe the overall project. should include a link to a ticket -->
<!-- that has a launch checklist -->
<!-- if this is an old project being documented, summarize the known -->
<!-- issues with the project. -->
donate-review was created as part of tpo/web/donate-neo#6 and tpo/tpa/team#41108. Donate-review's purpose is to provide a review app deploy target for tpo/web/donate-neo. Most of the other tpo/web sites are static lektor sites, and can be easily deployed to a review app target as simple static sites fronted by Apache. But because donate-neo is a django application, it needs a specially-created deploy target for review apps. donate-review clones the donate-neo codebase, checks out the right commit, sets up the applications virtuel environment, dependencies, database and static files, and then deploys an Apache vhost for the new review app.
## Security and risk assessment
...
...
@@ -155,18 +166,21 @@ donate-neo instances (files, databases, and gunicorn instances) are not cleaned
up automatically. Cleanup only happens when someone runs the `stop-review` job.
If this job isn't run, These instances will not be cleaned up. Currently, there
is no solution for this. There's an [open Gitlab ticket][] to implement running
an `after_script` when a job is cancelled, but no news on when it might be
implemented. A potential solution is to write a cron job that just cleans up
an `after_script` when a job is cancelled, and as of January 2024 is "targeting near term resolution" for the ticket. A potential solution is to write a cron job that just cleans up
### stop-review sometimes doesn't remove old deployments
The stop-review job is supposed to read the `gunicorn.pid` file, kill the gunicorn process, tear down the apache config, and remove the deployed code. Sometimes gunicorn dies early though, and the stop-review job fails early, leaving old configs and code lying around.
### Outdated databases
Donate Review has a known issue with updating existing MRs. The updated code
will get pulled to the rview app target, but necessary setup won't be re-run.
will get pulled to the review app target, but necessary setup won't be re-run.
This can lead to situations where a new Django app is installed, but the app's
migrations won't be run, leading to an outdated database. This bug was