Changes

Page history
complete the static-shim documentation (#40364) authored by anarcat's avatar anarcat
Show whitespace changes
Inline Side-by-side
service/static-shim.md
View page @ 8349c724
...@@ -5,31 +5,22 @@ hosted in the static mirror infrastructure. ...@@ -5,31 +5,22 @@ hosted in the static mirror infrastructure.
# Tutorial # Tutorial
<!-- simple, brainless step-by-step instructions requiring little or -->
<!-- no technical background -->
TODO: "how do users add/remove sites"
# How-to
<!-- more in-depth procedure that may require interpretation -->
TODO: review ticket for possible howtos
## Deploying a static site from GitLab CI ## Deploying a static site from GitLab CI
First, you will need to make sure the site builds in GitLab CI. A First, you will need to make sure the site builds in [GitLab CI][]. A
`build` stage MUST be used that will produce artifacts that can be `build` stage MUST be used that will produce artifacts that can be
used by the `deploy` job provided in the [`static-shim-deploy.yml` used by the `deploy` job provided in the [`static-shim-deploy.yml`
template][]. How to build the website will vary according to the site, template][]. How to build the website will vary according to the site,
obviously. See the [hugo build instructions below](#building-a-hugo-site). obviously. See the [Hugo build instructions below](#building-a-hugo-site) for that
specific generator.
[`static-shim-deploy.yml` template]: https://gitlab.torproject.org/tpo/tpa/ci-templates/-/blob/main/static-shim-deploy.yml [`static-shim-deploy.yml` template]: https://gitlab.torproject.org/tpo/tpa/ci-templates/-/blob/main/static-shim-deploy.yml
TODO: link to documentation on how to build Lektor sites. TODO: link to documentation on how to build Lektor sites in GitLab CI.
It is a good idea to also add a `pages` stage to preview the It is a good idea to also add a `pages` stage to preview the
build. The above template has an example `pages` stage. build. The above template has an example `pages` stage, see also the
[publishing GitLab pages](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/gitlab/#publishing-gitlab-pages) section our [GitLab documentation](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/gitlab/).
Then include the deploy job template in the `.gitlab-ci.yml` with a Then include the deploy job template in the `.gitlab-ci.yml` with a
snippet like this: snippet like this:
...@@ -79,14 +70,16 @@ variable`, with the following parameters: ...@@ -79,14 +70,16 @@ variable`, with the following parameters:
Then the *public* part of that key needs to be added in Puppet. This Then the *public* part of that key needs to be added in Puppet. This
can only be done by TPA, so file a ticket there if you need can only be done by TPA, so file a ticket there if you need
assistance. For TPA, see below for the remaining instructions. assistance. For TPA, [see below](#adding-a-new-static-site-shim-in-puppet) for the remaining instructions.
You can commit the above changes to the `.gitlab-ci.yml` file, but You can commit the above changes to the `.gitlab-ci.yml` file, but
when pushed, the pipeline's `deploy` stage is normal, TPA needs to do when pushed, the pipeline's `deploy` stage is normal, TPA needs to do
its magic for the deploy to work. Make sure the build works in GitLab its magic for the deploy to work. Make sure the build works in GitLab
pages before requesting the deploy in the static mirror system. pages before requesting the deploy in the static mirror system.
### TPA: adding a new static site shim in Puppet # How-to
## Adding a new static site shim in Puppet
The public key mentioned above should be added in the `tor-puppet.git` repository, in the The public key mentioned above should be added in the `tor-puppet.git` repository, in the
`hiera/common.yaml` file, in the `staticsync::gitlab_shim::ssh::sites` `hiera/common.yaml` file, in the `staticsync::gitlab_shim::ssh::sites`
...@@ -251,19 +244,40 @@ template][]. ...@@ -251,19 +244,40 @@ template][].
## Pager playbook ## Pager playbook
<!-- information about common errors from the monitoring system and --> A typical failure will be that users complains that their
<!-- how to deal with them. this should be easy to follow: think of --> `deploy_static` job fails. We have yet to see such a failure occur,
<!-- your future self, in a stressful situation, tired and hungry. --> but if if does, users should provide a link to the Job log, which
should provide more information.
TODO: pager?
## Disaster recovery ## Disaster recovery
<!-- what to do if all goes to hell. e.g. restore from backups? --> The service is "cattle" in that it can easily be rebuilt from scratch
<!-- rebuild from scratch? not necessarily those procedures (e.g. see --> if the server is completely lost. Naturally it strongly depends on
<!-- "Installation" below but some pointers. --> GitLab for operation. If GitLab would fail, it should still be
possible to deploy sites to the static mirror system by deploying them
by hand to the static shim and calling `static-update-component`
there. It would be preferable to build the site outside of the
static-shim server to avoid adding any extra packages we do not need
there.
The status site is particularly vulnerable to disasters here, see the
[status-site disaster recovery documentation](service/status#disaster-recovery) for pointers on where
to go in case things really go south.
Another possible disaster that could happen is a complete GitLab
compromise or hostile GitLab admin. Such an attacker could deploy any
site they wanted and therefore deface or sabotage critical websites,
introducing hostile code in thousands of users. If such an event would
occur:
1. **remove all SSH keys from the Puppet configuration**,
specifically in the `staticsync::gitlab_shim::ssh::sites`
variable, defined in `hiera/common.yaml`.
TODO: DR 2. restore sites from a known backup. the [backup service](howto/backup) should
have a copy of the static-shim content
3. redeploy the sites manually (`static-update-component $URL`)
# Reference # Reference
...@@ -283,30 +297,35 @@ during downtimes, updates to websites are not possible. ...@@ -283,30 +297,35 @@ during downtimes, updates to websites are not possible.
## Design ## Design
<!-- how this is built --> The static shim was built to allow [GitLab CI][] to deploy content to the
<!-- should reuse and expand on the "proposed solution", it's a --> [static mirror system][].
<!-- "as-built" documented, whereas the "Proposed solution" is an -->
<!-- "architectural" document, which the final result might differ --> [GitLab CI]: service/ci
<!-- from, sometimes significantly -->
They way it works is that GitLab CI jobs (defined in the
`.gitlab-ci.yml` file) build the site and then push it to a static
source (currently `static-gitlab-shim.torproject.org`) with rsync over
SSH. Then the CI job also calls the `static-update-component` script
for the master to pull the content just like any other static
component.
<!-- a good guide to "audit" an existing project's design: --> ![SSH deploy design of the static-shim](static-shim/architecture-static-shim-ssh.png)
<!-- https://bluesock.org/~willkg/blog/dev/auditing_projects.html -->
<!-- things to evaluate here: A [previous design](#webhook-deployment) involved a webhook written in Python, but now most
of the business logic resides in a [`static-shim-deploy.yml` template]
template which is basically a shell script embedded in a YAML
file. The CI hooks are deployed by users, which will typically include
the above template in their own `.gitlab-ci.yml` file.
* services [static mirror system]: howto/static-component
* storage (databases? plain text files? cloud/S3 storage?)
* queues (e.g. email queues, job queues, schedulers)
* interfaces (e.g. webserver, commandline)
* authentication (e.g. SSH, LDAP?)
* programming languages, frameworks, versions
* dependent services (e.g. authenticates against LDAP, or requires
git pushes)
* deployments: how is code for this deployed (see also Installation)
how is this thing built, basically? --> ### Storage
TODO: design still in flux, see "alternatives considered" below. Files are generated in GitLab CI as artifacts and stored there, which
makes it possible for them to be deployed by hand as well. A copy is
also kept on the static-shim server to make future deployments
faster. We use `rsync --checksum` to avoid updating the timestamps
even if the source file were just regenerated from scratch.
### Authentication ### Authentication
...@@ -351,12 +370,6 @@ There is no issue tracker specifically for this project, [File][] or ...@@ -351,12 +370,6 @@ There is no issue tracker specifically for this project, [File][] or
This service was designed in [ticket 40364](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40364). This service was designed in [ticket 40364](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40364).
* the webhook implementation fails if sites take more than 10 seconds
to deploy.
* the webhook implementation doesn't provide much visibility on
failures or progress, to see the list of recent webhook calls, head
to Settings -> Webhooks -> Edit -> Recent deliveries
## Maintainer, users, and upstream ## Maintainer, users, and upstream
The shim was written by anarcat and is maintained by TPA. It is used The shim was written by anarcat and is maintained by TPA. It is used
...@@ -364,36 +377,43 @@ by all "critical" websites managed in GitLab. ...@@ -364,36 +377,43 @@ by all "critical" websites managed in GitLab.
## Monitoring and testing ## Monitoring and testing
<!-- describe how this service is monitored and how it can be tested --> There is not specific monitoring for this service, other than the
<!-- after major changes like IP address changes or upgrades. describe --> usual server-level monitoring. If the service should fail, users will
<!-- CI, test suites, linting, how security issues and upgrades are --> notice because their pipelines start failing.
<!-- tracked -->
TODO: write unit tests Good sites to test that the deployment works are
TODO: how is this monitored? <https://research.torproject.org/> ([pipeline link](https://gitlab.torproject.org/tpo/web/research/-/pipelines), not critical)
or <https://status.torproject.org/> ([pipeline link](https://gitlab.torproject.org/tpo/tpa/status-site/-/pipelines),
semi-critical).
## Logs and metrics ## Logs and metrics
<!-- where are the logs? how long are they kept? any PII? --> Jobs in GitLab CI have their own logs and retention policies. The
<!-- what about performance metrics? same questions --> static shim should not add anything special to this, in theory. In
practice it's possible some private key leakage occurs if a user would
display the content of their own private SSH key in the job log. If
they use the provided template, this should not occur.
The webhook logs are available through `journalctl -u webhook` and in We do not maintain any metrics on this service, other than the
`/var/log/daemon.log`. They should not contain PII that is not already usual server-level metrics.
present in GitLab itself. Specifically, they might contain webhook
payloads, artifacts URL and webpages contents.
TODO: metrics?
## Backups ## Backups
No specific backup procedure is necessary for this server, outside of No specific backup procedure is necessary for this server, outside of
the automated basics. In fact, data on this host is mostly ephemeral the automated basics. In fact, data on this host is mostly ephemeral
and could be reconstructed from pipelines in case of a total disaster. and could be reconstructed from pipelines in case of a total server
loss.
## Other documentation As mentioned in the [disaster recovery section](#disaster-recovery), if the GitLab
server gets compromised, the backup should still contain previous
good copies of the websites, in any case.
<!-- references to upstream documentation, if relevant --> ## Other documentation
* GitLab's [CI deployment mechanism](https://about.gitlab.com/blog/2021/02/05/ci-deployment-and-environments/) blog post
* [Design and launch ticket](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40364)
* our [static mirror system][] documentation
* our [GitLab CI documentation][GitLab CI]
* [Webhook homepage](https://github.com/adnanh/webhook) * [Webhook homepage](https://github.com/adnanh/webhook)
* [hook definition documentation](https://github.com/adnanh/webhook/blob/master/docs/Hook-Definition.md) * [hook definition documentation](https://github.com/adnanh/webhook/blob/master/docs/Hook-Definition.md)
* [hook examples](https://github.com/adnanh/webhook/blob/master/docs/Hook-Examples.md) * [hook examples](https://github.com/adnanh/webhook/blob/master/docs/Hook-Examples.md)
...@@ -401,51 +421,51 @@ and could be reconstructed from pipelines in case of a total disaster. ...@@ -401,51 +421,51 @@ and could be reconstructed from pipelines in case of a total disaster.
* [how to refer to payload in hook configuration](https://github.com/adnanh/webhook/blob/master/docs/Referencing-Request-Values.md) * [how to refer to payload in hook configuration](https://github.com/adnanh/webhook/blob/master/docs/Referencing-Request-Values.md)
* [usage](https://github.com/adnanh/webhook/blob/master/docs/Webhook-Parameters.md) * [usage](https://github.com/adnanh/webhook/blob/master/docs/Webhook-Parameters.md)
* [GitLab webhook documentation](https://docs.gitlab.com/ee/user/project/integrations/webhooks.html) * [GitLab webhook documentation](https://docs.gitlab.com/ee/user/project/integrations/webhooks.html)
* [Design and launch ticket](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40364)
# Discussion # Discussion
## Overview ## Overview
<!-- describe the overall project. should include a link to a ticket --> The static shim was built to unblock the [Jenkins retirement
<!-- that has a launch checklist --> project](https://gitlab.torproject.org/groups/tpo/-/milestones/27). A key blocker was that the [static mirror system][] was
strongly coupled with Jenkins: many high traffic and critical websites
<!-- if this is an old project being documented, summarize the known --> are built and deployed by Jenkins. Unless we wanted to completely
<!-- issues with the project. to quote the "audit procedure": retire the static mirror system (in favor, say, of GitLab Pages), we
had to create a way for GitLab CI to deploy content to the static
5. When was the last security review done on the project? What was mirror system.
the outcome? Are there any security issues currently? Should it
have another security review?
6. When was the last risk assessment done? Something that would cover
risks from the data stored, the access required, etc.
7. Are there any in-progress projects? Technical debt cleanup?
Migrations? What state are they in? What's the urgency? What's the
next steps?
8. What urgent things need to be done on this project?
-->
## Goals ## Goals
<!-- include bugs to be fixed -->
### Must have ### Must have
* deploy sites from GitLab CI to the static mirror system
* site A cannot deploy to site B without being explicitly granted
permissions
* server-side (ie. in Puppet) access control (ie. user X can only
deploy site B)
### Nice to have ### Nice to have
* automate migration from Jenkins to avoid manually doing many sites
* reusable GitLab CI templates
### Non-Goals ### Non-Goals
* static mirror system replacement
## Approvals required ## Approvals required
<!-- for example, legal, "vegas", accounting, current maintainer --> TPA
## Proposed Solution ## Proposed Solution
We have decided to deploy sites over SSH from GitLab CI, see below for
a discussion.
## Cost ## Cost
One VM, 20-30 hours of work, see [tpo/tpa/team#40364](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40364) for time tracking.
## Alternatives considered ## Alternatives considered
This shim was designed to replace Jenkins with GitLab CI. As suche, This shim was designed to replace Jenkins with GitLab CI. As suche,
...@@ -478,6 +498,18 @@ webhooks, but originally decided against it for the following reasons: ...@@ -478,6 +498,18 @@ webhooks, but originally decided against it for the following reasons:
exception and is more error prone (e.g. if we somehow forget the exception and is more error prone (e.g. if we somehow forget the
`command=` override, we open full shell access) `command=` override, we open full shell access)
After trying the webhook deployment mechanism (below), we decided to
go back to the deployment mechanism instead, because:
* the webhook implementation fails if sites take more than 10 seconds
to deploy.
* the webhook implementation doesn't provide much visibility on
failures or progress, to see the list of recent webhook calls, head
to Settings -> Webhooks -> Edit -> Recent deliveries
See below for details on that, and above for the full design of the
current deployment.
### webhook deployment ### webhook deployment
A designed based on GitLab webhooks was established, with a workflow A designed based on GitLab webhooks was established, with a workflow
...@@ -489,8 +521,7 @@ that goes something like this: ...@@ -489,8 +521,7 @@ that goes something like this:
artifacts back to GitLab artifacts back to GitLab
4. GitLab fires a [webhook](https://gitlab.torproject.org/help/user/project/integrations/webhooks#pipeline-events), typically on [pipeline events](https://docs.gitlab.com/ee/user/project/integrations/webhooks.html#pipeline-events) 4. GitLab fires a [webhook](https://gitlab.torproject.org/help/user/project/integrations/webhooks#pipeline-events), typically on [pipeline events](https://docs.gitlab.com/ee/user/project/integrations/webhooks.html#pipeline-events)
5. webhook receives the ping and authenticates against a 5. webhook receives the ping and authenticates against a
configuration, mapping to a given `static-component` (TODO: allow configuration, mapping to a given `static-component`
list for gitlab?)
6. after authentication, the webhook fires a script 6. after authentication, the webhook fires a script
(`static-gitlab-shim-pull`) (`static-gitlab-shim-pull`)
7. `static-gitlab-shim-pull` parses the payload from the webhook and 7. `static-gitlab-shim-pull` parses the payload from the webhook and
... ...
......