- Tutorial
- Build a Lektor project on your machine
- Build a basic Lektor website in GitLab CI
- How-to
- Submit a website contribution
- As an occasional contributor
- As a regular contributor
- Pager playbook
- Disaster recovery
- Reference
- Installation
- SLA
- Design
- CI build/test pipelines
- CD pipelines and environments
- Staging and production
- Review apps
- Localization staging
- Issues
- Maintainer, users, and upstream
- Monitoring and testing
- Logs and metrics
- Backups
- Other documentation
- Discussion
- Overview
- Goals
- Must have
- Nice to have
- Non-Goals
- Approvals required
- Proposed Solution
- Cost
- Alternatives considered
Lektor is a static website generator written in Python, we use it to generate most of the websites of the Tor Project.
Tutorial
Build a Lektor project on your machine
See this page on the Web team wiki.
Build a basic Lektor website in GitLab CI
To enable automatic builds of a Lektor project in GitLab CI, add this snippet
in .gitlab.ci-yml
, at the root of the project:
include:
- project: tpo/tpa/ci-templates
file:
- lektor.yml
- pages-deploy.yml
The jobs defined in lektor.yml
will spawn a container to build the site, and
pages-deploy.yml
will deploy the build artifacts to GitLab Pages.
See howto/gitlab for more details on publishing to GitLab Pages.
How-to
Submit a website contribution
As an occasional contributor
The first step is to get a GitLab account.
This will allow you to fork the Lektor project in your personal GitLab namespace, where you can push commits with your changes.
As you do this, GitLab CI will continuously build a copy of the website with your changes and publish it to GitLab Pages. The location where these Pages are hosted can be displayed by navigation to the project Settings > Pages.
When you are satisfied, you can submit a Merge Request and one of the website maintainers will evaluate the proposed changes.
As a regular contributor
As someone who expects to submit contributions on a regular basis to one of the
Tor Project websites, the first step is to request access. This can be done by
joining the #tor-www
channel on IRC and asking!
The access level granted for website content contributors is normally
Developer. This role grants the ability to push new branches to the GitLab
project and submit Merge Requests to the default main
branch.
When a Merge Request is created, a CI pipeline status widget will appear under
the description, above the discussion threads. If GitLab CI succeeds building
the branch, it will publish the build artifacts and display a View app
button. Clicking the button will navigate to the build result hosted on
review.torproject.net
.
Project members with the Developer role on the TPO blog and main website have the permission to accept Merge Requests.
Once the branch is deleted, after the Merge Request is accepted, for example, the build artifacts are automatically unpublished.
Pager playbook
Disaster recovery
See #revert-a-deployment-mistake for instruction on how to roll-back an environment to its previous state after an accidental deployment.
Reference
Installation
Creating a new Lektor website is out of scope for this document.
Check out the Quickstart page in the Lektor documentation to get started.
SLA
Design
The workflows around Lektor websites is heavily dependent on GitLab CI: it handles building the sites, running tests and deploying them to various environments including staging and production.
See service/ci for general documentation about GitLab CI.
CI build/test pipelines
The lektor.yml CI template is used to configure pipelines for build and
testing Lektor website projects. Including this into the project's
gitlab-ci.yml
is usually sufficient for GitLab CI to "do the right thing".
There are several elements that can be used to customize the build process:
-
LEKTOR_BUILD_FLAGS
: this variable accepts a space separated list of flags to append to thelektor build
command. For example, setting this variable tonpm
will cause-f npm
to be appended to the build command. -
LEKTOR_PARTIAL_BUILD
: this variable can be used to alter the build process occurring on non-default branches and l10n-staging jobs. When set (to anything), it will append commands defined in.setup-lektor-partial-build
to the job'sbefore_script
. Its main purpose is to pre-process website sources to reduce the build times by trimming less-essential content which contribute a lot to build duration. See the web/tpo project CI for an example. -
TRANSLATION_BRANCH
: this variable must contain the name of the translation repository branch used to store localization files. If this variable is absent, the website will be built without l10n.
Another method of customizing the build process is by overriding keys from the
.lektor
hash (defined in the lektor.yml
template) from their own
.gitlab-ci.yml
file.
For example, this hash, added to gitlab-ci.yml
will cause the jobs defined in
the template to use a different image, and set GIT_STRATEGY
to clone
.
.lektor:
image: ubuntu:latest
variables:
GIT_STRATEGY: clone
This is in addition to the ability to override the named job parameters
directly in .gitlab-ci.yml
.
CD pipelines and environments
The Tor Project Lektor websites are deployed automatically by GitLab by a process of continuous deployment (CD).
Staging and production
Deployments to staging and production environments are handled by the static-shim-deploy.yml CI template. The service/static-shim wiki page describes the prerequisites for GitLab to be able to upload websites to the static mirror system.
A basic Lektor project that deploys to production would have a .gitlab-ci.yml
set up like this:
---
variables:
SITE_URL: example.torproject.org
include:
project: tpo/tpa/ci-templates
file:
- lektor.yml
- static-shim-deploy.yml
See the #template-variables documentation for details about the variables involved in the deployment process.
See the #working-with-a-staging-environments documentation for details about adding a staging environment to a project's deployment workflow.
Review apps
Lektor projects which include static-shim-deploy.yml
and have access to the
REVIEW_STATIC_GITLAB_SHIM_SSH_PRIVATE_KEY
CI variable (this includes all
projects in the tpo/web
namespace) have Review apps automatically enabled.
See the #working-with-review-apps documentation for details about how to use Review apps.
Localization staging
To support the work of translation contributors who work on the Tor Project
websites, we automatically build and deploy special localized versions of the
projects to reviews.torproject.net
.
The workflow can be described as follows:
-
Translations are contributed on Transifex
-
Every 30 minutes, these changes are merged to the corresponding branches in the translation repository and pushed to tpo/translation
-
A project pipeline is triggered and runs the jobs from the lektor-l10n-staging-trigger.yml CI template
-
If the changed files include any
.po
which is >15% translated, a pipeline will be triggered in the Lektor project with the specialL10N_STAGING
variable added -
In the Lektor project, the presence of the
L10N_STAGING
variable alters the regular build job: all languages >15% translated are built instead of only the officially supported languages for that project. The result is deployed toreviews.torproject.net/tpo/web/<project-name>/l10n
To enable localization staging for a Lektor project, it's sufficient to add this
snippet in .gitlab-ci.yml
in the relevant tpo/translation branch
variables:
TRANSLATION_BRANCH : $CI_COMMIT_REF_NAME
LEKTOR_PROJECT: tpo/web/<project-name>
include:
- project: tpo/tpa/ci-templates
file: lektor-l10n-staging-trigger.yml
Replace <project-name>
with the name of the Lektor GitLab project.
Issues
Lektor website projects on GitLab have individual issue trackers, so problems related to specific websites such as typos, bad links, missing content or build problems should be filed in the relevant tracker.
For problems related to deployments or CI templates specifically, File or search for issues in the ci-templates issue tracker.
Maintainer, users, and upstream
Lektor websites are maintained in collaboration by the Web team and TPA.
Monitoring and testing
Currently there is no monitoring beyond the supporting infrastructure (eg. DNS, host servers, httpd, etc.).
Logs and metrics
Backups
There are no backups specific to Lektor.
Source code of our Lektor projects is backed up along with GitLab itself, and the production build artifacts themselves are picked up with those of the hosts comprising the static mirror system.