lektor.md

Lektor is a static website generator written in Python, we use it to generate most of the websites of the Tor Project.

• Tutorial
  • Build a Lektor project on your machine
  • Build a basic Lektor website in GitLab CI
• How-to
  • Submit a website contribution
    • As an occasional contributor
    • As a regular contributor
  • Pager playbook
  • Disaster recovery
• Reference
  • Installation
  • SLA
  • Design
    • CI build/test pipelines
    • CD pipelines and environments
      • Staging and production
    • Review apps
    • Localization staging
  • Issues
  • Maintainer, users, and upstream
  • Monitoring and testing
  • Logs and metrics
  • Backups
  • Other documentation
• Discussion
  • Overview
  • Goals
    • Must have
    • Nice to have
    • Non-Goals
  • Approvals required
  • Proposed Solution
  • Cost
  • Alternatives considered

Tutorial

Build a Lektor project on your machine

See this page on the Web team wiki.

Build a basic Lektor website in GitLab CI

To enable automatic builds of a Lektor project in GitLab CI, add this snippet in .gitlab.ci-yml, at the root of the project:

include:
  - project: tpo/tpa/ci-templates
	file:
	  - lektor.yml
	  - pages-deploy.yml

The jobs defined in lektor.yml will spawn a container to build the site, and pages-deploy.yml will deploy the build artifacts to GitLab Pages.

See howto/gitlab for more details on publishing to GitLab Pages.

How-to

Submit a website contribution

As an occasional contributor

The first step is to get a GitLab account.

This will allow you to fork the Lektor project in your personal GitLab namespace, where you can push commits with your changes.

As you do this, GitLab CI will continuously build a copy of the website with your changes and publish it to GitLab Pages. The location where these Pages are hosted can be displayed by navigation to the project Settings > Pages.

When you are satisfied, you can submit a Merge Request and one of the website maintainers will evaluate the proposed changes.

As a regular contributor

As someone who expects to submit contributions on a regular basis to one of the Tor Project websites, the first step is to request access. This can be done by joining the #tor-www channel on IRC and asking!

The access level granted for website content contributors is normally Developer. This role grants the ability to push new branches to the GitLab project and submit Merge Requests to the default main branch.

When a Merge Request is created, a CI pipeline status widget will appear under the description, above the discussion threads. If GitLab CI succeeds building the branch, it will publish the build artifacts and display a View app button. Clicking the button will navigate to the build result hosted on review.torproject.net.

Project members with the Developer role on the TPO blog and main website have the permission to accept Merge Requests.

Once the branch is deleted, after the Merge Request is accepted, for example, the build artifacts are automatically unpublished.

Pager playbook

Disaster recovery

See #revert-a-deployment-mistake for instruction on how to roll-back an environment to its previous state after an accidental deployment.

Reference

Installation

Creating a new Lektor website is out of scope for this document.

Check out the Quickstart page in the Lektor documentation to get started.

SLA

Design

The workflows around Lektor websites is heavily dependent on GitLab CI: it handles building the sites, running tests and deploying them to various environments including staging and production.

See service/ci for general documentation about GitLab CI.

CI build/test pipelines

The lektor.yml CI template is used to configure pipelines for build and testing Lektor website projects. Including this into the project's gitlab-ci.yml is usually sufficient for GitLab CI to "do the right thing".

There are several elements that can be used to customize the build process:

• LEKTOR_BUILD_FLAGS: this variable accepts a space separated list of flags to append to the lektor build command. For example, setting this variable to npm will cause -f npm to be appended to the build command.

• LEKTOR_PARTIAL_BUILD: this variable can be used to alter the build process occurring on non-default branches and l10n-staging jobs. When set (to anything), it will append commands defined in .setup-lektor-partial-build to the job's before_script. Its main purpose is to pre-process website sources to reduce the build times by trimming less-essential content which contribute a lot to build duration. See the web/tpo project CI for an example.

• TRANSLATION_BRANCH: this variable must contain the name of the translation repository branch used to store localization files. If this variable is absent, the website will be built without l10n.

Another method of customizing the build process is by overriding keys from the .lektor hash (defined in the lektor.yml template) from their own .gitlab-ci.yml file.

For example, this hash, added to gitlab-ci.yml will cause the jobs defined in the template to use a different image, and set GIT_STRATEGY to clone.

.lektor:
  image: ubuntu:latest
  variables:
    GIT_STRATEGY: clone

This is in addition to the ability to override the named job parameters directly in .gitlab-ci.yml.

CD pipelines and environments

The Tor Project Lektor websites are deployed automatically by GitLab by a process of continuous deployment (CD).

Staging and production

Deployments to staging and production environments are handled by the static-shim-deploy.yml CI template. The service/static-shim wiki page describes the prerequisites for GitLab to be able to upload websites to the static mirror system.

A basic Lektor project that deploys to production would have a .gitlab-ci.yml set up like this:

---
variables:
  SITE_URL: example.torproject.org

include:
  project: tpo/tpa/ci-templates
  file:
    - lektor.yml
    - static-shim-deploy.yml

See the #template-variables documentation for details about the variables involved in the deployment process.

See the #working-with-a-staging-environments documentation for details about adding a staging environment to a project's deployment workflow.

Review apps

Lektor projects which include static-shim-deploy.yml and have access to the REVIEW_STATIC_GITLAB_SHIM_SSH_PRIVATE_KEY CI variable (this includes all projects in the tpo/web namespace) have Review apps automatically enabled.

See the #working-with-review-apps documentation for details about how to use Review apps.

Localization staging

To support the work of translation contributors who work on the Tor Project websites, we automatically build and deploy special localized versions of the projects to reviews.torproject.net.

The workflow can be described as follows:

1. Translations are contributed on Transifex

2. Every 30 minutes, these changes are merged to the corresponding branches in the translation repository and pushed to tpo/translation

3. A project pipeline is triggered and runs the jobs from the lektor-l10n-staging-trigger.yml CI template

4. If the changed files include any .po which is >15% translated, a pipeline will be triggered in the Lektor project with the special L10N_STAGING variable added

5. In the Lektor project, the presence of the L10N_STAGING variable alters the regular build job: all languages >15% translated are built instead of only the officially supported languages for that project. The result is deployed to reviews.torproject.net/tpo/web/<project-name>/l10n

To enable localization staging for a Lektor project, it's sufficient to add this snippet in .gitlab-ci.yml in the relevant tpo/translation branch

variables:
  TRANSLATION_BRANCH : $CI_COMMIT_REF_NAME
  LEKTOR_PROJECT: tpo/web/<project-name>

include:
  - project: tpo/tpa/ci-templates
    file: lektor-l10n-staging-trigger.yml

Replace <project-name> with the name of the Lektor GitLab project.

Issues

Lektor website projects on GitLab have individual issue trackers, so problems related to specific websites such as typos, bad links, missing content or build problems should be filed in the relevant tracker.

For problems related to deployments or CI templates specifically, File or search for issues in the ci-templates issue tracker.

Maintainer, users, and upstream

Lektor websites are maintained in collaboration by the Web team and TPA.

Monitoring and testing

Currently there is no monitoring beyond the supporting infrastructure (eg. DNS, host servers, httpd, etc.).

Logs and metrics

Backups

There are no backups specific to Lektor.

Source code of our Lektor projects is backed up along with GitLab itself, and the production build artifacts themselves are picked up with those of the hosts comprising the static mirror system.

Other documentation

• Lektor documentation

Discussion

Overview

Goals

Must have

Nice to have

Non-Goals

Approvals required

Proposed Solution

Cost

Alternatives considered