lektor.md

[Lektor][] is a static website generator written in Python, we use it to
generate most of the websites of the Tor Project.

[Lektor]: https://github.com/lektor/lektor

[[_TOC_]]

# Tutorial

## Build a Lektor project on your machine

See [this page](https://gitlab.torproject.org/tpo/web/wiki/-/wikis/Compiling-a-local-version-of-the-website)
on the Web team wiki.

## Build a basic Lektor website in GitLab CI

To enable automatic builds of a Lektor project in GitLab CI, add this snippet
in `.gitlab.ci-yml`, at the root of the project:

	include:
	  - project: tpo/tpa/ci-templates
		file:
		  - lektor.yml
		  - pages-deploy.yml

The jobs defined in `lektor.yml` will spawn a container to build the site, and
`pages-deploy.yml` will deploy the build artifacts to GitLab Pages.

See [howto/gitlab](howto/gitlab#publishing-gitlab-pages) for more details on
publishing to GitLab Pages.

# How-to

## Submit a website contribution

### As an occasional contributor

The first step is to [get a GitLab account](howto/gitlab#how-to-get-an-account).

This will allow you to fork the Lektor project in your personal GitLab
namespace, where you can push commits with your changes.

As you do this, GitLab CI will continuously build a copy of the website with
your changes and publish it to GitLab Pages. The location where these Pages are
hosted can be displayed by navigation to the project Settings > Pages.

When you are satisfied, you can submit a Merge Request and one of the website
maintainers will evaluate the proposed changes.

### As a regular contributor

As someone who expects to submit contributions on a regular basis to one of the
Tor Project websites, the first step is to request access. This can be done by
joining the `#tor-www` channel on IRC and asking!

The access level granted for website content contributors is normally
*Developer*. This role grants the ability to push new branches to the GitLab
project and submit Merge Requests to the default `main` branch.

When a Merge Request is created, a CI pipeline status widget will appear under
the description, above the discussion threads. If GitLab CI succeeds building
the branch, it will publish the build artifacts and display a **View app**
button. Clicking the button will navigate to the build result hosted on
`review.torproject.net`.

Project members with the *Developer* role on the TPO blog and main website
have the permission to accept Merge Requests.

Once the branch is deleted, after the Merge Request is accepted, for example,
the build artifacts are automatically unpublished.

## Pager playbook

<!-- information about common errors from the monitoring system and -->
<!-- how to deal with them. this should be easy to follow: think of -->
<!-- your future self, in a stressful situation, tired and hungry. -->

## Disaster recovery

See [#revert-a-deployment-mistake](service/static-shim#revert-a-deployment-mistake)
for instruction on how to roll-back an environment to its previous state after
an accidental deployment.

# Reference

## Installation

Creating a new Lektor website is out of scope for this document.

Check out the [Quickstart](https://www.getlektor.com/docs/quickstart/) page in
the Lektor documentation to get started.

## SLA

## Design

The workflows around Lektor websites is heavily dependent on GitLab CI: it
handles building the sites, running tests and deploying them to various
environments including staging and production.

See [service/ci](service/ci) for general documentation about GitLab CI.

### CI build/test pipelines

The [lektor.yml][] CI template is used to configure pipelines for build and
testing Lektor website projects. Including this into the project's
`gitlab-ci.yml` is usually sufficient for GitLab CI to "do the right thing".

There are several elements that can be used to customize the build process:

 * `LEKTOR_BUILD_FLAGS`: this variable accepts a space separated list of
   flags to append to the `lektor build` command. For example, setting this
   variable to `npm` will cause `-f npm` to be appended to the build command.

 * `LEKTOR_PARTIAL_BUILD`: this variable can be used to alter the build process
   occurring on non-default branches and l10n-staging jobs. When set (to
   anything), it will append commands defined in `.setup-lektor-partial-build`
   to the job's `before_script`. Its main purpose is to pre-process website
   sources to reduce the build times by trimming less-essential content which
   contribute a lot to build duration. See the [web/tpo project CI][] for an
   example.

 * `TRANSLATION_BRANCH`: this variable must contain the name of the
   [translation repository][] branch used to store localization files. If this
   variable is absent, the website will be built without l10n.

[lektor.yml]: https://gitlab.torproject.org/tpo/tpa/ci-templates/-/blob/main/lektor.yml
[translation repository]: https://gitlab.torproject.org/tpo/translation
[web/tpo project CI]: https://gitlab.torproject.org/tpo/web/tpo/-/blob/main/.gitlab-ci.yml

Another method of customizing the build process is by overriding keys from the
`.lektor` hash (defined in the `lektor.yml` template) from their own
`.gitlab-ci.yml` file.

For example, this hash, added to `gitlab-ci.yml` will cause the jobs defined in
the template to use a different image, and set `GIT_STRATEGY` to `clone`.

    .lektor:
      image: ubuntu:latest
      variables:
        GIT_STRATEGY: clone

This is in addition to the ability to override the named job parameters
directly in `.gitlab-ci.yml`.

### CD pipelines and environments

The Tor Project Lektor websites are deployed automatically by GitLab by a process
of continuous deployment (CD).

#### Staging and production

Deployments to staging and production environments are handled by the
[static-shim-deploy.yml][] CI template. The [service/static-shim][] wiki page
describes the prerequisites for GitLab to be able to upload websites to the
[static mirror system][].

A basic Lektor project that deploys to production would have a `.gitlab-ci.yml`
set up like this:

    ---
    variables:
      SITE_URL: example.torproject.org

    include:
      project: tpo/tpa/ci-templates
      file:
        - lektor.yml
        - static-shim-deploy.yml

See the [#template-variables](service/static-shim#template-variables)
documentation for details about the variables involved in the deployment process.

See the [#working-with-a-staging-environments](service/static-shim#working-with-a-staging-environment)
documentation for details about adding a staging environment to a project's
deployment workflow.

[static-shim-deploy.yml]: https://gitlab.torproject.org/tpo/tpa/ci-templates/-/blob/main/static-shim-deploy.yml
[service/static-shim]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/static-shim
[static mirror system]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/static-component

### Review apps

Lektor projects which include `static-shim-deploy.yml` and have access to the
`REVIEW_STATIC_GITLAB_SHIM_SSH_PRIVATE_KEY` CI variable (this includes all
projects in the `tpo/web` namespace) have [Review apps][] automatically enabled.

See the [#working-with-review-apps](service/static-shim#working-with-review-apps)
documentation for details about how to use Review apps.

[Review apps]: https://docs.gitlab.com/ee/ci/review_apps/

### Localization staging

To support the work of translation contributors who work on the Tor Project
websites, we automatically build and deploy special localized versions of the
projects to `reviews.torproject.net`.

The workflow can be described as follows:

  1. Translations are contributed on Transifex

  2. Every 30 minutes, these changes are merged to the corresponding branches in
     the translation repository and pushed to [tpo/translation][]

  3. A project pipeline is triggered and runs the jobs from the
     [lektor-l10n-staging-trigger.yml][] CI template

  4. If the changed files include any `.po` which is >15% translated, a pipeline
     will be triggered in the Lektor project with the special `L10N_STAGING`
     variable added

  5. In the Lektor project, the presence of the `L10N_STAGING` variable alters
     the regular build job: all languages >15% translated are built instead of
     only the officially supported languages for that project. The result is
     deployed to `reviews.torproject.net/tpo/web/<project-name>/l10n`

To enable localization staging for a Lektor project, it's sufficient to add this
snippet in `.gitlab-ci.yml` in the relevant [tpo/translation][] branch

    variables:
      TRANSLATION_BRANCH : $CI_COMMIT_REF_NAME
      LEKTOR_PROJECT: tpo/web/<project-name>

    include:
      - project: tpo/tpa/ci-templates
        file: lektor-l10n-staging-trigger.yml

Replace `<project-name>` with the name of the Lektor GitLab project.

[tpo/translation]: https://gitlab.torproject.org/tpo/translation
[lektor-l10n-staging-trigger.yml]: https://gitlab.torproject.org/tpo/tpa/ci-templates/-/blob/main/lektor-l10n-staging-trigger.yml

## Issues

Lektor website projects on GitLab have individual issue trackers, so problems
related to specific websites such as typos, bad links, missing content or
build problems should be filed in the relevant tracker.

For problems related to deployments or CI templates specifically, [File][] or
[search][] for issues in the [ci-templates issue tracker][search].

 [File]: https://gitlab.torproject.org/tpo/tpa/ci-templates/-/issues/new
 [search]: https://gitlab.torproject.org/tpo/tpa/ci-templates/-/issues

## Maintainer, users, and upstream

Lektor websites are maintained in collaboration by the Web team and TPA.

## Monitoring and testing

Currently there is no monitoring beyond the supporting infrastructure (eg. DNS,
host servers, httpd, etc.).

## Logs and metrics

<!-- where are the logs? how long are they kept? any PII? -->
<!-- what about performance metrics? same questions -->

## Backups

There are no backups specific to Lektor.

Source code of our Lektor projects is backed up along with GitLab itself, and the
production build artifacts themselves are picked up with those of the hosts
comprising the static mirror system.

## Other documentation

 * [Lektor documentation](https://www.getlektor.com/docs/)

# Discussion

## Overview

<!-- describe the overall project. should include a link to a ticket -->
<!-- that has a launch checklist -->

<!-- if this is an old project being documented, summarize the known -->
<!-- issues with the project. to quote the "audit procedure":

 5. When was the last security review done on the project? What was
    the outcome? Are there any security issues currently? Should it
    have another security review?

 6. When was the last risk assessment done? Something that would cover
    risks from the data stored, the access required, etc.

 7. Are there any in-progress projects? Technical debt cleanup?
    Migrations? What state are they in? What's the urgency? What's the
    next steps?

 8. What urgent things need to be done on this project?

-->

## Goals

<!-- include bugs to be fixed -->

### Must have

### Nice to have

### Non-Goals

## Approvals required

<!-- for example, legal, "vegas", accounting, current maintainer -->

## Proposed Solution

## Cost

## Alternatives considered

<!-- include benchmarks and procedure if relevant -->