Skip to content
Snippets Groups Projects

Compiling a local version of the website

Last edited by Gabriel Filion

All the lektor-based websites work the same, although they have different repositories. This howto was written for the Tor Project main website repository address, https://gitlab.torproject.org/tpo/web/tpo.git, but it can also work for:

Setting up a development environment

The process to set up a development environment for one of our sites is a bit complicated, but once you set it up, you won't need to do it again!

1. Install Python 3.11

First, check if you have the correct version installed by running the following shell command:

python3 --version

If not, download and install Python 3.11. If you're on linux, your package manager might provide a python3.11 package.

Note: this procedure is not currently working on python 3.12+ and multiple adjustments will need to be made in the tpo website repository to get it working for more recent versions of python.

2. Clone the repo from gitlab.torproject.org

Run the following shell command:

git clone https://gitlab.torproject.org/tpo/web/tpo.git && cd tpo

3. Set up a virtual Python environment

This step isn't strictly necessary, but without it Lektor will globally install plugins, causing build issues.

python3.11 -m venv venv
source venv/bin/activate
pip install --upgrade pip lektor

Now, exit the the virtual environment by running deactivate.

4. Install the dependencies for the lektor-i18n-plugin

lektor-i18n-plugin requires gettext and python3-babel. On Debian these can be installed with the following command:

sudo apt-get install gettext python3-babel

5. Clone the translations repository

Download the correct branch of the translations repo to the ./i18n/ folder:

git clone --single-branch --branch tpo-web https://gitlab.torproject.org/tpo/translation.git i18n

Make sure to change --branch tpo-web to the correct branch. For the other websites, the branches are: tbmanual-contentspot, support-portal and communitytpo-contentspot.

If it complains about the folder already existing and not being empty, read about common Lektor errors.

6. Initialize the building blocks submodule

We use a common "building blocks" submodule to make sure our sites look the same. We call this submodule lego. Run this command to initialize it:

git submodule update --init --recursive --remote

This will clone lego into the ./lego/ directory. Your local copy of the site will already have symlinks set up to point to lego files where needed.

7. Install the website's python dependencies

Our websites use a number of lektor plugins to work. To be consistent in what python packages are used for the plugins and their dependencies, we use a few different requirements.txt files to install python packages.

Activate your virtual environment, and then install the requirements:

source venv/bin/activate
pip install -r lego/lektor-requirements.txt
pip install -r requirements.txt
wget -qO- https://gitlab.torproject.org/tpo/web/lego/-/raw/main/scripts/download-sass.sh | bash

Once it has finished, add dart-sass to your PATH by executing the command the script has printed on output. It should be similar to echo PATH=/root/.local/bin:"$PATH" >> /root/.bashrc && source /root/.bashrc.

8. Build the website for the first time

Several of our plugins (lektor-i18n and lektor-scss) require that the lektor site be built twice. If you only build once, you'll be missing things like CSS or translations. Run lektor build -f scss and wait for the build to finish before developing on the website. Once you've waited for this initial build, you should be able to develop on the site normally.

You've finished setting up your development environment, congratulations!

9. Making changes to the website

Now that you've got your development environment, it's time to start actually making changes to the site.

Building the website

Run lektor build -f scss to build the website. The -f scss part of the command enables the lektor-scss plugin. Most of our sites use the scss plugin, but not all of them. As of right now the easiest way to find out if a site uses the scss plugin is to run this command:

grep -q 'LEKTOR_BUILD_FLAGS: *scss' .gitlab-ci.yml && echo 'this site uses the lektor-scss plugin' || echo "this site doesn't use the scss plugin"

Some websites need additional -f flags. You should check the README file of the site to find out what plugins it needs. If the site doesn't have a README file, you can also check the .gitlab-ci.yml file, and look for the line starting with LEKTOR_BUILD_FLAGS:

Check out the developing on the website page for more detailed information.

Tip: To save some time while you build in local, you can edit the configs/i18n.ini file and take some languages out of the 'translations' option.

Tip: When you build a site that uses more than one flag, remember to include -f before each one, like this: lektor build -f flag1 -f flag2

Changing the content

Lektor content uses a set of directories and contents.lr files to define the structure of the site. You can find these contents.lr files in the content/ directory. See the Writing Content wiki page for more information.

Translations

The translation mechanism is hooked into the build system, so translating a website just means building the website. You can learn more about the translation process here.

Running a local dev server

lektor s runs a lektor server and allows you to have the website updating on http://localhost:5000. You can try this out by running the server and accessing it on your browser. Remember to include any -f build flags in the lektor command!

Note: It will not work on Tor Browser. :)

Most common lektor errors

FileNotFoundError

FileNotFoundError: [Errno 2] No such file or directory: '/content/**some filename**/contents.lr'

This happens when a source file contents.lr has been deleted in the repository, but their associated language files are still there. So, when trying to update the language files, lektor fails because the source file is not there. Delete the folder with the translations and build again:

rm -r '/content/**some filename**/'

Update lego

Example:

/training/code-of-conduct/index.html (jinja2.exceptions.UndefinedError: 'render_text' is undefined)

Can be fixed with git submodule update --init --recursive

SyntaxError

except IOError, e: SyntaxError: invalid syntax

This can be the python version. Make sure you're using Python 3.8

  • If you encounter the error: from urllib import requestImportError: cannot import name request, you can fix it by going to ~tpo/lego/packages/xml-to-html/lektor_xml_to_html.py and changing the line from urllib import request to from urllib3 import request. You may also have to do the same change for the ~tpo/lego/packages/txt-to-html/lektor_txt_to_html.py file.

  • A possible error is UnicodeEncodeError: 'ascii' codec can't encode character u'\xae' stemming from ~tpo/lego/packages/i18n/lektor_i18n.py. This can be fixed by adding the following code snippet to the top of the page after the import sys phrase:

reload(sys)
sys.setdefaultencoding('utf8')

'i18n' already exists

If git clone https://gitlab.torproject.org/tpo/translation.git i18n fails with:

fatal: destination path 'i18n' already exists and is not an empty directory.

This happens if you tried to build lektor before downloading the translations. You need to remove that directory and try cloning again:

rm -r 'i18n'
git clone https://gitlab.torproject.org/tpo/translation.git i18n

babel.cfg not found

If when building the website you get the error:

"FileNotFoundError: [Errno 2] No such file or directory: 'babel.cfg'"

You are not on the root directory for lektor. Go to the root directory and try again.

Weird characters between lines on the website

Sometimes the paragraphs of the website will be mixed with phrases like: Project-Id-Version: PACKAGE VERSION Report-Msgid-Bugs-To: PO-Revision-Date: 2018-11-14 12:31+0000 Last-Translator: ...

scruff-on-lektor-local

This means that the i18n plugin has not been able to update the pages well. You need to run lektor build more times for this text to disappear.

If the unwanted text is still present after building the site multiple times, you might be using Python 3.9 or higher. The lektor-i18n-plugin requires Python 3.8 to work.

Setuptools error

If when building the website you get the error:

  Collecting setuptools>=38.2.5
    Using cached https://files.pythonhosted.org/packages/a8/e7/1440b0d19054a5616e9e5beeaa22f68485aa9de20d187f04e52880b7ae7a/setuptools-59.2.0.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'error'
    Complete output from command /usr/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-4e9g0x0f --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple --:
    ERROR: You must give at least one requirement to install (see "pip help install")

    ----------------------------------------
  Command "/usr/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-4e9g0x0f --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple --" failed with error code 1 in None

  ----------------------------------------
Command "/usr/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-xojjpvj9 --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- setuptools>=38.2.5 wheel" failed with error code 1 in None

You will need to upgrade your pip and run lektor server again:

pip3 install --upgrade pip

Changes to content or templates not showing up after building

If you build a site, make changes, and build the site again, Lektor might not rebuild the file. If you're wondering why your changes aren't showing up, try running lektor clean --yes and re-building. This forces lektor to build from scratch. It can take a while, but it might fix your problem. THe translation files will already be in place, so you won't need to build from scratch multiple times.

Comments

    Please register or sign in to add a comment.