|
|
# Introduction to tor-browser-build
|
|
|
|
|
|
The `tor-browser-build` repository contains all the configuration files that tell which projects should be built, where to find their sources, and the scripts that run the actual builds.
|
|
|
|
|
|
All the Tor Browser builds we release are produced in this way, and all modifications must produce the intended result in this process before being accepted.
|
|
|
|
|
|
The build scripts are run in strictly controlled environments to make the official builds of Tor Browser [reproducible](https://reproducible-builds.org/): if you follow these instructions, your output will match ours, byte by byte.
|
|
|
|
|
|
The base project `tor-browser-build` is based on is called [RBM](http://rbm.torproject.org).
|
|
|
|
|
|
# Prerequisites
|
|
|
|
|
|
The build process needs to be run on Linux on x86_64 machines (also called amd64, dependingly on the distribution).
|
|
|
We use cross-compilation for all the other environments (e.g., MinGW for Windows, etc...).
|
|
|
Our build servers use Debian stable, but other distributions should work without problems.
|
|
|
|
|
|
The essential requirement is support for `user_namespaces (7)`, as the **builds run in unprivileged containers**.
|
|
|
|
|
|
Thus, the account running the build needs some subordinate user ids. Usually, they are created when adding the user.
|
|
|
If not, you may need to consult `subuid (5)` and/or contact your system administrator.
|
|
|
|
|
|
Depending on your configuration, you may encounter additional difficulties (e.g., SELinux contexts or other security settings).
|
|
|
|
|
|
Your system also needs Perl with the following modules installed:
|
|
|
|
|
|
- YAML::XS
|
|
|
- File::Basename
|
|
|
- Getopt::Long
|
|
|
- Template
|
|
|
- IO::Handle
|
|
|
- IO::CaptureOutput
|
|
|
- JSON
|
|
|
- File::Temp
|
|
|
- Path::Tiny
|
|
|
- File::Path
|
|
|
- File::Slurp
|
|
|
- File::Copy::Recursive
|
|
|
- String::ShellQuote
|
|
|
- Sort::Versions
|
|
|
- Digest::SHA
|
|
|
- Data::UUID
|
|
|
- Data::Dump
|
|
|
- DateTime
|
|
|
|
|
|
In case of Debian, you will need to install them with `apt`:
|
|
|
|
|
|
```shell
|
|
|
apt install libyaml-libyaml-perl libtemplate-perl libdatetime-perl \
|
|
|
libio-handle-util-perl libio-all-perl \
|
|
|
libio-captureoutput-perl libjson-perl libpath-tiny-perl \
|
|
|
libstring-shellquote-perl libsort-versions-perl \
|
|
|
libdigest-sha-perl libdata-uuid-perl libdata-dump-perl \
|
|
|
libfile-copy-recursive-perl libfile-slurp-perl git uidmap
|
|
|
```
|
|
|
|
|
|
Please, refer to your system's documentation for detailed instructions on installing them.
|
|
|
|
|
|
The final requirement is git, which is also called during the build steps, not only to get the tor-browser-build repository.
|
|
|
|
|
|
# Getting the source
|
|
|
|
|
|
The only repository you will need to clone is `tor-browser-build`, which will automatically download all the other required projects.
|
|
|
Its URL is [https://git.torproject.org/builders/tor-browser-build.git](https://git.torproject.org/builders/tor-browser-build.git).
|
|
|
|
|
|
Generally speaking, keeping the git history instead of downloading a zip is desirable for reasons explained in the following sections.
|
|
|
|
|
|
In addition, git can embed **GPG signatures**, which you can use to check that you are actually building Tor Browser.
|
|
|
|
|
|
Our build system checks the integrity of the code it downloads whenever possible. But these checks are reliable only if the build system code itself has not been compromised.
|
|
|
|
|
|
# Build types
|
|
|
|
|
|
Tor Browser has three release channels:
|
|
|
|
|
|
* stable (also called release in the code)
|
|
|
* alpha
|
|
|
* nightly
|
|
|
|
|
|
and supports 4 platforms, with possible variants:
|
|
|
|
|
|
* Linux (x86_64, x86)
|
|
|
* macOS (x86_64)
|
|
|
* Windows (x86_64, x86)
|
|
|
* Android (ARMv7, ARMv8/aarch64, x86, x86_64)
|
|
|
|
|
|
The makefile contains one target for each channel on each platform variant, plus some targets that group many platforms together (desktop, Android, or both).
|
|
|
|
|
|
For example, the following targets are all valid:
|
|
|
|
|
|
* `make release`: makes the stable release for all platforms;
|
|
|
* `make alpha-desktop`: makes the alpha release for Linux, Windows, both 32 and 64 bit, and Mac;
|
|
|
* `make nightly-linux-x86_64`: makes a nightly build for x86_64 Linux.
|
|
|
|
|
|
## Testbuilds
|
|
|
|
|
|
We have an additional configuration that we call _testbuild_. It can be based on any channel (by default, alpha, but it can be configured), and its output is very similar to its base. However, some steps are skipped to make the process quicker.
|
|
|
|
|
|
The main difference with the base target is that desktop test builds are English-only (Tor Browser on desktop creates an installer for each language, which is time-consuming because of the compression).
|
|
|
|
|
|
The difference on Android is that while regular builds compile GeckoView for all architectures, testbuilds compile it only for the desired one.
|
|
|
This reduced the build time by half to 3 hours (or more) depending on the machine.
|
|
|
|
|
|
# Running the build
|
|
|
|
|
|
## Git branches and tags
|
|
|
|
|
|
**Targets are not guaranteed to work in every possible git tree**. You should `git checkout` the correct branch/tag before starting a build.
|
|
|
|
|
|
Nightlies should generally run everywhere, but we run them at the latest commit of the `master` branch.
|
|
|
|
|
|
Alphas are also based on `master`, but they are run on a specific annotated tag that we create before building an alpha for its release.
|
|
|
The tag name usually has this format: `tbb-VVVV-buildN`, where `VVVV` is the version and `N` is the build number.
|
|
|
If one version has multiple build numbers, some may not be buildable or have some runtime error or may not be reproducible; generally, you should refer to the highest one.
|
|
|
|
|
|
Stable builds are similar to alphas: they have tags in the same format, but the associated commits are in another branch (usually `maint-VV`).
|
|
|
|
|
|
**Both alpha and stable tags are always signed**.
|
|
|
|
|
|
## Customize your build environment
|
|
|
|
|
|
You will likely want to customize your build environment.
|
|
|
|
|
|
The first step to do is copy `rbm.local.conf.example` to `rbm.local.conf`.
|
|
|
|
|
|
The example file contains the most common settings to override, but all the options of `rbm.conf` could be changed in `rbm.local.conf`, if needed.
|
|
|
|
|
|
The most likely to be changed settings are:
|
|
|
|
|
|
* `buildconf/num_procs`: sets the number of concurrent threads. You can also set it temporarily with the `RBM_NUM_PROCS` environment variable, but if you build Tor Browser often, you can make the change permanent in this way;
|
|
|
* `targets/torbrowser-testbuild`: the target for the testbuilds. Usually, developers run them in the nightly profile.
|
|
|
|
|
|
## The actual build process
|
|
|
|
|
|
The makefile is very short, and it mainly contains calls to `rbm`, with the different targets set.
|
|
|
|
|
|
If you want to obtain a Tor Browser build, you can just need to call make with one of those targets.
|
|
|
|
|
|
Please notice that, for reproducibility purposes, we compile many of the tools we use, including GCC, Clang, Rust, Go, and others. Therefore, the first build will take a really long time. The following ones will be faster.
|
|
|
|
|
|
You can also call `rbm` directly to build only one component, for example: `./rbm/rbm build nasm --target nightly --target torbrowser-android-x86_64`.
|
|
|
|
|
|
It will also build any dependencies that are out of date.
|
|
|
|
|
|
## Build failures and temporary files
|
|
|
|
|
|
If the build succeeds, `rbm` exits with status code 0.
|
|
|
|
|
|
However, if it fails, the default option is to open an interactive shell in the container environment. Please see the section dedicated to it for further information on using it.
|
|
|
|
|
|
You should close it with the `exit` command or Ctrl-D. In this way, the build process will terminate gracefully and delete any temporary files.
|
|
|
|
|
|
If you click Ctrl-C instead, you have a premature exit, and the container will be left there. You can remove the temporary files with this command: `./rbm/container run -- rm -rf tmp/rbm-*` (you may want to adjust the directory to remove, and/or invoke `./rbm/container` to obtain useful information on how to use that command).
|
|
|
It can also be used in other cases that lead to the same condition, e.g., a power loss.
|
|
|
|
|
|
# Output
|
|
|
|
|
|
The build process writes in several subdirectories of the directory where you cloned tor-browser-build:
|
|
|
|
|
|
* `release`, `alpha:` here, you can find the installers of Tor Browser stable and alpha, respectively. They are organized by version, each of which has a `signed` and an `unsigned` subdirectory: you will find the installers created by you on the latter;
|
|
|
* `nightly`: here, you can find the installers for nightly builds grouped by date. Nightly are never signed;
|
|
|
* `testbuild`: here, you can find the installers created by the testbuilds. Testbuilds are neither signed nor versioned, they are only organized by platform, so when you run a new one, it will replace the already existing one.
|
|
|
|
|
|
There, you will find the correct file to test your builds.
|
|
|
|
|
|
In addition to that, you will notice that some other subdirectories occupy a lot of space:
|
|
|
|
|
|
* `git_clones`: as written above, `tor-browser-build` downloads the source code for all the components that it compiles, and, in the case of git repositories, it places them there;
|
|
|
* `out:` this directory is used mainly for artifacts (compilation output of a component), source code archives, precompiled dependencies downloaded from the Internet, and possibly other files.
|
|
|
|
|
|
Finally, `logs` contain the logs file, organized by project. By default, new builds append to the exiting logs, but that can be changed.
|
|
|
|
|
|
## Verify it
|
|
|
|
|
|
In the same directory as the installers, you will find the installer hashes.
|
|
|
|
|
|
If you followed these instructions without changing anything, you should obtain the same hashes as our unsigned builds.
|
|
|
|
|
|
Otherwise, please consider contacting us, as we may treat your case as an issue.
|
|
|
|
|
|
## Cleaning the artifacts
|
|
|
|
|
|
# Testing patches
|
|
|
|
|
|
The easiest way to try some changes in tor-browser-build is by telling it to use custom sources.
|
|
|
Every project has a configuration file in `projects/$PROJECT/config`.
|
|
|
|
|
|
If it is a git-based project, you can customize its `git_url` and `git_hash`.
|
|
|
The former can either be a URL or a local directory, like `/home/you/code/project-name/.git`.
|
|
|
|
|
|
So, the workflow to test new patches is:
|
|
|
|
|
|
1. clone the repository of the project to patch;
|
|
|
2. create a feature branch (optional but useful, especially if you want to contribute that patch back to us);
|
|
|
3. set the `git_hash` of the project configuration either to the commit hash or to the branch name (be advised: if fetching is disabled in `rbm.local.conf`, branch names may not work);
|
|
|
4. run the build.
|
|
|
|
|
|
Sources can also be patched at build time by modifying the build scripts, e.g., for projects we do not fork.
|
|
|
|
|
|
## Using the debug shell
|
|
|
|
|
|
### What and why
|
|
|
|
|
|
When a build fails, you are presented with a debug shell inside the building environment (unless you disabled it in `rbm.local.conf`).
|
|
|
|
|
|
You can use it to try to fix the problems.
|
|
|
However, there is no way to resume the build process, even if you manage to produce working artifacts in this environment.
|
|
|
Nonetheless, trying the fix there is a good idea because you may find other problems, and it can save a lot of time if the solution does not work as expected.
|
|
|
|
|
|
You can first try to understand where the build stopped by looking at the project's build log.
|
|
|
|
|
|
### Before doing anything
|
|
|
|
|
|
Before starting using the debug shell, there are a couple of usually optional but highly suggested steps.
|
|
|
|
|
|
The first one is setting the `TERM` variable, possibly to match your current terminal (usually `xterm-256color` or `tmux-256color`, etc..).
|
|
|
|
|
|
Then, you will likely want to start bash because the default shell has limited capabilities for interactive sessions.
|
|
|
|
|
|
```shell
|
|
|
export TERM=xterm-256color
|
|
|
bash
|
|
|
```
|
|
|
|
|
|
`TERM` must be set to launch some building systems (notably, Firefox's one) and some other programs, such as nano.
|
|
|
In bullseye-based containers (and following versions), the terminal type influences bash, too, so setting it before launching the shell is better.
|
|
|
|
|
|
### What to do there
|
|
|
|
|
|
The shell is launched in the initial directory of the container.
|
|
|
|
|
|
The automatic procedure launches the `build` script it finds there.
|
|
|
|
|
|
So, the usual debug process consists in:
|
|
|
|
|
|
1. analyze the build script and understand which command failed;
|
|
|
2. do something to prevent that command from failing again (e.g., change the source code);
|
|
|
3. run the commands coming before the failing one on the build script selectively:
|
|
|
- define all the environment variables like the script;
|
|
|
- **do not run** any non-idempotent command, i.e., do not extract/copy/move/patch files a second time;
|
|
|
4. run the failed step;
|
|
|
5. run all the remaining commands.
|
|
|
|
|
|
Try to keep track of any changes you make on the debug environment because they will be deleted once the shell is closed.
|
|
|
|
|
|
# Developer builds
|
|
|
|
|
|
`tor-browser-build` cannot produce incremental builds, so you may check out [this page](https://gitlab.torproject.org/tpo/applications/tor-browser/-/wikis/Hacking) for new developments.
|
|
|
Be careful to **use them only for testing**. |
|
|
\ No newline at end of file |