|
|
# Introduction to tor-browser-build
|
|
|
|
|
|
Tor Browser Bundle is a collection of several programs: tor, a modified fork of Firefox, the pluggable transports, etc...
|
|
|
|
|
|
The `tor-browser-build` repository contains all the configuration files that tell which projects should be built, where to find their sources, the scripts that run the actual builds, and how to create the final packages.
|
|
|
|
|
|
All the Tor Browser builds we release are produced in this way, and all modifications must produce the intended result in this process before being accepted.
|
|
|
|
|
|
The build scripts are run in strictly controlled environments to make the official builds of Tor Browser [reproducible](https://reproducible-builds.org/): if you follow these instructions, your output will match ours, byte by byte.
|
|
|
|
|
|
`tor-browser-build` is based on another, more generic project, called [RBM](http://rbm.torproject.org).
|
|
|
|
|
|
# Prerequisites
|
|
|
|
|
|
The build process needs to be run on Linux on x86_64 machines (also called amd64, dependingly on the distribution).
|
|
|
We use cross-compilation for all the other environments (e.g., MinGW for Windows, etc...).
|
|
|
Our build servers use Debian stable, but other distributions should work without problems.
|
|
|
|
|
|
The essential requirement is support for `user_namespaces (7)`, as the **builds run in unprivileged containers**.
|
|
|
|
|
|
Thus, the account running the build needs some subordinate user ids. Usually, they are created when adding the user.
|
|
|
If not, you may need to consult `subuid (5)` and/or contact your system administrator.
|
|
|
|
|
|
Depending on your configuration, you may encounter additional difficulties (e.g., SELinux contexts or other security settings).
|
|
|
|
|
|
Your system also needs Perl with the following modules installed:
|
|
|
|
|
|
- YAML::XS
|
|
|
- File::Basename
|
|
|
- Getopt::Long
|
|
|
- Template
|
|
|
- IO::Handle
|
|
|
- IO::CaptureOutput
|
|
|
- JSON
|
|
|
- File::Temp
|
|
|
- Path::Tiny
|
|
|
- File::Path
|
|
|
- File::Slurp
|
|
|
- File::Copy::Recursive
|
|
|
- String::ShellQuote
|
|
|
- Sort::Versions
|
|
|
- Digest::SHA
|
|
|
- Data::UUID
|
|
|
- Data::Dump
|
|
|
- DateTime
|
|
|
|
|
|
In case of Debian, you can install them with `apt`:
|
|
|
|
|
|
```shell
|
|
|
apt install libyaml-libyaml-perl libtemplate-perl libdatetime-perl \
|
|
|
libio-handle-util-perl libio-all-perl \
|
|
|
libio-captureoutput-perl libjson-perl libpath-tiny-perl \
|
|
|
libstring-shellquote-perl libsort-versions-perl \
|
|
|
libdigest-sha-perl libdata-uuid-perl libdata-dump-perl \
|
|
|
libfile-copy-recursive-perl libfile-slurp-perl git uidmap
|
|
|
```
|
|
|
|
|
|
For other distros, please refer to your system's documentation.
|
|
|
|
|
|
The final requirement is git, which is also called during the build steps, not only to fetch the tor-browser-build repository.
|
|
|
|
|
|
# Getting the source
|
|
|
|
|
|
The only repository you will need to clone is `tor-browser-build`, then it will automatically download all the other required projects:
|
|
|
|
|
|
```shell
|
|
|
git clone https://git.torproject.org/builders/tor-browser-build.git
|
|
|
```
|
|
|
|
|
|
Generally speaking, keeping the git history instead of downloading a zip is desirable for reasons explained in the following sections.
|
|
|
|
|
|
In addition, git can embed **GPG signatures**, which you can use to check that you are actually building Tor Browser.
|
|
|
|
|
|
Our build system checks the integrity of the code it downloads whenever possible. But these checks are reliable only if the build system code itself has not been compromised.
|
|
|
|
|
|
# Build types
|
|
|
|
|
|
Tor Browser has three release channels:
|
|
|
|
|
|
* stable (also called simply _release_ in the code)
|
|
|
* alpha
|
|
|
* nightly
|
|
|
|
|
|
and officially supports 4 platforms, each one on one or more architectures:
|
|
|
|
|
|
* Linux (x86_64, x86)
|
|
|
* macOS (x86_64, aarch64)
|
|
|
* Windows (x86_64, x86)
|
|
|
* Android (ARMv7, ARMv8/aarch64, x86, x86_64)
|
|
|
|
|
|
The makefile contains one target for each channel on each platform variant, plus some targets that group many platforms together (desktop, android, or both).
|
|
|
|
|
|
For example, the following targets are all valid:
|
|
|
|
|
|
* `make torbrowser-release`: makes the stable release for all platforms;
|
|
|
* `make torbrowser-alpha-desktop`: makes the alpha release for Linux, Windows (32 and 64 bit for both the OS), and Mac;
|
|
|
* `make torbrowser-nightly-linux-x86_64`: makes a nightly build for x86_64 Linux.
|
|
|
* `make torbrowser-release-android`: makes a release build of android for all 4 arches
|
|
|
|
|
|
You can examine the `Makefile` for a exhaustive list of all the targets
|
|
|
|
|
|
## Testbuilds
|
|
|
|
|
|
We have an additional configuration that we call _testbuild_. It can be based on any channel (by default, alpha, but it can be configured), and its output is very similar to its base. However, some steps are skipped to make the process quicker.
|
|
|
|
|
|
The main difference with the base target is that desktop test builds are English-only (Tor Browser on desktop creates an installer for each language, which is time-consuming because of the compression).
|
|
|
|
|
|
The difference on Android is that while regular builds compile GeckoView for all architectures, testbuilds compile it only for the desired one.
|
|
|
This reduces the build time by half to 3 hours (or more) depending on your machine.
|
|
|
|
|
|
# Running the build
|
|
|
|
|
|
## Git branches and tags
|
|
|
|
|
|
**Targets are not guaranteed to succeed in every possible git tree**. You should `git checkout` the correct branch/tag before starting a build.
|
|
|
|
|
|
We run nightlies at the latest commit of the `main` branch, so they should run at every commit of the history.
|
|
|
|
|
|
Alphas are also based on `main`, but they are run on a specific annotated tag (we create it just before an alpha release).
|
|
|
The tag name usually has this format: `tbb-VVVV-buildN`, where `VVVV` is the version and `N` is the build number.
|
|
|
If one version has multiple build numbers, some may not be buildable or have some runtime error or may not be reproducible; generally, you should refer to the highest one.
|
|
|
|
|
|
Stable builds are similar to alphas: they have tags in the same format, but the associated commits are in another branch (usually `maint-VV`).
|
|
|
|
|
|
**Both alpha and stable tags are always signed**.
|
|
|
|
|
|
## Customize your build environment
|
|
|
|
|
|
You will likely want to customize your build environment.
|
|
|
|
|
|
The first step to do is copy `rbm.local.conf.example` to `rbm.local.conf`.
|
|
|
|
|
|
The example file contains the most common settings to override, but all the options of `rbm.conf` could be changed in `rbm.local.conf`, if needed.
|
|
|
|
|
|
The most likely to be changed settings are:
|
|
|
|
|
|
* `buildconf/num_procs`: sets the number of concurrent threads. You can also set it temporarily with the `RBM_NUM_PROCS` environment variable, but if you build Tor Browser often, you can make the change permanent in this way;
|
|
|
* `targets/torbrowser-testbuild`: the target for the testbuilds. Usually, developers run them in the nightly profile.
|
|
|
|
|
|
## The actual build process
|
|
|
|
|
|
The makefile is very short, as it mainly contains calls to `rbm`, with different targets.
|
|
|
|
|
|
If you want to build Tor Browser without any changes, just call `make` with one of those targets (see above for more details on their meaning).
|
|
|
|
|
|
Please notice that, for reproducibility purposes, we compile many of the tools we use, including GCC, Clang, Rust, Go, and others. Therefore, the first build will take a really long time. The following ones will be faster because they will reuse all the artifacts that are up to date.
|
|
|
|
|
|
You can also call `rbm` directly to build only one component, for example:
|
|
|
|
|
|
```shell
|
|
|
./rbm/rbm build nasm --target nightly --target torbrowser-android-x86_64
|
|
|
```
|
|
|
|
|
|
It will also build any dependencies that have not been built yet, or that are out of date.
|
|
|
|
|
|
### Build failures and temporary files
|
|
|
|
|
|
If the build succeeds, `rbm` and `make` exit with status code 0.
|
|
|
|
|
|
If it fails, the default option is to open an interactive shell in the container used to build the project that failed.
|
|
|
Please see the section dedicated to it for further information on using it.
|
|
|
|
|
|
You should close it with the `exit` command or Ctrl-D.
|
|
|
In this way, the build process will terminate gracefully and delete any temporary files.
|
|
|
|
|
|
Clicking Ctrl-C will cause an immediate exit, without cleaning the container and any other temporary files.
|
|
|
The same will happen in case of power loss or any other sudden interruption of the build.
|
|
|
|
|
|
Your user will not be able to delete these files, because they are associated to your subuids/subguids.
|
|
|
You will need to remove them from the container environment with this command:
|
|
|
|
|
|
```shell
|
|
|
./rbm/container run -- rm -rf tmp/rbm-*
|
|
|
```
|
|
|
|
|
|
Also, you may want to adjust the directory to remove.
|
|
|
`./rbm/container` will give you useful information on how to use that command.
|
|
|
|
|
|
## Output
|
|
|
|
|
|
The build archives/installers will be in one of these directories of `tor-browser-build`:
|
|
|
|
|
|
* `release`, `alpha`: here, you can find the installers of Tor Browser stable and alpha, respectively. They are organized by version, each of which has a `signed` and an `unsigned` subdirectory: you will find the installers created by you on the latter;
|
|
|
* `nightly`: here, you can find the installers for nightly builds grouped by date. Nightly builds are never signed;
|
|
|
* `testbuild`: here, you can find the installers created by the testbuilds. Testbuilds are neither signed nor versioned, they are only organized by platform, so if you run one, it will replace any already existing one for the same platform/architecture combination.
|
|
|
|
|
|
In addition to that, you will notice that some other directories occupy a lot of space:
|
|
|
|
|
|
* `git_clones`: as written above, `tor-browser-build` downloads the source code for all the components that it compiles, and, in the case of git repositories, it places them there. RBM manages these directories automatically, and usually you should not need to do anything there;
|
|
|
* `out`: this directory is used for artifacts (compilation output of a component), source code archives, precompiled dependencies downloaded from the Internet, and possibly other files.
|
|
|
|
|
|
Finally, `logs` contains the log files, organized by project. By default, new builds append to the exiting files, but this behavior can be customized.
|
|
|
|
|
|
### Verify it
|
|
|
|
|
|
In the same directory as the installers, you will find the installer hashes.
|
|
|
|
|
|
If you followed these instructions without changing anything, you should obtain the same hashes as our unsigned builds.
|
|
|
|
|
|
We sign and upload our `sha256sums-unsigned-build.txt`, so you can also download them, verify their signatures, and run `sha256sum -c sha256sums-unsigned-build.txt`.
|
|
|
|
|
|
Our signatures are detached, so if you used the same `make` target as us, as an alternative you can also only download the signature, and it should match your `sha256sums-unsigned-build.txt`.
|
|
|
|
|
|
Otherwise, please check that you are using the correct hashes, and then consider contacting us, as we may treat your case as an issue.
|
|
|
|
|
|
For desktop builds, we produce also smaller, incremental archives (`make incrementals-{release,alpha,nightly}`): their hashes are in `sha256sums-unsigned-build.incrementals.txt`, instead, but they should match as well.
|
|
|
|
|
|
If you are a TPO dev verifying a build, you can use the `make` targets `$browser-upload-sha256sumes-$channel` to sign and upload the hashes to people.torproject.org, assuming you've set a `tpo_user` in `rmb.local.conf`. Example command: `make torbrowser-upload-sha256sums-release`.
|
|
|
|
|
|
## Cleaning the artifacts
|
|
|
|
|
|
Since we often switch git branches, `make clean` tries to remove old artifacts in a smart way.
|
|
|
But it needs to be configured to do so.
|
|
|
|
|
|
**TODO**: explain how to configure the clean phase.
|
|
|
|
|
|
# Testing patches
|
|
|
|
|
|
The easiest way to try some changes in `tor-browser-build` is by telling it to use custom sources.
|
|
|
|
|
|
Every component has a configuration file in `projects/$PROJECT_NAME/config`.
|
|
|
If it is a git-based project, you can customize its `git_url` and `git_hash`.
|
|
|
The former can either be a URL or a local directory, like `/home/you/code/project-name/.git`, the latter can be a commit hash, a tag, or a branch name.
|
|
|
|
|
|
The workflow to test new patches is:
|
|
|
|
|
|
1. clone the repository of the project to patch;
|
|
|
2. create a feature branch (optional but useful, especially if you want to contribute that patch back to us);
|
|
|
3. set the `git_hash` (be advised: if fetching is disabled in `rbm.local.conf`, branch names may not work, and messing with tags causes problems, too, which can be solved only by acting directly to `git_clones`);
|
|
|
4. run the build.
|
|
|
|
|
|
Sources can also be patched at build time by injecting `.patch`/`.diff` files into the build container and modifying the build scripts to apply them.
|
|
|
This is useful, for example, for projects we do not fork.
|
|
|
|
|
|
## Using the debug shell
|
|
|
|
|
|
### What, why and caveats
|
|
|
|
|
|
When a build fails, you are presented with a debug shell inside the building environment (unless you disabled it in `rbm.local.conf`).
|
|
|
|
|
|
You can use it to try to fix the problems.
|
|
|
However, there is no way to resume the build process, even if you manage to produce working artifacts in this environment.
|
|
|
Nonetheless, trying your fixes there is a good idea because you may find other problems, and it can save a lot of time if the solution does not work as expected.
|
|
|
|
|
|
### Before doing anything
|
|
|
|
|
|
Before starting using the debug shell, there are a couple of usually optional but highly suggested steps.
|
|
|
|
|
|
The first one is setting the `TERM` variable, possibly to match your current terminal (usually `xterm-256color` or `tmux-256color`, etc..).
|
|
|
|
|
|
Then, you will likely want to start Bash because the default shell has limited capabilities for interactive sessions.
|
|
|
|
|
|
```shell
|
|
|
export TERM=xterm-256color
|
|
|
bash
|
|
|
```
|
|
|
|
|
|
`TERM` must be set to launch some building systems (notably, Firefox's one) and some other programs, such as nano.
|
|
|
In bullseye-based containers (and following versions), the terminal type influences Bash, too, so setting it before launching the shell is better.
|
|
|
|
|
|
### What to do there
|
|
|
|
|
|
The automatic procedure runs a `build` script that is generated from `projects/$PROJECT_NAME/build`.
|
|
|
If this script exits with a status different from 0, the build is considered failed, and the debug shell is opened at the directory containing the script.
|
|
|
|
|
|
So, the usual debug process consists in:
|
|
|
|
|
|
1. analyze the project's build log and the `build` script to understand which command failed;
|
|
|
2. do something to prevent that command from failing again (e.g., change the source code);
|
|
|
3. run the commands coming before the failing one on the build script selectively:
|
|
|
- define all the environment variables like the script;
|
|
|
- **do not run** any non-idempotent command, i.e., do not extract/copy/move/patch files a second time;
|
|
|
4. run the failed step;
|
|
|
5. run all the remaining commands.
|
|
|
|
|
|
Try to keep track of any changes you make on the debug environment because they will be deleted once the shell is closed.
|
|
|
|
|
|
Also, `build` is the default name, but it can be customized on the project configuration, or some complex projects (such as GeckoView) need in multiple build phases, and each one has a different script name.
|
|
|
|
|
|
# Developer builds
|
|
|
|
|
|
`tor-browser-build` cannot produce incremental builds, so you may check out [this page](/Development-Information/Tor-Browser/dev-Build) for new developments that need frequent, incremental and quick builds.
|
|
|
Be careful to **use them only for testing**. |
|
|
\ No newline at end of file |