|
|
# Introduction to tor-browser-build
|
|
|
|
|
|
The `tor-browser-build` repository contains all the configuration files that tell which projects should be built, where to find their sources, and the scripts that run the actual builds.
|
|
|
Tor Browser Bundle is a collection of several programs: tor, a modified fork of Firefox, the pluggable transports, etc...
|
|
|
|
|
|
The `tor-browser-build` repository contains all the configuration files that tell which projects should be built, where to find their sources, the scripts that run the actual builds, and how to create the final packages.
|
|
|
|
|
|
All the Tor Browser builds we release are produced in this way, and all modifications must produce the intended result in this process before being accepted.
|
|
|
|
|
|
The build scripts are run in strictly controlled environments to make the official builds of Tor Browser [reproducible](https://reproducible-builds.org/): if you follow these instructions, your output will match ours, byte by byte.
|
|
|
|
|
|
The base project `tor-browser-build` is based on is called [RBM](http://rbm.torproject.org).
|
|
|
`tor-browser-build` is based on another, more generic project, called [RBM](http://rbm.torproject.org).
|
|
|
|
|
|
# Prerequisites
|
|
|
|
... | ... | @@ -42,7 +44,7 @@ Your system also needs Perl with the following modules installed: |
|
|
- Data::Dump
|
|
|
- DateTime
|
|
|
|
|
|
In case of Debian, you will need to install them with `apt`:
|
|
|
In case of Debian, you can install them with `apt`:
|
|
|
|
|
|
```shell
|
|
|
apt install libyaml-libyaml-perl libtemplate-perl libdatetime-perl \
|
... | ... | @@ -53,14 +55,17 @@ apt install libyaml-libyaml-perl libtemplate-perl libdatetime-perl \ |
|
|
libfile-copy-recursive-perl libfile-slurp-perl git uidmap
|
|
|
```
|
|
|
|
|
|
Please, refer to your system's documentation for detailed instructions on installing them.
|
|
|
For other distros, please refer to your system's documentation for instructions on installing them.
|
|
|
|
|
|
The final requirement is git, which is also called during the build steps, not only to get the tor-browser-build repository.
|
|
|
The final requirement is git, which is also called during the build steps, not only to fetch the tor-browser-build repository.
|
|
|
|
|
|
# Getting the source
|
|
|
|
|
|
The only repository you will need to clone is `tor-browser-build`, which will automatically download all the other required projects.
|
|
|
Its URL is [https://git.torproject.org/builders/tor-browser-build.git](https://git.torproject.org/builders/tor-browser-build.git).
|
|
|
The only repository you will need to clone is `tor-browser-build`, then it will automatically download all the other required projects:
|
|
|
|
|
|
```shell
|
|
|
git clone https://git.torproject.org/builders/tor-browser-build.git
|
|
|
```
|
|
|
|
|
|
Generally speaking, keeping the git history instead of downloading a zip is desirable for reasons explained in the following sections.
|
|
|
|
... | ... | @@ -76,7 +81,7 @@ Tor Browser has three release channels: |
|
|
* alpha
|
|
|
* nightly
|
|
|
|
|
|
and supports 4 platforms, with possible variants:
|
|
|
and officially supports 4 platforms, each one on one or more architectures:
|
|
|
|
|
|
* Linux (x86_64, x86)
|
|
|
* macOS (x86_64)
|
... | ... | @@ -88,7 +93,7 @@ The makefile contains one target for each channel on each platform variant, plus |
|
|
For example, the following targets are all valid:
|
|
|
|
|
|
* `make release`: makes the stable release for all platforms;
|
|
|
* `make alpha-desktop`: makes the alpha release for Linux, Windows, both 32 and 64 bit, and Mac;
|
|
|
* `make alpha-desktop`: makes the alpha release for Linux, Windows (32 and 64 bit for both the OS), and Mac;
|
|
|
* `make nightly-linux-x86_64`: makes a nightly build for x86_64 Linux.
|
|
|
|
|
|
## Testbuilds
|
... | ... | @@ -98,7 +103,7 @@ We have an additional configuration that we call _testbuild_. It can be based on |
|
|
The main difference with the base target is that desktop test builds are English-only (Tor Browser on desktop creates an installer for each language, which is time-consuming because of the compression).
|
|
|
|
|
|
The difference on Android is that while regular builds compile GeckoView for all architectures, testbuilds compile it only for the desired one.
|
|
|
This reduced the build time by half to 3 hours (or more) depending on the machine.
|
|
|
This reduces the build time by half to 3 hours (or more) depending on the machine.
|
|
|
|
|
|
# Running the build
|
|
|
|
... | ... | @@ -106,9 +111,9 @@ This reduced the build time by half to 3 hours (or more) depending on the machin |
|
|
|
|
|
**Targets are not guaranteed to work in every possible git tree**. You should `git checkout` the correct branch/tag before starting a build.
|
|
|
|
|
|
Nightlies should generally run everywhere, but we run them at the latest commit of the `master` branch.
|
|
|
We run nightlies at the latest commit of the `master` branch, but they should generally run everywhere.
|
|
|
|
|
|
Alphas are also based on `master`, but they are run on a specific annotated tag that we create before building an alpha for its release.
|
|
|
Alphas are also based on `master`, but they are run on a specific annotated tag (we create it just before an alpha release).
|
|
|
The tag name usually has this format: `tbb-VVVV-buildN`, where `VVVV` is the version and `N` is the build number.
|
|
|
If one version has multiple build numbers, some may not be buildable or have some runtime error or may not be reproducible; generally, you should refer to the highest one.
|
|
|
|
... | ... | @@ -131,82 +136,104 @@ The most likely to be changed settings are: |
|
|
|
|
|
## The actual build process
|
|
|
|
|
|
The makefile is very short, and it mainly contains calls to `rbm`, with the different targets set.
|
|
|
The makefile is very short, as it mainly contains calls to `rbm`, with the different targets set.
|
|
|
|
|
|
If you want to obtain a Tor Browser build, you can just need to call make with one of those targets.
|
|
|
If you want to build Tor Browser without any changes, just call `make` with one of those targets (see above for more details on the targets).
|
|
|
|
|
|
Please notice that, for reproducibility purposes, we compile many of the tools we use, including GCC, Clang, Rust, Go, and others. Therefore, the first build will take a really long time. The following ones will be faster.
|
|
|
|
|
|
You can also call `rbm` directly to build only one component, for example: `./rbm/rbm build nasm --target nightly --target torbrowser-android-x86_64`.
|
|
|
You can also call `rbm` directly to build only one component, for example:
|
|
|
|
|
|
```shell
|
|
|
./rbm/rbm build nasm --target nightly --target torbrowser-android-x86_64
|
|
|
```
|
|
|
|
|
|
It will also build any dependencies that are out of date.
|
|
|
It will also build any dependencies that have not been built yet, or that are out of date.
|
|
|
|
|
|
## Build failures and temporary files
|
|
|
### Build failures and temporary files
|
|
|
|
|
|
If the build succeeds, `rbm` exits with status code 0.
|
|
|
If the build succeeds, `rbm` and `make` exit with status code 0.
|
|
|
|
|
|
However, if it fails, the default option is to open an interactive shell in the container environment. Please see the section dedicated to it for further information on using it.
|
|
|
If it fails, the default option is to open an interactive shell in the container used to build the project that failed.
|
|
|
Please see the section dedicated to it for further information on using it.
|
|
|
|
|
|
You should close it with the `exit` command or Ctrl-D. In this way, the build process will terminate gracefully and delete any temporary files.
|
|
|
You should close it with the `exit` command or Ctrl-D.
|
|
|
In this way, the build process will terminate gracefully and delete any temporary files.
|
|
|
|
|
|
If you click Ctrl-C instead, you have a premature exit, and the container will be left there. You can remove the temporary files with this command: `./rbm/container run -- rm -rf tmp/rbm-*` (you may want to adjust the directory to remove, and/or invoke `./rbm/container` to obtain useful information on how to use that command).
|
|
|
It can also be used in other cases that lead to the same condition, e.g., a power loss.
|
|
|
Clicking Ctrl-C will cause an immediate exit, without cleaning the container and any other temporary files.
|
|
|
The same will happen in case of power loss or any other sudden interruption of the build.
|
|
|
|
|
|
# Output
|
|
|
Your user will not be able to delete these files, because they are associated to your subuids/subguids.
|
|
|
You will need to remove them from the container environment with this command:
|
|
|
|
|
|
The build process writes in several subdirectories of the directory where you cloned tor-browser-build:
|
|
|
```shell
|
|
|
./rbm/container run -- rm -rf tmp/rbm-*
|
|
|
```
|
|
|
|
|
|
* `release`, `alpha:` here, you can find the installers of Tor Browser stable and alpha, respectively. They are organized by version, each of which has a `signed` and an `unsigned` subdirectory: you will find the installers created by you on the latter;
|
|
|
* `nightly`: here, you can find the installers for nightly builds grouped by date. Nightly are never signed;
|
|
|
* `testbuild`: here, you can find the installers created by the testbuilds. Testbuilds are neither signed nor versioned, they are only organized by platform, so when you run a new one, it will replace the already existing one.
|
|
|
Also, you may want to adjust the directory to remove.
|
|
|
`./rbm/container` will give you useful information on how to use that command.
|
|
|
|
|
|
There, you will find the correct file to test your builds.
|
|
|
## Output
|
|
|
|
|
|
In addition to that, you will notice that some other subdirectories occupy a lot of space:
|
|
|
The build archives/installers will be in one of these directories of `tor-browser-build`:
|
|
|
|
|
|
* `git_clones`: as written above, `tor-browser-build` downloads the source code for all the components that it compiles, and, in the case of git repositories, it places them there;
|
|
|
* `out:` this directory is used mainly for artifacts (compilation output of a component), source code archives, precompiled dependencies downloaded from the Internet, and possibly other files.
|
|
|
* `release`, `alpha:` here, you can find the installers of Tor Browser stable and alpha, respectively. They are organized by version, each of which has a `signed` and an `unsigned` subdirectory: you will find the installers created by you on the latter;
|
|
|
* `nightly`: here, you can find the installers for nightly builds grouped by date. Nightly builds are never signed;
|
|
|
* `testbuild`: here, you can find the installers created by the testbuilds. Testbuilds are neither signed nor versioned, they are only organized by platform, so if you run one, it will replace any already existing one for the same platform/architecture combination.
|
|
|
|
|
|
In addition to that, you will notice that some other directories occupy a lot of space:
|
|
|
|
|
|
* `git_clones`: as written above, `tor-browser-build` downloads the source code for all the components that it compiles, and, in the case of git repositories, it places them there. RBM manages these directories automatically, and usually you should not need to do anything there;
|
|
|
* `out`: this directory is used for artifacts (compilation output of a component), source code archives, precompiled dependencies downloaded from the Internet, and possibly other files.
|
|
|
|
|
|
Finally, `logs` contain the logs file, organized by project. By default, new builds append to the exiting logs, but that can be changed.
|
|
|
Finally, `logs` contain the logs file, organized by project. By default, new builds append to the exiting logs, but this behavior can be customized.
|
|
|
|
|
|
## Verify it
|
|
|
### Verify it
|
|
|
|
|
|
In the same directory as the installers, you will find the installer hashes.
|
|
|
|
|
|
If you followed these instructions without changing anything, you should obtain the same hashes as our unsigned builds.
|
|
|
|
|
|
Otherwise, please consider contacting us, as we may treat your case as an issue.
|
|
|
We sign and upload our `sha256sums-unsigned-build.txt`, so you can also download them, and run `sha256sum -c sha256sums-unsigned-build.txt` (and possibly, verifying their signatures first).
|
|
|
|
|
|
Our signatures are detached, so if you used the same `make` target as us, you can also only download the signature, and it should match your `sha256sums-unsigned-build.txt`.
|
|
|
|
|
|
Otherwise, please check that you are using the correct hashes, and then consider contacting us, as we may treat your case as an issue.
|
|
|
|
|
|
## Cleaning the artifacts
|
|
|
|
|
|
Since we often switch git branches, `make clean` tries to remove old artifacts in a smart way.
|
|
|
But it needs to be configured to do so.
|
|
|
|
|
|
**TODO**: explain how to configure the clean phase.
|
|
|
|
|
|
# Testing patches
|
|
|
|
|
|
The easiest way to try some changes in tor-browser-build is by telling it to use custom sources.
|
|
|
Every project has a configuration file in `projects/$PROJECT/config`.
|
|
|
The easiest way to try some changes in `tor-browser-build` is by telling it to use custom sources.
|
|
|
|
|
|
Every component has a configuration file in `projects/$PROJECT_NAME/config`.
|
|
|
If it is a git-based project, you can customize its `git_url` and `git_hash`.
|
|
|
The former can either be a URL or a local directory, like `/home/you/code/project-name/.git`.
|
|
|
The former can either be a URL or a local directory, like `/home/you/code/project-name/.git`, the latter can be a commit hash, a tag, or a branch name.
|
|
|
|
|
|
So, the workflow to test new patches is:
|
|
|
The workflow to test new patches is:
|
|
|
|
|
|
1. clone the repository of the project to patch;
|
|
|
2. create a feature branch (optional but useful, especially if you want to contribute that patch back to us);
|
|
|
3. set the `git_hash` of the project configuration either to the commit hash or to the branch name (be advised: if fetching is disabled in `rbm.local.conf`, branch names may not work);
|
|
|
3. set the `git_hash` (be advised: if fetching is disabled in `rbm.local.conf`, branch names may not work, and messing with tags causes problems, too, which can be solved only by acting directly to `git_clones`);
|
|
|
4. run the build.
|
|
|
|
|
|
Sources can also be patched at build time by modifying the build scripts, e.g., for projects we do not fork.
|
|
|
Sources can also be patched at build time by injecting `.patch`/`.diff` files into the build container and modifying the build scripts to apply them.
|
|
|
This is useful, for example, for projects we do not fork.
|
|
|
|
|
|
## Using the debug shell
|
|
|
|
|
|
### What and why
|
|
|
### What, why and caveats
|
|
|
|
|
|
When a build fails, you are presented with a debug shell inside the building environment (unless you disabled it in `rbm.local.conf`).
|
|
|
|
|
|
You can use it to try to fix the problems.
|
|
|
However, there is no way to resume the build process, even if you manage to produce working artifacts in this environment.
|
|
|
Nonetheless, trying the fix there is a good idea because you may find other problems, and it can save a lot of time if the solution does not work as expected.
|
|
|
|
|
|
You can first try to understand where the build stopped by looking at the project's build log.
|
|
|
Nonetheless, trying your fixes there is a good idea because you may find other problems, and it can save a lot of time if the solution does not work as expected.
|
|
|
|
|
|
### Before doing anything
|
|
|
|
... | ... | @@ -226,13 +253,12 @@ In bullseye-based containers (and following versions), the terminal type influen |
|
|
|
|
|
### What to do there
|
|
|
|
|
|
The shell is launched in the initial directory of the container.
|
|
|
|
|
|
The automatic procedure launches the `build` script it finds there.
|
|
|
The automatic procedure runs a `build` script that is generated from `projects/$PROJECT_NAME/build`.
|
|
|
If this script exits with a status different from 0, the build is considered failed, and the debug shell is opened at the directory containing the script.
|
|
|
|
|
|
So, the usual debug process consists in:
|
|
|
|
|
|
1. analyze the build script and understand which command failed;
|
|
|
1. analyze the project's build log and the `build` script to understand which command failed;
|
|
|
2. do something to prevent that command from failing again (e.g., change the source code);
|
|
|
3. run the commands coming before the failing one on the build script selectively:
|
|
|
- define all the environment variables like the script;
|
... | ... | @@ -242,7 +268,9 @@ So, the usual debug process consists in: |
|
|
|
|
|
Try to keep track of any changes you make on the debug environment because they will be deleted once the shell is closed.
|
|
|
|
|
|
Also, `build` is the default name, but it can be customized on the project configuration, or some complex projects (such as GeckoView) need in multiple build phases, and each one has a different script name.
|
|
|
|
|
|
# Developer builds
|
|
|
|
|
|
`tor-browser-build` cannot produce incremental builds, so you may check out [this page](https://gitlab.torproject.org/tpo/applications/tor-browser/-/wikis/Hacking) for new developments.
|
|
|
`tor-browser-build` cannot produce incremental builds, so you may check out [this page](https://gitlab.torproject.org/tpo/applications/tor-browser/-/wikis/Hacking) for new developments that need frequent, incremental and quick builds.
|
|
|
Be careful to **use them only for testing**. |
|
|
\ No newline at end of file |