Draft: Tor and Mullvad Browser for linux-arm64 (native and cross builds)

Merge Info

Related Issues

• #12631
• #32355

Backporting

Timeline

• Immediate: patchset needed as soon as possible
• Next Minor Stable Release: patchset that needs to be verified in nightly before backport
• Eventually: patchset that needs to be verified in alpha before backport
• No Backport (preferred): patchset for the next major stable

(Optional) Justification

• Emergency security update: patchset fixes CVEs, 0-days, etc
• Censorship event: patchset enables censorship circumvention
• Critical bug-fix: patchset fixes a bug in core-functionality
• Consistency: patchset which would make development easier if it were in both the alpha and release branches; developer tools, build system changes, etc
• Sponsor required: patchset required for sponsor
• Other: new feature

Issue Tracking

• Link resolved issues with appropriate Release Prep issue for changelog generation

Review

Request Reviewer

• Request review from an applications developer depending on modified system:
  • NOTE: if the MR modifies multiple areas, please /cc all the relevant reviewers (since gitlab only allows 1 reviewer)
  • accessibility : henry
  • android : clairehurst, dan
  • build system : boklm
  • extensions : ma1
  • firefox internals (XUL/JS/XPCOM) : ma1
  • fonts : pierov
  • frontend (implementation) : henry
  • frontend (review) : donuts, richard
  • localization : henry, pierov
  • macos : clairehurst, dan
  • nightly builds : boklm
  • rebases/release-prep : boklm, dan, ma1, pierov, richard
  • security : ma1
  • signing : boklm, richard
  • updater : pierov
  • misc/other : pierov, richard

Change Description

This MR adds linux-arm64 support to the Tor and Mullvad Browsers. The {tor,mullvad,base}browser-$CHANNEL-linux-aarch64 targets support both x86_64 cross-compilation and arm64 native builds. Cross-compilation is the default. Native compilation is enabled by adding a linux-cross: 0 entry to the var key in rbm.local.conf, as in

var:
  local_conf: 1
  linux-cross: 0

(rbm.local.conf itself can of course be copied from rbm.local.conf.example). Both the browsers currently build on both platforms and run on Fedora Asahi Remix 39 (Fedora on Apple Silicon). I didn't test the basebrowser, but since the other two work, that one should too. The Tor Browser is able to connect to onion sites, and to connect via bridges too.

To set up cross-compilation, I took control of an apparently unused build variable: var/linux-cross. This is set to 0 in all but the {tor,base}browser-arm targets, which in turn are never built. Of course, if I am mistaken and that variable was actually used, the code can be refactored to a separate path. When possible I recycled the old var/linux-cross == 1 content.

Toolchain

nasm is not built as it is not needed for arm64.

When cross-compiling:

• two sets of binutils + gcc are built, native x86_64 and cross arm64
• clang and rust are built only once with native x86_64 and cross arm64 support
• python, cbindgen and node are built only for x86_64
• go can automatically compile for both x86_64 and arm64, so there was nothing to do here (except for !945 (closed))

When compiling natively:

• bootstrapping go from C code (as you know) requires building Go 1.4, which did not support arm64 as the host architecture. Thanks to a patch by Heikki Lindholm, we can however compile an arm (as in "32-bit arm") go1.4, which in turn can compile an arm64 go1.19. As it stands, therefore, the Tor Browser can be natively compiled only on arm64 platforms which support running 32-bit arm instructions, which excludes e.g. Apple Silicon. To natively build the Tor Browser on Asahi Linux, have a look at the instructions here and use my linux-arm64-asahi branch. (Fixed. Fully native compilation is now supported and this MR builds on Asahi Linux too)

Things that need to be worked out

• I was unable to template the target replacements in go projects, so I had to remove those replacements. This breaks non-linux builds, so it must be fixed as a priority. (Fixed)
• When and only when cross-compiling arm64 from x86_64, most of the binaries in the tor-expert-bundle are dynamically linked to libssp.so for stack-protection. My system does not have that library, so I had to include it in the bundle to be loaded via LD_LIBRARY_PATH to make the Tor Browser work. I suspect the reason why libssp.so is pulled in is I'm compiling the gcc cross-compiler without hardening (following a hint I found in projects/gcc/build itself; native builds, on the other hand, have hardening enabled when compiling gcc). One could try building the cross compiler with hardening enabled to see what happens. (I was wrong, hardening the compiler itself had nothing to do with this. The issue was caused by the fact that the cross gcc didn't know the target glibc version at configure time, so it couldn't assume that libssp.so is not needed and linked against it when hardening the binaries in tor-expert-bundle. This is fixed now.)
• I was unable to use clang as the linker for the aarch64-linux-gnu target while compiling rust from x86_64. When I selected clang, it tried to pull in the native x86_64 GNU ld when linking arm64 binaries, so it understandably failed. To solve this, I had to explicitly select aarch64-linux-gnu-gcc as the linker for the aarch64-linux-gnu target. This is not necessarily a bad thing, but I thought I should mention it anyway.
• Most of the patches in projects/firefox should really go into the respective Tor or Mullvad Browser repos. (Done)
• Choosing whether to compile natively or to cross-compile should be more user-friendly, e.g. it should be decided via an option in rbm.local.conf. I was unable to template an integer to put in rbm.conf's targets/linux-aarch64/var/linux-cross. (It was in fact already possible to do so, so I updated the instructions above)

Credits

While I rewrote most of the arm64 enablement from scratch, a huge thanks goes to Heikki Lindholm both for years of maintenance and builds of the Tor Browser for arm64, and for the content of some of the patches that went either into this MR or in its early versions - most notably the go-bootstrap, browser and parts of the firefox patches. Also, I had good knowledge of his patchset, so in many cases I ended up doing what he is doing (e.g. the target replacement suppression, not in use anymore).

How Tested

Tor and Mullvad Browser, cross and native, nightly and alpha builds tested, the browsers run on Debian 9 (Stretch) and Fedora Asahi Remix 39 with no apparent issues.

mentioned in issue #12631

assigned to @NoisyCoil

changed the description

added 1 commit

• 5cb85b11 - Enable cross-gcc hardening on linux

Compare with previous version

changed the description

changed the description

Apparently my last-minute commit enabling hardening for cross-gcc broke gcc. I brought everything back to this morning's version. Sorry for the confusion.

You will find yesterday's alpha release compiled for arm64 on x86_64 here.

I checked, and the cross build is reproducible. To be specific, if you compile Tor Browser v13.5a5 on the same x86_64 machine twice the sha256sums match.

For the record, if you were to then re-compile it on arm64, the shasums wouldn't match because of the different compilers used for building the browser. This already shows in the fact that the native compiler does not dynamically link against libssp.so, as I said, which in particular requires including that dependency in the bundle when cross-compiling.

added 7 commits

• dea3408f - Revert "Add arm64 support to projects/nasm"
• a99f6932 - Do not build nasm for the linux-arm64 target
• 34179f63 - Let gcc know the min glibc version when cross-compiling for linux-arm64
• 922b44fa - Revert "Add libssp.so to tor-expert-bundle when cross-compiling linux-arm64"
• 029f563c - Add 'build' configuration flag to tor when cross-compiling linux
• 7ac907cb - Disable gcc multilib support when building for linux-arm64
• 76d519d4 - Clean up cross-configure flags in projects/gcc

Compare with previous version

changed the description

The most recent commits fix the libssp.so issue and prevent nasm from building when the target is linux-arm64, as it is not needed.

The test builds in here have been refreshed and now include the Mullvad Browser. They seem to fully work on Debian 10 (Buster) too. I also checked that the native builds are reproducible, even across different arm64 machines.

This seems to be necessary only for native aarch64 builds.

I think it's out of scope for this MR.

As you wrote, let's avoid patches here, especially for mozconfigs.

Why is the syscall numbers patch needed?

I could cross-compile Firefox without this patch and run it.

Such a patch would require us constant maintenance.

We can't avoid them unless we merge them in Tor Browser and Mullvad Browser first. I could open MRs there, but I can't remove them from here ATM. The intention was to remove them later on.

The reason why you could compile (cross or native is irrelevant) and run Firefox without that patch is either you were compiling it on an OS later than Buster, or you were running it on an OS whose glibc is not the latest versions. The number of syscalls got larger since then on the arm64 side, and that patch is needed so that Firefox's sandbox does not make the browser crash on internal clone3 syscalls, which are only made by more recent OSes (like Fedora 39). For context, the number of syscalls is imported from the system's headers, and Buster still has the old one (and so does Stretch then of course).

This thread will be resolved by tor-browser!967 (merged), mullvad-browser!113 (merged), tor-browser!971 (merged) and mullvad-browser!114 (closed). The syscall patch needs to stay in tor-browser-build because it pertains the build system: it should not be needed when building linux-aarch64 on Debian Buster with backports, Debian Bullseye, or later (or when running the browser on the same system on which it is built).

In more detail, commit 8f6ccf6 introduced the clone3 syscall in mainline linux kernel, v5.3. clone3 is used internally on modern linux OSes (e.g., on arm64, it is used on systems which have glibc v2.38 or higher installed, such as Fedora 39, on which the issue first showed up after an upgrade from Fedora 38 and glibc v2.37) and its calling is detected by Firefox's sandbox. The sandbox blocks syscalls whose number is larger than MAX_PUBLIC_SYSCALL. On arm64, the latter is set equal to asm-generic/unistd.h's __NR_syscall at build time, which in turn is provided by the linux-libc-dev package. Debian Stretch's kernel packages are v4.9 (no backports) or 4.19 (with backports). On the most recent v4.19 __NR_syscalls equals 294, which is less than clone3's 435. Thus, when Firefox is built on Stretch and run on a modern arm64 system, the sandbox blocks the clone3 syscalls, which in turn makes the browser crash at startup.

Buster with backports and Bullseye are on linux v5.10, so they already include a large enough __NR_syscalls. Building on those should avoid the issue, but it may also break compatibility of the browser with older systems, so it's not a viable solution.

The clone3 issue does not affect linux-x86_64 nor linux-i686 since for those targets MAX_PUBLIC_SYSCALL is set to 1024u rather than being imported from the system's headers. In fact, the patch I originally sent to Heikki Lindholm had MAX_PUBLIC_SYSCALL set to 1024u like on x86. He went with 457u (the number still found in fix-aarch64-syscall-number.patch) instead because that was the mainline kernel's __NR_syscall at the time he merged the patch into his arm64 branch. That's since grown to 462.

@pierov since the MRs have been merged, I leave it to you to resolve this thread if you think the syscall patch can stay here.

Thanks for this MR! I've gone very quickly through it, but eventually I think someone else will be the reviewer.

My preference would be for not dealing both with cross-compiling and native compiling in the same MR. It makes the MR even bigger and we can't test it.

Moreover, I think that before switching to native compiling, we should add a way to have a target for host tools. It might make it easier to deal with stuff such as Go dependency tarballs.

Also, I'd like to see the changes more grouped. We usually start our commits with Bug 12345: ....

Multiple commits fine (they could all start with Bug 32355: ..., or with another issue, since #32355 is specific to armv7), but maybe 30 are a bit too much, you could group more of them together (e.g., one for getting the basics, such as rbm.conf, GCC, binutils and so on, another one for Firefox, which might include also changes to Rust, another one for Tor and PTs, etc...).

Anyway, I think we could deal with that only when closer to merge.

added 1 commit

• c896397b - Improve formatting in projects/firefox/build

Compare with previous version

added 1 commit

• d923d89e - Use template tags for SSE2 checks in projects/browser/RelativeLink/start-browser

Compare with previous version

Thank you for working on this.

added 3 commits

• dc9b6a25 - Restore target replacements in projects/{conjure,go,lyrebird,snowflake,webtunnel}
• 077a00b7 - Downgrade glibc to v2.24 when cross-compiling linux-arm64
• 0eb3a901 - Downgrade linux to v4.19.232 when cross-compiling linux-arm64

Compare with previous version

The last commits:

• add support for Debian 9 (Stretch) and, more generally, to distros whith glibc v2.24 or higher (tested only on Debian Stretch itself)
• solve the issue with target replacements in go projects, which in particular make the non-linux targets buildable again (tested with the torbrowser-alpha-macos and torbrowser-alpha-android-aarch64 targets)

changed the description

added 46 commits

• 0eb3a901...14d5ad27 - 11 commits from branch tpo/applications:main
• 14d5ad27...495078ea - 25 earlier commits
• 92685cd0 - Let gcc know the min glibc version when cross-compiling for linux-arm64
• 9093198b - Revert "Add libssp.so to tor-expert-bundle when cross-compiling linux-arm64"
• 1cc04938 - Add 'build' configuration flag to tor when cross-compiling linux
• 1dd5ff6d - Disable gcc multilib support when building for linux-arm64
• 683212a9 - Clean up cross-configure flags in projects/gcc
• 05f454a8 - Improve formatting in projects/firefox/build
• db42090e - Use template tags for SSE2 checks in projects/browser/RelativeLink/start-browser
• ffd9c00b - Restore target replacements in projects/{conjure,go,lyrebird,snowflake,webtunnel}
• 25c34077 - Downgrade glibc to v2.24 when cross-compiling linux-arm64
• 99a59dd0 - Downgrade linux to v4.19.232 when cross-compiling linux-arm64

Compare with previous version

added 39 commits

• 99a59dd0...b6e589f0 - 4 commits from branch tpo/applications:main
• b6e589f0...f04c169e - 25 earlier commits
• 756da7ee - Let gcc know the min glibc version when cross-compiling for linux-arm64
• ec47b4a9 - Revert "Add libssp.so to tor-expert-bundle when cross-compiling linux-arm64"
• bfc6d854 - Add 'build' configuration flag to tor when cross-compiling linux
• 21f8aa13 - Disable gcc multilib support when building for linux-arm64
• ce24c5b3 - Clean up cross-configure flags in projects/gcc
• f89e99a9 - Improve formatting in projects/firefox/build
• 46200e2d - Use template tags for SSE2 checks in projects/browser/RelativeLink/start-browser
• 2637fad7 - Restore target replacements in projects/{conjure,go,lyrebird,snowflake,webtunnel}
• 629a1069 - Downgrade glibc to v2.24 when cross-compiling linux-arm64
• 40d37feb - Downgrade linux to v4.19.232 when cross-compiling linux-arm64

Compare with previous version

added 36 commits

• 5a54c1cc - 1 commit from branch tpo/applications:main
• 5a54c1cc...656426cd - 25 earlier commits
• 34bd6385 - Let gcc know the min glibc version when cross-compiling for linux-arm64
• e6a82205 - Revert "Add libssp.so to tor-expert-bundle when cross-compiling linux-arm64"
• 33d33f21 - Add 'build' configuration flag to tor when cross-compiling linux
• 1f2fd772 - Disable gcc multilib support when building for linux-arm64
• 6e5c825b - Clean up cross-configure flags in projects/gcc
• e1cfb763 - Improve formatting in projects/firefox/build
• 61dfbda4 - Use template tags for SSE2 checks in projects/browser/RelativeLink/start-browser
• 3b5a526d - Restore target replacements in projects/{conjure,go,lyrebird,snowflake,webtunnel}
• 4c075616 - Downgrade glibc to v2.24 when cross-compiling linux-arm64
• 880321b9 - Downgrade linux to v4.19.232 when cross-compiling linux-arm64

Compare with previous version

added 36 commits

• 9e7707b6 - 1 commit from branch tpo/applications:main
• 9e7707b6...8d2c1cf5 - 25 earlier commits
• df148958 - Let gcc know the min glibc version when cross-compiling for linux-arm64
• 1ef57eb1 - Revert "Add libssp.so to tor-expert-bundle when cross-compiling linux-arm64"
• ce7ff12e - Add 'build' configuration flag to tor when cross-compiling linux
• d0f44abc - Disable gcc multilib support when building for linux-arm64
• 51bdd8ab - Clean up cross-configure flags in projects/gcc
• 31e80d44 - Improve formatting in projects/firefox/build
• 20b477fb - Use template tags for SSE2 checks in projects/browser/RelativeLink/start-browser
• be4b2cc7 - Restore target replacements in projects/{conjure,go,lyrebird,snowflake,webtunnel}
• d17c47f7 - Downgrade glibc to v2.24 when cross-compiling linux-arm64
• 92d385b9 - Downgrade linux to v4.19.232 when cross-compiling linux-arm64

Compare with previous version

/cc @richard

assigned to @boklm and unassigned @NoisyCoil

unassigned @boklm

requested review from @boklm

assigned to @NoisyCoil

added 37 commits

• 92d385b9...debb3452 - 2 commits from branch tpo/applications:main
• debb3452...bed86e3e - 25 earlier commits
• a60350a5 - Let gcc know the min glibc version when cross-compiling for linux-arm64
• 65a04db0 - Revert "Add libssp.so to tor-expert-bundle when cross-compiling linux-arm64"
• c4c9f82e - Add 'build' configuration flag to tor when cross-compiling linux
• be9f17f6 - Disable gcc multilib support when building for linux-arm64
• 23f5bf10 - Clean up cross-configure flags in projects/gcc
• 6ee080f1 - Improve formatting in projects/firefox/build
• 5c2e9e6c - Use template tags for SSE2 checks in projects/browser/RelativeLink/start-browser
• 3b12ae6a - Restore target replacements in projects/{conjure,go,lyrebird,snowflake,webtunnel}
• 7d34d8e3 - Downgrade glibc to v2.24 when cross-compiling linux-arm64
• db6577d4 - Downgrade linux to v4.19.232 when cross-compiling linux-arm64

Compare with previous version

mentioned in merge request tor-browser!967 (merged)

mentioned in merge request mullvad-browser!113 (merged)

added 18 commits

• db6577d4...2234116b - 8 earlier commits
• cc2aabf4 - Add arm64 support to projects/cbindgen
• 76357b19 - Add arm64 support to projects/firefox
• 2b1202c8 - Add arm64 support to projects/libevent
• fab62337 - Add arm64 support to projects/openssl
• 6e1e6119 - Add arm64 support to projects/tor
• 06a1ef9a - Add arm64 support to projects/go-bootstrap
• ca79f7f3 - Add arm64 support to projects/go
• 03987a22 - Add arm64 support to projects/{conjure,lyrebird,snowflake,webtunnel}
• 078eaab4 - Add arm64 support to projects/manual
• 837790e6 - Use template tags for SSE2 checks in projects/browser/RelativeLink/start-browser

Compare with previous version

The last changes

• rename the target from linux-arm64 to linux-aarch64
• add linux-aarch64 to the browser-all and browser-all-desktop targets in projects/release/config
• remove an unnecessary clang patch
• squash related commits for better code readability

Builds tested on Ubuntu 23.10 (cross, x86_64) and Debian 12 (native, arm64), browsers tested on Fedora 39 and Debian 9.

changed the description

added 1 commit

• 28122a72 - fixup! Add arm64 support to projects/tor

Compare with previous version

added 2 commits

• a9add2e1 - fixup! Add arm64 support to projects/manual
• 2e581f63 - fixup! Add arm64 support to projects/firefox

Compare with previous version

The last changes remove the mozconfig patches from projects/firefox, since they are now available in the main browser repos. The *-alpha-linux-aarch64 targets will fail to build until I rebase the patchset onto the next alpha release (nightlies work though).

changed the description

changed the description

added 23 commits

• 872cff7c - 1 commit from branch tpo/applications:main
• 872cff7c...4838f5ed - 12 earlier commits
• a9d6ddb4 - Add arm64 support to projects/tor
• 2f2ac510 - Add arm64 support to projects/go-bootstrap
• 2dc60287 - Add arm64 support to projects/go
• 4d43b985 - Add arm64 support to projects/{conjure,lyrebird,snowflake,webtunnel}
• 9c52f4ef - Add arm64 support to projects/manual
• 8b3f3fa9 - Use template tags for SSE2 checks in projects/browser/RelativeLink/start-browser
• 1cc36142 - fixup! Add arm64 support to projects/tor
• 78b38b03 - fixup! Add arm64 support to projects/manual
• a3af489b - fixup! Add arm64 support to projects/firefox
• 8a83aad3 - fixup! Add arm64 support to projects/go

Compare with previous version

I added !945 (closed) to this patchset. Depending on its fate in main I will change it (e.g. by adding linux-aarch64 guards around not using go1.4) or remove it.