tor-browser-build merge requestshttps://gitlab.torproject.org/tpo/applications/tor-browser-build/-/merge_requests2024-03-28T17:43:25Zhttps://gitlab.torproject.org/tpo/applications/tor-browser-build/-/merge_requests/944geckoview tools in tor-browser moved into their own directory (fat-aar)2024-03-28T17:43:25ZDan Ballardgeckoview tools in tor-browser moved into their own directory (fat-aar)## Merge Info
<!-- Bookkeeping information for release management -->
### Related Issues
### Backporting
#### Timeline
- [ ] **Immediate**: patchset needed as soon as possible
- [ ] **Next Minor Stable Release**: patchset that needs...## Merge Info
<!-- Bookkeeping information for release management -->
### Related Issues
### Backporting
#### Timeline
- [ ] **Immediate**: patchset needed as soon as possible
- [ ] **Next Minor Stable Release**: patchset that needs to be verified in nightly before backport
- [ ] **Eventually**: patchset that needs to be verified in alpha before backport
- [x] **No Backport (preferred)**: patchset for the next major stable
#### (Optional) Justification
- [ ] **Emergency security update**: patchset fixes CVEs, 0-days, etc
- [ ] **Censorship event**: patchset enables censorship circumvention
- [ ] **Critical bug-fix**: patchset fixes a bug in core-functionality
- [ ] **Consistency**: patchset which would make development easier if it were in both the alpha and release branches; developer tools, build system changes, etc
- [ ] **Sponsor required**: patchset required for sponsor
- [ ] **Other**: please explain
### Issue Tracking
- [ ] Link resolved issues with appropriate [Release Prep issue](https://gitlab.torproject.org/groups/tpo/applications/-/issues/?sort=updated_desc&state=opened&label_name%5B%5D=Release%20Prep&first_page_size=20) for changelog generation
### Review
#### Request Reviewer
- [ ] Request review from an applications developer depending on modified system:
- **NOTE**: if the MR modifies multiple areas, please `/cc` all the relevant reviewers (since gitlab only allows 1 reviewer)
- **accessibility** : henry
- **android** : clairehurst, dan
- **build system** : boklm
- **extensions** : ma1
- **firefox internals (XUL/JS/XPCOM)** : ma1
- **fonts** : pierov
- **frontend (implementation)** : henry
- **frontend (review)** : donuts, richard
- **localization** : henry, pierov
- **macos** : clairehurst, dan
- **nightly builds** : boklm
- **rebases/release-prep** : boklm, dan, ma1, pierov, richard
- **security** : ma1
- **signing** : boklm, richard
- **updater** : pierov
- **misc/other** : pierov, richard
### Change Description
<!-- Whatever context the reviewer needs to effectively review the patchset; if the patch includes UX updates be sure to include screenshots/video of how any new behaviour -->
#### How Tested
tor-browser!951 changes where the fat-aar tool is so this change is needed to continue using it to build geckoviewDan BallardDan Ballardhttps://gitlab.torproject.org/tpo/applications/tor-browser-build/-/merge_requests/920Draft: Tor and Mullvad Browser for linux-arm64 (native and cross builds)2024-03-28T13:16:11ZNoisyCoilDraft: Tor and Mullvad Browser for linux-arm64 (native and cross builds)## Merge Info
<!-- Bookkeeping information for release management -->
### Related Issues
- tor-browser-build#12631
- tor-browser-build#32355
### Backporting
#### Timeline
- [ ] **Immediate**: patchset needed as soon as possible
- [ ]...## Merge Info
<!-- Bookkeeping information for release management -->
### Related Issues
- tor-browser-build#12631
- tor-browser-build#32355
### Backporting
#### Timeline
- [ ] **Immediate**: patchset needed as soon as possible
- [ ] **Next Minor Stable Release**: patchset that needs to be verified in nightly before backport
- [ ] **Eventually**: patchset that needs to be verified in alpha before backport
- [x] **No Backport (preferred)**: patchset for the next major stable
#### (Optional) Justification
- [ ] **Emergency security update**: patchset fixes CVEs, 0-days, etc
- [ ] **Censorship event**: patchset enables censorship circumvention
- [ ] **Critical bug-fix**: patchset fixes a bug in core-functionality
- [ ] **Consistency**: patchset which would make development easier if it were in both the alpha and release branches; developer tools, build system changes, etc
- [ ] **Sponsor required**: patchset required for sponsor
- [x] **Other**: new feature
### Issue Tracking
- [ ] Link resolved issues with appropriate [Release Prep issue](https://gitlab.torproject.org/groups/tpo/applications/-/issues/?sort=updated_desc&state=opened&label_name%5B%5D=Release%20Prep&first_page_size=20) for changelog generation
### Review
#### Request Reviewer
- [ ] Request review from an applications developer depending on modified system:
- **NOTE**: if the MR modifies multiple areas, please `/cc` all the relevant reviewers (since gitlab only allows 1 reviewer)
- **accessibility** : henry
- **android** : clairehurst, dan
- **build system** : boklm
- **extensions** : ma1
- **firefox internals (XUL/JS/XPCOM)** : ma1
- **fonts** : pierov
- **frontend (implementation)** : henry
- **frontend (review)** : donuts, richard
- **localization** : henry, pierov
- **macos** : clairehurst, dan
- **nightly builds** : boklm
- **rebases/release-prep** : boklm, dan, ma1, pierov, richard
- **security** : ma1
- **signing** : boklm, richard
- **updater** : pierov
- **misc/other** : pierov, richard
### Change Description
This MR adds linux-arm64 support to the Tor and Mullvad Browsers. The `{tor,mullvad,base}browser-$CHANNEL-linux-arm64` targets support both x86_64 cross-compilation and arm64 native builds (with a caveat, more on this ahead). Cross-compilation is the default. Native compilation is enabled by setting `targets/linux-arm64/var/linux-cross` to `0` in `rbm.conf`. Both the browsers currently build on both platforms and run on Fedora Asahi Remix 39 (Fedora on Apple Silicon). I didn't test the `basebrowser`, but since the other two work, that one should too. The Tor Browser is able to connect to onion sites, and to connect via bridges too.
To set up cross-compilation, I took control of an apparently unused build variable: `var/linux-cross`. This is set to `0` in all but the `{tor,base}browser-arm` targets, which in turn are never built. Of course, if I am mistaken and that variable was actually used, the code can be refactored to a separate path. When possible I recycled the old `var/linux-cross == 1` content.
#### Toolchain
`nasm` is not built as it is not needed for arm64.
When cross-compiling:
- two sets of `binutils` + `gcc` are built, native x86_64 and cross arm64
- `clang` and `rust` are built only once with native x86_64 and cross arm64 support
- `python`, `cbindgen` and `node` are built only for x86_64
- `go` can automatically compile for both x86_64 and arm64, so there was nothing to do here
When compiling natively:
- bootstrapping `go` from C code (as you know) requires building Go 1.4, which did not support arm64 as the host architecture. Thanks to a patch by Heikki Lindholm, we can however compile an arm (as in "32-bit arm") `go1.4`, which in turn can compile an arm64 `go1.19`. As it stands, therefore, the Tor Browser can be natively compiled only on arm64 platforms which support running 32-bit arm instructions, which excludes e.g. Apple Silicon. To natively build the Tor Browser on Asahi Linux, have a look at the instructions [here](https://gitlab.com/NoisyCoil/tor-browser-build/-/wikis/Building-on-Asahi-Linux) and use my [linux-arm64-asahi](https://gitlab.torproject.org/NoisyCoil/tor-browser-build/-/tree/linux-arm64-asahi?ref_type=heads) branch.
#### Things that need to be worked out
- ~I was unable to template the target replacements in `go` projects, so I had to remove those replacements. This breaks non-linux builds, so it must be fixed as a priority.~ (Fixed)
- ~When and only when cross-compiling arm64 from x86_64, most of the binaries in the `tor-expert-bundle` are dynamically linked to `libssp.so` for stack-protection. My system does not have that library, so I had to include it in the bundle to be loaded via `LD_LIBRARY_PATH` to make the Tor Browser work. I suspect the reason why `libssp.so` is pulled in is I'm compiling the `gcc` cross-compiler without hardening (following a hint I found in `projects/gcc/build` itself; native builds, on the other hand, have hardening enabled when compiling `gcc`). One could try building the cross compiler with hardening enabled to see what happens.~ (I was wrong, hardening the compiler itself had nothing to do with this. The issue was caused by the fact that the cross `gcc` didn't know the target `glibc` version at configure time, so it couldn't assume that `libssp.so` is not needed and linked against it when hardening the binaries in `tor-expert-bundle`. This is fixed now.)
- I was unable to use `clang` as the linker for the `aarch64-linux-gnu` target while compiling `rust` from x86_64. When I selected clang, it tried to pull in the native x86_64 GNU `ld` when linking arm64 binaries, so it understandably failed. To solve this, I had to explicitly select `aarch64-linux-gnu-gcc` as the linker for the `aarch64-linux-gnu` target. This is not necessarily a bad thing, but I thought I should mention it anyway.
- Most of the patches in `projects/firefox` should really go into the respective Tor or Mullvad Browser repos.
- Choosing whether to compile natively or to cross-compile should be more user-friendly, e.g. it should be decided via an option in `rbm.local.conf`. I was unable to template an integer to put in `rbm.conf`'s `targets/linux-arm64/var/linux-cross`.
#### Credits
While I rewrote most of the arm64 enablement from scratch, a huge thanks goes to Heikki Lindholm both for years of maintenance and builds of the Tor Browser for arm64, and for the content of some of the patches that went into this MR - most notably the `go-bootstrap`, `browser` and parts of the `firefox` patches. Also, I had good knowledge of his patchset, so in many cases I ended up doing what he is doing (e.g. the target replacement suppression).
#### How Tested
Tor and Mullvad Browser, cross and native, nightly and alpha builds tested, the browsers run on Debian 9 (Stretch) and Fedora Asahi Remix 39 with no apparent issues.NoisyCoilNoisyCoilhttps://gitlab.torproject.org/tpo/applications/tor-browser-build/-/merge_requests/943Draft: Bug 41083: Make a deb package for Mullvad Browser2024-03-28T10:54:57ZboklmDraft: Bug 41083: Make a deb package for Mullvad Browser## Merge Info
<!-- Bookkeeping information for release management -->
### Related Issues
- tor-browser-build#41083
- tor-browser-build#41112
### Backporting
#### Timeline
- [ ] **Immediate**: patchset needed as soon as possible
- [ ]...## Merge Info
<!-- Bookkeeping information for release management -->
### Related Issues
- tor-browser-build#41083
- tor-browser-build#41112
### Backporting
#### Timeline
- [ ] **Immediate**: patchset needed as soon as possible
- [ ] **Next Minor Stable Release**: patchset that needs to be verified in nightly before backport
- [ ] **Eventually**: patchset that needs to be verified in alpha before backport
- [x] **No Backport (preferred)**: patchset for the next major stable
#### (Optional) Justification
- [ ] **Emergency security update**: patchset fixes CVEs, 0-days, etc
- [ ] **Censorship event**: patchset enables censorship circumvention
- [ ] **Critical bug-fix**: patchset fixes a bug in core-functionality
- [ ] **Consistency**: patchset which would make development easier if it were in both the alpha and release branches; developer tools, build system changes, etc
- [ ] **Sponsor required**: patchset required for sponsor
- [ ] **Other**: please explain
### Issue Tracking
- [ ] Link resolved issues with appropriate [Release Prep issue](https://gitlab.torproject.org/groups/tpo/applications/-/issues/?sort=updated_desc&state=opened&label_name%5B%5D=Release%20Prep&first_page_size=20) for changelog generation
### Review
#### Request Reviewer
- [x] Request review from an applications developer depending on modified system:
- **NOTE**: if the MR modifies multiple areas, please `/cc` all the relevant reviewers (since gitlab only allows 1 reviewer)
- **accessibility** : henry
- **android** : clairehurst, dan
- **build system** : boklm
- **extensions** : ma1
- **firefox internals (XUL/JS/XPCOM)** : ma1
- **fonts** : pierov
- **frontend (implementation)** : henry
- **frontend (review)** : donuts, richard
- **localization** : henry, pierov
- **macos** : clairehurst, dan
- **nightly builds** : boklm
- **rebases/release-prep** : boklm, dan, ma1, pierov, richard
- **security** : ma1
- **signing** : boklm, richard
- **updater** : pierov
- **misc/other** : pierov, richard
### Change Description
This create a `mullvad-browser`, `mullvad-browser-alpha` or `mullvad-browser-nightly` package (depending on channel we build). The package create a link to the start script in `/usr/bin` with the same name as the package.
I changed the start script to set `HOME` to `$HOME/.$package_name/home` (for system installs).
#### How Tested
I tested alpha and nightly builds. Installing the package on Debian bullseye worked, and running `mullvad-browser-alpha` or `mullvad-browser-nightly` started the browser correctly.
I also checked that the tar.xz bundle is still working after the changes in the start script.boklmboklmhttps://gitlab.torproject.org/tpo/applications/tor-browser-build/-/merge_requests/845Draft: Bug 40997: Add a reproducible-build.txt file in release directory2023-11-28T09:43:19ZboklmDraft: Bug 40997: Add a reproducible-build.txt file in release directory## Merge Info
<!-- Bookkeeping information for release management -->
### Related Issues
- tor-browser-build#40997
- tpo/community/hackweek#29
### Backporting
#### Timeline
- [x] **Immediate**: patchset needed as soon as possible
- [...## Merge Info
<!-- Bookkeeping information for release management -->
### Related Issues
- tor-browser-build#40997
- tpo/community/hackweek#29
### Backporting
#### Timeline
- [x] **Immediate**: patchset needed as soon as possible
- [ ] **Next Minor Stable Release**: patchset that needs to be verified in nightly before backport
- [ ] **Eventually**: patchset that needs to be verified in alpha before backport
- [ ] **No Backport (preferred)**: patchset for the next major stable
#### (Optional) Justification
- [ ] **Emergency security update**: patchset fixes CVEs, 0-days, etc
- [ ] **Censorship event**: patchset enables censorship circumvention
- [ ] **Critical bug-fix**: patchset fixes a bug in core-functionality
- [x] **Consistency**: patchset which would make development easier if it were in both the alpha and release branches; developer tools, build system changes, etc
- [ ] **Sponsor required**: patchset required for sponsor
- [ ] **Other**: please explain
### Issue Tracking
- [ ] Link resolved issues with appropriate [Release Prep issue](https://gitlab.torproject.org/groups/tpo/applications/-/issues/?sort=updated_desc&state=opened&label_name%5B%5D=Release%20Prep&first_page_size=20) for changelog generation
### Review
#### Request Reviewer
- [ ] Request review from an applications developer depending on modified system:
- **NOTE**: if the MR modifies multiple areas, please `/cc` all the relevant reviewers (since gitlab only allows 1 reviewer)
- **accessibility** : henry
- **android** : clairehurst, dan
- **build system** : boklm
- **extensions** : ma1
- **firefox internals (XUL/JS/XPCOM)** : ma1
- **fonts** : pierov
- **frontend (implementation)** : henry
- **frontend (review)** : donuts, richard
- **localization** : henry, pierov
- **macos** : clairehurst, dan
- **nightly builds** : boklm
- **rebases/release-prep** : dan, ma1, pierov, richard
- **security** : ma1
- **signing** : boklm, richard
- **updater** : pierov
- **misc/other** : pierov, richard
#### Change Description
<!-- Whatever context the reviewer needs to effectively review the patchset; if the patch includes UX updates be sure to include screenshots/video of how any new behaviour -->
#### How Tested
<!-- Description of steps taken to verify the change -->boklmboklmhttps://gitlab.torproject.org/tpo/applications/tor-browser-build/-/merge_requests/938Bug 41106: Force rebuild of application-services2024-03-19T12:43:37ZboklmBug 41106: Force rebuild of application-services## Merge Info
<!-- Bookkeeping information for release management -->
### Related Issues
- tor-browser-build#41106
### Backporting
#### Timeline
- [x] **Immediate**: patchset needed as soon as possible
- [ ] **Next Minor Stable Relea...## Merge Info
<!-- Bookkeeping information for release management -->
### Related Issues
- tor-browser-build#41106
### Backporting
#### Timeline
- [x] **Immediate**: patchset needed as soon as possible
- [ ] **Next Minor Stable Release**: patchset that needs to be verified in nightly before backport
- [ ] **Eventually**: patchset that needs to be verified in alpha before backport
- [ ] **No Backport (preferred)**: patchset for the next major stable
#### (Optional) Justification
- [ ] **Emergency security update**: patchset fixes CVEs, 0-days, etc
- [ ] **Censorship event**: patchset enables censorship circumvention
- [x] **Critical bug-fix**: patchset fixes a bug in core-functionality
- [ ] **Consistency**: patchset which would make development easier if it were in both the alpha and release branches; developer tools, build system changes, etc
- [ ] **Sponsor required**: patchset required for sponsor
- [ ] **Other**: please explain
### Issue Tracking
- [ ] Link resolved issues with appropriate [Release Prep issue](https://gitlab.torproject.org/groups/tpo/applications/-/issues/?sort=updated_desc&state=opened&label_name%5B%5D=Release%20Prep&first_page_size=20) for changelog generation
### Review
#### Request Reviewer
- [x] Request review from an applications developer depending on modified system:
- **NOTE**: if the MR modifies multiple areas, please `/cc` all the relevant reviewers (since gitlab only allows 1 reviewer)
- **accessibility** : henry
- **android** : clairehurst, dan
- **build system** : boklm
- **extensions** : ma1
- **firefox internals (XUL/JS/XPCOM)** : ma1
- **fonts** : pierov
- **frontend (implementation)** : henry
- **frontend (review)** : donuts, richard
- **localization** : henry, pierov
- **macos** : clairehurst, dan
- **nightly builds** : boklm
- **rebases/release-prep** : boklm, dan, ma1, pierov, richard
- **security** : ma1
- **signing** : boklm, richard
- **updater** : pierov
- **misc/other** : pierov, richard
### Change Description
This does not fix the issue, but forces a rebuild as a temporary workaround.
#### How Tested
Not tested.boklmboklmhttps://gitlab.torproject.org/tpo/applications/tor-browser-build/-/merge_requests/849Bug 41013: Add a README to each project2024-03-18T20:40:30ZPier Angelo VendrameBug 41013: Add a README to each project## Merge Info
<!-- Bookkeeping information for release management -->
### Related Issues
- tor-browser#xxxxx
- mullvad-browser#xxxxx
- tor-browser-build#41013
- tpo/community/hackweek#25
### Backporting
#### Timeline
- [ ] **Immediat...## Merge Info
<!-- Bookkeeping information for release management -->
### Related Issues
- tor-browser#xxxxx
- mullvad-browser#xxxxx
- tor-browser-build#41013
- tpo/community/hackweek#25
### Backporting
#### Timeline
- [ ] **Immediate**: patchset needed as soon as possible
- [ ] **Next Minor Stable Release**: patchset that needs to be verified in nightly before backport
- [ ] **Eventually**: patchset that needs to be verified in alpha before backport
- [x] **No Backport (preferred)**: patchset for the next major stable
#### (Optional) Justification
- [ ] **Emergency security update**: patchset fixes CVEs, 0-days, etc
- [ ] **Censorship event**: patchset enables censorship circumvention
- [ ] **Critical bug-fix**: patchset fixes a bug in core-functionality
- [ ] **Consistency**: patchset which would make development easier if it were in both the alpha and release branches; developer tools, build system changes, etc
- [ ] **Sponsor required**: patchset required for sponsor
- [ ] **Other**: please explain
### Issue Tracking
- [ ] Link resolved issues with appropriate [Release Prep issue](https://gitlab.torproject.org/groups/tpo/applications/-/issues/?sort=updated_desc&state=opened&label_name%5B%5D=Release%20Prep&first_page_size=20) for changelog generation
### Review
#### Request Reviewer
- [ ] Request review from an applications developer depending on modified system:
- **NOTE**: if the MR modifies multiple areas, please `/cc` all the relevant reviewers (since gitlab only allows 1 reviewer)
- **accessibility** : henry
- **android** : clairehurst, dan
- **build system** : boklm
- **extensions** : ma1
- **firefox internals (XUL/JS/XPCOM)** : ma1
- **fonts** : pierov
- **frontend (implementation)** : henry
- **frontend (review)** : donuts, richard
- **localization** : henry, pierov
- **macos** : clairehurst, dan
- **nightly builds** : boklm
- **rebases/release-prep** : dan, ma1, pierov, richard
- **security** : ma1
- **signing** : boklm, richard
- **updater** : pierov
- **misc/other** : pierov, richard
#### Change Description
In this MR I added a lot of READMEs.
I'm missing a few ones for which I'd need additional inputs:
- `lox-wasm` and `wasm-bindgen`
- `release`
- `yubihsm-shell`
Probably I haven't been coherent with the detail level, but we can improve that :slight_smile:.
#### How Tested
Browsed the tree of the source branch.
/cc @richardPier Angelo VendramePier Angelo Vendramehttps://gitlab.torproject.org/tpo/applications/tor-browser-build/-/merge_requests/846Bug 41012: Add a "component" information to our projects and a script to crea...2023-12-18T08:23:17ZPier Angelo VendrameBug 41012: Add a "component" information to our projects and a script to create a dependency graph## Merge Info
<!-- Bookkeeping information for release management -->
### Related Issues
- tor-browser#xxxxx
- mullvad-browser#xxxxx
- tor-browser-build#41012
### Backporting
#### Timeline
- [ ] **Immediate**: patchset needed as soon...## Merge Info
<!-- Bookkeeping information for release management -->
### Related Issues
- tor-browser#xxxxx
- mullvad-browser#xxxxx
- tor-browser-build#41012
### Backporting
#### Timeline
- [ ] **Immediate**: patchset needed as soon as possible
- [ ] **Next Minor Stable Release**: patchset that needs to be verified in nightly before backport
- [ ] **Eventually**: patchset that needs to be verified in alpha before backport
- [x] **No Backport (preferred)**: patchset for the next major stable
#### (Optional) Justification
- [ ] **Emergency security update**: patchset fixes CVEs, 0-days, etc
- [ ] **Censorship event**: patchset enables censorship circumvention
- [ ] **Critical bug-fix**: patchset fixes a bug in core-functionality
- [ ] **Consistency**: patchset which would make development easier if it were in both the alpha and release branches; developer tools, build system changes, etc
- [ ] **Sponsor required**: patchset required for sponsor
- [ ] **Other**: please explain
### Issue Tracking
- [ ] Link resolved issues with appropriate [Release Prep issue](https://gitlab.torproject.org/groups/tpo/applications/-/issues/?sort=updated_desc&state=opened&label_name%5B%5D=Release%20Prep&first_page_size=20) for changelog generation
### Review
#### Request Reviewer
- [x] Request review from an applications developer depending on modified system:
- **NOTE**: if the MR modifies multiple areas, please `/cc` all the relevant reviewers (since gitlab only allows 1 reviewer)
- **accessibility** : henry
- **android** : clairehurst, dan
- **build system** : boklm
- **extensions** : ma1
- **firefox internals (XUL/JS/XPCOM)** : ma1
- **fonts** : pierov
- **frontend (implementation)** : henry
- **frontend (review)** : donuts, richard
- **localization** : henry, pierov
- **macos** : clairehurst, dan
- **nightly builds** : boklm
- **rebases/release-prep** : dan, ma1, pierov, richard
- **security** : ma1
- **signing** : boklm, richard
- **updater** : pierov
- **misc/other** : pierov, richard
/cc @richard
#### Change Description
I've created a script to create a dependency graph in the dot format.
I decided to group some dependencies together, so I added to each project (except for a few ones) a component information.
Maybe we can discuss a few projects (e.g., `node` and `cbindgen`: they're toolchain, but used only by Firefox. Maybe nasm should be with them).
The script is set to ignore the dependency on some projects, notably:
- container-image
- binutils
- compiler
- go (at least until we have Namecoin dependencies instead of using go.mod/go.sum)
Also, the script ignores firefox-l10n because it isn't trivial to parse (but it's the only project, so not worth to implement a better parser only for it).
#### How Tested
Created this graph:
<details><summary>Graph</summary>
![graphviz_ignore.svg](/uploads/22aedd59d45c5c88320434841bef2c18/graphviz_ignore.svg)
</details>
<details><summary>Graph without exceptions (for reference)</summary>
![graphviz.svg](/uploads/c42e9af52785c8037ce812d52bd662bd/graphviz.svg)
</details>Pier Angelo VendramePier Angelo Vendrame