The Tor Project should provide a Tor Browser compatible with the ARMv7 processor. This would provide a safe way of using Tor for users of the Samsung ARM Chromebook, the Samsung Chromebook 2, the Raspberry Pi, the Novena Open Laptop, and probably other platforms too.
This ticket is specifically for building Tor Browser for ARMv7 targets with the x86_64 build arch.
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Would it be acceptable (at least for the purposes of an initial merge) for the Linux/ARMv7 target to use the clang/rustc packages from Debian Buster rather than the rbm-built equivalents (at least for building the firefox project)? I'm having no luck getting Firefox to build for Linux/ARMv7 with the self-built clang/rustc (seems to be some kind of weird path/sysroot issue), but I'm able to build it fine with the Buster-packaged compilers in rbm.
I assume that the main reason for building clang/rustc inside rbm is for bootstrapping-related security, but given that legacy/trac#32523 (moved) might make that obsolete anyway, I tend to figure that anything that builds working binaries is better than nothing for now.
Would it be acceptable (at least for the purposes of an initial merge) for the Linux/ARMv7 target to use the clang/rustc packages from Debian Buster rather than the rbm-built equivalents (at least for building the firefox project)? I'm having no luck getting Firefox to build for Linux/ARMv7 with the self-built clang/rustc (seems to be some kind of weird path/sysroot issue), but I'm able to build it fine with the Buster-packaged compilers in rbm.
So, both clang/rustc compiled yourself are problematic or just one of them? I.e. what happens if you take, say, the clang we compile with the buster-packaged rustc and vice versa? Is the issue still the same as in comment:2? If not what is the problem now?
Good questions @gk, I'll do some further diagnosis and hopefully see what happens if one is from Buster and the other is rbm-built. Might not get to it for a few weeks, things are hectic here (and this port is mostly a weekend project for me). I don't have my current work pushed to NotABug yet, but will hopefully do so next weekend.
Updated branch at https://notabug.org/JeremyRand/tor-browser-build/src/armhf-esr68 (commit hash 224663ea2f7b5096f25ef7ed3e7170d21b16fbe5 in the armhf-esr68 branch). Successfully got the rbm-built clang working for building firefox for linux-arm. Still very much a WIP branch, but it does build working binaries.
Updated branch at https://notabug.org/JeremyRand/tor-browser-build/src/armhf-esr68 (commit hash bcd0e21237a0aeb598d1cf388b08d6f4e28eecd0). Builds binaries that work fine on my Asus C201 (running PrawnOS, based on Debian Buster). Binutils, Clang, Rust, Rust's stdlib, and cbindgen are all built from source in rbm, as with x86. libgcc/libstdc++ from Debian Buster's repos are used as build-time dependencies (removing them is likely to be nontrivial).
As discussed with Georg on IRC, I'm now requesting informal review, not for the purpose of determining eligibility to be merged at this time, but for the purpose of determining whether there are any likely conflicts between this branch and the eventual ESR78 port of Tor Browser. Georg informs me that this will definitely not be considered for merge until after ESR78 is active; that's totally fine from my point of view. I would appreciate any informal review that could point out things that might be merge blockers after ESR78 is active, or things that may need a rewrite/refactor to be compatible with ESR78. (However, I recognize that this is a low priority for the Tor Browser team, probably more so due to the recent staff losses, so I will accept whatever delays happen in reviewing this.) I am already aware that the use of network access while building the tor project is a blocker; I intend to fix that in the same way that I already fixed it in the firefox project (i.e. switching the build container to Debian Buster, and installing the relevant armhf packages via apt-get as part of container construction).
That's impressive work, thanks! I have three high-level questions for now:
IIRC the linux-cross target was the idea to abstract the cross-compilation part away from ARM and other platforms. Do I remember that right? Right now we have for that:
That does not strike me overly useful. fteproxy is gone. We'll want to have snowflake support on those platforms anyway and I doubt that the container won't be amd64. So, overall we don't seem to need that abstraction? Or am I missing something?
We build usually on latest Debian stable for our targets when we are cross-compiling. I think it is fine thinking about something else than building on Debian Wheezy for this project, in particular if it would make things easier (which it seems it would). We are only still on Wheezy for non-cross builds because of CentOS 6 and we'll soon move away from it because CentOS 6 will finally be dead later this year. We'll likely end up using Jessie or even Stretch (but very likely not Buster). I see you are using Glibc 2.26 in your cross-build which is even newer than what is included in Stretch. Thus, you probably already excluded a bunch of potentially still supported distro versions for armv7 (which is fine to me right now). So, a good step forward here we would thinking about what we/you want to support and what minimum Debian version that would mean (it's not Wheezy :)) and then start using that one (as I said above I bet newer Debian versions would greatly help in the cross-compilation effort)
We use essentially for all of our cross-compilation efforts clang nowadays and not GCC as the former has quite good support (which got better over the years) and GCC is not enough (anymore) to build for macOS/Winodws/Android. Would your linux arm work benefit from using clang here too? Or would that be harder? You are using clang for the Firefox part anyway (which is already the by far most complicated piece in our projects).
That does not strike me overly useful. fteproxy is gone. We'll want to have snowflake support on those platforms anyway and I doubt that the container won't be amd64. So, overall we don't seem to need that abstraction? Or am I missing something?
@gk The var/linux-cross variable is used quite a lot in the build scripts / config files for projects (far more than in rbm.conf). I've found it to be very useful in those areas.
We build usually on latest Debian stable for our targets when we are cross-compiling. I think it is fine thinking about something else than building on Debian Wheezy for this project, in particular if it would make things easier (which it seems it would). We are only still on Wheezy for non-cross builds because of CentOS 6 and we'll soon move away from it because CentOS 6 will finally be dead later this year. We'll likely end up using Jessie or even Stretch (but very likely not Buster). I see you are using Glibc 2.26 in your cross-build which is even newer than what is included in Stretch. Thus, you probably already excluded a bunch of potentially still supported distro versions for armv7 (which is fine to me right now). So, a good step forward here we would thinking about what we/you want to support and what minimum Debian version that would mean (it's not Wheezy :)) and then start using that one (as I said above I bet newer Debian versions would greatly help in the cross-compilation effort)
I would have no objection to experimenting with either Jessie or Stretch to see how well they handle cross-compiling. Buster worked well for my prototyping (and Wheezy definitely did not play well), but if Jessie or Stretch can be used with minimal extra effort compared to Buster (which seems plausible) then I think that's a good thing to aim for.
We use essentially for all of our cross-compilation efforts clang nowadays and not GCC as the former has quite good support (which got better over the years) and GCC is not enough (anymore) to build for macOS/Winodws/Android. Would your linux arm work benefit from using clang here too? Or would that be harder? You are using clang for the Firefox part anyway (which is already the by far most complicated piece in our projects).
Clang may actually be easier to use everywhere for cross-compiling than GCC, because building GCC from source for cross-compiling purposes requires building glibc from source as well (which produces potential for conflicts with the Debian-packaged glibc). An arguable drawback of Clang is that building glibc from source is good for auditability, so having Clang use the Debian-packaged glibc is bad. I don't really buy that argument, since (1) the x86 Linux targets already use the Debian-packaged glibc, and (2) the glibc is linked dynamically, so AFAICT that code never actually gets shipped to end-user machines anyway. So, if you're okay with switching everything for ARM to Clang, I'm definitely okay with giving that a try. Might simplify the code substantially.
That does not strike me overly useful. fteproxy is gone. We'll want to have snowflake support on those platforms anyway and I doubt that the container won't be amd64. So, overall we don't seem to need that abstraction? Or am I missing something?
@gk The var/linux-cross variable is used quite a lot in the build scripts / config files for projects (far more than in rbm.conf). I've found it to be very useful in those areas.
We build usually on latest Debian stable for our targets when we are cross-compiling. I think it is fine thinking about something else than building on Debian Wheezy for this project, in particular if it would make things easier (which it seems it would). We are only still on Wheezy for non-cross builds because of CentOS 6 and we'll soon move away from it because CentOS 6 will finally be dead later this year. We'll likely end up using Jessie or even Stretch (but very likely not Buster). I see you are using Glibc 2.26 in your cross-build which is even newer than what is included in Stretch. Thus, you probably already excluded a bunch of potentially still supported distro versions for armv7 (which is fine to me right now). So, a good step forward here we would thinking about what we/you want to support and what minimum Debian version that would mean (it's not Wheezy :)) and then start using that one (as I said above I bet newer Debian versions would greatly help in the cross-compilation effort)
I would have no objection to experimenting with either Jessie or Stretch to see how well they handle cross-compiling. Buster worked well for my prototyping (and Wheezy definitely did not play well), but if Jessie or Stretch can be used with minimal extra effort compared to Buster (which seems plausible) then I think that's a good thing to aim for.
We use essentially for all of our cross-compilation efforts clang nowadays and not GCC as the former has quite good support (which got better over the years) and GCC is not enough (anymore) to build for macOS/Winodws/Android. Would your linux arm work benefit from using clang here too? Or would that be harder? You are using clang for the Firefox part anyway (which is already the by far most complicated piece in our projects).
Clang may actually be easier to use everywhere for cross-compiling than GCC, because building GCC from source for cross-compiling purposes requires building glibc from source as well (which produces potential for conflicts with the Debian-packaged glibc). An arguable drawback of Clang is that building glibc from source is good for auditability, so having Clang use the Debian-packaged glibc is bad. I don't really buy that argument, since (1) the x86 Linux targets already use the Debian-packaged glibc, and (2) the glibc is linked dynamically, so AFAICT that code never actually gets shipped to end-user machines anyway. So, if you're okay with switching everything for ARM to Clang, I'm definitely okay with giving that a try. Might simplify the code substantially.
Clang may actually be easier to use everywhere for cross-compiling than GCC, because building GCC from source for cross-compiling purposes requires building glibc from source as well (which produces potential for conflicts with the Debian-packaged glibc). An arguable drawback of Clang is that building glibc from source is good for auditability, so having Clang use the Debian-packaged glibc is bad. I don't really buy that argument, since (1) the x86 Linux targets already use the Debian-packaged glibc, and (2) the glibc is linked dynamically, so AFAICT that code never actually gets shipped to end-user machines anyway. So, if you're okay with switching everything for ARM to Clang, I'm definitely okay with giving that a try. Might simplify the code substantially.
I recently tried building Tor Browser's Firefox fork for ppc64le with Clang (without cross-compiling or RBM) and ended up with a binary that segfaulted on launch. I checked Talospace and found that this is a known issue; building Firefox for ppc64le requires GCC. I confirmed this myself by replacing Clang with GCC, which fixed the segfault.
So, given that the linux-cross target is intended to work for both ARM and POWER with minimal changes, I no longer think it's a good idea to spend effort on using Clang everywhere.
Clang may actually be easier to use everywhere for cross-compiling than GCC, because building GCC from source for cross-compiling purposes requires building glibc from source as well (which produces potential for conflicts with the Debian-packaged glibc). An arguable drawback of Clang is that building glibc from source is good for auditability, so having Clang use the Debian-packaged glibc is bad. I don't really buy that argument, since (1) the x86 Linux targets already use the Debian-packaged glibc, and (2) the glibc is linked dynamically, so AFAICT that code never actually gets shipped to end-user machines anyway. So, if you're okay with switching everything for ARM to Clang, I'm definitely okay with giving that a try. Might simplify the code substantially.
I recently tried building Tor Browser's Firefox fork for ppc64le with Clang (without cross-compiling or RBM) and ended up with a binary that segfaulted on launch. I checked Talospace and found that this is a known issue; building Firefox for ppc64le requires GCC. I confirmed this myself by replacing Clang with GCC, which fixed the segfault.
So, given that the linux-cross target is intended to work for both ARM and POWER with minimal changes, I no longer think it's a good idea to spend effort on using Clang everywhere.
Okay, that is unfortunate. @JeremyRand: Is there a clang bug open where this is tracked and I can learn more about the problem?
Is there a clang bug open where this is tracked and I can learn more about the problem?
@gk I am not aware of any open bug, but based on asking around on #talos-workstation, it sounds to me that "QA testing for Clang on anything besides x86 and ARM is near nonexistent" is common knowledge among the kind of people who work with this stuff regularly. I can maybe ask the Talospace guy whether he knows of a bug, but it sounds to me like even if this particular bug gets fixed, the lack of QA testing for Clang with POWER will probably cause future breakage that may not get fixed promptly.
Updated ARM branch at https://notabug.org/JeremyRand/tor-browser-build/src/armhf-esr78 . It no longer needs any network access inside the build containers. Builds fine for me, and runs fine on my Asus C201 running PrawnOS (derived from Debian Buster). Currently about 8 months behind master; I will try to finish catching up with master ASAP. Git history is a tad messy right now; I will squash things while I catch up with master.
@gk I am not aware of any open bug, but based on asking around on #talos-workstation, it sounds to me that "QA testing for Clang on anything besides x86 and ARM is near nonexistent" is common knowledge among the kind of people who work with this stuff regularly. I can maybe ask the Talospace guy whether he knows of a bug, but it sounds to me like even if this particular bug gets fixed, the lack of QA testing for Clang with POWER will probably cause future breakage that may not get fixed promptly.
Okay, fair enough.
Updated ARM branch at https://notabug.org/JeremyRand/tor-browser-build/src/armhf-esr78 . It no longer needs any network access inside the build containers. Builds fine for me, and runs fine on my Asus C201 running PrawnOS (derived from Debian Buster). Currently about 8 months behind master; I will try to finish catching up with master ASAP. Git history is a tad messy right now; I will squash things while I catch up with master.
Good stuff! Yeah, getting your work rebased onto master seems to be like a good next step. Note, though, that we are about to switch our nightly builds to pick up esr91 (with all the toolchain updates this involves and potential build breakage for you :( )... So, there might be a new hill to climb after the master rebase.
Updated ARM branch at https://notabug.org/JeremyRand/tor-browser-build/src/armhf-esr91 ; successfully rebased onto ESR91. This branch builds without errors, but I haven't been able to test the binaries because my Asus C201 is undergoing repairs. Hoping to have a working ARM test machine again within a week or two. In the meantime, review is welcome.
My ARM branch names the target linux-arm. Would it be desirable for me to rename it to linux-armv7 for consistency with the existing android-armv7 target name?
Hi @boklm. Do you want me to send in an MR on GitLab, or is providing the branch URL (as I did above) sufficient for you to review? (I haven't yet gotten familiar with the workflow you guys use since the Trac to GitLab transition).
I think adding several new bundles (one for each locale) would increase too much the size used on dist.tpo, and the time for the signing process. So I think we have two options to avoid that:
having the linux-arm bundles only in English, and maybe a few other locales
having one multi-lingual bundle including all locales
Since the multi-lingual bundle is also something we might want to have on the other platforms in the future, I think it's a good idea to try it now with the linux-arm bundles.
I had a look through their maint-13.0-arm branch. It's a bit out of date (seems it they don't keep up with our minor releases) but the patchset is well organised, it's a number of per-project patches on top of the the 13.0 build tag.
Given the current improved state of our build+signing infra we should review these patches and at least see if they are reproducible.
@boklm we had a closer look in IRC yesterday, and it turns out one caveat here is that they are building from a arm64 host OS, not cross compiling. So its a cool starting point but not a ready-working solution.