Use mingw-w64/clang toolchain to build Firefox
We want to use the clang-based mingw-w64 toolchain to build Firefox to be finally able to enabled Stylo on Windows as well. This is the ticket tracking the implementation work
Activity
We likely start small and just use the toolchain to build the Firefox part which means we'd have two mingw-w64 toolchains for the time being (we had this back then in the old days for macOS, too). I guess other parts of the whole bundle might need quite some amount of work to getting built with that new toolchain and we want to move fast here aiming for Tor Browser 8.5.
However, if it turns out I am wrong and it is indeed easy to use mingw-w64/clang for everything, fine with me.
-
Testing
bug_28238shows that we have some reproducibility issues with mingw-w64/clang. It seems we have to deal at least with timestamp issues in the COFF header. Comparing e.g.firefox.exefrom two different runs shows:--- /dev/fd/63 2019-01-10 09:38:08.159230047 +0100 +++ /dev/fd/62 2019-01-10 09:38:08.159230047 +0100 @@ -6,7 +6,7 @@ 00000050: 6973 2070 726f 6772 616d 2063 616e 6e6f is program canno 00000060: 7420 6265 2072 756e 2069 6e20 444f 5320 t be run in DOS 00000070: 6d6f 6465 2e24 0000 5045 0000 4c01 0600 mode.$..PE..L... -00000080: 3b1c 365c 003c 0000 a201 0000 e000 2201 ;.6\.<........". +00000080: 45ff 365c 003c 0000 a201 0000 e000 2201 E.6\.<........". 00000090: 0b01 0e00 001e 0000 001a 0000 0000 0000 ................ 000000a0: c013 0000 0010 0000 0000 0000 0000 4000 ..............@. 000000b0: 0010 0000 0002 0000 0600 0000 0000 0000 ................ -
Replying to gk:
Testing
bug_28238shows that we have some reproducibility issues with mingw-w64/clang. It seems we have to deal at least with timestamp issues in the COFF header. Comparing e.g.firefox.exefrom two different runs shows: {{{ --- /dev/fd/63 2019-01-10 09:38:08.159230047 +0100 +++ /dev/fd/62 2019-01-10 09:38:08.159230047 +0100 @@ -6,7 +6,7 @@ 00000050: 6973 2070 726f 6772 616d 2063 616e 6e6f is program canno 00000060: 7420 6265 2072 756e 2069 6e20 444f 5320 t be run in DOS 00000070: 6d6f 6465 2e24 0000 5045 0000 4c01 0600 mode.$..PE..L... -00000080: 3b1c 365c 003c 0000 a201 0000 e000 2201 ;.6.<........". +00000080: 45ff 365c 003c 0000 a201 0000 e000 2201 E.6.<........". 00000090: 0b01 0e00 001e 0000 001a 0000 0000 0000 ................ 000000a0: c013 0000 0010 0000 0000 0000 0000 4000 ..............@. 000000b0: 0010 0000 0002 0000 0600 0000 0000 0000 ................ }}}FWIW: That's still an unsolved issue after bumping the LLVM revision to r348363.
-
Replying to ld:
https://bugs.chromium.org/p/chromium/issues/detail?id=533657 https://bugs.llvm.org/show_bug.cgi?id=24740#c3
That's about building clang deterministic, yes. Thus, this is more suited for #29041 (moved).
https://bugs.llvm.org/show_bug.cgi?id=35914
Trac:
Username: ld-
https://reviews.llvm.org/rL332613 seems like the relevant revision for the timestamp issue.
-
So, I have been fighting today with building the whole Firefox related part (including
fxc2) without the old mingw-w64/gcc-based toolchain.fxc2does not want and I don't know why yet. It fails to run with things like0:03.01 err:module:import_dll Library api-ms-win-crt-convert-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found 0:03.01 err:module:import_dll Library api-ms-win-crt-heap-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found 0:03.01 err:module:import_dll Library api-ms-win-crt-private-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found 0:03.01 err:module:import_dll Library api-ms-win-crt-runtime-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found 0:03.01 err:module:import_dll Library api-ms-win-crt-stdio-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found 0:03.02 err:module:import_dll Library api-ms-win-crt-string-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found 0:03.02 err:module:import_dll Library api-ms-win-crt-environment-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found 0:03.02 err:module:import_dll Library api-ms-win-crt-math-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found 0:03.02 err:module:import_dll Library api-ms-win-crt-time-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found 0:03.03 err:module:import_dll Library api-ms-win-crt-locale-l1-1-0.dll (which is needed by L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe") not found 0:03.03 err:module:LdrInitializeThunk Main exe initialization for L"Z:\\var\\tmp\\dist\\fxc2\\bin\\fxc2.exe" failed, status c0000135I need to look closer at the way this is built on Mozilla infra tomorrow where this seems to be working.
-
Replying to gk:
So, I have been fighting today with building the whole Firefox related part (including
fxc2) without the old mingw-w64/gcc-based toolchain. No, no, don't fight with it! The new tool chain is much more sane. And it will make you happy.fxc2does not want and I don't know why yet. It fails to run with things like {{{ 0:03.01 err:module:import_dll Library api-ms-win-crt-convert-l1-1-0.dll (which is needed by L"Z:\var\tmp\dist\fxc2\bin\fxc2.exe") not found 0:03.01 err:module:import_dll Library api-ms-win-crt-heap-l1-1-0.dll (which is needed by L"Z:\var\tmp\dist\fxc2\bin\fxc2.exe") not found 0:03.01 err:module:import_dll Library api-ms-win-crt-private-l1-1-0.dll (which is needed by L"Z:\var\tmp\dist\fxc2\bin\fxc2.exe") not found 0:03.01 err:module:import_dll Library api-ms-win-crt-runtime-l1-1-0.dll (which is needed by L"Z:\var\tmp\dist\fxc2\bin\fxc2.exe") not found 0:03.01 err:module:import_dll Library api-ms-win-crt-stdio-l1-1-0.dll (which is needed by L"Z:\var\tmp\dist\fxc2\bin\fxc2.exe") not found 0:03.02 err:module:import_dll Library api-ms-win-crt-string-l1-1-0.dll (which is needed by L"Z:\var\tmp\dist\fxc2\bin\fxc2.exe") not found 0:03.02 err:module:import_dll Library api-ms-win-crt-environment-l1-1-0.dll (which is needed by L"Z:\var\tmp\dist\fxc2\bin\fxc2.exe") not found 0:03.02 err:module:import_dll Library api-ms-win-crt-math-l1-1-0.dll (which is needed by L"Z:\var\tmp\dist\fxc2\bin\fxc2.exe") not found 0:03.02 err:module:import_dll Library api-ms-win-crt-time-l1-1-0.dll (which is needed by L"Z:\var\tmp\dist\fxc2\bin\fxc2.exe") not found 0:03.03 err:module:import_dll Library api-ms-win-crt-locale-l1-1-0.dll (which is needed by L"Z:\var\tmp\dist\fxc2\bin\fxc2.exe") not found 0:03.03 err:module:LdrInitializeThunk Main exe initialization for L"Z:\var\tmp\dist\fxc2\bin\fxc2.exe" failed, status c0000135 }}} Do you have wine with ucrt? FWIW, wine 4.0 is out. I need to look closer at the way this is built on Mozilla infra tomorrow where this seems to be working. They use https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/scripts/misc/build-wine.shTrac:
Username: fxc2 -
Okay, I am close with this ticket. Here is a test bundle which compiles fxc2 and firefox with mingw-w64/clang while mingw-w64/gcc is used for the remaining components. I'd be interested in whether that explodes on anyone's machine:
https://people.torproject.org/~gk/testbuilds/torbrowser-install-win64-tbb-nightly_28238_1_en-US.exe https://people.torproject.org/~gk/testbuilds/torbrowser-install-win64-tbb-nightly_28238_1_en-US.exe.asc
-
Here is a 32bit one to try out:
https://people.torproject.org/~gk/testbuilds/torbrowser-install-tbb-nightly_28238_1_en-US.exe https://people.torproject.org/~gk/testbuilds/torbrowser-install-tbb-nightly_28238_1_en-US.exe.asc
Trac:
Parent: N/A to #29318 (moved) Whoa! Cypherpunks operational!
Of course, that explodes, but whether I'm the only one who is interested in it?
Trac:
Parent: #29318 (moved) to N/Aoops
Trac:
Parent: N/A to #29318 (moved)-
Replying to cypherpunks:
Whoa! Cypherpunks operational!
Of course, that explodes, but whether I'm the only one who is interested in it?
Care to give more context? Which of the bundles did you test? On which Windows version? What happened? What error did you get, if any?
-
Replying to gk:
Replying to gk:
Testing
bug_28238shows that we have some reproducibility issues with mingw-w64/clang. It seems we have to deal at least with timestamp issues in the COFF header. Comparing e.g.firefox.exefrom two different runs shows: {{{ --- /dev/fd/63 2019-01-10 09:38:08.159230047 +0100 +++ /dev/fd/62 2019-01-10 09:38:08.159230047 +0100 @@ -6,7 +6,7 @@ 00000050: 6973 2070 726f 6772 616d 2063 616e 6e6f is program canno 00000060: 7420 6265 2072 756e 2069 6e20 444f 5320 t be run in DOS 00000070: 6d6f 6465 2e24 0000 5045 0000 4c01 0600 mode.$..PE..L... -00000080: 3b1c 365c 003c 0000 a201 0000 e000 2201 ;.6.<........". +00000080: 45ff 365c 003c 0000 a201 0000 e000 2201 E.6.<........". 00000090: 0b01 0e00 001e 0000 001a 0000 0000 0000 ................ 000000a0: c013 0000 0010 0000 0000 0000 0000 4000 ..............@. 000000b0: 0010 0000 0002 0000 0600 0000 0000 0000 ................ }}}FWIW: That's still an unsolved issue after bumping the LLVM revision to r348363.
That is solved and in the process of being upstreamed. I now get identical builds on one machine. However, compared to the other one I tested they significantly differ. Not all .dlls/.exe files are affected, though, just
firefox.exe,libGLESv2.dll,mozglue.dll,pingsender.exe,plugin-container.exe,plugin-hang-ui.exe, andxul.dll. -
Trac:
fxexe_diff.bz2 Replying to gk:
Replying to cypherpunks:
Whoa! Cypherpunks operational!
Of course, that explodes, but whether I'm the only one who is interested in it?
Care to give more context? Which of the bundles did you test? On which Windows version? What happened? What error did you get, if any? No answers to my questions - no answers to your questions. But don't worry, it's a blocker. There are over 1M daily users on Windows and no one here. How so? Are they fakes? Or don't care? And you, do you care about security on Windows as a primary target, or Windows is incompatible with security for you? (5 vs 5 questions)
x64 build: On first run; tor failed; noting:
2/5/19, 16:46:34.346 [NOTICE] Bootstrapped 40% (loading_keys): Loading authority key certs 2/5/19, 16:46:34.952 [WARN] ISO time "2019-02-05 16:00:00\r" was unparseable 2/5/19, 16:46:34.952 [WARN] Unable to parse networkstatus consensus
On second run (I ran firefox.exe directly but I don't think that mattered) it worked and I was able to load a website, youtube, play a video, and hit an onion site.
x86 build: same problem with the authoritity certificates. On second run (this time I used the Start Tor Browser shortcut) it worked; again with a website, onion, and youtube. (I actually exited using IPv6 too!) I confirmed ASLR on the x86 build.
All on Windows 10.
Replying to tom:
x64 build: On first run; tor failed; noting:
2/5/19, 16:46:34.346 [NOTICE] Bootstrapped 40% (loading_keys): Loading authority key certs 2/5/19, 16:46:34.952 [WARN] ISO time "2019-02-05 16:00:00\r" was unparseable 2/5/19, 16:46:34.952 [WARN] Unable to parse networkstatus consensus
On second run (I ran firefox.exe directly but I don't think that mattered) it worked and I was able to load a website, youtube, play a video, and hit an onion site.
x86 build: same problem with the authoritity certificates. On second run (this time I used the Start Tor Browser shortcut) it worked; again with a website, onion, and youtube. (I actually exited using IPv6 too!) I confirmed ASLR on the x86 build.
All on Windows 10.
This looks like #28614 (moved).
-
Trac:
Parent: #29318 (moved) to #30321 (moved) -
Trac:
Parent: #30321 (moved) to #30322 (moved) -
bug_28238_v14(https://gitweb.torproject.org/user/gk/tor-browser-build.git/log/?h=bug_28238_v14) has the patches to get this going. pospeselr checked that a 64bit build is working, which is nice!The patches are on top for those for #28716 (moved) to make testing easier. One caveat are the 32bit builds wich are currently failing due to memory issues when building
gkrust. I need to find some configure flags that help here or we need to bite the bullet of using a 64bit host for 32bit builds as well.Trac:
Status: new to needs_review
Keywords: TorBrowserTeam201908 deleted, TorBrowserTeam201908R added -
Replying to boklm:
Switching to 64bit host for the 32bit builds sounds like a good idea.
I agree with that and we should probably do that in our alpha cycle for Tor Browser 9. However, I fear that's more work involved than we have capacity in, say, the next two, three days given all the other stuff on our plate.
It seems I managed to solve the build problems for now with
bug_28238_v15(https://gitweb.torproject.org/user/gk/tor-browser-build.git/log/?h=bug_28238_v15). Please use that one for review. -
bug_28238_v16(https://gitweb.torproject.org/user/gk/tor-browser-build.git/log/?h=bug_28238_v16) picks up the new change frombug_28716_v14(but does not contain new Firefox related ones. -
- Have you confirmed you don't need the spec thing for https://bugzilla.mozilla.org/show_bug.cgi?id=1460801 ? (If so, I can close that bug.)
- fxc2 requires the winpthread dll to be in its bin directly IIRC; but I don't see where you're copying that. (I might have missed it. If you're not getting errors on fxc2 it must be working.)
- I see some flags in our mozconfig you don't have:
- I'm not sure what you do about PDBs; but it would be good to get a bug on file about generating/outputting them. (Perhaps in some configuration generating the static clang libraries with debug info also.)
-
Replying to tom:
- Have you confirmed you don't need the spec thing for https://bugzilla.mozilla.org/show_bug.cgi?id=1460801 ? (If so, I can close that bug.)
I think it worked for me back then a couple of months ago when I worked on the toolchain. I'll double-check with nightly builds once things landed but am optimistic.
- fxc2 requires the winpthread dll to be in its bin directly IIRC; but I don't see where you're copying that. (I might have missed it. If you're not getting errors on fxc2 it must be working.)
I think we don't need it when building with
mingw-w64-clang. At least the build passes and I'd suspect compile time issues if that were a problem.- I see some flags in our mozconfig you don't have:
Yes, I am not sure why we don't need those but the build is not breaking, so we could leave that investigation for later.
- I'm not sure what you do about PDBs; but it would be good to get a bug on file about generating/outputting them. (Perhaps in some configuration generating the static clang libraries with debug info also.)
I agree. Right now don't generate them.
-
- fxc2 requires the winpthread dll to be in its bin directly IIRC; but I don't see where you're copying that. (I might have missed it. If you're not getting errors on fxc2 it must be working.)
I think we don't need it when building with
mingw-w64-clang. At least the build passes and I'd suspect compile time issues if that were a problem.No, from my recollection it will compile fine but won't run if the dll is missing from the directory. (However if firefox builds, then fxc2 didn't error and it worked...)
- I'm not sure what you do about PDBs; but it would be good to get a bug on file about generating/outputting them. (Perhaps in some configuration generating the static clang libraries with debug info also.)
I agree. Right now don't generate them.
I suspect you do generate them; but without setting MOZ_COPY_PDBs they are not winding up next to the outputed files so they're not winding up in the final tarball. You could stick that into the mozconfig and see if they show up though.
-
Replying to tom:
- fxc2 requires the winpthread dll to be in its bin directly IIRC; but I don't see where you're copying that. (I might have missed it. If you're not getting errors on fxc2 it must be working.)
I think we don't need it when building with
mingw-w64-clang. At least the build passes and I'd suspect compile time issues if that were a problem.No, from my recollection it will compile fine but won't run if the dll is missing from the directory. (However if firefox builds, then fxc2 didn't error and it worked...)
Another thought here: how are you compiling
fxc2? I recall that I needed to resort to the .dll when compilingfxc2withmingw-w64-gcc, which was my main motivation to usemingw-w64-clangwhen buildingfxc2as well.