Skip to content
Snippets Groups Projects
New look: Off

Rust 1.60 not working to build 102 on Debian Jessie

    • Closed Issue created by Pier Angelo Vendrame @pierov

    Mozilla uses Rust 1.60 to build Firefox 102.

    For some reason our binaries cannot complete a build, but official binaries can.

    It seems like we hit a deadlock in the phase

    Compiling gkrust-shared v0.1.0 (/home/rbm/tor-browser/toolkit/library/rust/shared)

    Only one rustc process remains alive, but it uses only 40-50MB of memory, and 0% of CPU, and it never ends.

    I have also tried to add a RUSTC_LOGS="info"/RUSTC_LOGS="debug", but I don't get any output.

    Our binaries seem to work on the current stable of Debian (I have tried to set bullseye as a container in projects/firefox/config).

    As a workaround we can run ./install.sh --prefix=$distdir on the downloaded 1.60, to convert the downloaded archive (which is organized by project) to a more standard directory structure (the usual bin/, lib/, etc...).

    Also, I have opened a thread on Rust's forum: https://users.rust-lang.org/t/how-to-reproduce-official-rustc-builds-for-linux/79569. My idea is that we could start by building a rustc in the same environment and with the same configuration as the official binaries, and then make it more and more similar to our builds, to see when it breaks.

    Edited
    To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information

    Child items ...

    • Activity

    • Pier Angelo Vendrame changed the description

      changed the description

    • morgan added Next Sponsor 131 labels

      added Next Sponsor 131 labels

    • morgan changed milestone to %Sponsor 131 - Phase 3 - Major ESR 102 Migration

      changed milestone to %Sponsor 131 - Phase 3 - Major ESR 102 Migration

    • Pier Angelo Vendrame removed Next label

      removed Next label

    • Pier Angelo Vendrame assigned to @pierov

      assigned to @pierov

    • Pier Angelo Vendrame added Doing label

      added Doing label

      • Pier Angelo Vendrame
        Pier Angelo Vendrame @pierov ·
        Author Maintainer

        So, I've tried to build rustc in CentOS 7 with almost the same recipe of the official Dockerfile.

        "almost" because I used GCC 8 (more info here, I don't remember if more modern versions are packed, too), instead of compiling some GCC 5 from sources.

        Also, I've compiled clang with the same options we use in porjects/clang/build, and then set it as CC and CXX.

        This configuration worked:

        ./configure --prefix=$HOME/rust --enable-local-rust --enable-vendor --enable-full-tools --enable-sanitizers  --enable-profiler --disable-docs --disable-compiler-docs --set target.x86_64-unknown-linux-gnu.linker=clang  --set target.x86_64-unknown-linux-gnu.ar=llvm-ar --set target.x86_64-unknown-linux-gnu.ranlib=llvm-ranlib --set llvm.ninja=false --set rust.jemalloc --set rust.use-lld=true --llvm-root=$HOME/clang --sysconfdir=etc

        I have added --enable-local-rust and --enable-vendor from our command line (to simulate our offline builds), and --llvm-root=$HOME/clang is prevent Rust from compiling too many LLVM tools (which I'd love to do for #40592 (closed)).

        However, this configuration doesn't work in Debian Jessie, because Rust wants to compile its own lld with --enable-full-tools. The problem was some cmake nonsense, that checked the version of libstdc++. For some reason, it's ignoring --gcc-toolchain=/home/rbm/gcc. I wonder if I can write a pair of wrappers to clang and clang++ that always have this option.

        So, I've tried to replace --enable-full-tools with --enable-extended, that enables some tools like cargo, and is the command line option we currently have on projects/rust/config.

        This allows us to finish gkrust-shared, but goes to sigsegv in gkrust :scream:

        The output of mach build 
        0:06.09    Compiling gkrust v0.1.0 (/home/rbm/tor-browser/toolkit/library/rust)
 0:06.80 /home/rbm/rust/lib/librustc_driver-2b06d4f7701cf2a7.so(+0xc912e3)[0x7f966f1e62e3]
 0:06.80 /lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7f966df3b890]
 0:06.80 /home/rbm/rust/lib/librustc_driver-2b06d4f7701cf2a7.so(_RNvMs9_NtNtCsd62QaXPQfQz_12rustc_middle2ty7contextNtB5_6TyCtxt23def_path_hash_to_def_id+0x72)[0x7f9674932822]
 0:06.80 /home/rbm/rust/lib/librustc_driver-2b06d4f7701cf2a7.so(_RNvXs_NtNtCsd62QaXPQfQz_12rustc_middle9dep_graph8dep_nodeINtNtNtCshHhGpJjbd1P_18rustc_query_system9dep_graph8dep_node7DepNodeNtB4_7DepKindENtB4_10DepNodeExt14extract_def_id+0x47)[0x7f96749a7427]
 0:06.80 /home/rbm/rust/lib/librustc_driver-2b06d4f7701cf2a7.so(+0x54fa6fc)[0x7f9673a4f6fc]
 0:06.80 /home/rbm/rust/lib/librustc_driver-2b06d4f7701cf2a7.so(_RNvXs_NtCsd62QaXPQfQz_12rustc_middle9dep_graphNtNtNtB6_2ty7context6TyCtxtNtNtCshHhGpJjbd1P_18rustc_query_system9dep_graph10DepContext27try_load_from_on_disk_cache+0x40)[0x7f9674913040]
 0:06.80 /home/rbm/rust/lib/librustc_driver-2b06d4f7701cf2a7.so(+0x5681e8d)[0x7f9673bd6e8d]
 0:06.80 /home/rbm/rust/lib/librustc_driver-2b06d4f7701cf2a7.so(_RNvXs0_NtCs8OaLfcDXK2A_16rustc_query_impl13on_disk_cacheNtB5_11OnDiskCacheNtNtNtCsd62QaXPQfQz_12rustc_middle2ty7context11OnDiskCache20drop_serialized_data+0x14)[0x7f9673ac3d24]
 0:06.80 /home/rbm/rust/lib/librustc_driver-2b06d4f7701cf2a7.so(+0x595c81b)[0x7f9673eb181b]
 0:06.80 /home/rbm/rust/lib/librustc_driver-2b06d4f7701cf2a7.so(+0x593bada)[0x7f9673e90ada]
 0:06.80 /home/rbm/rust/lib/librustc_driver-2b06d4f7701cf2a7.so(+0x5954483)[0x7f9673ea9483]
 0:06.80 /home/rbm/rust/lib/librustc_driver-2b06d4f7701cf2a7.so(_RNvNtNtCsfNrynS0TUcA_17rustc_incremental7persist4save14save_dep_graph+0x12)[0x7f9673e689e2]
 0:06.80 /home/rbm/rust/lib/librustc_driver-2b06d4f7701cf2a7.so(+0xc17287)[0x7f966f16c287]
 0:06.80 /home/rbm/rust/lib/librustc_driver-2b06d4f7701cf2a7.so(+0xc36438)[0x7f966f18b438]
 0:06.80 /home/rbm/rust/lib/librustc_driver-2b06d4f7701cf2a7.so(+0xc1a506)[0x7f966f16f506]
 0:06.80 /home/rbm/rust/lib/librustc_driver-2b06d4f7701cf2a7.so(+0xcaac49)[0x7f966f1ffc49]
 0:06.80 /home/rbm/rust/lib/libstd-c5b9f4170faf6900.so(rust_metadata_std_dedd32e72e9060aa+0x8f4f3)[0x7f96786614f3]
 0:06.80 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8064)[0x7f966df34064]
 0:06.80 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f966dc6962d]
 0:06.98 error: could not compile `gkrust`
 0:06.98 Caused by:
 0:06.98   process didn't exit successfully: `/home/rbm/rust/bin/rustc --crate-name gkrust toolkit/library/rust/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type staticlib --emit=dep-info,link -C opt-level=2 -C panic=abort -C embed-bitcode=no -Clto --cfg 'feature="cubeb-remoting"' --cfg 'feature="cubeb_pulse_rust"' --cfg 'feature="gecko_profiler"' --cfg 'feature="gecko_profiler_parse_elf"' --cfg 'feature="glean_with_gecko"' --cfg 'feature="moz_memory"' --cfg 'feature="moz_places"' --cfg 'feature="simd-accel"' --cfg 'feature="webmidi_midir_impl"' --cfg 'feature="with_dbus"' -C metadata=b944790933c28ace -C extra-filename=-b944790933c28ace --out-dir /home/rbm/tor-browser/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -C linker=/home/rbm/tor-browser/build/cargo-linker -C incremental=/home/rbm/tor-browser/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/incremental -L dependency=/home/rbm/tor-browser/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/deps -L dependency=/home/rbm/tor-browser/obj-x86_64-pc-linux-gnu/release/deps --extern gkrust_shared=/home/rbm/tor-browser/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/deps/libgkrust_shared-54906c81a521dbec.rlib --extern lmdb_sys=/home/rbm/tor-browser/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/deps/liblmdb_sys-48e39fd79b04a389.rlib --extern mozglue_static=/home/rbm/tor-browser/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/deps/libmozglue_static-d649421a313d02ae.rlib --extern mozilla_central_workspace_hack=/home/rbm/tor-browser/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/deps/libmozilla_central_workspace_hack-1d224b27d5e9321d.rlib --extern swgl=/home/rbm/tor-browser/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/deps/libswgl-e13c1aba7b1ce90d.rlib -C debuginfo=2 --cap-lints warn -Cembed-bitcode=yes -C codegen-units=1 -L native=/usr/lib/x86_64-linux-gnu -L native=/home/rbm/tor-browser/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/build/audioipc-ac45fe0acb331ad6/out -L native=/home/rbm/tor-browser/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/build/audioipc2-25d4bc98d8e3e639/out -L native=/home/rbm/tor-browser/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/build/lmdb-rkv-sys-e0dbc2746796db32/out -L native=/home/rbm/tor-browser/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/build/mozglue-static-6aa4f545d3c1b6e8/out -L native=/usr/lib/x86_64-linux-gnu -L native=/home/rbm/tor-browser/obj-x86_64-pc-linux-gnu/dist/bin -L native=/home/rbm/tor-browser/obj-x86_64-pc-linux-gnu/security/nss/lib/nss/nss_nss3 -L native=/home/rbm/tor-browser/obj-x86_64-pc-linux-gnu/security/nss/lib/ssl/ssl_ssl3 -L native=/home/rbm/tor-browser/obj-x86_64-pc-linux-gnu/config/external/nspr/pr -L native=/home/rbm/tor-browser/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/build/swgl-b43fd4033fe6de8f/out` (signal: 11, SIGSEGV: invalid memory reference)
 0:07.00 /home/rbm/tor-browser/config/makefiles/rust.mk:429: recipe for target 'force-cargo-library-build' failed
 0:07.00 make[4]: *** [force-cargo-library-build] Error 101
 0:07.00 /home/rbm/tor-browser/config/recurse.mk:72: recipe for target 'toolkit/library/rust/target' failed
 0:07.00 make[3]: *** [toolkit/library/rust/target] Error 2
 0:07.00 /home/rbm/tor-browser/config/recurse.mk:32: recipe for target 'compile' failed
 0:07.00 make[2]: *** [compile] Error 2
 0:07.00 /home/rbm/tor-browser/config/rules.mk:352: recipe for target 'default' failed
 0:07.00 make[1]: *** [default] Error 2
 0:07.00 client.mk:63: recipe for target 'build' failed
 0:07.00 make: *** [build] Error 2
 0:07.02 350 compiler warnings present.

        So, at the moment I have the following ideas:

        1. try to compile again Rust in CentOS with the same options I used in Debian
        2. add the wrappers to force --gcc-toolchain
        3. use the clang binaries compiled on CentOS to compile Rust on Debian (to exclude bugs in LLVM)

        Please notice that CentOS 7 uses glibc 2.17, Debian Jessie glibc 2.18, so the binaries are CentOS -> Debian only, at the moment.

      • Pier Angelo Vendrame
        Pier Angelo Vendrame @pierov ·
        Author Maintainer

        try to compile again Rust in CentOS with the same options I used in Debian

        This worked, so I think we don't need to test 2.

        I think I could try the binaries I've produced on Debian again, after cleaning obj-whatever.

      • Pier Angelo Vendrame
        Pier Angelo Vendrame @pierov ·
        Author Maintainer

        On Debian we need also to use a combination of these vars (with the correct paths):

        export CC=clang
export CXX=clang++
export CFLAGS='--gcc-toolchain=/home/rbm/gcc'
export CXXFLAGS='--gcc-toolchain=/home/rbm/gcc'
export RUSTFLAGS="-L$HOME/gcc/lib64"
      • Pier Angelo Vendrame
        Pier Angelo Vendrame @pierov ·
        Author Maintainer

        Okay, I got a working build with my custom Rust!! :tada:

        This is the configure of Rust I eventually used:

        ./configure --prefix=$HOME/rust --enable-local-rust --enable-vendor --enable-full-tools --enable-sanitizers  --enable-profiler --disable-docs --disable-compiler-docs --set target.x86_64-unknown-linux-gnu.linker=clang  --set target.x86_64-unknown-linux-gnu.ar=llvm-ar --set target.x86_64-unknown-linux-gnu.ranlib=llvm-ranlib --set llvm.ninja=false --set rust.jemalloc --set rust.use-lld=true --llvm-root=$HOME/clang --sysconfdir=etc

        So, still --enable-full-tools (how did I even convince cmake to use the correct libstdc++?!).

        I'd love to get a build with --enable-extended, though, so we could also avoid --set llvm.ninja=false and maybe also --set rust.use-lld=true.

        Anyway, this was in a container, but not in tor-browser-build.

      • Pier Angelo Vendrame
        Pier Angelo Vendrame @pierov ·
        Author Maintainer

        This also worked:

        ./configure --prefix=$HOME/rust --enable-local-rust --enable-vendor --enable-extended --enable-sanitizers  --enable-profiler --disable-docs --disable-compiler-docs --set target.x86_64-unknown-linux-gnu.linker=clang  --set target.x86_64-unknown-linux-gnu.ar=llvm-ar --set target.x86_64-unknown-linux-gnu.ranlib=llvm-ranlib --set rust.jemalloc --llvm-root=$HOME/clang --sysconfdir=etc

        I guess the previous segmentation fault was due to me not cleaning the obj-... directory.

        Anyway, we could keep --enable-sanitizers, and remove --enable-profiler.

        But I'll remove both for a test, so that the only difference with our current configuration will be only the additional tools from LLVM and jemalloc (our current config with --llvm-root still leads the build not ending).

      • Pier Angelo Vendrame
        Pier Angelo Vendrame @pierov ·
        Author Maintainer

        Okay, I think we have the culprit: jemalloc. Without it, the compilation never ends.

        These are the latest flags I've tried:

        ./configure --prefix=$HOME/rust --sysconfdir=etc --enable-local-rust --enable-vendor --enable-extended --disable-docs --disable-compiler-docs --target=x86_64-unknown-linux-gnu,i686-unknown-linux-gnu --set target.x86_64-unknown-linux-gnu.linker=clang --set target.x86_64-unknown-linux-gnu.ar=llvm-ar --set target.x86_64-unknown-linux-gnu.ranlib=llvm-ranlib --set target.i686-unknown-linux-gnu.linker=clang --set target.i686-unknown-linux-gnu.ar=llvm-ar --set target.i686-unknown-linux-gnu.ranlib=llvm-ranlib --llvm-root=$HOME/clang

        (unless it's the 32-bit target).

      • Pier Angelo Vendrame
        Pier Angelo Vendrame @pierov ·
        Author Maintainer

        Yep, it was jemalloc, this completes a build, too:

        ./configure --prefix=$HOME/rust --sysconfdir=etc --enable-local-rust --enable-vendor --enable-extended --disable-docs --disable-compiler-docs --target=x86_64-unknown-linux-gnu,i686-unknown-linux-gnu --llvm-root=$HOME/clang --set rust.jemalloc
      • Please register or sign in to reply
    • Pier Angelo Vendrame marked this issue as related to #40595 (closed)

      marked this issue as related to #40595 (closed)

    • Pier Angelo Vendrame mentioned in merge request !503 (merged)

      mentioned in merge request !503 (merged)

    • Gitolite Merge Bot closed with merge request !503 (merged)

      closed with merge request !503 (merged)

    • Pier Angelo Vendrame marked this issue as related to #40589 (closed)

      marked this issue as related to #40589 (closed)

    • Pier Angelo Vendrame removed Doing label

      removed Doing label

    Please register or sign in to reply