Be smarter about vendoring for Rust projects

In legacy/trac#30490 (moved) we added cbindgen to our projects which resulted in a large extra archive with all the vendored stuff that is specified. However, we probably don't need a lot of that stuff (like all the big Windows .a files). We should figure out a smarter way of handling that than just shipping everything.

added TorBrowserTeam202004 in Legacy / Trac component::applications/tor browser in Legacy / Trac gitlab-tb-tor-browser-build in Legacy / Trac owner::tbb-team in Legacy / Trac priority::medium in Legacy / Trac reviewer::sysrqb in Legacy / Trac severity::normal in Legacy / Trac status::new in Legacy / Trac tbb-maint in Legacy / Trac tbb-rbm in Legacy / Trac type::task in Legacy / Trac labels

legacy/trac#32436 (moved) has similar problems. So, let's be more generic and make this ticket into getting to a proper vendoring process/strategy.

Trac:
Summary: Be smarter about vendoring for cbindgen to Be smarter about vendoring for Rust projects

If Mozilla is vendoring most of the rust dependencies we need, I think we could generate a tarball from tor-browser.git/third_party/rust, and include it in the cbindgen project, and in other rust projects that we build. That way we don't need to manually update a tarball of vendored projects.

If we want to avoid generating a new tarball (which will probably involve generating a tarball containing all the firefox tree, extracting it, and generating a new tarball from the third_party/rust directory only), we can re-use the src-firefox-$version.tar.xz tarball which we already generate.

Replying to boklm:

If Mozilla is vendoring most of the rust dependencies we need, I think we could generate a tarball from tor-browser.git/third_party/rust, and include it in the cbindgen project, and in other rust projects that we build. That way we don't need to manually update a tarball of vendored projects.

If we want to avoid generating a new tarball (which will probably involve generating a tarball containing all the firefox tree, extracting it, and generating a new tarball from the third_party/rust directory only), we can re-use the src-firefox-$version.tar.xz tarball which we already generate.

Yeah, I've thought about that. The problem here is, though, those the projects in question are external ones which are not needed to build Mozilla's Rust code. Thus, Mozilla has not vendored them in but has them rather as a build dependency. For cbindgen we could think about using Debian packages at least once we move don't have versions anymore where cbindgen is not shipped for in a sufficiently recent version. For lucetc this option is not available.

Replying to gk:

Replying to boklm:

If Mozilla is vendoring most of the rust dependencies we need, I think we could generate a tarball from tor-browser.git/third_party/rust, and include it in the cbindgen project, and in other rust projects that we build. That way we don't need to manually update a tarball of vendored projects.

If we want to avoid generating a new tarball (which will probably involve generating a tarball containing all the firefox tree, extracting it, and generating a new tarball from the third_party/rust directory only), we can re-use the src-firefox-$version.tar.xz tarball which we already generate.

Yeah, I've thought about that. The problem here is, though, those the projects in question are external ones which are not needed to build Mozilla's Rust code. Thus, Mozilla has not vendored them in but has them rather as a build dependency. For cbindgen we could think about using Debian packages at least once we move don't have versions anymore where cbindgen is not shipped for in a sufficiently recent version. For lucetc this option is not available.

Hmm, I don't understand what you mean here. When I look at the content of cbindgen-vendor.tar.bz2, all the directories included in it are also present in tor-browser.git/third_party/rust. Or are you talking about other projects than cbindgen?

Replying to boklm:

Replying to gk:

Replying to boklm:

If Mozilla is vendoring most of the rust dependencies we need, I think we could generate a tarball from tor-browser.git/third_party/rust, and include it in the cbindgen project, and in other rust projects that we build. That way we don't need to manually update a tarball of vendored projects.

If we want to avoid generating a new tarball (which will probably involve generating a tarball containing all the firefox tree, extracting it, and generating a new tarball from the third_party/rust directory only), we can re-use the src-firefox-$version.tar.xz tarball which we already generate.

Yeah, I've thought about that. The problem here is, though, those the projects in question are external ones which are not needed to build Mozilla's Rust code. Thus, Mozilla has not vendored them in but has them rather as a build dependency. For cbindgen we could think about using Debian packages at least once we move don't have versions anymore where cbindgen is not shipped for in a sufficiently recent version. For lucetc this option is not available.

Hmm, I don't understand what you mean here. When I look at the content of cbindgen-vendor.tar.bz2, all the directories included in it are also present in tor-browser.git/third_party/rust. Or are you talking about other projects than cbindgen?

I am talking right now about cbindgen and lucetc. But the specific third_party/rust part I had in mind is only applying to the former.

Yes, the packages are there but not all the versions are the same as cargo vendor gives us. This, the risk is high that either compilation fails or some other weird behavior would happen. If the packages available and the versions they are in matched, I agree, using the code from third_party/rust would work. But, alas, that's not the case.

If we trust that cargo vendor will provide reproducible output, and is correctly checking downloaded files, then maybe we could add some step in the build where we allow network access in the container, in order to run cargo vendor and generate a vendor tarball.

Marking as triaged, but not putting it on the roadmap yet.

Trac:
Keywords: N/A deleted, TorBrowserTeamTriaged added

Replying to boklm:

If we trust that cargo vendor will provide reproducible output, and is correctly checking downloaded files, then maybe we could add some step in the build where we allow network access in the container, in order to run cargo vendor and generate a vendor tarball.

Moving this to February. If we make the above assumptions (and they need to be confirmed), then after legacy/trac#28325 (moved) is solved this ticket should (hopefully) follow soon after.

Trac:
Keywords: TorBrowserTeamTriaged deleted, TorBrowserTeam202002 added

Release Train Migration.

Trac:
Keywords: N/A deleted, ReleaseTrainMigration added

We are no longer in February, moving tickets

Trac:
Keywords: TorBrowserTeam202002 deleted, TorBrowserTeam202003 added

We are no longer in March

Trac:
Keywords: TorBrowserTeam202003 deleted, TorBrowserTeam202004 added

Trac:
Status: new to assigned
Sponsor: N/A to Sponsor58
Reviewer: N/A to sysrqb
Owner: tbb-team to boklm

Trac:
Cc: N/A to tbb-team

Trac:
Parent: N/A to legacy/trac#33659 (moved)

Release all this tickets back into tbb-team.

Trac:
Owner: boklm to tbb-team

Trac:
Status: assigned to new
Keywords: ReleaseTrainMigration deleted, tbb-maint added
Sponsor: Sponsor58 to N/A
Parent: legacy/trac#33659 (moved) to N/A

Add magic gitlab keyword.

Trac:
Keywords: N/A deleted, gitlab-tb-tor-browser-build added

moved from legacy/trac#31588 (moved)

added Task label and removed 1 deleted label

added Maintenance label and removed 1 deleted label

I'm wondering if we could make cargo vendor log the URLs from files it is downloading, to be able generate something similar to gradle-dependencies-list.txt.

added Next label

added Icebox label

removed Next label

removed Icebox label

added Q2 label

added Backlog label

added Q3 label and removed Q2 label

removed Q3 label

added Roadmap::Future label

removed Backlog label

At the reproducible build summit I have been suggested to have a look at the Cargo.lock file and cargo fetch command that can use it.

But we did not try to hack on, so I am not sure it will solve our problem.

We had a quick look at the format of the Cargo.lock and it references the crates registry, so using cargo fetch might just be the easiest solution.

If needed, we could create a dependency tarball on our machines first, then version the resulting hash, in case we want to be more sure about the integrity.

added Build System label

assigned to @pierov

removed Roadmap::Future label

added Doing FF115-esr labels

marked this issue as related to #40855 (closed)

mentioned in merge request !738 (merged)

marked this issue as related to #40848 (closed)

added Needs Review label and removed Doing label

removed the relation with #40848 (closed)

removed Needs Review label

In !762 (merged) I moved application-services and cbindgen to the new pkg_type: cargo_vendor that creates itself automatically with cargo vendor --locked vendor. So far it worked everywhere, even though I created the hashes a while ago (e.g., tb-build-06 built the vendor packages today for the first time, and it succeeded).

uniffi-rs still uses an old archive hosted in people.tpo. However, we build an old version (0.7.0), and application services include a more recent one (0.23.0) in their Rust dependencies. So, I will try to build application services without uniffi-rs, and if it succeeds I will open an issue to drop the project and close this one as solved.

Also, the patch we used to bundle has been upstreamed: https://github.com/mozilla/uniffi-rs/commit/621a84fad6cb5449afda779eaef59692d3e8420c

Opened #40917 (closed) to follow on this.

marked this issue as related to #40901 (closed)

closed

marked this issue as related to #40965 (closed)

marked this issue as related to #40966 (closed)