reproducible builds without shm

fix #614 (closed)

I've determined the two source of nondeterminism in Arti builds are lzma-sys and zstd-sys.

There was a PR, zstd-rs!162, making zstd-sys reproducible. Sadly there isn't a release yet, so it's available by git dependency only.
I made a similar PR on xz2-rs, xz2-rs!102, to fix the same issue with lzma-sys. I was able to verify it indeed fix reproducibility issues. As of now, it's only available via git dependency too.

requested review from @nickm

Thanks for working on this! I'm happy for us to upgrade once the patches are upstream. (I'm guessing that this branch is here so you can test; please let me know if you want earlier comment.)

this MR is there so 1/ people know #614 is being worked on 2/ I don't forget I'm working on it (which for the record is what happened with !819 (merged), I just found I had that branch from a month ago).

Feel free to have a look, but there is honestly not much to look at

added 2 commits

• 08c19b28 - use future version of zstd and lzma
• afafeefc - always run build-repro (for tests purpose)

Compare with previous version

we got a new version of lzma-sys! Now we just need a release of zstd-sys and this will be ready

added 366 commits

• afafeefc...8cf9e2a6 - 364 commits from branch tpo/core:main
• 3d5d4ada - use newer version of zstd and lzma
• 9d741b07 - always run build-repro (for tests purpose)

Compare with previous version

marked this merge request as ready

expected result (tested in CI and on my own machine, but more confirmation is always better):

branch       : repro-less-shm
commit       : 3d5d4ada0190ac935b3786d9d36614c75cb5fec6
Linux hash   : 2c534238d69da28e4222ad351447b8e3bbf69259dd8b334b650ab6fe8ed2db80
Windows hash : 733f4fcf6c0e3b10890c700e77a556abdff2a3bbdcd703c5c61dad547dd1286e
MacOS hash   : 551d5961dee3adbe44e4860184c6210b2c622245d1f3fab9355cc522db1c939d

No objection from me.

enabled an automatic merge when the pipeline for 3d5d4ada succeeds

merged

mentioned in commit e317c0ad

mentioned in issue tpo/tpa/gitlab#135 (closed)