Confirm Tor Project tor.git package builds are reproducible
Our deb.torproject.org debs are now built on Gitlab (and possibly our RPMs too?). I believe the deb.torproject.org packages come from https://gitlab.torproject.org/tpo/core/debian/tor/-/pipelines, but @weasel will have to confirm. I am less sure where the RPMs come from. @kushal will have to let us know.
We should confirm that these packages are reproducible. If they are based on the Debian build system, I believe they should be. And tor.git itself might be reproducible by default. However, @weasel was not sure if this was the case, and @ahf and I were not either.
It would be a sad day if Gitlab 0day got someone the whole Tor network, and reproducible builds prevent this possibility, and also allow us to check this after-the-fact.
So I am creating this ticket to check reproducibility, and then fix any issues.
I think the best way forward is to export the gitlab runner script into a standard docker container and see if the sha256sums from https://deb.torproject.org/torproject.org/pool/main/t/tor/ match from the resulting debs from said docker container, or even just a build on a random debian machine. But there may be other ways. I know @jnewsome is good at spinning up ephemeral runners, so that could be another route too, but that does not fully eliminate Gitlab from the picture.
If the test build's sha256sums do not match against the ones in https://deb.torproject.org/torproject.org/pool/main/t/tor/, we may want to use @boklm's RBM tool in our gitlab tor.git build runners, to make it reproducible. (RBM is what Tor Browser now uses, to produce reproducible Tor Browser releases.)
Cc: @boklm, @jnewsome, @kushal, @ahf, @dgoulet, @weasel, @anarcat, @lavamind
Activity
Adding the conversation I had with weasel here:
170522 20:45:24 ahf: weasel: yo yo, is the packages for debian stable on deb.torproject.org build reproducibly? 200522 13:21:09 weasel: ahf: I haven't heard that it isn't, at least 200522 14:42:46 ahf: weasel: cool! i just couldn't figure out what was up and down with the gitlab CI runners. you still build things locally, right? 200522 14:51:35 weasel: ahf: the most recent backports uploads I built manually 200522 14:52:08 weasel: (but I don't upload binaries to debian.org anyway -- unless in the weird corner cases where that is required) 200522 14:52:22 ahf: ya 200522 14:52:26 ahf: but this was for deb.torproject.org i meant 200522 14:52:28 ahf: not debian itself 200522 14:52:54 weasel: the deb.tpo binaries usually come from gitlab 200522 14:53:21 ahf: ah-ha, so they are probably *not* reproducible here? 200522 14:53:30 weasel: unsure 200522 14:53:34 ahf: ack, thank you! 200522 14:53:34 weasel: why wouldn't they? 200522 14:53:51 ahf: i do not know, it was my impression there was a lot of infrastructure around the goal of building things reproducibly 200522 14:54:02 ahf: but maybe the debian tooling takes care of that 200522 14:54:26 ahf: my biggest experience with this stuff is via TBB's nice "rbm" tool that does all the hard work for me (well, it's the team that does the hard work, but i can just run their stuff) 200522 14:54:27 weasel: I haven't played with it a lot 200522 14:54:51 weasel: but I suspect that given the right metainfo, one could build a second set of binaries that match the ones on deb.tpo 200522 14:55:02 ahf: ya 200522 14:55:04 weasel: now, i'm not sure we actually collect the right metainfo 200522 14:55:29 ahf: does debian.org have some fancy queue system that builds the binary packages from scm automatically? 200522 14:55:35 weasel: no 200522 14:55:52 weasel: things get built from source packages 200522 14:56:37 weasel: how you get to those is a different beast. plenty of methods 200522 14:57:41 ahf: ack 200522 14:59:36 ahf: thanks for the info!Times are UTC.
I know @jnewsome is good at spinning up ephemeral runners
Here's the "2-liner" I have saved for spinning up extra runners for shadow simulations:
sudo gitlab-runner register --non-interactive --url=https://gitlab.torproject.org/ --registration-token=<scrubbed> --docker-shm-size=100000000000 --docker-security-opt=seccomp:unconfined --executor=docker --docker-image=ubuntu:latest --output-limit 100000 --tag-list=shadow --docker-volumes=/srv/shadow:/shadow sudo service gitlab-runner startWe probably don't need some of those options for the build job. I'm guessing what we need is more like:
sudo gitlab-runner register --non-interactive --url=https://gitlab.torproject.org/ --registration-token=<scrubbed> --executor=docker --docker-image=ubuntu:latest --tag-list=my-ephemeral-runner-tag sudo service gitlab-runner startThe registration token comes from the Settings -> CI/CD page in the gitlab UI.
Otoh a compromised gitlab could insert a
- patch < backdoor.diffline into the script that gets sent to the runner. Ideally we'd figure out a way to audit what script the runner actually executes to rule out this sort of attack.Happy to take a shot at reproducing that way from a runner on my desktop if you like. We may need to edit the workflow to allow dynamically choosing a runner tag (which we can in turn use to pick a specific runner).
Happy to take a shot at reproducing that way from a runner on my desktop if you like. We may need to edit the workflow to allow dynamically choosing a runner tag (which we can in turn use to pick a specific runner).
How easy is it to use this process to extract a script that can be run on a regular docker container, or on a bare metal machine? Is that a feasible way to quickly rule out gitlab from this process?
-
How easy is it to use this process to extract a script that can be run on a regular docker container, or on a bare metal machine? Is that a feasible way to quickly rule out gitlab from this process?
I don't know of a way to do that.
It looks like .gitlab-ci.yml already delegates most of the work to ci-driver.sh. I think it already shouldn't be too difficult to arrange to invoke that script directly via Docker.
I think the main thing missing from ci-driver.sh is installing system packages. Btw afaict that itself is not done in a reproducible way - it just gets the latest version of the debian packages. I think we'd need to find some way to ensure we get the exact same versions when reproducing. It'd be nice if apt-get provided a way to say "install the versions that were the latest at time X" but I don't see such a feature. Maybe in the build we could generate a manifest of exactly which versions get installed, and optionally take and use such a manifest (when reproducing)?
Happy to take a shot at reproducing that way from a runner on my desktop if you like. We may need to edit the workflow to allow dynamically choosing a runner tag (which we can in turn use to pick a specific runner).
How easy is it to use this process to extract a script that can be run on a regular docker container, or on a bare metal machine? Is that a feasible way to quickly rule out gitlab from this process?
If there is a script like this, then I think it would be quite easy to add something to
tor-browser-build.gitwhere amake tor-deb-matchwould build a package inside a container (using the same system we use for building Tor Browser), and compare it with the package on deb.torproject.org.-
I think the main thing missing from ci-driver.sh is installing system packages. Btw afaict that itself is not done in a reproducible way - it just gets the latest version of the debian packages. I think we'd need to find some way to ensure we get the exact same versions when reproducing. It'd be nice if apt-get provided a way to say "install the versions that were the latest at time X" but I don't see such a feature. Maybe in the build we could generate a manifest of exactly which versions get installed, and optionally take and use such a manifest (when reproducing)?
Unfortunately it's not easy to get always exactly the same package versions installed, since old versions of the packages are removed from mirrors when there is a new debian point release.
It would be possible to use snapshot.debian.org to get packages from a certain date, but it's rate limited so doesn't work very well.
It seems Qubes people have been working on their own snapshot service: https://www.qubes-os.org/news/2021/10/08/reproducible-builds-for-debian-a-big-step-forward/
However, not having exactly the same packages is usually not a problem since most updates don't affect build outputs. It can be a problem when there is a gcc update, but it doesn't happen very often in stable releases.
For Tor Browser we have this ticket (which is not fixed yet): tor-browser-build#28102
-
Maybe the gitlab workflow could include all downloaded deb packages (which are signed?) as an additional artifact, and those could be verified and used when reproducing?
Possibly at a much later step. It is likely not necessary for this purpose. In my experience with the Tor Browser gitian build system, even years ago it was never the case that package updates influenced things. In fact, for something like C-Tor, I bet it will only happen if the entire base system and toolchain upgrades to a new version, or if there is an emergency compiler bugfix.
If it is still looking tricky to reproduce the build in a standalone setup, let's just try the ephemeral Gitlab runner idea you had in #40615 (comment 2805974) and re-run the job on a new runner. That will eliminate much of this uncertainty if that matches, and if it does not match, we also will have someplace concrete to start.
Edited by Mike Perry -
Reproducing this way requires the ability to register a runner in the project.
Oh do you need @weasel to add you to the project to do #40615 (comment 2805974)? Or can you add yourself?
-
It looks like I have the capability to add a runner - I went ahead and added one.
I'd have to change the CI script a bit to allow overriding the runner tag, so that I could force the x86 build to happen on my runner.
I'll probably try just using Docker directly first; I think it should be pretty easy. If it's not I'll send a PR with the changes needed to the CI script.
-
So I hacked up reproduce.sh. It's basically a copy-paste of the gitlab yml (
debian/.debian-ci.yml), fed through Docker. It's currently hard-coded to build branch debian-0.4.6 for amd64. It's made to be executed from a checkout of debian/tor repo. It mounts the current directory, in the docker containers, creating directories source-packages and binary-packages.The last build job for the debian-0.4.6 branch, which this would hopefully reproduce, is https://gitlab.torproject.org/tpo/core/debian/tor/-/jobs/131635. i.e. the artifacts for that job ought to match the
binary-packagesdirectory that we create.Unfortunately it doesn't.
Taking a step back, the build_source artifacts probably also ought to match the generated
source-packages, but that differs as well. We probably need to make this step reproducible first. -
Trying again from e6683856 (the version of the debian/tor repo that the job I'm try to reproduce ran at)
Edited by Jim Newsome -
Trying again from e6683856 (the version of this repo that the job I'm try to reproduce ran at)
That seems to have fixed at least some of the differences in the source artifacts. There are some different paths, and different paths inside the patches, but those might not affect the final output. (I haven't checked thoroughly whether there are other differences)
diffing the buildinfo files in what's ultimately built though, it looks like we are ending up with different versions of libssl and libxml; i.e. it looks like they updated in debian in the meantime.
$ diff artifacts/binary-packages/debian-amd64-bullseye/*.buildinfo binary-packages/debian-amd64-bullseye/*.buildinfo 5c5 < Version: 0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1 --- > Version: 0.4.6.10-dev-20220525T203017Z-1~d11.bullseye+1 7,9c7,9 < f612e3a60859452beccc86a172062954 5329608 tor-dbgsym_0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1_amd64.deb < bff1c161fc10ec1b851b3cdcf7f91286 1445460 tor-geoipdb_0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1_all.deb < a8613c81c74ccac2e4c4e768ec5a8a0c 1984768 tor_0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1_amd64.deb --- > 4a0ab0141bcf5358bd903224da01e266 5329564 tor-dbgsym_0.4.6.10-dev-20220525T203017Z-1~d11.bullseye+1_amd64.deb > 3e5d05e30a4a13e0f452004c57d36996 1445184 tor-geoipdb_0.4.6.10-dev-20220525T203017Z-1~d11.bullseye+1_all.deb > 0537e37832084ec2a4b4fbd6610c1d99 1984600 tor_0.4.6.10-dev-20220525T203017Z-1~d11.bullseye+1_amd64.deb 11,13c11,13 < aaba305c08ae491ec937833bb25ca95e52168b32 5329608 tor-dbgsym_0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1_amd64.deb < e846a5c588279a078e3969b28bdf2d658e3932ad 1445460 tor-geoipdb_0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1_all.deb < d8f46299535ad84ac35ee4a126237d2192d60d6d 1984768 tor_0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1_amd64.deb --- > 0df75d7badca1efc8bfe95315c66d8a573a52605 5329564 tor-dbgsym_0.4.6.10-dev-20220525T203017Z-1~d11.bullseye+1_amd64.deb > fbc33eb713f3d3657573d0693ac781a6825aa3c1 1445184 tor-geoipdb_0.4.6.10-dev-20220525T203017Z-1~d11.bullseye+1_all.deb > 5663e4558f2a5d43d4f2671f00d2912b63686a0d 1984600 tor_0.4.6.10-dev-20220525T203017Z-1~d11.bullseye+1_amd64.deb 15,17c15,17 < 923aca0a5aefa0c8475e06e98fb737fc6b34db59843f8e73338582ed6dea5182 5329608 tor-dbgsym_0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1_amd64.deb < 825c5a024d69f1cd37f6b62761e1d39123752f594348877316892b8b7866b743 1445460 tor-geoipdb_0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1_all.deb < bc9fa4bbefb633313ad4c6688b7740c12d6080161d65b06f6dcd8cf8e7d163c2 1984768 tor_0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1_amd64.deb --- > b58ccb6a0644f1c371b4bebf2e180c566ff2de66c7d0c40308caff2f1be81b68 5329564 tor-dbgsym_0.4.6.10-dev-20220525T203017Z-1~d11.bullseye+1_amd64.deb > 1a174fd16e2be4ad28450b61ce4ff33b4d570d507fca83e66730c5512a3652fa 1445184 tor-geoipdb_0.4.6.10-dev-20220525T203017Z-1~d11.bullseye+1_all.deb > b4d94961e554a22d0e148684ec16a94f831f182817415e56381a4792cd917d94 1984600 tor_0.4.6.10-dev-20220525T203017Z-1~d11.bullseye+1_amd64.deb 20c20,21 < Build-Date: Mon, 09 May 2022 16:33:14 +0000 --- > Build-Date: Wed, 25 May 2022 20:37:33 +0000 > Build-Path: /build/build-env/build-tree/tor-0.4.6.10-dev-20220525T203017Z 54,55c55,56 < dpkg (= 1.20.9), < dpkg-dev (= 1.20.9), --- > dpkg (= 1.20.10), > dpkg-dev (= 1.20.10), 100c101 < libdpkg-perl (= 1.20.9), --- > libdpkg-perl (= 1.20.10), 163,164c164,165 < libssl-dev (= 1.1.1n-0+deb11u1), < libssl1.1 (= 1.1.1n-0+deb11u1), --- > libssl-dev (= 1.1.1n-0+deb11u2), > libssl1.1 (= 1.1.1n-0+deb11u2), 183,184c184,185 < libxml2 (= 2.9.10+dfsg-6.7+deb11u1), < libxml2-utils (= 2.9.10+dfsg-6.7+deb11u1), --- > libxml2 (= 2.9.10+dfsg-6.7+deb11u2), > libxml2-utils (= 2.9.10+dfsg-6.7+deb11u2), 225,226c226,227 < DEB_BUILD_OPTIONS="parallel=8" < SOURCE_DATE_EPOCH="1652112856" --- > DEB_BUILD_OPTIONS="parallel=12" > SOURCE_DATE_EPOCH="1653510667" -
Cleaned up the script a little bit; maybe I should check this in somewhere? reproduce.sh. I fixed the image for the build step to use debian-bullseye instead of debian-stable, but afaik they're equivalent, and I still got the diff above.
-
I tried running my script twice; that also ends up with different checksums of the final deb files:
$ diff -u binary-packages/debian-amd64-bullseye/*.buildinfo binary-packages-v1/debian-amd64-bullseye/*.buildinfo --- binary-packages/debian-amd64-bullseye/tor_0.4.6.10-dev-20220525T205701Z-1~d11.bullseye+1_amd64.buildinfo 2022-05-25 16:00:23.879326754 -0500 +++ binary-packages-v1/debian-amd64-bullseye/tor_0.4.6.10-dev-20220525T191131Z-1~d11.bullseye+1_amd64.buildinfo 2022-05-25 14:37:53.872782197 -0500 @@ -2,23 +2,23 @@ Source: tor Binary: tor tor-dbgsym tor-geoipdb Architecture: all amd64 -Version: 0.4.6.10-dev-20220525T205701Z-1~d11.bullseye+1 +Version: 0.4.6.10-dev-20220525T191131Z-1~d11.bullseye+1 Checksums-Md5: - e46af438a0ebabf2516081d43e172a48 5329568 tor-dbgsym_0.4.6.10-dev-20220525T205701Z-1~d11.bullseye+1_amd64.deb - 01ac0b96f92fe862cc53ea7c6282a828 1438224 tor-geoipdb_0.4.6.10-dev-20220525T205701Z-1~d11.bullseye+1_all.deb - 10ca990f940d74c6e8ad11bafdd75d24 1985064 tor_0.4.6.10-dev-20220525T205701Z-1~d11.bullseye+1_amd64.deb + 542c06944f8ffef8fb46afde07fe194c 5329608 tor-dbgsym_0.4.6.10-dev-20220525T191131Z-1~d11.bullseye+1_amd64.deb + 04bb25170c920ce1456814372464dcc3 1443872 tor-geoipdb_0.4.6.10-dev-20220525T191131Z-1~d11.bullseye+1_all.deb + c99164b196d902bda15a075000817038 1985516 tor_0.4.6.10-dev-20220525T191131Z-1~d11.bullseye+1_amd64.deb Checksums-Sha1: - 9770c7bb745a6be473adfda6617c6ab6efc78bf5 5329568 tor-dbgsym_0.4.6.10-dev-20220525T205701Z-1~d11.bullseye+1_amd64.deb - 574ec6dcc59c8e1d0831833f97cb1e6509a5fed7 1438224 tor-geoipdb_0.4.6.10-dev-20220525T205701Z-1~d11.bullseye+1_all.deb - 2d407f46147e3ccf2eb65a9bdf08cbdfe21063fd 1985064 tor_0.4.6.10-dev-20220525T205701Z-1~d11.bullseye+1_amd64.deb + 5b530ef8bd6f0cdb80e47b794ea26ef6cc0448f0 5329608 tor-dbgsym_0.4.6.10-dev-20220525T191131Z-1~d11.bullseye+1_amd64.deb + 47485b3131ae024d387967c912d808726991b013 1443872 tor-geoipdb_0.4.6.10-dev-20220525T191131Z-1~d11.bullseye+1_all.deb + 157681c4dcce586bc8edd2791af1c6c88c622b1d 1985516 tor_0.4.6.10-dev-20220525T191131Z-1~d11.bullseye+1_amd64.deb Checksums-Sha256: - 6c4139f9349d8d5c90e2b06efc02b304f55395f53dbcb88b1f34f3dc2bd7ed1a 5329568 tor-dbgsym_0.4.6.10-dev-20220525T205701Z-1~d11.bullseye+1_amd64.deb - 59565b88d6d0543b89283a1715162a3a296ab5954b69f87c0fd1b0c7da566c0b 1438224 tor-geoipdb_0.4.6.10-dev-20220525T205701Z-1~d11.bullseye+1_all.deb - 8aa5437f98ee9231525b1e822f8e6b1bf5a694ded790d3e685c8e2f9c70a4e1e 1985064 tor_0.4.6.10-dev-20220525T205701Z-1~d11.bullseye+1_amd64.deb + 14a779bab401c4bba36d5ea74ee880c2844e751505a035a3321ad515e6877c45 5329608 tor-dbgsym_0.4.6.10-dev-20220525T191131Z-1~d11.bullseye+1_amd64.deb + 20e3af9f950f0009d6ba82297af197491f4dc5e9e94f8ca9956dd2817d3ba529 1443872 tor-geoipdb_0.4.6.10-dev-20220525T191131Z-1~d11.bullseye+1_all.deb + 78abc805b48e29c2beed76debe92a498c24e9a6567c64466089662864ab17162 1985516 tor_0.4.6.10-dev-20220525T191131Z-1~d11.bullseye+1_amd64.deb Build-Origin: Debian Build-Architecture: amd64 -Build-Date: Wed, 25 May 2022 21:00:23 +0000 -Build-Path: /build/build-env/build-tree/tor-0.4.6.10-dev-20220525T205701Z +Build-Date: Wed, 25 May 2022 19:37:53 +0000 +Build-Path: /build/build-env/build-tree/tor-0.4.6.10-dev-20220525T191131Z Installed-Build-Depends: asciidoc (= 9.0.0~rc2-1), asciidoc-base (= 9.0.0~rc2-1), @@ -224,4 +224,4 @@ zlib1g-dev (= 1:1.2.11.dfsg-2+deb11u1) Environment: DEB_BUILD_OPTIONS="parallel=12" - SOURCE_DATE_EPOCH="1653512273" + SOURCE_DATE_EPOCH="1653505943" -
Did more cleanup of the script and checked into a personal repo: https://gitlab.torproject.org/jnewsome/reproduce-tor-debian-build
I also unpacked the debs and found that the tor binaries are the same, though I'm a bit surprised that's the case given the library version differences. Details in the README.
-
Added another layer of wrapping: https://gitlab.torproject.org/jnewsome/reproduce-tor-debian-build/-/blob/main/reproduce_pipeline.py takes a CI pipeline number and tries to reproduce the build from that pipeline and validate that it ultimately generates the same tor binary as the pipeline did.
So I think we've ~passed our sanity check that it's possible (though maybe we should check more then just the
torbinary itself?). We should figure out if this reproduce + validate workflow is something we want to maintain and how. -
Poked around a bit more. It looks like the remaining differences are primarily timestamps. If we want the whole .deb to be reproducible, I think we need to run with a frozen clock (e.g. using
faketime) and then use the same time when reproducing.$ for f in `(cd artifacts-38207/ && find usr -type f)`; do diff artifacts-38207/$f reproduce-38207/$f; done Binary files artifacts-38207/usr/share/doc/tor/changelog.Debian.gz and reproduce-38207/usr/share/doc/tor/changelog.Debian.gz differ Binary files artifacts-38207/usr/share/man/man8/tor-instance-create.8.gz and reproduce-38207/usr/share/man/man8/tor-instance-create.8.gz differ$ diff <(gunzip -c artifacts-38207/usr/share/doc/tor/changelog.Debian.gz) <(gunzip -c reproduce-38207/usr/share/doc/tor/changelog.Debian.gz) 1c1 < tor (0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1) tor-nightly-0.4.6.x-bullseye; urgency=medium --- > tor (0.4.6.10-dev-20220526T191917Z-1~d11.bullseye+1) tor-nightly-0.4.6.x-bullseye; urgency=medium 5c5 < -- Peter Palfrader <build@runner-y9surhn-project-1218-concurrent-0> Mon, 09 May 2022 16:14:16 +0000 --- > -- Peter Palfrader <build@2b183bfec45f> Thu, 26 May 2022 19:20:07 +0000 7c7 < tor (0.4.6.10-dev-20220509T161328Z-1) tor-nightly-0.4.6.x; urgency=medium --- > tor (0.4.6.10-dev-20220526T191917Z-1) tor-nightly-0.4.6.x; urgency=medium 9c9 < * Automated build of tor-nightly at 20220509T161328Z, git revision --- > * Automated build of tor-nightly at 20220526T191917Z, git revision 12c12 < -- Peter Palfrader <build@runner-y9surhn-project-1218-concurrent-0> Mon, 09 May 2022 16:14:12 +0000 --- > -- Peter Palfrader <build@2b183bfec45f> Thu, 26 May 2022 19:20:04 +0000$ diff <(gunzip -c artifacts-38207/usr/share/man/man8/tor-instance-create.8.gz) <(gunzip -c reproduce-38207/usr/share/man/man8/tor-instance-create.8.gz) 5c5 < .\" Date: 05/09/2022 --- > .\" Date: 05/26/2022 10c10 < .TH "TOR\-INSTANCE\-CREAT" "8" "05/09/2022" "Tor" "Tor Manual" --- > .TH "TOR\-INSTANCE\-CREAT" "8" "05/26/2022" "Tor" "Tor Manual"Edited by Jim Newsome 
first off, if you're poking around artifacts to try to figure out why they differ, i strongly encourage you try out the "diffoscope" tool written by the reproducible builds people.
the other thing you have there is that yes, there's something that injects a new changelog entry in your build system. i'm not sure what that is, but that is of course bound to generate a diff.
that thing should respect the SOURCE_EPOCH variable so that it can be run reproducibly, without having to mess with LD_PRELOAD (which is what faketime does, i believe).
...
On 2022-05-26 20:36:05, Jim Newsome (@jnewsome) wrote:
Jim Newsome commented on a discussion: #40615 (comment 2808087)
Poked around a bit more. It looks like the remaining differences are primarily timestamps. If we want the whole .deb to be reproducible, I think we need to run with a frozen clock (e.g. using
faketime) and then use the same time when reproducing.$ for f in `(cd artifacts-38207/ && find usr -type f)`; do diff artifacts-38207/$f reproduce-38207/$f; done Binary files artifacts-38207/usr/share/doc/tor/changelog.Debian.gz and reproduce-38207/usr/share/doc/tor/changelog.Debian.gz differ Binary files artifacts-38207/usr/share/man/man8/tor-instance-create.8.gz and reproduce-38207/usr/share/man/man8/tor-instance-create.8.gz differ$ diff <(gunzip -c artifacts-38207/usr/share/doc/tor/changelog.Debian.gz) <(gunzip -c reproduce-38207/usr/share/doc/tor/changelog.Debian.gz) 1c1 < tor (0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1) tor-nightly-0.4.6.x-bullseye; urgency=medium --- > tor (0.4.6.10-dev-20220526T191917Z-1~d11.bullseye+1) tor-nightly-0.4.6.x-bullseye; urgency=medium 5c5 < -- Peter Palfrader <build@runner-y9surhn-project-1218-concurrent-0> Mon, 09 May 2022 16:14:16 +0000 --- > -- Peter Palfrader <build@2b183bfec45f> Thu, 26 May 2022 19:20:07 +0000 7c7 < tor (0.4.6.10-dev-20220509T161328Z-1) tor-nightly-0.4.6.x; urgency=medium --- > tor (0.4.6.10-dev-20220526T191917Z-1) tor-nightly-0.4.6.x; urgency=medium 9c9 < * Automated build of tor-nightly at 20220509T161328Z, git revision --- > * Automated build of tor-nightly at 20220526T191917Z, git revision 12c12 < -- Peter Palfrader <build@runner-y9surhn-project-1218-concurrent-0> Mon, 09 May 2022 16:14:12 +0000 --- > -- Peter Palfrader <build@2b183bfec45f> Thu, 26 May 2022 19:20:04 +0000-- Antoine Beaupré torproject.org system administration
-
first off, if you're poking around artifacts to try to figure out why they differ, i strongly encourage you try out the "diffoscope" tool written by the reproducible builds people.
Nifty! The build-date is also in the name of the deb files, which is presumably preventing diffoscope from drilling down further:
$ diffoscope artifacts-38207/binary-packages reproduce-38207/binary-packages --- artifacts-38207/binary-packages +++ reproduce-38207/binary-packages ├── file list │ @@ -1,6 +1,6 @@ │ debian-amd64-bullseye │ -debian-amd64-bullseye/tor-dbgsym_0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1_amd64.deb │ -debian-amd64-bullseye/tor-geoipdb_0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1_all.deb │ -debian-amd64-bullseye/tor_0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1_amd64.buildinfo │ -debian-amd64-bullseye/tor_0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1_amd64.changes │ -debian-amd64-bullseye/tor_0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1_amd64.deb │ +debian-amd64-bullseye/tor-dbgsym_0.4.6.10-dev-20220526T191917Z-1~d11.bullseye+1_amd64.deb │ +debian-amd64-bullseye/tor-geoipdb_0.4.6.10-dev-20220526T191917Z-1~d11.bullseye+1_all.deb │ +debian-amd64-bullseye/tor_0.4.6.10-dev-20220526T191917Z-1~d11.bullseye+1_amd64.buildinfo │ +debian-amd64-bullseye/tor_0.4.6.10-dev-20220526T191917Z-1~d11.bullseye+1_amd64.changes │ +debian-amd64-bullseye/tor_0.4.6.10-dev-20220526T191917Z-1~d11.bullseye+1_amd64.deb ├── stat {} │ @@ -1,8 +1,8 @@ │ │ Size: 4096 Blocks: 8 IO Block: 4096 directory │ Links: 3 │ -Access: (0775/drwxrwxr-x) Uid: ( 1000/jnewsome) Gid: ( 1000/jnewsome) │ +Access: (0755/drwxr-xr-x) Uid: ( 1000/jnewsome) Gid: ( 1000/jnewsome) │ │ -Modify: 2022-05-26 19:22:41.771190046 +0000 │ +Modify: 2022-05-26 19:22:37.951167898 +0000 │ │ Birth: - ├── getfacl -p -c {} │ @@ -1,4 +1,4 @@ │ user::rwx │ -group::rwx │ +group::r-x │ other::r-x ├── debian-amd64-bullseye │ ├── file list │ │ @@ -1,5 +1,5 @@ │ │ -tor-dbgsym_0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1_amd64.deb │ │ -tor-geoipdb_0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1_all.deb │ │ -tor_0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1_amd64.buildinfo │ │ -tor_0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1_amd64.changes │ │ -tor_0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1_amd64.deb │ │ +tor-dbgsym_0.4.6.10-dev-20220526T191917Z-1~d11.bullseye+1_amd64.deb │ │ +tor-geoipdb_0.4.6.10-dev-20220526T191917Z-1~d11.bullseye+1_all.deb │ │ +tor_0.4.6.10-dev-20220526T191917Z-1~d11.bullseye+1_amd64.buildinfo │ │ +tor_0.4.6.10-dev-20220526T191917Z-1~d11.bullseye+1_amd64.changes │ │ +tor_0.4.6.10-dev-20220526T191917Z-1~d11.bullseye+1_amd64.deb │ ├── stat {} │ │ @@ -1,8 +1,8 @@ │ │ │ │ Size: 4096 Blocks: 8 IO Block: 4096 directory │ │ Links: 2 │ │ -Access: (0755/drwxr-xr-x) Uid: ( 1000/jnewsome) Gid: ( 1000/jnewsome) │ │ +Access: (0755/drwxr-xr-x) Uid: (100100/ UNKNOWN) Gid: (100100/ UNKNOWN) │ │ │ │ -Modify: 2022-05-09 16:33:15.000000000 +0000 │ │ +Modify: 2022-05-26 19:22:37.951167898 +0000 │ │ │ │ Birth: - │ ├── stat {} │ │ @@ -1,8 +1,8 @@ │ │ │ │ Size: 4096 Blocks: 8 IO Block: 4096 directory │ │ Links: 2 │ │ -Access: (0755/drwxr-xr-x) Uid: ( 1000/jnewsome) Gid: ( 1000/jnewsome) │ │ +Access: (0755/drwxr-xr-x) Uid: (100100/ UNKNOWN) Gid: (100100/ UNKNOWN) │ │ │ │ -Modify: 2022-05-09 16:33:15.000000000 +0000 │ │ +Modify: 2022-05-26 19:22:37.951167898 +0000 │ │ │ │ Birth: -that thing should respect the SOURCE_EPOCH variable so that it can be run reproducibly, without having to mess with LD_PRELOAD (which is what faketime does, i believe).
It sure does. Agreed SOURCE_EPOCH sounds like a more principled way to do it.
-
Ah yes, you would need to use it to compare the .debs individually, i guess.
...
On 2022-05-26 20:58:43, Jim Newsome (@jnewsome) wrote:
Jim Newsome commented on a discussion: #40615 (comment 2808092)
first off, if you're poking around artifacts to try to figure out why they differ, i strongly encourage you try out the "diffoscope" tool written by the reproducible builds people.
Nifty! The build-date is also in the name of the deb files, which is presumably preventing diffoscope from drilling down further:
-- Antoine Beaupré torproject.org system administration
-
Here's the output for the "main" deb. Still looks like mostly timestamps, and
runner-y9surhn-project-1218-concurrent-0vs2b183bfec45f.$ diffoscope *-38207/binary-packages/debian-amd64-bullseye/tor_0.4.6.10-dev-*.bullseye+1_amd64.deb --- artifacts-38207/binary-packages/debian-amd64-bullseye/tor_0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1_amd64.deb +++ reproduce-38207/binary-packages/debian-amd64-bullseye/tor_0.4.6.10-dev-20220526T191917Z-1~d11.bullseye+1_amd64.deb ├── file list │ @@ -1,3 +1,3 @@ │ --rw-r--r-- 0 0 0 4 2022-05-09 16:14:16.000000 debian-binary │ --rw-r--r-- 0 0 0 5564 2022-05-09 16:14:16.000000 control.tar.xz │ --rw-r--r-- 0 0 0 1979012 2022-05-09 16:14:16.000000 data.tar.xz │ +-rw-r--r-- 0 0 0 4 2022-05-26 19:20:07.000000 debian-binary │ +-rw-r--r-- 0 0 0 5564 2022-05-26 19:20:07.000000 control.tar.xz │ +-rw-r--r-- 0 0 0 1979956 2022-05-26 19:20:07.000000 data.tar.xz ├── control.tar.xz │ ├── control.tar │ │ ├── file list │ │ │ @@ -1,8 +1,8 @@ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./ │ │ │ --rw-r--r-- 0 root (0) root (0) 214 2022-05-09 16:14:16.000000 ./conffiles │ │ │ --rw-r--r-- 0 root (0) root (0) 2258 2022-05-09 16:14:16.000000 ./control │ │ │ --rw-r--r-- 0 root (0) root (0) 2159 2022-05-09 16:14:16.000000 ./md5sums │ │ │ --rwxr-xr-x 0 root (0) root (0) 7272 2022-05-09 16:14:16.000000 ./postinst │ │ │ --rwxr-xr-x 0 root (0) root (0) 1999 2022-05-09 16:14:16.000000 ./postrm │ │ │ --rwxr-xr-x 0 root (0) root (0) 187 2022-05-09 16:14:16.000000 ./preinst │ │ │ --rwxr-xr-x 0 root (0) root (0) 603 2022-05-09 16:14:16.000000 ./prerm │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./ │ │ │ +-rw-r--r-- 0 root (0) root (0) 214 2022-05-26 19:20:07.000000 ./conffiles │ │ │ +-rw-r--r-- 0 root (0) root (0) 2258 2022-05-26 19:20:07.000000 ./control │ │ │ +-rw-r--r-- 0 root (0) root (0) 2159 2022-05-26 19:20:07.000000 ./md5sums │ │ │ +-rwxr-xr-x 0 root (0) root (0) 7272 2022-05-26 19:20:07.000000 ./postinst │ │ │ +-rwxr-xr-x 0 root (0) root (0) 1999 2022-05-26 19:20:07.000000 ./postrm │ │ │ +-rwxr-xr-x 0 root (0) root (0) 187 2022-05-26 19:20:07.000000 ./preinst │ │ │ +-rwxr-xr-x 0 root (0) root (0) 603 2022-05-26 19:20:07.000000 ./prerm │ │ ├── ./control │ │ │ @@ -1,9 +1,9 @@ │ │ │ Package: tor │ │ │ -Version: 0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1 │ │ │ +Version: 0.4.6.10-dev-20220526T191917Z-1~d11.bullseye+1 │ │ │ Architecture: amd64 │ │ │ Maintainer: Peter Palfrader <weasel@debian.org> │ │ │ Installed-Size: 5512 │ │ │ Depends: libc6 (>= 2.29), libcap2 (>= 1:2.10), libevent-2.1-7 (>= 2.1.8-stable), liblzma5 (>= 5.1.1alpha+20120614), libseccomp2 (>= 0.0.0~20120605), libssl1.1 (>= 1.1.1), libsystemd0, libzstd1 (>= 1.4.0), zlib1g (>= 1:1.1.4), adduser, runit-helper (>= 2.10.0~), lsb-base │ │ │ Recommends: logrotate, tor-geoipdb, torsocks │ │ │ Suggests: mixmaster, torbrowser-launcher, socat, apparmor-utils, nyx, obfs4proxy │ │ │ Conflicts: libssl0.9.8 (<< 0.9.8g-9) │ │ ├── ./md5sums │ │ │ ├── ./md5sums │ │ │ │┄ Files differ ├── data.tar.xz │ ├── data.tar │ │ ├── file list │ │ │ @@ -1,86 +1,86 @@ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./etc/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./etc/apparmor.d/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./etc/apparmor.d/abstractions/ │ │ │ --rw-r--r-- 0 root (0) root (0) 581 2022-05-09 16:14:16.000000 ./etc/apparmor.d/abstractions/tor │ │ │ --rw-r--r-- 0 root (0) root (0) 684 2022-05-09 16:14:16.000000 ./etc/apparmor.d/system_tor │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./etc/cron.weekly/ │ │ │ --rwxr-xr-x 0 root (0) root (0) 322 2022-05-09 16:14:16.000000 ./etc/cron.weekly/tor │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./etc/default/ │ │ │ --rw-r--r-- 0 root (0) root (0) 2623 2022-05-09 16:14:16.000000 ./etc/default/tor │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./etc/init.d/ │ │ │ --rwxr-xr-x 0 root (0) root (0) 6072 2022-05-09 16:14:16.000000 ./etc/init.d/tor │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./etc/logrotate.d/ │ │ │ --rw-r--r-- 0 root (0) root (0) 242 2022-05-09 16:14:16.000000 ./etc/logrotate.d/tor │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./etc/runit/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./etc/runit/runsvdir/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./etc/runit/runsvdir/default/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./etc/sv/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./etc/sv/tor/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./etc/sv/tor/.meta/ │ │ │ --rw-r--r-- 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./etc/sv/tor/.meta/installed │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./etc/sv/tor/log/ │ │ │ --rwxr-xr-x 0 root (0) root (0) 140 2022-05-09 16:14:16.000000 ./etc/sv/tor/log/run │ │ │ --rwxr-xr-x 0 root (0) root (0) 962 2022-05-09 16:14:16.000000 ./etc/sv/tor/run │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./etc/tor/ │ │ │ --rw-r--r-- 0 root (0) root (0) 8888 2022-05-09 16:14:16.000000 ./etc/tor/torrc │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./lib/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./lib/systemd/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./lib/systemd/system/ │ │ │ --rw-r--r-- 0 root (0) root (0) 312 2022-05-09 16:14:16.000000 ./lib/systemd/system/tor.service │ │ │ --rw-r--r-- 0 root (0) root (0) 1286 2022-05-09 16:14:16.000000 ./lib/systemd/system/tor@.service │ │ │ --rw-r--r-- 0 root (0) root (0) 1061 2022-05-09 16:14:16.000000 ./lib/systemd/system/tor@default.service │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./lib/systemd/system-generators/ │ │ │ --rwxr-xr-x 0 root (0) root (0) 751 2022-05-09 16:14:16.000000 ./lib/systemd/system-generators/tor-generator │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./usr/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./usr/bin/ │ │ │ --rwxr-xr-x 0 root (0) root (0) 3347176 2022-05-09 16:14:16.000000 ./usr/bin/tor │ │ │ --rwxr-xr-x 0 root (0) root (0) 541848 2022-05-09 16:14:16.000000 ./usr/bin/tor-gencert │ │ │ --rwxr-xr-x 0 root (0) root (0) 463992 2022-05-09 16:14:16.000000 ./usr/bin/tor-print-ed-signing-cert │ │ │ --rwxr-xr-x 0 root (0) root (0) 178360 2022-05-09 16:14:16.000000 ./usr/bin/tor-resolve │ │ │ --rwxr-xr-x 0 root (0) root (0) 1375 2022-05-09 16:13:27.000000 ./usr/bin/torify │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./usr/sbin/ │ │ │ --rwxr-xr-x 0 root (0) root (0) 2643 2022-05-09 16:14:16.000000 ./usr/sbin/tor-instance-create │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./usr/share/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./usr/share/doc/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./usr/share/doc/tor/ │ │ │ --rw-r--r-- 0 root (0) root (0) 460 2022-05-09 16:14:16.000000 ./usr/share/doc/tor/NEWS.Debian.gz │ │ │ --rw-r--r-- 0 root (0) root (0) 2964 2022-05-09 16:14:16.000000 ./usr/share/doc/tor/README.Debian │ │ │ --rw-r--r-- 0 root (0) root (0) 30394 2022-05-09 16:14:16.000000 ./usr/share/doc/tor/changelog.Debian.gz │ │ │ --rw-r--r-- 0 root (0) root (0) 520345 2022-05-09 16:13:28.000000 ./usr/share/doc/tor/changelog.gz │ │ │ --rw-r--r-- 0 root (0) root (0) 11342 2022-05-09 16:14:16.000000 ./usr/share/doc/tor/copyright │ │ │ --rw-r--r-- 0 root (0) root (0) 17557 2022-05-09 16:13:28.000000 ./usr/share/doc/tor/tor-exit-notice.html │ │ │ --rw-r--r-- 0 root (0) root (0) 20874 2022-05-09 16:14:16.000000 ./usr/share/doc/tor/tor-gencert.html │ │ │ --rw-r--r-- 0 root (0) root (0) 17821 2022-05-09 16:14:16.000000 ./usr/share/doc/tor/tor-print-ed-signing-cert.html │ │ │ --rw-r--r-- 0 root (0) root (0) 18708 2022-05-09 16:14:16.000000 ./usr/share/doc/tor/tor-resolve.html │ │ │ --rw-r--r-- 0 root (0) root (0) 280427 2022-05-09 16:14:16.000000 ./usr/share/doc/tor/tor.html │ │ │ --rw-r--r-- 0 root (0) root (0) 17798 2022-05-09 16:14:16.000000 ./usr/share/doc/tor/torify.html │ │ │ --rw-r--r-- 0 root (0) root (0) 4696 2022-05-09 16:14:16.000000 ./usr/share/doc/tor/torrc.sample.gz │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./usr/share/lintian/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./usr/share/lintian/overrides/ │ │ │ --rw-r--r-- 0 root (0) root (0) 46 2022-05-09 16:14:16.000000 ./usr/share/lintian/overrides/tor │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./usr/share/man/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./usr/share/man/man1/ │ │ │ --rw-r--r-- 0 root (0) root (0) 1645 2022-05-09 16:14:16.000000 ./usr/share/man/man1/tor-gencert.1.gz │ │ │ --rw-r--r-- 0 root (0) root (0) 816 2022-05-09 16:14:16.000000 ./usr/share/man/man1/tor-print-ed-signing-cert.1.gz │ │ │ --rw-r--r-- 0 root (0) root (0) 1012 2022-05-09 16:14:16.000000 ./usr/share/man/man1/tor-resolve.1.gz │ │ │ --rw-r--r-- 0 root (0) root (0) 54069 2022-05-09 16:14:16.000000 ./usr/share/man/man1/tor.1.gz │ │ │ --rw-r--r-- 0 root (0) root (0) 763 2022-05-09 16:14:16.000000 ./usr/share/man/man1/torify.1.gz │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./usr/share/man/man5/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./usr/share/man/man8/ │ │ │ --rw-r--r-- 0 root (0) root (0) 1222 2022-05-09 16:14:16.000000 ./usr/share/man/man8/tor-instance-create.8.gz │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./usr/share/runit/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./usr/share/runit/meta/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./usr/share/runit/meta/tor/ │ │ │ --rw-r--r-- 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./usr/share/runit/meta/tor/installed │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./usr/share/tor/ │ │ │ --rw-r--r-- 0 root (0) root (0) 351 2022-05-09 16:14:16.000000 ./usr/share/tor/tor-service-defaults-torrc │ │ │ --rw-r--r-- 0 root (0) root (0) 431 2022-05-09 16:14:16.000000 ./usr/share/tor/tor-service-defaults-torrc-instances │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./var/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./var/log/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./var/log/runit/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./var/log/runit/tor/ │ │ │ -lrwxrwxrwx 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./etc/sv/tor/log/supervise -> /run/runit/supervise/tor.log │ │ │ -lrwxrwxrwx 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./etc/sv/tor/supervise -> /run/runit/supervise/tor │ │ │ -lrwxrwxrwx 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./usr/sbin/tor -> ../bin/tor │ │ │ -lrwxrwxrwx 0 root (0) root (0) 0 2022-05-09 16:14:16.000000 ./usr/share/man/man5/torrc.5.gz -> ../man1/tor.1.gz │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./etc/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./etc/apparmor.d/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./etc/apparmor.d/abstractions/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 581 2022-05-26 19:20:07.000000 ./etc/apparmor.d/abstractions/tor │ │ │ +-rw-r--r-- 0 root (0) root (0) 684 2022-05-26 19:20:07.000000 ./etc/apparmor.d/system_tor │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./etc/cron.weekly/ │ │ │ +-rwxr-xr-x 0 root (0) root (0) 322 2022-05-26 19:20:07.000000 ./etc/cron.weekly/tor │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./etc/default/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 2623 2022-05-26 19:20:07.000000 ./etc/default/tor │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./etc/init.d/ │ │ │ +-rwxr-xr-x 0 root (0) root (0) 6072 2022-05-26 19:20:07.000000 ./etc/init.d/tor │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./etc/logrotate.d/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 242 2022-05-26 19:20:07.000000 ./etc/logrotate.d/tor │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./etc/runit/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./etc/runit/runsvdir/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./etc/runit/runsvdir/default/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./etc/sv/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./etc/sv/tor/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./etc/sv/tor/.meta/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./etc/sv/tor/.meta/installed │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./etc/sv/tor/log/ │ │ │ +-rwxr-xr-x 0 root (0) root (0) 140 2022-05-26 19:20:07.000000 ./etc/sv/tor/log/run │ │ │ +-rwxr-xr-x 0 root (0) root (0) 962 2022-05-26 19:20:07.000000 ./etc/sv/tor/run │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./etc/tor/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 8888 2022-05-26 19:20:07.000000 ./etc/tor/torrc │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./lib/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./lib/systemd/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./lib/systemd/system/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 312 2022-05-26 19:20:07.000000 ./lib/systemd/system/tor.service │ │ │ +-rw-r--r-- 0 root (0) root (0) 1286 2022-05-26 19:20:07.000000 ./lib/systemd/system/tor@.service │ │ │ +-rw-r--r-- 0 root (0) root (0) 1061 2022-05-26 19:20:07.000000 ./lib/systemd/system/tor@default.service │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./lib/systemd/system-generators/ │ │ │ +-rwxr-xr-x 0 root (0) root (0) 751 2022-05-26 19:20:07.000000 ./lib/systemd/system-generators/tor-generator │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./usr/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./usr/bin/ │ │ │ +-rwxr-xr-x 0 root (0) root (0) 3347176 2022-05-26 19:20:07.000000 ./usr/bin/tor │ │ │ +-rwxr-xr-x 0 root (0) root (0) 541848 2022-05-26 19:20:07.000000 ./usr/bin/tor-gencert │ │ │ +-rwxr-xr-x 0 root (0) root (0) 463992 2022-05-26 19:20:07.000000 ./usr/bin/tor-print-ed-signing-cert │ │ │ +-rwxr-xr-x 0 root (0) root (0) 178360 2022-05-26 19:20:07.000000 ./usr/bin/tor-resolve │ │ │ +-rwxr-xr-x 0 root (0) root (0) 1375 2022-05-26 19:19:17.000000 ./usr/bin/torify │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./usr/sbin/ │ │ │ +-rwxr-xr-x 0 root (0) root (0) 2643 2022-05-26 19:20:07.000000 ./usr/sbin/tor-instance-create │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./usr/share/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./usr/share/doc/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./usr/share/doc/tor/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 460 2022-05-26 19:20:07.000000 ./usr/share/doc/tor/NEWS.Debian.gz │ │ │ +-rw-r--r-- 0 root (0) root (0) 2964 2022-05-26 19:20:07.000000 ./usr/share/doc/tor/README.Debian │ │ │ +-rw-r--r-- 0 root (0) root (0) 30378 2022-05-26 19:20:07.000000 ./usr/share/doc/tor/changelog.Debian.gz │ │ │ +-rw-r--r-- 0 root (0) root (0) 520345 2022-05-26 19:19:17.000000 ./usr/share/doc/tor/changelog.gz │ │ │ +-rw-r--r-- 0 root (0) root (0) 11342 2022-05-26 19:20:07.000000 ./usr/share/doc/tor/copyright │ │ │ +-rw-r--r-- 0 root (0) root (0) 17557 2022-05-26 19:19:17.000000 ./usr/share/doc/tor/tor-exit-notice.html │ │ │ +-rw-r--r-- 0 root (0) root (0) 20874 2022-05-26 19:20:07.000000 ./usr/share/doc/tor/tor-gencert.html │ │ │ +-rw-r--r-- 0 root (0) root (0) 17821 2022-05-26 19:20:07.000000 ./usr/share/doc/tor/tor-print-ed-signing-cert.html │ │ │ +-rw-r--r-- 0 root (0) root (0) 18708 2022-05-26 19:20:07.000000 ./usr/share/doc/tor/tor-resolve.html │ │ │ +-rw-r--r-- 0 root (0) root (0) 280427 2022-05-26 19:20:07.000000 ./usr/share/doc/tor/tor.html │ │ │ +-rw-r--r-- 0 root (0) root (0) 17798 2022-05-26 19:20:07.000000 ./usr/share/doc/tor/torify.html │ │ │ +-rw-r--r-- 0 root (0) root (0) 4696 2022-05-26 19:20:07.000000 ./usr/share/doc/tor/torrc.sample.gz │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./usr/share/lintian/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./usr/share/lintian/overrides/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 46 2022-05-26 19:20:07.000000 ./usr/share/lintian/overrides/tor │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./usr/share/man/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./usr/share/man/man1/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 1645 2022-05-26 19:20:07.000000 ./usr/share/man/man1/tor-gencert.1.gz │ │ │ +-rw-r--r-- 0 root (0) root (0) 816 2022-05-26 19:20:07.000000 ./usr/share/man/man1/tor-print-ed-signing-cert.1.gz │ │ │ +-rw-r--r-- 0 root (0) root (0) 1012 2022-05-26 19:20:07.000000 ./usr/share/man/man1/tor-resolve.1.gz │ │ │ +-rw-r--r-- 0 root (0) root (0) 54069 2022-05-26 19:20:07.000000 ./usr/share/man/man1/tor.1.gz │ │ │ +-rw-r--r-- 0 root (0) root (0) 763 2022-05-26 19:20:07.000000 ./usr/share/man/man1/torify.1.gz │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./usr/share/man/man5/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./usr/share/man/man8/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 1222 2022-05-26 19:20:07.000000 ./usr/share/man/man8/tor-instance-create.8.gz │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./usr/share/runit/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./usr/share/runit/meta/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./usr/share/runit/meta/tor/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./usr/share/runit/meta/tor/installed │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./usr/share/tor/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 351 2022-05-26 19:20:07.000000 ./usr/share/tor/tor-service-defaults-torrc │ │ │ +-rw-r--r-- 0 root (0) root (0) 431 2022-05-26 19:20:07.000000 ./usr/share/tor/tor-service-defaults-torrc-instances │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./var/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./var/log/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./var/log/runit/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./var/log/runit/tor/ │ │ │ +lrwxrwxrwx 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./etc/sv/tor/log/supervise -> /run/runit/supervise/tor.log │ │ │ +lrwxrwxrwx 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./etc/sv/tor/supervise -> /run/runit/supervise/tor │ │ │ +lrwxrwxrwx 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./usr/sbin/tor -> ../bin/tor │ │ │ +lrwxrwxrwx 0 root (0) root (0) 0 2022-05-26 19:20:07.000000 ./usr/share/man/man5/torrc.5.gz -> ../man1/tor.1.gz │ │ ├── ./usr/share/doc/tor/changelog.Debian.gz │ │ │ ├── changelog.Debian │ │ │ │ @@ -1,19 +1,19 @@ │ │ │ │ -tor (0.4.6.10-dev-20220509T161328Z-1~d11.bullseye+1) tor-nightly-0.4.6.x-bullseye; urgency=medium │ │ │ │ +tor (0.4.6.10-dev-20220526T191917Z-1~d11.bullseye+1) tor-nightly-0.4.6.x-bullseye; urgency=medium │ │ │ │ │ │ │ │ * Build for tor-nightly-0.4.6.x-bullseye. │ │ │ │ │ │ │ │ - -- Peter Palfrader <build@runner-y9surhn-project-1218-concurrent-0> Mon, 09 May 2022 16:14:16 +0000 │ │ │ │ + -- Peter Palfrader <build@2b183bfec45f> Thu, 26 May 2022 19:20:07 +0000 │ │ │ │ │ │ │ │ -tor (0.4.6.10-dev-20220509T161328Z-1) tor-nightly-0.4.6.x; urgency=medium │ │ │ │ +tor (0.4.6.10-dev-20220526T191917Z-1) tor-nightly-0.4.6.x; urgency=medium │ │ │ │ │ │ │ │ - * Automated build of tor-nightly at 20220509T161328Z, git revision │ │ │ │ + * Automated build of tor-nightly at 20220526T191917Z, git revision │ │ │ │ 5568961cbf0749f5+4ba89c0ccc9452fa with debiantree e668385. │ │ │ │ │ │ │ │ - -- Peter Palfrader <build@runner-y9surhn-project-1218-concurrent-0> Mon, 09 May 2022 16:14:12 +0000 │ │ │ │ + -- Peter Palfrader <build@2b183bfec45f> Thu, 26 May 2022 19:20:04 +0000 │ │ │ │ │ │ │ │ tor (0.4.6.10-1) unstable; urgency=medium │ │ │ │ │ │ │ │ * New upstream version. │ │ │ │ │ │ │ │ -- Peter Palfrader <weasel@debian.org> Sun, 27 Feb 2022 13:58:14 +0100 │ │ ├── ./usr/share/man/man8/tor-instance-create.8.gz │ │ │ ├── tor-instance-create.8 │ │ │ │ @@ -1,17 +1,17 @@ │ │ │ │ '\" t │ │ │ │ .\" Title: tor-instance-create │ │ │ │ .\" Author: Peter Palfrader │ │ │ │ .\" Generator: DocBook XSL Stylesheets vsnapshot <http://docbook.sf.net/> │ │ │ │ -.\" Date: 05/09/2022 │ │ │ │ +.\" Date: 05/26/2022 │ │ │ │ .\" Manual: Tor Manual │ │ │ │ .\" Source: Tor │ │ │ │ .\" Language: English │ │ │ │ .\" │ │ │ │ -.TH "TOR\-INSTANCE\-CREAT" "8" "05/09/2022" "Tor" "Tor Manual" │ │ │ │ +.TH "TOR\-INSTANCE\-CREAT" "8" "05/26/2022" "Tor" "Tor Manual" │ │ │ │ .\" ----------------------------------------------------------------- │ │ │ │ .\" * Define some portability stuff │ │ │ │ .\" ----------------------------------------------------------------- │ │ │ │ .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ │ │ │ │ .\" http://bugs.debian.org/507673 │ │ │ │ .\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html │ │ │ │ .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -
Jim Newsome commented on a discussion: #40615 (comment 2808094)
Here's the output for the "main" deb. Still looks like mostly timestamps, and
runner-y9surhn-project-1218-concurrent-0vs2b183bfec45f.I think most of this stuff would be solved by having a consistent debian/changelog entry, except:
│ │ ├── ./usr/share/man/man8/tor-instance-create.8.gz │ │ │ ├── tor-instance-create.8 │ │ │ │ @@ -1,17 +1,17 @@ │ │ │ │ '\" t │ │ │ │ .\" Title: tor-instance-create │ │ │ │ .\" Author: Peter Palfrader │ │ │ │ .\" Generator: DocBook XSL Stylesheets vsnapshot <http://docbook.sf.net/> │ │ │ │ -.\" Date: 05/09/2022 │ │ │ │ +.\" Date: 05/26/2022 │ │ │ │ .\" Manual: Tor Manual │ │ │ │ .\" Source: Tor │ │ │ │ .\" Language: English │ │ │ │ .\" │ │ │ │ -.TH "TOR\-INSTANCE\-CREAT" "8" "05/09/2022" "Tor" "Tor Manual" │ │ │ │ +.TH "TOR\-INSTANCE\-CREAT" "8" "05/26/2022" "Tor" "Tor Manual" │ │ │ │ .\" ----------------------------------------------------------------- │ │ │ │ .\" * Define some portability stuff │ │ │ │ .\" ----------------------------------------------------------------- │ │ │ │ .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ │ │ │ │ .\" http://bugs.debian.org/507673 │ │ │ │ .\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html │ │ │ │ .\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~That! That's evil! :) Surely there's other docbook stuff like this elsewhere in debian, it's worth looking at how this was solved elsewhere.
...
On 2022-05-26 21:09:17, Jim Newsome wrote:
-- Antoine Beaupré torproject.org system administration
-
reproducibility, in general, requires some tuning like setting the system time:
https://reproducible-builds.org/docs/source-date-epoch/
... or tweaking the build path:
https://reproducible-builds.org/docs/build-path/
a lot of changes have been made to Debian's build infrastructure to make those kind of "just work", so when you build a debian package "the right way", you actually benefit from a lot of those.
it would actually be important to more clearly state what the concern is: has there been non-reproducible builds found in the wild? how are you rebuilding the debian packages (and why...)?
in general, the reproducible-builds.org website is a great resource for all of this, and they can actually help us if we need to figure out some of those things. from what i remember there's paid consultants working on this stuff over there now...
oh, and if I might add, the RBM tool is quite a beast, from what I remember. we shouldn't need this for tor-little-t, really.
(and rust is a whole different reproducibility problem, i'll just assume that problem space doesn't exist for now. :p)
-
The first step is just to check the build outside gitlab (or even in gitlab). For simple things, the toolchain has been updated not to include timestamps, etc. Reproducibility only requires RBM if there's additional stuff like zips, scripting, etc that re-inserts things like timestamps, filesystem ordering, etc.
We should not over-complicate this before doing a simple test. And if that test fails, then we should try using the tools we have, if we can.
-
Apparently gitlab-runner has a feature to execute jobs locally: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2797. Might be worth seeing if this would suit our needs.
It uses a local .gitlab-ci.yml, so I think the user doesn't need any special permission to the repo (e.g. to register a runner). Compared to registering and using a local runner this also removes the attack vector of a compromised gitlab server injecting backdoored commands into the script generated from .gitlab-ci.yml.
We'd still need a wrapper script to "orchestrate" the jobs since it only runs a single job, and we may need to inject some environment variables, but at least the repro script wouldn't need to duplicate the shell snippets in the gitlab yml. (We could alternatively move those shell snippets out to external shell scripts in the repo, but the extra layer of indirection would make the yml itself a bit more opaque)
-
Jim Newsome commented:
Apparently gitlab-runner has a feature to execute jobs locally: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2797. Might be worth seeing if this would suit our needs.
That looks like a feature... request, not an actual shipped feature. :) But it's an interesting idea.
It uses a local .gitlab-ci.yml, so I think the user doesn't need any special permission to the repo (e.g. to register a runner). Compared to registering and using a local runner this also removes the attack vector of a compromised gitlab server injecting backdoored commands into the script generated from .gitlab-ci.yml.
... or compromised code. Yes, that's really interesting!
We'd still need a wrapper script to "orchestrate" the jobs since it only runs a single job, and we may need to inject some environment variables, but at least the repro script wouldn't need to duplicate the shell snippets in the gitlab yml. (We could alternatively move those shell snippets out to external shell scripts in the repo, but the extra layer of indirection would make the yml itself a bit more opaque)
I tend to keep as little business logic in the YML as i can, exactly for that reason. If the YML is simple, then you can reproduce the build in a simple docker container, and it's pretty close to the environment provided by the GitLab runner.
At least in theory. :) In practice, there's a ton of stuff that happens during the runner setup, including environment variables, config files and so on, which are obviously hard to reproduce...
...
On 2022-05-27 14:14:25, Jim Newsome (@jnewsome) wrote:
-- Antoine Beaupré torproject.org system administration
-
That looks like a feature... request, not an actual shipped feature. :) But it's an interesting idea.
It's a request for a v2 of the feature that's more maintainable and has more features :)
I tend to keep as little business logic in the YML as i can, exactly for that reason. If the YML is simple, then you can reproduce the build in a simple docker container, and it's pretty close to the environment provided by the GitLab runner. At least in theory. :) In practice, there's a ton of stuff that happens during the runner setup, including environment variables, config files and so on, which are obviously hard to reproduce...
Yeah, that's the road I was starting to go down. Then I thought maybe it'd be easier to just write a script that parses the yml into scripts, then thought but maybe gitlab already exposes that somewhere... so here we are :). Using this might be a nice shortcut, but yeah if it doesn't work out we can fall back to moving more of the logic out of yml and into scripts invoked by the yml...
-
Jim Newsome commented on a discussion: #40615 (comment 2808198)
That looks like a feature... request, not an actual shipped feature. :) But it's an interesting idea.
It's a request for a v2 of the feature that's more maintainable and has more features :)
Oh nice. okay, then I definitely need to look more into this and probably document this in our wiki.
I tend to keep as little business logic in the YML as i can, exactly for that reason. If the YML is simple, then you can reproduce the build in a simple docker container, and it's pretty close to the environment provided by the GitLab runner. At least in theory. :) In practice, there's a ton of stuff that happens during the runner setup, including environment variables, config files and so on, which are obviously hard to reproduce...
Yeah, that's the road I was starting to go down. Then I thought maybe it'd be easier to just write a script that parses the yml into scripts, then thought but maybe gitlab already exposes that somewhere... so here we are :). Using this might be a nice shortcut, but yeah if it doesn't work out we can fall back to moving more of the logic out of yml and into scripts invoked by the yml...
hehe... good thing you stopped in time.
do you have a link to the upstream docs on this?
...
On 2022-05-27 14:28:38, Jim Newsome (@jnewsome) wrote:
-- Antoine Beaupré torproject.org system administration
-
do you have a link to the upstream docs on this?
-
I got it partly working: https://gitlab.torproject.org/jnewsome/reproduce-tor-debian-build/-/blob/main/reproduce_pipeline_with_runner.py
It successfully runs the
build_sourcejob and grabs the artifacts, but I got stuck at trying to execute just one of the parameterized build jobs (build_binary: [debian, bullseye, amd64, amd64, -slim]). Maybe there's a way to do it that I don't see... Otherwise maybe just doing it the "boring" way of moving everything out of yml into helper scripts s.t. it can be used with a thin docker wrapper is the way to go :) -
Started to pick apart the yml into shell scripts, but it's pretty gnarly with all the templating etc.
It ended up being pretty simple to just parse the yml and munge it into a script I can feed through docker: https://gitlab.torproject.org/jnewsome/reproduce-tor-debian-build/-/blob/main/reproduce_pipeline.py#L22
The script could still stand to be generalized some more (e.g. currently hardcoded to build for debian-bullseye; we should also be able to point it at alternative repos for the tor and/or build source to cross-validate), but you should be able to just edit the pipeline_id and it should automagically build at the same branch and commit as that pipeline did.
Edited by Jim Newsome -
We had some more discussion about
gitlab-runner exechere: https://forum.torproject.net/t/tor-project-gitlab-runner-updates/3668/4One thing I mentioned there that I hadn't mentioned here was that the planned next version of
gitlab-runner execinteracts with the GitLab instance, and it's unclear exactly what the GitLab instance will provide and whether it will be auditable. I just raised this concern on the gitlab issue: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2797#note_1003160535
mentioned in commit tpo/tpa/wiki-replica@2ab59b69
added Backlog label
changed milestone to %Tor: 0.4.7.x-post-stable
-
The RPMs come from a local build machine using Molecule+docker as build system. Sadly the RPMs in general are yet to be properly reproducible.
It seems it's possible to have reproducible RPM packages by setting some macros:
%source_date_epoch_from_changelog Y %clamp_mtime_to_source_date_epoch Y %use_source_date_epoch_as_buildtime Y %_buildhost reproduciblehttps://lists.reproducible-builds.org/pipermail/rb-general/2018-December/001301.html
assigned to @Diziet
added Q4 label
-
There has been a bit of deep compromise of infrastructure the past few months, and some gitlab security issues. I am also getting a bit nervous about the adversarial activity we've seen lately on the network, and the fact that it is known that this ticket is just sitting here, unfunded :/.
Could one of @jnewsome, @Diziet, or @boklm check out the build status of the 0.4.7.13 debs using https://gitlab.torproject.org/jnewsome/reproduce-tor-debian-build/ and some of our debs, and/or provide instructions to point that reproducing at key debs (ie: debian/stable x64 and arm64) from https://gitlab.torproject.org/tpo/core/debian/tor/-/jobs.
I have a debian 11 arm64 vm I could try it on, if someone updates the script it for the right job(s) to check.
-
Taking a look.
I have a debian 11 arm64 vm I could try it on, if someone updates the script it for the right job(s) to check.
Theoretically you should be able to just override the config on L127, but I'll check whether the script still works as-is first and report back...
-
[debian, bullseye, amd64, amd64, tpa, -slim, ]LGTM - mostly timestamp differences. Will push some updates to the script shortly.$ diffoscope reproduce-62714/theirs/binary-packages/debian-amd64-bullseye/tor_*bullseye+1_amd64.deb reproduce-*/ours/binary-packages/debian-amd64-bullseye/tor_*bullseye+1_amd64.deb --- reproduce-62714/theirs/binary-packages/debian-amd64-bullseye/tor_0.4.8.0-alpha-dev-20230117T020355Z-1~d11.bullseye+1_amd64.deb +++ reproduce-62714/ours/binary-packages/debian-amd64-bullseye/tor_0.4.8.0-alpha-dev-20230117T190809Z-1~d11.bullseye+1_amd64.deb ├── file list │ @@ -1,3 +1,3 @@ │ --rw-r--r-- 0 0 0 4 2023-01-17 02:04:44.000000 debian-binary │ --rw-r--r-- 0 0 0 5572 2023-01-17 02:04:44.000000 control.tar.xz │ --rw-r--r-- 0 0 0 2027700 2023-01-17 02:04:44.000000 data.tar.xz │ +-rw-r--r-- 0 0 0 4 2023-01-17 19:08:52.000000 debian-binary │ +-rw-r--r-- 0 0 0 5576 2023-01-17 19:08:52.000000 control.tar.xz │ +-rw-r--r-- 0 0 0 2026440 2023-01-17 19:08:52.000000 data.tar.xz ├── control.tar.xz │ ├── control.tar │ │ ├── file list │ │ │ @@ -1,8 +1,8 @@ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./ │ │ │ --rw-r--r-- 0 root (0) root (0) 214 2023-01-17 02:04:44.000000 ./conffiles │ │ │ --rw-r--r-- 0 root (0) root (0) 2263 2023-01-17 02:04:44.000000 ./control │ │ │ --rw-r--r-- 0 root (0) root (0) 2159 2023-01-17 02:04:44.000000 ./md5sums │ │ │ --rwxr-xr-x 0 root (0) root (0) 7272 2023-01-17 02:04:44.000000 ./postinst │ │ │ --rwxr-xr-x 0 root (0) root (0) 1999 2023-01-17 02:04:44.000000 ./postrm │ │ │ --rwxr-xr-x 0 root (0) root (0) 187 2023-01-17 02:04:44.000000 ./preinst │ │ │ --rwxr-xr-x 0 root (0) root (0) 603 2023-01-17 02:04:44.000000 ./prerm │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./ │ │ │ +-rw-r--r-- 0 root (0) root (0) 214 2023-01-17 19:08:52.000000 ./conffiles │ │ │ +-rw-r--r-- 0 root (0) root (0) 2263 2023-01-17 19:08:52.000000 ./control │ │ │ +-rw-r--r-- 0 root (0) root (0) 2159 2023-01-17 19:08:52.000000 ./md5sums │ │ │ +-rwxr-xr-x 0 root (0) root (0) 7272 2023-01-17 19:08:52.000000 ./postinst │ │ │ +-rwxr-xr-x 0 root (0) root (0) 1999 2023-01-17 19:08:52.000000 ./postrm │ │ │ +-rwxr-xr-x 0 root (0) root (0) 187 2023-01-17 19:08:52.000000 ./preinst │ │ │ +-rwxr-xr-x 0 root (0) root (0) 603 2023-01-17 19:08:52.000000 ./prerm │ │ ├── ./control │ │ │ @@ -1,9 +1,9 @@ │ │ │ Package: tor │ │ │ -Version: 0.4.8.0-alpha-dev-20230117T020355Z-1~d11.bullseye+1 │ │ │ +Version: 0.4.8.0-alpha-dev-20230117T190809Z-1~d11.bullseye+1 │ │ │ Architecture: amd64 │ │ │ Maintainer: Peter Palfrader <weasel@debian.org> │ │ │ Installed-Size: 5643 │ │ │ Depends: libc6 (>= 2.29), libcap2 (>= 1:2.10), libevent-2.1-7 (>= 2.1.8-stable), liblzma5 (>= 5.1.1alpha+20120614), libseccomp2 (>= 0.0.0~20120605), libssl1.1 (>= 1.1.1), libsystemd0, libzstd1 (>= 1.4.0), zlib1g (>= 1:1.1.4), adduser, runit-helper (>= 2.10.0~), lsb-base │ │ │ Recommends: logrotate, tor-geoipdb, torsocks │ │ │ Suggests: mixmaster, torbrowser-launcher, socat, apparmor-utils, nyx, obfs4proxy │ │ │ Conflicts: libssl0.9.8 (<< 0.9.8g-9) │ │ ├── ./md5sums │ │ │ ├── ./md5sums │ │ │ │┄ Files differ ├── data.tar.xz │ ├── data.tar │ │ ├── file list │ │ │ @@ -1,86 +1,86 @@ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./etc/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./etc/apparmor.d/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./etc/apparmor.d/abstractions/ │ │ │ --rw-r--r-- 0 root (0) root (0) 615 2023-01-17 02:04:44.000000 ./etc/apparmor.d/abstractions/tor │ │ │ --rw-r--r-- 0 root (0) root (0) 684 2023-01-17 02:04:44.000000 ./etc/apparmor.d/system_tor │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./etc/cron.weekly/ │ │ │ --rwxr-xr-x 0 root (0) root (0) 322 2023-01-17 02:04:44.000000 ./etc/cron.weekly/tor │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./etc/default/ │ │ │ --rw-r--r-- 0 root (0) root (0) 2623 2023-01-17 02:04:44.000000 ./etc/default/tor │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./etc/init.d/ │ │ │ --rwxr-xr-x 0 root (0) root (0) 6072 2023-01-17 02:04:44.000000 ./etc/init.d/tor │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./etc/logrotate.d/ │ │ │ --rw-r--r-- 0 root (0) root (0) 242 2023-01-17 02:04:44.000000 ./etc/logrotate.d/tor │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./etc/runit/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./etc/runit/runsvdir/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./etc/runit/runsvdir/default/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./etc/sv/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./etc/sv/tor/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./etc/sv/tor/.meta/ │ │ │ --rw-r--r-- 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./etc/sv/tor/.meta/installed │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./etc/sv/tor/log/ │ │ │ --rwxr-xr-x 0 root (0) root (0) 140 2023-01-17 02:04:44.000000 ./etc/sv/tor/log/run │ │ │ --rwxr-xr-x 0 root (0) root (0) 962 2023-01-17 02:04:44.000000 ./etc/sv/tor/run │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./etc/tor/ │ │ │ --rw-r--r-- 0 root (0) root (0) 8888 2023-01-17 02:04:44.000000 ./etc/tor/torrc │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./lib/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./lib/systemd/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./lib/systemd/system/ │ │ │ --rw-r--r-- 0 root (0) root (0) 312 2023-01-17 02:04:44.000000 ./lib/systemd/system/tor.service │ │ │ --rw-r--r-- 0 root (0) root (0) 1293 2023-01-17 02:04:44.000000 ./lib/systemd/system/tor@.service │ │ │ --rw-r--r-- 0 root (0) root (0) 1068 2023-01-17 02:04:44.000000 ./lib/systemd/system/tor@default.service │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./lib/systemd/system-generators/ │ │ │ --rwxr-xr-x 0 root (0) root (0) 751 2023-01-17 02:04:44.000000 ./lib/systemd/system-generators/tor-generator │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./usr/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./usr/bin/ │ │ │ --rwxr-xr-x 0 root (0) root (0) 3437736 2023-01-17 02:04:44.000000 ./usr/bin/tor │ │ │ --rwxr-xr-x 0 root (0) root (0) 545944 2023-01-17 02:04:44.000000 ./usr/bin/tor-gencert │ │ │ --rwxr-xr-x 0 root (0) root (0) 468088 2023-01-17 02:04:44.000000 ./usr/bin/tor-print-ed-signing-cert │ │ │ --rwxr-xr-x 0 root (0) root (0) 178360 2023-01-17 02:04:44.000000 ./usr/bin/tor-resolve │ │ │ --rwxr-xr-x 0 root (0) root (0) 1375 2023-01-17 02:03:54.000000 ./usr/bin/torify │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./usr/sbin/ │ │ │ --rwxr-xr-x 0 root (0) root (0) 2643 2023-01-17 02:04:44.000000 ./usr/sbin/tor-instance-create │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./usr/share/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./usr/share/doc/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./usr/share/doc/tor/ │ │ │ --rw-r--r-- 0 root (0) root (0) 460 2023-01-17 02:04:44.000000 ./usr/share/doc/tor/NEWS.Debian.gz │ │ │ --rw-r--r-- 0 root (0) root (0) 2964 2023-01-17 02:04:44.000000 ./usr/share/doc/tor/README.Debian │ │ │ --rw-r--r-- 0 root (0) root (0) 30627 2023-01-17 02:04:44.000000 ./usr/share/doc/tor/changelog.Debian.gz │ │ │ --rw-r--r-- 0 root (0) root (0) 532934 2023-01-17 02:03:54.000000 ./usr/share/doc/tor/changelog.gz │ │ │ --rw-r--r-- 0 root (0) root (0) 11342 2023-01-17 02:04:44.000000 ./usr/share/doc/tor/copyright │ │ │ --rw-r--r-- 0 root (0) root (0) 32679 2023-01-17 02:03:54.000000 ./usr/share/doc/tor/tor-exit-notice.html │ │ │ --rw-r--r-- 0 root (0) root (0) 20874 2023-01-17 02:04:44.000000 ./usr/share/doc/tor/tor-gencert.html │ │ │ --rw-r--r-- 0 root (0) root (0) 17821 2023-01-17 02:04:44.000000 ./usr/share/doc/tor/tor-print-ed-signing-cert.html │ │ │ --rw-r--r-- 0 root (0) root (0) 18708 2023-01-17 02:04:44.000000 ./usr/share/doc/tor/tor-resolve.html │ │ │ --rw-r--r-- 0 root (0) root (0) 287239 2023-01-17 02:04:44.000000 ./usr/share/doc/tor/tor.html │ │ │ --rw-r--r-- 0 root (0) root (0) 17798 2023-01-17 02:04:44.000000 ./usr/share/doc/tor/torify.html │ │ │ --rw-r--r-- 0 root (0) root (0) 4703 2023-01-17 02:04:44.000000 ./usr/share/doc/tor/torrc.sample.gz │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./usr/share/lintian/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./usr/share/lintian/overrides/ │ │ │ --rw-r--r-- 0 root (0) root (0) 46 2023-01-17 02:04:44.000000 ./usr/share/lintian/overrides/tor │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./usr/share/man/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./usr/share/man/man1/ │ │ │ --rw-r--r-- 0 root (0) root (0) 1645 2023-01-17 02:04:44.000000 ./usr/share/man/man1/tor-gencert.1.gz │ │ │ --rw-r--r-- 0 root (0) root (0) 817 2023-01-17 02:04:44.000000 ./usr/share/man/man1/tor-print-ed-signing-cert.1.gz │ │ │ --rw-r--r-- 0 root (0) root (0) 1013 2023-01-17 02:04:44.000000 ./usr/share/man/man1/tor-resolve.1.gz │ │ │ --rw-r--r-- 0 root (0) root (0) 55282 2023-01-17 02:04:44.000000 ./usr/share/man/man1/tor.1.gz │ │ │ --rw-r--r-- 0 root (0) root (0) 773 2023-01-17 02:04:44.000000 ./usr/share/man/man1/torify.1.gz │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./usr/share/man/man5/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./usr/share/man/man8/ │ │ │ --rw-r--r-- 0 root (0) root (0) 1223 2023-01-17 02:04:44.000000 ./usr/share/man/man8/tor-instance-create.8.gz │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./usr/share/runit/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./usr/share/runit/meta/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./usr/share/runit/meta/tor/ │ │ │ --rw-r--r-- 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./usr/share/runit/meta/tor/installed │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./usr/share/tor/ │ │ │ --rw-r--r-- 0 root (0) root (0) 351 2023-01-17 02:04:44.000000 ./usr/share/tor/tor-service-defaults-torrc │ │ │ --rw-r--r-- 0 root (0) root (0) 431 2023-01-17 02:04:44.000000 ./usr/share/tor/tor-service-defaults-torrc-instances │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./var/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./var/log/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./var/log/runit/ │ │ │ -drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./var/log/runit/tor/ │ │ │ -lrwxrwxrwx 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./etc/sv/tor/log/supervise -> /run/runit/supervise/tor.log │ │ │ -lrwxrwxrwx 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./etc/sv/tor/supervise -> /run/runit/supervise/tor │ │ │ -lrwxrwxrwx 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./usr/sbin/tor -> ../bin/tor │ │ │ -lrwxrwxrwx 0 root (0) root (0) 0 2023-01-17 02:04:44.000000 ./usr/share/man/man5/torrc.5.gz -> ../man1/tor.1.gz │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./etc/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./etc/apparmor.d/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./etc/apparmor.d/abstractions/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 615 2023-01-17 19:08:52.000000 ./etc/apparmor.d/abstractions/tor │ │ │ +-rw-r--r-- 0 root (0) root (0) 684 2023-01-17 19:08:52.000000 ./etc/apparmor.d/system_tor │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./etc/cron.weekly/ │ │ │ +-rwxr-xr-x 0 root (0) root (0) 322 2023-01-17 19:08:52.000000 ./etc/cron.weekly/tor │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./etc/default/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 2623 2023-01-17 19:08:52.000000 ./etc/default/tor │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./etc/init.d/ │ │ │ +-rwxr-xr-x 0 root (0) root (0) 6072 2023-01-17 19:08:52.000000 ./etc/init.d/tor │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./etc/logrotate.d/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 242 2023-01-17 19:08:52.000000 ./etc/logrotate.d/tor │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./etc/runit/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./etc/runit/runsvdir/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./etc/runit/runsvdir/default/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./etc/sv/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./etc/sv/tor/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./etc/sv/tor/.meta/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./etc/sv/tor/.meta/installed │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./etc/sv/tor/log/ │ │ │ +-rwxr-xr-x 0 root (0) root (0) 140 2023-01-17 19:08:52.000000 ./etc/sv/tor/log/run │ │ │ +-rwxr-xr-x 0 root (0) root (0) 962 2023-01-17 19:08:52.000000 ./etc/sv/tor/run │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./etc/tor/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 8888 2023-01-17 19:08:52.000000 ./etc/tor/torrc │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./lib/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./lib/systemd/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./lib/systemd/system/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 312 2023-01-17 19:08:52.000000 ./lib/systemd/system/tor.service │ │ │ +-rw-r--r-- 0 root (0) root (0) 1293 2023-01-17 19:08:52.000000 ./lib/systemd/system/tor@.service │ │ │ +-rw-r--r-- 0 root (0) root (0) 1068 2023-01-17 19:08:52.000000 ./lib/systemd/system/tor@default.service │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./lib/systemd/system-generators/ │ │ │ +-rwxr-xr-x 0 root (0) root (0) 751 2023-01-17 19:08:52.000000 ./lib/systemd/system-generators/tor-generator │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./usr/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./usr/bin/ │ │ │ +-rwxr-xr-x 0 root (0) root (0) 3437736 2023-01-17 19:08:52.000000 ./usr/bin/tor │ │ │ +-rwxr-xr-x 0 root (0) root (0) 545944 2023-01-17 19:08:52.000000 ./usr/bin/tor-gencert │ │ │ +-rwxr-xr-x 0 root (0) root (0) 468088 2023-01-17 19:08:52.000000 ./usr/bin/tor-print-ed-signing-cert │ │ │ +-rwxr-xr-x 0 root (0) root (0) 178360 2023-01-17 19:08:52.000000 ./usr/bin/tor-resolve │ │ │ +-rwxr-xr-x 0 root (0) root (0) 1375 2023-01-17 19:08:09.000000 ./usr/bin/torify │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./usr/sbin/ │ │ │ +-rwxr-xr-x 0 root (0) root (0) 2643 2023-01-17 19:08:52.000000 ./usr/sbin/tor-instance-create │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./usr/share/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./usr/share/doc/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./usr/share/doc/tor/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 460 2023-01-17 19:08:52.000000 ./usr/share/doc/tor/NEWS.Debian.gz │ │ │ +-rw-r--r-- 0 root (0) root (0) 2964 2023-01-17 19:08:52.000000 ./usr/share/doc/tor/README.Debian │ │ │ +-rw-r--r-- 0 root (0) root (0) 30606 2023-01-17 19:08:52.000000 ./usr/share/doc/tor/changelog.Debian.gz │ │ │ +-rw-r--r-- 0 root (0) root (0) 532934 2023-01-17 19:08:09.000000 ./usr/share/doc/tor/changelog.gz │ │ │ +-rw-r--r-- 0 root (0) root (0) 11342 2023-01-17 19:08:52.000000 ./usr/share/doc/tor/copyright │ │ │ +-rw-r--r-- 0 root (0) root (0) 32679 2023-01-17 19:08:09.000000 ./usr/share/doc/tor/tor-exit-notice.html │ │ │ +-rw-r--r-- 0 root (0) root (0) 20874 2023-01-17 19:08:52.000000 ./usr/share/doc/tor/tor-gencert.html │ │ │ +-rw-r--r-- 0 root (0) root (0) 17821 2023-01-17 19:08:52.000000 ./usr/share/doc/tor/tor-print-ed-signing-cert.html │ │ │ +-rw-r--r-- 0 root (0) root (0) 18708 2023-01-17 19:08:52.000000 ./usr/share/doc/tor/tor-resolve.html │ │ │ +-rw-r--r-- 0 root (0) root (0) 287239 2023-01-17 19:08:52.000000 ./usr/share/doc/tor/tor.html │ │ │ +-rw-r--r-- 0 root (0) root (0) 17798 2023-01-17 19:08:52.000000 ./usr/share/doc/tor/torify.html │ │ │ +-rw-r--r-- 0 root (0) root (0) 4703 2023-01-17 19:08:52.000000 ./usr/share/doc/tor/torrc.sample.gz │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./usr/share/lintian/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./usr/share/lintian/overrides/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 46 2023-01-17 19:08:52.000000 ./usr/share/lintian/overrides/tor │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./usr/share/man/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./usr/share/man/man1/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 1645 2023-01-17 19:08:52.000000 ./usr/share/man/man1/tor-gencert.1.gz │ │ │ +-rw-r--r-- 0 root (0) root (0) 817 2023-01-17 19:08:52.000000 ./usr/share/man/man1/tor-print-ed-signing-cert.1.gz │ │ │ +-rw-r--r-- 0 root (0) root (0) 1013 2023-01-17 19:08:52.000000 ./usr/share/man/man1/tor-resolve.1.gz │ │ │ +-rw-r--r-- 0 root (0) root (0) 55282 2023-01-17 19:08:52.000000 ./usr/share/man/man1/tor.1.gz │ │ │ +-rw-r--r-- 0 root (0) root (0) 773 2023-01-17 19:08:52.000000 ./usr/share/man/man1/torify.1.gz │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./usr/share/man/man5/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./usr/share/man/man8/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 1223 2023-01-17 19:08:52.000000 ./usr/share/man/man8/tor-instance-create.8.gz │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./usr/share/runit/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./usr/share/runit/meta/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./usr/share/runit/meta/tor/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./usr/share/runit/meta/tor/installed │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./usr/share/tor/ │ │ │ +-rw-r--r-- 0 root (0) root (0) 351 2023-01-17 19:08:52.000000 ./usr/share/tor/tor-service-defaults-torrc │ │ │ +-rw-r--r-- 0 root (0) root (0) 431 2023-01-17 19:08:52.000000 ./usr/share/tor/tor-service-defaults-torrc-instances │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./var/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./var/log/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./var/log/runit/ │ │ │ +drwxr-xr-x 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./var/log/runit/tor/ │ │ │ +lrwxrwxrwx 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./etc/sv/tor/log/supervise -> /run/runit/supervise/tor.log │ │ │ +lrwxrwxrwx 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./etc/sv/tor/supervise -> /run/runit/supervise/tor │ │ │ +lrwxrwxrwx 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./usr/sbin/tor -> ../bin/tor │ │ │ +lrwxrwxrwx 0 root (0) root (0) 0 2023-01-17 19:08:52.000000 ./usr/share/man/man5/torrc.5.gz -> ../man1/tor.1.gz │ │ ├── ./usr/share/doc/tor/changelog.Debian.gz │ │ │ ├── changelog.Debian │ │ │ │ @@ -1,19 +1,19 @@ │ │ │ │ -tor (0.4.8.0-alpha-dev-20230117T020355Z-1~d11.bullseye+1) tor-nightly-main-bullseye; urgency=medium │ │ │ │ +tor (0.4.8.0-alpha-dev-20230117T190809Z-1~d11.bullseye+1) tor-nightly-main-bullseye; urgency=medium │ │ │ │ │ │ │ │ * Build for tor-nightly-main-bullseye. │ │ │ │ │ │ │ │ - -- Peter Palfrader <build@runner-rkxrxyrk-project-1218-concurrent-0> Tue, 17 Jan 2023 02:04:44 +0000 │ │ │ │ + -- Peter Palfrader <build@5bb2eeed6591> Tue, 17 Jan 2023 19:08:52 +0000 │ │ │ │ │ │ │ │ -tor (0.4.8.0-alpha-dev-20230117T020355Z-1) tor-nightly-main; urgency=medium │ │ │ │ +tor (0.4.8.0-alpha-dev-20230117T190809Z-1) tor-nightly-main; urgency=medium │ │ │ │ │ │ │ │ - * Automated build of tor-nightly at 20230117T020355Z, git revision │ │ │ │ + * Automated build of tor-nightly at 20230117T190809Z, git revision │ │ │ │ ce5ec4d82548c8f2 with debiantree c70eb56. │ │ │ │ │ │ │ │ - -- Peter Palfrader <build@runner-rkxrxyrk-project-1218-concurrent-0> Tue, 17 Jan 2023 02:04:40 +0000 │ │ │ │ + -- Peter Palfrader <build@5bb2eeed6591> Tue, 17 Jan 2023 19:08:49 +0000 │ │ │ │ │ │ │ │ tor (0.4.7.13-1) unstable; urgency=medium │ │ │ │ │ │ │ │ * New upstream version. │ │ │ │ │ │ │ │ -- Peter Palfrader <weasel@debian.org> Thu, 12 Jan 2023 18:31:32 +0100 -
When you update the script, can you point me at the arm64 jobs + things I should tweak for arm64?
Can I build other debs than Deb 11 from a Deb 11 VM, with different docker images? Does the script do this?
Also, for the diffoscope, do you download the artifacts for the job manually, or does your script handle it?
-
Updated the script to accommodate changes in the pipeline (new parameters in the build matrix), and to make it easier to reproduce different configs. I checked the ubuntu-xenial-amd64 build as well with similar results.
Exactly which build to reproduce can now be changed by editing
job_paramsCan I build other debs than Deb 11 from a Deb 11 VM, with different docker images? Does the script do this?
Yup, the script uses Docker, and pulls the same image name as was used in the build.
Also, for the diffoscope, do you download the artifacts for the job manually, or does your script handle it?
The script downloads the artifacts for you.
-
I think you picked the wrong pipeline? This builds 0.4.8.0 for me, not 0.4.7.13. It looks like your diffoscope output also is comparing 0.4.8.0.
The correct pipeline for 0.4.7.13 seems to be 62395.
Additionally, I had to do some fiddling to get it to find the right job. At least for arm64, the arguments for the job name change order from what you had in your refactoring commit. I had to revert to the hardcoded commit and do:
+ their_build_source = get_successful_job(project_id, pipeline_id, 'build_source-release') + their_build_binary = get_successful_job(project_id, pipeline_id, 'build_binary-release: [debian, arm64v8, bullseye, aarch64, osuosl, -slim, ]')But now, I am still hitting a tag checkout issue in the build container. It seems like it is taking too much of the debian package version into the tag name, plus some other weird stuff? It is trying to do a git checkout of checkout maint-tor-0.4.7.13-1, but the right tag is just tor-0.4.7.13 (or debian-tor-0.4.7.13-1)..
Not sure how to fix this.. Hardcoding pipeline_branch and commit don't work. The bad tag seems to be being made in the build script itself.
Edited by Mike Perry -
I think you picked the wrong pipeline? This builds 0.4.8.0 for me, not 0.4.7.13. It looks like your diffoscope output also is comparing 0.4.8.0. The correct pipeline for 0.4.7.13 seems to be 62395.
Ah ok. For documentation/future-reference, how did you go about identifying that pipeline id btw?
Additionally, I had to do some fiddling to get it to find the right job. At least for arm64, the arguments for the job name change order from what you had in your refactoring commit.
Ugh, yeah I wasn't sure whether I could count on the order being consistent or not. I'll see if I can make that a bit more robust.
But now, I am still hitting a tag checkout issue in the build container. It seems like it is taking too much of the debian package version into the tag name, plus some other weird stuff? It is trying to do a git checkout of checkout maint-tor-0.4.7.13-1, but the right tag is just tor-0.4.7.13 (or debian-tor-0.4.7.13-1)..
It was trying to run the wrong job. I updated the script to run
build_source-releaseandbuild_binary-release.I also needed to set
CI_COMMIT_TAG, which those jobs depend on.It gets a bit further now but is failing in the
pristine-tarcommand:+ pristine-tar checkout tor-0.4.7.13.tar.gz fatal: ambiguous argument 'cb08422f82dbd92fc6dac9a6962d81afd8a93377^{tree}': unknown revision or path not in the working tree.Continuing to debug...
-
Ah ok. For documentation/future-reference, how did you go about identifying that pipeline id btw?
I took a look at https://gitlab.torproject.org/tpo/core/debian/tor/-/pipelines and guessed based on the tag label name for the pipeline.
fatal: ambiguous argument 'cb08422f82dbd92fc6dac9a6962d81afd8a93377^{tree}': unknown revision or path not in the working tree.I had issues like this due to the usage of --depth for shallow checkouts. I removed those in my branch from your python, but there are also some usage of them in the build scripts themselves maybe?
-
I poked around a bit in the intermediate state (
reproduce-62395/pipeline_src/build-env/src/pristine-upstream)I'm not familiar with the
pristine-tartool, so I'm flying a little blind here.Experimentally, it looks like it's getting the commit hash
cb0842...fromtor-0.4.7.13.tar.gz.id:.../src/pristine-upstream$ cat tor-0.4.7.13.tar.gz.id cb08422f82dbd92fc6dac9a6962d81afd8a93377I think it's looking for it in the debian tor repo (
gitlab.torproject.org/tpo/core/debian/tor.git). I don't see it there, nor in thetorrepo (gitlab.torproject.org/tpo/core/tor.git).I spot checked the hashes in a couple of the older files and couldn't find those either. Probably I'm not understanding what it's actually trying to do here...
Edited by Jim Newsome 
I don't see a commit with that hash in any of the relevant repos. I think it's looking for it in the debian tor repo (
https://gitlab.torproject.org/tpo/core/debian/tor.git). I don't see it there, nor the in thetorrepo (git@gitlab.torproject.org:tpo/core/tor.git).See https://git.torproject.org/debian/tor-pristine-upstream.git (or https://gitweb.torproject.org/debian/tor-pristine-upstream.git/ )
Also see debian/.debian-ci.yml.
The release builds need the upstream source tarball, and they get it from git with the help of pristine-tar.
And while we are at it, we also verify that the upstream tarball has a proper signature.
...
On Wed, 18 Jan 2023, Jim Newsome (@jnewsome) wrote:
-- | .''
. ** Debian ** Peter Palfrader | : :' : The universal https://www.palfrader.org/ |.' Operating System |- https://www.debian.org/-
See https://git.torproject.org/debian/tor-pristine-upstream.git (or https://gitweb.torproject.org/debian/tor-pristine-upstream.git/ )
https://git.torproject.org/debian/tor-pristine-upstream.gitalso doesn't have that commit hash. -
Jim Newsome commented on a discussion: #40615 (comment 2870147)
See https://git.torproject.org/debian/tor-pristine-upstream.git (or https://gitweb.torproject.org/debian/tor-pristine-upstream.git/ )
https://git.torproject.org/debian/tor-pristine-upstream.gitalso doesn't have that commit hash.Right, tor-pristine-upstream has the reference to the tree object (cb08422f82dbd92fc6dac9a6962d81afd8a93377) in tor-0.4.7.13.tar.gz.id.
cb08422f82dbd... then is a tree in the tor repo (git show cb08422f82dbd).
...
On Wed, 18 Jan 2023, Jim Newsome (@jnewsome) wrote:
| .''`. ** Debian ** Peter Palfrader | : :' : The universalhttps://www.palfrader.org/ |
.' Operating System | `- https://www.debian.org/ -
cb08422f82dbd... then is a tree in the tor repo (git show cb08422f82dbd).
Ah ha - I'd make a mistake when looking for cb08422f82dbd in my full checkout of the debian tor repo. It is there.
Updated my script to:
- do a non-shallow clone of that repo
- set up the pipeline variables containing public keys used by the script to verify signatures
@mikeperry I also updated the script to fix the parameter ordering for the arm build, so hopefully it should work for you now.
-
For historical reference, I reproduced 0.4.7.13 on arm64 successfully. diffoscope differences are similar to earlier: just file list metadata and the package info replacement strings.
I also had to comment out the env line from subprocess.Popen(), because my $DOCKER_HOST is empty (local docker). Else it failed:
+ #env={'DOCKER_HOST': os.getenv('DOCKER_HOST')},It does concern me a bit that the build script that is run comes from gitlab itself, which could be tainted. Perhaps there could be a step to save/review it, and optionally use a local cached copy?
Edited by Mike Perry -
It does concern me a bit that the build script that is run comes from gitlab itself, which could be tainted. Perhaps there could be a step to save/review it, and optionally use a local cached copy?
The cloned repo is in
reproduce-*/pipeline_src. The scripts can be inspected there. Or if we want to cross-validate with another clone of that repo, I think we just need to validate that the given tag has the same hash in that checkout as in the other repo. e.g.git rev-parse debian-tor-0.4.7.13-1^gives the answer for me in that directory vs my working checkout of that repo.
Hi. Thanks for the mention @mikeperry. I feel I'm coming into the middle of this rather, and I'm not familiar with the context. I skimread this thread.
My starting point is that I would like to use the Reproducible Builds project's notion of reproducibility, which involves producing the exact same deliverables, bit-for-bit identical. So there should be no question of anyone reviewing things in diffoscope. (Or rather, having to review something in diffoscope means "our build is not reproducible, and we are debugging it".)
And ideally actual reproduction would be done routinely, on systems sharing as little as possible in terms of non-source inputs.
It seems we're using a roughly normal-Debian-package build system; there's a debian/rules. So we ought to be using Debian's official reproduction tooling, and a normal Debian build rune (dpkg-buildpackage, sbuild, or something), not some ad-hoc script. And we could use Debian's tooling for producing the repro environment, rather than a docker image. After all, subverting the build environment is a way to subvert the binaries.
So. I think "the Tor binaries on deb.torproject.org are reproducible" means this:
I should be able to get the tor source code from the signed tag, and run
debrebuild, and get the same binaries. I haven't tried to do this. Is it likely to work? I think from reading this thread that the answer is "no".I see that http://deb.torproject.org/torproject.org/pool/main/t/tor/ doesn't contain buildinfo files so that would be the first hurdle!
It may seem like this is an extreme notion of reproducibility, but it is usually achieveable in practice in my experience. Recently, with my personal hat on, I built .debs for the Xen Hypervisor on my laptop and uploaded them to Debian experimental (for foolish reasons to do with broken Debian processes), and these turned out to be identical to the binaries one of my collaborators built from the same git commit, and we hadn't even bothered with a tool like debrepro.
-
I completely agree on all points, Ian. The main issue we're running into right now is that we migrated the package building into gitlab a couple years back but did not make use of the Debian reproducibility tooling for this. So we have had to check this stuff manually with diffoscope. This ticket is mostly filled with that kind of triage, to make sure we weren't pwnt after weird shit happened on gitlab.
It is also my opinion that doing this right is not hard. We just haven't had someone familiar with the debian reproducibility tooling to do it. We also got in managerial disagreements about getting funding for this, hiring/contracting someone, etc.
However, this is one of those situations where trying to get funding will literally cost more than the activity itself, by several orders of magnitude. Unfortunately, this also means that these kinds of simple yet crucial things tend to never get done, for this exactly reason :/.
-
I should be able to get the tor source code from the signed tag, and run
debrebuild, and get the same binaries. I haven't tried to do this. Is it likely to work? I think from reading this thread that the answer is "no".I see that http://deb.torproject.org/torproject.org/pool/main/t/tor/ doesn't contain buildinfo files so that would be the first hurdle!
They exist -- they are in the artifacts we produce, even.
However, last time I checked, reprepro couldn't handle them.
If that's changed, we should make reprepro export them.
...
On Thu, 19 Jan 2023, Ian Jackson (@Diziet) wrote:
| .''`. ** Debian ** Peter Palfrader | : :' : The universalhttps://www.palfrader.org/ |
.' Operating System | `- https://www.debian.org/ -
Broken buildinfo support in reprepro seems to be noted here https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=869961
-
Tor 0.4.7.13 is in Debian unstable and is reproducible there: https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/tor.html
-
Ian Jackson commented: Tor 0.4.7.13 is in Debian unstable and is reproducible there: https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/tor.html
Same source package :)
...
On Thu, 19 Jan 2023, Ian Jackson (@Diziet) wrote:
-- | .''
. ** Debian ** Peter Palfrader | : :' : The universal https://www.palfrader.org/ |.' Operating System |- https://www.debian.org/
removed Q4 label
