Release Tor Browser on F-Droid
Can we create a build script and add the app now? What are the blockers? How difficult will this become after we begin building using tor-browser-build?
The Guardian Project have their own F-Droid repository, do we need our own? If we do, can ours be included in the f-droid app by default (but disabled), too?
Designs
- tpo/ux/team #77
Activity
added TorBrowserTeam202006 in Legacy / Trac component::applications/tor browser in Legacy / Trac owner::sysrqb in Legacy / Trac parent::26318 in Legacy / Trac points::1 in Legacy / Trac priority::medium in Legacy / Trac severity::normal in Legacy / Trac sponsor::8 in Legacy / Trac status::accepted in Legacy / Trac tbb-9.5 in Legacy / Trac tbb-mobile in Legacy / Trac type::enhancement in Legacy / Trac labels
Good luck! Keeping an eye to add the link to the website when it's ready (legacy/trac#27305 (moved)).
Trac:
Cc: igt0, gk, dmr, eighthave to igt0, gk, dmr, eighthave, traumschule
Trac:
Username: darkspirit
Cc: igt0, gk, dmr, eighthave, traumschule to igt0, gk, dmr, eighthave, traumschule, jan@ikenmeyer.eu
sysrqb: Could we try to map out the pros and cons for having Tor Browser built and distributed via the official F-Droid repo vs. making it available via our own one?
So, assuming we are going the official F-Droid repo route and are getting a build script ready that is essentially doing the tor-browser-build on F-Droid's infra, how would we make sure that we are actually releasing what we got with our own builds? I mean without verification before we go live reproducible builds are just a promise. :)
eighthave: are there ways to incorporate that verification step in the whole build process?
Do we add additional attack surface by relying on F-Droid infra vs. setting up our own (and if so, is that worth the benefit)?
The ideal setup is to have both a Tor Project repo and TBA building in f-droid.org. F-Droid handles getting the same app from multiple sources already.
Since TBA will be released only with Tor Project's signature (and not an f-droid.org signature), Tor Project can publish to its own repo on its own schedule. Then the f-droid.org builders will trigger builds based on git tags (or other means), and can fetch the APK from the Tor Project's own repo. It will then use the signatures from those APKs as the test of a successful build.
If you want more control of when the f-droid.org builders make their builds, you can manually submit merge/pull requests for each release, and include just the APK signature there (that can be extracted using
fdroid signatures tor-browser.apk).Having a Tor Project repo means TBA updates can be shipped regardless of timing/status of f-droid.org. Having f-droid.org also build and ship the TBA APKs provides an extra level of reproducibility and distribution resilience, since the f-droid.org repo is mirrored on many standard free software mirror servers (ftp.fau.de, cyberbits.eu, osuosl, etc). It is also easy to add mirrors to the Tor Project repo, they can be any webserver via ssh/rsync, GitHub Pages, GitLab Pages, and AmazonS3 or any service compatible with
s3cmd.-
Another nice feature of getting Tor Browser integrated into the F-Droid ecosystem is the "Update Channels" library for making an app be able to find/download/install it own updates. One way it can operate is that TBA would check the Tor Project repo for updates. This requires only a HTTP HEAD request, then comparing the ETag, so like < 1kb. One mode of operation is starting to check for updates if it hasn't been updated in the past month.
https://gitlab.com/fdroid/update-channels
This is relevant when TBA inevitably distributed outside of F-Droid/Play. People will get it from APKMirror, APKPure, Aptoide, or any of the myriad third party app stores out there.
-
Replying to eighthave:
Another nice feature of getting Tor Browser integrated into the F-Droid ecosystem is the "Update Channels" library for making an app be able to find/download/install it own updates. One way it can operate is that TBA would check the Tor Project repo for updates. This requires only a HTTP HEAD request, then comparing the ETag, so like < 1kb. One mode of operation is starting to check for updates if it hasn't been updated in the past month.
https://gitlab.com/fdroid/update-channels
This is relevant when TBA inevitably distributed outside of F-Droid/Play. People will get it from APKMirror, APKPure, Aptoide, or any of the myriad third party app stores out there.
That's indeed an interesting point, thanks for mentioning it.
quick update here. I like this plan in theory, where we run our own repo and publish updates on f-droid.org and our own. My main worry is our ability for running more services like this as an organization. I think this is something we can think about more and make plans for in the (not too distant) future - maybe. In the meantime, I'd like to get f-droid.org builds working now.
I currently have a metadata file ready and it successfully builds tor browser using tor-browser-build within a fdroidserver instance I installed locally. My main worry here is how long the build process takes. On a bare-metal older laptop I'm using, the entire build process for armv7 takes roughly 3.5 hours. I'm under the impression F-Droid use VMs for each app build, and there's an upper bound of 7 hours(?) per build. If this is true, then I won't be surprised if we hit that timeout limit. One of the main reasons for this is artifacts are not cached across builds - so all repos and files are cloned/downloaded for every build iteration. I understand why this is important for a general apk builder, but this is painful for a large project like tor browser.
In addition, the Tor Browser build process requires 40+ GB of disk space and 2+ GB of RAM, and I'm not sure if F-Droid's default build server will support this.
I haven't confirmed reproducible builds between our apk and fdroid yet.
Replying to sysrqb:
I currently have a metadata file ready and it successfully builds tor browser using tor-browser-build within a fdroidserver instance I installed locally. My main worry here is how long the build process takes. On a bare-metal older laptop I'm using, the entire build process for armv7 takes roughly 3.5 hours. I'm under the impression F-Droid use VMs for each app build, and there's an upper bound of 7 hours(?) per build. If this is true, then I won't be surprised if we hit that timeout limit. One of the main reasons for this is artifacts are not cached across builds - so all repos and files are cloned/downloaded for every build iteration. I understand why this is important for a general apk builder, but this is painful for a large project like tor browser.
I'm wondering if there could be an option in the F-Droid build to save some files at the end of a build, and restore them when starting a new builds.
I think the directories we would want to keep are
out/,git_clones/,gclient/.-
Replying to sysrqb:
quick update here. I like this plan in theory, where we run our own repo and publish updates on f-droid.org and our own. My main worry is our ability for running more services like this as an organization. I think this is something we can think about more and make plans for in the (not too distant) future - maybe. In the meantime, I'd like to get f-droid.org builds working now.
I currently have a metadata file ready and it successfully builds tor browser using tor-browser-build within a fdroidserver instance I installed locally.
Nice. One thing that makes me nervous with that idea is that we don't have a good way to test if any of the changes we make to
tor-browser-build/rbmwould suddenly break in an f-droid environment. I don't want to find that out while doing a critical release. Not sure if f-droid supports something like try pushes to test that. Or maybe we should add that as part of our nightly builds. -
The F-Droid community would generally love to support Tor Project, so don't consider the current setup something to workaround. I think there can even be special, hard-coded exceptions for
org.torproject.*packages. The builder VM has 4-8GB RAM, and could have more if that would be useful. There is no disk space limitation so far, just timeouts on the whole process.I imagine that the large majority of the TBA build time comes from building Fennec. f-droid.org has been building and shipping Fennec builds for years, so we know they work. They've always been handled by one volunteer, so they haven't required tons of work. Per-build timeouts are specified in the metadata file, the default is 2 hours. f-droid.org currently has a 36 hour timeout per build cycle, that could be spent all on a single job. Looking at IceCat's most recent build log, it took 4.5 hours. You can see all the build logs here:
- https://f-droid.org/wiki/page/org.gnu.icecat
- https://f-droid.org/wiki/page/org.mozilla.fennec_fdroid
fdroid has its
srclibsmechanism for caching reusable library builds, that might be useful. The F-Droid community is also open to ideas for improving things, even if specifically for Tor Project. One simple thing would be to make the buildserver always tryorg.torproject.*builds first in each build cycle. There is a separate process that queues updates, so that quirk would only take effect when there is aorg.torproject.*update to build.CI builds on fdroid infrastructure
We have a few instances of the complete build infrastructure running for CI tests. We also have an alpha gitlab-runner of the full buildserver VM. For TBA, there could be a GitLab repo somewhere where TBA is built on an fdroid buildserver on every push. Here are two publicly visible instances:
Running the repo
If the fdroid repo was run on https://www.torproject.org/fdroid, then no new server or system needed. It is just copying static files to an directory in an existing webserver. Then the direct download links can just be symlinks or hard links to the files in the fdroid repo, that is easily scripted. It could be a static link for the latest release, since he
fdroid updatecan generate a symlink to the 'current version'. The whole fdroidserver tool suite is build/sign/publish automation, since f-droid.org is only possible with a large amount of automation.