Trac issueshttps://gitlab.torproject.org/legacy/trac/-/issues2020-06-16T01:26:27Zhttps://gitlab.torproject.org/legacy/trac/-/issues/34228Update Linux toolchain for ESR 782020-06-16T01:26:27ZGeorg KoppenUpdate Linux toolchain for ESR 78We need to go over out Linux toolchain and update it wherever we think we need it for Firefox 78 ESR.We need to go over out Linux toolchain and update it wherever we think we need it for Firefox 78 ESR.Georg KoppenGeorg Koppenhttps://gitlab.torproject.org/legacy/trac/-/issues/34227Update desktop toolchains for ESR 782020-06-16T01:26:26ZGeorg KoppenUpdate desktop toolchains for ESR 78This is the parent ticket for all desktop toolchain related workThis is the parent ticket for all desktop toolchain related workhttps://gitlab.torproject.org/legacy/trac/-/issues/34203Some of the static libraries we build are not reproducible2020-06-16T01:26:26ZGeorg KoppenSome of the static libraries we build are not reproducibleI just realized that the `.a` archives we create (e.g.) for `libevent` on android are not reproducible while their contents are. We should fix that as it makes it easier to compare results and spot problems.
While we are at it we should...I just realized that the `.a` archives we create (e.g.) for `libevent` on android are not reproducible while their contents are. We should fix that as it makes it easier to compare results and spot problems.
While we are at it we should check other outputs as well as I bet not only `lilbevent` is affected.
FWIW: In the `libevent` case it seems timestamps play a role when creating the `.a` files.https://gitlab.torproject.org/legacy/trac/-/issues/34187Update zlib build script to pick up new android toolchain2020-06-16T01:26:25ZGeorg KoppenUpdate zlib build script to pick up new android toolchainIt seems in order to pick up the new android toolchain we need to update our `zlib` project as well.It seems in order to pick up the new android toolchain we need to update our `zlib` project as well.Georg KoppenGeorg Koppenhttps://gitlab.torproject.org/legacy/trac/-/issues/34163testbuild target is broken for Tor Browser 64 bit (versionCode can't get comp...2020-06-16T01:26:24ZGeorg Koppentestbuild target is broken for Tor Browser 64 bit (versionCode can't get computed){{{
0:03.87 Traceback (most recent call last):
0:03.87 File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
0:03.88 "__main__", fname, loader, pkg_name)
0:03.88 File "/usr/lib/python2.7/runpy.py", line 72, in _ru...{{{
0:03.87 Traceback (most recent call last):
0:03.87 File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
0:03.88 "__main__", fname, loader, pkg_name)
0:03.88 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
0:03.88 exec code in run_globals
0:03.88 File "/var/tmp/build/firefox-4516c5951e84/python/mozbuild/mozbuild/action/file_generate.py", line 120, in <module>
0:03.88 sys.exit(main(sys.argv[1:]))
0:03.88 File "/var/tmp/build/firefox-4516c5951e84/python/mozbuild/mozbuild/action/file_generate.py", line 71, in main
0:03.88 ret = module.__dict__[method](output, *args.additional_arguments, **kwargs)
0:03.88 File "/var/tmp/build/firefox-4516c5951e84/mobile/android/base/generate_build_config.py", line 145, in generate_android_manifest
0:03.88 defines=_defines(),
0:03.88 File "/var/tmp/build/firefox-4516c5951e84/mobile/android/base/generate_build_config.py", line 129, in _defines
0:03.88 max_sdk=max_sdk)
0:03.88 File "/var/tmp/build/firefox-4516c5951e84/python/mozbuild/mozbuild/android_version_code.py", line 140, in android_version_code
0:03.88 return android_version_code_v0(buildid, *args, **kwargs)
0:03.88 File "/var/tmp/build/firefox-4516c5951e84/python/mozbuild/mozbuild/android_version_code.py", line 31, in android_version_code_v0
0:03.88 "for CPU arch %s" % cpu_arch)
0:03.88 ValueError: Don't know how to compute android:versionCode for CPU arch arm64-v8a
}}}
I *think* this will be fixed once we move away from ESR 68 as I've been doing test builds for #33626. Still, even if so, this is a bug in our current setup.https://gitlab.torproject.org/legacy/trac/-/issues/34110Investigate `./mach android gradle-dependencies` for our use cases2020-06-16T01:26:24ZGeorg KoppenInvestigate `./mach android gradle-dependencies` for our use casesMozilla has a neat way of automating the gradle dependencies it needs during build time and making them available: https://firefox-source-docs.mozilla.org/build/buildsystem/toolchains.html#firefox-for-android-with-gradle
We should think...Mozilla has a neat way of automating the gradle dependencies it needs during build time and making them available: https://firefox-source-docs.mozilla.org/build/buildsystem/toolchains.html#firefox-for-android-with-gradle
We should think about how we could use that either just for Fenix or in general for our mobile related projects.https://gitlab.torproject.org/legacy/trac/-/issues/34101Add tor-browser-build project for application-services2020-06-16T01:26:23ZGeorg KoppenAdd tor-browser-build project for application-servicesUnless we rip out all of the `application-services` (which I currently don't think we'll do) we need to have a project for it in `tor-browser-build`.
This will be a fun one to build. See the `lib` dir for some build scripts and [the meg...Unless we rip out all of the `application-services` (which I currently don't think we'll do) we need to have a project for it in `tor-browser-build`.
This will be a fun one to build. See the `lib` dir for some build scripts and [the megazord design](https://github.com/mozilla/application-services/blob/master/docs/design/megazords.md).Georg KoppenGeorg Koppenhttps://gitlab.torproject.org/legacy/trac/-/issues/34052Create a fonts tarball during tor browser build process2020-06-16T01:26:22ZboklmCreate a fonts tarball during tor browser build processTor Browser for Linux/Windows/macOS includes a specific list of fonts, which we get using `projects/fonts/`.
To make packaging for other OSs such as OpenBSD and NetBSD easier, we can create and distribute a tarball containing the fonts ...Tor Browser for Linux/Windows/macOS includes a specific list of fonts, which we get using `projects/fonts/`.
To make packaging for other OSs such as OpenBSD and NetBSD easier, we can create and distribute a tarball containing the fonts from the linux build.
An other file we might want to include in/with this tarball is `projects/tor-browser/Bundle-Data/linux/Data/fontconfig/fonts.conf`.https://gitlab.torproject.org/legacy/trac/-/issues/34051Generate list of all dependencies and additional files2020-06-16T01:26:22ZMatthew FinkelGenerate list of all dependencies and additional filesExternal Tor Browser packages (for other platforms) would find it helpful if we produce a list of all dependencies used for building Tor Browser for a platform and if those dependencies were build using custom patches. This list should i...External Tor Browser packages (for other platforms) would find it helpful if we produce a list of all dependencies used for building Tor Browser for a platform and if those dependencies were build using custom patches. This list should include any additional files we inject into the final packages (such as licenses, start script, fonts, etc.).https://gitlab.torproject.org/legacy/trac/-/issues/34046Sign commits with gpg2020-06-16T01:26:22ZboklmSign commits with gpgAs discussed in ticket:25102#comment:20, we should sign all top commits from branches that are used in nightly builds.As discussed in ticket:25102#comment:20, we should sign all top commits from branches that are used in nightly builds.https://gitlab.torproject.org/legacy/trac/-/issues/34014Support sqlite3 in our python project2020-06-16T01:26:21ZGeorg KoppenSupport sqlite3 in our python projectPython3 we use needs sqlite3 support now.Python3 we use needs sqlite3 support now.Georg KoppenGeorg Koppenhttps://gitlab.torproject.org/legacy/trac/-/issues/34013Bump node version to v10.21.02020-06-16T01:26:20ZGeorg KoppenBump node version to v10.21.0Update our node version to what is used in mozilla-central.Update our node version to what is used in mozilla-central.Georg KoppenGeorg Koppenhttps://gitlab.torproject.org/legacy/trac/-/issues/34012Bump cbindgen version to 0.14.12020-06-16T01:26:20ZGeorg KoppenBump cbindgen version to 0.14.1Update to latest cbindgen used on mozilla-central.Update to latest cbindgen used on mozilla-central.Georg KoppenGeorg Koppenhttps://gitlab.torproject.org/legacy/trac/-/issues/34011Bump clang version to 9.0.12020-06-16T01:26:19ZGeorg KoppenBump clang version to 9.0.1Let's go away from clang 8.0.1Let's go away from clang 8.0.1Georg KoppenGeorg Koppenhttps://gitlab.torproject.org/legacy/trac/-/issues/33973Create fat .aar for geckoview2020-06-16T01:26:18ZGeorg KoppenCreate fat .aar for geckoviewDownstream consumers like `android-components` and `fenix` use fat .aar files. We need to create them out of ouf per-arch ones. https://bugzilla.mozilla.org/show_bug.cgi?id=1508976 is the bug where this got implemented on Mozilla's side.Downstream consumers like `android-components` and `fenix` use fat .aar files. We need to create them out of ouf per-arch ones. https://bugzilla.mozilla.org/show_bug.cgi?id=1508976 is the bug where this got implemented on Mozilla's side.Georg KoppenGeorg Koppenhttps://gitlab.torproject.org/legacy/trac/-/issues/33953Provide a way for easily updating Go dependencies of projects2020-06-16T01:26:16ZGeorg KoppenProvide a way for easily updating Go dependencies of projectsIn #28325 we are working on enabling `go.mod` support for our Go based projects. While that will be an improvement in that we don't need to work around a new default anymore, it does not solve the question how we can easily update all th...In #28325 we are working on enabling `go.mod` support for our Go based projects. While that will be an improvement in that we don't need to work around a new default anymore, it does not solve the question how we can easily update all the dependencies of a project and the dependencies of those dependencies. That's the scope for this ticket. Besides #28325 we should keep in mind that we might want to think about how we want to handle go modules in the future. Right now, we have a project per module in `tor-browser-build`. Maybe that's not a smart thing with the growing amount of projects we have. Either way, the solution for this ticket will have an impact on that, so we should take the question about how to handle the module-project relationship into account.
For the dependency update path forward we a bunch of possible options, some already mentioned in #28325 (in no particular order):
1) Use `go mod vendor` to vendor in the dependencies and then build with `-mod=vendor` to use the `vendor` folder with the dependencies.
2) Refine the `gomodtorbm` script written by dcf/cohosh (see: #28942 and #33576)
3) Use `go mod download` to fetch dependencies into the cache and then point, during the build, with `GOPROXY` to the cached files (that's feeling similar to what we use for our Gradle dependencies right now).
There might be more than those three.
boklm had a nice idea of restructuring our go projects in a way that we'd only have one `go-module` project and we could list all needed dependencies directly in the respective project under `input_files` (see: comment:42:ticket:28942). That might be orthogonal to 1) and 3) but might actually help with 2). not sure.https://gitlab.torproject.org/legacy/trac/-/issues/33936Make our fetch-gradle-dependencies script smarter regarding dependency location2020-06-16T01:26:16ZGeorg KoppenMake our fetch-gradle-dependencies script smarter regarding dependency locationWe fetch gradle dependencies with our `fetch-gradle-dependencies` script but don't really care where exactly we put them in that process. We should change that, though, because otherwise we need to do in build scripts things like
`cp -r ...We fetch gradle dependencies with our `fetch-gradle-dependencies` script but don't really care where exactly we put them in that process. We should change that, though, because otherwise we need to do in build scripts things like
`cp -r $gradle_repo/m2/* $gradle_repo` to adjust the paths so that gradle can find them.
That's a thing our script should do in the first place to avoid complexity in our build scripts.Georg KoppenGeorg Koppenhttps://gitlab.torproject.org/legacy/trac/-/issues/33935Fenix's classes5.dex files are not reproducible2020-06-16T01:26:15ZGeorg KoppenFenix's classes5.dex files are not reproducibleWhen building .apk files with the https://gitweb.torproject.org/user/gk/tor-browser-build.git/commit/?h=bug_33927_v2&id=a9eca52f63d049f0cb1a03c14971c8743d79a2b1 it turns out the `classes5.dex` files are not getting built deterministicall...When building .apk files with the https://gitweb.torproject.org/user/gk/tor-browser-build.git/commit/?h=bug_33927_v2&id=a9eca52f63d049f0cb1a03c14971c8743d79a2b1 it turns out the `classes5.dex` files are not getting built deterministically for all 4 architectures. Looking at the diff for armv7 (attached) there is at least a timestamp issues we need to fix, but maybe more.https://gitlab.torproject.org/legacy/trac/-/issues/33934Fix Fenix reproducibility issues2020-06-16T01:26:15ZGeorg KoppenFix Fenix reproducibility issuesFenix has reproducibility issues it seems. We should collect them in child tickets and address them.Fenix has reproducibility issues it seems. We should collect them in child tickets and address them.https://gitlab.torproject.org/legacy/trac/-/issues/33932Improve steps for creating gradle dependencies lists for projects2020-06-16T01:26:14ZGeorg KoppenImprove steps for creating gradle dependencies lists for projectsThe steps we have for creating lists of gradle dependencies are still not bullet proof. I stumbled across a case where a download got attempted but then a redirect happened which included a different URL and which finally failed. Our cur...The steps we have for creating lists of gradle dependencies are still not bullet proof. I stumbled across a case where a download got attempted but then a redirect happened which included a different URL and which finally failed. Our current instructions don't cope with that case (and probably other corner cases neither).
I've thinking about that and feel we can do better if we mimmick gradle's behavior more:
1) We keep the first step we currently have (extracting all the attempted downloads)
2) We actually try to download all those resources ourselves and compute their sha256 sum.
3) Download failures are easily removable (no sha256 sum is created in that case) and we just remove the successful duplicates.Georg KoppenGeorg Koppen