tor-browser-build issueshttps://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues2024-03-05T13:58:33Zhttps://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/41091Check if we can export the debug symbols also for Android2024-03-05T13:58:33ZPier Angelo VendrameCheck if we can export the debug symbols also for AndroidSometimes we would like to be able to debug Geckoview's native code (e.g., if it crashes).
We will need to force the APK to be debuggable (which we can turn on only if needed), but at least we could avoid to rebuild GeckoView.
I think ...Sometimes we would like to be able to debug Geckoview's native code (e.g., if it crashes).
We will need to force the APK to be debuggable (which we can turn on only if needed), but at least we could avoid to rebuild GeckoView.
I think that a command like this could help:
```bash
find toolkit/library/build mozglue/build security -name '*.so' -exec bash -c 'mkdir -p $(dirname "/var/tmp/dist/geckoview/libraries/{}"); cp -a {} /var/tmp/dist/geckoview/libraries/{}' \;
```
I ran it inside the `obj-$arch...` directory (notice that at the end we have two `obj-...` directories in testbuilds, since we create the fake fat AAR).
These directories come from the fact that they are the ones [Mozilla is adding to lldb's configuration](https://firefox-source-docs.mozilla.org/mobile/android/geckoview/contributor/native-debugging.html#set-up-lldb-to-find-your-symbols).
Notice that I'm doing a full `cp` rather than an `objcopy`: the difference is 0.1GB in uncompressed files, and I haven't tried if symbols without an explicit gnu link work (maybe we'd have to extract the aar and re-create it to add the debug link, and still I'm not sure it will work).
However, the final size of each artifact will grow from around 50MB to almost 500MB!
So, we maybe should we apply it only to testbuilds?https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/41011Remove tor-onion-proxy-library project2023-11-06T23:31:41ZrichardRemove tor-onion-proxy-library projectThis will be deprecated with the remaining ~"Sponsor 96" work.This will be deprecated with the remaining ~"Sponsor 96" work.https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/41010Create a project to ship tor binaries in an Android-developer friendly way2024-02-05T18:15:26ZrichardCreate a project to ship tor binaries in an Android-developer friendly way`tor-onion-proxy-library` is going away, we need to setup the torrc and populate tor+PTs using the `tor-expert-bundle` project`tor-onion-proxy-library` is going away, we need to setup the torrc and populate tor+PTs using the `tor-expert-bundle` projectSponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibethttps://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/40992Updated torbrowser_version number is not enough to change firefox-android ver...2024-03-05T16:18:43ZboklmUpdated torbrowser_version number is not enough to change firefox-android versionCode numberWhen a new Tor Browser version doesn't include any new firefox-android
commit, its versionCode number will not change, although
`MOZ_BUILD_DATE` is changing.
The reason is that the function `generateFennecVersionCode` in
`fenix/buildSrc...When a new Tor Browser version doesn't include any new firefox-android
commit, its versionCode number will not change, although
`MOZ_BUILD_DATE` is changing.
The reason is that the function `generateFennecVersionCode` in
`fenix/buildSrc/src/main/java/Config.kt` is taking the date from
`MOZ_BUILD_DATE` and using the number of hours from December 28, 2014,
to generate a versionCode. However, `MOZ_BUILD_DATE` is based on the
commit date, on which we add the Tor Browser version as seconds, so
when a new version doesn't change the `firefox-android`,
`MOZ_BUILD_DATE` will only be change by a few seconds, which is not
enough to make `generateFennecVersionCode` update the versionCode.
To fix this we can add an empty firefox-android commit as a quick
workaround for this release, before finding a better way to handle
changing `MOZ_BUILD_DATE` for new releases.boklmboklmhttps://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/40701Make GeckoView take the branch/tags from Firefox2023-08-26T06:05:48ZPier Angelo VendrameMake GeckoView take the branch/tags from FirefoxA while ago, we made GeckoView use the same branch as Firefox for desktop (tor-browser#41308).
However, we did not change the GeckoView project to use the same branch as Firefox, but we're keeping in sync always (even when we do desktop...A while ago, we made GeckoView use the same branch as Firefox for desktop (tor-browser#41308).
However, we did not change the GeckoView project to use the same branch as Firefox, but we're keeping in sync always (even when we do desktop-only changes or Android-only changes, even though the latter are definitely rarer).
We should modify GV's config to take the same branch as FF automatically.
Last nightly, for example, failed because I forgot to update the nightly has on GV.https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/40570Don't include version information in GeckoView project2022-12-22T10:58:23ZGeorg KoppenDon't include version information in GeckoView projectWe include Tor Browser version information when building the Firefox
part, e.g. in order to get the (incremental) update files right (see:
tor-browser-build#18326) or before accessing the profile has been loaded (see: tor-browser-build#1...We include Tor Browser version information when building the Firefox
part, e.g. in order to get the (incremental) update files right (see:
tor-browser-build#18326) or before accessing the profile has been loaded (see: tor-browser-build#18325).
But that should not be needed when building the GeckoView .aar as the
application parts move to Fenix.
We should investigate and fix that as not needing to rebuild the whole
GeckoView for Android could speed up our mobile releases a lot in case we
need to get new ones out fast due to GeckoView unrelated reasons.https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/40556Show the Tor Browser version number on f-droid (instead of firefox version)2022-08-02T12:17:24ZboklmShow the Tor Browser version number on f-droid (instead of firefox version)It seems that some users are confused by the version number that is shown on f-droid:
https://blog.torproject.org/comment/285989#comment-285989
It looks like the version number that is shown is the Firefox version on which it is based, ...It seems that some users are confused by the version number that is shown on f-droid:
https://blog.torproject.org/comment/285989#comment-285989
It looks like the version number that is shown is the Firefox version on which it is based, instead of the Tor Browser version.https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/40544Bump Java 8 version to something more recent2023-01-05T12:37:45ZGeorg KoppenBump Java 8 version to something more recentWe take the currently used Java 8 version from a snapshot from 2019. That's pretty old and missing a bunch of security fixes. We should pick up a newer version.We take the currently used Java 8 version from a snapshot from 2019. That's pretty old and missing a bunch of security fixes. We should pick up a newer version.https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/40539Investigate optimizing APK for Smart App Updating2022-06-23T19:57:53ZMatthew FinkelInvestigate optimizing APK for Smart App UpdatingIn mid-2012, beginning with Jelly Bean (4.1), Google Play began using Smart App Updates [0]
```
Smart App Updates
Smart app updates is a new feature of Google Play that introduces a better way
of delivering app updates to devices. When ...In mid-2012, beginning with Jelly Bean (4.1), Google Play began using Smart App Updates [0]
```
Smart App Updates
Smart app updates is a new feature of Google Play that introduces a better way
of delivering app updates to devices. When developers publish an update, Google
Play now delivers only the bits that have changed to devices, rather than the
entire APK. This makes the updates much lighter-weight in most cases, so they
are faster to download, save the device’s battery, and conserve bandwidth usage
on users’ mobile data plan. On average, a smart app update is about 1/3 the
size of a full APK update.
```
Considering Orfox is now ~30MB, and this may increase with additional locales, we should track the current diff patch size Google Play distributes for incremental updates, and we should investigate if we can tweak our APK so it provides minimal diffs.
Can we (nearly) reproduce their results?
StackOverflow suggests they are using GDIFF [1].
Google was nice enough to publish some more details (4 years later) about how they're now using a newer diff method in some situations [2]:
```
For approximately 98% of app updates from the Play Store, only changes (deltas)
to APK files are downloaded and merged with the existing files, reducing the
size of updates. Google Play has used delta algorithms since 2012, and we
recently rolled out an additional delta algorithm, bsdiff (created by Colin
Percival1), that our experimentation shows can reduce delta size by up to 50%
or more compared to the previous algorithm for some APKs. Bsdiff is
specifically targeted to produce more efficient deltas of native libraries by
taking advantage of the specific ways in which compiled native code changes
between versions. To be most effective, native libraries should be stored
uncompressed (compression interferes with delta algorithms).
```
Unfortunately, we are restricted by our user base and by which platforms Mozilla is targeting. At this point, Mozilla are still targeting SDK 16, which is good for us, but we don't get new optimizations:
```
However, please note, native libraries should only be uncompressed when the
minimum SDK version for an APK is 23 (Marshmallow) or later
```
So we should keep bsdiff optimizations in mind for when Mozilla eventually move to min-sdk 23+.
[0] https://developer.android.com/about/versions/jelly-bean.html
[1] https://stackoverflow.com/questions/12860938/smart-app-updates-on-google-play-store-how-does-it-work/12877791#12877791
[2] https://android-developers.googleblog.com/2016/07/improvements-for-smaller-app-downloads.htmlhttps://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/40480Compile compiler-rt with the new Clang on Android2022-12-22T11:11:19ZPier Angelo VendrameCompile compiler-rt with the new Clang on AndroidIn #40433 we updated LLVM/clang to version 13.0.1, however there is still room for improvement:
- [ ] build `compiler-rt` with the newly built compiler, instead of the NDK
- we need to investigate how to use Clang without the NDK
- [ ...In #40433 we updated LLVM/clang to version 13.0.1, however there is still room for improvement:
- [ ] build `compiler-rt` with the newly built compiler, instead of the NDK
- we need to investigate how to use Clang without the NDK
- [ ] make sure `lld` works with the x86/x86_64 NDK libraries that have compressed parts (see this [similar issue](https://github.com/android/ndk/issues/888))
- we've always used the NDK's linker, so this isn't a big deal, I think (I haven't found what Mozilla does, yet)
- I may have resolved this while switching desktop to 102
- [ ] move Android-specific parts from projects/clang to a new project
- Windows and macOS build projects/clang, and then use it to compile their compiler-rt and libc++ **in another project** (`mingw-w64-clang` and `macosx-toolchain`, respectively).
- If we removed all the platform-specific settings from clang, we could compile LLVM once (at least share it between Android and macOS, since they both target x86, arm and aarch64, or enable all the architectures also for Windows; Linux is built on an old host, so no way to share it at the moment...)https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/40421Expand/revise android build docs2022-12-22T10:54:03ZaguestuserExpand/revise android build docs
# context
- this build workflow has several layers with lots of moving parts, some stale docs, and non-obvious gotchas could use some documentation to make it easier/faster to avoid/fix mistakes
- we could also (IMHO) use some common kn...
# context
- this build workflow has several layers with lots of moving parts, some stale docs, and non-obvious gotchas could use some documentation to make it easier/faster to avoid/fix mistakes
- we could also (IMHO) use some common knowledge of steps we take during builds (gleaned from discussion in tor-browser#40784 and tor-browser-build#40418) to communicate more smoothly about our work
# tasks
- [ ] expand `/docs/how-to-create-gradle-dependencies-list.txt` to include more explicit pointers of when manual overrides for `git_url` and `git_target` are needed in step (1), and how copying/renaming `gradle-dependencies.txt` works in step (3)
- [ ] add `/docs/how-to-run-android-build` to capture all steps for generating and publishing a build for a new in a TODO-list-like formathttps://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/40281Make it easier to verify that the unsigned apk from a build is matching the s...2022-12-22T11:11:10ZboklmMake it easier to verify that the unsigned apk from a build is matching the signed apk we publishedI think we can have a script or makefile command to make it easier for someone who rebuilt tor browser to check that the unsigned apk from their build is matching the signed apk we published.
To do that I think we can use apksigcopier:
...I think we can have a script or makefile command to make it easier for someone who rebuilt tor browser to check that the unsigned apk from their build is matching the signed apk we published.
To do that I think we can use apksigcopier:
https://github.com/obfusk/apksigcopierboklmboklmhttps://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/40202Use constant placeholder for pom file instead of file hash2023-07-12T14:09:09ZMatthew FinkelUse constant placeholder for pom file instead of file hashIn #40163 we stopped comparing the hash of the pom files. We should stop including it entirely and use a placeholder instead. This will make comparing modifications easier.In #40163 we stopped comparing the hash of the pom files. We should stop including it entirely and use a placeholder instead. This will make comparing modifications easier.https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/40069Remove the armeabi parts in tor-onion-proxy-library's build script2022-12-22T11:04:38ZGeorg KoppenRemove the armeabi parts in tor-onion-proxy-library's build scriptWe only need `armeabi-v7a` but not `armeabi`, so no need to copy `obfs4`
and `snowflake` over. We probably should just nuke that part before
starting the build.We only need `armeabi-v7a` but not `armeabi`, so no need to copy `obfs4`
and `snowflake` over. We probably should just nuke that part before
starting the build.https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/40056Get all mobile dependencies during dry run2023-07-17T12:52:26ZGeorg KoppenGet all mobile dependencies during dry runThe first build after a version bump is usually with networking allowed
to get all the Gradle dependencies extracted. That step is currently
failing for `application-services`, `android-components`, and `fenix` for unknown reasons: the G...The first build after a version bump is usually with networking allowed
to get all the Gradle dependencies extracted. That step is currently
failing for `application-services`, `android-components`, and `fenix` for unknown reasons: the Gradle
`--debug` output does not show artifact downloads for some dependencies
for later re-use in our mavenLocal setup.https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/34203Some of the static libraries we build are not reproducible2023-01-05T14:34:24ZGeorg KoppenSome of the static libraries we build are not reproducibleI just realized that the `.a` archives we create (e.g.) for `libevent` on android are not reproducible while their contents are. We should fix that as it makes it easier to compare results and spot problems.
While we are at it we should...I just realized that the `.a` archives we create (e.g.) for `libevent` on android are not reproducible while their contents are. We should fix that as it makes it easier to compare results and spot problems.
While we are at it we should check other outputs as well as I bet not only `lilbevent` is affected.
FWIW: In the `libevent` case it seems timestamps play a role when creating the `.a` files.https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/34110Investigate `./mach android gradle-dependencies` for our use cases2022-08-03T12:18:56ZGeorg KoppenInvestigate `./mach android gradle-dependencies` for our use casesMozilla has a neat way of automating the gradle dependencies it needs during build time and making them available: https://firefox-source-docs.mozilla.org/build/buildsystem/toolchains.html#firefox-for-android-with-gradle
We should think...Mozilla has a neat way of automating the gradle dependencies it needs during build time and making them available: https://firefox-source-docs.mozilla.org/build/buildsystem/toolchains.html#firefox-for-android-with-gradle
We should think about how we could use that either just for Fenix or in general for our mobile related projects.https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/33936Make our fetch-gradle-dependencies script smarter regarding dependency location2022-08-03T12:22:36ZGeorg KoppenMake our fetch-gradle-dependencies script smarter regarding dependency locationWe fetch gradle dependencies with our `fetch-gradle-dependencies` script but don't really care where exactly we put them in that process. We should change that, though, because otherwise we need to do things like
`cp -r $gradle_repo/m2/*...We fetch gradle dependencies with our `fetch-gradle-dependencies` script but don't really care where exactly we put them in that process. We should change that, though, because otherwise we need to do things like
`cp -r $gradle_repo/m2/* $gradle_repo` in build scripts to adjust the paths so that gradle can find them.
That's a thing our script should do in the first place to avoid complexity in our build scripts.https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/32477Clean up obfs4/go build on Android2023-01-05T14:16:33ZboklmClean up obfs4/go build on AndroidWith legacy/trac#28803 (commit eee5d30a9ab1d727caac262cb62f72aaab75e0a0), we added support for Android for the obfs4 build.
In some of the go dependencies, we setup `var/compiler` (for example `gobsaes`), but not in some others (for exa...With legacy/trac#28803 (commit eee5d30a9ab1d727caac262cb62f72aaab75e0a0), we added support for Android for the obfs4 build.
In some of the go dependencies, we setup `var/compiler` (for example `gobsaes`), but not in some others (for example `ed25519`).
We should:
- understand why it is needed for some projects and not others
- check if it is really needed for all the projects where we add it
- understand why it is needed for Android, but not for the other platformshttps://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/32200only include required bits of OpenSSL in Android builds2022-10-06T01:19:52Zeighthaveonly include required bits of OpenSSL in Android buildsI've been doing some experiments to make the Android _libtor.so_ binaries smaller. One of them is building OpenSSL with as many things as possible turned off. This does make the resulting binary smaller. Here's what I tried:
```
$ ./...I've been doing some experiments to make the Android _libtor.so_ binaries smaller. One of them is building OpenSSL with as many things as possible turned off. This does make the resulting binary smaller. Here's what I tried:
```
$ ./Configure \
no-comp no-dtls no-ec2m no-psk no-srp no-ssl2 no-ssl3 \
no-camellia no-idea no-md2 no-md4 no-mdc2 no-rc2 no-rc4 no-rc5 no-rmd160 no-whirlpool \
no-dso no-engine no-hw no-ui-console \
no-shared no-unit-test \
```
The open question is whether the test coverage is good enough to know whether this breaks anything.
Additionally, I think Android _ndk-build_ used to 'gcc' "gc sections" to mark unused code blocks which were then stripped out at the end. They seemed to have stopped doing this with _clang_, but I don't know why. In the past, I have seen the "gc sections" stripping reduce binary size quite a bit.
Also related: I tried building with `-Os` and `-Oz` instead of `-O2`. That made a big difference:
https://github.com/guardianproject/tor-android/issues/18
This is related to legacy/trac#28764