Get all mobile dependencies during dry run

The first build after a version bump is usually with networking allowed to get all the Gradle dependencies extracted. That step is currently failing for application-services, android-components, and fenix for unknown reasons: the Gradle --debug output does not show artifact downloads for some dependencies for later re-use in our mavenLocal setup.

changed title from Get all application-services dependency during dry run to "Get all application-services dependencies during dry run"

changed title from "Get all application-services dependencies during dry run" to Get all application-services dependencies during dry run

added 1 deleted label

mentioned in issue tor-browser#33184 (closed)

changed title from Get all application-services dependencies during dry run to Get all mobile dependencies during dry run

changed the description

It used to work for android-components and fenix, so this is probably a good hint for digging deeper.

assigned to @gk

added Maintenance Next labels

changed time estimate to 5h

changed milestone to %Tor Browser: 10.5

added Doing label and removed Next label

added Next label and removed Doing label

removed 1 deleted label

For #40217 (closed) we now get

./android-components/gradle-dependencies-list.txt:4cd24a06b2a253110d8afd250e9eec6c6faafea6463d740824743d637e761f12 | https://repo.maven.apache.org/maven2/org/jetbrains/kotlinx/kotlinx-coroutines-core-jvm/1.4.2/kotlinx-coroutines-core-jvm-1.4.2.jar
./android-components/gradle-dependencies-list.txt:4cd24a06b2a253110d8afd250e9eec6c6faafea6463d740824743d637e761f12 | https://repo.maven.apache.org/maven2/org/jetbrains/kotlinx/kotlinx-coroutines-core/1.4.2/kotlinx-coroutines-core-1.4.2.jar

so I am wondering whether Gradle is smarter here realizing that it already got one dependency with hash 4cd24a06b2a253110d8afd250e9eec6c6faafea6463d740824743d637e761f12 and does not download kotlinx-coroutines-core-1.4.2.jar additionally. That would essentially be a hash-based deps resolution instead of a file-/URL-based one we do.

However, spot-checking that for previous artifacts we needed to download manually this does not seem to hold: there are no resources with the same hashes in the dependency list file.

unassigned @gk

changed time estimate to 48h

changed milestone to %Tor Browser: 11.0 Issues with previous release

I have noticed that usually the file that is missing is a .jar or .aar file, but with a .pom file with the same name that we have.

Maybe we can update the script to always try to download .jar and .aar files for all .pom files?

This could lead to build errors if the pom is expected not to have an associated jar/aar: #40582 (comment 2823598).

mentioned in merge request !387 (closed)

mentioned in merge request tor-browser!243 (merged)

mentioned in issue #40418 (closed)

marked this issue as related to #40454 (closed)

added Q2 label

added temp label

added Roadmap::Future label and removed Next label

added Next label and removed Roadmap::Future temp labels

removed Next label

added Backlog label

added Q3 label and removed Q2 label

removed milestone %Tor Browser: 11.0 Issues with previous release

removed Backlog label

added Roadmap::Future label

removed Q3 label

added Feature label

added Android label

While I'd prefer a way to get dependencies directly without a dry run, I made some progress to use Gradle's cache from the dry-run build, instead of the logs while updating to 115: #40855 (comment 2906041).

In replies I've also attached a few scripts to scrape that cache and to try to create some URLs from that URL with some strategies.

We could try to automate these scripts more.

In replies I've also attached a few scripts to scrape that cache and to try to create some URLs from that URL with some strategies.

We could try to automate these scripts more.

Yes, I think the match.py script can be adapted to replace projects/common/gen_gradle_deps_file.sh.

My scripts failed to gather these files:

• https://maven.google.com/androidx/annotation/annotation/1.6.0/annotation-1.6.0.jar (it's transformed from annotation-metadata-1.6.0.jar, or something similar)
• https://repo1.maven.org/maven2/org/jetbrains/kotlinx/atomicfu/0.17.3/atomicfu-0.17.3.jar (not sure why)
• https://repo1.maven.org/maven2/org/jetbrains/kotlinx/atomicfu/0.17.3/atomicfu-0.17.3-all.jar (maybe because of the -all suffix?)
• https://repo1.maven.org/maven2/org/jetbrains/kotlinx/kotlinx-coroutines-core/1.6.4/kotlinx-coroutines-core-1.6.4-all.jar (I suppose because of the -all suffix again)
• https://jcenter.bintray.com/net/freehaven/tor/control/jtorctl/0.2/jtorctl-0.2.jar, https://jcenter.bintray.com/net/freehaven/tor/control/jtorctl/0.2/jtorctl-0.2.pom (maybe I was missing part of the tor integration)

It's a miss of 6 files. Not great, but quick enough to fix (it can be done with an offline build only).