Add CI to combine translation files across versions.

Merge Info

Related Issues

• #42305 (closed)

Backporting

Timeline

• Immediate: patchset needed as soon as possible
• Next Minor Stable Release: patchset that needs to be verified in nightly before backport
• Eventually: patchset that needs to be verified in alpha before backport
• No Backport (preferred): patchset for the next major stable

(Optional) Justification

• Emergency security update: patchset fixes CVEs, 0-days, etc
• Censorship event: patchset enables censorship circumvention
• Critical bug-fix: patchset fixes a bug in core-functionality
• Consistency: patchset which would make development easier if it were in both the alpha and release branches; developer tools, build system changes, etc
• Sponsor required: patchset required for sponsor
• Localization: typos and other localization changes that should be also in the release branch
• Other: please explain

Merging

• Merge to tor-browser - !fixups to tor-browser-specific commits, new features, security backports
• Merge to base-browser - !fixups to base-browser-specific commits, new features to be shared with mullvad-browser, and security backports
  • NOTE: if your changeset includes patches to both base-browser and tor-browser please clearly label in the change description which commits should be cherry-picked to base-browser after merging

Issue Tracking

• Link resolved issues with appropriate Release Prep issue for changelog generation

Review

Request Reviewer

• Request review from an applications developer depending on modified system:
  • NOTE: if the MR modifies multiple areas, please /cc all the relevant reviewers (since gitlab only allows 1 reviewer)
  • accessibility : henry
  • android : clairehurst, dan
  • build system : boklm
  • extensions : ma1
  • firefox internals (XUL/JS/XPCOM) : ma1
  • fonts : pierov
  • frontend (implementation) : henry
  • frontend (review) : donuts, richard
  • localization : henry, pierov
  • macos : clairehurst, dan
  • nightly builds : boklm
  • rebases/release-prep : dan, ma1, pierov, richard
  • security : ma1
  • signing : boklm, richard
  • updater : pierov
  • misc/other : pierov, richard

Change Description

/cc @richard for review as well

/cc @emmapeel since this will effect the translation repo

/cc @ma1 since you sometimes merge to tpo/tor-browser and will be triggering these pipelines.

All of you have been given Reporter roles in tor-browser-translation-bot/translation so you can view the other merge request: https://gitlab.torproject.org/tor-browser-translation-bot/translation/-/merge_requests/3

On tor-browser

This adds a gitlab CI pipeline for the tor-browser repository, which includes a single stage: "update-translations".

This stage will be triggered for the gitlab default branch whenever either:

• A commit is added to the gitlab default branch, which changes a ".ftl", ".properties" or ".dtd" file.
• Or, the pipeline is run manually with FORCE_UPDATE_TRANSLATIONS set to true.

NOTE: gitlab pipelines are assigned to a user who triggered them, and will have their permissions. In the first case, the user who pushed the commit is the user who triggers the pipeline.

This will call the combine-translation-versions.py, which will:

1. Assume the current HEAD (i.e. the tip of the branch this pipeline was triggered for) contains the most recent translation files, used in nightly (and alpha).
2. Assume that the most recently created git tag containing the word "stable" in its annotations contains the translation used for the stable release. This version should be within 0.5 of HEAD's branch.
3. For each translation file, take all the strings from the nightly version as the basis, then append any extra strings that are still used in the stable release. These extra strings have a comment added to them, noting that they won't stick around for long.
4. Combine the strings into a new translation file. Basically this file contains both the strings needed for nightly as well as the stable release. Strings that are in the alpha release but not nightly or stable will be missing, but I think this would be rare and generally ok for alpha.

The resulting translation file content is packaged into a JSON, along with some metadata. This JSON is saved as a CI job artifact.

Once we have the JSON artifact, this will trigger a pipeline for tor-browser-translation-bot/translation:tor-browser-ci, passing in the JSON artifact url as a variable.

NOTE: The pipeline in the other project can only be triggered if the user who triggered the tor-browser pipeline also has permissions for the other project. So for this to work, all tor-browser members that can merge should be added to tor-browser-translation-bot/translation with Developer roles, which I will add once we're sure about this approach. Is this currently just @pierov, @richard and @ma1?

NOTE: I wrote the CI so that it could work for firefox-android as well. But we can test it with tor-browser repo first.

On tor-browser-translation-bot/translation

I have a bot account, just for this task: @tor-browser-translation-bot.

They have their own translation repository containing the tor-browser-ci branch. The project is private, so only members should be able to see it.

Basically, this branch contains a gitlab CI to accept the JSON from tpo/applications/tor-browser, and create new merge requests in tpo/translation to update the tor-browser or base-browser branches. It should be reviewed in tandem with this, found at: https://gitlab.torproject.org/tor-browser-translation-bot/translation/-/merge_requests/3. Please add comments about the other script there instead.

@emmapeel once we're sure about this, I think you will need to give tor-browser-translation-bot sufficient permission to open and edit merge requests in tpo/translation. I think Developer role would be required.

How Tested

Tested with my henry/tor-browser and henry/translation repositories. I'll give the final version another test before marking this as not a Draft.