tor-browser CI still causes Gitlab instance slowdowns after mitigations
Since the linter CI was merged July 11th developers started noticing whenever an MR was merged that usually resulted in Gitlab slowness. The overall bouts of sloness in Gitlab is being investigated over on tpo/tpa/team#41677 (closed). One of the leads on the issue is a large number of pack-object downloads on the tor-browser repository, which led us to the conclusion that CI fetching multiple times in parallel is one of the leading causes of the issue.
On !1081 (merged) I implemented a potential fix to the problem, by adding a CI job that runs before all other jobs, does the fetch, then bundles the repository and uploads the bundle as an artifact. The artifact is what gets downloaded by other jobs which can therefore skip the git fetch.
After that was merged, we noticed there were still latency spikes showing up on Gitlab. After further investigation I confirmed the latency issues still have a strong correlation with the tor-browser CI -- see tpo/tpa/team#41677 (comment 3072217) for a comprehensive report on that investigation.
I am creating this ticket to track my new attempt to address the tor-browser CI vs. Gitlab issue. We have a new lead on the problem, which at the moment remains unconfirmed. The git-bundle approach might be inadequate, because when we upload the repository as an artifact the artifact is still uploaded to the Gitlab server. Therefore each CI job is still making a request to download a ~800M artifact from the Gitlab server, arriving at the same issue we had when git fetching.