Jenkins workers' connectivity with Fastly is unreliable → breaks many builds & tests
For GitLab CI the problem was workaround'ed on #18195 (closed).
But we also have the problem with:
- APT using Fastly sources in Tails (https://gitlab.tails.boum.org/tails/tails/-/issues/20870)
- Impact: this has been the most common source of build failures on Jenkins since March 7.
- Workarounds: we have a couple workarounds in mind but they all have significant drawbacks (either they cause UX regressions for Tails users, or they increase the gap between what we're testing and what users actually use, which can cause both false negatives & false positives).
- downloading from
downloads.openwrt.org
over HTTPS while building Tails (https://gitlab.tails.boum.org/tails/tails/-/issues/20887)- Impact: this makes many builds fail on Jenkins.
- Workarounds: none so far, but we did not think about it much yet. Worst case we can run the failing operation in a retry loop, hopefully it'll end up succeeding.
Given how popular Fastly is, I anticipate we'll keep running into such problems, as long as our CI cannot reach Fastly servers reliably. It won't always be realistic to workaround the problems on the developers' side in a satisfying manner. Ideally the problem would be solved at its root. What are the chances this happens and on what timeline?
The answer to this last question will inform if/how/when the Tails Team puts efforts into implementing workarounds.