Skip to content

Disable chutney test again

Ian Jackson requested to merge Diziet/arti:chutney into main

Fixes #810 (closed) by disabling the test.

I appreciate that pwople have been trying to fix this test. In !1393 (merged), for example, where it was re-enabled after a period of failing every run (due to ssl library incompatibility between different images).

However, this test is still failing one time in 5. This is IMO far far outside the tolerable test failure rate, even for an allow-failure test. (Having a "passed with warnings" happen spuriously 20% of the time means that someone who introduces a warning for the first time will assume it's just the flaky test; as a result our tree always has warnings; so someone who sees warnings from CI will even more assume it's not their fault.)

We don't have formal guidelines on this, but I would suggest that this test should not be re-enabled until the failure rate can be brought down to 1:1000 (which I guess would put it roughly on a par with our handful of other flaky tests).

CC @trinity-1686a @nickm (because this is, effectively, reverting something they did, so I feel I should make sure they sew it; rather than because I think it's their job to fix)

Merge request reports