Track chutney CI failures, and tweak the allow failures settings

CI failed one job on 0.2.9, but the debug run succeeded: https://travis-ci.org/torproject/chutney/jobs/505547963

We might need to increase the allowed failures for 0.2.9, or put a sleep in between the allow failures tests.

But we won't know which one to do until we see how CI works for a few days.

Trac:
Parent Ticket: legacy/trac#29729 (moved)

added chutney-ci in Legacy / Trac component::core tor/chutney in Legacy / Trac network-team-roadmap-2019-Q1Q2 in Legacy / Trac owner::teor in Legacy / Trac parent::29729 in Legacy / Trac points::1 in Legacy / Trac priority::medium in Legacy / Trac resolution::fixed in Legacy / Trac severity::normal in Legacy / Trac sponsor::19-must in Legacy / Trac status::closed in Legacy / Trac type::defect in Legacy / Trac labels

Here's another failure on 0.2.9, but the debug job succeeds: https://travis-ci.org/torproject/chutney/jobs/505552407

Maybe we should add a sleep between the allow failures tests, until 0.2.9 is unsupported.

The test sometimes succeeds, so 0.2.9 isn't reliably broken: https://travis-ci.org/torproject/chutney/jobs/505552915

(We also had a Travis network failure in another test, so we should tolerate some level of failure.)

Trac:
Owner: N/A to teor
Status: new to assigned

Same 0.2.9 failures: https://travis-ci.org/torproject/chutney/jobs/505567525 https://travis-ci.org/teor2345/chutney/jobs/505046093

0.3.4 failure, before test-network-forgiving: https://travis-ci.org/teor2345/chutney/jobs/505554142

Travis network failures: https://travis-ci.org/teor2345/chutney/jobs/505554152 https://travis-ci.org/teor2345/chutney/jobs/505562574

There have been about 13 builds, times 14 jobs per build.

0.2.9 failed on 4/13 jobs, failing 4 builds. 0.3.4 failed on 1/13 jobs, failing 1 build. Travis network failed on 2/182 jobs, failing 2 builds.

Let's get the chutney failure rate below the Travis failure rate?

Here's what I'm thinking of doing:

• add a sleep between the allow failure rounds, to hopefully decorrelate the 0.2.9 failures
• continue to monitor the Travis and 0.3.4 failure rate

These chutney tickets are on the network team roadmap, or they are required for tickets that are on the network team roadmap.

Trac:
Keywords: N/A deleted, network-team-roadmap-2019-Q1Q2 added

I haven't seen any of these failures since we implemented legacy/trac#22132 (moved). Let's continue to monitor and close this bug if we don't see any in the next week?

We might need this for legacy/trac#29729 (moved).

Trac:
Parent: N/A to legacy/trac#29729 (moved)

This issue seems to be fixed by the bootstrap changes, let's open specific bugs for failures.

Trac:
Resolution: N/A to fixed
Status: assigned to closed

closed

changed time estimate to 8h

mentioned in issue legacy/trac#29763 (moved)

mentioned in issue legacy/trac#29729 (moved)

moved from legacy/trac#29761 (moved)

added Bug + 1 deleted label and removed 1 deleted label

added CI label and removed 1 deleted label