CI Failures
This page lists the CI failures for the network team.
It's based on the Network Team's 0.4.0 Release Status page, and teor's user page.
CI Builders
- Jenkins: https://jenkins.torproject.org/view/tor/
- Travis: https://travis-ci.org/torproject/tor/branches
- Appveyor: https://ci.appveyor.com/project/torproject/tor/history
Jenkins builds our supported branches and Debian packages for deb.torproject.org. Notifications go to #tor-bots on irc.oftc.net.
Travis and Appveyor build GitHub pull requests, personal repositories, and our supported branches. Notifications go to #tor-ci on irc.oftc.net.
Handling CI Failures
We want to log every CI failure in trac, so we can see which CI issues we need to fix first.
Supported Branches
We want CI to always pass on supported branches.
When CI fails on:
- master, a supported maint branch, or a supported release branch, or
- a pull request, but it's not related to the changes in the pull request.
- Find the failure log message, the failed branch, and the merge head (if it's a pull request).
- If there is an existing ticket, log that information on the existing ticket.
- Otherwise, log a ticket tagged "tor-ci-fail" with the information, and mark it as high priority.
- If the failure is on an earlier supported branch, update the version and backport tags to that branch.
Pull Requests
When CI fails on a pull request, and it was caused by the changes in the pull request, mark the ticket for that pull request as needs_revision.
CI Failure Tickets
Unassigned tickets
These open tickets are tagged ci-fail, and they don't have an owner.
TODO: who will assign them to the best person to fix the ticket?
[[TicketQuery(status!=closed,owner=,keywords~=ci-fail,group=owner,format=table,order=changetime,desc=true,col=id|summary|status|owner|reviewer|priority|severity|changetime|sponsor,max=100)]]
Owner Action tickets
These tickets are tagged ci-fail, they have an owner, and they are in a status that needs action from the owner.
Anyone can change the assignments:
- assign themselves one of these tickets,
- un-assign ticket from themselves, or
- ask someone else to take one of their tickets.
[[TicketQuery(status!=needs_review|merge_ready|closed,owner!=,keywords~=ci-fail,group=owner,format=table,order=changetime,desc=true,col=id|summary|status|owner|reviewer|priority|severity|changetime|sponsor,max=100)]]
Needs Review tickets
These tickets are tagged ci-fail, and they are in needs_review.
[[TicketQuery(status=needs_review,keywords~=ci-fail,group=reviewer,format=table,order=changetime,desc=true,col=id|summary|status|owner|reviewer|priority|severity|changetime|sponsor,max=100)]]
Merge Ready tickets
These tickets are tagged ci-fail, and they are in merge_ready.
[[TicketQuery(status=merge_ready,keywords~=ci-fail,group=milestone|reviewer,format=table,order=changetime,desc=true,col=id|summary|status|owner|reviewer|priority|severity|changetime|sponsor,max=100)]]
CI failures being diagnosed
As of 23 Sep 2019:
-
tor-ci-linux-0.2.9 (old-ish)transient; cleared up by itself in the following build
see https://jenkins.torproject.org/job/tor-ci-linux-0.2.9/292/console
This looks like a travis failure. There's a java exception caused by a timeout, and no builders actually happen.
-
tor-ci-linux-0.4.1 (new-ish)transient; cleared up by itself in the following build
see https://jenkins.torproject.org/job/tor-ci-linux-0.4.1/ARCHITECTURE=amd64,SUITE=buster/28/console
Buster amd64 is failing with what appears to be
08:50:53 08:50:53 FAIL: src/test/test_workqueue_efd.sh
08:50:53 ====================================
08:50:53 08:50:53 Illegal instruction
08:50:53 FAIL src/test/test_workqueue_efd.sh (exit status: 132)I wonder if this is transient, or if it is a real problem? We didn't change this code any time recently.
-
tor-ci-linux-master-clang (new-ish)transient? seems to have fixed itself
This is only happening on i686 sid, and it is a compiler segfault. This makes me think it is a bug in the clang package in debian unstable.
Looks like there's now a complaint about an unrecognized warning flag? -catalyst see #31769 (moved)
15:39:37 cc1: error: command line option '-Wextra-semi' is valid for C++/ObjC++ but not for C [-Werror]
15:39:37 cc1: all warnings being treated as errors
15:39:37 make[1]: *** [Makefile:11538: src/feature/control/control_bootstrap.o] Error 1- tor-ci-mingwcross-* (old-ish)
This looks like a misconfiguration to me -- it seems to be complaining about not having found a zlib package maybe?
- tor-ci-windows-master (old?)
https://jenkins.torproject.org/job/tor-ci-windows-master/78/console
This looks like it could be a real bug: a unit test is failing. It's in "slow/process/callbacks" -- I believe ahf knows that code best.
07:58:07 addr/virtaddr_persist: [forking]
07:58:07 FAIL ../tor/src/test/test_addr.c:1415: assert(a OP_EQ "192.168.3.4"): <192.168.7.240> vs <192.168.3.4>
07:58:07 [virtaddr_persist FAILED]could be a network virtualization artifact?