|
|
|
# CI Failures
|
|
|
|
|
|
|
|
[[PageOutline]]
|
|
|
|
|
|
|
|
This page lists the CI failures for the network team.
|
|
|
|
|
|
|
|
It's based on the [Network Team's](./org/teams/NetworkTeam) [0.4.0 Release Status](./org/teams/NetworkTeam/CoreTorReleases/040Status) page, and [teor's user page](./user/teor).
|
|
|
|
|
|
|
|
## CI Builders
|
|
|
|
|
|
|
|
* Jenkins: https://jenkins.torproject.org/view/tor/
|
|
|
|
* Travis: https://travis-ci.org/torproject/tor/branches
|
|
|
|
* Appveyor: https://ci.appveyor.com/project/torproject/tor/history
|
|
|
|
|
|
|
|
Jenkins builds our supported branches and Debian packages for deb.torproject.org. Notifications go to #tor-bots on irc.oftc.net.
|
|
|
|
|
|
|
|
Travis and Appveyor build GitHub pull requests, personal repositories, and our supported branches. Notifications go to #tor-ci on irc.oftc.net.
|
|
|
|
|
|
|
|
## Handling CI Failures
|
|
|
|
|
|
|
|
We want to log every CI failure in trac, so we can see which CI issues we need to fix first.
|
|
|
|
|
|
|
|
### Supported Branches
|
|
|
|
|
|
|
|
We want CI to always pass on supported branches.
|
|
|
|
|
|
|
|
When CI fails on:
|
|
|
|
* master, a supported maint branch, or a supported release branch, or
|
|
|
|
* a pull request, but it's not related to the changes in the pull request.
|
|
|
|
|
|
|
|
1. Find the failure log message, the failed branch, and the merge head (if it's a pull request).
|
|
|
|
2. If there is an existing ticket, log that information on the existing ticket.
|
|
|
|
3. Otherwise, log a ticket tagged "tor-ci-fail" with the information, and mark it as high priority.
|
|
|
|
4. If the failure is on an earlier supported branch, update the version and backport tags to that branch.
|
|
|
|
|
|
|
|
### Pull Requests
|
|
|
|
|
|
|
|
When CI fails on a pull request, and it was caused by the changes in the pull request, mark the ticket for that pull request as needs_revision.
|
|
|
|
|
|
|
|
## CI Failure Tickets
|
|
|
|
|
|
|
|
### Unassigned tickets
|
|
|
|
|
|
|
|
These open tickets are tagged ci-fail, and they don't have an owner.
|
|
|
|
|
|
|
|
TODO: who will assign them to the best person to fix the ticket?
|
|
|
|
|
|
|
|
[[TicketQuery(status!=closed,owner=,keywords~=ci-fail,group=owner,format=table,order=changetime,desc=true,col=id|summary|status|owner|reviewer|priority|severity|changetime|sponsor,max=100)]]
|
|
|
|
|
|
|
|
## Owner Action tickets
|
|
|
|
|
|
|
|
These tickets are tagged ci-fail, they have an owner, and they are in a status that needs action from the owner.
|
|
|
|
|
|
|
|
Anyone can change the assignments:
|
|
|
|
* assign themselves one of these tickets,
|
|
|
|
* un-assign ticket from themselves, or
|
|
|
|
* ask someone else to take one of their tickets.
|
|
|
|
|
|
|
|
[[TicketQuery(status!=needs_review|merge_ready|closed,owner!=,keywords~=ci-fail,group=owner,format=table,order=changetime,desc=true,col=id|summary|status|owner|reviewer|priority|severity|changetime|sponsor,max=100)]]
|
|
|
|
|
|
|
|
## Needs Review tickets
|
|
|
|
|
|
|
|
These tickets are tagged ci-fail, and they are in needs_review.
|
|
|
|
|
|
|
|
[[TicketQuery(status=needs_review,keywords~=ci-fail,group=reviewer,format=table,order=changetime,desc=true,col=id|summary|status|owner|reviewer|priority|severity|changetime|sponsor,max=100)]]
|
|
|
|
|
|
|
|
## Merge Ready tickets
|
|
|
|
|
|
|
|
These tickets are tagged ci-fail, and they are in merge_ready.
|
|
|
|
|
|
|
|
[[TicketQuery(status=merge_ready,keywords~=ci-fail,group=milestone|reviewer,format=table,order=changetime,desc=true,col=id|summary|status|owner|reviewer|priority|severity|changetime|sponsor,max=100)]]
|
|
|
|
|
|
|
|
## CI failures being diagnosed
|
|
|
|
|
|
|
|
As of 23 Sep 2019:
|
|
|
|
|
|
|
|
* ~~tor-ci-linux-0.2.9 (old-ish)~~ transient; cleared up by itself in the following build
|
|
|
|
|
|
|
|
~~see https://jenkins.torproject.org/job/tor-ci-linux-0.2.9/292/console
|
|
|
|
This looks like a travis failure. There's a java exception caused by a timeout, and no builders actually happen.~~
|
|
|
|
|
|
|
|
* ~~tor-ci-linux-0.4.1 (new-ish)~~ transient; cleared up by itself in the following build
|
|
|
|
|
|
|
|
~~see https://jenkins.torproject.org/job/tor-ci-linux-0.4.1/ARCHITECTURE=amd64,SUITE=buster/28/console
|
|
|
|
Buster amd64 is failing with what appears to be~~
|
|
|
|
```
|
|
|
|
08:50:53 08:50:53 FAIL: src/test/test_workqueue_efd.sh
|
|
|
|
08:50:53 ====================================
|
|
|
|
08:50:53 08:50:53 Illegal instruction
|
|
|
|
08:50:53 FAIL src/test/test_workqueue_efd.sh (exit status: 132)
|
|
|
|
```
|
|
|
|
~~I wonder if this is transient, or if it is a real problem? We didn't change this code any time recently.~~
|
|
|
|
|
|
|
|
* ~~tor-ci-linux-master-clang (new-ish)~~ transient? seems to have fixed itself
|
|
|
|
|
|
|
|
~~https://jenkins.torproject.org/job/tor-ci-linux-master-clang/3640/ARCHITECTURE=i386,SUITE=sid/consoleFull ~~
|
|
|
|
|
|
|
|
~~This is only happening on i686 sid, and it is a compiler segfault. This makes me think it is a bug in the clang package in debian unstable.~~
|
|
|
|
|
|
|
|
~~Looks like there's now a complaint about an unrecognized warning flag? -catalyst~~ see #31769
|
|
|
|
|
|
|
|
```
|
|
|
|
15:39:37 cc1: error: command line option '-Wextra-semi' is valid for C++/ObjC++ but not for C [-Werror]
|
|
|
|
15:39:37 cc1: all warnings being treated as errors
|
|
|
|
15:39:37 make[1]: *** [Makefile:11538: src/feature/control/control_bootstrap.o] Error 1
|
|
|
|
```
|
|
|
|
|
|
|
|
* tor-ci-mingwcross-* (old-ish)
|
|
|
|
|
|
|
|
https://jenkins.torproject.org/job/tor-ci-mingwcross-master/2499/ARCHITECTURE=amd64,SUITE=buster/console
|
|
|
|
|
|
|
|
This looks like a misconfiguration to me -- it seems to be complaining about not having found a zlib package maybe?
|
|
|
|
|
|
|
|
|
|
|
|
* tor-ci-windows-master (old?)
|
|
|
|
|
|
|
|
https://jenkins.torproject.org/job/tor-ci-windows-master/78/console
|
|
|
|
|
|
|
|
This looks like it could be a real bug: a unit test is failing. It's in "slow/process/callbacks" -- I believe ahf knows that code best.
|
|
|
|
|
|
|
|
* https://jenkins.torproject.org/view/tor/job/tor-ci-linux-master-expensive-hardening/2527/ARCHITECTURE=amd64,SUITE=bionic/console
|
|
|
|
```
|
|
|
|
07:58:07 addr/virtaddr_persist: [forking]
|
|
|
|
07:58:07 FAIL ../tor/src/test/test_addr.c:1415: assert(a OP_EQ "192.168.3.4"): <192.168.7.240> vs <192.168.3.4>
|
|
|
|
07:58:07 [virtaddr_persist FAILED]
|
|
|
|
```
|
|
|
|
could be a network virtualization artifact? |
|
|
|
\ No newline at end of file |