As with the previous month, I figured I would show a sign of life here
and try to keep you up to date with what's happening in sysadmin-land,
even though we're not having regular meetings. I'm still experimenting
with structure here, and this is totally un-edited, so please bear
with me.

# Important announcements

You might have missed this:

 * Jenkins will be retired in December 2021, and it's time to move
   your jobs away
 * if you want old Trac wiki redirects to go to the right place, do
   let us know, see [ticket 40233][]
 * we do not have ARM 32 builders anymore, the last one was shut down
   recently ([ticket 32920][]) and they had been removed from CI
   (Jenkins) anyways before that. the core team is looking at
   alternatives for building Tor on armhf in the future, see [ticket
   40347][]
 * we have setup a Prometheus Alertmanager during the hack week, which
   means we can do alerting based on Prometheus metrics, see the
   [altering documentation][] for more information

As usual, if you have any questions, comments, or issues, please do
contact us following this "how to get help" procedure:

<https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-2-support#how-to-get-help>

Yes, that's a terrible URL. Blame GitLab. :)

[altering documentation]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/prometheus#alerting
[ticket 40347]: https://gitlab.torproject.org/tpo/core/tor/-/issues/40347
[ticket 32920]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/32920
[ticket 40233]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40233

# Crash of the month

Your sysadmin crashed a Ganeti node, creating a split-brain scenario
([ticket 40229][]). He would love to say that was planned and a
routine exercise to test the documentation but (a) it wasn't and (b)
the document had to be made up as he went, so that was actually a
stressful experience.

Remember kids: never start a migration before the weekend or going to
bed unless you're willing and ready to stay up all night (or
weekend).

[ticket 40229]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40229

# Metrics of the month

 * hosts in Puppet: 86, LDAP: 89, Prometheus exporters: 140
 * number of Apache servers monitored: 28, hits per second: 147
 * number of Nginx servers: 2, hits per second: 2, hit ratio: 0.86
 * number of self-hosted nameservers: 6, mail servers: 7
 * pending upgrades: 1, reboots: 0
 * average load: 0.68, memory available: 2.00 TiB/2.77 TiB, running processes: 552
 * bytes sent: 276.43 MB/s, received: 162.75 MB/s
 * [GitLab tickets][]: ? tickets including...
   * open: 0
   * icebox: 109
   * backlog: 15
   * next: 2
   * doing: 2
   * (closed: 2266)
    
 [Gitlab tickets]: https://gitlab.torproject.org/tpo/tpa/team/-/boards

# Ticket analysis

Here's an update of the ticket table, which we last saw in February:

| date       | open | icebox | backlog | next | doing | closed | delta | sum    | new  | spill |
|------------|------|--------|---------|------|-------|--------|-------|--------|------|-------|
| 2020-11-18 | 1    | 84     | 32      | 5    | 4     | 2119   | NA    | 2245   | NA   | NA    |
| 2020-12-02 | 0    | 92     | 20      | 9    | 8     | 2130   | 11    | 2259   | 14   | -3    |
| 2021-01-19 | 0    | 91     | 20      | 12   | 10    | 2165   | 35    | 2298   | 39   | -4    |
| 2021-02-02 | 0    | 96     | 18      | 10   | 7     | 2182   | 17    | 2313   | 15   | 2     |
| 2021-03-02 | 0    | 107    | 15      | 9    | 7     | 2213   | 31    | 2351   | 38   | -7    |
| 2021-04-07 | 0    | 106    | 22      | 7    | 4     | 2225   | 12    | 2364   | 13   | -1    |
| 2021-05-03 | 0    | 109    | 15      | 2    | 2     | 2266   | 41    | 2394   | 30   | 11    |
|------------|------|--------|---------|------|-------|--------|-------|--------|------|-------|
| total      | NA   | NA     | NA      | NA   | NA    | NA     | 147   | NA     | 149  | -2    |
|------------|------|--------|---------|------|-------|--------|-------|--------|------|-------|
| mean       | 0.1  | 97.9   | 20.3    | 7.7  | 6.0   | 2185.7 | 21.0  | 2317.7 | 21.3 | -0.3  |
<!-- #+TBLFM: $9=vsum($2..$7)::$10=@0$-1-@-1$-1::$11=$8-$10::@2$10=NA::@2$11=NA::@9$9=NA::@9$10=vsum(@I..@II);EN::@9$8=vsum(@I..@II);EN::@10$2..$11=vmean(@I..@II);%.1fEN -->
<!-- the above is an org-mode table and can be reculated by -->
<!-- uncommenting the above formula and hitting "C-c C-c" -->

I added a "delta" column which shows how many additional tickets were
closed since the previous period. April is our record so far, with a
record of 41 tickets closed in less than 30 days, more than one ticket
per day!

I also added a "new" column that shows how many new tickets, in total,
were created in the period. And the "spill" is the difference between
the two. If positive, we're winning the ticket game, if negative,
we're losing ground and more tickets are being created than we are
closing. Overall, we're slightly behind (-2), but that's only because
of the epic month of April.

And while I'm here, I went crazy with Emacs' `orgtbl-mode` and added
totals and averages.

In other news, the Icebox keeps growing, which should keep us cool and
breezy during the northern hemisphere summer that's coming up. ;) At
least the Backlog is not growing too wildly, and the actual current
queue (Next/Doing) is pretty reasonable. So things seem to be under
control, but the new hiring process is taking significant time so this
might upset our roadmap a little.

Regardless of those numbers: don't hesitate to make new tickets!

# Ticket of the month

Ticket [40218][] tracks the progress of the CI migration from Jenkins
to GitLab CI. Jenkins is scheduled for retirement in December 2021,
and progress has been excellent, with the network team actually
*asking* for the Jenkins jobs to be disabled ([ticket 40225](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40225))
which, if it gets completed, will means the retirement of 4 virtual
machines already.

Exciting cleanup!

[40218]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40218
