minutes

howto/upgrades/bullseye/data.csv

@@ -8,3 +8,5 @@ Date,release,count
 2021-11-11,buster,84
 2021-11-11,bullseye,4
 2021-11-11,stretch,1
+2022-01-11,buster,77
+2022-01-11,bullseye,12

meeting/2022-01-11.md

+# Roll call: who's there and emergencies
+
+ * anarcat
+ * kez
+ * lavamind
+
+No emergencies.
+
+# Holidays debrief
+
+Holidays went fine, some minor issues, but nothing that needed to be
+urgently dealt with (e.g. [40569][], [40567][], [commit][], runner
+bug). Rotation worked well.
+
+[40569]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40569
+[40567]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40567
+[commit]: https://gitweb.torproject.org/admin/tsa-misc.git/commit/?id=33fb8ded635aa620e57edc2625a7325a9eee09ca
+
+anarcat went cowboy and setup two new nodes before the holidays, which
+is not great because it's against our general "don't launch on a
+friday". (It wasn't on a friday, but it was close enough to the
+holidays to be a significant risk.) Thankfully things worked out fine:
+one of the runners ended up failing just as lavamind was starting work
+again last week. (!)
+
+# 2021 roadmap review
+
+## sysadmin
+
+We did a review directly in the wiki page. Notable changes:
+
+ * jenkins is marked as completed, as rouyi will be retired this week
+   (!)
+ * the blog migration was completed!
+ * we consider we managed to deal with the day-to-day while still
+   reserving time for the unexpected (e.g. the rushed web migration
+   from Jenkins to GitLab CI)
+ * we loved that team work and should plan to do it again
+ * we were mostly on budget: we had an extra 100EUR/mth at hetzner for
+   a new Ganeti node in the gnt-fsn cluster, and extra costs
+   (54EUR/mth!) for the [Hetzner IPv4 billing changes][], and more for
+   extra bandwidth use
+ 
+ [Hetzner Ipv4 billing changes]: https://docs.hetzner.com/general/others/ipv4-pricing/
+
+## web
+
+Did a review of the 2021 web roadmap (from the [wiki homepage][]), copied below:
+
+ [wiki homepage]: https://gitlab.torproject.org/tpo/web/team/-/wikis/home
+
+* [ ] [Donations page redesign][] - 10-50%
+* [ ] [Improves bridges.torproject.org][] - 80% done!
+* [ ] [Remove outdated documentation from the header][] - the
+      "docs.tpo ticket", considering using dev.tpo instead, focus on
+      launching dev.tpo next instead
+* [x] Migrate blog.torproject.org from Drupal To Lektor: it needs a milestone and planning
+* [x] [Support forum][]
+* [ ] [Developer portal][] AKA dev.tpo
+* [x] Get website build from Jenkins into to GitLabCI for the static mirror pool (before December)
+* [ ] Get up to speed on maintenance tasks: 
+    * [ ] Bootstrap upgrade - uh god.
+    * [ ] browser documentation update - what is this?
+    * [ ] get translation stats available - what is this?
+    * [x] rename 'master' branch as 'main'
+    * [ ] fix wiki for documentation - what is this?
+    * [ ] get [onion service tooling][] into TPO GitLab namespace - what is this?
+
+[Donations page redesign]: https://gitlab.torproject.org/groups/tpo/-/milestones/22
+[Improves bridges.torproject.org]: https://gitlab.torproject.org/groups/tpo/-/milestones/7
+[Remove outdated documentation from the header]: https://gitlab.torproject.org/tpo/web/team/-/issues/8
+[Support forum]: https://gitlab.torproject.org/groups/tpo/-/milestones/26
+[Developer portal]: https://gitlab.torproject.org/groups/tpo/-/milestones/23
+[onion service tooling]: https://gitlab.torproject.org/hiro/roid
+
+# Syadmin+web OKRs for 2022 Q1
+
+We want to take more time to plan for the web team, in particular, and especially focused on this in the meeting.
+
+## web team
+
+We did the following brainstorm. Anarcat will come up with a proposal for a better-formatted OKR set for next week, at which point we'll prioritize this and the sysadmin OKRs for Q1.
+
+ * OKR: rewrite of the donate page ([milestone 22][])
+   * [new lektor frontend][]
+   * [we can donate through the .onion][]
+   * [vanilla JS rewrite][]
+ * OKR: make it easier for translators to contribute
+   * help the translation team to switch to Weblate
+   * it is easier for translators to find their built copy of the website
+   * bring build time to 15 minutes to accelerate feedback to translators
+   * allow the web team to trigger manual builds for reviews
+ * OKR: documentation overhaul:
+   * [launch dev.tpo][]
+   * "Remove outdated documentation from the header", stop pointing to dead docs
+   * come with ideas on how to manage the wiki situation
+   * cleanup the queues and workflow
+ * OKR: resurrect bridge port scan
+   * do not scan private IP blocks
+   * make it pretty
+
+[milestone 22]: https://gitlab.torproject.org/groups/tpo/-/milestones/22
+[new lektor frontend]: https://gitlab.torproject.org/tpo/web/donate-static/-/issues/37
+[we can donate through the .onion]: https://gitlab.torproject.org/tpo/web/donate-static/-/issues/36
+[vanilla JS rewrite]: https://gitlab.torproject.org/tpo/web/donate-static/-/issues/45
+[launch dev.tpo]: https://gitlab.torproject.org/tpo/web/dev/-/issues/6
+
+Missed from the last meeting:
+
+ * sponsor 9 stuff: collected UX feedback for portals, which involves web to fix issues we found, need to prioritise
+
+We also need to organise with the new people:
+
+ * onion SRE: new OTF project USAGM, starting in February
+ * new community person
+
+# Other discussions
+
+# Next meeting
+
+We're going to hold another meeting next week, same time, to review the web OKRs and prioritize Q1.
+
+# Metrics of the month
+
+ * hosts in Puppet: 89, LDAP: 91, Prometheus exporters: 139
+ * number of Apache servers monitored: 27, hits per second: 185
+ * number of Nginx servers: 0, hits per second: 0, hit ratio: 0.00
+ * number of self-hosted nameservers: 6, mail servers: 8
+ * pending upgrades: 7, reboots: 0
+ * average load: 0.35, memory available: 4.01 TiB/5.13 TiB, running processes: 643
+ * disk free/total: 84.95 TiB/39.99 TiB
+ * bytes sent: 325.45 MB/s, received: 190.66 MB/s
+ * planned bullseye upgrades completion date: 2024-09-07
+ * [GitLab tickets][]: 159 tickets including...
+   * open: 2
+   * icebox: 143
+   * backlog: 8
+   * next: 2
+   * doing: 2
+   * needs information: 2
+   * (closed: 2573)
+
+ [Gitlab tickets]: https://gitlab.torproject.org/tpo/tpa/team/-/boards
+
+Upgrade prediction graph now lives at:
+
+https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bullseye/
+
+... with someone accurate values, although the 2024 estimate above
+should be taken with a grain of salt, as we haven't really started the
+upgrade at all.
+
+We should also note that we turned off our two Nginx servers (ignoring
+the Nginx server in the GitLab instance, which is not really monitored
+correctly), when we migrated the blog to a static site. Those two
+servers were the caching server sitting in front of the Drupal blog
+for cost savings. They served us well but are now retired since they
+are not necessary for the static version.

meeting/monthly-report

@@ -251,9 +251,9 @@ def main():
            prom.query(args, 'sum(rate(apache_accesses_total[30d]))')))
 
     print(" * number of Nginx servers: %d, hits per second: %d, hit ratio: %0.2f" %  # noqa: E501
-          (prom.query(args, 'count(count(nginx_http_request_details_total) by (alias))'),  # noqa: E501
+          (prom.query(args, 'count(count(nginx_http_request_details_total) by (alias))') or 0,  # noqa: E501
            prom.query(args, 'sum(rate(nginx_http_request_details_total[30d]))'),  # noqa: E501
-           prom.query(args, 'sum(nginx_http_request_details_total{upstream_cache_status="HIT"})/sum(nginx_http_request_details_total)'),  # noqa: E501
+           prom.query(args, 'sum(nginx_http_request_details_total{upstream_cache_status="HIT"})/sum(nginx_http_request_details_total)') or 0.0,  # noqa: E501
           ))
 
     print(" * number of self-hosted nameservers: %d, mail servers: %d" %