meeting minutes

howto/upgrades/bookworm/data.csv

@@ -19,3 +19,6 @@ Date,release,count
 2023-09-26,bookworm,41
 2023-09-26,bullseye,42
 2023-09-26,buster,5
+2023-10-02,buster,5
+2023-10-02,bookworm,42
+2023-10-02,bullseye,42

meeting.md

@@ -28,6 +28,7 @@ Those are just for TPA, there are broader notes on meetings in the
 
 # 2023
 
+ * [2023-10-02](meeting/2023-10-02)
  * [2023-06-05](meeting/2023-06-05)
  * [2023-05-08](meeting/2023-05-08)
  * [2023-03-13](meeting/2023-03-13)

meeting/2023-10-02.md

+# Roll call: who's there and emergencies
+
+onionoo-backend running out of disk space ([tpo/tpa/team#41343][])
+
+[tpo/tpa/team#41343]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41343
+
+# Dashboard cleanup
+
+Normal per-user check-in:
+
+ * https://gitlab.torproject.org/groups/tpo/-/boards?scope=all&utf8=%E2%9C%93&ssignee_username=anarcat
+ * https://gitlab.torproject.org/groups/tpo/-/boards?scope=all&utf8=%E2%9C%93&assignee_username=kez
+ * https://gitlab.torproject.org/groups/tpo/-/boards?scope=all&utf8=%E2%9C%93&assignee_username=lavamind
+
+General dashboards:
+
+ * https://gitlab.torproject.org/tpo/tpa/team/-/boards/117
+ * https://gitlab.torproject.org/groups/tpo/web/-/boards
+ * https://gitlab.torproject.org/groups/tpo/tpa/-/boards
+
+Nextcloud roadmap / spreadsheet.
+
+Overall, it seems we are as you would expect when returning from a
+rather chaotic vacation. Backlog is large, but things seem to be under
+control.
+
+We added SVN back on the roadmap after one too many tickets asking for
+setup.
+
+# Metrics of the month
+
+ * hosts in Puppet: 89, LDAP: 89, Prometheus exporters: 166
+ * number of Apache servers monitored: 37, hits per second: 626
+ * number of self-hosted nameservers: 6, mail servers: 10
+ * pending upgrades: 1, reboots: 0
+ * average load: 0.69, memory available: 3.58 TiB/4.98 TiB, running processes: 424
+ * disk free/total: 53.19 TiB/126.72 TiB
+ * bytes sent: 403.47 MB/s, received: 269.04 MB/s
+ * planned bullseye upgrades completion date: 2024-08-02
+ * [GitLab tickets][]: 196 tickets including...
+   * open: 0
+   * icebox: 163
+   * needs information: 5
+   * backlog: 13
+   * next: 9
+   * doing: 4
+   * needs review: 2
+   * (closed: 3301)
+
+ [Gitlab tickets]: https://gitlab.torproject.org/tpo/tpa/team/-/boards
+
+Upgrade prediction graph lives at:
+
+https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bookworm/
+
+Now also available as the main Grafana dashboard. Head to
+<https://grafana.torproject.org/>, change the time period to 30 days,
+and wait a while for results to render.
+
+# Number of the month: 42
+
+34 machines were upgraded from bullseye to bookworm in the two first
+days of last week! We calculated this was an average of 20 minutes per
+host to upgrade.
+
+The trick, of course, is that things often break *after* the upgrade,
+and that "fixing" time is not counted here. That said, last estimate
+for this was one hour per machine, and we're doing a whole fleet
+upgrade every 2-3 years, which means about ten hours of work saved per
+year.
+
+But the number of the month is, of course, 42, as we now have an equal
+number of bookworm and bullseye machine, after the upgrade. And that
+number is, naturally, [42][].
+
+See also https://xkcd.com/1205/ which, interestingly, we fall out of
+scope of.
+
+[42]: https://en.wikipedia.org/wiki/42

meeting/monthly-report

@@ -140,7 +140,7 @@ def host_count_puppet(args):
 
 def host_count_ldap(args):
     ldap_data = subprocess.check_output(['ssh', args.ldap,
-                                         'ldapsearch -ZZ -vLx -h db.torproject.org -b "ou=hosts,dc=torproject,dc=org" 2>/dev/null'])  # noqa: E501
+                                         'ldapsearch -ZZ -vLx -H ldap://db.torproject.org -b "ou=hosts,dc=torproject,dc=org" 2>/dev/null'])  # noqa: E501
     return len(re.findall(r'^dn: host', ldap_data.decode('ascii'), re.M))
 
 
@@ -272,7 +272,7 @@ def main():
           (sizeof_fmt_decimal(prom.query(args, 'sum(rate(node_network_transmit_bytes_total[30d]))')),  # noqa: E501
            sizeof_fmt_decimal(prom.query(args, 'sum(rate(node_network_receive_bytes_total[30d]))'))))  # noqa: E501
 
-    print(" * planned bullseye upgrades completion date: ???")
+    print(" * planned bookworm upgrades completion date: ???")
     print(""" * [GitLab tickets][]: ? tickets including...
    * open: ?
    * icebox: ?
@@ -284,7 +284,7 @@ def main():
     print(" [Gitlab tickets]: https://gitlab.torproject.org/tpo/tpa/team/-/boards")  # noqa: E501
 
     print()
-    print("Upgrade prediction graph lives at https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bullseye/")  # noqa: E501
+    print("Upgrade prediction graph lives at https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bookworm/")  # noqa: E501
     print()
     print("Now also available as the main Grafana dashboard. Head to <https://grafana.torproject.org/>, change the time period to 30 days, and wait a while for results to render.")  # noqa: E501
     # TODO: talk with the gitlab API to extract those numbers