TPA issueshttps://gitlab.torproject.org/groups/tpo/tpa/-/issues2023-12-05T16:51:36Zhttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41400obsolete package on weather-01: python3-flask-assets2023-12-05T16:51:36Zanarcatobsolete package on weather-01: python3-flask-assetsSince we upgraded weather-01 to Debian bookworm, the `python3-flask-assets` package is marked as "obsolete". It was indeed removed before the stable release, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1000555
@sarthikg @gk : ...Since we upgraded weather-01 to Debian bookworm, the `python3-flask-assets` package is marked as "obsolete". It was indeed removed before the stable release, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1000555
@sarthikg @gk : is that package still required by weather-01? if so, could you vendor the dependency and manage it yourself so we can purge the unmaintained package?
thanks!Debian 12 bookworm upgradeanarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41388Launching Tor Weather breaks with password authentication error2024-01-19T19:44:52ZGeorg KoppenLaunching Tor Weather breaks with password authentication errorI think I fixed a bunch of things related to the recent Debian upgrade, so the website is up and accessible again. Great. However, our Onionoo job is still crashing and I am inclined to think that I don't have the powers to fix that:
```...I think I fixed a bunch of things related to the recent Debian upgrade, so the website is up and accessible again. Great. However, our Onionoo job is still crashing and I am inclined to think that I don't have the powers to fix that:
```
11/08/2023 02:00:15 PM : JOB : INFO : Onionoo Job Crashed - (psycopg2.OperationalError) connection to server at "localhost" (::1), port 5432 failed: FATAL: password authentication failed for user "torweather"
connection to server at "localhost" (::1), port 5432 failed: FATAL: password authentication failed for user "torweather"
(Background on this error at: https://sqlalche.me/e/20/e3q8)
```
I was looking over the [service description](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/tor-weather) but it's not obvious to me right now how to debug this further...
/cc @sarthikg
# Emergency checklist
- [x] status site update (@gk, https://gitlab.torproject.org/tpo/tpa/status-site/-/merge_requests/39)
- [x] check if we can recover from backups (@anarcat yes, barely)
- [x] recover PostgreSQL 15 cluster from backups (@anarcat)
- [x] confirm service works properly (tested by @anarcat, @trinity-1686a, would like confirmation from @sarthikg and @gk)
# Cleanup work
- [x] post-mortem (@anarcat)
- [x] timeline analysis (@anarcat )
- [x] document data recovery procedures (@anarcat)
- [x] analyze upgrade logs to figure out what happened and how to prevent it (@anarcat)
- [x] recommend (monitoring?) improvements to prevent this from happening again (@anarcat)
- [x] mark incident as resolved in status site (@anarcat, see https://gitlab.torproject.org/tpo/tpa/status-site/-/merge_requests/40)
- [x] delete `/srv/backup/bacula/recup_dir.*` and `/root/RECOVER*` on bungei (@anarcat)
- [x] delete backups in `/srv/f*` and `/srv/dump` on weather-01 (@anarcat)
- [x] remove old psql 13 cluster on weather-01 (@anarcat)
- [x] delete `/root/weather-01-rootfs-20231108T154128.dump` on fsn-node-01 (@anarcat)
# Future improvements
- [x] improve upgrade procedure to avoid dropping non-empty clusters (@anarcat, done in wiki-replica@199627ca but really not pretty)
- [x] ~~ensure proper contact information for servers~~ (we already have contact information in the service) and that the upgrade procedure MUST notify server admins to test their services (@anarcat)
- [x] puppetize all PostgreSQL servers (@lavamind, moved to issue #41401 )
- [x] properly monitor tor-weather service (@gk, moved to issue https://gitlab.torproject.org/tpo/network-health/metrics/monitoring-and-alerting/-/issues/24)
- [x] extend backup retention period on upgrade, ~~or just keep in the same backup rotation~~ (@anarcat) (for now we're going to assume there's a good reason to move the old version aside and keep this setup, we matched the 21 days rotation now in wiki-replica@267a9818)
- [x] remove staging component from status site (@lavamind, https://gitlab.torproject.org/tpo/tpa/status-site/-/issues/32)Debian 12 bookworm upgradeanarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41376Update howto/postgresql recovery procedure2023-11-02T18:49:34ZJérôme Charaouilavamind@torproject.orgUpdate howto/postgresql recovery procedureThe current recovery steps in the wiki don't work for PostgreSQL version 12 and later.
This is because `recovery.conf` in the database directory is no longer supported. The `restore_command` config must be placed in `postgresql.conf`.
...The current recovery steps in the wiki don't work for PostgreSQL version 12 and later.
This is because `recovery.conf` in the database directory is no longer supported. The `restore_command` config must be placed in `postgresql.conf`.
```
Nov 01 22:25:43 rude postgresql@15-main[30560]: 2023-11-01 22:25:43 UTC [30565-1] LOG: starting PostgreSQL 15.3 (Debian 15.3-0+deb12u1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
Nov 01 22:25:43 rude postgresql@15-main[30560]: 2023-11-01 22:25:43 UTC [30565-2] LOG: listening on IPv4 address "0.0.0.0", port 5432 Nov 01 22:25:43 rude postgresql@15-main[30560]: 2023-11-01 22:25:43 UTC [30565-3] LOG: listening on IPv6 address "::", port 5432
Nov 01 22:25:43 rude postgresql@15-main[30560]: 2023-11-01 22:25:43 UTC [30565-4] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
Nov 01 22:25:43 rude postgresql@15-main[30560]: 2023-11-01 22:25:43 UTC [30568-1] LOG: database system was interrupted; last known up at 2023-11-01 18:52:52 UTC
Nov 01 22:25:43 rude postgresql@15-main[30560]: 2023-11-01 22:25:43 UTC [30568-2] FATAL: using recovery command file "recovery.conf" is not supported
Nov 01 22:25:43 rude postgresql@15-main[30560]: 2023-11-01 22:25:43 UTC [30565-5] LOG: startup process (PID 30568) exited with exit code 1
Nov 01 22:25:43 rude postgresql@15-main[30560]: 2023-11-01 22:25:43 UTC [30565-6] LOG: aborting startup due to startup process failure
Nov 01 22:25:43 rude postgresql@15-main[30560]: 2023-11-01 22:25:43 UTC [30565-7] LOG: database system is shut down
```
From the [12.0 release notes](https://www.postgresql.org/docs/release/12.0/):
> recovery.conf is no longer used, and the server will not start if that file exists. recovery.signal and standby.signal files are now used to switch into non-primary mode.
In addition, after adding the restore directive, a `restore.signal` file must be create din the database directory to trigger the recovery process.
```
Nov 01 22:33:31 rude postgresql@15-main[31198]: 2023-11-01 22:33:31 UTC [31203-1] LOG: starting PostgreSQL 15.3 (Debian 15.3-0+deb12u1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
Nov 01 22:33:31 rude postgresql@15-main[31198]: 2023-11-01 22:33:31 UTC [31203-2] LOG: listening on IPv4 address "0.0.0.0", port 5432
Nov 01 22:33:31 rude postgresql@15-main[31198]: 2023-11-01 22:33:31 UTC [31203-3] LOG: listening on IPv6 address "::", port 5432
Nov 01 22:33:31 rude postgresql@15-main[31198]: 2023-11-01 22:33:31 UTC [31203-4] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
Nov 01 22:33:31 rude postgresql@15-main[31198]: 2023-11-01 22:33:31 UTC [31206-1] LOG: database system was interrupted; last known up at 2023-11-01 18:52:52 UTC
Nov 01 22:33:31 rude postgresql@15-main[31198]: 2023-11-01 22:33:31 UTC [31206-2] LOG: invalid checkpoint record
Nov 01 22:33:31 rude postgresql@15-main[31198]: 2023-11-01 22:33:31 UTC [31206-3] FATAL: could not locate required checkpoint record
Nov 01 22:33:31 rude postgresql@15-main[31198]: 2023-11-01 22:33:31 UTC [31206-4] HINT: If you are restoring from a backup, touch "/var/lib/postgresql/15/main/recovery.signal" and add required recovery options.
Nov 01 22:33:31 rude postgresql@15-main[31198]: If you are not restoring from a backup, try removing the file "/var/lib/postgresql/15/main/backup_label".
Nov 01 22:33:31 rude postgresql@15-main[31198]: Be careful: removing "/var/lib/postgresql/15/main/backup_label" will result in a corrupt cluster if restoring from a backup.
Nov 01 22:33:31 rude postgresql@15-main[31198]: 2023-11-01 22:33:31 UTC [31203-5] LOG: startup process (PID 31206) exited with exit code 1
Nov 01 22:33:31 rude postgresql@15-main[31198]: 2023-11-01 22:33:31 UTC [31203-6] LOG: aborting startup due to startup process failure
Nov 01 22:33:31 rude postgresql@15-main[31198]: 2023-11-01 22:33:31 UTC [31203-7] LOG: database system is shut down
```Debian 12 bookworm upgradeJérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41350hash-slinger generates incorrect TLSA certificates on pauli/nevii2023-12-12T16:05:44Zanarcathash-slinger generates incorrect TLSA certificates on pauli/neviiIn Debian bullseye, running the following command here generates an invalid DNS record:
```
pauli# ./tlsa --create --usage=3 --selector=1 --mtype=1 --certificate /srv/puppet.torproject.org/from-letsencrypt/cdn-fastly-backend.torproject....In Debian bullseye, running the following command here generates an invalid DNS record:
```
pauli# ./tlsa --create --usage=3 --selector=1 --mtype=1 --certificate /srv/puppet.torproject.org/from-letsencrypt/cdn-fastly-backend.torproject.org.crt --port 443 cdn-fastly-backend.torproject.org --output=generic
Got a certificate for cdn-fastly-backend.torproject.org. with Subject: /CN=cdn-fastly-backend.torproject.org
_443._tcp.cdn-fastly-backend.torproject.org. IN TYPE52 \# 35.0 030101e86cb4aa5bec41b44c5e78c0b3b05992ab276d540376aca18eb494d8e229cd4c
```
Notice the float (35.0) there? That, of course, crashes bind with:
```
Notice: /Stage[main]/Dnsextras::Entries/Exec[rebuild torproject.org zone]/returns: dns_rdata_fromtext: /srv/dns.torproject.org/puppet-extra/include-torproject.org:945: near '35.0': not a valid number
```
I suspect this wasn't caught by other users because it happens when the len() of the cert string is an odd number, which, oddly, I guess it is here.
----
This affects pauli, which has a patch to `tlsa` to workaround the issue, and has the package pinned. This needs to be fixed before we upgrade to bookworm, sooner rather than later, so that we can get the fix in bookworm itself.
This has been reported upstream at https://github.com/letoams/hash-slinger/issues/45 and in Debian as https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1053483.Debian 12 bookworm upgradeanarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41341Deploy new PuppetDB server2023-10-04T21:47:02ZJérôme Charaouilavamind@torproject.orgDeploy new PuppetDB serverAs part of tpo/tpa/team#40696 we decided to deploy a new PuppetDB server alongside the old puppet master.
The hope is we can make progress on the Puppet upgrade plan by replacing the current PuppetDB running on `pauli` with this new one.As part of tpo/tpa/team#40696 we decided to deploy a new PuppetDB server alongside the old puppet master.
The hope is we can make progress on the Puppet upgrade plan by replacing the current PuppetDB running on `pauli` with this new one.Debian 12 bookworm upgradeJérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41339onionprobe broke during the bookworm upgrade2023-12-22T14:02:32Zanarcatonionprobe broke during the bookworm upgradeDuring the first batch of bookworm upgrades (#41251), we found that onionprobe stopped working on the main prometheus server (`prometheus1.torproject.org`, AKA `hetzner-nbg1-01`):
```
root@hetzner-nbg1-01:~# systemctl status onionprobe
...During the first batch of bookworm upgrades (#41251), we found that onionprobe stopped working on the main prometheus server (`prometheus1.torproject.org`, AKA `hetzner-nbg1-01`):
```
root@hetzner-nbg1-01:~# systemctl status onionprobe
× onionprobe.service - Onionprobe
Loaded: loaded (/lib/systemd/system/onionprobe.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Wed 2023-09-27 11:26:31 UTC; 2h 36min ago
Duration: 247ms
Docs: file:///usr/share/doc/onionprobe/README.Debian
man:onionprobe(1)
https://gitlab.torproject.org/tpo/onion-services/onionprobe
Process: 78817 ExecStart=/usr/bin/onionprobe -c $CONFIG $DAEMON_OPTS (code=exited, status=1/FAILURE)
Main PID: 78817 (code=exited, status=1/FAILURE)
CPU: 245ms
Sep 27 11:26:30 hetzner-nbg1-01 systemd[1]: Started onionprobe.service - Onionprobe.
Sep 27 11:26:31 hetzner-nbg1-01 onionprobe[78817]: 2023-09-27 11:26:31,048 INFO: Starting Onionprobe version 1.0.0...
Sep 27 11:26:31 hetzner-nbg1-01 onionprobe[78817]: 2023-09-27 11:26:31,049 INFO: Initializing Tor process...
Sep 27 11:26:31 hetzner-nbg1-01 onionprobe[78817]: 2023-09-27 11:26:31,073 ERROR: Process terminated: Unknown option '16:578F4DEC1A07C7FF600CB53BD614DB4E175369A9A5FB4813DBD30C1527'. Failing.
Sep 27 11:26:31 hetzner-nbg1-01 onionprobe[78817]: 2023-09-27 11:26:31,073 CRITICAL: Error initializing Tor
Sep 27 11:26:31 hetzner-nbg1-01 systemd[1]: onionprobe.service: Main process exited, code=exited, status=1/FAILURE
Sep 27 11:26:31 hetzner-nbg1-01 systemd[1]: onionprobe.service: Failed with result 'exit-code'.
```
Here's a config dump:
<details><summary>Click to expand</summary>
```
root@hetzner-nbg1-01:~# cat /etc/default/onionprobe
# This file is managed by Puppet, local changes will be lost
#
# Default settings for the Onionprobe system-wide service
#
# This is used only for the system-wide service and has no
# effect when using the onionprobe command line directly.
# The configuration file to use
CONFIG="/etc/onionprobe/tpo.yaml"
# Additional Onionprobe command line options to use
DAEMON_OPTS=""
root@hetzner-nbg1-01:~# cat /etc/onionprobe/tpo.yaml
circuit_stream_timeout: 60
control_port: 19051
descriptor_max_retries: 5
descriptor_timeout: 30
endpoints:
2019.www.torproject.org:
address: jqyzxhjk6psc6ul5jnfwloamhtyh7si74b4743k2qgpskwwxrzhsxmad.onion
paths:
- path: ''
port: 80
protocol: http
api.donate.torproject.org:
address: rbi3fpvpz4vlrx67scoqef2zxz7k4xyiludszg655favvkygjmhz6wyd.onion
paths:
- path: ''
port: 80
protocol: http
archive.torproject.org:
address: uy3qxvwzwoeztnellvvhxh7ju7kfvlsauka7avilcjg7domzxptbq7qd.onion
paths:
- path: ''
port: 80
protocol: http
aus1.torproject.org:
address: ot3ivcdxmalbsbponeeq5222hftpf3pqil24q3s5ejwo5t52l65qusid.onion
paths:
- path: ''
port: 80
protocol: http
aus2.torproject.org:
address: b5t7emfr2rn3ixr4lvizpi3stnni4j4p6goxho7lldf4qg4hz5hvpqid.onion
paths:
- path: ''
port: 80
protocol: http
blog.torproject.org:
address: pzhdfe7jraknpj2qgu5cz2u3i4deuyfwmonvzu5i3nyw4t4bmg7o5pad.onion
paths:
- path: ''
port: 80
protocol: http
bridges.torproject.org:
address: yq5jjvr7drkjrelzhut7kgclfuro65jjlivyzfmxiq2kyv5lickrl4qd.onion
paths:
- path: ''
port: 80
protocol: http
cloud.torproject.org:
address: ui3cpcohcoko6aydhuhlkwqqtvadhaflcc5zb7mwandqmcal7sbwzwqd.onion
paths:
- path: ''
port: 80
protocol: http
collector.torproject.org:
address: pgmrispjerzzf2tdzbfp624cg5vpbvdw2q5a3hvtsbsx25vnni767yad.onion
paths:
- path: ''
port: 80
protocol: http
collector2.torproject.org:
address: urscdffm73o4y6hpp3r43bgmudq42hq2ibdpkld6a7hy3qa44qbvc2yd.onion
paths:
- path: ''
port: 80
protocol: http
community.torproject.org:
address: xmrhfasfg5suueegrnc4gsgyi2tyclcy5oz7f5drnrodmdtob6t2ioyd.onion
paths:
- path: ''
port: 80
protocol: http
consensus-health.torproject.org:
address: tkskz5dkjel4xqyw5d5l3k52kgglotwn6vgb5wrl2oa5yi2szvywiyid.onion
paths:
- path: ''
port: 80
protocol: http
crm.torproject.org:
address: 6ojylpznauimd2fga3m7g24vd7ebkzlemxdprxckevqpzs347ugmynqd.onion
paths:
- path: ''
port: 80
protocol: http
deb.torproject.org:
address: apow7mjfryruh65chtdydfmqfpj5btws7nbocgtaovhvezgccyjazpqd.onion
paths:
- path: ''
port: 80
protocol: http
dev.crm.torproject.org:
address: eewp4iydzyu2a5d6bvqadadkozxdbhsdtmsoczu2joexfrjjsheaecad.onion
paths:
- path: ''
port: 80
protocol: http
dist.torproject.org:
address: scpalcwstkydpa3y7dbpkjs2dtr7zvtvdbyj3dqwkucfrwyixcl5ptqd.onion
paths:
- path: ''
port: 80
protocol: http
donate-api.torproject.org:
address: lkfkuhcx62yfvzuz5o3ju4divuf4bsh2bybwd3oierq47kyp2ig2gvid.onion
paths:
- path: ''
port: 80
protocol: http
donate.torproject.org:
address: yoaenchicimox2qdc47p36zm3cuclq7s7qxx6kvxqaxjodigfifljqqd.onion
paths:
- path: ''
port: 80
protocol: http
exonerator.torproject.org:
address: pm46i5h2lfewyx6l7pnicbxhts2sxzacvsbmqiemqaspredf2gm3dpad.onion
paths:
- path: ''
port: 80
protocol: http
extra.torproject.org:
address: kkr72iohlfix5ipjg776eyhplnl2oiv5tz4h2y2bkhjix3quafvjd5ad.onion
paths:
- path: ''
port: 80
protocol: http
forum.torproject.org:
address: v236xhqtyullodhf26szyjepvkbv6iitrhjgrqj4avaoukebkk6n6syd.onion
paths:
- path: ''
port: 80
protocol: http
gettor.torproject.org:
address: ueghr2hzndecdntou33mhymbbxj7pir74nwzhqr6drhxpbz3j272p4id.onion
paths:
- path: ''
port: 80
protocol: http
git.torproject.org:
address: xtlfhaspqtkeeqxk6umggfbr3gyfznvf4jhrge2fujz53433i2fcs3id.onion
paths:
- path: ''
port: 80
protocol: http
gitlab.torproject.org:
address: eweiibe6tdjsdprb4px6rqrzzcsi22m4koia44kc5pcjr7nec2rlxyad.onion
paths:
- path: ''
port: 80
protocol: http
gitweb.torproject.org:
address: gzgme7ov25seqjbphab4fkcph3jkobfwwpivt5kzbv3kqx2y2qttl4yd.onion
paths:
- path: ''
port: 80
protocol: http
grafana1.torproject.org:
address: 7zjnw5lx2x27rwiocxkqdquo7fawj46mf2wiu2l7e6z6ng6nivmdxnad.onion
paths:
- path: ''
port: 80
protocol: http
grafana2.torproject.org:
address: f3vd6fyiccuppybkxiblgigej3pfvvqzjnhd3wyv7h4ee5asawf2fhqd.onion
paths:
- path: ''
port: 80
protocol: http
ircbouncer.torproject.org:
address: moz5kotsnjony4oxccxfo4lwk3pvoxmdoljibhgoonzgzjs5oemtjmqd.onion
paths:
- path: ''
port: 80
protocol: http
metrics-api.torproject.org:
address: yc3galza3gejn3taziuhhgrwt4bdtwmom25zby7jphfwbeirvkmcdvqd.onion
paths:
- path: ''
port: 80
protocol: http
metrics-db.torproject.org:
address: lk6lj36rfj32u2rjceujj3o7otgujm6fw5hyyxr6jko6pkfasb2z6eid.onion
paths:
- path: ''
port: 80
protocol: http
metrics.torproject.org:
address: hctxrvjzfpvmzh2jllqhgvvkoepxb4kfzdjm6h7egcwlumggtktiftid.onion
paths:
- path: ''
port: 80
protocol: http
moat.torproject.org:
address: z7m7ogzdhu43nosvjtsuplfmuqa3ge5obahixydhmzdox6owwxfoxzid.onion
paths:
- path: ''
port: 80
protocol: http
nagios.torproject.org:
address: w6vizvw4ckesva5fvlkrepynemxdq6pgo5sh4r76ec6msq5notkhqryd.onion
paths:
- path: ''
port: 80
protocol: http
newsletter.torproject.org:
address: a4ygisnerpgtc5ayerl22pll6cls3oyj54qgpm7qrmb66xrxts6y3lyd.onion
paths:
- path: ''
port: 80
protocol: http
nightlies.tbb.torproject.org:
address: umj4zbqdfcyevlkgqgpq6foxk3z75zzxsbgt5jqmfxofrbrjh3crbnad.onion
paths:
- path: ''
port: 80
protocol: http
nyx.torproject.org:
address: 3ewfgrt4gzfccp6bnquhqb266r3zepiqpnsk3falwygkegtluwuyevid.onion
paths:
- path: ''
port: 80
protocol: http
onion.torproject.org:
address: xao2lxsmia2edq2n5zxg6uahx6xox2t7bfjw6b5vdzsxi7ezmqob6qid.onion
paths:
- path: ''
port: 80
protocol: http
onionoo.torproject.org:
address: dud2sxm6feahhuwj4y4lzktduy7v3qpaqsfkggtj2ojmzathttkegoid.onion
paths:
- path: ''
port: 80
protocol: http
openpgpkey.torproject.org:
address: 2yldcptk56shc7lwieozoglw3t5ghty7m6mf2faysvfnzccqavbu2mad.onion
paths:
- path: ''
port: 80
protocol: http
people.torproject.org:
address: 5ecey6oe4rocdsfoigr4idu42cecm2j7zfogc3xc7kfn4uriehwrs6qd.onion
paths:
- path: ''
port: 80
protocol: http
prometheus1.torproject.org:
address: ydok5jiruh3ak6hcfdlm2g7iuraaxcomeckj2nucjsxif6qmrrda2byd.onion
paths:
- path: ''
port: 80
protocol: http
prometheus2.torproject.org:
address: vyo6yrqhl3by7d6n5t6hjkflaqbarjpqjnvapr5u5rafk4imnfrmcjyd.onion
paths:
- path: ''
port: 80
protocol: http
rbm.torproject.org:
address: nkuz2tpok7ctwd5ueer5bytj3bm42vp7lgjcsnznal3stotg6vyaakyd.onion
paths:
- path: ''
port: 80
protocol: http
research.torproject.org:
address: xhqthou6scpfnwjyzc3ekdgcbvj76ccgyjyxp6cgypxjlcuhnxiktnqd.onion
paths:
- path: ''
port: 80
protocol: http
review.torproject.net:
address: zhkhhhnppc5k6xju7n25rjba3wuip73jnodicxl65qdpchrwvvsilcyd.onion
paths:
- path: ''
port: 80
protocol: http
rpm.torproject.org:
address: 4ayyzfoh5qdrokqaejis3rdredhvf22n3migyxfudpkpunngfc7g4lqd.onion
paths:
- path: ''
port: 80
protocol: http
snowflake.torproject.org:
address: oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion
paths:
- path: ''
port: 80
protocol: http
spec.torproject.org:
address: i3xi5qxvbrngh3g6o7czwjfxwjzigook7zxzjmgwg5b7xnjcn5hzciad.onion
paths:
- path: ''
port: 80
protocol: http
staging-api.donate.torproject.org:
address: vorwws6g6mx23djlznmlqva4t5olulpnet6fxyiyytcu5dorp3fstdqd.onion
paths:
- path: ''
port: 80
protocol: http
staging.crm.torproject.org:
address: pt34uujusar4arrvsqljndqlt7tck2d5cosaav5xni4nh7bmvshyp2yd.onion
paths:
- path: ''
port: 80
protocol: http
staging.donate-api.torproject.org:
address: 7niqsyixinnhxvh33zh5dqnplxnc2yd6ktvats3zmtbbpzcphpbsa6qd.onion
paths:
- path: ''
port: 80
protocol: http
status.torproject.org:
address: eixoaclv7qvnmu5rolbdwba65xpdiditdoyp6edsre3fitad777jr3ad.onion
paths:
- path: ''
port: 80
protocol: http
stem.torproject.org:
address: mf34jlghauz5pxjcmdymdqbe5pva4v24logeys446tdrgd5lpsrocmqd.onion
paths:
- path: ''
port: 80
protocol: http
styleguide.torproject.org:
address: 7khzpw47s35pwo3lvtctwf2szvnq3kgglvzc22elx7of2awdzpovqmqd.onion
paths:
- path: ''
port: 80
protocol: http
submission.torproject.org:
address: givpjczyrb5jjseful3o5tn3tg7tidbu4gydl4sa5ekpcipivqaqnpad.onion
paths:
- path: ''
port: 80
protocol: http
support.torproject.org:
address: rzuwtpc4wb3xdzrj3yeajsvm3fkq4vbeubm2tdxaqruzzzgs5dwemlad.onion
paths:
- path: ''
port: 80
protocol: http
survey.torproject.org:
address: eh5esdnd6fkbkapfc6nuyvkjgbtnzq2is72lmpwbdbxepd2z7zbgzsqd.onion
paths:
- path: ''
port: 80
protocol: http
svn-archive.torproject.org:
address: b63iq6es4biaawfilwftlfkw6a6putogxh4iakei2ioppb7dsfucekyd.onion
paths:
- path: ''
port: 80
protocol: http
tagtor.torproject.org:
address: lx75vwrdgdgzpnnewquw2kngajieq6lqbblawoufjkf6fyqexhu4iiad.onion
paths:
- path: ''
port: 80
protocol: http
tb-manual.torproject.org:
address: dsbqrprgkqqifztta6h3w7i2htjhnq7d3qkh3c7gvc35e66rrcv66did.onion
paths:
- path: ''
port: 80
protocol: http
test-api.donate.torproject.org:
address: wiofesr5qt2k7qrlljpk53isgedxi6ddw6z3o7iay2l7ne3ziyagxaid.onion
paths:
- path: ''
port: 80
protocol: http
test-data.tbb.torproject.org:
address: umbk3kbgov4ekg264yulvbrpykfye7ohguqbds53qn547mdpt6o4qkad.onion
paths:
- path: ''
port: 80
protocol: http
test.crm.torproject.org:
address: a4d52y2erv4eijii66cpnyqn7rsnnq3gmtrsdxzt2laoutvu4gz7fwid.onion
paths:
- path: ''
port: 80
protocol: http
test.donate-api.torproject.org:
address: i4zhrn4md3ucd5dfgeo5lnqd3jy2z2kzp3lt4tdisvivzoqqtlrymkid.onion
paths:
- path: ''
port: 80
protocol: http
www.onion-router.net:
address: tttyx2vwp7ihml3vkhywwcizv6nbwrikpgeciy3qrow7l7muak2pnhad.onion
paths:
- path: ''
port: 80
protocol: http
www.torproject.org:
address: 2gzyxa5ihm7nsggfxnu52rck2vv4rvmdlkiu3zzui5du4xyclen53wid.onion
paths:
- path: ''
port: 80
protocol: http
http_connect_max_retries: 3
http_connect_timeout: 30
http_read_timeout: 30
interval: 60
launch_tor: true
log_level: info
loop: true
new_circuit: false
prometheus_exporter: true
prometheus_exporter_port: 9935
randomize: false
rounds: 0
shuffle: false
sleep: 60
socks_port: 19050
tor_address: 127.0.0.1
```
</details>Debian 12 bookworm upgradeanarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41338changes@db.torproject.org swallows emails instead fo replying2023-10-03T06:17:41ZGeorg Koppenchanges@db.torproject.org swallows emails instead fo replying```
07:38 < GeKo> i try to change a sudo password over at db.torproject.org, however
changes@db.torproject.org does not get back to me
07:38 < GeKo> is it stuck by chance?
07:39 < GeKo> going over my past encounters it eit...```
07:38 < GeKo> i try to change a sudo password over at db.torproject.org, however
changes@db.torproject.org does not get back to me
07:38 < GeKo> is it stuck by chance?
07:39 < GeKo> going over my past encounters it either immediately yelled at me
because of failures or immediately confirmed the change
07:40 < GeKo> but this time nothing happens
07:40 < GeKo> neither yesterday nor today. i just sent my requests and they seem to
vanish in a black hole :smile:
```Debian 12 bookworm upgradeanarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41337port puppet-managed configuration files to Debian bookworm2024-02-08T16:28:16Zanarcatport puppet-managed configuration files to Debian bookwormDuring the first batch of bookworm upgrades (#41251), we found a few issues with the Puppet configs that should probably be tweaked before the next batch to remove noise.
We have some slight diffs in our Puppet-managed NTP configuration...During the first batch of bookworm upgrades (#41251), we found a few issues with the Puppet configs that should probably be tweaked before the next batch to remove noise.
We have some slight diffs in our Puppet-managed NTP configuration:
```
Notice: /Stage[main]/Ntp/File[/etc/ntpsec/ntp.conf]/content:
--- /etc/ntpsec/ntp.conf 2023-09-26 14:41:08.648258079 +0000
+++ /tmp/puppet-file20230926-35001-x7hntz 2023-09-26 14:47:56.547991158 +0000
@@ -4,13 +4,13 @@
# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help
-driftfile /var/lib/ntpsec/ntp.drift
+driftfile /var/lib/ntp/ntp.drift
# Leap seconds definition provided by tzdata
leapfile /usr/share/zoneinfo/leap-seconds.list
# Enable this if you want statistics to be logged.
-#statsdir /var/log/ntpsec/
+#statsdir /var/log/ntpstats/
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
Notice: /Stage[main]/Ntp/File[/etc/ntpsec/ntp.conf]/content: content changed '{sha256}c5d627a596de1c67aa26dfbd472a4f07039f4664b1284cf799d4e1eb43c92c80' to '{sha256}18de87983c2f8491852390acc21c466611d6660083b0d0810bb6509470949be3'
Notice: /Stage[main]/Ntp/File[/etc/ntpsec/ntp.conf]/mode: mode changed '0644' to '0444'
Info: /Stage[main]/Ntp/File[/etc/ntpsec/ntp.conf]: Scheduling refresh of Exec[service ntpsec restart]
Info: /Stage[main]/Ntp/File[/etc/ntpsec/ntp.conf]: Scheduling refresh of Exec[service ntpsec restart]
Notice: /Stage[main]/Ntp/File[/etc/default/ntpsec]/content:
--- /etc/default/ntpsec 2023-07-29 20:51:53.000000000 +0000
+++ /tmp/puppet-file20230926-35001-d4tltp 2023-09-26 14:47:56.579990910 +0000
@@ -1,9 +1 @@
-NTPD_OPTS="-g -N"
-
-# Set to "yes" to ignore DHCP servers returned by DHCP.
-IGNORE_DHCP=""
-
-# If you use certbot to obtain a certificate for ntpd, provide its name here.
-# The ntpsec deploy hook for certbot will handle copying and permissioning the
-# certificate and key files.
-NTPSEC_CERTBOT_CERT_NAME=""
+NTPD_OPTS='-g'
Notice: /Stage[main]/Ntp/File[/etc/default/ntpsec]/content: content changed '{sha256}26bcfca8526178fc5e0df1412fbdff120a0d744cfbd023fef7b9369e0885f84b' to '{sha256}1bb4799991836109d4733e4aaa0e1754a1c0fee89df225598319efb83aa4f3b1'
Notice: /Stage[main]/Ntp/File[/etc/default/ntpsec]/mode: mode changed '0644' to '0444'
Info: /Stage[main]/Ntp/File[/etc/default/ntpsec]: Scheduling refresh of Exec[service ntpsec restart]
Info: /Stage[main]/Ntp/File[/etc/default/ntpsec]: Scheduling refresh of Exec[service ntpsec restart]
Notice: /Stage[main]/Ntp/Exec[service ntpsec restart]: Triggered 'refresh' from 4 events
```
Note that this is a "reverse diff", that is Puppet restoring the old
bullseye config, so we should apply the reverse of this in Puppet.
### sudo configuration lacks limits.conf?
Just notice this diff on all hosts:
```
--- /etc/pam.d/sudo 2021-12-14 19:59:20.613496091 +0000
+++ /etc/pam.d/sudo.dpkg-dist 2023-06-27 11:45:00.000000000 +0000
@@ -1,12 +1,8 @@
-##
-## THIS FILE IS UNDER PUPPET CONTROL. DON'T EDIT IT HERE.
-##
#%PAM-1.0
-# use the LDAP-derived password file for sudo access
-auth requisite pam_pwdfile.so pwdfile=/var/lib/misc/thishost/sudo-passwd
+# Set up user limits from /etc/security/limits.conf.
+session required pam_limits.so
-# disable /etc/password for sudo authentication, see #6367
-#@include common-auth
+@include common-auth
@include common-account
@include common-session-noninteractive
```
Why don't we have `pam_limits` setup? Historical oddity? To investigatte.Debian 12 bookworm upgradehttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41327test and possibly replace docker with podman in GitLab runners2023-09-25T15:00:57Zanarcattest and possibly replace docker with podman in GitLab runnersGitLab *finally* managed to port the GitLab runner infrastructure to be useable with Podman:
- https://docs.gitlab.com/runner/executors/docker.html#use-podman-to-run-docker-commands
- https://about.gitlab.com/releases/2022/08/22/gitlab-...GitLab *finally* managed to port the GitLab runner infrastructure to be useable with Podman:
- https://docs.gitlab.com/runner/executors/docker.html#use-podman-to-run-docker-commands
- https://about.gitlab.com/releases/2022/08/22/gitlab-15-3-released/#gitlab-runner-153
- https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27119
this could be tremendously useful for us, in many ways:
1. podman makes it much easier to run "rootless" containers, which could significantly improve the security of our runners
2. that, in turn, could make it easier to build container images in runners (see gitlab#123, gitlab#90, gitlab#89 for background on that work, and https://docs.gitlab.com/runner/executors/docker.html#using-podman-to-build-container-images-from-a-dockerfile for the upstream docs)
3. podman doesn't require a daemon, so runner jobs would could directly under systemd which, in turn, might make gitlab-runner upgrades less disruptive
4. podman is simpler than docker and therefore easier to package in Debian, which means the package may be more up to date (for example, upstream docker is at 22.06-beta0, but unstable has 20.10.17, and stable 20.10.5, while podman is at 4.2.0, which is already packaged in experimental, unstable has 3.4.7 and stable 3.0.1)
Unfortunately, GitLab and Podman has fixed their things in version 15.3 (which we run) and 4.2.0 (which we don't), respectively. So we're not quite ready to run this from the Debian side of things. First the 4.2.0 podman release would need to get into unstable, and there testing. *Then* we could see if we can either get a backport running, or setup a bookworm runner, which therefore might make this part of the %"Debian 12 bookworm upgrade" milestone.
See this page for progress on the podman packaging: https://tracker.debian.org/pkg/libpodDebian 12 bookworm upgradeJérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41321bookworm upgrades, third batch: High complexity2024-02-01T04:09:39Zanarcatbookworm upgrades, third batch: High complexityupgrade the following servers, if they still exist, one by one, carefully.
- [x] `alberti.torproject.org` - ~~bullseye upgrade was #40693~~ done in one shot, oops!
- [ ] `eugeni.torproject.org` - bullseye upgrade was #40694
- [ ] `hetz...upgrade the following servers, if they still exist, one by one, carefully.
- [x] `alberti.torproject.org` - ~~bullseye upgrade was #40693~~ done in one shot, oops!
- [ ] `eugeni.torproject.org` - bullseye upgrade was #40694
- [ ] `hetzner-hel1-01.torproject.org` - bullseye upgrade was #40695
- [ ] `pauli.torproject.org` - bullseye upgrade was #40696, puppetdb upgraded separately in #41341
Note that some of those machines might not be running bullseye yet, see the related tickets for more information. An upgrade to bullseye MUST be performed first, even if we batch-upgrade them to bookworm, as there are significant issues with skipping the bullseye upgrade. Those were found while accidentally upgrading `alberti` from buster to bookworm, see #40693.
In [TPA-RFC-57](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-57-bookworm-upgrades#high-complexity-individually-done) this also included the ganeti cluster upgrades, but those have been split up in separate tickets (#41254 and #41253).Debian 12 bookworm upgrade2024-02-14https://gitlab.torproject.org/tpo/tpa/team/-/issues/41254upgrade gnt-fsn Ganeti cluster to bookworm2023-09-14T18:32:29Zanarcatupgrade gnt-fsn Ganeti cluster to bookwormupgrade the gnt-fsn cluster to bookworm, which means those nodes
- [ ] fsn-node-01
- [ ] fsn-node-02
- [ ] fsn-node-03
- [ ] fsn-node-04
- [ ] fsn-node-05
- [ ] fsn-node-06
- [ ] fsn-node-07
- [ ] fsn-node-08
mind the [Ganeti upgrade p...upgrade the gnt-fsn cluster to bookworm, which means those nodes
- [ ] fsn-node-01
- [ ] fsn-node-02
- [ ] fsn-node-03
- [ ] fsn-node-04
- [ ] fsn-node-05
- [ ] fsn-node-06
- [ ] fsn-node-07
- [ ] fsn-node-08
mind the [Ganeti upgrade procedure](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bookworm#ganeti-upgrades) which still needs to be updated from the bullseye procedure. also note that [this patch](https://github.com/ganeti/ganeti/pull/1694) is necessary... be careful with this one, it might not be ready yet.
possibly consider upgrading the (smaller?) gnt-dal cluster (#41253) first.Debian 12 bookworm upgradehttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41253upgrade gnt-dal Ganeti cluster to bookworm2023-09-14T18:32:29Zanarcatupgrade gnt-dal Ganeti cluster to bookwormupgrade the gnt-dal cluster to bookworm, which means those nodes
- [ ] dal-node-01
- [ ] dal-node-02
- [ ] dal-node-03
mind the [Ganeti upgrade procedure](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bookworm#ganet...upgrade the gnt-dal cluster to bookworm, which means those nodes
- [ ] dal-node-01
- [ ] dal-node-02
- [ ] dal-node-03
mind the [Ganeti upgrade procedure](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bookworm#ganeti-upgrades) which still needs to be updated from the bullseye procedure. also note that [this patch](https://github.com/ganeti/ganeti/pull/1694) is necessary... be careful with this one, it might not be ready yet.Debian 12 bookworm upgradehttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41252bookworm upgrades, second batch2024-02-06T18:22:35Zanarcatbookworm upgrades, second batchupgrade all those servers to Debian bookworm
- [x] bacula-director-01.torproject.org (@lavamind)
- [x] btcpayserver-02.torproject.org (@anarcat)
- [x] bungei.torproject.org (@lavamind, `crypttab` had a typo, had to boot rescue to recove...upgrade all those servers to Debian bookworm
- [x] bacula-director-01.torproject.org (@lavamind)
- [x] btcpayserver-02.torproject.org (@anarcat)
- [x] bungei.torproject.org (@lavamind, `crypttab` had a typo, had to boot rescue to recover, grub KVM console seems inaccessible)
- [x] carinatum.torproject.org (@kez, had issues during upgrade, followup in https://gitlab.torproject.org/tpo/network-health/doctor/-/issues/40034)
- [x] check-01.torproject.org (@kez, had a prolonged outage: https://gitlab.torproject.org/tpo/network-health/metrics/tor-check/-/issues/40017, mod_qos broken, followup in https://gitlab.torproject.org/tpo/tpa/team/-/issues/41509)
- [x] colchicifolium.torproject.org (@anarcat)
- [x] collector-02.torproject.org (@anarcat)
- [x] crm-ext-01.torproject.org (@anarcat, PHP 8 compatibility issue, followup in #41511)
- [x] crm-int-01.torproject.org (@anarcat)
- [x] dangerzone-01.torproject.org (@kez)
- [x] donate-review-01.torproject.org (@kez)
- [x] gayi.torproject.org (@anarcat)
- [x] gitlab-02.torproject.org (@anarcat, migrated to standalone postgresql following upgrade issue #41426)
- [x] henryi.torproject.org (@kez)
- [x] majus.torproject.org (@anarcat, obsolete transifex-client package left around)
- [x] materculae.torproject.org (@lavamind, noticed extra load on the server, filed https://gitlab.torproject.org/tpo/tpa/team/-/issues/41507)
- [x] meronense.torproject.org (@lavamind, possible OOM regression see #41515)
- [x] metrics-store-01.torproject.org (@anarcat)
- [x] nevii.torproject.org (@anarcat, handful of issues with paths moved from `/usr/sbin` to `/usr/bin`)
- [x] onionbalance-02.torproject.org (@anarcat)
- [x] onionoo-backend-01.torproject.org (@anarcat)
- [x] onionoo-backend-02.torproject.org (@anarcat)
- [x] onionoo-frontend-01.torproject.org (@anarcat)
- [x] onionoo-frontend-02.torproject.org (@anarcat)
- [x] polyanthum.torproject.org (@lavamind)
- [x] probetelemetry-01.torproject.org (@anarcat)
- [x] rdsys-frontend-01.torproject.org (@anarcat)
- [x] rude.torproject.org (@lavamind)
- [x] survey-01.torproject.org (@lavamind)
- [x] telegram-bot-01.torproject.org (@anarcat)
- [x] weather-01.torproject.org (@anarcat, catastrophic data loss, see #41388)
31 machines
like the first batch, due date is approximate here, used to ping the team to organise this before the actual week planned in TPA-RFC-57, which is "last week of october".
an announcement need to be sent to remind people of this upcoming batch when the first due date is hit.Debian 12 bookworm upgradeanarcatanarcat2024-01-29https://gitlab.torproject.org/tpo/tpa/team/-/issues/41251bookworm upgrades, first batch2023-09-27T14:04:30Zanarcatbookworm upgrades, first batchupgrade all those servers to Debian bookworm
- [x] archive-01.torproject.org (@lavamind)
- [x] cdn-backend-sunet-02.torproject.org (@lavamind)
- [x] ci-runner-x86-03.torproject.org (@lavamind)
- [x] chives.torproject.org (@lavamind)
- [...upgrade all those servers to Debian bookworm
- [x] archive-01.torproject.org (@lavamind)
- [x] cdn-backend-sunet-02.torproject.org (@lavamind)
- [x] ci-runner-x86-03.torproject.org (@lavamind)
- [x] chives.torproject.org (@lavamind)
- [x] ~~ci-runner-x86-01.torproject.org~~ (retired)
- [x] dal-rescue-01.torproject.org (@lavamind)
- [x] dal-rescue-02.torproject.org (@lavamind)
- [x] hetzner-hel1-02.torproject.org (@lavamind)
- [x] hetzner-hel1-03.torproject.org (@lavamind)
- [x] hetzner-nbg1-01.torproject.org (@anarcat )
- [x] hetzner-nbg1-02.torproject.org (@anarcat)
- [x] loghost01.torproject.org (@lavamind)
- [x] mandos-01.torproject.org (@lavamind)
- [x] media-01.torproject.org (@lavamind)
- [x] neriniflorum.torproject.org (@lavamind)
- [x] ns3.torproject.org (@lavamind)
- [x] ns5.torproject.org (@anarcat)
- [x] palmeri.torproject.org (@lavamind)
- [x] perdulce.torproject.org (@anarcat)
- [x] relay-01.torproject.org (@anarcat)
- [x] static-gitlab-shim.torproject.org (@anarcat)
- [x] static-master-fsn.torproject.org (@anarcat)
- [x] staticiforme.torproject.org (@anarcat)
- [x] submit-01.torproject.org (@anarcat)
- [x] ~~tb-build-01.torproject.org~~ (retired)
- [x] tb-build-04.torproject.org (@anarcat)
- [x] tb-build-05.torproject.org (@anarcat)
- [x] tb-pkgstage-01.torproject.org (@anarcat)
- [x] tb-tester-01.torproject.org (@anarcat)
- [x] tbb-nightlies-master.torproject.org (@anarcat)
- [x] web-dal-07.torproject.org (@lavamind)
- [x] web-dal-08.torproject.org (@anarcat)
- [x] web-fsn-01.torproject.org (@anarcat)
- [x] web-fsn-02.torproject.org (@anarcat)
33 machines
due date is set as an alarm to organise this for the second or third week of September, with a 2 week advance notice. the actual due date should be more like 2023-09-22 or something like that.
an announcement need to be sent to remind people of this upcoming batch when the first due date is hit.Debian 12 bookworm upgradeanarcatanarcat2023-09-25https://gitlab.torproject.org/tpo/tpa/team/-/issues/41245TPA-RFC-57: bookworm upgrade plan2024-01-09T16:42:45ZanarcatTPA-RFC-57: bookworm upgrade planin the %"Debian 11 bullseye upgrade" we did three main batches of upgrades, and that worked extremely well.
do something similar to https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-20-bullseye-upgrades but take into acc...in the %"Debian 11 bullseye upgrade" we did three main batches of upgrades, and that worked extremely well.
do something similar to https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-20-bullseye-upgrades but take into account the [bullseye post-mortem](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bullseye#post-mortem). specifically, don't block the upgrade on complex services, if at all possible.
also need to consider the trailing %"Debian 11 bullseye upgrade" servers (6 at the current count), although those are mostly going to be rebuilt (pauli/eugeni/nagios) or retired (cupani, vineale). The only one that still needs a kick is alberti, and it's *possible* it could be double-upgraded to bookworm as well.
this is being drafted in https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-57-bookworm-upgradesDebian 12 bookworm upgradeanarcatanarcat2023-07-26https://gitlab.torproject.org/tpo/tpa/team/-/issues/41244switch installers to bookworm2023-07-04T19:46:29Zanarcatswitch installers to bookwormas part of the %"Debian 12 bookworm upgrade" milestone, we should switch the installers to setup only new bookworm machines from now on.
* [x] ganeti: @lavamind added this to `hiera/common.yaml`, but maybe it should be made upstream as ...as part of the %"Debian 12 bookworm upgrade" milestone, we should switch the installers to setup only new bookworm machines from now on.
* [x] ganeti: @lavamind added this to `hiera/common.yaml`, but maybe it should be made upstream as well:
```
ganeti::debootstrap_variants:
bookworm:
suite: "bookworm"
```
* [x] @lavamind fix `Puppetfile` to point at https://gitlab.com/shared-puppet-modules-group/puppet-ganeti/-/commit/ff3814b6572dc03621fcd6027d23c89f55a7fe8f
* [x] wiki instructions (https://gitlab.torproject.org/tpo/tpa/wiki-replica/-/commit/c97207ab874ab9f49946c3c88e1b45468d5914a7)
* [x] fabric_tpaDebian 12 bookworm upgradeJérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41199Clean up PGDG packages on bungei after bookworm upgrade2024-01-29T22:22:45ZJérôme Charaouilavamind@torproject.orgClean up PGDG packages on bungei after bookworm upgradeAfter setting up a PostgreSQL instance on a new `bookworm` machine, backups didn't work:
```
root@bungei:~# sudo -u torbackup postgres-make-one-base-backup $(grep ^metricsdb-01.torproject.org $(which postgres-make-base-backups )) ...After setting up a PostgreSQL instance on a new `bookworm` machine, backups didn't work:
```
root@bungei:~# sudo -u torbackup postgres-make-one-base-backup $(grep ^metricsdb-01.torproject.org $(which postgres-make-base-backups ))
Doing metricsdb-01.torproject.org:5432 15/main: bungei.torproject.org-20230531-143231-metricsdb-01.torproject.org-main-15-backup
could not change directory to "/root": Permission denied
pg_basebackup: error: incompatible server version 15.3 (Debian 15.3-0+deb12u1)
pg_basebackup failed with exit code 1
```
So I upgraded the PostgreSQL-15 client packages using the [PDGD Apt repository](https://wiki.postgresql.org/wiki/Apt) which provides packages that play nice with the Debian ecosystem:
```
root@bungei:~# dpkg --list | grep .pgdg
ii libpq5:amd64 15.3-1.pgdg110+1 amd64 PostgreSQL C client library
ii postgresql-client 15+250.pgdg110+1 all front-end programs for PostgreSQL (supported version)
ii postgresql-client-15 15.3-1.pgdg110+1 amd64 front-end programs for PostgreSQL 15
ii postgresql-client-common 250.pgdg110+1 all manager for multiple PostgreSQL client versions
ii python3-psycopg2 2.9.6-1.pgdg110+1 amd64 Python 3 module for PostgreSQL
```
This ticket is here to ensure that once `bungei` is upgraded to `bookworm`, those packages are replaced with the Debian archive versions.Debian 12 bookworm upgradeJérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41197Puppet agent emits warning about Ssh_authorized_key_type on bookworm2023-05-31T15:56:28ZJérôme Charaouilavamind@torproject.orgPuppet agent emits warning about Ssh_authorized_key_type on bookwormOn `bookworm` machines, puppet agent (version 7.23) emits the following warning every time it runs:
puppet-agent[472436]: (/Stage[main]/Profile::Admins/User[root]) Ssh_authorized_key type is not available. Cannot purge SSH keys.
Th...On `bookworm` machines, puppet agent (version 7.23) emits the following warning every time it runs:
puppet-agent[472436]: (/Stage[main]/Profile::Admins/User[root]) Ssh_authorized_key type is not available. Cannot purge SSH keys.
This is likely related to the removal of sshkey management in core Puppet 7, and installing `puppetlabs-module-sshkeys-core` on the node doesn't fix the problem. The server is likely simply unaware that the agent cannot deal with this resource type anymore.
We may want to manage `root`'s `authorized_keys` another way if we can't find a fix.Debian 12 bookworm upgradeJérôme Charaouilavamind@torproject.orgJérôme Charaouilavamind@torproject.orghttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41191systemd-binfmt.service failing on bookworm2023-05-31T17:05:39ZJérôme Charaouilavamind@torproject.orgsystemd-binfmt.service failing on bookwormOn a regular `bookworm` installation, after Puppetization I noticed that `systemd-binfmt.service` would fail to start up.
I traced it to this tidbit in our `torproject-org` Puppet class:
```puppet
file { '/etc/systemd/system/proc-sys-f...On a regular `bookworm` installation, after Puppetization I noticed that `systemd-binfmt.service` would fail to start up.
I traced it to this tidbit in our `torproject-org` Puppet class:
```puppet
file { '/etc/systemd/system/proc-sys-fs-binfmt_misc.automount':
ensure => 'link',
target => '/dev/null',
notify => Exec['systemctl daemon-reload'],
}
```
It was introduced in tor-puppet.git commit b1b027dadda4b5cabbe225340d4627b71b0d9f92, with no explanation.
Why are we deploying that? Do we need to keep it around?Debian 12 bookworm upgradeanarcatanarcathttps://gitlab.torproject.org/tpo/tpa/team/-/issues/41160Upgrade userdir-ldap to Python 32023-05-11T17:27:30ZJérôme Charaouilavamind@torproject.orgUpgrade userdir-ldap to Python 3Currently it's not possible to fully integrate a Debian `bookworm` machine (such as `forum-test-01`) into the fleet because `userdir-ldap` requires Python 2.7, and it's been removed from Debian.
Please port `userdir-ldap` to Python 3.Currently it's not possible to fully integrate a Debian `bookworm` machine (such as `forum-test-01`) into the fleet because `userdir-ldap` requires Python 2.7, and it's been removed from Debian.
Please port `userdir-ldap` to Python 3.Debian 12 bookworm upgradeanarcatanarcat