onionprobe broke during the bookworm upgrade

During the first batch of bookworm upgrades (#41251 (closed)), we found that onionprobe stopped working on the main prometheus server (prometheus1.torproject.org, AKA hetzner-nbg1-01):

root@hetzner-nbg1-01:~# systemctl status onionprobe
× onionprobe.service - Onionprobe
     Loaded: loaded (/lib/systemd/system/onionprobe.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Wed 2023-09-27 11:26:31 UTC; 2h 36min ago
   Duration: 247ms
       Docs: file:///usr/share/doc/onionprobe/README.Debian
             man:onionprobe(1)
             https://gitlab.torproject.org/tpo/onion-services/onionprobe
    Process: 78817 ExecStart=/usr/bin/onionprobe -c $CONFIG $DAEMON_OPTS (code=exited, status=1/FAILURE)
   Main PID: 78817 (code=exited, status=1/FAILURE)
        CPU: 245ms

Sep 27 11:26:30 hetzner-nbg1-01 systemd[1]: Started onionprobe.service - Onionprobe.
Sep 27 11:26:31 hetzner-nbg1-01 onionprobe[78817]: 2023-09-27 11:26:31,048 INFO: Starting Onionprobe version 1.0.0...
Sep 27 11:26:31 hetzner-nbg1-01 onionprobe[78817]: 2023-09-27 11:26:31,049 INFO: Initializing Tor process...
Sep 27 11:26:31 hetzner-nbg1-01 onionprobe[78817]: 2023-09-27 11:26:31,073 ERROR: Process terminated: Unknown option '16:578F4DEC1A07C7FF600CB53BD614DB4E175369A9A5FB4813DBD30C1527'.  Failing.
Sep 27 11:26:31 hetzner-nbg1-01 onionprobe[78817]: 2023-09-27 11:26:31,073 CRITICAL: Error initializing Tor
Sep 27 11:26:31 hetzner-nbg1-01 systemd[1]: onionprobe.service: Main process exited, code=exited, status=1/FAILURE
Sep 27 11:26:31 hetzner-nbg1-01 systemd[1]: onionprobe.service: Failed with result 'exit-code'.

Here's a config dump:

Click to expand

root@hetzner-nbg1-01:~# cat /etc/default/onionprobe 
# This file is managed by Puppet, local changes will be lost
#
# Default settings for the Onionprobe system-wide service
#
# This is used only for the system-wide service and has no
# effect when using the onionprobe command line directly.

# The configuration file to use
CONFIG="/etc/onionprobe/tpo.yaml"

# Additional Onionprobe command line options to use
DAEMON_OPTS=""
root@hetzner-nbg1-01:~# cat /etc/onionprobe/tpo.yaml
circuit_stream_timeout: 60
control_port: 19051
descriptor_max_retries: 5
descriptor_timeout: 30
endpoints:
  2019.www.torproject.org:
    address: jqyzxhjk6psc6ul5jnfwloamhtyh7si74b4743k2qgpskwwxrzhsxmad.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  api.donate.torproject.org:
    address: rbi3fpvpz4vlrx67scoqef2zxz7k4xyiludszg655favvkygjmhz6wyd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  archive.torproject.org:
    address: uy3qxvwzwoeztnellvvhxh7ju7kfvlsauka7avilcjg7domzxptbq7qd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  aus1.torproject.org:
    address: ot3ivcdxmalbsbponeeq5222hftpf3pqil24q3s5ejwo5t52l65qusid.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  aus2.torproject.org:
    address: b5t7emfr2rn3ixr4lvizpi3stnni4j4p6goxho7lldf4qg4hz5hvpqid.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  blog.torproject.org:
    address: pzhdfe7jraknpj2qgu5cz2u3i4deuyfwmonvzu5i3nyw4t4bmg7o5pad.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  bridges.torproject.org:
    address: yq5jjvr7drkjrelzhut7kgclfuro65jjlivyzfmxiq2kyv5lickrl4qd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  cloud.torproject.org:
    address: ui3cpcohcoko6aydhuhlkwqqtvadhaflcc5zb7mwandqmcal7sbwzwqd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  collector.torproject.org:
    address: pgmrispjerzzf2tdzbfp624cg5vpbvdw2q5a3hvtsbsx25vnni767yad.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  collector2.torproject.org:
    address: urscdffm73o4y6hpp3r43bgmudq42hq2ibdpkld6a7hy3qa44qbvc2yd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  community.torproject.org:
    address: xmrhfasfg5suueegrnc4gsgyi2tyclcy5oz7f5drnrodmdtob6t2ioyd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  consensus-health.torproject.org:
    address: tkskz5dkjel4xqyw5d5l3k52kgglotwn6vgb5wrl2oa5yi2szvywiyid.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  crm.torproject.org:
    address: 6ojylpznauimd2fga3m7g24vd7ebkzlemxdprxckevqpzs347ugmynqd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  deb.torproject.org:
    address: apow7mjfryruh65chtdydfmqfpj5btws7nbocgtaovhvezgccyjazpqd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  dev.crm.torproject.org:
    address: eewp4iydzyu2a5d6bvqadadkozxdbhsdtmsoczu2joexfrjjsheaecad.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  dist.torproject.org:
    address: scpalcwstkydpa3y7dbpkjs2dtr7zvtvdbyj3dqwkucfrwyixcl5ptqd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  donate-api.torproject.org:
    address: lkfkuhcx62yfvzuz5o3ju4divuf4bsh2bybwd3oierq47kyp2ig2gvid.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  donate.torproject.org:
    address: yoaenchicimox2qdc47p36zm3cuclq7s7qxx6kvxqaxjodigfifljqqd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  exonerator.torproject.org:
    address: pm46i5h2lfewyx6l7pnicbxhts2sxzacvsbmqiemqaspredf2gm3dpad.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  extra.torproject.org:
    address: kkr72iohlfix5ipjg776eyhplnl2oiv5tz4h2y2bkhjix3quafvjd5ad.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  forum.torproject.org:
    address: v236xhqtyullodhf26szyjepvkbv6iitrhjgrqj4avaoukebkk6n6syd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  gettor.torproject.org:
    address: ueghr2hzndecdntou33mhymbbxj7pir74nwzhqr6drhxpbz3j272p4id.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  git.torproject.org:
    address: xtlfhaspqtkeeqxk6umggfbr3gyfznvf4jhrge2fujz53433i2fcs3id.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  gitlab.torproject.org:
    address: eweiibe6tdjsdprb4px6rqrzzcsi22m4koia44kc5pcjr7nec2rlxyad.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  gitweb.torproject.org:
    address: gzgme7ov25seqjbphab4fkcph3jkobfwwpivt5kzbv3kqx2y2qttl4yd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  grafana1.torproject.org:
    address: 7zjnw5lx2x27rwiocxkqdquo7fawj46mf2wiu2l7e6z6ng6nivmdxnad.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  grafana2.torproject.org:
    address: f3vd6fyiccuppybkxiblgigej3pfvvqzjnhd3wyv7h4ee5asawf2fhqd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  ircbouncer.torproject.org:
    address: moz5kotsnjony4oxccxfo4lwk3pvoxmdoljibhgoonzgzjs5oemtjmqd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  metrics-api.torproject.org:
    address: yc3galza3gejn3taziuhhgrwt4bdtwmom25zby7jphfwbeirvkmcdvqd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  metrics-db.torproject.org:
    address: lk6lj36rfj32u2rjceujj3o7otgujm6fw5hyyxr6jko6pkfasb2z6eid.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  metrics.torproject.org:
    address: hctxrvjzfpvmzh2jllqhgvvkoepxb4kfzdjm6h7egcwlumggtktiftid.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  moat.torproject.org:
    address: z7m7ogzdhu43nosvjtsuplfmuqa3ge5obahixydhmzdox6owwxfoxzid.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  nagios.torproject.org:
    address: w6vizvw4ckesva5fvlkrepynemxdq6pgo5sh4r76ec6msq5notkhqryd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  newsletter.torproject.org:
    address: a4ygisnerpgtc5ayerl22pll6cls3oyj54qgpm7qrmb66xrxts6y3lyd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  nightlies.tbb.torproject.org:
    address: umj4zbqdfcyevlkgqgpq6foxk3z75zzxsbgt5jqmfxofrbrjh3crbnad.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  nyx.torproject.org:
    address: 3ewfgrt4gzfccp6bnquhqb266r3zepiqpnsk3falwygkegtluwuyevid.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  onion.torproject.org:
    address: xao2lxsmia2edq2n5zxg6uahx6xox2t7bfjw6b5vdzsxi7ezmqob6qid.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  onionoo.torproject.org:
    address: dud2sxm6feahhuwj4y4lzktduy7v3qpaqsfkggtj2ojmzathttkegoid.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  openpgpkey.torproject.org:
    address: 2yldcptk56shc7lwieozoglw3t5ghty7m6mf2faysvfnzccqavbu2mad.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  people.torproject.org:
    address: 5ecey6oe4rocdsfoigr4idu42cecm2j7zfogc3xc7kfn4uriehwrs6qd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  prometheus1.torproject.org:
    address: ydok5jiruh3ak6hcfdlm2g7iuraaxcomeckj2nucjsxif6qmrrda2byd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  prometheus2.torproject.org:
    address: vyo6yrqhl3by7d6n5t6hjkflaqbarjpqjnvapr5u5rafk4imnfrmcjyd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  rbm.torproject.org:
    address: nkuz2tpok7ctwd5ueer5bytj3bm42vp7lgjcsnznal3stotg6vyaakyd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  research.torproject.org:
    address: xhqthou6scpfnwjyzc3ekdgcbvj76ccgyjyxp6cgypxjlcuhnxiktnqd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  review.torproject.net:
    address: zhkhhhnppc5k6xju7n25rjba3wuip73jnodicxl65qdpchrwvvsilcyd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  rpm.torproject.org:
    address: 4ayyzfoh5qdrokqaejis3rdredhvf22n3migyxfudpkpunngfc7g4lqd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  snowflake.torproject.org:
    address: oljlphash3bpqtrvqpr5gwzrhroziw4mddidi5d2qa4qjejcbrmoypqd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  spec.torproject.org:
    address: i3xi5qxvbrngh3g6o7czwjfxwjzigook7zxzjmgwg5b7xnjcn5hzciad.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  staging-api.donate.torproject.org:
    address: vorwws6g6mx23djlznmlqva4t5olulpnet6fxyiyytcu5dorp3fstdqd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  staging.crm.torproject.org:
    address: pt34uujusar4arrvsqljndqlt7tck2d5cosaav5xni4nh7bmvshyp2yd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  staging.donate-api.torproject.org:
    address: 7niqsyixinnhxvh33zh5dqnplxnc2yd6ktvats3zmtbbpzcphpbsa6qd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  status.torproject.org:
    address: eixoaclv7qvnmu5rolbdwba65xpdiditdoyp6edsre3fitad777jr3ad.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  stem.torproject.org:
    address: mf34jlghauz5pxjcmdymdqbe5pva4v24logeys446tdrgd5lpsrocmqd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  styleguide.torproject.org:
    address: 7khzpw47s35pwo3lvtctwf2szvnq3kgglvzc22elx7of2awdzpovqmqd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  submission.torproject.org:
    address: givpjczyrb5jjseful3o5tn3tg7tidbu4gydl4sa5ekpcipivqaqnpad.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  support.torproject.org:
    address: rzuwtpc4wb3xdzrj3yeajsvm3fkq4vbeubm2tdxaqruzzzgs5dwemlad.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  survey.torproject.org:
    address: eh5esdnd6fkbkapfc6nuyvkjgbtnzq2is72lmpwbdbxepd2z7zbgzsqd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  svn-archive.torproject.org:
    address: b63iq6es4biaawfilwftlfkw6a6putogxh4iakei2ioppb7dsfucekyd.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  tagtor.torproject.org:
    address: lx75vwrdgdgzpnnewquw2kngajieq6lqbblawoufjkf6fyqexhu4iiad.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  tb-manual.torproject.org:
    address: dsbqrprgkqqifztta6h3w7i2htjhnq7d3qkh3c7gvc35e66rrcv66did.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  test-api.donate.torproject.org:
    address: wiofesr5qt2k7qrlljpk53isgedxi6ddw6z3o7iay2l7ne3ziyagxaid.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  test-data.tbb.torproject.org:
    address: umbk3kbgov4ekg264yulvbrpykfye7ohguqbds53qn547mdpt6o4qkad.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  test.crm.torproject.org:
    address: a4d52y2erv4eijii66cpnyqn7rsnnq3gmtrsdxzt2laoutvu4gz7fwid.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  test.donate-api.torproject.org:
    address: i4zhrn4md3ucd5dfgeo5lnqd3jy2z2kzp3lt4tdisvivzoqqtlrymkid.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  www.onion-router.net:
    address: tttyx2vwp7ihml3vkhywwcizv6nbwrikpgeciy3qrow7l7muak2pnhad.onion
    paths:
    - path: ''
    port: 80
    protocol: http
  www.torproject.org:
    address: 2gzyxa5ihm7nsggfxnu52rck2vv4rvmdlkiu3zzui5du4xyclen53wid.onion
    paths:
    - path: ''
    port: 80
    protocol: http
http_connect_max_retries: 3
http_connect_timeout: 30
http_read_timeout: 30
interval: 60
launch_tor: true
log_level: info
loop: true
new_circuit: false
prometheus_exporter: true
prometheus_exporter_port: 9935
randomize: false
rounds: 0
shuffle: false
sleep: 60
socks_port: 19050
tor_address: 127.0.0.1