onionbalance-02 (onionbalance v3) doesn't survive tor upgrades
17:29:52 <+anarcat> weasel: any idea why the www.tpo onionv3 service might be down? http://2gzyxa5ihm7nsggfxnu52rck2vv4rvmdlkiu3zzui5du4xyclen53wid.onion/
17:30:54 <+weasel> the balancers seem all dead?
17:31:19 <+anarcat> oh onionbalance-02 is degraded
17:31:31 <+anarcat> intersting, i remember something like this happening during hiro's first tests
17:31:46 <+anarcat> should i kick them back up?
17:32:07 <+weasel> May 27 16:41:47 onionbalance-02 onionbalance[17129]: 2021-05-27 16:41:47,518 [WARNING]: Descriptor action failed for unknown service
cjdtxt7e3yo6mr42llqlxh6lggjdbsmcnt5un3l5nlfrg3camonaoqad to $6C333B5BDBA3DFE7A782E40C
17:32:08 <+weasel> May 27 17:46:47 onionbalance-02 onionbalance[17129]: 2021-05-27 17:46:47,379 [WARNING]: Descriptor action failed for unknown service
ueghr2hzndecdntou33mhymbbxj7pir74nwzhqr6drhxpbz3j272p4id to $CF714ECC37179E667F9C1AA8
17:32:08 <+weasel> May 27 19:23:49 onionbalance-02 systemd[1]: Stopping OnionBalance - Tor Onion Service load balancer for nyx.torproject.org...
17:32:08 <+weasel> May 27 19:23:54 onionbalance-02 systemd[1]: torproject-onionbalance@nyx.torproject.org.service: State 'stop-sigterm' timed out. Killing.
17:32:08 <+weasel> May 27 19:23:54 onionbalance-02 systemd[1]: torproject-onionbalance@nyx.torproject.org.service: Killing process 17129 (/usr/bin/python) with signal SIGKILL.
17:32:09 <+weasel> May 27 19:23:54 onionbalance-02 systemd[1]: torproject-onionbalance@nyx.torproject.org.service: Main process exited, code=killed, status=9/KILL
17:32:09 <+weasel> May 27 19:23:54 onionbalance-02 systemd[1]: torproject-onionbalance@nyx.torproject.org.service: Failed with result 'timeout'.
17:32:11 <+weasel> May 27 19:23:54 onionbalance-02 systemd[1]: Stopped OnionBalance - Tor Onion Service load balancer for nyx.torproject.org.
17:32:44 <+weasel> that matches the time the tor package was upgraded.
17:33:01 <+weasel> root@onionbalance-02:~# ls -lrt /var/lib/dpkg/info/ | tail -n1
17:33:01 <+weasel> -rw-r--r-- 1 root root 1659 May 27 19:23 tor.list
17:33:58 <+Sebastian> anarcat: sorry, I meant just the onion of course. Hope that didn't inspire a panic
17:35:04 <+weasel> torproject-onionbalance@.service has a BindsTo=tor.service
17:35:19 <+weasel> so I'm not quite sure why things were, apparently, stopped but not restarted.
17:35:22 <+anarcat> Sebastian: not at all :)
17:35:45 <+anarcat> sigh, another package that fails to upgrade?
17:35:57 <+weasel> onionbalance has never survived tor restarts.
17:36:08 <+anarcat> thanks for the investigation weasel, maybe that desserves a ticket somewhere ;)
17:36:12 <+weasel> we fixed that in our old setup by modifying the service file and adding a BindsTo=tor.service
17:36:14 <+anarcat> but i'm heading for dinner here
17:36:20 <+weasel> which, I think on v2, caused things to work
17:36:28 <+anarcat> i see
17:36:31 <+weasel> I'll just restart things for now
17:36:35 <+anarcat> thanks
17:36:40 <+anarcat> is that the tor or the onionbalance upgrade?
17:36:42 <+anarcat> i assume the latter?
17:36:47 <+weasel> the tor upgrade
17:36:53 <+weasel> it got updated in testing
17:36:55 <+anarcat> i meant the former
17:36:56 <+anarcat> right
17:37:00 <+anarcat> i did see that fly by
17:37:06 <+anarcat> okay thanks, gtg :)
17:37:12 <+weasel> cheers
17:37:23 <+weasel> the underlying issue deserves a ticket, but I'm on my out
17:37:23 <+Sebastian> danke weasel and anarcat
17:38:20 <+anarcat> ack
/cc @weasel