Trac issueshttps://gitlab.torproject.org/legacy/trac/-/issues2020-06-13T15:53:26Zhttps://gitlab.torproject.org/legacy/trac/-/issues/34088circuit_build_times_update_alpha(): Bug: Could not determine largest build time2020-06-13T15:53:26Zs7rcircuit_build_times_update_alpha(): Bug: Could not determine largest build timeI don't think this is HSv3 ONLY related, but I can only make it happen while building large amounts of rendezvous circuits with reasonable concurrency (over 50). I am not assigning the tor-hs keyword for this reason, but sticking it to s...I don't think this is HSv3 ONLY related, but I can only make it happen while building large amounts of rendezvous circuits with reasonable concurrency (over 50). I am not assigning the tor-hs keyword for this reason, but sticking it to same master parent ticket.
All lines are the same. It is seen for like 130-160 times during the build of little over 100.000 rendezvous circuits.
```
Apr 03 04:05:33.000 [warn] circuit_build_times_update_alpha(): Bug: Could not determine largest build time (0). Xm is 20025ms and we've abandoned 0 out of 136 circuits. (on Tor 0.4.4.0-alpha-dev )
Apr 03 04:05:33.000 [warn] circuit_build_times_update_alpha(): Bug: Could not determine largest build time (0). Xm is 20025ms and we've abandoned 0 out of 137 circuits. (on Tor 0.4.4.0-alpha-dev )
Apr 03 04:05:33.000 [warn] circuit_build_times_update_alpha(): Bug: Could not determine largest build time (0). Xm is 20025ms and we've abandoned 0 out of 138 circuits. (on Tor 0.4.4.0-alpha-dev )
Apr 03 04:05:33.000 [warn] circuit_build_times_update_alpha(): Bug: Could not determine largest build time (0). Xm is 20025ms and we've abandoned 0 out of 139 circuits. (on Tor 0.4.4.0-alpha-dev )
Apr 03 04:05:33.000 [warn] circuit_build_times_update_alpha(): Bug: Could not determine largest build time (0). Xm is 20025ms and we've abandoned 0 out of 140 circuits. (on Tor 0.4.4.0-alpha-dev )
```
EDIT: All lines are the same except it always abandons 0 out of N circuits and N is always different of course, increasing with +1 most of the times until the Bug warn disappears.Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/34087HSv3: Bug: Non-fatal assertion in hs_client.c:981: close_or_reextend_intro_circ2020-06-13T15:53:26Zs7rHSv3: Bug: Non-fatal assertion in hs_client.c:981: close_or_reextend_intro_circClient side HSv3 non fatal bug. It occurs like between ~20 - 30 times in a total of 100.000 built rendezvous circuits:
```
Apr 11 08:58:25.000 [warn] tor_bug_occurred_(): Bug: ../src/feature/hs/hs_client.c:981: close_or_reextend_intro_c...Client side HSv3 non fatal bug. It occurs like between ~20 - 30 times in a total of 100.000 built rendezvous circuits:
```
Apr 11 08:58:25.000 [warn] tor_bug_occurred_(): Bug: ../src/feature/hs/hs_client.c:981: close_or_reextend_intro_circ: Non-fatal assertion !(desc == NULL) failed. (on Tor 0.4.4.0-alpha-dev )
Apr 11 08:58:25.000 [warn] Bug: Tor 0.4.4.0-alpha-dev: Non-fatal assertion !(desc == NULL) failed in close_or_reextend_intro_circ at ../src/feature/hs/hs_client.c:981. Stack trace: (on Tor 0.4.4.0-alpha-dev )
Apr 11 08:58:25.000 [warn] Bug: tor(log_backtrace_impl+0x56) [0x55d874ac7ee6] (on Tor 0.4.4.0-alpha-dev )
Apr 11 08:58:25.000 [warn] Bug: tor(tor_bug_occurred_+0x16c) [0x55d874ac30ec] (on Tor 0.4.4.0-alpha-dev )
Apr 11 08:58:25.000 [warn] Bug: tor(hs_client_receive_introduce_ack+0x2f5) [0x55d8749cfe95] (on Tor 0.4.4.0-alpha-dev )
Apr 11 08:58:25.000 [warn] Bug: tor(rend_process_relay_cell+0x226) [0x55d874a1e796] (on Tor 0.4.4.0-alpha-dev )
Apr 11 08:58:25.000 [warn] Bug: tor(+0xbe368) [0x55d874966368] (on Tor 0.4.4.0-alpha-dev )
Apr 11 08:58:25.000 [warn] Bug: tor(+0xbf1c5) [0x55d8749671c5] (on Tor 0.4.4.0-alpha-dev )
Apr 11 08:58:25.000 [warn] Bug: tor(circuit_receive_relay_cell+0x2b4) [0x55d874968a44] (on Tor 0.4.4.0-alpha-dev )
Apr 11 08:58:25.000 [warn] Bug: tor(command_process_cell+0x2c8) [0x55d87494b788] (on Tor 0.4.4.0-alpha-dev )
Apr 11 08:58:25.000 [warn] Bug: tor(channel_tls_handle_cell+0x4eb) [0x55d87492af0b] (on Tor 0.4.4.0-alpha-dev )
Apr 11 08:58:25.000 [warn] Bug: tor(+0xac5ca) [0x55d8749545ca] (on Tor 0.4.4.0-alpha-dev )
Apr 11 08:58:25.000 [warn] Bug: tor(connection_handle_read+0xa92) [0x55d8749183a2] (on Tor 0.4.4.0-alpha-dev )
Apr 11 08:58:25.000 [warn] Bug: tor(+0x75609) [0x55d87491d609] (on Tor 0.4.4.0-alpha-dev )
Apr 11 08:58:25.000 [warn] Bug: /lib/x86_64-linux-gnu/libevent-2.1.so.6(+0x229ba) [0x7f1984b569ba] (on Tor 0.4.4.0-alpha-dev )
Apr 11 08:58:25.000 [warn] Bug: /lib/x86_64-linux-gnu/libevent-2.1.so.6(event_base_loop+0x5a7) [0x7f1984b57537] (on Tor 0.4.4.0-alpha-dev )
Apr 11 08:58:25.000 [warn] Bug: tor(do_main_loop+0xff) [0x55d87491e88f] (on Tor 0.4.4.0-alpha-dev )
Apr 11 08:58:25.000 [warn] Bug: tor(tor_run_main+0x10b5) [0x55d87490bbc5] (on Tor 0.4.4.0-alpha-dev )
Apr 11 08:58:25.000 [warn] Bug: tor(tor_main+0x3a) [0x55d8749092ca] (on Tor 0.4.4.0-alpha-dev )
Apr 11 08:58:25.000 [warn] Bug: tor(main+0x19) [0x55d874908e89] (on Tor 0.4.4.0-alpha-dev )
Apr 11 08:58:25.000 [warn] Bug: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7f198443809b] (on Tor 0.4.4.0-alpha-dev )
Apr 11 08:58:25.000 [warn] Bug: tor(_start+0x2a) [0x55d874908eda] (on Tor 0.4.4.0-alpha-dev )
Apr 11 08:58:25.000 [warn] tor_bug_occurred_(): Bug: ../src/feature/hs/hs_client.c:981: close_or_reextend_intro_circ: Non-fatal assertion !(desc == NULL) failed. (on Tor 0.4.4.0-alpha-dev )
```Tor: 0.4.4.x-finalDavid Gouletdgoulet@torproject.orgDavid Gouletdgoulet@torproject.orghttps://gitlab.torproject.org/legacy/trac/-/issues/34086HSv3: Bug Non-fatal assertion in hs_client.c:776: client_rendezvous_circ_has_...2020-06-13T15:53:25Zs7rHSv3: Bug Non-fatal assertion in hs_client.c:776: client_rendezvous_circ_has_openedClient side HSv3 non fatal bug. It occurs like between ~50 - 70 times in a total of 100.000 built rendezvous circuits:
```
Apr 03 14:53:04.000 [warn] tor_bug_occurred_(): Bug: ../src/feature/hs/hs_client.c:776: client_rendezvous_circ_ha...Client side HSv3 non fatal bug. It occurs like between ~50 - 70 times in a total of 100.000 built rendezvous circuits:
```
Apr 03 14:53:04.000 [warn] tor_bug_occurred_(): Bug: ../src/feature/hs/hs_client.c:776: client_rendezvous_circ_has_opened: Non-fatal assertion !(!node_supports_v3_rendezvous_point(rp_node)) failed. (on Tor 0.4.4.0-alpha-dev )
Apr 03 14:53:04.000 [warn] Bug: Tor 0.4.4.0-alpha-dev: Non-fatal assertion !(!node_supports_v3_rendezvous_point(rp_node)) failed in client_rendezvous_circ_has_opened at ../src/feature/hs/hs_client.c:776. Stack trace: (on Tor 0.4.4.0-alpha-dev )
Apr 03 14:53:04.000 [warn] Bug: tor(log_backtrace_impl+0x56) [0x55d874ac7ee6] (on Tor 0.4.4.0-alpha-dev )
Apr 03 14:53:04.000 [warn] Bug: tor(tor_bug_occurred_+0x16c) [0x55d874ac30ec] (on Tor 0.4.4.0-alpha-dev )
Apr 03 14:53:04.000 [warn] Bug: tor(circuit_has_opened+0x80) [0x55d874947810] (on Tor 0.4.4.0-alpha-dev )
Apr 03 14:53:04.000 [warn] Bug: tor(circuit_send_next_onion_skin+0x2b8) [0x55d874930d38] (on Tor 0.4.4.0-alpha-dev )
Apr 03 14:53:04.000 [warn] Bug: tor(+0xbe4ba) [0x55d8749664ba] (on Tor 0.4.4.0-alpha-dev )
Apr 03 14:53:04.000 [warn] Bug: tor(+0xbf1c5) [0x55d8749671c5] (on Tor 0.4.4.0-alpha-dev )
Apr 03 14:53:04.000 [warn] Bug: tor(circuit_receive_relay_cell+0x2b4) [0x55d874968a44] (on Tor 0.4.4.0-alpha-dev )
Apr 03 14:53:04.000 [warn] Bug: tor(command_process_cell+0x2c8) [0x55d87494b788] (on Tor 0.4.4.0-alpha-dev )
Apr 03 14:53:04.000 [warn] Bug: tor(channel_tls_handle_cell+0x4eb) [0x55d87492af0b] (on Tor 0.4.4.0-alpha-dev )
Apr 03 14:53:04.000 [warn] Bug: tor(+0xac5ca) [0x55d8749545ca] (on Tor 0.4.4.0-alpha-dev )
Apr 03 14:53:04.000 [warn] Bug: tor(connection_handle_read+0xa92) [0x55d8749183a2] (on Tor 0.4.4.0-alpha-dev )
Apr 03 14:53:04.000 [warn] Bug: tor(+0x75609) [0x55d87491d609] (on Tor 0.4.4.0-alpha-dev )
Apr 03 14:53:04.000 [warn] Bug: /lib/x86_64-linux-gnu/libevent-2.1.so.6(+0x229ba) [0x7f1984b569ba] (on Tor 0.4.4.0-alpha-dev )
Apr 03 14:53:04.000 [warn] Bug: /lib/x86_64-linux-gnu/libevent-2.1.so.6(event_base_loop+0x5a7) [0x7f1984b57537] (on Tor 0.4.4.0-alpha-dev )
Apr 03 14:53:04.000 [warn] Bug: tor(do_main_loop+0xff) [0x55d87491e88f] (on Tor 0.4.4.0-alpha-dev )
Apr 03 14:53:04.000 [warn] Bug: tor(tor_run_main+0x10b5) [0x55d87490bbc5] (on Tor 0.4.4.0-alpha-dev )
Apr 03 14:53:04.000 [warn] Bug: tor(tor_main+0x3a) [0x55d8749092ca] (on Tor 0.4.4.0-alpha-dev )
Apr 03 14:53:04.000 [warn] Bug: tor(main+0x19) [0x55d874908e89] (on Tor 0.4.4.0-alpha-dev )
Apr 03 14:53:04.000 [warn] Bug: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7f198443809b] (on Tor 0.4.4.0-alpha-dev )
Apr 03 14:53:04.000 [warn] Bug: tor(_start+0x2a) [0x55d874908eda] (on Tor 0.4.4.0-alpha-dev )
```Tor: 0.4.4.x-finalDavid Gouletdgoulet@torproject.orgDavid Gouletdgoulet@torproject.orghttps://gitlab.torproject.org/legacy/trac/-/issues/34085HSv3: Bug Non-fatal assertion in hs_client.c:518: intro_circ_is_ok2020-06-13T15:53:24Zs7rHSv3: Bug Non-fatal assertion in hs_client.c:518: intro_circ_is_okClient side HSv3 non fatal bug. It occurs like between ~80-120 times in a total of 100.000 built rendezvous circuits:
```
Mar 31 18:44:17.000 [warn] tor_bug_occurred_(): Bug: ../src/feature/hs/hs_client.c:518: intro_circ_is_ok: Non-fata...Client side HSv3 non fatal bug. It occurs like between ~80-120 times in a total of 100.000 built rendezvous circuits:
```
Mar 31 18:44:17.000 [warn] tor_bug_occurred_(): Bug: ../src/feature/hs/hs_client.c:518: intro_circ_is_ok: Non-fatal assertion !(!hs_ident_intro_circ_is_valid(circ->hs_ident)) failed. (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: Tor 0.4.4.0-alpha-dev: Non-fatal assertion !(!hs_ident_intro_circ_is_valid(circ->hs_ident)) failed in intro_circ_is_ok at ../src/feature/hs/hs_client.c:518. Stack trace: (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(log_backtrace_impl+0x56) [0x55d874ac7ee6] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(tor_bug_occurred_+0x16c) [0x55d874ac30ec] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(hs_client_send_introduce1+0x271) [0x55d8749ce5e1] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(connection_ap_handshake_attach_circuit+0x3bd) [0x55d874949d5d] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(connection_ap_attach_pending+0x178) [0x55d87494e108] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(circuit_send_next_onion_skin+0x2b8) [0x55d874930d38] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(+0xbe4ba) [0x55d8749664ba] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(+0xbf1c5) [0x55d8749671c5] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(circuit_receive_relay_cell+0x2b4) [0x55d874968a44] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(command_process_cell+0x2c8) [0x55d87494b788] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(channel_tls_handle_cell+0x4eb) [0x55d87492af0b] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(+0xac5ca) [0x55d8749545ca] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(connection_handle_read+0xa92) [0x55d8749183a2] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(+0x75609) [0x55d87491d609] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: /lib/x86_64-linux-gnu/libevent-2.1.so.6(+0x229ba) [0x7f1984b569ba] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: /lib/x86_64-linux-gnu/libevent-2.1.so.6(event_base_loop+0x5a7) [0x7f1984b57537] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(do_main_loop+0xff) [0x55d87491e88f] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(tor_run_main+0x10b5) [0x55d87490bbc5] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(tor_main+0x3a) [0x55d8749092ca] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(main+0x19) [0x55d874908e89] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7f198443809b] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(_start+0x2a) [0x55d874908eda] (on Tor 0.4.4.0-alpha-dev )
```https://gitlab.torproject.org/legacy/trac/-/issues/34084HSv3: Bug at setup_intro_circ_auth_key at ../src/feature/hs/hs_client.c:7392020-06-13T15:53:24Zs7rHSv3: Bug at setup_intro_circ_auth_key at ../src/feature/hs/hs_client.c:739Client side HSv3 non fatal bug. It occurs like between ~80-120 times in a total of 100.000 built rendezvous circuits.
Seems like Tor is unable to find the right pubkey to assign to the introduction circuit and this causes a wave of asse...Client side HSv3 non fatal bug. It occurs like between ~80-120 times in a total of 100.000 built rendezvous circuits.
Seems like Tor is unable to find the right pubkey to assign to the introduction circuit and this causes a wave of asserts (also see dup #34085).
```
Mar 31 18:44:17.000 [warn] tor_bug_occurred_(): Bug: ../src/feature/hs/hs_client.c:739: setup_intro_circ_auth_key: This line should not have been reached. (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: Tor 0.4.4.0-alpha-dev: Line unexpectedly reached at setup_intro_circ_auth_key at ../src/feature/hs/hs_client.c:739. Stack trace: (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(log_backtrace_impl+0x56) [0x55d874ac7ee6] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(tor_bug_occurred_+0x16c) [0x55d874ac30ec] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(hs_client_circuit_has_opened+0x342) [0x55d8749ceb22] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(circuit_send_next_onion_skin+0x2b8) [0x55d874930d38] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(+0xbe4ba) [0x55d8749664ba] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(+0xbf1c5) [0x55d8749671c5] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(circuit_receive_relay_cell+0x2b4) [0x55d874968a44] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(command_process_cell+0x2c8) [0x55d87494b788] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(channel_tls_handle_cell+0x4eb) [0x55d87492af0b] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(+0xac5ca) [0x55d8749545ca] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(connection_handle_read+0xa92) [0x55d8749183a2] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(+0x75609) [0x55d87491d609] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: /lib/x86_64-linux-gnu/libevent-2.1.so.6(+0x229ba) [0x7f1984b569ba] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: /lib/x86_64-linux-gnu/libevent-2.1.so.6(event_base_loop+0x5a7) [0x7f1984b57537] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(do_main_loop+0xff) [0x55d87491e88f] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(tor_run_main+0x10b5) [0x55d87490bbc5] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(tor_main+0x3a) [0x55d8749092ca] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(main+0x19) [0x55d874908e89] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7f198443809b] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(_start+0x2a) [0x55d874908eda] (on Tor 0.4.4.0-alpha-dev )
```
This gets shortly followed by:
```
Mar 31 18:44:17.000 [warn] tor_bug_occurred_(): Bug: ../src/feature/hs/hs_client.c:518: intro_circ_is_ok: Non-fatal assertion !(!hs_ident_intro_circ_is_valid(circ->hs_ident)) failed. (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: Tor 0.4.4.0-alpha-dev: Non-fatal assertion !(!hs_ident_intro_circ_is_valid(circ->hs_ident)) failed in intro_circ_is_ok at ../src/feature/hs/hs_client.c:518. Stack trace: (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(log_backtrace_impl+0x56) [0x55d874ac7ee6] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(tor_bug_occurred_+0x16c) [0x55d874ac30ec] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(hs_client_send_introduce1+0x271) [0x55d8749ce5e1] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(connection_ap_handshake_attach_circuit+0x3bd) [0x55d874949d5d] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(connection_ap_attach_pending+0x178) [0x55d87494e108] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(circuit_send_next_onion_skin+0x2b8) [0x55d874930d38] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(+0xbe4ba) [0x55d8749664ba] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(+0xbf1c5) [0x55d8749671c5] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(circuit_receive_relay_cell+0x2b4) [0x55d874968a44] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(command_process_cell+0x2c8) [0x55d87494b788] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(channel_tls_handle_cell+0x4eb) [0x55d87492af0b] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(+0xac5ca) [0x55d8749545ca] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(connection_handle_read+0xa92) [0x55d8749183a2] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(+0x75609) [0x55d87491d609] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: /lib/x86_64-linux-gnu/libevent-2.1.so.6(+0x229ba) [0x7f1984b569ba] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: /lib/x86_64-linux-gnu/libevent-2.1.so.6(event_base_loop+0x5a7) [0x7f1984b57537] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(do_main_loop+0xff) [0x55d87491e88f] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(tor_run_main+0x10b5) [0x55d87490bbc5] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(tor_main+0x3a) [0x55d8749092ca] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(main+0x19) [0x55d874908e89] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7f198443809b] (on Tor 0.4.4.0-alpha-dev )
Mar 31 18:44:17.000 [warn] Bug: tor(_start+0x2a) [0x55d874908eda] (on Tor 0.4.4.0-alpha-dev )
```Tor: 0.4.4.x-finalGeorge KadianakisGeorge Kadianakishttps://gitlab.torproject.org/legacy/trac/-/issues/34083Client rendezvous circuit is no longer in circuit_wait but in pending_entry_c...2020-06-13T15:53:22Zs7rClient rendezvous circuit is no longer in circuit_wait but in pending_entry_connectionsWhen you are creating many rendezvous client circuits with a reasonable concurrency, you get tons of messages in the log file marked as bug like this:
```
May 01 08:55:43.000 [warn] connection_ap_attach_pending(): Bug: 0x55d877d94940 is...When you are creating many rendezvous client circuits with a reasonable concurrency, you get tons of messages in the log file marked as bug like this:
```
May 01 08:55:43.000 [warn] connection_ap_attach_pending(): Bug: 0x55d877d94940 is no longer in circuit_wait. Its current state is waiting for rendezvous desc. Why is it on pending_entry_connections? (on Tor 0.4.4.0-alpha-dev )
May 01 08:55:43.000 [warn] connection_ap_attach_pending(): Bug: 0x55d8789c16c0 is no longer in circuit_wait. Its current state is waiting for rendezvous desc. Why is it on pending_entry_connections? (on Tor 0.4.4.0-alpha-dev )
May 01 08:55:43.000 [warn] connection_ap_attach_pending(): Bug: 0x55d875eef5a0 is no longer in circuit_wait. Its current state is waiting for rendezvous desc. Why is it on pending_entry_connections? (on Tor 0.4.4.0-alpha-dev )
May 01 08:55:43.000 [warn] connection_ap_attach_pending(): Bug: 0x55d876063640 is no longer in circuit_wait. Its current state is waiting for rendezvous desc. Why is it on pending_entry_connections? (on Tor 0.4.4.0-alpha-dev )
May 01 08:55:43.000 [warn] connection_ap_attach_pending(): Bug: 0x55d877b92960 is no longer in circuit_wait. Its current state is waiting for rendezvous desc. Why is it on pending_entry_connections? (on Tor 0.4.4.0-alpha-dev )
May 01 08:55:43.000 [warn] connection_ap_attach_pending(): Bug: 0x55d8764ae550 is no longer in circuit_wait. Its current state is waiting for rendezvous desc. Why is it on pending_entry_connections? (on Tor 0.4.4.0-alpha-dev )
May 01 08:55:43.000 [warn] connection_ap_attach_pending(): Bug: 0x55d878a83f00 is no longer in circuit_wait. Its current state is waiting for rendezvous desc. Why is it on pending_entry_connections? (on Tor 0.4.4.0-alpha-dev )
May 01 08:55:43.000 [warn] connection_ap_attach_pending(): Bug: 0x55d877854530 is no longer in circuit_wait. Its current state is waiting for rendezvous desc. Why is it on pending_entry_connections? (on Tor 0.4.4.0-alpha-dev )
May 01 08:55:43.000 [warn] connection_ap_attach_pending(): Bug: 0x55d878cac3a0 is no longer in circuit_wait. Its current state is waiting for rendezvous desc. Why is it on pending_entry_connections? (on Tor 0.4.4.0-alpha-dev )
May 01 08:55:43.000 [warn] connection_ap_attach_pending(): Bug: 0x55d875b8d290 is no longer in circuit_wait. Its current state is waiting for rendezvous desc. Why is it on pending_entry_connections? (on Tor 0.4.4.0-alpha-dev )
May 01 08:55:43.000 [warn] connection_ap_attach_pending(): Bug: 0x55d8788a4d70 is no longer in circuit_wait. Its current state is waiting for rendezvous desc. Why is it on pending_entry_connections? (on Tor 0.4.4.0-alpha-dev )
May 01 08:55:43.000 [warn] connection_ap_attach_pending(): Bug: 0x55d878144a30 is no longer in circuit_wait. Its current state is waiting for rendezvous desc. Why is it on pending_entry_connections? (on Tor 0.4.4.0-alpha-dev )
May 01 08:55:43.000 [warn] connection_ap_attach_pending(): Bug: 0x55d877a2dc30 is no longer in circuit_wait. Its current state is waiting for rendezvous desc. Why is it on pending_entry_connections? (on Tor 0.4.4.0-alpha-dev )
```
No sense to send more lines since they are all the same, but just with different circuit ID. The number of such messages exceeds 1000 in a total say at least 100.000 built rendezvous circuits.Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/34082Master ticket for client side rendezvous circuit related bugs that cause reac...2020-06-13T15:53:22Zs7rMaster ticket for client side rendezvous circuit related bugs that cause reachability problems in HSv3 landThis is the master ticket for some reachability issues I discovered while stress testing my onionbalance v3 setup. They all occurred while handling HSv3 services.
At least two of them always occur together, but handling them as separate...This is the master ticket for some reachability issues I discovered while stress testing my onionbalance v3 setup. They all occurred while handling HSv3 services.
At least two of them always occur together, but handling them as separate tickets for now and keeping this master ticket to glue them together, since they all mention different stack traces.Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/33545assertion failure when "all zero" client auth key provided2020-06-13T15:53:04ZMark Smithassertion failure when "all zero" client auth key providedWhile doing some Tor Browser testing for Sponsor 27, I experienced the following after I intentionally used an incorrect client auth key for a v3 onion service:
```
... [err] tor_assertion_failed_: Bug: src/feature/hs/hs_descriptor.c:142...While doing some Tor Browser testing for Sponsor 27, I experienced the following after I intentionally used an incorrect client auth key for a v3 onion service:
```
... [err] tor_assertion_failed_: Bug: src/feature/hs/hs_descriptor.c:1423: decrypt_descriptor_cookie: Assertion !fast_mem_is_zero((char *) client_auth_sk, sizeof(*client_auth_sk)) failed; aborting. (on Tor 0.4.4.0-alpha-dev 1da0b05a5cace6ed)
```
As it turns out, I happened to enter a key that is consists entirely of zero bits. This is an unusual thing to do, but I do not think tor should exit.
Steps to reproduce in Tor Browser:
1. Try to load an http or https page for a v3 onion service that requires client authentication, e.g., dgoulet's test server.
2. Enter 56 'A's when prompted for a client auth key.
Result: tor exits due to the assertion failure. Behind the scenes, the browser installs the key via a control port command like the following:
```
onion_client_auth_add <onion-addr> x25519:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
```
and then tries to access the onion service again (page reload).Tor: 0.4.3.x-finalGeorge KadianakisGeorge Kadianakishttps://gitlab.torproject.org/legacy/trac/-/issues/33794hs-v3: OnionBalance backend instances all fail after more introductions by hi...2020-06-13T15:52:46Zs7rhs-v3: OnionBalance backend instances all fail after more introductions by hitting descriptor upload time limitsWe fixed #33762 and thought that was the problem (which it also was) but after fixing that I have discovered that all onionbalance backend instances continue to die after testing and testing (establishing successful RP circuits with the ...We fixed #33762 and thought that was the problem (which it also was) but after fixing that I have discovered that all onionbalance backend instances continue to die after testing and testing (establishing successful RP circuits with the frontend) and never heal unless Tor processes of the backends are restarted.
I was instructed by asn on IRC to look between the time of SIGHUP (logrotate log file change) and time of:
```
[warn] tor_bug_occurred_(): Bug: ../src/feature/hs/hs_circuit.c:987: get_subcredential_for_handling_intro2_cell: Non-fatal assertion !(!service->ob_subcreds) failed. (on Tor 0.4.4.0-alpha-dev )
```
for the message
```
log_info(LD_REND, "Hidden service %s next descriptor successfully "
"built. Now scheduled for upload.",
```
Which meant that it eventually recovered by itself. I can see two of those messages in ~24 hours, but the backend instances continue to be unavailable. Frontend does not work, and even if you try to connect to the backend address directly instead of the frontend one it does not work.
It looks like it is never able to upload further descriptors at some point, flooding the log file with:
```
[info] log_cant_upload_desc(): Service [scrubbed] can't upload its current descriptor: Next upload time is 1585751537, it is now 1585748071. [598 similar message(s) suppressed in last 600 seconds]
[info] log_cant_upload_desc(): Service [scrubbed] can't upload its next descriptor: Next upload time is 1585751987, it is now 1585748071. [598 similar message(s) suppressed in last 600 seconds]
[info] log_cant_upload_desc(): Service [scrubbed] can't upload its current descriptor: Next upload time is 1585751537, it is now 1585748671. [598 similar message(s) suppressed in last 600 seconds]
[info] log_cant_upload_desc(): Service [scrubbed] can't upload its next descriptor: Next upload time is 1585751987, it is now 1585748671. [598 similar message(s) suppressed in last 600 seconds]
[info] log_cant_upload_desc(): Service [scrubbed] can't upload its current descriptor: Next upload time is 1585751537, it is now 1585749271. [598 similar message(s) suppressed in last 600 seconds]
[info] log_cant_upload_desc(): Service [scrubbed] can't upload its next descriptor: Next upload time is 1585751987, it is now 1585749271. [598 similar message(s) suppressed in last 600 seconds]
[info] log_cant_upload_desc(): Service [scrubbed] can't upload its current descriptor: Next upload time is 1585751537, it is now 1585749871. [598 similar message(s) suppressed in last 600 seconds]
[info] log_cant_upload_desc(): Service [scrubbed] can't upload its next descriptor: Next upload time is 1585751987, it is now 1585749871. [598 similar message(s) suppressed in last 600 seconds]
```
I have info log level files from 2 backend instances of like ~47 MB each.Tor: 0.4.4.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/33762OnionService v3 running as onionbalance backend fails to reload failing with ...2020-06-13T15:52:39Zs7rOnionService v3 running as onionbalance backend fails to reload failing with get_subcredential_for_handling_intro2_cell: Non-fatal assertion !(!service->ob_subcreds) failed.When you do a **reload** on a Tor process that is running an onion service v3 as an onionbalance backend instance (with `HiddenServiceOnionBalanceInstance 1` in torrc), it fails to come up, making the backend onion service **unavailable*...When you do a **reload** on a Tor process that is running an onion service v3 as an onionbalance backend instance (with `HiddenServiceOnionBalanceInstance 1` in torrc), it fails to come up, making the backend onion service **unavailable** while not exiting the Tor process entirely. The only thing that makes it heal is a _restart_' of the process.
It can be triggered any time you **reload** (SIGHUP) a Tor process with an onion service running with `HiddenServiceOnionBalanceInstance 1`. This is Tor 0.4.4.0-alpha-dev-20200327T132009Z-1~d10.buster+1 from deb.tpo nightly master.
Here is the full stack trace:
```
Mar 29 15:59:12.000 [notice] Received reload signal (hup). Reloading config and resetting internal state.
Mar 29 15:59:12.000 [notice] Read configuration file "/usr/share/tor/tor-service-defaults-torrc".
Mar 29 15:59:12.000 [notice] Read configuration file "/etc/tor/torrc".
Mar 29 15:59:12.000 [info] config_generic_service(): HiddenServiceDir="/var/lib/tor/ob-hs". Configuring...
Mar 29 15:59:12.000 [info] config_generic_service(): HiddenServicePort=80 127.0.0.1:80 for "/var/lib/tor/ob-hs"
Mar 29 15:59:12.000 [info] tor_getpwnam(): Caching new entry debian-tor for debian-tor
Mar 29 15:59:12.000 [info] helper_parse_uint64(): HiddenServiceOnionBalanceInstance was parsed to 1
Mar 29 15:59:12.000 [info] ob_option_parse(): OnionBalance: MasterOnionAddress *******.onion registered
Mar 29 15:59:12.000 [notice] Tor 0.4.4.0-alpha-dev opening log file.
Mar 29 16:00:42.000 [warn] tor_bug_occurred_(): Bug: ../src/feature/hs/hs_circuit.c:987: get_subcredential_for_handling_intro2_cell: Non-fatal assertion !(!service->ob_subcreds) failed. (on Tor 0.4.4.0-alpha-dev )
Mar 29 16:00:42.000 [warn] Bug: Tor 0.4.4.0-alpha-dev: Non-fatal assertion !(!service->ob_subcreds) failed in get_subcredential_for_handling_intro2_cell at ../src/feature/hs/hs_circuit.c:987. Stack trace: (on Tor 0.4.4.0-alpha-dev )
Mar 29 16:00:42.000 [warn] Bug: /usr/bin/tor(log_backtrace_impl+0x56) [0x558eb518fee6] (on Tor 0.4.4.0-alpha-dev )
Mar 29 16:00:42.000 [warn] Bug: /usr/bin/tor(tor_bug_occurred_+0x16c) [0x558eb518b0ec] (on Tor 0.4.4.0-alpha-dev )
Mar 29 16:00:42.000 [warn] Bug: /usr/bin/tor(hs_circ_handle_introduce2+0x5eb) [0x558eb5091e3b] (on Tor 0.4.4.0-alpha-dev )
Mar 29 16:00:42.000 [warn] Bug: /usr/bin/tor(hs_service_receive_introduce2+0xc3) [0x558eb50a53d3] (on Tor 0.4.4.0-alpha-dev )
Mar 29 16:00:42.000 [warn] Bug: /usr/bin/tor(rend_process_relay_cell+0x1be) [0x558eb50e672e] (on Tor 0.4.4.0-alpha-dev )
Mar 29 16:00:42.000 [warn] Bug: /usr/bin/tor(+0xbe368) [0x558eb502e368] (on Tor 0.4.4.0-alpha-dev )
Mar 29 16:00:42.000 [warn] Bug: /usr/bin/tor(+0xbf1c5) [0x558eb502f1c5] (on Tor 0.4.4.0-alpha-dev )
Mar 29 16:00:42.000 [warn] Bug: /usr/bin/tor(circuit_receive_relay_cell+0x2b4) [0x558eb5030a44] (on Tor 0.4.4.0-alpha-dev )
Mar 29 16:00:42.000 [warn] Bug: /usr/bin/tor(command_process_cell+0x2c8) [0x558eb5013788] (on Tor 0.4.4.0-alpha-dev )
Mar 29 16:00:42.000 [warn] Bug: /usr/bin/tor(channel_tls_handle_cell+0x4eb) [0x558eb4ff2f0b] (on Tor 0.4.4.0-alpha-dev )
Mar 29 16:00:42.000 [warn] Bug: /usr/bin/tor(+0xac5ca) [0x558eb501c5ca] (on Tor 0.4.4.0-alpha-dev )
Mar 29 16:00:42.000 [warn] Bug: /usr/bin/tor(connection_handle_read+0xa92) [0x558eb4fe03a2] (on Tor 0.4.4.0-alpha-dev )
Mar 29 16:00:42.000 [warn] Bug: /usr/bin/tor(+0x75609) [0x558eb4fe5609] (on Tor 0.4.4.0-alpha-dev )
Mar 29 16:00:42.000 [warn] Bug: /lib/x86_64-linux-gnu/libevent-2.1.so.6(+0x229ba) [0x7f34833269ba] (on Tor 0.4.4.0-alpha-dev )
Mar 29 16:00:42.000 [warn] Bug: /lib/x86_64-linux-gnu/libevent-2.1.so.6(event_base_loop+0x5a7) [0x7f3483327537] (on Tor 0.4.4.0-alpha-dev )
Mar 29 16:00:42.000 [warn] Bug: /usr/bin/tor(do_main_loop+0xff) [0x558eb4fe688f] (on Tor 0.4.4.0-alpha-dev )
Mar 29 16:00:42.000 [warn] Bug: /usr/bin/tor(tor_run_main+0x10b5) [0x558eb4fd3bc5] (on Tor 0.4.4.0-alpha-dev )
Mar 29 16:00:42.000 [warn] Bug: /usr/bin/tor(tor_main+0x3a) [0x558eb4fd12ca] (on Tor 0.4.4.0-alpha-dev )
Mar 29 16:00:42.000 [warn] Bug: /usr/bin/tor(main+0x19) [0x558eb4fd0e89] (on Tor 0.4.4.0-alpha-dev )
Mar 29 16:00:42.000 [warn] Bug: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7f3482c0809b] (on Tor 0.4.4.0-alpha-dev )
Mar 29 16:00:42.000 [warn] Bug: /usr/bin/tor(_start+0x2a) [0x558eb4fd0eda] (on Tor 0.4.4.0-alpha-dev )
```
One instance has 26 such stack traces across a time period of 6 mins and 54 seconds and the other one 32 such stack traces across a time period of 4 mins and 28 seconds.
----
While we are at this, `config_generic_service()` , `tor_getpwnam()`, `helper_parse_uint64()` and `ob_option_parse()` report **[info]** level logs in a **Log notice file** configured Tor instance, is this normal?Tor: 0.4.4.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/33538v3bw files with too large of weights lead to relays being selected nearly uni...2020-06-13T15:52:00Zpastlyv3bw files with too large of weights lead to relays being selected nearly uniformly at randomAs part of working on the FlashFlow paper (currently under submission) we ran Shadow simulations and compared it to TorFlow. Not surprising.
Summary of the Shadow network used:
- 5% of the real Tor network in size
- 44 exits
- ...As part of working on the FlashFlow paper (currently under submission) we ran Shadow simulations and compared it to TorFlow. Not surprising.
Summary of the Shadow network used:
- 5% of the real Tor network in size
- 44 exits
- 104 guards
- 180 middles
- 3 auths
- 10 "markov" clients. It's not terribly important to know what they're doing, other than knowing they're making lots of 3-hop exit circuits and exchanging traffic with servers. 2 of the clients have tor debug logs. All 10 contribute to the relay selection data.
- Tor version used is a63b4148229ae8ce46494fd6a0f99149c231605c (master branch as of March 5th, 2020) plus a small logging patch. [Branch here](https://github.com/pastly/public-tor/tree/log-relay-weights). This existed in 0.3.5.7 as well. I don't know when this problem started because I don't know exactly what the problem is.
- Shadow 292cd89ba52fc2972fdd9d2e27e384db9601663b (as of Jan 10th, 2020).
- Shadow-plugin-tor 8deab15a032f5173ba7c12ad6dd0bcb1cb0c3463 (as of Oct 2019) plus patch so it works with new Tors. [Branch here](https://github.com/pastly/shadow-plugin-tor/tree/three-stubs).
The only difference in the simulations are the v3bw files used.
There are three simulations:
1. Torflow-derived weights (TF)
1. FlashFlow-derived weights (FF init)
1. FlashFlow-derived weights that have all been divided by 136 (FF scaled)
weight-dist.pdf shows the distribution of the weights in the v3bw files, both with the raw absolute weights and as normalized (norm_weight = weight / total_weight). Despite having nearly identical normalized weight distributions (note: FF init and FF scaled are obviously identical), FF init results in (1) relays being selected seemingly uniformly at random, and (2) significantly worse performance as a consequence.
selection-v-weight.pdf shows how often the 10 markov clients picked each relay. Focus on the scatter plots. Notice how in TF and FF scaled there is basically a 1:1 linear relationship between additional weight and selection frequency, while in FF init the selection frequency is roughly the same regardless of the relay's weight.
I am also attaching the three v3bw files, combined into one file to reduce email spam.
I am also attaching small snippits from the debug logs (again: combined into one file) of one of the markov clients. The snippits show some of the relay weights the client is using when deciding which relays to use. You can see in the FF initial one that the weights are much more similar than in FF scaled and TF.Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/33511Unknown option 'HiddenServiceOnionBalanceInstance'2020-06-13T15:51:56ZTracUnknown option 'HiddenServiceOnionBalanceInstance'I did git clone master branch which was merged with ticket32709_044_02 already, but I do receive this error when start tor:
```
Tor 0.4.4.0-alpha-dev (git-9892cc3b12db4dc1) running on Linux with Libevent 2.0.21-stable, OpenSSL 1.0.2k-fip...I did git clone master branch which was merged with ticket32709_044_02 already, but I do receive this error when start tor:
```
Tor 0.4.4.0-alpha-dev (git-9892cc3b12db4dc1) running on Linux with Libevent 2.0.21-stable, OpenSSL 1.0.2k-fips, Zlib 1.2.7, Liblzma 5.2.2, and Libzstd N/A.
Mar 03 08:06:55.980 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Mar 03 08:06:55.980 [notice] This version is not a stable Tor release. Expect more bugs than usual.
Mar 03 08:06:55.980 [notice] Read configuration file "/etc/tor/torrc".
Mar 03 08:06:55.982 [warn] Failed to parse/validate config: Unknown option 'HiddenServiceOnionBalanceInstance'. Failing.
**Trac**:
**Username**: roman.kofevarka@gmail.comTor: 0.4.4.x-final