Trac issueshttps://gitlab.torproject.org/legacy/trac/-/issues2020-06-16T11:27:52Zhttps://gitlab.torproject.org/legacy/trac/-/issues/32040HS intro padding machine reactivates after receiving INTRODUCE_ACK2020-06-16T11:27:52ZGeorge KadianakisHS intro padding machine reactivates after receiving INTRODUCE_ACKWhile reseaching wtf-pad I noticed that the intro machines display a weird shutdown/reactivation pattern.
In particular, what happens is that after INTRODUCE1 is sent, the machine starts up and sends all its padding, and then shuts down...While reseaching wtf-pad I noticed that the intro machines display a weird shutdown/reactivation pattern.
In particular, what happens is that after INTRODUCE1 is sent, the machine starts up and sends all its padding, and then shuts down. Then when INTRODUCE_ACK arrives, it reactivates (probably because INTRODUCE_ACK is part of the accepted purposes and the machine is shutdown/freed at that time) and sends again padding, then again shuts down...
This does not work as intended and causes extra cells to fly in with a pattern that probably does not help them blend in.
Here are some Tor logs (with padanalysis incoming/outgoing cells logs):
```
Oct 11 13:25:54.000 [warn] outgoing-cell: EXTEND2 0
Oct 11 13:25:54.000 [warn] incoming-cell: EXTENDED2 0 66
Oct 11 13:25:54.000 [warn] outgoing-cell: EXTEND2 0
Oct 11 13:25:54.000 [warn] incoming-cell: EXTENDED2 0 66
Oct 11 13:25:57.000 [warn] outgoing-cell: EXTEND 0
Oct 11 13:25:58.000 [warn] incoming-cell: EXTENDED 0 148
Oct 11 13:25:58.000 [warn] outgoing-cell: INTRODUCE1 4
Oct 11 13:25:58.000 [warn] outgoing-cell: PADDING_NEGOTIATE 4
Oct 11 13:25:58.000 [warn] incoming-cell: PADDING_NEGOTIATED 4 4
Oct 11 13:25:58.000 [warn] incoming-cell: DROP 4 0
Oct 11 13:25:58.000 [warn] incoming-cell: DROP 4 0
Oct 11 13:25:58.000 [warn] incoming-cell: DROP 4 0
Oct 11 13:25:58.000 [warn] incoming-cell: DROP 4 0
Oct 11 13:25:58.000 [warn] incoming-cell: DROP 4 0
Oct 11 13:25:58.000 [warn] incoming-cell: DROP 4 0
Oct 11 13:25:58.000 [warn] incoming-cell: DROP 4 0
Oct 11 13:25:58.000 [warn] incoming-cell: DROP 4 0
Oct 11 13:25:58.000 [warn] incoming-cell: DROP 4 0
Oct 11 13:25:58.000 [warn] incoming-cell: PADDING_NEGOTIATED 4 4
Oct 11 13:25:58.000 [info] circpad_handle_padding_negotiated(): Received STOP command on PADDING_NEGOTIATED for circuit 5
Oct 11 13:25:58.000 [info] circpad_circuit_machineinfo_free_idx(): Freeing padding info idx 0 on circuit 5 (7)
Oct 11 13:25:58.000 [warn] incoming-cell: INTRODUCE_ACK 4 0
Oct 11 13:25:58.000 [info] rend_client_introduction_acked(): Received ack. Telling rend circ...
Oct 11 13:25:58.000 [info] circpad_setup_machine_on_circ(): Registering machine client_ip_circ to origin circ 5 (8)
Oct 11 13:25:58.000 [info] circpad_node_supports_padding(): Checking padding: supported
Oct 11 13:25:58.000 [info] circpad_negotiate_padding(): Negotiating padding on circuit 5 (8), command 2
Oct 11 13:25:58.000 [warn] outgoing-cell: 8 PADDING_NEGOTIATE 4
Oct 11 13:25:58.000 [info] circpad_machine_spec_transition(): Circuit 5 circpad machine 0 transitioning from 0 to 1
Oct 11 13:25:58.000 [info] circpad_marked_circuit_for_padding(): Circuit 5 is not marked for close because of a pending padding machine in index 0.
Oct 11 13:25:58.000 [warn] incoming-cell: PADDING_NEGOTIATED 4 4
Oct 11 13:25:58.000 [warn] incoming-cell: DROP 4 0
Oct 11 13:25:58.000 [warn] incoming-cell: DROP 4 0
Oct 11 13:25:58.000 [warn] incoming-cell: DROP 4 0
Oct 11 13:25:58.000 [warn] incoming-cell: DROP 4 0
Oct 11 13:25:58.000 [warn] incoming-cell: DROP 4 0
Oct 11 13:25:58.000 [warn] incoming-cell: DROP 4 0
Oct 11 13:25:58.000 [warn] incoming-cell: DROP 4 0
Oct 11 13:25:58.000 [warn] incoming-cell: DROP 4 0
Oct 11 13:25:58.000 [warn] incoming-cell: PADDING_NEGOTIATED 4 4
Oct 11 13:25:58.000 [info] circpad_handle_padding_negotiated(): Received STOP command on PADDING_NEGOTIATED for circuit 5
Oct 11 13:25:58.000 [info] circpad_circuit_machineinfo_free_idx(): Freeing padding info idx 0 on circuit 5 (15)
...
Oct 11 13:36:11.000 [info] circuit_mark_for_close_(): Circuit 4130340667 (id: 5) marked for close at src/core/or/circuituse.c:1509 (orig reason: 9, new reason: 0)
```
I'm not sure what the fix should be here. It might be that we need to remove INTRODUCE_ACK from the list of accepted purposes, but if we do that then if INTRODUCE_ACK arrives earlier we will stop padding after. Hmm.Tor: 0.4.4.x-finalGeorge KadianakisGeorge Kadianakishttps://gitlab.torproject.org/legacy/trac/-/issues/31806Update padding-spec.txt to cover hs circuit setup obfuscation2020-06-13T15:45:49ZMike PerryUpdate padding-spec.txt to cover hs circuit setup obfuscationhttps://github.com/mikeperry-tor/torspec/tree/hs-padding-spec is the updated padding spec with the necessary changes. Current HEAD to merge is 479f0dc4bd2a873ab153ed0e7d7066e975069929https://github.com/mikeperry-tor/torspec/tree/hs-padding-spec is the updated padding spec with the necessary changes. Current HEAD to merge is 479f0dc4bd2a873ab153ed0e7d7066e975069929Tor: 0.4.2.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/30173Ensure circuit padding can be safely disabled from consensus2020-06-13T15:40:42ZMike PerryEnsure circuit padding can be safely disabled from consensusRight now, to "disable" circuit padding, we have to set consensus parameters circpad_global_allowed_cells=1 and circpad_global_max_padding_pct=1, which actually only sets a limit of 1% of our traffic to be padding. Additionally, each pad...Right now, to "disable" circuit padding, we have to set consensus parameters circpad_global_allowed_cells=1 and circpad_global_max_padding_pct=1, which actually only sets a limit of 1% of our traffic to be padding. Additionally, each padding cell that is *not* sent because this limit is reached causes an info log line to be written.
We need another way to fully disable padding, for A/B performance testing. And we should rate limit that info log so relays logging at info don't print a line for every single padding cell they don't send.Tor: 0.4.1.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/30172Always send PADDING_NEGOTIATE if middle supports it2022-06-24T16:07:31ZMike PerryAlways send PADDING_NEGOTIATE if middle supports itWe should define some kind of NULL machine for whatever hop is most common in our padding machine list, and negotiate that machine if no other machines apply to the current circuit. This machine shouldn't take up a slot or count as negot...We should define some kind of NULL machine for whatever hop is most common in our padding machine list, and negotiate that machine if no other machines apply to the current circuit. This machine shouldn't take up a slot or count as negotiated, tho, so we can still negotiate other machines at later points if the circuit purpose changes, etc.
Similarly, this NULL machine should (maybe) set should_negotiate_end and send a PADDING_NEGOTIATE at circuit close.
We need to do this so that there isn't an obvious PADDING_NEGOTIATE cell request/response pair with obvious timings that it went to the middle node (since the PADDING_NEGOTIATED response will come faster than all other responses on the circuit). See also #30092 for similar motivating reasoning.Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/30092Add a probability-to-apply field for circuitpadidng machines2022-06-16T18:10:05ZMike PerryAdd a probability-to-apply field for circuitpadidng machinesIn #28634, we realized that we may want to make some fraction of pre-built GENERAL and HS_VANGUARDS circuits look like padded onion service circuits, as a defense in depth against a classifier that can still recognize our specially padde...In #28634, we realized that we may want to make some fraction of pre-built GENERAL and HS_VANGUARDS circuits look like padded onion service circuits, as a defense in depth against a classifier that can still recognize our specially padded onion service circuits as, well, special, and still interesting.
But we don't want to make all general circuits look this way. Just some fraction. So it would be nice if the machine conditions could somehow toss a coin to decide to apply the machine to a circuit. Unfortunately, right now the conditions are memoryless, so we have nothing that can say "you already tossed the coin", but we could special case just this to have a flag on the circuit or something.Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/29990Handle zero monotonic time differences in the circuit padding code2020-06-13T15:40:07ZteorHandle zero monotonic time differences in the circuit padding codeI can reliably get the following failure on my macOS VM when its wall clock time is out of sync with the host time.
Failures like this also intermittently happen when the underlying API is low-resolution, or not actually monotonic (for ...I can reliably get the following failure on my macOS VM when its wall clock time is out of sync with the host time.
Failures like this also intermittently happen when the underlying API is low-resolution, or not actually monotonic (for example, Windows).
```
circuitpadding/circuitpadding_circuitsetup_machine: [forking]
FAIL ../src/test/test_circuitpadding.c:1900: assert(client_side->padding_info[0]->padding_scheduled_at_usec OP_NE 0): 0 vs 0
[circuitpadding_circuitsetup_machine FAILED]
```
I've worked around the issue in #29500 by weakening the test conditions.
But we need to fix the circuitpadding code so it handles zero monotonic time differences correctly.Tor: 0.4.1.x-finalMike PerryMike Perryhttps://gitlab.torproject.org/legacy/trac/-/issues/29840WTF-PAD: Log padding counters into the heartbeat2020-06-13T15:39:39ZGeorge KadianakisWTF-PAD: Log padding counters into the heartbeatWe should tie the padding counters in circuitpadding to the heartbeat so that we can see (at least on the relays) how much padding we are sending/receiving.We should tie the padding counters in circuitpadding to the heartbeat so that we can see (at least on the relays) how much padding we are sending/receiving.Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/29821Make circuit/channel padding correctly respect dormant mode2020-06-13T15:39:35ZGeorge KadianakisMake circuit/channel padding correctly respect dormant modeAs part of #28636, we are doing a minimal version of dormant mode in circpadding, which does not schedule padding if dormant mode is enabled.
However, that's only part of the work since already existing timers won't stop counting.
It a...As part of #28636, we are doing a minimal version of dormant mode in circpadding, which does not schedule padding if dormant mode is enabled.
However, that's only part of the work since already existing timers won't stop counting.
It also seems like channel padding does not respect dormant mode either.Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/29593channelpadding_timers test failed on tor-0.3.5.82020-06-13T15:38:37ZGeorge Kadianakischannelpadding_timers test failed on tor-0.3.5.8weasel reported the following test failure today:
```
channelpadding/channelpadding_timers: [forking] Feb 22 09:58:55.345 [err] tor_assertion_failed_(): Bug: ../src/test/testing_rsakeys.c:540:
init_pregenerated_keys: Assertion r == 0 fa...weasel reported the following test failure today:
```
channelpadding/channelpadding_timers: [forking] Feb 22 09:58:55.345 [err] tor_assertion_failed_(): Bug: ../src/test/testing_rsakeys.c:540:
init_pregenerated_keys: Assertion r == 0 failed; aborting. (on Tor 0.3.5.8 )
Feb 22 09:58:55.347 [err] Bug: Assertion r == 0 failed in init_pregenerated_keys at ../src/test/testing_rsakeys.c:540. Stack trace: (on Tor 0.3.5.8 )
Feb 22 09:58:55.348 [err] Bug: ./src/test/test(log_backtrace_impl+0x84) [0x569abc28] (on Tor 0.3.5.8 )
Feb 22 09:58:55.348 [err] Bug: ./src/test/test(tor_assertion_failed_+0xc4) [0x569a67b8] (on Tor 0.3.5.8 )
Feb 22 09:58:55.348 [err] Bug: ./src/test/test(init_pregenerated_keys+0x198) [0x5678f770] (on Tor 0.3.5.8 )
Feb 22 09:58:55.349 [err] Bug: ./src/test/test(testcase_run_one+0x284) [0x5678fb14] (on Tor 0.3.5.8 )
Feb 22 09:58:55.349 [err] Bug: ./src/test/test(tinytest_main+0x188) [0x56790490] (on Tor 0.3.5.8 )
Feb 22 09:58:55.349 [err] Bug: ./src/test/test(main+0x43c) [0x563dca5c] (on Tor 0.3.5.8 )
Feb 22 09:58:55.349 [err] Bug: /lib/mips-linux-gnu/libc.so.6(__libc_start_main+0x108) [0x76d422b8] (on Tor 0.3.5.8 )
Feb 22 09:58:55.350 [err] Bug: ./src/test/test(+0x5fbd4) [0x563dcbd4] (on Tor 0.3.5.8 )
[Lost connection!]
[channelpadding_timers FAILED]
channeltls/create: [forking] OK
```
from https://buildd.debian.org/status/fetch.php?pkg=tor&arch=mips&ver=0.3.5.8-1&stamp=1550830462&raw=0
Apparently, if this is not dealt with, it'll mean 0.3.5.8 will not ship with the next DebianTor: 0.4.0.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/legacy/trac/-/issues/29500Broken circuitpadding unittests on appveyor2020-06-13T15:40:08ZGeorge KadianakisBroken circuitpadding unittests on appveyorThis x86 appveyor build failed two unittests: https://ci.appveyor.com/project/torproject/tor/builds/22350196/job/457arqlgfgf6b0a3
```
circuitpadding/circuitpadding_tokens: [forking]
FAIL ../src/test/test_circuitpadding.c:1125: assert...This x86 appveyor build failed two unittests: https://ci.appveyor.com/project/torproject/tor/builds/22350196/job/457arqlgfgf6b0a3
```
circuitpadding/circuitpadding_tokens: [forking]
FAIL ../src/test/test_circuitpadding.c:1125: assert(mi->histogram[4] OP_EQ 2): 0 vs 2
[circuitpadding_tokens FAILED]
circuitpadding/circuitpadding_rtt: [forking]
FAIL ../src/test/test_circuitpadding.c:324: assert(relay_side->padding_info[0]->last_received_time_usec OP_NE 0): 0 vs 0
[circuitpadding_rtt FAILED]
```Tor: 0.4.0.x-finalMike PerryMike Perryhttps://gitlab.torproject.org/legacy/trac/-/issues/29231Relays vastly underreport write-total in padding-counts line in extrainfo des...2020-06-13T15:37:36ZRoger DingledineRelays vastly underreport write-total in padding-counts line in extrainfo descriptorHere is a snippet from WhatsGoingOn's extrainfo descriptor:
```
write-history 2019-01-27 18:10:06 (86400 s) 7709989888,6911394816,6582383616,7393989632,6065009664
read-history 2019-01-27 18:10:06 (86400 s) 7644498944,6824735744,655076147...Here is a snippet from WhatsGoingOn's extrainfo descriptor:
```
write-history 2019-01-27 18:10:06 (86400 s) 7709989888,6911394816,6582383616,7393989632,6065009664
read-history 2019-01-27 18:10:06 (86400 s) 7644498944,6824735744,6550761472,7355173888,6059125760
[...]
padding-counts 2019-01-27 22:06:23 (86400 s) bin-size=10000 write-drop=0 write-pad=220000 write-total=250000 read-drop=0 read-pad=210000 read-total=12310000 enabled-read-pad=10000 enabled-read-total=20000 enabled-write-pad=10000 enabled-write-total=10000 max-chanpad-timers=6
```
Notice how the read-total is reasonable compared to read-history, but the write-total is way undercounting compared to write-history.
I suspect that write-total is the wrong one.Tor: 0.4.1.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/29085WTF-PAD: Reduce monotime usage because of performance issues2020-06-13T15:36:49ZGeorge KadianakisWTF-PAD: Reduce monotime usage because of performance issuesThere are various concerns about the monotime usage of circpadding. We should look into this before enabling any machines here.
See https://github.com/torproject/tor/pull/624#discussion_r246443113
and https://trac.torproject.org/project...There are various concerns about the monotime usage of circpadding. We should look into this before enabling any machines here.
See https://github.com/torproject/tor/pull/624#discussion_r246443113
and https://trac.torproject.org/projects/tor/ticket/28142#comment:32Tor: 0.4.1.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/29084Ensure circuit padding RTT estimate handes var cells/wide creates2020-06-13T15:36:48ZGeorge KadianakisEnsure circuit padding RTT estimate handes var cells/wide createsThe use_rtt_estimate field in the circuit padding machines lets machines offset the inter-packet delays by a middle-node estimated RTT value of packets that go from the middle to the exit/website.
We abort this measurement if we get two...The use_rtt_estimate field in the circuit padding machines lets machines offset the inter-packet delays by a middle-node estimated RTT value of packets that go from the middle to the exit/website.
We abort this measurement if we get two or more cells back-to-back in either direction, as this indicates that the half-duplex request/response circuit setup and RELAY_BEGIN sequence has finished.
However, if we switch to a multi-cell circuit handshake, then we will need to take that into account when measuring RTT.
If RELAY_EARLY is used only for the first cell of a multi-cell EXTEND2 payload,
then we can just count time between RELAY_EARLIES. But the proposal currently says MAY, but not MUST for this behavior.Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/29083WTF-PAD: Specify exit policy for machine conditions2020-06-13T15:36:48ZGeorge KadianakisWTF-PAD: Specify exit policy for machine conditionsFrom the TODO file:
```
- Specify exit policy for machine conditions?
- short_policy_t looks good, except for its flexible array member :/
- Can we make our own struct with a small, fixed number of policy
entries? Say...From the TODO file:
```
- Specify exit policy for machine conditions?
- short_policy_t looks good, except for its flexible array member :/
- Can we make our own struct with a small, fixed number of policy
entries? Say 3-4? Or is that a bad idea to lose this flexibility?
- Check conditions based on attached streams on the circuit
- Accept should mean "only apply if matched"
- Reject should mean "don't apply if matched"
- If a policy is specified, Reject *:* is implicit default (so reject
policies need an Accept entry).
- With no policy, Accept *:* is implicit default.
```Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28878WTF-PAD: Improve deterministic randomness in tests2020-06-13T15:35:46ZGeorge KadianakisWTF-PAD: Improve deterministic randomness in testsAs part of test_prob_distr.c in #28142, we implemented a deterministic randomness system for the stochastic tests. There are various things we can improve there:
a) Make it into its own subsystem so that other tests can also use it.
b) ...As part of test_prob_distr.c in #28142, we implemented a deterministic randomness system for the stochastic tests. There are various things we can improve there:
a) Make it into its own subsystem so that other tests can also use it.
b) Don't use a uint32_t counter as the seed, because the byte order is machine dependent. Instead make it a `uint8_t[4]` or something.
c) Allow users to overwrite the seed using environment variables or the CLI. For now users have to tweak the `init_deterministic_rand()` func.Tor: 0.4.1.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/legacy/trac/-/issues/28821Introduce timer_is_scheduled() method and replace padding_scheduled_at_us2020-06-13T15:35:32ZGeorge KadianakisIntroduce timer_is_scheduled() method and replace padding_scheduled_at_usThis came up during #28142 review: https://github.com/torproject/tor/pull/461#discussion_r231889687
When #28142 gets merged, we can do a small refactoring by removing `padding_scheduled_at_us` and replacing it with a new timer method th...This came up during #28142 review: https://github.com/torproject/tor/pull/461#discussion_r231889687
When #28142 gets merged, we can do a small refactoring by removing `padding_scheduled_at_us` and replacing it with a new timer method that returns true if the timer is scheduled, and false if it's disabled.Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28804Add circuit padding to padding-spec.txt and write a doc for researchers2020-06-13T15:35:28ZGeorge KadianakisAdd circuit padding to padding-spec.txt and write a doc for researchersThere are a few things that can be done to the spec to improve them.
For the basics, we can improve the sectioning and also add some indentation to make it more clear how sections are structured (for example, to make it clear the distin...There are a few things that can be done to the spec to improve them.
For the basics, we can improve the sectioning and also add some indentation to make it more clear how sections are structured (for example, to make it clear the distinctions between histograms and prob distrs).Tor: unspecifiedMike PerryMike Perryhttps://gitlab.torproject.org/legacy/trac/-/issues/28780circpadding: Add machine flag for not closing circuit if machine is active2020-06-13T15:42:10ZGeorge Kadianakiscircpadding: Add machine flag for not closing circuit if machine is activeSee https://github.com/torproject/tor/pull/461#discussion_r231546320 for more details .
Related to #28634.See https://github.com/torproject/tor/pull/461#discussion_r231546320 for more details .
Related to #28634.Tor: 0.4.1.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/28779Remove double function declarations in circuitpadding.c/test_circuitpadding.c2020-06-13T15:35:22ZGeorge KadianakisRemove double function declarations in circuitpadding.c/test_circuitpadding.cWe can use the STATIC macro to avoid defining every function twice.
We can also rearrange the circuitpadding.c functions so that we don't need to define them on top of the .c file.We can use the STATIC macro to avoid defining every function twice.
We can also rearrange the circuitpadding.c functions so that we don't need to define them on top of the .c file.Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28778Rename some circuitpadding structs2020-06-13T15:35:21ZGeorge KadianakisRename some circuitpadding structsSee https://github.com/torproject/tor/pull/461#discussion_r231885343 and https://github.com/torproject/tor/pull/461#discussion_r232343640 for a discussion to rename some of the essential structs of circuitpadding to better reflect their ...See https://github.com/torproject/tor/pull/461#discussion_r231885343 and https://github.com/torproject/tor/pull/461#discussion_r232343640 for a discussion to rename some of the essential structs of circuitpadding to better reflect their semantics.Tor: unspecified