Tor incorrectly stores stats on incoming PT connections

changed milestone to %Tor: 0.4.8.x-post-stable

added Bug For Anticensorship Team Metrics labels

my patch is currently in trinity-1686a/tor@43490ca7 if anybody else wants to deploy it.

I'll make a merge request of it once I've seen it does indeed fix the issue. It's currently deployed on 2BFC64E5E97508554BFAF52B03C5044385861209, which has a firewalled ORPort, and enough clients that I should be able to see the result once the stats gets updated

Would you be up for running your Tor instance with a patch that potentially fixes this issue, @dcf ?

We're not set up for conveniently running from source. I'll wait for @trinity-1686a's test results.

mentioned in commit tpo/network-health/metrics/timeline@6820270b

That bridge now has zero <OR> connections
before:

bridge-stats-end 2023-10-09 17:26:36 (86400 s)
bridge-ips ru=528,us=64,de=16,eg=16,ir=16,nl=16,sg=16,ae=8,am=8,ap=8,bg=8,by=8,ch=8,cy=8,es=8,fr=8,gb=8,hk=8,it=8,jp=8,mx=8,my=8,no=8,nz=8,pe=8,pl=8,pr=8,sa=8,tr=8,tw=8,ua=8
bridge-ip-versions v4=720,v6=0
bridge-ip-transports <OR>=376,obfs4=344

after:

bridge-stats-end 2023-10-11 19:34:58 (86400 s)
bridge-ips ru=248,us=40,ae=8,am=8,by=8,ca=8,cy=8,de=8,ee=8,eg=8,es=8,fr=8,gb=8,gr=8,hk=8,ir=8,is=8,it=8,jp=8,kr=8,kz=8,lt=8,lu=8,ma=8,md=8,mx=8,my=8,nl=8,no=8,pr=8,pt=8,sa=8,tr=8,tw=8,za=8
bridge-ip-versions v4=352,v6=0
bridge-ip-transports obfs4=352

I'm not sure why the overall number of user got divided by 2, my best guess is restarting the bridge made it temporarily less popular

Reducing the number of unique IP addresses by half is what I would expect. Currently every unique IP address is being double-counted; with the fix each one is only counted once. I'm a little surprised that it extends even to bridge-ips and bridge-ip-versions, but I can imagine how that might happen.

Note that user count estimates come from counting directory requests, not unique IP addresses. bridge-ip-transports doesn't influence the estimate of the number of users. The total number of users, rather, comes from the "ok" part of dirreq-v3-resp. If you look at the before-and-after of dirreq-v3-resp, you will probably not see a 50% change. The only effect of bridge-ip-transports is deciding what fractions of the total number of users should be attributed to each transport. Only the ratio of transports in bridge-ip-transports matters, not the absolute numbers.

https://metrics.torproject.org/reproducible-metrics.html#bridge-users

Parse successful requests from the "ok" part of the "dirreq-v3-resp" line, subtract 4 to undo the binning operation that has been applied by the bridge, and discard the resulting number if it's zero or negative. ... Bridges do not report directory requests by transport or IP version. We approximate these numbers by multiplying the total number of requests with the fraction of unique IP addresses by transport or IP version.

Someone who knows the code better than me should confirm, but I don't think there is a double-counting in either case. Before my patch, we would count a connection either early (in channeltls.c), and with the wrong transport, or late (in channel.c), and with the right transport, but not both. And with the patch, we count with the right transport both on the early and late codepath.

If you look at the before-and-after of dirreq-v3-resp, you will probably not see a 50% change.

before:
dirreq-v3-resp ok=528,not-enough-sigs=0,unavailable=0,not-found=0,not-modified=0,busy=0
after:
dirreq-v3-resp ok=504,not-enough-sigs=0,unavailable=0,not-found=0,not-modified=0,busy=0

looks about right

I don't know, it looks like double counting in bridge-ip-transports to me: once with the proper transport, once with a null transport. Otherwise why would the split be so close to 50/50? "With the patch, we count with the right transport both on the early and late codepath": that makes sense, because bridge-ip-tranports counts unique IP addresses (not connections); the same IP address seen twice with the same transport would be deduplicated and only count as one.

Here's the history of one of the tor instances on the snowflake-01 bridge before and after the upgrade at 2023-10-03 23:00. See how the snowflake count remains steady, while <OR> rises up to become approximately equal to it:

published	<OR>	snowflake
2023-10-01 13:36:42	4	24996
2023-10-02 14:36:43	4	25276
2023-10-03 13:36:43	4	25260
2023-10-04 15:24:09	4	25692
2023-10-06 05:09:20	28092	27316
2023-10-07 05:17:28	26732	25972
2023-10-08 05:27:34	27324	26572
2023-10-09 04:04:42	28284	27500

I got these numbers from the intermediate data file (not versioned) transport_ips/bridge-extra-infos-2023-10.transport_ips.csv in snowflake-graphs.

published,fingerprint,begin,end,nickname,transport,ips
2023-10-01 13:36:42,5481936581E23D2D178105D44DB6915AB06BFB7F,2023-09-29 15:36:07,2023-09-30 15:36:07,flakey1,<OR>,4
2023-10-01 13:36:42,5481936581E23D2D178105D44DB6915AB06BFB7F,2023-09-29 15:36:07,2023-09-30 15:36:07,flakey1,snowflake,24996
2023-10-02 14:36:43,5481936581E23D2D178105D44DB6915AB06BFB7F,2023-09-30 15:36:07,2023-10-01 15:36:07,flakey1,<OR>,4
2023-10-02 14:36:43,5481936581E23D2D178105D44DB6915AB06BFB7F,2023-09-30 15:36:07,2023-10-01 15:36:07,flakey1,snowflake,25276
2023-10-03 13:36:43,5481936581E23D2D178105D44DB6915AB06BFB7F,2023-10-01 15:36:07,2023-10-02 15:36:07,flakey1,<OR>,4
2023-10-03 13:36:43,5481936581E23D2D178105D44DB6915AB06BFB7F,2023-10-01 15:36:07,2023-10-02 15:36:07,flakey1,snowflake,25260
2023-10-04 15:24:09,5481936581E23D2D178105D44DB6915AB06BFB7F,2023-10-02 15:36:07,2023-10-03 15:36:07,flakey1,<OR>,4
2023-10-04 15:24:09,5481936581E23D2D178105D44DB6915AB06BFB7F,2023-10-02 15:36:07,2023-10-03 15:36:07,flakey1,snowflake,25692
2023-10-06 05:09:20,5481936581E23D2D178105D44DB6915AB06BFB7F,2023-10-04 07:52:00,2023-10-05 07:52:00,flakey1,<OR>,28092
2023-10-06 05:09:20,5481936581E23D2D178105D44DB6915AB06BFB7F,2023-10-04 07:52:00,2023-10-05 07:52:00,flakey1,snowflake,27316
2023-10-07 05:17:28,5481936581E23D2D178105D44DB6915AB06BFB7F,2023-10-05 07:52:00,2023-10-06 07:52:00,flakey1,<OR>,26732
2023-10-07 05:17:28,5481936581E23D2D178105D44DB6915AB06BFB7F,2023-10-05 07:52:00,2023-10-06 07:52:00,flakey1,snowflake,25972
2023-10-08 05:27:34,5481936581E23D2D178105D44DB6915AB06BFB7F,2023-10-06 07:52:00,2023-10-07 07:52:00,flakey1,<OR>,27324
2023-10-08 05:27:34,5481936581E23D2D178105D44DB6915AB06BFB7F,2023-10-06 07:52:00,2023-10-07 07:52:00,flakey1,snowflake,26572
2023-10-09 04:04:42,5481936581E23D2D178105D44DB6915AB06BFB7F,2023-10-07 07:52:00,2023-10-08 07:52:00,flakey1,<OR>,28284
2023-10-09 04:04:42,5481936581E23D2D178105D44DB6915AB06BFB7F,2023-10-07 07:52:00,2023-10-08 07:52:00,flakey1,snowflake,27500

mentioned in merge request !776 (merged)

assigned to @trinity-1686a

added Doing label

mentioned in issue tpo/network-health/metrics/website#40102 (closed)

mentioned in commit tpo/network-health/metrics/timeline@1c3f4596

mentioned in issue tpo/network-health/metrics/website#40103 (closed)

should be fixed in 0.4.8.10

closed

mentioned in issue tpo/anti-censorship/pluggable-transports/snowflake#40311 (closed)

Tor incorrectly stores stats on incoming PT connections

Child items 0

Activity