The Tor Project issueshttps://gitlab.torproject.org/groups/tpo/-/issues2021-08-23T13:18:03Zhttps://gitlab.torproject.org/tpo/network-health/metrics/collector/-/issues/40002CollecTor should archive sanitised bridgestrap results2021-08-23T13:18:03ZirlCollecTor should archive sanitised bridgestrap resultsTo monitor the health of bridges in the network over time, and also to support providing better information to bridge operators via Onionoo/Relay Search, CollecTor should archive sanitised bridgestrap results.
c.f. https://gitlab.torpro...To monitor the health of bridges in the network over time, and also to support providing better information to bridge operators via Onionoo/Relay Search, CollecTor should archive sanitised bridgestrap results.
c.f. https://gitlab.torproject.org/tpo/metrics/relay-search/-/issues/40003irlirl2021-07-31https://gitlab.torproject.org/tpo/core/tor/-/issues/1889Contradictory bandwidth reports in overlapping extra-info descriptors2020-06-27T14:09:09ZRobert HoganContradictory bandwidth reports in overlapping extra-info descriptorsOn a randomly sampled day in July there were 2000+ cases where the bw reported by extra-info for a given timeslice on a router was two different measurements, depending on which extra-info descriptor you looked at.
Here is an example in...On a randomly sampled day in July there were 2000+ cases where the bw reported by extra-info for a given timeslice on a router was two different measurements, depending on which extra-info descriptor you looked at.
Here is an example in the read history for relay Pounet27TorRelay:
BW NOT EQUAL for 2010-07-30T02:31. Is 439296 : Was 4738048
read-history 2010-07-30 21:46:14 (900 s) 11229184,13546496,11268096,13392896,16567296,9265152,6163456,4869120,8190976,0,0,0,0,0,0,0,0,0,734208,439296,0,0,0,0,0,0,0,0,0,6400000,11721728,12796928,12404736,14963712,13637632,11209728,12163072,10721280,6233088,10669056,9666560,9474048,11543552,11773952,10487808,10580992,10299392,13220864,12548096,11171840,7584768,4970496,3948544,2686976,4140032,4179968,4777984,8782848,9647104,9838592,13700096,10819584,11526144,9906176,14349312,12468224,13254656,8672256,5794816,7155712,3275776,3682304,3339264,6937600,6926336,7901184,8549376,10533888,7639040,9754624,6426624,4097024,5630976,4674560,5093376,6675456,4874240,5531648,3744768,6883328,9871360,11807744,11097088,9028608,9231360,9802752
read-history 2010-07-30 04:46:14 (900 s) 8813568,8123392,9978880,8428544,9339904,6733824,9417728,11633664,11225088,9106432,11148288,7831552,11962368,9401344,8095744,8654848,9501696,7830528,6435840,8958976,4248576,4089856,4747264,6569984,4973568,2717696,5286912,0,0,1968128,0,0,0,0,0,0,0,0,0,2446336,5278720,6597632,4627456,7857152,7080960,8280064,5863424,7086080,5991424,2252800,1883136,5161984,3418112,4377600,3938304,6874112,5746688,2485248,3495936,4019200,6891520,3680256,4489216,4912128,3629056,1320960,7661568,11542528,11229184,13546496,11268096,13392896,16567296,9265152,6163456,4869120,8190976,0,0,0,0,0,0,0,0,0,245760,4738048,5920768,3057664,4745216,6483968,5610496,5480448,5497856,5471232
The difference starts after the zeros in the following sequence in each extra-info:
21:46: 11229184,13546496,11268096,13392896,16567296,9265152,6163456,4869120,8190976,0,0,0,0,0,0,0,0,0,245760,4738048
04:46: 11229184,13546496,11268096,13392896,16567296,9265152,6163456,4869120,8190976,0,0,0,0,0,0,0,0,0,734208,439296
I need to characterize the samples I have better to see if there's a pattern but any idea why this might happen?Tor: unspecifiedhttps://gitlab.torproject.org/tpo/core/tor/-/issues/1907Unbalanced bandwidth reported in extra-info2020-06-27T14:09:08ZRobert HoganUnbalanced bandwidth reported in extra-infoWhen I scan the extra-info data from July for routers that reported more bandwidth read than written in a given 15 minute interval I find quite a few examples. Here's one:
write-history 2010-07-30 05:23:10 (900 s) 631808,48128,28672,143...When I scan the extra-info data from July for routers that reported more bandwidth read than written in a given 15 minute interval I find quite a few examples. Here's one:
write-history 2010-07-30 05:23:10 (900 s) 631808,48128,28672,1436672,1487872,1076224,860160,1084416,1443840,1492992,334848,1538048,1835008,1281024,21504,2844672,1456128,1641472,21504,1600512,2584576,1123328,24576,1881088,1755136,1750016,26624,1568768,1318912,753664,1013760,991232,15360,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1824768,1720320,1221632,25600,1684480,1898496,363520,916480,1755136,842752,14336,0,0,11264,0,0,0,0,0,0,0,0,1847296,2082816,1767424,23552,664576,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1115136
read-history 2010-07-30 05:23:10 (900 s) 1223680,41984,24576,4365312,4619264,3951616,2774016,4028416,6646784,7943168,1826816,8904704,10902528,6670336,18432,14535680,8086528,7943168,19456,6751232,14049280,6567936,21504,9063424,10653696,10608640,23552,7103488,5774336,3519488,5557248,5390336,11264,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,5951488,10020864,7044096,447488,9719808,10972160,331776,4963328,9638912,4784128,11264,0,0,12288,0,0,0,0,0,0,0,0,5955584,11724800,10200064,25600,1619968,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3531776
Note:
1847296,2082816,1767424,23552,664576
written against
5955584,11724800,10200064,25600,1619968
read. That's probably not just directory fetches. My guess it's client bandwidth. There may even be cases where it's hidden service bandwidth - if the service is running 'locally'.
I ran a scan against the 30th July to pull the routers this happens to. I only pulled the guys that reported a 4MB plus difference for any given 15m interval. The same guys show up again and again, so they are doing something different to everyone else.Tor: unspecifiedhttps://gitlab.torproject.org/tpo/core/tor/-/issues/1926Extra-info descriptors have corrupt {write,read}-history lines2020-06-27T14:09:07ZKarsten LoesingExtra-info descriptors have corrupt {write,read}-history linesWhile parsing extra-info descriptors for the dir bytes analysis, I found that some of them have weird write-history or read-history lines with obviously wrong dates. I wonder what leads to these false reports and if there are more errors...While parsing extra-info descriptors for the dir bytes analysis, I found that some of them have weird write-history or read-history lines with obviously wrong dates. I wonder what leads to these false reports and if there are more errors like this in extra-info descriptors.
Here are some examples:
```
published 2010-09-03 08:21:00
write-history 2002-01-01 00:47:32 (900 s) 0,109568,44032,189440,219136,
92160,190464,354304,126976
read-history 2002-01-01 00:47:32 (900 s) 816128,1042432,163840,180224,
2770944,221184,133120,359424,770048
```
```
published 2010-06-08 19:20:14
write-history 2003-01-01 02:20:36 (900 s) 8192,26624,18432,49152,169984
read-history 2003-01-01 02:20:36 (900 s) 147456,154624,13312,23552,2411520
```
```
published 2010-08-17 17:14:51
write-history 2003-01-01 03:16:39 (900 s) 3454976
read-history 2010-08-17 04:16:39 (900 s)
```
```
published 2010-08-31 12:12:21
write-history 2002-01-01 03:06:31 (900 s) 117760,69632,59392,52224,61440,
871424,3474432,3022848,4219904,2000896,160768,138240,138240,175104,131072,
47104,59392,33792,37888,87040
read-history 2002-01-01 03:06:31 (900 s) 1923072,52224,194560,35840,
1046528,906240,3657728,3198976,5303296,2452480,338944,423936,1155072,
308224,119808,193536,1045504,91136,191488,104448
```
```
published 2010-08-31 12:12:28
write-history 2002-01-01 03:06:31 (900 s) 117760,69632,59392,52224,61440,
871424,3474432,3022848,4219904,2000896,160768,138240,138240,175104,131072,
47104,59392,33792,37888,87040
read-history 2010-08-31 12:06:31 (900 s) 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0
```
```
published 2010-05-28 21:30:25
write-history 2010-05-28 21:15:26 (900 s) 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0
read-history 1970-01-02 07:45:26 (900 s) 1245184,447488,1458176,1916928,
324608,479232,133120,272384,1698816,791552,573440,381952,352256,667648,
451584,861184,515072,902144,301056,5548032,14390272,11052032,395264,
372736,250880,560128,202752,199680,1078272,915456,2571264,227328,194560,
602112,189440,325632,194560,450560,320512,180224,263168,488448,167936,
172032,135168,506880,175104,205824,23552,401408,21504,21504,180224,274432,
18432,139264,15360,399360,13312,143360,20480,405504,16384,143360,16384,
398336,11264,142336,15360,270336,15360,143360,15360,398336,15360,14336,
20480,407552,18432,147456,13312,414720,13312,144384,10240,406528,3072,
135168,8192,270336,4096,7168,5120,404480,7168,5120
```
Any idea what causes this problem? I didn't find an obvious pattern in these examples. Is it just memory corruption on the relays?Tor: unspecifiedhttps://gitlab.torproject.org/tpo/core/tor/-/issues/5825Bridges without geoip file report empty statistics2021-07-09T17:21:59ZKarsten LoesingBridges without geoip file report empty statisticsA bridge that doesn't have a geoip file reports empty statistics, implying that it saw zero users. That's confusing, because we cannot distinguish this case from a bridge that actually didn't see any users. The bridge should simply lea...A bridge that doesn't have a geoip file reports empty statistics, implying that it saw zero users. That's confusing, because we cannot distinguish this case from a bridge that actually didn't see any users. The bridge should simply leave out its statistics if it doesn't have a geoip file.
There are [not many bridges affected](https://metrics.torproject.org/papers/bridge-report-usage-stats-2012-04-30.pdf) by this bug, but we should probably fix it anyway. 0.2.3.x or 0.2.4.x?https://gitlab.torproject.org/tpo/core/tor/-/issues/7134Add statistics on time spent on crypto operations2020-07-23T17:53:24ZKarsten LoesingAdd statistics on time spent on crypto operationsWe'd like to find out how much of Tor's time is spent on which crypto operations, and we'd want to answer this question over time.
These statistics would help us understand the implications of other design shifts -- like directory guard...We'd like to find out how much of Tor's time is spent on which crypto operations, and we'd want to answer this question over time.
These statistics would help us understand the implications of other design shifts -- like directory guards, where one of the desired side effects is a reduced number of TLS handshakes for relays because clients handshake with fewer relays. Another benefit could be that we'd have better data on hardware requirements for the Torouter and similar projects, in particular crypto operations performed by an average relay or bridge.
Probably the easiest way to achieve the "answer over time" part would be to have relays/bridges add a new line to their extra-info descriptors, and later look at metrics archives to see how things changed over time. There should not be too big privacy implications if we make the granularity sufficiently big.
We might want to make this feature optional, since it takes time to check the time. (Gotta find out how many nanoseconds for a hi-resolution timers vs how many nanoseconds for encrypting a cell. That would be part of the design.)Tor: unspecifiedhttps://gitlab.torproject.org/tpo/core/tor/-/issues/8190Relays should publish number of refill intervals where the token bucket went dry2022-05-23T20:32:01ZRoger DingledineRelays should publish number of refill intervals where the token bucket went dryOur extrainfo descriptors have 15-minute intervals where we list the sum of bytes handled in that interval.
Intuitively, it seems that congestion should have to do with how many times we run out of tokens in our token bucket.
We could ...Our extrainfo descriptors have 15-minute intervals where we list the sum of bytes handled in that interval.
Intuitively, it seems that congestion should have to do with how many times we run out of tokens in our token bucket.
We could track the number of refills where our global buckets run dry, and publish the number of such refills in each 15-minute interval.
We probably want to track read bucket and write bucket separately; differences between the two might teach us something about legacy/trac#4682 as well.https://gitlab.torproject.org/tpo/core/tor/-/issues/8786Add extra-info line that tracks the number of consensus downloads over each p...2022-05-23T20:34:06ZGeorge KadianakisAdd extra-info line that tracks the number of consensus downloads over each pluggable transportIn legacy/trac#5040, Karsten suggested to add yet another line for measuring obfsbridge stats.
He wants a `dirreq-v3-transport` line with the exact same format as `bridge-ip-transports`, that counts consensus fetches instead of direct c...In legacy/trac#5040, Karsten suggested to add yet another line for measuring obfsbridge stats.
He wants a `dirreq-v3-transport` line with the exact same format as `bridge-ip-transports`, that counts consensus fetches instead of direct connections. This will improve the granularity of bridge statistics, and it will help us count users accurately in scenarios like flashproxy (where each client is actually a flashproxy bridge).
This means that we should be considering the `GEOIP_CLIENT_NETWORKSTATUS_V2` and `GEOIP_CLIENT_NETWORKSTATUS` events in this case, instead of `GEOIP_CLIENT_CONNECT`.https://gitlab.torproject.org/tpo/core/tor/-/issues/13167Export dirauth files via directory protocol2022-03-17T20:07:02ZSebastian HahnExport dirauth files via directory protocolMetrics downloads a few files (consensus, descriptors, extrainfo, v3 votes) from dirauths for further processing. It'd be good if all these files could be served by Tor directly, as this would alleviate the need for the dirauth ops to ta...Metrics downloads a few files (consensus, descriptors, extrainfo, v3 votes) from dirauths for further processing. It'd be good if all these files could be served by Tor directly, as this would alleviate the need for the dirauth ops to take special steps to make these files available.https://gitlab.torproject.org/tpo/core/tor/-/issues/13194Track time between ESTABLISH_RENDEZVOUS and RENDEZVOUS1 cell2020-07-24T15:55:20ZRoger DingledineTrack time between ESTABLISH_RENDEZVOUS and RENDEZVOUS1 cellIn messing with legacy/trac#13193, I ended up with lines like
```
Sep 19 02:12:42.656 [info] stats_label_circ(): circ has newly been used for hs: ESTABLISH_RENDEZVOUS
Sep 19 02:12:55.131 [info] stats_label_circ(): circ has newly been use...In messing with legacy/trac#13193, I ended up with lines like
```
Sep 19 02:12:42.656 [info] stats_label_circ(): circ has newly been used for hs: ESTABLISH_RENDEZVOUS
Sep 19 02:12:55.131 [info] stats_label_circ(): circ has newly been used for hs: RENDEZVOUS1
```
and
```
Sep 19 02:05:53.253 [info] stats_label_circ(): circ has newly been used for hs: ESTABLISH_RENDEZVOUS
Sep 19 02:05:54.033 [info] stats_label_circ(): circ has newly been used for hs: RENDEZVOUS1
```
That second one looks pretty good speed-wise, and that first one looks pretty ugly. I assume some of this slow-down is due to legacy/trac#13151, but I don't know how much.
Wouldn't it be cool to track this delay value over time, to see if things are getting better or worse, to be able to measure how big a deal things like legacy/trac#13151 are, etc?Tor: unspecifiedhttps://gitlab.torproject.org/tpo/core/tor/-/issues/13195Collect aggregate stats around hidden service descriptor publishes and fetches2020-07-24T15:54:27ZRoger DingledineCollect aggregate stats around hidden service descriptor publishes and fetchesHow many hidden services are there, total? Is that number going up or down? Are there interesting recent trends? Does ahmia's list of hidden services represent most of them or few of them?
We have no idea.
If each relay that serves as ...How many hidden services are there, total? Is that number going up or down? Are there interesting recent trends? Does ahmia's list of hidden services represent most of them or few of them?
We have no idea.
If each relay that serves as an HSDir fetch track of number of .onion addresses they've seen, total number of descriptor publishes they've seen, and total number of descriptor fetches they've seen, we'd get closer to answering some of these questions.
There is also a privacy question here: a given HSDir is responsible for a given slice of the keyspace, so if indeed hidden services vary widely in popularity, then I expect the answers here to vary widely by relay. How bad is it to reveal stats that might be used to guess about popularity of hidden services? If it is bad, this might be a perfect situation for using a tool like PrivEx, to get the network-wide total without revealing per-relay contribution to that total.Tor: unspecifiedhttps://gitlab.torproject.org/tpo/core/tor/-/issues/13258Keep stats on effectiveness of consensus diffs2020-07-24T15:53:08ZSebastian HahnKeep stats on effectiveness of consensus diffsWe could have all dir mirrors report how many clients they've seen requesting diffs to a consensus X hours old. That way, we could fine-tune the amount of time dir mirrors have to store old diffs, and we could estimate how much of a win ...We could have all dir mirrors report how many clients they've seen requesting diffs to a consensus X hours old. That way, we could fine-tune the amount of time dir mirrors have to store old diffs, and we could estimate how much of a win they are in general.Tor: unspecifiedhttps://gitlab.torproject.org/tpo/core/tor/-/issues/13466Collect aggregate stats of ntor-using hidden service interactions vs tap-usin...2020-07-24T16:11:21ZRoger DingledineCollect aggregate stats of ntor-using hidden service interactions vs tap-using interactionsIn legacy/trac#13195 I have a patch to better understand the fraction of Tor traffic that has to do with hidden services, vs the fraction that has to do with exiting.
For the hidden service traffic, we can see if the handshake that exte...In legacy/trac#13195 I have a patch to better understand the fraction of Tor traffic that has to do with hidden services, vs the fraction that has to do with exiting.
For the hidden service traffic, we can see if the handshake that extended that hop of the circuit was done using ntor or with tap.
By remembering this bit, we can track fraction of hidden service traffic that came from old Tor 0.2.3.x style bots, vs newer clients. It would be very useful to see this fraction over time.Tor: unspecifiedhttps://gitlab.torproject.org/tpo/core/tor/-/issues/13792HS statistics for private tor network to gather info on services, clients and...2020-06-27T14:02:13ZDavid Gouletdgoulet@torproject.orgHS statistics for private tor network to gather info on services, clients and relaysThis ticket is for gathering HS statistics for a private tor network.
Reference: legacy/trac#13509This ticket is for gathering HS statistics for a private tor network.
Reference: legacy/trac#13509Tor: unspecifiedhttps://gitlab.torproject.org/tpo/core/tor/-/issues/16846Include sizeof(void *) in your extrainfo.2020-06-27T14:00:48ZYawning AngelInclude sizeof(void *) in your extrainfo.From legacy/trac#16535, it appears that it would be useful to get an idea of how much x86 vs x86_64 we have as part of the tor network. If we don't also include CPU architecture information, it would be good to have that as well if we ca...From legacy/trac#16535, it appears that it would be useful to get an idea of how much x86 vs x86_64 we have as part of the tor network. If we don't also include CPU architecture information, it would be good to have that as well if we can do it safely.https://gitlab.torproject.org/tpo/core/tor/-/issues/16997Gather and report metrics for the number of channels a relay is servicing.2020-07-24T18:06:15ZYawning AngelGather and report metrics for the number of channels a relay is servicing.It would be really nice to have statistics on the min/max/average number of simultaneous channels (TCP/IP connections) a relay is servicing for a few reasons:
* If we are to add per-channel padding like in legacy/trac#16861 (or somethi...It would be really nice to have statistics on the min/max/average number of simultaneous channels (TCP/IP connections) a relay is servicing for a few reasons:
* If we are to add per-channel padding like in legacy/trac#16861 (or something more aggressive like people discuss), we need this to figure out how much extra bandwidth a relay will use, and possibly for parameter tuning.
* This information will be helpful in determining if someone can/should run a relay, given their environment (Eg: To figure out if their relay will end up underutilized/bad for the network due to NAT connection table/OS limitations).
I'm uncertain as to how coarse grained this will need to be, particularly for Guards. Bridges **SHOULD NOT** report this information.Tor: unspecifiedhttps://gitlab.torproject.org/tpo/core/tor/-/issues/19009bandwidth testing circuits should be allowed to use our guards2022-06-22T15:21:09ZRoger Dingledinebandwidth testing circuits should be allowed to use our guardsIn git commit 267e61d0, we fixed bug legacy/trac#654, where relays were putting all their bandwidth testing circuits over the same small set of guards, thus having an accidental bottleneck, meaning we got an inaccurate bandwidth estimate...In git commit 267e61d0, we fixed bug legacy/trac#654, where relays were putting all their bandwidth testing circuits over the same small set of guards, thus having an accidental bottleneck, meaning we got an inaccurate bandwidth estimate.
But we fixed it by having them *never* use a guard for their first hop of a testing circuit, which in turn produced surprising behavior in tiny test networks, because relays can't make testing circuits if all the available relays are in their guard list.
teor fixed that in commit 22a1e9cac by making us not avoid our guards if testingtornetwork, and not avoid our guards if all the nodes in the consensus are on our guard list. It turns out that latter check isn't quite good enough, because we're picking two hops, so having at least one relay in the network that isn't in our guard list isn't enough to complete a circuit.
The underlying problem is that when UseEntryGuards is true, the rest of choose_good_entry_server() is entirely about picking a new entry guard. Except we reuse it for some edge cases where we want to just pick some entry point and not use our guard list. Some refactoring or something seems wise.https://gitlab.torproject.org/tpo/core/tor/-/issues/21345Do relays count dir reqs as completed before they're complete?2020-09-07T05:39:10ZRoger DingledineDo relays count dir reqs as completed before they're complete?I've been eyeing the spike in .ae users, and I proposed to dgoulet that directory mirrors are mis-counting the number of completed consensus fetches. In particular, I proposed that they are counting consensus fetches as complete before t...I've been eyeing the spike in .ae users, and I proposed to dgoulet that directory mirrors are mis-counting the number of completed consensus fetches. In particular, I proposed that they are counting consensus fetches as complete before they actually complete.
This ticket is for documenting my investigation of whether I think this misbehavior is really happening.Tor: unspecifiedhttps://gitlab.torproject.org/tpo/core/tor/-/issues/26694dir-spec: DirAuths should expose bwauth bandwidth files2020-06-27T13:52:58Zjugadir-spec: DirAuths should expose bwauth bandwidth filesThis ticket is for changing dir-spec to implement legacy/trac#21377This ticket is for changing dir-spec to implement legacy/trac#21377Tor: 0.4.1.x-finaljugajugahttps://gitlab.torproject.org/tpo/core/tor/-/issues/26797DirAuths should only read the V3BandwidthsFile once per vote2022-06-17T13:55:19ZteorDirAuths should only read the V3BandwidthsFile once per voteEvery time we read the file, we increase the risk of race conditions, where some code reads one version of the file, and other code reads another version.
See legacy/trac#26702 for details.
If we implement legacy/trac#21377, we should ...Every time we read the file, we increase the risk of race conditions, where some code reads one version of the file, and other code reads another version.
See legacy/trac#26702 for details.
If we implement legacy/trac#21377, we should do this refactor first.