Trac issueshttps://gitlab.torproject.org/legacy/trac/-/issues2020-07-02T19:47:20Zhttps://gitlab.torproject.org/legacy/trac/-/issues/33262Prop 313: 3. Write a Script that Counts IPv6 Relays in the Consensus2020-07-02T19:47:20ZteorProp 313: 3. Write a Script that Counts IPv6 Relays in the ConsensusWe want to write a script that generates statistics for relays that:
1. have an IPv6 ORPort,
2. support IPv6 clients,
3. support IPv6 reachability checks, and
4. support IPv6 reachability checks, and IPv6 clients.
The first two ...We want to write a script that generates statistics for relays that:
1. have an IPv6 ORPort,
2. support IPv6 clients,
3. support IPv6 reachability checks, and
4. support IPv6 reachability checks, and IPv6 clients.
The first two statistics have no dependencies. The last two statistics depend on the "Relay=3" subprotocol in #33226.
The script should calculate:
* the number of relays, and
* the consensus weight fraction of relays.
In order to provide easy access to these statistics, we propose
that the script should:
* download a consensus (or read an existing consensus), and
* calculate and report these statistics.
We could write this script using Python 3 and Stem:
https://stem.torproject.org
The following consensus weight fractions should divide by the total
consensus weight:
* have an IPv6 ORPort (all relays have an IPv4 ORPort), and
* support IPv6 reachability checks (all relays support IPv4 reachability).
The following consensus weight fractions should divide by the
"usable Guard" consensus weight:
* support IPv6 clients, and
* support IPv6 reachability checks and IPv6 clients.
"Usable Guards" have the Guard flag, but do not have the Exit flag. If the
Guard also has the BadExit flag, the Exit flag should be ignored.
The script should check that Wgd is 0. If it is not, the script
should log a warning about the accuracy of the "Usable Guard" statistics.
See proposal 313, section 3:
https://gitweb.torproject.org/torspec.git/tree/proposals/313-relay-ipv6-stats.txt#n82Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/33266Prop 313: 7.2. Show IPv6 Relay Counts on Consensus Health2020-06-13T17:54:14ZteorProp 313: 7.2. Show IPv6 Relay Counts on Consensus HealthHi Tom,
Do you have some time to add some IPv6 relay counts to https://consensus-health.torproject.org/ ?
This change is an optional part of proposal 313, which adds some IPv6 statistics to tor. There's no hurry, any time in the next f...Hi Tom,
Do you have some time to add some IPv6 relay counts to https://consensus-health.torproject.org/ ?
This change is an optional part of proposal 313, which adds some IPv6 statistics to tor. There's no hurry, any time in the next few months would be great.
Before you make these changes, we need to implement a bunch of IPv6 relay changes in tor, and add an IPv6 Relay subprotocol version to tor in #33226.
Then consensus health can add an IPv6 section, which counts relays in the consensus that:
* have an IPv6 ORPort, and
* support IPv6 reachability checks.
We'd like to see the number of relays, and the consensus weight fraction.
The definitions of these statistics are in proposal 313, section 3:
https://gitweb.torproject.org/torspec.git/tree/proposals/313-relay-ipv6-stats.txt#n82
These changes can be tested using the script in ticket #33262.
We hope that this change will:
* help tor developers improve IPv6 support on the tor network,
* help diagnose issues with IPv6 on the tor network, and
* drive IPv6 adoption on tor relays.
This consensus health change is described in proposal 313, section 7.2:
https://gitweb.torproject.org/torspec.git/tree/proposals/313-relay-ipv6-stats.txt#n236Tom Rittertom@ritter.vgTom Rittertom@ritter.vghttps://gitlab.torproject.org/legacy/trac/-/issues/33273Prop 313: 8.2. Analyse and Monitor IPv6 Stats2020-06-13T17:48:14ZteorProp 313: 8.2. Analyse and Monitor IPv6 StatsAs part of Sponsor 55, the Tor Metrics team will analyse the new IPv6 connection and bandwidth statistics.
We will use their analysis to provide IPv6 progress reports.
We might also discover some bugs in tor's IPv6 stats code during th...As part of Sponsor 55, the Tor Metrics team will analyse the new IPv6 connection and bandwidth statistics.
We will use their analysis to provide IPv6 progress reports.
We might also discover some bugs in tor's IPv6 stats code during the first analysis.
See proposal 313, section 8.2, metrics analysis part:
https://gitweb.torproject.org/torspec.git/tree/proposals/313-relay-ipv6-stats.txt#n355
The definitions of these statistics are in proposal 313, sections 4 and 5:
https://gitweb.torproject.org/torspec.git/tree/proposals/313-relay-ipv6-stats.txt#n118https://gitlab.torproject.org/legacy/trac/-/issues/33956Define and use TOR_ADDRPORT_BUF_LEN2020-06-13T15:53:13ZteorDefine and use TOR_ADDRPORT_BUF_LENIn #33918, we discovered a bug where IPv6 addresses were being truncated in logs.
During the fix, we noticed that we had a TOR_ADDR_BUF_LEN, but no equivalent constant for addresses and ports. The new TOR_ADDRPORT_BUF_LEN should allow s...In #33918, we discovered a bug where IPv6 addresses were being truncated in logs.
During the fix, we noticed that we had a TOR_ADDR_BUF_LEN, but no equivalent constant for addresses and ports. The new TOR_ADDRPORT_BUF_LEN should allow space for:
* TOR_ADDR_BUF_LEN
* IPv6 brackets (2, if not included in TOR_ADDR_BUF_LEN already)
* the port separator (1)
* the port (5)
We should check for other truncation errors while making this change.Tor: 0.4.4.x-finalNeel Chauhanneel@neelc.orgNeel Chauhanneel@neelc.orghttps://gitlab.torproject.org/legacy/trac/-/issues/33812Add unit tests for bandwidth statistics functions2020-06-13T15:52:48ZMrSquancheeAdd unit tests for bandwidth statistics functionsWe currently don't have tests for bandwidth statistics
reporting code.
We need to write tests for the following functions.
1. commit_max()
2. advance_obs()
3. add_obs()
4. rep_hist_bandwidth_assess()
5. rep_hist_fill_bandwidth_history()
...We currently don't have tests for bandwidth statistics
reporting code.
We need to write tests for the following functions.
1. commit_max()
2. advance_obs()
3. add_obs()
4. rep_hist_bandwidth_assess()
5. rep_hist_fill_bandwidth_history()
6. rep_hist_get_bandwidth_lines()
This will be useful for #33617Tor: unspecifiedMrSquancheeMrSquancheehttps://gitlab.torproject.org/legacy/trac/-/issues/33617Add a BandwidthStatistics option and consensus parameter2020-06-13T15:52:12ZteorAdd a BandwidthStatistics option and consensus parameterStage 1:
We propose adding a new BandwidthStatistics torrc option, which activates reporting of all the bandwidth statistics. At the moment, all statistics are controlled by ExtraInfoStatistics, but we propose using the new BandwidthSt...Stage 1:
We propose adding a new BandwidthStatistics torrc option, which activates reporting of all the bandwidth statistics. At the moment, all statistics are controlled by ExtraInfoStatistics, but we propose using the new BandwidthStatistics option specifically for the bandwidth statistics.
The default value of this option should be 1. (The existing bandwidth statistics are reported by default.)
The new option should have a manual page entry, and test_parseconf.sh tests in src/test/conf_examples.
Stage 2:
Add a BandwidthStatistics consensus paramter.
Change the default value of the BandwidthStatistics option to "auto". If the value is "auto", use the value of the consensus parameter. If the consensus parameter is not set, use "1".
Update the manual page to describe the new consensus parameter.
See the proposal:
https://gitweb.torproject.org/torspec.git/tree/proposals/313-relay-ipv6-stats.txt#n298Tor: 0.4.4.x-finalMrSquancheeMrSquancheehttps://gitlab.torproject.org/legacy/trac/-/issues/33272Prop 313: 8.2. Test IPv6 Stats on the Tor Network2020-06-13T15:51:18ZteorProp 313: 8.2. Test IPv6 Stats on the Tor NetworkWe want to test the IPv6 connection and bandwidth statistics on the public network, with a small number of relays and bridges.
See proposal 313, section 8.2, public tor network tests part:
https://gitweb.torproject.org/torspec.git/tree/...We want to test the IPv6 connection and bandwidth statistics on the public network, with a small number of relays and bridges.
See proposal 313, section 8.2, public tor network tests part:
https://gitweb.torproject.org/torspec.git/tree/proposals/313-relay-ipv6-stats.txt#n355Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/33270Prop 313: 8.1. Monitor IPv6 Relay Counts in the Consensus2020-06-13T15:51:17ZteorProp 313: 8.1. Monitor IPv6 Relay Counts in the ConsensusMonitor the output of the script in #33262.
Continue monitoring as more volunteer relay operators deploy the relevant tor versions. (And as the number of IPv6 relays in the consensus increases.)
There isn't actually much to check here....Monitor the output of the script in #33262.
Continue monitoring as more volunteer relay operators deploy the relevant tor versions. (And as the number of IPv6 relays in the consensus increases.)
There isn't actually much to check here. If we do find issues, we should open separate bugs for them.
See proposal 313, section 8.1, ongoing monitoring part:
https://gitweb.torproject.org/torspec.git/tree/proposals/313-relay-ipv6-stats.txt#n337Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/33268Prop 313: 8.1. Test IPv6 Relay Consensus Counts on the Tor Network2020-06-13T15:51:17ZteorProp 313: 8.1. Test IPv6 Relay Consensus Counts on the Tor NetworkThis task depends on #33262.
Test the IPv6 relay count script in proposal 313, using the public tor network.
There isn't really much to test here. There are only 4 calculations. We've done counts like this before.
See proposal 313, se...This task depends on #33262.
Test the IPv6 relay count script in proposal 313, using the public tor network.
There isn't really much to test here. There are only 4 calculations. We've done counts like this before.
See proposal 313, section 8.1, public tor network tests part:
https://gitweb.torproject.org/torspec.git/tree/proposals/313-relay-ipv6-stats.txt#n337Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/33265Prop 313: 6. Update Directory Spec for IPv6 Stats2020-06-13T15:51:16ZteorProp 313: 6. Update Directory Spec for IPv6 StatsWe want to add some IPv6 statistics to extra-info documents in the tor directory protocol.
See proposal 313, section 6, for a draft of this change:
https://gitweb.torproject.org/torspec.git/tree/proposals/313-relay-ipv6-stats.txt#n168We want to add some IPv6 statistics to extra-info documents in the tor directory protocol.
See proposal 313, section 6, for a draft of this change:
https://gitweb.torproject.org/torspec.git/tree/proposals/313-relay-ipv6-stats.txt#n168Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/33264Prop 313: 5. Collect IPv6 Connection Stats on Relays2020-06-13T15:51:15ZteorProp 313: 5. Collect IPv6 Connection Stats on RelaysWe propose that relays (but not bridges) collect IPv6 connection statistics.
Bridges refuse to collect the existing ConnDirectionStatistics, so we do not
believe it is safe to collect the smaller IPv6 totals on bridges.
To minimise dev...We propose that relays (but not bridges) collect IPv6 connection statistics.
Bridges refuse to collect the existing ConnDirectionStatistics, so we do not
believe it is safe to collect the smaller IPv6 totals on bridges.
To minimise development and testing effort, we propose re-using the existing
"bidi" code in rephist.c. (This code may require some refactoring, because
the "bidi" totals are globals, rather than a struct.)
(We might also want to move this code into separate relay-only code and
header files, because it is relay-specific.)
In particular, tor currently counts these connection statistics:
* below threshold,
* mostly read,
* mostly written, and
* both read and written.
We propose adding IPv6 variants of all these statistics. (The IPv4
statistics can be calculated by subtracting the IPv6 statistics from the
existing total connection statistics.)
See proposal 313, section 5:
https://gitweb.torproject.org/torspec.git/tree/proposals/313-relay-ipv6-stats.txt#n144Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/33263Prop 313: 4. Collect IPv6 Bandwidth Stats on Relays and Bridges2020-06-13T15:51:15ZteorProp 313: 4. Collect IPv6 Bandwidth Stats on Relays and BridgesWe propose that relays (and bridges) collect IPv6 consumed bandwidth
statistics.
To minimise development and testing effort, we propose re-using the existing
"bw_array" code in rephist.c.
(We might want to move this code into separate ...We propose that relays (and bridges) collect IPv6 consumed bandwidth
statistics.
To minimise development and testing effort, we propose re-using the existing
"bw_array" code in rephist.c.
(We might want to move this code into separate relay-only code and header files, because it is relay-specific.)
In particular, tor currently counts these bandwidth statistics:
* read,
* write,
* dir_read, and
* dir_write.
We propose adding the following bandwidth statistics:
* ipv6_read, and
* ipv6_write.
(The IPv4 statistics can be calculated by subtracting the IPv6 statistics
from the existing total consumed bandwidth statistics.)
We believe that collecting IPv6 consumed bandwidth statistics is about as
safe as the existing IPv4+IPv6 total consumed bandwidth statistics.
See proposal 313, section 4:
https://gitweb.torproject.org/torspec.git/tree/proposals/313-relay-ipv6-stats.txt#n118Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/33271Prop 313: 8.2. Test IPv6 Statistics using Chutney2020-06-13T15:50:22ZteorProp 313: 8.2. Test IPv6 Statistics using ChutneyWe want to test IPv6 connection and consumed bandwidth statistics using
chutney networks. However, chutney runs for a short amount of time, and
creates a limited amount of traffic, so we also need to test these changes
on the public tor ...We want to test IPv6 connection and consumed bandwidth statistics using
chutney networks. However, chutney runs for a short amount of time, and
creates a limited amount of traffic, so we also need to test these changes
on the public tor network.
In particular, we have struggled to test statistics using chutney, because
tor's hard-coded statistics period is 24 hours. (And most chutney networks
run for under 1 minute.)
Maybe we can change the statistics interval to 10-30 seconds, and see if that works?
See proposal 313, section 8.2, chutney tests part:
https://gitweb.torproject.org/torspec.git/tree/proposals/313-relay-ipv6-stats.txt#n355Tor: 0.4.4.x-finalteorteorhttps://gitlab.torproject.org/legacy/trac/-/issues/33267Prop 313: 8.1. Test IPv6 Relay Consensus Counts using Chutney2020-06-13T15:50:20ZteorProp 313: 8.1. Test IPv6 Relay Consensus Counts using ChutneyTest the IPv6 relay count script in proposal 313, using chutney networks.
Since chutney creates a limited number of relays, we also need to test these changes on consensuses from the public tor network. So we shouldn't spend much time t...Test the IPv6 relay count script in proposal 313, using chutney networks.
Since chutney creates a limited number of relays, we also need to test these changes on consensuses from the public tor network. So we shouldn't spend much time testing the script with chutney.
See proposal 313, section 8.1, chutney tests part:
https://gitweb.torproject.org/torspec.git/tree/proposals/313-relay-ipv6-stats.txt#n337Tor: 0.4.4.x-finalteorteor