Trac issueshttps://gitlab.torproject.org/legacy/trac/-/issues2020-06-13T16:14:14Zhttps://gitlab.torproject.org/legacy/trac/-/issues/28158Vote for relay bandwidths even if the min percentage has not been reached2020-06-13T16:14:14ZjugaVote for relay bandwidths even if the min percentage has not been reachedIn #28062 we add additional header lines when the minimum percentage of measured relays that pass some restrictions and do not write any bandwidth lines.
A dirauth asked if that is better that writing the few bandwidth lines.
The reason ...In #28062 we add additional header lines when the minimum percentage of measured relays that pass some restrictions and do not write any bandwidth lines.
A dirauth asked if that is better that writing the few bandwidth lines.
The reason why we do it is because torflow does it and we want sbws 1.0 to behave as torflow.
In future sbws versions, we might prefer to include the bandwidth linessbws: 1.2.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/28547Monitor relays that are not measured by each sbws instance2020-06-13T16:15:34ZteorMonitor relays that are not measured by each sbws instanceIf sbws isn't measuring a relay, either:
* the relay is misconfigured, or
* there is a bug in sbws (like #28519)
We should decide how long it should take sbws to measure a relay. Then we should create a list of the relays that aren't me...If sbws isn't measuring a relay, either:
* the relay is misconfigured, or
* there is a bug in sbws (like #28519)
We should decide how long it should take sbws to measure a relay. Then we should create a list of the relays that aren't measured by sbws.
When we work out why a relay isn't being measured by sbws, we should contact the operator (or fix the bug).sbws: 1.1.x-finaljugajugahttps://gitlab.torproject.org/legacy/trac/-/issues/28563Work out how sbws can report excluded relays in the bandwidth file2020-06-13T16:15:47ZteorWork out how sbws can report excluded relays in the bandwidth fileIf we report excluded relays in the bandwidth file, then they will be publicly archived and available for anyone to analyse.
We just need to work out a syntax that makes tor ignore excluded relays.If we report excluded relays in the bandwidth file, then they will be publicly archived and available for anyone to analyse.
We just need to work out a syntax that makes tor ignore excluded relays.sbws: 1.1.x-finaljugajugahttps://gitlab.torproject.org/legacy/trac/-/issues/28565Report excluded results in a relay's bandwidth line2020-06-13T16:14:36ZteorReport excluded results in a relay's bandwidth lineWhen sbws excludes some results for a relay, we should:
* count how many results were excluded and how many were used
* report the excluded and used result counts in the relay's bandwidth line
If the number of excluded results is high, ...When sbws excludes some results for a relay, we should:
* count how many results were excluded and how many were used
* report the excluded and used result counts in the relay's bandwidth line
If the number of excluded results is high, we should split it into subcategories.sbws: 1.1.x-finaljugajugahttps://gitlab.torproject.org/legacy/trac/-/issues/28567Report relays that sbws wants to test, but the test doesn't work2020-06-13T16:14:38ZteorReport relays that sbws wants to test, but the test doesn't workWe should count:
* the number of times sbws tries to test a relay
* the number of times the test works and fails
If there is a high number of failures, we should also count:
* the number of times that pair selection works and fails
* th...We should count:
* the number of times sbws tries to test a relay
* the number of times the test works and fails
If there is a high number of failures, we should also count:
* the number of times that pair selection works and fails
* the number of times that connecting works and fails
* if there are a lot of these, count the hop that fails
* the number of times that data transfer works and failssbws: 1.1.x-finaljugajugahttps://gitlab.torproject.org/legacy/trac/-/issues/28582Document the load-balancing goal for sbws2020-06-13T15:34:30ZteorDocument the load-balancing goal for sbwsWe should include a section early in the bandwidth spec that documents the load balancing goal for sbws.
Here's my initial draft:
```
Bandwidth scanners should have a well-defined network equilibrium goal.
For sbws and Torflow without ...We should include a section early in the bandwidth spec that documents the load balancing goal for sbws.
Here's my initial draft:
```
Bandwidth scanners should have a well-defined network equilibrium goal.
For sbws and Torflow without PID control, the network equilibrium goals are:
1. clients experience reasonably consistent performance
2. clients experience performance that is as fast as possible, without compromising goal 1
These performance goals include both throughput and latency.
For Torflow with PID control, the goal was:
1. Each relay has exactly the same spare capacity for an additional stream
This goal was unachievable because Tor did not provide enough feedback on circuit failure.
These non-goals are common suggestions:
1. Bandwidth is allocated equally across relays
2. All relay bandwidth is used
These goals are unachievable, because they conflict with the consistent client performance goal.
```
Based on mike's response from this thread:
```
I believe quite strongly that even if the Tor network gets faster on
average, if this comes at the cost of increased performance variance,
user experience and perceived speed of Tor will be much worse. There's
nothing more annoying than a system that is *usually* fast enough to do
what you need it to do, but fails to be fast enough for that activity at
unpredictable times.
```
https://lists.torproject.org/pipermail/tor-dev/2018-August/013419.htmlTor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28694When CircuitPadding is implemented in Tor, set it to 0 in sbws2020-06-13T16:14:58ZteorWhen CircuitPadding is implemented in Tor, set it to 0 in sbwsIn #28693, we will implement a tor option for circuit padding.
When that option is implemented, we should set it to 0 in sbws.In #28693, we will implement a tor option for circuit padding.
When that option is implemented, we should set it to 0 in sbws.sbws: 1.1.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/28737Redesign sbws torrc option configuration2020-06-13T16:15:03ZteorRedesign sbws torrc option configurationsbws' torrc option handling is broken.
Here's a better scheme:
1. sbws config.default.ini and config.ini contain the following sections:
* tor.options.launch: a minimal set of options that must be configured when tor is launched. The ...sbws' torrc option handling is broken.
Here's a better scheme:
1. sbws config.default.ini and config.ini contain the following sections:
* tor.options.launch: a minimal set of options that must be configured when tor is launched. The minimal set contains the data directory config, control config, and log config. The network is disabled at launch. The initial options are:
```
'DataDirectory': conf.getpath('tor', 'datadir'),
'PidFile': conf.getpath('tor', 'pid'),
# Because we need things from full server descriptors (namely for now: the
# bandwidth line)
'UseMicrodescriptors': '0',
'ControlSocket': conf.getpath('tor', 'control_socket'),
# Easier than password authentication
'CookieAuthentication': '1',
'Log': [
'NOTICE file {}'.format(os.path.join(conf.getpath('tor', 'log'),
'notice.log')),
],
# useful logging options for clients that don't care about anonymity
'SafeLogging': '0',
'LogTimeGranularity': '1',
'ProtocolWarnings': '1',
'DisableNetwork': '1',
```
* tor.options.runtime.(group_name): Zero or more groups of tor options that can be set at runtime. The network is enabled at runtime. The "50-default" group of options is:
```
# We will find out via the ControlPort and not setting something static
# means a lower chance of conflict
'SocksPort': 'auto',
# To avoid path bias warnings
'UseEntryGuards': '0',
c.set_conf('__LeaveStreamsUnattached', '1')
# Things needed to make circuits fail a little faster. We get the
# circuit_timeout as a string instead of an int on purpose: stem only
# accepts strings.
'LearnCircuitBuildTimeout': '0',
'CircuitBuildTimeout': conf['general']['circuit_timeout'],
'DisableNetwork': '0',
```
* Zero or more tor.runtime.ignore_failure.(group name): tor options that are set in groups at runtime, but ignored if they fail. #28692 and #28694 will add options to this list.
Options in config.ini override options with the same name in sbws.default.ini, with + and / having the same meaning as in a torrc file (+ appends, / removes). Launch options are applied first, then runtime options are applied in group name order. Tor will make later runtime options replace earlier runtime options, and launch options.
2. sbws gets its control socket from the launch_options ControlSocket option(s)
3. sbws gets its data directory, pid, log(s), and circuit build timeout using GETCONF
4. For backwards compatibility:
* if tor.extra_lines is present, it should be appended to tor.options.launch. sbws' option merging code never worked, so appending shouldn't cause any more issues than the existing code.
* if any of these sbws options are present in an old config file, synthesise options.launch from the sbws options. If options.launch is also present in the same config file, it overrides the synthetic options. (sbws' option merging never worked for these options.)
```
'DataDirectory': conf.getpath('tor', 'datadir'),
'PidFile': conf.getpath('tor', 'pid'),
'ControlSocket': conf.getpath('tor', 'control_socket'),
'Log': [
'NOTICE file {}'.format(os.path.join(conf.getpath('tor', 'log'),
'notice.log')),
],
# Things needed to make circuits fail a little faster. We get the
# circuit_timeout as a string instead of an int on purpose: stem only
# accepts strings.
'CircuitBuildTimeout': conf['general']['circuit_timeout'],
```
The final option order is:
* sbws creates options.launch, using the last option from:
* synthetic legacy config options from config.default.ini
* options.launch from config.default.ini
* synthetic legacy config options from config.ini
* options.launch from config.ini
* sbws appends extra_lines to options.launch, using the last extra_lines from:
* config.default.ini
* config.ini
* sbws creates options.runtime groups, using the last one of each group from:
* config.default.ini
* config.ini
sbws launches tor with options.launch, then applies each group of options.runtime in order.
This is a new feature, so it should go in sbws 1.1.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28774Stop the integration tests http server when the tests end2020-06-13T16:15:08ZjugaStop the integration tests http server when the tests endThe integration tests launch an HTTP server (tox.ini), but it is not stop after tests finish.
This affect the developer running the integration tests, not the operator.The integration tests launch an HTTP server (tox.ini), but it is not stop after tests finish.
This affect the developer running the integration tests, not the operator.sbws: unspecifiedjugajugahttps://gitlab.torproject.org/legacy/trac/-/issues/28985Does sbws need Tor to report observed bandwidths more often?2020-06-13T16:15:17ZteorDoes sbws need Tor to report observed bandwidths more often?In #23856 in 2017, we updated the observed bandwidth to 5 days from 1 day.
We should review this change, and decide if Tor should report observed bandwidths more frequently.
In #28983, we'll work out how long it takes sbws to measure mo...In #23856 in 2017, we updated the observed bandwidth to 5 days from 1 day.
We should review this change, and decide if Tor should report observed bandwidths more frequently.
In #28983, we'll work out how long it takes sbws to measure most of the network. We could use this time as a guide: Tor probably shouldn't report observed bandwidths faster than sbws can measure them. But Tor should report observed bandwidths every few times sbws measures Tor.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/29047Improve code style following PEP8 and PEP2572020-06-13T16:15:19ZjugaImprove code style following PEP8 and PEP257sbws: unspecifiedjugajugahttps://gitlab.torproject.org/legacy/trac/-/issues/29057Adapt bandwidth file classes to be compatible with stem (descriptors, etc) do...2020-06-13T16:15:21ZjugaAdapt bandwidth file classes to be compatible with stem (descriptors, etc) documentsso that the code can be moved to stem, as commented in #29056so that the code can be moved to stem, as commented in #29056sbws: unspecifiedjugajugahttps://gitlab.torproject.org/legacy/trac/-/issues/29291Work out a better way to choose the data size to download2020-06-13T16:15:24ZjugaWork out a better way to choose the data size to downloadInstead of starting to download 16MB and adjust the size depending on how long it takes to download until a maximum of 1GB, we could:
- Choose the initial size based on the bandwidth that the relay report
- (Try to download for a maximu...Instead of starting to download 16MB and adjust the size depending on how long it takes to download until a maximum of 1GB, we could:
- Choose the initial size based on the bandwidth that the relay report
- (Try to download for a maximum of 10secs? and) stop downloading data when we have learnt enoughsbws: 1.2.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/29292Document number of threads configuration depending on the machine available b...2020-06-13T16:15:25ZjugaDocument number of threads configuration depending on the machine available bandwidthFor instance, how many threads can we have when the machine available bandwidth is 100Mbps or 1Gbps.
Based on what we talked in https://trac.torproject.org/projects/tor/wiki/org/meetings/2019BrusselsNetworkTeam/Notes/SBWSRoadmap#QuestionsFor instance, how many threads can we have when the machine available bandwidth is 100Mbps or 1Gbps.
Based on what we talked in https://trac.torproject.org/projects/tor/wiki/org/meetings/2019BrusselsNetworkTeam/Notes/SBWSRoadmap#Questionssbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/29294Create an script to automate releases2020-06-13T16:15:26ZjugaCreate an script to automate releasessbws: unspecifiedjugajugahttps://gitlab.torproject.org/legacy/trac/-/issues/29355Include scanner nickname and UUID in the bandwidth file headers?2020-06-13T16:15:28ZjugaInclude scanner nickname and UUID in the bandwidth file headers?In #28741 the scanner sends HTTP metadata in every request to make it easier to debug issues, but this information is not include in the bandwidth file header.In #28741 the scanner sends HTTP metadata in every request to make it easier to debug issues, but this information is not include in the bandwidth file header.sbws: 1.2.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/29358Stop overloading the CPU when the test network is run in the integration tests2020-06-13T16:15:29ZjugaStop overloading the CPU when the test network is run in the integration testsIt might be caused by the test network relays' configuration.
In the case it's not possible to do not possible to overload the CPU, it's still possible to simplify the relays' configuration.It might be caused by the test network relays' configuration.
In the case it's not possible to do not possible to overload the CPU, it's still possible to simplify the relays' configuration.sbws: 1.2.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/29853sbws should show relays for diagnostics, even when MIN_REPORT has not been re...2020-06-13T16:15:47Zteorsbws should show relays for diagnostics, even when MIN_REPORT has not been reachedHi juga,
There's a serious bug in #28563: sbws is not reporting any relays until MIN_REPORT has been reached.
But we want sbws to always report every relay for diagnostics. If the bandwidth file doesn't have minimum_number_eligible_rel...Hi juga,
There's a serious bug in #28563: sbws is not reporting any relays until MIN_REPORT has been reached.
But we want sbws to always report every relay for diagnostics. If the bandwidth file doesn't have minimum_number_eligible_relays, they should all be marked as "under_min_report=1 vote=0".
https://github.com/torproject/sbws/blob/6070d36022dc2130518dd0c68332166b2bf76c73/sbws/lib/v3bwfile.py#L1352sbws: 1.1.x-finaljugajugahttps://gitlab.torproject.org/legacy/trac/-/issues/29854Missing diagnostic keys in relay lines, but the data is in the header2020-06-13T16:15:48ZteorMissing diagnostic keys in relay lines, but the data is in the headerThese keys are in the bandwidth file header, but there are not any similar keys in the relay lines:
* recent_priority_list_count
* recent_priority_relay_count
* recent_measurement_attempt_count
* recent_measurement_failure_count
Countin...These keys are in the bandwidth file header, but there are not any similar keys in the relay lines:
* recent_priority_list_count
* recent_priority_relay_count
* recent_measurement_attempt_count
* recent_measurement_failure_count
Counting the number of times that a relay was put in the priority list is useful, so we can find out if a relay is not being prioritised.
Similarly, counting the number of times that we can't measure a relay is also useful.
Can we have these keys in each relay line:
* relay_recent_measurement_attempt_count
* relay_recent_measurement_failure_countsbws: 1.1.x-finaljugajuga