Trac issueshttps://gitlab.torproject.org/legacy/trac/-/issues2020-06-13T16:15:05Zhttps://gitlab.torproject.org/legacy/trac/-/issues/28758Add requirements files2020-06-13T16:15:05ZanadahzAdd requirements filesAdd a requirements.txt file with the required sbws dependencies.Add a requirements.txt file with the required sbws dependencies.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28737Redesign sbws torrc option configuration2020-06-13T16:15:03ZteorRedesign sbws torrc option configurationsbws' torrc option handling is broken.
Here's a better scheme:
1. sbws config.default.ini and config.ini contain the following sections:
* tor.options.launch: a minimal set of options that must be configured when tor is launched. The ...sbws' torrc option handling is broken.
Here's a better scheme:
1. sbws config.default.ini and config.ini contain the following sections:
* tor.options.launch: a minimal set of options that must be configured when tor is launched. The minimal set contains the data directory config, control config, and log config. The network is disabled at launch. The initial options are:
```
'DataDirectory': conf.getpath('tor', 'datadir'),
'PidFile': conf.getpath('tor', 'pid'),
# Because we need things from full server descriptors (namely for now: the
# bandwidth line)
'UseMicrodescriptors': '0',
'ControlSocket': conf.getpath('tor', 'control_socket'),
# Easier than password authentication
'CookieAuthentication': '1',
'Log': [
'NOTICE file {}'.format(os.path.join(conf.getpath('tor', 'log'),
'notice.log')),
],
# useful logging options for clients that don't care about anonymity
'SafeLogging': '0',
'LogTimeGranularity': '1',
'ProtocolWarnings': '1',
'DisableNetwork': '1',
```
* tor.options.runtime.(group_name): Zero or more groups of tor options that can be set at runtime. The network is enabled at runtime. The "50-default" group of options is:
```
# We will find out via the ControlPort and not setting something static
# means a lower chance of conflict
'SocksPort': 'auto',
# To avoid path bias warnings
'UseEntryGuards': '0',
c.set_conf('__LeaveStreamsUnattached', '1')
# Things needed to make circuits fail a little faster. We get the
# circuit_timeout as a string instead of an int on purpose: stem only
# accepts strings.
'LearnCircuitBuildTimeout': '0',
'CircuitBuildTimeout': conf['general']['circuit_timeout'],
'DisableNetwork': '0',
```
* Zero or more tor.runtime.ignore_failure.(group name): tor options that are set in groups at runtime, but ignored if they fail. #28692 and #28694 will add options to this list.
Options in config.ini override options with the same name in sbws.default.ini, with + and / having the same meaning as in a torrc file (+ appends, / removes). Launch options are applied first, then runtime options are applied in group name order. Tor will make later runtime options replace earlier runtime options, and launch options.
2. sbws gets its control socket from the launch_options ControlSocket option(s)
3. sbws gets its data directory, pid, log(s), and circuit build timeout using GETCONF
4. For backwards compatibility:
* if tor.extra_lines is present, it should be appended to tor.options.launch. sbws' option merging code never worked, so appending shouldn't cause any more issues than the existing code.
* if any of these sbws options are present in an old config file, synthesise options.launch from the sbws options. If options.launch is also present in the same config file, it overrides the synthetic options. (sbws' option merging never worked for these options.)
```
'DataDirectory': conf.getpath('tor', 'datadir'),
'PidFile': conf.getpath('tor', 'pid'),
'ControlSocket': conf.getpath('tor', 'control_socket'),
'Log': [
'NOTICE file {}'.format(os.path.join(conf.getpath('tor', 'log'),
'notice.log')),
],
# Things needed to make circuits fail a little faster. We get the
# circuit_timeout as a string instead of an int on purpose: stem only
# accepts strings.
'CircuitBuildTimeout': conf['general']['circuit_timeout'],
```
The final option order is:
* sbws creates options.launch, using the last option from:
* synthetic legacy config options from config.default.ini
* options.launch from config.default.ini
* synthetic legacy config options from config.ini
* options.launch from config.ini
* sbws appends extra_lines to options.launch, using the last extra_lines from:
* config.default.ini
* config.ini
* sbws creates options.runtime groups, using the last one of each group from:
* config.default.ini
* config.ini
sbws launches tor with options.launch, then applies each group of options.runtime in order.
This is a new feature, so it should go in sbws 1.1.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28718Simplify configuration options2020-06-13T16:15:01ZjugaSimplify configuration optionsPrograms should allow in this order: cli options, user configuration files passed by cli, environment variables, system configuration files and program defaults.
There seems there's not a common easy way to achieve this in Python.
sbws a...Programs should allow in this order: cli options, user configuration files passed by cli, environment variables, system configuration files and program defaults.
There seems there's not a common easy way to achieve this in Python.
sbws accepts cli and user configuration files in a complicated way.
The program defaults are provided as a configuration file which might not be that convenient seems confuses where to find the defaults.
Maybe an external library like https://github.com/bw2/ConfigArgParse (supported in Debian) or some other design can help to simplify this.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28712Make conf['tor']['extra_lines'] into a torrc file2020-06-13T16:15:00ZteorMake conf['tor']['extra_lines'] into a torrc filesbws tries to parse extra torrc options from its config file, and add them to the existing options. But this code is buggy: it tries to merge extra options with existing options with the same name. That doesn't work if the option is only...sbws tries to parse extra torrc options from its config file, and add them to the existing options. But this code is buggy: it tries to merge extra options with existing options with the same name. That doesn't work if the option is only allowed to occur once in the torrc file.
Tor can get options from these places, in this order:
* torrc-defaults file
* torrc file
* command-line (stem's launch command)
* control port (stem's control port command)
Here's a better design for sbws:
* hard-coded options in a torrc-defaults file
* extra user options in a torrc file
* configured options on the command-line
* runtime options over the control port
This means that:
* users can override hard-coded options using the torrc file
* users can modify configured options using the sbws config.ini
* users can't modify runtime options (for simplicity)sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28706Maybe implement resolving destination domain using Tor's RESOLVE and ADDRMAP ...2020-06-13T16:14:59ZjugaMaybe implement resolving destination domain using Tor's RESOLVE and ADDRMAP eventsIn #28458 the domain of the destination was being resolved locally to check whether and exit policy allows to exit to the IP, which had 2 problems:
- in the case that the destination is a CDN, the IP resolved locally would be different t...In #28458 the domain of the destination was being resolved locally to check whether and exit policy allows to exit to the IP, which had 2 problems:
- in the case that the destination is a CDN, the IP resolved locally would be different to the IP resolved by the exit.
- it was returning the first IP found, without checking whether the scanner supported IPv6.
The correct way would be to resolve the domain via Tor itself using RESOLVE and ADDRMAP events with that exit.
While there are not too many circuits that fails (because the policy doesn't allow to exit to the destination IP), this is not a prioritysbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28684Make sbws easy to understand and maintain2020-06-13T16:14:56ZjugaMake sbws easy to understand and maintainThere was not an initial design and we are still adding functionality. Some of the new functionality require modifying several parts of the code and it's becoming harder to maintain and understand.
This ticket is to identify the changes ...There was not an initial design and we are still adding functionality. Some of the new functionality require modifying several parts of the code and it's becoming harder to maintain and understand.
This ticket is to identify the changes needed (refactors, etc.).sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28667Obtain the new descriptors and network status documents by signals instead of...2022-03-03T15:23:50ZjugaObtain the new descriptors and network status documents by signals instead of every 5minThe network status and descriptors are obtained every 5min https://gitweb.torproject.org/sbws.git/tree/sbws/lib/relaylist.py#n144.
Is there any reason for that?.
I think torflow obtain them as they come.
It could be done with stem using:...The network status and descriptors are obtained every 5min https://gitweb.torproject.org/sbws.git/tree/sbws/lib/relaylist.py#n144.
Is there any reason for that?.
I think torflow obtain them as they come.
It could be done with stem using:
```
controller.add_event_listener(
listener_fn, EventType.NS, EventType.NEWDESC)
```
Also, instead overwriting those values, Relay could have a dictionary (key would be the timestamp) with a subset of the attributes of document, in case we might need to calculate median/mean of those attributes
Edit: fix typosbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28589Open trac tickets for every open sbws GitHub issue2020-06-13T16:14:41ZteorOpen trac tickets for every open sbws GitHub issueWe still have some open issues in:
https://github.com/torproject/sbws/issues/
Let's copy them across to trac, or close them.We still have some open issues in:
https://github.com/torproject/sbws/issues/
Let's copy them across to trac, or close them.sbws: unspecifiedjugajugahttps://gitlab.torproject.org/legacy/trac/-/issues/28545Use an 11 second download in sbws2020-06-13T16:14:31ZteorUse an 11 second download in sbwsIn #22453, we might remove relay bandwidth self-tests in Tor. So we want sbws to send bandwidth for Tor's entire 10 second bandwidth interval.
Let's set the download_target to 11 seconds (to cover the 10 second interval), and adjust the...In #22453, we might remove relay bandwidth self-tests in Tor. So we want sbws to send bandwidth for Tor's entire 10 second bandwidth interval.
Let's set the download_target to 11 seconds (to cover the 10 second interval), and adjust the min to 7 and the max to 14?sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28519Prioritize relays with no measured bandwidth in the consensus2020-06-13T16:14:30ZjugaPrioritize relays with no measured bandwidth in the consensussbws currently prioritizes relays for which it has not measurements.
It should also prioritize relays with no measure bandwidth in the consensus (https://trac.torproject.org/projects/tor/ticket/22453#comment:33).
Maybe it should also pri...sbws currently prioritizes relays for which it has not measurements.
It should also prioritize relays with no measure bandwidth in the consensus (https://trac.torproject.org/projects/tor/ticket/22453#comment:33).
Maybe it should also prioritize new relays (uptime).sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28463Measure exits as non-exits with 50% probability2020-06-13T16:15:37ZteorMeasure exits as non-exits with 50% probabilityIf an exit can't be measured as an exit for any reason, we still want to measure it.
If we choose to measure exits as non-exits with 50% probability, then non-working exits will still get measured.
This might be a bad thing: maybe we w...If an exit can't be measured as an exit for any reason, we still want to measure it.
If we choose to measure exits as non-exits with 50% probability, then non-working exits will still get measured.
This might be a bad thing: maybe we want exits that don't work as exits to have low bandwidths.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28447improve new SBWS rounding to exhibit uniform percent deltas2020-06-13T16:14:23Zstarlightimprove new SBWS rounding to exhibit uniform percent deltasIn ticket #27337 bandwidth vote rounding was changed from Torflow's three significant decimal digits to two places in order to reduce inconsequential consensus diff updates. The change is a good one except that the resulting distributio...In ticket #27337 bandwidth vote rounding was changed from Torflow's three significant decimal digits to two places in order to reduce inconsequential consensus diff updates. The change is a good one except that the resulting distribution of values exhibits large unevenness of relative value changes, an effect which with was much smaller with three digit rounding. To obtain the same benefit with an uniform change distribution, rounding constructed on a natural log scale should be applied. For example instead of
```
100000 -> 110000 +10.0%
40000 -> 41000 +2.5%
```
the proper way to accomplish this is to round by taking the natural log of the value, round that to the nearest 1/20th for uniform 5% steps and then raise `e` by the result:
```
100000 rounds to 98716
105000 rounds to 103777
98716 -> 103777 +5.1%
40000 rounds to 40135
42000 rounds to 42193
40135 -> 42193 +5.1%
1100 rounds to 1096
1050 rounds to 1043
1043 -> 1096 +5.1%
```
The overall goal of SBWS is improved consensus construction and while the importance of this refinement can be debated, it is clearly beneficial, is easy to implement and therefore a reasonable adjustment at this early stage of testing deployment.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28282Refactor bandwidth file generation code2020-06-13T16:14:19ZjugaRefactor bandwidth file generation codeInitially, `v3bwfile.py` was just converting measurement `Result`s to a `Bandwidth file` lines format.
Then we started to apply scaling methods and filtering the measurements to satisfy certain restrictions.
To don not have to modify cod...Initially, `v3bwfile.py` was just converting measurement `Result`s to a `Bandwidth file` lines format.
Then we started to apply scaling methods and filtering the measurements to satisfy certain restrictions.
To don not have to modify code in other file/classes, all of that was implemented in `v3bwfile.py` classes/methods, but it should be moved to either an intermediate file/classes that deal with the statistics calculations (and not the format) or modify `Results` to be able to perform those calculations.
Edit: fix typosbws: unspecifiedjugajugahttps://gitlab.torproject.org/legacy/trac/-/issues/28106Change integration tests from bash to shell2020-06-13T16:14:12ZjugaChange integration tests from bash to shellgman999 reported that tests need to change from bash to shell to include them in openbsdgman999 reported that tests need to change from bash to shell to include them in openbsdsbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28099Make a policy for adding new sbws relay keys2020-06-13T16:14:11ZteorMake a policy for adding new sbws relay keysSplit from #28085:
> I think we should also come up with some policy to add/remove key/values.
> Even if all the ones we have so far seem useful, we are not using it.
I don't understand. Do you mean:
There are some keys in the spec th...Split from #28085:
> I think we should also come up with some policy to add/remove key/values.
> Even if all the ones we have so far seem useful, we are not using it.
I don't understand. Do you mean:
There are some keys in the spec that are not in the code.
Or
There are some keys in the code that are not in the spec.
Or
There are some keys in the spec and the code that are not being used by anyone.
> For instance, do we have tickets of cases where these key/values were needed/requested?.
When we find this information, we should update the spec so it tells us why we need each value.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28087Investigate circuit timeout times and if sbws is properly cleaning up circuit...2020-06-13T16:14:10ZpastlyInvestigate circuit timeout times and if sbws is properly cleaning up circuits when it gives up on themAnd it seems to be garbage circuits, or circuits it gave up on but didn't clean up.
Here is the output of a script that shows the in-use circuits. 6 measurements happening. **43** circuits in existence??? What the hell is that???
```
*...And it seems to be garbage circuits, or circuits it gave up on but didn't clean up.
Here is the output of a script that shows the in-use circuits. 6 measurements happening. **43** circuits in existence??? What the hell is that???
```
* 21983 tornodenumber9004 Quintex42 (185.21.216.198:443)
* 21994 romero niftyllipika (185.21.216.198:443)
* 21995 ffffm CalyxInstitute09 (185.21.216.198:443)
* 21996 AgentBolek Unnamed (185.21.216.198:443)
* 21997 BeastieJoy61 whatconfig (185.21.216.198:443)
* 21998 blueberryTORnode1 anonymous3 (185.21.216.198:443)
6 streams; 6/43 circuits in use
```sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28076When sbws has measured less than 60% of relays, write a header with progress2020-06-13T16:14:09ZteorWhen sbws has measured less than 60% of relays, write a header with progressIn #28062, we remove the link to the bandwidth file when fewer than 60% of relays are measured.
Instead, we could write a bandwidth file containing a header field with:
* number of relays measured / target number of relays / total numbe...In #28062, we remove the link to the bandwidth file when fewer than 60% of relays are measured.
Instead, we could write a bandwidth file containing a header field with:
* number of relays measured / target number of relays / total number of relays
* percentage of relays measured / target percentage
Edit: the bandwidth file would not contain any relays - just the headersbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/27791sbws: compare relays against other similar relays2020-06-13T16:14:00Zteorsbws: compare relays against other similar relaysFrom https://trac.torproject.org/projects/tor/ticket/25687#comment:13
Evaluating relays against others of the same class seems a good idea. (I.e. process guards, exits and unflagged middle relays as isolated collections.) Torflow impl...From https://trac.torproject.org/projects/tor/ticket/25687#comment:13
Evaluating relays against others of the same class seems a good idea. (I.e. process guards, exits and unflagged middle relays as isolated collections.) Torflow implemented it when PID logic was active, but presently it is disabled.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/27790sbws: design and construct bias curves2020-06-13T16:14:00Zteorsbws: design and construct bias curvesFrom https://trac.torproject.org/projects/tor/ticket/25687#comment:13
The essence of Torflow's active approach is that observed bandwidth capacity at each relay is the key measurement and that it can only be reliably determined locally ...From https://trac.torproject.org/projects/tor/ticket/25687#comment:13
The essence of Torflow's active approach is that observed bandwidth capacity at each relay is the key measurement and that it can only be reliably determined locally but that it requires adjustment, principally to account for used vs unused capacity and secondly the relative performance of each node in the asymmetric domain of internet traffic routing. IMO indisputably correct. The Peerflow paper tacitly recognizes this.
However the simple linear adjustment algorithm cannot be fine-tuned for better results across the vast range of relay performance. IIRC polynomial equations of sufficient order can describe curves of near arbitrary complexity and therefore parameterized polynomials can be used interactively, in a gradual empirical search, to describe an improving set of adjustment biases for applying scanner measurements to advertised bandwidths. This link illustrates the general principal, though the idea is to design and construct bias curves with polynomials rather then to fit them somehow.
​https://en.wikipedia.org/wiki/Polynomial_regressionsbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/27789sbws: use a decaying average for measured and observed bandwidths2020-06-13T16:13:59Zteorsbws: use a decaying average for measured and observed bandwidthsSplit off #27346:
use a decaying average for measured and observed bandwidths, because:
* recent bandwidths are closer to the relay's current capacitySplit off #27346:
use a decaying average for measured and observed bandwidths, because:
* recent bandwidths are closer to the relay's current capacitysbws: unspecified