Trac issueshttps://gitlab.torproject.org/legacy/trac/-/issues2020-06-13T16:15:25Zhttps://gitlab.torproject.org/legacy/trac/-/issues/29292Document number of threads configuration depending on the machine available b...2020-06-13T16:15:25ZjugaDocument number of threads configuration depending on the machine available bandwidthFor instance, how many threads can we have when the machine available bandwidth is 100Mbps or 1Gbps.
Based on what we talked in https://trac.torproject.org/projects/tor/wiki/org/meetings/2019BrusselsNetworkTeam/Notes/SBWSRoadmap#QuestionsFor instance, how many threads can we have when the machine available bandwidth is 100Mbps or 1Gbps.
Based on what we talked in https://trac.torproject.org/projects/tor/wiki/org/meetings/2019BrusselsNetworkTeam/Notes/SBWSRoadmap#Questionssbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/33473Document which branches developers should base their patches on2020-06-13T16:16:30ZjugaDocument which branches developers should base their patches onsbws: unspecifiedjugajugahttps://gitlab.torproject.org/legacy/trac/-/issues/28985Does sbws need Tor to report observed bandwidths more often?2020-06-13T16:15:17ZteorDoes sbws need Tor to report observed bandwidths more often?In #23856 in 2017, we updated the observed bandwidth to 5 days from 1 day.
We should review this change, and decide if Tor should report observed bandwidths more frequently.
In #28983, we'll work out how long it takes sbws to measure mo...In #23856 in 2017, we updated the observed bandwidth to 5 days from 1 day.
We should review this change, and decide if Tor should report observed bandwidths more frequently.
In #28983, we'll work out how long it takes sbws to measure most of the network. We could use this time as a guide: Tor probably shouldn't report observed bandwidths faster than sbws can measure them. But Tor should report observed bandwidths every few times sbws measures Tor.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/29892Easy deployments in test server running in public network2020-06-13T16:15:48ZjugaEasy deployments in test server running in public networkSome bugs can only be detected after running the scanner several hours in the public network.
Having a fabfile or similar would easir that deploying automatically in a test server.Some bugs can only be detected after running the scanner several hours in the public network.
Having a fabfile or similar would easir that deploying automatically in a test server.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/4708Implement bwauth cap for latency2022-03-01T15:29:15ZMike PerryImplement bwauth cap for latencyRobert, Sebastian and I hashed out an idea for another feedback mechanism for the bw auths based on latency.
Basically, the idea is to create another cap called latency_error similar to how we use circ_error. If a node's latency exceeds...Robert, Sebastian and I hashed out an idea for another feedback mechanism for the bw auths based on latency.
Basically, the idea is to create another cap called latency_error similar to how we use circ_error. If a node's latency exceeds some quantile of the population (by being higher than the latencies of say 75% of all nodes), we would then compute and use a pid_error-style error value based on the distance from this 75% quantile setpoint, and use it if it is a more negative number than pid_error and circ_error.
I think the way we want to measure this latency is from CREATE to STREAM FAILED EXITPOLICY for a circuit creation + stream failure for a 1-hop stream exit attempt to localhost. This way we measure both cryptoworker queue latency as well as orconn, circuit, and stream latency.
I think the simplest way to build this is as a separate process from bwauthority.py that simply builds 1-hop circuits and attempts to exit to localhost from them. It would then output a separate, additional measurement file for the network that would be read in by aggregate.py, and used to compute latency_error.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/4709Implement bwauth cap for TCP socket exhaustion2020-06-13T16:13:22ZMike PerryImplement bwauth cap for TCP socket exhaustionStep 0 is to determine if any of the Gbit+ tor relays (especially Guard+Exit nodes) ever come close to running out of TCP sockets.
Step 1 is find some way to measure stream failures from the bwauths, compute a stream_error value, and us...Step 0 is to determine if any of the Gbit+ tor relays (especially Guard+Exit nodes) ever come close to running out of TCP sockets.
Step 1 is find some way to measure stream failures from the bwauths, compute a stream_error value, and use it. See #4708 for more details on that.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/27365Implement sbws features in the tor authority code2020-06-13T16:13:52ZteorImplement sbws features in the tor authority codesbws and torflow measure relay bandwidth, and then process the results. Some of this processing could be implemented on tor directory authorities instead. The authority implementation would be more consistent, and reduce the amount of ef...sbws and torflow measure relay bandwidth, and then process the results. Some of this processing could be implemented on tor directory authorities instead. The authority implementation would be more consistent, and reduce the amount of effort required to implement new bandwidth measurement tools.
Here are some potential authority features:
* an absolute node consensus weight cap
* a relative node consensus weight cap
* the MaxAdvertisedBandwidth cap
* other post-processing in the bandwidth file spec:
https://gitweb.torproject.org/torspec.git/tree/bandwidth-file-spec.txtsbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/29047Improve code style following PEP8 and PEP2572020-06-13T16:15:19ZjugaImprove code style following PEP8 and PEP257sbws: unspecifiedjugajugahttps://gitlab.torproject.org/legacy/trac/-/issues/28447improve new SBWS rounding to exhibit uniform percent deltas2020-06-13T16:14:23Zstarlightimprove new SBWS rounding to exhibit uniform percent deltasIn ticket #27337 bandwidth vote rounding was changed from Torflow's three significant decimal digits to two places in order to reduce inconsequential consensus diff updates. The change is a good one except that the resulting distributio...In ticket #27337 bandwidth vote rounding was changed from Torflow's three significant decimal digits to two places in order to reduce inconsequential consensus diff updates. The change is a good one except that the resulting distribution of values exhibits large unevenness of relative value changes, an effect which with was much smaller with three digit rounding. To obtain the same benefit with an uniform change distribution, rounding constructed on a natural log scale should be applied. For example instead of
```
100000 -> 110000 +10.0%
40000 -> 41000 +2.5%
```
the proper way to accomplish this is to round by taking the natural log of the value, round that to the nearest 1/20th for uniform 5% steps and then raise `e` by the result:
```
100000 rounds to 98716
105000 rounds to 103777
98716 -> 103777 +5.1%
40000 rounds to 40135
42000 rounds to 42193
40135 -> 42193 +5.1%
1100 rounds to 1096
1050 rounds to 1043
1043 -> 1096 +5.1%
```
The overall goal of SBWS is improved consensus construction and while the importance of this refinement can be debated, it is clearly beneficial, is easy to implement and therefore a reasonable adjustment at this early stage of testing deployment.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/27346Improve sbws bandwidth accuracy2020-06-13T16:13:49ZteorImprove sbws bandwidth accuracyBetter designs SHOULD:
* use at least 4 measurements that are at least 6 hours apart, because:
- there is a daily cycle
- each day contains 2 similar points in the cycle (it is an up and down cycle)
- if all 4 measurements happen w...Better designs SHOULD:
* use at least 4 measurements that are at least 6 hours apart, because:
- there is a daily cycle
- each day contains 2 similar points in the cycle (it is an up and down cycle)
- if all 4 measurements happen within a few hours, they will still be biased
* use at least 3 days of observed bandwidths, because:
- a single download at the changeover point can affect 2 days
* weight bandwidths based on the time since the last bandwidth, because:
- if we only record bandwidths when they change, bandwidths that are updated soon after the last bandwidth are weighted too high
- we can either:
* record the bandwidths every hour, even if they haven't changed
* weight each bandwidth by the time since the last bandwidth
* use a decaying average for measured and observed bandwidths, because:
- recent bandwidths are closer to the relay's current capacity
- and we want accurate resultssbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/29718Include a refactor plan2020-06-13T16:15:36ZjugaInclude a refactor planThe parent ticket have some children tickets and we collected some ideas in https://pad.riseup.net/p/rGfvR7ZsvtoZ, but there are other ideas not collected.The parent ticket have some children tickets and we collected some ideas in https://pad.riseup.net/p/rGfvR7ZsvtoZ, but there are other ideas not collected.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28087Investigate circuit timeout times and if sbws is properly cleaning up circuit...2020-06-13T16:14:10ZpastlyInvestigate circuit timeout times and if sbws is properly cleaning up circuits when it gives up on themAnd it seems to be garbage circuits, or circuits it gave up on but didn't clean up.
Here is the output of a script that shows the in-use circuits. 6 measurements happening. **43** circuits in existence??? What the hell is that???
```
*...And it seems to be garbage circuits, or circuits it gave up on but didn't clean up.
Here is the output of a script that shows the in-use circuits. 6 measurements happening. **43** circuits in existence??? What the hell is that???
```
* 21983 tornodenumber9004 Quintex42 (185.21.216.198:443)
* 21994 romero niftyllipika (185.21.216.198:443)
* 21995 ffffm CalyxInstitute09 (185.21.216.198:443)
* 21996 AgentBolek Unnamed (185.21.216.198:443)
* 21997 BeastieJoy61 whatconfig (185.21.216.198:443)
* 21998 blueberryTORnode1 anonymous3 (185.21.216.198:443)
6 streams; 6/43 circuits in use
```sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28099Make a policy for adding new sbws relay keys2020-06-13T16:14:11ZteorMake a policy for adding new sbws relay keysSplit from #28085:
> I think we should also come up with some policy to add/remove key/values.
> Even if all the ones we have so far seem useful, we are not using it.
I don't understand. Do you mean:
There are some keys in the spec th...Split from #28085:
> I think we should also come up with some policy to add/remove key/values.
> Even if all the ones we have so far seem useful, we are not using it.
I don't understand. Do you mean:
There are some keys in the spec that are not in the code.
Or
There are some keys in the code that are not in the spec.
Or
There are some keys in the spec and the code that are not being used by anyone.
> For instance, do we have tickets of cases where these key/values were needed/requested?.
When we find this information, we should update the spec so it tells us why we need each value.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28712Make conf['tor']['extra_lines'] into a torrc file2020-06-13T16:15:00ZteorMake conf['tor']['extra_lines'] into a torrc filesbws tries to parse extra torrc options from its config file, and add them to the existing options. But this code is buggy: it tries to merge extra options with existing options with the same name. That doesn't work if the option is only...sbws tries to parse extra torrc options from its config file, and add them to the existing options. But this code is buggy: it tries to merge extra options with existing options with the same name. That doesn't work if the option is only allowed to occur once in the torrc file.
Tor can get options from these places, in this order:
* torrc-defaults file
* torrc file
* command-line (stem's launch command)
* control port (stem's control port command)
Here's a better design for sbws:
* hard-coded options in a torrc-defaults file
* extra user options in a torrc file
* configured options on the command-line
* runtime options over the control port
This means that:
* users can override hard-coded options using the torrc file
* users can modify configured options using the sbws config.ini
* users can't modify runtime options (for simplicity)sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28684Make sbws easy to understand and maintain2020-06-13T16:14:56ZjugaMake sbws easy to understand and maintainThere was not an initial design and we are still adding functionality. Some of the new functionality require modifying several parts of the code and it's becoming harder to maintain and understand.
This ticket is to identify the changes ...There was not an initial design and we are still adding functionality. Some of the new functionality require modifying several parts of the code and it's becoming harder to maintain and understand.
This ticket is to identify the changes needed (refactors, etc.).sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/34310Maybe add new type of errors in the bandwidth file2020-06-13T16:16:42ZjugaMaybe add new type of errors in the bandwidth fileAs commented in https://trac.torproject.org/projects/tor/ticket/30905#comment:9
Things that are not really failures:
when loading the results, the relays that changed ip are excluded. I could add a patch to count all the results fi...As commented in https://trac.torproject.org/projects/tor/ticket/30905#comment:9
Things that are not really failures:
when loading the results, the relays that changed ip are excluded. I could add a patch to count all the results first, which reduces the number of failures in around 1000.
when sbws is restarted, all the queued relays to measure that didn't start to be measured yet, are not saved in the results, but the attempt has already been count. We could also add a patch to count the attempt in measure_relay, instead of main_loop, though then we won't know the number of attempts counting from the moment they were queued.
real failures would happen when result_putter_error is triggered, or some bug makes sbws stall, which then will print the TICKET_MSG error. In the 1st case, we could actually count an error.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28706Maybe implement resolving destination domain using Tor's RESOLVE and ADDRMAP ...2020-06-13T16:14:59ZjugaMaybe implement resolving destination domain using Tor's RESOLVE and ADDRMAP eventsIn #28458 the domain of the destination was being resolved locally to check whether and exit policy allows to exit to the IP, which had 2 problems:
- in the case that the destination is a CDN, the IP resolved locally would be different t...In #28458 the domain of the destination was being resolved locally to check whether and exit policy allows to exit to the IP, which had 2 problems:
- in the case that the destination is a CDN, the IP resolved locally would be different to the IP resolved by the exit.
- it was returning the first IP found, without checking whether the scanner supported IPv6.
The correct way would be to resolve the domain via Tor itself using RESOLVE and ADDRMAP events with that exit.
While there are not too many circuits that fails (because the policy doesn't allow to exit to the destination IP), this is not a prioritysbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28463Measure exits as non-exits with 50% probability2020-06-13T16:15:37ZteorMeasure exits as non-exits with 50% probabilityIf an exit can't be measured as an exit for any reason, we still want to measure it.
If we choose to measure exits as non-exits with 50% probability, then non-working exits will still get measured.
This might be a bad thing: maybe we w...If an exit can't be measured as an exit for any reason, we still want to measure it.
If we choose to measure exits as non-exits with 50% probability, then non-working exits will still get measured.
This might be a bad thing: maybe we want exits that don't work as exits to have low bandwidths.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/4359Minimize time between new relay appearing and having some bw vote for it2020-06-13T16:13:21ZRoger DingledineMinimize time between new relay appearing and having some bw vote for itIn #2286 I point out a huge security problem in Tor, which is that new relays can lie about their bandwidth and get away with it.
One of the components of my suggested fix is to minimize the period of time between when a new relay appea...In #2286 I point out a huge security problem in Tor, which is that new relays can lie about their bandwidth and get away with it.
One of the components of my suggested fix is to minimize the period of time between when a new relay appears in the network, and when we have an opinion about its bandwidth.
So it would be great for the bwauths to recognize new relays and schedule them for high-priority tests.
Or is that already done? What's the expected turnaround time?
This ticket is perhaps related to #2550.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28990Move all or part of the documentation about Tor and the bandwidth scanner2020-06-13T16:15:18ZjugaMove all or part of the documentation about Tor and the bandwidth scannerI've writing documentation in https://onbasca.readthedocs.io/ that i didn't know where else could go.
Some of it might be useful only for me but some might be useful for other people and should be in Tor project domain.I've writing documentation in https://onbasca.readthedocs.io/ that i didn't know where else could go.
Some of it might be useful only for me but some might be useful for other people and should be in Tor project domain.sbws: unspecified