Trac issueshttps://gitlab.torproject.org/legacy/trac/-/issues2020-06-13T16:19:10Zhttps://gitlab.torproject.org/legacy/trac/-/issues/8688bwauths need to upgrade (to start measuring even non-Fast relays)2020-06-13T16:19:10ZRoger Dingledinebwauths need to upgrade (to start measuring even non-Fast relays)With the recent commit of #8435 to Tor, directory authorities will leave off the Fast flag from any non-measured relay.
I believe bwauths currently don't measure relays that don't have the Fast flag.
Bad cycle we're about to have here....With the recent commit of #8435 to Tor, directory authorities will leave off the Fast flag from any non-measured relay.
I believe bwauths currently don't measure relays that don't have the Fast flag.
Bad cycle we're about to have here.
See https://trac.torproject.org/projects/tor/ticket/8273#comment:6 and comments below it for context. Aaron has a proposed patch.Aaron GibsonAaron Gibsonhttps://gitlab.torproject.org/legacy/trac/-/issues/8684bwauth files don't include opinions about Authorities2020-06-13T16:19:09ZRoger Dingledinebwauth files don't include opinions about AuthoritiesIt appears that moria1's bwauth doesn't provide an opinion about moria1, or any authority for that matter.
And no authorities provide Measured lines for turtles.
I suspect there's code in the bwauth to skip measuring authorities.
That...It appears that moria1's bwauth doesn't provide an opinion about moria1, or any authority for that matter.
And no authorities provide Measured lines for turtles.
I suspect there's code in the bwauth to skip measuring authorities.
That's going to be bad now that we've turned on #8435.
The right fix might be to change the bwauths. But for now I'm filing as a Tor bug until we figure out where to fix it.https://gitlab.torproject.org/legacy/trac/-/issues/8417BwAuthority doesn't come with bwfiles file, and so makes no progress.2020-06-13T16:19:07ZRoger DingledineBwAuthority doesn't come with bwfiles file, and so makes no progress.I've been running my new freshly-checked-out bwauth for six days, and my periodic cron job tells me:
```
NOTICE[Mon Mar 04 06:45:01 2013]:No scan results yet.
```
Mike tells me this means it is making no progress.
I have frequent trace...I've been running my new freshly-checked-out bwauth for six days, and my periodic cron job tells me:
```
NOTICE[Mon Mar 04 06:45:01 2013]:No scan results yet.
```
Mike tells me this means it is making no progress.
I have frequent tracebacks in the logs:
```
ERROR[Wed Mar 06 17:31:35 2013]:An unexpected error occured.
Traceback (most recent call last):
File "bwauthority_child.py", line 391, in <module>
else: main(sys.argv)
File "bwauthority_child.py", line 331, in main
min_streams, sql_file)
File "bwauthority_child.py", line 235, in speedrace
url = choose_url(start_pct)
File "bwauthority_child.py", line 105, in choose_url
f = file("./data/bwfiles", "r")
IOError: [Errno 2] No such file or directory: './data/bwfiles'
```
This data/bwfiles file is nowhere to be found. It looks like write_file_list() in aggregate.py is what's supposed to generate it. But maybe we only call that function once we have working stats?Aaron GibsonAaron Gibsonhttps://gitlab.torproject.org/legacy/trac/-/issues/8399Torflow traceback: ZeroDivisionError: float division2020-06-13T16:19:07ZRoger DingledineTorflow traceback: ZeroDivisionError: float division```
Traceback (most recent call last):
File "../../TorCtl/TorCtl.py", line 720, in _eventLoop
self._handleFn(timestamp, reply)
File "../../TorCtl/TorCtl.py", line 1346, in _handle1
self._map1.get(event.event_name, self.unknow...```
Traceback (most recent call last):
File "../../TorCtl/TorCtl.py", line 720, in _eventLoop
self._handleFn(timestamp, reply)
File "../../TorCtl/TorCtl.py", line 1346, in _handle1
self._map1.get(event.event_name, self.unknown_event)(event)
File "../../TorCtl/PathSupport.py", line 1837, in new_consensus_event
TorCtl.ConsensusTracker.new_consensus_event(self, n)
File "../../TorCtl/TorCtl.py", line 1694, in new_consensus_event
self._read_routers(self.ns_map.values())
File "../../TorCtl/TorCtl.py", line 1653, in _read_routers
ratio_r.sort(lambda x, y: cmp(float(y.bw)/y.get_unmeasured_bw(),
File "../../TorCtl/TorCtl.py", line 1654, in <lambda>
float(x.bw)/x.get_unmeasured_bw()))
ZeroDivisionError: float division
```
Preceded by
```
WARN[Thu Feb 28 23:02:26 2013]:No event for: 650 NEWCONSENSUS
```
and post-ceded by
```
NOTICE[Thu Feb 28 23:02:26 2013]:Tor closed control connection. Exiting event thread.
WARN[Thu Feb 28 23:02:26 2013]:Child process recieved SIGTERM
INFO[Thu Feb 28 23:02:26 2013]:Beginning time loop
DEBUG[Thu Feb 28 23:02:27 2013]:Child Process Spawning...
INFO[Thu Feb 28 23:02:27 2013]:Connecting to Tor at 127.0.0.1:9111
```
so it looks like it took down something-er-other too.Aaron GibsonAaron Gibsonhttps://gitlab.torproject.org/legacy/trac/-/issues/8372IOError: [Errno 2] No such file or directory: './data/bwfiles'2020-06-13T16:19:07ZRoger DingledineIOError: [Errno 2] No such file or directory: './data/bwfiles'```
ERROR[Thu Feb 28 21:40:33 2013]:An unexpected error occured.
Traceback (most recent call last):
File "bwauthority_child.py", line 391, in <module>
else: main(sys.argv)
File "bwauthority_child.py", line 331, in main
min_st...```
ERROR[Thu Feb 28 21:40:33 2013]:An unexpected error occured.
Traceback (most recent call last):
File "bwauthority_child.py", line 391, in <module>
else: main(sys.argv)
File "bwauthority_child.py", line 331, in main
min_streams, sql_file)
File "bwauthority_child.py", line 235, in speedrace
url = choose_url(start_pct)
File "bwauthority_child.py", line 105, in choose_url
f = file("./data/bwfiles", "r")
IOError: [Errno 2] No such file or directory: './data/bwfiles'
```
Running torflow caf001962.
(Maybe it's ok? But in that case, maybe it shouldn't say that it's an unexpected error?)Aaron GibsonAaron Gibsonhttps://gitlab.torproject.org/legacy/trac/-/issues/8371UnboundLocalError: local variable 'hdlr' referenced before assignment2020-06-13T16:19:06ZRoger DingledineUnboundLocalError: local variable 'hdlr' referenced before assignment```
ERROR[Thu Feb 28 21:33:17 2013]:An unexpected error occured.
Traceback (most recent call last):
File "bwauthority_child.py", line 391, in <module>
else: main(sys.argv)
File "bwauthority_child.py", line 313, in main
hdlr.a...```
ERROR[Thu Feb 28 21:33:17 2013]:An unexpected error occured.
Traceback (most recent call last):
File "bwauthority_child.py", line 391, in <module>
else: main(sys.argv)
File "bwauthority_child.py", line 313, in main
hdlr.attach_sql_listener('sqlite://')
UnboundLocalError: local variable 'hdlr' referenced before assignment
```
Running Torflow caf001962.Aaron GibsonAaron Gibsonhttps://gitlab.torproject.org/legacy/trac/-/issues/8370torflow checks out old torctl as submodule2020-06-13T16:19:06ZRoger Dingledinetorflow checks out old torctl as submodule```
bwauth@moria:~/work$ git clone https://git.torproject.org/git/torflow.git
Cloning into torflow...
remote: Counting objects: 3757, done.
remote: Compressing objects: 100% (1055/1055), done.
remote: Total 3757 (delta 2668), reused 3718...```
bwauth@moria:~/work$ git clone https://git.torproject.org/git/torflow.git
Cloning into torflow...
remote: Counting objects: 3757, done.
remote: Compressing objects: 100% (1055/1055), done.
remote: Total 3757 (delta 2668), reused 3718 (delta 2647)
Receiving objects: 100% (3757/3757), 1.13 MiB, done.
Resolving deltas: 100% (2668/2668), done.
bwauth@moria:~/work$ cd torflow
bwauth@moria:~/work/torflow$ ./add_torctl.sh
Submodule 'TorCtl' (https://git.torproject.org/pytorctl.git) registered for path 'TorCtl'
Cloning into TorCtl...
remote: Counting objects: 725, done.
remote: Compressing objects: 100% (236/236), done.
remote: Total 725 (delta 491), reused 718 (delta 488)
Receiving objects: 100% (725/725), 187.32 KiB, done.
Resolving deltas: 100% (491/491), done.
Submodule path 'TorCtl': checked out '305a759d99dd01f60faed9aa036b37746d3c54c5'
```
305a759 is way old (the head is from Apr 2012). For example, it doesn't include aagbsn's fix from today.
What am I doing wrong?Aaron GibsonAaron Gibsonhttps://gitlab.torproject.org/legacy/trac/-/issues/7281Bandwidth auths should publish average and weighted onionskin failure rates2020-06-13T16:19:06ZMike PerryBandwidth auths should publish average and weighted onionskin failure ratesWhile testing for path bias rates, I've noticed that the onionskin failure rate in the network can vary as much as +/- 30% in the matter of a few hours. This is rather insane.
We should publish statistics on this failure rate over time ...While testing for path bias rates, I've noticed that the onionskin failure rate in the network can vary as much as +/- 30% in the matter of a few hours. This is rather insane.
We should publish statistics on this failure rate over time somehow. This might be a metrics task, but if the bandwidth auths do it, we can have them set consensus parameters for the path bias code to decide on how loudly to scream and when (see Proposal 209).
For that, I think I want network-wide averages, as well as consensus-bandwidth weighted averages. The path bias levels should be set using consensus-bandwidth weighted values.
See also #5457, but I don't think we should do that anymore, based on how easy it seems to be to cause onionskin overload.https://gitlab.torproject.org/legacy/trac/-/issues/7025Support Tor 0.2.3.x control port event keyword reordering2020-06-13T16:19:05ZMike PerrySupport Tor 0.2.3.x control port event keyword reorderingRight now, the bandwidth auths are not going to work with Tor 0.2.3.x because for some reason we needed to deliberately break the control-spec.txt by reordering keywords... #3679 attempted to fix this by rewriting all of our event parsin...Right now, the bandwidth auths are not going to work with Tor 0.2.3.x because for some reason we needed to deliberately break the control-spec.txt by reordering keywords... #3679 attempted to fix this by rewriting all of our event parsing, but that caused strange issues with the bw auths.
We should just try to update the regex instead, as a simpler change.Aaron GibsonAaron Gibsonhttps://gitlab.torproject.org/legacy/trac/-/issues/7023Circ fail capping should also work with descriptor bandwidth2022-03-04T13:16:11ZMike PerryCirc fail capping should also work with descriptor bandwidthThe circuit failure capping we did in #1984 assumed that the PID feedback would actually work. Turns out there's a few barriers to that.
We should make the circuit failure stuff work a little better with descriptor bandwidth values. For...The circuit failure capping we did in #1984 assumed that the PID feedback would actually work. Turns out there's a few barriers to that.
We should make the circuit failure stuff work a little better with descriptor bandwidth values. For that, we probably need to do something like subtract the circuit fail rate from the pid_error, not use the min().Aaron GibsonAaron Gibsonhttps://gitlab.torproject.org/legacy/trac/-/issues/6582Adapt Exit Scanner to Firefox automation framework2020-06-13T16:19:03ZMike PerryAdapt Exit Scanner to Firefox automation frameworkAssuming the forensic automation works out as a stepping stone to automated testing, it wouldn't be too much of a stretch to adapt TorFlow to guide the exit choices of the Firefox tests, to see if any exits cause Firefox to either leak t...Assuming the forensic automation works out as a stepping stone to automated testing, it wouldn't be too much of a stretch to adapt TorFlow to guide the exit choices of the Firefox tests, to see if any exits cause Firefox to either leak to disk, bypass proxy, or simply do weird stuff or fail to load webpages.
This would also involve integration with prototype sandboxes and possibly even VM instrumentation, to watch for evidence of exploit/violation.
Putting this component in TorFlow for now, but it is probably shard with QA and sandboxing.Aaron GibsonAaron Gibsonhttps://gitlab.torproject.org/legacy/trac/-/issues/6385NetworkScanners/BwAuthority/run_scan.sh fixes2020-06-13T16:19:03Zweasel (Peter Palfrader)NetworkScanners/BwAuthority/run_scan.sh fixesa couple of minor things for the run_scan script.a couple of minor things for the run_scan script.Aaron GibsonAaron Gibsonhttps://gitlab.torproject.org/legacy/trac/-/issues/6133bwauths need to let us know when they've been running long enough2020-06-13T16:19:02ZRoger Dingledinebwauths need to let us know when they've been running long enoughWith #2286, if any three directory authorities vote one Measured line, potentially all the relays will get capped to some small constant weight.
So the bwauths need to let the directory authority know when they have 'enough' measurement...With #2286, if any three directory authorities vote one Measured line, potentially all the relays will get capped to some small constant weight.
So the bwauths need to let the directory authority know when they have 'enough' measurements, so the directory authority can only admit to having Measured lines when it has enough.
But the threshold shouldn't just be a simple fraction of relays, since then an adversary can flood the network with new relays to drive down the fraction and put us under the threshold.
One simple answer is to pick some parameter n where n is enough days for us to comfortably measure most relays, and output a bit "have we been running for most of the last n days or not".
We could put that bit into the V3BandwidthsFile file and have Tor read it. Or we could look at the bit ourselves and not update the V3BandwidthsFile file until the bit is true. Or some other smart thing. Whatever is easiest from bwauth's side.Aaron GibsonAaron Gibsonhttps://gitlab.torproject.org/legacy/trac/-/issues/6131bwauths learn to recognize Unmeasured=1 in consensus line and treat it differ...2020-06-13T16:19:02ZRoger Dingledinebwauths learn to recognize Unmeasured=1 in consensus line and treat it differentlyIn #2286 we're going to change directory authorities to say something like "w Bandwidth=100 Capped=1" for relays that don't have enough Measured lines.
Torflow's BWAuthority needs to learn not to get snookered by the 100 into thinking t...In #2286 we're going to change directory authorities to say something like "w Bandwidth=100 Capped=1" for relays that don't have enough Measured lines.
Torflow's BWAuthority needs to learn not to get snookered by the 100 into thinking that the relay's peers are all the other "Bandwidth=100" relays.Mike PerryMike Perryhttps://gitlab.torproject.org/legacy/trac/-/issues/5953Soat NoneType traceback in mime_type2020-06-13T16:19:01ZAaron GibsonSoat NoneType traceback in mime_typeMy exit scanner died with the following traceback.
```
DEBUG[Tue May 22 18:20:20 2012]:Starting request for: http://phys.org/pdf253371830.pdf
DEBUG[Tue May 22 18:20:21 2012]:Mime type is application/x-download, length 7681
INFO[Tue May ...My exit scanner died with the following traceback.
```
DEBUG[Tue May 22 18:20:20 2012]:Starting request for: http://phys.org/pdf253371830.pdf
DEBUG[Tue May 22 18:20:21 2012]:Mime type is application/x-download, length 7681
INFO[Tue May 22 18:20:21 2012]:Completed read
INFO[Tue May 22 18:20:21 2012]:Completed HTTP Reqest for: http://phys.org/pdf253371830.pdf
ERROR[Tue May 22 18:20:21 2012]:An unexpected error occured.
Traceback (most recent call last):
File "soat.py", line 2948, in <module>
main(sys.argv)
File "soat.py", line 2933, in main
test.rewind()
File "soat.py", line 1885, in rewind
BaseHTTPTest.rewind(self)
File "soat.py", line 876, in rewind
map(self.add_target, self.get_targets())
File "soat.py", line 1918, in get_targets
urllist.update(map(lambda x: (x, filetype), self.get_search_urls_for_filetype(filetype, self.results_per_type)))
File "soat.py", line 1840, in get_search_urls_for_filetype
url, success, code, cur_filetype = self.first_load(url,filetype)
File "soat.py", line 1186, in first_load
loaded_filetype = mime_to_filetype(req.mime_type)
File "soat.py", line 1479, in mime_to_filetype
return mimetypes.guess_extension(mime_type)[1:]
TypeError: 'NoneType' object is unsubscriptable
INFO[Tue May 22 18:20:21 2012]:Resetting __LeaveStreamsUnattached=0 and FetchUselessDescriptors=1
```Aaron GibsonAaron Gibsonhttps://gitlab.torproject.org/legacy/trac/-/issues/5933Soat does not report FailureExitTruncation on truncated content.2020-06-13T16:19:00ZAaron GibsonSoat does not report FailureExitTruncation on truncated content.Soat/Snakeinspector should report the failure type as FailureExitTruncation. I have noticed several reports emailed to the exit-scanner list that are marked FailureExitOnly; investigation reveals that the exit-specific .content is a trun...Soat/Snakeinspector should report the failure type as FailureExitTruncation. I have noticed several reports emailed to the exit-scanner list that are marked FailureExitOnly; investigation reveals that the exit-specific .content is a truncated version of the original .content.Aaron GibsonAaron Gibsonhttps://gitlab.torproject.org/legacy/trac/-/issues/5885snakeinspector.py is not attaching the bad content file2020-06-13T16:19:00ZAaron Gibsonsnakeinspector.py is not attaching the bad content filesnakeinspector.py is attaching the original content file but is not attaching the exit content file.
See snakeinspector.py lines 277 - 287snakeinspector.py is attaching the original content file but is not attaching the exit content file.
See snakeinspector.py lines 277 - 287Aaron GibsonAaron Gibsonhttps://gitlab.torproject.org/legacy/trac/-/issues/5464Decentralized measurement for network load balancing2020-06-13T16:18:59ZMike PerryDecentralized measurement for network load balancingEigenSpeed could provide a lot of security improvements to the Tor network in the face of all sorts of amplification attacks. It just sort of sucks because the passive version could not measure fast relays, and so we've never used it.
H...EigenSpeed could provide a lot of security improvements to the Tor network in the face of all sorts of amplification attacks. It just sort of sucks because the passive version could not measure fast relays, and so we've never used it.
However, an active version based on CapProbe, PacketPair, etc could possibly measure capacity in as little as a handful of UDP packets, enabling distributed active lightweight measurements. We could also blend in circuit failure rate information.
As an alternative, we could also try using the passive EigenSpeed for slow relays and the bw authorities only for the faster ones...
The big problem is that this is basically a research effort. We're going to need to try at least a couple different versions of these designs and compare them to each other, and then compare them to the current bandwidth authorities, to make sure everything is as performant and abuse tolerant as possible.Roger DingledineRoger Dingledinehttps://gitlab.torproject.org/legacy/trac/-/issues/5459Exit scanner should scan for Guard <-> Exit reachability2020-06-13T16:18:58ZMike PerryExit scanner should scan for Guard <-> Exit reachabilityThe Tor Exit Scanner should be checking to ensure that all Guard nodes can create circuits to all Exit nodes, to attempt to detect tagging and path bias attackers who fail circuits that don't go through colluding nodes.The Tor Exit Scanner should be checking to ensure that all Guard nodes can create circuits to all Exit nodes, to attempt to detect tagging and path bias attackers who fail circuits that don't go through colluding nodes.Aaron GibsonAaron Gibsonhttps://gitlab.torproject.org/legacy/trac/-/issues/4912line 34 of run_scan.sh uses rm incorrectly2020-06-13T16:18:58ZAndrew Lewmanline 34 of run_scan.sh uses rm incorrectlyWhen running run_scan.sh to start a bwauth, the following output appears:
```
Killing off scanner 1.
Killing off scanner 2.
Killing off scanner 3.
Killing off scanner 4.
rm: missing operand
Try `rm --help' for more information.
rm: miss...When running run_scan.sh to start a bwauth, the following output appears:
```
Killing off scanner 1.
Killing off scanner 2.
Killing off scanner 3.
Killing off scanner 4.
rm: missing operand
Try `rm --help' for more information.
rm: missing operand
Try `rm --help' for more information.
rm: missing operand
Try `rm --help' for more information.
rm: missing operand
Try `rm --help' for more information.
Waiting for 60 seconds to refresh tors...
```
Line 34 of run_scan.sh is missing the rm operands, it is currently
```
find $i/scan-data/ -depth -type f -print | egrep -v -- "-done-|\/.svn" | xargs -P 1024 rm
```
it should have -rf added to the end of that xargs rm statement, such that the correct line is
```
find $i/scan-data/ -depth -type f -print | egrep -v -- "-done-|\/.svn" | xargs -P 1024 rm -rf
```Mike PerryMike Perry