Trac issueshttps://gitlab.torproject.org/legacy/trac/-/issues2020-06-13T18:15:36Zhttps://gitlab.torproject.org/legacy/trac/-/issues/34397Onion for metrics.torproject.org makes clearnet XMLHttpRequests2020-06-13T18:15:36ZcOnion for metrics.torproject.org makes clearnet XMLHttpRequestsA query such as http://rougmnvswfsmd4dq.onion/rs.html#toprelays will send an XHR through onionoo.torproject.org rather than through tgel7v4rpcllsrk2.onion
For a hidden service, it seems preferred not to have any third-party clearnet req...A query such as http://rougmnvswfsmd4dq.onion/rs.html#toprelays will send an XHR through onionoo.torproject.org rather than through tgel7v4rpcllsrk2.onion
For a hidden service, it seems preferred not to have any third-party clearnet requests.https://gitlab.torproject.org/legacy/trac/-/issues/34352Bump OP version in setup.py2020-06-13T18:04:43ZAna CusturaBump OP version in setup.pyI ran into a weird bug where installing the latest OP on a system that already has an installation from a version prior to commit [de26da3](https://gitweb.torproject.org/onionperf.git/commit/?id=de26da351d859b64a35b81eb2d07dbf325a86960) ...I ran into a weird bug where installing the latest OP on a system that already has an installation from a version prior to commit [de26da3](https://gitweb.torproject.org/onionperf.git/commit/?id=de26da351d859b64a35b81eb2d07dbf325a86960) results in the following error message:
```
$ onionperf
Traceback (most recent call last):
File "/usr/local/bin/onionperf", line 4, in <module>
__import__('pkg_resources').run_script('OnionPerf==0.2rc0', 'onionperf')
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 667, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1452, in run_script
raise ResolutionError(
pkg_resources.ResolutionError: Script 'scripts/onionperf' not found in metadata at None
```
It looks like `setuptools` is renaming the OnionPerf version from `0.2.pre` (as it is currently specified in `setup.py`) to `0.2rc0` when installing, and they somehow conflict. I've tested this can be solved by bumping the version to 0.3 in `setup.py` (patch attached).https://gitlab.torproject.org/legacy/trac/-/issues/34297Explore different options other than crontab to have a flexible scheduling sy...2020-06-13T17:56:18ZBarkin SimsekExplore different options other than crontab to have a flexible scheduling systemCurrently, I use crontab for fetching web pages at regular intervals. I believe it is not the best option, and it is worth exploring the options available.Currently, I use crontab for fetching web pages at regular intervals. I believe it is not the best option, and it is worth exploring the options available.https://gitlab.torproject.org/legacy/trac/-/issues/34296Develop a mechanism for disabling/enabling cookie, JavaScript, etc. functiona...2020-06-13T17:56:18ZBarkin SimsekDevelop a mechanism for disabling/enabling cookie, JavaScript, etc. functionality of the web browsershttps://gitlab.torproject.org/legacy/trac/-/issues/34295Develop a module for parsing and modifying the HTTP headers2020-06-13T17:56:18ZBarkin SimsekDevelop a module for parsing and modifying the HTTP headersWe need to develop a module for parsing and modifying the HTTP headers to test different conditions.We need to develop a module for parsing and modifying the HTTP headers to test different conditions.https://gitlab.torproject.org/legacy/trac/-/issues/34294Integrate Tor Stem2020-06-13T17:56:17ZBarkin SimsekIntegrate Tor StemIntegrate [Tor Stem](https://stem.torproject.org/) to choose specific Tor exit nodes for testing.Integrate [Tor Stem](https://stem.torproject.org/) to choose specific Tor exit nodes for testing.https://gitlab.torproject.org/legacy/trac/-/issues/34293Create an API for running the system on the user-provided websites2020-06-13T17:56:17ZBarkin SimsekCreate an API for running the system on the user-provided websitesThis part of the API aims to run certain tests on user-provided websites. So that people are not limited to the types of websites provided by the CAPTCHA Monitor.This part of the API aims to run certain tests on user-provided websites. So that people are not limited to the types of websites provided by the CAPTCHA Monitor.https://gitlab.torproject.org/legacy/trac/-/issues/34292Create an API for people to fetch data easily2020-06-13T17:56:16ZBarkin SimsekCreate an API for people to fetch data easilyThis API might feature customized queries to get a specific part of the dataset rather than the whole dataset.This API might feature customized queries to get a specific part of the dataset rather than the whole dataset.https://gitlab.torproject.org/legacy/trac/-/issues/34291Create an API for enabling 3rd party interactions with the system2020-06-13T17:56:16ZBarkin SimsekCreate an API for enabling 3rd party interactions with the systemhttps://gitlab.torproject.org/legacy/trac/-/issues/34290Enhance the available visualizations on the dashboard2020-06-13T17:56:15ZBarkin SimsekEnhance the available visualizations on the dashboardThe visualizations are not easy to understand at first sight. Make them as simple and as meaningful possible.The visualizations are not easy to understand at first sight. Make them as simple and as meaningful possible.https://gitlab.torproject.org/legacy/trac/-/issues/34289Integrate Cloudflare API not to change Cloudflare settings manually2020-06-13T17:56:15ZBarkin SimsekIntegrate Cloudflare API not to change Cloudflare settings manuallyUsing different zone settings and security levels makes a huge difference. So, integrate the [Cloudflare API](https://api.cloudflare.com/#zone-settings-properties) to check how these settings affect the results.Using different zone settings and security levels makes a huge difference. So, integrate the [Cloudflare API](https://api.cloudflare.com/#zone-settings-properties) to check how these settings affect the results.https://gitlab.torproject.org/legacy/trac/-/issues/34288Integrate more web browsers/fetchers and their older versions2020-06-13T17:56:15ZBarkin SimsekIntegrate more web browsers/fetchers and their older versionsCurrently, only the Tor Browser itself is a part of the system. We need to be able to fetch webpages with other browsers like (Firefox, Chromium, etc.) and other methods for fetching webpages (cURL, python libraries, etc.).
Additionall...Currently, only the Tor Browser itself is a part of the system. We need to be able to fetch webpages with other browsers like (Firefox, Chromium, etc.) and other methods for fetching webpages (cURL, python libraries, etc.).
Additionally, we need to be able to have multiple versions of these browsers/fetchers to see if using different versions makes a difference.https://gitlab.torproject.org/legacy/trac/-/issues/34287Make the dataset accessible and downloadable2020-06-13T17:56:14ZBarkin SimsekMake the dataset accessible and downloadableThe data collected is stored in an InfluxDB database. The data needs to be stored in a more accessible/common format which can be easily downloaded.The data collected is stored in an InfluxDB database. The data needs to be stored in a more accessible/common format which can be easily downloaded.https://gitlab.torproject.org/legacy/trac/-/issues/34257Analyze unusual distribution of time to extend to first hop in circuit2020-06-13T18:04:40ZKarsten LoesingAnalyze unusual distribution of time to extend to first hop in circuitI spent some time looking at OnionPerf measurements today. I found something that I did not expect: It seems like the time required to build the first hop in a circuit has a huge variance and rather unusual distribution. I'll attach a gr...I spent some time looking at OnionPerf measurements today. I found something that I did not expect: It seems like the time required to build the first hop in a circuit has a huge variance and rather unusual distribution. I'll attach a graph shortly that visualizes that.
Looking at that graph, we can see a few things:
- The three OnionPerf instances have very different performance regarding circuit extension. (I checked the earlier instances op-hk, op-nl, and op-us, and they had the same characteristics.)
- There are huge plateaus for op-us2 and op-nl2 in the first hop graph where _some_ circuits have been successfully extended and others not. Typically, we'd expect a distribution like op-nl2's, just pulled to the right. But that's not the case here. The blue line is special at around 0.6 seconds and the red line at around 0.9 seconds. In fact, the green line is also a bit special at around 0.3 seconds when it almost flattens, only to increase linearly until it reaches 100% at around 0.6 seconds.
- If we assume that the U.S. and Hong Kong are simply far away from many relays in a geographical sense, that doesn't explain why extending to the middle node and to the exit goes relatively fast even for those two hosts. Keep in mind that extending to the second hop requires a round-trip to the first hop, and that extending to the third hop requires a round-trip to the first and the second hop.
What's going on here? What properties of these relays should we be looking at? I already looked at:
- consensus weight,
- date/time of building these circuits, and
- whether these are just a small number of guards being reused over and over;
but nothing of these explained the shape of these ECDFs. I'm going to attach the data file that this graph is based on, if others want to take a look.
And would it make sense to try out running an OnionPerf instance on another set of hosts that are geographically close to our current hosts? Maybe it's related to how these hosts are set up, including their network? (Just in case that other set of hosts produces different results, we would still have to investigate how that affects our overall measurements of things like time to first byte or throughput.)
This is relevant to Sponsor 59, because we need to make sure that our current measurements are going to be a solid baseline for future experiments. Classifying as potential defect.https://gitlab.torproject.org/legacy/trac/-/issues/34231Document and maybe improve how we're mapping TGen transfers to Tor streams/ci...2020-06-13T18:04:39ZKarsten LoesingDocument and maybe improve how we're mapping TGen transfers to Tor streams/circuitsOnionPerf uses TGen to make transfers using a local Tor client. OnionPerf also uses Stem to connect to the Tor client's control port and register for control events.
This ticket is about documenting how we can map TGen transfers to Tor ...OnionPerf uses TGen to make transfers using a local Tor client. OnionPerf also uses Stem to connect to the Tor client's control port and register for control events.
This ticket is about documenting how we can map TGen transfers to Tor streams and circuits. OnionPerf did this to produce the .tpf output format (which we just killed in #34141). But we'll also need this functionality to implement #34218 or #33260.
Here's what we're doing in [metrics-lib](https://gitweb.torproject.org/metrics-lib.git/tree/src/main/java/org/torproject/descriptor/onionperf/OnionPerfAnalysisConverter.java#n121) right now to map transfers and streams:
- Index Tor circuits by their circuit ID.
- Index Tor streams by their source port; if there are two or more streams with the same source port, remember them all.
- Go through TGen transfers one by one. For each, extract the local source port.
- Go through Tor streams with the same source port and check if transfer end and stream end happened within 150 seconds.
- If there's a match, look up the corresponding circuit by circuit ID.
Note that OnionPerf took a simpler approach for producing .tpf files by remembering just one stream by source port and not applying that 150 seconds heuristic. The result was that some mappings were wrong. The approach taken by metrics-lib leads to a few missing mappings (probably as many as OnionPerf had), and apparently no wrong mappings.
Is there a way to have an exact mapping that doesn't require a heuristic? And is there a way to do it without having to wait for transfer and stream to end?https://gitlab.torproject.org/legacy/trac/-/issues/34224Update analysis results file version to 2.02020-06-13T18:04:38ZKarsten LoesingUpdate analysis results file version to 2.0We added a new field to the analysis results file format in #26673, but we did not yet increment the format version number. The current version field is a floating point number, which doesn't work well for versioning. We should use a ver...We added a new field to the analysis results file format in #26673, but we did not yet increment the format version number. The current version field is a floating point number, which doesn't work well for versioning. We should use a version string here. Given that this change alone is going to be backward-incompatible, we'll have to call the new version 2.0. I'm preparing a patch.https://gitlab.torproject.org/legacy/trac/-/issues/34218Split PROXY errors into whatever reasons are given in TorCtl logs2020-06-13T18:04:37ZKarsten LoesingSplit PROXY errors into whatever reasons are given in TorCtl logsIf we combine TGen transfers with Tor streams and circuits we can learn a lot more about why our proxy (Tor) failed. Including that information in visualizations might be very useful.
When we match TGen transfers with TorCtl events, we'...If we combine TGen transfers with Tor streams and circuits we can learn a lot more about why our proxy (Tor) failed. Including that information in visualizations might be very useful.
When we match TGen transfers with TorCtl events, we'll have to look [how metrics-lib does this](https://gitweb.torproject.org/metrics-lib.git/tree/src/main/java/org/torproject/descriptor/onionperf/OnionPerfAnalysisConverter.java).
Maybe we can use some ideas from #30612 where we discussed putting an error code graph on Tor Metrics.https://gitlab.torproject.org/legacy/trac/-/issues/34217Include partial downloads in existing TTLB graphs2020-06-13T18:04:37ZKarsten LoesingInclude partial downloads in existing TTLB graphsWith #26673 being resolved we should include partial downloads in existing TTLB graphs. Maybe there's a way to put less emphasis on the different file sizes and combine the various partial downloads in a single TTLB graph.With #26673 being resolved we should include partial downloads in existing TTLB graphs. Maybe there's a way to put less emphasis on the different file sizes and combine the various partial downloads in a single TTLB graph.https://gitlab.torproject.org/legacy/trac/-/issues/34213Replace TorCtlParser with OnionTrace's control log parser2020-06-13T18:04:35ZKarsten LoesingReplace TorCtlParser with OnionTrace's control log parserSimilar to our plan to use TGen's analysis.py instead of OnionPerf's in #33974, the idea would be to avoid keeping our own copy of a Tor control even log parser in OnionPerf.
This work should start with an analysis how much work this is...Similar to our plan to use TGen's analysis.py instead of OnionPerf's in #33974, the idea would be to avoid keeping our own copy of a Tor control even log parser in OnionPerf.
This work should start with an analysis how much work this is going to save in the long run, and whether that justifies adding another dependency.
Feel free to adjust estimated points before starting to work on this if 2.0 doesn't make any sense.https://gitlab.torproject.org/legacy/trac/-/issues/34208Onionperf errors when no subcommand is given (python3)2020-06-13T18:04:35ZAna CusturaOnionperf errors when no subcommand is given (python3)Not a major issue by any means, but OP errors when ran without any arguments. This seems to be a behaviour change from Python2 to Python3. The expected behaviour was previously to print help.
```
~/onionperf$ onionperf
Traceback (most re...Not a major issue by any means, but OP errors when ran without any arguments. This seems to be a behaviour change from Python2 to Python3. The expected behaviour was previously to print help.
```
~/onionperf$ onionperf
Traceback (most recent call last):
File "/usr/local/bin/onionperf", line 532, in <module>
if __name__ == '__main__': sys.exit(main())
File "/usr/local/bin/onionperf", line 351, in main
args.func(args)
AttributeError: 'Namespace' object has no attribute 'func'
```
I've attached a patch to get back that behaviour. See also https://bugs.python.org/issue16308.