Trac issueshttps://gitlab.torproject.org/legacy/trac/-/issues2020-06-13T18:04:42Zhttps://gitlab.torproject.org/legacy/trac/-/issues/34316Make -o/-i arguments mutually exclusive2020-06-13T18:04:42ZKarsten LoesingMake -o/-i arguments mutually exclusiveRight now, it's possible to specify both `-o` and `-i` at the same time, which doesn't make any sense.
Also, it's rather non-intuitive that both argument defaults are `True` and that setting either of them changes their value to `False`...Right now, it's possible to specify both `-o` and `-i` at the same time, which doesn't make any sense.
Also, it's rather non-intuitive that both argument defaults are `True` and that setting either of them changes their value to `False`.
I found this issue while working on #34216, but this issue seemed sufficiently different to justify having its own ticket. I'll post a patch for review in a minute.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/34303Find out why onion service measurements have gotten slower2020-06-13T15:53:42ZKarsten LoesingFind out why onion service measurements have gotten slowerToday I changed the ["Time to download files over Tor" graph](https://metrics.torproject.org/torperf.html) to include version 3 onion service measurements.
By chance, I also looked at the [onion server measurements](https://metrics.torp...Today I changed the ["Time to download files over Tor" graph](https://metrics.torproject.org/torperf.html) to include version 3 onion service measurements.
By chance, I also looked at the [onion server measurements](https://metrics.torproject.org/torperf.html?start=2020-02-25&end=2020-05-25&server=onion&filesize=50kb) and found that the onion service measurements made by op-nl2, op-us2, and op-hk2 have gotten much slower as compared to their op-nl, op-us, and op-hk predecessors.
I'm 95% certain that this is not a bug in the graphs.
My current best guess is that something in `tor` has changed. I'd like to set up a small number of experimental OnionPerf instances all in the same place but with different `tor` versions. Any suggestions on locations (Amsterdam, Florida, Hong Kong) or `tor` versions?
This is also relevant to Sponsor 59 in order to make sure that our current measurements are going to be a baseline for future experiments. Classifying as potential defect.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/34302Include version 3 onion service measurements in all performance graphs2020-06-13T18:15:35ZKarsten LoesingInclude version 3 onion service measurements in all performance graphsIn #33434 we found that the results of version 2 and version 3 onion service measurements are almost identical. But so far we're not plotting the version 3 measurements at all. We need to fix that, in particular due to version 2 measurem...In #33434 we found that the results of version 2 and version 3 onion service measurements are almost identical. But so far we're not plotting the version 3 measurements at all. We need to fix that, in particular due to version 2 measurements going away soon.
I started working on a patch over the weekend that includes version 3 onion service measurements in all performance graphs. I'll merge and deploy that today.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/34261Remove pandas warning when using bullseye version of pandas2020-06-13T18:04:41ZKarsten LoesingRemove pandas warning when using bullseye version of pandasThe following warning is shown when using OnionPerf's visualize mode with version 0.25.3 of pandas (contained in Debian bullseye). It is not shown when using buster's version of pandas. If there's an easy way to remove the warning when u...The following warning is shown when using OnionPerf's visualize mode with version 0.25.3 of pandas (contained in Debian bullseye). It is not shown when using buster's version of pandas. If there's an easy way to remove the warning when using either version, let's do it.
```
/usr/lib/python3/dist-packages/pandas/plotting/_matplotlib/converter.py:103: FutureWarning: Using an implicitly registered datetime converter for a matplotlib plotting method. The converter was registered by pandas on import. Future versions of pandas will require you to explicitly register matplotlib converters.
To register the converters:
>>> from pandas.plotting import register_matplotlib_converters
>>> register_matplotlib_converters()
warnings.warn(msg, FutureWarning)
```Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/34239Release CollecTor 1.15.22020-06-13T17:52:32ZKarsten LoesingRelease CollecTor 1.15.2I need to put out CollecTor 1.15.2 today with an update to metrics-lib 2.13.0 in order to archive OnionPerf's analysis results file format version 2.0.I need to put out CollecTor 1.15.2 today with an update to metrics-lib 2.13.0 in order to archive OnionPerf's analysis results file format version 2.0.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/34235Release metrics-lib 2.13.02020-06-13T17:58:28ZKarsten LoesingRelease metrics-lib 2.13.0I'm putting out metrics-lib 2.13.0 today to parse OnionPerf's analysis results file format 2.0 (#34224).I'm putting out metrics-lib 2.13.0 today to parse OnionPerf's analysis results file format 2.0 (#34224).Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/34216Split visualizations into public server vs. v2 onion server vs. v3 onion serv...2020-06-13T18:04:37ZKarsten LoesingSplit visualizations into public server vs. v2 onion server vs. v3 onion server measurementsRight now, OnionPerf visualizations contain all measurements found in a data set. This can be very confusing, because this includes measurements performed via exits, via v2 onion service, and via v3 onion service. We should split visuali...Right now, OnionPerf visualizations contain all measurements found in a data set. This can be very confusing, because this includes measurements performed via exits, via v2 onion service, and via v3 onion service. We should split visualizations, or we should provide a filter and only graph one subset of measurements at a time.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/34215Harmonize TTFB/TTLB definitions with Tor Metrics plots2020-06-13T18:04:36ZKarsten LoesingHarmonize TTFB/TTLB definitions with Tor Metrics plotsRight now, OnionPerf plots 'command' to 'first_byte', whereas Tor Metrics plots 'start' to 'first_byte' (as found in #33258). We should harmonize these definitions, which likely means using what we have been using for Tor Metrics plots f...Right now, OnionPerf plots 'command' to 'first_byte', whereas Tor Metrics plots 'start' to 'first_byte' (as found in #33258). We should harmonize these definitions, which likely means using what we have been using for Tor Metrics plots for the past decade.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/34214Remove existing Tor control log visualizations2020-06-13T18:04:36ZKarsten LoesingRemove existing Tor control log visualizationsAs found in #33258, the existing Tor control log visualizations are not really useful to us. Let's consider removing them.
As part of this change, we should rename the TGen visualizations .pdf and .csv to something starting with `onionp...As found in #33258, the existing Tor control log visualizations are not really useful to us. Let's consider removing them.
As part of this change, we should rename the TGen visualizations .pdf and .csv to something starting with `onionperf.*`, as these files will contain parts from TGen and TorCtl in the future (error codes from TorCtl logs, measurements filtered by relay fingerprints).Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/34142Integrate reprocessing mode into analysis mode2020-06-13T18:04:33ZKarsten LoesingIntegrate reprocessing mode into analysis modeWhen we added the _reprocess_ mode a while back, we should have tried harder to add that functionality as part of the existing _analyze_ mode which does almost the same.
Right now, the reprocessing module is a wrapper on top of the curr...When we added the _reprocess_ mode a while back, we should have tried harder to add that functionality as part of the existing _analyze_ mode which does almost the same.
Right now, the reprocessing module is a wrapper on top of the current analyze mode. It matches log files, has filtering capabilities, and multi-processes the jobs in a pool.
Maybe it's as simple as accepting a directory where we accept a single file right now.
First step here is to brainstorm possible user interface changes and discuss them here. Coding will be step two.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/34109Download and parse OnionPerf analysis .json files instead of .tpf files2020-06-13T18:09:38ZKarsten LoesingDownload and parse OnionPerf analysis .json files instead of .tpf filesWith #34070 and #34072 being merged and deployed we can now change metrics-web to download and parse OnionPerf analysis .json files instead of .tpf files.With #34070 and #34072 being merged and deployed we can now change metrics-web to download and parse OnionPerf analysis .json files instead of .tpf files.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/34103Stop rounding y axis labels with units2020-06-13T18:15:35ZKarsten LoesingStop rounding y axis labels with unitsThe [Time to download files over Tor graph](https://metrics.torproject.org/torperf.html) now displays the following labels, all using seconds as units: 0, 0, 1, 2, 2, 2. Better labels would be 0.0, 0.5, 1.0, 1.5, 2.0, 2.5.
Similarly, th...The [Time to download files over Tor graph](https://metrics.torproject.org/torperf.html) now displays the following labels, all using seconds as units: 0, 0, 1, 2, 2, 2. Better labels would be 0.0, 0.5, 1.0, 1.5, 2.0, 2.5.
Similarly, the [Advertised bandwidth distribution graph](https://metrics.torproject.org/advbwdist-perc.html?start=2015-01-19&end=2020-04-18&p=50) showing the median displays the following labels, all using Gbit/s: 0.00, 0.02, 0.05, 0.08. Better labels would be 0.000, 0.025, 0.050, 0.075.
The underlying issue, which we didn't fix in #33933 nor in #33066, is that we have to pick a number of digits for a graph which then needs to work for whichever scale is being displayed. In some cases this is difficult to do right (first graph above with measurements apparently getting faster over time), in other cases it's impossible (second graph above with 1st and 99th percentile having different orders of magnitude).
The fix is to stop using the `unit_format` function from the somewhat outdated `scales` package that we're using and instead write our own function for formatting labels with units.
I'm going to attach two example graphs where this went wrong plus a patch that I'll review more carefully tomorrow before I deploy it on the server.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/34073Release CollecTor 1.15.02020-06-13T17:52:31ZKarsten LoesingRelease CollecTor 1.15.0I'm going to release CollecTor 1.15.0 to get the #34072 patch deployed.I'm going to release CollecTor 1.15.0 to get the #34072 patch deployed.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/34072Archive OnionPerf analysis .json files2020-06-13T18:09:38ZKarsten LoesingArchive OnionPerf analysis .json filesOnionperf produces two output file formats: .tpf and .json files. So far we have been archiving only the first format, but now that metrics-lib supports parsing .json files (#34070) we can archive the second format as well. In the near f...Onionperf produces two output file formats: .tpf and .json files. So far we have been archiving only the first format, but now that metrics-lib supports parsing .json files (#34070) we can archive the second format as well. In the near future we'll want OnionPerf to stop producing .tpf files and CollecTor to stop archiving them.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/34071Release metrics-lib 2.12.02020-06-13T17:58:28ZKarsten LoesingRelease metrics-lib 2.12.0I'm going to put out a new metrics-lib release for #34070 today.I'm going to put out a new metrics-lib release for #34070 today.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/34070Add parsing support for OnionPerf analysis files2020-06-13T18:09:38ZKarsten LoesingAdd parsing support for OnionPerf analysis filesBack in the days, Torperf wrote its results to .tpf files using one line per measurement with space-separated key=value pairs. OnionPerf continued using this format in addition to writing a more sophisticated JSON format. It's time to te...Back in the days, Torperf wrote its results to .tpf files using one line per measurement with space-separated key=value pairs. OnionPerf continued using this format in addition to writing a more sophisticated JSON format. It's time to teach metrics-lib how to read that JSON format, so that we can abandon the .tpf format. That is going to simplify upcoming changes to OnionPerf where we then only have to write a single output format.
The first step is to add parsing support for OnionPerf analysis files to metrics-lib. I worked on a patch over the past weeks that parses JSON files and internally converts them to Torperf results. The idea here is to keep changes to applications like metrics-web as small as possible. They can simply read different descriptor files and obtain the same output as before. We can later extract more parts from OnionPerf analysis files and provide them in a new interface, but we don't have to do that right now.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/34024Reduce timeout and stallout values2020-06-13T18:04:30ZKarsten LoesingReduce timeout and stallout valuesOn #33974 we discussed a suggestion to reduce timeouts for our three downloads as follows:
- 50 KiB download with 15 seconds timeout rather than 295 seconds,
- 1 MiB download with 60 seconds timeout rather than 1795 seconds, and
- 5 ...On #33974 we discussed a suggestion to reduce timeouts for our three downloads as follows:
- 50 KiB download with 15 seconds timeout rather than 295 seconds,
- 1 MiB download with 60 seconds timeout rather than 1795 seconds, and
- 5 MiB download with 120 seconds timeout rather than 3595 seconds.
Similarly, stallouts would be dropped entirely:
- 50 KiB download with 0 seconds stallout rather than 300 seconds,
- 1 MiB download with 0 seconds stallout rather than 1800 seconds, and
- 5 MiB download with 0 seconds stallout rather than 3600 seconds.
After discussing this with irl we concluded that we might want to pick values somewhere in the middle. The smaller values above are being used by TGen for generating load for Shadow simulations, in that case it makes sense to use timeouts similar to how users would behave. But in the measurements we're doing with OnionPerf we can easily record more data even after a human user would have given up and later filter out measurements taking longer than whatever timeouts we want to use.
In particular, it would be important for us to use timeouts that are higher than timeouts used internally by the Tor client, so that we can observe what happens in those cases. Even if a human user would long have given up.
How about we use timeouts and stallouts close to 5 minutes, so that we avoid overlapping measurements? Like 270 seconds for all three download sizes? What would we use as stallout value here? 0?Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/34023Reduce the number of 50 KiB downloads2020-06-13T18:04:30ZKarsten LoesingReduce the number of 50 KiB downloadsOn #33076 we discussed whether we should kill the 50 KiB downloads in deployed OnionPerfs and only keep the 1 MiB and 5 MiB downloads. The primary reason would be that our [throughput](https://metrics.torproject.org/onionperf-throughput....On #33076 we discussed whether we should kill the 50 KiB downloads in deployed OnionPerfs and only keep the 1 MiB and 5 MiB downloads. The primary reason would be that our [throughput](https://metrics.torproject.org/onionperf-throughput.html) graphs would be based on five times as many data points per day, because they only include 1 MiB and 5 MiB downloads, but not 50 KiB downloads. This would not affect our [circuit round-trip latencies graphs](https://metrics.torproject.org/onionperf-latencies.html) which include all three downloaded file sizes.
The main reason against killing 50 KiB downloads is that OnionPerfs would consume more bandwidth and also put more load on the Tor network. Let's consider two scenarios with and without 50 KiB downloads. In both scenarios we're making a new download every 5 minutes, randomly chosen with a weight of 1.0 for 5 MiB runs, 2.0 for 1 MiB runs, and either 12.0 or 0.0 for 50 KiB runs:
- With 50 KiB downloads we're downloading on average `12/15 * 50 KiB + 2/15 * 1 MiB + 1/15 * 5 MiB = 517 KiB` every 5 minutes, or `517 * 8 * 1024 / (300 * 1000) = 14 kbps`.
- Without 50 KiB downloads we're downloading on average `2/3 * 1 MiB + 1/3 * 5 MiB = 2389 KiB` every 5 minutes, or `2389 * 8 * 1024 / (300 * 1000) = 65 kbps`.
These numbers are both tiny in comparison to the overall network capacity and to other services like the bandwidth scanners.
I'm going to make this change and deploy it on new OnionPerf instances tomorrow, unless I hear objections here.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/33549Simplify logging configuration across code bases2020-06-13T18:10:36ZKarsten LoesingSimplify logging configuration across code basesWe're using different logging configurations in our code bases, some of which are broken and others are more complex than we need them to be. Some examples:
- CollecTor has a shutdown console logger, but it's not being printed to anymo...We're using different logging configurations in our code bases, some of which are broken and others are more complex than we need them to be. Some examples:
- CollecTor has a shutdown console logger, but it's not being printed to anymore since we refactored package names.
- CollecTor has five separate log files for the modules that existed back when we added those log files, but these are not being written to since we refactored package names. We also added more modules but did not update the logging configuration for them.
- Onionoo and a few others have a separate error log file which logs only ERROR messages, but we're not doing anything with that file.
- Some code bases have a separate statistics log file that we likewise do not pay any attention to.
- All code bases rotate log files whenever they reach a size of 1 MiB, but they do not restrict the number or total size of log files being written. It would be easier to handle 1 file per day even if that grows to 100 MiB or more.
- There could be a more reasonable default for the log directory than `LOGBASE_IS_UNDEFINED`.
- It would be convenient to be able to configure the log level that doesn't require providing a full logging configuration, like `-DLOGLEVEL=DEBUG`.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/33502Do not let appended descriptor files grow too large2020-06-13T17:52:29ZKarsten LoesingDo not let appended descriptor files grow too largeI revisited #20395 last week. The issue is that metrics-lib cannot handle large descriptor files, because it first reads the entire file into memory before splitting it into single descriptors and parsing them. While it would be possible...I revisited #20395 last week. The issue is that metrics-lib cannot handle large descriptor files, because it first reads the entire file into memory before splitting it into single descriptors and parsing them. While it would be possible to parse large descriptor files after making some major code changes (using `FileChannel` and doing lazy parsing), I don't think that we have to do that. After all, we're writing these large descriptor files ourselves in CollecTor, and it's up to us to stop doing that.
Going back in time, the original reason for concatenating multiple descriptors into a single file was that rsyncing many tiny files from one host to another host was just slow. So we appended server descriptors and extra-info descriptors into a single file. This works well with server descriptors or extra-info descriptors published within 1 hour or even 10 hours. It does not work that well anymore with all server descriptors or extra-info descriptors synced from another CollecTor instance when starting a new instance (#20335). It works even less well when importing one or more monthly tarballs containing server descriptors or extra-info descriptors (#27716).
My suggestion is that we define a configurable limit for appended descriptor files of, say, 20 MiB. And when storing a descriptor, we check whether appending a descriptor to an existing descriptor file would exceed this limit and start a new descriptor file in that case.
There are some technical details to work out, but I think they can be solved. I also don't expect this to produce a lot of code, not even complex code changes. The benefit would be that we could resolve #20395 and #27716 by implementing this.
Thoughts on the general idea?Karsten LoesingKarsten Loesing