Commit 575741e3 authored by Karsten Loesing's avatar Karsten Loesing
Browse files

Include v3 onion OnionPerf measurements. 

In commit 7af8f108 we started ignoring v3 onion OnionPerf measurements,
because including a third option in addition to "public" and "onion"
would have required writing more code.

However, an analysis in #33434 revealed that v3 onion measurements are
basically no different than v2 onion measurements. As a result we're
turning off v2 onion measurements and combining v2 and v3 onion
measurements in existing graphs.

Implements #34032.
parent f83d7599

src/main/resources/web/json/metrics.json

+4 −4
Original line number Diff line number Diff line
@@ -274,7 +274,7 @@ 
    "id": "torperf",

    "title": "Time to download files over Tor",

    "type": "Graph",

    "description": "<p>This graph shows overall performance when downloading static files of different sizes over Tor, either from a server on the public internet or from a version 2 onion server.  The graph shows the range of measurements from first to third quartile, and highlights the median.  The slowest and fastest quarter of measurements are omitted from the graph.</p>",

    "description": "<p>This graph shows overall performance when downloading static files of different sizes over Tor, either from a server on the public internet or from an onion server.  The graph shows the range of measurements from first to third quartile, and highlights the median.  The slowest and fastest quarter of measurements are omitted from the graph.</p>",

    "function": "torperf",

    "parameters": [

      "start",
@@ -287,7 +287,7 @@ 
    "id": "torperf-failures",

    "title": "Timeouts and failures of downloading files over Tor",

    "type": "Graph",

    "description": "<p>This graph shows the fraction of timeouts and failures when downloading static files of different sizes over Tor, either from a server on the public internet or from a version 2 onion server.  A timeout occurs when a download does not complete within the scheduled time, in which case it is aborted in order not to overlap with the next scheduled download.  A failure occurs when the download completes, but the response is smaller than expected.</p>",

    "description": "<p>This graph shows the fraction of timeouts and failures when downloading static files of different sizes over Tor, either from a server on the public internet or from an onion server.  A timeout occurs when a download does not complete within the scheduled time, in which case it is aborted in order not to overlap with the next scheduled download.  A failure occurs when the download completes, but the response is smaller than expected.</p>",

    "function": "torperf_failures",

    "parameters": [

      "start",
@@ -311,7 +311,7 @@ 
    "id": "onionperf-latencies",

    "title": "Circuit round-trip latencies",

    "type": "Graph",

    "description": "<p>This graph shows round-trip latencies of circuits used for downloading static files of different sizes over Tor, either from a server on the public internet or from a version 2 onion server. Round-trip latencies are measured as the time between sending the HTTP request and receiving the HTTP response header. The graph shows the median of measurements as thick line, the range of measurements from first to third quartile as ribbon, and the highest and lowest non-outlier measurements as thin lines.</p>",

    "description": "<p>This graph shows round-trip latencies of circuits used for downloading static files of different sizes over Tor, either from a server on the public internet or from an onion server. Round-trip latencies are measured as the time between sending the HTTP request and receiving the HTTP response header. The graph shows the median of measurements as thick line, the range of measurements from first to third quartile as ribbon, and the highest and lowest non-outlier measurements as thin lines.</p>",

    "function": "onionperf_latencies",

    "parameters": [

      "start",
@@ -323,7 +323,7 @@ 
    "id": "onionperf-throughput",

    "title": "Throughput",

    "type": "Graph",

    "description": "<p>This graph shows throughput when downloading static files of different sizes over Tor, either from a server on the public internet or from a version 2 onion server. Throughput is calculated from the time between receiving 0.5 and 1 MiB of the response. The graph shows the median of measurements as thick line, the range of measurements from first to third quartile as ribbon, and the highest and lowest non-outlier measurements as thin lines.</p>",

    "description": "<p>This graph shows throughput when downloading static files of different sizes over Tor, either from a server on the public internet or from an onion server. Throughput is calculated from the time between receiving 0.5 and 1 MiB of the response. The graph shows the median of measurements as thick line, the range of measurements from first to third quartile as ribbon, and the highest and lowest non-outlier measurements as thin lines.</p>",

    "function": "onionperf_throughput",

    "parameters": [

      "start",

src/main/resources/web/jsps/stats.jsp

+9 −8
Original line number Diff line number Diff line
@@ -55,6 +55,7 @@ https://metrics.torproject.org/identifier.csv 
<li><b>August 5, 2019:</b> Re-added the <a href="#bandwidth">Total relay bandwidth</a> graph due to popular demand.</li>

<li><b>October 2, 2019:</b> Added <a href="#webstats-tb-channel">Tor Browser updates by release channel</a> graph.</li>

<li><b>December 18, 2019:</b> Added <a href="#bridgedb-transport">BridgeDB requests by requested transport</a> and <a href="#bridgedb-distributor">BridgeDB requests by distributor</a> graphs.</li>

<li><b>May 24, 2020:</b> Included version 3 onion service measurements in <a href="#torperf">Time to download files over Tor</a>, <a href="#torperf-failures">Timeouts and failures of downloading files over Tor</a>, <a href="#onionperf-buildtimes">Circuit build times</a>, <a href="#onionperf-latencies">Circuit round-trip latencies</a> graphs.</li>

</ul>



</div>
@@ -547,7 +548,7 @@ Performance <a href="#performance" name="performance" class="anchor">#</a></h2> 
<ul>

<li><b>start:</b> First UTC date (YYYY-MM-DD) to include in the file.</li>

<li><b>end:</b> Last UTC date (YYYY-MM-DD) to include in the file.</li>

<li><b>server:</b> Either <b>"public"</b> for requests to a server on the public internet, or <b>"onion"</b> for requests to a version 2 onion server.</li>

<li><b>server:</b> Either <b>"public"</b> for requests to a server on the public internet, or <b>"onion"</b> for requests to an onion server.</li>

<li><b>filesize:</b> Size of the downloaded file in bytes, with pre-defined possible values: <b>"50kb"</b>, <b>"1mb"</b>, or <b>"5mb"</b>.</li>

</ul>
@@ -557,7 +558,7 @@ Performance <a href="#performance" name="performance" class="anchor">#</a></h2> 
<li><b>date:</b> UTC date (YYYY-MM-DD) when download performance was measured.</li>

<li><b>filesize:</b> Size of the downloaded file in bytes.</li>

<li><b>source:</b> Name of the OnionPerf or Torperf service performing measurements.</li>

<li><b>server:</b> Either <b>"public"</b> if the request was made to a server on the public internet, or <b>"onion"</b> if the request was made to a version 2 onion server.</li>

<li><b>server:</b> Either <b>"public"</b> if the request was made to a server on the public internet, or <b>"onion"</b> if the request was made to an onion server.</li>

<li><b>q1:</b> First quartile of time in milliseconds until receiving the last byte.</li>

<li><b>md:</b> Median of time in milliseconds until receiving the last byte.</li>

<li><b>q3:</b> Third quartile of time in milliseconds until receiving the last byte.</li>
@@ -573,7 +574,7 @@ Performance <a href="#performance" name="performance" class="anchor">#</a></h2> 
<ul>

<li><b>start:</b> First UTC date (YYYY-MM-DD) to include in the file.</li>

<li><b>end:</b> Last UTC date (YYYY-MM-DD) to include in the file.</li>

<li><b>server:</b> Either <b>"public"</b> for requests to a server on the public internet, or <b>"onion"</b> for requests to a version 2 onion server.</li>

<li><b>server:</b> Either <b>"public"</b> for requests to a server on the public internet, or <b>"onion"</b> for requests to an onion server.</li>

<li><b>filesize:</b> Size of the downloaded file in bytes, with pre-defined possible values: <b>"50kb"</b>, <b>"1mb"</b>, or <b>"5mb"</b>.</li>

</ul>
@@ -583,7 +584,7 @@ Performance <a href="#performance" name="performance" class="anchor">#</a></h2> 
<li><b>date:</b> UTC date (YYYY-MM-DD) when download performance was measured.</li>

<li><b>filesize:</b> Size of the downloaded file in bytes.</li>

<li><b>source:</b> Name of the OnionPerf or Torperf service performing measurements.</li>

<li><b>server:</b> Either <b>"public"</b> if the request was made to a server on the public internet, or <b>"onion"</b> if the request was made to a version 2 onion server.</li>

<li><b>server:</b> Either <b>"public"</b> if the request was made to a server on the public internet, or <b>"onion"</b> if the request was made to an onion server.</li>

<li><b>timeouts:</b> Fraction of requests that timed out when attempting to download the static file over Tor.</li>

<li><b>failures:</b> Fraction of requests that failed when attempting to download the static file over Tor.</li>

</ul>
@@ -621,7 +622,7 @@ Performance <a href="#performance" name="performance" class="anchor">#</a></h2> 
<ul>

<li><b>start:</b> First UTC date (YYYY-MM-DD) to include in the file.</li>

<li><b>end:</b> Last UTC date (YYYY-MM-DD) to include in the file.</li>

<li><b>server:</b> Either <b>"public"</b> for requests to a server on the public internet, or <b>"onion"</b> for requests to a version 2 onion server.</li>

<li><b>server:</b> Either <b>"public"</b> for requests to a server on the public internet, or <b>"onion"</b> for requests to an onion server.</li>

</ul>



<h4>Columns</h4>
@@ -629,7 +630,7 @@ Performance <a href="#performance" name="performance" class="anchor">#</a></h2> 
<ul>

<li><b>date:</b> UTC date (YYYY-MM-DD) when download performance was measured.</li>

<li><b>source:</b> Name of the OnionPerf or Torperf service performing measurements.</li>

<li><b>server:</b> Either <b>"public"</b> if the request was made to a server on the public internet, or <b>"onion"</b> if the request was made to a version 2 onion server.</li>

<li><b>server:</b> Either <b>"public"</b> if the request was made to a server on the public internet, or <b>"onion"</b> if the request was made to an onion server.</li>

<li><b>low:</b> Lowest time in milliseconds between sending the HTTP request and receiving the HTTP response header within 1.5 IQR of lower quartile (lower whisker in a boxplot).</li>

<li><b>q1:</b> First quartile of time in milliseconds between sending the HTTP request and receiving the HTTP response header.</li>

<li><b>md:</b> Median of time in milliseconds between sending the HTTP request and receiving the HTTP response header.</li>
@@ -647,7 +648,7 @@ Performance <a href="#performance" name="performance" class="anchor">#</a></h2> 
<ul>

<li><b>start:</b> First UTC date (YYYY-MM-DD) to include in the file.</li>

<li><b>end:</b> Last UTC date (YYYY-MM-DD) to include in the file.</li>

<li><b>server:</b> Either <b>"public"</b> for requests to a server on the public internet, or <b>"onion"</b> for requests to a version 2 onion server.</li>

<li><b>server:</b> Either <b>"public"</b> for requests to a server on the public internet, or <b>"onion"</b> for requests to an onion server.</li>

</ul>



<h4>Columns</h4>
@@ -655,7 +656,7 @@ Performance <a href="#performance" name="performance" class="anchor">#</a></h2> 
<ul>

<li><b>date:</b> UTC date (YYYY-MM-DD) when download performance was measured.</li>

<li><b>source:</b> Name of the OnionPerf or Torperf service performing measurements.</li>

<li><b>server:</b> Either <b>"public"</b> if the request was made to a server on the public internet, or <b>"onion"</b> if the request was made to a version 2 onion server.</li>

<li><b>server:</b> Either <b>"public"</b> if the request was made to a server on the public internet, or <b>"onion"</b> if the request was made to an onion server.</li>

<li><b>low:</b> Lowest measured throughput in kilobits per second within 1.5 IQR of lower quartile (lower whisker in a boxplot).</li>

<li><b>q1:</b> First quartile of measured throughput in kilobits per second.</li>

<li><b>md:</b> Median of measured throughput in kilobits per second.</li>

src/main/sql/onionperf/init-onionperf.sql

+0 −4
Original line number Diff line number Diff line
@@ -80,7 +80,6 @@ SELECT DATE(start) AS date, 
  COUNT(*) AS requests

FROM measurements

WHERE DATE(start) < current_date - 1

AND (endpointremote IS NULL OR endpointremote NOT SIMILAR TO '_{56}.onion%')

GROUP BY date, filesize, source, server

UNION

SELECT DATE(start) AS date,
@@ -98,7 +97,6 @@ SELECT DATE(start) AS date, 
  COUNT(*) AS requests

FROM measurements

WHERE DATE(start) < current_date - 1

AND (endpointremote IS NULL OR endpointremote NOT SIMILAR TO '_{56}.onion%')

GROUP BY date, filesize, 3, server) sub

ORDER BY date, filesize, source, server;
@@ -140,7 +138,6 @@ WITH filtered_measurements AS ( 
  WHERE DATE(start) < current_date - 1

  AND datarequest > 0

  AND dataresponse > 0

  AND (endpointremote IS NULL OR endpointremote NOT SIMILAR TO '_{56}.onion%')

), quartiles AS (

  SELECT date,

    source,
@@ -185,7 +182,6 @@ WITH filtered_measurements AS ( 
      ELSE NULL END AS kbps

  FROM measurements

  WHERE DATE(start) < current_date - 1

  AND (endpointremote IS NULL OR endpointremote NOT SIMILAR TO '_{56}.onion%')

), quartiles AS (

  SELECT date,

    source,