GitLab

Pre-deployment code fixes/additions

See merge request !13

Ensure that onionperf ignores measurements and timeouts, when no timeout is learned yet, while changing guards

See merge request !12

Add a new argument to the filter function, '--exclude-cbt',
that can be used to mark circuits as filtered based on
the cbt_set field. The filter is useful for visualizing
results for measurements that use guards, where the analysis
file still includes circuits built before CBT was computed.

A new field, cbt_set, is added for every circuit; this is set to
false at the beginning of a measurement, and gets set to True
after a BUILDTIMEOUT_SET COMPUTED event. Each
BUILDTIMEOUT_SET RESET event resets the field to False.
Any measurements that specify `--drop-guards` will use this
field to exclude circuits built when CBT was not computed
from the daily analysis by default.

Analysis files for these measurements are also tagged by default with
the 'exclude_cbt' filter, which is then used by the visualize command to
only visualize results that used non-excluded circuits.

Compute 5 MiB transfer throughput based on the last MiB of the response

See merge request !11

Task 33421

See merge request !10

Adds code to track guards on a per-circuit basis.
Adds country information per fingerprint.
Removes guard-related visualisations, to be
rewritten as per #40024.
Includes all Guard information in CSV file.
This has been rebased on the latest develop branch.

Implements #33421

Remaining steps:
 - Wait for #33260 is merged to avoid
   implementing matching of TGen streams/transfers and Tor
   streams/circuits twice.
 - Carefully review and possibly rewrite patch to be clearer.
 - Add change log entry.

Implements #33421.

Suggested by acute, still related to #40012.

Implements #40008.

Attempting to visualize analysis files containing only unsuccessful
measurements results in various tracebacks.

With this patch we check more carefully whether a data frame is empty
before adding a plot.

Another related change is that we always include
"time_to_{first,last}_byte" and "mbps" columns in the CSV output,
regardless of whether there are there are any non-null values in the
data. See also #40004 for a previous related change.

And we check whether a Tor stream identifier exists before retrieving
a Tor stream.

Finally, we only include TTFB/TTLB if the usecs value is non-zero.

Fixes #44012.

Before this change we would have replaced an existing "filters" field
with the filters applied in this run. But if the user runs `onionperf
filter` more than once to apply different filters, we'll want to list
all filters in the "filters" field.

This change also removes a couple of `self.`s, in particular
`self.analysis` and `self.filters`, because one Filtering object can
apply filters to more than one analysis.

This commit tweaks the recently added --drop-guards switch as follows:

 - Guards are dropped right at startup and then every N hours.
   Otherwise we might not receive the first round of GUARD NEW/UP
   events. It's unclear why we don't receive those events, but finding
   out might be time-consuming whereas dropping guards at startup is
   basically free.

 - Right after guards are dropped, circuit build timeouts are dropped,
   too, if supported by the Tor version. If the Tor version does not
   support this, there's going to be a warning, and the control log
   will simply not contain BUILDTIMEOUT_SET events.

Still part of the reopened #33399.

This new filter mode removes Tor circuits that don't match the
provided fingerprints and leaves TGen transfers/streams untouched. At
the same time the visualize mode only includes TGen transfers/streams
with an existing mapping between TGen transfers/streams and Tor
streams/circuits.

This patch changes the default behavior of the visualize mode. The
original behavior of visualizing TGen transfers/streams *without* an
existing mapping to Tor streams/circuits can be selected with the
--outer-join switch, even though that's rather an edge use case.

Another minor change is that the filtered analysis files is not
written with sort_keys=True anymore, which would have produced a newly
sorted file with keys in alphabetic order rather than the original
insert order. The result is an actually useful diff.