Write script to reconstruct hidden service time components from controller events

When we get #1944 (closed) going, we're going to start wondering what the breakdown of "time before connected cell" is.

Does torperf get its statistics of time breakdown just from the socks port, or does it learn things over the control port too, or what? We should figure out what time components we want to track, and then figure out how to export them so torperf can track them for posterity.

We may also find that we want to remember the circuits for the other components of the rendezvous, similar to the extension we're considering for #2551 (moved).

changed milestone to %Tor: 0.2.9.x-final

added component::archived/torperf milestone::Tor: 0.2.9.x-final owner::dgoulet points::medium priority::medium resolution::fixed severity::normal sponsor::R-must status::closed tor-hs type::task labels

Torperf, the C program, gets its timing information only from talking to Tor's SOCKS port. Torperf, the Python part, does learn things from the control port, namely path information and maybe circuit build times when we implement #2551 (moved). It should be possible to add hidden service time components to one of the Python scripts.

See also #2565 (closed) for some thoughts to redesign Torperf to add hidden service time components.

torperf2 does this by parsing the log events that come from the controller port.

https://github.com/aaronsw/torperf/

Trac:
Status: new to needs_review

Trac:
Owner: karsten to N/A
Status: needs_review to assigned

Trac:
Keywords: N/A deleted, TorPerfIteration20110305 added

Added points from sprint planning meeting.

Trac:
Points: N/A to 2

We didn't start working on this ticket in our March 5 Torperf sprint. Removing the keyword and bumping points to 4. Also setting priority to minor, because we should finish our substantial changes to Torperf first before looking at Torperf for hidden services.

Trac:
Component: Metrics to Torperf
Priority: normal to minor
Points: 2 to 4
Keywords: TorPerfIteration20110305 deleted, N/A added

See #3036 (closed) for thoughts on adding the various hidden service substeps to the .mergedata format.

Trac:
Priority: minor to normal
Keywords: N/A deleted, SponsorR added

Trac:
Cc: N/A to teor

I spoke to David about this process yesterday. Rob wrote a stem script to dump a bunch of controller events while we load an onion service. David is working on a parsing script to piece together the events for each onion service we're visiting, so we can a) understand which components of the onion service rendezvous worked, and b) track how long each component takes.

As a double bonus, Rob can run the same stem script on his OnionPerf, and then we can use the same parsing script to reconstruct the component timing and success rates on the OnionPerf actions.

And as a triple bonus, onion crawlers like Ahmia or SRI's crawler can dump these events too, and we can use the same script to reconstruct in that case too.

I'm going to revise the title here to indicate that torperf probably is no longer the place to do this reconstruction.

Trac:
Summary: extend torperf to record hidden service time components to write script to reconstruct hidden service time components from controller events
Cc: teor to teor, dgoulet

Here is Rob's stem script: https://github.com/robgjansen/tor-ctl-logger

Trac:
Keywords: SponsorR deleted, tor-hs added
Sponsor: N/A to SponsorR
Parent: #2552 (moved) to N/A
Cc: teor, dgoulet to teor
Milestone: N/A to Tor: 0.2.8.x-final
Status: assigned to accepted
Points: 4 to medium
Summary: write script to reconstruct hidden service time components from controller events to Write script to reconstruct hidden service time components from controller events
Owner: N/A to dgoulet
Type: enhancement to task

Trac:
Cc: teor to teor, rob.g.jansen@nrl.navy.mil
Severity: N/A to Normal

These seem like features, or like other stuff unlikely to be possible this month. Bumping them to 0.2.9

Trac:
Milestone: Tor: 0.2.8.x-final to Tor: 0.2.9.x-final

@rob: Is this something that OnionPerf does now and if yes, we should open specific tickets if we are missing some control events that we would like to track for the HS dance and close this ticket.

Trac:
Sponsor: SponsorR to SponsorR-must
Reviewer: N/A to N/A

Replying to dgoulet:

@rob: Is this something that OnionPerf does now and if yes, we should open specific tickets if we are missing some control events that we would like to track for the HS dance and close this ticket.

Hrm, actually I found #17627 (moved) about missing events for the "HS dance". I'm closing this in favor of that ticket and from the fact that we have OnionPerf doing what we want. Please re-open if you think otherwise.

Trac:
Status: accepted to closed
Resolution: N/A to fixed

OnionPerf does not currently do anything with HS_DESC events.

https://github.com/robgjansen/onionperf/blob/master/onionperf/analysis.py#L673

AIUI, #17627 (moved) is about adding additional events to Tor so that something like OnionPerf can actually link up everything. You may want to use this ticket to track the task of actually doing something with those new Tor events once they exist (i.e., updating OnionPerf to support HS_DESC events, or writing a new script if you don't want to do it in OnionPerf).

IMHO, OnionPerf is the correct place to add the new logic.

closed

mentioned in issue #2565 (closed)