Instrument Tor client to count directory requests and bridge connections

When estimating user numbers, we have to make assumptions a) on the number of directory requests that an average user makes and b) on the average number of bridge connections that a bridge client opens.

We should instrument a Tor client to log all directory requests and bridge connections, so that we can derive better data from these logs. In fact, the info-level logs of the Tor clients that we use for Torperf may already tell us something about directory requests.

added component::metrics/analysis in Legacy / Trac priority::medium in Legacy / Trac resolution::wontfix in Legacy / Trac severity::normal in Legacy / Trac status::closed in Legacy / Trac type::task in Legacy / Trac labels

Trac:
Status: new to assigned
Owner: karsten to N/A

Trac:
Component: Metrics to Analysis

In legacy/trac#7009 (moved) we wanted to know how often clients download new consensuses. Changed log level of the torperfs on ferrinii to info, will report some stats here in a week from now.

Trac:
consensus-downloads.csv

Trac:

Attached a .csv and a .png file containing successfully loaded consensuses by ferrinii's tor clients. In particular, I'm counting occurrences of the following log line:

[info] connection_dir_client_reached_eof(): Successfully loaded consensus.

Trac:
Cc: N/A to arma

Excellent.

These data points do not match up with the assumption "clients fetch a new consensus every 2-4 hours"!

So there is either (1) a bug in how you're deciding that we fetched a new consensus, or (2) we are in fact fetching them more often than we thought (and, it would seem, more often than we need to).

Or I guess (3) Torperf's Tors are fetching them more often than regular Tors.

Trac:
torperf50kb-2013-07-04.log.gz

I just attached the info-level log of the 50KiB torperf from July 4. This is Tor version 0.2.2.35 (git-b04388f9e7546a9f). Also, here's the torrc:

DataDirectory .
SocksPort 9020
MaxCircuitDirtiness 1 minute
UseEntryGuards 0
RunAsDaemon 1
Log info file log
ControlPort 10020
CookieAuthentication 1
PidFile tor.pid

Am I looking at the right log lines, or what lines should I look at?

Also, please note that any change we make to the consensus download interval has huge impact on our user-counting algorithm. If we make future tors smarter about downloading consensuses less frequently, we'll have a hard time adapting our user-counting algorithm.

Trac:

Attached an updated graph and switched back to notice-level logging on ferrinii.

ahf: you might find this ticket interesting, as the torperf traces have a bunch of logs that could be reconstructed to answer questions about frequency of dir fetches.

Trac:
Reviewer: N/A to N/A
Severity: N/A to Normal
Sponsor: N/A to N/A
Cc: arma to arma, ahf

Closing tickets in Metrics/Analysis that have been created 5+ years ago and not seen progress recently, except for the ones that "nickm-cares" about.

Trac:
Status: assigned to closed
Resolution: N/A to wontfix

closed

mentioned in issue legacy/trac#7009 (moved)

moved from legacy/trac#2963 (moved)

added Task label and removed 1 deleted label

removed 1 deleted label