Develop design sketch and prototype for bandwidth inflation detection tool

We want to be able to detect bandwidth lying by operators on the network. This ticket is the parent ticket for related tasks (which might need to be done in different projects) and could be used for sketching and sorting out all the design details.

added Next Sponsor 112 Q3 S112-O3 labels

assigned to @juga

/cc @mikeperry for help with the design sketch

marked this issue as related to tpo/network-health/metrics/library#40020 (closed)

marked this issue as related to tpo/network-health/metrics/descriptorParser#45 (closed)

marked this issue as related to tpo/network-health/metrics/descriptorParser#25 (closed)

marked this issue as related to tpo/network-health/metrics/onionperf#40066 (closed)

marked this issue as related to sbws#40156 (closed)

changed title from Develop design sketch and prototype for bandwidth inflation tool to Develop design sketch and prototype for bandwidth inflation detection tool

We're going to proceed in phases for this work: the early phase will be focused on just gathering relays with a low ratio according to either sbws or onionperf, and examining them. We want to answer the following questions:

• Are they mostly Exits, or Long-term guards? Do they share any other noticable characteristics?
• What does their reported bandwidth and bandwidth history look like?
• What does the circuit failure rate from onionperf look like for these relays?
• What are the relays with the highest onionperf circuit failure count, overall?
• Do any relays only appear in onionperf data, or only in sbws? Or do both tend to agree?
• What is the right ratio_param value? (We want more relays to examine early on, so higher is better, but later we should lower it, to reduce the relays reported)
• What is the right time_interval for onionperf? Is 1 week enough to cover most relays?
• Do the individual onionperf and sbws locations agree, or do they differ?
• Should we add filtering to onionperf data?
• Can we use any tools to get an additional measurement of the relay?

Here is a checklist for this initial prototype:

• Extract low ratio sbws relays (related: sbws#40156 (closed)) below ratio_param=0.50,0.66,0.75 (keep raising the ratio until a reasonable # of relays show up)
• Compute similar ratios in onionperf:
  • Calculate a network-wide download throughput average, for time_interval=1week, for each op7 location (separately)
  • Calculate a relay download throughput average, for time_interval=1week, for each relay as it appears in the Middle position
  • If any Guards or Exits do not appear in the middle position, use their values for other positions
  • Filter out any throughput values for a relay below the average/median for that relay
  • Divide the per-relay throughput average by the network-wide throughput average
• Record circuit failures in onionperf by recording a failure for a relay any time a circuit or download fails, and it is in the attempted path
• Answer questions based on initial analysis of this data

List of potential additional measurement tools:

• Manual test tool to test relays as a middle, via Tor
• Tools to measure IP address BW:
  • Can be used if relay answers ICMP:
    • https://github.com/oleg-derevenetz/bwping
    • https://github.com/hpcn-uam/iDPDK-LatencyMeter
  • Needs two endpoints (ie: cooperation of relay operator):
    • https://github.com/PearLab/abget
    • https://github.com/esnet/iperf
    • https://github.com/jsommers/yaz
    • https://github.com/brentondwalker/pathchirp
    • https://github.com/kristrev/bandwidth-estimator
    • https://github.com/microsoft/ethr

Cc: @hiro for the onionperf pieces

@mikeperry, i've some questions that maybe i figure out once we've some data and aren't urgent to reply:

What does their reported bandwidth and bandwidth history look like?

With reported bandwidth you mean the the observed bandwidth reported in the relay's server descriptor, the one reported by a bwauth in a bandwidth file that ends up as consensus weight or the measured stream bandwidth (also reported by a bwauth in a bandwidth file)?

And by bandwidth history, do you mean the previous bandwidth or the {read, write}-history reported in the server descriptor?

Extract low ratio sbws relays (related: sbws#40156 (closed)) below ratio_param=0.50,0.66,0.75 (keep raising the ratio until a reasonable # of relays show up)

What would be a reasonable number of relays?, 90%?, I guess we decide the threshold after having several consensuses (like a month of them?) and not on each consensus?

With reported bandwidth you mean the the observed bandwidth reported in the relay's server descriptor, the one reported by a bwauth in a bandwidth file that ends up as consensus weight or the measured stream bandwidth (also reported by a bwauth in a bandwidth file)?

Descriptor bandwidth.

And by bandwidth history, do you mean the previous bandwidth or the {read, write}-history reported in the server descriptor?

Yes, this.

What would be a reasonable number of relays?, 90%?, I guess we decide the threshold after having several consensuses (like a month of them?) and not on each consensus?

A reasonable number of low-ratio relays to investigate manually, to help answer the questions I wrote. Like 10-20 relays.

mentioned in issue tpo/network-health/metrics/monitoring-and-alerting#29 (closed)

mentioned in issue #319 (closed)

marked this issue as related to #319 (closed)

mentioned in issue #320 (closed)

marked this issue as related to #320 (closed)

mentioned in issue #321 (closed)

marked this issue as related to #321 (closed)

mentioned in issue tpo/network-health/metrics/onionperf#40069 (closed)