Skip to content

extend shadow simulation testbed to measure performance with arti relays

We currently have a test-bed for running large (and small) scale tor network simulations under shadow using our gitlab CI runners. We would like to extend this test-bed to support running arti relays, to evaluate end-to-end network performance of a tor network using arti relays.

The test-bed currently lives at https://gitlab.torproject.org/jnewsome/sponsor-61-sims

That test-bed uses tornettools to generate the shadow network configurations, including tor configuration files. We'll need to extend tornettools to support configuring arti relays, which use a different configuration format. Given that we've already extended chutney to support configuring test networks with arti or tor nodes, and that chutney is now usable as a library, we should be able to deduplicate the config-generation in tornettools with chutney by having tornettools use chutney to generate the configurations, which will give us the option to configure some or all of the relays as arti nodes.

tornettools configures an oniontrace instance on each relay, which uses the tor control port to log some events, and uses the logs of those events for some metrics such as circuit build times and bandwidth usage. Since arti doesn't have a compatible control port, we'd either need to expose this data another way in arti and update the tornettools to use that, or just update tornettools to optionally not try to use oniontrace in this way and omit these metrics. We can do the latter initially to start running simulations sooner.

  • Migrate or fork the "sponsor 61 sims" project (which is used to configure and run tor shadow simulations on our gitlab runners) from jnewsome's personal namespace to the network team namespace
  • Extend that project to support using arti for a configurable fraction of relay nodes and get metrics that don't require oniontrace / control-port (transfer times for 50 KiB, 1 MiB, 5 MiB time to completion)
    • Extend chutney to support configuring nodes on different IP addresses (e.g. using alternate loopback addresses like 127.0.0.2 and 127.0.0.3 when run natively; or public addresses for use under shadow)
    • Migrate tornettools to use chutney to generate its tor configuration files
    • Expose options in tornettools to configure some or all relays to use arti instead of tor. This will also suppress setting up oniontrace on those relays, since it doesn't support arti. (Blocked on MVP arti relay implementation; issue # ?)
    • Expose options in our testbed script and CI variable to tell tornettools to use arti instead of tor for some relays.

Additional work to consider:

  • Schedule regular simulation runs (e.g. nightly or weekly) to track and monitor performance of arti @ head. (We do something very similar now in the shadow project to track performance and behavior of large tor simulations using shadow @ head)
  • Add support for arti clients in addition to tor clients. In one sense this isn't strictly in scope since this project is focused on arti relay performance. On the other hand we might expect that a significant fraction of clients will be using arti by the time that arti-relay rolls out, and hence will want to be sure that an arti-relay network is performant for arti clients as much as for tor clients. This also shouldn't take much more work, but we'll initially lose some metrics that we gather from the client control port (circuit build times).
  • Add support for metrics were previously gathered via control port (circuit build times; bandwidth usage). e.g. expose data via the arti metrics port or RPC API and extend tornettools etc to get it from there. This overlaps with #1555, since onionperf, used by the network health team's metrics pipeline, likewise uses the tor control port to gather this data.
  • Update tornettools to support adding "onionperf" nodes for gathering end-to-end metrics, instead of or in addition to its cobbled together mostly-equivalent (which I think was done instead of using onionperf because at the time shadow didn't support simulation processes launching other processes). This would ensure a more apples-to-apples comparison between simulation data and the metrics pipeline, and ease maintenance if we drop support for the previous "cobbled together onionperf" nodes.
  • Attempt to model the effects of merging multiple c-tor relays running on the same hardware with a single arti relay. We may need to do some work on CPU-time modeling in shadow to do a useful analysis here.