Skip to content

Unified shadow/chutney integration testing

Current state

We currently have two integration test frameworks in the arti repository:

  • In tests/shadow there is a test using the shadow simulator. It includes a ~manually generated tor network with checked-in keys, certificates, etc. Different "tests" are implemented primarily as arti clients and hidden services in various configurations, performing network transfers via tgen via tor exit nodes and/or to hidden services (running with tor and/or arti).
  • In tests/chutney there is a test using chutney, which runs natively on a local testing tor network. It doesn't do much right now, partly because chutney currently has fairly limited support for arti. In CI we run the whole chutney test in shadow to give us determinism and speed up "wait" times e.g. during network bootstrap. In the shadow test, the whole test still runs on one simulated host inside the simulation.

Goals

Functional goals

  • Deduplicate these 2 frameworks.
  • Avoid checked-in keys, certificates, etc as we currently have in the shadow test.
  • Make it easy to add tests and differently configured network nodes (currently somewhat true in the chutney test, less true in the shadow test)
  • Be able to take advantage of shadow's capabilities to run tests under various network conditions (high latency, low bandwidth, packet loss, etc) and with deterministic execution.
  • Also be able to run the tests natively (not under shadow) to ease interactive experimentation and debugging, potentially run on more platforms than just Linux (does chutney support other platforms? it seems like it could without much trouble), exercise functionality not supported by shadow (AF_UNIX, ipv6) and to give an "escape hatch" in case of any incompatibilities/issues that crop up with shadow.
  • It should be possible to run the tests under shadow, and run the tests natively, with no more than one shell script invocation.
  • It should be possible to re-run isolated parts of the integration test suite that might be failing.
  • We should able to write additional tests that run in the same environment as the network, including RPC tests and integration tests written in Rust. (This might require teaching Chutney how to pass information to tests not written as a part of Chutney)

Quality goals

  • Failures should be as easy as possible to diagnose.
  • The chutney test suite, run natively, should be over 99% reliable.
  • The chutney test suite, run under shadow, should be just about 100% reliable.
  • Minimize the divergence between "what happens in shadow" and "what happens without shadow". It's reasonable that some tests will only work under shadow (e.g. requiring delicate/deterministic timing) and some might work only without it (e.g. using OS functionality that shadow doesn't support, such as named pipes or ipv6), but these exceptions should be minimized and well documented.

Proposed approach

MVP

  • Extend chutney to:
    • Be able to bring up arti nodes in various configurations (e.g. clients, hidden services, and soon relays)
  • Migrate tests from tests/shadow to tests/chutney. This mostly means a custom chutney network with the arti nodes in various configurations that we have in tests/shadow. We can use chutney's current verify test to validate transfers between various node pairs analogously to what is being done in the tests/shadow tests instead of using tgen.
  • Additional functional tests can be added in the same style as chutney's verify test. chutney currently has a framework for network tests, where each test is a python module that provides a test function that takes a chutney Network object. verify is the only test in this framework, but we can add others. e.g. some of these could be adapters to invoke other tests or test frameworks while passing them the information they need to run against the configured chutney network.

Additional functionality

  • Teach chutney to generate (and maybe run) a shadow config file corresponding to running a given set of chutney tests against a given chutney network. We could then restructure the top level of tests/chutney a bit so that instead of using a separate wrapper to run the whole chutney test under shadow, we just have a suite of chutney tests that chutney runs either natively or under shadow.
  • Add support to distribute the chutney nodes on different shadow simulated hosts instead of running them all on a single host. (Maybe concurrently add support for running nodes on different IPs when run natively; e.g. 127.0.0.2, 127.0.0.3, ...)
  • Add support for specifying network link characteristics (bandwidth, latency, etc) when run under shadow. (Would probably be not supported when run natively, at least initially)

Additional code quality improvements

I've been doing a bit of work to make chutney more modular and bring it up to "modern" python practices that scale a bit better to chutney's complexity. e.g. getting rid of global variables, adding type annotations, etc. It's probably worth doing more in this area to ease further development and maintenance. This may include:

  • Continuing to flesh out type annotations; ideally being able to pass mypy --strict. (It now already passes mypy).
  • (Maybe) Migrate away from the Environment bag-of-attributes approach to a more structured class hierarchy. This would give us more benefit from python's type checking and modern IDE features (and is probably easier to do now that those exist)
  • (Maybe) Migrate from the bespoke config file templating system to inline templating. This is a bit nicer these days with e.g. python format string literals.

Development plan

  • For now, tests can continue to be added to tests/shadow
  • Once I've implemented the functionality needed to bring up arti nodes in chutney, I'll migrate tests from tests/shadow to tests/chutney
  • Similarly I should be able to migrate or help migrate any other integration tests that need to be written in the meantime (such as RPC tests)
Edited by Jim Newsome