Skip to content

Unified shadow/chutney integration testing

Current state

We currently have two integration test frameworks in the arti repository:

  • In tests/shadow there is a test using the shadow simulator. It includes a ~manually generated tor network with checked-in keys, certificates, etc. Different "tests" are implemented primarily as arti clients and hidden services in various configurations, performing network transfers via tgen via tor exit nodes and/or to hidden services (running with tor and/or arti).
  • In tests/chutney there is a test using chutney, which runs natively on a local testing tor network. It doesn't do much right now, partly because chutney currently has fairly limited support for arti. In CI we run the whole chutney test in shadow to give us determinism and speed up "wait" times e.g. during network bootstrap. In the shadow test, the whole test still runs on one simulated host inside the simulation.

Goals

Functional goals

  • Deduplicate these 2 frameworks.
  • Avoid checked-in keys, certificates, etc as we currently have in the shadow test.
  • Make it easy to add tests and differently configured network nodes (currently somewhat true in the chutney test, less true in the shadow test)
  • Be able to take advantage of shadow's capabilities to run tests under various network conditions (high latency, low bandwidth, packet loss, etc) and with deterministic execution.
  • Also be able to run the tests natively (not under shadow) to ease interactive experimentation and debugging, potentially run on more platforms than just Linux (does chutney support other platforms? it seems like it could without much trouble), exercise functionality not supported by shadow (AF_UNIX, ipv6) and to give an "escape hatch" in case of any incompatibilities/issues that crop up with shadow.
  • It should be possible to run the tests under shadow, and run the tests natively, with no more than one shell script invocation.
  • It should be possible to re-run isolated parts of the integration test suite that might be failing.
  • We should able to write additional tests that run in the same environment as the network, including RPC tests and integration tests written in Rust. (This might require teaching Chutney how to pass information to tests not written as a part of Chutney)

Quality goals

  • Failures should be as easy as possible to diagnose.
  • The chutney test suite, run natively, should be over 99% reliable.
  • The chutney test suite, run under shadow, should be just about 100% reliable.
  • Minimize the divergence between "what happens in shadow" and "what happens without shadow". It's reasonable that some tests will only work under shadow (e.g. requiring delicate/deterministic timing) and some might work only without it (e.g. using OS functionality that shadow doesn't support, such as named pipes or ipv6), but these exceptions should be minimized and well documented.

Development plan/record

  • Support running arti's CI chutney tests under shadow

    • Add options to disable functionality incompatible with shadow. chutney!21 (merged)
    • Add support to shadow to directly execute scripts, e.g. allowing tests that have outer scripts invoking chutney to be run under shadow https://github.com/shadow/shadow/pull/3401
    • Add a job that runs arti's current chutney tests under shadow. !2427 (merged)
    • Add a python package to shadow to support programmatic type-checked generation of configuration files, and shadow-exec wrapper for simplified usage when running a single "root" process in the simulation https://github.com/shadow/shadow/pull/3449
    • Move tests using chutney's network into "inner" chutney tests
    • Simplify using shadow-exec wrapper

  • chutney code-health and testing improvements to facilitate the planned changes while minimizing chance of breakage.

    • Support using chutney as a python module. It's not clear this is precisely needed with the current plan, but is generally useful and heavily overlaps with general code health improvements and making chutney work like a more idiomatic python project.
    • Ensure we don't break c-tor's usage of chutney.
      • Run c-tor's CI in chutney's CI. chutney!23 (merged)
      • Update c-tor to install chutney as a pip module so that python dependencies are automatically installed. tor!840 (merged)
    • Use python type annotations and mypy to statically detect and prevent errors.
    • Add CI tests for chutney's built-in networks
      • Add chutney test to validate that all built-in networks can be configured. chutney!35 (merged)
      • Make generated configs deterministic and diffable, so that changes can be detected and examined. chutney!44 (merged)
      • Add a (can-fail) test that fails whenever there are changes to the generated config files for the built-in networks. chutney!40 (merged)
    • Improve configuration management
      • Replace Environments (graphs of untyped key-value pairs) with more strongly typed alternatives and make data flow clearer. chutney!33 (merged)
      • General cleanup to make the configuration specification and output more understandable chutney!41 (merged)
    • Improve torrc config-file generation.
      • Replace bespoke templating system with python f-strings. chutney!33 (merged)
      • Unify torrc templates into a single base template, so that all configuration is specified via explicit configuration parameters. chutney!37 (merged)

  • Extend chutney to support adding arti nodes in various configurations (e.g. clients, hidden services, and soon relays)

  • Migrate tests from tests/shadow to tests/chutney. This mostly means a custom chutney network with the arti nodes in various configurations that we have in tests/shadow. We can use chutney's current verify test to validate transfers between various node pairs analogously to what is being done in the tests/shadow tests instead of using tgen.

  • Fix chutney bugs discovered during and/or blocking other work on this ticket

Future chutney/shadow roadmap

Items we want to do, but are not currently planned as part of this ticket, which will be closed when the MVP is completed. Any items left here when closing this ticket should probably first be broken into their own (backlog?) tickets.

  • Teach chutney to generate (and maybe run) a shadow config file corresponding to running a given set of chutney tests against a given chutney network. We could then restructure the top level of tests/chutney a bit so that instead of using a separate wrapper to run the whole chutney test under shadow, we just have a suite of chutney tests that chutney runs either natively or under shadow.
  • Add support to distribute the chutney nodes on different shadow simulated hosts instead of running them all on a single host. (Maybe concurrently add support for running nodes on different IPs when run natively; e.g. 127.0.0.2, 127.0.0.3, ...)
  • Add support for specifying network link characteristics (bandwidth, latency, etc) when run under shadow. (Would probably be not supported when run natively, at least initially)
  • Additional functional tests to be added in the same style as chutney's verify test. chutney currently has a framework for network tests, where each test is a python module that provides a test function that takes a chutney Network object. verify is the only test in this framework, but we can add others. e.g. some of these could be adapters to invoke other tests or test frameworks while passing them the information they need to run against the configured chutney network.

arti integration testing in the meantime

  • For now, tests can continue to be added to tests/shadow
  • Once I've implemented the functionality needed to bring up arti nodes in chutney, I'll migrate tests from tests/shadow to tests/chutney
  • Similarly I should be able to migrate or help migrate any other integration tests that need to be written in the meantime (such as RPC tests)
Edited by Jim Newsome