Skip to content
Snippets Groups Projects

IntegrationTesting

Last edited by Gaba

Content goes here

[org/sponsors/SponsorS/IntegrationTesting/Proposal]

Possible timeline:

(DRAFT DRAFT DRAFT)

The (1), (2), and (3) on items below refer to the three major proposal headings.

Start date for technical work: 2014 Oct 1

6 month milestone: 2015 April 1

Designs and specifications for controller improvements to allow better monitoring of performance, scalability, and overhead, and to allow better testing of Tor features. (3)

Design and specifications for a unified pluggable integration test framework where developers can add their own network tests. (1,2)

Design and specifications for improved means to decouple Tor modules for better module-level testing. (2)

Identification of highest-priority areas for integration test coverage, and specifications for test modules to validate those areas. (2)

Identification of least-testable areas in current code/design, and plans for replacing/mitigating/testing them. (2)

Alpha version of testing framework, possibly missing major areas of functionality, is implemented, and in use as a regular part of tor development. (1,2)

12 month milestone: 2015 Oct 1

Improve chutney templating to better handle complex multi-version configurations. (1)

Alpha version of unified test framework running, including chutney for launching a tor network, stem for scripting Tor nodes, a set of test-clients, and a set of ill-behaved testing clients to check for correct handling of protocol violations. (2)

Improvements to chutney to launch more kinds of Tor networks and test more Tor features. (1)

Alpha version of unified test framework running, including chutney for launching a tor network, stem for scripting Tor nodes, a set of test-clients, and a set of ill-behaved testing clients to check for correct handling of protocol violations. (2)

Initial data-driven work to simplify Tor's module interdependencies (*)

Improvements to chutney and test framework based on early reports from external developers. (1,2)

Test framework is stable enough to recommend to external developers. (2)

Tor test coverage (integration plus unit tests) at 60% or higher.

18 month milestone: 2016 Apr 1

Initial implementations of some high-value controller features, so we can start getting experience with how they work. (3) #17284

Further data-driven work to simplify Tor's module interdependencies. Most functions are extracted from the "spaghetti" core of the program. (*) (DONE)

Tor test coverage (integration plus unit tests) at 68% or higher. (*) (DONE)

Alpha version of module-isolation mechanism. (2) #17285

Test infrastructure includes a stable API and documentation on how to write extensions. All its components have received stable production-quality releases (2). #17282 , #17283

24 month milestone: 2016 Oct 1

Tor test coverage (integration plus unit tests) at 75% or higher. (*) #17289

Testing framework includes ill-behaved servers to verify correct behavior of network and clients in their presence. (2) #17290

All highest-priority areas identified by initial research have high (>>80%) test coverage. Most low-testability areas have been replaced, mitigated, or tested. (2) #17288

Further controller improvements TBD based on experiences using initial round of new controller features. (3) #17287

All desired controller features implemented. (3) #17287

Module isolation and module-level tests integrated into Tor proper (2). #17291

Further improvements TBD based on first 18 months' experience. (*)

Tracking our work

Core Tor:

"The first task is [1a] streamlining and automating the process of launching a complete testing Tor network, including Tor directory authorities, relays, bridges, clients, and hidden services, as well as client applications and destination services like websites. [1b] We need to continue to identify and resolve bugs in Tor's "testing network" configuration. [1c] We need to improve packaging and portability for these tools, so other researchers and developers can reproduce and extend our results."

"The second task is designing and scripting an automated test suite to exercise and stress as much of Tor's functionality as possible. These tests should come in the form of [2a] a) standalone unit tests, [2b] b) scripted clients that perform specific actions and expect certain results, and c) [2c] Tor controllers that exercise the control port interface as much as possible to observe and modify Tor's internal state. This test framework will allow us to investigate and reproduce bugs reported in the wild in a safe (with respect to user privacy) and controlled environment. We should also focus on getting our test framework to the point where other developers can devise and run their own tests."

"The third task, in parallel with the first two, is to extend Tor's controller interface to allow better monitoring. So far the control protocol has focused on exposing client state to local controllers (like the graphical interface); we will change it to export more details for relays, directory authorities, and hidden services as well. Combined with build automation (a separate project), the extended interface will also provide a more powerful framework for continuous evaluation of performance, scalability, and bandwidth overhead."

Comments

    Please register or sign in to add a comment.