End-to-end network correctness testing
Achieving a privacy advantage at all with the VPN requires careful attention to app isolation, and right now there are no end-to-end tests for this mechanism and it contains several points of fragility:
- Race condition in evaluating status of ongoing connections
- Conflation of multiple apps via UID lookup failures
- Various paths to look up DNS, some of which may not be isolated reliably
- Potential quirks with specific apps
- Potential bugs in specific Android versions
I'd argue that before we put this in front of users we should have a stronger source of confidence that our isolation system is working correctly on supported devices. Some form of automated end-to-end testing for this seems essential. It seems appropriate for the client to run in an Android emulator or similar platform, while the server could be a mock replacement for Arti or it could be a full simulated Tor network.
While isolation is the most critical thing to get right, this same testing mechanism could also be used to test the correctness of the stack of transformations we're doing. Right now there's no testing that would catch bugs in our TCP implementation, and this could cover that ground.