From 46f144ec4d1cb825ea3d92de43226dc68c85bac0 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Thu, 17 Feb 2022 20:14:36 -0300 Subject: [PATCH 01/48] Feat: specs: inital threat model considerations and deployment overview --- specs.md | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/specs.md b/specs.md index 4cb375f..e9797fb 100644 --- a/specs.md +++ b/specs.md @@ -55,6 +55,9 @@ maintaining Onion Services. site, but experiments could be made to test if a site could have a dynamic number of backends. + This limitation means that Oniongroove would not initially benefit + from having a cloud native deployment, but that should also be considered. + 4. Uniformity with flexibility: looks like most (if not all) sites can have the same "CDN" fronting setup, while it's "last mile"/endpoints might be all different. That said, the "first half" of the solution could be based in the @@ -81,7 +84,36 @@ maintaining Onion Services. ## Threat Model -To be written. +Initial considerations: + +* Some setups might have security requirements such as hardened hosting + on premises/baremetal (i.e, outside the cloud or third-party providers). + + If backend keys are disposable, then the operators could consider whether + only the frontend instance(s) should be in an evironment of maximum control + (such as a hardened/secured baremetal on their datacenter). Backend keys + could then be periodically rotated (as a mitigation for the current lack + of implementation for offline .onion keys) ou replaced in case of detected + compromise, reducing eventual damages to shorter periods (if Onionbalance + support such rotations). + +## Deployment + +### Overview + +Consider a two-tier approach, where sysadmins keep a copy of all .onion keys +(and optionally TLS certs) an the entire Onion Service Operation can be +redeployed if needed (eg. disaster recovery, switching providers or even in +case the deployment procedure/stack changes and is found that a redeployment +is less costly than keeping an existing installation). + +1. Vendor-specific tier: module(s) that bootstrap the basic environment (like a + system with shell, could be based on Terraform), taking care of + vendor-specific logic depending on the hosting platform chosen. + +2. Vendor neutral tier: a module that bootstraps the Onionbalance frontend(s), + backends load balancers etc (could use Ansible for that, even Docker). + This step should be vendor-neutral. ## Relevant issues -- GitLab From d9d8fa6827976900581c259715937c441156b8d8 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Fri, 18 Feb 2022 15:42:12 -0300 Subject: [PATCH 02/48] Feat: incorporate content from the skill test --- specs.md | 252 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 252 insertions(+) diff --git a/specs.md b/specs.md index e9797fb..72c4496 100644 --- a/specs.md +++ b/specs.md @@ -97,6 +97,99 @@ Initial considerations: compromise, reducing eventual damages to shorter periods (if Onionbalance support such rotations). +## Architecture + +Architectural choices for the Onionbalance backends: + +* Given that a single Tor daemon can host many services, a single horizontal + layer or "wall" of nodes/VMs/containers running Tor could act as Onionbalance + backends for many sites. + +* Another alternative, which is more resilient but demands more + resources, is to have one horizontal layer of nodes acting as Onionbalance + backends for each onion site. + +Topological choices for the Onionbalance frontends: + +* From the Onionbalance docs, looks like that a single instance can manage + many frontends, each one with multiple associated backends. + +* Additional frontend nodes could act as failover, being activated only + if the main one fails. + +For the connection between the Onionbalance backends and the onion site +endpoint: + +* Preferably run the Onionbalance backend in the same host as the application + endpoint. + +* For third parties endpoints (partner's sites): + * Additional Onion Service link? + + * VPN (Wireguard) link? + + * [OnionCat](https://www.onioncat.org) link? + + * Authenticated (or verified) HTTPS link? + +## Topology + +### Overall functioning + +The proposed sollution is composed by services in the following rough sketch: + + onionbalance1 -> frontend1 + + backend1 -> proxy1 -> endpoint1 ... endpointN + backend2 -> proxy2 -> endpoint1 ... endpointN + ... + backendN -> proxyN -> endpoint1 ... endpointN + +### Types of services + +Main services: + +* endpoints: each endpoint service can host an arbitrary number of sites + using Apache Virtual Hosts and with a default `ServerName`, and + hence no `ServerName blablabla.onion` is needed; a layer of endpoints gives + redundancy up to `N - 1` failing endpoints. +* proxies: each proxy service runs an NGINX instance configured to serve all websites + from all endpoints; each website is also run on it's own port and a reverse proxy + is set with all endpoints as upstreams; this also gives additional redundancy; + port number to server name conversion happens at this stage. +* backends: each backend service runs a Tor instance running an Onion Service + for each site; `HiddenServicePort` is set according to each sites' TCP port + in the proxy; so, for redundancy, the Onion Service for each site is linked to + an NGINX instance which is linked to all application endpoints. +* frontends: runs a Tor instance to act as the Onion Service frontend for each + Onion Site; right now just a single frontend is supported, so redundancy is + this level should be implemented with high availability config where a spare + service with the same Tor data starts only if the current frontend goes + unresponsive. +* onionbalance: runs an instance of Onionbalance; like the frontend service, + only a single service is supported and needs additional high availability + setup; controls the frontend service for all the sites. + +Additional services: + +* monitor: simple monitoring nodes using `mon`. +* tor: nodes with Tor daemons used by the monitoring nodes. + +### Service grouping + +* Each backends and it's proxy counterpart (say `backend1` and `proxy1` might + be grouped together in the same machine). + +* Each frontend and it's onionbalance counterpart (say `frontend1` and + `onionbalance`) might be also grouped in a single machine. + +* Each backend might stay in a separate machine/location. + +* Each endpoint might stay in a separate machine/location. + +* Each monitor node could be grouped with it's tor counterpart (say `monitor1` + and `tor1`). + ## Deployment ### Overview @@ -115,6 +208,165 @@ is less costly than keeping an existing installation). backends load balancers etc (could use Ansible for that, even Docker). This step should be vendor-neutral. +### Strategies + +Assumptions: + +* Given: + * The Onion Service key of the Onionbalance frontend. + * The configuration. + * The software sollution. + * A bunch of nodes. + +* It might be possible to (re)bootstrap the whole "CDN" by using an + orquestration tool. + +Workflow: + +* Use a orchestration tool such as Ansible, Puppet, Terraform: + * I would not stick at first with and specific tool but instead + would check if the Tor Project and Community have some conventions + and preferences regarding the usage of those tools. + +* Onion Service keys could be either: + 1. Generated locally at the operator laptop and then deployed + remotelly along with the configuration OR + + 2. Generated directly in the machines hosting the Onion Service. + +* Generating keys locally have the following advantages: + * Easier to backup the keys (using GnuPG optionally with a tool + like keyringer with multiple recipients and digital signature). + + * Easy to redeploy the whole frontend/backend infrastructure if + needed. + +* They Onionbalance frontend address should be considered as + long term and hence it's keys needs to be stored on encrypted + backups with some redundancy (both physical and human). + +Additional checklist: + +* Basic security requirements: running Onion Services is more sensitive than + relays, so additional security checks and measures are needed: maybe an Onion + Service for mission critical (in terms of data protection) is even more + sensitive than an exit node: + * More careful service provider selection, or even better: run Onion Services + on premises. + + * System checks: disk encryption, firewall and other protections. + + * Using Tor packages from the official Tor repositories instead of the + operating system's distributtion repositories (Debian for instance). + + * The application server (be an webserver or anything else) cannot leak + identifiable information such as hostnames or IP addresses (disabling + ServerTokens on Apache). + + * Other recomended and best practices from the Tor and Riseup documentations. + +* Defenses and SPOFs (single points of failure) avoidance: + * Use Onionbalance for Onion Service scalability. + + * Apply other anti-DoS measures. + + * Setup The Vanguards Onion Service Addon on each Tor daemon (both + Onionbalance frontend and backend). + + * Encrypted offline backups of Onion Service keys: at least from the Onion + Balance frontend service, because the backends _might_ be disposable + (needs checking). + + * Using automated and deterministic admin tasks prevents SPOFs represented + by the absency of the Onion Site operator, as automation allows more people + do take care of the maintenance. + + * [Stealth mode (`HiddenServiceAuthorizeClient`) between any connections between services + behind Onion Services?] + +* Performance: + * For public-facing (known IP addresses), + consider [single onion service](https://support.torproject.org/glossary/single-onion-service/), + using `HiddenServiceSingleHopMode`. + + * Use UNIX sockets if possible: they seems faster and more secure as no + TCP is involved. + +* Service discovery for public-facing sites: + * Onion-Location HTTP header in the endpoint. + + * Alt-Svc Header. + +## Performance & Anti-DoS measures + +* Basic CDN functionalities such as caching to avoid excessive access of the endpoints. + +* Adopt recomendations suggested at the Tor docs, such as rate limiting + (`HiddenServiceExportCircuitID`), client authorization and CAPTCHAS, + depending of course on the site characteristics. + +* Consider to avoid HTTPS in the Onion Service: + * HTTPS required additional CPU consumption, especially for the handshake, + which could worsen the consequences of a DoS attack (CLIENT HELLO flooding). + + * Since HTTPS is served from an Onion Service, tradditional DoS HTTPS mitigations + would not work (like `fail2ban` or firewall rate limiting). + +* OR consider HTTPS with a special a cipher suite: + * Hypothesis do be researched/tested: perhaps an Onion Service with HTTPS + with a proper cipher suite might provide *both* an additional layer + of security and some rough kind of "proof of work" by letting the the client + use more CPU than the server; but right now I don't know the state of the + art on cipher suites and if there are a choice which provides both properties. + + * But even with this properties, HTTPS over Onion Service might be still costly + in comparison with plain HTTP over Onion Service. + + * Only with more research and testing this can be decided. + + * So, initially, it would be best to start by only using HTTP over Onion Service + and, if HTTPS proves to be useful, introduce it later with HSTS header and + HTTP-to-HTTPS redirection. + +## Metrics + +Collecting: some approaches do collect metrics from the Onion Service: + +* Use a proxy middleware that parses HTTP requests and responses, processing the + minimum metadata needed for stats on pages hits etc. + +* Read data from the ControlPort, preferably as an unix socket, but that needs + additional security protections. + +* Use the new MetricsPort config. But does this param support unix sockets as well? + +* Perhaps the Stem python library has some functionality to monitor Onion Services. + +* Onionbalance also supports a unix status socket. + +* Log parsing. But which useful information on Onion Service is available at + Tor daemon's log? + +Aggregating: + +* Temporal and other metrics aggregation to avoid any traffic correlation. + +* Send it to Prometheus or other system. + +* Setup a private metrics/monitoring dashboard for the CDN. + +Testing: + +* Setup external probes using OnionScan and other monitoring tools to test + the heaht of the Onion Service from the outside. They could probe *both* + the Onionbalance frontend and the backends. They do not need to be + positioned in any special place in the internet as long as they can + access the Tor network, but preferably in a different network from + the Onionbalance CDN. + +* Internal probe between each Onion Service backend and the application + endpoint. + ## Relevant issues Some of the relevant [Onion Services issues](https://gitlab.torproject.org/groups/tpo/-/issues?label_name%5B%5D=Onion+Services&scope=all&state=opened) for the Oniongroove project: -- GitLab From 90ee613808548969944b79ec31cc334937c9da54 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Fri, 18 Feb 2022 18:04:16 -0300 Subject: [PATCH 03/48] Feat: specs: some more assumptions --- specs.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/specs.md b/specs.md index 72c4496..a51cc85 100644 --- a/specs.md +++ b/specs.md @@ -212,7 +212,8 @@ is less costly than keeping an existing installation). Assumptions: -* Given: +* System should behave like an "Onion As a Service". This means that, + given: * The Onion Service key of the Onionbalance frontend. * The configuration. * The software sollution. @@ -221,6 +222,11 @@ Assumptions: * It might be possible to (re)bootstrap the whole "CDN" by using an orquestration tool. +* Redeployment: not only needed, but planned with minimized downtimes and + wiping out keys from the old, decomissioned systems. + +* Versioned config file format (YAML). + Workflow: * Use a orchestration tool such as Ansible, Puppet, Terraform: -- GitLab From e6f9b6ee00e69c686c0df1bf6162471f0090aa28 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Mon, 21 Feb 2022 18:12:48 -0300 Subject: [PATCH 04/48] Feat: docs structure * Move docs to separate folder and files * Adds threat model references --- specs.md => doc/specs.md | 15 --------------- doc/threat.md | 24 ++++++++++++++++++++++++ 2 files changed, 24 insertions(+), 15 deletions(-) rename specs.md => doc/specs.md (96%) create mode 100644 doc/threat.md diff --git a/specs.md b/doc/specs.md similarity index 96% rename from specs.md rename to doc/specs.md index a51cc85..2e2061a 100644 --- a/specs.md +++ b/doc/specs.md @@ -82,21 +82,6 @@ maintaining Onion Services. the Tor daemon currently does not support this feature, requiring a high level of operational security to protect Onion Services' keys. -## Threat Model - -Initial considerations: - -* Some setups might have security requirements such as hardened hosting - on premises/baremetal (i.e, outside the cloud or third-party providers). - - If backend keys are disposable, then the operators could consider whether - only the frontend instance(s) should be in an evironment of maximum control - (such as a hardened/secured baremetal on their datacenter). Backend keys - could then be periodically rotated (as a mitigation for the current lack - of implementation for offline .onion keys) ou replaced in case of detected - compromise, reducing eventual damages to shorter periods (if Onionbalance - support such rotations). - ## Architecture Architectural choices for the Onionbalance backends: diff --git a/doc/threat.md b/doc/threat.md new file mode 100644 index 0000000..cdfb29b --- /dev/null +++ b/doc/threat.md @@ -0,0 +1,24 @@ +# Oniongroove Threat Model + +## Threat Model + +Initial considerations: + +* To not be repetitive or duplicate efforts, Oniongroove threat model begins by + inheriting the following more general analysis: + * [Tor: The Second-Generation Onion Router - Threat Model](https://svn-archive.torproject.org/svn/projects/design-paper/tor-design.html#subsec:threat-model). + + * [The Design and Implementation of the Tor Browser - Adversary Model](https://2019.www.torproject.org/projects/torbrowser/design/#adversary) + + * [GitHub - Attacks-on-Tor/Attacks-on-Tor: Thirteen Years of Tor Attacks](https://github.com/Attacks-on-Tor/Attacks-on-Tor#threat-model) + +* Some setups might have security requirements such as hardened hosting + on premises/baremetal (i.e, outside the cloud or third-party providers). + + If backend keys are disposable, then the operators could consider whether + only the frontend instance(s) should be in an evironment of maximum control + (such as a hardened/secured baremetal on their datacenter). Backend keys + could then be periodically rotated (as a mitigation for the current lack + of implementation for offline .onion keys) ou replaced in case of detected + compromise, reducing eventual damages to shorter periods (if Onionbalance + support such rotations). -- GitLab From 80b02830fb5dbe9a09b8ad19d998dc05547d3aed Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Tue, 22 Feb 2022 18:53:21 -0300 Subject: [PATCH 05/48] Feat: specs: section for open questions --- doc/specs.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/specs.md b/doc/specs.md index 2e2061a..20794b5 100644 --- a/doc/specs.md +++ b/doc/specs.md @@ -402,3 +402,7 @@ Some of the relevant [Onion Services issues](https://gitlab.torproject.org/group * load balancers (to be plugged in the endpoint) * monitor nodes * probe nodes + +## Open questions + +See [these questions about current Onionbalance limits](https://gitlab.torproject.org/tpo/community/team/-/issues/57#note_2780708). -- GitLab From 73d66301c23f14478d5cef079b37799d6e86fa97 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Wed, 23 Feb 2022 15:26:33 -0300 Subject: [PATCH 06/48] Fix: move logo to assets folder --- README.md | 2 +- logo.jpg => assets/logo.jpg | Bin 2 files changed, 1 insertion(+), 1 deletion(-) rename logo.jpg => assets/logo.jpg (100%) diff --git a/README.md b/README.md index 69b1b62..c6cc632 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Oniongroove -![Oniongroove Logo](logo.jpg "Oniongroove") +![Oniongroove Logo](assets/logo.jpg "Oniongroove") The Oniongroove plans to be a High Availability [Onion Service](https://community.torproject.org/onion-services/) Deployment and diff --git a/logo.jpg b/assets/logo.jpg similarity index 100% rename from logo.jpg rename to assets/logo.jpg -- GitLab From 9a335b306601ca18aa4c6d965a46bb74744bfe87 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Wed, 23 Feb 2022 15:27:14 -0300 Subject: [PATCH 07/48] Feat: specs: folder scheme, basic UX and open questions --- doc/specs.md | 58 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 57 insertions(+), 1 deletion(-) diff --git a/doc/specs.md b/doc/specs.md index 20794b5..6bab689 100644 --- a/doc/specs.md +++ b/doc/specs.md @@ -403,6 +403,62 @@ Some of the relevant [Onion Services issues](https://gitlab.torproject.org/group * monitor nodes * probe nodes +## Folder scheme + +Folder scheme relies on the [XDG Base Directory +Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) +v0.8: + +* `$XDG_CONFIG_HOME/oniongroove`: local configurations for oniongroove, + i.e, not shared between operators. Subfolders: + * `$XDG_CONFIG_HOME/oniongroove/providers/`: + operator configuration for a given provider. Per-persona settings, + not meant to be shared between operators, so it's left outside the + provider git repository. + +* `$PROVIDER_PATH`: arbitrary path specified by the operator when + setting up (or cloning) a new provider repository. Used to store + the local working copy of the provider's Git repository. Subfolders + and files: + * `$PROVIDER_PATH/oniongroove.yaml`: main config file. + +## UX + +Example CLI invocations and behaviors. + +### Creating a provider + + oniongroove create \ + [--path + + -> Check if provider already exists + -> Check if a local operator config for that provider exists + -> Check if OpenPGP keypair is available + -> Create OpenPGP keypair if needed + -> Setup provider folder + -> Clone provider git repository if that was specified + -> (Optionally) check latest commit OpenPGP signature + -> Create provider git repository from template and using operator name/email + -> (Optionally) Setup OpenPGP repository signature + +### Adding nodes + +* To be defined. + +### Adding Onion Services + +* To be defined. + +### Other subcommands + +* Git wrapper. + ## Open questions -See [these questions about current Onionbalance limits](https://gitlab.torproject.org/tpo/community/team/-/issues/57#note_2780708). +* Onionbalance: see [these questions about current Onionbalance limits](https://gitlab.torproject.org/tpo/community/team/-/issues/57#note_2780708). +* Password sharing mechanism: see discussion in this ticket: [evaluate password management options (#29677)](https://gitlab.torproject.org/tpo/tpa/team/-/issues/29677). -- GitLab From ba7f1ed7177b5626aeab32d9e01cc911f1206dca Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Fri, 25 Feb 2022 17:30:07 -0300 Subject: [PATCH 08/48] Some work in the docs: * Setup mkdocs. * Some changes in the specs. --- .gitignore | 1 + README.md | 2 +- docs/index.md | 5 +++ {doc => docs}/specs.md | 70 ++++++++++++++++++++++++++++++++--------- {doc => docs}/threat.md | 0 mkdocs.yml | 22 +++++++++++++ www/.gitignore | 0 7 files changed, 84 insertions(+), 16 deletions(-) create mode 100644 .gitignore create mode 100644 docs/index.md rename {doc => docs}/specs.md (86%) rename {doc => docs}/threat.md (100%) create mode 100644 mkdocs.yml create mode 100644 www/.gitignore diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..f8d728c --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +www diff --git a/README.md b/README.md index c6cc632..3503931 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ ![Oniongroove Logo](assets/logo.jpg "Oniongroove") -The Oniongroove plans to be a High Availability [Onion +The Oniongroove is planned to be a High Availability [Onion Service](https://community.torproject.org/onion-services/) Deployment and Monitoring system, giving groove to the [Onionbalance](https://gitlab.torproject.org/tpo/core/onionbalance/) with diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..eb44aed --- /dev/null +++ b/docs/index.md @@ -0,0 +1,5 @@ +# The Oniongroove Documentation + +* [Repository](https://gitlab.torproject.org/rhatto/oniongroove). +* [Design and specification](specs). +* [Threat Model](threat). diff --git a/doc/specs.md b/docs/specs.md similarity index 86% rename from doc/specs.md rename to docs/specs.md index 6bab689..263232a 100644 --- a/doc/specs.md +++ b/docs/specs.md @@ -12,9 +12,24 @@ > 4. (music) A pronounced, enjoyable rhythm. Goal: to have balance, rhythm and smoothness in setting up, monitoring and -maintaining Onion Services. +maintaining [Tor Onion Services](https://community.torproject.org/onion-services/). -## Discovery phase +## Preliminaries + +* The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", + "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to + be interpreted as described in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). + +* This specifications takes the meaning for terms like Operator, User and + Client from the + [Tor Rendezvous Specification - Version 3](https://gitlab.torproject.org/tpo/core/torspec/-/blob/main/rend-spec-v3.txt). + +* It's assumed familiarity with Tor Onion Services, [Onion Balance](https://onionbalance.readthedocs.io) + and the [Vanguards Add-On](https://blog.torproject.org/announcing-vanguards-add-onion-services/). + +## Guiding questions + +The following basic questions oriented the overall Oniongroove design: 0. Architecture: which are the possible architectures to build an Onion balanced service? @@ -27,23 +42,25 @@ maintaining Onion Services. ## Scope -1. Key generation: such suite could generate all .onion keys locally (sysadmin's - box), encrypting and pushing to a private repository (at least for the - frontend onion keypair for each site/service). Then other sysadmins could clone - that internal repository and start managing the sites/machines. +1. Key generation: all .onion keys MUST be generated locally -- i.e, at the + operators's box prior to deployment -- encrypted and optionally pushed to a + private repository. This allows other members of the operators team to clone + that internal repository and start managing sites and nodes. This ensures fast + disaster recovery and also avoid the need for a backup system just for onion + balancing. -2. Disposability: depending on design choice, the frontend .onion address - could be the only persistent data, and everything else could be - disposable/recycled/recreated in case of failure or major infrastructure/design - revamp.. +2. Disposability: depending on design choice and implementation availability at + Oniobalance, the frontend .onion address MAY be the only persistent data, + and everything else MAY be disposable/recycled/recreated in case of failure or + major infrastructure/design refactoring. That of course depends if Onionbalance supports backend rotation. But even in the case that backend .onion keys need also to be backed up, the system - could be renegerated with only the combination with the software suite, + could be regenerated with only the combination with the software suite, the keys and the custom configuration. -3. Elasticity: that leads to the follow requirement: shall this system be such - that backend nodes can be added and removed at will? In a way that enables +3. Elasticity: disposability properties leads to the OPTIONAL feature in which + backend nodes can be added and removed at will? In a way that enables the system to be elastic in the future, adding and removing nodes according to average load or sysadmin command? Or that would need the Onionbalance instance to be restarted (resulting in unwanted downtimes), or even breaking it? @@ -82,6 +99,10 @@ maintaining Onion Services. the Tor daemon currently does not support this feature, requiring a high level of operational security to protect Onion Services' keys. +9. No lock-in: operators aren't required to use the Oniongroove suite in order to + implement it's functionality. They can instead selectively pick some subsystems + and plug into their existing deployment workflow. + ## Architecture Architectural choices for the Onionbalance backends: @@ -460,5 +481,24 @@ Example CLI invocations and behaviors. ## Open questions -* Onionbalance: see [these questions about current Onionbalance limits](https://gitlab.torproject.org/tpo/community/team/-/issues/57#note_2780708). -* Password sharing mechanism: see discussion in this ticket: [evaluate password management options (#29677)](https://gitlab.torproject.org/tpo/tpa/team/-/issues/29677). +* Onionbalance: + * See [these questions about current Onionbalance limits](https://gitlab.torproject.org/tpo/community/team/-/issues/57#note_2780708). + + * Backend rotation: + * Does Onionbalance really need persistent backend addresses? From what was + investigated until now, it seems that what matters is that: + * The backend addresses are currently published . + + * Every backend has both `HiddenServiceOnionBalanceInstance` and + `MasterOnionAddress` configuration options set. + + * When rotating backends, it's important to not just turn off the older + backends, but to keep then online until all copies of the older + superdescriptor are expired/replaced with the updated list of introduction + points. + + * But this behavior MUST be tested in practice before considering any + backend rotation feature. + +* Password sharing mechanism: see discussion in this ticket: [evaluate password + management options (#29677)](https://gitlab.torproject.org/tpo/tpa/team/-/issues/29677). diff --git a/doc/threat.md b/docs/threat.md similarity index 100% rename from doc/threat.md rename to docs/threat.md diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 0000000..21d44d8 --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,22 @@ +# +# Mkdocs configuration +# + +site_name: The Oniongroove +docs_dir : docs +site_dir : www +dev_addr : '127.0.0.1:8080' + +#theme: +# name : material +# collapse_navigation: true +# titles_only : false +# +#extra_css: +# - site.css +# +#plugins: +# - search +# +#markdown_extensions: +# - footnotes diff --git a/www/.gitignore b/www/.gitignore new file mode 100644 index 0000000..e69de29 -- GitLab From ae76df9bbc82a7f01da7e619b7c55690389a7ed8 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Tue, 1 Mar 2022 19:48:24 -0300 Subject: [PATCH 09/48] Feat: specs: enhancements --- README.md | 4 +-- docs/specs.md | 74 ++++++++++++++++++++++++++++++++++++++++++--------- 2 files changed, 64 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 3503931..92bf1fe 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Oniongroove +# Oniongroove: Deployment tool for the Onion Web ![Oniongroove Logo](assets/logo.jpg "Oniongroove") @@ -15,4 +15,4 @@ using the [Tor Network](https://torproject.org). ## Status This software suite is still in the planning stage! -See the [specs document](specs.md) for details. +See the [specs document](docs/specs.md) for details. diff --git a/docs/specs.md b/docs/specs.md index 263232a..574012c 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -1,4 +1,4 @@ -# Oniongroove Design and Specification +# Oniongroove Design and Specification v0.0.1 > [groove](https://en.wiktionary.org/wiki/groove), n: > @@ -27,20 +27,27 @@ maintaining [Tor Onion Services](https://community.torproject.org/onion-services * It's assumed familiarity with Tor Onion Services, [Onion Balance](https://onionbalance.readthedocs.io) and the [Vanguards Add-On](https://blog.torproject.org/announcing-vanguards-add-onion-services/). +* The terms "suite" and "software suite" both refers implicitly to Oniongroove. + ## Guiding questions The following basic questions oriented the overall Oniongroove design: 0. Architecture: which are the possible architectures to build an Onion - balanced service? + balanced service? Knowing that there's no "one size fits all" choice, + which service topology could work best for most use cases? 1. Implementation: which are the available stacks/tools to implement those - architectures? + architectures? Ideally, the suite should not be tightly coupled or + heavily dependent with any specific implementation. 2. Support: which are the most common hosting environments that should be supported? -## Scope +## Features + +Oniongroove design focuses in some useful properties for Onion Services +security and high availability: 1. Key generation: all .onion keys MUST be generated locally -- i.e, at the operators's box prior to deployment -- encrypted and optionally pushed to a @@ -130,26 +137,69 @@ endpoint: endpoint. * For third parties endpoints (partner's sites): - * Additional Onion Service link? - * VPN (Wireguard) link? * [OnionCat](https://www.onioncat.org) link? * Authenticated (or verified) HTTPS link? + * Could even be a collection of .onion addresses (perhaps each one even be + set using Oniongroove), but that would probably defeat the purpose of + having an .onion fronting in the first place ;) + ## Topology ### Overall functioning -The proposed sollution is composed by services in the following rough sketch: +The proposed solution is composed by services in the following diagram: + +```mermaid +graph TD + + subgraph "Oniongroove CDN" + + B1[Tor backend 1] --> L1[Load balancer 1] + B2[Tor backend 2] --> L2[Load balancer 2] + B3[Tor backend 3] --> L3[Load balancer 3] + Bn[Tor backend n] --> Ln[Load balancer n] + + end + + subgraph "Application" + + L1 --> E1 + L1 --> E2 + L1 --> E3 + L1 --> En + + L2 --> E1 + L2 --> E2 + L2 --> E3 + L2 --> En + + L3 --> E1 + L3 --> E2 + L3 --> E3 + L3 --> En + + Ln --> E1 + Ln --> E2 + Ln --> E3 + Ln --> En + + E1[Endpoint 1] + E2[Endpoint 2] + E3[Endpoint 3] + En[Endpoint n] + + end + + subgraph "Oniongroove fronting" - onionbalance1 -> frontend1 + O1[Onionbalance 1] --> F1[Tor Frontend 1] --> H1[Tor HSDir] - backend1 -> proxy1 -> endpoint1 ... endpointN - backend2 -> proxy2 -> endpoint1 ... endpointN - ... - backendN -> proxyN -> endpoint1 ... endpointN + end +``` ### Types of services -- GitLab From e7f67373f8e4d9b2b13d13d7280e3a449b22c88c Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Wed, 2 Mar 2022 10:36:10 -0300 Subject: [PATCH 10/48] Feat: specs: some more scattered details --- docs/specs.md | 75 ++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 60 insertions(+), 15 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index 574012c..bbc6bba 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -11,8 +11,12 @@ > > 4. (music) A pronounced, enjoyable rhythm. -Goal: to have balance, rhythm and smoothness in setting up, monitoring and -maintaining [Tor Onion Services](https://community.torproject.org/onion-services/). +## Goals + +1. Give balance, rhythm and smoothness in setting up, monitoring and + maintaining [Tor Onion Services](https://community.torproject.org/onion-services/). + +2. Increase Onion Service adoption by providing such a tool. ## Preliminaries @@ -29,9 +33,15 @@ maintaining [Tor Onion Services](https://community.torproject.org/onion-services * The terms "suite" and "software suite" both refers implicitly to Oniongroove. +* Every specification deals with the difficulty of defining all the relevant + details of a would-be universe where instances of a system exists, provided + that the specification is carefully followed. Question remains of not only what + should be added but what should be removed until only the necessary (while + never sufficient) is there. This current specification is no exception. + ## Guiding questions -The following basic questions oriented the overall Oniongroove design: +The following basic questions oriented the overall Oniongroove planning: 0. Architecture: which are the possible architectures to build an Onion balanced service? Knowing that there's no "one size fits all" choice, @@ -49,6 +59,9 @@ The following basic questions oriented the overall Oniongroove design: Oniongroove design focuses in some useful properties for Onion Services security and high availability: +0. Reusability: a single Oniongroove instance MAY host more than a single + .onion site. The total number of hosted .onion sites SHOULD not be limited. + 1. Key generation: all .onion keys MUST be generated locally -- i.e, at the operators's box prior to deployment -- encrypted and optionally pushed to a private repository. This allows other members of the operators team to clone @@ -110,6 +123,14 @@ security and high availability: implement it's functionality. They can instead selectively pick some subsystems and plug into their existing deployment workflow. +10. Requirements: Oniongroove CDN requirements MUST be low. + The system can run in parallel along with the existing web sites available + via the regular internet and can be hosted in a different infrastructure if + needed. + Minor mofifications in existing systems are needed only for OPTIONAL + security and circunvention features like HTTPS for .onion addresses and the + Onion-Location header for service discovery. + ## Architecture Architectural choices for the Onionbalance backends: @@ -201,21 +222,28 @@ graph TD end ``` +Please note that the diagram above is simplified: + +1. In fact, each .onion site can have different number of endpoints, even a + single one. +2. Metrics and monitoring subsystems are not displayed. + ### Types of services Main services: -* endpoints: each endpoint service can host an arbitrary number of sites - using Apache Virtual Hosts and with a default `ServerName`, and - hence no `ServerName blablabla.onion` is needed; a layer of endpoints gives - redundancy up to `N - 1` failing endpoints. -* proxies: each proxy service runs an NGINX instance configured to serve all websites - from all endpoints; each website is also run on it's own port and a reverse proxy - is set with all endpoints as upstreams; this also gives additional redundancy; - port number to server name conversion happens at this stage. +* endpoints: each endpoint service can be a entry point for upstream websites; + as an example, they could be Apache web servers using Apache Virtual Hosts + and with a default `ServerName`, and hence no `ServerName blablabla.onion` is + needed; a layer of endpoints gives redundancy up to `N - 1` failing endpoints. +* balancers: each load balancer service runs an NGINX instance + configured to serve all websites from all endpoints; each website is also run + on it's own port and a reverse proxy is set with all endpoints as upstreams; + this also gives additional redundancy; port number to server name conversion + happens at this stage. * backends: each backend service runs a Tor instance running an Onion Service for each site; `HiddenServicePort` is set according to each sites' TCP port - in the proxy; so, for redundancy, the Onion Service for each site is linked to + in the load balancer; so, for redundancy, the Onion Service for each site is linked to an NGINX instance which is linked to all application endpoints. * frontends: runs a Tor instance to act as the Onion Service frontend for each Onion Site; right now just a single frontend is supported, so redundancy is @@ -228,13 +256,13 @@ Main services: Additional services: -* monitor: simple monitoring nodes using `mon`. +* monitor: monitoring nodes. * tor: nodes with Tor daemons used by the monitoring nodes. ### Service grouping -* Each backends and it's proxy counterpart (say `backend1` and `proxy1` might - be grouped together in the same machine). +* Each backends and it's load balancer counterpart (say `backend1` and + `balancer` might be grouped together in the same machine). * Each frontend and it's onionbalance counterpart (say `frontend1` and `onionbalance`) might be also grouped in a single machine. @@ -531,6 +559,17 @@ Example CLI invocations and behaviors. ## Open questions +* Preliminaries: + * This is a living specs/design: is not meant to be finished, but + continuously updated as needed. Paraphrasing the + [Tor Protocol Specification](https://gitlab.torproject.org/tpo/core/torspec/-/blob/main/tor-spec.txt), + + > Note: This document aims to specify Oniongroove as currently implemented, though + > it may take it a little time to become fully up to date. Future versions + > of Oniongroove may implement improved protocols, and compatibility is not + > guaranteed. We may or may not remove compatibility notes for other + > obsolete versions of Oniongroove as they become obsolete. + * Onionbalance: * See [these questions about current Onionbalance limits](https://gitlab.torproject.org/tpo/community/team/-/issues/57#note_2780708). @@ -552,3 +591,9 @@ Example CLI invocations and behaviors. * Password sharing mechanism: see discussion in this ticket: [evaluate password management options (#29677)](https://gitlab.torproject.org/tpo/tpa/team/-/issues/29677). + +* Metrics: + * Needs description and diagram about metrics collection and reporting. + +* Monitoring: + * Needs description and diagram about monitoring and notifications. -- GitLab From ec750fd2ecc367b09369521f3c5a01a1bb16c494 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Wed, 2 Mar 2022 19:48:18 -0300 Subject: [PATCH 11/48] Feat: specs: Oniongroove as a config generator and dispatcher --- docs/specs.md | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index bbc6bba..bad3dfc 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -292,6 +292,12 @@ is less costly than keeping an existing installation). backends load balancers etc (could use Ansible for that, even Docker). This step should be vendor-neutral. +Oniongroove MUST operate as a translation between an implementation-neutral +configuration format and the specific configuration of any module/recipe +such as Ansible cookbooks. In this sense, Oniongroove behaves basically +as a configuration generator and dispatcher, abstracting details and +presenting a clean configuration structure. + ### Strategies Assumptions: @@ -490,11 +496,13 @@ Some of the relevant [Onion Services issues](https://gitlab.torproject.org/group * The "suite" can manage many * "providers", * which hosts many - * "onion services", each one split in - * a "frontend" Onionbalance instance - * "backend" Onion Services + * "onion services" sites, each one split in + * a "frontend" Onionbalance service instance + * "backend" Onion Service instances * a "monitor" instance (also gathers stats) * "probes" + * which connects to one or more: + * endpoints / upstreams * which is split in * frontend node(s) * backend nodes @@ -520,6 +528,13 @@ v0.8: the local working copy of the provider's Git repository. Subfolders and files: * `$PROVIDER_PATH/oniongroove.yaml`: main config file. + * `$PROVIDER_PATH/sites/.yaml`: configuration for a single site name. + * `$PROVIDER_PATH/nodes//.yaml`: configuration for a single node from a given type. + * `$PROVIDER_PATH/provisioners/`: generated configuration for a given provisioning technology. + +## Configuration format + +* YAML configuration format defining all entity instances in the model. ## UX -- GitLab From 0ff3aab4832a423199e0487f7598ba715f7f1e62 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Thu, 3 Mar 2022 07:13:10 -0300 Subject: [PATCH 12/48] Feat: specs: additional overview --- docs/specs.md | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/docs/specs.md b/docs/specs.md index bad3dfc..d542165 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -298,6 +298,34 @@ such as Ansible cookbooks. In this sense, Oniongroove behaves basically as a configuration generator and dispatcher, abstracting details and presenting a clean configuration structure. +* Every site created MUST trigger keypair generation for the frontend and all + backends as well as an Onionbalance config update. + +* Every backend node creation MUST trigger keypair generation for all sites as + well as an Oniobalance config update. + +* Ideally, Onionbalance SHOULD pick backend descriptor in a randomized order to + ensure a regular distribution of backend node usage. This depends on + Onionbalance behavior. + +* Every site has a set of endpoints (min. of 1) which are compiled into an + `upstream` block of the NGINX configuration of each load balance instance. + This ensures regular distribution in the upstream connection. If the endpoint + is already a CDN then then the `upstream` is composed of a single item. + +* OPTIONAL TLS certificate verification in the upstream connection: leave that + to the client browser or display a customized warning page in case of an + invalid certificate. + +* OPTIONAL upstream connection via Tor circuit, to prevent detection of the + Oniongroove CDN location. + +* Custom NGINX page in case of any reverse proxy connection error is OPTIONAL. + +* Oniongroove CDN MUST NOT cache content. It MUST act only as a load balanced + proxy. Any caching SHOULD be implemented by the upstream application logic + (separation of concerns). + ### Strategies Assumptions: @@ -610,5 +638,18 @@ Example CLI invocations and behaviors. * Metrics: * Needs description and diagram about metrics collection and reporting. + * Can metrics be submited/collected via .onion services? + * Monitoring: * Needs description and diagram about monitoring and notifications. + + * Don't prioritize a domain for the monitoring dashboard: use an .onion by + default with authentication. Currently Onionbalance does not support + authentication so this service MUST be implemented outside de CDN pool. + +* Failover: + * How to do failover for probing/monitoring/stats/notifications? + + * How to do the Onionbalance failover? Config and keys might be easily + replicated but needs a way for backup frontend nodes to takeover if others + are unavailable. -- GitLab From 539734883eaa2a740243aa23060e19793d7a0713 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Thu, 3 Mar 2022 07:28:55 -0300 Subject: [PATCH 13/48] Feat: specs: clean architecture and CRUD --- docs/specs.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index d542165..761f9eb 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -133,7 +133,10 @@ security and high availability: ## Architecture -Architectural choices for the Onionbalance backends: +Inspired by [rdsys](https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/blob/main/doc/architecture.md), +Oniongroove aspires to implement the design philosophy of [The Clean +Architecture](https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html), +with the following architectural choices for the Onionbalance backends: * Given that a single Tor daemon can host many services, a single horizontal layer or "wall" of nodes/VMs/containers running Tor could act as Onionbalance @@ -568,7 +571,9 @@ v0.8: Example CLI invocations and behaviors. -### Creating a provider +### Provider CRUD + +Creating a provider: oniongroove create \ [--path Create provider git repository from template and using operator name/email -> (Optionally) Setup OpenPGP repository signature -### Adding nodes +### Node CRUD + +* To be defined. + +### Onion Services CRUD * To be defined. -### Adding Onion Services +### Applying the configuration * To be defined. -- GitLab From d2d6d09911bc0ac0f5d3b13a5dd4be0c837eff1b Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Thu, 3 Mar 2022 07:32:33 -0300 Subject: [PATCH 14/48] Feat: specs: TOC --- docs/specs.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/specs.md b/docs/specs.md index 761f9eb..b87a7a6 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -11,6 +11,10 @@ > > 4. (music) A pronounced, enjoyable rhythm. +## Index + +[TOC] + ## Goals 1. Give balance, rhythm and smoothness in setting up, monitoring and -- GitLab From a84937b4243dd71ee2a7e2e3e95ab6646bae30e2 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Thu, 3 Mar 2022 07:34:39 -0300 Subject: [PATCH 15/48] Feat: specs: move relevant issues and inspirations to the bottom --- docs/specs.md | 56 +++++++++++++++++++++++++-------------------------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index b87a7a6..130c223 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -498,34 +498,6 @@ Testing: * Internal probe between each Onion Service backend and the application endpoint. -## Relevant issues - -Some of the relevant [Onion Services issues](https://gitlab.torproject.org/groups/tpo/-/issues?label_name%5B%5D=Onion+Services&scope=all&state=opened) for the Oniongroove project: - -* [prop224: Implement offline keys for v3 onion services (#29054) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/29054) -* [prop224: Implement tor-genkey tool for offline HS key creation (#18098) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/18098) -* [integrate v3 onion service key generation into tor (#40475) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/40475) -* [hs: Implement self reachability test (#40064) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/40064) -* [Understand code performance of onion services under DoS (#33704) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/33704) -* [Research approaches for improving the availability of services under DoS (#31223) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/31223) -* [Design a PoW scheme for HS DoS defence (#33712) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/33712) -* [controller: events for hidden service intro point changes, descriptor changes, uploads, etc (#19324) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/19324) -* [Tor should ship docker images of onion services (and other services) (#30240) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/30240) -* [Consider adopting vanguard's security suggestions for onionbalance (#31857) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/31857) -* [Onion service node pinning by default (#25308) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/25308) -* [Add features improving onion services' interaction with Tor. (#32511) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/32511) -* [v3: Support distinct descriptor mode (#7) · Issues · The Tor Project / Core / Onionbalance · GitLab](https://gitlab.torproject.org/tpo/core/onionbalance/-/issues/7) -* [TorBrowser should advertise Onion Networking capability (#32256) · Issues · The Tor Project / Applications / Tor Browser · GitLab](https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/32256) - -## Inspirations - -* [Hiro / Roid · GitLab](https://gitlab.torproject.org/hiro/roid) -* [Hiro / onions on heroku · GitLab](https://gitlab.torproject.org/hiro/onions-on-heroku) -* [GitHub - alecmuffett/eotk: Enterprise Onion Toolkit](https://github.com/alecmuffett/eotk) -* [GitHub - nusenu/ansible-relayor: An Ansible Role for Tor Relay Operators](https://github.com/nusenu/ansible-relayor): good reference in how to build an ansible role. -* [leap / leap_cli · GitLab](https://0xacab.org/leap/leap_cli), see [archived docs on LEAP providers](http://web.archive.org/web/20160314111917/https://leap.se/docs/platform): good UX reference for a CLI tool. -* [stats: Create a stat exporter port for monitoring purposes (#40063) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/40063) - ## Model sketch * The "suite" can manage many @@ -613,6 +585,34 @@ Creating a provider: * Git wrapper. +## Relevant issues + +Some of the relevant [Onion Services issues](https://gitlab.torproject.org/groups/tpo/-/issues?label_name%5B%5D=Onion+Services&scope=all&state=opened) for the Oniongroove project: + +* [prop224: Implement offline keys for v3 onion services (#29054) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/29054) +* [prop224: Implement tor-genkey tool for offline HS key creation (#18098) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/18098) +* [integrate v3 onion service key generation into tor (#40475) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/40475) +* [hs: Implement self reachability test (#40064) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/40064) +* [Understand code performance of onion services under DoS (#33704) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/33704) +* [Research approaches for improving the availability of services under DoS (#31223) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/31223) +* [Design a PoW scheme for HS DoS defence (#33712) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/33712) +* [controller: events for hidden service intro point changes, descriptor changes, uploads, etc (#19324) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/19324) +* [Tor should ship docker images of onion services (and other services) (#30240) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/30240) +* [Consider adopting vanguard's security suggestions for onionbalance (#31857) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/31857) +* [Onion service node pinning by default (#25308) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/25308) +* [Add features improving onion services' interaction with Tor. (#32511) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/32511) +* [v3: Support distinct descriptor mode (#7) · Issues · The Tor Project / Core / Onionbalance · GitLab](https://gitlab.torproject.org/tpo/core/onionbalance/-/issues/7) +* [TorBrowser should advertise Onion Networking capability (#32256) · Issues · The Tor Project / Applications / Tor Browser · GitLab](https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/32256) + +## Inspirations + +* [Hiro / Roid · GitLab](https://gitlab.torproject.org/hiro/roid) +* [Hiro / onions on heroku · GitLab](https://gitlab.torproject.org/hiro/onions-on-heroku) +* [GitHub - alecmuffett/eotk: Enterprise Onion Toolkit](https://github.com/alecmuffett/eotk) +* [GitHub - nusenu/ansible-relayor: An Ansible Role for Tor Relay Operators](https://github.com/nusenu/ansible-relayor): good reference in how to build an ansible role. +* [leap / leap_cli · GitLab](https://0xacab.org/leap/leap_cli), see [archived docs on LEAP providers](http://web.archive.org/web/20160314111917/https://leap.se/docs/platform): good UX reference for a CLI tool. +* [stats: Create a stat exporter port for monitoring purposes (#40063) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/40063) + ## Open questions * Preliminaries: -- GitLab From b7a62eab563bd497afcf5ab1fb1019361b41d6d3 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Thu, 3 Mar 2022 07:47:20 -0300 Subject: [PATCH 16/48] Fix: specs: reorganize sections --- docs/specs.md | 70 ++++++++++++++++++++++++++------------------------- 1 file changed, 36 insertions(+), 34 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index 130c223..97c1886 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -24,6 +24,8 @@ ## Preliminaries +### Assumptions + * The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). @@ -43,7 +45,7 @@ should be added but what should be removed until only the necessary (while never sufficient) is there. This current specification is no exception. -## Guiding questions +### Guiding questions The following basic questions oriented the overall Oniongroove planning: @@ -58,10 +60,10 @@ The following basic questions oriented the overall Oniongroove planning: 2. Support: which are the most common hosting environments that should be supported? -## Features +## Overview -Oniongroove design focuses in some useful properties for Onion Services -security and high availability: +Oniongroove design focuses in some useful properties/features for Onion +Services security and high availability: 0. Reusability: a single Oniongroove instance MAY host more than a single .onion site. The total number of hosted .onion sites SHOULD not be limited. @@ -137,6 +139,8 @@ security and high availability: ## Architecture +### Overall functioning + Inspired by [rdsys](https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/blob/main/doc/architecture.md), Oniongroove aspires to implement the design philosophy of [The Clean Architecture](https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html), @@ -175,9 +179,7 @@ endpoint: set using Oniongroove), but that would probably defeat the purpose of having an .onion fronting in the first place ;) -## Topology - -### Overall functioning +### Topology The proposed solution is composed by services in the following diagram: @@ -281,7 +283,26 @@ Additional services: * Each monitor node could be grouped with it's tor counterpart (say `monitor1` and `tor1`). -## Deployment +### Entity-relationships model + +* The "suite" can manage many + * "providers", + * which hosts many + * "onion services" sites, each one split in + * a "frontend" Onionbalance service instance + * "backend" Onion Service instances + * a "monitor" instance (also gathers stats) + * "probes" + * which connects to one or more: + * endpoints / upstreams + * which is split in + * frontend node(s) + * backend nodes + * load balancers (to be plugged in the endpoint) + * monitor nodes + * probe nodes + +## Implementation ### Overview @@ -333,9 +354,9 @@ presenting a clean configuration structure. proxy. Any caching SHOULD be implemented by the upstream application logic (separation of concerns). -### Strategies +### Deployment -Assumptions: +Assumptions for the deployment strategy: * System should behave like an "Onion As a Service". This means that, given: @@ -428,7 +449,7 @@ Additional checklist: * Alt-Svc Header. -## Performance & Anti-DoS measures +### Performance & Anti-DoS measures * Basic CDN functionalities such as caching to avoid excessive access of the endpoints. @@ -459,7 +480,7 @@ Additional checklist: and, if HTTPS proves to be useful, introduce it later with HSTS header and HTTP-to-HTTPS redirection. -## Metrics +### Metrics Collecting: some approaches do collect metrics from the Onion Service: @@ -498,26 +519,7 @@ Testing: * Internal probe between each Onion Service backend and the application endpoint. -## Model sketch - -* The "suite" can manage many - * "providers", - * which hosts many - * "onion services" sites, each one split in - * a "frontend" Onionbalance service instance - * "backend" Onion Service instances - * a "monitor" instance (also gathers stats) - * "probes" - * which connects to one or more: - * endpoints / upstreams - * which is split in - * frontend node(s) - * backend nodes - * load balancers (to be plugged in the endpoint) - * monitor nodes - * probe nodes - -## Folder scheme +### Folder scheme Folder scheme relies on the [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) @@ -539,11 +541,11 @@ v0.8: * `$PROVIDER_PATH/nodes//.yaml`: configuration for a single node from a given type. * `$PROVIDER_PATH/provisioners/`: generated configuration for a given provisioning technology. -## Configuration format +### Configuration format * YAML configuration format defining all entity instances in the model. -## UX +### UX Example CLI invocations and behaviors. -- GitLab From 0b99be58b8724a93d9df55eec14768657a0a34e7 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Thu, 3 Mar 2022 07:48:29 -0300 Subject: [PATCH 17/48] Fix: specs: typo --- docs/specs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/specs.md b/docs/specs.md index 97c1886..289b884 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -283,7 +283,7 @@ Additional services: * Each monitor node could be grouped with it's tor counterpart (say `monitor1` and `tor1`). -### Entity-relationships model +### Entity-relationship model * The "suite" can manage many * "providers", -- GitLab From 886f16ba3b043fbb230e5f1f3d087d6e339162c8 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Fri, 4 Mar 2022 16:15:46 -0300 Subject: [PATCH 18/48] Feat: specs: vanguards references --- docs/specs.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index 289b884..b8ad816 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -422,8 +422,10 @@ Additional checklist: * Apply other anti-DoS measures. - * Setup The Vanguards Onion Service Addon on each Tor daemon (both - Onionbalance frontend and backend). + * Setup The Vanguards Onion Service Addon on each Tor daemon. For reference, + check: + * https://gitlab.torproject.org/tpo/core/tor/-/issues/31857 + * https://github.com/mikeperry-tor/vanguards/blob/master/README_SECURITY.md#how-to-onionbalance * Encrypted offline backups of Onion Service keys: at least from the Onion Balance frontend service, because the backends _might_ be disposable -- GitLab From 096ebfd834cb559ca717a124542030135987df9f Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Mon, 7 Mar 2022 17:49:37 -0300 Subject: [PATCH 19/48] Feature: specs: problem, proposed solution and enhanced overview --- README.md | 20 ++++++-- docs/specs.md | 135 ++++++++++++++++++++++++++++++++++++++++++-------- 2 files changed, 130 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index 92bf1fe..b26e2ae 100644 --- a/README.md +++ b/README.md @@ -9,10 +9,24 @@ Monitoring system, giving groove to the [Vanguards Addon](https://github.com/mikeperry-tor/vanguards) and many more features. -Oniongroove acts as a load balancer or even a Content Delivery Network (CDN) -using the [Tor Network](https://torproject.org). +Oniongroove acts as a proxy, load balancer or even a Content Delivery Network +(CDN) using the [Tor Network](https://torproject.org). + +Oniongroove use case is similar to [EOTK](https://github.com/alecmuffett/eotk), +essentially implementing a "reverse .onion proxy" between Tor Users and +backend websites: + +```mermaid +graph LR + C[Client] -- .onion address via Tor Network --> O[Oniongroove CDN] -- Some network link --> U[upstream site] +``` + +It can be used to: + +1. Create Onion Services for any existing website. +2. Setting up "pure-onion" sites, i.e, those available only inside the Tor network. ## Status This software suite is still in the planning stage! -See the [specs document](docs/specs.md) for details. +See the [documentation](docs/index.md) for details. diff --git a/docs/specs.md b/docs/specs.md index b8ad816..19a2f74 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -17,10 +17,30 @@ ## Goals -1. Give balance, rhythm and smoothness in setting up, monitoring and +1. Give balance, rhythm and smoothness in configuring, monitoring and maintaining [Tor Onion Services](https://community.torproject.org/onion-services/). -2. Increase Onion Service adoption by providing such a tool. +2. Increase Onion Service adoption by providing such a tool that abstracts + most of the details involved in Onion Services management. + +## The problem + +Setting up Onion Services sites with load balancing, Denial of Service (DoS) +defenses, guard protection and other best practices involves many moving parts, +from installing different software to proceed with many incremental steps, +making it difficult to go beyond creating a simple .onion address. + +## Proposed solution + +The solution proposed in this specification tackles the problem using the +Infrastruture as Code (IaC) paradigm to manage high availability Onion Services +sites by: + +1. Offering a simplified interface, configuration format and convention. +2. Providing a deployment tool that takes care of most of the setup. + +The rest of this document details design choices, the architecture and +implementation of Oniongroove. ## Preliminaries @@ -47,7 +67,8 @@ ### Guiding questions -The following basic questions oriented the overall Oniongroove planning: +The basic questions oriented the overall Oniongroove planning where split +in three dimensions: 0. Architecture: which are the possible architectures to build an Onion balanced service? Knowing that there's no "one size fits all" choice, @@ -55,10 +76,12 @@ The following basic questions oriented the overall Oniongroove planning: 1. Implementation: which are the available stacks/tools to implement those architectures? Ideally, the suite should not be tightly coupled or - heavily dependent with any specific implementation. + heavily dependent with any specific implementation, leaving room from + refactoring and internals replacement while keeping the same user + experience and configuration format. -2. Support: which are the most common hosting environments that should be - supported? +2. Support: which are the most common hosting environments and use cases + that should be supported? ## Overview @@ -66,11 +89,54 @@ Oniongroove design focuses in some useful properties/features for Onion Services security and high availability: 0. Reusability: a single Oniongroove instance MAY host more than a single - .onion site. The total number of hosted .onion sites SHOULD not be limited. + .onion site. The total number of hosted .onion sites SHOULD not be limited + by any software constraint but only by the available computing resources. + Such Oniongroove instance is called a _provider_. Teams of Operators MAY + manage multiple providers. + +```mermaid +flowchart LR + subgraph Operator Computer + Oniongroove[Oniongroove tool] + + R1[Provider Repository 1] + R2[Provider Repository 2] + Rn[Provider Repository n] + + Oniongroove -- Manages --> R1 + Oniongroove -- Manages --> R2 + Oniongroove -- Manages --> Rn + + S11[Onion Site 1.1] + S12[Onion Site 1.2] + S1n[Onion Site 1.n] + + R1 -- Defines --> S11 + R1 -- Defines --> S12 + R1 -- Defines --> S1n + + S21[Onion Site 2.1] + S21[Onion Site 2.2] + S2n[Onion Site 2.n] + + R2 -- Defines --> S21 + R2 -- Defines --> S22 + R2 -- Defines --> S2n + + S31[Onion Site 3.1] + S31[Onion Site 3.2] + S3n[Onion Site 3.n] + + Rn -- Defines --> S31 + Rn -- Defines --> S32 + Rn -- Defines --> S3n + + end +``` 1. Key generation: all .onion keys MUST be generated locally -- i.e, at the - operators's box prior to deployment -- encrypted and optionally pushed to a - private repository. This allows other members of the operators team to clone + Operators's box prior to deployment -- encrypted and optionally pushed to a + private repository. This allows other members of the Operators team to clone that internal repository and start managing sites and nodes. This ensures fast disaster recovery and also avoid the need for a backup system just for onion balancing. @@ -85,21 +151,40 @@ Services security and high availability: could be regenerated with only the combination with the software suite, the keys and the custom configuration. +```mermaid +flowchart LR + subgraph Operator Computer + Conf[Configuration] + Keys + + OC[Oniongroove configurator] -- Generates --> Conf + OC -- Generates --> Keys + OP[Oniongroove provisioner] + end + + Conf --> OP -- Applies into --> P[Remote machines] + Keys --> OP + P -- Produces --> CDN[Oniongroove CDN] +``` + 3. Elasticity: disposability properties leads to the OPTIONAL feature in which - backend nodes can be added and removed at will? In a way that enables - the system to be elastic in the future, adding and removing nodes according to - average load or sysadmin command? Or that would need the Onionbalance instance - to be restarted (resulting in unwanted downtimes), or even breaking it? + backend nodes can be added and removed at will, making it from the start + compatible with any elastic capability to be speficied in the future -- like + adding and removing nodes according to backend overload. - Currently Onionbalance - [supports only up to 8 backends](https://gitlab.torproject.org/tpo/core/onionbalance/-/issues/7) + This depends if the Onionbalance instance to be restarted (resulting in + unwanted downtimes) every time the set of backend nodes changes -- either by + adding or removing nodes. - Initial implementation would then use a fixed number of 8 backends per .onion - site, but experiments could be made to test if a site could have a dynamic - number of backends. + This should also be done in sync with the timeframe of Onion Services' + descriptor updates, so Clients don't end up trying to access backends + that are no longer online. - This limitation means that Oniongroove would not initially benefit - from having a cloud native deployment, but that should also be considered. + By the time this spec was written, Onionbalance [maximum number of backends + is reported to be 8](https://gitlab.torproject.org/tpo/core/onionbalance/-/issues/7), + so initial implementation MAY use a fixed number of max. 8 backends per + .onion site, but OPTIONAL experiments can be made to test if a site could have + a dynamic number of backends. 4. Uniformity with flexibility: looks like most (if not all) sites can have the same "CDN" fronting setup, while it's "last mile"/endpoints might be all @@ -108,7 +193,10 @@ Services security and high availability: "last mile" (endpoint) configurations. 5. Migration support: the previous point would depend in built-in support to - migrate existing onion services into the CDN instance. + migrate existing onion services into the CDN instance, by + * Importing their frontend and backend keys along with configuration. + * Turning off the old system. + * Deploying configuration and keys to fresh nodes using Oniongroove. 7. Testing: Shall this suite be tested using Chutney or via the shadow simulator (Gitlab CI)? Makes sense? Which other tests should be considered? @@ -245,7 +333,7 @@ Main services: as an example, they could be Apache web servers using Apache Virtual Hosts and with a default `ServerName`, and hence no `ServerName blablabla.onion` is needed; a layer of endpoints gives redundancy up to `N - 1` failing endpoints. -* balancers: each load balancer service runs an NGINX instance +* balancers: each load balancer service runs an NGINX/OpenResty instance configured to serve all websites from all endpoints; each website is also run on it's own port and a reverse proxy is set with all endpoints as upstreams; this also gives additional redundancy; port number to server name conversion @@ -670,3 +758,6 @@ Some of the relevant [Onion Services issues](https://gitlab.torproject.org/group * How to do the Onionbalance failover? Config and keys might be easily replicated but needs a way for backup frontend nodes to takeover if others are unavailable. + +* FAQ: + * Why not just use/extend EOTK? -- GitLab From 66a58e3fcf74b971b0617a56fa4dd5c958decc3f Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Tue, 8 Mar 2022 15:24:55 -0300 Subject: [PATCH 20/48] Feat: specs: roadmap and misc changes --- README.md | 2 +- docs/specs.md | 39 +++++++++++++++++++++++++++++---------- 2 files changed, 30 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index b26e2ae..217eafc 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ backend websites: ```mermaid graph LR - C[Client] -- .onion address via Tor Network --> O[Oniongroove CDN] -- Some network link --> U[upstream site] + C[Client] -- .onion address via Tor Network --> O[Oniongroove CDN] -- Some network link --> U[Upstream site] ``` It can be used to: diff --git a/docs/specs.md b/docs/specs.md index 19a2f74..a82d512 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -39,6 +39,9 @@ sites by: 1. Offering a simplified interface, configuration format and convention. 2. Providing a deployment tool that takes care of most of the setup. +It's important to note what **Oniongroove is not a caching/mirror solution, +but an Onion Services proxy layer**. + The rest of this document details design choices, the architecture and implementation of Oniongroove. @@ -186,20 +189,21 @@ flowchart LR .onion site, but OPTIONAL experiments can be made to test if a site could have a dynamic number of backends. -4. Uniformity with flexibility: looks like most (if not all) sites can have the - same "CDN" fronting setup, while it's "last mile"/endpoints might be all - different. That said, the "first half" of the solution could be based in the - same software suite and workflow which could be flexible enough to accept distinct - "last mile" (endpoint) configurations. +4. Uniformity with flexibility: Oniongroove works with the assumption that all + sites can have the same "CDN" fronting setup, while their "last + mile"/endpoints might be all different. That said, the "first half" of the + solution MUST be based in the same software suite and workflow which MUST be + flexible enough to accept distinct "last mile" (endpoint) configurations. -5. Migration support: the previous point would depend in built-in support to - migrate existing onion services into the CDN instance, by +5. Migration support: built-in support to migrate existing onion services into + the CDN instance, by just * Importing their frontend and backend keys along with configuration. * Turning off the old system. * Deploying configuration and keys to fresh nodes using Oniongroove. -7. Testing: Shall this suite be tested using Chutney or via the shadow simulator (Gitlab - CI)? Makes sense? Which other tests should be considered? +7. Testing: this suite MAY include test suites using either + [Chutney](https://gitlab.torproject.org/tpo/core/chutney), the [Shadow + Simulator](https://shadow.github.io/) (Gitlab CI) or custom procedures. 8. Limits: which limits we should expect from the current technology, and how we could work around those? Examples: @@ -215,7 +219,7 @@ flowchart LR 9. No lock-in: operators aren't required to use the Oniongroove suite in order to implement it's functionality. They can instead selectively pick some subsystems - and plug into their existing deployment workflow. + and plug into their existing deployment workflow or just follow the conventions. 10. Requirements: Oniongroove CDN requirements MUST be low. The system can run in parallel along with the existing web sites available @@ -242,6 +246,10 @@ with the following architectural choices for the Onionbalance backends: resources, is to have one horizontal layer of nodes acting as Onionbalance backends for each onion site. +Topology is different from "classic" content delivery networks: an Onionbalance +frontend acts as a publisher for backend servers in Tor’s Onion Service’s +hashring (DHT). A single .onion address allows access to any backend server. + Topological choices for the Onionbalance frontends: * From the Onionbalance docs, looks like that a single instance can manage @@ -677,6 +685,17 @@ Creating a provider: * Git wrapper. +## Roadmap + +Implementation is planned as follows: + +* Milestone #0: Prototype: a working CDN uning vendor-neutral modules. +* Milestone #1: MVP: statistics. +* Milestone #2: Monitoring. +* Milestone #4: Traffic control / protections. +* Milestone #5: Redundancy at the frontend, stats and monitoring nodes. +* Milestone #7: Vendor-specific tiers. + ## Relevant issues Some of the relevant [Onion Services issues](https://gitlab.torproject.org/groups/tpo/-/issues?label_name%5B%5D=Onion+Services&scope=all&state=opened) for the Oniongroove project: -- GitLab From f49685d7e86072c7afe34fa5e1e09f2e97dca4cf Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Tue, 8 Mar 2022 17:25:29 -0300 Subject: [PATCH 21/48] Feat: specs: ER diagram --- docs/specs.md | 34 ++++++++++++++++++++++++++++++---- 1 file changed, 30 insertions(+), 4 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index a82d512..dd1ac74 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -381,16 +381,18 @@ Additional services: ### Entity-relationship model +Simply put, this is the ER model for Oniongroove: + * The "suite" can manage many * "providers", * which hosts many * "onion services" sites, each one split in * a "frontend" Onionbalance service instance - * "backend" Onion Service instances + * "backend" Onion Service instances, + * which connects to one or more: + * endpoints / upstreams * a "monitor" instance (also gathers stats) * "probes" - * which connects to one or more: - * endpoints / upstreams * which is split in * frontend node(s) * backend nodes @@ -398,6 +400,30 @@ Additional services: * monitor nodes * probe nodes +Which in an ER diagram looks like this: + +```mermaid +erDiagram + Oniongroove ||--o{ Provider : manages + Provider ||--o{ OnionSite : hosts + Provider ||--o{ FrontendNode : "split in" + Provider ||--o{ BackendNode : "split in" + Provider ||--o{ BalancerNode : "split in" + Provider ||--o{ MonitorNode : "split in" + Provider ||--o{ ProbeNode : "split in" + + OnionSite ||--o| FrontendInstance : "split in" + OnionSite ||--o{ BackendInstance : "split in" + OnionSite ||--o| MonitorInstance : "split in" + OnionSite ||--o{ ProbeInstance : "split in" + + BackendInstance ||--o{ Upstream : "connects to" +``` + +The cardinality between `OnionSite` and `FrontendInstance` and +`MonitorInstance` depends if the implementation has failover between frontend +and monitoring instances. + ## Implementation ### Overview @@ -478,7 +504,7 @@ Workflow: * Onion Service keys could be either: 1. Generated locally at the operator laptop and then deployed - remotelly along with the configuration OR + remotely along with the configuration OR 2. Generated directly in the machines hosting the Onion Service. -- GitLab From 4eeefebabdbaf0e9d993f8dc087fd4afd1c9fa76 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Tue, 8 Mar 2022 18:11:20 -0300 Subject: [PATCH 22/48] Fix: specs: overview --- docs/specs.md | 35 ++++++++++++++++++++--------------- 1 file changed, 20 insertions(+), 15 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index dd1ac74..dc92ed1 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -39,8 +39,9 @@ sites by: 1. Offering a simplified interface, configuration format and convention. 2. Providing a deployment tool that takes care of most of the setup. -It's important to note what **Oniongroove is not a caching/mirror solution, -but an Onion Services proxy layer**. +It's important to note what **Oniongroove is not a caching/mirroring solution, +but an Onion Services proxy layer**. It requires active endpoint(s) to connect +to. The rest of this document details design choices, the architecture and implementation of Oniongroove. @@ -205,17 +206,21 @@ flowchart LR [Chutney](https://gitlab.torproject.org/tpo/core/chutney), the [Shadow Simulator](https://shadow.github.io/) (Gitlab CI) or custom procedures. -8. Limits: which limits we should expect from the current technology, and how we - could work around those? Examples: - * Tor daemon is single process, no threads. How it scales under load for Onion - Services and with a varying number of Onion Services? +8. Limits: while Oniongroove aims to scale and be as secure as possible, some + limits might exists from the current technology. Some can be worked around, + but others require improvements in the underlying stack, like the following: + * Tor daemon is single process, with limited threading support. How it + scales under load for Onion Services and with a varying number of Onion + Services? - * Which other limits are important to be considered in the scope of this project, - like the current upper bound of 8 Onionbalance backend servers? + * The reported + [maximum number of Onionbalance backends](https://gitlab.torproject.org/tpo/core/onionbalance/-/issues/7). - * Despite the support for offline keys in the Onion Services v3 specification, - the Tor daemon currently does not support this feature, requiring a high - level of operational security to protect Onion Services' keys. + * Despite the support for offline keys in the Onion Services v3 specification, + the Tor daemon currently does not support this feature, requiring a high + level of operational security to protect Onion Services' keys. + + * Other limits important to be considered in the scope of this project. 9. No lock-in: operators aren't required to use the Oniongroove suite in order to implement it's functionality. They can instead selectively pick some subsystems @@ -223,10 +228,10 @@ flowchart LR 10. Requirements: Oniongroove CDN requirements MUST be low. The system can run in parallel along with the existing web sites available - via the regular internet and can be hosted in a different infrastructure if - needed. - Minor mofifications in existing systems are needed only for OPTIONAL - security and circunvention features like HTTPS for .onion addresses and the + via the "regular/vanilla" internet and can be hosted in a different + infrastructure if needed. + Minor modifications in existing systems are needed only for OPTIONAL + security and circumvention features like HTTPS for .onion addresses and the Onion-Location header for service discovery. ## Architecture -- GitLab From 5c29a8fb1daeaab7fe431b33f5dae84f95b96119 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Tue, 8 Mar 2022 18:19:54 -0300 Subject: [PATCH 23/48] Fix: specs: typos --- docs/specs.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index dc92ed1..7bbfa94 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -71,7 +71,7 @@ implementation of Oniongroove. ### Guiding questions -The basic questions oriented the overall Oniongroove planning where split +The basic questions oriented the overall Oniongroove planning are split in three dimensions: 0. Architecture: which are the possible architectures to build an Onion @@ -120,7 +120,7 @@ flowchart LR R1 -- Defines --> S1n S21[Onion Site 2.1] - S21[Onion Site 2.2] + S22[Onion Site 2.2] S2n[Onion Site 2.n] R2 -- Defines --> S21 @@ -128,7 +128,7 @@ flowchart LR R2 -- Defines --> S2n S31[Onion Site 3.1] - S31[Onion Site 3.2] + S32[Onion Site 3.2] S3n[Onion Site 3.n] Rn -- Defines --> S31 @@ -426,8 +426,8 @@ erDiagram ``` The cardinality between `OnionSite` and `FrontendInstance` and -`MonitorInstance` depends if the implementation has failover between frontend -and monitoring instances. +`MonitorInstance` depends whether the implementation has failover between +frontend and monitoring instances. ## Implementation -- GitLab From 40937a1c675c5613ea3280ddf11cc0de90312774 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Wed, 9 Mar 2022 21:06:23 -0300 Subject: [PATCH 24/48] Fix: specs: architecture and implementation review --- docs/specs.md | 157 ++++++++++++++++++++++++++++++-------------------- 1 file changed, 96 insertions(+), 61 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index 7bbfa94..9f43785 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -240,45 +240,60 @@ flowchart LR Inspired by [rdsys](https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/blob/main/doc/architecture.md), Oniongroove aspires to implement the design philosophy of [The Clean -Architecture](https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html), -with the following architectural choices for the Onionbalance backends: +Architecture](https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html). -* Given that a single Tor daemon can host many services, a single horizontal - layer or "wall" of nodes/VMs/containers running Tor could act as Onionbalance - backends for many sites. +The topology is different from "classic" content delivery networks: an +Onionbalance frontend acts as a publisher for backend servers in Tor’s Onion +Service’s hashring (DHT). A single .onion address allows access to any backend +server. -* Another alternative, which is more resilient but demands more - resources, is to have one horizontal layer of nodes acting as Onionbalance - backends for each onion site. +Backends MUST be configured to host Onion Services with the following topology: -Topology is different from "classic" content delivery networks: an Onionbalance -frontend acts as a publisher for backend servers in Tor’s Onion Service’s -hashring (DHT). A single .onion address allows access to any backend server. +* A single horizontal layer (or "wall") of backend instances (in different + nodes/VMs/containers) act as Onionbalance backends for many sites, given + that a single Tor daemon can host many services. -Topological choices for the Onionbalance frontends: +* There's no limit in the number of backend instances, but with no guarantee + that all instances will be used by every frontend instance due to the size + limit of each Onion Service descriptor. -* From the Onionbalance docs, looks like that a single instance can manage - many frontends, each one with multiple associated backends. +* Operators wishing to offer a single Onion Service in an entire horizontal + layer of backend nodes MUST setup different Oniongroove providers, one for + each site. -* Additional frontend nodes could act as failover, being activated only - if the main one fails. +Frontend architecture MUST be the following: -For the connection between the Onionbalance backends and the onion site -endpoint: +* A single frontend instance can manage many frontends, each one with multiple + associated backends. -* Preferably run the Onionbalance backend in the same host as the application - endpoint. +* Additional frontend nodes MAY act as failover, being activated only if the + main one fails. + +* Operators wishing to run more than a single frontend instance at the same + time, each one managing different sets of sites MUST setup different + Oniongroove providers. Frontend instances MUST be considered only as a + publishing node of superdescriptors, so currently there is no actual need to + scale these nodes, but only having failover capabilities. This might be + an issue in the future if providers contain a huge number of Onion Services + to be managed (many superdescriptors to handle), but in that case those + "superproviders" may be acting as a single point of failure. This also + might be solved by patching Onionbalance to allow for multiprocessing. + +The connection between the Onionbalance backends and the Onion Sites +endpoints it MAY be set in a different number of ways: -* For third parties endpoints (partner's sites): - * VPN (Wireguard) link? +* Running the Onionbalance backend in the same host or network as the + application endpoint. - * [OnionCat](https://www.onioncat.org) link? +* Using VPN (like [Wireguard](https://www.wireguard.com)) links. - * Authenticated (or verified) HTTPS link? +* Setting up [OnionCat](https://www.onioncat.org) links. - * Could even be a collection of .onion addresses (perhaps each one even be - set using Oniongroove), but that would probably defeat the purpose of - having an .onion fronting in the first place ;) +* HTTPS link, with certificate verification and OPTIONAL authentication. + +* A collection of .onion addresses (perhaps each one even be set using + Oniongroove), but that would probably defeat the purpose of having an .onion + fronting in the first place. ### Topology @@ -332,7 +347,7 @@ graph TD end ``` -Please note that the diagram above is simplified: +Please note that the graph above is simplified: 1. In fact, each .onion site can have different number of endpoints, even a single one. @@ -372,14 +387,14 @@ Additional services: ### Service grouping * Each backends and it's load balancer counterpart (say `backend1` and - `balancer` might be grouped together in the same machine). + `balancer1` MAY be grouped together in the same `worker` machine). * Each frontend and it's onionbalance counterpart (say `frontend1` and - `onionbalance`) might be also grouped in a single machine. + `onionbalance1`) MAY be also grouped in a single machine. -* Each backend might stay in a separate machine/location. +* Each worker MAY stay in a separate machine/location. -* Each endpoint might stay in a separate machine/location. +* Each endpoint MAY stay in a separate machine/location. * Each monitor node could be grouped with it's tor counterpart (say `monitor1` and `tor1`). @@ -433,18 +448,18 @@ frontend and monitoring instances. ### Overview -Consider a two-tier approach, where sysadmins keep a copy of all .onion keys -(and optionally TLS certs) an the entire Onion Service Operation can be -redeployed if needed (eg. disaster recovery, switching providers or even in -case the deployment procedure/stack changes and is found that a redeployment -is less costly than keeping an existing installation). +Implementations MUST consider a two-tier approach, where sysadmins keep a +copy of all .onion keys (and optionally TLS certs) so the entire Onion Service +Operation can be redeployed if needed (eg. disaster recovery, switching +providers or even in case the deployment procedure/stack changes and is found +that a redeployment is less costly than keeping an existing installation). 1. Vendor-specific tier: module(s) that bootstrap the basic environment (like a system with shell, could be based on Terraform), taking care of vendor-specific logic depending on the hosting platform chosen. 2. Vendor neutral tier: a module that bootstraps the Onionbalance frontend(s), - backends load balancers etc (could use Ansible for that, even Docker). + backends, load balancers etc (could use Ansible for that, even Docker). This step should be vendor-neutral. Oniongroove MUST operate as a translation between an implementation-neutral @@ -453,11 +468,17 @@ such as Ansible cookbooks. In this sense, Oniongroove behaves basically as a configuration generator and dispatcher, abstracting details and presenting a clean configuration structure. -* Every site created MUST trigger keypair generation for the frontend and all - backends as well as an Onionbalance config update. +Deployments MUST happen according to this workflow: + +* Every site created MUST trigger a keypair generation for the frontend and all + backends as well as an Onionbalance config update. Since all key generation + happens locally during "compile time", there's no need for a back-and-forth + between the frontend and the backends in order to discover all backend + addresses/pubkeys for all sites. -* Every backend node creation MUST trigger keypair generation for all sites as - well as an Oniobalance config update. +* Every backend node creation MUST trigger a backend keypair generation for all + sites as well as an Oniobalance config update. This step also happens during + compilation time. * Ideally, Onionbalance SHOULD pick backend descriptor in a randomized order to ensure a regular distribution of backend node usage. This depends on @@ -468,19 +489,6 @@ presenting a clean configuration structure. This ensures regular distribution in the upstream connection. If the endpoint is already a CDN then then the `upstream` is composed of a single item. -* OPTIONAL TLS certificate verification in the upstream connection: leave that - to the client browser or display a customized warning page in case of an - invalid certificate. - -* OPTIONAL upstream connection via Tor circuit, to prevent detection of the - Oniongroove CDN location. - -* Custom NGINX page in case of any reverse proxy connection error is OPTIONAL. - -* Oniongroove CDN MUST NOT cache content. It MUST act only as a load balanced - proxy. Any caching SHOULD be implemented by the upstream application logic - (separation of concerns). - ### Deployment Assumptions for the deployment strategy: @@ -490,22 +498,22 @@ Assumptions for the deployment strategy: * The Onion Service key of the Onionbalance frontend. * The configuration. * The software sollution. - * A bunch of nodes. + * A bunch of nodes (like VPSs). * It might be possible to (re)bootstrap the whole "CDN" by using an orquestration tool. -* Redeployment: not only needed, but planned with minimized downtimes and +* Redeployment is not only needed, but planned with minimized downtimes and wiping out keys from the old, decomissioned systems. * Versioned config file format (YAML). Workflow: -* Use a orchestration tool such as Ansible, Puppet, Terraform: - * I would not stick at first with and specific tool but instead - would check if the Tor Project and Community have some conventions - and preferences regarding the usage of those tools. +* Use a orchestration tool such as Ansible, Puppet and Terraform: + * Not sticking at first with an specific tool but instead would check if the + Tor Project and Community have some conventions and preferences regarding + the usage of those tools. * Onion Service keys could be either: 1. Generated locally at the operator laptop and then deployed @@ -787,6 +795,24 @@ Some of the relevant [Onion Services issues](https://gitlab.torproject.org/group * But this behavior MUST be tested in practice before considering any backend rotation feature. +* Proxy: + * URL rewriting, blocking and similar features from EOTK. + + * OPTIONAL TLS certificate verification in the upstream connection: leave that + to the client browser or display a customized warning page in case of an + invalid certificate. + + * OPTIONAL upstream connection via Tor circuit, to prevent detection of the + Oniongroove CDN location. + + * Custom NGINX page in case of any reverse proxy connection error is OPTIONAL. + + * Oniongroove CDN MUST NOT cache content? It MUST act only as a load balanced + proxy. Any caching SHOULD be implemented by the upstream application logic + (separation of concerns)? EOTK uses temporary caching for optimization, which + is an interesting feature for public sites. Some applications MAY require + that cache is always off to prevent upstream content being stored in caches. + * Password sharing mechanism: see discussion in this ticket: [evaluate password management options (#29677)](https://gitlab.torproject.org/tpo/tpa/team/-/issues/29677). @@ -811,3 +837,12 @@ Some of the relevant [Onion Services issues](https://gitlab.torproject.org/group * FAQ: * Why not just use/extend EOTK? + * Need for an idempotent Infrastructure as Code (IaC) implementation + ensuring that systems are in a defined state. + + * Different use-cases: EOTK is focused mainly in public-face sites, + while a more general tool might be useful. + + * Onionbalance support. + + * Key management (encrypted backups). -- GitLab From 0cf0278332d4b0033919d74d4cf6ee99affc1a3f Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Wed, 9 Mar 2022 21:08:54 -0300 Subject: [PATCH 25/48] Fix: specs: implementation: compilation and provisioning --- docs/specs.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/specs.md b/docs/specs.md index 9f43785..a6acfc8 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -468,6 +468,11 @@ such as Ansible cookbooks. In this sense, Oniongroove behaves basically as a configuration generator and dispatcher, abstracting details and presenting a clean configuration structure. +```mermaid +graph LR + OC[Provider configuration] -- Compilation --> VNC[Provisioning configuration] -- Provisioning --> AP[Applied configuration] +``` + Deployments MUST happen according to this workflow: * Every site created MUST trigger a keypair generation for the frontend and all -- GitLab From de25a4dfcf2c75ae589b72a0cc573cb4444c4b1d Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Wed, 9 Mar 2022 21:16:11 -0300 Subject: [PATCH 26/48] Fix: specs: removes stealth mode feature --- docs/specs.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index a6acfc8..48b92c0 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -466,7 +466,7 @@ Oniongroove MUST operate as a translation between an implementation-neutral configuration format and the specific configuration of any module/recipe such as Ansible cookbooks. In this sense, Oniongroove behaves basically as a configuration generator and dispatcher, abstracting details and -presenting a clean configuration structure. +presenting a clean configuration structure: ```mermaid graph LR @@ -575,9 +575,6 @@ Additional checklist: by the absency of the Onion Site operator, as automation allows more people do take care of the maintenance. - * [Stealth mode (`HiddenServiceAuthorizeClient`) between any connections between services - behind Onion Services?] - * Performance: * For public-facing (known IP addresses), consider [single onion service](https://support.torproject.org/glossary/single-onion-service/), -- GitLab From 1259647630c29939d9ceeba87299587f6f69fff3 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Wed, 9 Mar 2022 21:20:55 -0300 Subject: [PATCH 27/48] Fix: specs: implementation: compilation and provisioning (2) --- docs/specs.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/specs.md b/docs/specs.md index 48b92c0..efd18e1 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -470,7 +470,13 @@ presenting a clean configuration structure: ```mermaid graph LR - OC[Provider configuration] -- Compilation --> VNC[Provisioning configuration] -- Provisioning --> AP[Applied configuration] + OC[Provider configuration] + Keys[Provider keys] + + subgraph "Oniongroove provisioner" + Keys -- Compilation --> VNC + OC -- Compilation --> VNC[Provisioning configuration] -- Provisioning --> AP[Applied configuration] + end ``` Deployments MUST happen according to this workflow: -- GitLab From 59e22aa8dc7962467b34b23df48a654272f8d57e Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Wed, 9 Mar 2022 21:29:37 -0300 Subject: [PATCH 28/48] Fix: specs: open questions: existing solutions --- docs/specs.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index efd18e1..c7f9eaa 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -843,8 +843,8 @@ Some of the relevant [Onion Services issues](https://gitlab.torproject.org/group replicated but needs a way for backup frontend nodes to takeover if others are unavailable. -* FAQ: - * Why not just use/extend EOTK? +* Existing solutions: + * EOTK: why not just use/extend it? * Need for an idempotent Infrastructure as Code (IaC) implementation ensuring that systems are in a defined state. @@ -853,4 +853,4 @@ Some of the relevant [Onion Services issues](https://gitlab.torproject.org/group * Onionbalance support. - * Key management (encrypted backups). + * Key management with encrypted backups. -- GitLab From 2452441afd84109c79996aa69ce0c82377838ba4 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Thu, 10 Mar 2022 16:27:56 -0300 Subject: [PATCH 29/48] Fix: specs: deployment and key management --- docs/specs.md | 87 +++++++++++++++++++++++++++++---------------------- 1 file changed, 49 insertions(+), 38 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index c7f9eaa..ac36b08 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -466,7 +466,17 @@ Oniongroove MUST operate as a translation between an implementation-neutral configuration format and the specific configuration of any module/recipe such as Ansible cookbooks. In this sense, Oniongroove behaves basically as a configuration generator and dispatcher, abstracting details and -presenting a clean configuration structure: +presenting a clean configuration structure. + +This means that it MUST be possible to (re)bootstrap a whole provider by using +just: + +* The relevant Onion Service keys. +* The configuration. +* The software solution. +* A bunch of nodes (like VPSs). + +A system which such capacity would then behave like an "Onion As a Service": ```mermaid graph LR @@ -479,6 +489,12 @@ graph LR end ``` +Not sticking with an specific tool allows Oniongroove to be sustainable in the +long term if the uderlying deployment technology changes or if the community +prefers to change implementations. + +### Deployment + Deployments MUST happen according to this workflow: * Every site created MUST trigger a keypair generation for the frontend and all @@ -495,55 +511,40 @@ Deployments MUST happen according to this workflow: ensure a regular distribution of backend node usage. This depends on Onionbalance behavior. -* Every site has a set of endpoints (min. of 1) which are compiled into an - `upstream` block of the NGINX configuration of each load balance instance. - This ensures regular distribution in the upstream connection. If the endpoint - is already a CDN then then the `upstream` is composed of a single item. +* Every site has a set of endpoints (min. of 1) which are compiled into + the configuration of each `balance` instance -- in the case of NGINX, + it's the `upstream` block. -### Deployment - -Assumptions for the deployment strategy: - -* System should behave like an "Onion As a Service". This means that, - given: - * The Onion Service key of the Onionbalance frontend. - * The configuration. - * The software sollution. - * A bunch of nodes (like VPSs). - -* It might be possible to (re)bootstrap the whole "CDN" by using an - orquestration tool. + This ensures regular distribution in the upstream connection. If the endpoint + is already a CDN then the set endpoint is composed of a single item. -* Redeployment is not only needed, but planned with minimized downtimes and +* Redeployment MAY not only needed, but planned with minimized downtimes and wiping out keys from the old, decomissioned systems. -* Versioned config file format (YAML). +### Key management -Workflow: +When designing this solution, two choices for Onion Services key handling and +certificates were considered: -* Use a orchestration tool such as Ansible, Puppet and Terraform: - * Not sticking at first with an specific tool but instead would check if the - Tor Project and Community have some conventions and preferences regarding - the usage of those tools. +1. Locally generated at the Operators computed and then deployed + remotely along with the configuration OR -* Onion Service keys could be either: - 1. Generated locally at the operator laptop and then deployed - remotely along with the configuration OR +2. Generated directly in the machines hosting the Onion Service. - 2. Generated directly in the machines hosting the Onion Service. +In the end, local generation was chosen as this approach have some +advantages: -* Generating keys locally have the following advantages: - * Easier to backup the keys (using GnuPG optionally with a tool - like keyringer with multiple recipients and digital signature). +* Easier to backup the keys (using GnuPG optionally with a password/secret + sharing application with multiple recipients and with digital signature + support). - * Easy to redeploy the whole frontend/backend infrastructure if - needed. +* Easy to redeploy the whole frontend/backend infrastructure if needed. -* They Onionbalance frontend address should be considered as - long term and hence it's keys needs to be stored on encrypted - backups with some redundancy (both physical and human). +The Onionbalance frontend address MUST be considered as long term and hence +their keys MUST be stored on encrypted backups with some redundancy (both +physical and human). -Additional checklist: +### Basic checklist * Basic security requirements: running Onion Services is more sensitive than relays, so additional security checks and measures are needed: maybe an Onion @@ -690,6 +691,8 @@ v0.8: * YAML configuration format defining all entity instances in the model. +* Should be versioned. + ### UX Example CLI invocations and behaviors. @@ -836,6 +839,11 @@ Some of the relevant [Onion Services issues](https://gitlab.torproject.org/group default with authentication. Currently Onionbalance does not support authentication so this service MUST be implemented outside de CDN pool. + * The nature of Tor networking might make unnecessary the use of many probes + in terms of network perspectives (like OONI does), making additional + probes only a matter of failover. But in any case probes could report + it's status in case they cannot reach the Tor network. + * Failover: * How to do failover for probing/monitoring/stats/notifications? @@ -854,3 +862,6 @@ Some of the relevant [Onion Services issues](https://gitlab.torproject.org/group * Onionbalance support. * Key management with encrypted backups. + +* This document is very big! Perhaps the design discussion should be + separated from the specifications (a more synthetic document). -- GitLab From b862cef417732b5326fb49689e35bc81b7ca2155 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Thu, 10 Mar 2022 17:13:23 -0300 Subject: [PATCH 30/48] Feat: specs: security considerations --- docs/specs.md | 136 +++++++++++++++++++++++++++++--------------------- 1 file changed, 79 insertions(+), 57 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index ac36b08..dd06fe6 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -69,6 +69,20 @@ implementation of Oniongroove. should be added but what should be removed until only the necessary (while never sufficient) is there. This current specification is no exception. +* At the same time, it's not the intention of this document to be "written in + the rock" or to serve as a perfect model to future implementations as what + happens in waterfall-like models. It's mainly intended to aid the development. + + This is a living specs/design: is not meant to be finished, but + continuously updated as needed: in the spirit (and paraphrasing) the + [Tor Protocol Specification](https://gitlab.torproject.org/tpo/core/torspec/-/blob/main/tor-spec.txt): + + > Note: This document aims to specify Oniongroove as currently implemented, though + > it may take it a little time to become fully up to date. Future versions + > of Oniongroove may implement improved protocols, and compatibility is not + > guaranteed. We may or may not remove compatibility notes for other + > obsolete versions of Oniongroove as they become obsolete. + ### Guiding questions The basic questions oriented the overall Oniongroove planning are split @@ -544,58 +558,64 @@ The Onionbalance frontend address MUST be considered as long term and hence their keys MUST be stored on encrypted backups with some redundancy (both physical and human). -### Basic checklist +### Security considerations -* Basic security requirements: running Onion Services is more sensitive than - relays, so additional security checks and measures are needed: maybe an Onion - Service for mission critical (in terms of data protection) is even more - sensitive than an exit node: - * More careful service provider selection, or even better: run Onion Services - on premises. +Running Onion Services is more sensitive than Tor relays (not to say even more +than exit nodes), so not only basic/general but also additional security checks +and measures are needed, including but not limited to: - * System checks: disk encryption, firewall and other protections. +* SHOULD use UNIX sockets whenever possible: they're faster and more secure as + no TCP is involved. When in use, they MUST be used properly (like with + correct permissions and ownership). - * Using Tor packages from the official Tor repositories instead of the - operating system's distributtion repositories (Debian for instance). +* MUST use Tor packages from the official Tor repositories instead of the + operating system's distributtion repositories (e.g. Debian). - * The application server (be an webserver or anything else) cannot leak - identifiable information such as hostnames or IP addresses (disabling - ServerTokens on Apache). +* The application server (be an webserver or anything else) MUST not leak + identifiable information such as hostnames or IP addresses (which can + be achieved by disabling `ServerTokens` and `mod_status` on Apache). - * Other recomended and best practices from the Tor and Riseup documentations. +* SHOULD follow other recomended and best practices from the + [Tor](https://community.torproject.org/onion-services/) and [Riseup + documentations](https://help.riseup.net/en/security/network-security/tor/onionservices-best-practices). -* Defenses and SPOFs (single points of failure) avoidance: - * Use Onionbalance for Onion Service scalability. +Defenses and SPOFs (single points of failure) avoidance SHOULD also be +considered: - * Apply other anti-DoS measures. +* Setting The Vanguards Onion Service Addon on each Tor daemon. For reference, + check: + * [Consider adopting vanguard's security suggestions for onionbalance](https://gitlab.torproject.org/tpo/core/tor/-/issues/31857) + * [How to OnionBalance - Vanguards security documentation](https://github.com/mikeperry-tor/vanguards/blob/master/README_SECURITY.md#how-to-onionbalance). - * Setup The Vanguards Onion Service Addon on each Tor daemon. For reference, - check: - * https://gitlab.torproject.org/tpo/core/tor/-/issues/31857 - * https://github.com/mikeperry-tor/vanguards/blob/master/README_SECURITY.md#how-to-onionbalance +* Encrypted offline backups of Onion Service keys: at least from the Onion + Balance frontend service. If backend keys cannot be disposable, they SHOULD + also be backed up. - * Encrypted offline backups of Onion Service keys: at least from the Onion - Balance frontend service, because the backends _might_ be disposable - (needs checking). +* Using automated and deterministic admin tasks prevents SPOFs represented + by the absency of the Onion Site Operator, as automation allows more people + do take care of the maintenance. While this requirement cannot be fulfilled + by Oniongroove itself, it's implementation can lower the entrance barrier + for new Operators. - * Using automated and deterministic admin tasks prevents SPOFs represented - by the absency of the Onion Site operator, as automation allows more people - do take care of the maintenance. +For mission-critical applications, additional precautions SHOULD also be +implemented whenever possible: -* Performance: - * For public-facing (known IP addresses), - consider [single onion service](https://support.torproject.org/glossary/single-onion-service/), - using `HiddenServiceSingleHopMode`. +* System checks: disk encryption, firewall and other protections. - * Use UNIX sockets if possible: they seems faster and more secure as no - TCP is involved. +* More careful service provider selection, or even better: run Onion Services + on premises. While this is out of Oniongroove scope, it's something that + needs to be documented and explained to Operators. -* Service discovery for public-facing sites: - * Onion-Location HTTP header in the endpoint. +### Performance & Anti-DoS measures - * Alt-Svc Header. +* Use Onionbalance for Onion Service scalability. -### Performance & Anti-DoS measures +* SHOULD Apply other relevant [anti-DoS + measures](https://community.torproject.org/onion-services/advanced/dos/). + +* For public-facing (known IP addresses), + consider [single onion service](https://support.torproject.org/glossary/single-onion-service/), + using `HiddenServiceSingleHopMode`. * Basic CDN functionalities such as caching to avoid excessive access of the endpoints. @@ -626,6 +646,13 @@ physical and human). and, if HTTPS proves to be useful, introduce it later with HSTS header and HTTP-to-HTTPS redirection. +### Service discovery + +* Service discovery for public-facing sites: + * Onion-Location HTTP header in the endpoint. + + * Alt-Svc Header. + ### Metrics Collecting: some approaches do collect metrics from the Onion Service: @@ -734,6 +761,8 @@ Creating a provider: ### Other subcommands * Git wrapper. +* Key regeneration? +* X.509 keys and certs management? ## Roadmap @@ -776,16 +805,17 @@ Some of the relevant [Onion Services issues](https://gitlab.torproject.org/group ## Open questions -* Preliminaries: - * This is a living specs/design: is not meant to be finished, but - continuously updated as needed. Paraphrasing the - [Tor Protocol Specification](https://gitlab.torproject.org/tpo/core/torspec/-/blob/main/tor-spec.txt), +* Existing solutions: + * EOTK: why not just use/extend it? + * Need for an idempotent Infrastructure as Code (IaC) implementation + ensuring that systems are in a defined state. + + * Different use-cases: EOTK is focused mainly in public-face sites, + while a more general tool might be useful. - > Note: This document aims to specify Oniongroove as currently implemented, though - > it may take it a little time to become fully up to date. Future versions - > of Oniongroove may implement improved protocols, and compatibility is not - > guaranteed. We may or may not remove compatibility notes for other - > obsolete versions of Oniongroove as they become obsolete. + * Onionbalance support. + + * Key management with encrypted backups. * Onionbalance: * See [these questions about current Onionbalance limits](https://gitlab.torproject.org/tpo/community/team/-/issues/57#note_2780708). @@ -851,17 +881,9 @@ Some of the relevant [Onion Services issues](https://gitlab.torproject.org/group replicated but needs a way for backup frontend nodes to takeover if others are unavailable. -* Existing solutions: - * EOTK: why not just use/extend it? - * Need for an idempotent Infrastructure as Code (IaC) implementation - ensuring that systems are in a defined state. - - * Different use-cases: EOTK is focused mainly in public-face sites, - while a more general tool might be useful. - - * Onionbalance support. - - * Key management with encrypted backups. +* Security considerations: + * Some of them can be moved to an official documentation an be used as + a reference of what SHOULD be done. * This document is very big! Perhaps the design discussion should be separated from the specifications (a more synthetic document). -- GitLab From 4a746cbea39132830c46bc21bfa03a8e1863467c Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Thu, 10 Mar 2022 17:25:55 -0300 Subject: [PATCH 31/48] Fix: specs: some adjusts --- docs/specs.md | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index dd06fe6..826fa50 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -408,7 +408,9 @@ Additional services: * Each worker MAY stay in a separate machine/location. -* Each endpoint MAY stay in a separate machine/location. +* Each endpoint MAY stay in a separate machine/location. Isolation between the + Oniongroove CDN and the endpoints is RECOMMENDED as a [best + practice](https://help.riseup.net/en/security/network-security/tor/onionservices-best-practices#be-careful-of-localhost-bypasses). * Each monitor node could be grouped with it's tor counterpart (say `monitor1` and `tor1`). @@ -584,7 +586,7 @@ considered: * Setting The Vanguards Onion Service Addon on each Tor daemon. For reference, check: - * [Consider adopting vanguard's security suggestions for onionbalance](https://gitlab.torproject.org/tpo/core/tor/-/issues/31857) + * [Consider adopting vanguard's security suggestions for onionbalance](https://gitlab.torproject.org/tpo/core/tor/-/issues/31857). * [How to OnionBalance - Vanguards security documentation](https://github.com/mikeperry-tor/vanguards/blob/master/README_SECURITY.md#how-to-onionbalance). * Encrypted offline backups of Onion Service keys: at least from the Onion @@ -606,16 +608,19 @@ implemented whenever possible: on premises. While this is out of Oniongroove scope, it's something that needs to be documented and explained to Operators. +* Regular audits. + ### Performance & Anti-DoS measures -* Use Onionbalance for Onion Service scalability. +* MUST use Onionbalance for Onion Service scalability. * SHOULD Apply other relevant [anti-DoS measures](https://community.torproject.org/onion-services/advanced/dos/). -* For public-facing (known IP addresses), - consider [single onion service](https://support.torproject.org/glossary/single-onion-service/), - using `HiddenServiceSingleHopMode`. +* For public-facing (known IP addresses), MUST offer + [single onion service](https://support.torproject.org/glossary/single-onion-service/), + using `HiddenServiceSingleHopMode` to enhance performance (with the cost of losing + server-side anonymity). * Basic CDN functionalities such as caching to avoid excessive access of the endpoints. @@ -682,12 +687,12 @@ Aggregating: Testing: -* Setup external probes using OnionScan and other monitoring tools to test - the heaht of the Onion Service from the outside. They could probe *both* - the Onionbalance frontend and the backends. They do not need to be - positioned in any special place in the internet as long as they can - access the Tor network, but preferably in a different network from - the Onionbalance CDN. +* Setup external probes using OnionScan and other monitoring tools to test the + health, quality and security of the Onion Service from the outside. They + could probe *both* the Onionbalance frontend and the backends. They do not need + to be positioned in any special place in the internet as long as they can + access the Tor network, but preferably in a different network from the + Onionbalance CDN. * Internal probe between each Onion Service backend and the application endpoint. -- GitLab From 0367d9b654299bfab649ce2b6487c9d533986568 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Thu, 10 Mar 2022 17:59:36 -0300 Subject: [PATCH 32/48] Fix: specs: moving some content to the open questions section --- docs/specs.md | 209 ++++++++++++++++++++++++++++---------------------- 1 file changed, 116 insertions(+), 93 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index 826fa50..4948cca 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -610,11 +610,13 @@ implemented whenever possible: * Regular audits. -### Performance & Anti-DoS measures +### Scalability + +Performance measures Anti-DoS mitigations to be taken into account: * MUST use Onionbalance for Onion Service scalability. -* SHOULD Apply other relevant [anti-DoS +* SHOULD Apply relevant [anti-DoS measures](https://community.torproject.org/onion-services/advanced/dos/). * For public-facing (known IP addresses), MUST offer @@ -622,41 +624,15 @@ implemented whenever possible: using `HiddenServiceSingleHopMode` to enhance performance (with the cost of losing server-side anonymity). -* Basic CDN functionalities such as caching to avoid excessive access of the endpoints. - -* Adopt recomendations suggested at the Tor docs, such as rate limiting - (`HiddenServiceExportCircuitID`), client authorization and CAPTCHAS, - depending of course on the site characteristics. - -* Consider to avoid HTTPS in the Onion Service: - * HTTPS required additional CPU consumption, especially for the handshake, - which could worsen the consequences of a DoS attack (CLIENT HELLO flooding). - - * Since HTTPS is served from an Onion Service, tradditional DoS HTTPS mitigations - would not work (like `fail2ban` or firewall rate limiting). - -* OR consider HTTPS with a special a cipher suite: - * Hypothesis do be researched/tested: perhaps an Onion Service with HTTPS - with a proper cipher suite might provide *both* an additional layer - of security and some rough kind of "proof of work" by letting the the client - use more CPU than the server; but right now I don't know the state of the - art on cipher suites and if there are a choice which provides both properties. - - * But even with this properties, HTTPS over Onion Service might be still costly - in comparison with plain HTTP over Onion Service. - - * Only with more research and testing this can be decided. - - * So, initially, it would be best to start by only using HTTP over Onion Service - and, if HTTPS proves to be useful, introduce it later with HSTS header and - HTTP-to-HTTPS redirection. - -### Service discovery - -* Service discovery for public-facing sites: - * Onion-Location HTTP header in the endpoint. +* Basic CDN functionalities such as temporary caching to avoid excessive access + of the endpoints SHOULD be considered, without converting the suite into a + caching/mirroring application, but only relying on caching to alleviate load in + the endpoints. - * Alt-Svc Header. + Oniongroove SHOULD mainly be a load balanced proxy solution. + Temporary caching SHOULD be available for optimization, which is an + interesting feature for public sites. But some applications MAY require that + cache is always off to prevent upstream content being stored in caches. ### Metrics @@ -810,85 +786,132 @@ Some of the relevant [Onion Services issues](https://gitlab.torproject.org/group ## Open questions -* Existing solutions: - * EOTK: why not just use/extend it? - * Need for an idempotent Infrastructure as Code (IaC) implementation - ensuring that systems are in a defined state. +### Meta + +* This document is very big! Perhaps the design discussion should be + separated from the specifications (a more synthetic document). + +### Existing solutions - * Different use-cases: EOTK is focused mainly in public-face sites, - while a more general tool might be useful. +* EOTK: why not just use/extend it? + * Need for an idempotent Infrastructure as Code (IaC) implementation + ensuring that systems are in a defined state. - * Onionbalance support. + * Different use-cases: EOTK is focused mainly in public-face sites, + while a more general tool might be useful. - * Key management with encrypted backups. + * Onionbalance support. -* Onionbalance: - * See [these questions about current Onionbalance limits](https://gitlab.torproject.org/tpo/community/team/-/issues/57#note_2780708). + * Key management with encrypted backups. - * Backend rotation: - * Does Onionbalance really need persistent backend addresses? From what was - investigated until now, it seems that what matters is that: - * The backend addresses are currently published . +### Onionbalance - * Every backend has both `HiddenServiceOnionBalanceInstance` and - `MasterOnionAddress` configuration options set. +* See [these questions about current Onionbalance limits](https://gitlab.torproject.org/tpo/community/team/-/issues/57#note_2780708). - * When rotating backends, it's important to not just turn off the older - backends, but to keep then online until all copies of the older - superdescriptor are expired/replaced with the updated list of introduction - points. +* Backend rotation: + * Does Onionbalance really need persistent backend addresses? From what was + investigated until now, it seems that what matters is that: + * The backend addresses are currently published . - * But this behavior MUST be tested in practice before considering any - backend rotation feature. + * Every backend has both `HiddenServiceOnionBalanceInstance` and + `MasterOnionAddress` configuration options set. -* Proxy: - * URL rewriting, blocking and similar features from EOTK. + * When rotating backends, it's important to not just turn off the older + backends, but to keep then online until all copies of the older + superdescriptor are expired/replaced with the updated list of introduction + points. - * OPTIONAL TLS certificate verification in the upstream connection: leave that - to the client browser or display a customized warning page in case of an - invalid certificate. + * But this behavior MUST be tested in practice before considering any + backend rotation feature. - * OPTIONAL upstream connection via Tor circuit, to prevent detection of the - Oniongroove CDN location. +### Proxy - * Custom NGINX page in case of any reverse proxy connection error is OPTIONAL. +* Should `balance` nodes be renamed to `proxy`, `middleware` or something else, + highlighting their roles both as load balancers and filters? - * Oniongroove CDN MUST NOT cache content? It MUST act only as a load balanced - proxy. Any caching SHOULD be implemented by the upstream application logic - (separation of concerns)? EOTK uses temporary caching for optimization, which - is an interesting feature for public sites. Some applications MAY require - that cache is always off to prevent upstream content being stored in caches. +* URL rewriting, blocking and similar features from EOTK. + +* OPTIONAL TLS certificate verification in the upstream connection: leave that + to the client browser or display a customized warning page in case of an + invalid certificate. + +* OPTIONAL upstream connection via Tor circuit, to prevent detection of the + Oniongroove CDN location. + +* Custom NGINX page in case of any reverse proxy connection error is OPTIONAL. + +### Key management * Password sharing mechanism: see discussion in this ticket: [evaluate password management options (#29677)](https://gitlab.torproject.org/tpo/tpa/team/-/issues/29677). -* Metrics: - * Needs description and diagram about metrics collection and reporting. +### Metrics - * Can metrics be submited/collected via .onion services? +* Needs description and diagram about metrics collection and reporting. -* Monitoring: - * Needs description and diagram about monitoring and notifications. +* Can metrics be submited/collected via .onion services? - * Don't prioritize a domain for the monitoring dashboard: use an .onion by - default with authentication. Currently Onionbalance does not support - authentication so this service MUST be implemented outside de CDN pool. +### Monitoring - * The nature of Tor networking might make unnecessary the use of many probes - in terms of network perspectives (like OONI does), making additional - probes only a matter of failover. But in any case probes could report - it's status in case they cannot reach the Tor network. +* Needs description and diagram about monitoring and notifications. -* Failover: - * How to do failover for probing/monitoring/stats/notifications? +* Don't prioritize a domain for the monitoring dashboard: use an .onion by + default with authentication. Currently Onionbalance does not support + authentication so this service MUST be implemented outside de CDN pool. - * How to do the Onionbalance failover? Config and keys might be easily - replicated but needs a way for backup frontend nodes to takeover if others - are unavailable. +* The nature of Tor networking might make unnecessary the use of many probes + in terms of network perspectives (like OONI does), making additional + probes only a matter of failover. But in any case probes could report + it's status in case they cannot reach the Tor network. -* Security considerations: - * Some of them can be moved to an official documentation an be used as - a reference of what SHOULD be done. +### Failover -* This document is very big! Perhaps the design discussion should be - separated from the specifications (a more synthetic document). +* How to do failover for probing/monitoring/stats/notifications? + +* How to do the Onionbalance failover? Config and keys might be easily + replicated but needs a way for backup frontend nodes to takeover if others + are unavailable. + +### Security considerations + +* Some of them can be moved to an official documentation an be used as + a reference of what SHOULD be done. + +### Scalability + +* HTTPS support MAY be considered depending also on performance considerations: + * Arguments to avoid HTTPS in the Onion Service related to performance: + * HTTPS required additional CPU consumption, especially for the handshake, + which could worsen the consequences of a DoS attack (CLIENT HELLO flooding). + + * Since HTTPS is served from an Onion Service, tradditional DoS HTTPS mitigations + would not work (like `fail2ban` or firewall rate limiting). + + * Arguments in favour of HTTPS with a special cipher suite: + * Hypothesis do be researched/tested: perhaps an Onion Service with HTTPS + with a proper cipher suite might provide *both* an additional layer + of security and some rough kind of "proof of work" by letting the the client + use more CPU than the server; but right now I don't know the state of the + art on cipher suites and if there are a choice which provides both properties. + + * But even with this properties, HTTPS over Onion Service might be still costly + in comparison with plain HTTP over Onion Service. + + * Only with more research and testing this can be decided. + + * So, if performance is a concern, it would be best to start by only using + HTTP over Onion Service and, if HTTPS proves to be useful, introduce it + later with HSTS header and HTTP-to-HTTPS redirection. + +### Service discovery + +Service discovery for public-facing sites is functionalities are outside of the +Oniongroove scope as they require to be set directly in the endpoints +public-facing servers, but MAY be mentioned as recommendations to be used +altogether with the CND deployment. + +Some technologies are available for that: + +* Onion-Location HTTP header in the endpoint. + +* Alt-Svc Header. -- GitLab From 1c43c79cbb12b659a262515b5cbd1566303b1a73 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Thu, 10 Mar 2022 18:31:49 -0300 Subject: [PATCH 33/48] Fix: specs: moving some content to the open questions section (2) --- docs/specs.md | 82 +++++++++++++++++++++++++++------------------------ 1 file changed, 43 insertions(+), 39 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index 4948cca..453ee0f 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -634,45 +634,6 @@ Performance measures Anti-DoS mitigations to be taken into account: interesting feature for public sites. But some applications MAY require that cache is always off to prevent upstream content being stored in caches. -### Metrics - -Collecting: some approaches do collect metrics from the Onion Service: - -* Use a proxy middleware that parses HTTP requests and responses, processing the - minimum metadata needed for stats on pages hits etc. - -* Read data from the ControlPort, preferably as an unix socket, but that needs - additional security protections. - -* Use the new MetricsPort config. But does this param support unix sockets as well? - -* Perhaps the Stem python library has some functionality to monitor Onion Services. - -* Onionbalance also supports a unix status socket. - -* Log parsing. But which useful information on Onion Service is available at - Tor daemon's log? - -Aggregating: - -* Temporal and other metrics aggregation to avoid any traffic correlation. - -* Send it to Prometheus or other system. - -* Setup a private metrics/monitoring dashboard for the CDN. - -Testing: - -* Setup external probes using OnionScan and other monitoring tools to test the - health, quality and security of the Onion Service from the outside. They - could probe *both* the Onionbalance frontend and the backends. They do not need - to be positioned in any special place in the internet as long as they can - access the Tor network, but preferably in a different network from the - Onionbalance CDN. - -* Internal probe between each Onion Service backend and the application - endpoint. - ### Folder scheme Folder scheme relies on the [XDG Base Directory @@ -915,3 +876,46 @@ Some technologies are available for that: * Onion-Location HTTP header in the endpoint. * Alt-Svc Header. + +### Metrics + +Regarding metric collection, some approaches do collect metrics from the Onion +Service needs to be considered: + +* Use a proxy middleware that parses HTTP requests and responses, processing the + minimum metadata needed for stats on pages hits etc. + +* Read data from the ControlPort, preferably as an unix socket, but that needs + additional security protections. + +* Use the new MetricsPort config. But does this param support unix sockets as well? + +* Perhaps the Stem python library has some functionality to monitor Onion Services. + +* Onionbalance also supports a unix status socket. + +* Log parsing. But which useful information on Onion Service is available at + Tor daemon's log? + +Aggregation can be done in a number of ways: + +* Sending them to Prometheus or other system. + +* Setting up a private metrics/monitoring dashboard for the CDN. + +* With temporal and other metrics aggregation to avoid any traffic correlation. + +### Testing + +The whole infrastructure needs to be tested, with notifications in case of incidents like +failures or high load: + +* By setting up external probes using OnionScan and other monitoring tools to + test the health, quality and security of the Onion Service from the outside. + They could probe *both* the Onionbalance frontend and the backends. They do not + need to be positioned in any special place in the internet as long as they can + access the Tor network, but preferably in a different network from the + Onionbalance CDN. + +* By using internal probes between each Onion Service backend and the + application endpoint. -- GitLab From 6b4fa67d5754c63f03a49b770d4f4122b5660661 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Thu, 10 Mar 2022 18:37:57 -0300 Subject: [PATCH 34/48] Feat: specs: encrypted keystore path and section organization --- docs/specs.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index 453ee0f..b7115b7 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -655,6 +655,7 @@ v0.8: * `$PROVIDER_PATH/sites/.yaml`: configuration for a single site name. * `$PROVIDER_PATH/nodes//.yaml`: configuration for a single node from a given type. * `$PROVIDER_PATH/provisioners/`: generated configuration for a given provisioning technology. + * `$PROVIDER_PATH/keyring/`: encrypted keystore. ### Configuration format @@ -666,7 +667,7 @@ v0.8: Example CLI invocations and behaviors. -### Provider CRUD +#### Provider CRUD Creating a provider: @@ -688,19 +689,19 @@ Creating a provider: -> Create provider git repository from template and using operator name/email -> (Optionally) Setup OpenPGP repository signature -### Node CRUD +#### Node CRUD * To be defined. -### Onion Services CRUD +#### Onion Services CRUD * To be defined. -### Applying the configuration +#### Applying the configuration * To be defined. -### Other subcommands +#### Other subcommands * Git wrapper. * Key regeneration? @@ -745,7 +746,7 @@ Some of the relevant [Onion Services issues](https://gitlab.torproject.org/group * [leap / leap_cli · GitLab](https://0xacab.org/leap/leap_cli), see [archived docs on LEAP providers](http://web.archive.org/web/20160314111917/https://leap.se/docs/platform): good UX reference for a CLI tool. * [stats: Create a stat exporter port for monitoring purposes (#40063) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/40063) -## Open questions +## Open topics and questions ### Meta -- GitLab From f362dfdc0f3a3ad34aa3fc6d18d120d7f49f662c Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Fri, 11 Mar 2022 09:04:02 -0300 Subject: [PATCH 35/48] Feat: specs: config file format and some UX CRUD --- docs/specs.md | 65 ++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 62 insertions(+), 3 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index b7115b7..08f205d 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -645,7 +645,9 @@ v0.8: * `$XDG_CONFIG_HOME/oniongroove/providers/`: operator configuration for a given provider. Per-persona settings, not meant to be shared between operators, so it's left outside the - provider git repository. + provider git repository: + * `$XDG_CONFIG_HOME/oniongroove/providers//config.yaml`: + main configuration file for per-persona and provider specific settings. * `$PROVIDER_PATH`: arbitrary path specified by the operator when setting up (or cloning) a new provider repository. Used to store @@ -659,9 +661,49 @@ v0.8: ### Configuration format -* YAML configuration format defining all entity instances in the model. +Oniongroove MUST use a [semantic versioned](https://semver.org/) +configuration in the [YAML](https://yaml.org/) format for defining +all entity instances in the model. -* Should be versioned. +Structure for the `$XDG_CONFIG_HOME/oniongroove/providers//config.yaml` config file: + +```yaml +--- +operator_name : 'Some Name' # Operator persona name (nickname) +operator_email : 'someone@example.org' # Operator persona email +openpgpg_fingerprint : 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' # OpenPGP key fingerprint + +provider_path: "~/some/path/to/oniongroove/provider" # Provider path in the local machine + +``` + +Structure for the `$PROVIDER_PATH/oniongroove.yaml` config file: + +```yaml +--- +version: '0.0.1' # Config version +name : 'Example' # Provider name + +``` + +Structure for the `$PROVIDER_PATH/sites/.yaml` config file: + +```yaml +--- +endpoints: # List of endpoints + 'endpoint1.example.org' + 'endpoint2.example.org' + 'endpoint3.example.org' + 'endpoint4.example.org' +``` + +Structure for the `$PROVIDER_PATH/nodes//.yaml` config file: + +```yaml +--- +type: 'backend' # Node type +fqdn: 'node-name.example.org' # FQDN for SSH access. Can be an onion service. +``` ### UX @@ -689,6 +731,23 @@ Creating a provider: -> Create provider git repository from template and using operator name/email -> (Optionally) Setup OpenPGP repository signature +Reading a provider: + + oniongroove info + +Updating a provider: + +* Should be done by editing the configuration files and re-provisioning. + +Removing a provider: + + oniongroove remove [-y] [-wipe] + + -> Ask for confirmation if needed + -> Removes or wipes $XDG_CONFIG_HOME/oniongroove/providers/ + -> Removes or wipes $PROVIDER_PATH + -> Informs Operator that all remote nodes should be removed manually + #### Node CRUD * To be defined. -- GitLab From c210ab9481d9068d31fa4db2414ccbf16dc7797c Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Fri, 11 Mar 2022 09:19:38 -0300 Subject: [PATCH 36/48] Feat: specs: more UX CRUD --- docs/specs.md | 72 +++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 59 insertions(+), 13 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index 08f205d..ce00fc3 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -560,6 +560,8 @@ The Onionbalance frontend address MUST be considered as long term and hence their keys MUST be stored on encrypted backups with some redundancy (both physical and human). +Keys MUST not be stored in plaintext in the provider repository. + ### Security considerations Running Onion Services is more sensitive than Tor relays (not to say even more @@ -709,11 +711,11 @@ fqdn: 'node-name.example.org' # FQDN for SSH access. Can be an onion service. Example CLI invocations and behaviors. -#### Provider CRUD +#### Provider config CRUD Creating a provider: - oniongroove create \ + oniongroove create provider \ [--path + oniongroove info provider + + -> Prints config, nodes etc Updating a provider: @@ -741,30 +745,64 @@ Updating a provider: Removing a provider: - oniongroove remove [-y] [-wipe] + oniongroove remove provider [-y] [-wipe] -> Ask for confirmation if needed -> Removes or wipes $XDG_CONFIG_HOME/oniongroove/providers/ -> Removes or wipes $PROVIDER_PATH -> Informs Operator that all remote nodes should be removed manually -#### Node CRUD +#### Node config CRUD + +Creating a node: + + oniongroove create node \ + [--type Check if node already exists + -> Create config file with (default?) node type + -> If backend, generate .onion keys for this backend for each defined site + -> If frontend, generate main .onion keys for each defined site + -> Encrypt keys for storage + +Reading a node: + + oniongroove info node + +Updating a node: + +* Should be done by editing the configuration files and re-provisioning. + +Removing a node: + + oniongroove remove node [-y] + +#### Onion Services config CRUD + +Creating an onion site: + + oniongroove create site -* To be defined. +Reading a site: -#### Onion Services CRUD + oniongroove info site -* To be defined. +Updating a node: + +* Should be done by editing the configuration files and re-provisioning. + +Removing a site: + + oniongroove remove site #### Applying the configuration -* To be defined. +Deployment is done with: -#### Other subcommands + oniongroove deploy -* Git wrapper. -* Key regeneration? -* X.509 keys and certs management? + -> Compiles configuration + -> Provision each node ## Roadmap @@ -979,3 +1017,11 @@ failures or high load: * By using internal probes between each Onion Service backend and the application endpoint. + +### UX + +Other subcommands to be considered: + +* Git wrapper. +* Key regeneration? +* X.509 keys and certs management? -- GitLab From cc5b56910251c74de013ebcfbce94b33d5fa5f6b Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Mon, 14 Mar 2022 09:19:39 -0300 Subject: [PATCH 37/48] Feat: specs: scalability: on huge number of backends --- docs/specs.md | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/docs/specs.md b/docs/specs.md index ce00fc3..0f14224 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -616,7 +616,27 @@ implemented whenever possible: Performance measures Anti-DoS mitigations to be taken into account: -* MUST use Onionbalance for Onion Service scalability. +* MUST use Onionbalance for Onion Service scalability: + * The number of backends MAY be way greater than those included in each + superdescriptor. This means that the number of introduction points + available $I_a$ for each onion site MAY be much bigger than the + number of introduction points included in the superdescriptor $I_s$ + for that same onion site, i.e., $I_a >> I_s$. + + * The task of introduction point selection remains to Onionbalance, which MAY + be patched to select introduction points randomly each time it publishes a + superdescriptor, providing automatic backend failover selection and an + additional balancing mechanism. + + * As an hypothetical example, the number of backends nodes -- each one + publishing one backend onion service per site in the provider -- could + yield $I_a = 200$ per site, whereas each frontend Onion Service could publish + up to $I_s = 20$. + + * It's important to note that having a huge number of published backend onion + services with established introduction points connections MAY imply in some + overhead to the Tor network, although this is believed to be low but MAY be + subject of further research. * SHOULD Apply relevant [anti-DoS measures](https://community.torproject.org/onion-services/advanced/dos/). -- GitLab From f7e8070f16d43f1c074778e14b440f1ba6d136e1 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Mon, 14 Mar 2022 11:08:57 -0300 Subject: [PATCH 38/48] Fix: specs: inline math --- docs/specs.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index 0f14224..97668f4 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -619,9 +619,9 @@ Performance measures Anti-DoS mitigations to be taken into account: * MUST use Onionbalance for Onion Service scalability: * The number of backends MAY be way greater than those included in each superdescriptor. This means that the number of introduction points - available $I_a$ for each onion site MAY be much bigger than the - number of introduction points included in the superdescriptor $I_s$ - for that same onion site, i.e., $I_a >> I_s$. + available $`I_a`$ for each onion site MAY be much bigger than the + number of introduction points included in the superdescriptor $`I_s`$ + for that same onion site, i.e., $`I_a >> I_s`$. * The task of introduction point selection remains to Onionbalance, which MAY be patched to select introduction points randomly each time it publishes a @@ -630,8 +630,8 @@ Performance measures Anti-DoS mitigations to be taken into account: * As an hypothetical example, the number of backends nodes -- each one publishing one backend onion service per site in the provider -- could - yield $I_a = 200$ per site, whereas each frontend Onion Service could publish - up to $I_s = 20$. + yield $`I_a = 200`$ per site, whereas each frontend Onion Service could publish + up to $`I_s = 20`$. * It's important to note that having a huge number of published backend onion services with established introduction points connections MAY imply in some -- GitLab From 1e72fff2875d9d7dc76cf42d46421ffd8cafb0fe Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Thu, 24 Mar 2022 13:46:09 -0300 Subject: [PATCH 39/48] Feat: specs: use cases --- docs/specs.md | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/docs/specs.md b/docs/specs.md index 97668f4..88d5dfe 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -46,6 +46,38 @@ to. The rest of this document details design choices, the architecture and implementation of Oniongroove. +## Use cases + +### Fit + +Onionprobe fits the following use cases: + +* Onionizing existing websites (as [EOTK](https://github.com/alecmuffett/eotk) + does). +* Hosting only one Onion Service websites with scalability (Oniongroove + provider with an unique site). +* Hosting many Onion Services websites. +* Hosting anonymous websites (i.e., those not served via regular HTTP/HTTPS + connections outside the Tor network). +* Hosting websites with or without HTTPS. + +### Don't fit + +Onionprobe does not fits the following use cases: + +* Hosting basic Onion Services websites (like Tor and Apache in a shared + hosting environment). +* Adding the + [Onion-Location](https://community.torproject.org/onion-services/advanced/onion-location/) + header into existing sites (that's the job of the existing site infrastructure + that serves content outside the Tor network). + +### May fit + +Oniongroove MAY fit the following use cases in the future: + +* Hosting Onion Services that do not rely on HTTP (Onion Services that aren't websites). + ## Preliminaries ### Assumptions -- GitLab From 1573e4d653660cb967ac22ee9aadc1460d89603a Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Tue, 29 Mar 2022 11:13:01 -0300 Subject: [PATCH 40/48] Feat: specs: Onionprobe --- docs/specs.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/specs.md b/docs/specs.md index 88d5dfe..937207a 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -975,6 +975,10 @@ Some of the relevant [Onion Services issues](https://gitlab.torproject.org/group probes only a matter of failover. But in any case probes could report it's status in case they cannot reach the Tor network. +* Oniongroove SHOULD use + [Onionprobe](https://gitlab.torproject.org/tpo/onion-services/onionprobe) or + a compatible tool. + ### Failover * How to do failover for probing/monitoring/stats/notifications? -- GitLab From ee34032557e0fb9fd5f1c9088991cea2e61ef892 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Wed, 30 Mar 2022 12:16:46 -0300 Subject: [PATCH 41/48] Feat: specs: node discovery and .onion subdomains --- docs/specs.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/specs.md b/docs/specs.md index 937207a..205a2b1 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -915,6 +915,10 @@ Some of the relevant [Onion Services issues](https://gitlab.torproject.org/group * Key management with encrypted backups. +### Deployment + +* OPTIONAL support for node discovery using and external service/source/API. + ### Onionbalance * See [these questions about current Onionbalance limits](https://gitlab.torproject.org/tpo/community/team/-/issues/57#note_2780708). @@ -951,6 +955,8 @@ Some of the relevant [Onion Services issues](https://gitlab.torproject.org/group * Custom NGINX page in case of any reverse proxy connection error is OPTIONAL. +* OPTIONAL Onion Service sites with subdomains in the URL. + ### Key management * Password sharing mechanism: see discussion in this ticket: [evaluate password -- GitLab From 4b1305da775e20a28fa98a56d327db86c58f9acd Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Wed, 6 Apr 2022 08:27:56 -0300 Subject: [PATCH 42/48] Fix: specs: typos --- docs/specs.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index 205a2b1..931371e 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -90,7 +90,7 @@ Oniongroove MAY fit the following use cases in the future: Client from the [Tor Rendezvous Specification - Version 3](https://gitlab.torproject.org/tpo/core/torspec/-/blob/main/rend-spec-v3.txt). -* It's assumed familiarity with Tor Onion Services, [Onion Balance](https://onionbalance.readthedocs.io) +* It's assumed familiarity with Tor Onion Services, [Onionbalance](https://onionbalance.readthedocs.io) and the [Vanguards Add-On](https://blog.torproject.org/announcing-vanguards-add-onion-services/). * The terms "suite" and "software suite" both refers implicitly to Oniongroove. @@ -510,7 +510,7 @@ that a redeployment is less costly than keeping an existing installation). backends, load balancers etc (could use Ansible for that, even Docker). This step should be vendor-neutral. -Oniongroove MUST operate as a translation between an implementation-neutral +Oniongroove MUST operate as a translator between an implementation-neutral configuration format and the specific configuration of any module/recipe such as Ansible cookbooks. In this sense, Oniongroove behaves basically as a configuration generator and dispatcher, abstracting details and -- GitLab From 9af58e1964b8181f91ac2d238454cae140dabf64 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Wed, 6 Apr 2022 09:39:22 -0300 Subject: [PATCH 43/48] Fix: specs: typos and corrections --- docs/specs.md | 45 +++++++++++++++++++++++---------------------- 1 file changed, 23 insertions(+), 22 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index 931371e..82ffb16 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -219,7 +219,7 @@ flowchart LR 3. Elasticity: disposability properties leads to the OPTIONAL feature in which backend nodes can be added and removed at will, making it from the start - compatible with any elastic capability to be speficied in the future -- like + compatible with any elastic capability to be specified in the future -- like adding and removing nodes according to backend overload. This depends if the Onionbalance instance to be restarted (resulting in @@ -245,8 +245,8 @@ flowchart LR 5. Migration support: built-in support to migrate existing onion services into the CDN instance, by just * Importing their frontend and backend keys along with configuration. - * Turning off the old system. * Deploying configuration and keys to fresh nodes using Oniongroove. + * Turning off the old system. 7. Testing: this suite MAY include test suites using either [Chutney](https://gitlab.torproject.org/tpo/core/chutney), the [Shadow @@ -262,9 +262,10 @@ flowchart LR * The reported [maximum number of Onionbalance backends](https://gitlab.torproject.org/tpo/core/onionbalance/-/issues/7). - * Despite the support for offline keys in the Onion Services v3 specification, - the Tor daemon currently does not support this feature, requiring a high - level of operational security to protect Onion Services' keys. + * Despite the support for offline keys in the Onion Services [v3 + specification](https://gitlab.torproject.org/tpo/core/torspec/-/blob/main/rend-spec-v3.txt) + (section 1.7), the Tor daemon currently does not support this feature, + requiring a high level of operational security to protect Onion Services' keys. * Other limits important to be considered in the scope of this project. @@ -403,18 +404,16 @@ Please note that the graph above is simplified: Main services: -* endpoints: each endpoint service can be a entry point for upstream websites; - as an example, they could be Apache web servers using Apache Virtual Hosts - and with a default `ServerName`, and hence no `ServerName blablabla.onion` is - needed; a layer of endpoints gives redundancy up to `N - 1` failing endpoints. +* endpoints: each endpoint service can be a entry point for upstream sites + available via HTTP (available only in the case of local connection) or HTTPS; + a layer of endpoints gives redundancy up to `N - 1` failing endpoints. * balancers: each load balancer service runs an NGINX/OpenResty instance configured to serve all websites from all endpoints; each website is also run on it's own port and a reverse proxy is set with all endpoints as upstreams; this also gives additional redundancy; port number to server name conversion happens at this stage. * backends: each backend service runs a Tor instance running an Onion Service - for each site; `HiddenServicePort` is set according to each sites' TCP port - in the load balancer; so, for redundancy, the Onion Service for each site is linked to + for each site; for redundancy, the Onion Service for each site is linked to an NGINX instance which is linked to all application endpoints. * frontends: runs a Tor instance to act as the Onion Service frontend for each Onion Site; right now just a single frontend is supported, so redundancy is @@ -438,13 +437,13 @@ Additional services: * Each frontend and it's onionbalance counterpart (say `frontend1` and `onionbalance1`) MAY be also grouped in a single machine. -* Each worker MAY stay in a separate machine/location. +* Each `worker` MAY stay in a separate machine/location. -* Each endpoint MAY stay in a separate machine/location. Isolation between the +* Each `endpoint` MAY stay in a separate machine/location. Isolation between the Oniongroove CDN and the endpoints is RECOMMENDED as a [best practice](https://help.riseup.net/en/security/network-security/tor/onionservices-best-practices#be-careful-of-localhost-bypasses). -* Each monitor node could be grouped with it's tor counterpart (say `monitor1` +* Each `monitor` node could be grouped with it's tor counterpart (say `monitor1` and `tor1`). ### Entity-relationship model @@ -499,7 +498,7 @@ frontend and monitoring instances. Implementations MUST consider a two-tier approach, where sysadmins keep a copy of all .onion keys (and optionally TLS certs) so the entire Onion Service Operation can be redeployed if needed (eg. disaster recovery, switching -providers or even in case the deployment procedure/stack changes and is found +providers or even in case the deployment procedure/stack changes and is decided that a redeployment is less costly than keeping an existing installation). 1. Vendor-specific tier: module(s) that bootstrap the basic environment (like a @@ -522,7 +521,7 @@ just: * The relevant Onion Service keys. * The configuration. * The software solution. -* A bunch of nodes (like VPSs). +* A bunch of nodes (like VPSes). A system which such capacity would then behave like an "Onion As a Service": @@ -567,14 +566,14 @@ Deployments MUST happen according to this workflow: is already a CDN then the set endpoint is composed of a single item. * Redeployment MAY not only needed, but planned with minimized downtimes and - wiping out keys from the old, decomissioned systems. + wiping out keys from the old, decommissioned systems. ### Key management When designing this solution, two choices for Onion Services key handling and certificates were considered: -1. Locally generated at the Operators computed and then deployed +1. Locally generated at the Operators computer and then deployed remotely along with the configuration OR 2. Generated directly in the machines hosting the Onion Service. @@ -582,7 +581,7 @@ certificates were considered: In the end, local generation was chosen as this approach have some advantages: -* Easier to backup the keys (using GnuPG optionally with a password/secret +* Easier to backup the keys (using GnuPG and optionally with a password/secret sharing application with multiple recipients and with digital signature support). @@ -605,7 +604,7 @@ and measures are needed, including but not limited to: correct permissions and ownership). * MUST use Tor packages from the official Tor repositories instead of the - operating system's distributtion repositories (e.g. Debian). + operating system's distribution repositories (e.g. Debian). * The application server (be an webserver or anything else) MUST not leak identifiable information such as hostnames or IP addresses (which can @@ -676,7 +675,9 @@ Performance measures Anti-DoS mitigations to be taken into account: * For public-facing (known IP addresses), MUST offer [single onion service](https://support.torproject.org/glossary/single-onion-service/), using `HiddenServiceSingleHopMode` to enhance performance (with the cost of losing - server-side anonymity). + server-side anonymity). In this case, using the Vanguards Addon is OPTIONAL, but + note that disabling it is a provider-wide configuration, so it is RECOMMENDED that + public-facing and pure-.onion services should not me mixed in the same provider. * Basic CDN functionalities such as temporary caching to avoid excessive access of the endpoints SHOULD be considered, without converting the suite into a @@ -1011,7 +1012,7 @@ Some of the relevant [Onion Services issues](https://gitlab.torproject.org/group * Arguments in favour of HTTPS with a special cipher suite: * Hypothesis do be researched/tested: perhaps an Onion Service with HTTPS with a proper cipher suite might provide *both* an additional layer - of security and some rough kind of "proof of work" by letting the the client + of security and some rough kind of "proof of work" by letting the client use more CPU than the server; but right now I don't know the state of the art on cipher suites and if there are a choice which provides both properties. -- GitLab From 48fcf18c5bb799b23d484272440bd813418b6aea Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Wed, 6 Apr 2022 18:39:52 -0300 Subject: [PATCH 44/48] Fix(specs): roadmap numbers --- docs/specs.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index 82ffb16..b5de251 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -864,9 +864,9 @@ Implementation is planned as follows: * Milestone #0: Prototype: a working CDN uning vendor-neutral modules. * Milestone #1: MVP: statistics. * Milestone #2: Monitoring. -* Milestone #4: Traffic control / protections. -* Milestone #5: Redundancy at the frontend, stats and monitoring nodes. -* Milestone #7: Vendor-specific tiers. +* Milestone #3: Traffic control / protections. +* Milestone #4: Redundancy at the frontend, stats and monitoring nodes. +* Milestone #5: Vendor-specific tiers. ## Relevant issues -- GitLab From f8a68a87b39280e3a6b01527c3dd3b6966d0ef04 Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Thu, 7 Apr 2022 14:48:13 -0300 Subject: [PATCH 45/48] Feat: specs: some more inspirations --- docs/specs.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/specs.md b/docs/specs.md index b5de251..3d1a465 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -891,8 +891,13 @@ Some of the relevant [Onion Services issues](https://gitlab.torproject.org/group * [Hiro / Roid · GitLab](https://gitlab.torproject.org/hiro/roid) * [Hiro / onions on heroku · GitLab](https://gitlab.torproject.org/hiro/onions-on-heroku) +* [Hiro / terraform-onions · GitLab](https://gitlab.torproject.org/hiro/terraform-onions) * [GitHub - alecmuffett/eotk: Enterprise Onion Toolkit](https://github.com/alecmuffett/eotk) * [GitHub - nusenu/ansible-relayor: An Ansible Role for Tor Relay Operators](https://github.com/nusenu/ansible-relayor): good reference in how to build an ansible role. +* SR2 [Terraform Registry](https://registry.terraform.io/modules/sr2c/tor-bridge) providers: + * [sr2c/tor/cloudinit | Terraform Registry](https://registry.terraform.io/modules/sr2c/tor/cloudinit/latest) + * [sr2c/torrc/null | Terraform Registry](https://registry.terraform.io/modules/sr2c/torrc/null/latest) + * [sr2c/contactinfo/null | Terraform Registry](https://registry.terraform.io/modules/sr2c/contactinfo/null/latest) * [leap / leap_cli · GitLab](https://0xacab.org/leap/leap_cli), see [archived docs on LEAP providers](http://web.archive.org/web/20160314111917/https://leap.se/docs/platform): good UX reference for a CLI tool. * [stats: Create a stat exporter port for monitoring purposes (#40063) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/40063) -- GitLab From 13db123dc039a44895c4cd8e6820a9c7f2c2e85a Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Fri, 8 Apr 2022 10:43:01 -0300 Subject: [PATCH 46/48] Feat(specs): implementation: proxy balancers --- docs/specs.md | 55 +++++++++++++++++++++++++++------------------------ 1 file changed, 29 insertions(+), 26 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index 3d1a465..42cb4bd 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -351,10 +351,10 @@ graph TD subgraph "Oniongroove CDN" - B1[Tor backend 1] --> L1[Load balancer 1] - B2[Tor backend 2] --> L2[Load balancer 2] - B3[Tor backend 3] --> L3[Load balancer 3] - Bn[Tor backend n] --> Ln[Load balancer n] + B1[Tor backend 1] --> L1[Proxy balancer 1] + B2[Tor backend 2] --> L2[Proxy balancer 2] + B3[Tor backend 3] --> L3[Proxy balancer 3] + Bn[Tor backend n] --> Ln[Proxy balancer n] end @@ -407,7 +407,7 @@ Main services: * endpoints: each endpoint service can be a entry point for upstream sites available via HTTP (available only in the case of local connection) or HTTPS; a layer of endpoints gives redundancy up to `N - 1` failing endpoints. -* balancers: each load balancer service runs an NGINX/OpenResty instance +* balancers: each proxy balancer service runs an NGINX/OpenResty instance configured to serve all websites from all endpoints; each website is also run on it's own port and a reverse proxy is set with all endpoints as upstreams; this also gives additional redundancy; port number to server name conversion @@ -431,7 +431,7 @@ Additional services: ### Service grouping -* Each backends and it's load balancer counterpart (say `backend1` and +* Each backends and it's proxy balancer counterpart (say `backend1` and `balancer1` MAY be grouped together in the same `worker` machine). * Each frontend and it's onionbalance counterpart (say `frontend1` and @@ -463,7 +463,7 @@ Simply put, this is the ER model for Oniongroove: * which is split in * frontend node(s) * backend nodes - * load balancers (to be plugged in the endpoint) + * proxy balancers (to be plugged in the endpoint) * monitor nodes * probe nodes @@ -506,7 +506,7 @@ that a redeployment is less costly than keeping an existing installation). vendor-specific logic depending on the hosting platform chosen. 2. Vendor neutral tier: a module that bootstraps the Onionbalance frontend(s), - backends, load balancers etc (could use Ansible for that, even Docker). + backends, proxy balancers etc (could use Ansible for that, even Docker). This step should be vendor-neutral. Oniongroove MUST operate as a translator between an implementation-neutral @@ -689,6 +689,27 @@ Performance measures Anti-DoS mitigations to be taken into account: interesting feature for public sites. But some applications MAY require that cache is always off to prevent upstream content being stored in caches. +### Proxy balancers + +These are the requisites for the proxy balancer instances: + +* Rewriting proxy capabilities: the proxy balance instances MUST be equiped with + URL rewriting to ensure that links are changed to their .onion counterparts. + +* Blocking capabilities: proxy balance instances also MUST provide support for + URL restrictions in a way that Operators can block access to login, admin + and other pages from .onion access. + +* REQUIRED TLS certificate verification in the upstream connection with OPTIONAL + custom warning page in case of an invalid certificate. + +* OPTIONAL upstream connection via Tor circuit, to prevent detection of the + Oniongroove CDN location. + +* OPTIONAL page in case of any reverse proxy connection error. + +* SHOULD support for Onion Service sites with subdomains in the URL. + ### Folder scheme Folder scheme relies on the [XDG Base Directory @@ -945,24 +966,6 @@ Some of the relevant [Onion Services issues](https://gitlab.torproject.org/group * But this behavior MUST be tested in practice before considering any backend rotation feature. -### Proxy - -* Should `balance` nodes be renamed to `proxy`, `middleware` or something else, - highlighting their roles both as load balancers and filters? - -* URL rewriting, blocking and similar features from EOTK. - -* OPTIONAL TLS certificate verification in the upstream connection: leave that - to the client browser or display a customized warning page in case of an - invalid certificate. - -* OPTIONAL upstream connection via Tor circuit, to prevent detection of the - Oniongroove CDN location. - -* Custom NGINX page in case of any reverse proxy connection error is OPTIONAL. - -* OPTIONAL Onion Service sites with subdomains in the URL. - ### Key management * Password sharing mechanism: see discussion in this ticket: [evaluate password -- GitLab From 0b9ff15c6da4c10b4a5f08fcd26e8621062f8fbc Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Fri, 8 Apr 2022 15:10:06 -0300 Subject: [PATCH 47/48] Feat(specs): misc additions from the specs meeting --- docs/specs.md | 58 ++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 41 insertions(+), 17 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index 42cb4bd..a69b593 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -39,6 +39,10 @@ sites by: 1. Offering a simplified interface, configuration format and convention. 2. Providing a deployment tool that takes care of most of the setup. +Addressing the problem helps to make the Onion Services technology an usual +website feature and not a special case, similar to what happened along the +years regarding HTTPS adoption. + It's important to note what **Oniongroove is not a caching/mirroring solution, but an Onion Services proxy layer**. It requires active endpoint(s) to connect to. @@ -60,6 +64,11 @@ Onionprobe fits the following use cases: * Hosting anonymous websites (i.e., those not served via regular HTTP/HTTPS connections outside the Tor network). * Hosting websites with or without HTTPS. +* A command line tool (CLI) for Onion Service management supporting teamwork. +* A library for Onion Service management, available to be integrated into + deployment systems. +* A service API, enabling systems as "click-to-deploy" dashboards to offer + Onion Service deployment an additional feature when creating websites. ### Don't fit @@ -540,6 +549,12 @@ Not sticking with an specific tool allows Oniongroove to be sustainable in the long term if the uderlying deployment technology changes or if the community prefers to change implementations. +### Modularization + +Such a system MUST not be composed of a single, monolitic codebase but instead +composed of many smaller modules that can be used as standalone applications +or reused in other implementations. + ### Deployment Deployments MUST happen according to this workflow: @@ -783,7 +798,7 @@ fqdn: 'node-name.example.org' # FQDN for SSH access. Can be an onion service. ### UX -Example CLI invocations and behaviors. +Example command line invocations and behaviors. #### Provider config CRUD @@ -882,12 +897,12 @@ Deployment is done with: Implementation is planned as follows: -* Milestone #0: Prototype: a working CDN uning vendor-neutral modules. -* Milestone #1: MVP: statistics. -* Milestone #2: Monitoring. -* Milestone #3: Traffic control / protections. +* Milestone #0: Prototype: a working CDN running vendor-neutral modules. +* Milestone #1: MVP: statistic modules. +* Milestone #2: Monitoring modules. +* Milestone #3: Traffic control / protections modules. * Milestone #4: Redundancy at the frontend, stats and monitoring nodes. -* Milestone #5: Vendor-specific tiers. +* Milestone #5: Vendor-specific module tiers. ## Relevant issues @@ -910,17 +925,26 @@ Some of the relevant [Onion Services issues](https://gitlab.torproject.org/group ## Inspirations -* [Hiro / Roid · GitLab](https://gitlab.torproject.org/hiro/roid) -* [Hiro / onions on heroku · GitLab](https://gitlab.torproject.org/hiro/onions-on-heroku) -* [Hiro / terraform-onions · GitLab](https://gitlab.torproject.org/hiro/terraform-onions) -* [GitHub - alecmuffett/eotk: Enterprise Onion Toolkit](https://github.com/alecmuffett/eotk) -* [GitHub - nusenu/ansible-relayor: An Ansible Role for Tor Relay Operators](https://github.com/nusenu/ansible-relayor): good reference in how to build an ansible role. -* SR2 [Terraform Registry](https://registry.terraform.io/modules/sr2c/tor-bridge) providers: - * [sr2c/tor/cloudinit | Terraform Registry](https://registry.terraform.io/modules/sr2c/tor/cloudinit/latest) - * [sr2c/torrc/null | Terraform Registry](https://registry.terraform.io/modules/sr2c/torrc/null/latest) - * [sr2c/contactinfo/null | Terraform Registry](https://registry.terraform.io/modules/sr2c/contactinfo/null/latest) -* [leap / leap_cli · GitLab](https://0xacab.org/leap/leap_cli), see [archived docs on LEAP providers](http://web.archive.org/web/20160314111917/https://leap.se/docs/platform): good UX reference for a CLI tool. -* [stats: Create a stat exporter port for monitoring purposes (#40063) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/40063) +* Generic: + * [GitHub - alecmuffett/eotk: Enterprise Onion Toolkit](https://github.com/alecmuffett/eotk) + * [Hiro / Roid · GitLab](https://gitlab.torproject.org/hiro/roid) +* Heroky: + * [Hiro / onions on heroku · GitLab](https://gitlab.torproject.org/hiro/onions-on-heroku) +* Ansible: + * [GitHub - nusenu/ansible-relayor: An Ansible Role for Tor Relay Operators](https://github.com/nusenu/ansible-relayor): good reference in how to build an ansible role. +* Terraform: + * [CDK for Terraform | Terraform by HashiCorp](https://www.terraform.io/cdktf) + * SR2 [Terraform Registry](https://registry.terraform.io/modules/sr2c/tor-bridge) providers: + * [sr2c/tor/cloudinit | Terraform Registry](https://registry.terraform.io/modules/sr2c/tor/cloudinit/latest) + * [sr2c/torrc/null | Terraform Registry](https://registry.terraform.io/modules/sr2c/torrc/null/latest) + * [sr2c/contactinfo/null | Terraform Registry](https://registry.terraform.io/modules/sr2c/contactinfo/null/latest) + * [Hiro / terraform-onions · GitLab](https://gitlab.torproject.org/hiro/terraform-onions) +* UX: + * [leap / leap_cli · GitLab](https://0xacab.org/leap/leap_cli), see [archived docs on LEAP providers](http://web.archive.org/web/20160314111917/https://leap.se/docs/platform): good UX reference for a CLI tool. +* Metrics: + * [stats: Create a stat exporter port for monitoring purposes (#40063) · Issues · The Tor Project / Core / Tor · GitLab](https://gitlab.torproject.org/tpo/core/tor/-/issues/40063) +* Other: + * Check the [Oniongroove deployment research](https://gitlab.torproject.org/tpo/onion-services/onion-support/-/issues/42) ticket. ## Open topics and questions -- GitLab From a100d493b10927061cdaef55d26b014db7a4841a Mon Sep 17 00:00:00 2001 From: Silvio Rhatto Date: Fri, 8 Apr 2022 16:58:18 -0300 Subject: [PATCH 48/48] Feat(specs): implementation roadmap via modules --- docs/specs.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/specs.md b/docs/specs.md index a69b593..460ef22 100644 --- a/docs/specs.md +++ b/docs/specs.md @@ -552,8 +552,8 @@ prefers to change implementations. ### Modularization Such a system MUST not be composed of a single, monolitic codebase but instead -composed of many smaller modules that can be used as standalone applications -or reused in other implementations. +by of many smaller modules that can be used as standalone applications or +reused in other implementations. ### Deployment @@ -895,9 +895,13 @@ Deployment is done with: ## Roadmap -Implementation is planned as follows: +It's RECOMMENDED that implementation start by doing simple usable modules, +planned as follows: -* Milestone #0: Prototype: a working CDN running vendor-neutral modules. +* Milestone #0: Prototype: a working CDN running vendor-neutral modules, + including: + * Key generation module. + * Provider configuration management module. * Milestone #1: MVP: statistic modules. * Milestone #2: Monitoring modules. * Milestone #3: Traffic control / protections modules. -- GitLab