A CDN is a "Content delivery network", "a geographically distributed network of proxy servers and their data centers. The goal is to provide high availability and performance by distributing the service spatially relative to end users." -- (Wikipedia)

Tor operates its own CDN in the form of the static-component system, but also uses external providers for certain edge cases like domain fronting and Tor browser upgrades (but not installs), since they are delivered over Tor.

This page documents mostly the commercial provider, see the static-component page for our own CDN.

Tutorial

For managing web sites in our own CDN, see doc/static-sites.

How-to

Changing components in the static site system

See the static-component documentation.

Domain fronting

The basic idea here is that you setup Fastly as a proxy for a service that is being censored. Let's call that service example.torproject.net for the purpose of this demonstration.

In the Fastly control panel (password in tor-passwords.git, hosts-extra-info):

Press "Create a Delivery service"
Choose example.torproject.net as the "Domain", so that requests with that domain in the Host: will route to this configuration, add name of the service and ticket reference number as a comment
then add a "host" (a backend, really) named example.torproject.net (yes, again), so that requests to this service will go to the backend (those are also called "origins" in Fastly)
"Activate" the configuration, this will give you a URL the domain fronting client should be able to use (the "test domain link"), which should be something like example.torproject.net.global.prod.fastly.net

Note that this does not support subpaths (e.g. example.torproject.net/foo), make a new virtual host for the service instead of using a subpath.

Also note that there might be other URLs you can use to reach the service in Fastly, see choosing the right hostname in Fastly's documentation.

Pager playbook

For problems with our own system, see the static-component playbook.

Disaster recovery

For problems with our own system, see the static-component disaster recovery.

Reference

We have two main CDNs systems that are managed by TPA. The first and more elaborate one is the static-component system, and the other is a commercial CDN provider, Fastly.

We have both for privacy reasons: we do not want to send our users to an external provider, where we do not control what they do with the user data, specifically their logging (retention) policies, law enforcement collaboration policies and, generally, we want to retain control over that data.

We have two main exceptions: one is for Tor browser upgrades, which are performed over Tor so should not cause any security issues to end users, and the other is domain fronting, which is specifically designed to use commercial CDNs to work around censorship.

Most of the following documentation pertains to the commercial CDN provider, see the static-component documentation for the reference guide on the other CDN.

Installation

For static site components, see the static-component installation documentation.

TODO: document how a site gets added into Fastly's CDN.

Upgrades

Not relevant for external CDNs.

SLA

There is no SLA specifically written for this service, but see also the static-component SLA.

Design and architecture

TODO: make a small architecture diagram of how Fastly works for TB upgrades and another for domain fronting

Services

Fastly provides services mostly over the web, so HTTPS all the way. It communicates with backends over HTTPS as well.

Storage

N/A

Queues

N/A

Interfaces

Fastly has an administrative interface over HTTPS and also an API that we leverage to configure services through the cdn-config-fastly.git. There we define the domains managed by Fastly and their backends.

Unfortunately, that code has somewhat bitrotten, is hard to deploy and to use, and has been abandoned.

The domain fronting stuff is manually configured through the https://manage.fastly.com/ interface.

Authentication

Fastly credentials are available in the TPA password manager, in tor-passwords.git.

Implementation

Fastly is a mostly proprietary service, but apparently uses Varnish (as of 2020).

Related services

Depends on TLS, DNS and relates to the static-component services.

Issues

There is no issue tracker specifically for this project, File or search for issues in the team issue tracker with the label Fastly or static-component label.

Fastly support is at https://support.fastly.com/.

Maintainer

Weasel setup the connection with Fastly in the first place, through his contacts at Debian.org. Anarcat is the current service admin.

Users

This service is used by the anti-censorship and applications teams.

Upstream

Fastly.com is our provider. The deal was negotiated in 2016 thanks to our Debian connections. Confirmation was in the Message-Id <CANeg4+d1St_bwU0JNbihhRMzniZnAhakX2O9Ha5b7b13D1pcvQ@mail.gmail.com>.

We have 20k

/mth credits. Effectively, we are billed bandwidth at 0

per month so it's hard to estimate how much of that we currently use, but according to the latest invoice (in April 2021), we were using about 186,000GB (so ~180TB) per month through 1300 requests (!?). According to their calculator that would be ~15000

/mth. So, back in April 2021, we had about 5k

/mth extra.

Monitoring and metrics

No monitoring of the Fastly service, see also tpo/tpa/team#21303.

Tests

Unclear. TODO: document how to test if Fastly works for TB and domain fronting.

Logs

TODO: document where the fastly logs are.

Backups

No backups, ephemeral service.

Discussion

Overview

The CDN service is stable in the sense that it doesn't see much change.

Its main challenge at this point is the duality between Fastly and our bespoke static-component system, with a lot of technical debt in the latter.

Security and risk assessment

There hasn't been a official security review done of the Fastly hosting service or its privacy policy, but it is rumoured that Fastly's privacy policies are relatively innocuous.

In principle, Varnish doesn't keep logs which, out of the box, should expose our users less, but Varnish is probably severely modified from the upstream. They do provide dashboards and statistics which show they inject some VCL in their configuration to at least add those analytics.

TODO: explicitly review the Fastly privacy policies and terms of service

Technical debt and next steps

The biggest technical debt is on the site of the static-component system, which will not be explicitly discussed here.

There is also no automation done for domain fronting, the cdn-config-fastly.git framework covering only the static-component parts.

Proposed Solution

No change is being proposed to the CDN service at this time.

Other alternatives

See static-component.

Comments

Please register or sign in to add a comment.

cdn