Stop overloading Riseup's router on every Tails release

Summary

When tails does a release, all volunteer mirrors pull from an Rsync server hosted in Lizard. This causes upload spikes, saturating Lizard's 1 Gb/s network port.

90-day bandwitdh for Lizard as seen by the Riseup router tails-bw-90
Zoom in one event tails-bw-event
90-day bandwith for full router, note Lizard's spikes on upper half seattle-bw-90

When the physical interface gets saturated, it's riseup's router that needs to start dropping packets (rather than lizard itself). This causes negative effects on riseup's router and network, and downloads are less efficient.

Proposal 1: SQM

(thanks @taggart!)

  • Setup Smart Queue Management (SQM) on Lizard
  • Set UPLINK and DOWNLINK to 925000 (aka 925Mbps) to force Lizard to handle bandwith spikes

This would:

  • make things more fair for each of the mirrors downloading things (rather than the ones with more resources dominating)
  • ensure that latency remains good for all clients
  • allow other lizard traffic (including sysadmin access) to function responsively
  • make your mirror clients back off some rather than riseup's router having to drop their packets
  • won't effect other traffic going through riseup's seattle router as much. Riseup will still get alerts about the spikes, but it shouldn't degrade other services
Here is a short howto:
  • apt-get install make git iproute2 (if these aren't already installed)
  • git clone https://github.com/tohojo/sqm-scripts.git
  • cd sqm-scripts
  • make install PLATFORM=linux
  • cp /etc/sqm/default.conf /etc/sqm/eth0.iface.conf (adjust interface name as needed)
  • edit eth0.iface.conf and adjust UPLINK and DOWNLINK
  • reboot (ifupdown isn't quite enough)

Proposal 2: Enforce limit on the Rsync level

(thanks, @anarcat!)

Add --bwlimit 925M to the systemd unit ExecStart.

Other options

  • Push to mirrors (instead of having them pull from us)
  • Decrease the pull frequency (impacts RM)
  • Adapt Tor's CDN so it's able to distribute Tails
  • Have a tiered mirror network
  • Update mirrors in sets
  • Leverage Tails mirrors to circumvent censorship (eg. mirror Tor's website and binaries)
  • Distribute to mirrors via Torrent

To-do

Edited by zen