The current 4 tor instances on the snowflake-01 bridge are saturated near 100% CPU since yesterday, due to increased usage. (Right now the bridge is at about 200 MB/s outgoing, 140 MB/s incoming.) We need to increase the number of instances.
I am a bit worried that 8 instances will permit more traffic than the hardware of the bridge will permit. I'll watch the performance and be prepared to step it back to 6.
While making the change, also increase clientIDAddrMapCapacity because snowflake-server is reporting constant "no address in clientID-to-IP map (capacity 10240)". Previous issue: #40084 (closed).
# INSTANCES=$(for n in $(seq 5 8); do echo "snowflake$n"; done)# for instance in $INSTANCES; do tor-instance-create "$instance"; done# vi /etc/tor/instances/*/torrc # This is one of the instances of tor to which snowflake-server forwards # incoming traffic, via a load balancer. Each of the instances has a different # ServerTransportListenAddr. BridgeRelay 1 AssumeReachable 1 BridgeDistribution none ORPort 127.0.0.1:auto ExtORPort auto SocksPort 0 ServerTransportPlugin snowflake exec /usr/local/bin/extor-static-cookie /etc/extor-static-cookie/static_extended_orport_auth_cookie ServerTransportListenAddr snowflake 127.0.0.1:1000NUM Nickname flakeyNUM# for instance in $INSTANCES; do cp -r -v /var/lib/tor-instances/snowflake1/keys "/var/lib/tor-instances/$instance/" && chown -R -v _tor-"$instance" "/var/lib/tor-instances/$instance/keys"; done# for instance in $INSTANCES; do rm -rfv /var/lib/tor-instances/"$instance"/keys/secret_onion_key{,_ntor}.old; done# for instance in $INSTANCES; do mkdir -m 700 -p -v /var/lib/tor-instances/"$instance"/keys/secret_onion_key{,_ntor}.old; done# for instance in $INSTANCES; do for dir in /var/lib/tor-instances/"$instance"/keys/secret_onion_key{,_ntor}.old; do echo >"$dir/README" "This directory exists to prevent onion key rotation. See https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Guides/Snowflake-Bridge-Installation-Guide"; done; done# for instance in $INSTANCES; do chmod -v -w /var/lib/tor-instances/"$instance"/keys/secret_onion_key{,_ntor}; done# systemctl start tor@snowflake5 tor@snowflake6 tor@snowflake7 tor@snowflake8# cat /var/lib/tor-instances/*/fingerprintflakey1 2B280B23E1107BB62ABFC40DDCC8824814F80A72flakey2 2B280B23E1107BB62ABFC40DDCC8824814F80A72flakey3 2B280B23E1107BB62ABFC40DDCC8824814F80A72flakey4 2B280B23E1107BB62ABFC40DDCC8824814F80A72flakey5 2B280B23E1107BB62ABFC40DDCC8824814F80A72flakey6 2B280B23E1107BB62ABFC40DDCC8824814F80A72flakey7 2B280B23E1107BB62ABFC40DDCC8824814F80A72flakey8 2B280B23E1107BB62ABFC40DDCC8824814F80A72
I added the new instances to the end of the backend tor-instances section:
server snowflake5 127.0.0.1:10005 server snowflake6 127.0.0.1:10006 server snowflake7 127.0.0.1:10007 server snowflake8 127.0.0.1:10008
Then I reloaded the haproxy process. This is supposed to send a SIGUSR2 to make it reload its configuration without dropping existing connections.
# systemctl reload haproxy
I reloaded haproxy at 2022-09-22 20:12:32. It seems to be working. The new instances, after just a few minutes, are already up to ≈40% CPU. I was able to bootstrap a Tor Browser and see flakey8 in the logs.
About an hour later, all eight instances of tor and extor-static-cookie look like they have reached an equilibrium. The %MEM and TIME+ columns for snowflake-server a smaller than they would otherwise be, since the process OOM-restarted about 25 minutes ago.