Snowflake issueshttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues2022-07-09T04:20:46Zhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40088Snowflake server keeps failing unexpectedly2022-07-09T04:20:46ZCecylia BocovichSnowflake server keeps failing unexpectedlyThe snowflake server has failed twice now in the last two weeks.
[After the first failure](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40060#note_2767868), we upgraded the capacity of the Sn...The snowflake server has failed twice now in the last two weeks.
[After the first failure](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40060#note_2767868), we upgraded the capacity of the Snowflake bridge (#40085).
Today when it failed we received an alert from monit: https://lists.torproject.org/pipermail/anti-censorship-alerts/2021-December/000786.html
Unfortunately, in my rush to fix it I didn't check the CPU usage :/ let's use this ticket to track the issue and note what we notice the next time it occurs.Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetshelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40085Upgrade CPU capacity of bridge2022-10-03T01:01:14ZDavid Fifielddcf@torproject.orgUpgrade CPU capacity of bridgeThe snowflake bridge is currently using most of its 4 CPUs. Most of the use is by snowflake-server. This was noticed by @meskio at https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40060#note_27678...The snowflake bridge is currently using most of its 4 CPUs. Most of the use is by snowflake-server. This was noticed by @meskio at https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40060#note_2767868.
```
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1822 debian-+ 20 0 4020400 2.0g 8192 R 250.7 26.4 156:08.20 snowflake-server
1821 debian-+ 20 0 831256 521804 106116 R 82.7 6.4 53:23.49 tor
1779 snowfla+ 20 0 1305556 47036 9068 S 16.0 0.6 6:27.23 proxy-go
1788 snowfla+ 20 0 1231824 43688 8988 S 10.7 0.5 3:18.65 proxy-go
```
![CPU Usage (%), 2021-12-21 05:11:27 to 2021-12-22 05:11:27](/uploads/d0392acb126b9385884f1f304e91f1c0/graph.png)
We should probably profile snowflake-server to find the hotspots to reduce its CPU usage (#40086), but for now we can redeploy on hardware with more CPUs. We're still doing okay with memory.
We are currently using
* 8 GiB / 4 CPU cores (40.75 credits)
And our options for upgrading are
* 12 GiB / 6 CPU cores (60.75 credits)
* 16 GiB / 8 CPU cores (80.75 credits)
I would tend to push it to the max, seeing how user growth is going.
A previous bridge upgrade issue was tpo/anti-censorship/pluggable-transports/snowflake#40051.David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40084Increase `clientIDAddrMapCapacity` in snowflake-server2023-03-29T16:05:53ZDavid Fifielddcf@torproject.orgIncrease `clientIDAddrMapCapacity` in snowflake-serverThe snowflake PT server has a data structure [`clientIDAddrMap`](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/aeb0794d2843d0cf9dfba2d8d4d0a9719b5636cd/server/lib/http.go#L42-49) that conveys cli...The snowflake PT server has a data structure [`clientIDAddrMap`](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/aeb0794d2843d0cf9dfba2d8d4d0a9719b5636cd/server/lib/http.go#L42-49) that conveys client IP addresses from the outer HTTPS "transport" layer and the inner KCP layer that actually talks to tor.
```go
// clientIDAddrMap stores short-term mappings from ClientIDs to IP addresses.
// When we call pt.DialOr, tor wants us to provide a USERADDR string that
// represents the remote IP address of the client (for metrics purposes, etc.).
// This data structure bridges the gap between ServeHTTP, which knows about IP
// addresses, and handleStream, which is what calls pt.DialOr. The common piece
// of information linking both ends of the chain is the ClientID, which is
// attached to the WebSocket connection and every session.
var clientIDAddrMap = newClientIDMap(clientIDAddrMapCapacity)
```
With the recent increase in Snowflake users, the number of client is tending to exceed the capacity of the map. The server is logging the following error, which is from [here](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/aeb0794d2843d0cf9dfba2d8d4d0a9719b5636cd/server/lib/snowflake.go#L209-213):
```
2021/12/21 17:04:59 no address in clientID-to-IP map (capacity 1024)
```
So we need to increase the fixed size of the mapping, `clientIDAddrMapCapacity`. Maybe from 1k to 10k.
The occurrences of the log message seem to be increasing. I initially thought this problem might be the cause of the apparent dip in users from https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40060#note_2767868, but I don't see an especially large count on 2021/12/19. The large count on 2021/12/21 may be because of the restart in https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40060#note_2767868.
```
# zgrep -h clientID-to-IP /var/log/tor/snowflake-server.log* | awk '{print $1}' | sort | uniq -c | tail -n 20
1 2021/12/02
14 2021/12/03
10 2021/12/04
20 2021/12/05
8 2021/12/06
15 2021/12/07
26 2021/12/08
30 2021/12/09
6 2021/12/10
8 2021/12/11
31 2021/12/12
36 2021/12/13
46 2021/12/14
52 2021/12/15
47 2021/12/16
48 2021/12/17
33 2021/12/18
49 2021/12/19
63 2021/12/20
464 2021/12/21
```
Our current log rotation has logs going back to 2021/06/21, but the first occurrence of the clientID-to-IP log message is on 2021/07/23.
```
1 2021/07/23
2 2021/07/29
2 2021/07/30
5 2021/07/31
1 2021/08/01
6 2021/08/03
8 2021/08/05
14 2021/08/10
2 2021/08/14
1 2021/08/15
1 2021/08/18
```https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40083Upgrade our standalone proxies for pion/dtls@v2.0.12 fingerprint changes2021-12-21T06:34:55ZDavid Fifielddcf@torproject.orgUpgrade our standalone proxies for pion/dtls@v2.0.12 fingerprint changes!66 updated the version of the pion/dtls dependency, in order to get a modified DTLS fingerprint in response to blocking in Russia. This update has already shipped for clients in [Tor Browser 11.5a1](https://blog.torproject.org/new-relea...!66 updated the version of the pion/dtls dependency, in order to get a modified DTLS fingerprint in response to blocking in Russia. This update has already shipped for clients in [Tor Browser 11.5a1](https://blog.torproject.org/new-release-tor-browser-115a1/). But we believe [the DTLS fingerprinting is bidirectional](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40014#note_2765074), and so both the client and the proxy need to have a good fingerprint, in order for the connection to work in Russia.
This issue is to upgrade the standalone proxies we run to commit https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/commit/738bd464eac631dd296ebf932f0a98f9bb9868e3 or later.
Discussion at the 2021-12-09 anti-censorship team meeting:
http://meetbot.debian.net/tor-meeting/2021/tor-meeting.2021-12-09-16.00.log.html#l-98
```
16:33:49 <shelikhoo> To make snowflake work for peoples influenced by this DTLS block, we might need to encourage standalone proxy operators to update software version
16:34:12 <cohosh> yes good point shelikhoo
16:34:22 <cohosh> we can use this module replacement trick and update the docker container
16:34:31 <cohosh> this makes that process easier than i thought
16:35:33 <dcf1> yes, on the point of standalone proxies, we need to encourage people to upgrade, or if we need to, we can potentially exclude proxies that have not upgraded, at the broker
16:36:21 <shelikhoo> or make sure updated client only match with updated standalone proxy
16:36:35 <shelikhoo> (but that will be a little complex)
16:37:10 <cohosh> the less complexity we add in the broker matching, probably the better
16:37:25 <cohosh> it is nice that if the client fails to connect it will keep trying
16:37:51 <arma2> shelikhoo: right, cohosh and i discussed that last night, and the direction we were heading is: try to get headless snowflakes to upgrade, and eventually stop handling the old ones, and then the broker matching algorithm can stay simple
16:38:46 <shelikhoo> Yes, so we wants to send proxy version in the broker request
16:38:58 <arma2> yep. and apparently we already do.
16:39:03 <shelikhoo> Yes
16:39:15 <cohosh> we used this to exclude old proxies before
```David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40082lockscreen, screensaver, disabled by snowflake2021-12-01T16:43:37Zpromeneurlockscreen, screensaver, disabled by snowflakeopenSUSE 15.3
Chrome 96
snowflake 0.5.4
When someone uses snowflake and uses webrtc protocol
then
my PC lokscreen and screensaver are disabled.
It's normal if I use webrtc but not if someone uses webrtc via snowflake.openSUSE 15.3
Chrome 96
snowflake 0.5.4
When someone uses snowflake and uses webrtc protocol
then
my PC lokscreen and screensaver are disabled.
It's normal if I use webrtc but not if someone uses webrtc via snowflake.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40081debian package fails to build in i3862022-05-03T19:23:11Zmeskiomeskio@torproject.orgdebian package fails to build in i386It looks like the tests are failing on the debian package build:
https://buildd.debian.org/status/fetch.php?pkg=snowflake&arch=i386&ver=1.1.0-2&stamp=1637172884&raw=0
```
=== RUN TestBrokerInteractions
Proxy connections to broker ✔...It looks like the tests are failing on the debian package build:
https://buildd.debian.org/status/fetch.php?pkg=snowflake&arch=i386&ver=1.1.0-2&stamp=1637172884&raw=0
```
=== RUN TestBrokerInteractions
Proxy connections to broker ✔
polls broker correctly ✔✔✔
handles poll error ✔2021/11/17 18:04:40 Error reading broker response: invalid character 'e' in literal true (expecting 'r')
2021/11/17 18:04:40 body: test
✔✔
sends answer to broker ✔✔✔✔✔
handles answer error panic: test timed out after 10m0s
```meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40078Problem with docker-compose on Raspberry Pi 3B+2021-11-09T20:35:46ZrichysProblem with docker-compose on Raspberry Pi 3B+Hi, I'm testing SNOWFLAKE on docker, and it's giving me an error that never finishes booting. Attached records.
Tutorial:
[view](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/wikis/home#option-3-stan...Hi, I'm testing SNOWFLAKE on docker, and it's giving me an error that never finishes booting. Attached records.
Tutorial:
[view](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/wikis/home#option-3-standalone)
Docker version on Raspberry Pi 3B +: docker-compose version 1.25.0, build unknown.
Logs:
```
docker-compose logs -f snowflake-proxy
Attaching to snowflake-proxy
snowflake-proxy | standard_init_linux.go: 219: exec user process caused: exec format error
snowflake-proxy | standard_init_linux.go: 219: exec user process caused: exec format error
snowflake-proxy | standard_init_linux.go: 219: exec user process caused: exec format error
snowflake-proxy | standard_init_linux.go: 219: exec user process caused: exec format error
snowflake-proxy | standard_init_linux.go: 219: exec user process caused: exec format error
snowflake-proxy | standard_init_linux.go:219: exec user process caused: exec format error
snowflake-proxy exited with code 1
snowflake-proxy | standard_init_linux.go:219: exec user process caused: exec format error
```
You can review it, I am interested in putting it on my RP3, I currently have it in the Firefox Browser, but I would like to have it in docker.
Best regardshttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40075Have standalone proxy retest their NAT assignment every 24 hours2021-11-16T19:37:42ZCecylia BocovichHave standalone proxy retest their NAT assignment every 24 hoursThe webextension will retest their NAT type every 24 hours. We originally didn't have the standalone proxy do this because of an assumption that they will be less mobile than the web-based proxies. However, after an issue with our NAT pr...The webextension will retest their NAT type every 24 hours. We originally didn't have the standalone proxy do this because of an assumption that they will be less mobile than the web-based proxies. However, after an issue with our NAT probe service we've seen our capacity of unrestricted proxies drop and not fully recover. This might be due to several unrestricted proxies failing their NAT check and are unable to recover. See the drop in "nat-unrestricted" snowflakes from the metrics:
```
snowflake-stats-end 2021-10-20 22:53:34 (86400 s)
snowflake-ips US=3777,DE=1271,RU=752,FR=655,JP=592,GB=468,CA=459,NL=336,BR=316,IN=292,AU=259,IT=240,CH=195,PL=162,ES=152,SE=146,MX=130,AT=128,IR=110,UA=108,TH=99,RO=92,ID=86,CN=83,BE=80,PT=79,DK=76,PH=74,CZ=68,FI=68,ZA=67,GR=66,HK=61,KR=60,AR=58,DZ=54,HU=52,BD=46,BG=46,SG=46,TR=46,CL=45,NO=44,IE=39,MY=36,SK=34,BY=33,TW=33,IL=29,NG=27,NZ=25,MA=23,SA=22,LU=21,CO=20,VN=20,LY=18,EE=17,KZ=17,LV=17,LT=16,EG=14,HR=14,IQ=12,NP=12,RS=12,TN=11,AE=10,KE=10,UY=10,EC=9,EU=9,MD=9,??=8,PE=8,BH=7,CR=7,IS=7,LK=7,UZ=7,AF=6,SI=6,SV=6,JO=5,PA=5,PK=5,AL=4,BA=4,MK=4,MM=4,PS=4,TZ=4,VE=4,AZ=3,CI=3,GT=3,JM=3,MT=3,MV=3,PR=3,QA=3,RW=3,BO=2,CY=2,DO=2,FO=2,GH=2,HT=2,KG=2,OM=2,PY=2,TT=2,BM=1,CW=1,GE=1,GF=1,GP=1,GQ=1,HN=1,KH=1,ME=1,MO=1,MR=1,RE=1,SY=1,UG=1,ZM=1
snowflake-ips-total 12797
snowflake-ips-standalone 3331
snowflake-ips-badge 34
snowflake-ips-webext 9432
snowflake-idle-count 3739888
client-denied-count 6208
client-restricted-denied-count 6208
client-unrestricted-denied-count 0
client-snowflake-match-count 147872
snowflake-ips-nat-restricted 7259
snowflake-ips-nat-unrestricted 241
snowflake-ips-nat-unknown 5275
snowflake-stats-end 2021-10-21 22:53:34 (86400 s)
snowflake-ips US=3706,DE=1221,RU=819,FR=635,JP=533,CA=465,GB=418,NL=347,BR=334,IN=273,IT=270,CH=229,AU=191,PL=157,ES=145,SE=136,AT=135,IR=128,TH=109,ID=106,CN=105,MX=98,BE=92,FI=82,UA=82,PH=79,PT=75,RO=74,CZ=69,HK=69,DK=66,IL=65,GR=62,ZA=61,NO=58,HU=55,DZ=54,BD=52,KR=50,AR=49,TR=46,SG=41,BG=39,CL=37,TW=36,SK=35,IE=31,NZ=27,BY=26,MY=26,SA=26,CO=24,LU=24,NG=24,EU=23,VN=21,EG=20,KZ=18,MA=17,MD=17,RS=17,LV=16,??=14,LT=13,SV=13,IS=12,LY=12,HR=11,PE=11,EE=9,KE=9,AE=8,NP=8,TN=8,TZ=8,UY=8,JO=7,PA=7,CR=6,PK=6,SI=6,EC=5,UZ=5,AF=4,CY=4,GT=4,LK=4,MM=4,PS=4,AL=3,AZ=3,BA=3,BH=3,CI=3,IQ=3,JM=3,KG=3,MK=3,MV=3,PR=3,QA=3,BJ=2,BO=2,DO=2,FO=2,MO=2,PY=2,SY=2,TT=2,UG=2,VE=2,ZM=2,AW=1,BM=1,CG=1,CU=1,CW=1,ET=1,GE=1,GF=1,GH=1,GP=1,GQ=1,HN=1,HT=1,ME=1,MR=1,MT=1,RE=1,RW=1,SC=1,SD=1,SS=1,YE=1
snowflake-ips-total 12635
snowflake-ips-standalone 3263
snowflake-ips-badge 47
snowflake-ips-webext 9325
snowflake-idle-count 3795824
client-denied-count 9136
client-restricted-denied-count 9136
client-unrestricted-denied-count 0
client-snowflake-match-count 140248
snowflake-ips-nat-restricted 7155
snowflake-ips-nat-unrestricted 221
snowflake-ips-nat-unknown 5234
snowflake-stats-end 2021-10-22 22:53:34 (86400 s)
snowflake-ips US=3443,DE=1264,RU=811,FR=606,JP=512,GB=456,CA=402,NL=312,BR=280,IN=258,AU=217,IT=215,CH=201,PL=159,ES=147,SE=135,IR=132,AT=123,BE=119,ID=113,CN=106,FI=88,PH=87,RO=85,PT=82,DK=81,MX=78,TH=78,UA=74,CZ=61,GR=61,ZA=61,HK=58,DZ=52,KR=52,HU=51,NG=48,SG=48,AR=47,NO=45,BG=44,TR=43,BD=42,IL=34,TW=34,CL=33,MY=31,NZ=28,SA=27,SK=27,BY=26,VN=25,IE=24,MA=22,LU=20,MD=19,KZ=17,CO=16,LT=16,AE=14,EE=13,HR=13,LV=13,RS=13,EG=12,LY=12,NP=11,IS=10,PE=10,SV=10,BH=9,LK=9,SI=9,EC=8,PA=8,UY=8,??=7,CR=7,EU=7,KE=7,YE=7,TN=6,UZ=6,AF=5,GE=5,IQ=5,JM=5,JO=5,MK=5,MM=5,AL=4,BA=4,CI=4,KG=4,QA=4,TZ=4,BO=3,GT=3,HN=3,PK=3,PR=3,VE=3,AO=2,CY=2,FO=2,GF=2,GH=2,MO=2,MV=2,PS=2,PY=2,RE=2,TT=2,AD=1,AZ=1,BJ=1,BM=1,BS=1,CM=1,CW=1,DO=1,ET=1,GP=1,GQ=1,HT=1,MT=1,SS=1,SY=1,ZM=1
snowflake-ips-total 12120
snowflake-ips-standalone 3186
snowflake-ips-badge 31
snowflake-ips-webext 8903
snowflake-idle-count 3574000
client-denied-count 5968
client-restricted-denied-count 5968
client-unrestricted-denied-count 0
client-snowflake-match-count 152912
snowflake-ips-nat-restricted 6769
snowflake-ips-nat-unrestricted 200
snowflake-ips-nat-unknown 5132
snowflake-stats-end 2021-10-23 22:53:34 (86400 s)
snowflake-ips US=3327,DE=1134,RU=823,JP=512,FR=501,GB=394,CA=337,NL=284,BR=265,IN=255,IT=206,CH=179,AU=165,ES=141,PL=140,IR=123,SE=122,ID=106,CN=104,AT=100,PT=96,TH=92,BE=83,UA=83,MX=80,FI=72,DK=69,PH=69,GR=62,RO=61,HU=59,HK=55,KR=55,DZ=52,CZ=51,ZA=51,BD=45,SG=45,CL=42,NO=42,AR=41,TW=41,TR=34,BY=33,IE=32,NG=32,MY=31,SK=31,LV=30,VN=27,CO=26,IL=25,EU=24,MA=24,NZ=24,SA=22,AE=21,LU=20,LY=20,EG=18,KZ=18,BG=17,HR=15,??=14,LT=14,EE=13,SV=12,CR=11,MD=11,PE=11,RS=10,KE=9,NP=9,PK=9,JO=8,LK=8,TN=8,BH=7,IS=7,JM=7,PA=7,SI=7,IQ=6,UY=6,UZ=6,EC=5,GE=5,KG=5,AZ=4,GH=4,MK=4,MM=4,AL=3,CY=3,MV=3,SN=3,YE=3,BO=2,CU=2,FO=2,HN=2,HT=2,LI=2,MO=2,MT=2,MU=2,PR=2,PS=2,PY=2,QA=2,SY=2,TT=2,VE=2,ZM=2,AW=1,BA=1,BM=1,BZ=1,GF=1,GQ=1,GT=1,MC=1,PG=1,RE=1,SS=1,TG=1,TZ=1
snowflake-ips-total 11381
snowflake-ips-standalone 3290
snowflake-ips-badge 28
snowflake-ips-webext 8063
snowflake-idle-count 3506824
client-denied-count 3112
client-restricted-denied-count 3112
client-unrestricted-denied-count 0
client-snowflake-match-count 165664
snowflake-ips-nat-restricted 6518
snowflake-ips-nat-unrestricted 266
snowflake-ips-nat-unknown 4574
snowflake-stats-end 2021-10-24 22:53:34 (86400 s)
snowflake-ips US=2956,DE=1269,RU=799,FR=578,JP=552,GB=385,CA=313,NL=277,IN=264,BR=240,IT=206,AU=183,CH=161,PL=157,ES=141,CN=129,IR=121,SE=107,AT=98,TH=95,MX=86,BE=84,ID=82,UA=73,FI=71,PH=65,PT=65,DK=64,HK=63,RO=57,DZ=56,GR=51,BD=48,HU=48,ZA=46,CZ=43,CO=40,NO=40,SG=40,AR=39,NG=37,TR=36,TW=36,IE=35,LV=30,CL=29,KR=29,IL=26,KE=25,NZ=25,SK=24,BY=23,VN=23,MY=22,SA=20,EU=19,KZ=19,EG=18,LU=18,AE=17,BG=17,RS=17,HR=16,LT=16,LY=16,MA=15,EE=14,NP=14,SV=13,TN=12,??=9,CR=9,LK=8,MD=8,PE=8,UZ=8,IQ=7,UY=7,IS=6,PA=6,PK=6,SI=6,TO=6,GE=5,GT=5,MK=5,SN=5,BO=4,MM=4,PR=4,AL=3,AZ=3,BA=3,BH=3,CY=3,EC=3,JM=3,KG=3,MV=3,UG=3,AF=2,CI=2,FO=2,QA=2,RE=2,SY=2,TT=2,VE=2,AO=1,BM=1,BZ=1,CG=1,ET=1,GF=1,GQ=1,HN=1,HT=1,JO=1,KH=1,MO=1,MT=1,PG=1,PY=1,TZ=1,ZM=1
snowflake-ips-total 11042
snowflake-ips-standalone 2955
snowflake-ips-badge 32
snowflake-ips-webext 8055
snowflake-idle-count 3421584
client-denied-count 10048
client-restricted-denied-count 10048
client-unrestricted-denied-count 0
client-snowflake-match-count 147144
snowflake-ips-nat-restricted 6101
snowflake-ips-nat-unrestricted 186
snowflake-ips-nat-unknown 4731
snowflake-stats-end 2021-10-25 22:53:40 (86400 s)
snowflake-ips US=3317,DE=1301,RU=787,FR=640,JP=523,GB=434,CA=392,IN=319,NL=306,IT=256,BR=250,AU=211,CH=205,PL=167,ES=157,IR=157,SE=133,CN=122,BE=107,TH=106,MX=104,AT=103,PT=90,PH=89,FI=86,ID=86,UA=82,RO=76,DK=73,DZ=69,BD=68,ZA=68,HK=64,CZ=63,GR=59,TR=55,NO=49,CL=46,SG=45,AR=43,KR=41,HU=40,TW=37,IE=35,SK=34,CO=33,MY=32,NZ=31,IL=29,BY=28,VN=28,EG=26,SA=26,BG=24,LU=24,MA=21,HR=20,KZ=18,EE=17,LV=17,AE=15,KE=15,LT=15,NP=15,TN=15,RS=14,LY=13,SV=13,LK=12,NG=11,UY=11,IS=10,EU=9,MD=8,PE=8,UZ=8,??=7,KG=7,SI=7,BA=6,BH=6,CR=6,MM=6,PA=6,EC=5,PK=5,PR=5,SN=5,AZ=4,GE=4,MK=4,VE=4,BO=3,CY=3,MO=3,MT=3,MV=3,PS=3,AL=2,CI=2,FO=2,GT=2,IQ=2,ME=2,QA=2,TT=2,AF=1,BM=1,BS=1,DO=1,GF=1,GH=1,GQ=1,HN=1,HT=1,JM=1,LA=1,RE=1,SY=1,TZ=1,UG=1,ZM=1
snowflake-ips-total 12198
snowflake-ips-standalone 3119
snowflake-ips-badge 33
snowflake-ips-webext 9046
snowflake-idle-count 3633752
client-denied-count 10560
client-restricted-denied-count 10560
client-unrestricted-denied-count 0
client-snowflake-match-count 135544
snowflake-ips-nat-restricted 5439
snowflake-ips-nat-unrestricted 141
snowflake-ips-nat-unknown 6589
snowflake-stats-end 2021-10-26 22:53:40 (86400 s)
snowflake-ips US=3865,DE=1279,RU=804,FR=661,JP=501,GB=443,CA=413,NL=312,IN=278,BR=277,IT=248,AU=246,PL=187,ES=175,CH=170,IR=152,SE=141,BE=106,MX=102,ID=94,CN=92,PT=88,AT=84,FI=82,RO=82,GR=79,PH=76,TH=74,UA=73,DK=71,ZA=67,HK=62,CZ=61,DZ=61,TR=54,AR=52,BD=50,TW=49,CL=46,KR=46,SG=45,HU=38,NO=36,MY=33,IE=30,SK=29,BY=26,NZ=26,IL=25,CO=23,LU=22,RS=22,EG=21,EU=21,LV=19,VN=19,BG=18,NG=18,HR=17,LY=17,SA=17,LT=16,MA=16,EE=15,IS=12,KZ=12,MD=12,AE=11,TN=11,UY=10,KE=9,CR=8,LK=8,MM=8,NP=8,PE=8,SV=8,PA=7,??=6,PR=6,SI=6,IQ=5,MK=5,UZ=5,BA=4,BH=4,CY=4,EC=4,GE=4,JM=4,KG=4,UG=4,FO=3,PK=3,PS=3,VE=3,AF=2,AL=2,AO=2,BJ=2,BO=2,CI=2,GF=2,GQ=2,ME=2,MV=2,QA=2,RE=2,SD=2,SY=2,TO=2,TT=2,AW=1,AZ=1,BM=1,BS=1,CU=1,CW=1,ET=1,GH=1,GP=1,GT=1,HN=1,JO=1,LA=1,ML=1,MO=1,MT=1,PY=1,SN=1,TZ=1,ZM=1
snowflake-ips-total 12633
snowflake-ips-standalone 3224
snowflake-ips-badge 19
snowflake-ips-webext 9390
snowflake-idle-count 3725384
client-denied-count 47864
client-restricted-denied-count 47864
client-unrestricted-denied-count 0
client-snowflake-match-count 131832
snowflake-ips-nat-restricted 3304
snowflake-ips-nat-unrestricted 74
snowflake-ips-nat-unknown 9230
snowflake-stats-end 2021-10-27 22:53:40 (86400 s)
snowflake-ips US=3319,DE=1330,RU=752,FR=572,JP=508,GB=405,CA=373,NL=314,BR=279,AU=271,IT=267,IN=242,CH=200,PL=194,ES=152,SE=141,IR=138,AT=109,DZ=107,ID=105,BE=102,MX=99,CN=98,UA=87,PT=84,RO=81,FI=79,TH=79,PH=76,HK=73,DK=70,GR=63,TR=63,ZA=62,AR=59,SG=53,MY=52,TW=50,CL=48,BD=47,CZ=43,HU=41,NO=41,KR=40,BY=38,IE=38,SK=36,LV=35,EG=33,NZ=33,LU=31,VN=30,BG=26,NG=26,CO=25,AE=22,IL=21,SA=21,EU=20,MA=19,RS=18,EE=17,LY=17,PE=16,HR=14,KZ=14,NP=13,IS=12,LT=12,MD=12,TN=11,??=10,KE=10,SI=10,UY=10,JM=8,CR=7,SV=7,BH=6,EC=6,GQ=6,KG=6,LK=6,MM=6,TZ=6,PA=5,PR=5,TD=5,UZ=5,CI=4,MK=4,PK=4,PS=4,AZ=3,BA=3,CY=3,GE=3,MV=3,SN=3,SY=3,VE=3,AF=2,DO=2,FO=2,GT=2,JO=2,KH=2,MO=2,MT=2,PG=2,QA=2,RE=2,TT=2,UG=2,AL=1,AO=1,BM=1,BO=1,BS=1,ET=1,GF=1,GH=1,GP=1,HN=1,KW=1,KY=1,ML=1,PY=1,TO=1,YE=1,ZM=1
snowflake-ips-total 12185
snowflake-ips-standalone 2902
snowflake-ips-badge 21
snowflake-ips-webext 9262
snowflake-idle-count 3773896
client-denied-count 118128
client-restricted-denied-count 118128
client-unrestricted-denied-count 0
client-snowflake-match-count 120208
snowflake-ips-nat-restricted 2484
snowflake-ips-nat-unrestricted 38
snowflake-ips-nat-unknown 9640
snowflake-stats-end 2021-10-28 22:53:40 (86400 s)
snowflake-ips US=3437,DE=1282,RU=784,FR=644,JP=548,GB=410,CA=343,NL=341,BR=267,IN=267,IT=263,AU=243,PL=184,CH=171,IR=152,SE=152,ES=142,AT=107,ID=102,CN=94,PT=92,MX=89,BE=87,FI=83,UA=81,DZ=80,RO=78,DK=77,TH=75,HK=73,ZA=66,PH=65,BD=61,CZ=57,GR=56,MY=50,AR=49,HU=49,SG=49,CL=45,IE=44,TW=41,NO=39,NZ=37,TR=35,KR=33,SK=33,BG=31,BY=28,IL=28,NG=27,VN=25,LU=23,SA=23,AE=20,CO=18,RS=18,EE=17,SV=17,EU=16,HR=15,KE=15,LV=15,KZ=14,LK=14,EG=13,LT=13,LY=12,MA=12,MM=12,PE=12,UY=12,MD=11,NP=10,PR=10,SI=10,CR=9,IS=8,MU=8,PK=8,TN=8,VE=8,EC=7,JM=7,SN=7,BH=6,PG=6,SY=6,??=5,GL=5,PS=5,AL=4,MK=4,PA=4,SZ=4,UG=4,UZ=4,FO=3,GE=3,JO=3,KG=3,MV=3,RE=3,TT=3,AF=2,BA=2,CI=2,CY=2,GQ=2,GT=2,MT=2,PY=2,QA=2,AO=1,AW=1,AZ=1,BM=1,BO=1,DO=1,ET=1,GF=1,GH=1,GP=1,HN=1,KH=1,KY=1,LI=1,MN=1,MO=1,SC=1,TZ=1
snowflake-ips-total 12232
snowflake-ips-standalone 2942
snowflake-ips-badge 31
snowflake-ips-webext 9259
snowflake-idle-count 3670632
client-denied-count 80976
client-restricted-denied-count 80976
client-unrestricted-denied-count 0
client-snowflake-match-count 109376
snowflake-ips-nat-restricted 1933
snowflake-ips-nat-unrestricted 34
snowflake-ips-nat-unknown 10218
snowflake-stats-end 2021-10-29 22:53:40 (86400 s)
snowflake-ips US=3679,DE=1233,RU=775,FR=593,JP=537,GB=420,NL=358,CA=347,IN=295,BR=266,IT=249,PL=174,AU=172,CH=164,IR=149,ES=148,SE=135,AT=116,MX=110,ID=99,DZ=95,BE=91,UA=91,TH=89,PT=84,CN=83,DK=82,FI=81,RO=75,CZ=65,ZA=60,BD=59,GR=59,HK=59,SG=56,PH=55,HU=52,AR=50,TW=45,TR=42,CL=40,NO=40,MY=38,NZ=32,SK=32,IE=31,SA=31,BY=29,KR=28,BG=24,IL=22,AE=21,EE=20,EG=20,EU=20,RS=20,VN=20,LT=18,HR=17,LU=17,CO=15,LV=14,LY=14,KZ=12,MM=12,??=11,CR=10,MA=10,MD=10,PR=10,AL=9,EC=9,KE=9,NG=9,SV=9,IS=8,UY=8,PE=7,UG=7,BH=6,CY=6,JM=6,KG=6,NP=6,SY=6,LK=5,PA=5,SI=5,TN=5,VE=5,PK=4,SZ=4,TZ=4,UZ=4,AZ=3,BA=3,BO=3,ET=3,JO=3,MK=3,MV=3,OM=3,PS=3,RE=3,SN=3,TT=3,AD=2,CU=2,FO=2,QA=2,SD=2,AO=1,AW=1,BM=1,CG=1,CI=1,CW=1,GE=1,GF=1,GH=1,GP=1,GQ=1,HN=1,IQ=1,KH=1,KY=1,LI=1,MO=1,MT=1,PY=1,SC=1,YE=1,ZM=1,ZW=1
snowflake-ips-total 12251
snowflake-ips-standalone 3061
snowflake-ips-badge 30
snowflake-ips-webext 9160
snowflake-idle-count 3687184
client-denied-count 100824
client-restricted-denied-count 100824
client-unrestricted-denied-count 0
client-snowflake-match-count 107520
snowflake-ips-nat-restricted 1790
snowflake-ips-nat-unrestricted 28
snowflake-ips-nat-unknown 10395
snowflake-stats-end 2021-10-30 22:53:40 (86400 s)
snowflake-ips US=3244,DE=1098,RU=715,JP=604,FR=515,CA=441,GB=374,IN=289,NL=289,BR=262,IT=200,AU=175,IR=161,CH=149,PL=148,ES=134,AT=110,SE=107,MX=104,DZ=96,BE=89,UA=89,TH=83,SG=77,FI=74,RO=72,CN=71,PT=69,ID=66,HK=60,CZ=59,BD=58,PH=57,ZA=55,DK=54,GR=51,CL=46,NO=44,TR=44,TW=40,AR=38,NZ=35,SA=33,NG=31,CO=28,VN=28,BY=27,EG=27,IE=26,KR=26,HU=25,SK=23,MY=22,PR=22,BG=19,LU=19,MA=19,KZ=15,CR=13,EU=13,LT=13,MD=13,MM=13,RS=13,EE=12,HR=12,IL=12,LV=12,NP=11,SV=11,KE=10,PE=10,TN=10,AE=9,??=7,EC=6,LK=6,LY=6,BA=5,BH=5,MV=5,PA=5,SI=5,SN=5,AL=4,BO=4,IS=4,JM=4,KG=4,PK=4,SY=4,VE=4,AZ=3,GT=3,QA=3,RE=3,UG=3,AD=2,CY=2,FO=2,GE=2,JO=2,MK=2,MT=2,OM=2,PS=2,TT=2,UY=2,UZ=2,YE=2,AW=1,BM=1,CU=1,CW=1,ET=1,GH=1,GP=1,GQ=1,HN=1,IQ=1,KH=1,ME=1,MO=1,PG=1,PY=1,ZM=1
snowflake-ips-total 11267
snowflake-ips-standalone 2821
snowflake-ips-badge 25
snowflake-ips-webext 8421
snowflake-idle-count 3695008
client-denied-count 168672
client-restricted-denied-count 168672
client-unrestricted-denied-count 0
client-snowflake-match-count 109680
snowflake-ips-nat-restricted 1446
snowflake-ips-nat-unrestricted 26
snowflake-ips-nat-unknown 9755
snowflake-stats-end 2021-11-01 20:26:47 (86400 s)
snowflake-ips US=3201,DE=1268,RU=765,FR=598,JP=576,GB=430,CA=389,NL=366,IN=294,IT=271,AU=249,BR=245,CH=185,PL=168,MX=157,SE=140,ES=137,IR=123,AT=101,PT=98,FI=95,ID=92,BE=91,TW=87,TH=85,UA=83,HK=79,DZ=76,CN=75,DK=70,GR=70,CZ=64,RO=62,PH=60,AR=55,TR=53,ZA=53,BD=50,IE=39,NO=39,SG=39,NZ=38,CL=34,HU=32,BY=31,MA=31,BG=30,SK=30,KR=29,SA=28,MY=25,PR=25,VN=24,CO=23,IL=23,HR=22,NG=22,RS=21,EG=19,LU=19,SV=16,AE=15,EE=15,IS=15,LT=15,KZ=14,LK=14,LY=13,EU=12,LV=10,NP=10,CR=9,MD=8,MM=8,TN=8,UY=8,PA=7,PE=7,BH=6,PK=6,SI=6,KE=5,SY=5,UZ=5,VE=5,BO=4,EC=4,KG=4,MV=4,YE=4,ZM=4,AZ=3,GH=3,IQ=3,JM=3,JO=3,PS=3,RE=3,??=2,AL=2,BA=2,BB=2,CU=2,CY=2,FO=2,GE=2,GQ=2,GT=2,HN=2,KW=2,MK=2,MT=2,QA=2,TT=2,TZ=2,AP=1,AW=1,BM=1,CI=1,CW=1,ET=1,GL=1,GP=1,LB=1,MO=1,MU=1,PG=1,PY=1,TJ=1
snowflake-ips-total 12019
snowflake-ips-standalone 2918
snowflake-ips-badge 19
snowflake-ips-webext 9082
snowflake-idle-count 3597496
client-denied-count 148832
client-restricted-denied-count 148824
client-unrestricted-denied-count 8
client-snowflake-match-count 100544
snowflake-ips-nat-restricted 3875
snowflake-ips-nat-unrestricted 65
snowflake-ips-nat-unknown 8062
```
If we implement this, we should "fail optimistic" as we do with the web proxies. That is, if a proxy is unrestricted and then the NAT check returns "unknown", they will not change their NAT. They only change it if they receive a definitive "restricted" answer.shelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40074Snowflake should include a settings icon2021-11-01T10:12:27ZcypherpunksSnowflake should include a settings iconCurrently, Snowflake has no icon, and browsers generate a default one for settings pages (the extension shows an icon in the browser bar, but not extension settings pages). This could be fixed by including a larger version of the icon (w...Currently, Snowflake has no icon, and browsers generate a default one for settings pages (the extension shows an icon in the browser bar, but not extension settings pages). This could be fixed by including a larger version of the icon (which I believe extensions ask for in the manifest, for icons on settings pages).https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40073Restart bridge and broker for update to virtualization platform2021-10-31T20:41:28ZDavid Fifielddcf@torproject.orgRestart bridge and broker for update to virtualization platformhttps://lists.torproject.org/pipermail/anti-censorship-team/2021-October/000196.html
> We are performing an update on our virtualization platform, this requires a change in the configuration of the storage backend. Due to this update yo...https://lists.torproject.org/pipermail/anti-censorship-team/2021-October/000196.html
> We are performing an update on our virtualization platform, this requires a change in the configuration of the storage backend. Due to this update your VPS needs to be stopped/started. We will execute this for you next week.
>
> If you prefer to plan this on your own, please feel free to stop/start your VPS yourself. By doing so, the VPS will be moved to the updated platform.
>
> Note: A reboot from within the machine itself will not be sufficient.
We [talked about this](http://meetbot.debian.net/tor-meeting/2021/tor-meeting.2021-10-28-16.00.log.html#l-34) at the 2021-10-28 anti-censorship team meeting.
@dcf will do the reboots, aiming for early on 2021-10-31 UTC.David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.org2021-10-31https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40072Make a prometheus alert for abnormal NAT assignments from probetest2021-11-04T16:05:57ZDavid Fifielddcf@torproject.orgMake a prometheus alert for abnormal NAT assignments from probetestRelated to #40071:
https://lists.torproject.org/pipermail/anti-censorship-team/2021-October/000197.html
> ...looking into the broker graphs there is something weird since 2 days. The number of proxies with 'unknown' type of nat has rise...Related to #40071:
https://lists.torproject.org/pipermail/anti-censorship-team/2021-October/000197.html
> ...looking into the broker graphs there is something weird since 2 days. The number of proxies with 'unknown' type of nat has rised heavily at the same time the 'restricted' nat has gone down. There are long periods without idle proxies and many requests being denied of nat type uknown. It doesn't look like the proxy capacity has gone down, can it be something broken on the way we test the nat type?
We want to get an automated alert when something like this happens.
At the 2021-10-28 anti-censorship team meeting [we discussed how to add new alerts](http://meetbot.debian.net/tor-meeting/2021/tor-meeting.2021-10-28-16.00.log.html#l-111):
```
<+meskio> who can do the alertmanager config? do we have access to that machine? or do we need to ask the metrics team?
<+cohosh> oh we can do it
<+cohosh> i set it up with anarcat during the last hackweek that all we need to do is make a MR
<+meskio> ahh, cool, so the config file is in a repo
<+meskio> I can do that, never touched alertmanager, but is in my list of things to learn
<+cohosh> https://gitlab.torproject.org/tpo/tpa/prometheus-alerts
```meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40069Snowflake needs outbound proxy support2023-10-25T15:40:24ZtlaSnowflake needs outbound proxy supportFor continued iOS support, we will need to run Snowflake behind a proxy, since with its Go runtime it's way to big to run in a [Network Extension](https://developer.apple.com/documentation/networkextension/packet_tunnel_provider), which ...For continued iOS support, we will need to run Snowflake behind a proxy, since with its Go runtime it's way to big to run in a [Network Extension](https://developer.apple.com/documentation/networkextension/packet_tunnel_provider), which has a hard 15 MByte RAM usage limit.
Currently, Snowflake doesn't seem to support that scenario.
Please point me to the code, if it actually has, so I can understand how to leverage it.
If not, I suggest having a look at Obfs4proxy for reference on how this could be implemented:
https://gitlab.com/yawning/obfs4/-/blob/master/obfs4proxy/obfs4proxy.go#L67-158
Thank you!shelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40067Add 'Number of users your Snowflake has helped so far' feature in the extension2021-10-04T10:10:56ZShakilAdd 'Number of users your Snowflake has helped so far' feature in the extensionCurrently, it has 'Number of users your Snowflake has helped circumvent censorship in the last 24 hours'. I really love watching the number grow from 0 to 10-15 every day. If it is possible to see all the people I have helped so far, tha...Currently, it has 'Number of users your Snowflake has helped circumvent censorship in the last 24 hours'. I really love watching the number grow from 0 to 10-15 every day. If it is possible to see all the people I have helped so far, that would be even more encouraging.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40066O1.1: Prepare Snowflake to handle a surge of operators and users.2022-09-27T18:43:09ZGabagaba@torproject.orgO1.1: Prepare Snowflake to handle a surge of operators and users.Although we already deployed Snowflake in Tor Browser, we want to be sure that Snowflake can handle all new users from China by preparing it with:
- [x] add many additional Snowflake operators (coordinate with @ggus on campaign),
- [ ]...Although we already deployed Snowflake in Tor Browser, we want to be sure that Snowflake can handle all new users from China by preparing it with:
- [x] add many additional Snowflake operators (coordinate with @ggus on campaign),
- [ ] monitor bottlenecks & blocking events (ongoing task for @tpo/anti-censorship),
- [x] set up at least one more snowflake bridge (1. prepare snowflake to give more than 2 bridge, 2. coordinate with @ggus for when partnering to have more bridges)
- [ ] respond to blocking events and prevent users from getting Snowflakes that have been blocked (ongoing task for @tpo/anti-censorship).Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetshelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40065Old CentOS Linux snowflake: go.mod at revision v1.1.0: unknown revision v1.1.02022-10-13T17:50:19ZslrslrOld CentOS Linux snowflake: go.mod at revision v1.1.0: unknown revision v1.1.0Hello, i have followed https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/wikis/home#source-code and "go build" output:
`go: git.torproject.org/pluggable-transports/goptlib.git@v1.1.0: reading git.torproj...Hello, i have followed https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/wikis/home#source-code and "go build" output:
`go: git.torproject.org/pluggable-transports/goptlib.git@v1.1.0: reading git.torproject.org/pluggable-transports/goptlib.git/go.mod at revision v1.1.0: unknown revision v1.1.0
`
also in the tutorial i would replace:
Build the Snowflake proxy.
```
go build
```
by:
Build the Snowflake proxy.
```
cd snowflake;go build
```
as it does not work without going to the git cloned dir.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40063Restructure snowflake library for v2 release2021-11-11T14:11:55ZCecylia BocovichRestructure snowflake library for v2 releaseThe next time we bump the version of Snowflake, we have to make a major version bump because of the changes to `NewSnowflakeClient` needed for the new AMP cache rendezvous method. This is a good opportunity to evaluate our API, which fun...The next time we bump the version of Snowflake, we have to make a major version bump because of the changes to `NewSnowflakeClient` needed for the new AMP cache rendezvous method. This is a good opportunity to evaluate our API, which functions/structures are exported, and which API features will be most useful for others who want to use our library. We should also take a look at best practices for library structure and naming mechanics and how much room we have to change there.
Some things to consider:
- [ ] [OONI has requested we introduce a way to direct Snowflake logs to a chosen output](https://github.com/ooni/probe/issues/1730)
- [x] As discussed in !50, if we export the `RendezvousMethod` interface, callers of the library can implement their own rendezvous methods
- [ ] Implementing callbacks for specific events would make it easier for calling programs to debug or react different to different types of connection failures or censorship attempts (see #40062 for some inspiration for this)
- [x] Can we get some official Go documentation/reference for this library? Let's at least evaluate if we need to improve what we have
- [x] We're probably exporting way more things than we need to be. Cutting down on what we export will make documentation and usage of the library simpler, and also [might help reduce binary size](#40004) (though I haven't confirmed this).Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40062Snowflake should self-diagnose where it fails in the connection process, and ...2022-04-08T10:56:52ZRoger DingledineSnowflake should self-diagnose where it fails in the connection process, and inform TorWe have periodic reports (e.g. #40044) of people in China saying that Tor Browser + Snowflake gets to 10% bootstrapped and can't get past it. We got another one on irc just now. Our own internal tests show that Snowflake bootstraps succe...We have periodic reports (e.g. #40044) of people in China saying that Tor Browser + Snowflake gets to 10% bootstrapped and can't get past it. We got another one on irc just now. Our own internal tests show that Snowflake bootstraps successfully on the VPS we're trying it from, but clearly that's not the end of the story. For example, I bet the mobile carriers have different constraints.
I was first thinking to suggest some standalone Snowflake debugging tool that would try a bunch of things and see how they go and summarize it for the user.
But then I realized: Snowflake itself should do this, because it *does* try things, and it learns how they go, and our users already have it. So all that remains is (a) figuring out which conclusions are worth escalating to the user, possibly including some refactoring inside Snowflake to do the steps in a way where we're confident in our results, and then (b) deciding what the right pathway is for escalating the information.
For 'b', we should be careful to avoid getting bogged down picking between options, since there are plenty of approaches that will do adequately. Maybe the PT log command is useful here, and (if I understand it correctly) in that case the way users can see the output is by preferences->tor->view logs.
And then I imagine the bulk of the work will be in step 'a'.
To get us started: what is the taxonomy of ways that Snowflake can fail to make its connection? And for each of those ways, is there an obvious point where Snowflake can self-assess that it has failed?Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibethttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40061It wont connect2022-10-05T15:37:32ZcypherpunksIt wont connectIt's stuck at 10 %It's stuck at 10 %https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40060server is still logging io.ErrClosedPipe errors because of wrapped errors2022-01-07T16:46:16ZDavid Fifielddcf@torproject.orgserver is still logging io.ErrClosedPipe errors because of wrapped errorsDespite !30, the Snowflake server is still logging `io.ErrClosedPipe` errors:
```
2021/06/24 17:41:12 error copying WebSocket to ORPort readfrom tcp [scrubbed]->[scrubbed]: io: read/write on closed pipe
2021/06/24 17:46:11 acceptStreams...Despite !30, the Snowflake server is still logging `io.ErrClosedPipe` errors:
```
2021/06/24 17:41:12 error copying WebSocket to ORPort readfrom tcp [scrubbed]->[scrubbed]: io: read/write on closed pipe
2021/06/24 17:46:11 acceptStreams: io: read/write on closed pipe
2021/06/24 17:46:33 error copying WebSocket to ORPort readfrom tcp [scrubbed]->[scrubbed]: io: read/write on closed pipe
2021/06/24 18:20:20 error copying ORPort to WebSocket io: read/write on closed pipe
```
The reason is that the errors are not really `io.ErrClosedPipe`; they are wrapped by [`errors.WithStack`](https://pkg.go.dev/github.com/pkg/errors#WithStack) in kcp-go. You can see the different using `log.Printf("%T", err)`, which yields `*errors.withStack`.
I was having the same problem in the dnstt server. I solved it by using [`errors.Is`](https://pkg.go.dev/errors#Is) from the [go1.13 errors interface](https://blog.golang.org/go1.13-errors), rather than plain equality.
https://repo.or.cz/dnstt.git/commitdiff/e4dc2883efea932f1da62ef35c3e88806aed9eeahttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40059Change how Snowflake handles client arguments2022-03-02T15:29:57ZCecylia BocovichChange how Snowflake handles client arguments@richard just pointed out on IRC that the way Snowflake's client-side arguments are passed to the executable make them difficult to dynamically change through Tor Browser's preferences. For Snowflake, these are specified through the `Cli...@richard just pointed out on IRC that the way Snowflake's client-side arguments are passed to the executable make them difficult to dynamically change through Tor Browser's preferences. For Snowflake, these are specified through the `ClientTransportPlugin` torrc option in the [`torrc-defaults`](https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/blob/5e61e15a2b71412538b3be5e9b62180f4d2686ce/projects/tor-browser/Bundle-Data/PTConfigs/linux/torrc-defaults-appendix) file:
```
## obfs4proxy configuration
ClientTransportPlugin meek_lite,obfs2,obfs3,obfs4,scramblesuit exec ./TorBrowser/Tor/PluggableTransports/obfs4proxy
## snowflake configuration
ClientTransportPlugin snowflake exec ./TorBrowser/Tor/PluggableTransports/snowflake-client -url https://snowflake-broker.torproject.net.global.prod.fastly.net/ -front cdn.sstatic.net -ice stun:stun.l.google.com:19302,stun:stun.voip.blackberry.com:3478,stun:stun.altar.com.pl:3478,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:3478,stun:stun.epygi.com:3478,stun:stun.sonetel.com:3478,stun:stun.sonetel.net:3478,stun:stun.stunprotocol.org:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478
```
Bridge lines, on the other hand, are specified in a seperate torrc file. See the [built-in preferences](https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/blob/5e61e15a2b71412538b3be5e9b62180f4d2686ce/projects/tor-browser/Bundle-Data/PTConfigs/bridge_prefs.js) for obfs4 and snowflake bridges.
Right now the only way to make changes to Snowflake client-side options (which have a huge impact on censorship) is to ship a new verison of Tor Browser or tell users to manually modify their torrc files.
@dcf also mentioned in !50 that we need to reconsider command-line options for Snowflake with the addition of new rendezvous methods. This is a related concern and we should make sure that how we chose to move forward works well with this scenario.
One option would be to instead specify command-line arguments through the pluggable transport specification PT args (as obfs4 does with the `cert` and `iat-mode` arguments). I haven't tried this, so I'm not sure it would work if two different bridge lines have the same fingerprint, but I believe it would allow us to specify multiple Snowflake configurations as separate bridges:
```
Bridge snowflake 192.0.2.3:1 2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://snowflake-broker.torproject.net.global.prod.fastly.net/ front=cdn.sstatic.net ice=stun:stun.l.google.com:19302
Bridge snowflake 192.0.2.3:2 2B280B23E1107BB62ABFC40DDCC8824814F80A72 ampcache=https://cdn.ampproject.org/ ice=stun:stun.l.google.com:19302
```Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & TibetCecylia BocovichCecylia Bocovich