Snowflake issueshttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues2022-09-27T18:43:09Zhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40066O1.1: Prepare Snowflake to handle a surge of operators and users.2022-09-27T18:43:09ZGabagaba@torproject.orgO1.1: Prepare Snowflake to handle a surge of operators and users.Although we already deployed Snowflake in Tor Browser, we want to be sure that Snowflake can handle all new users from China by preparing it with:
- [x] add many additional Snowflake operators (coordinate with @ggus on campaign),
- [ ]...Although we already deployed Snowflake in Tor Browser, we want to be sure that Snowflake can handle all new users from China by preparing it with:
- [x] add many additional Snowflake operators (coordinate with @ggus on campaign),
- [ ] monitor bottlenecks & blocking events (ongoing task for @tpo/anti-censorship),
- [x] set up at least one more snowflake bridge (1. prepare snowflake to give more than 2 bridge, 2. coordinate with @ggus for when partnering to have more bridges)
- [ ] respond to blocking events and prevent users from getting Snowflakes that have been blocked (ongoing task for @tpo/anti-censorship).Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetshelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/28651Prepare all pieces of the snowflake pipeline for a second snowflake bridge2023-05-15T19:25:40ZRoger DingledinePrepare all pieces of the snowflake pipeline for a second snowflake bridgeRight now there is one snowflake bridge, and its fingerprint is hard-coded in tor browser.
Eventually we will have enough load, and/or want more resiliency, that we want to set up a second snowflake bridge.
To be able to do that, I thi...Right now there is one snowflake bridge, and its fingerprint is hard-coded in tor browser.
Eventually we will have enough load, and/or want more resiliency, that we want to set up a second snowflake bridge.
To be able to do that, I think we need changes at the client, changes at the snowflake, and changes at the broker.
[Edit 2022-03: the three items I list next are no longer quite the best way to do this ticket. See the extensive and ongoing discussions below for what we currently think is the best plan.]
(A) At the snowflake side, the snowflake needs to tell the broker which bridge(s) it is willing to send traffic to. Additionally, we either want to declare that each snowflake sends to only one bridge, or we need to add a way for the client to tell the snowflake which bridge it wants to reach.
(B) At the broker side, we need it to be able to learn from snowflakes which bridge(s) they use, and we need it to be able to learn from clients which bridge they want to use, and we need it to match clients with snowflakes that will reach that bridge.
(C) At the client side, we need it to tell the broker which bridge it wants to use, and (depending on our design choice in A above) we might also need the client to be able to tell the snowflake which bridge it wants to use.
(There is an alternative approach, where we assume that every snowflake is always running the newest javascript, so it is willing to reach every bridge on our master list. Then the broker doesn't need to do anything new, and we just need to add a way for the client to tell the snowflake which bridge it wants. I don't have a good handle on how realistic this assumption is.)Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetshelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/34075Implement metrics to measure snowflake churn2023-07-18T18:57:33ZCecylia BocovichImplement metrics to measure snowflake churnAs discussed in the meeting this week, it would be useful to know how often snowflake proxy IP addresses actually change. We collect counts of unique IPs on any given day, but not how much variance we get in IP addresses over time.
This...As discussed in the meeting this week, it would be useful to know how often snowflake proxy IP addresses actually change. We collect counts of unique IPs on any given day, but not how much variance we get in IP addresses over time.
This relates to our ability to resist censorship, as snowflake relies in part on the claim that snowflakes are ephemeral, changing, and difficult to block exhaustively.
Let's implement some metrics to see how much snowflake IPs change.Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetshelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40095Add load balancing to bridge2022-10-17T05:15:38ZDavid Fifielddcf@torproject.orgAdd load balancing to bridgeWe rehearsed a load-balanced bridge installation in #40091. Now let's do it for the production bridge. To reduce risk, we plan to do a staged upgrade using a secondary bridge.
We still do not have a permanent solution to the [onion key ...We rehearsed a load-balanced bridge installation in #40091. Now let's do it for the production bridge. To reduce risk, we plan to do a staged upgrade using a secondary bridge.
We still do not have a permanent solution to the [onion key rotation issue](https://forum.torproject.net/t/tor-relays-how-to-reduce-tor-cpu-load-on-a-single-bridge/1483/11). The current plan is to periodically reset [LastRotatedOnionKey](https://gitweb.torproject.org/tor.git/tree/doc/state-contents.txt?h=tor-0.4.6.9#n82) in the state file of all tor instances.
- [x] ask sysadmin team to reduce TTL for snowflake.torproject.net to 60 seconds tpo/tpa/team#40594
- [x] copy user accounts to staging bridge https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40091#note_2768855
- [x] install new staging bridge ([installation guide](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40091#note_2770360))
- [x] refresh the LastRotatedOnionKey line in the state file of the production bridge and restart tor
- [x] back up identity and onion keys from production bridge
- [x] copy identity and onion keys from production bridge to the staging bridge
- [x] copy HTTPS TLS keys and certificates from the production bridge to the staging bridge
- [x] test HTTPS of staging bridge using `curl --connect-to`
- [x] test tor bootstrap on staging bridge using local broker and proxy, and temporary domain name https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40091#note_2770360
- [x] switch DNS for snowflake.torproject.net to point to staging bridge tpo/tpa/team#40598
- [x] monitor for a day, and be ready to switch DNS back to production if connections fail on the staging bridge
- [x] disable and mask tor@default instance on production bridge
- [x] install load balancing configuration on production bridge [installation guide](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40091#note_2770360)
- [x] test HTTPS of production bridge using `curl --connect-to`
- [x] test tor bootstrap on production bridge using local broker and proxy, and temporary domain name https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40091#note_2770360
- [x] switch DNS for snowflake.torproject.net to point back to production bridge tpo/tpa/team#40602
- [x] monitor for 2 days, and be ready to switch DNS back to staging if connections fail on the production bridge
- [x] ask sysadmin team to restore TTL for snowflake.torproject.net to normal tpo/tpa/team#40595
- [x] shut down staging bridge
- [x] post new instructions to [Snowflake Bridge Installation Guide](https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Guides/Snowflake-Bridge-Installation-Guide) and [Snowflake Bridge Survival Guide](https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Guides/Snowflake-Bridge-Survival-Guide)
References
* #40091
* https://forum.torproject.net/t/tor-relays-how-to-reduce-tor-cpu-load-on-a-single-bridge/1483
* http://meetbot.debian.net/tor-meeting/2022/tor-meeting.2022-01-20-15.59.log.html#l-60Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & TibetDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40088Snowflake server keeps failing unexpectedly2022-07-09T04:20:46ZCecylia BocovichSnowflake server keeps failing unexpectedlyThe snowflake server has failed twice now in the last two weeks.
[After the first failure](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40060#note_2767868), we upgraded the capacity of the Sn...The snowflake server has failed twice now in the last two weeks.
[After the first failure](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40060#note_2767868), we upgraded the capacity of the Snowflake bridge (#40085).
Today when it failed we received an alert from monit: https://lists.torproject.org/pipermail/anti-censorship-alerts/2021-December/000786.html
Unfortunately, in my rush to fix it I didn't check the CPU usage :/ let's use this ticket to track the issue and note what we notice the next time it occurs.Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetshelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/19409Make a deb of snowflake (proxy and client) and get into Debian2022-03-05T21:42:54ZadrelanosMake a deb of snowflake (proxy and client) and get into Debianaka
`apt-get install snowflake`
Speaking for Whonix, this would be very useful. Perhaps for Tails as well, but I am not speaking for them.
(Similar to legacy/trac#13160.)aka
`apt-get install snowflake`
Speaking for Whonix, this would be very useful. Perhaps for Tails as well, but I am not speaking for them.
(Similar to legacy/trac#13160.)Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetmeskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40055Standalone proxy reconnects in a tight loop when server refuses connection2022-10-11T19:44:19ZDavid Fifielddcf@torproject.orgStandalone proxy reconnects in a tight loop when server refuses connectionRun the proxy and let it connect to a broker port that refuses connections. After the probetest (about 30 s), it starts trying to poll the broker in a tight loop. We should probably wait at least `pollInterval` between connection attempt...Run the proxy and let it connect to a broker port that refuses connections. After the probetest (about 30 s), it starts trying to poll the broker in a tight loop. We should probably wait at least `pollInterval` between connection attempts in case of error.
```
snowflake/proxy$ ./proxy -broker http://localhost:9999/ 2>&1 | head -n 50
2021/07/19 18:00:04 starting
2021/07/19 18:00:04 WebRTC: Created offer
2021/07/19 18:00:04 WebRTC: Set local description
2021/07/19 18:00:04 Offer: {...}
2021/07/19 18:00:35 error polling probe: http2: timeout awaiting response headers
2021/07/19 18:00:35 NAT type: unknown
2021/07/19 18:00:35 error polling broker: dial tcp [scrubbed]: connect: connection refused
2021/07/19 18:00:35 Error reading broker response: unexpected end of JSON input
2021/07/19 18:00:35 body:
2021/07/19 18:00:35 bad offer from broker
2021/07/19 18:00:35 error polling broker: dial tcp [scrubbed]: connect: connection refused
2021/07/19 18:00:35 Error reading broker response: unexpected end of JSON input
2021/07/19 18:00:35 body:
2021/07/19 18:00:35 bad offer from broker
2021/07/19 18:00:35 error polling broker: dial tcp [scrubbed]: connect: connection refused
...
```Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetmeskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40062Snowflake should self-diagnose where it fails in the connection process, and ...2022-04-08T10:56:52ZRoger DingledineSnowflake should self-diagnose where it fails in the connection process, and inform TorWe have periodic reports (e.g. #40044) of people in China saying that Tor Browser + Snowflake gets to 10% bootstrapped and can't get past it. We got another one on irc just now. Our own internal tests show that Snowflake bootstraps succe...We have periodic reports (e.g. #40044) of people in China saying that Tor Browser + Snowflake gets to 10% bootstrapped and can't get past it. We got another one on irc just now. Our own internal tests show that Snowflake bootstraps successfully on the VPS we're trying it from, but clearly that's not the end of the story. For example, I bet the mobile carriers have different constraints.
I was first thinking to suggest some standalone Snowflake debugging tool that would try a bunch of things and see how they go and summarize it for the user.
But then I realized: Snowflake itself should do this, because it *does* try things, and it learns how they go, and our users already have it. So all that remains is (a) figuring out which conclusions are worth escalating to the user, possibly including some refactoring inside Snowflake to do the steps in a way where we're confident in our results, and then (b) deciding what the right pathway is for escalating the information.
For 'b', we should be careful to avoid getting bogged down picking between options, since there are plenty of approaches that will do adequately. Maybe the PT log command is useful here, and (if I understand it correctly) in that case the way users can see the output is by preferences->tor->view logs.
And then I imagine the bulk of the work will be in step 'a'.
To get us started: what is the taxonomy of ways that Snowflake can fail to make its connection? And for each of those ways, is there an obvious point where Snowflake can self-assess that it has failed?Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibethttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40059Change how Snowflake handles client arguments2022-03-02T15:29:57ZCecylia BocovichChange how Snowflake handles client arguments@richard just pointed out on IRC that the way Snowflake's client-side arguments are passed to the executable make them difficult to dynamically change through Tor Browser's preferences. For Snowflake, these are specified through the `Cli...@richard just pointed out on IRC that the way Snowflake's client-side arguments are passed to the executable make them difficult to dynamically change through Tor Browser's preferences. For Snowflake, these are specified through the `ClientTransportPlugin` torrc option in the [`torrc-defaults`](https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/blob/5e61e15a2b71412538b3be5e9b62180f4d2686ce/projects/tor-browser/Bundle-Data/PTConfigs/linux/torrc-defaults-appendix) file:
```
## obfs4proxy configuration
ClientTransportPlugin meek_lite,obfs2,obfs3,obfs4,scramblesuit exec ./TorBrowser/Tor/PluggableTransports/obfs4proxy
## snowflake configuration
ClientTransportPlugin snowflake exec ./TorBrowser/Tor/PluggableTransports/snowflake-client -url https://snowflake-broker.torproject.net.global.prod.fastly.net/ -front cdn.sstatic.net -ice stun:stun.l.google.com:19302,stun:stun.voip.blackberry.com:3478,stun:stun.altar.com.pl:3478,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:3478,stun:stun.epygi.com:3478,stun:stun.sonetel.com:3478,stun:stun.sonetel.net:3478,stun:stun.stunprotocol.org:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478
```
Bridge lines, on the other hand, are specified in a seperate torrc file. See the [built-in preferences](https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/blob/5e61e15a2b71412538b3be5e9b62180f4d2686ce/projects/tor-browser/Bundle-Data/PTConfigs/bridge_prefs.js) for obfs4 and snowflake bridges.
Right now the only way to make changes to Snowflake client-side options (which have a huge impact on censorship) is to ship a new verison of Tor Browser or tell users to manually modify their torrc files.
@dcf also mentioned in !50 that we need to reconsider command-line options for Snowflake with the addition of new rendezvous methods. This is a related concern and we should make sure that how we chose to move forward works well with this scenario.
One option would be to instead specify command-line arguments through the pluggable transport specification PT args (as obfs4 does with the `cert` and `iat-mode` arguments). I haven't tried this, so I'm not sure it would work if two different bridge lines have the same fingerprint, but I believe it would allow us to specify multiple Snowflake configurations as separate bridges:
```
Bridge snowflake 192.0.2.3:1 2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://snowflake-broker.torproject.net.global.prod.fastly.net/ front=cdn.sstatic.net ice=stun:stun.l.google.com:19302
Bridge snowflake 192.0.2.3:2 2B280B23E1107BB62ABFC40DDCC8824814F80A72 ampcache=https://cdn.ampproject.org/ ice=stun:stun.l.google.com:19302
```Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & TibetCecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40056O1.2: Increase the number of Snowflake proxies.2022-07-05T18:08:28ZGabagaba@torproject.orgO1.2: Increase the number of Snowflake proxies.With limited ramp up so far, we already have 6,000 volunteers running Snowflakes. Increasing Snowflake operators through more aggressive promotion will allow us to rapidly increase unblocked bridge entry points to the Tor network by thou...With limited ramp up so far, we already have 6,000 volunteers running Snowflakes. Increasing Snowflake operators through more aggressive promotion will allow us to rapidly increase unblocked bridge entry points to the Tor network by thousands.
- O1.2.1: Run a public campaign to encourage Internet users to install the Snowflake extension in Firefox or Chrome and use their web browser to act as bridges for censored people. This campaign will target individuals who care about free access to information but who may have limited capacity to operate a traditional bridge.
- O1.2.2: Scale Tor reachability through mobile Snowflakes. It’s also possible to build the Snowflake into mobile applications. We can rapidly increase the availability and diversity of bridge IP addresses that are not blocked in China by asking the millions of active Orbot users to “help censored users connect to Tor.” Through the Internews DISRUPT program, Guardian Project is designing and implementing “Orbot as Bridge” capability with the use of Snowflake, and as a result of this Activity, any Orbot user will be able to become a Snowflake bridge to Tor when they are not actively using their device. Guardian Project is also implementing as much of this capability as possible in Onion Browser for iOS.1 To do so, Guardian Project will: continue to improve Tor+Snowflake stability & performance on mobile devices and tune mobile-as-Snowflake bridge usability and performance, especially considering connections from the target regions.Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibet2022-12-31https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40089Possibly malicious snowflake polls?2022-02-21T14:35:42ZCecylia BocovichPossibly malicious snowflake polls?I just took a look at the exported prometheus metrics: https://snowflake-broker.torproject.net/prometheus
and saw a bunch of very unusual snowflake proxy types:
```
snowflake_rounded_proxy_poll_total{nat="restricted'\"\\(",status="idle"...I just took a look at the exported prometheus metrics: https://snowflake-broker.torproject.net/prometheus
and saw a bunch of very unusual snowflake proxy types:
```
snowflake_rounded_proxy_poll_total{nat="restricted'\"\\(",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="restricted'/**/and(select'1'from/**/pg_sleep(0))>'0",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="restricted'/**/and(select'1'from/**/pg_sleep(18))>'0",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="restricted'/**/and/**/DBMS_PIPE.RECEIVE_MESSAGE('p',18)='p",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="restricted'/**/and/**/DBMS_PIPE.RECEIVE_MESSAGE('q',0)='q",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="restricted'and(select'1'from/**/cast(md5(1634804485)as/**/int))>'0",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="restricted'and(select*from(select+sleep(0))a/**/union/**/select+1)='",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="restricted'and(select*from(select+sleep(18))a/**/union/**/select+1)='",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="restricted'and(select+1)>0waitfor/**/delay'0:0:0",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="restricted'and(select+1)>0waitfor/**/delay'0:0:18",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="restricted'and/**/convert(int,sys.fn_sqlvarbasetostr(HashBytes('MD5','1720393988')))>'0",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="restricted'and/**/extractvalue(1,concat(char(126),md5(1199007671)))and'",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="restricted/**/and/**/cast(md5('1204196037')as/**/int)>0",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="restricted|expr 809667990 + 908363838 ",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="restricted鎈'\"\\(",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown",status="idle"} 2.9086904e+07
snowflake_rounded_proxy_poll_total{nat="unknown",status="matched"} 2.757224e+06
snowflake_rounded_proxy_poll_total{nat="unknown\nexpr 965825977 + 974216935\n",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="unknown\"and(select*from(select+sleep(0))a/**/union/**/select+1)=\"",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="unknown\"and(select*from(select+sleep(14))a/**/union/**/select+1)=\"",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown\"and/**/extractvalue(1,concat(char(126),md5(1145896139)))and\"",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="unknown$(expr 898219840 + 955106000)",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown&set /A 859877269+929132787",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'\"\\(",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'/**/and(select'1'from/**/pg_sleep(0))>'0",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'/**/and(select'1'from/**/pg_sleep(14))>'0",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'/**/and/**/DBMS_PIPE.RECEIVE_MESSAGE('f',0)='f",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'/**/and/**/DBMS_PIPE.RECEIVE_MESSAGE('u',14)='u",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'and(select'1'from/**/cast(md5(1946869959)as/**/int))>'0",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'and(select*from(select+sleep(0))a/**/union/**/select+1)='",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'and(select*from(select+sleep(14))a/**/union/**/select+1)='",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'and(select+1)>0waitfor/**/delay'0:0:0",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'and(select+1)>0waitfor/**/delay'0:0:14",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'and/**/convert(int,sys.fn_sqlvarbasetostr(HashBytes('MD5','1587856848')))>'0",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'and/**/extractvalue(1,concat(char(126),md5(1875029567)))and'",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="unknown/**/and/**/cast(md5('1500206875')as/**/int)>0",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown|expr 881080391 + 963149055 ",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown鎈'\"\\(",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unrestricted",status="idle"} 1.801232e+06
snowflake_rounded_proxy_poll_total{nat="unrestricted",status="matched"} 7.7092e+06
```
I'm keeping this confidential for now in case this is a security vulnerability being actively exploited.Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & TibetCecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40091Set up a Snowflake bridge staging server2022-04-06T05:00:23ZDavid Fifielddcf@torproject.orgSet up a Snowflake bridge staging serverThe goals of this issue are:
1. Share practical experience setting up a snowflake bridge.
2. Write a [Snowflake Bridge Installation Guide](https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Guides/Snowflake-Bridge-I...The goals of this issue are:
1. Share practical experience setting up a snowflake bridge.
2. Write a [Snowflake Bridge Installation Guide](https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Guides/Snowflake-Bridge-Installation-Guide), like the existing [Snowflake Broker Installation Guide](https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Guides/Snowflake-Broker-Installation-Guide).
3. Install an [experimental load balancing configuration](https://lists.torproject.org/pipermail/tor-relays/2022-January/020183.html) in an attempt to increase the capacity of the bridge.
If the load balancing configuration works, we can then apply it to the production bridge. (Or change DNS so as to swap production and staging.)
We talked about scaling the snowflake bridge and load balancing ideas at the [2022-01-06 team meeting](http://meetbot.debian.net/tor-meeting/2022/tor-meeting.2022-01-06-15.59.html).
<details>
<summary>Meeting notes</summary>
* Scaling Snowflake bridge
* Already increased from 4 to 8 CPUs
* https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40085
* We should profile snowflake-server to reduce its CPU use
* https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40086
* But dcf feels that the bottleneck is not the snowflake-server process (which is multithreaded and can use multiple CPU cores), but the tor process (which can only use 1 CPU core and is already constantly at 100%). See https://lists.torproject.org/pipermail/tor-relays/2021-December/020156.html
* It turns out that it is possible to run multiple instances of tor with the same identity keys (hence the same fingerprint), either on the same host or on different hosts: https://lists.torproject.org/pipermail/tor-relays/2021-December/020157.html. Multiple instances of tor is a way of scaling tor beyond 1 CPU core, with snowflake-server distributing traffic over the available instances: https://lists.torproject.org/pipermail/tor-relays/2021-December/020182.html
* With another shim (similar to moat-shim) you can keep ExtORPort metrics reporting in the load-balanced configuration:
* https://gitlab.torproject.org/dcf/extor-static-cookie
* https://lists.torproject.org/pipermail/tor-relays/2022-January/020183.html
* Though I'm not sure, if Metrics gets multiple descriptors per day with the same fingerprint, whether it will sum all of them, or only keep one of them
* dcf has a bridge running now (with obfs4proxy, not snowflake-server) with this extor-static-cookie + load balancing configuration
* https://metrics.torproject.org/rs.html#details/07B9C6D7BE9685E83FA8C7A4FEB34CAD6CB77503
* Though I have not tested Roger's caveat about DEFAULT_ONION_KEY_LIFETIME_DAYS yet https://lists.torproject.org/pipermail/tor-relays/2022-January/020196.html
* We could apply this same thing to the snowflake bridge. But it is kind of a big configuration change: we need to stop snowflake-server being managed by tor, and instead run it independently (i.e., using runit or systemd) and have it talk to the tor instances' static ExtORPorts through a load balancer. It would be best to do this on a staging server separate from production first.
* we tentatively plan to do this next tuesday, 2022-01-11. dcf will be in touch and have a host ready for installation.
* Once the tor bottleneck is removed, though, we are not too far from the next likely bottleneck, which is total CPU of the host, shared between tor and snowflake-server. It might be time to think about migrating to different hosting for the snowflake bridge. Or we can experiment with running snowflake-server on one host, and N instances of tor on another (preferably nearby) host.
</details>Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & TibetDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40094Disable standalone proxies on bridge2022-01-19T07:40:48ZDavid Fifielddcf@torproject.orgDisable standalone proxies on bridgeWe want to increase the tor capacity of the snowflake bridge. We should disable the two standalone proxies we run on that host, in order to open up some more CPU headroom. Cf. #40091.We want to increase the tor capacity of the snowflake bridge. We should disable the two standalone proxies we run on that host, in order to open up some more CPU headroom. Cf. #40091.Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & TibetDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40143update debian package2022-06-07T08:00:21Zmeskiomeskio@torproject.orgupdate debian packageThere are several security issues in pion that requires updating the package.There are several security issues in pion that requires updating the package.Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetmeskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40193Snowflake Broker Deployment 22-10-032022-10-25T16:32:20ZshelikhooSnowflake Broker Deployment 22-10-03We are going to deploy a new version of snowflake broker configuration to snowflake broker.
The broker binary isn't updated, and remain [v2.3.1](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/tags/v2.3...We are going to deploy a new version of snowflake broker configuration to snowflake broker.
The broker binary isn't updated, and remain [v2.3.1](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/tags/v2.3.1).
## Deployment Script
```
sv stop snowflake-broker
cp /etc/service/snowflake-broker/run ./snowflake-broker-run-22-10-03-backup-$(date +%N)
install --owner root ./snowflake-broker-run-22-10-03-candidcate /etc/service/snowflake-broker/run
sv start snowflake-broker
```
## Rollback Script(will be located at /home/shelikhoo/deployment-22-10-03)
```
sv stop snowflake-broker
install --owner root ./snowflake-broker-run-22-10-03-backup-???? /etc/service/snowflake-broker/run
sv start snowflake-broker
```
## New Run File
(the difference is at --allowed-relay-pattern)
(-ip-count-mask's value is not real value used on the production system)
```
#!/bin/sh -e
setcap 'cap_net_bind_service=+ep' /usr/local/bin/broker
export GOMAXPROCS=1
exec chpst -u snowflake-broker -o 32768 /usr/local/bin/broker --metrics-log /home/snowflake-broker/metrics.log --acme-hostnames snowflake-broker.bamsoftware.com,snowflake-broker.freehaven.net,snowflake-broker.torproject.net --acme-email dcf@torproject.org --acme-cert-cache /home/snowflake-broker/acme-cert-cache --bridge-list-path /home/snowflake-broker/bridge_lists.json --default-relay-pattern ^snowflake.torproject.net$ --allowed-relay-pattern snowflake.torproject.net$ -ip-count-log /home/snowflake-broker/metrics-ip-salted.jsonl -ip-count-interval 1h -ip-count-mask ****** 2>&1
```
## Old Run File
```
#!/bin/sh -e
setcap 'cap_net_bind_service=+ep' /usr/local/bin/broker
export GOMAXPROCS=1
exec chpst -u snowflake-broker -o 32768 /usr/local/bin/broker --metrics-log /home/snowflake-broker/metrics.log --acme-hostnames snowflake-broker.bamsoftware.com,snowflake-broker.freehaven.net,snowflake-broker.torproject.net --acme-email dcf@torproject.org --acme-cert-cache /home/snowflake-broker/acme-cert-cache --bridge-list-path /home/snowflake-broker/bridge_lists.json --default-relay-pattern ^snowflake.torproject.net$ --allowed-relay-pattern ^snowflake.torproject.net$ -ip-count-log /home/snowflake-broker/metrics-ip-salted.jsonl -ip-count-interval 1h -ip-count-mask ****** 2>&1
```
## Side effect to be watched
The network capacity of the snowflake may be decreased. However, if we can take this hit, we should be able to roll out distributed snowflake support.Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetshelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40225Snowflake Broker Deployment 22-10-252022-11-08T18:21:09ZshelikhooSnowflake Broker Deployment 22-10-25We are going to deploy a new version of snowflake broker configuration to snowflake broker.
The broker binary isn't updated, and remain [v2.3.1](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/tags/v2.3...We are going to deploy a new version of snowflake broker configuration to snowflake broker.
The broker binary isn't updated, and remain [v2.3.1](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/tags/v2.3.1).
This will rollout the change we did in [Snowflake Broker Deployment 22-10-03](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40193) plus [include](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40212) [secondary bridge](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40122) definition, and [Remove GOMAXPROCS=1](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40205).
## Deployment Script
```
sv stop snowflake-broker
cp /etc/service/snowflake-broker/run ./snowflake-broker-run-22-10-25-backup-$(date +%N)
cp /home/snowflake-broker/bridge_lists.json ./bridge_lists_json-22-10-25-backup-$(date +%N)
install --owner root ./snowflake-broker-run-22-10-25-candidcate /etc/service/snowflake-broker/run
install --owner root ./bridge_lists_json-22-10-25-candidcate /home/snowflake-broker/bridge_lists.json
sv start snowflake-broker
```
## New Run File
(the difference is at --allowed-relay-pattern)
(-ip-count-mask's value is not real value used on the production system)
```
#!/bin/sh -e
setcap 'cap_net_bind_service=+ep' /usr/local/bin/broker
exec chpst -u snowflake-broker -o 32768 /usr/local/bin/broker --metrics-log /home/snowflake-broker/metrics.log --acme-hostnames snowflake-broker.bamsoftware.com,snowflake-broker.freehaven.net,snowflake-broker.torproject.net --acme-email dcf@torproject.org --acme-cert-cache /home/snowflake-broker/acme-cert-cache --bridge-list-path /home/snowflake-broker/bridge_lists.json --default-relay-pattern ^snowflake.torproject.net$ --allowed-relay-pattern snowflake.torproject.net$ -ip-count-log /home/snowflake-broker/metrics-ip-salted.jsonl -ip-count-interval 1h -ip-count-mask ****** 2>&1
```
## Old Run File
```
#!/bin/sh -e
setcap 'cap_net_bind_service=+ep' /usr/local/bin/broker
export GOMAXPROCS=1
exec chpst -u snowflake-broker -o 32768 /usr/local/bin/broker --metrics-log /home/snowflake-broker/metrics.log --acme-hostnames snowflake-broker.bamsoftware.com,snowflake-broker.freehaven.net,snowflake-broker.torproject.net --acme-email dcf@torproject.org --acme-cert-cache /home/snowflake-broker/acme-cert-cache --bridge-list-path /home/snowflake-broker/bridge_lists.json --default-relay-pattern ^snowflake.torproject.net$ --allowed-relay-pattern ^snowflake.torproject.net$ -ip-count-log /home/snowflake-broker/metrics-ip-salted.jsonl -ip-count-interval 1h -ip-count-mask ****** 2>&1
```
## New bridge_lists.json
```
{"displayName":"default", "webSocketAddress":"wss://snowflake.torproject.net/", "fingerprint":"2B280B23E1107BB62ABFC40DDCC8824814F80A72"}
{"displayName":"Bridge02", "webSocketAddress":"wss://02.snowflake.torproject.net/", "fingerprint":"8838024498816A039FCBBAB14E6F40A0843051FA"}
```
## Old bridge_lists.json
```
{"displayName":"default", "webSocketAddress":"wss://snowflake.torproject.net/", "fingerprint":"2B280B23E1107BB62ABFC40DDCC8824814F80A72"}
```
## Side effect to be watched
The network capacity of the snowflake may be decreased(again).Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetshelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40278Snowflake failed to connect on Android 11 and above2024-01-10T16:06:45ZGedshSnowflake failed to connect on Android 11 and aboveThis issue causes Snowflake to be unable to establish a connection on Android 11 and above when the app's target SDK level is above 29.
It is related to the Go x/mobile issue https://github.com/golang/go/issues/40569
The problem is loc...This issue causes Snowflake to be unable to establish a connection on Android 11 and above when the app's target SDK level is above 29.
It is related to the Go x/mobile issue https://github.com/golang/go/issues/40569
The problem is located in the Pion library https://github.com/pion/ice/blob/214c86c5cfbc40fa9adbbea59081bd64c8b65d7c/agent.go#L327 which calls https://pkg.go.dev/github.com/mingyech/transport/v2/stdnet#Net.UpdateInterfaces which is forbidden on Android 11.
Snowflake log:
```go
2023/07/09 17:40:03 snowflake-client 2.5.1
2023/07/09 17:40:03 Started SOCKS listener at [scrubbed].
2023/07/09 17:40:17 SOCKS accepted: {[scrubbed] utls-imitate=hellorandomizedalpn map[utls-imitate:[hellorandomizedalpn]]}
2023/07/09 17:40:17
--- Starting Snowflake Client ---
2023/07/09 17:40:17 Using ICE servers:
2023/07/09 17:40:17 url: stun:stun.bluesip.net:3478
2023/07/09 17:40:17 url: stun:stun.epygi.com:3478
2023/07/09 17:40:17 url: stun:stun.stunprotocol.org:3478
2023/07/09 17:40:17 url: stun:stun.voys.nl:3478
2023/07/09 17:40:17 url: stun:stun.uls.co.za:3478
2023/07/09 17:40:17 url: stun:stun.sonetel.net:3478
2023/07/09 17:40:17 url: stun:stun.antisip.com:3478
2023/07/09 17:40:17 Rendezvous using Broker at: https://snowflake-broker.torproject.net.global.prod.fastly.net/
2023/07/09 17:40:17 Domain fronting using: cdn.sstatic.net
2023/07/09 17:40:17 ---- SnowflakeConn: begin collecting snowflakes ---
2023/07/09 17:40:17 ---- SnowflakeConn: starting a new session ---
2023/07/09 17:40:17 ---- SnowflakeConn: begin stream 3 ---
2023/07/09 17:40:17 redialing on same connection
2023/07/09 17:40:17 WebRTC: Collecting a new Snowflake. Currently at [0/1]
2023/07/09 17:40:17 snowflake-421fa3a84195d6e9 connecting...
2023/07/09 17:40:17 WebRTC: DataChannel created
2023/07/09 17:40:17 Failed to prepare offer failed to create network: route ip+net: netlinkrib: permission denied
2023/07/09 17:40:17 WebRTC: closing DataChannel
2023/07/09 17:40:17 WebRTC: closing PeerConnection
2023/07/09 17:40:17 WebRTC: Closing
2023/07/09 17:40:17 WebRTC: failed to create network: route ip+net: netlinkrib: permission denied Retrying...
2023/07/09 17:40:17 NAT Type: unrestricted
```Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetshelikhooshelikhoo2023-08-31