Blocking of Snowflake in Turkmenistan, 2021-10-24

On 2021-10-24, the number of Snowflake users in Turkmenistan dropped from 20–30 to almost zero:

Previously discussed at:

• tpo/community/support#40030 (comment 2759213)
• http://meetbot.debian.net/tor-meeting/2021/tor-meeting.2021-11-04-15.59.log.html#l-55

   16:19:56 <anadahz> Confused about the meek client metrics in Turkmenistan -- https://metrics.torproject.org/userstats-bridge-combined.png?start=2021-08-02&end=2021-11-04&country=tm
   16:20:39 <anadahz> How come and there are so many meek clients in Turkmenistan?
   16:20:54 <dcf1> Here is a graph with some more context
   16:20:56 <dcf1> https://people.torproject.org/~dcf/metrics-country.html?start=2021-08-01&end=2021-11-05&country=tm
   16:21:00 <meskio> does it look like related to snowflake going down?
   16:21:13 <dcf1> however zoom out a bit to get even *more* context (esp. wrt relay users)
   16:21:15 <dcf1> https://people.torproject.org/~dcf/metrics-country.html?start=2021-07-01&end=2021-11-05&country=tm
   16:21:23 <cohosh> related info on tor blocking in TM: https://gitlab.torproject.org/tpo/community/support/-/issues/40030
   16:22:10 <dcf1> to me it looks like OR and meek were rising simultaneously, then snowflake and OR got blocked.
   16:22:33 <cohosh> wow
   16:22:49 <meskio> blocked? or our failure with probetest?
   16:23:45 <dcf1> but snowflake users globally did not go to zero in the same way https://metrics.torproject.org/userstats-bridge-transport.html?start=2021-08-06&end=2021-11-04&transport=snowflake
   16:24:05 <anadahz> On 2021-10-31 the amount of meek clients count were almost spike to 1,5 times than before.
   16:24:05 <meskio> I see what you mean :(
   16:24:09 <cohosh> yeah this looks suspiciously close to zero

Using the bidirectional nature of blocking in Turkmenistan, we can see that the block on Snowflake is effected by (at least) DNS and SNI blocking of the broker's front domain, cdn.sstatic.net.

$ dig +short telecom.tm
95.85.120.6
$ TARGET=95.85.120.6

$ dig @$TARGET +noedns +short +timeout=5 cdn.sstatic.net
127.0.0.1
$ curl --connect-to ::$TARGET: --connect-timeout 5 https://cdn.sstatic.net/ -D -
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to cdn.sstatic.net:443 

But www.google.com does not appear to be blocked (which, in the case of HTTPS, is a change from a Hyperquack measurement in August 2021), so it's possible that AMP cache rendezvous could work.

$ dig @$TARGET +noedns +short +timeout=5 www.google.com
;; connection timed out; no servers could be reached
$ curl --connect-to ::$TARGET: --connect-timeout 5 https://www.google.com/ -D -
curl: (60) SSL: no alternative certificate subject name matches target host name 'www.google.com'

AFAIK we use the same SNI for moat so I assume moat is also affected by this block. I'm curious how the bridge usage hasn't gone down, maybe because they only need the domain fronting to work to refresh the bridges and most users already have working bridges.

Maybe is time to rotate the SNI we use in both places (tpo/anti-censorship/pluggable-transports/snowflake#40068).

Will be nice to have a new SNI for the next TB release.

marked this issue as related to tpo/anti-censorship/pluggable-transports/snowflake#40068

mentioned in issue tpo/anti-censorship/pluggable-transports/snowflake#40068

Turkmenistan Snowflake Block Workaround Trial

Hi @gus :

Here are some snowflake bridge lines that uses an alternative fronting domain. These bridge lines instruct snowflake to use an alternative version of fronting domains that have the potential to bypass the censorship on the current fronting domain.

To use these bridges, go to about:preferences#tor check use a Bridge, select Provide a bridge (These terms may differ in local languages), and paste one of the following bridge lines below into it. This need to be done before connecting to the Tor network.

snowflake 192.0.2.3:1 url=https://snowflake-broker.torproject.net.global.prod.fastly.net/ front=fastly.jsdelivr.net ice=stun:stun.voip.blackberry.com:3478,stun:stun.altar.com.pl:3478,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:3478,stun:stun.epygi.com:3478,stun:stun.sonetel.com:3478,stun:stun.sonetel.net:3478,stun:stun.stunprotocol.org:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478

snowflake 192.0.2.3:1 url=https://snowflake-broker.torproject.net.global.prod.fastly.net/ front=b.stripecdn.com ice=stun:stun.voip.blackberry.com:3478,stun:stun.altar.com.pl:3478,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:3478,stun:stun.epygi.com:3478,stun:stun.sonetel.com:3478,stun:stun.sonetel.net:3478,stun:stun.stunprotocol.org:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478

snowflake 192.0.2.3:1 url=https://snowflake-broker.torproject.net.global.prod.fastly.net/ front=cdn.shopify.com ice=stun:stun.voip.blackberry.com:3478,stun:stun.altar.com.pl:3478,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:3478,stun:stun.epygi.com:3478,stun:stun.sonetel.com:3478,stun:stun.sonetel.net:3478,stun:stun.stunprotocol.org:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478

snowflake 192.0.2.3:1 url=https://snowflake-broker.torproject.net.global.prod.fastly.net/ front=foursquare.com ice=stun:stun.voip.blackberry.com:3478,stun:stun.altar.com.pl:3478,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:3478,stun:stun.epygi.com:3478,stun:stun.sonetel.com:3478,stun:stun.sonetel.net:3478,stun:stun.stunprotocol.org:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478

It would be helpful for us to inform users living in Turkmenistan about this workaround, and collect their feedback.

Shell

Hi Shell, I'm reaching out to some TM users.

Yes! Thanks for relaying this information. I will be waiting for the feedback!

First TM feedback: none of them worked. I will check with other users.

Thanks for letting me know.

This is an unexpected result and we currently don't have enough information to understand what has gone wrong. Once snowflake!67 is merged and released, some information about the reason for failure will be included in the Tor log. This should allow us to gain insight into failure.

Based on external testing of these four front domains, it seems that only foursquare.com is blocked, and it is blocked only by SNI, not DNS.

$ TARGET="$(dig +short telecom.tm)" # 95.85.120.6

$ dig @$TARGET +noedns +timeout=5 fastly.jsdelivr.net
;; connection timed out; no servers could be reached
$ dig @$TARGET +noedns +timeout=5 b.stripecdn.com
;; connection timed out; no servers could be reached
$ dig @$TARGET +noedns +timeout=5 cdn.shopify.com
;; connection timed out; no servers could be reached
$ dig @$TARGET +noedns +timeout=5 foursquare.com
;; connection timed out; no servers could be reached

$ curl --connect-to ::$TARGET: --connect-timeout 5 https://fastly.jsdelivr.net/
curl: (60) SSL: no alternative certificate subject name matches target host name 'fastly.jsdelivr.net'
$ curl --connect-to ::$TARGET: --connect-timeout 5 https://b.stripecdn.com/
curl: (60) SSL: no alternative certificate subject name matches target host name 'b.stripecdn.com'
$ curl --connect-to ::$TARGET: --connect-timeout 5 https://cdn.shopify.com/
curl: (60) SSL: no alternative certificate subject name matches target host name 'cdn.shopify.com'
$ curl --connect-to ::$TARGET: --connect-timeout 5 https://foursquare.com/
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to foursquare.com:443 

added Ongoing label

assigned to @shelikhoo

mentioned in issue tpo/anti-censorship/pluggable-transports/snowflake#40066 (closed)

marked this issue as related to tpo/anti-censorship/pluggable-transports/snowflake#40066 (closed)

What is the status here? We think Snowflake is currently broken in .tm, because they block our front domain, and we've explored some other front domains that ought to work, but they didn't work in a real-world test, and we're not sure why?

That matches my understanding.

I asked a user who reported blocking for help testing Snowflake, but did not get a reply. https://ntc.party/t/dpi/1518/3

I asked the user form Turkmenistan to check if the snowflake bridges would help to connect. The user tried them and reported back that the bridges did not work for them

There are some new (April 2022) OONI tests results here: https://explorer.ooni.org/search?since=2022-03-12&until=2022-04-12&failure=false&probe_cc=TM&test_name=torsf

@nina if you are in contact with someone from Turkmenistan, it would be helpful to ask them to try the AMP cache rendezvous. The normal domain-fronting rendezvous does not work in Turkmenistan because the front domain, cdn.sstatic.net, is blocked by DNS injection, which can be tested from outside the country (more info):

$ dig @telecom.tm +short +noedns cdn.sstatic.net
127.0.0.1

Thank you @dcf for reminding about this instruction. I've sent it to the user. Let's see, may be we will get some feedback

mentioned in issue team#69 (closed)

marked this issue as related to #40029 (closed)

removed the relation with tpo/anti-censorship/pluggable-transports/snowflake#40066 (closed)

added S96-maybe-O3 label

changed milestone to %Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibet

added Sponsor 96 label

added Q2 label

@gus Let's talk about this in our s96 coordination meeting to see how to move forward.

@kerosene this one is other ticket we could get help with.

I have received feedback from a V2Ray user living in Turkmenistan that Cloudflare(a CDN network)'s IP address space is now being partially blocked(some of Cloudflare's IP address is inaccessible, based on https://github.com/TulvL/cloudflare-ip-tester ). The hypothesis is that fastly is now blocked by IP in a similar way, which could explain our test result and user feedback. We can:

• Request a user to test if the fronted website is still accessible
• Setup a vantage point in Turkmenistan
• Use idle scan(https://en.wikipedia.org/wiki/Idle_scan) to try to test the network connectivity status between Turkmenistan devices and CDN networks. (This is a hostile network activity and should proceed with caution)

Here is the screenshot from user:

added Q3 label and removed Q2 label

It seems that meek-azure is working quite well as TM gov is using Skype (and Gmail) for their internal comms. @shelikhoo would it be possible to use meek-azure as domain fronting for the snowflake broker?

Technologically, yes. There is nothing preventing us from using the azure as fronting. I will discuss this option with other team members.

mentioned in issue #40029 (closed)

mentioned in merge request rdsys!48 (merged)

8/18/22, 17:39:56.434 [NOTICE] DisableNetwork is set. Tor will not make or accept non-control network connections. Shutting down all existing connections.
8/18/22, 17:39:56.458 [NOTICE] Opening Socks listener on 127.0.0.1:9150
8/18/22, 17:39:56.458 [NOTICE] Opened Socks listener connection (ready) on 127.0.0.1:9150
8/18/22, 17:39:57.443 [NOTICE] Bootstrapped 1% (conn_pt): Connecting to pluggable transport
8/18/22, 17:39:57.444 [NOTICE] Bootstrapped 2% (conn_done_pt): Connected to pluggable transport
8/18/22, 17:39:57.452 [NOTICE] Bootstrapped 10% (conn_done): Connected to a relay
8/18/22, 17:40:03.430 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 17:40:03.433 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker failure read tcp 192.168.1.101:60570->151.101.129.69:443: wsarecv: An existing connection was forcibly closed by the remote host.
8/18/22, 17:40:13.438 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 17:40:13.438 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker failure read tcp 192.168.1.101:60573->151.101.129.69:443: wsarecv: An existing connection was forcibly closed by the remote host.

Snowflake used azure as the default domain front until just over a year ago. You can see when we made the change from the commit where we updated the torrc file.

We switched to fastly shortly after microsoft released a statement saying they would start actively blocking domain fronted connections through microsoft azure but the azure account we used for Snowflake is still up and working.

The following bridge line should just work in Tor Browser:

Bridge snowflake 192.0.2.3:1 2B280B23E1107BB62ABFC40DDCC8824814F80A72 fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://snowflake-broker.azureedge.net/ front=ajax.aspnetcdn.com ice=stun:stun.voip.blackberry.com:3478,stun:stun.altar.com.pl:3478,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:3478,stun:stun.epygi.com:3478,stun:stun.sonetel.com:3478,stun:stun.sonetel.net:3478,stun:stun.stunprotocol.org:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478

I've just confirmed that this does work. So as a quick workaround, before rdsys!48 (merged) gets merged, users should be able to use this as a custom bridge.

8/18/22, 18:43:20.223 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 18:43:25.828 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 18:43:25.829 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 18:43:35.832 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 18:43:40.933 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 18:43:41.836 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 18:43:51.840 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 18:43:56.842 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 18:43:56.843 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 18:44:06.848 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 18:44:11.932 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 18:44:12.852 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 18:44:22.845 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 18:44:27.858 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 18:44:27.859 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 18:44:37.845 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 18:44:43.266 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 18:44:43.266 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 18:44:53.272 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 18:44:58.873 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 18:44:58.933 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 18:45:08.933 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 18:45:14.283 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 18:45:14.882 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 18:45:22.887 [WARN] Problem bootstrapping. Stuck at 10% (conn_done): Connected to a relay. (DONE; DONE; count 2; recommendation warn; host 2B280B23E1107BB62ABFC40DDCC8824814F80A72 at 192.0.2.3:1)
8/18/22, 18:45:22.888 [WARN] 2 connections have failed:
8/18/22, 18:45:22.888 [WARN] 2 connections died in state handshaking (TLS) with SSL state SSLv3/TLS write client hello in HANDSHAKE
8/18/22, 18:45:22.896 [NOTICE] Closing no-longer-configured Socks listener on 127.0.0.1:9150
8/18/22, 18:45:22.897 [NOTICE] DisableNetwork is set. Tor will not make or accept non-control network connections. Shutting down all existing connections.
8/18/22, 18:45:23.580 [WARN] Pluggable Transport process terminated with status code 0

the user I asked to check have sent the same log. And port 3478 is blocked in TM

Shoot :( there's some progress in the sense that the broker connection is working now, but it looks like the client is having trouble connecting to the proxies.

And port 3478 is blocked in TM

This could be the reason why, if the client can't connect to the STUN servers, then it's possible the offers don't contain enough information to allow the connection to the proxies to happen. Will dig around to see if there's a STUN server that's not running on port 3478.

Do you know whether port 3479 is blocked? You can try this bridge line to check to see if it works:

Bridge snowflake 192.0.2.3:1 2B280B23E1107BB62ABFC40DDCC8824814F80A72 fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://snowflake-broker.azureedge.net/ front=ajax.aspnetcdn.com ice=stun:stun.voip.blackberry.com:3479,stun:stun.altar.com.pl:3479,stun:stun.antisip.com:3479,stun:stun.bluesip.net:3479,stun:stun.dus.net:3479,stun:stun.epygi.com:3479,stun:stun.sonetel.com:3479,stun:stun.sonetel.net:3479,stun:stun.stunprotocol.org:3479,stun:stun.uls.co.za:3479,stun:stun.voipgate.com:3479,stun:stun.voys.nl:3479

I did some testing myself, and confirmed that when the STUN servers are not reachable, Snowflake doesn't log an error and instead proceeds to send an offer SDP without any externally accessible candidates:

2022/08/18 19:37:10 {"type":"offer","sdp":"v=0\r\no=- 537061262799921275 1660851424 IN IP4 0.0.0.0\r
\ns=-\r\nt=0 0\r\na=fingerprint:sha-256 
9E:52:E8:E8:9E:4A:8D:5C:8F:98:71:B8:74:79:D3:9F:F3:B1:0B:70:C6:6F:FC:14:D9:FE:65:39:EB:B4:96:FB\r
\na=extmap-allow-mixed\r\na=group:BUNDLE 0\r\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 
0.0.0.0\r\na=setup:actpass\r\na=mid:0\r\na=sendrecv\r\na=sctp-port:5000\r\na=ice-ufrag:jraUcufTglGajELZ
\r\na=ice-pwd:tzjWlEcnShIlOdCoOiDgSMdefmWMXdIV\r\na=end-of-candidates\r\n"}

Whether or not the connection succeeds has to do with the client's NAT behaviour. Some NAT setups require both sides to have valid candidates, while others only require one side. My guess is that if the users are on mobile, we'll see the behaviour pasted in the logs above where the client keeps asking the broker for more proxies and then times out, failing to connect.

I did some digging in the old sources we used to find our current list of STUN servers as well as a new list and found a few with non standard ports. Here's another bridge line to try:

Bridge snowflake 192.0.2.3:1 2B280B23E1107BB62ABFC40DDCC8824814F80A72 fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://snowflake-broker.azureedge.net/ front=ajax.aspnetcdn.com ice=stun:stun.sipgate.net:10000,stun:stun.nextcloud.com:443,stun:stun1.l.google.com:19305

The Google stun server is likely to be blocked, but I'm hopeful about the nextcloud one. Hosting it on port 443 is clever.

@cohosh this line...

Bridge snowflake 192.0.2.3:1 2B280B23E1107BB62ABFC40DDCC8824814F80A72 fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://snowflake-broker.azureedge.net/ front=ajax.aspnetcdn.com                     
ice=stun:stun.voip.blackberry.com:3479,stun:stun.altar.com.pl:3479,stun:stun.antisip.com:3479,stun:stun.bluesip.net:3479,stun:stun.dus.net:3479,stun:stun.epygi.com:3479,stun:stun.sonetel.com:3479,stun:stun.sonetel.net:3479,stun:stun.stunprotocol.org:3479,stun:stun.uls.co.za:3479,stun:stun.voipgate.com:3479,stun:stun.voys.nl:3479 

Works!

8/18/22, 19:45:35.955 [NOTICE] New control connection opened from 127.0.0.1.
8/18/22, 19:45:35.956 [NOTICE] New control connection opened from 127.0.0.1.
8/18/22, 19:45:35.994 [NOTICE] Opening Socks listener on 127.0.0.1:9150
8/18/22, 19:45:35.994 [NOTICE] Opened Socks listener connection (ready) on 127.0.0.1:9150
8/18/22, 19:45:36.241 [NOTICE] Bootstrapped 1% (conn_pt): Connecting to pluggable transport
8/18/22, 19:45:36.243 [NOTICE] Bootstrapped 2% (conn_done_pt): Connected to pluggable transport
8/18/22, 19:45:36.292 [NOTICE] Bootstrapped 10% (conn_done): Connected to a relay
8/18/22, 19:45:42.870 [NOTICE] Bootstrapped 14% (handshake): Handshaking with a relay
8/18/22, 19:45:48.238 [NOTICE] Bootstrapped 15% (handshake_done): Handshake with a relay done
8/18/22, 19:45:48.239 [NOTICE] Bootstrapped 75% (enough_dirinfo): Loaded enough directory info to build circuits
8/18/22, 19:45:48.240 [NOTICE] Bootstrapped 95% (circuit_create): Establishing a Tor circuit
8/18/22, 19:45:59.547 [NOTICE] Guard $97700DFE9F483596DDA6264C4D7DF7641E1E39CE ($97700DFE9F483596DDA6264C4D7DF7641E1E39CE) is failing more circuits than usual. Most likely this means the Tor network is overloaded. Success counts are 160/230. Use counts are 86/95. 209 circuits completed, 7 were unusable, 25 collapsed, and 30 timed out. For reference, your timeout cutoff is 60 seconds.
8/18/22, 19:46:06.269 [NOTICE] Bootstrapped 100% (done): Done
8/18/22, 19:46:06.395 [NOTICE] New control connection opened from 127.0.0.1.

hmmm... The user I asked to check reported as non-working:

8/18/22, 20:20:33.512 [NOTICE] DisableNetwork is set. Tor will not make or accept non-control network connections. Shutting down all existing connections.
8/18/22, 20:20:39.750 [NOTICE] DisableNetwork is set. Tor will not make or accept non-control network connections. Shutting down all existing connections.
8/18/22, 20:20:41.455 [NOTICE] Opening Socks listener on 127.0.0.1:9150
8/18/22, 20:20:41.455 [NOTICE] Opened Socks listener connection (ready) on 127.0.0.1:9150
8/18/22, 20:20:41.589 [NOTICE] Bootstrapped 1% (conn_pt): Connecting to pluggable transport
8/18/22, 20:20:41.590 [NOTICE] Bootstrapped 2% (conn_done_pt): Connected to pluggable transport
8/18/22, 20:20:41.592 [NOTICE] Bootstrapped 10% (conn_done): Connected to a relay
8/18/22, 20:20:47.463 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:20:48.455 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:20:57.732 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:21:02.776 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:21:03.450 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:21:13.456 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:21:18.455 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:21:23.845 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:21:33.941 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:21:38.985 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:21:40.100 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:21:50.930 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:21:55.135 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:21:55.274 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received

@nina what about the bridge line here: #40024 (comment 2829155)

the same:

8/18/22, 20:31:21.474 [NOTICE] DisableNetwork is set. Tor will not make or accept non-control network connections. Shutting down all existing connections.
8/18/22, 20:31:28.234 [NOTICE] DisableNetwork is set. Tor will not make or accept non-control network connections. Shutting down all existing connections.
8/18/22, 20:31:28.234 [NOTICE] Switching to guard context "bridges" (was using "default")
8/18/22, 20:31:30.993 [NOTICE] Opening Socks listener on 127.0.0.1:9150
8/18/22, 20:31:30.993 [NOTICE] Opened Socks listener connection (ready) on 127.0.0.1:9150
8/18/22, 20:31:31.607 [NOTICE] Bootstrapped 1% (conn_pt): Connecting to pluggable transport
8/18/22, 20:31:31.608 [NOTICE] Bootstrapped 2% (conn_done_pt): Connected to pluggable transport
8/18/22, 20:31:31.609 [NOTICE] Bootstrapped 10% (conn_done): Connected to a relay
8/18/22, 20:31:36.989 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:31:37.639 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:31:47.730 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:31:52.769 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:31:57.819 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:32:07.934 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:32:12.979 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:32:13.446 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:32:23.710 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:32:28.135 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:32:28.556 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:32:38.670 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:32:43.731 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:32:43.996 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:32:53.995 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:32:59.000 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:32:59.483 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:33:09.592 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:33:14.637 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:33:15.651 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:33:25.722 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:33:30.769 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:33:30.770 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received

And this line:

Bridge snowflake 192.0.2.3:1 2B280B23E1107BB62ABFC40DDCC8824814F80A72 fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://snowflake-broker.azureedge.net/ front=ajax.aspnetcdn.com ice=stun:stun.sipgate.net:10000,stun:stun.nextcloud.com:443,stun:stun1.l.google.com:19305

doesn't work:

8/18/22, 20:46:27.500 [NOTICE] Switching to guard context "bridges" (was using "default")
8/18/22, 20:46:32.350 [NOTICE] Opening Socks listener on 127.0.0.1:9150
8/18/22, 20:46:32.350 [NOTICE] Opened Socks listener connection (ready) on 127.0.0.1:9150
8/18/22, 20:46:32.846 [NOTICE] Bootstrapped 10% (conn_done): Connected to a relay
8/18/22, 20:46:38.588 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:46:38.954 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:46:49.589 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:46:54.589 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:46:54.590 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:47:04.588 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:47:09.965 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:47:10.588 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:47:20.456 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:47:25.589 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:47:25.590 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:47:35.588 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:47:40.734 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:47:40.980 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:47:50.985 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:47:56.589 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:47:56.589 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:48:06.588 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:48:11.589 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:48:11.998 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:48:22.200 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:48:27.588 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:48:27.589 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:48:37.588 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:48:42.690 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:48:48.160 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:48:58.220 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:49:03.588 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:49:03.700 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:49:13.706 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:49:19.370 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:49:19.587 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:49:29.589 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:49:34.588 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:49:34.588 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:49:44.588 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:49:50.520 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:49:51.588 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:50:01.588 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:50:06.587 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:50:06.768 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:50:16.775 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:50:22.710 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:50:22.587 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:50:32.588 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:50:37.783 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:50:38.588 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:50:48.590 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:50:53.588 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:50:53.791 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:51:03.796 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:51:08.861 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:51:09.960 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:51:19.100 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/18/22, 20:51:24.588 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/18/22, 20:51:25.104 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/18/22, 20:51:32.876 [WARN] Problem bootstrapping. Stuck at 10% (conn_done): Connected to a relay. (DONE; DONE; count 13; recommendation warn; host 2B280B23E1107BB62ABFC40DDCC8824814F80A72 at 192.0.2.3:1)
8/18/22, 20:51:32.876 [WARN] 18 connections have failed:

Okay

So the line where we switch to port 3479 only works for some people but not others?

@cohosh Yes,I've just got a report from another user who successfully connected with 3479 port bridge

@nina can you collect on which ISPs the snowflake line isn't working?

In AGTS snowflake seems to work.

@gus worked for AGTS

So it sounds like the bridge line in #40024 (comment 2829141) works in AGTS (ASN 51495)

Here is another bridge line to try in ISPs where the above bridge line is not working:

Bridge snowflake 192.0.2.3:1 2B280B23E1107BB62ABFC40DDCC8824814F80A72 fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://snowflake-broker.azureedge.net/ front=ajax.aspnetcdn.com ice=stun:206.53.159.130:3479,stun:176.119.42.11:3479,stun:94.23.17.185:3479,stun:217.74.179.29:3479,stun:83.125.8.47:3479,stun:23.253.102.137:3479,stun:52.26.251.34:3479,stun:52.26.251.34:3479,stun:18.191.223.12:3479,stun:154.73.34.8:3479,stun:185.125.180.70:3479,stun:195.35.115.37:3479

this one uses the same STUN servers as #40024 (comment 2829141), but replaces the domain name with the IPv4 address. In case the censor is filtering DNS requests for domains with the key word "stun".

That bridges worked well for AGTS, but did not work for Tukmentelecom. I asked several other users to check it too. And the logs:

8/23/22, 10:33:33.978 [NOTICE] Opening Socks listener on 127.0.0.1:9150
8/23/22, 10:33:33.978 [NOTICE] Opened Socks listener connection (ready) on 127.0.0.1:9150
8/23/22, 10:33:34.174 [NOTICE] Closing no-longer-configured Socks listener on 127.0.0.1:9150
8/23/22, 10:33:34.175 [NOTICE] DisableNetwork is set. Tor will not make or accept non-control network connections. Shutting down all existing connections.
8/23/22, 10:33:34.951 [WARN] Pluggable Transport process terminated with status code 0
8/23/22, 10:33:38.276 [NOTICE] Opening Socks listener on 127.0.0.1:9150
8/23/22, 10:33:38.278 [NOTICE] Opened Socks listener connection (ready) on 127.0.0.1:9150
8/23/22, 10:33:38.971 [NOTICE] Bootstrapped 1% (conn_pt): Connecting to pluggable transport
8/23/22, 10:33:38.973 [NOTICE] Bootstrapped 2% (conn_done_pt): Connected to pluggable transport
8/23/22, 10:33:38.981 [NOTICE] Bootstrapped 10% (conn_done): Connected to a relay
8/23/22, 10:33:43.430 [NOTICE] New control connection opened from 127.0.0.1.
8/23/22, 10:33:44.263 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/23/22, 10:33:49.951 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/23/22, 10:33:59.952 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen
8/23/22, 10:34:04.952 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
8/23/22, 10:34:05.269 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
8/23/22, 10:34:15.272 [NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": connection failed timeout waiting for DataChannel.OnOpen 

got a feedback form the user that the bridge below works faster than previous one from #40024 (comment 2829141)

Bridge snowflake 192.0.2.3:1 2B280B23E1107BB62ABFC40DDCC8824814F80A72 fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://snowflake-broker.azureedge.net/ front=ajax.aspnetcdn.com ice=stun:206.53.159.130:3479,stun:176.119.42.11:3479,stun:94.23.17.185:3479,stun:217.74.179.29:3479,stun:83.125.8.47:3479,stun:23.253.102.137:3479,stun:52.26.251.34:3479,stun:52.26.251.34:3479,stun:18.191.223.12:3479,stun:154.73.34.8:3479,stun:185.125.180.70:3479,stun:195.35.115.37:3479

This is now the bridgeline that will be autoconfigured by Connect Assist in TM.

For your information, as of 7 January 2023, none of the servers listed above work on Turkmentelecom. Tested from 95.85.105.0/24.

root@tmvm:~# SRVS="206.53.159.130:3479
> 176.119.42.11:3479
> 94.23.17.185:3479
> 217.74.179.29:3479
> 83.125.8.47:3479
> 23.253.102.137:3479
> 52.26.251.34:3479
> 52.26.251.34:3479
> 18.191.223.12:3479
> 154.73.34.8:3479
> 185.125.180.70:3479
> 195.35.115.37:3479"
root@tmvm:~# 
root@tmvm:~# for i in $SRVS; do s="${i%:*}"; p="${i#*:}"; echo testing $s $p; (echo -n 00010008014b142afe8bf45ada430e0a81863a100003000400000000 | xxd -ps -r; sleep 5) | timeout 5 nc -p 3467 -u $s $p; done
testing 206.53.159.130 3479
testing 176.119.42.11 3479
testing 94.23.17.185 3479
testing 217.74.179.29 3479
testing 83.125.8.47 3479
testing 23.253.102.137 3479
testing 52.26.251.34 3479
testing 52.26.251.34 3479
testing 18.191.223.12 3479
testing 154.73.34.8 3479
testing 185.125.180.70 3479
testing 195.35.115.37 3479

@gus @ValdikSS to follow up on the STUN blocking from earlier, I just set up a STUN server on ports 8080 and 8443 to test the theory that if Turkmentelecom and/or AGTS are blocking STUN ports, STUN messages may still go through if the server is running on unusual ports. You can use it with the following bridge line:

Bridge snowflake 192.0.2.3:80 2B280B23E1107BB62ABFC40DDCC8824814F80A72 fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://snowflake-broker.torproject.net.global.prod.fastly.net/ front=cdn.sstatic.net ice=stun:161.35.66.96:8080 utls-imitate=hellorandomizedalpn

I'm curious about whether this works in either AGTS or Turkmentelecom. I can also rotate these ports or try ports that may have potentially more collateral damage associated with them. If it works, perhaps we can attempt to set up more STUN servers on ports other than 3478 or 3479.

I'm afraid that finding non-blocked hosting provider would be a challenging task. The IP address you've provided, 161.35.66.96, is a Digital Ocean, and it's IP ranges, as well as many other hostings, are blocked.

[NOTICE] New control connection opened from 127.0.0.1.
[NOTICE] DisableNetwork is set. Tor will not make or accept non-control network connections. Shutting down all existing connections.
[NOTICE] Switching to guard context "bridges" (was using "default")
[NOTICE] Opening Socks listener on 127.0.0.1:9150
[NOTICE] Opened Socks listener connection (ready) on 127.0.0.1:9150
[NOTICE] Bootstrapped 1% (conn_pt): Connecting to pluggable transport
[NOTICE] Bootstrapped 2% (conn_done_pt): Connected to pluggable transport
[NOTICE] Bootstrapped 10% (conn_done): Connected to a relay
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": trying a new proxy: timeout waiting for DataChannel.OnOpen
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": trying a new proxy: timeout waiting for DataChannel.OnOpen
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": trying a new proxy: timeout waiting for DataChannel.OnOpen
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": trying a new proxy: timeout waiting for DataChannel.OnOpen
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": trying a new proxy: timeout waiting for DataChannel.OnOpen
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": trying a new proxy: timeout waiting for DataChannel.OnOpen
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker failure read tcp [scrubbed]->[scrubbed]: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": trying a new proxy: timeout waiting for DataChannel.OnOpen
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": trying a new proxy: timeout waiting for DataChannel.OnOpen
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": trying a new proxy: timeout waiting for DataChannel.OnOpen
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": trying a new proxy: timeout waiting for DataChannel.OnOpen
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": trying a new proxy: timeout waiting for DataChannel.OnOpen
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": trying a new proxy: timeout waiting for DataChannel.OnOpen
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": trying a new proxy: timeout waiting for DataChannel.OnOpen
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": trying a new proxy: timeout waiting for DataChannel.OnOpen
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": offer created
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": broker rendezvous peer received
[NOTICE] Managed proxy "TorBrowser\Tor\PluggableTransports\snowflake-client.exe": trying a new proxy: timeout waiting for DataChannel.OnOpen
[WARN] Problem bootstrapping. Stuck at 10% (conn_done): Connected to a relay. (DONE; DONE; count 1; recommendation warn; host 2B280B23E1107BB62ABFC40DDCC8824814F80A72 at 192.0.2.3:80)
[WARN] 1 connections have failed:
[WARN] 1 connections died in state handshaking (TLS) with SSL state SSLv3/TLS write client hello in HANDSHAKE
[NOTICE] Closing no-longer-configured Socks listener on 127.0.0.1:9150
[NOTICE] DisableNetwork is set. Tor will not make or accept non-control network connections. Shutting down all existing connections.
[WARN] Pluggable Transport process terminated with status code 0

Thanks for the update @ValdikSS and @gus, I missed this issue here tpo/community/relays#65 (closed) that states that pretty much only residential bridges are working.

There's something bothering me here. Even if the user is unable to connect to any STUN servers and sends an SDP offer with no candidates, they should still be able to connect to proxies that are not behind a NAT or behind a full cone NAT, regardless of what kind of NAT they have. The reason behind this is that if the proxy receives a binding request user message from the client, it will "learn" the client's candidate from the source address of the request and their binding request success response will contain an XOR-MAPPED-ADDRESS field with the client's candidate back to the client. So in general, the connection will look like this:

Client SDP: no candidates
Proxy SDP: candidate=1.2.3.4:5678

Proxy                                                    Client

  <--------- STUN binding request user ---------------------
1.2.3.4:5678                                            8.7.6.5:4321


  -----------------STUN binding request success ------------>
1.2.3.4:5678                                            8.7.6.5:4321


  ---------- STUN binding request user --------------------->
1.2.3.4:5678                                            8.7.6.5:4321


  <----------------STUN binding request success ------------
1.2.3.4:5678                                            8.7.6.5:4321


  <------------------------- DTLS -------------------------->
1.2.3.4:5678                                            8.7.6.5:4321

You can check this yourself by configuring a Tor bridge line without any ice servers:

Bridge snowflake 192.0.2.3:80 2B280B23E1107BB62ABFC40DDCC8824814F80A72 fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://snowflake-broker.torproject.net.global.prod.fastly.net/ front=cdn.sstatic.net utls-imitate=hellorandomizedalpn

You'll see a lot of datachannel timeouts like in the log shared above, because there are restricted cone NATs in the proxy pool for clients with symmetric or unknown NAT types. But the client should eventually find a proxy without a NAT or with a full cone NAT. I don't know what the ratio or NAT types are in the "unrestricted" proxy pool, but the fact that clients aren't ever able connect is suspicious.

I wonder if STUN is being blocked by protocol rather than port. If the binding request user messages aren't going through to each peer, then the webrtc connection won't be attempted. Other possibilities are that the proxies are being enumerated (unlikely, I think), or that DTLS is being blocked.

If it really is the case that clients just aren't getting full cone or un-NAT'd proxies, we could introduce another pool at the broker given preferentially to clients without any SDP candidates in their offer.

If STUN is being blocked by protocol, perhaps tpo/anti-censorship/pluggable-transports/snowflake#40240 (closed) would be a useful fix. If DTLS is being blocked, perhaps we can circumvent that blocking by adjusting the fingerprint or adding another obfuscation layer. If proxies are being enumerated, the fix will be a lot more complicated.

I forgot to add: most of the proxies without NATs or with full cone NATs are probably run on large hosting providers. So it could be the case that all of the proxies that should work, even if the client can't connect to STUN servers, don't because they are in IP ranges that are blocked wholesale.

So perhaps what we need are proxies on full cone NATs on home networks, but filtering these out to give to users in TM would be difficult.

I wonder if STUN is being blocked by protocol rather than port.

Doesn't seem so. I just set up port redirection from the unblocked VPS IP port 80 to a public STUN server, and tried to use it from AGTS and Turkmentelecom — it works.

I can't say for all of Turkmenistan users in different regions, but according to the probes in Ashgabat which I have, you can hardly call their network access as "an internet". Out of 6000+ Tor Relays, only about 50 are reachable, most of which are set up on residential connections in Europe or Canada, and sometimes you might see a relay on an (yet) not blocked hosting address.

And this is not a targeted Tor block, this is just how Turkmenistan blocks IP ranges. If you try typing innocent query to the search engine, like "how to cook", you'd be lucky if 3 search results out of 20 would open.

So to summarize:

Both AGTS and Turkmentelecom are blocking all of our STUN servers as well as most large hosting providers. We don't think they are blocking Snowflake by protocol anywhere, and we can successfully get a connection to the broker via domain fronting.

While STUN isn't necessary for a client to use Snowflake, in the case of TM it practically is because most of the Snowflakes that work for clients without candidates in their SDP are run on large hosting providers. So it's possible that if we found a way for clients to make a connection to a STUN server, that would "unlock" the Snowflake proxies that are being run on residential IP ranges.

@ValdikSS in your opinion, is it worth it to try and find a way to run a STUN server that clients in TM can get access to? If running it on most VPS providers doesn't work, what about the following:

• run a STUN server on a residential connection in US or CA
• run a STUN server from a hosting provider in TM or RU (this might be risky to the operator however)

I just set up port redirection from the unblocked VPS IP port 80 to a public STUN server, and tried to use it from AGTS and Turkmentelecom

Could this be a viable way forward? Or is this unblocked VPS just used for measurements and too risky to be used for censorship circumvention?

What I'm unsure about is whether this provides more value than the existing call to run obfs4 relays on residential connections. From what I understand, Snowflake will still be strictly worse in terms of reliability and performance than an obfs4 server run from a residential (unblocked) IP range.

mentioned in merge request rdsys-admin!4 (merged)

removed Q3 label

added Q4 label

removed milestone %Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibet

removed Sponsor 96 label

removed S96-maybe-O3 label

mentioned in issue team#96 (closed)

removed Q4 label

added Q1 label

added Backlog label

mentioned in issue tpo/anti-censorship/pluggable-transports/snowflake#40108 (closed)

mentioned in issue tpo/applications/tor-browser#40552 (closed)

mentioned in issue tpo/anti-censorship/pluggable-transports/snowflake#40257

mentioned in issue tpo/community/relays#61 (closed)

mentioned in issue tpo/anti-censorship/pluggable-transports/snowflake#40231 (closed)

mentioned in merge request tpo/anti-censorship/pluggable-transports/snowflake!141 (merged)

removed Q1 label

added Q2 label