Snowflake issueshttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues2022-12-20T20:57:52Zhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/25596Configure TURN servers for the proxy and/or client2022-12-20T20:57:52ZArlo BreaultConfigure TURN servers for the proxy and/or client> This may require spinning up our own TURN servers somewhere.
>
>> In most cases, the public ip candidates given by STUN servers are sufficient for peers to negotiate a working p2p path. But sometimes, maybe 14% of the time [1], this is...> This may require spinning up our own TURN servers somewhere.
>
>> In most cases, the public ip candidates given by STUN servers are sufficient for peers to negotiate a working p2p path. But sometimes, maybe 14% of the time [1], this isn't enough -- usually due to symmetric NAT / lack of port preservation, and a TURN relay is required as a last resort.
>>
>> Also, once snowflake proxies & clients are multiplexed, perhaps the likelihood that TURN continuous to be absolutely required for every pair of peers greatly decreases and we might be fine again. More thought is needed here.
\\ Migrated from https://github.com/keroserene/snowflake/issues/18https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/25595Test suite for Snowflake on various NAT topologies2023-08-01T23:45:54ZArlo BreaultTest suite for Snowflake on various NAT topologiesMigrated from https://github.com/keroserene/snowflake/issues/20Migrated from https://github.com/keroserene/snowflake/issues/20max-bmax-bhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/25594Broker: investigate non-domain-fronting secure client / proxy registrations2024-02-27T18:55:26ZArlo BreaultBroker: investigate non-domain-fronting secure client / proxy registrations> Pasting discussion from email,
>
>> in the Flashproxy case, registration wasn't
>> bidi, and I think they imagined using insecure
>> channels to register like OSSes. In Snowflake,
>> the client is making TLS connections with the
>> bro...> Pasting discussion from email,
>
>> in the Flashproxy case, registration wasn't
>> bidi, and I think they imagined using insecure
>> channels to register like OSSes. In Snowflake,
>> the client is making TLS connections with the
>> broker, which amounts to the same thing as
>> encrypting the payload with the facilitator's
>> public key.
>
> Also,
>
>> There's also the case where an adversary DOSes the facilitator with a
>> bunch of fake client or proxy registrations and things like that.
>
> This is now legacy/trac#25593
>
>> Also, there is the potential that in the future we might need some
>> sort of non-domain-fronting rendezvous. It seems that right now we
>> have an ecosystem of tools growing that assumes domain-fronting will
>> always be available & effective. May be worth considering how to
>> prepare for regions where this might not work as well in the future.
>
> So this ticket should probably be for that.
\\ Migrated from https://github.com/keroserene/snowflake/issues/13
Ideas that have been proposed or done:
* #25985 AMP cache
* #25874 DNS
* #26151 Amazon SQSDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/25593Broker needs better resilience against DoS2024-02-14T14:59:35ZArlo BreaultBroker needs better resilience against DoSMigrated from https://github.com/keroserene/snowflake/issues/25Migrated from https://github.com/keroserene/snowflake/issues/25Roger DingledineRoger Dingledinehttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/25592Consider webrtc<-->webrtc vs webrtc<-->websocket for the browser proxy2020-06-27T13:40:39ZArlo BreaultConsider webrtc<-->webrtc vs webrtc<-->websocket for the browser proxy> webrtc <-->websocket is the current situation, which works.
> webrtc <--> webrtc would take a bit more work.
\\
> Quoting @yawning,
>
> What are your plans for actually getting the server side to scale well? Since you're using cgo yo...> webrtc <-->websocket is the current situation, which works.
> webrtc <--> webrtc would take a bit more work.
\\
> Quoting @yawning,
>
> What are your plans for actually getting the server side to scale well? Since you're using cgo you will run into Really Interesting behavior wrt OS threads as you try to increase concurrency.
>
> https://lists.torproject.org/pipermail/tor-dev/2016-January/010311.html
\\
> Yes, that does complicate the potential of using a second WebRTC connection to the relay...
>
> The benefit of WebRTC on both sides should be performance / latency, whereas right now the websocket connection to the relay is most likely bottlenecking the first WebRTC data channel from the client.
>
> But maybe it's actually not, at least significantly. It's quite possible the typical latency of the network is already most of what the user experiences compared to the websocket relay, in which case the benefit of the 2nd WebRTC would be negligible with respect to the effort of making it happen.... maybe we need to measure this (I haven't found any obvious latency numbers on metrics.torproject.org), or we could just decide not to bother and close this.
\\ Migrated from https://github.com/keroserene/snowflake/issues/8https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/25472snowflake-client's `-url` option is intolerant of a missing path2020-06-27T13:40:39ZDavid Fifielddcf@torproject.orgsnowflake-client's `-url` option is intolerant of a missing pathI was following [server-webrtc's README](https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/server-webrtc/README.md?id=ff8f3851082e8f7f8b4c8b99b161be35020aeb67) and I made a small mistake in the client torrc. Instead o...I was following [server-webrtc's README](https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/server-webrtc/README.md?id=ff8f3851082e8f7f8b4c8b99b161be35020aeb67) and I made a small mistake in the client torrc. Instead of
```
-url http://127.0.0.1:8080/
```
I had (no trailing slash):
```
-url http://127.0.0.1:8080
```
This caused the client to fail with this error:
```
2018/03/13 03:10:40 Negotiating via BrokerChannel...
Target URL:
Front URL: 127.0.0.1:8080
2018/03/13 03:10:40 BrokerChannel Error: dial tcp: unknown port tcp/8080client
2018/03/13 03:10:40 Failed to retrieve answer. Retrying in 10 seconds
```
This is because the URL to request is built using string concatenation. It built the string `"http://127.0.0.1:8080client"` instead of `"http://127.0.0.1:8080/client"`.
https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/client/rendezvous.go?id=ff8f3851082e8f7f8b4c8b99b161be35020aeb67#n83
```
request, err := http.NewRequest("POST", bc.url.String()+"client", data)
```https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/25429Need something better than client's `checkForStaleness`2023-08-01T23:54:09ZArlo BreaultNeed something better than client's `checkForStaleness`If no message has been received on the datachannel on the client for `SnowflakeTimeout` (30 seconds), `checkForStaleness` closes the connection. The comment says this is to,
> // Prevent long-lived broken remotes.
but there's no hear...If no message has been received on the datachannel on the client for `SnowflakeTimeout` (30 seconds), `checkForStaleness` closes the connection. The comment says this is to,
> // Prevent long-lived broken remotes.
but there's no heartbeat at this level of abstraction so the connection is constantly being reset anytime the user pauses their activity (for example, to read a webpage).
This greatly exacerbated legacy/trac#21312Sponsor 28: Reliable Anonymous Communication Evading Censors and Repressors (RACECAR)Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/25346Adapt snowflake-server to use ACME HTTP-01 challenge for automatic certificates2024-01-08T00:03:43ZDavid Fifielddcf@torproject.orgAdapt snowflake-server to use ACME HTTP-01 challenge for automatic certificatesAs with the broker (tpo/anti-censorship/pluggable-transports/snowflake#25345), we need to make the Snowflake server transport plugin use the HTTP-01 challenge.As with the broker (tpo/anti-censorship/pluggable-transports/snowflake#25345), we need to make the Snowflake server transport plugin use the HTTP-01 challenge.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/25345Adapt broker to use ACME HTTP-01 challenge for automatic certificates2024-01-08T00:03:43ZDavid Fifielddcf@torproject.orgAdapt broker to use ACME HTTP-01 challenge for automatic certificatesSee tpo/anti-censorship/pluggable-transports/meek#24928 for the equivalent ticket for meek. The TLS-SNI-01 challenge that we were using doesn't work anymore. Because of this, the standalone broker has been broken since 2018-01-18 :/See tpo/anti-censorship/pluggable-transports/meek#24928 for the equivalent ticket for meek. The TLS-SNI-01 challenge that we were using doesn't work anymore. Because of this, the standalone broker has been broken since 2018-01-18 :/David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/25344proxy-go needs to relax between polls2020-06-27T13:40:40ZDavid Fifielddcf@torproject.orgproxy-go needs to relax between pollsThe JavaScript proxy has `DEFAULT_BROKER_POLL_INTERVAL`, but there's nothing like that in proxy-go. The standalone broker is getting 2 to 5 /proxy requests per second from the 3 round-the-clock proxy-go instances.The JavaScript proxy has `DEFAULT_BROKER_POLL_INTERVAL`, but there's nothing like that in proxy-go. The standalone broker is getting 2 to 5 /proxy requests per second from the 3 round-the-clock proxy-go instances.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/25123Snowflake not work2020-06-27T13:40:40ZcypherpunksSnowflake not workhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/23780Tor repeatedly tells me that "Your Guard is failing an extremely large amount...2020-06-27T13:40:41ZcypherpunksTor repeatedly tells me that "Your Guard is failing an extremely large amount of circuits" when using snowflakeOf course, Snowflake works and I have no idea why the guard must be failing. Moreover, it says that message right after the 85% in the bootstrap and 90%:
```
[notice] Bootstrapped 80%: Connecting to the Tor network
[notice] Bootstrapped...Of course, Snowflake works and I have no idea why the guard must be failing. Moreover, it says that message right after the 85% in the bootstrap and 90%:
```
[notice] Bootstrapped 80%: Connecting to the Tor network
[notice] Bootstrapped 85%: Finishing handshake with first hop
[warn] Your Guard [scrubbed] is failing an extremely large amount of circuits. This could indicate a route manipulation attack, extreme network overload, or a bug. Success counts are 50/169. Use counts are 73/76. 162 circuits completed, 1 were unusable, 112 collapsed, and 4 timed out. For reference, your timeout cutoff is 60 seconds.
[notice] Tor has successfully opened a circuit. Looks like client functionality is working.
[notice] Bootstrapped 100%: Done
[warn] Your Guard [scrubbed] is failing a very large amount of circuits. Most likely this means the Tor network is overloaded, but it could also mean an attack against you or potentially the guard itself. Success counts are 52/171. Use counts are 73/76. 164 circuits completed, 1 were unusable, 112 collapsed, and 4 timed out. For reference, your timeout cutoff is 60 seconds.
```https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/23749Snowflake-client potentially suffers from memory leaks2020-06-27T13:40:41ZcypherpunksSnowflake-client potentially suffers from memory leaksIt uses up to 160.5 MiB on my system after 6 hours of use, so it seems that it's due to some memory leak, no?It uses up to 160.5 MiB on my system after 6 hours of use, so it seems that it's due to some memory leak, no?https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/23356proxy-go starts using 100% CPU when network is disconnected2020-06-27T13:40:41ZDavid Fifielddcf@torproject.orgproxy-go starts using 100% CPU when network is disconnectedA power outage disconnected the network of a laptop on which I was running proxy-go. The laptop battery kept the computer running until the power came back on, but the network was still down. When I checked on it, the log was filled with...A power outage disconnected the network of a laptop on which I was running proxy-go. The laptop battery kept the computer running until the power came back on, but the network was still down. When I checked on it, the log was filled with 2.4 GB of
```
2017/08/29 10:02:33 error polling broker: Post https://snowflake-reg.appspot.com/proxy: dial tcp: lookup snowflake-reg.appspot.com on <dns server>:53: dial udp <dns server>:53: connect: network is unreachable
2017/08/29 10:02:33 error polling broker: Post https://snowflake-reg.appspot.com/proxy: dial tcp: lookup snowflake-reg.appspot.com on <dns server>:53: dial udp <dns server>:53: connect: network is unreachable
2017/08/29 10:02:33 error polling broker: Post https://snowflake-reg.appspot.com/proxy: dial tcp: lookup snowflake-reg.appspot.com on <dns server>:53: dial udp <dns server>:53: connect: network is unreachable
```
There were 11,879,088 of these messages over the course of about an hour (according to the log timestamps), so about 3,330 messages per second. I'm guessing the code was in a tight failure loop.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/23257Snowflake doesn't connect on the CalVisitor network2022-07-09T04:20:15ZDavid Fifielddcf@torproject.orgSnowflake doesn't connect on the CalVisitor networkSnowflake doesn't work on [one of the wireless networks](https://telecom.berkeley.edu/calvisitor) at UC Berkeley. It sends an offer and receives an answer, but then hangs here:
```
2017/08/16 13:56:38 ---- Handler: snowflake assigned ---...Snowflake doesn't work on [one of the wireless networks](https://telecom.berkeley.edu/calvisitor) at UC Berkeley. It sends an offer and receives an answer, but then hangs here:
```
2017/08/16 13:56:38 ---- Handler: snowflake assigned ----
2017/08/16 13:56:38 Buffered 203 bytes --> WebRTC
2017/08/16 13:56:43 Traffic Bytes (in|out): 0 | 203 -- (0 OnMessages, 1 Sends)
```
I initially suspected that it had something to do with the network not assigning a public IPv4 address, only a public IPv6 address. I moved our existing proxy-go instances to a host with an IPv6 address, and that didn't help.
Here's the part of the log where client gathers its candidates. Notice the IPv6 address 2607:f140:6000:2:c53e:4d3e:19d5:d94c and the IPv4 address 10.105.136.190.
```
2017/08/16 13:55:58 WebRTC: DataChannel created.
2017/08/16 13:55:58 candidate:1109156821 1 udp 2122262783 2607:f140:6000:2:c53e:4d3e:19d5:d94c 56748 typ host generation 0 ufrag 3h4p network-id 2 network-cost 50
2017/08/16 13:55:58 candidate:2027054270 1 udp 2122194687 10.105.136.190 61395 typ host generation 0 ufrag 3h4p network-id 1 network-cost 50
2017/08/16 13:55:58 candidate:211787557 1 tcp 1518283007 2607:f140:6000:2:c53e:4d3e:19d5:d94c 49259 typ host tcptype passive generation 0 ufrag 3h4p network-id 2 network-cost 50
2017/08/16 13:55:58 candidate:911317070 1 tcp 1518214911 10.105.136.190 49260 typ host tcptype passive generation 0 ufrag 3h4p network-id 1 network-cost 50
2017/08/16 13:55:59 SOCKS accepted: {0.0.3.0:1 map[]}
```
Here's an abridged form of the answer. Notice how the candidates (`a=candidate:`) include both IPv6 2a00:c6c0:0:151:4:8f94:69f5:7c01 and IPv4 37.218.242.151. However, the connection address (`c=`) is the IPv4 address.
```
o=- 5000276783403536546 2 IN IP4 127.0.0.1
m=application 41242 DTLS/SCTP 5000
c=IN IP4 37.218.242.151
a=candidate:836092752 1 udp 2122262783 2a00:c6c0:0:151:4:8f94:69f5:7c01 40571 typ host generation 0 network-id 1 network-cost 50
a=candidate:404726760 1 udp 2122194687 37.218.242.151 41242 typ host generation 0 network-id 2 network-cost 50
a=candidate:2136358816 1 tcp 1518283007 2a00:c6c0:0:151:4:8f94:69f5:7c01 39103 typ host tcptype passive generation 0 network-id 1 network-cost 50
a=candidate:1453088536 1 tcp 1518214911 37.218.242.151 49703 typ host tcptype passive generation 0 network-id 2 network-cost 50
```
Is this one of those weird NAT cases where a TURN server is needed? Is that why it's not working? The CalVisitor does block some applications like IMAP.
I tested this with Tor Browser 7.5a4 for mac.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/23209When I opened Tor Launcher and changed my pluggable transport to snowflake a ...2022-07-09T04:20:15ZTracWhen I opened Tor Launcher and changed my pluggable transport to snowflake a message from my firewall popped upWhen I opened Tor Launcher and changed my pluggable transport to snowflake a message from my firewall popped up, it said "do you want to allow snowflake-client to allow incoming connections", some users may be concerned because this mess...When I opened Tor Launcher and changed my pluggable transport to snowflake a message from my firewall popped up, it said "do you want to allow snowflake-client to allow incoming connections", some users may be concerned because this message has never popped up when using Tor Browser Bundle.
TBB 7.5a4
**Trac**:
**Username**: Dbryrtfbcbhgfhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/21954Snowflake breaks the 7.0a3 build2022-07-09T04:20:15ZGeorg KoppenSnowflake breaks the 7.0a3 buildBuilding Snowflake breaks our 7.0a3 build with the following error:
```
+ mkdir -p /home/debian/go/src/github.com/keroserene/
+ ln -sf /home/debian/build/go-webrtc /home/debian/go/src/github.com/keroserene/go-webrtc
+ CGO_CXXFLAGS='-D_GL...Building Snowflake breaks our 7.0a3 build with the following error:
```
+ mkdir -p /home/debian/go/src/github.com/keroserene/
+ ln -sf /home/debian/build/go-webrtc /home/debian/go/src/github.com/keroserene/go-webrtc
+ CGO_CXXFLAGS='-D_GLIBCXX_USE_CXX11_ABI=1 -D__STDC_FORMAT_MACROS=1'
+ CGO_LDFLAGS=-latomic
+ go install github.com/keroserene/go-webrtc
# github.com/keroserene/go-webrtc
<command-line>:0:0: warning: "_GLIBCXX_USE_CXX11_ABI" redefined
<command-line>:0:0: note: this is the location of the previous definition
# github.com/keroserene/go-webrtc
<command-line>:0:0: warning: "_GLIBCXX_USE_CXX11_ABI" redefined
<command-line>:0:0: note: this is the location of the previous definition
# github.com/keroserene/go-webrtc
<command-line>:0:0: warning: "_GLIBCXX_USE_CXX11_ABI" redefined
<command-line>:0:0: note: this is the location of the previous definition
# github.com/keroserene/go-webrtc
/home/debian/install/binutils/bin/ld.gold.real: error: ../../go/src/github.com/keroserene/go-webrtc/lib/libwebrtc-linux-386-magic.a(dump_syms_regtest.o): unsupported ELF file type 2
collect2: error: ld returned 1 exit status
```
I double-checked that the commits I am on are the same as in the patch for legacy/trac#21748. Furthermore, I made a `make clean-webrtc` and the build started with `tbb-7.0a3-build3`.
There are actually two issues here:
1) We need to unblock the 7.0a3 build as soon as possible as we want to release the first ESR52-based Tor Browser alpha next week.
2) I'd like to understand how this can happen at all given that we did not change the commits compared to the fix in legacy/trac#21748 and I tested that the building worked back then. I somehow doubt that arlolra and I made the same mistake (by using the wrong commit or something).
I need to have a fix for that within the next 24 hours, otherwise do I need to disable Snowflake in the alphas.Arlo BreaultArlo Breaulthttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/21748Snowflake breaks nightly builds as of March 152023-04-01T17:44:34ZGeorg KoppenSnowflake breaks nightly builds as of March 15Our nightly builds are broken now due to Snowflake compilation being unhappy:
```
+ go install github.com/keroserene/go-webrtc
# github.com/keroserene/go-webrtc
<command-line>:0:0: warning: "_GLIBCXX_USE_CXX11_ABI" redefined
<command-lin...Our nightly builds are broken now due to Snowflake compilation being unhappy:
```
+ go install github.com/keroserene/go-webrtc
# github.com/keroserene/go-webrtc
<command-line>:0:0: warning: "_GLIBCXX_USE_CXX11_ABI" redefined
<command-line>:0:0: note: this is the location of the previous definition
# github.com/keroserene/go-webrtc
<command-line>:0:0: warning: "_GLIBCXX_USE_CXX11_ABI" redefined
<command-line>:0:0: note: this is the location of the previous definition
# github.com/keroserene/go-webrtc
<command-line>:0:0: warning: "_GLIBCXX_USE_CXX11_ABI" redefined
<command-line>:0:0: note: this is the location of the previous definition
../../go/src/github.com/keroserene/go-webrtc/peerconnection.cc:13:51: fatal error: webrtc/pc/test/fakeaudiocapturemodule.h: No such file or directory
compilation terminated.
```Arlo BreaultArlo Breaulthttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/21314snowflake-client needs to stop using my network when I'm not giving it requests2020-11-23T18:41:46ZRoger Dingledinesnowflake-client needs to stop using my network when I'm not giving it requestsI started my Tor Browser, and told it to use snowflake, and it did. Then I changed my mind and told it to stop using snowflake. Now, apparently there's a bug in Tor where Tor is supposed to kill snowflake-client when there are no more br...I started my Tor Browser, and told it to use snowflake, and it did. Then I changed my mind and told it to stop using snowflake. Now, apparently there's a bug in Tor where Tor is supposed to kill snowflake-client when there are no more bridge lines in my torrc that want to use it. But ignoring that Tor bug, snowflake-client should also be defensive for me. Right now it is touching the broker every 10 seconds, looking for a snowflake, even though it is getting no requests. That can't be good for scalability or for the broker or for the users.Snowflake in Tor Browser 10.5Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/21312Memory and file descriptor leaks in programs that use go-webrtc2022-03-16T15:45:50ZArlo BreaultMemory and file descriptor leaks in programs that use go-webrtc```
1:30:49 AM - arma5: 2017/01/24 18:57:44 candidate:3131354255 1 tcp 1518280447 192.168.1.154 36729 typ host tcptype passive generation 0 ufrag YXKXffPBBnA3SJ3K network-id 1
1:30:52 AM - arma5: this is the last line in its log
1:31:20 ...```
1:30:49 AM - arma5: 2017/01/24 18:57:44 candidate:3131354255 1 tcp 1518280447 192.168.1.154 36729 typ host tcptype passive generation 0 ufrag YXKXffPBBnA3SJ3K network-id 1
1:30:52 AM - arma5: this is the last line in its log
1:31:20 AM - arma5: and snowflake-client is pegged at 100% cpu
1:32:39 AM - arma5: % strace -p3074
1:32:39 AM - arma5: Process 3074 attached
1:32:39 AM - arma5: futex(0x1059710, FUTEX_WAIT, 0, NULL
```Arlo BreaultArlo Breault