Snowflake issueshttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues2023-01-03T16:00:26Zhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40242Order of parameters may suppress other parameters2023-01-03T16:00:26Zn0tooseOrder of parameters may suppress other parameters### Steps to reproduce
- `snowflake-proxy -broker https://snowflake-broker.torproject.net/ -capacity 10 -nat-retest-interval 24h -relay wss://snowflake.bamsoftware.com/ -stun stun:stun.stunprotocol.org:3478 -summary-interval 1s -verbose...### Steps to reproduce
- `snowflake-proxy -broker https://snowflake-broker.torproject.net/ -capacity 10 -nat-retest-interval 24h -relay wss://snowflake.bamsoftware.com/ -stun stun:stun.stunprotocol.org:3478 -summary-interval 1s -verbose true -keep-local-addresses false` prints a summary once every second. Works fine.
- `snowflake-proxy -broker https://snowflake-broker.torproject.net/ -capacity 10 -keep-local-addresses false -nat-retest-interval 24h -relay wss://snowflake.bamsoftware.com/ -stun stun:stun.stunprotocol.org:3478 -summary-interval 1s -verbose true` prints a summary once every hour, ignoring the summary-interval argument. I wrote it that way, as that was the order in `snowflake-proxy -help`.
This was attempted in a Docker container running on Debian Bookworm.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40241Remove stunprotocol.org as default for proxies and clients2023-05-24T16:51:53ZCecylia BocovichRemove stunprotocol.org as default for proxies and clientsThe STUN server we use as a default gets hit with millions of requests/day making it costly for the operator. Checklist for rolling out these changes:
- [x] Update the code (!129)
- [x] Update the default bridges in tor browser (tpo/app...The STUN server we use as a default gets hit with millions of requests/day making it costly for the operator. Checklist for rolling out these changes:
- [x] Update the code (!129)
- [x] Update the default bridges in tor browser (tpo/applications/tor-browser-build!617)
- [x] Update the suggested bridge lines given out by the circumvention settings API
- [x] Update the proxy docker container
- [x] Update the debian package
- [x] Update orbot bridge line (merged in https://github.com/guardianproject/orbot/commit/07e416d3a9475caf17ebb46bd437d235a440206f, but not shipped)meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40240Investigate if STUN over TCP/TLS is beneficial to us2023-03-26T18:19:13ZitchyonionInvestigate if STUN over TCP/TLS is beneficial to usFrom client [README:](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/main/client/README.md?plain=1#L48)
> `-ice` is a comma-separated list of ICE servers. These can be STUN or TURN servers.
This...From client [README:](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/main/client/README.md?plain=1#L48)
> `-ice` is a comma-separated list of ICE servers. These can be STUN or TURN servers.
This is not true, as right now only [STUN](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/main/client/lib/snowflake.go#L245) over [UDP](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/main/common/nat/nat.go#L147) is supported. We don't support any TURN servers, nor STUN over TLS (TCP).
We should clarify this in README, or implement support for those that are missing.
The STUN RFC contains a list of [attacks](https://www.rfc-editor.org/rfc/rfc5389#section-16.1) that can be mitigated by STUN over TLS. I'm not sure if these are relevant for Tor's usecase.itchyonionitchyonionhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40239Experiment with increasing conntrack table size on snowflake-012024-01-22T15:19:37ZDavid Fifielddcf@torproject.orgExperiment with increasing conntrack table size on snowflake-01During performance experimentation we (@linus and I) [disabled connection tracking](tpo/anti-censorship/pluggable-transports/snowflake#40189), suspecting it as a cause of high CPU use and because the conntrack table appeared to be danger...During performance experimentation we (@linus and I) [disabled connection tracking](tpo/anti-censorship/pluggable-transports/snowflake#40189), suspecting it as a cause of high CPU use and because the conntrack table appeared to be dangerously close to being full.
Let we [re-enabled connection tracking](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40203#note_2851633) to no apparent ill effect.
[Performance optimizations in snowflake-server](tpo/anti-censorship/pluggable-transports/snowflake!100) were enough to bring the CPU use under control.
The conntrack table still appears to be close to overflowing, but we are not actually seeing any "nf_conntrack: table full, dropping packet" kernel logs that would indicate an actual problem.
At this moment the conntrack table is 92% full (`nf_conntrack_count` / `nf_conntrack_max` = 240316 / 262144).
It could be that this is normal and nothing to worry about.
Let's try doubling the maximum number of entries and see if it reaches a new equilibrium.
Currently we have:
```
# cat /proc/sys/net/netfilter/nf_conntrack_{count,max,buckets}
240316
262144
65536
```
I'm going to do this:
```
echo 131072 > /proc/sys/net/netfilter/nf_conntrack_buckets
echo 524288 > /proc/sys/net/netfilter/nf_conntrack_max
```
And meanwhile I'll run a tracking script to record `nf_conntrack_count` once per minute.David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.org2022-12-21https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40238Server versionFlag duplicates -unsafe-logging option, causes immediate panic2022-12-14T17:20:49ZDavid Fifielddcf@torproject.orgServer versionFlag duplicates -unsafe-logging option, causes immediate panicRecent versions of the Snowflake server (e.g. 839d2218837dfbd1682ff39b375f45660b3974b5) panic immediately on startup:
```
snowflake/server$ ./server
./server flag redefined: unsafe-logging
panic: ./server flag redefined: unsafe-logging...Recent versions of the Snowflake server (e.g. 839d2218837dfbd1682ff39b375f45660b3974b5) panic immediately on startup:
```
snowflake/server$ ./server
./server flag redefined: unsafe-logging
panic: ./server flag redefined: unsafe-logging
```
The cause is a copy-paste error at https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/commit/33248f3dec5594c985cfd11e6c6143ddaa5613c0#2bfd4afce737f8dc3bb3ba69c068895274dac4e5_140_142,
which was part of !111.
The option name should have been `"version"`, but it duplicates `"unsafe-logging"` instead.
```go
flag.BoolVar(&unsafeLogging, "unsafe-logging", false, "prevent logs from being scrubbed")
flag.BoolVar(&versionFlag, "unsafe-logging", false, "display version info to stderr and quit")
```itchyonionitchyonionhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40237client suggested torrc does not suggest utls-imitate2022-12-09T23:08:29ZRoger Dingledineclient suggested torrc does not suggest utls-imitatehttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/main/client/torrc
no longer matches the snowflake bridge line in Tor Browser.
In particular, the utls-imitate line in not present in the recommended...https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/main/client/torrc
no longer matches the snowflake bridge line in Tor Browser.
In particular, the utls-imitate line in not present in the recommended example torrc file.
[Discovered while helping a user in iran who was running snowflake + tor directly rather than inside Tor Browser]https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40236Make `test -z` more verbose so we can easily catch when it fails2022-11-17T16:28:34ZCecylia BocovichMake `test -z` more verbose so we can easily catch when it failsA few of the recent MRs have consistently failed *all* of their CI tests:
- https://gitlab.torproject.org/dcf/snowflake/-/pipelines/56708
- https://gitlab.torproject.org/dcf/snowflake/-/pipelines/56707
These MRs pass all go tests and ve...A few of the recent MRs have consistently failed *all* of their CI tests:
- https://gitlab.torproject.org/dcf/snowflake/-/pipelines/56708
- https://gitlab.torproject.org/dcf/snowflake/-/pipelines/56707
These MRs pass all go tests and vetting when run manually outside of the CI environment. It doesn't seem to matter which runner they are executed on.
Edit: figured out the problem here: `test -z "$(go fmt ./...)"` was failing, but it's silent on the CI output because it doesn't produce stderr or stdout output. Changing the purpose of this issue from debugging to making sure this doesn't confuse us again.Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40234Signaling through TURN2023-08-24T18:14:07ZWofWcawofwca@protonmail.comSignaling through TURNThis one is an epic.
I was thinking about https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/22945#note_2823413 and #40164 and came up with an interesting idea.
How about we do signaling through a...This one is an epic.
I was thinking about https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/22945#note_2823413 and #40164 and came up with an interesting idea.
How about we do signaling through a WebRTC peer connection itself? In order to avoid [leaking the peers' private data](https://w3c.github.io/webrtc-pc/#revealing-ip-addresses), let's establish the peer connection through a TURN relay initially (with the help of `iceTransportPolicy: "relay"` WebRTC option), then set `iceTransportPolicy` to `"all"` (enabling STUN and true P2P) and `restartIce()` and continue signaling (ICE trickling).
Where do we get a TURN server, you might ask? Let's host it along with the broker, I say. Of course we'll probably need some gatekeeping for it (like limiting bandwidth, connection duration, only allowing peers that have communicated with the broker, rotating passwords) so that it doesn't get overloaded by outsiders. Conveniently, [Pion also offers a powerful TURN library](https://github.com/pion/turn) ([example](https://github.com/pion/turn/blob/v2.0.8/examples/turn-server/simple/main.go)).
Biggest problem - looks like the client has to tunnel the TURN traffic through a domain-fronting HTTPS (WSS?) tunnel (or some other censorship-resistant thing (#25594 )?) because the TURN server might be blocked. I'm not sure how hard it is to achieve, but here's [an example of traffic manipulation in Pion](https://github.com/pion/webrtc/blob/v3.1.47/examples/ice-single-port/main.go), so I guess it shouldn't be super hard.
So
Pros:
* Solves #22945
* Practically Solves #40164
* Solves the verification part of #40165 because it's not two different connections, it's the same one.
* Makes the broker more future-proof because it doesn't have to process data that the proxy and the client want to exchange, it simply passes it along.
* Can allow faster bootstrapping by relaying (non-signaling) proxy-to-client data initially, before true P2P has been established.
* (maybe, need to verify) better DPI resistance due to handshake being performed in a secure (domain-fronted) channel (see https://gitlab.torproject.org/tpo/anti-censorship/censorship-analysis/-/issues/40030).
Cons:
* The broker codebase needs a major overhaul.
[Some chat logs](https://matrix.to/#/!hNphRlWKcRVXnwAWJy:matrix.org/$EWgGZ38YotRK9zhqpMjBJo98wQo1HFapnLRXqzsBSCg?via=matrix.org&via=nitro.chat&via=systemli.org) (nothing particularly important).https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40233package version 2.3 in debian2022-11-15T17:02:18Zmeskiomeskio@torproject.orgpackage version 2.3 in debianThe current version of snowflake-proxy in debian doesn't include the support for multiple bridges. It is only present in testing and sid, but people is complaining that their proxies are not working anymore. Let's upgrade the package.The current version of snowflake-proxy in debian doesn't include the support for multiple bridges. It is only present in testing and sid, but people is complaining that their proxies are not working anymore. Let's upgrade the package.meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40232snowflake proxy README includes an old version of ./proxy --help2023-07-29T22:27:43ZRoger Dingledinesnowflake proxy README includes an old version of ./proxy --helpIn snowflake/proxy/README, we have this section:
```
The Snowflake proxy can be run with the following options:
Usage of ./proxy:
-broker string
broker URL (default "https://snowflake-broker.torproject.net/")
-capacity uint...In snowflake/proxy/README, we have this section:
```
The Snowflake proxy can be run with the following options:
Usage of ./proxy:
-broker string
broker URL (default "https://snowflake-broker.torproject.net/")
-capacity uint
maximum concurrent clients
-keep-local-addresses
keep local LAN address ICE candidates
-log string
log filename
-relay string
websocket relay URL (default "wss://snowflake.torproject.net/")
-stun string
stun URL (default "stun:stun.stunprotocol.org:3478")
-unsafe-logging
prevent logs from being scrubbed
```
but ./proxy --help now shows more options than that:
```
Usage of ./proxy:
-allow-non-tls-relay
allow relay without tls encryption
-allowed-relay-hostname-pattern string
a pattern to specify allowed hostname pattern for relay URL. (default "snowflake.torproject.net$")
-broker string
broker URL (default "https://snowflake-broker.torproject.net/")
-capacity uint
maximum concurrent clients
-ephemeral-ports-range string
ICE UDP ephemeral ports range (format:"<min>:<max>")
-keep-local-addresses
keep local LAN address ICE candidates
-log string
log filename
-nat-retest-interval duration
the time interval in second before NAT type is retested, 0s disables retest. Valid time units are "s", "m", "h". (default 24h0m0s)
-relay string
websocket relay URL (default "wss://snowflake.bamsoftware.com/")
-stun string
STUN URL (default "stun:stun.stunprotocol.org:3478")
-summary-interval duration
the time interval to output summary, 0s disables summaries. Valid time units are "s", "m", "h". (default 1h0m0s)
-unsafe-logging
prevent logs from being scrubbed
-verbose
increase log verbosity
```
The really simple fix would be to update the README with the newer text. The more robust approach would be to stop trying to maintain the same data in both places (because it's clearly not working), and change the README to teach you how to run ./proxy --help to get the usage, and/or change it to list just the most important options.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40231fix: client sometimes sends offer with no ICE candidates2024-03-21T20:25:28ZWofWcawofwca@protonmail.comfix: client sometimes sends offer with no ICE candidatesNot sure if it's normal.
This is what has been returned by the broker to a `/proxy` request:
```
{
"Status": "client match",
"Offer": "{\"type\":\"offer\",\"sdp\":\"v=0\\r\\no=- <SCRUBBED> <SCRUBBED> IN IP4 0.0.0.0\\r\\ns=-\\r\\nt=0...Not sure if it's normal.
This is what has been returned by the broker to a `/proxy` request:
```
{
"Status": "client match",
"Offer": "{\"type\":\"offer\",\"sdp\":\"v=0\\r\\no=- <SCRUBBED> <SCRUBBED> IN IP4 0.0.0.0\\r\\ns=-\\r\\nt=0 0\\r\\na=fingerprint:sha-256 <SCRUBBED>\\r\\na=extmap-allow-mixed\\r\\na=group:BUNDLE 0\\r\\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\\r\\nc=IN IP4 0.0.0.0\\r\\na=setup:actpass\\r\\na=mid:0\\r\\na=sendrecv\\r\\na=sctp-port:5000\\r\\na=ice-ufrag:<SCRUBBED>\\r\\na=ice-pwd:<SCRUBBED>\\r\\na=end-of-candidates\\r\\n\"}",
"NAT": "unrestricted",
"RelayURL": "wss://snowflake.torproject.net/"
}
```
Offer:
```
v=0
o=- <SCRUBBED> <SCRUBBED> IN IP4 0.0.0.0
s=-
t=0 0
a=fingerprint:sha-256 <SCRUBBED>
a=extmap-allow-mixed
a=group:BUNDLE 0
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=setup:actpass
a=mid:0
a=sendrecv
a=sctp-port:5000
a=ice-ufrag:<SCRUBBED>
a=ice-pwd:<SCRUBBED>
a=end-of-candidates
```
As you can see, it contains no ICE candidates. Also note the `a=end-of-candidates`. My Snowflake sent an answer but connection failed.
I have noticed this twice already.
Can this be due to the fact we're trying to filter out local network ICE candidates? And maybe STUN being blocked for the clients?itchyonionitchyonionhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40230fix: send SDP answer if ICE gathering is taking long to complete2022-12-13T16:59:46ZWofWcawofwca@protonmail.comfix: send SDP answer if ICE gathering is taking long to completeSee https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake-webext/-/merge_requests/55. Probably need to implement it here as well (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake...See https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake-webext/-/merge_requests/55. Probably need to implement it here as well (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake-webext/-/merge_requests/55#note_2851693).
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40164 has the same title, but suggests a different approach, maybe close then this in favor of that.
Also I think it's best if we can still send the `end-of-candidates` marker in the SDP so that the remote peer knows not to expect any more ICE candidates.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40228perf: proxy: don't wait for WebRTC to establish before connecting to server2022-11-14T09:51:43ZWofWcawofwca@protonmail.comperf: proxy: don't wait for WebRTC to establish before connecting to serverIs there a benefit in this waiting (other than when WebRTC fails and we spare doing the connection for nothing)?
We can start connecting as soon as we get the client's offer. This can make bootstrapping a little faster. Same goes for the...Is there a benefit in this waiting (other than when WebRTC fails and we spare doing the connection for nothing)?
We can start connecting as soon as we get the client's offer. This can make bootstrapping a little faster. Same goes for the extension.
* https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/97dea533da7b6b3b2b1dfbffe7dca3a8350fab0b/proxy/lib/snowflake.go#L328
* https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake-webext/-/blob/44bb28a15a4ebb5193e5d72b84d9259de7ea633d/proxypair.js#L140-142https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40227Non-Tor Snowflake2022-11-03T16:23:51ZcypherpunksNon-Tor SnowflakeMany Chinese/Iranians/Russians/Belarusian... these days have at least one relative in the west.
It would be great to offshoot a new branch from the Snowflake chrome extension where it operates as a non-Tor proxy.
This way, even non-tec...Many Chinese/Iranians/Russians/Belarusian... these days have at least one relative in the west.
It would be great to offshoot a new branch from the Snowflake chrome extension where it operates as a non-Tor proxy.
This way, even non-technical people can ask a relative abroad to install it in their browser or open it as a webpage, and share their IP & Snowflake key (over iMsg/Gmail/Insta/... for example) for them to connect to.
For the end-user, this has the benefit of higher bandwidth and not getting rejected by websites blocking Tor exit nodes.
However, I'm not sure how the end user could reach out to this WebRTC. Probably an app or extension would be needed there as well, unless HTTPs/SOCKs could be used?!
I'm sure the guys who figured out how to do it for Tor, can figure out a way for this case as well.
Thank youhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40226Abbreviate `ice` list in bridge lines2023-01-30T19:29:46ZDavid Fifielddcf@torproject.orgAbbreviate `ice` list in bridge linesIn tpo/applications/tor-browser-build#40665 we had a problem with snowflake bridge lines being too long.
As a mitigation we can abbreviate the largest part of the line, the `ice` list of STUN servers,
by omitting the `stun:` scheme, and ...In tpo/applications/tor-browser-build#40665 we had a problem with snowflake bridge lines being too long.
As a mitigation we can abbreviate the largest part of the line, the `ice` list of STUN servers,
by omitting the `stun:` scheme, and the port number when it is 3478.
A current bridge line is 500 bytes (counting only the part after the bridge fingerprint, which is what counts):
`fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://snowflake-broker.torproject.net.global.prod.fastly.net/ front=cdn.sstatic.net ice=stun:stun.l.google.com:19302,stun:stun.altar.com.pl:3478,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:3478,stun:stun.epygi.com:3478,stun:stun.sonetel.com:3478,stun:stun.sonetel.net:3478,stun:stun.stunprotocol.org:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478 utls-imitate=hellorandomizedalpn`
The `ice` part is 324 bytes, or 65%. We can shorten that to 209 bytes by abbreviating the entries as proposed above. That reduces the total length by 115 bytes, or 23%.
`fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://snowflake-broker.torproject.net.global.prod.fastly.net/ front=cdn.sstatic.net ice=stun.l.google.com:19302,stun.altar.com.pl,stun.antisip.com,stun.bluesip.net,stun.dus.net,stun.epygi.com,stun.sonetel.com,stun.sonetel.net,stun.stunprotocol.org,stun.uls.co.za,stun.voipgate.com,stun.voys.nl utls-imitate=hellorandomizedalpn`
To make this work, we need to have the parser supply defaults for the scheme and port of each STUN URL if not specified.
Discussion from the [2022-10-27 team meeting](http://meetbot.debian.net/tor-meeting/2022/tor-meeting.2022-10-27-15.59.html):
http://meetbot.debian.net/tor-meeting/2022/tor-meeting.2022-10-27-15.59.log.html#l-95itchyonionitchyonionhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40225Snowflake Broker Deployment 22-10-252022-11-08T18:21:09ZshelikhooSnowflake Broker Deployment 22-10-25We are going to deploy a new version of snowflake broker configuration to snowflake broker.
The broker binary isn't updated, and remain [v2.3.1](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/tags/v2.3...We are going to deploy a new version of snowflake broker configuration to snowflake broker.
The broker binary isn't updated, and remain [v2.3.1](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/tags/v2.3.1).
This will rollout the change we did in [Snowflake Broker Deployment 22-10-03](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40193) plus [include](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40212) [secondary bridge](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40122) definition, and [Remove GOMAXPROCS=1](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40205).
## Deployment Script
```
sv stop snowflake-broker
cp /etc/service/snowflake-broker/run ./snowflake-broker-run-22-10-25-backup-$(date +%N)
cp /home/snowflake-broker/bridge_lists.json ./bridge_lists_json-22-10-25-backup-$(date +%N)
install --owner root ./snowflake-broker-run-22-10-25-candidcate /etc/service/snowflake-broker/run
install --owner root ./bridge_lists_json-22-10-25-candidcate /home/snowflake-broker/bridge_lists.json
sv start snowflake-broker
```
## New Run File
(the difference is at --allowed-relay-pattern)
(-ip-count-mask's value is not real value used on the production system)
```
#!/bin/sh -e
setcap 'cap_net_bind_service=+ep' /usr/local/bin/broker
exec chpst -u snowflake-broker -o 32768 /usr/local/bin/broker --metrics-log /home/snowflake-broker/metrics.log --acme-hostnames snowflake-broker.bamsoftware.com,snowflake-broker.freehaven.net,snowflake-broker.torproject.net --acme-email dcf@torproject.org --acme-cert-cache /home/snowflake-broker/acme-cert-cache --bridge-list-path /home/snowflake-broker/bridge_lists.json --default-relay-pattern ^snowflake.torproject.net$ --allowed-relay-pattern snowflake.torproject.net$ -ip-count-log /home/snowflake-broker/metrics-ip-salted.jsonl -ip-count-interval 1h -ip-count-mask ****** 2>&1
```
## Old Run File
```
#!/bin/sh -e
setcap 'cap_net_bind_service=+ep' /usr/local/bin/broker
export GOMAXPROCS=1
exec chpst -u snowflake-broker -o 32768 /usr/local/bin/broker --metrics-log /home/snowflake-broker/metrics.log --acme-hostnames snowflake-broker.bamsoftware.com,snowflake-broker.freehaven.net,snowflake-broker.torproject.net --acme-email dcf@torproject.org --acme-cert-cache /home/snowflake-broker/acme-cert-cache --bridge-list-path /home/snowflake-broker/bridge_lists.json --default-relay-pattern ^snowflake.torproject.net$ --allowed-relay-pattern ^snowflake.torproject.net$ -ip-count-log /home/snowflake-broker/metrics-ip-salted.jsonl -ip-count-interval 1h -ip-count-mask ****** 2>&1
```
## New bridge_lists.json
```
{"displayName":"default", "webSocketAddress":"wss://snowflake.torproject.net/", "fingerprint":"2B280B23E1107BB62ABFC40DDCC8824814F80A72"}
{"displayName":"Bridge02", "webSocketAddress":"wss://02.snowflake.torproject.net/", "fingerprint":"8838024498816A039FCBBAB14E6F40A0843051FA"}
```
## Old bridge_lists.json
```
{"displayName":"default", "webSocketAddress":"wss://snowflake.torproject.net/", "fingerprint":"2B280B23E1107BB62ABFC40DDCC8824814F80A72"}
```
## Side effect to be watched
The network capacity of the snowflake may be decreased(again).Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetshelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40224utls RoundTripper does not work when it is supposed to use HTTP/12022-12-06T14:17:27ZDavid Fifielddcf@torproject.orgutls RoundTripper does not work when it is supposed to use HTTP/1Try setting `utls=hellorandomizednoalpn` in the bridge line. snowflake-client's rendezvous will fail with:
```
WebRTC: incorrect ALPN negotiated Retrying...
```
Lack of ALPN should cause a fallback to HTTP/1, and the connection should ...Try setting `utls=hellorandomizednoalpn` in the bridge line. snowflake-client's rendezvous will fail with:
```
WebRTC: incorrect ALPN negotiated Retrying...
```
Lack of ALPN should cause a fallback to HTTP/1, and the connection should still work. What's going wrong.
The problem is the internal `connectWithH1` map, which is inconsistently keyed. Entries are written to the map with `host:port` keys. Entries are read from the map using sometimes `host:port` keys, and sometimes `host` keys.
* [`RoundTrip`](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/v2.3.1/common/utls/roundtripper.go#L64) calls gets its domain name from `req.URL.Host`; i.e., it uses a key like `"cdn.sstatic.net"`.
* [`dialOrGetTLSWithExpectedALPN`](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/v2.3.1/common/utls/roundtripper.go#L124) gets its domain name from an `addr` argument to `DialTLS`; i.e., it uses a key like `"cdn.sstatic.net:443"`.
Adding a few debugging logs makes it clear what is going wrong:
```
2022/10/21 23:26:48 Target URL: snowflake-broker.torproject.net.global.prod.fastly.net
2022/10/21 23:26:48 Front URL: cdn.sstatic.net
2022/10/21 23:26:48 getShouldConnectWithH1("cdn.sstatic.net") -> false
2022/10/21 23:26:48 getShouldConnectWithH1("cdn.sstatic.net:443") -> false
2022/10/21 23:26:48 setShouldConnectWithH1("cdn.sstatic.net:443")
2022/10/21 23:26:48 getShouldConnectWithH1("cdn.sstatic.net") -> false
2022/10/21 23:26:48 getShouldConnectWithH1("cdn.sstatic.net:443") -> true
2022/10/21 23:26:48 getShouldConnectWithH1("cdn.sstatic.net") -> false
2022/10/21 23:26:48 getShouldConnectWithH1("cdn.sstatic.net:443") -> true
2022/10/21 23:26:48 getShouldConnectWithH1("cdn.sstatic.net") -> false
2022/10/21 23:26:48 getShouldConnectWithH1("cdn.sstatic.net:443") -> true
2022/10/21 23:26:48 getShouldConnectWithH1("cdn.sstatic.net") -> false
2022/10/21 23:26:48 getShouldConnectWithH1("cdn.sstatic.net:443") -> true
2022/10/21 23:26:48 WebRTC: closing DataChannel
2022/10/21 23:26:48 WebRTC: closing PeerConnection
2022/10/21 23:26:48 WebRTC: Closing
2022/10/21 23:26:48 WebRTC: incorrect ALPN negotiated Retrying...
```shelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40222Inactive connections with 510 second duration2023-06-28T11:15:43ZVortInactive connections with 510 second durationAfter commit 1bc54948 (!108), I started to notice lots of connections in logs with similar duration (510s-512s), which were closed due to inactivity.
I see several possible reasons for such behaviour:
1. All such connections come from ...After commit 1bc54948 (!108), I started to notice lots of connections in logs with similar duration (510s-512s), which were closed due to inactivity.
I see several possible reasons for such behaviour:
1. All such connections come from non-standard clients.
2. Timeout value from !108 is too small.
3. Lots of connections broke because of high rate of UDP losses in my network.
```bash
$ grep -c "over 51[012]" log_crop2.txt && grep -c "Traffic throughput" log_crop2.txt
52
459
```
If connections are closed because of problems with my network (3), then, most likely, nothing should be done from snowflake side.
If this is a sign of attack or other bot activity (1), then it may need further investigation.
And, most importantly, if all of them are legit and my network is good enough for snowflake (2), it means losing >11% of user connections just because of bug, which I think is not acceptable.
```bash
$ grep "over 51[012]" log_crop2.txt | tail -20
2022/10/20 09:20:16 Traffic throughput (up|down): 1 MB|45 KB -- (209 OnMessages, 2641 Sends, over 510 seconds)
2022/10/20 09:49:23 Traffic throughput (up|down): 7 MB|367 KB -- (750 OnMessages, 6452 Sends, over 510 seconds)
2022/10/20 10:18:56 Traffic throughput (up|down): 184 KB|12 KB -- (53 OnMessages, 1236 Sends, over 510 seconds)
2022/10/20 11:05:02 Traffic throughput (up|down): 1 MB|26 KB -- (193 OnMessages, 1156 Sends, over 510 seconds)
2022/10/20 11:11:46 Traffic throughput (up|down): 51 KB|16 KB -- (93 OnMessages, 853 Sends, over 510 seconds)
2022/10/20 11:52:27 Traffic throughput (up|down): 11 MB|180 KB -- (428 OnMessages, 11799 Sends, over 510 seconds)
2022/10/20 12:16:46 Traffic throughput (up|down): 48 KB|13 KB -- (169 OnMessages, 334 Sends, over 510 seconds)
2022/10/20 12:30:28 Traffic throughput (up|down): 3 MB|353 KB -- (1053 OnMessages, 4517 Sends, over 510 seconds)
2022/10/20 12:31:58 Traffic throughput (up|down): 4 MB|47 KB -- (500 OnMessages, 3478 Sends, over 511 seconds)
2022/10/20 12:39:16 Traffic throughput (up|down): 27 KB|28 KB -- (58 OnMessages, 297 Sends, over 510 seconds)
2022/10/20 12:56:22 Traffic throughput (up|down): 1 MB|12 KB -- (72 OnMessages, 1206 Sends, over 510 seconds)
2022/10/20 13:16:58 Traffic throughput (up|down): 29 KB|5 KB -- (28 OnMessages, 224 Sends, over 510 seconds)
2022/10/20 13:29:10 Traffic throughput (up|down): 159 KB|4 KB -- (17 OnMessages, 1997 Sends, over 510 seconds)
2022/10/20 13:30:19 Traffic throughput (up|down): 244 KB|199 KB -- (444 OnMessages, 1257 Sends, over 510 seconds)
2022/10/20 13:52:30 Traffic throughput (up|down): 259 KB|77 KB -- (236 OnMessages, 1230 Sends, over 510 seconds)
2022/10/20 13:58:37 Traffic throughput (up|down): 33 KB|7 KB -- (39 OnMessages, 760 Sends, over 511 seconds)
2022/10/20 14:00:25 Traffic throughput (up|down): 23 KB|15 KB -- (89 OnMessages, 398 Sends, over 510 seconds)
2022/10/20 14:51:09 Traffic throughput (up|down): 102 KB|30 KB -- (344 OnMessages, 538 Sends, over 511 seconds)
2022/10/20 15:06:10 Traffic throughput (up|down): 132 KB|31 KB -- (87 OnMessages, 305 Sends, over 510 seconds)
2022/10/20 15:10:26 Traffic throughput (up|down): 161 KB|111 KB -- (194 OnMessages, 1157 Sends, over 510 seconds)
```https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40218Can we include the pieces IPtProxy needs into the client library?2024-02-01T12:07:48Zmeskiomeskio@torproject.orgCan we include the pieces IPtProxy needs into the client library?IPtProxy does patch snowflake to use it, to avoid rewriting what we already have in the cli client side:
https://github.com/tladesignz/IPtProxy/blob/master/snowflake.patch
Would be nice if we find a way for the client library to acomoda...IPtProxy does patch snowflake to use it, to avoid rewriting what we already have in the cli client side:
https://github.com/tladesignz/IPtProxy/blob/master/snowflake.patch
Would be nice if we find a way for the client library to acomodate IPtPRoxy needs and don't need to patch it.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40217Upgrade utls dependency to v1.1.3 or later2023-10-11T05:28:38ZDavid Fifielddcf@torproject.orgUpgrade utls dependency to v1.1.3 or laterOur [current version](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/v2.3.1/go.mod#L15) is [v1.0.0](https://github.com/refraction-networking/utls/releases/tag/v1.0.0) from 2021-11-02. Newer releas...Our [current version](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/v2.3.1/go.mod#L15) is [v1.0.0](https://github.com/refraction-networking/utls/releases/tag/v1.0.0) from 2021-11-02. Newer releases have important enhancements, notably new and corrected fingerprints:
* [Implement certificate compression #95](https://github.com/refraction-networking/utls/pull/95)
* [new ClientHellos and Extensions #116](https://github.com/refraction-networking/utls/pull/116)
* Chrome_87
* Chrome_96
* Chrome_100
* Chrome_102
* Firefox_99
* Firefox_102
* iOS_13
* iOS_14
* Android_11
* ApplicationSettingsExtension
* SignatureAlgorithmsCertExtension
* DelegatedCredentialsExtension
* StatusRequestExtension
* [Add new ClientHellos #122](https://github.com/refraction-networking/utls/pull/122)
* Firefox 105
* Chrome 106
* Edge 106
* Safari 16.0
* 360 Secure Browser 11.0
* QQ Browser 11.1
* [Fix Google Parrots #125](https://github.com/refraction-networking/utls/pull/125)
You may not want to automatically include every possible new supported fingerprint. For example, see [this caveat](https://github.com/refraction-networking/utls/pull/122#issue-1401840671):
> Unfortunately, the specs based on Edge 106 and 360 11.0 seem to incompatible with this library.shelikhooshelikhoo