Snowflake issueshttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues2023-10-07T17:22:33Zhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40282Release tar.gz file is not available and "go build" possibly not work on armhf2023-10-07T17:22:33ZslrslrRelease tar.gz file is not available and "go build" possibly not work on armhfHello,
on 6.1.28+ armv7l GNU/Linux
$ cd /mnt/tmpfs;git clone https://git.torproject.org/pluggable-transports/snowflake.git && cd snowflake/proxy;go build
```
Cloning into 'snowflake'...
warning: redirecting to https://gitlab.torproject...Hello,
on 6.1.28+ armv7l GNU/Linux
$ cd /mnt/tmpfs;git clone https://git.torproject.org/pluggable-transports/snowflake.git && cd snowflake/proxy;go build
```
Cloning into 'snowflake'...
warning: redirecting to https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake.git/
remote: Enumerating objects: 6098, done.
remote: Counting objects: 100% (199/199), done.
remote: Compressing objects: 100% (114/114), done.
remote: Total 6098 (delta 83), reused 194 (delta 83), pack-reused 5899
Receiving objects: 100% (6098/6098), 2.64 MiB | 3.70 MiB/s, done.
Resolving deltas: 100% (4117/4117), done.
go: downloading github.com/prometheus/client_golang v1.16.0
...
go: downloading gopkg.in/yaml.v3 v3.0.1
/home/user/go/pkg/mod/golang.org/x/crypto@v0.10.0/curve25519/curve25519_go120.go:9:8: package crypto/ecdh is not in GOROOT (/usr/lib/go-1.15/src/crypto/ecdh)
```
user@host /mnt/tmpfs/snowflake/proxy $ ls
```
lib main.go README.md
```
And when i copy [latest release](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/releases) .tar.gz file link and try to download it:
$ cd /mnt/tmpfs/ && wget https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/jobs/295269/artifacts/file/snowflake-.tar.gz
```
...
HTTP request sent, awaiting response... 404 Not Found
```
Can you fix that 404 file please, or if you have idea how to allow build on mine mentioned system?
In the meantime, i have tried to "go built" on a different (also Debian based - 5.10.0-20-amd64 x86_64 GNU/Linux) computer and copied resulting "proxy" directory to 6.1.28+ armv7l GNU/Linux computer: scp -r ../proxy/ user@192.168.1.10:/home/user/snowflake/
Yet it does not launch it: /home/user/snowflake/proxy/proxy: cannot execute binary file: Exec format error
Building using "GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build" [mentioned in my previous issue](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40213#note_2845175) and copying to othr computer also end in "Exec format error".
On source, it launch ./proxy OK.meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40281Umbrel Snowflake Tor Proxy down for days2023-08-09T10:41:18ZXiangcaiUmbrel Snowflake Tor Proxy down for days### Umbrel SnowFlake Log
```plaintext
snowflake
Attaching to snowflake_tor_server_1, snowflake_web_1, snowflake_app_proxy_1, snowflake_proxy_1
app_proxy_1 | [HPM] Client disconnected
app_proxy_1 | Validating token: 1b19614ff477 ......### Umbrel SnowFlake Log
```plaintext
snowflake
Attaching to snowflake_tor_server_1, snowflake_web_1, snowflake_app_proxy_1, snowflake_proxy_1
app_proxy_1 | [HPM] Client disconnected
app_proxy_1 | Validating token: 1b19614ff477 ...
app_proxy_1 | Validating token: 1b19614ff477 ...
app_proxy_1 | Validating token: 1b19614ff477 ...
app_proxy_1 | [HPM] Upgrading to WebSocket
app_proxy_1 | Validating token: 1b19614ff477 ...
app_proxy_1 | [HPM] Client disconnected
app_proxy_1 | Validating token: 1b19614ff477 ...
app_proxy_1 | [HPM] Upgrading to WebSocket
app_proxy_1 | [HPM] Client disconnected
proxy_1 | 2023/07/27 13:57:22 Timed out waiting for client to open data channel.
proxy_1 | 2023/07/27 13:57:23 sdp offer successfully received.
proxy_1 | 2023/07/27 13:57:23 Generating answer...
proxy_1 | 2023/07/27 13:57:49 Timed out waiting for client to open data channel.
proxy_1 | 2023/07/27 13:58:42 sdp offer successfully received.
proxy_1 | 2023/07/27 13:58:42 Generating answer...
proxy_1 | 2023/07/27 13:59:10 Timed out waiting for client to open data channel.
proxy_1 | 2023/07/27 14:01:44 sdp offer successfully received.
proxy_1 | 2023/07/27 14:01:44 Generating answer...
proxy_1 | 2023/07/27 14:02:10 Timed out waiting for client to open data channel.
web_1 | 2023/07/27 13:41:50 websocket: close 1001
web_1 | 2023/07/27 13:41:50 Connection closed: 10.21.0.8:59112, connections: 0
web_1 | 2023/07/27 13:41:50 Command exited for: 10.21.0.8:59112
web_1 | 2023/07/27 13:57:00 10.21.0.8:59760 200 GET /auth_token.js
web_1 | 2023/07/27 13:57:00 New client connected: 10.21.0.8:59764
web_1 | 2023/07/27 13:57:00 Command is running for client 10.21.0.8:59764 with PID 23 (args="-c tail -n 10000 -f /snowflake/snowflake.log | grep \"Traffic Relayed\""), connections: 1
web_1 | 2023/07/27 13:57:00 10.21.0.8:59764 101 GET /ws
web_1 | 2023/07/27 13:57:03 websocket: close 1001
web_1 | 2023/07/27 13:57:03 Command exited for: 10.21.0.8:59764
web_1 | 2023/07/27 13:57:03 Connection closed: 10.21.0.8:59764, connections: 0
```meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40279Try reducing num-turbotunnel in server2024-01-22T15:19:37ZDavid Fifielddcf@torproject.orgTry reducing num-turbotunnel in server#40200 (and later #40246) caused snowflake-server to use multiple parallel KCP engines, which had an observable beneficial effect on performance. But those changes happened during the time between #40199 and #40262, and it may be that th...#40200 (and later #40246) caused snowflake-server to use multiple parallel KCP engines, which had an observable beneficial effect on performance. But those changes happened during the time between #40199 and #40262, and it may be that their apparent benefit was actually an accidental partial mitigation of #40260. (Partitioning clients over multiple KCP engines may have reduced the chance of packet corruption, which had its own negative effect on performance.)
I want to try reducing `num-turbotunnel` again, now that #40260 is fixed.
/cc @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40277Specialize ClientMap for ClientID2024-01-07T23:43:45ZDavid Fifielddcf@torproject.orgSpecialize ClientMap for ClientIDThe [`ClientMap.SendQueue`](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/58c3121c6b3afe54ddb31edc5efca613142bbe4e/common/turbotunnel/clientmap.go#L56)
function is the first actual Snowflake func...The [`ClientMap.SendQueue`](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/58c3121c6b3afe54ddb31edc5efca613142bbe4e/common/turbotunnel/clientmap.go#L56)
function is the first actual Snowflake function
that appears in a CPU profile of snowflake-server:
[snowflake-server.2023-06-28T10_47_29Z.cpu.prof](/uploads/7b0f409a90bdc95b4bba169df794649a/snowflake-server.2023-06-28T10_47_29Z.cpu.prof)
```
$ go tool pprof -text 'snowflake-server.2023-06-28T10:47:29Z.cpu.prof'
File: snowflake-server.20230628.c3e2f91b.prof
Type: cpu
Time: Jun 28, 2023 at 10:47am (UTC)
Duration: 1hrs, Total samples = 63696.55s (1769.24%)
Showing nodes accounting for 52749.47s, 82.81% of 63696.55s total
Dropped 1622 nodes (cum <= 318.48s)
flat flat% sum% cum cum%
17749.19s 27.87% 27.87% 17749.19s 27.87% runtime/internal/syscall.Syscall6
3813.60s 5.99% 33.85% 3813.60s 5.99% runtime.epollwait
2380.75s 3.74% 37.59% 8172.14s 12.83% runtime.selectgo
1915.76s 3.01% 40.60% 1915.76s 3.01% runtime.memmove
1770.22s 2.78% 43.38% 2580.39s 4.05% runtime.lock2
1269.98s 1.99% 45.37% 1341.07s 2.11% runtime.casgstatus
1046.43s 1.64% 47.01% 1291.14s 2.03% runtime.unlock2
998.18s 1.57% 48.58% 2658.96s 4.17% runtime.sellock
905.56s 1.42% 50.00% 3656.57s 5.74% runtime.mallocgc
867.74s 1.36% 51.36% 871.58s 1.37% runtime.(*waitq).dequeue (inline)
826.43s 1.30% 52.66% 826.43s 1.30% crypto/aes.gcmAesEnc
789.16s 1.24% 53.90% 2709.70s 4.25% github.com/xtaci/kcp-go/v5.(*KCP).flush
617.51s 0.97% 54.87% 7628.15s 11.98% runtime.schedule
572.20s 0.9% 55.77% 2542.47s 3.99% gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/v2/common/turbotunnel.(*ClientMap).SendQueue
538.09s 0.84% 56.61% 538.09s 0.84% runtime.futex
513.24s 0.81% 57.42% 513.24s 0.81% runtime.procyield
445.32s 0.7% 58.12% 466.23s 0.73% runtime.findObject
```
It only accounts for 1% of total CPU time,
or 4% if you count the cumulative contribution of the mutex lock/unlock
and the operations on the inner map,
but it's still a hot spot.
In the branch https://gitlab.torproject.org/dcf/snowflake/-/commits/sendqueue-clientid/
I have a change to make `ClientMap` assume it is always dealing with a `ClientID`,
not just any `net.Addr`.
The change makes `ClientMap.SendQueue` faster in `BenchmarkSendQueue`.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40276Try reducing allocations in encapsulation.ReadData2023-11-21T04:19:58ZDavid Fifielddcf@torproject.orgTry reducing allocations in encapsulation.ReadDataIn the branch https://gitlab.torproject.org/dcf/snowflake/-/commits/encapsulation-readdata-buffer
(commit https://gitlab.torproject.org/dcf/snowflake/-/commit/9ac64239b4bff07cb016d7c2609eae66a92483c8)
I have a patch to make `encapsulatio...In the branch https://gitlab.torproject.org/dcf/snowflake/-/commits/encapsulation-readdata-buffer
(commit https://gitlab.torproject.org/dcf/snowflake/-/commit/9ac64239b4bff07cb016d7c2609eae66a92483c8)
I have a patch to make `encapsulation.ReadData` fill a provided buffer rather than allocate a new buffer on every call.
This function is part of the hot read loop and is probably respondible
for a large part of garbage collection pressure.
I am going to test this change on a production bridge to see if it helps.
/cc @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40275Bump verison of snowflake to v2.6.02023-06-20T18:48:10ZCecylia BocovichBump verison of snowflake to v2.6.0Let's do a library version bump and update the version shipped in Tor BrowserLet's do a library version bump and update the version shipped in Tor BrowserCecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40274AVG blocked connection of Chrome+Snowflake due to "URL:Blacklist" threat2023-06-21T09:28:53ZcypherpunksAVG blocked connection of Chrome+Snowflake due to "URL:Blacklist" threatSnowflake 0.7.2 on Chrome was running in the background and I got a pop-up alert from AVG telling me that it had blocked a connection to https://02.snowflake.torproject.net/?client_ip=xxx.xxx.xxx.xxx
The IP listed in the actual URL belo...Snowflake 0.7.2 on Chrome was running in the background and I got a pop-up alert from AVG telling me that it had blocked a connection to https://02.snowflake.torproject.net/?client_ip=xxx.xxx.xxx.xxx
The IP listed in the actual URL belongs to a Russian telco, according to a whois query.
I tried searching for information about this but google was not helpful.
I thought I'd leave a comment here just in case.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40273Automated Testing Script for Potential Domain Fronting Domains2023-10-25T15:42:23ZshelikhooAutomated Testing Script for Potential Domain Fronting DomainsIn order to make it easier to deal with blocking of domain fronting domains that is being observed in Iran, Russia, and possibly China, a [script](https://gist.github.com/xiaokangwang/3e6b7ca99a30b13f82ed5fe6bf8d8f43) was written to auto...In order to make it easier to deal with blocking of domain fronting domains that is being observed in Iran, Russia, and possibly China, a [script](https://gist.github.com/xiaokangwang/3e6b7ca99a30b13f82ed5fe6bf8d8f43) was written to automate the screening of domain name for domain fronting.
The following domain name was evaluated to be potentially useable for domain fronting. We should try to reduce solo reliance on cdn.sstatic.net with a fronting domain rotation system to reduce the chance any of them get blocked.
See also: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40068 , https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/123shelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40272go snowflake picks 'restricted' but stun-nat-behaviour picked endpoint indepe...2023-05-16T18:32:06ZRoger Dingledinego snowflake picks 'restricted' but stun-nat-behaviour picked endpoint independentOn my new Debian bookworm-rc2, I installed the stun-nat-behaviour testing tool:
```
$ go install github.com/pion/stun/cmd/stun-nat-behaviour@latest
go: downloading github.com/pion/stun v0.5.2
go: downloading github.com/pion/dtls/v2 v2.2....On my new Debian bookworm-rc2, I installed the stun-nat-behaviour testing tool:
```
$ go install github.com/pion/stun/cmd/stun-nat-behaviour@latest
go: downloading github.com/pion/stun v0.5.2
go: downloading github.com/pion/dtls/v2 v2.2.6
go: downloading github.com/pion/transport/v2 v2.2.0
go: downloading golang.org/x/crypto v0.5.0
go: downloading github.com/pion/udp/v2 v2.0.1
$ cd go/bin/
$ ./stun-nat-behaviour
INFO: 2023/05/13 15:45:02 connecting to STUN server: stun.voip.blackberry.com:3478
INFO: 2023/05/13 15:45:02 Local address: 0.0.0.0:33731
INFO: 2023/05/13 15:45:02 Remote address: 20.15.169.7:3478
INFO: 2023/05/13 15:45:02 Mapping Test I: Regular binding request
INFO: 2023/05/13 15:45:02 Sending to 20.15.169.7:3478: (20 bytes)
INFO: 2023/05/13 15:45:03 Response from 20.15.169.7:3478: (92 bytes)
INFO: 2023/05/13 15:45:03 Error: NAT discovery feature not supported by this server
WARNING: 2023/05/13 15:45:03 NAT mapping behavior: inconclusive
INFO: 2023/05/13 15:45:03 connecting to STUN server: stun.voip.blackberry.com:3478
INFO: 2023/05/13 15:45:03 Local address: 0.0.0.0:60922
INFO: 2023/05/13 15:45:03 Remote address: 20.15.169.7:3478
INFO: 2023/05/13 15:45:03 Filtering Test I: Regular binding request
INFO: 2023/05/13 15:45:03 Sending to 20.15.169.7:3478: (20 bytes)
INFO: 2023/05/13 15:45:03 Response from 20.15.169.7:3478: (92 bytes)
WARNING: 2023/05/13 15:45:03 Error: NAT discovery feature not supported by this server
WARNING: 2023/05/13 15:45:03 NAT filtering behavior: inconclusive
$ ./stun-nat-behaviour -server stun.voipgate.com:3478
INFO: 2023/05/13 15:46:13 connecting to STUN server: stun.voipgate.com:3478
INFO: 2023/05/13 15:46:13 Local address: 0.0.0.0:52168
INFO: 2023/05/13 15:46:13 Remote address: 185.125.180.70:3478
INFO: 2023/05/13 15:46:13 Mapping Test I: Regular binding request
INFO: 2023/05/13 15:46:13 Sending to 185.125.180.70:3478: (20 bytes)
INFO: 2023/05/13 15:46:13 Response from 185.125.180.70:3478: (100 bytes)
INFO: 2023/05/13 15:46:13 Received XOR-MAPPED-ADDRESS: 173.56.90.221:52168
INFO: 2023/05/13 15:46:13 Mapping Test II: Send binding request to the other address but primary port
INFO: 2023/05/13 15:46:13 Sending to 185.125.180.71:3478: (20 bytes)
INFO: 2023/05/13 15:46:13 Response from 185.125.180.71:3478: (100 bytes)
INFO: 2023/05/13 15:46:13 Received XOR-MAPPED-ADDRESS: 173.56.90.221:52168
WARNING: 2023/05/13 15:46:13 => NAT mapping behavior: endpoint independent
INFO: 2023/05/13 15:46:13 connecting to STUN server: stun.voipgate.com:3478
INFO: 2023/05/13 15:46:13 Local address: 0.0.0.0:55735
INFO: 2023/05/13 15:46:13 Remote address: 185.125.180.70:3478
INFO: 2023/05/13 15:46:13 Filtering Test I: Regular binding request
INFO: 2023/05/13 15:46:13 Sending to 185.125.180.70:3478: (20 bytes)
INFO: 2023/05/13 15:46:14 Response from 185.125.180.70:3478: (100 bytes)
INFO: 2023/05/13 15:46:14 Filtering Test II: Request to change both IP and port
INFO: 2023/05/13 15:46:14 Sending to 185.125.180.70:3478: (28 bytes)
INFO: 2023/05/13 15:46:14 Response from 185.125.180.71:3479: (100 bytes)
WARNING: 2023/05/13 15:46:14 => NAT filtering behavior: endpoint independent
```
which sure makes me think that I successfully put my laptop in dmz mode behind my router. (As another data point, I can telnet in to services on this laptop from the outside).
But then running headless snowflake, I have
```
2023/05/13 19:57:49 snowflake-proxy 2.4.1
023/05/13 19:57:49 Proxy starting
2023/05/13 19:57:49 WebRTC: Created offer
2023/05/13 19:57:49 WebRTC: Set local description
2023/05/13 19:57:54 Offer: {"type":"offer","sdp":"v=0\r\no=- 4036211437389836707 1684007869 IN IP4 [scrubbed]\r\ns=-\r\nt=0 0\r\na=fingerprint:sha-256 15:17:2D:41:43:C5:5D:D4:BD:B0:20:17:01:36:DB:0A:60:2B:E5:C4:A0:01:69:55:8F:77:71:17:72:F2:64:57\r\na=extmap-allow-mixed\r\na=group:BUNDLE 0\r\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 [scrubbed]\r\na=setup:actpass\r\na=mid:0\r\na=sendrecv\r\na=sctp-port:5000\r\na=ice-ufrag:TKUQqKORrkMnHIMw\r\na=ice-pwd:xLDVhUwJYEBnMGBjSgqOnluskGZjDVNg\r\na=candidate:1961522861 1 udp 2130706431 [scrubbed] 39181 typ host\r\na=candidate:1961522861 2 udp 2130706431 [scrubbed] 39181 typ host\r\na=end-of-candidates\r\n"}
2023/05/13 19:58:19 NAT Type measurement: unknown -> restricted = restricted
2023/05/13 19:58:19 NAT type: restricted
```
It sure looks like my snowflake is mistakenly calling me restricted when I'm not.
Did the newer go libs or something cause a regression where we think people like me are restricted?https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40269Snowflake web badge need 3rd-party cookies to run, which is unskilled in toda...2023-04-20T16:09:34ZsolidsSnowflake web badge need 3rd-party cookies to run, which is unskilled in today's most browsersToday most browser set the default setting to disable 3rd-party cookie, in Safari it's called "Prevent corss-site tracking", in Chromium-based browser it's called "Block third-party cookies". For example, the badge in website [relay.love...Today most browser set the default setting to disable 3rd-party cookie, in Safari it's called "Prevent corss-site tracking", in Chromium-based browser it's called "Block third-party cookies". For example, the badge in website [relay.love](https://relay.love) will show "Cookies are not enabled." in my browser and it's not possible to run without re-enabling 3rd-party cookies, which will allow tracking websites sneak in my privacy.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40265Standalone proxy - error polling probe - 'certificate is not standards compli...2023-03-16T17:40:24ZMarkCStandalone proxy - error polling probe - 'certificate is not standards compliant'@meskio As requested.
Yesterday I updated one of my proxies after !136 completed. When I run the proxy it starts, then begins probetest, then immediately (and repeatedly) returns an error:
2023/03/16 00:45:47 error polling probe: x509:...@meskio As requested.
Yesterday I updated one of my proxies after !136 completed. When I run the proxy it starts, then begins probetest, then immediately (and repeatedly) returns an error:
2023/03/16 00:45:47 error polling probe: x509: “snowflake-broker.torproject.net” certificate is not standards compliant
2023/03/16 00:45:47 NAT type: unknown
2023/03/16 00:45:48 error polling broker: x509: “snowflake-broker.torproject.net” certificate is not standards compliant
2023/03/16 00:45:48 Error reading broker response: unexpected end of JSON input
2023/03/16 00:45:48 body:
2023/03/16 00:45:48 bad offer from broker
Here’s the log of the start up (note that the git pull in this sequence says 'already up to date' because I ran it a second time as a double check).
[proxy_error_polling_broker_-_03_15_23.txt](/uploads/e8d343dc26e784d66e59880530374774/proxy_error_polling_broker_-_03_15_23.txt)
On the previous build everything was functioning properly.
@itchyonion Might this result be caused by the bug you mentioned in #40108 yesterday?itchyonionitchyonionhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40264Add option to configure private snowflake proxy2023-03-28T18:36:08ZRendezvousPointAdd option to configure private snowflake proxyJust like obfs4 private bridges.Just like obfs4 private bridges.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40262Deploy snowflake-server for QueuePacketConn buffer reuse fix (#40260)2023-07-18T00:47:30ZDavid Fifielddcf@torproject.orgDeploy snowflake-server for QueuePacketConn buffer reuse fix (#40260)#40260 describes a buffer reuse error in `QueuePacketConn` resulting from #40199.
!140 promises to fix it.
After merging !140, deploy to the bridges and restart.
- [x] snowflake-01
- [x] snowflake-02
/cc @cohosh @linus#40260 describes a buffer reuse error in `QueuePacketConn` resulting from #40199.
!140 promises to fix it.
After merging !140, deploy to the bridges and restart.
- [x] snowflake-01
- [x] snowflake-02
/cc @cohosh @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40260Weird KCP packets received by the client2023-09-25T19:36:14ZCecylia BocovichWeird KCP packets received by the clientWe were notified by @hazae41 of some suspicious looking kcp traffic traffic arriving at the client. This traffic looks like it belongs to KCP sessions that do not correspond to the client's own KCP sessions.
I'm in the process of trying...We were notified by @hazae41 of some suspicious looking kcp traffic traffic arriving at the client. This traffic looks like it belongs to KCP sessions that do not correspond to the client's own KCP sessions.
I'm in the process of trying to reproduce this. Here is the patch they used to see the contents of the KCP packets:
<details>
<summary> kcp library logging patch </summary>
```diff
diff --git a/readloop.go b/readloop.go
index 697395a..1d8e17b 100644
--- a/readloop.go
+++ b/readloop.go
@@ -1,6 +1,7 @@
package kcp
import (
+ "log"
"sync/atomic"
"github.com/pkg/errors"
@@ -11,6 +12,7 @@ func (s *UDPSession) defaultReadLoop() {
var src string
for {
if n, addr, err := s.conn.ReadFrom(buf); err == nil {
+ log.Println("kcp <- ", buf)
// make sure the packet is from the same source
if src == "" { // set source address
src = addr.String()
```
</details>
and the following smux patch:
<details>
<summary> smux library logging patch </summary>
```diff
diff --git a/session.go b/session.go
index bc56066..1fe1122 100644
--- a/session.go
+++ b/session.go
@@ -96,7 +96,7 @@ func newSession(config *Config, conn io.ReadWriteCloser, client bool) *Session {
s.chProtoError = make(chan struct{})
if client {
- s.nextStreamID = 1
+ s.nextStreamID = 5
} else {
s.nextStreamID = 0
}
```
</details>
They report seeing smux headers that have a streamid of 3 regardless of the above change that starts stream ids at 5, and seeing kcp headers that do not contain recognizable kcp conv ids followed by a stream id of 3 and a TLS application packet (despite the fact that new KCP sessions should always contain a TLS handshake first since they represent a new OR conn).Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40259Make nf_conntrack changes persistent2023-04-25T15:13:40ZDavid Fifielddcf@torproject.orgMake nf_conntrack changes persistentInvestigating the [loss of users from Iran](https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/115#note_2883298)
and the [increase of users from Russia that only affected snowflake-02](https://gitlab.torproject.org/tpo/anti-...Investigating the [loss of users from Iran](https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/115#note_2883298)
and the [increase of users from Russia that only affected snowflake-02](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/merge_requests/131#note_2883300)
that happened in the second half of February 2023,
I discovered that the [increased nf_conntrack table size](https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Guides/Snowflake-Bridge-Installation-Guide?version_id=a7bcf0502102bd8d820f590bc9856a0a738a2225#general-system-setup)
from #40239 had not taken effect since
the bridges were rebooted on 2023-02-16 for #40253.
```
root@snowflake-01:~# date -u --iso=sec
2023-03-04T07:13:32+00:00
root@snowflake-01:~# cat /proc/sys/net/netfilter/nf_conntrack_{count,max,buckets}
186671
262144
65536
```
```
root@snowflake-02:~# date -u --iso=sec
2023-03-04T07:36:45+00:00
root@snowflake-02:~# cat /proc/sys/net/netfilter/nf_conntrack_{count,max,buckets}
21155
262144
65536
```
The default settings were supposed to be overridden
by /etc/sysctl.d/nf_conntrack.conf, which is loaded during boot by sysctl:
```
root@snowflake-01:~# cat /etc/sysctl.d/nf_conntrack.conf
net.netfilter.nf_conntrack_max = 524288
net.netfilter.nf_conntrack_buckets = 524288
```
I think the problem is that the sysctl settings are made
before the nf_conntrack module is first loaded, so they have no effect,
as the [sysctl.d(5)](https://manpages.debian.org/bullseye/systemd/sysctl.d.5.en.html#CONFIGURATION_FORMAT)
man page cautions:
> Many sysctl parameters only become available when certain kernel modules are loaded. Modules are usually loaded on demand, e.g. when certain hardware is plugged in or network brought up. This means that [systemd-sysctl.service(8)](https://manpages.debian.org/bullseye/systemd/systemd-sysctl.service.8.en.html) which runs during early boot will not configure such parameters if they become available after it has run. To set such parameters, it is recommended to add an [udev(7)](https://manpages.debian.org/bullseye/udev/udev.7.en.html) rule to set those parameters when they become available. Alternatively, a slightly simpler and less efficient option is to add the module to [modules-load.d(5)](https://manpages.debian.org/bullseye/systemd/modules-load.d.5.en.html), causing it to be loaded statically before sysctl settings are applied (see example below).
The workaround, as the man page suggests,
is to add `nf_conntrack` to /etc/modules to make it be loaded earlier in the boot sequence,
or add a udev rule to re-load the `net.netfilter` sysctl settings
when nf_conntrack is loaded.
* https://github.com/systemd/systemd/issues/1113#issuecomment-138051408
* https://serverfault.com/a/676721
* https://github.com/coreos/bugs/issues/785
I've re-started the [conntrack.sh script](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40239#note_2864515)
to track `nf_conntrack_count` over time.
I will let it run for a couple of days to get a baseline,
then I will increase the nf_conntrack limits again
to see if it has an effect on usership.
Then, I will try one of the above workarounds to make the change actually persistent this time.
/cc @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40258Resynchronization with Upsteamed Remove HelloVerify countermeasure2023-04-06T13:05:39ZshelikhooResynchronization with Upsteamed Remove HelloVerify countermeasureAs of https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40249, we have finished upstreaming Remove HelloVerify countermeasure changes to upstream.
We should try to resynchronize our dependency wit...As of https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40249, we have finished upstreaming Remove HelloVerify countermeasure changes to upstream.
We should try to resynchronize our dependency with upstream.shelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40255snowflake-02: increase number of tor instances from 4 to 122023-02-17T03:15:20ZDavid Fifielddcf@torproject.orgsnowflake-02: increase number of tor instances from 4 to 12The idea is that snowflake-02 will soon be handling a roughly equal amount of traffic as snowflake-01, which currently runs 12 instances.The idea is that snowflake-02 will soon be handling a roughly equal amount of traffic as snowflake-01, which currently runs 12 instances.David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40253Restart snowflake bridges for haproxy CVE-2023-0056, CVE-2023-257252023-05-25T15:04:32ZDavid Fifielddcf@torproject.orgRestart snowflake bridges for haproxy CVE-2023-0056, CVE-2023-25725https://security-tracker.debian.org/tracker/DSA-5348-1
https://lists.debian.org/debian-security-announce/2023/msg00037.html
> Two vulnerabilities were discovered in HAProxy, a fast and reliable load
> balancing reverse proxy, which may ...https://security-tracker.debian.org/tracker/DSA-5348-1
https://lists.debian.org/debian-security-announce/2023/msg00037.html
> Two vulnerabilities were discovered in HAProxy, a fast and reliable load
> balancing reverse proxy, which may result in denial of service, or
> bypass of access controls and routing rules via specially crafted
> requests.
>
> For the stable distribution (bullseye), these problems have been fixed in
> version 2.2.9-2+deb11u4.
>
> We recommend that you upgrade your haproxy packages.
https://ubuntu.com/security/notices/USN-5869-1
> HAProxy could allow unintended access to network services.
>
> The problem can be corrected by updating your system to the following package versions:
>
> Ubuntu 20.04
>
> [haproxy - 2.0.29-0ubuntu1.3](https://launchpad.net/ubuntu/+source/haproxy/2.0.29-0ubuntu1.3)
- [x] snowflake-01
- [x] snowflake-02
/cc @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40252Standalone proxy probetest Wiki inconsistency and implementation details2023-03-07T18:13:49ZitchyonionStandalone proxy probetest Wiki inconsistency and implementation details1. Our [Wiki](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/wikis/NAT-matching#determining-nat-behaviour) states that:
> We determine the NAT behaviour of clients by using the tricks in [RFC 5780](ht...1. Our [Wiki](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/wikis/NAT-matching#determining-nat-behaviour) states that:
> We determine the NAT behaviour of clients by using the tricks in [RFC 5780](https://tools.ietf.org/html/rfc5780) ... For standalone proxies written in Go, we use the same method.
Which is not true since we switched to probetest https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/commit/00f8f85f412878c2066fcb5d3f4739e50912a925
Cecylia linked the issue for the change: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40013. I will update the Wiki.
2. Right now we are [logging the SDP offer](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/main/proxy/lib/snowflake.go#L676) used for probetest, which might be misleading because before looking into this in detail I always thought we were logging the SDP candidates used for WebRTC connection. Does this help the user in any way or is simplying logging the resultant NAT type enough? Some options to consider:
- keep the same logging, but made it extra clear that this is only used for probetest, not peer connection
- log SDP candidates for WebRTC. I think this would be helpful for debugging, but could produce much more logs
- log both
- [x] Update Wiki with reason to use probetest
- [x] Research on whether we should respect proxy options in probetest
- [x] Decide what to logitchyonionitchyonionhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40249Upstreaming Remove HelloVerify countermeasure2023-03-03T11:45:36ZshelikhooUpstreaming Remove HelloVerify countermeasureWe have already released a version of snowflake with [Remove HelloVerify](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/merge_requests/134) [Orig](https://gitlab.torproject.org/tpo/anti-censorship/plu...We have already released a version of snowflake with [Remove HelloVerify](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/merge_requests/134) [Orig](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/merge_requests/131) countermeasure against Russia's censorship of snowflake. However, as we are currently applying the changes as a series of patch in the forked repo, this situation isn't ideal if that means we need to constantly rebase and maintain this fork.
Thus, we are currently seeking to upstream this change.
Changes to be upstreamed:
1. [DTLS](https://github.com/xiaokangwang/dtls/tree/dev-skiphelloverify) -> [Github Pull Request](https://github.com/pion/dtls/pull/513)
2. [WebRTC](https://github.com/xiaokangwang/webrtc/tree/dev-skiphelloverify) -> [Github Pull Request](https://github.com/pion/webrtc/pull/2407)
See Also: https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/merge_requests/637shelikhooshelikhoo