Snowflake issueshttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues2022-07-09T04:20:15Zhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40135Cannot build snowflake/proxy2022-07-09T04:20:15ZcypherpunksCannot build snowflake/proxyHello! I want to install the snowflake proxy on my raspberry pi, but I can't seem to build it. I updated my system via apt-get update/update succesfully and the installation of git and golang were also successful. But after cloning the ...Hello! I want to install the snowflake proxy on my raspberry pi, but I can't seem to build it. I updated my system via apt-get update/update succesfully and the installation of git and golang were also successful. But after cloning the git, I went to /home/pi/snowflake/proxy and when I enter go build the system gives me this:
```
pi@raspberrypi:~/snowflake/proxy $ go build
main.go:5:2: cannot find package "git.torproject.org/pluggable-transports/snowflake.git/v2/common/event" in any of:
/usr/lib/go-1.7/src/git.torproject.org/pluggable-transports/snowflake.git/v2/common/event (from $GOROOT)
($GOPATH not set)
main.go:12:2: cannot find package "git.torproject.org/pluggable-transports/snowflake.git/v2/common/safelog" in any of:
/usr/lib/go-1.7/src/git.torproject.org/pluggable-transports/snowflake.git/v2/common/safelog (from $GOROOT)
($GOPATH not set)
main.go:13:2: cannot find package "git.torproject.org/pluggable-transports/snowflake.git/v2/proxy/lib" in any of:
/usr/lib/go-1.7/src/git.torproject.org/pluggable-transports/snowflake.git/v2/proxy/lib (from $GOROOT)
($GOPATH not set)
```
What is the problem here, and how can I fix it? Whe I look into /usr/lib/go-1.7/src/ there is no git.torproject.org subdirectory
I hope, you can help me.itchyonionitchyonionhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40103mix of past and present in snowflake proxy log2022-07-09T04:20:15Ztoralfmix of past and present in snowflake proxy log"In the last 1h0m0s, there are 28 connections. Traffic Relayed ↑ 273 MB, ↓ 273 MB."
are -> were
or ?"In the last 1h0m0s, there are 28 connections. Traffic Relayed ↑ 273 MB, ↓ 273 MB."
are -> were
or ?https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40098snowflake_server.httpHandler.ln is not initialized, leading to panic in onesh...2022-07-09T04:20:16ZDavid Fifielddcf@torproject.orgsnowflake_server.httpHandler.ln is not initialized, leading to panic in oneshotModeThere are a few of these in the snowflake-server log:
```
2022/02/01 16:44:11 http: panic serving [scrubbed]: runtime error: invalid memory address or nil pointer dereference
goroutine 2513699 [running]:
net/http.(*conn).serve.func1(0xc...There are a few of these in the snowflake-server log:
```
2022/02/01 16:44:11 http: panic serving [scrubbed]: runtime error: invalid memory address or nil pointer dereference
goroutine 2513699 [running]:
net/http.(*conn).serve.func1(0xc007576dc0)
/usr/lib/go-1.15/src/net/http/server.go:1801 +0x147
panic(0x7ea7a0, 0xb0c450)
/usr/lib/go-1.15/src/runtime/panic.go:975 +0x47a
git.torproject.org/pluggable-transports/snowflake.git/v2/server/lib.(*SnowflakeListener).queueConn(0x0, 0x8dc820, 0xc04b943740, 0xc03557fbb0, 0xc04b9436e0)
./snowflake/server/lib/snowflake.go:275 +0x37
git.torproject.org/pluggable-transports/snowflake.git/v2/server/lib.oneshotMode(...)
./snowflake/server/lib/http.go:117
git.torproject.org/pluggable-transports/snowflake.git/v2/server/lib.(*httpHandler).ServeHTTP(0xc0001a0e80, 0x8d9800, 0xc057a75880, 0xc0dda2cd00)
./snowflake/server/lib/http.go:105 +0x5b7
net/http.serverHandler.ServeHTTP(0xc0001ee0e0, 0x8d9800, 0xc057a75880, 0xc0dda2cd00)
/usr/lib/go-1.15/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc007576dc0, 0x8da1c0, 0xc013b86b00)
/usr/lib/go-1.15/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
/usr/lib/go-1.15/src/net/http/server.go:2969 +0x36c
```
`oneshotMode` is called with `httpHandler.ln` as the `*SnowflakeListener` argument:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/e828b0607662c7325f4d1bbf4c5072ce81f38fb9/server/lib/http.go#L100-105
```go
// We didn't find a matching token, which means that we are
// dealing with a client that doesn't know about such things.
// "Unread" the token by constructing a new Reader and pass it
// to the old one-session-per-WebSocket mode.
conn2 := &overrideReadConn{Conn: conn, Reader: io.MultiReader(bytes.NewReader(token[:]), conn)}
err = oneshotMode(conn2, addr, handler.ln)
```
`httpHandler` is:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/e828b0607662c7325f4d1bbf4c5072ce81f38fb9/server/lib/http.go#L63-68
```go
type httpHandler struct {
// pconn is the adapter layer between stream-oriented WebSocket
// connections and the packet-oriented KCP layer.
pconn *turbotunnel.QueuePacketConn
ln *SnowflakeListener
}
```
But in the only place `httpHandler` is instantiated, its `ln` field is left uninitialized:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/e828b0607662c7325f4d1bbf4c5072ce81f38fb9/server/lib/snowflake.go#L80-85
```go
handler := httpHandler{
// pconn is shared among all connections to this server. It
// overlays packet-based client sessions on top of ephemeral
// WebSocket connections.
pconn: turbotunnel.NewQueuePacketConn(addr, clientMapTimeout),
}
```Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40088Snowflake server keeps failing unexpectedly2022-07-09T04:20:46ZCecylia BocovichSnowflake server keeps failing unexpectedlyThe snowflake server has failed twice now in the last two weeks.
[After the first failure](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40060#note_2767868), we upgraded the capacity of the Sn...The snowflake server has failed twice now in the last two weeks.
[After the first failure](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40060#note_2767868), we upgraded the capacity of the Snowflake bridge (#40085).
Today when it failed we received an alert from monit: https://lists.torproject.org/pipermail/anti-censorship-alerts/2021-December/000786.html
Unfortunately, in my rush to fix it I didn't check the CPU usage :/ let's use this ticket to track the issue and note what we notice the next time it occurs.Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetshelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40058Snowflake embed refuses to work on Chromium?2022-07-09T04:20:16ZkoutsieSnowflake embed refuses to work on Chromium?Snowflake (embed) simply refuses to enable on Chromium and Chrome.
At first i thought it was one of my extensions but a couple of friends reported the inability to enable Snowflake's embed while stating that the web extension version wor...Snowflake (embed) simply refuses to enable on Chromium and Chrome.
At first i thought it was one of my extensions but a couple of friends reported the inability to enable Snowflake's embed while stating that the web extension version works fine.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/34435Update bug-reporting links for gitlab2021-07-09T18:26:25ZDavid Fifielddcf@torproject.orgUpdate bug-reporting links for gitlabAfter the [gitlab migration](https://lists.torproject.org/pipermail/tor-project/2020-June/002866.html) we need to update bug-reporting instructions.
* https://snowflake.torproject.org/#bugs<br>
https://gitweb.torproject.org/pluggabl...After the [gitlab migration](https://lists.torproject.org/pipermail/tor-project/2020-June/002866.html) we need to update bug-reporting instructions.
* https://snowflake.torproject.org/#bugs<br>
https://gitweb.torproject.org/pluggable-transports/snowflake-webext.git/tree/static/index.html?h=webext-0.3.1&id=d2a9a8fd136ac6bbba393de3d51fb7ca85e17b8a#n75
* https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/wikis/home#reporting-bugs
I checked in snowflake.git and did not find anything that needs to be changed.Gabagaba@torproject.orgGabagaba@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/34147Remove redundant languages from snowflake page2021-07-09T18:26:26ZRoger DingledineRemove redundant languages from snowflake pagehttps://snowflake.torproject.org/
offers me en, en_GB, and en_US in the language drop-down menu. We should have one English, not three.
More generally, I bet we have a policy somewhere about which languages we want to be sure to include...https://snowflake.torproject.org/
offers me en, en_GB, and en_US in the language drop-down menu. We should have one English, not three.
More generally, I bet we have a policy somewhere about which languages we want to be sure to include when we have translated something, and we should see if we're missing any of those; and also I hope we have a policy about which languages to *not* include (and ideally we should go back to transifex and remove those from the set that people can translate, to avoid giving people the impression that they will be doing something useful if they e.g. translate en_US to en_GB).
I'm sorry I don't know more -- I bet Emma or Antonela or Gus will know more about whether we have languages policies and habits in place. :)https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/34080Avoid double delays from ReconnectTimeout2022-07-09T04:20:16ZDavid Fifielddcf@torproject.orgAvoid double delays from ReconnectTimeout[ReconnectTimeout](https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/client/lib/snowflake.go?id=72cfb96edeb7c9a3c93d38539bc31a51e30dbe8d#n18) is used in 2 places:
* In [exchangeSDP](https://gitweb.torproject.org/plug...[ReconnectTimeout](https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/client/lib/snowflake.go?id=72cfb96edeb7c9a3c93d38539bc31a51e30dbe8d#n18) is used in 2 places:
* In [exchangeSDP](https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/client/lib/webrtc.go?id=72cfb96edeb7c9a3c93d38539bc31a51e30dbe8d#n223), where it is a delay inserted between calls to `broker.Negotiate` until one of them succeeds.
`Failed to retrieve answer. Retrying in 10s`
* In the main [ConnectLoop](https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/client/snowflake.go?id=72cfb96edeb7c9a3c93d38539bc31a51e30dbe8d#n27), where it is a delay inserted between every check for getting a new snowflake.
```
WebRTC: <errmsg> Retrying in 10s...
```
The broker itself also terminates requests after 10s when the chosen proxy doesn't respond: `BrokerChannel Response: 504 Gateway Timeout`.
This situation sometimes results in double delays. Here are two cases I've identified.
* The client requests a proxy, the broker responds immediately with an answer, but the proxy doesn't work. After waiting the `DataChannelTimeout` to decide that the proxy doesn't work, the client waits an _additional_ `ReconnectTimeout` in `ConnectLoop`.
Here, I've set `DataChannelTimeout` to 10s. Notice that between `DataChannel created` and `Collecting a new Snowflake` there are 20s (which is `DataChannelTimeout` + `ReconnectTimeout`), when it really should only be 10s.
```
2020/04/30 22:38:29 Received Answer.
2020/04/30 22:38:29 WebRTC: DataChannel created.
2020/04/30 22:38:39 establishDataChannel: timeout waiting for DataChannel.OnOpen
2020/04/30 22:38:39 WebRTC: closing PeerConnection
2020/04/30 22:38:39 WebRTC: Closing
2020/04/30 22:38:39 WebRTC: WebRTC: Could not establish DataChannel Retrying in 10s...
2020/04/30 22:38:49 WebRTC: Collecting a new Snowflake. Currently at [0/1]
```
* The client requests a proxy, and the broker waits for 10s to respond with a 504 Gateway Timeout (indicating that the chosen proxy did not return an answer to the broker in time). The client waits 10s for the broker to respond, then waits another `ReconnectTimeout` in exchangeSDP before trying the broker again.
```
2020/04/30 22:39:30 Negotiating via BrokerChannel...
2020/04/30 22:39:41 BrokerChannel Response: 504 Gateway Timeout
2020/04/30 22:39:41 BrokerChannel Error: Unexpected error, no answer.
2020/04/30 22:39:41 Failed to retrieve answer. Retrying in 10s
2020/04/30 22:39:51 Negotiating via BrokerChannel...
```
Both these cases can probably be fixed by running the timer in parallel with the periodic operation they are rate limiting. That is, instead of
```
for {
operation()
<-time.After(ReconnectTimeout)
}
```
it can be
```
for {
timer := time.After(ReconnectTimeout)
operation()
<-timer
}
```
That way, if the operation itself takes more than 10s, `ReconnectTimeout` doesn't impose any additional delay.Sponsor 28: Reliable Anonymous Communication Evading Censors and Repressors (RACECAR)Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/33744Remove local LAN address ICE candidates from JS proxy answer2021-07-09T18:26:25ZArlo BreaultRemove local LAN address ICE candidates from JS proxy answerThis is a follow up from legacy/trac#19026 where it was done for the clients and golang proxies.This is a follow up from legacy/trac#19026 where it was done for the clients and golang proxies.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/33519Support multiple simultaneous SOCKS connections2022-03-14T05:36:27ZDavid Fifielddcf@torproject.orgSupport multiple simultaneous SOCKS connectionsThe Snowflake client accepts multiple simultaneous SOCKS connections from tor, but it only tries to collect one proxy at a time, and each proxy can service only one SOCKS connection (this is true in the turbotunnel branch as well). One o...The Snowflake client accepts multiple simultaneous SOCKS connections from tor, but it only tries to collect one proxy at a time, and each proxy can service only one SOCKS connection (this is true in the turbotunnel branch as well). One of the SOCKS connections gets the only available proxy, while the others starve.
I can think of a few ways to approach this.
1. Dynamically adjust the `max` parameter according to how many SOCKS connections there are currently. If there's one SOCKS connection, we need only one proxy. If there's another SOCKS connection, raise the limit to allow the proxy-collecting thread to pick up another one, and lower the limit again if the number of SOCKS connections drops back down.
2. Start up a separate proxy-collecting thread for each SOCKS connection, as suggested at comment:12:ticket:21314. Each SOCKS connection will make its own broker requests and collect its own proxies, not interacting with those of any other SOCKS connection. A downside of this is that the number of Snowflake proxies you are contacting leaks the number of SOCKS connections you have ongoing. (Which can also be seen as a benefit in that if there are zero SOCKS connections, you don't even bother to contact the broker.)
3. Make it possible for multiple SOCKS connections to share the same proxy. Continue using a global proxy-collecting thread, and make there be a single shared `RedialPacketConn` instead of a separate one for each SOCKS connection. As things work now, this would require tagging _every packet_ with the ClientID, instead of [sending the ClientID once](https://gitweb.torproject.org/user/dcf/snowflake.git/tree/client/lib/snowflake.go?h=turbotunnel&id=47312dd1eccc8456652853bd66f8ed396e9ba6ec#n52) and letting it be the same implicitly for all packets that follow.
4. Make it possible for multiple SOCKS connections to share the same proxy, and use a single KCP/QUIC connection for all SOCKS connections. Separate SOCKS connections go into separate streams within the KCP/QUIC connection. In other words, rather than doing both `sess = kcp.NewConn2/quic.Dial` and `sess.OpenStream` in the SOCKS handler, we do `sess = kcp.NewConn2/quic.Dial` in `main` and then `sess.OpenStream` in the SOCKS handler. This way we could continue tagging the ClientID just once, because the program would only ever work with one ClientID at a time. However this way would make it harder to do the "stop using the network when not being used" of legacy/trac#21314, because that single KCP/QUIC connection would try to keep itself alive all the time and would contact the broker every time it needed a new proxy. Perhaps we could make it so that if there are zero streams, we close the KCP/QUIC connection, and lazily create a new one if and when we get another SOCKS connection.
| |= status quo=|= 1=|= 2=|= 3=|= 4=|
|---------------------------|--------------|--------------|--------------|--------------|--------------|
|= proxy-collecting threads=| one global| one global| one per SOCKS| one global| one global|
|= proxy limit per thread=| 1| # of SOCKS| 1| 1| 1|
|= proxies shared between SOCKSes?=| dedicated| dedicated| dedicated| shared| shared|
|= `PacketConn`s=| one per SOCKS| one per SOCKS| one per SOCKS| one global| one global|
|= KCP/QUIC connections=| one per SOCKS| one per SOCKS| one per SOCKS| one per SOCKS| one global|
|= KCP/QUIC streams=| one per SOCKS| one per SOCKS| one per SOCKS| one per SOCKS| one per SOCKS|
|= ClientID on every packet?=| no| no| no| yes| no|https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/33401turbotunnel-quic snowflake-client occasionally uses lots of CPU2022-07-09T04:20:15ZDavid Fifielddcf@torproject.orgturbotunnel-quic snowflake-client occasionally uses lots of CPUAs originally noted at comment:7:ticket:33211, the quic-go turbotunnel client sometimes uses 100+% CPU for a few minutes before returning to normal operation. It is specific to the quic-go implementation; it doesn't happen with the kcp-g...As originally noted at comment:7:ticket:33211, the quic-go turbotunnel client sometimes uses 100+% CPU for a few minutes before returning to normal operation. It is specific to the quic-go implementation; it doesn't happen with the kcp-go implementation nor the non-turbotunnel client.
As best I can figure, the cause has something to do with timers created under `(*session) maybeResetTimer`.
[profile001.png](None/profile001.png)David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/33367Goroutine leak in websocketconn2022-07-09T04:20:16ZDavid Fifielddcf@torproject.orgGoroutine leak in websocketconnThinking about legacy/trac#33364, I found that snowflake-server is chewing a lot of memory. It may be some memory leak or something.
```
$ top -o%MEM
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26910 debi...Thinking about legacy/trac#33364, I found that snowflake-server is chewing a lot of memory. It may be some memory leak or something.
```
$ top -o%MEM
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26910 debian-+ 20 0 1916628 1.522g 0 S 0.0 77.8 58:51.37 snowflake-serve
```
The memory use seems to be inhibiting other processes. `runsvdir` puts status messages in its own `argv` so you can inspect them with `ps`. Currently it's reflecting `xz` not being able to allocate memory to compress logs:
```
$ ps ax | grep runsvdir
1358 ? Ss 94:01 runsvdir -P /etc/service log: locate memory \
svlogd: warning: processor failed, restart: /home/snowflake-proxy/snowflake-proxy-standalone-17h.log.d xz: (stdin): Cannot allocate memory \
svlogd: warning: processor failed, restart: /home/snowflake-proxy/snowflake-proxy-standalone-17h.log.d xz: (stdin): Cannot allocate memory \
svlogd: warning: processor failed, restart: /home/snowflake-proxy/snowflake-proxy-standalone-17h.log.d
```
I even got it just now trying to run a diagnostic command (it doesn't always happen):
```
$ ps ax | grep standal
-bash: fork: Cannot allocate memory
```
In the short term, looks like we need to restart the server. Then we need to figure out what's causing it to use so much memory.
The server was last restarted 2020-02-10 18:57 (one week ago) at [ca9ae12c383405bc9a755e1bc902e9755495c1f1](https://gitweb.torproject.org/pluggable-transports/snowflake.git/log/?id=ca9ae12c383405bc9a755e1bc902e9755495c1f1) for legacy/trac#32964.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/33365Probe Snowflake bridge from proxy 1x a day2021-07-09T18:26:25ZCecylia BocovichProbe Snowflake bridge from proxy 1x a dayWe're getting reports that the Snowflake bridge isn't reachable in legacy/trac#33364, but it's taking awhile for volunteers to notice because the probe check only happens once at installation or if you disable/enable the proxy.
Perhaps ...We're getting reports that the Snowflake bridge isn't reachable in legacy/trac#33364, but it's taking awhile for volunteers to notice because the probe check only happens once at installation or if you disable/enable the proxy.
Perhaps we can do the probe check 1x a day (e.g., when we do the stats refresh)?Arlo BreaultArlo Breaulthttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/33211proxy-go sometimes gets into a 100+% CPU state2021-07-09T18:26:26ZDavid Fifielddcf@torproject.orgproxy-go sometimes gets into a 100+% CPU stateproxy-go sometimes works itself into a state where it is still running and working, but using more than 100% CPU. I have had it happen locally a couple of times while testing turbotunnel stuff, and it's currently happening with proxy-go-...proxy-go sometimes works itself into a state where it is still running and working, but using more than 100% CPU. I have had it happen locally a couple of times while testing turbotunnel stuff, and it's currently happening with proxy-go-restartless:
```
$ top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13844 snowfla+ 20 0 551292 320692 8844 R 161.1 15.6 129356:18 proxy-go
```
Or looking at single threads:
```
$ top -H
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24628 snowfla+ 20 0 551292 320692 8844 R 39.7 15.6 15219:01 proxy-go
13844 snowfla+ 20 0 551292 320692 8844 R 35.4 15.6 15431:52 proxy-go
1637 snowfla+ 20 0 551292 320692 8844 R 34.8 15.6 16057:40 proxy-go
13848 snowfla+ 20 0 551292 320692 8844 S 27.5 15.6 13669:02 proxy-go
13846 snowfla+ 20 0 551292 320692 8844 S 22.5 15.6 17021:57 proxy-go
```
I caught it once and attached to the process with GDB, but didn't know what to make of it. `thread apply all bt` seemed to show all the threads being somewhere in the Go runtime; the thread that wasn't was not one of the threads using a lot of CPU. (Matching up the `PID` field from `top -H` with the `LWP` identifiers in gdb.)
I had the idea to make proxy-go emit profiling output, and then exmine the call chain that was resulting the in the most CPU using [profiling tools](https://blog.golang.org/profiling-go-programs). A patch to do that is [proxy-go-profile.patch.](None/proxy-go-profile.patch.) But I haven't been able to reproduce the high CPU usage yet.Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/33112snowflake-webextension "Could not connect to the bridge."2021-07-09T18:26:25Zcypherpunkssnowflake-webextension "Could not connect to the bridge."My snowflake webextension has been working well up to this point but recently started having the error "Could not connect to the bridge." for the past few days. I'm not sure if this is something on my end or something with Snowflake.My snowflake webextension has been working well up to this point but recently started having the error "Could not connect to the bridge." for the past few days. I'm not sure if this is something on my end or something with Snowflake.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/32576Fix race condition in snowflake broker2022-07-09T04:20:15ZCecylia BocovichFix race condition in snowflake brokerThere is a race condition with the snowflake heap that has been causing the broker to crash several times a day. This race condition has existed in the broker for several years, but some recent updates as well as the host migration manag...There is a race condition with the snowflake heap that has been causing the broker to crash several times a day. This race condition has existed in the broker for several years, but some recent updates as well as the host migration managed to shake it loose.
----
This race condition is causing the snowflake broker to crash repeatedly and often since the migration. We noticed because CollecTor stopped collecting metrics since the restart on 14 November 2019.Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/31902Add a short FAQ to snowflake.tp.o2021-07-09T18:26:26ZArlo BreaultAdd a short FAQ to snowflake.tp.oThis should include explanations for the missing feature error messages. See comment:13:ticket:31391This should include explanations for the missing feature error messages. See comment:13:ticket:31391https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/31425Snowflake broker is sluggish and sometimes fails2022-07-09T04:20:15ZCecylia BocovichSnowflake broker is sluggish and sometimes failsThis morning while trying to deploy a change I noticed that my SSH connection is extremely sluggish.
I'm also getting a lot of timeouts at the client:
```
2019/08/16 15:37:54 Negotiating via BrokerChannel...
Target URL: snowflake-broke...This morning while trying to deploy a change I noticed that my SSH connection is extremely sluggish.
I'm also getting a lot of timeouts at the client:
```
2019/08/16 15:37:54 Negotiating via BrokerChannel...
Target URL: snowflake-broker.azureedge.net
Front URL: ajax.aspnetcdn.com
2019/08/16 15:38:05 BrokerChannel Response:
504 Gateway Timeout
2019/08/16 15:38:05 BrokerChannel Error: Unexpected error, no answer.
2019/08/16 15:38:05 Failed to retrieve answer. Retrying in 10 seconds
```
Some requests go through, but the timeouts seem to be occurring more frequently. At first glance, this doesn't seem to be a CPU or memory resources consumption problem. Maybe there's a data transfer limit we're hitting?David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/31407Create a broker spec for metrics collection2020-07-14T22:23:11ZCecylia BocovichCreate a broker spec for metrics collectionWe're in the process of creating a module for the collection of snowflake metrics (legacy/trac#21315, legacy/trac#29461). We need a better place to put the spec for the metrics data output by the snowflake broker than a comment in the so...We're in the process of creating a module for the collection of snowflake metrics (legacy/trac#21315, legacy/trac#29461). We need a better place to put the spec for the metrics data output by the snowflake broker than a comment in the source code (see https://trac.torproject.org/projects/tor/ticket/29461#comment:5)
A spec for the broker will also be useful to expand upon later to specify how the broker interacts with other pieces of either Snowflake or the Tor ecosystem in the case that the broker assumes more of the responsibilities of BridgeDB.Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/31391Do a reachability check before polling for clients2023-11-12T02:27:25ZcypherpunksDo a reachability check before polling for clientsI'm looking at https://trac.torproject.org/projects/tor/attachment/ticket/29461/example_metrics.log and I've seen one `TR=2` and another `TR=5` record. I'm not aware of the current situation in TR but as far as I remember Tor was censore...I'm looking at https://trac.torproject.org/projects/tor/attachment/ticket/29461/example_metrics.log and I've seen one `TR=2` and another `TR=5` record. I'm not aware of the current situation in TR but as far as I remember Tor was censored there (it is even stated in the Torlauncher).
We can have a snowflake do a reachability check of the bridge(s) and broker (and maybe Tor domains/dirauths) before trying to poll for clients. If this check fails, we should display a user-facing error message or warning and hold off on polling while the bridge is unreachableArlo BreaultArlo Breault