Snowflake issueshttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues2022-07-09T04:20:16Zhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/34080Avoid double delays from ReconnectTimeout2022-07-09T04:20:16ZDavid Fifielddcf@torproject.orgAvoid double delays from ReconnectTimeout[ReconnectTimeout](https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/client/lib/snowflake.go?id=72cfb96edeb7c9a3c93d38539bc31a51e30dbe8d#n18) is used in 2 places:
* In [exchangeSDP](https://gitweb.torproject.org/plug...[ReconnectTimeout](https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/client/lib/snowflake.go?id=72cfb96edeb7c9a3c93d38539bc31a51e30dbe8d#n18) is used in 2 places:
* In [exchangeSDP](https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/client/lib/webrtc.go?id=72cfb96edeb7c9a3c93d38539bc31a51e30dbe8d#n223), where it is a delay inserted between calls to `broker.Negotiate` until one of them succeeds.
`Failed to retrieve answer. Retrying in 10s`
* In the main [ConnectLoop](https://gitweb.torproject.org/pluggable-transports/snowflake.git/tree/client/snowflake.go?id=72cfb96edeb7c9a3c93d38539bc31a51e30dbe8d#n27), where it is a delay inserted between every check for getting a new snowflake.
```
WebRTC: <errmsg> Retrying in 10s...
```
The broker itself also terminates requests after 10s when the chosen proxy doesn't respond: `BrokerChannel Response: 504 Gateway Timeout`.
This situation sometimes results in double delays. Here are two cases I've identified.
* The client requests a proxy, the broker responds immediately with an answer, but the proxy doesn't work. After waiting the `DataChannelTimeout` to decide that the proxy doesn't work, the client waits an _additional_ `ReconnectTimeout` in `ConnectLoop`.
Here, I've set `DataChannelTimeout` to 10s. Notice that between `DataChannel created` and `Collecting a new Snowflake` there are 20s (which is `DataChannelTimeout` + `ReconnectTimeout`), when it really should only be 10s.
```
2020/04/30 22:38:29 Received Answer.
2020/04/30 22:38:29 WebRTC: DataChannel created.
2020/04/30 22:38:39 establishDataChannel: timeout waiting for DataChannel.OnOpen
2020/04/30 22:38:39 WebRTC: closing PeerConnection
2020/04/30 22:38:39 WebRTC: Closing
2020/04/30 22:38:39 WebRTC: WebRTC: Could not establish DataChannel Retrying in 10s...
2020/04/30 22:38:49 WebRTC: Collecting a new Snowflake. Currently at [0/1]
```
* The client requests a proxy, and the broker waits for 10s to respond with a 504 Gateway Timeout (indicating that the chosen proxy did not return an answer to the broker in time). The client waits 10s for the broker to respond, then waits another `ReconnectTimeout` in exchangeSDP before trying the broker again.
```
2020/04/30 22:39:30 Negotiating via BrokerChannel...
2020/04/30 22:39:41 BrokerChannel Response: 504 Gateway Timeout
2020/04/30 22:39:41 BrokerChannel Error: Unexpected error, no answer.
2020/04/30 22:39:41 Failed to retrieve answer. Retrying in 10s
2020/04/30 22:39:51 Negotiating via BrokerChannel...
```
Both these cases can probably be fixed by running the timer in parallel with the periodic operation they are rate limiting. That is, instead of
```
for {
operation()
<-time.After(ReconnectTimeout)
}
```
it can be
```
for {
timer := time.After(ReconnectTimeout)
operation()
<-timer
}
```
That way, if the operation itself takes more than 10s, `ReconnectTimeout` doesn't impose any additional delay.Sponsor 28: Reliable Anonymous Communication Evading Censors and Repressors (RACECAR)Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/27385https://snowflake.torproject.org/embed is confusing2022-07-09T04:20:15Zcypherpunkshttps://snowflake.torproject.org/embed is confusing`embed.html` should be the go to for embedding snowflake badges in webpages but it currently has a couple of problems,
1. No description, nothing to suggest to the lambda visitor about what to do.
2. Clicking on the badge redirects to ...`embed.html` should be the go to for embedding snowflake badges in webpages but it currently has a couple of problems,
1. No description, nothing to suggest to the lambda visitor about what to do.
2. Clicking on the badge redirects to the options page hosted in torproject.org, but this means that users who have first-party isolation manually enabled (as can be done in Firefox) won't be able to enable it on the page where embed.html is embedded.
Ideally what should be done is:
1. Small description ("Do you want to help censored users access the Tor network?") with a snowflake logo.
2. If user clicks on yes, then in that same iframe there's some JS check to see if WebRTC is disabled, if it is inform the user that WebRTC is necessary and perhaps add a link on how to enable it back.
3. If WebRTC is enabled, then load up snowflake.js and modernizr.js. Description should contain if the connection to the broker is done and is waiting for a client request, and if it is then maybe the logo should change as well as the description. (Since everything is done on the same page then there won't be any problems with first party isolation -- except with 3rd party cookies disabled)
Ideally it should be easy to embed into webpages if what's above is done, and should be small enough.
(cc'ing antonela of the ux-team for her opinions :)Sponsor 28: Reliable Anonymous Communication Evading Censors and Repressors (RACECAR)Arlo BreaultArlo Breaulthttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/25429Need something better than client's `checkForStaleness`2023-08-01T23:54:09ZArlo BreaultNeed something better than client's `checkForStaleness`If no message has been received on the datachannel on the client for `SnowflakeTimeout` (30 seconds), `checkForStaleness` closes the connection. The comment says this is to,
> // Prevent long-lived broken remotes.
but there's no hear...If no message has been received on the datachannel on the client for `SnowflakeTimeout` (30 seconds), `checkForStaleness` closes the connection. The comment says this is to,
> // Prevent long-lived broken remotes.
but there's no heartbeat at this level of abstraction so the connection is constantly being reset anytime the user pauses their activity (for example, to read a webpage).
This greatly exacerbated legacy/trac#21312Sponsor 28: Reliable Anonymous Communication Evading Censors and Repressors (RACECAR)Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/20813Start producing snowflakes2023-08-01T19:29:39ZArlo BreaultStart producing snowflakesOnce `snowflake-client` gets in the alpha Tor Browser builds (tpo/applications/tor-browser#20735), we're going to have some unhappy users if we don't have a sufficient number of proxies available.
We should start ramping up production a...Once `snowflake-client` gets in the alpha Tor Browser builds (tpo/applications/tor-browser#20735), we're going to have some unhappy users if we don't have a sufficient number of proxies available.
We should start ramping up production asap.
Some ideas in,<br>
https://github.com/glamrock/cupcake<br>
https://github.com/keroserene/snowflake/issues/30
We probably also want to close out the opt-in issue,<br>
https://github.com/keroserene/snowflake/issues/21Sponsor 28: Reliable Anonymous Communication Evading Censors and Repressors (RACECAR)https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40088Snowflake server keeps failing unexpectedly2022-07-09T04:20:46ZCecylia BocovichSnowflake server keeps failing unexpectedlyThe snowflake server has failed twice now in the last two weeks.
[After the first failure](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40060#note_2767868), we upgraded the capacity of the Sn...The snowflake server has failed twice now in the last two weeks.
[After the first failure](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40060#note_2767868), we upgraded the capacity of the Snowflake bridge (#40085).
Today when it failed we received an alert from monit: https://lists.torproject.org/pipermail/anti-censorship-alerts/2021-December/000786.html
Unfortunately, in my rush to fix it I didn't check the CPU usage :/ let's use this ticket to track the issue and note what we notice the next time it occurs.Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetshelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40135Cannot build snowflake/proxy2022-07-09T04:20:15ZcypherpunksCannot build snowflake/proxyHello! I want to install the snowflake proxy on my raspberry pi, but I can't seem to build it. I updated my system via apt-get update/update succesfully and the installation of git and golang were also successful. But after cloning the ...Hello! I want to install the snowflake proxy on my raspberry pi, but I can't seem to build it. I updated my system via apt-get update/update succesfully and the installation of git and golang were also successful. But after cloning the git, I went to /home/pi/snowflake/proxy and when I enter go build the system gives me this:
```
pi@raspberrypi:~/snowflake/proxy $ go build
main.go:5:2: cannot find package "git.torproject.org/pluggable-transports/snowflake.git/v2/common/event" in any of:
/usr/lib/go-1.7/src/git.torproject.org/pluggable-transports/snowflake.git/v2/common/event (from $GOROOT)
($GOPATH not set)
main.go:12:2: cannot find package "git.torproject.org/pluggable-transports/snowflake.git/v2/common/safelog" in any of:
/usr/lib/go-1.7/src/git.torproject.org/pluggable-transports/snowflake.git/v2/common/safelog (from $GOROOT)
($GOPATH not set)
main.go:13:2: cannot find package "git.torproject.org/pluggable-transports/snowflake.git/v2/proxy/lib" in any of:
/usr/lib/go-1.7/src/git.torproject.org/pluggable-transports/snowflake.git/v2/proxy/lib (from $GOROOT)
($GOPATH not set)
```
What is the problem here, and how can I fix it? Whe I look into /usr/lib/go-1.7/src/ there is no git.torproject.org subdirectory
I hope, you can help me.itchyonionitchyonionhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40103mix of past and present in snowflake proxy log2022-07-09T04:20:15Ztoralfmix of past and present in snowflake proxy log"In the last 1h0m0s, there are 28 connections. Traffic Relayed ↑ 273 MB, ↓ 273 MB."
are -> were
or ?"In the last 1h0m0s, there are 28 connections. Traffic Relayed ↑ 273 MB, ↓ 273 MB."
are -> were
or ?https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40098snowflake_server.httpHandler.ln is not initialized, leading to panic in onesh...2022-07-09T04:20:16ZDavid Fifielddcf@torproject.orgsnowflake_server.httpHandler.ln is not initialized, leading to panic in oneshotModeThere are a few of these in the snowflake-server log:
```
2022/02/01 16:44:11 http: panic serving [scrubbed]: runtime error: invalid memory address or nil pointer dereference
goroutine 2513699 [running]:
net/http.(*conn).serve.func1(0xc...There are a few of these in the snowflake-server log:
```
2022/02/01 16:44:11 http: panic serving [scrubbed]: runtime error: invalid memory address or nil pointer dereference
goroutine 2513699 [running]:
net/http.(*conn).serve.func1(0xc007576dc0)
/usr/lib/go-1.15/src/net/http/server.go:1801 +0x147
panic(0x7ea7a0, 0xb0c450)
/usr/lib/go-1.15/src/runtime/panic.go:975 +0x47a
git.torproject.org/pluggable-transports/snowflake.git/v2/server/lib.(*SnowflakeListener).queueConn(0x0, 0x8dc820, 0xc04b943740, 0xc03557fbb0, 0xc04b9436e0)
./snowflake/server/lib/snowflake.go:275 +0x37
git.torproject.org/pluggable-transports/snowflake.git/v2/server/lib.oneshotMode(...)
./snowflake/server/lib/http.go:117
git.torproject.org/pluggable-transports/snowflake.git/v2/server/lib.(*httpHandler).ServeHTTP(0xc0001a0e80, 0x8d9800, 0xc057a75880, 0xc0dda2cd00)
./snowflake/server/lib/http.go:105 +0x5b7
net/http.serverHandler.ServeHTTP(0xc0001ee0e0, 0x8d9800, 0xc057a75880, 0xc0dda2cd00)
/usr/lib/go-1.15/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc007576dc0, 0x8da1c0, 0xc013b86b00)
/usr/lib/go-1.15/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
/usr/lib/go-1.15/src/net/http/server.go:2969 +0x36c
```
`oneshotMode` is called with `httpHandler.ln` as the `*SnowflakeListener` argument:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/e828b0607662c7325f4d1bbf4c5072ce81f38fb9/server/lib/http.go#L100-105
```go
// We didn't find a matching token, which means that we are
// dealing with a client that doesn't know about such things.
// "Unread" the token by constructing a new Reader and pass it
// to the old one-session-per-WebSocket mode.
conn2 := &overrideReadConn{Conn: conn, Reader: io.MultiReader(bytes.NewReader(token[:]), conn)}
err = oneshotMode(conn2, addr, handler.ln)
```
`httpHandler` is:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/e828b0607662c7325f4d1bbf4c5072ce81f38fb9/server/lib/http.go#L63-68
```go
type httpHandler struct {
// pconn is the adapter layer between stream-oriented WebSocket
// connections and the packet-oriented KCP layer.
pconn *turbotunnel.QueuePacketConn
ln *SnowflakeListener
}
```
But in the only place `httpHandler` is instantiated, its `ln` field is left uninitialized:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/e828b0607662c7325f4d1bbf4c5072ce81f38fb9/server/lib/snowflake.go#L80-85
```go
handler := httpHandler{
// pconn is shared among all connections to this server. It
// overlays packet-based client sessions on top of ephemeral
// WebSocket connections.
pconn: turbotunnel.NewQueuePacketConn(addr, clientMapTimeout),
}
```Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40058Snowflake embed refuses to work on Chromium?2022-07-09T04:20:16ZkoutsieSnowflake embed refuses to work on Chromium?Snowflake (embed) simply refuses to enable on Chromium and Chrome.
At first i thought it was one of my extensions but a couple of friends reported the inability to enable Snowflake's embed while stating that the web extension version wor...Snowflake (embed) simply refuses to enable on Chromium and Chrome.
At first i thought it was one of my extensions but a couple of friends reported the inability to enable Snowflake's embed while stating that the web extension version works fine.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/34435Update bug-reporting links for gitlab2021-07-09T18:26:25ZDavid Fifielddcf@torproject.orgUpdate bug-reporting links for gitlabAfter the [gitlab migration](https://lists.torproject.org/pipermail/tor-project/2020-June/002866.html) we need to update bug-reporting instructions.
* https://snowflake.torproject.org/#bugs<br>
https://gitweb.torproject.org/pluggabl...After the [gitlab migration](https://lists.torproject.org/pipermail/tor-project/2020-June/002866.html) we need to update bug-reporting instructions.
* https://snowflake.torproject.org/#bugs<br>
https://gitweb.torproject.org/pluggable-transports/snowflake-webext.git/tree/static/index.html?h=webext-0.3.1&id=d2a9a8fd136ac6bbba393de3d51fb7ca85e17b8a#n75
* https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/wikis/home#reporting-bugs
I checked in snowflake.git and did not find anything that needs to be changed.Gabagaba@torproject.orgGabagaba@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/34147Remove redundant languages from snowflake page2021-07-09T18:26:26ZRoger DingledineRemove redundant languages from snowflake pagehttps://snowflake.torproject.org/
offers me en, en_GB, and en_US in the language drop-down menu. We should have one English, not three.
More generally, I bet we have a policy somewhere about which languages we want to be sure to include...https://snowflake.torproject.org/
offers me en, en_GB, and en_US in the language drop-down menu. We should have one English, not three.
More generally, I bet we have a policy somewhere about which languages we want to be sure to include when we have translated something, and we should see if we're missing any of those; and also I hope we have a policy about which languages to *not* include (and ideally we should go back to transifex and remove those from the set that people can translate, to avoid giving people the impression that they will be doing something useful if they e.g. translate en_US to en_GB).
I'm sorry I don't know more -- I bet Emma or Antonela or Gus will know more about whether we have languages policies and habits in place. :)https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/33744Remove local LAN address ICE candidates from JS proxy answer2021-07-09T18:26:25ZArlo BreaultRemove local LAN address ICE candidates from JS proxy answerThis is a follow up from legacy/trac#19026 where it was done for the clients and golang proxies.This is a follow up from legacy/trac#19026 where it was done for the clients and golang proxies.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/33519Support multiple simultaneous SOCKS connections2022-03-14T05:36:27ZDavid Fifielddcf@torproject.orgSupport multiple simultaneous SOCKS connectionsThe Snowflake client accepts multiple simultaneous SOCKS connections from tor, but it only tries to collect one proxy at a time, and each proxy can service only one SOCKS connection (this is true in the turbotunnel branch as well). One o...The Snowflake client accepts multiple simultaneous SOCKS connections from tor, but it only tries to collect one proxy at a time, and each proxy can service only one SOCKS connection (this is true in the turbotunnel branch as well). One of the SOCKS connections gets the only available proxy, while the others starve.
I can think of a few ways to approach this.
1. Dynamically adjust the `max` parameter according to how many SOCKS connections there are currently. If there's one SOCKS connection, we need only one proxy. If there's another SOCKS connection, raise the limit to allow the proxy-collecting thread to pick up another one, and lower the limit again if the number of SOCKS connections drops back down.
2. Start up a separate proxy-collecting thread for each SOCKS connection, as suggested at comment:12:ticket:21314. Each SOCKS connection will make its own broker requests and collect its own proxies, not interacting with those of any other SOCKS connection. A downside of this is that the number of Snowflake proxies you are contacting leaks the number of SOCKS connections you have ongoing. (Which can also be seen as a benefit in that if there are zero SOCKS connections, you don't even bother to contact the broker.)
3. Make it possible for multiple SOCKS connections to share the same proxy. Continue using a global proxy-collecting thread, and make there be a single shared `RedialPacketConn` instead of a separate one for each SOCKS connection. As things work now, this would require tagging _every packet_ with the ClientID, instead of [sending the ClientID once](https://gitweb.torproject.org/user/dcf/snowflake.git/tree/client/lib/snowflake.go?h=turbotunnel&id=47312dd1eccc8456652853bd66f8ed396e9ba6ec#n52) and letting it be the same implicitly for all packets that follow.
4. Make it possible for multiple SOCKS connections to share the same proxy, and use a single KCP/QUIC connection for all SOCKS connections. Separate SOCKS connections go into separate streams within the KCP/QUIC connection. In other words, rather than doing both `sess = kcp.NewConn2/quic.Dial` and `sess.OpenStream` in the SOCKS handler, we do `sess = kcp.NewConn2/quic.Dial` in `main` and then `sess.OpenStream` in the SOCKS handler. This way we could continue tagging the ClientID just once, because the program would only ever work with one ClientID at a time. However this way would make it harder to do the "stop using the network when not being used" of legacy/trac#21314, because that single KCP/QUIC connection would try to keep itself alive all the time and would contact the broker every time it needed a new proxy. Perhaps we could make it so that if there are zero streams, we close the KCP/QUIC connection, and lazily create a new one if and when we get another SOCKS connection.
| |= status quo=|= 1=|= 2=|= 3=|= 4=|
|---------------------------|--------------|--------------|--------------|--------------|--------------|
|= proxy-collecting threads=| one global| one global| one per SOCKS| one global| one global|
|= proxy limit per thread=| 1| # of SOCKS| 1| 1| 1|
|= proxies shared between SOCKSes?=| dedicated| dedicated| dedicated| shared| shared|
|= `PacketConn`s=| one per SOCKS| one per SOCKS| one per SOCKS| one global| one global|
|= KCP/QUIC connections=| one per SOCKS| one per SOCKS| one per SOCKS| one per SOCKS| one global|
|= KCP/QUIC streams=| one per SOCKS| one per SOCKS| one per SOCKS| one per SOCKS| one per SOCKS|
|= ClientID on every packet?=| no| no| no| yes| no|https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/33401turbotunnel-quic snowflake-client occasionally uses lots of CPU2022-07-09T04:20:15ZDavid Fifielddcf@torproject.orgturbotunnel-quic snowflake-client occasionally uses lots of CPUAs originally noted at comment:7:ticket:33211, the quic-go turbotunnel client sometimes uses 100+% CPU for a few minutes before returning to normal operation. It is specific to the quic-go implementation; it doesn't happen with the kcp-g...As originally noted at comment:7:ticket:33211, the quic-go turbotunnel client sometimes uses 100+% CPU for a few minutes before returning to normal operation. It is specific to the quic-go implementation; it doesn't happen with the kcp-go implementation nor the non-turbotunnel client.
As best I can figure, the cause has something to do with timers created under `(*session) maybeResetTimer`.
[profile001.png](None/profile001.png)David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/33367Goroutine leak in websocketconn2022-07-09T04:20:16ZDavid Fifielddcf@torproject.orgGoroutine leak in websocketconnThinking about legacy/trac#33364, I found that snowflake-server is chewing a lot of memory. It may be some memory leak or something.
```
$ top -o%MEM
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26910 debi...Thinking about legacy/trac#33364, I found that snowflake-server is chewing a lot of memory. It may be some memory leak or something.
```
$ top -o%MEM
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26910 debian-+ 20 0 1916628 1.522g 0 S 0.0 77.8 58:51.37 snowflake-serve
```
The memory use seems to be inhibiting other processes. `runsvdir` puts status messages in its own `argv` so you can inspect them with `ps`. Currently it's reflecting `xz` not being able to allocate memory to compress logs:
```
$ ps ax | grep runsvdir
1358 ? Ss 94:01 runsvdir -P /etc/service log: locate memory \
svlogd: warning: processor failed, restart: /home/snowflake-proxy/snowflake-proxy-standalone-17h.log.d xz: (stdin): Cannot allocate memory \
svlogd: warning: processor failed, restart: /home/snowflake-proxy/snowflake-proxy-standalone-17h.log.d xz: (stdin): Cannot allocate memory \
svlogd: warning: processor failed, restart: /home/snowflake-proxy/snowflake-proxy-standalone-17h.log.d
```
I even got it just now trying to run a diagnostic command (it doesn't always happen):
```
$ ps ax | grep standal
-bash: fork: Cannot allocate memory
```
In the short term, looks like we need to restart the server. Then we need to figure out what's causing it to use so much memory.
The server was last restarted 2020-02-10 18:57 (one week ago) at [ca9ae12c383405bc9a755e1bc902e9755495c1f1](https://gitweb.torproject.org/pluggable-transports/snowflake.git/log/?id=ca9ae12c383405bc9a755e1bc902e9755495c1f1) for legacy/trac#32964.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/33365Probe Snowflake bridge from proxy 1x a day2021-07-09T18:26:25ZCecylia BocovichProbe Snowflake bridge from proxy 1x a dayWe're getting reports that the Snowflake bridge isn't reachable in legacy/trac#33364, but it's taking awhile for volunteers to notice because the probe check only happens once at installation or if you disable/enable the proxy.
Perhaps ...We're getting reports that the Snowflake bridge isn't reachable in legacy/trac#33364, but it's taking awhile for volunteers to notice because the probe check only happens once at installation or if you disable/enable the proxy.
Perhaps we can do the probe check 1x a day (e.g., when we do the stats refresh)?Arlo BreaultArlo Breaulthttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/33211proxy-go sometimes gets into a 100+% CPU state2021-07-09T18:26:26ZDavid Fifielddcf@torproject.orgproxy-go sometimes gets into a 100+% CPU stateproxy-go sometimes works itself into a state where it is still running and working, but using more than 100% CPU. I have had it happen locally a couple of times while testing turbotunnel stuff, and it's currently happening with proxy-go-...proxy-go sometimes works itself into a state where it is still running and working, but using more than 100% CPU. I have had it happen locally a couple of times while testing turbotunnel stuff, and it's currently happening with proxy-go-restartless:
```
$ top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13844 snowfla+ 20 0 551292 320692 8844 R 161.1 15.6 129356:18 proxy-go
```
Or looking at single threads:
```
$ top -H
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24628 snowfla+ 20 0 551292 320692 8844 R 39.7 15.6 15219:01 proxy-go
13844 snowfla+ 20 0 551292 320692 8844 R 35.4 15.6 15431:52 proxy-go
1637 snowfla+ 20 0 551292 320692 8844 R 34.8 15.6 16057:40 proxy-go
13848 snowfla+ 20 0 551292 320692 8844 S 27.5 15.6 13669:02 proxy-go
13846 snowfla+ 20 0 551292 320692 8844 S 22.5 15.6 17021:57 proxy-go
```
I caught it once and attached to the process with GDB, but didn't know what to make of it. `thread apply all bt` seemed to show all the threads being somewhere in the Go runtime; the thread that wasn't was not one of the threads using a lot of CPU. (Matching up the `PID` field from `top -H` with the `LWP` identifiers in gdb.)
I had the idea to make proxy-go emit profiling output, and then exmine the call chain that was resulting the in the most CPU using [profiling tools](https://blog.golang.org/profiling-go-programs). A patch to do that is [proxy-go-profile.patch.](None/proxy-go-profile.patch.) But I haven't been able to reproduce the high CPU usage yet.Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/33112snowflake-webextension "Could not connect to the bridge."2021-07-09T18:26:25Zcypherpunkssnowflake-webextension "Could not connect to the bridge."My snowflake webextension has been working well up to this point but recently started having the error "Could not connect to the bridge." for the past few days. I'm not sure if this is something on my end or something with Snowflake.My snowflake webextension has been working well up to this point but recently started having the error "Could not connect to the bridge." for the past few days. I'm not sure if this is something on my end or something with Snowflake.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/32576Fix race condition in snowflake broker2022-07-09T04:20:15ZCecylia BocovichFix race condition in snowflake brokerThere is a race condition with the snowflake heap that has been causing the broker to crash several times a day. This race condition has existed in the broker for several years, but some recent updates as well as the host migration manag...There is a race condition with the snowflake heap that has been causing the broker to crash several times a day. This race condition has existed in the broker for several years, but some recent updates as well as the host migration managed to shake it loose.
----
This race condition is causing the snowflake broker to crash repeatedly and often since the migration. We noticed because CollecTor stopped collecting metrics since the restart on 14 November 2019.Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/31902Add a short FAQ to snowflake.tp.o2021-07-09T18:26:26ZArlo BreaultAdd a short FAQ to snowflake.tp.oThis should include explanations for the missing feature error messages. See comment:13:ticket:31391This should include explanations for the missing feature error messages. See comment:13:ticket:31391