Snowflake issueshttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues2024-02-20T07:58:00Zhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40106proxy-go support for IGD2024-02-20T07:58:00ZJillproxy-go support for IGDI tried using the proxy-go in two setup, one on a laptop behind NAT, the other on a server with its own restrictive firewall.
In both cases the NAT type was detected as "restricted". My understanding is that at most one end of the conne...I tried using the proxy-go in two setup, one on a laptop behind NAT, the other on a server with its own restrictive firewall.
In both cases the NAT type was detected as "restricted". My understanding is that at most one end of the connection can be restricted,
so a restricted proxy can't talk to a restricted client. Getting an unrestricted NAT is better as it's compatible with more clients.
To that effect, proxy-go could use [IGD](https://en.wikipedia.org/wiki/Internet_Gateway_Device_Protocol) to ask the NAT to create a dynamic
port forwarding so it is effectively unrestricted. This would help with the laptop situation (assuming the router doing NAT supports IGD,
which mine does), and using something like [miniupnp](https://miniupnp.tuxfamily.org/), it would also be possible to dynamically open ports
on a local, restrictive, firewall.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40105Make the snowflake deb more accessible to Debian stable users2024-01-28T01:38:34ZRoger DingledineMake the snowflake deb more accessible to Debian stable usersWe made a Snowflake deb in https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/19409 and it is apparently now in Debian sid: <br>
https://tracker.debian.org/pkg/snowflake
Yay!
These versions are on...We made a Snowflake deb in https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/19409 and it is apparently now in Debian sid: <br>
https://tracker.debian.org/pkg/snowflake
Yay!
These versions are only in sid, right? Do we have a path in mind for normal Debian users to be able to install this deb? In particular, either a debian backports repository, or our deb.torproject.org repo.
(We have a policy that we only take things from Debian for our deb.tpo repo, but I think we would be following that policy here, since that deb *is* in Debian, it's just only in sid.)meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40104proxy lib - be able to configure the proxy type2022-03-21T18:25:21Zmeskiomeskio@torproject.orgproxy lib - be able to configure the proxy typeThe snowflake proxy library is being used by more clients than our standalone proxy. Library users should be able to set the proxy type that will be reported to the broker.
Currently the proxy type is hardcoded: https://gitlab.torprojec...The snowflake proxy library is being used by more clients than our standalone proxy. Library users should be able to set the proxy type that will be reported to the broker.
Currently the proxy type is hardcoded: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/main/proxy/lib/snowflake.go#L202
A use case for it now will be to have a different type in orbot, so we know how many snowflakes are provided by orbot users in comparison to other users.
We should take into account that currently the broker has a [hardcoded list of proxy types](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/main/broker/metrics.go#L26) and the rest is treated as 'unknown'. This was motivated by having a lot of requests with estrange proxy types (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40089). I guess we can extend the proxy type list for the mayor types we know off or we could do some simple validation of what kind of proxy types are meaningful.meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40103mix of past and present in snowflake proxy log2022-07-09T04:20:15Ztoralfmix of past and present in snowflake proxy log"In the last 1h0m0s, there are 28 connections. Traffic Relayed ↑ 273 MB, ↓ 273 MB."
are -> were
or ?"In the last 1h0m0s, there are 28 connections. Traffic Relayed ↑ 273 MB, ↓ 273 MB."
are -> were
or ?https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40102Several devices on the same network ?2022-03-01T15:55:27ZcypherpunksSeveral devices on the same network ?Hi !
I got in touch with you once with that question :
- Is it possible to install Snowflake on all the devices connected to the same local network ? (Ex : On a family, on 2 PCs)
Your answer :
- Mmm, it's not recommended.
My request :
...Hi !
I got in touch with you once with that question :
- Is it possible to install Snowflake on all the devices connected to the same local network ? (Ex : On a family, on 2 PCs)
Your answer :
- Mmm, it's not recommended.
My request :
- To know if your answer is still relevant
- Whatever, can you write in the Wiki/website FAQ, etc. ?
Thank you ! Tons of love. :) ♥https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40101Failures at standalone proxy2023-05-16T20:13:44ZCecylia BocovichFailures at standalone proxyWe got the following error log from a proxy volunteer:
```
2022/02/11 00:38:05 In the last 1h0m0s, there are 103 connections. Traffic Relayed ↑ 2 GB, ↓ 2 GB.
ortc ERROR: 2022/02/11 02:28:16 Failed to accept data channel: failed to se...We got the following error log from a proxy volunteer:
```
2022/02/11 00:38:05 In the last 1h0m0s, there are 103 connections. Traffic Relayed ↑ 2 GB, ↓ 2 GB.
ortc ERROR: 2022/02/11 02:28:16 Failed to accept data channel: failed to send ChannelOpen ACK: sending payload data in non-established state: state=Closed
ortc ERROR: 2022/02/11 02:29:25 Failed to accept data channel: failed to send ChannelOpen ACK: sending payload data in non-established state: state=Closed
2022/02/11 01:38:05 In the last 1h0m0s, there are 66 connections. Traffic Relayed ↑ 1 GB, ↓ 1 GB.
2022/02/11 02:38:05 In the last 1h0m0s, there are 74 connections. Traffic Relayed ↑ 560 MB, ↓ 560 MB.
2022/02/11 03:38:05 In the last 1h0m0s, there are 69 connections. Traffic Relayed ↑ 5 GB, ↓ 5 GB.
2022/02/11 04:38:05 In the last 1h0m0s, there are 68 connections. Traffic Relayed ↑ 1 GB, ↓ 1 GB.
2022/02/11 05:38:05 In the last 1h0m0s, there are 86 connections. Traffic Relayed ↑ 2 GB, ↓ 2 GB.
2022/02/11 06:38:05 In the last 1h0m0s, there are 91 connections. Traffic Relayed ↑ 2 GB, ↓ 2 GB.
2022/02/11 07:38:05 In the last 1h0m0s, there are 111 connections. Traffic Relayed ↑ 613 MB, ↓ 613 MB.
sctp ERROR: 2022/02/11 08:53:20 [0xc073c969c0] stream 1 not found)
sctp ERROR: 2022/02/11 08:53:20 [0xc073c969c0] stream 1 not found)
sctp ERROR: 2022/02/11 09:03:19 [0xc0a7166340] stream 1 not found)
ortc ERROR: 2022/02/11 09:15:45 Failed to accept data channel: failed to send ChannelOpen ACK: sending payload data in non-established state: state=Closed
sctp ERROR: 2022/02/11 09:37:53 [0xc0103c36c0] stream 1 not found)
sctp ERROR: 2022/02/11 09:37:53 [0xc0103c36c0] stream 1 not found)
2022/02/11 08:38:05 In the last 1h0m0s, there are 121 connections. Traffic Relayed ↑ 212 MB, ↓ 212 MB.
ortc ERROR: 2022/02/11 09:41:30 Failed to accept data channel: failed to send ChannelOpen ACK: sending payload data in non-established state: state=Closed
sctp ERROR: 2022/02/11 09:54:55 [0xc02e8e1860] stream 1 not found)
sctp ERROR: 2022/02/11 09:54:55 [0xc02e8e1860] stream 1 not found)
sctp ERROR: 2022/02/11 09:54:55 [0xc02e8e1860] stream 1 not found)
sctp ERROR: 2022/02/11 09:54:55 [0xc02e8e1860] stream 1 not found)
sctp ERROR: 2022/02/11 09:54:58 [0xc073d15380] stream 1 not found)
sctp ERROR: 2022/02/11 09:55:01 [0xc0103c29c0] stream 1 not found)
sctp ERROR: 2022/02/11 09:55:03 [0xc003828ea0] stream 1 not found)
sctp ERROR: 2022/02/11 09:55:03 [0xc003828ea0] stream 1 not found)
ortc ERROR: 2022/02/11 10:27:42 Failed to accept data channel: failed to send ChannelOpen ACK: sending payload data in non-established state: state=Closed
Killed
(by kernel OOM killer.)
```shelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40100Performance modelling of Snowflake2024-02-27T18:25:37ZCecylia BocovichPerformance modelling of SnowflakeAs a followup to previous discussions on Snowflake performance, the purpose of this issue is to track work on modelling and measuring the impact of Snowflake improvements on network performance. This work will be primarily done with the ...As a followup to previous discussions on Snowflake performance, the purpose of this issue is to track work on modelling and measuring the impact of Snowflake improvements on network performance. This work will be primarily done with the [Shadow](https://shadow.github.io/) network simulation tool. This tool can measure the impact that changes to Snowflake can have on the throughput of traffic for clients, as well as resource consumption of the broker and bridge.
Snowflake shadow simulation scripts can be found at https://gitlab.torproject.org/cohosh/snowflake-simulation
There are a few tasks to complete before we are ready to conduct performance experiments:
- [ ] Help Shadow developers debug outstanding issues with go network code
- [ ] Improve the Snowflake network model to accurately reflect the network conditions faced by both snowflake clients and proxy volunteers
- [ ] Improve the output format of test results so they can be easily interpreted
Once these pieces are in place, I plan to conduct the following experiments:
- [ ] Tune turbotunnel parameters by experimenting with the space of probable configurations (#40026)
- [ ] Splitting traffic across multiple snowflake proxies (#25723)
- [ ] The impact of geographic location of proxies on client performance (#31661)
Shadow simulations do have some limitations. We have also deployed onionperf instances to measure real-world Snowflake performance. If evidence for performance improvements is compelling enough, we can measure the impact of the change in deployment from these locations.Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40099`closed` field of SnowflakeListener is never initialized2022-02-11T15:08:07ZDavid Fifielddcf@torproject.org`closed` field of SnowflakeListener is never initialized`SnowflakeListener` is:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/e828b0607662c7325f4d1bbf4c5072ce81f38fb9/server/lib/snowflake.go#L165-172
```go
type SnowflakeListener struct {
addr ...`SnowflakeListener` is:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/e828b0607662c7325f4d1bbf4c5072ce81f38fb9/server/lib/snowflake.go#L165-172
```go
type SnowflakeListener struct {
addr net.Addr
queue chan net.Conn
server *http.Server
ln *kcp.Listener
closed chan struct{}
closeOnce sync.Once
}
```
When it is initialized, its `queue` channel is initialized but its `closed` channel is left `nil`:
```go
listener := &SnowflakeListener{addr: addr, queue: make(chan net.Conn, 65534)}
```
The effect of this is that in functions such as `SnowflakeListener.queueConn`, the `select` will never take the `closed` case, because reads from `nil` channels always block:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/e828b0607662c7325f4d1bbf4c5072ce81f38fb9/server/lib/snowflake.go#L273-280
```go
func (l *SnowflakeListener) queueConn(conn net.Conn) error {
select {
case <-l.closed:
return fmt.Errorf("accepted connection on closed listener")
case l.queue <- conn:
return nil
}
}
```https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40098snowflake_server.httpHandler.ln is not initialized, leading to panic in onesh...2022-07-09T04:20:16ZDavid Fifielddcf@torproject.orgsnowflake_server.httpHandler.ln is not initialized, leading to panic in oneshotModeThere are a few of these in the snowflake-server log:
```
2022/02/01 16:44:11 http: panic serving [scrubbed]: runtime error: invalid memory address or nil pointer dereference
goroutine 2513699 [running]:
net/http.(*conn).serve.func1(0xc...There are a few of these in the snowflake-server log:
```
2022/02/01 16:44:11 http: panic serving [scrubbed]: runtime error: invalid memory address or nil pointer dereference
goroutine 2513699 [running]:
net/http.(*conn).serve.func1(0xc007576dc0)
/usr/lib/go-1.15/src/net/http/server.go:1801 +0x147
panic(0x7ea7a0, 0xb0c450)
/usr/lib/go-1.15/src/runtime/panic.go:975 +0x47a
git.torproject.org/pluggable-transports/snowflake.git/v2/server/lib.(*SnowflakeListener).queueConn(0x0, 0x8dc820, 0xc04b943740, 0xc03557fbb0, 0xc04b9436e0)
./snowflake/server/lib/snowflake.go:275 +0x37
git.torproject.org/pluggable-transports/snowflake.git/v2/server/lib.oneshotMode(...)
./snowflake/server/lib/http.go:117
git.torproject.org/pluggable-transports/snowflake.git/v2/server/lib.(*httpHandler).ServeHTTP(0xc0001a0e80, 0x8d9800, 0xc057a75880, 0xc0dda2cd00)
./snowflake/server/lib/http.go:105 +0x5b7
net/http.serverHandler.ServeHTTP(0xc0001ee0e0, 0x8d9800, 0xc057a75880, 0xc0dda2cd00)
/usr/lib/go-1.15/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc007576dc0, 0x8da1c0, 0xc013b86b00)
/usr/lib/go-1.15/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
/usr/lib/go-1.15/src/net/http/server.go:2969 +0x36c
```
`oneshotMode` is called with `httpHandler.ln` as the `*SnowflakeListener` argument:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/e828b0607662c7325f4d1bbf4c5072ce81f38fb9/server/lib/http.go#L100-105
```go
// We didn't find a matching token, which means that we are
// dealing with a client that doesn't know about such things.
// "Unread" the token by constructing a new Reader and pass it
// to the old one-session-per-WebSocket mode.
conn2 := &overrideReadConn{Conn: conn, Reader: io.MultiReader(bytes.NewReader(token[:]), conn)}
err = oneshotMode(conn2, addr, handler.ln)
```
`httpHandler` is:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/e828b0607662c7325f4d1bbf4c5072ce81f38fb9/server/lib/http.go#L63-68
```go
type httpHandler struct {
// pconn is the adapter layer between stream-oriented WebSocket
// connections and the packet-oriented KCP layer.
pconn *turbotunnel.QueuePacketConn
ln *SnowflakeListener
}
```
But in the only place `httpHandler` is instantiated, its `ln` field is left uninitialized:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/e828b0607662c7325f4d1bbf4c5072ce81f38fb9/server/lib/snowflake.go#L80-85
```go
handler := httpHandler{
// pconn is shared among all connections to this server. It
// overlays packet-based client sessions on top of ephemeral
// WebSocket connections.
pconn: turbotunnel.NewQueuePacketConn(addr, clientMapTimeout),
}
```Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40097Help OONI developers to calibrate the new torsf OONI experiment2022-03-11T05:12:10ZsbsHelp OONI developers to calibrate the new torsf OONI experimentGood day, Snowflake developers!
We recently did QA to understand and calibrate the now `torsf` (tor + snowflake) experiment we have written and (TL;DR) we were quite surprised by the amount of bootstrap timeouts we observed on mobile ne...Good day, Snowflake developers!
We recently did QA to understand and calibrate the now `torsf` (tor + snowflake) experiment we have written and (TL;DR) we were quite surprised by the amount of bootstrap timeouts we observed on mobile networks.
I've explained the problem we're having in great details and posed questions for you here https://github.com/ooni/probe/issues/2004. It would be awesome if someone of you could take a look and help us out!
Thanks a lot! :pray: :slightly_smiling_face:Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40096Too many open files error at probetest2022-09-24T14:42:19ZCecylia BocovichToo many open files error at probetestI took a look at the NAT probetest logs this morning after the [alert yesterday](https://lists.torproject.org/pipermail/anti-censorship-alerts/2022-January/000924.html). It doesn't look like probetest is stuck, but I did notice a familia...I took a look at the NAT probetest logs this morning after the [alert yesterday](https://lists.torproject.org/pipermail/anti-censorship-alerts/2022-January/000924.html). It doesn't look like probetest is stuck, but I did notice a familiar message in the logs:
```
2022/01/29 21:02:32 http: Accept error: accept tcp [scrubbed]: accept4: too many
open files; retrying in 5ms
```
And when I run a proxy, I frequently get http timeout errors, like due to this issue on the server.
We ran into this recently with the load balanced bridge (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40095#note_2772325) and previously with the broker (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/31425).
The NAT probetest is run with sv, and it's `run` file contains the following:
```
#!/bin/sh -e
export GOMAXPROCS=1
setcap 'cap_net_bind_service=+ep' /usr/local/bin/probetest
exec ip netns exec net0 /usr/local/bin/probetest --acme-hostnames snowflake-broker.bamsoftware.com,snowflake-broker.freehaven.net,snowflake-broker.torproject.net --acme-email dcf@torproject.org --acme-cert-cache /home/snowflake-broker/acme-cert-cache --addr :8443 2>&1
```https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40095Add load balancing to bridge2022-10-17T05:15:38ZDavid Fifielddcf@torproject.orgAdd load balancing to bridgeWe rehearsed a load-balanced bridge installation in #40091. Now let's do it for the production bridge. To reduce risk, we plan to do a staged upgrade using a secondary bridge.
We still do not have a permanent solution to the [onion key ...We rehearsed a load-balanced bridge installation in #40091. Now let's do it for the production bridge. To reduce risk, we plan to do a staged upgrade using a secondary bridge.
We still do not have a permanent solution to the [onion key rotation issue](https://forum.torproject.net/t/tor-relays-how-to-reduce-tor-cpu-load-on-a-single-bridge/1483/11). The current plan is to periodically reset [LastRotatedOnionKey](https://gitweb.torproject.org/tor.git/tree/doc/state-contents.txt?h=tor-0.4.6.9#n82) in the state file of all tor instances.
- [x] ask sysadmin team to reduce TTL for snowflake.torproject.net to 60 seconds tpo/tpa/team#40594
- [x] copy user accounts to staging bridge https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40091#note_2768855
- [x] install new staging bridge ([installation guide](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40091#note_2770360))
- [x] refresh the LastRotatedOnionKey line in the state file of the production bridge and restart tor
- [x] back up identity and onion keys from production bridge
- [x] copy identity and onion keys from production bridge to the staging bridge
- [x] copy HTTPS TLS keys and certificates from the production bridge to the staging bridge
- [x] test HTTPS of staging bridge using `curl --connect-to`
- [x] test tor bootstrap on staging bridge using local broker and proxy, and temporary domain name https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40091#note_2770360
- [x] switch DNS for snowflake.torproject.net to point to staging bridge tpo/tpa/team#40598
- [x] monitor for a day, and be ready to switch DNS back to production if connections fail on the staging bridge
- [x] disable and mask tor@default instance on production bridge
- [x] install load balancing configuration on production bridge [installation guide](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40091#note_2770360)
- [x] test HTTPS of production bridge using `curl --connect-to`
- [x] test tor bootstrap on production bridge using local broker and proxy, and temporary domain name https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40091#note_2770360
- [x] switch DNS for snowflake.torproject.net to point back to production bridge tpo/tpa/team#40602
- [x] monitor for 2 days, and be ready to switch DNS back to staging if connections fail on the production bridge
- [x] ask sysadmin team to restore TTL for snowflake.torproject.net to normal tpo/tpa/team#40595
- [x] shut down staging bridge
- [x] post new instructions to [Snowflake Bridge Installation Guide](https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Guides/Snowflake-Bridge-Installation-Guide) and [Snowflake Bridge Survival Guide](https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Guides/Snowflake-Bridge-Survival-Guide)
References
* #40091
* https://forum.torproject.net/t/tor-relays-how-to-reduce-tor-cpu-load-on-a-single-bridge/1483
* http://meetbot.debian.net/tor-meeting/2022/tor-meeting.2022-01-20-15.59.log.html#l-60Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & TibetDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40094Disable standalone proxies on bridge2022-01-19T07:40:48ZDavid Fifielddcf@torproject.orgDisable standalone proxies on bridgeWe want to increase the tor capacity of the snowflake bridge. We should disable the two standalone proxies we run on that host, in order to open up some more CPU headroom. Cf. #40091.We want to increase the tor capacity of the snowflake bridge. We should disable the two standalone proxies we run on that host, in order to open up some more CPU headroom. Cf. #40091.Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & TibetDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40093Rendez-vous via tailscale derp server2024-02-27T19:09:31ZArlo BreaultRendez-vous via tailscale derp serverhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40091Set up a Snowflake bridge staging server2022-04-06T05:00:23ZDavid Fifielddcf@torproject.orgSet up a Snowflake bridge staging serverThe goals of this issue are:
1. Share practical experience setting up a snowflake bridge.
2. Write a [Snowflake Bridge Installation Guide](https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Guides/Snowflake-Bridge-I...The goals of this issue are:
1. Share practical experience setting up a snowflake bridge.
2. Write a [Snowflake Bridge Installation Guide](https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Guides/Snowflake-Bridge-Installation-Guide), like the existing [Snowflake Broker Installation Guide](https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Guides/Snowflake-Broker-Installation-Guide).
3. Install an [experimental load balancing configuration](https://lists.torproject.org/pipermail/tor-relays/2022-January/020183.html) in an attempt to increase the capacity of the bridge.
If the load balancing configuration works, we can then apply it to the production bridge. (Or change DNS so as to swap production and staging.)
We talked about scaling the snowflake bridge and load balancing ideas at the [2022-01-06 team meeting](http://meetbot.debian.net/tor-meeting/2022/tor-meeting.2022-01-06-15.59.html).
<details>
<summary>Meeting notes</summary>
* Scaling Snowflake bridge
* Already increased from 4 to 8 CPUs
* https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40085
* We should profile snowflake-server to reduce its CPU use
* https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40086
* But dcf feels that the bottleneck is not the snowflake-server process (which is multithreaded and can use multiple CPU cores), but the tor process (which can only use 1 CPU core and is already constantly at 100%). See https://lists.torproject.org/pipermail/tor-relays/2021-December/020156.html
* It turns out that it is possible to run multiple instances of tor with the same identity keys (hence the same fingerprint), either on the same host or on different hosts: https://lists.torproject.org/pipermail/tor-relays/2021-December/020157.html. Multiple instances of tor is a way of scaling tor beyond 1 CPU core, with snowflake-server distributing traffic over the available instances: https://lists.torproject.org/pipermail/tor-relays/2021-December/020182.html
* With another shim (similar to moat-shim) you can keep ExtORPort metrics reporting in the load-balanced configuration:
* https://gitlab.torproject.org/dcf/extor-static-cookie
* https://lists.torproject.org/pipermail/tor-relays/2022-January/020183.html
* Though I'm not sure, if Metrics gets multiple descriptors per day with the same fingerprint, whether it will sum all of them, or only keep one of them
* dcf has a bridge running now (with obfs4proxy, not snowflake-server) with this extor-static-cookie + load balancing configuration
* https://metrics.torproject.org/rs.html#details/07B9C6D7BE9685E83FA8C7A4FEB34CAD6CB77503
* Though I have not tested Roger's caveat about DEFAULT_ONION_KEY_LIFETIME_DAYS yet https://lists.torproject.org/pipermail/tor-relays/2022-January/020196.html
* We could apply this same thing to the snowflake bridge. But it is kind of a big configuration change: we need to stop snowflake-server being managed by tor, and instead run it independently (i.e., using runit or systemd) and have it talk to the tor instances' static ExtORPorts through a load balancer. It would be best to do this on a staging server separate from production first.
* we tentatively plan to do this next tuesday, 2022-01-11. dcf will be in touch and have a host ready for installation.
* Once the tor bottleneck is removed, though, we are not too far from the next likely bottleneck, which is total CPU of the host, shared between tor and snowflake-server. It might be time to think about migrating to different hosting for the snowflake bridge. Or we can experiment with running snowflake-server on one host, and N instances of tor on another (preferably nearby) host.
</details>Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & TibetDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40090Snowflake Server Deployment 2022-Jan-072022-01-07T19:55:42ZshelikhooSnowflake Server Deployment 2022-Jan-07* Code to be deployed: 7c8fc33243fd5b83b16e7103d467de4af12c57de
* Binary to be deployed:
* 2cad03a0160e68cf974497936802703dd13fa25fdb263cef72d03cedac2ffb81 deploy-snowflake-server-22-01-07-candidcate
* 330aaf72054f0272ef387f60550820...* Code to be deployed: 7c8fc33243fd5b83b16e7103d467de4af12c57de
* Binary to be deployed:
* 2cad03a0160e68cf974497936802703dd13fa25fdb263cef72d03cedac2ffb81 deploy-snowflake-server-22-01-07-candidcate
* 330aaf72054f0272ef387f60550820a5236f67c69435dec1cea82390103f9263 snowflake-server-22-01-07-candidcate
### Deployment Script ###
```
service tor stop
cp /usr/local/bin/snowflake-server ./snowflake-server-22-01-07-backup-$(date +%N)
install --owner root ./snowflake-server-22-01-07-candidcate /usr/local/bin/snowflake-server
setcap 'cap_net_bind_service=+ep' /usr/local/bin/snowflake-server
service tor start
```https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40089Possibly malicious snowflake polls?2022-02-21T14:35:42ZCecylia BocovichPossibly malicious snowflake polls?I just took a look at the exported prometheus metrics: https://snowflake-broker.torproject.net/prometheus
and saw a bunch of very unusual snowflake proxy types:
```
snowflake_rounded_proxy_poll_total{nat="restricted'\"\\(",status="idle"...I just took a look at the exported prometheus metrics: https://snowflake-broker.torproject.net/prometheus
and saw a bunch of very unusual snowflake proxy types:
```
snowflake_rounded_proxy_poll_total{nat="restricted'\"\\(",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="restricted'/**/and(select'1'from/**/pg_sleep(0))>'0",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="restricted'/**/and(select'1'from/**/pg_sleep(18))>'0",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="restricted'/**/and/**/DBMS_PIPE.RECEIVE_MESSAGE('p',18)='p",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="restricted'/**/and/**/DBMS_PIPE.RECEIVE_MESSAGE('q',0)='q",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="restricted'and(select'1'from/**/cast(md5(1634804485)as/**/int))>'0",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="restricted'and(select*from(select+sleep(0))a/**/union/**/select+1)='",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="restricted'and(select*from(select+sleep(18))a/**/union/**/select+1)='",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="restricted'and(select+1)>0waitfor/**/delay'0:0:0",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="restricted'and(select+1)>0waitfor/**/delay'0:0:18",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="restricted'and/**/convert(int,sys.fn_sqlvarbasetostr(HashBytes('MD5','1720393988')))>'0",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="restricted'and/**/extractvalue(1,concat(char(126),md5(1199007671)))and'",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="restricted/**/and/**/cast(md5('1204196037')as/**/int)>0",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="restricted|expr 809667990 + 908363838 ",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="restricted鎈'\"\\(",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown",status="idle"} 2.9086904e+07
snowflake_rounded_proxy_poll_total{nat="unknown",status="matched"} 2.757224e+06
snowflake_rounded_proxy_poll_total{nat="unknown\nexpr 965825977 + 974216935\n",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="unknown\"and(select*from(select+sleep(0))a/**/union/**/select+1)=\"",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="unknown\"and(select*from(select+sleep(14))a/**/union/**/select+1)=\"",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown\"and/**/extractvalue(1,concat(char(126),md5(1145896139)))and\"",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="unknown$(expr 898219840 + 955106000)",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown&set /A 859877269+929132787",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'\"\\(",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'/**/and(select'1'from/**/pg_sleep(0))>'0",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'/**/and(select'1'from/**/pg_sleep(14))>'0",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'/**/and/**/DBMS_PIPE.RECEIVE_MESSAGE('f',0)='f",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'/**/and/**/DBMS_PIPE.RECEIVE_MESSAGE('u',14)='u",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'and(select'1'from/**/cast(md5(1946869959)as/**/int))>'0",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'and(select*from(select+sleep(0))a/**/union/**/select+1)='",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'and(select*from(select+sleep(14))a/**/union/**/select+1)='",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'and(select+1)>0waitfor/**/delay'0:0:0",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'and(select+1)>0waitfor/**/delay'0:0:14",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'and/**/convert(int,sys.fn_sqlvarbasetostr(HashBytes('MD5','1587856848')))>'0",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown'and/**/extractvalue(1,concat(char(126),md5(1875029567)))and'",status="idle"} 8
snowflake_rounded_proxy_poll_total{nat="unknown/**/and/**/cast(md5('1500206875')as/**/int)>0",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown|expr 881080391 + 963149055 ",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unknown鎈'\"\\(",status="matched"} 8
snowflake_rounded_proxy_poll_total{nat="unrestricted",status="idle"} 1.801232e+06
snowflake_rounded_proxy_poll_total{nat="unrestricted",status="matched"} 7.7092e+06
```
I'm keeping this confidential for now in case this is a security vulnerability being actively exploited.Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & TibetCecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40092Improve docs on network_mode: host (and network in general)2023-01-18T16:18:15ZchmacImprove docs on network_mode: host (and network in general)When I found this repo, the example line `network_mode: host` jumped out at me as suspicious. I looked up the docs and figured that it's probably because snowflake requires lots of ports or so. I figured that my trust in the tor project ...When I found this repo, the example line `network_mode: host` jumped out at me as suspicious. I looked up the docs and figured that it's probably because snowflake requires lots of ports or so. I figured that my trust in the tor project is pretty high, and so I'm running a snowflake node.
But, I'm not really sure what network conditions it needs. Does it expect that `network_mode: host` means it's running on a host which has a publicly accessible IP? Does it needs ports on that host's firewall open?
The idea behind this issue is to improve the docs in this area so that snowflake hosts like myself can figure out what network conditions are required for snowflake to work. For example, I have no idea if my node is actually functional right now, I also have no idea how to test it.
Some example questions we could aim to answer:
- What ports does snowflake run on?
- Does snowflake need to be run on a machine with a public IP?
- Does snowflake run properly if behind a NAT?
- Does snowflake require specific ports to be opened in the system firewall?
- How can a server admin test if snowflake is properly configured and working?
As an add on, it would be great to see answers to questions like these:
- How much bandwidth can one expect snowflake to use?
- Does it make sense to add any kind of limits?
- If so, how would that be done?
- Are there any security considerations to running a snowflake server?
- What sort of system resources (CPU, memory) does snowflake use?
- Does it make sense to check on this periodically for memory leaks, etc?
- How can one be notified when updates are published to the docker image?
- Is there a security mailing list where one could be notified of any security issues that require urgent update of the snowflake server?
Finally, thanks for making the tor network more resilient, snowflake looks to be an awesome improvement for people in locations with internet censorship, and thanks for working on tor in general, it's a phenomenal resource supporting the human experience.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40088Snowflake server keeps failing unexpectedly2022-07-09T04:20:46ZCecylia BocovichSnowflake server keeps failing unexpectedlyThe snowflake server has failed twice now in the last two weeks.
[After the first failure](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40060#note_2767868), we upgraded the capacity of the Sn...The snowflake server has failed twice now in the last two weeks.
[After the first failure](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40060#note_2767868), we upgraded the capacity of the Snowflake bridge (#40085).
Today when it failed we received an alert from monit: https://lists.torproject.org/pipermail/anti-censorship-alerts/2021-December/000786.html
Unfortunately, in my rush to fix it I didn't check the CPU usage :/ let's use this ticket to track the issue and note what we notice the next time it occurs.Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetshelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40087Let's Encrypt "DST Root X3" root expiration affects old Android clients acces...2024-01-22T16:33:10ZcybertaLet's Encrypt "DST Root X3" root expiration affects old Android clients accessing brokerRunning Snowflake with the default config mentioned in this repository and shown below Snowflake fails to create a connection on some Android devices (apparently older Android versions, I could reproduce that issue using Android 4 and An...Running Snowflake with the default config mentioned in this repository and shown below Snowflake fails to create a connection on some Android devices (apparently older Android versions, I could reproduce that issue using Android 4 and Android 6 on a real device and on an emulator).
The error log tells me the cause of the connection failure is an expired certificate.
`WebRTC: x509: certificate has expired or is not yet valid: current time 2021-12-28T16:12:58Z is after 2021-09-30T14:01:15Z Retrying... `
Default config, I'm referring to:
```
snowflake-target https://snowflake-broker.torproject.net.global.prod.fastly.net/
snowflake-front cdn.sstatic.net
```
Using a different broker and domain-fronting I can work around the issue (config taken from https://github.com/cohosh/snowflake)
Could you please have a look at the broker / domain fronting setup or adapt the documentation here?Cecylia BocovichCecylia Bocovich