Snowflake issueshttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues2023-06-21T09:15:10Zhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40113Surveying Snowflake CPU/RAM utilization2023-06-21T09:15:10ZGeorgeSurveying Snowflake CPU/RAM utilizationAs per issue raised at 20220305 Relay Operators meetup, we discussed possible high CPU usage with Snowflake with browser addon. Relates to "High CPU load on idle proxies" #40112
We should get contributions of users detailing:
snowflake...As per issue raised at 20220305 Relay Operators meetup, we discussed possible high CPU usage with Snowflake with browser addon. Relates to "High CPU load on idle proxies" #40112
We should get contributions of users detailing:
snowflake_version,browser,browser_version,installed_addons,CPU,CPU_snowflake_usage,RAM,RAM_snowflake_usage,operating_system
We should likely include average Snowflake usage and maybe consider basic hardware specs on device. Contributions should no have other applications running to better control the results.
Any enhancements on this survey welcome.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40112High CPU load on idle proxies2024-03-05T18:23:24ZPeter GerberHigh CPU load on idle proxiesI've been running Snowflake proxies for some time now and I'm seeing rather high CPU load even when proxies are idle.
I'm seeing CPU usage similar to this on all my proxies:
```
systemctl status tor-snowflake
● tor-snowflake.service - ...I've been running Snowflake proxies for some time now and I'm seeing rather high CPU load even when proxies are idle.
I'm seeing CPU usage similar to this on all my proxies:
```
systemctl status tor-snowflake
● tor-snowflake.service - Tor Snowflake bridge
Loaded: loaded (/etc/systemd/system/tor-snowflake.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2022-03-12 11:18:37 CET; 1 day 5h ago
Main PID: 3242240 (tor-snowflake)
IP: 1.5G in, 1.5G out
Tasks: 9 (limit: 19046)
Memory: 55.6M
CPU: 7h 59min 35.486s
CGroup: /system.slice/tor-snowflake.service
└─3242240 /usr/local/bin/tor-snowflake
```
This corresponds to about a **~28% CPU load at an average speed of 15 kiB/s**.
I looked at the CPU load over the timespan of a few days. I see a CPU load of around 30% even with no traffic whatsoever. Thus, I suspect there are CPU cycles wasted somewhere.
OS: Debian 11 "bullseye"
Version: 19e9e384154adc6251579dc6843f11f53cbd0146https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40111Move bridge to a permanent faster server2022-10-19T20:40:14ZDavid Fifielddcf@torproject.orgMove bridge to a permanent faster serverBackground: [\[tor-project\] More resources required for Snowflake bridge](https://forum.torproject.net/t/tor-project-more-resources-required-for-snowflake-bridge/2353)
I expect to be able to move the snowflake bridge to a more permanen...Background: [\[tor-project\] More resources required for Snowflake bridge](https://forum.torproject.net/t/tor-project-more-resources-required-for-snowflake-bridge/2353)
I expect to be able to move the snowflake bridge to a more permanent home on a faster server after 2022-03-21.
#40110 is to use a *different* faster server in the meantime, until the permanent one is prepared.
- [x] get access to new server hardware
- [x] install new bridge ([installation guide](https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Guides/Snowflake-Bridge-Installation-Guide))
- [x] copy user accounts https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40091#note_2768855
- [x] copy identity and onion keys from existing bridge
- [x] double check onion keys
```
# md5sum /var/lib/tor-instances/*/keys/secret_onion_key{,_ntor}
f57a05262f65beea15ec05bbeefe404c /var/lib/tor-instances/snowflake1/keys/secret_onion_key
a16c5403d18509c79fa7b863eb66892a /var/lib/tor-instances/snowflake1/keys/secret_onion_key_ntor
```
- [x] copy HTTPS TLS keys and certificates from existing bridge
- [x] test tor bootstrap on new bridge using local broker and proxy, and /etc/hosts domain name record https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40095#note_2773365
- [x] test rebooting the server to make sure everything comes back up
- [x] start the tor@snowflake* services
- [x] make DNS for snowflake.torproject.net point to the new bridge tpo/tpa/team#40716
- [x] monitor for a day, be ready to switch DNS back if connections fail
- [x] after a week or so, shut down temporary bridge
Cc @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40110Move bridge to an interim faster server2022-05-17T00:24:17ZDavid Fifielddcf@torproject.orgMove bridge to an interim faster serverBackground: [\[tor-project\] More resources required for Snowflake bridge](https://forum.torproject.net/t/tor-project-more-resources-required-for-snowflake-bridge/2353)
I have or have been offered access to a faster server for the snowf...Background: [\[tor-project\] More resources required for Snowflake bridge](https://forum.torproject.net/t/tor-project-more-resources-required-for-snowflake-bridge/2353)
I have or have been offered access to a faster server for the snowflake bridge. I anticipate being able to change DNS and start using the faster server on Wednesday, 2022-03-16.
The current situation is: I have control right now of a suitable *paid* server, and I have an offer of a suitable *free* server that I am supposed to get control of on Tuesday, 2022-03-15. I will prefer to switch to the free server, but if there are any unforeseen problems with that deal, the paid server will be ready as a backup.
I expect to have to move the bridge *again*, to a more permanent home, but not before 2022-03-21 (#40111). The purpose of the migration in this ticket is to improve service until that other server is ready, and to mitigate any possible delays.
- [x] install new bridge ([installation guide](https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Guides/Snowflake-Bridge-Installation-Guide)), try 8 tor instances to start
- [x] copy user accounts https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40091#note_2768855
- [x] copy identity and onion keys from existing bridge
- [x] copy HTTPS TLS keys and certificates from existing bridge
- [x] test tor bootstrap on new bridge using local broker and proxy, and /etc/hosts domain name record https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40095#note_2773365
- [x] make DNS for snowflake.torproject.net point to the new bridge tpo/tpa/team#40664
- [x] monitor for a day, be ready to switch DNS back if connections fail
Cc @artDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40109Reboot broker and bridge for Dirty Pipe kernel vulnerability (CVE-2022-0847)2022-03-11T16:02:06ZDavid Fifielddcf@torproject.orgReboot broker and bridge for Dirty Pipe kernel vulnerability (CVE-2022-0847)> Subject: DirtyPipe vulnerability for VPSes - action required<br>
> Date: Fri, 11 Mar 2022 13:21:43 +0000
>
> Dear Eclips.is user,
>
> You may have heard of a recent vulnerability in the mainstream Linux Kernel, called the 'Dirty Pipe' ...> Subject: DirtyPipe vulnerability for VPSes - action required<br>
> Date: Fri, 11 Mar 2022 13:21:43 +0000
>
> Dear Eclips.is user,
>
> You may have heard of a recent vulnerability in the mainstream Linux Kernel, called the 'Dirty Pipe' vulnerability. This vulnerability may also be present in the kernel you are running on your VPS. The vulnerability means there is a risk that hackers have been able to inject code onto your VPS, but there is no
> indication that this vulnerability has actually been used to gain access to any of the machines.
>
> In order to fix the vulnerability, updated kernels are available. To switch to this updated kernel, follow the below steps:
>
> 1. Most users are using our default kernel. In this case we supply the kernel, you only have to reboot your VPS to make sure the latest version of the kernel is loaded.
> 2. Once you restarted your VPS, you can check to see if you are using the fixed version of the kernel. You can check if the update was successful by running: `uname -r`
> If the version is 5.10.104 or newer, you are no longer vulnerable to this exploit.
> 3. If you have restarted and the kernel version is still below 5.10.104 you should update your kernel yourself running:
> ```
> apt-get update && apt-get dist-upgrade
> ```
> After this update, you will need to reboot your VPS, and you will no longer be vulnerable to this exploit.
> 4. (Optional checks) Although we think chances are very slim, there is a theoretical possibility your system has already been compromised over the past days.
> If you want to make sure this has not happened, you could install debsums (`apt install debsums`) and run a full check on all installed packages to possibly detect files that were tampered with. You could also consider checking your /root/.ssh/authorized_keys for unknown entries.
>
> Note: Although for most use cases this check will be sufficient, it does not consitute a comprehensive security scan. If you suspect you have been hacked you should (hire someone to) perform a more thorough check.
>
> Please make sure to look into all of your VPSes.
>
> If you are interested in the full story, the reporter of this vulnerability wrote an extensive blogpost on this here: https://dirtypipe.cm4all.com/
>
> Please let us know if you have any remaining questions on this.David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40108standalone snowflake operators want a way to bind a specific outbound address2023-05-06T12:48:00ZRoger Dingledinestandalone snowflake operators want a way to bind a specific outbound addressThis is similar to Tor's ```OutboundBindAddress``` config option: if you're running headless snowflake on a server with multiple IP addresses, you want a way to control which IP address your outgoing connections will come from.
Tricks c...This is similar to Tor's ```OutboundBindAddress``` config option: if you're running headless snowflake on a server with multiple IP addresses, you want a way to control which IP address your outgoing connections will come from.
Tricks can be done with routing and iptables and etc, but those are OS-specific, require messing with stuff at the root level, etc. Maybe it is a two-line patch to let snowflake call bind() on the outbound connection socket?itchyonionitchyonionhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40107Do we have any pre-Dec2021 standalone snowflakes still? If so stop using them2022-07-07T12:17:59ZRoger DingledineDo we have any pre-Dec2021 standalone snowflakes still? If so stop using themPart of our Dec 2021 plan, when Russia blocked our pion webrtc traffic by DPI, was to get everybody to upgrade. The browser extension snowflakes don't use pion so they're immune, and the client side was handled by upgrading Tor Browser.
...Part of our Dec 2021 plan, when Russia blocked our pion webrtc traffic by DPI, was to get everybody to upgrade. The browser extension snowflakes don't use pion so they're immune, and the client side was handled by upgrading Tor Browser.
That left the standalone snowflakes. There are tickets like https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/32677 which are about how to communicate with standalone snowflake operators, how to get messages to them, how to make it easier for them to upgrade. But this ticket is much smaller in scope:
If people are still running standalone snowflakes that use the pion from before Dec 2021, we should decide, at the broker, to not make use of them.
(My notes from Dec 2021 say that standalone snowflakes say their version when they ask the broker if anybody needs a connect-back, so I am hoping this is an easy task. If it isn't easy, maybe we should triage it away and instead focus on making it easy next time.)https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40106proxy-go support for IGD2024-02-20T07:58:00ZJillproxy-go support for IGDI tried using the proxy-go in two setup, one on a laptop behind NAT, the other on a server with its own restrictive firewall.
In both cases the NAT type was detected as "restricted". My understanding is that at most one end of the conne...I tried using the proxy-go in two setup, one on a laptop behind NAT, the other on a server with its own restrictive firewall.
In both cases the NAT type was detected as "restricted". My understanding is that at most one end of the connection can be restricted,
so a restricted proxy can't talk to a restricted client. Getting an unrestricted NAT is better as it's compatible with more clients.
To that effect, proxy-go could use [IGD](https://en.wikipedia.org/wiki/Internet_Gateway_Device_Protocol) to ask the NAT to create a dynamic
port forwarding so it is effectively unrestricted. This would help with the laptop situation (assuming the router doing NAT supports IGD,
which mine does), and using something like [miniupnp](https://miniupnp.tuxfamily.org/), it would also be possible to dynamically open ports
on a local, restrictive, firewall.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40105Make the snowflake deb more accessible to Debian stable users2024-01-28T01:38:34ZRoger DingledineMake the snowflake deb more accessible to Debian stable usersWe made a Snowflake deb in https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/19409 and it is apparently now in Debian sid: <br>
https://tracker.debian.org/pkg/snowflake
Yay!
These versions are on...We made a Snowflake deb in https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/19409 and it is apparently now in Debian sid: <br>
https://tracker.debian.org/pkg/snowflake
Yay!
These versions are only in sid, right? Do we have a path in mind for normal Debian users to be able to install this deb? In particular, either a debian backports repository, or our deb.torproject.org repo.
(We have a policy that we only take things from Debian for our deb.tpo repo, but I think we would be following that policy here, since that deb *is* in Debian, it's just only in sid.)meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40104proxy lib - be able to configure the proxy type2022-03-21T18:25:21Zmeskiomeskio@torproject.orgproxy lib - be able to configure the proxy typeThe snowflake proxy library is being used by more clients than our standalone proxy. Library users should be able to set the proxy type that will be reported to the broker.
Currently the proxy type is hardcoded: https://gitlab.torprojec...The snowflake proxy library is being used by more clients than our standalone proxy. Library users should be able to set the proxy type that will be reported to the broker.
Currently the proxy type is hardcoded: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/main/proxy/lib/snowflake.go#L202
A use case for it now will be to have a different type in orbot, so we know how many snowflakes are provided by orbot users in comparison to other users.
We should take into account that currently the broker has a [hardcoded list of proxy types](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/main/broker/metrics.go#L26) and the rest is treated as 'unknown'. This was motivated by having a lot of requests with estrange proxy types (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40089). I guess we can extend the proxy type list for the mayor types we know off or we could do some simple validation of what kind of proxy types are meaningful.meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40103mix of past and present in snowflake proxy log2022-07-09T04:20:15Ztoralfmix of past and present in snowflake proxy log"In the last 1h0m0s, there are 28 connections. Traffic Relayed ↑ 273 MB, ↓ 273 MB."
are -> were
or ?"In the last 1h0m0s, there are 28 connections. Traffic Relayed ↑ 273 MB, ↓ 273 MB."
are -> were
or ?https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40102Several devices on the same network ?2022-03-01T15:55:27ZcypherpunksSeveral devices on the same network ?Hi !
I got in touch with you once with that question :
- Is it possible to install Snowflake on all the devices connected to the same local network ? (Ex : On a family, on 2 PCs)
Your answer :
- Mmm, it's not recommended.
My request :
...Hi !
I got in touch with you once with that question :
- Is it possible to install Snowflake on all the devices connected to the same local network ? (Ex : On a family, on 2 PCs)
Your answer :
- Mmm, it's not recommended.
My request :
- To know if your answer is still relevant
- Whatever, can you write in the Wiki/website FAQ, etc. ?
Thank you ! Tons of love. :) ♥https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40101Failures at standalone proxy2023-05-16T20:13:44ZCecylia BocovichFailures at standalone proxyWe got the following error log from a proxy volunteer:
```
2022/02/11 00:38:05 In the last 1h0m0s, there are 103 connections. Traffic Relayed ↑ 2 GB, ↓ 2 GB.
ortc ERROR: 2022/02/11 02:28:16 Failed to accept data channel: failed to se...We got the following error log from a proxy volunteer:
```
2022/02/11 00:38:05 In the last 1h0m0s, there are 103 connections. Traffic Relayed ↑ 2 GB, ↓ 2 GB.
ortc ERROR: 2022/02/11 02:28:16 Failed to accept data channel: failed to send ChannelOpen ACK: sending payload data in non-established state: state=Closed
ortc ERROR: 2022/02/11 02:29:25 Failed to accept data channel: failed to send ChannelOpen ACK: sending payload data in non-established state: state=Closed
2022/02/11 01:38:05 In the last 1h0m0s, there are 66 connections. Traffic Relayed ↑ 1 GB, ↓ 1 GB.
2022/02/11 02:38:05 In the last 1h0m0s, there are 74 connections. Traffic Relayed ↑ 560 MB, ↓ 560 MB.
2022/02/11 03:38:05 In the last 1h0m0s, there are 69 connections. Traffic Relayed ↑ 5 GB, ↓ 5 GB.
2022/02/11 04:38:05 In the last 1h0m0s, there are 68 connections. Traffic Relayed ↑ 1 GB, ↓ 1 GB.
2022/02/11 05:38:05 In the last 1h0m0s, there are 86 connections. Traffic Relayed ↑ 2 GB, ↓ 2 GB.
2022/02/11 06:38:05 In the last 1h0m0s, there are 91 connections. Traffic Relayed ↑ 2 GB, ↓ 2 GB.
2022/02/11 07:38:05 In the last 1h0m0s, there are 111 connections. Traffic Relayed ↑ 613 MB, ↓ 613 MB.
sctp ERROR: 2022/02/11 08:53:20 [0xc073c969c0] stream 1 not found)
sctp ERROR: 2022/02/11 08:53:20 [0xc073c969c0] stream 1 not found)
sctp ERROR: 2022/02/11 09:03:19 [0xc0a7166340] stream 1 not found)
ortc ERROR: 2022/02/11 09:15:45 Failed to accept data channel: failed to send ChannelOpen ACK: sending payload data in non-established state: state=Closed
sctp ERROR: 2022/02/11 09:37:53 [0xc0103c36c0] stream 1 not found)
sctp ERROR: 2022/02/11 09:37:53 [0xc0103c36c0] stream 1 not found)
2022/02/11 08:38:05 In the last 1h0m0s, there are 121 connections. Traffic Relayed ↑ 212 MB, ↓ 212 MB.
ortc ERROR: 2022/02/11 09:41:30 Failed to accept data channel: failed to send ChannelOpen ACK: sending payload data in non-established state: state=Closed
sctp ERROR: 2022/02/11 09:54:55 [0xc02e8e1860] stream 1 not found)
sctp ERROR: 2022/02/11 09:54:55 [0xc02e8e1860] stream 1 not found)
sctp ERROR: 2022/02/11 09:54:55 [0xc02e8e1860] stream 1 not found)
sctp ERROR: 2022/02/11 09:54:55 [0xc02e8e1860] stream 1 not found)
sctp ERROR: 2022/02/11 09:54:58 [0xc073d15380] stream 1 not found)
sctp ERROR: 2022/02/11 09:55:01 [0xc0103c29c0] stream 1 not found)
sctp ERROR: 2022/02/11 09:55:03 [0xc003828ea0] stream 1 not found)
sctp ERROR: 2022/02/11 09:55:03 [0xc003828ea0] stream 1 not found)
ortc ERROR: 2022/02/11 10:27:42 Failed to accept data channel: failed to send ChannelOpen ACK: sending payload data in non-established state: state=Closed
Killed
(by kernel OOM killer.)
```shelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40100Performance modelling of Snowflake2024-02-27T18:25:37ZCecylia BocovichPerformance modelling of SnowflakeAs a followup to previous discussions on Snowflake performance, the purpose of this issue is to track work on modelling and measuring the impact of Snowflake improvements on network performance. This work will be primarily done with the ...As a followup to previous discussions on Snowflake performance, the purpose of this issue is to track work on modelling and measuring the impact of Snowflake improvements on network performance. This work will be primarily done with the [Shadow](https://shadow.github.io/) network simulation tool. This tool can measure the impact that changes to Snowflake can have on the throughput of traffic for clients, as well as resource consumption of the broker and bridge.
Snowflake shadow simulation scripts can be found at https://gitlab.torproject.org/cohosh/snowflake-simulation
There are a few tasks to complete before we are ready to conduct performance experiments:
- [ ] Help Shadow developers debug outstanding issues with go network code
- [ ] Improve the Snowflake network model to accurately reflect the network conditions faced by both snowflake clients and proxy volunteers
- [ ] Improve the output format of test results so they can be easily interpreted
Once these pieces are in place, I plan to conduct the following experiments:
- [ ] Tune turbotunnel parameters by experimenting with the space of probable configurations (#40026)
- [ ] Splitting traffic across multiple snowflake proxies (#25723)
- [ ] The impact of geographic location of proxies on client performance (#31661)
Shadow simulations do have some limitations. We have also deployed onionperf instances to measure real-world Snowflake performance. If evidence for performance improvements is compelling enough, we can measure the impact of the change in deployment from these locations.Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40099`closed` field of SnowflakeListener is never initialized2022-02-11T15:08:07ZDavid Fifielddcf@torproject.org`closed` field of SnowflakeListener is never initialized`SnowflakeListener` is:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/e828b0607662c7325f4d1bbf4c5072ce81f38fb9/server/lib/snowflake.go#L165-172
```go
type SnowflakeListener struct {
addr ...`SnowflakeListener` is:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/e828b0607662c7325f4d1bbf4c5072ce81f38fb9/server/lib/snowflake.go#L165-172
```go
type SnowflakeListener struct {
addr net.Addr
queue chan net.Conn
server *http.Server
ln *kcp.Listener
closed chan struct{}
closeOnce sync.Once
}
```
When it is initialized, its `queue` channel is initialized but its `closed` channel is left `nil`:
```go
listener := &SnowflakeListener{addr: addr, queue: make(chan net.Conn, 65534)}
```
The effect of this is that in functions such as `SnowflakeListener.queueConn`, the `select` will never take the `closed` case, because reads from `nil` channels always block:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/e828b0607662c7325f4d1bbf4c5072ce81f38fb9/server/lib/snowflake.go#L273-280
```go
func (l *SnowflakeListener) queueConn(conn net.Conn) error {
select {
case <-l.closed:
return fmt.Errorf("accepted connection on closed listener")
case l.queue <- conn:
return nil
}
}
```https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40098snowflake_server.httpHandler.ln is not initialized, leading to panic in onesh...2022-07-09T04:20:16ZDavid Fifielddcf@torproject.orgsnowflake_server.httpHandler.ln is not initialized, leading to panic in oneshotModeThere are a few of these in the snowflake-server log:
```
2022/02/01 16:44:11 http: panic serving [scrubbed]: runtime error: invalid memory address or nil pointer dereference
goroutine 2513699 [running]:
net/http.(*conn).serve.func1(0xc...There are a few of these in the snowflake-server log:
```
2022/02/01 16:44:11 http: panic serving [scrubbed]: runtime error: invalid memory address or nil pointer dereference
goroutine 2513699 [running]:
net/http.(*conn).serve.func1(0xc007576dc0)
/usr/lib/go-1.15/src/net/http/server.go:1801 +0x147
panic(0x7ea7a0, 0xb0c450)
/usr/lib/go-1.15/src/runtime/panic.go:975 +0x47a
git.torproject.org/pluggable-transports/snowflake.git/v2/server/lib.(*SnowflakeListener).queueConn(0x0, 0x8dc820, 0xc04b943740, 0xc03557fbb0, 0xc04b9436e0)
./snowflake/server/lib/snowflake.go:275 +0x37
git.torproject.org/pluggable-transports/snowflake.git/v2/server/lib.oneshotMode(...)
./snowflake/server/lib/http.go:117
git.torproject.org/pluggable-transports/snowflake.git/v2/server/lib.(*httpHandler).ServeHTTP(0xc0001a0e80, 0x8d9800, 0xc057a75880, 0xc0dda2cd00)
./snowflake/server/lib/http.go:105 +0x5b7
net/http.serverHandler.ServeHTTP(0xc0001ee0e0, 0x8d9800, 0xc057a75880, 0xc0dda2cd00)
/usr/lib/go-1.15/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc007576dc0, 0x8da1c0, 0xc013b86b00)
/usr/lib/go-1.15/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
/usr/lib/go-1.15/src/net/http/server.go:2969 +0x36c
```
`oneshotMode` is called with `httpHandler.ln` as the `*SnowflakeListener` argument:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/e828b0607662c7325f4d1bbf4c5072ce81f38fb9/server/lib/http.go#L100-105
```go
// We didn't find a matching token, which means that we are
// dealing with a client that doesn't know about such things.
// "Unread" the token by constructing a new Reader and pass it
// to the old one-session-per-WebSocket mode.
conn2 := &overrideReadConn{Conn: conn, Reader: io.MultiReader(bytes.NewReader(token[:]), conn)}
err = oneshotMode(conn2, addr, handler.ln)
```
`httpHandler` is:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/e828b0607662c7325f4d1bbf4c5072ce81f38fb9/server/lib/http.go#L63-68
```go
type httpHandler struct {
// pconn is the adapter layer between stream-oriented WebSocket
// connections and the packet-oriented KCP layer.
pconn *turbotunnel.QueuePacketConn
ln *SnowflakeListener
}
```
But in the only place `httpHandler` is instantiated, its `ln` field is left uninitialized:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/e828b0607662c7325f4d1bbf4c5072ce81f38fb9/server/lib/snowflake.go#L80-85
```go
handler := httpHandler{
// pconn is shared among all connections to this server. It
// overlays packet-based client sessions on top of ephemeral
// WebSocket connections.
pconn: turbotunnel.NewQueuePacketConn(addr, clientMapTimeout),
}
```Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40097Help OONI developers to calibrate the new torsf OONI experiment2022-03-11T05:12:10ZsbsHelp OONI developers to calibrate the new torsf OONI experimentGood day, Snowflake developers!
We recently did QA to understand and calibrate the now `torsf` (tor + snowflake) experiment we have written and (TL;DR) we were quite surprised by the amount of bootstrap timeouts we observed on mobile ne...Good day, Snowflake developers!
We recently did QA to understand and calibrate the now `torsf` (tor + snowflake) experiment we have written and (TL;DR) we were quite surprised by the amount of bootstrap timeouts we observed on mobile networks.
I've explained the problem we're having in great details and posed questions for you here https://github.com/ooni/probe/issues/2004. It would be awesome if someone of you could take a look and help us out!
Thanks a lot! :pray: :slightly_smiling_face:Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40096Too many open files error at probetest2022-09-24T14:42:19ZCecylia BocovichToo many open files error at probetestI took a look at the NAT probetest logs this morning after the [alert yesterday](https://lists.torproject.org/pipermail/anti-censorship-alerts/2022-January/000924.html). It doesn't look like probetest is stuck, but I did notice a familia...I took a look at the NAT probetest logs this morning after the [alert yesterday](https://lists.torproject.org/pipermail/anti-censorship-alerts/2022-January/000924.html). It doesn't look like probetest is stuck, but I did notice a familiar message in the logs:
```
2022/01/29 21:02:32 http: Accept error: accept tcp [scrubbed]: accept4: too many
open files; retrying in 5ms
```
And when I run a proxy, I frequently get http timeout errors, like due to this issue on the server.
We ran into this recently with the load balanced bridge (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40095#note_2772325) and previously with the broker (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/31425).
The NAT probetest is run with sv, and it's `run` file contains the following:
```
#!/bin/sh -e
export GOMAXPROCS=1
setcap 'cap_net_bind_service=+ep' /usr/local/bin/probetest
exec ip netns exec net0 /usr/local/bin/probetest --acme-hostnames snowflake-broker.bamsoftware.com,snowflake-broker.freehaven.net,snowflake-broker.torproject.net --acme-email dcf@torproject.org --acme-cert-cache /home/snowflake-broker/acme-cert-cache --addr :8443 2>&1
```https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40093Rendez-vous via tailscale derp server2024-02-27T19:09:31ZArlo BreaultRendez-vous via tailscale derp serverhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40092Improve docs on network_mode: host (and network in general)2023-01-18T16:18:15ZchmacImprove docs on network_mode: host (and network in general)When I found this repo, the example line `network_mode: host` jumped out at me as suspicious. I looked up the docs and figured that it's probably because snowflake requires lots of ports or so. I figured that my trust in the tor project ...When I found this repo, the example line `network_mode: host` jumped out at me as suspicious. I looked up the docs and figured that it's probably because snowflake requires lots of ports or so. I figured that my trust in the tor project is pretty high, and so I'm running a snowflake node.
But, I'm not really sure what network conditions it needs. Does it expect that `network_mode: host` means it's running on a host which has a publicly accessible IP? Does it needs ports on that host's firewall open?
The idea behind this issue is to improve the docs in this area so that snowflake hosts like myself can figure out what network conditions are required for snowflake to work. For example, I have no idea if my node is actually functional right now, I also have no idea how to test it.
Some example questions we could aim to answer:
- What ports does snowflake run on?
- Does snowflake need to be run on a machine with a public IP?
- Does snowflake run properly if behind a NAT?
- Does snowflake require specific ports to be opened in the system firewall?
- How can a server admin test if snowflake is properly configured and working?
As an add on, it would be great to see answers to questions like these:
- How much bandwidth can one expect snowflake to use?
- Does it make sense to add any kind of limits?
- If so, how would that be done?
- Are there any security considerations to running a snowflake server?
- What sort of system resources (CPU, memory) does snowflake use?
- Does it make sense to check on this periodically for memory leaks, etc?
- How can one be notified when updates are published to the docker image?
- Is there a security mailing list where one could be notified of any security issues that require urgent update of the snowflake server?
Finally, thanks for making the tor network more resilient, snowflake looks to be an awesome improvement for people in locations with internet censorship, and thanks for working on tor in general, it's a phenomenal resource supporting the human experience.