Snowflake issueshttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues2023-06-08T19:26:08Zhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40198Out of localhost ephemeral ports ("cannot assign requested address") on link ...2023-06-08T19:26:08ZDavid Fifielddcf@torproject.orgOut of localhost ephemeral ports ("cannot assign requested address") on link between snowflake-server and haproxySince 2022-09-28 10:57:26 (53 hours ago) the snowflake-server log is full of:
```
# grep 'cannot assign requested address$' /var/log/snowflake-server/snowflake-server.log | head
2022/09/28 10:57:26 handleConn: failed to connect to ORPor...Since 2022-09-28 10:57:26 (53 hours ago) the snowflake-server log is full of:
```
# grep 'cannot assign requested address$' /var/log/snowflake-server/snowflake-server.log | head
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
# grep 'cannot assign requested address$' /var/log/snowflake-server/snowflake-server.log | tail
2022/09/30 15:44:17 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/30 15:44:18 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/30 15:44:18 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/30 15:44:18 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/30 15:44:18 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
```
The error message means that the kernel could not allocate an ephemeral port number
for a localhost TCP connection.
In this case it for the connection between snowflake-server and haproxy.
My analysis at https://lists.torproject.org/pipermail/anti-censorship-team/2022-September/000265.html was incomplete:
the total number of localhost sockets does not matter for the purpose of counting 4-tuples,
but it *does* matter for the purpose of allocating ephemeral ports.
There are currently only 21712 sockets open between snowflake-server and haproxy,
which means the remainder of the ephemeral port range is taken up by other
localhost communication.
My immediate plan for this is to move haproxy to 127.0.0.2:10000 rather than 127.0.0.1:10000,
and maybe similarly with the individual tor instances if necessary.
/cc @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.org2022-09-30https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40197nil pointer dereference in PeerConnection.PendingLocalDescription2022-10-03T11:53:14Zcypherpunksnil pointer dereference in PeerConnection.PendingLocalDescription```
info] {EDGE} connection_edge_process_inbuf(): data from edge while in 'waiting for circuit' state. Leaving it on buffer.
info] {EDGE} connection_edge_process_inbuf(): data from edge while in 'waiting for circuit' state. Leaving it on...```
info] {EDGE} connection_edge_process_inbuf(): data from edge while in 'waiting for circuit' state. Leaving it on buffer.
info] {EDGE} connection_edge_process_inbuf(): data from edge while in 'waiting for circuit' state. Leaving it on buffer.
info] {NET} parse_socks_client(): SOCKS 5 client: continuing without authentication
info] {NET} connection_read_proxy_handshake(): Proxy Client: OR connection (handshaking (proxy)) with 192.0.2.3:80 ID=1zOHpg+FxqQfi/6jDLtCpHHqBTH8gjYmCKXkus1D5Ko RSA_ID=2B280B23E1107BB62ABFC40DDCC8824814F80A72 successful
info] {BTRACK} bto_update_best(): ORCONN BEST_ANY state 2->3 gid=4
notice] {CONTROL} Bootstrapped 10% (conn_done): Connected to a relay
info] {BTRACK} bto_update_best(): ORCONN BEST_AP state 2->3 gid=4
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: panic: runtime error: invalid memory address or nil pointer dereference
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: [signal 0xc0000005 code=0x0 addr=0x0 pc=0x5ed17d]
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error:
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: goroutine 53 [running]:
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: github.com/pion/webrtc.(*PeerConnection).PendingLocalDescription(0x0)
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: /var/tmp/dist/gopath/src/github.com/pion/webrtc/peerconnection.go:2026 +0x1d
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: github.com/pion/webrtc.(*PeerConnection).LocalDescription(0xc00034c000)
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: /var/tmp/dist/gopath/src/github.com/pion/webrtc/peerconnection.go:1007 +0x1e
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib.(*WebRTCPeer).connect(0xc00034c000, 0x0, 0xc000345d48)
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: /var/tmp/dist/gopath/src/git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib/webrtc.go:150 +0xd8
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib.NewWebRTCPeerWithEvents(0x35ee40, 0xc0000d6000, {0x223f8f30008, 0xc00022e220})
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: /var/tmp/dist/gopath/src/git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib/webrtc.go:73 +0x38b
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib.WebRTCDialer.Catch(...)
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: /var/tmp/dist/gopath/src/git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib/rendezvous.go:172
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib.(*Peers).Collect(0xc000234080)
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: /var/tmp/dist/gopath/src/git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib/peers.go:69 +0x223
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib.connectLoop({0x846bd0, 0xc000234080})
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: /var/tmp/dist/gopath/src/git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib/snowflake.go:345 +0x56
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: created by git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib.(*Transport).Dial
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: /var/tmp/dist/gopath/src/git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib/snowflake.go:170 +0x206
info] {NET} TLS error: <syscall error while handshaking> (errno=10054: Connection reset by peer [WSAECONNRESET ]; state=SSLv3/TLS write client hello)
info] {OR} connection_tls_continue_handshake(): tls error [connection reset]. breaking connection.
info] {CIRC} circuit_n_chan_done(): Channel failed; closing circ.
info] {GENERAL} circuit_mark_for_close_(): Circuit 0 (id: 1) marked for close at circuitbuild.c:687 (orig reason: 8, new reason: 0)
info] {HANDSHAKE} connection_or_note_state_when_broken(): Connection died in state 'handshaking (TLS) with SSL state SSLv3/TLS write client hello in HANDSHAKE'
info] {BTRACK} bto_status_rcvr(): ORCONN DELETE gid=4 status=2 reason=4
warn] {CONTROL} Problem bootstrapping. Stuck at 10% (conn_done): Connected to a relay. (CONNECTRESET; CONNECTRESET; count 1; recommendation warn; host 2B280B23E1107BB62ABFC40DDCC8824814F80A72 at 192.0.2.3:80)
warn] {HANDSHAKE} 1 connections have failed:
warn] {HANDSHAKE} 1 connections died in state handshaking (TLS) with SSL state SSLv3/TLS write client hello in HANDSHAKE
info] {OR} circuit_build_failed(): Our circuit 0 (id: 1) died before the first hop with no connection
info] {GUARD} entry_guards_note_guard_failure(): Recorded failure for primary guard $2B280B23E1107BB62ABFC40DDCC8824814F80A72 ($2B280B23E1107BB62ABFC40DDCC8824814F80A72)
info] {CIRC} circuit_free_(): Circuit 0 (id: 1) has been freed.
warn] {PT} Pluggable Transport process terminated with status code 2
```https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40196snowflake plugin not working2024-02-14T16:43:33Zcypherpunkssnowflake plugin not workingsnowflake plugin on firefox is down. saying canßt connect to bridge.
i am in Germany.
_edited to have a clear title_snowflake plugin on firefox is down. saying canßt connect to bridge.
i am in Germany.
_edited to have a clear title_https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40195OOM killer invoked at broker2022-10-12T19:26:27ZCecylia BocovichOOM killer invoked at brokerThere have been outages the last two nights at the snowflake broker, I was puzzled at first because there's no errors in the broker logs. I just checked the syslog and it looks like the OOM killer has been invoked:
```
Sep 30 09:36:17 s...There have been outages the last two nights at the snowflake broker, I was puzzled at first because there's no errors in the broker logs. I just checked the syslog and it looks like the OOM killer has been invoked:
```
Sep 30 09:36:17 snowflake-broker kernel: [17516237.845995] probetest invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
```
This one is for the probetest, but it's likely the broker outages was due to this as well.Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40194Perform more regular dependency updates2023-07-04T14:27:43ZCecylia BocovichPerform more regular dependency updatesDependency updates happen in an adhoc way when we think of it or if there's a CVE we know about. I'd like to look into doing this more regularly, perhaps with the help of bots. I'd also like to revisit the pion dependency update process ...Dependency updates happen in an adhoc way when we think of it or if there's a CVE we know about. I'd like to look into doing this more regularly, perhaps with the help of bots. I'd also like to revisit the pion dependency update process for browser builds because the script at https://gitlab.torproject.org/-/snippets/145 needs hacks every time and the process still takes ages.meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40192Experiment with bypassing extor-static-cookie on snowflake-012022-12-12T01:03:33ZDavid Fifielddcf@torproject.orgExperiment with bypassing extor-static-cookie on snowflake-01The [12](tpo/anti-censorship/pluggable-transports/snowflake#40176) extor-static-cookie processes
collectively use about 300% of a CPU core (25% each).
We can open up some CPU headroom by cutting them out of the pipeline—but
then we need ...The [12](tpo/anti-censorship/pluggable-transports/snowflake#40176) extor-static-cookie processes
collectively use about 300% of a CPU core (25% each).
We can open up some CPU headroom by cutting them out of the pipeline—but
then we need another way doing static ExtORPort authentication.
https://lists.torproject.org/pipermail/anti-censorship-team/2022-September/000253.html
> For CPU pressure, I don't see any quick fixes. In an emergency, we could
> hack the tor binary to use a static ExtORPort authentication cookie, and
> remove the extor-static-cookie shim from the pipeline.
There's also the idea that the extra localhost communication required by
extor-static-cookie is a cause of the current performance bottleneck.
https://lists.torproject.org/pipermail/anti-censorship-team/2022-September/000263.html
> First, let's patch tor to get rid of the extor processes, as suggested
> by David earlier when discussing RAM pressure. This should bring down
> context switches.
Cf. [Two features that would help load-balanced bridges](https://lists.torproject.org/pipermail/tor-dev/2022-February/014695.html).
As a preliminary test to see if removing extor-static-cookie actually has an effect,
this issue is to try bypassing extor-static-cookie and having haproxy connect directly
to the "regular" ORPort of the tor processes.
The downside of this is that we will not be counting transport- and country-specific
metrics while the experiment is in place.
But it should take only a few hours maximum to see if it has an effect.
/cc @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40191snowflake-01: Performance experiments and enhancements2022-11-05T20:26:17ZLinus Nordberglinus@torproject.orgsnowflake-01: Performance experiments and enhancementsCollecting information and actions related to performance on snowflake-01 here.
General situation 2022-09-29 is that we won't push more than 3Gbps with a flatline on both bw and pps (a bit over 400kpps) and we're uncertain where the bot...Collecting information and actions related to performance on snowflake-01 here.
General situation 2022-09-29 is that we won't push more than 3Gbps with a flatline on both bw and pps (a bit over 400kpps) and we're uncertain where the bottleneck is at.Linus Nordberglinus@torproject.orgLinus Nordberglinus@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40189snowflake-01: Disable conntrack2022-12-13T20:25:17ZLinus Nordberglinus@torproject.orgsnowflake-01: Disable conntrackBased on output from perf(1) we should disable conntrack. We would have to deal with that regardless, since we're running out of slots in the conntrack table.Based on output from perf(1) we should disable conntrack. We would have to deal with that regardless, since we're running out of slots in the conntrack table.Linus Nordberglinus@torproject.orgLinus Nordberglinus@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40188snowflake-01: Reboot2022-10-02T07:51:26ZLinus Nordberglinus@torproject.orgsnowflake-01: RebootIn order to restart dbus, a reboot might be the best option.
```
Service restarts being deferred:
/etc/needrestart/restart.d/dbus.service
```
LMK if you think there are better ways. Note that we lack all OOB on this server (like IPMI,...In order to restart dbus, a reboot might be the best option.
```
Service restarts being deferred:
/etc/needrestart/restart.d/dbus.service
```
LMK if you think there are better ways. Note that we lack all OOB on this server (like IPMI, ILO access), so we really don't want to mess up.
I'm planning on doing the reboot in connection to #40186.Linus Nordberglinus@torproject.orgLinus Nordberglinus@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40187Reduce allocation in `QueuePacketConn.QueueIncoming`.2023-03-13T19:04:22ZDavid Fifielddcf@torproject.orgReduce allocation in `QueuePacketConn.QueueIncoming`.`QueueIncoming` [makes a copy](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/v2.3.1/common/turbotunnel/queuepacketconn.go#L54) of the passed in slice
before queuing it.
The idea behind this desig...`QueueIncoming` [makes a copy](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/v2.3.1/common/turbotunnel/queuepacketconn.go#L54) of the passed in slice
before queuing it.
The idea behind this design is to make the function harder to misuse:
the caller cannot modify the queued packet after calling `QueueIncoming`.
But it means an extra malloc and memmove per queued packet.
As long as the caller does not reuse the slice,
it is okay not to make a copy and it's more efficient.
These allocations are suspected to be a cause of frequent
garbage collections in
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40086#note_2838372.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40186snowflake-01: Connect the other NIC and move wireguard and sshd to it2022-09-30T16:00:58ZLinus Nordberglinus@torproject.orgsnowflake-01: Connect the other NIC and move wireguard and sshd to itThis will allow for
- skipping conntracking on the 10G NIC
- ethtool tuning of the 10G NIC without risking losing the hostThis will allow for
- skipping conntracking on the 10G NIC
- ethtool tuning of the 10G NIC without risking losing the hostLinus Nordberglinus@torproject.orgLinus Nordberglinus@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40185overflow in bandwidth reporting2022-11-21T15:34:42Ztrinity-1686aoverflow in bandwidth reportingA user reported on `#tor` they see strange bandwidth report on their snowflake proxy.
![image](/uploads/2840f9598f1d194a89058c04a84023e4/image.png)
It looks very much like an overflowed signed 32b integer. They use snowflake on raspber...A user reported on `#tor` they see strange bandwidth report on their snowflake proxy.
![image](/uploads/2840f9598f1d194a89058c04a84023e4/image.png)
It looks very much like an overflowed signed 32b integer. They use snowflake on raspberry pi 3 (64 bit), however I've heard more than one time of things going 32b on raspberries, so may be reproducible only in 32b modehttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40184fixed unit for bandwidth logging2022-11-11T14:38:01Zinvokedfixed unit for bandwidth loggingCurrently, the proxy logs bandwidth in changing units as the bandwidth scales up (KB->MB->GB). It might be preferable for the proxy operators to have a fixed unit instead. The problems associated with the way it currently works is:
1....Currently, the proxy logs bandwidth in changing units as the bandwidth scales up (KB->MB->GB). It might be preferable for the proxy operators to have a fixed unit instead. The problems associated with the way it currently works is:
1. If the proxy operator wants to scrape the logs for bandwidth data for use in other programs, the unit keeps changing.
2. Units like GB can cause the values of 1 or 2 to be far less meaningful when the proxy won't see bandwidth in the GB range consistently.
I would suggest fixing the values to KB unit.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40183Change snowflake proxy log verbosity2022-11-16T18:19:47ZCecylia BocovichChange snowflake proxy log verbosityWe've had a few operators of the standalone snowflake proxy approach us thinking that their proxy is not working because they are not seeing any logs.
Right now, the proxy will output nothing unless the `-verbose` argument is given. I'm...We've had a few operators of the standalone snowflake proxy approach us thinking that their proxy is not working because they are not seeing any logs.
Right now, the proxy will output nothing unless the `-verbose` argument is given. I'm thinking we can change this to log just a very few things by default:
- the starting message
- the NAT type
- the periodic metrics
and leave everything else as verbose logging.itchyonionitchyonionhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40182Snowflake Broker Deployment 26-09-222022-09-27T15:43:56ZshelikhooSnowflake Broker Deployment 26-09-22Code to be deployed: v2.3.1
## Deployment Script
```
sv stop snowflake-broker
cp /usr/local/bin/broker ./snowflake-broker-22-09-26-backup-$(date +%N)
install --owner root ./snowflake-broker-22-09-26-candidcate /usr/local/bin/broker
s...Code to be deployed: v2.3.1
## Deployment Script
```
sv stop snowflake-broker
cp /usr/local/bin/broker ./snowflake-broker-22-09-26-backup-$(date +%N)
install --owner root ./snowflake-broker-22-09-26-candidcate /usr/local/bin/broker
sv start snowflake-broker
```shelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40181Container doesn't start2022-09-29T18:27:59ZcypherpunksContainer doesn't startWhen running docker compose up, I don't get any further output after "Attaching to snowflake-proxy":
```
$ docker compose up snowflake-proxy
[+] Running 1/1
⠿ Container snowflake-proxy Created ...When running docker compose up, I don't get any further output after "Attaching to snowflake-proxy":
```
$ docker compose up snowflake-proxy
[+] Running 1/1
⠿ Container snowflake-proxy Created 0.2s
Attaching to snowflake-proxy
```
I'm running Docker Compose version v2.10.2 and Docker Engine - Community Version: 20.10.18.
I've also downloaded the docker-compose.yml and didn't change anything:
```
$ cat docker-compose.yml
version: "3.8"
services:
snowflake-proxy:
network_mode: host
image: thetorproject/snowflake-proxy:latest
container_name: snowflake-proxy
restart: unless-stopped
```
I'm running this on a Ubuntu 22.04 LTS LXC. I'm running similar LXCs with more than 20 different docker containers and never experienced an issue like this.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40180snowflake-01 planned hardware maintenance2022-09-26T19:02:18ZLinus Nordberglinus@torproject.orgsnowflake-01 planned hardware maintenancesnowflake-01 will be offline for about 10 minutes starting in approx 3h from now (ie 2022-09-26 13:10 to 13:20).
All times in UTC.snowflake-01 will be offline for about 10 minutes starting in approx 3h from now (ie 2022-09-26 13:10 to 13:20).
All times in UTC.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40179Reduce turbotunnel queueSize from 2048 to 5122022-12-08T15:04:48ZDavid Fifielddcf@torproject.orgReduce turbotunnel queueSize from 2048 to 512After yesterday's deployment of `io.Copy` memory mitigations (tpo/anti-censorship/pluggable-transports/snowflake#40177), memory use has crept up above 90% again.
I recommend implementing the [next most effective memory save](https://list...After yesterday's deployment of `io.Copy` memory mitigations (tpo/anti-censorship/pluggable-transports/snowflake#40177), memory use has crept up above 90% again.
I recommend implementing the [next most effective memory save](https://lists.torproject.org/pipermail/anti-censorship-team/2022-September/000253.html)
revealed by [profiling](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40086#note_2837328): decreasing the turbotunnel `queueSize`.
This is a fixed block of memory that's allocated per connected client.
In https://gitlab.torproject.org/dcf/snowflake/-/commit/328f0f4137145c08e58311888ad359362abaea79
I've reduced the parameter from 2048 to 512.
I don't know what effect this might have on performance,
apart from reducing memory use.
Past discussion about `queueSize`:
* https://lists.torproject.org/pipermail/anti-censorship-team/2021-July/000188.html
* https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/merge_requests/48#note_2744619David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.org2022-09-27https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40178Handle unknown client NAT type better to reduce load on restricted proxy pool2022-10-12T18:01:31ZCecylia BocovichHandle unknown client NAT type better to reduce load on restricted proxy poolWe're seeing a large number of clientpolls with unknown NAT types:
![image](/uploads/af5a7125fb7f8168b2bf6d2637a2ebaf/image.png)
To be safe, we treat unknown client NATs the same as restricted client NATs, so they pull from the smaller...We're seeing a large number of clientpolls with unknown NAT types:
![image](/uploads/af5a7125fb7f8168b2bf6d2637a2ebaf/image.png)
To be safe, we treat unknown client NATs the same as restricted client NATs, so they pull from the smaller pool of proxies that are known to work with symmetrics NATs. It's possible we can relieve at least some of the pressure on this proxy pool by having unknown clients first try proxies from the unrestricted pool and then fall back to the restricted pool if there is a failure to connect.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40177Deploy further snowflake-server performance improvements 2022-09-242023-03-10T21:31:54ZDavid Fifielddcf@torproject.orgDeploy further snowflake-server performance improvements 2022-09-24The snowflake-01 bridge is getting close to its memory limits again after yesterday's deployments
(tpo/anti-censorship/pluggable-transports/snowflake#40173, tpo/anti-censorship/pluggable-transports/snowflake#40175).
The little dip marked...The snowflake-01 bridge is getting close to its memory limits again after yesterday's deployments
(tpo/anti-censorship/pluggable-transports/snowflake#40173, tpo/anti-censorship/pluggable-transports/snowflake#40175).
The little dip marked in the graph below, I investigated it,
and it occurred during a time when the memory was almost 100% full.
You can also tell the server is reaching its limits because the send and recv traces
become a little bit decorrelated.
For comparison, currently, at the right edge of the graph, RAM use is 85%.
![bandwidth on network interface](/uploads/a05f736ab902e70d569fd7dd78bb9abe/g2.png)
[Profiling yesterday](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40086#note_2837328) revealed two major contributors to memory usage:
`io.Copy` buffers and client send queues, each of which accounts for about 25%
of the total memory used by the process.
I have a branch that is meant to resolve the `io.Copy` buffer one.
It exposes the `io.WriterTo` interface of `smux.Stream` through `SnowflakeClientConn`,
which will prevent `io.Copy` from allocating a buffer per client.
I also put in a minor optimization to `ClientMap.SendQueue`,
the benefit of which turns out to be tiny,
but there's no reason not to do it.
https://gitlab.torproject.org/dcf/snowflake/-/compare/a16133ff03d4a7f05bea375e5aa34ff794ee316c...4d4fad30c429bba9062d1b67d56787d45721627d?from_project_id=850
/cc @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.org2022-09-25