Snowflake issueshttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues2022-12-04T17:19:49Zhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40208"./proxy -h" does not show default value of a -capacity parameter2022-12-04T17:19:49Zslrslr"./proxy -h" does not show default value of a -capacity parameterHello at
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/tree/main/proxy
"proxy -h"
Regarding “-capacity” parameter it would be handy to know what value is used when i do not use the -capacity switch.Hello at
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/tree/main/proxy
"proxy -h"
Regarding “-capacity” parameter it would be handy to know what value is used when i do not use the -capacity switch.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40207Sudden reduction in snowflake-01 bridge bandwidth, 2022-10-04 17:152024-01-22T15:19:37ZDavid Fifielddcf@torproject.orgSudden reduction in snowflake-01 bridge bandwidth, 2022-10-04 17:15At around 2022-10-04 17:15 (about 2.75 hours ago), bandwidth on the bridge suddenly dropped
from about 3.2/2.4 Gbps TX/RX to about 1.0/0.8 Gbps TX/RX.
![snowflake-01 bandwidth on eno1](/uploads/96531ead7639867efd60f00881131ff8/snowflake...At around 2022-10-04 17:15 (about 2.75 hours ago), bandwidth on the bridge suddenly dropped
from about 3.2/2.4 Gbps TX/RX to about 1.0/0.8 Gbps TX/RX.
![snowflake-01 bandwidth on eno1](/uploads/96531ead7639867efd60f00881131ff8/snowflake-procnetdev-bw-20221006.png)
There is normally a nightly drop in usage, but this drop is several hours too early for that. It's also unusually flat.
/cc @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40206Broker status 500 when client requests fingerprint of snowflake-022023-03-14T16:42:43ZDavid Fifielddcf@torproject.orgBroker status 500 when client requests fingerprint of snowflake-02I was not sure whether distributed bridge support
(#40129, #28651) was supposed to be working yet, so I tried it.
I am using the client from commit 9ce1de4eee4e23c918c7c5e96666ff5c6ddc654e.
When I tried rendezvousing with the fingerprint...I was not sure whether distributed bridge support
(#40129, #28651) was supposed to be working yet, so I tried it.
I am using the client from commit 9ce1de4eee4e23c918c7c5e96666ff5c6ddc654e.
When I tried rendezvousing with the fingerprint of the
[snowflake-02 bridge](tpo/anti-censorship/pluggable-transports/snowflake#40122),
I got a 500 Internal Server Error, which was unexpected.
It makes me wonder if the broker is encountering an internal error and panicking,
or if 500 is the intended response code.
This torrc file works:
```
UseBridges 1
SocksPort auto
DataDirectory datadir
ClientTransportPlugin snowflake exec ./client -log snowflake.log
Bridge snowflake 192.0.2.3:80 2B280B23E1107BB62ABFC40DDCC8824814F80A72 fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://snowflake-broker.torproject.net/ ice=stun:stun.voip.blackberry.com:3478,stun:stun.altar.com.pl:3478,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:3478,stun:stun.epygi.com:3478,stun:stun.sonetel.com:3478,stun:stun.sonetel.net:3478,stun:stun.stunprotocol.org:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478
```
This torrc file does not work. The only difference is changing the fingerprint
`2B280B23E1107BB62ABFC40DDCC8824814F80A72` to `8838024498816A039FCBBAB14E6F40A0843051FA`
in both places.
```
UseBridges 1
SocksPort auto
DataDirectory datadir
ClientTransportPlugin snowflake exec ./client -log snowflake.log
Bridge snowflake 192.0.2.3:80 8838024498816A039FCBBAB14E6F40A0843051FA fingerprint=8838024498816A039FCBBAB14E6F40A0843051FA url=https://snowflake-broker.torproject.net/ ice=stun:stun.voip.blackberry.com:3478,stun:stun.altar.com.pl:3478,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:3478,stun:stun.epygi.com:3478,stun:stun.sonetel.com:3478,stun:stun.sonetel.net:3478,stun:stun.stunprotocol.org:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478
```
The broker's HTTP response is:
```
500 Internal Server Error
Access-Control-Allow-Headers: Origin, X-Session-ID
Access-Control-Allow-Origin: *
Content-Length: 0
Date: Tue, 04 Oct 2022 16:33:06 GMT
```https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40205Remove GOMAXPROCS=1 in the next deployment2022-10-25T16:34:16ZshelikhooRemove GOMAXPROCS=1 in the next deploymentWe should consider removing this unnecessary setting in the next deployment.
```
[6:12:21 pm] <dcf1> shelikhoo: it is also likely that GOMAXPROCS=1 can be removed from the run script. That was an experiment 2 years ago that turned out no...We should consider removing this unnecessary setting in the next deployment.
```
[6:12:21 pm] <dcf1> shelikhoo: it is also likely that GOMAXPROCS=1 can be removed from the run script. That was an experiment 2 years ago that turned out not to have an effect: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/32576#note_2594746
```shelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40204pion errors don't go into the log2022-12-03T13:45:30ZRoger Dingledinepion errors don't go into the logMy snowflake proxy tells me, I guess on either stdout or stderr,
```
sctp ERROR: 2022/10/03 13:47:32 [0xc002986380] stream 1 not found)
sctp ERROR: 2022/10/03 13:47:32 [0xc002986380] stream 1 not found)
sctp ERROR: 2022/10/03 13:47:32 [0...My snowflake proxy tells me, I guess on either stdout or stderr,
```
sctp ERROR: 2022/10/03 13:47:32 [0xc002986380] stream 1 not found)
sctp ERROR: 2022/10/03 13:47:32 [0xc002986380] stream 1 not found)
sctp ERROR: 2022/10/03 13:47:32 [0xc002986380] stream 1 not found)
sctp ERROR: 2022/10/03 13:47:32 [0xc002986380] stream 1 not found)
sctp ERROR: 2022/10/03 13:47:32 [0xc002986380] stream 1 not found)
sctp ERROR: 2022/10/03 13:47:32 [0xc002986380] stream 1 not found)
sctp ERROR: 2022/10/03 13:47:32 [0xc002986380] stream 1 not found)
sctp ERROR: 2022/10/03 13:47:32 [0xc002986380] stream 1 not found)
```
but I am using -log, and these lines don't show up in the log. It is unexpected that "error" category messages would be the ones that are transient and not captured for posterity.
(Also, the timestamps in the log seem to be utc, and the timestamps on my stdout/stderr appear to be local timezone. Not sure if that merits a separate ticket -- let me know if yes and I can open it.)Linus Nordberglinus@torproject.orgLinus Nordberglinus@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40203Restore firewall rules on snowflake-01 (minus conntrack)2023-10-06T03:11:45ZDavid Fifielddcf@torproject.orgRestore firewall rules on snowflake-01 (minus conntrack)In tpo/anti-censorship/pluggable-transports/snowflake#40189 we disabled conntrack
by removing the firewall rules.
We should restore a sensible firewall configuration,
minus the conntrack rules,
if it proves not to harm performance.
/cc ...In tpo/anti-censorship/pluggable-transports/snowflake#40189 we disabled conntrack
by removing the firewall rules.
We should restore a sensible firewall configuration,
minus the conntrack rules,
if it proves not to harm performance.
/cc @linusLinus Nordberglinus@torproject.orgLinus Nordberglinus@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40202Upgrade memory capacity of broker VPS2022-10-06T19:07:09ZDavid Fifielddcf@torproject.orgUpgrade memory capacity of broker VPSPer tpo/anti-censorship/pluggable-transports/snowflake#40195,
move the broker host to a configuration with more memory.
Past issues about VPS upgrades:
* tpo/anti-censorship/pluggable-transports/snowflake#40051
* tpo/anti-censorship/plu...Per tpo/anti-censorship/pluggable-transports/snowflake#40195,
move the broker host to a configuration with more memory.
Past issues about VPS upgrades:
* tpo/anti-censorship/pluggable-transports/snowflake#40051
* tpo/anti-censorship/pluggable-transports/snowflake#40085David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40201Out of ephemeral ports on link between haproxy and extor-static-cookie2023-05-22T16:22:31ZDavid Fifielddcf@torproject.orgOut of ephemeral ports on link between haproxy and extor-static-cookieThe multi-KCP optimization parallelization of #40200
has enabled a large increase in bandwidth, currently above 3.5 Gbps outgoing / 2.5 Gbps incoming.
But now we are experiencing another ephemeral port exhaustion issue like #40198.
This ...The multi-KCP optimization parallelization of #40200
has enabled a large increase in bandwidth, currently above 3.5 Gbps outgoing / 2.5 Gbps incoming.
But now we are experiencing another ephemeral port exhaustion issue like #40198.
This time it's on the link between haproxy and extor-static-cookie.
The snowflake-server log says:
```
2022/10/02 05:06:47 handleConn: failed to connect to ORPort: EOF
```
The haproxy log says:
```
Oct 2 05:06:53 snowflake-01 haproxy[9134]: Connect() failed for backend tor-instances: no free ports.
```
For context, there are about 38000 live connections between snowflake-server and haproxy.
```
# ss -f inet -n state established | grep '127.0.0.1:10000\s*$' | wc -l
37918
```
/cc @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40200Use multiple parallel KCP state machines2023-07-18T00:47:29ZDavid Fifielddcf@torproject.orgUse multiple parallel KCP state machinesI have a suspicion that KCP packet scheduling may be a bottleneck in the server.
Most of the server's processing is concurrent and scales across multiple CPU cores,
but all the traffic passes through the centralized `kcp.Listener` schedu...I have a suspicion that KCP packet scheduling may be a bottleneck in the server.
Most of the server's processing is concurrent and scales across multiple CPU cores,
but all the traffic passes through the centralized `kcp.Listener` scheduler
that is created [in `Transport.Listen`](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/v2.3.1/server/lib/snowflake.go#L145).
We can actually use multiple independent KCP schedulers,
as long as we consistently assign the packets of one particular session
to the same KCP.
And we can do that using the ClientID associated with each session.
I have a branch for this at
https://gitlab.torproject.org/dcf/snowflake/-/commits/multi-kcp.
I'm planning to test it as part of !100.David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40199Reduce allocation in `QueuePacketConn.WriteTo`2023-07-18T00:47:30ZDavid Fifielddcf@torproject.orgReduce allocation in `QueuePacketConn.WriteTo`Like with tpo/anti-censorship/pluggable-transports/snowflake#40187, this is safe to do
and more efficient when the caller does not reuse the passed-in buffer.
In kcp-go, `WriteTo` is called [inside `UDPSession.defaultTx`](https://github....Like with tpo/anti-censorship/pluggable-transports/snowflake#40187, this is safe to do
and more efficient when the caller does not reuse the passed-in buffer.
In kcp-go, `WriteTo` is called [inside `UDPSession.defaultTx`](https://github.com/xtaci/kcp-go/blob/v5.6.1/tx.go#L14)
via [`UDPSession.tx`](https://github.com/xtaci/kcp-go/blob/v5.6.1/tx_linux.go#L17)
from [`UDPSession.uncork`](https://github.com/xtaci/kcp-go/blob/v5.6.1/sess.go#L332),
which immediately discards the buffers it has just written (`s.txqueue = s.txqueue[:0]`).
`UDPSession.defaultTx` is the largest single allocator of temporary memory in
the memory profile of https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40086#note_2838372.David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40198Out of localhost ephemeral ports ("cannot assign requested address") on link ...2023-06-08T19:26:08ZDavid Fifielddcf@torproject.orgOut of localhost ephemeral ports ("cannot assign requested address") on link between snowflake-server and haproxySince 2022-09-28 10:57:26 (53 hours ago) the snowflake-server log is full of:
```
# grep 'cannot assign requested address$' /var/log/snowflake-server/snowflake-server.log | head
2022/09/28 10:57:26 handleConn: failed to connect to ORPor...Since 2022-09-28 10:57:26 (53 hours ago) the snowflake-server log is full of:
```
# grep 'cannot assign requested address$' /var/log/snowflake-server/snowflake-server.log | head
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
# grep 'cannot assign requested address$' /var/log/snowflake-server/snowflake-server.log | tail
2022/09/30 15:44:17 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/30 15:44:18 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/30 15:44:18 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/30 15:44:18 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/30 15:44:18 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
```
The error message means that the kernel could not allocate an ephemeral port number
for a localhost TCP connection.
In this case it for the connection between snowflake-server and haproxy.
My analysis at https://lists.torproject.org/pipermail/anti-censorship-team/2022-September/000265.html was incomplete:
the total number of localhost sockets does not matter for the purpose of counting 4-tuples,
but it *does* matter for the purpose of allocating ephemeral ports.
There are currently only 21712 sockets open between snowflake-server and haproxy,
which means the remainder of the ephemeral port range is taken up by other
localhost communication.
My immediate plan for this is to move haproxy to 127.0.0.2:10000 rather than 127.0.0.1:10000,
and maybe similarly with the individual tor instances if necessary.
/cc @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.org2022-09-30https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40197nil pointer dereference in PeerConnection.PendingLocalDescription2022-10-03T11:53:14Zcypherpunksnil pointer dereference in PeerConnection.PendingLocalDescription```
info] {EDGE} connection_edge_process_inbuf(): data from edge while in 'waiting for circuit' state. Leaving it on buffer.
info] {EDGE} connection_edge_process_inbuf(): data from edge while in 'waiting for circuit' state. Leaving it on...```
info] {EDGE} connection_edge_process_inbuf(): data from edge while in 'waiting for circuit' state. Leaving it on buffer.
info] {EDGE} connection_edge_process_inbuf(): data from edge while in 'waiting for circuit' state. Leaving it on buffer.
info] {NET} parse_socks_client(): SOCKS 5 client: continuing without authentication
info] {NET} connection_read_proxy_handshake(): Proxy Client: OR connection (handshaking (proxy)) with 192.0.2.3:80 ID=1zOHpg+FxqQfi/6jDLtCpHHqBTH8gjYmCKXkus1D5Ko RSA_ID=2B280B23E1107BB62ABFC40DDCC8824814F80A72 successful
info] {BTRACK} bto_update_best(): ORCONN BEST_ANY state 2->3 gid=4
notice] {CONTROL} Bootstrapped 10% (conn_done): Connected to a relay
info] {BTRACK} bto_update_best(): ORCONN BEST_AP state 2->3 gid=4
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: panic: runtime error: invalid memory address or nil pointer dereference
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: [signal 0xc0000005 code=0x0 addr=0x0 pc=0x5ed17d]
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error:
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: goroutine 53 [running]:
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: github.com/pion/webrtc.(*PeerConnection).PendingLocalDescription(0x0)
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: /var/tmp/dist/gopath/src/github.com/pion/webrtc/peerconnection.go:2026 +0x1d
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: github.com/pion/webrtc.(*PeerConnection).LocalDescription(0xc00034c000)
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: /var/tmp/dist/gopath/src/github.com/pion/webrtc/peerconnection.go:1007 +0x1e
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib.(*WebRTCPeer).connect(0xc00034c000, 0x0, 0xc000345d48)
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: /var/tmp/dist/gopath/src/git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib/webrtc.go:150 +0xd8
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib.NewWebRTCPeerWithEvents(0x35ee40, 0xc0000d6000, {0x223f8f30008, 0xc00022e220})
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: /var/tmp/dist/gopath/src/git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib/webrtc.go:73 +0x38b
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib.WebRTCDialer.Catch(...)
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: /var/tmp/dist/gopath/src/git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib/rendezvous.go:172
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib.(*Peers).Collect(0xc000234080)
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: /var/tmp/dist/gopath/src/git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib/peers.go:69 +0x223
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib.connectLoop({0x846bd0, 0xc000234080})
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: /var/tmp/dist/gopath/src/git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib/snowflake.go:345 +0x56
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: created by git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib.(*Transport).Dial
info] {PT} managed_proxy_stderr_callback(): Managed proxy at 'PluggableTransports\snowflake-client.exe' reported via standard error: /var/tmp/dist/gopath/src/git.torproject.org/pluggable-transports/snowflake.git/v2/client/lib/snowflake.go:170 +0x206
info] {NET} TLS error: <syscall error while handshaking> (errno=10054: Connection reset by peer [WSAECONNRESET ]; state=SSLv3/TLS write client hello)
info] {OR} connection_tls_continue_handshake(): tls error [connection reset]. breaking connection.
info] {CIRC} circuit_n_chan_done(): Channel failed; closing circ.
info] {GENERAL} circuit_mark_for_close_(): Circuit 0 (id: 1) marked for close at circuitbuild.c:687 (orig reason: 8, new reason: 0)
info] {HANDSHAKE} connection_or_note_state_when_broken(): Connection died in state 'handshaking (TLS) with SSL state SSLv3/TLS write client hello in HANDSHAKE'
info] {BTRACK} bto_status_rcvr(): ORCONN DELETE gid=4 status=2 reason=4
warn] {CONTROL} Problem bootstrapping. Stuck at 10% (conn_done): Connected to a relay. (CONNECTRESET; CONNECTRESET; count 1; recommendation warn; host 2B280B23E1107BB62ABFC40DDCC8824814F80A72 at 192.0.2.3:80)
warn] {HANDSHAKE} 1 connections have failed:
warn] {HANDSHAKE} 1 connections died in state handshaking (TLS) with SSL state SSLv3/TLS write client hello in HANDSHAKE
info] {OR} circuit_build_failed(): Our circuit 0 (id: 1) died before the first hop with no connection
info] {GUARD} entry_guards_note_guard_failure(): Recorded failure for primary guard $2B280B23E1107BB62ABFC40DDCC8824814F80A72 ($2B280B23E1107BB62ABFC40DDCC8824814F80A72)
info] {CIRC} circuit_free_(): Circuit 0 (id: 1) has been freed.
warn] {PT} Pluggable Transport process terminated with status code 2
```https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40196snowflake plugin not working2024-02-14T16:43:33Zcypherpunkssnowflake plugin not workingsnowflake plugin on firefox is down. saying canßt connect to bridge.
i am in Germany.
_edited to have a clear title_snowflake plugin on firefox is down. saying canßt connect to bridge.
i am in Germany.
_edited to have a clear title_https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40195OOM killer invoked at broker2022-10-12T19:26:27ZCecylia BocovichOOM killer invoked at brokerThere have been outages the last two nights at the snowflake broker, I was puzzled at first because there's no errors in the broker logs. I just checked the syslog and it looks like the OOM killer has been invoked:
```
Sep 30 09:36:17 s...There have been outages the last two nights at the snowflake broker, I was puzzled at first because there's no errors in the broker logs. I just checked the syslog and it looks like the OOM killer has been invoked:
```
Sep 30 09:36:17 snowflake-broker kernel: [17516237.845995] probetest invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
```
This one is for the probetest, but it's likely the broker outages was due to this as well.Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40194Perform more regular dependency updates2023-07-04T14:27:43ZCecylia BocovichPerform more regular dependency updatesDependency updates happen in an adhoc way when we think of it or if there's a CVE we know about. I'd like to look into doing this more regularly, perhaps with the help of bots. I'd also like to revisit the pion dependency update process ...Dependency updates happen in an adhoc way when we think of it or if there's a CVE we know about. I'd like to look into doing this more regularly, perhaps with the help of bots. I'd also like to revisit the pion dependency update process for browser builds because the script at https://gitlab.torproject.org/-/snippets/145 needs hacks every time and the process still takes ages.meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40193Snowflake Broker Deployment 22-10-032022-10-25T16:32:20ZshelikhooSnowflake Broker Deployment 22-10-03We are going to deploy a new version of snowflake broker configuration to snowflake broker.
The broker binary isn't updated, and remain [v2.3.1](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/tags/v2.3...We are going to deploy a new version of snowflake broker configuration to snowflake broker.
The broker binary isn't updated, and remain [v2.3.1](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/tags/v2.3.1).
## Deployment Script
```
sv stop snowflake-broker
cp /etc/service/snowflake-broker/run ./snowflake-broker-run-22-10-03-backup-$(date +%N)
install --owner root ./snowflake-broker-run-22-10-03-candidcate /etc/service/snowflake-broker/run
sv start snowflake-broker
```
## Rollback Script(will be located at /home/shelikhoo/deployment-22-10-03)
```
sv stop snowflake-broker
install --owner root ./snowflake-broker-run-22-10-03-backup-???? /etc/service/snowflake-broker/run
sv start snowflake-broker
```
## New Run File
(the difference is at --allowed-relay-pattern)
(-ip-count-mask's value is not real value used on the production system)
```
#!/bin/sh -e
setcap 'cap_net_bind_service=+ep' /usr/local/bin/broker
export GOMAXPROCS=1
exec chpst -u snowflake-broker -o 32768 /usr/local/bin/broker --metrics-log /home/snowflake-broker/metrics.log --acme-hostnames snowflake-broker.bamsoftware.com,snowflake-broker.freehaven.net,snowflake-broker.torproject.net --acme-email dcf@torproject.org --acme-cert-cache /home/snowflake-broker/acme-cert-cache --bridge-list-path /home/snowflake-broker/bridge_lists.json --default-relay-pattern ^snowflake.torproject.net$ --allowed-relay-pattern snowflake.torproject.net$ -ip-count-log /home/snowflake-broker/metrics-ip-salted.jsonl -ip-count-interval 1h -ip-count-mask ****** 2>&1
```
## Old Run File
```
#!/bin/sh -e
setcap 'cap_net_bind_service=+ep' /usr/local/bin/broker
export GOMAXPROCS=1
exec chpst -u snowflake-broker -o 32768 /usr/local/bin/broker --metrics-log /home/snowflake-broker/metrics.log --acme-hostnames snowflake-broker.bamsoftware.com,snowflake-broker.freehaven.net,snowflake-broker.torproject.net --acme-email dcf@torproject.org --acme-cert-cache /home/snowflake-broker/acme-cert-cache --bridge-list-path /home/snowflake-broker/bridge_lists.json --default-relay-pattern ^snowflake.torproject.net$ --allowed-relay-pattern ^snowflake.torproject.net$ -ip-count-log /home/snowflake-broker/metrics-ip-salted.jsonl -ip-count-interval 1h -ip-count-mask ****** 2>&1
```
## Side effect to be watched
The network capacity of the snowflake may be decreased. However, if we can take this hit, we should be able to roll out distributed snowflake support.Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetshelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40192Experiment with bypassing extor-static-cookie on snowflake-012022-12-12T01:03:33ZDavid Fifielddcf@torproject.orgExperiment with bypassing extor-static-cookie on snowflake-01The [12](tpo/anti-censorship/pluggable-transports/snowflake#40176) extor-static-cookie processes
collectively use about 300% of a CPU core (25% each).
We can open up some CPU headroom by cutting them out of the pipeline—but
then we need ...The [12](tpo/anti-censorship/pluggable-transports/snowflake#40176) extor-static-cookie processes
collectively use about 300% of a CPU core (25% each).
We can open up some CPU headroom by cutting them out of the pipeline—but
then we need another way doing static ExtORPort authentication.
https://lists.torproject.org/pipermail/anti-censorship-team/2022-September/000253.html
> For CPU pressure, I don't see any quick fixes. In an emergency, we could
> hack the tor binary to use a static ExtORPort authentication cookie, and
> remove the extor-static-cookie shim from the pipeline.
There's also the idea that the extra localhost communication required by
extor-static-cookie is a cause of the current performance bottleneck.
https://lists.torproject.org/pipermail/anti-censorship-team/2022-September/000263.html
> First, let's patch tor to get rid of the extor processes, as suggested
> by David earlier when discussing RAM pressure. This should bring down
> context switches.
Cf. [Two features that would help load-balanced bridges](https://lists.torproject.org/pipermail/tor-dev/2022-February/014695.html).
As a preliminary test to see if removing extor-static-cookie actually has an effect,
this issue is to try bypassing extor-static-cookie and having haproxy connect directly
to the "regular" ORPort of the tor processes.
The downside of this is that we will not be counting transport- and country-specific
metrics while the experiment is in place.
But it should take only a few hours maximum to see if it has an effect.
/cc @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40191snowflake-01: Performance experiments and enhancements2022-11-05T20:26:17ZLinus Nordberglinus@torproject.orgsnowflake-01: Performance experiments and enhancementsCollecting information and actions related to performance on snowflake-01 here.
General situation 2022-09-29 is that we won't push more than 3Gbps with a flatline on both bw and pps (a bit over 400kpps) and we're uncertain where the bot...Collecting information and actions related to performance on snowflake-01 here.
General situation 2022-09-29 is that we won't push more than 3Gbps with a flatline on both bw and pps (a bit over 400kpps) and we're uncertain where the bottleneck is at.Linus Nordberglinus@torproject.orgLinus Nordberglinus@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40189snowflake-01: Disable conntrack2022-12-13T20:25:17ZLinus Nordberglinus@torproject.orgsnowflake-01: Disable conntrackBased on output from perf(1) we should disable conntrack. We would have to deal with that regardless, since we're running out of slots in the conntrack table.Based on output from perf(1) we should disable conntrack. We would have to deal with that regardless, since we're running out of slots in the conntrack table.Linus Nordberglinus@torproject.orgLinus Nordberglinus@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40188snowflake-01: Reboot2022-10-02T07:51:26ZLinus Nordberglinus@torproject.orgsnowflake-01: RebootIn order to restart dbus, a reboot might be the best option.
```
Service restarts being deferred:
/etc/needrestart/restart.d/dbus.service
```
LMK if you think there are better ways. Note that we lack all OOB on this server (like IPMI,...In order to restart dbus, a reboot might be the best option.
```
Service restarts being deferred:
/etc/needrestart/restart.d/dbus.service
```
LMK if you think there are better ways. Note that we lack all OOB on this server (like IPMI, ILO access), so we really don't want to mess up.
I'm planning on doing the reboot in connection to #40186.Linus Nordberglinus@torproject.orgLinus Nordberglinus@torproject.org