Anti-censorship issueshttps://gitlab.torproject.org/groups/tpo/anti-censorship/-/issues2022-10-07T08:43:12Zhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake-webext/-/issues/60Personal Information Released on Web2022-10-07T08:43:12ZcypherpunksPersonal Information Released on WebNot sure if it's a coincidence, but my home address, name, phone, and email were all released yesterday, the same day I installed the extension. Several websites said the info was updated yesterday, specifically. I will continue using th...Not sure if it's a coincidence, but my home address, name, phone, and email were all released yesterday, the same day I installed the extension. Several websites said the info was updated yesterday, specifically. I will continue using the extension for now, as it's important and an easy way to help, but I am concerned about data being shared, even if it is encrypted.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40201Out of ephemeral ports on link between haproxy and extor-static-cookie2023-05-22T16:22:31ZDavid Fifielddcf@torproject.orgOut of ephemeral ports on link between haproxy and extor-static-cookieThe multi-KCP optimization parallelization of #40200
has enabled a large increase in bandwidth, currently above 3.5 Gbps outgoing / 2.5 Gbps incoming.
But now we are experiencing another ephemeral port exhaustion issue like #40198.
This ...The multi-KCP optimization parallelization of #40200
has enabled a large increase in bandwidth, currently above 3.5 Gbps outgoing / 2.5 Gbps incoming.
But now we are experiencing another ephemeral port exhaustion issue like #40198.
This time it's on the link between haproxy and extor-static-cookie.
The snowflake-server log says:
```
2022/10/02 05:06:47 handleConn: failed to connect to ORPort: EOF
```
The haproxy log says:
```
Oct 2 05:06:53 snowflake-01 haproxy[9134]: Connect() failed for backend tor-instances: no free ports.
```
For context, there are about 38000 live connections between snowflake-server and haproxy.
```
# ss -f inet -n state established | grep '127.0.0.1:10000\s*$' | wc -l
37918
```
/cc @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40200Use multiple parallel KCP state machines2023-07-18T00:47:29ZDavid Fifielddcf@torproject.orgUse multiple parallel KCP state machinesI have a suspicion that KCP packet scheduling may be a bottleneck in the server.
Most of the server's processing is concurrent and scales across multiple CPU cores,
but all the traffic passes through the centralized `kcp.Listener` schedu...I have a suspicion that KCP packet scheduling may be a bottleneck in the server.
Most of the server's processing is concurrent and scales across multiple CPU cores,
but all the traffic passes through the centralized `kcp.Listener` scheduler
that is created [in `Transport.Listen`](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/v2.3.1/server/lib/snowflake.go#L145).
We can actually use multiple independent KCP schedulers,
as long as we consistently assign the packets of one particular session
to the same KCP.
And we can do that using the ClientID associated with each session.
I have a branch for this at
https://gitlab.torproject.org/dcf/snowflake/-/commits/multi-kcp.
I'm planning to test it as part of !100.David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40199Reduce allocation in `QueuePacketConn.WriteTo`2023-07-18T00:47:30ZDavid Fifielddcf@torproject.orgReduce allocation in `QueuePacketConn.WriteTo`Like with tpo/anti-censorship/pluggable-transports/snowflake#40187, this is safe to do
and more efficient when the caller does not reuse the passed-in buffer.
In kcp-go, `WriteTo` is called [inside `UDPSession.defaultTx`](https://github....Like with tpo/anti-censorship/pluggable-transports/snowflake#40187, this is safe to do
and more efficient when the caller does not reuse the passed-in buffer.
In kcp-go, `WriteTo` is called [inside `UDPSession.defaultTx`](https://github.com/xtaci/kcp-go/blob/v5.6.1/tx.go#L14)
via [`UDPSession.tx`](https://github.com/xtaci/kcp-go/blob/v5.6.1/tx_linux.go#L17)
from [`UDPSession.uncork`](https://github.com/xtaci/kcp-go/blob/v5.6.1/sess.go#L332),
which immediately discards the buffers it has just written (`s.txqueue = s.txqueue[:0]`).
`UDPSession.defaultTx` is the largest single allocator of temporary memory in
the memory profile of https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40086#note_2838372.David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/gettor-project/OnionSproutsBot/-/issues/39abstract dialogue options2023-04-22T22:39:21Zn0tooseabstract dialogue optionsAs I was working on #15, I realized that I will most likely have to create two separate functions -- one for the desktop, and one for the Android versions of TBB. The functionality will be very similar (all database-related operations an...As I was working on #15, I realized that I will most likely have to create two separate functions -- one for the desktop, and one for the Android versions of TBB. The functionality will be very similar (all database-related operations and dialogue options), apart from details (such as the way the correct URL containing the correct file is determined).
As the same dialogue options may be used repeatedly across different functions, it may be a good idea to 'abstract' them (aka. put them in a different file) to make maintenance easier and improve readability.n0toosen0toosehttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake-webext/-/issues/59Snowflake is off - Could not connect to the bridge.2023-03-08T12:05:04ZcypherpunksSnowflake is off - Could not connect to the bridge.I am getting the "Snowflake is off - Could not connect to the bridge" message when I use the webextension in Opera, Chrome and the one that I have embedded on my website. I have tried to bin my extension and then download and activate it...I am getting the "Snowflake is off - Could not connect to the bridge" message when I use the webextension in Opera, Chrome and the one that I have embedded on my website. I have tried to bin my extension and then download and activate it again and I still get the same message. It has worked fine for four days before crashing.
What to do and try now?
Best
Ehttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40196snowflake plugin not working2024-02-14T16:43:33Zcypherpunkssnowflake plugin not workingsnowflake plugin on firefox is down. saying canßt connect to bridge.
i am in Germany.
_edited to have a clear title_snowflake plugin on firefox is down. saying canßt connect to bridge.
i am in Germany.
_edited to have a clear title_https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40195OOM killer invoked at broker2022-10-12T19:26:27ZCecylia BocovichOOM killer invoked at brokerThere have been outages the last two nights at the snowflake broker, I was puzzled at first because there's no errors in the broker logs. I just checked the syslog and it looks like the OOM killer has been invoked:
```
Sep 30 09:36:17 s...There have been outages the last two nights at the snowflake broker, I was puzzled at first because there's no errors in the broker logs. I just checked the syslog and it looks like the OOM killer has been invoked:
```
Sep 30 09:36:17 snowflake-broker kernel: [17516237.845995] probetest invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
```
This one is for the probetest, but it's likely the broker outages was due to this as well.Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40194Perform more regular dependency updates2023-07-04T14:27:43ZCecylia BocovichPerform more regular dependency updatesDependency updates happen in an adhoc way when we think of it or if there's a CVE we know about. I'd like to look into doing this more regularly, perhaps with the help of bots. I'd also like to revisit the pion dependency update process ...Dependency updates happen in an adhoc way when we think of it or if there's a CVE we know about. I'd like to look into doing this more regularly, perhaps with the help of bots. I'd also like to revisit the pion dependency update process for browser builds because the script at https://gitlab.torproject.org/-/snippets/145 needs hacks every time and the process still takes ages.meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40193Snowflake Broker Deployment 22-10-032022-10-25T16:32:20ZshelikhooSnowflake Broker Deployment 22-10-03We are going to deploy a new version of snowflake broker configuration to snowflake broker.
The broker binary isn't updated, and remain [v2.3.1](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/tags/v2.3...We are going to deploy a new version of snowflake broker configuration to snowflake broker.
The broker binary isn't updated, and remain [v2.3.1](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/tags/v2.3.1).
## Deployment Script
```
sv stop snowflake-broker
cp /etc/service/snowflake-broker/run ./snowflake-broker-run-22-10-03-backup-$(date +%N)
install --owner root ./snowflake-broker-run-22-10-03-candidcate /etc/service/snowflake-broker/run
sv start snowflake-broker
```
## Rollback Script(will be located at /home/shelikhoo/deployment-22-10-03)
```
sv stop snowflake-broker
install --owner root ./snowflake-broker-run-22-10-03-backup-???? /etc/service/snowflake-broker/run
sv start snowflake-broker
```
## New Run File
(the difference is at --allowed-relay-pattern)
(-ip-count-mask's value is not real value used on the production system)
```
#!/bin/sh -e
setcap 'cap_net_bind_service=+ep' /usr/local/bin/broker
export GOMAXPROCS=1
exec chpst -u snowflake-broker -o 32768 /usr/local/bin/broker --metrics-log /home/snowflake-broker/metrics.log --acme-hostnames snowflake-broker.bamsoftware.com,snowflake-broker.freehaven.net,snowflake-broker.torproject.net --acme-email dcf@torproject.org --acme-cert-cache /home/snowflake-broker/acme-cert-cache --bridge-list-path /home/snowflake-broker/bridge_lists.json --default-relay-pattern ^snowflake.torproject.net$ --allowed-relay-pattern snowflake.torproject.net$ -ip-count-log /home/snowflake-broker/metrics-ip-salted.jsonl -ip-count-interval 1h -ip-count-mask ****** 2>&1
```
## Old Run File
```
#!/bin/sh -e
setcap 'cap_net_bind_service=+ep' /usr/local/bin/broker
export GOMAXPROCS=1
exec chpst -u snowflake-broker -o 32768 /usr/local/bin/broker --metrics-log /home/snowflake-broker/metrics.log --acme-hostnames snowflake-broker.bamsoftware.com,snowflake-broker.freehaven.net,snowflake-broker.torproject.net --acme-email dcf@torproject.org --acme-cert-cache /home/snowflake-broker/acme-cert-cache --bridge-list-path /home/snowflake-broker/bridge_lists.json --default-relay-pattern ^snowflake.torproject.net$ --allowed-relay-pattern ^snowflake.torproject.net$ -ip-count-log /home/snowflake-broker/metrics-ip-salted.jsonl -ip-count-interval 1h -ip-count-mask ****** 2>&1
```
## Side effect to be watched
The network capacity of the snowflake may be decreased. However, if we can take this hit, we should be able to roll out distributed snowflake support.Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetshelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40192Experiment with bypassing extor-static-cookie on snowflake-012022-12-12T01:03:33ZDavid Fifielddcf@torproject.orgExperiment with bypassing extor-static-cookie on snowflake-01The [12](tpo/anti-censorship/pluggable-transports/snowflake#40176) extor-static-cookie processes
collectively use about 300% of a CPU core (25% each).
We can open up some CPU headroom by cutting them out of the pipeline—but
then we need ...The [12](tpo/anti-censorship/pluggable-transports/snowflake#40176) extor-static-cookie processes
collectively use about 300% of a CPU core (25% each).
We can open up some CPU headroom by cutting them out of the pipeline—but
then we need another way doing static ExtORPort authentication.
https://lists.torproject.org/pipermail/anti-censorship-team/2022-September/000253.html
> For CPU pressure, I don't see any quick fixes. In an emergency, we could
> hack the tor binary to use a static ExtORPort authentication cookie, and
> remove the extor-static-cookie shim from the pipeline.
There's also the idea that the extra localhost communication required by
extor-static-cookie is a cause of the current performance bottleneck.
https://lists.torproject.org/pipermail/anti-censorship-team/2022-September/000263.html
> First, let's patch tor to get rid of the extor processes, as suggested
> by David earlier when discussing RAM pressure. This should bring down
> context switches.
Cf. [Two features that would help load-balanced bridges](https://lists.torproject.org/pipermail/tor-dev/2022-February/014695.html).
As a preliminary test to see if removing extor-static-cookie actually has an effect,
this issue is to try bypassing extor-static-cookie and having haproxy connect directly
to the "regular" ORPort of the tor processes.
The downside of this is that we will not be counting transport- and country-specific
metrics while the experiment is in place.
But it should take only a few hours maximum to see if it has an effect.
/cc @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40191snowflake-01: Performance experiments and enhancements2022-11-05T20:26:17ZLinus Nordberglinus@torproject.orgsnowflake-01: Performance experiments and enhancementsCollecting information and actions related to performance on snowflake-01 here.
General situation 2022-09-29 is that we won't push more than 3Gbps with a flatline on both bw and pps (a bit over 400kpps) and we're uncertain where the bot...Collecting information and actions related to performance on snowflake-01 here.
General situation 2022-09-29 is that we won't push more than 3Gbps with a flatline on both bw and pps (a bit over 400kpps) and we're uncertain where the bottleneck is at.Linus Nordberglinus@torproject.orgLinus Nordberglinus@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake-webext/-/issues/58Increase capacity and have unrestricted proxies poll more frequently2023-01-02T16:53:54ZCecylia BocovichIncrease capacity and have unrestricted proxies poll more frequentlyAs suggested on the mailing list: https://lists.torproject.org/pipermail/anti-censorship-team/2022-September/000257.htmlAs suggested on the mailing list: https://lists.torproject.org/pipermail/anti-censorship-team/2022-September/000257.htmlCecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40189snowflake-01: Disable conntrack2022-12-13T20:25:17ZLinus Nordberglinus@torproject.orgsnowflake-01: Disable conntrackBased on output from perf(1) we should disable conntrack. We would have to deal with that regardless, since we're running out of slots in the conntrack table.Based on output from perf(1) we should disable conntrack. We would have to deal with that regardless, since we're running out of slots in the conntrack table.Linus Nordberglinus@torproject.orgLinus Nordberglinus@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40188snowflake-01: Reboot2022-10-02T07:51:26ZLinus Nordberglinus@torproject.orgsnowflake-01: RebootIn order to restart dbus, a reboot might be the best option.
```
Service restarts being deferred:
/etc/needrestart/restart.d/dbus.service
```
LMK if you think there are better ways. Note that we lack all OOB on this server (like IPMI,...In order to restart dbus, a reboot might be the best option.
```
Service restarts being deferred:
/etc/needrestart/restart.d/dbus.service
```
LMK if you think there are better ways. Note that we lack all OOB on this server (like IPMI, ILO access), so we really don't want to mess up.
I'm planning on doing the reboot in connection to #40186.Linus Nordberglinus@torproject.orgLinus Nordberglinus@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40187Reduce allocation in `QueuePacketConn.QueueIncoming`.2023-03-13T19:04:22ZDavid Fifielddcf@torproject.orgReduce allocation in `QueuePacketConn.QueueIncoming`.`QueueIncoming` [makes a copy](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/v2.3.1/common/turbotunnel/queuepacketconn.go#L54) of the passed in slice
before queuing it.
The idea behind this desig...`QueueIncoming` [makes a copy](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/v2.3.1/common/turbotunnel/queuepacketconn.go#L54) of the passed in slice
before queuing it.
The idea behind this design is to make the function harder to misuse:
the caller cannot modify the queued packet after calling `QueueIncoming`.
But it means an extra malloc and memmove per queued packet.
As long as the caller does not reuse the slice,
it is okay not to make a copy and it's more efficient.
These allocations are suspected to be a cause of frequent
garbage collections in
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40086#note_2838372.https://gitlab.torproject.org/tpo/anti-censorship/gettor-project/OnionSproutsBot/-/issues/38Offer Android builds2022-09-28T16:43:50ZGeopJrOffer Android builds`downloads.json` doesn't include Android links yet (https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/40110) but it might be useful to just manually add them for now.
The format is: `https://dist.torproject.org/t...`downloads.json` doesn't include Android links yet (https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/40110) but it might be useful to just manually add them for now.
The format is: `https://dist.torproject.org/torbrowser/{VERSION}/tor-browser-{VERSION}-android-{ARCH}-multi.apk`(`.asc`)
You can use the version from the `downloads.json`.
The available archs are: `["aarch64", "armv7", "x86_64", "x86"]`
edit:
Actually, getting the version [this way](https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/42#note_2822272) seems much safer.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40186snowflake-01: Connect the other NIC and move wireguard and sshd to it2022-09-30T16:00:58ZLinus Nordberglinus@torproject.orgsnowflake-01: Connect the other NIC and move wireguard and sshd to itThis will allow for
- skipping conntracking on the 10G NIC
- ethtool tuning of the 10G NIC without risking losing the hostThis will allow for
- skipping conntracking on the 10G NIC
- ethtool tuning of the 10G NIC without risking losing the hostLinus Nordberglinus@torproject.orgLinus Nordberglinus@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40185overflow in bandwidth reporting2022-11-21T15:34:42Ztrinity-1686aoverflow in bandwidth reportingA user reported on `#tor` they see strange bandwidth report on their snowflake proxy.
![image](/uploads/2840f9598f1d194a89058c04a84023e4/image.png)
It looks very much like an overflowed signed 32b integer. They use snowflake on raspber...A user reported on `#tor` they see strange bandwidth report on their snowflake proxy.
![image](/uploads/2840f9598f1d194a89058c04a84023e4/image.png)
It looks very much like an overflowed signed 32b integer. They use snowflake on raspberry pi 3 (64 bit), however I've heard more than one time of things going 32b on raspberries, so may be reproducible only in 32b modehttps://gitlab.torproject.org/tpo/anti-censorship/gettor-project/OnionSproutsBot/-/issues/37put user's preferred language on top of list of languages2022-12-19T16:14:48Zn0tooseput user's preferred language on top of list of languagesFor example, in this case, we should probably use English as the first option here.
![image](/uploads/6b5050a4bc17ddf656e79fe1d7d741e9/image.png)For example, in this case, we should probably use English as the first option here.
![image](/uploads/6b5050a4bc17ddf656e79fe1d7d741e9/image.png)n0toosen0toose