Snowflake issueshttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues2023-08-01T19:29:39Zhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/23888Creating a Snowflake WebExtension addon2023-08-01T19:29:39ZTracCreating a Snowflake WebExtension addonThe idea is to create a WebExtension that allows one to become a Snowflake bridge by just installing it. That way it only suffices to install an extension and forget about it, unlike the approach of keeping a tab always open with the sno...The idea is to create a WebExtension that allows one to become a Snowflake bridge by just installing it. That way it only suffices to install an extension and forget about it, unlike the approach of keeping a tab always open with the snowflake JS code.
Since it's based on WebExtensions it can be easily deployed for other browsers in their addon store.
I did try to make one myself but I don't have the expertise and time to debug all the problems that resulted. One of the important take aways that I learned in that process was that automatically loading scripts from external sites is prohibited and will result in the addon not passing the review in the addon store, so the `snowflake.js` and `modernizr.js` should be embedded with the addon. However, this would require modifying `snowflake.js` since when it's loaded locally it throws some typeError and doesn't show that there's some connection to snowflake.bamsoftware.com in the browser console. For debugging, to verify that the addon works as intended one may load it from `about:debug` and check `about:networking` in the DNS and WebSockets part.
For the implementation these resources should be loaded in the background to ensure a permanent state with this in the `manifest.json`,
```
"background": {
"page": "pages/Snowflake.html"
},
```
**Trac**:
**Username**: oarelSponsor 28: Reliable Anonymous Communication Evading Censors and Repressors (RACECAR)Arlo BreaultArlo Breaulthttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/20813Start producing snowflakes2023-08-01T19:29:39ZArlo BreaultStart producing snowflakesOnce `snowflake-client` gets in the alpha Tor Browser builds (tpo/applications/tor-browser#20735), we're going to have some unhappy users if we don't have a sufficient number of proxies available.
We should start ramping up production a...Once `snowflake-client` gets in the alpha Tor Browser builds (tpo/applications/tor-browser#20735), we're going to have some unhappy users if we don't have a sufficient number of proxies available.
We should start ramping up production asap.
Some ideas in,<br>
https://github.com/glamrock/cupcake<br>
https://github.com/keroserene/snowflake/issues/30
We probably also want to close out the opt-in issue,<br>
https://github.com/keroserene/snowflake/issues/21Sponsor 28: Reliable Anonymous Communication Evading Censors and Repressors (RACECAR)https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40030Increase number of proxies available for restricted clients2023-01-06T12:58:15ZCecylia BocovichIncrease number of proxies available for restricted clientsIt looks like our solution to #33666 has solved the connectivity issues for clients behind restricted NATs, but we still have a relatively small number of unrestricted proxies available for restricted clients.
See the counts of proxy by...It looks like our solution to #33666 has solved the connectivity issues for clients behind restricted NATs, but we still have a relatively small number of unrestricted proxies available for restricted clients.
See the counts of proxy by type:
![proxy_types](/uploads/0edc55334e2fdeef8e295a18e7850377/proxy_types.png)
And a close-up on the counts of unrestricted proxies:
![unrestricted_proxies](/uploads/1be8466c9dd85144196606fc2c22b47a/unrestricted_proxies.png)
So, we're making progress! But, I'm still seeing a lot of restricted clients getting turned away at the broker because there were no available proxies. Here's a plot of the times a restricted client failed to be matched with a snowflake:
![unmatches](/uploads/cefc6c1067d3d78a62064ab59da4ea41/unmatches.png)
For reference, November (when we start seeing the spikes) was around the time we rolled out our remote probe test and started correctly classifying browser-based proxies. This both increased the amount of unrestricted proxies we had, but also decreased the number of unknown proxies we had. The end result for restricted clients (since they are handed either unknown or unrestricted proxies), is that while each proxy they get is more likely to work for them, the pool of proxies they can pull from is greatly reduced.
The spikes themselves are interesting. Some days we don't get any denied counts and some days we get way more than I'd expect. We almost surely need more unrestricted proxies, but it would also be useful to get some more metrics on what proportion of successful matches are for unrestricted vs restricted clients, and how many times a restricted client has to poll to get a working proxy.
So this ticket is both for measurements, but I also think we should try to run some reliable proxy-go instances that have unrestricted NATs.Snowflake in Tor Browser 10.5Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40023Snowflake client's global session manager is tied to melted SnowflakeCollector2020-12-04T16:24:11ZCecylia BocovichSnowflake client's global session manager is tied to melted SnowflakeCollectorThere was a bug introduced by our solution to #21314. We moved the broker poll loop inside the SOCKS handler, which means that each SOCKS connection effectively has their own pool of snowflakes (i.e., their own `SnowflakeCollector`). How...There was a bug introduced by our solution to #21314. We moved the broker poll loop inside the SOCKS handler, which means that each SOCKS connection effectively has their own pool of snowflakes (i.e., their own `SnowflakeCollector`). However, the session manager was kept global (shared by all connections). The session manager defines the dialContext for `RedialPacketConn` and is tied to the `SnowflakeCollector` it was created with.
The problem with this showed up most clearly in #40018, where mobile applications using Snowflake as a library are stopping and restarting Snowflake, but will happen in any scenario where the first `SnowflakeCollector` is melted, but a new SOCKS connection is created.Snowflake in Tor Browser 10.5Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40007Get more alpha users of Snowflake to stress-test before stable2021-05-27T18:18:26ZCecylia BocovichGet more alpha users of Snowflake to stress-test before stableIn a discussion before the stable release of Tor Browser 9.5, we discussed that we'd like to get more snowflake users before moving it out of alpha so that we can stress-test the system to make sure it will work with the massive increase...In a discussion before the stable release of Tor Browser 9.5, we discussed that we'd like to get more snowflake users before moving it out of alpha so that we can stress-test the system to make sure it will work with the massive increase of users we get in stable.
This ticket is to track that progress and perform measurements. We have https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/40027 open as a way to get more alpha testers on mobile.Snowflake in Tor Browser 10.5Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/33666Investigate Snowflake proxy failures2022-06-01T19:51:48ZCecylia BocovichInvestigate Snowflake proxy failuresSometimes a client will get a useless proxy from the broker. At times this happens occasionally, and at times more often. It could be a NAT problem.Sometimes a client will get a useless proxy from the broker. At times this happens occasionally, and at times more often. It could be a NAT problem.Snowflake in Tor Browser 10.5Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/21314snowflake-client needs to stop using my network when I'm not giving it requests2020-11-23T18:41:46ZRoger Dingledinesnowflake-client needs to stop using my network when I'm not giving it requestsI started my Tor Browser, and told it to use snowflake, and it did. Then I changed my mind and told it to stop using snowflake. Now, apparently there's a bug in Tor where Tor is supposed to kill snowflake-client when there are no more br...I started my Tor Browser, and told it to use snowflake, and it did. Then I changed my mind and told it to stop using snowflake. Now, apparently there's a bug in Tor where Tor is supposed to kill snowflake-client when there are no more bridge lines in my torrc that want to use it. But ignoring that Tor bug, snowflake-client should also be defensive for me. Right now it is touching the broker every 10 seconds, looking for a snowflake, even though it is getting no requests. That can't be good for scalability or for the broker or for the users.Snowflake in Tor Browser 10.5Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/19001Ship Snowflake in Tor Browser2022-03-14T05:36:27ZDavid Fifielddcf@torproject.orgShip Snowflake in Tor BrowserThis is now a summary ticket for the things we need to complete to move Snowflake out of alpha and ship it with regular versions of Tor Browser
- [x] #21314
- [ ] #25723
- [x] #33666
- [ ] #40007
- [ ] https://gitlab.torproject.org/t...This is now a summary ticket for the things we need to complete to move Snowflake out of alpha and ship it with regular versions of Tor Browser
- [x] #21314
- [ ] #25723
- [x] #33666
- [ ] #40007
- [ ] https://gitlab.torproject.org/tpo/core/tor/-/issues/33669Snowflake in Tor Browser 10.5https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/30868Modify client rendezvous library to remove hard-coded responses2020-06-27T13:40:27ZCecylia BocovichModify client rendezvous library to remove hard-coded responsesClient tests rely of `client/lib/rendezvous.go` rely on specific HTTP response bodies which are prone to change and unnecessaryClient tests rely of `client/lib/rendezvous.go` rely on specific HTTP response bodies which are prone to change and unnecessaryTor: unspecifiedhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40351broker is not publishing metrics to collector2024-03-21T15:06:48Zmeskiomeskio@torproject.orgbroker is not publishing metrics to collectorYesterday (2024-03-20) the snowflake broker didn't publish any metrics for collector. I have checked /home/snowflake-broker/metrics.log in the broker server and the last published metric is:
```
snowflake-stats-end 2024-03-19 23:21:24 (8...Yesterday (2024-03-20) the snowflake broker didn't publish any metrics for collector. I have checked /home/snowflake-broker/metrics.log in the broker server and the last published metric is:
```
snowflake-stats-end 2024-03-19 23:21:24 (86400 s)
```
Looking at grafana everything seems to be working normally.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40347Deploy new SQS features and fixes2024-03-21T13:41:39ZCecylia BocovichDeploy new SQS features and fixesNow that we have country-specific metrics for client rendezvous methods (!258), and some client-side fixes (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/merge_requests/264, https://gitlab.torproject....Now that we have country-specific metrics for client rendezvous methods (!258), and some client-side fixes (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/merge_requests/264, https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/merge_requests/264), we should make sure these changes are deployed. That involves:
- [X] releasing a new Snowflake version [v2.9.2](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/releases/v2.9.2)
- [x] deploying a new version of the broker
- [x] opening a tor-browser-build MR to update the client (https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/merge_requests/936)Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40346Start disabled2024-03-18T17:30:14ZcypherpunksStart disabledRegardless of any other settings, I would suggest Snowflake never begin operating automatically upon installation, instead requiring the first use on any given device to be initiated manually.
I briefly had Snowflake installed on a pers...Regardless of any other settings, I would suggest Snowflake never begin operating automatically upon installation, instead requiring the first use on any given device to be initiated manually.
I briefly had Snowflake installed on a personal device, where it was disabled while I looked into the possibility of using a DNS sinkhole to prevent the use of my connection for undesirable purposes. I had preemptively turned services.sync.addons.ignoreUserEnabledChanges on so that, once I was comfortable, enabling Snowflake on my personal device I would not inadvertently enable it on my work computer. I unexpectedly needed to have the work machine reset and did not disable this flag, so Snowflake was installed and enabled when I synchronised my settings. I responded quickly and uninstalled the extension entirely, but it appears to have been active for long enough to have routed a connection to the website of a violent extremist group that was identified and flagged by our IT systems. This incident has caused me to seriously reconsider the risk using Snowflake creates, not just to myself but also by inadvertently enabling uses like the connection in question despite my efforts to prevent doing so, and as a result I am highly unlikely to reinstall it.
That this situation involved a mistake on my part does not justify it as a possibility. It cannot be expected that no user will ever make such a mistake - even advanced users cannot be expected to never forget things - and if such a simple and potentially-unavoidable mistake can cause automatic operation to put the user at risk like this then safeguards should be put in place both to protect them and to avoid deterring them entirely.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40342Shadow integration tests occasionally panic2024-03-07T22:51:40ZCecylia BocovichShadow integration tests occasionally panicA recent job failed: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/jobs/491691
This is likely runner-dependent, since no changes were made to the Shadow tests since it last passed:
```
$ shadow --log...A recent job failed: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/jobs/491691
This is likely runner-dependent, since no changes were made to the Shadow tests since it last passed:
```
$ shadow --log-level=debug --model-unblocked-syscall-latency=true snowflake-minimal.yaml > shadow.log
** Starting Shadow v3.0.0-557-g193924aa 2023-08-25--13:24:51 with GLib v2.66.8
thread 'shadow-worker' panicked at 'called `Result::unwrap()` on an `Err` value: ENOSYS', main/utility/childpid_watcher.rs:269:37
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'shadow-worker' panicked at 'called `Result::unwrap()` on an `Err` value: PoisonError { .. }', main/utility/childpid_watcher.rs:268:43
thread 'shadow-worker' panicked at 'called `Result::unwrap()` on an `Err` value: PoisonError { .. }thread '', shadow-workermain/utility/childpid_watcher.rs' panicked at ':assertion failed: self.shim_shmem_lock.borrow().is_none()268', :main/host/host.rs43:
971:9
fatal runtime error: thread local panicked on drop
thread 'shadow-worker' panicked at 'called `Result::unwrap()` on an `Err` value: PoisonError { .. }', main/utility/childpid_watcher.rs:268:43
thread 'shadow-worker' panicked at 'assertion failed: self.shim_shmem_lock.borrow().is_none()', main/host/host.rs:971:9/bin/bash: line 210: 30403 Aborted (core dumped) shadow --log-level=debug --model-unblocked-syscall-latency=true snowflake-minimal.yaml > shadow.log
```https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40341Encode AWS credentials for SQS rendezvous2024-03-12T11:26:12ZCecylia BocovichEncode AWS credentials for SQS rendezvousAmazon's automatic scraping of Github has found our public credentials shared on https://github.com/net4people/bbs/issues/335 which leads to their support team requiring us to rotate them. We may be able to avoid this by encoding our cre...Amazon's automatic scraping of Github has found our public credentials shared on https://github.com/net4people/bbs/issues/335 which leads to their support team requiring us to rotate them. We may be able to avoid this by encoding our credentials (for example with base64) and having users pass in the encoded strings.
cc @mpuhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40339Avoid SQS queue reuse errors2024-03-05T17:40:02ZCecylia BocovichAvoid SQS queue reuse errorsAs described in https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40323#note_3002284, the reuse of the `sqsClientID` can cause errors on subsequent rendezvous attempts.As described in https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40323#note_3002284, the reuse of the `sqsClientID` can cause errors on subsequent rendezvous attempts.mpumpuhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40335No release for version 2.9.02024-02-27T16:41:36ZPonchoNo release for version 2.9.0Hi there
Some time ago, you've tagged version 2.9.0
It's available under https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/tags
But there is no corresponding release under https://gitlab.torproject.org/...Hi there
Some time ago, you've tagged version 2.9.0
It's available under https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/tags
But there is no corresponding release under https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/releases and the release job was skipped https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/jobs/471273
Not sure whether this is all on purpose or if something went wrong. Therefore, opening this issue.Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40334Post upgrade2024-02-17T22:19:10ZLinus Nordberglinus@torproject.orgPost upgrade- [x] apt autoremove; apt remove '~c'
- [x] apt-mark auto rsyslog && apt autoremove # https://www.debian.org/releases/bookworm/amd64/release-notes/ch-information.en.html- [x] apt autoremove; apt remove '~c'
- [x] apt-mark auto rsyslog && apt autoremove # https://www.debian.org/releases/bookworm/amd64/release-notes/ch-information.en.htmlhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40333Perform upgrade2024-02-17T21:55:05ZLinus Nordberglinus@torproject.orgPerform upgrade- [x] APT sources prepared
- [x] apt update && apt -o APT::Get::Trivial-Only=true full-upgrade
- [x] apt upgrade --without-new-pkgs
- [x] apt full-upgrade
- [x] reboot- [x] APT sources prepared
- [x] apt update && apt -o APT::Get::Trivial-Only=true full-upgrade
- [x] apt upgrade --without-new-pkgs
- [x] apt full-upgrade
- [x] reboothttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40332Prepare upgrade2024-02-17T10:41:33ZLinus Nordberglinus@torproject.orgPrepare upgrade- [x] systemd-resolved installed? no
- [x] apt purge ifupdown
- [x] upgrade 11.8 -> 11.9
- [x] apt autopurge; apt purge \\~c
- [x] find /etc -name '*.dpkg-*' -o -name '*.ucf-*' -o -name '*.merge-error'
- [x] dpkg --audit
- [x] apt-mark s...- [x] systemd-resolved installed? no
- [x] apt purge ifupdown
- [x] upgrade 11.8 -> 11.9
- [x] apt autopurge; apt purge \\~c
- [x] find /etc -name '*.dpkg-*' -o -name '*.ucf-*' -o -name '*.merge-error'
- [x] dpkg --audit
- [x] apt-mark showhold
- [x] dpkg --get-selections '*' > /root/dpkg-get-selections && (umask 0077; tar cf /root/2024-02-17-backup.tar -C / root etc var/lib/dpkg var/lib/apt/extended_states)https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40331Verify console access2024-02-17T09:27:48ZLinus Nordberglinus@torproject.orgVerify console access