Snowflake issueshttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues2024-03-26T10:41:06Zhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40354Extract reusable parts to a shared library2024-03-26T10:41:06Zmeskiomeskio@torproject.orgExtract reusable parts to a shared library[RoundCounter](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/main/broker/prometheus.go?ref_type=heads) is a useful wrapper on top of prometheus to round metrics to 8. We want to use it in other p...[RoundCounter](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/main/broker/prometheus.go?ref_type=heads) is a useful wrapper on top of prometheus to round metrics to 8. We want to use it in other projects like rdsys.
Another useful piece for other projects is [safelog](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/tree/main/common/safelog?ref_type=heads) that is already being imported by bridgestrap and conjure. Maybe we want to be able to import it without snowflake.
We could bundle both into a single library as this might make it easier to add other pieces in the future and each extra library makes it harder to package software to distros.meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40353Rename Container Image Tag for containers built from main branch to nightly2024-03-25T15:21:37ZshelikhooRename Container Image Tag for containers built from main branch to nightlyThe current container image tag for container images built from main branch is latest, which is typically expected to the most recent stable release, instead of the unstable main branch build result. The tag should be renamed to `nightly...The current container image tag for container images built from main branch is latest, which is typically expected to the most recent stable release, instead of the unstable main branch build result. The tag should be renamed to `nightly` as [discussed](https://lists.torproject.org/pipermail/tor-project/2024-March/003787.html) in the during the weekly meeting.shelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40352Use unreliable and unordered WebRTC data channels2024-03-21T20:15:25ZDavid Fifielddcf@torproject.orgUse unreliable and unordered WebRTC data channels@shelikhoo:
Actually here are some observation from me related to https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40243:
Snowflake is currently using network resource in a so suboptimal way I ...@shelikhoo:
Actually here are some observation from me related to https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40243:
Snowflake is currently using network resource in a so suboptimal way I think it would make sense to also consider make protocol level change on how kcp is interacting with webrtc before considering to add forward error correction. This would be in the form of enabling unreliable mode of webrtc and make necessary change to get it to work.
Right now, kcp packets are sent in webrtc data channel in a reliable way that deliver all packets in order and retransmit any lost message repeatedly. However, kcp also retransmit its packet itself, which as a result, queue all those retransmitted packets somewhere, like in webrtc's buffer.
This means lost packets are required to be retransmitted more than once in different protocol, and retransmit & timeout get compounded. More retransmit result in more lost packets and more retransmission, which eventually lead to [connection melt down](https://openvpn.net/faq/what-is-tcp-meltdown/) <- please read.
back pressure like the one introduced in https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/merge_requests/144 only move the problem, and block kcp's send in unexpected way, as kcp don't expect send to block as it is usually over udp.
See also: https://lists.torproject.org/pipermail/anti-censorship-team/2023-March/000286.html
(@dcf split this issue off from #40251 to separate the analysis of speed in China from the proposed remedy.)shelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40351broker is not publishing metrics to collector2024-03-21T15:06:48Zmeskiomeskio@torproject.orgbroker is not publishing metrics to collectorYesterday (2024-03-20) the snowflake broker didn't publish any metrics for collector. I have checked /home/snowflake-broker/metrics.log in the broker server and the last published metric is:
```
snowflake-stats-end 2024-03-19 23:21:24 (8...Yesterday (2024-03-20) the snowflake broker didn't publish any metrics for collector. I have checked /home/snowflake-broker/metrics.log in the broker server and the last published metric is:
```
snowflake-stats-end 2024-03-19 23:21:24 (86400 s)
```
Looking at grafana everything seems to be working normally.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40349Upgrade snowflake broker machine from Debian 102024-03-19T16:23:25ZCecylia BocovichUpgrade snowflake broker machine from Debian 10Looks like Debian 10 will not be supported by the LTS team after this June: https://wiki.debian.org/LTSLooks like Debian 10 will not be supported by the LTS team after this June: https://wiki.debian.org/LTShttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40348Snowflake addon not found in firefox store2024-03-16T17:04:01ZSven GottwaldSnowflake addon not found in firefox storeI followed the link on [Browser Snowflake proxy](https://community.torproject.org/relay/setup/snowflake/browser/) that leads to https://addons.mozilla.org/en-US/firefox/addon/torproject-snowflake/. The page says:
> **Oops! We can’t find...I followed the link on [Browser Snowflake proxy](https://community.torproject.org/relay/setup/snowflake/browser/) that leads to https://addons.mozilla.org/en-US/firefox/addon/torproject-snowflake/. The page says:
> **Oops! We can’t find that page**
>
> If you’ve followed a link from another site for an extension or theme, that item is no longer available. This could be because:
> - The developer removed it. Developers commonly do this because they no longer support the extension or theme, or have replaced it.
> - Mozilla removed it. This can happen when issues are found during the review of the extension or theme, or the extension or theme has been abusing the terms and conditions for addons.mozilla.org. The developer has the opportunity to resolve the issues and make the add-on available again.
>
> Try visiting the page later, as the theme or extension may become available again. Alternatively, you may be able to find what you’re looking for in one of the available extensions or themes, or by asking for help on our community forums.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40347Deploy new SQS features and fixes2024-03-21T13:41:39ZCecylia BocovichDeploy new SQS features and fixesNow that we have country-specific metrics for client rendezvous methods (!258), and some client-side fixes (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/merge_requests/264, https://gitlab.torproject....Now that we have country-specific metrics for client rendezvous methods (!258), and some client-side fixes (https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/merge_requests/264, https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/merge_requests/264), we should make sure these changes are deployed. That involves:
- [X] releasing a new Snowflake version [v2.9.2](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/releases/v2.9.2)
- [x] deploying a new version of the broker
- [x] opening a tor-browser-build MR to update the client (https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/merge_requests/936)Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40346Start disabled2024-03-18T17:30:14ZcypherpunksStart disabledRegardless of any other settings, I would suggest Snowflake never begin operating automatically upon installation, instead requiring the first use on any given device to be initiated manually.
I briefly had Snowflake installed on a pers...Regardless of any other settings, I would suggest Snowflake never begin operating automatically upon installation, instead requiring the first use on any given device to be initiated manually.
I briefly had Snowflake installed on a personal device, where it was disabled while I looked into the possibility of using a DNS sinkhole to prevent the use of my connection for undesirable purposes. I had preemptively turned services.sync.addons.ignoreUserEnabledChanges on so that, once I was comfortable, enabling Snowflake on my personal device I would not inadvertently enable it on my work computer. I unexpectedly needed to have the work machine reset and did not disable this flag, so Snowflake was installed and enabled when I synchronised my settings. I responded quickly and uninstalled the extension entirely, but it appears to have been active for long enough to have routed a connection to the website of a violent extremist group that was identified and flagged by our IT systems. This incident has caused me to seriously reconsider the risk using Snowflake creates, not just to myself but also by inadvertently enabling uses like the connection in question despite my efforts to prevent doing so, and as a result I am highly unlikely to reinstall it.
That this situation involved a mistake on my part does not justify it as a possibility. It cannot be expected that no user will ever make such a mistake - even advanced users cannot be expected to never forget things - and if such a simple and potentially-unavoidable mistake can cause automatic operation to put the user at risk like this then safeguards should be put in place both to protect them and to avoid deterring them entirely.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40345migrate docker image to this repo2024-03-23T19:38:24Zmeskiomeskio@torproject.orgmigrate docker image to this repoWe used to develop the docker image in a separated repo: https://gitlab.torproject.org/tpo/anti-censorship/docker-snowflake-proxy/
But now we have a CI building the docker image in this repo: !246
Let's deprecate the original docker re...We used to develop the docker image in a separated repo: https://gitlab.torproject.org/tpo/anti-censorship/docker-snowflake-proxy/
But now we have a CI building the docker image in this repo: !246
Let's deprecate the original docker repo and move everything here. Things that might be missing:
* [ ] move docker-compose.yml to this repo or somewhere
* [ ] update the community documentation to use our repo
* [ ] integrate publishing the docker image in the release process
* [ ] are we cross building in the CI?
* [ ] how are we going to push to dockerhub the image?
* [ ] archive docker-snowflake-proxy reposhelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40344Snowflake works unreliably in China, 2024 Q12024-03-05T12:21:14ZshelikhooSnowflake works unreliably in China, 2024 Q1We have been receiving conflicting report about connectivity interruptions in China.
There was one report from user that highlighted this issue: https://github.com/net4people/bbs/issues/325. We was able to observe similar interruption o...We have been receiving conflicting report about connectivity interruptions in China.
There was one report from user that highlighted this issue: https://github.com/net4people/bbs/issues/325. We was able to observe similar interruption on our vantage point: https://gitlab.torproject.org/tpo/anti-censorship/connectivity-measurement/bridgestatus/-/blob/dc663e36d7dc81467a63f59c5d435b9f93e9e3ab/recentResult_cnnext#L89 .
The exact way connection get interrupted differ from report to report. The report from github user shows the connection can be established, but was interrupted soon. The report from vantage point show dtls connection handshake was unsuccessful, or the remote server was unreachable.
As of now, the censorship we are observing is decreasing, as some report's subsequent report show successful connection after waiting sufficiently long.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40343Make the client automatic try and report to user what snowflake options combi...2024-03-04T10:16:38Zsnowflake_user_40314Make the client automatic try and report to user what snowflake options combination workI think current we have too many snowflake options combination(e.g. front domains, STUN servers, URLs of brokers at various CDN, and others).\
Thus, I think perhaps we should provide a way let user input the potential options as front do...I think current we have too many snowflake options combination(e.g. front domains, STUN servers, URLs of brokers at various CDN, and others).\
Thus, I think perhaps we should provide a way let user input the potential options as front domains list, STUN servers list, URLs of brokers list, and others; then automatic try report to user what options combination(or bridge line) work.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40342Shadow integration tests occasionally panic2024-03-07T22:51:40ZCecylia BocovichShadow integration tests occasionally panicA recent job failed: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/jobs/491691
This is likely runner-dependent, since no changes were made to the Shadow tests since it last passed:
```
$ shadow --log...A recent job failed: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/jobs/491691
This is likely runner-dependent, since no changes were made to the Shadow tests since it last passed:
```
$ shadow --log-level=debug --model-unblocked-syscall-latency=true snowflake-minimal.yaml > shadow.log
** Starting Shadow v3.0.0-557-g193924aa 2023-08-25--13:24:51 with GLib v2.66.8
thread 'shadow-worker' panicked at 'called `Result::unwrap()` on an `Err` value: ENOSYS', main/utility/childpid_watcher.rs:269:37
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'shadow-worker' panicked at 'called `Result::unwrap()` on an `Err` value: PoisonError { .. }', main/utility/childpid_watcher.rs:268:43
thread 'shadow-worker' panicked at 'called `Result::unwrap()` on an `Err` value: PoisonError { .. }thread '', shadow-workermain/utility/childpid_watcher.rs' panicked at ':assertion failed: self.shim_shmem_lock.borrow().is_none()268', :main/host/host.rs43:
971:9
fatal runtime error: thread local panicked on drop
thread 'shadow-worker' panicked at 'called `Result::unwrap()` on an `Err` value: PoisonError { .. }', main/utility/childpid_watcher.rs:268:43
thread 'shadow-worker' panicked at 'assertion failed: self.shim_shmem_lock.borrow().is_none()', main/host/host.rs:971:9/bin/bash: line 210: 30403 Aborted (core dumped) shadow --log-level=debug --model-unblocked-syscall-latency=true snowflake-minimal.yaml > shadow.log
```https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40341Encode AWS credentials for SQS rendezvous2024-03-12T11:26:12ZCecylia BocovichEncode AWS credentials for SQS rendezvousAmazon's automatic scraping of Github has found our public credentials shared on https://github.com/net4people/bbs/issues/335 which leads to their support team requiring us to rotate them. We may be able to avoid this by encoding our cre...Amazon's automatic scraping of Github has found our public credentials shared on https://github.com/net4people/bbs/issues/335 which leads to their support team requiring us to rotate them. We may be able to avoid this by encoding our credentials (for example with base64) and having users pass in the encoded strings.
cc @mpuhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40340Add a mechanism to retest the client NAT type2024-03-04T08:42:02ZCecylia BocovichAdd a mechanism to retest the client NAT typeWhile we do periodically retest the NAT type of proxies, a client's NAT type is only checked once on startup. The result is that if, after the initial check, a client's network conditions change, they may have difficulty connecting to pr...While we do periodically retest the NAT type of proxies, a client's NAT type is only checked once on startup. The result is that if, after the initial check, a client's network conditions change, they may have difficulty connecting to proxies in their pool. Since client usage of snowflake is much more time-sensitive than proxies, the trigger for a retest could be a threshold of a certain number of failed Datachannel attempts.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40339Avoid SQS queue reuse errors2024-03-05T17:40:02ZCecylia BocovichAvoid SQS queue reuse errorsAs described in https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40323#note_3002284, the reuse of the `sqsClientID` can cause errors on subsequent rendezvous attempts.As described in https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40323#note_3002284, the reuse of the `sqsClientID` can cause errors on subsequent rendezvous attempts.mpumpuhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40337AWS warning about public IAM credentials for SQS rendezvous2024-03-08T12:58:24ZCecylia BocovichAWS warning about public IAM credentials for SQS rendezvousI got the following email from AWS:
```
We have become aware that the AWS Access Key AKIA5AIF4WJJXS7YHEG3 , belonging to IAM User SQS-client ,
along with the corresponding Secret Key is publicly available online at
https://github.com/n...I got the following email from AWS:
```
We have become aware that the AWS Access Key AKIA5AIF4WJJXS7YHEG3 , belonging to IAM User SQS-client ,
along with the corresponding Secret Key is publicly available online at
https://github.com/net4people/bbs/issues/335#issue-2157478835 .
Your security is important to us and this exposure of your account’s IAM credentials poses a security
risk to your AWS account, could lead to excessive charges from unauthorized activity, and violates
the AWS Customer Agreement or other agreement with us governing your use of our Services.
```
They probably have some automated tools to search for secret keys in Github repositories.
I have replied to the open support ticket to confirm that the sharing of credentials was intentional. Hopefully they will allow us to continue to use them.Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40336Should we default enable SQS rendezvous of Snowflake in built-in bridge?2024-02-28T17:55:43Zsnowflake_user_40314Should we default enable SQS rendezvous of Snowflake in built-in bridge?The current default rendezvous(domain fronting) is [expect to stop work](https://lists.torproject.org/pipermail/anti-censorship-team/2023-October/000328.html).
Current 13.0.10 do not enable SQS rendezvous by default.The current default rendezvous(domain fronting) is [expect to stop work](https://lists.torproject.org/pipermail/anti-censorship-team/2023-October/000328.html).
Current 13.0.10 do not enable SQS rendezvous by default.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40335No release for version 2.9.02024-02-27T16:41:36ZPonchoNo release for version 2.9.0Hi there
Some time ago, you've tagged version 2.9.0
It's available under https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/tags
But there is no corresponding release under https://gitlab.torproject.org/...Hi there
Some time ago, you've tagged version 2.9.0
It's available under https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/tags
But there is no corresponding release under https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/releases and the release job was skipped https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/jobs/471273
Not sure whether this is all on purpose or if something went wrong. Therefore, opening this issue.Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40334Post upgrade2024-02-17T22:19:10ZLinus Nordberglinus@torproject.orgPost upgrade- [x] apt autoremove; apt remove '~c'
- [x] apt-mark auto rsyslog && apt autoremove # https://www.debian.org/releases/bookworm/amd64/release-notes/ch-information.en.html- [x] apt autoremove; apt remove '~c'
- [x] apt-mark auto rsyslog && apt autoremove # https://www.debian.org/releases/bookworm/amd64/release-notes/ch-information.en.htmlhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40333Perform upgrade2024-02-17T21:55:05ZLinus Nordberglinus@torproject.orgPerform upgrade- [x] APT sources prepared
- [x] apt update && apt -o APT::Get::Trivial-Only=true full-upgrade
- [x] apt upgrade --without-new-pkgs
- [x] apt full-upgrade
- [x] reboot- [x] APT sources prepared
- [x] apt update && apt -o APT::Get::Trivial-Only=true full-upgrade
- [x] apt upgrade --without-new-pkgs
- [x] apt full-upgrade
- [x] reboot