Snowflake issueshttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues2020-08-06T16:36:36Zhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40006Reconfirm eclips.is account2020-08-06T16:36:36ZDavid Fifielddcf@torproject.orgReconfirm eclips.is accountI will take care of doing this, just relaying for transparency.
> Date: Thu, 6 Aug 2020 09:34:31 +0000<br>
> Subject: [eclips.is] Reconfirm account
>
> Dear eclips.is user,
>
> Once in a while we request you to reconfirm your account. T...I will take care of doing this, just relaying for transparency.
> Date: Thu, 6 Aug 2020 09:34:31 +0000<br>
> Subject: [eclips.is] Reconfirm account
>
> Dear eclips.is user,
>
> Once in a while we request you to reconfirm your account. This way we know your account is in active use.
>
> If your account is still in use, please confirm so by following this link:
>
> [redacted]
>
> Please do this before: Sep 1, 2020
>
> If we do not receive confirmation before this date, we will disable your resources first and eventually remove associated data
>
> Updates
> - Eclips.is depends on funding from the Open Technology Fund for platform operation. Due to a cut in this funding, the long-term future of the platform is uncertain. However, we expect to be able to continue providing services until at least April 2021.
> - Greenhost offers also a commercial platform. Run by the same people with the same care. Moving to the commercial platform will create more space for those who depend fully on the free services. Contact us if you are interested in moving to commercial.
> - The Hong Kong location will be decommissioned. Users of the Hong Kong infrastructure received a notification about this. All data will be removed after September 25th.
> Kind regards,
>
> The eclips.is team
Further details in a separate email:
> Date: Thu, 6 Aug 2020 12:51:18 +0200<br>
> Subject: eclips.is update
>
> Hello everyone!
>
> With this e-mail we want to update you on the latest about eclips.is platform.
>
> [Funding and future]<br>
> Eclips.is depends on funding from the Open Technology Fund for platform
> operation. With recent changes within OTF this funding is under pressure, and
> the long-term future of the platform is uncertain. We hereby want to clarify the
> future of eclips.is, especially as many projects using eclips.is services, are
> facing comparable challenges. They may therefore become even more dependent on
> this free service.
>
> We are working closely together with OTF to be able to keep the platform
> running. We expect to be able to maintain our current services until at least
> April 2021.
>
> We are also working on finding other funding options to continue offering these
> services after April 2021. Once we have more news on that we will of course
> share this with you.
>
> If you have the funds to move your eclips.is infrastructure to our Greenhost
> (commercial) platform, we encourage you to consider that move. By moving to
> Greenhost commercial, more funds and resources will be available for eclips.is
> to support those who can not afford commercial solutions. Please contact us this
> would be an option for you.
>
> [Account reconfirmation]<br>
> To make more efficient use of the eclips.is platform, we will ask you to confirm
> you account once in a while (~ every 3 months). With this new procedure we can
> optimize the use of the available resources.
> We sent an email on this to all email adresses of eclips.is users we have on
> record. If you did not receive this e-mail, please contact us. It is important
> we have the right e-mail address registered to support you.
>
> If we do not receive confirmation, we will first disable your resources and
> eventually remove the associated data and configuration..
>
> [Decommission Hong Kong Infrastructure]<br>
> It is with great regret we have to inform you we have to terminate our eclips.is
> services in Asia/Hong Kong on 25 September, 2020.
>
> Over the course of the last period the situation in Hong Kong has changed. On 1
> July, 2020, the PRC (China) government implemented a new security law, making it
> possible to extradite people from Hong Kong to mainland China, for those forming
> a threat to the PRC's authority.
>
> This new security law will make it more complex for Greenhost staff to operate
> and maintain the equipment in the Hong Kong data center. Apart from that, a
> significant amount of users of eclips.is no longer want to store data in Hong
> Kong.
>
> This decision was taken irrespective of, and before the changes at OTF. However,
> after this news we stopped investigating options to set up a new location in
> Asia.
>
> Last but not least, we want to thank everybody at OTF and in the community for
> remaining positive and energetic despite the amount of pressure everybody is
> facing right now!David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.org2020-09-01https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40050Upgrade snowflake bridge to 0.3.5.15 by 2021-06-15 for TROVE-2021-003, -004, ...2021-06-18T18:35:07ZDavid Fifielddcf@torproject.orgUpgrade snowflake bridge to 0.3.5.15 by 2021-06-15 for TROVE-2021-003, -004, -005, -006https://lists.torproject.org/pipermail/tor-talk/2021-June/045737.html
> In around two weeks–likely on the 14th or 15th– we plan to put out new stable Tor releases to fix issues in all currently released versions of Tor. There are three ...https://lists.torproject.org/pipermail/tor-talk/2021-June/045737.html
> In around two weeks–likely on the 14th or 15th– we plan to put out new stable Tor releases to fix issues in all currently released versions of Tor. There are three issues that will be fixed, with severity levels between "Medium" and "High" according to our classification system. The most severe issue, by our reckoning, is a denial-of-service issue affecting onion service clients. We'll share more details after people have time to patch. To the best of our knowledge, these vulnerabilities are not being exploited in the wild.
>
> The new releases will be 0.3.5.15, 0.4.4.9, 0.4.5.9, 0.4.6.5. The issues to be fixed are TROVE-2021-003 through TROVE-2021-006. When these releases are out, we will recommend that everybody upgrade, including clients *and* relays.David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.org2021-06-18https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40073Restart bridge and broker for update to virtualization platform2021-10-31T20:41:28ZDavid Fifielddcf@torproject.orgRestart bridge and broker for update to virtualization platformhttps://lists.torproject.org/pipermail/anti-censorship-team/2021-October/000196.html
> We are performing an update on our virtualization platform, this requires a change in the configuration of the storage backend. Due to this update yo...https://lists.torproject.org/pipermail/anti-censorship-team/2021-October/000196.html
> We are performing an update on our virtualization platform, this requires a change in the configuration of the storage backend. Due to this update your VPS needs to be stopped/started. We will execute this for you next week.
>
> If you prefer to plan this on your own, please feel free to stop/start your VPS yourself. By doing so, the VPS will be moved to the updated platform.
>
> Note: A reboot from within the machine itself will not be sufficient.
We [talked about this](http://meetbot.debian.net/tor-meeting/2021/tor-meeting.2021-10-28-16.00.log.html#l-34) at the 2021-10-28 anti-censorship team meeting.
@dcf will do the reboots, aiming for early on 2021-10-31 UTC.David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.org2021-10-31https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40161Snowflake Broker IP Change Rate Data HMAC Key Used the default value2023-07-29T18:04:14ZshelikhooSnowflake Broker IP Change Rate Data HMAC Key Used the default valueThis is an issue discovered in the https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40151 's configuration that the `ip-count-mask` used the default value, and make the counting result unsuitable ...This is an issue discovered in the https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40151 's configuration that the `ip-count-mask` used the default value, and make the counting result unsuitable for publishing.
I will deploy it again tomorrow with a randomly generated HMAC key. The collected unsalted IP count data will be retained and used for internal research only. It is not useless either, since it has collected data for over a month, and could be used for validation of this feature.shelikhooshelikhoo2022-08-03https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40173Increase number of tor instances from 4 to 8 on snowflake-01 bridge2022-12-03T20:40:46ZDavid Fifielddcf@torproject.orgIncrease number of tor instances from 4 to 8 on snowflake-01 bridgeBackground:
https://lists.torproject.org/pipermail/anti-censorship-team/2022-September/000247.html
The current 4 tor instances on the snowflake-01 bridge are saturated near 100% CPU since yesterday, due to increased usage. (Right now th...Background:
https://lists.torproject.org/pipermail/anti-censorship-team/2022-September/000247.html
The current 4 tor instances on the snowflake-01 bridge are saturated near 100% CPU since yesterday, due to increased usage. (Right now the bridge is at about 200 MB/s outgoing, 140 MB/s incoming.) We need to increase the number of instances.
I am a bit worried that 8 instances will permit more traffic than the hardware of the bridge will permit. I'll watch the performance and be prepared to step it back to 6.
While making the change, also increase `clientIDAddrMapCapacity` because snowflake-server is reporting constant "no address in clientID-to-IP map (capacity 10240)". Previous issue: #40084.
/cc @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.org2022-09-23https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40175Deploy snowflake-server performance improvements 2022-09-232022-11-16T17:48:41ZDavid Fifielddcf@torproject.orgDeploy snowflake-server performance improvements 2022-09-23The background is greatly increased usage and resulting load on the server in the past 2 days.
* [[anti-censorship-team] Need to increase number of tor instances on snowflake-01 bridge, increased usage since yesterday](https://lists.torp...The background is greatly increased usage and resulting load on the server in the past 2 days.
* [[anti-censorship-team] Need to increase number of tor instances on snowflake-01 bridge, increased usage since yesterday](https://lists.torproject.org/pipermail/anti-censorship-team/2022-September/000247.html)
* tpo/anti-censorship/pluggable-transports/snowflake#40173
* https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40086#note_2836854
snowflake-server is remaining functional, but restarting every few hours due to out-of-memory conditions. I did some profiling yesterday and found a few optimizations. They may help, or they may not.
https://gitlab.torproject.org/dcf/snowflake/-/compare/03b2b56f879879bb379cff8d7352ace1102d8811...c2f7003e2d316112db062540dcbe5ee569e2bd71?from_project_id=43
The plan is to first deploy the optimizations with profiling enabled, for a few hours, to collect profiles under load. Then re-deploy without profiling, later today if nothing goes wrong.
/cc @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.org2022-09-24https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40177Deploy further snowflake-server performance improvements 2022-09-242023-03-10T21:31:54ZDavid Fifielddcf@torproject.orgDeploy further snowflake-server performance improvements 2022-09-24The snowflake-01 bridge is getting close to its memory limits again after yesterday's deployments
(tpo/anti-censorship/pluggable-transports/snowflake#40173, tpo/anti-censorship/pluggable-transports/snowflake#40175).
The little dip marked...The snowflake-01 bridge is getting close to its memory limits again after yesterday's deployments
(tpo/anti-censorship/pluggable-transports/snowflake#40173, tpo/anti-censorship/pluggable-transports/snowflake#40175).
The little dip marked in the graph below, I investigated it,
and it occurred during a time when the memory was almost 100% full.
You can also tell the server is reaching its limits because the send and recv traces
become a little bit decorrelated.
For comparison, currently, at the right edge of the graph, RAM use is 85%.
![bandwidth on network interface](/uploads/a05f736ab902e70d569fd7dd78bb9abe/g2.png)
[Profiling yesterday](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40086#note_2837328) revealed two major contributors to memory usage:
`io.Copy` buffers and client send queues, each of which accounts for about 25%
of the total memory used by the process.
I have a branch that is meant to resolve the `io.Copy` buffer one.
It exposes the `io.WriterTo` interface of `smux.Stream` through `SnowflakeClientConn`,
which will prevent `io.Copy` from allocating a buffer per client.
I also put in a minor optimization to `ClientMap.SendQueue`,
the benefit of which turns out to be tiny,
but there's no reason not to do it.
https://gitlab.torproject.org/dcf/snowflake/-/compare/a16133ff03d4a7f05bea375e5aa34ff794ee316c...4d4fad30c429bba9062d1b67d56787d45721627d?from_project_id=850
/cc @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.org2022-09-25https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40176Increase number of tor instances from 8 to 12 on snowflake-01 bridge2022-10-15T02:40:41ZDavid Fifielddcf@torproject.orgIncrease number of tor instances from 8 to 12 on snowflake-01 bridgeAfter the instances were increased from 4 to 8 in tpo/anti-censorship/pluggable-transports/snowflake#40173, traffic has continued to increase, and now the 8 tor processes are each reaching above 90% CPU at times (cf. https://gitlab.torpr...After the instances were increased from 4 to 8 in tpo/anti-censorship/pluggable-transports/snowflake#40173, traffic has continued to increase, and now the 8 tor processes are each reaching above 90% CPU at times (cf. https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40175#note_2837026).
Increase the number of instances from 8 to 12.
/cc @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.org2022-09-25https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40179Reduce turbotunnel queueSize from 2048 to 5122022-12-08T15:04:48ZDavid Fifielddcf@torproject.orgReduce turbotunnel queueSize from 2048 to 512After yesterday's deployment of `io.Copy` memory mitigations (tpo/anti-censorship/pluggable-transports/snowflake#40177), memory use has crept up above 90% again.
I recommend implementing the [next most effective memory save](https://list...After yesterday's deployment of `io.Copy` memory mitigations (tpo/anti-censorship/pluggable-transports/snowflake#40177), memory use has crept up above 90% again.
I recommend implementing the [next most effective memory save](https://lists.torproject.org/pipermail/anti-censorship-team/2022-September/000253.html)
revealed by [profiling](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40086#note_2837328): decreasing the turbotunnel `queueSize`.
This is a fixed block of memory that's allocated per connected client.
In https://gitlab.torproject.org/dcf/snowflake/-/commit/328f0f4137145c08e58311888ad359362abaea79
I've reduced the parameter from 2048 to 512.
I don't know what effect this might have on performance,
apart from reducing memory use.
Past discussion about `queueSize`:
* https://lists.torproject.org/pipermail/anti-censorship-team/2021-July/000188.html
* https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/merge_requests/48#note_2744619David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.org2022-09-27https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40198Out of localhost ephemeral ports ("cannot assign requested address") on link ...2023-06-08T19:26:08ZDavid Fifielddcf@torproject.orgOut of localhost ephemeral ports ("cannot assign requested address") on link between snowflake-server and haproxySince 2022-09-28 10:57:26 (53 hours ago) the snowflake-server log is full of:
```
# grep 'cannot assign requested address$' /var/log/snowflake-server/snowflake-server.log | head
2022/09/28 10:57:26 handleConn: failed to connect to ORPor...Since 2022-09-28 10:57:26 (53 hours ago) the snowflake-server log is full of:
```
# grep 'cannot assign requested address$' /var/log/snowflake-server/snowflake-server.log | head
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
# grep 'cannot assign requested address$' /var/log/snowflake-server/snowflake-server.log | tail
2022/09/30 15:44:17 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/30 15:44:18 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/30 15:44:18 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/30 15:44:18 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/30 15:44:18 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
```
The error message means that the kernel could not allocate an ephemeral port number
for a localhost TCP connection.
In this case it for the connection between snowflake-server and haproxy.
My analysis at https://lists.torproject.org/pipermail/anti-censorship-team/2022-September/000265.html was incomplete:
the total number of localhost sockets does not matter for the purpose of counting 4-tuples,
but it *does* matter for the purpose of allocating ephemeral ports.
There are currently only 21712 sockets open between snowflake-server and haproxy,
which means the remainder of the ephemeral port range is taken up by other
localhost communication.
My immediate plan for this is to move haproxy to 127.0.0.2:10000 rather than 127.0.0.1:10000,
and maybe similarly with the individual tor instances if necessary.
/cc @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.org2022-09-30https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40125Update Landing Page (snowflake.torproject.org)2023-03-15T14:48:30ZsereneUpdate Landing Page (snowflake.torproject.org)We've had feedback that the snowflake.torproject.org landing page has been somewhat confusing / unclear. Many users are unsure whether to install the Snowflake Browser Extension, or run Tor Browser. (eg. Many users in Russia have install...We've had feedback that the snowflake.torproject.org landing page has been somewhat confusing / unclear. Many users are unsure whether to install the Snowflake Browser Extension, or run Tor Browser. (eg. Many users in Russia have installed the extension instead, being proxies instead of clients!)
I threw together a draft of an [updated landing page](https://serene.cx/snowflake-torproject-org-draft/ ) by slightly modifying the [current one](https://snowflake.torproject.org/).
To be improved further, but it's an idea:
![Screen_Shot_2022-04-11_at_18.16.33](/uploads/dccb730fb60564a0b9a4cf26a946a223/Screen_Shot_2022-04-11_at_18.16.33.png)
- I think the Tor Browser screenshot also needs to be updated.
- I have another idea, which I'll include in a separate ticket.
- Is this managed / on a separate host from main torproject.org?
This Snowflake landing page can be the very first place people see, so we should make it much more clear & helpful for the users. Let me know next steps @gus @gaba :)meskiomeskio@torproject.orgmeskiomeskio@torproject.org2022-09-30https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40239Experiment with increasing conntrack table size on snowflake-012024-01-22T15:19:37ZDavid Fifielddcf@torproject.orgExperiment with increasing conntrack table size on snowflake-01During performance experimentation we (@linus and I) [disabled connection tracking](tpo/anti-censorship/pluggable-transports/snowflake#40189), suspecting it as a cause of high CPU use and because the conntrack table appeared to be danger...During performance experimentation we (@linus and I) [disabled connection tracking](tpo/anti-censorship/pluggable-transports/snowflake#40189), suspecting it as a cause of high CPU use and because the conntrack table appeared to be dangerously close to being full.
Let we [re-enabled connection tracking](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40203#note_2851633) to no apparent ill effect.
[Performance optimizations in snowflake-server](tpo/anti-censorship/pluggable-transports/snowflake!100) were enough to bring the CPU use under control.
The conntrack table still appears to be close to overflowing, but we are not actually seeing any "nf_conntrack: table full, dropping packet" kernel logs that would indicate an actual problem.
At this moment the conntrack table is 92% full (`nf_conntrack_count` / `nf_conntrack_max` = 240316 / 262144).
It could be that this is normal and nothing to worry about.
Let's try doubling the maximum number of entries and see if it reaches a new equilibrium.
Currently we have:
```
# cat /proc/sys/net/netfilter/nf_conntrack_{count,max,buckets}
240316
262144
65536
```
I'm going to do this:
```
echo 131072 > /proc/sys/net/netfilter/nf_conntrack_buckets
echo 524288 > /proc/sys/net/netfilter/nf_conntrack_max
```
And meanwhile I'll run a tracking script to record `nf_conntrack_count` once per minute.David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.org2022-12-21https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40056O1.2: Increase the number of Snowflake proxies.2022-07-05T18:08:28ZGabagaba@torproject.orgO1.2: Increase the number of Snowflake proxies.With limited ramp up so far, we already have 6,000 volunteers running Snowflakes. Increasing Snowflake operators through more aggressive promotion will allow us to rapidly increase unblocked bridge entry points to the Tor network by thou...With limited ramp up so far, we already have 6,000 volunteers running Snowflakes. Increasing Snowflake operators through more aggressive promotion will allow us to rapidly increase unblocked bridge entry points to the Tor network by thousands.
- O1.2.1: Run a public campaign to encourage Internet users to install the Snowflake extension in Firefox or Chrome and use their web browser to act as bridges for censored people. This campaign will target individuals who care about free access to information but who may have limited capacity to operate a traditional bridge.
- O1.2.2: Scale Tor reachability through mobile Snowflakes. It’s also possible to build the Snowflake into mobile applications. We can rapidly increase the availability and diversity of bridge IP addresses that are not blocked in China by asking the millions of active Orbot users to “help censored users connect to Tor.” Through the Internews DISRUPT program, Guardian Project is designing and implementing “Orbot as Bridge” capability with the use of Snowflake, and as a result of this Activity, any Orbot user will be able to become a Snowflake bridge to Tor when they are not actively using their device. Guardian Project is also implementing as much of this capability as possible in Onion Browser for iOS.1 To do so, Guardian Project will: continue to improve Tor+Snowflake stability & performance on mobile devices and tune mobile-as-Snowflake bridge usability and performance, especially considering connections from the target regions.Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibet2022-12-31https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40268Restart snowflake bridges for haproxy CVE-2023-08362023-04-15T03:18:15ZDavid Fifielddcf@torproject.orgRestart snowflake bridges for haproxy CVE-2023-0836The vulnerability has to do with FastCGI, which we don't use.
https://security-tracker.debian.org/tracker/DSA-5388-1<br>
https://lists.debian.org/debian-security-announce/2023/msg00078.html
> An information leak vulnerability was disco...The vulnerability has to do with FastCGI, which we don't use.
https://security-tracker.debian.org/tracker/DSA-5388-1<br>
https://lists.debian.org/debian-security-announce/2023/msg00078.html
> An information leak vulnerability was discovered in HAProxy 2.1, 2.2 before 2.2.27, 2.3, 2.4 before 2.4.21, 2.5 before 2.5.11, 2.6 before 2.6.8, 2.7 before 2.7.1. There are 5 bytes left uninitialized in the connection buffer when encoding the FCGI_BEGIN_REQUEST record. Sensitive data may be disclosed to configured FastCGI backends in an unexpected way.
https://ubuntu.com/security/notices/USN-5994-1
> It was discovered that HAProxy incorrectly initialized certain connection
buffers. A remote attacker could possibly use this issue to obtain
sensitive information.
- [x] snowflake-01
- [x] snowflake-02
/cc @linus
Past haproxy upgrade issue: #40253.David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.org2023-04-18https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40270Upgrade tor on snowflake-01 to 0.4.72024-01-22T15:19:37ZDavid Fifielddcf@torproject.orgUpgrade tor on snowflake-01 to 0.4.7snowflake-01 is currently running 0.4.5.16, which is end of life and may get removed from the network as early as next week:
* https://mastodon.social/@torproject/110256009579078954
* https://forum.torproject.net/t/tor-relays-psa-tor-0-4...snowflake-01 is currently running 0.4.5.16, which is end of life and may get removed from the network as early as next week:
* https://mastodon.social/@torproject/110256009579078954
* https://forum.torproject.net/t/tor-relays-psa-tor-0-4-5-reaches-end-of-life-eol-on-2023-02-15/6338
We need to upgrade to the 0.4.7 series.
This probably means switching to the [torproject.org deb repository](https://support.torproject.org/apt/)
rather than the Debian one.
But we need to be careful with this one and be ready to revert the upgrade quickly if necessary.
0.4.7.13 has new support for the [`IP_BIND_ADDRESS_NO_PORT`](https://forum.torproject.net/t/stable-release-0-4-5-16-and-0-4-7-13/6216)
socket option when using `OutboundBindAddress` in torrc.
(As [we currently do](tpo/anti-censorship/pluggable-transports/snowflake#40223)
as an attempted mitigation for anti-DDoS scripts at middle relays.)
Past analysis has shown that different processes interact badly with one another
when they bind sockets to a particular address and differ in whether they use `IP_BIND_ADDRESS_NO_PORT`:
a situation where everything uses the option is stable,
as is one where nothing uses the option;
but when one process uses `IP_BIND_ADDRESS_NO_PORT` and other do not,
the one that does may have its ephemeral ports "stolen" by those that do,
leading to `EADDRNOTAVAIL` errors.
It's why we
[force a source port range](https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Guides/Snowflake-Bridge-Installation-Guide?version_id=831f28b730b50c6656ada23ceee0abdad2a134db#haproxy)
in haproxy.cfg, because that has the side effect of disabling `IP_BIND_ADDRESS_NO_PORT` in haproxy.
If tor's new use of `IP_BIND_ADDRESS_NO_PORT` starts giving `EADDRNOTAVAIL`,
the immediate remedy is to downgrade to an old version.
Another possible quick fix is to remove `OutboundBindAddress` from the torrc files:
if the socket is not pre-bound, there's no problem.
The longer-term solution is to make the other programs on the system that bind to a source address
also use `EADDRNOTAVAIL`.
For haproxy it's easy, just remove the source port ranges from haproxy.cfg.
The other programs to worry about are
[snowflake-server](tpo/anti-censorship/pluggable-transports/snowflake!120)
and [extor-static-cookie](https://gitlab.torproject.org/dcf/extor-static-cookie/-/commit/61e0684e40f84aa57a032e3fa9b80d2d24986bde),
both of which
[bind to a random localhost IP address in a range](https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Survival-Guides/Snowflake-Bridge-Installation-Guide?version_id=831f28b730b50c6656ada23ceee0abdad2a134db#localhost-address-ranges)
to avoid running out of localhost 4-tuples
(which is a separate issue from the `IP_BIND_ADDRESS_NO_PORT` one).
Last time I checked, [`net.Dialer`](https://pkg.go.dev/net#Dialer)
with `LocalAddr` set did *not* use `IP_BIND_ADDRESS_NO_PORT`.
Background on the `IP_BIND_ADDRESS_NO_PORT` issue:
* https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40201#note_2868367
* https://forum.torproject.net/t/tor-relays-inet-csk-bind-conflict/5757/16
* https://blog.cloudflare.com/the-quantum-state-of-a-tcp-port/
snowflake-02 is already running 0.4.7.13 without any apparent ill effects,
though it is at only about 15% the clients of snowflake-01.
/cc @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.org2023-05-10https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40280Remove proxy churn measurement from broker2023-10-16T14:33:23ZDavid Fifielddcf@torproject.orgRemove proxy churn measurement from broker#34075 and !95 added records to the broker
to estimate proxy churn.
We now have [done the analysis](https://github.com/turfed/snowflake-paper/tree/1c217201393bc15d2bf131b4be564f4bd00764af/figures/proxy-churn).
metrics-ip-salted.jsonl is ...#34075 and !95 added records to the broker
to estimate proxy churn.
We now have [done the analysis](https://github.com/turfed/snowflake-paper/tree/1c217201393bc15d2bf131b4be564f4bd00764af/figures/proxy-churn).
metrics-ip-salted.jsonl is now over 600 MB.
If we do not have plans for analyzing future measurements,
we should stop recording them.David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.org2023-08-10https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40278Snowflake failed to connect on Android 11 and above2024-01-10T16:06:45ZGedshSnowflake failed to connect on Android 11 and aboveThis issue causes Snowflake to be unable to establish a connection on Android 11 and above when the app's target SDK level is above 29.
It is related to the Go x/mobile issue https://github.com/golang/go/issues/40569
The problem is loc...This issue causes Snowflake to be unable to establish a connection on Android 11 and above when the app's target SDK level is above 29.
It is related to the Go x/mobile issue https://github.com/golang/go/issues/40569
The problem is located in the Pion library https://github.com/pion/ice/blob/214c86c5cfbc40fa9adbbea59081bd64c8b65d7c/agent.go#L327 which calls https://pkg.go.dev/github.com/mingyech/transport/v2/stdnet#Net.UpdateInterfaces which is forbidden on Android 11.
Snowflake log:
```go
2023/07/09 17:40:03 snowflake-client 2.5.1
2023/07/09 17:40:03 Started SOCKS listener at [scrubbed].
2023/07/09 17:40:17 SOCKS accepted: {[scrubbed] utls-imitate=hellorandomizedalpn map[utls-imitate:[hellorandomizedalpn]]}
2023/07/09 17:40:17
--- Starting Snowflake Client ---
2023/07/09 17:40:17 Using ICE servers:
2023/07/09 17:40:17 url: stun:stun.bluesip.net:3478
2023/07/09 17:40:17 url: stun:stun.epygi.com:3478
2023/07/09 17:40:17 url: stun:stun.stunprotocol.org:3478
2023/07/09 17:40:17 url: stun:stun.voys.nl:3478
2023/07/09 17:40:17 url: stun:stun.uls.co.za:3478
2023/07/09 17:40:17 url: stun:stun.sonetel.net:3478
2023/07/09 17:40:17 url: stun:stun.antisip.com:3478
2023/07/09 17:40:17 Rendezvous using Broker at: https://snowflake-broker.torproject.net.global.prod.fastly.net/
2023/07/09 17:40:17 Domain fronting using: cdn.sstatic.net
2023/07/09 17:40:17 ---- SnowflakeConn: begin collecting snowflakes ---
2023/07/09 17:40:17 ---- SnowflakeConn: starting a new session ---
2023/07/09 17:40:17 ---- SnowflakeConn: begin stream 3 ---
2023/07/09 17:40:17 redialing on same connection
2023/07/09 17:40:17 WebRTC: Collecting a new Snowflake. Currently at [0/1]
2023/07/09 17:40:17 snowflake-421fa3a84195d6e9 connecting...
2023/07/09 17:40:17 WebRTC: DataChannel created
2023/07/09 17:40:17 Failed to prepare offer failed to create network: route ip+net: netlinkrib: permission denied
2023/07/09 17:40:17 WebRTC: closing DataChannel
2023/07/09 17:40:17 WebRTC: closing PeerConnection
2023/07/09 17:40:17 WebRTC: Closing
2023/07/09 17:40:17 WebRTC: failed to create network: route ip+net: netlinkrib: permission denied Retrying...
2023/07/09 17:40:17 NAT Type: unrestricted
```Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetshelikhooshelikhoo2023-08-31https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40305Upgrade tor on snowflake bridges to 0.4.8.8 for TROVE 2023 0042023-11-07T00:11:57ZDavid Fifielddcf@torproject.orgUpgrade tor on snowflake bridges to 0.4.8.8 for TROVE 2023 004https://lists.torproject.org/pipermail/tor-relays/2023-November/021373.html
> Today (2023-11-03) the Network Team has released [new Tor versions 0.4.7.16 and 0.4.8.8](https://forum.torproject.org/t/security-release-0-4-7-16-and-0-4-8-8/...https://lists.torproject.org/pipermail/tor-relays/2023-November/021373.html
> Today (2023-11-03) the Network Team has released [new Tor versions 0.4.7.16 and 0.4.8.8](https://forum.torproject.org/t/security-release-0-4-7-16-and-0-4-8-8/10064). These updates contains a fix to a remote crash bug (TROVE 2023 004). It is highly recommended that all relay operators upgrade to the new versions as soon as possible to maintain the network stability and security.
>
> For those running their Tor relay using the Tor Debian repository, expect the new deb package to be available soon.
>
> The patches prevents the issue from causing a crash in Tor. However, it will make Tor more noisy when the bug is triggered, including logging information about the remote peer that is the source or destination of the circuit in the path. Such information is important for our developers to diagnose the specific invariant within Tor's TLS logic that does not hold.
- [x] snowflake-01
- [x] snowflake-02
/cc @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.org2023-11-06https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40319Figure out why acme/autocert has not renewed bridge TLS certificates since Oc...2024-01-11T06:27:48ZDavid Fifielddcf@torproject.orgFigure out why acme/autocert has not renewed bridge TLS certificates since OctoberI got emails from Let's Encrypt on 2023-12-28 and 2024-01-05 saying that the certificates for 02.snowflake.torproject.net and snowflake.torproject.net respectively would soon expire.
<details>
<summary>Subject: Let's Encrypt certificate...I got emails from Let's Encrypt on 2023-12-28 and 2024-01-05 saying that the certificates for 02.snowflake.torproject.net and snowflake.torproject.net respectively would soon expire.
<details>
<summary>Subject: Let's Encrypt certificate expiration notice for domain "02.snowflake.torproject.net"</summary>
Date: Thu, 28 Dec 2023 09:46:04 +0000<br>
From: Let's Encrypt Expiry Bot <expiry@letsencrypt.org><br>
To: dcf@torproject.org<br>
Subject: Let's Encrypt certificate expiration notice for domain "02.snowflake.torproject.net"
Hello,
Your certificate (or certificates) for the names listed below will expire in 19 days (on 2024-01-17). Please make sure to renew your certificate before then, or visitors to your web site will encounter errors.
We recommend renewing certificates automatically when they have a third of their total lifetime left. For Let's Encrypt's current 90-day certificates, that means renewing 30 days before expiration. See https://letsencrypt.org/docs/integration-guide/ for details.
02.snowflake.torproject.net
For details about when we send these emails, please visit: https://letsencrypt.org/docs/expiration-emails/ In particular, note that this reminder email is still sent if you've obtained a slightly different certificate by adding or removing names. If you've replaced this certificate with a newer one that covers more or fewer names than the list above, you may be able to ignore this message.
</details>
<details>
<summary>Subject: Let's Encrypt certificate expiration notice for domain "snowflake.torproject.net"</summary>
Date: Fri, 05 Jan 2024 17:19:27 +0000<br>
From: Let's Encrypt Expiry Bot <expiry@letsencrypt.org><br>
To: dcf@torproject.org<br>
Subject: Let's Encrypt certificate expiration notice for domain "snowflake.torproject.net"
Hello,
Your certificate (or certificates) for the names listed below will expire in 19 days (on 2024-01-25). Please make sure to renew your certificate before then, or visitors to your web site will encounter errors.
We recommend renewing certificates automatically when they have a third of their total lifetime left. For Let's Encrypt's current 90-day certificates, that means renewing 30 days before expiration. See https://letsencrypt.org/docs/integration-guide/ for details.
snowflake.torproject.net
For details about when we send these emails, please visit: https://letsencrypt.org/docs/expiration-emails/ In particular, note that this reminder email is still sent if you've obtained a slightly different certificate by adding or removing names. If you've replaced this certificate with a newer one that covers more or fewer names than the list above, you may be able to ignore this message.
</details>
Certificate renewal is supposed to happen automatically using the
[acme/autocert package](https://pkg.go.dev/golang.org/x/crypto@v0.17.0/acme/autocert).
But the current certificates are both from last October:
<details>
<summary>
<pre>
Validity
Not Before: Oct 15 14:03:54 2023 GMT
Not After : Jan 13 14:03:53 2024 GMT
Subject: CN = 02.snowflake.torproject.net
</pre>
</summary>
```
-----BEGIN CERTIFICATE-----
MIIEODCCAyCgAwIBAgISBILJIV5T+hp2SFg3D2z5RGrDMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMzEwMTUxNDAzNTRaFw0yNDAxMTMxNDAzNTNaMCYxJDAiBgNVBAMT
GzAyLnNub3dmbGFrZS50b3Jwcm9qZWN0Lm5ldDBZMBMGByqGSM49AgEGCCqGSM49
AwEHA0IABH5DrT87QcJxDEhNFZ15YHUR16BQAwNQpQ52qWKV1Y4oA8elZvE/XKHY
op2p6JEO3Sz/kjbcNH6dWXWIkRXz0CmjggIdMIICGTAOBgNVHQ8BAf8EBAMCB4Aw
HQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYD
VR0OBBYEFIetn19sSDfKBLDyYT7PfpQELn08MB8GA1UdIwQYMBaAFBQusxe3WFbL
rlAJQOYfr52LFMLGMFUGCCsGAQUFBwEBBEkwRzAhBggrBgEFBQcwAYYVaHR0cDov
L3IzLm8ubGVuY3Iub3JnMCIGCCsGAQUFBzAChhZodHRwOi8vcjMuaS5sZW5jci5v
cmcvMCYGA1UdEQQfMB2CGzAyLnNub3dmbGFrZS50b3Jwcm9qZWN0Lm5ldDATBgNV
HSAEDDAKMAgGBmeBDAECATCCAQQGCisGAQQB1nkCBAIEgfUEgfIA8AB3ADtTd3U+
LbmAToswWwb+QDtn2E/D9Me9AA0tcm/h+tQXAAABizPdegsAAAQDAEgwRgIhAPgR
OZAl/LXlhK8XIqa6MWEct6xODHoT4/pVBMRqsV0mAiEA7fO6V1+Gdc6rzViQBRTk
xJjsHzd5qHhf/ahbpZ/rJUMAdQDatr9rP7W2Ip+bwrtca+hwkXFsu1GEhTS9pD0w
SNf7qwAAAYsz3XogAAAEAwBGMEQCIHV/7jyPSrJe9bjMa2HMvwdu+FWChjvaPnY7
XYM55exqAiA8cdIDAZJymwt6PX/eWtyLhswbBD7iLbhlIRO6kfpcUTANBgkqhkiG
9w0BAQsFAAOCAQEASXh2o1Gh5nhhwKYEeU+wZWpqhIkvf+zbO7PEd/X1IMGRheRJ
UnobWUwzIn8Rrks6y3ktjSRtY2wY5QQgfXClYCMeleZLlp7IY1RDgG4oiqiXQ1Xr
ZJMQ+2cGnDGbdW+Jy2ISo3Mlc6H/TlfC7w6Ef+4NeTgVGbyuGKhHD0szASWfWsdO
jPtKVdxOYAyENAr0Xk/slNfgHubnKz5m1qHl0Lm8IEDgW56PjpLahQNkJM+7XUgz
52ANjQaToIzuafxGTCM2Ik0xry1/P7skI6KenuLfvxevS90qZ6GRTI6aAw/8PMR3
dWAv4LRds9DRL/NBRWTCNRNHwki9fNHciOiGzw==
-----END CERTIFICATE-----
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
04:82:c9:21:5e:53:fa:1a:76:48:58:37:0f:6c:f9:44:6a:c3
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, O = Let's Encrypt, CN = R3
Validity
Not Before: Oct 15 14:03:54 2023 GMT
Not After : Jan 13 14:03:53 2024 GMT
Subject: CN = 02.snowflake.torproject.net
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
pub:
04:7e:43:ad:3f:3b:41:c2:71:0c:48:4d:15:9d:79:
60:75:11:d7:a0:50:03:03:50:a5:0e:76:a9:62:95:
d5:8e:28:03:c7:a5:66:f1:3f:5c:a1:d8:a2:9d:a9:
e8:91:0e:dd:2c:ff:92:36:dc:34:7e:9d:59:75:88:
91:15:f3:d0:29
ASN1 OID: prime256v1
NIST CURVE: P-256
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
87:AD:9F:5F:6C:48:37:CA:04:B0:F2:61:3E:CF:7E:94:04:2E:7D:3C
X509v3 Authority Key Identifier:
14:2E:B3:17:B7:58:56:CB:AE:50:09:40:E6:1F:AF:9D:8B:14:C2:C6
Authority Information Access:
OCSP - URI:http://r3.o.lencr.org
CA Issuers - URI:http://r3.i.lencr.org/
X509v3 Subject Alternative Name:
DNS:02.snowflake.torproject.net
X509v3 Certificate Policies:
Policy: 2.23.140.1.2.1
CT Precertificate SCTs:
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : 3B:53:77:75:3E:2D:B9:80:4E:8B:30:5B:06:FE:40:3B:
67:D8:4F:C3:F4:C7:BD:00:0D:2D:72:6F:E1:FA:D4:17
Timestamp : Oct 15 15:03:54.635 2023 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:46:02:21:00:F8:11:39:90:25:FC:B5:E5:84:AF:17:
22:A6:BA:31:61:1C:B7:AC:4E:0C:7A:13:E3:FA:55:04:
C4:6A:B1:5D:26:02:21:00:ED:F3:BA:57:5F:86:75:CE:
AB:CD:58:90:05:14:E4:C4:98:EC:1F:37:79:A8:78:5F:
FD:A8:5B:A5:9F:EB:25:43
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : DA:B6:BF:6B:3F:B5:B6:22:9F:9B:C2:BB:5C:6B:E8:70:
91:71:6C:BB:51:84:85:34:BD:A4:3D:30:48:D7:FB:AB
Timestamp : Oct 15 15:03:54.656 2023 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:44:02:20:75:7F:EE:3C:8F:4A:B2:5E:F5:B8:CC:6B:
61:CC:BF:07:6E:F8:55:82:86:3B:DA:3E:76:3B:5D:83:
39:E5:EC:6A:02:20:3C:71:D2:03:01:92:72:9B:0B:7A:
3D:7F:DE:5A:DC:8B:86:CC:1B:04:3E:E2:2D:B8:65:21:
13:BA:91:FA:5C:51
Signature Algorithm: sha256WithRSAEncryption
Signature Value:
49:78:76:a3:51:a1:e6:78:61:c0:a6:04:79:4f:b0:65:6a:6a:
84:89:2f:7f:ec:db:3b:b3:c4:77:f5:f5:20:c1:91:85:e4:49:
52:7a:1b:59:4c:33:22:7f:11:ae:4b:3a:cb:79:2d:8d:24:6d:
63:6c:18:e5:04:20:7d:70:a5:60:23:1e:95:e6:4b:96:9e:c8:
63:54:43:80:6e:28:8a:a8:97:43:55:eb:64:93:10:fb:67:06:
9c:31:9b:75:6f:89:cb:62:12:a3:73:25:73:a1:ff:4e:57:c2:
ef:0e:84:7f:ee:0d:79:38:15:19:bc:ae:18:a8:47:0f:4b:33:
01:25:9f:5a:c7:4e:8c:fb:4a:55:dc:4e:60:0c:84:34:0a:f4:
5e:4f:ec:94:d7:e0:1e:e6:e7:2b:3e:66:d6:a1:e5:d0:b9:bc:
20:40:e0:5b:9e:8f:8e:92:da:85:03:64:24:cf:bb:5d:48:33:
e7:60:0d:8d:06:93:a0:8c:ee:69:fc:46:4c:23:36:22:4d:31:
af:2d:7f:3f:bb:24:23:a2:9e:9e:e2:df:bf:17:af:4b:dd:2a:
67:a1:91:4c:8e:9a:03:0f:fc:3c:c4:77:75:60:2f:e0:b4:5d:
b3:d0:d1:2f:f3:41:45:64:c2:35:13:47:c2:48:bd:7c:d1:dc:
88:e8:86:cf
```
</details>
<details>
<summary>
<pre>
Validity
Not Before: Oct 22 11:23:44 2023 GMT
Not After : Jan 20 11:23:43 2024 GMT
Subject: CN = snowflake.torproject.net
</pre>
</summary>
```
-----BEGIN CERTIFICATE-----
MIIEMjCCAxqgAwIBAgISBMnl0UqZyd2Ds/DcWpQdF13aMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMzEwMjIxMTIzNDRaFw0yNDAxMjAxMTIzNDNaMCMxITAfBgNVBAMT
GHNub3dmbGFrZS50b3Jwcm9qZWN0Lm5ldDBZMBMGByqGSM49AgEGCCqGSM49AwEH
A0IABDGdN4mjaNp0o+146WrLD5iuzDBpd4DrR9FmYbISj9KNQOVbty07VIWtdhel
KO95Phbj8Gmo8OlU2GCsRFuVczOjggIaMIICFjAOBgNVHQ8BAf8EBAMCB4AwHQYD
VR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0O
BBYEFMGfwy1PcMQaSDTB+69POTFrNxHRMB8GA1UdIwQYMBaAFBQusxe3WFbLrlAJ
QOYfr52LFMLGMFUGCCsGAQUFBwEBBEkwRzAhBggrBgEFBQcwAYYVaHR0cDovL3Iz
Lm8ubGVuY3Iub3JnMCIGCCsGAQUFBzAChhZodHRwOi8vcjMuaS5sZW5jci5vcmcv
MCMGA1UdEQQcMBqCGHNub3dmbGFrZS50b3Jwcm9qZWN0Lm5ldDATBgNVHSAEDDAK
MAgGBmeBDAECATCCAQQGCisGAQQB1nkCBAIEgfUEgfIA8AB2ANq2v2s/tbYin5vC
u1xr6HCRcWy7UYSFNL2kPTBI1/urAAABi1dXW9IAAAQDAEcwRQIgVx42UBDJsH1f
R48qhQrRUqGHF5Xn0W16ONOiTDf3m0gCIQCoUlDJjucW8suM+ZZll1IJrT2by9eM
ik+dLfJfeEIfAwB2ADtTd3U+LbmAToswWwb+QDtn2E/D9Me9AA0tcm/h+tQXAAAB
i1dXW8IAAAQDAEcwRQIhAJgErCNbhsx8+wrpmWJ3NO04EpRrw1qTMyecCM/viUL4
AiAMGXmerZwFANih+eRqPNfPpGYollCcCRsRYVgf9XHcbzANBgkqhkiG9w0BAQsF
AAOCAQEAccSUiBqi7/D7HZSLw6Ji1t3Dvy50wB13QAAJJY1juv8Izxgltcqd2Hvb
NHVI2GQ0gQ0DxHmbRX77t9CmjCYnhFQrVvzPguYECqDfBpxjjezr8T9axfGm2CIU
n2PnUP65p/E49D9PAidUkmz2NW5IkWhvm3tfdbkdbJNpHfnM1rPmcIA3Q2FGE+jj
bGTmBsngUL/XV38crKTssOny5Lcp8W7iYlWTpWKw1r3Ja/yW2aG89oF5RvgPLf1z
Q9bqG7Y7uYtSz1hGNiy4wGypiGD6Oj14cAKOKiPL2MH9j1NYosvi2LbFMN2cIByW
cV8VCezuZbnVpvubv5HTxDyW0gLtuw==
-----END CERTIFICATE-----
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
04:c9:e5:d1:4a:99:c9:dd:83:b3:f0:dc:5a:94:1d:17:5d:da
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, O = Let's Encrypt, CN = R3
Validity
Not Before: Oct 22 11:23:44 2023 GMT
Not After : Jan 20 11:23:43 2024 GMT
Subject: CN = snowflake.torproject.net
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
pub:
04:31:9d:37:89:a3:68:da:74:a3:ed:78:e9:6a:cb:
0f:98:ae:cc:30:69:77:80:eb:47:d1:66:61:b2:12:
8f:d2:8d:40:e5:5b:b7:2d:3b:54:85:ad:76:17:a5:
28:ef:79:3e:16:e3:f0:69:a8:f0:e9:54:d8:60:ac:
44:5b:95:73:33
ASN1 OID: prime256v1
NIST CURVE: P-256
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
C1:9F:C3:2D:4F:70:C4:1A:48:34:C1:FB:AF:4F:39:31:6B:37:11:D1
X509v3 Authority Key Identifier:
14:2E:B3:17:B7:58:56:CB:AE:50:09:40:E6:1F:AF:9D:8B:14:C2:C6
Authority Information Access:
OCSP - URI:http://r3.o.lencr.org
CA Issuers - URI:http://r3.i.lencr.org/
X509v3 Subject Alternative Name:
DNS:snowflake.torproject.net
X509v3 Certificate Policies:
Policy: 2.23.140.1.2.1
CT Precertificate SCTs:
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : DA:B6:BF:6B:3F:B5:B6:22:9F:9B:C2:BB:5C:6B:E8:70:
91:71:6C:BB:51:84:85:34:BD:A4:3D:30:48:D7:FB:AB
Timestamp : Oct 22 12:23:44.850 2023 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:45:02:20:57:1E:36:50:10:C9:B0:7D:5F:47:8F:2A:
85:0A:D1:52:A1:87:17:95:E7:D1:6D:7A:38:D3:A2:4C:
37:F7:9B:48:02:21:00:A8:52:50:C9:8E:E7:16:F2:CB:
8C:F9:96:65:97:52:09:AD:3D:9B:CB:D7:8C:8A:4F:9D:
2D:F2:5F:78:42:1F:03
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : 3B:53:77:75:3E:2D:B9:80:4E:8B:30:5B:06:FE:40:3B:
67:D8:4F:C3:F4:C7:BD:00:0D:2D:72:6F:E1:FA:D4:17
Timestamp : Oct 22 12:23:44.834 2023 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:45:02:21:00:98:04:AC:23:5B:86:CC:7C:FB:0A:E9:
99:62:77:34:ED:38:12:94:6B:C3:5A:93:33:27:9C:08:
CF:EF:89:42:F8:02:20:0C:19:79:9E:AD:9C:05:00:D8:
A1:F9:E4:6A:3C:D7:CF:A4:66:28:96:50:9C:09:1B:11:
61:58:1F:F5:71:DC:6F
Signature Algorithm: sha256WithRSAEncryption
Signature Value:
71:c4:94:88:1a:a2:ef:f0:fb:1d:94:8b:c3:a2:62:d6:dd:c3:
bf:2e:74:c0:1d:77:40:00:09:25:8d:63:ba:ff:08:cf:18:25:
b5:ca:9d:d8:7b:db:34:75:48:d8:64:34:81:0d:03:c4:79:9b:
45:7e:fb:b7:d0:a6:8c:26:27:84:54:2b:56:fc:cf:82:e6:04:
0a:a0:df:06:9c:63:8d:ec:eb:f1:3f:5a:c5:f1:a6:d8:22:14:
9f:63:e7:50:fe:b9:a7:f1:38:f4:3f:4f:02:27:54:92:6c:f6:
35:6e:48:91:68:6f:9b:7b:5f:75:b9:1d:6c:93:69:1d:f9:cc:
d6:b3:e6:70:80:37:43:61:46:13:e8:e3:6c:64:e6:06:c9:e0:
50:bf:d7:57:7f:1c:ac:a4:ec:b0:e9:f2:e4:b7:29:f1:6e:e2:
62:55:93:a5:62:b0:d6:bd:c9:6b:fc:96:d9:a1:bc:f6:81:79:
46:f8:0f:2d:fd:73:43:d6:ea:1b:b6:3b:b9:8b:52:cf:58:46:
36:2c:b8:c0:6c:a9:88:60:fa:3a:3d:78:70:02:8e:2a:23:cb:
d8:c1:fd:8f:53:58:a2:cb:e2:d8:b6:c5:30:dd:9c:20:1c:96:
71:5f:15:09:ec:ee:65:b9:d5:a6:fb:9b:bf:91:d3:c4:3c:96:
d2:02:ed:bb
```
</details>
Note also that the "Not After" dates in the certificates are 4–5 days earlier than
the dates in the emails.
I started thinking about what might have changed at
[the most recent deployment](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/merge_requests/154#note_2967886),
which was on 2023-11-21.
[This is the diff](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/compare/58c3121c...aa06e7be?from_project_id=43&straight=false)
with the [next most recent deployment](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40277#note_2918780)
on 2023-07-01.
What's weird is that certificates for the snowflake-01 bridge's alternative domain names
snowflake.freehaven.net and snowflake.bamsoftware.com have been renewed as recently
as a week ago:
```
2023-10-03 05:45 snowflake.bamsoftware.com+rsa
2023-10-22 12:23 snowflake.torproject.net
2023-10-27 18:12 snowflake.torproject.net+rsa
2023-12-08 22:50 snowflake.freehaven.net+rsa
2023-12-10 04:39 snowflake.bamsoftware.com
2023-12-30 05:16 snowflake.freehaven.net
```
About every 20–30 minutes I can see a file being created in the pt_state
directory like:
```
-rw------- 1 snowflake-server nogroup 903 Jan 7 21:04 snowflake.torproject.net+token
```
So it seems like the HTTP-01 proof renewal machinery is getting invoked,
but it's not getting completed for some reason.David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.org2024-01-12https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40298snowflake-01: Upgrade Debian 11 -> 122024-02-17T22:21:09ZLinus Nordberglinus@torproject.orgsnowflake-01: Upgrade Debian 11 -> 12I'd like to upgrade snowflake-01 to bookworm (12).
Which packages are crucial to have tested before, and have they been tested elsewhere?\
Is snowflake-02 running Debian?
/cc @dcfI'd like to upgrade snowflake-01 to bookworm (12).
Which packages are crucial to have tested before, and have they been tested elsewhere?\
Is snowflake-02 running Debian?
/cc @dcfLinus Nordberglinus@torproject.orgLinus Nordberglinus@torproject.org2024-02-18