Anti-censorship issueshttps://gitlab.torproject.org/groups/tpo/anti-censorship/-/issues2023-04-19T11:43:07Zhttps://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/158Telegram Bridge Distributor responding with a blank message2023-04-19T11:43:07Zchampionquizzerchampionquizzer@torproject.orgTelegram Bridge Distributor responding with a blank messageThe @GetBridgesBot on Telegram is currently responding with a blank message and no bridge lines.
This is the response I am getting with `/bridges` or `Bridges`:
```
Your bridges:
```The @GetBridgesBot on Telegram is currently responding with a blank message and no bridge lines.
This is the response I am getting with `/bridges` or `Bridges`:
```
Your bridges:
```https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/157S3 existence objects cause archive.org torrent files to change every hour2023-08-23T18:59:04ZVortS3 existence objects cause archive.org torrent files to change every hourFor torrents to be useful, their content should be as stable as possible.
This is not true for TorBrowser torrents right now:
They contain `.exist-gettor` files, which are updated every hour.
And since they are updated, `_meta.sqli...For torrents to be useful, their content should be as stable as possible.
This is not true for TorBrowser torrents right now:
They contain `.exist-gettor` files, which are updated every hour.
And since they are updated, `_meta.sqlite` file gets updated too.
I think either `.exist-gettor` files should be removed from archives at all or they should be updated only when necessary.
Example: few hours ago https://archive.org/download/torbrowser-12.0.4-dbd1a7abad9ee3d1/torbrowser-12.0.4-dbd1a7abad9ee3d1_archive.torrent file had BTIH of `A3445D681702D17AEBB4886332CAF6DC7EEF6B3F`, now it have BTIH of `FEBA482D46AB6B853F1140A66EC2F307B3CFACE7`.meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/lox/-/issues/9Reasonable and effective integration with rdsys2023-06-20T19:08:49ZonyinyangReasonable and effective integration with rdsysThis issue may actually lead to several new issues in various repos but for now I am placing it here.
Through experimentation with rdsys and attempting to integrate it with the existing Lox library, it has become apparent that Lox and rd...This issue may actually lead to several new issues in various repos but for now I am placing it here.
Through experimentation with rdsys and attempting to integrate it with the existing Lox library, it has become apparent that Lox and rdsys may not play nicely together in both of their current forms. This may be more of a problem with myself having an unequal understanding of both sides of this integration. Here are some things that I plan to look into with this issue:
- [x] Understand what exactly rdsys sends to a distributor and why
- what is a gone/changed/new resource?
- is rdsys behaving as expected for each of these resource types?
- [x] What information does Lox need from rdsys and how should it behave with the resources it is provided
- how should syncing between Lox and rdsys work?
- how are bridges that are down/slow/unreliable handled?
- does it make most sense to make changes in Lox or rdsys
- [x] Actually do the integration based on the above findingsonyinyangonyinyanghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake-webext/-/issues/82WS.makeWebsocket ignores params (i.e. `client_ip`), losing country statistics2024-02-13T21:15:19ZDavid Fifielddcf@torproject.orgWS.makeWebsocket ignores params (i.e. `client_ip`), losing country statisticsI am investigating a sudden partial loss of [`client_ip` statistics](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/18628) that occurred after 2022-06-27,
when the fraction of connections that h...I am investigating a sudden partial loss of [`client_ip` statistics](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/18628) that occurred after 2022-06-27,
when the fraction of connections that had `client_ip` set dropped from 99% to 94%.
I have tracked it down to commit 15768f50c0ddd68d3ffb815cd532ddbd3d85fd41, part of !29,
which was released in [snowflake-webextension-0.6.0](https://archive.org/details/snowflake-webextension-0.6.0)
on the day in question, 2022-06-27.
The bug is in [`WS.makeWebsocket`](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake-webext/-/blob/995a42710044ae1df0fb839b12906049549c2074/websocket.js#L13-29).
Notice how `params` is used to create `parsedURL`—but then `parsedURL` is thrown away
and the `WebSocket` constructor is called on the unmodified `url` argument:
```js
static makeWebsocket(url, params) {
let parsedURL = new URL(url);
let urlpa = new URLSearchParams(params);
urlpa.forEach(function (value, key) {
parsedURL.searchParams.set(key, value);
});
let ws = new WebSocket(url);
ws.binaryType = 'arraybuffer';
return ws;
}
```
The effect of this bug is a partial loss of client IP geolocation statistics at the bridge.
Connections will be attributed to `??` rather than the proper country code.
snowflake-server [logs once a day](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/commit/d9e8f8f6479a8b4abe31eb5bfb9023de1bbca8af)
the number of connections that had `client_ip` set:
```
$ grep 'connections had client_ip$' /var/log/snowflake-server/snowflake-server.log
2022/06/25 17:53:16 in the past 86400 s, 194793/195854 connections had client_ip
2022/06/26 17:53:16 in the past 86400 s, 194526/195608 connections had client_ip
2022/06/27 17:53:16 in the past 86400 s, 185234/187519 connections had client_ip
2022/06/28 17:53:16 in the past 86400 s, 172629/183925 connections had client_ip
2022/06/29 17:53:16 in the past 86400 s, 169512/180011 connections had client_ip
2022/06/30 17:53:16 in the past 86400 s, 162445/172427 connections had client_ip
```
Converting to percentages, we see that after 2022-06-27 the proportion dropped
from 99+% to about 94%:
```
$ grep 'connections had client_ip$' /var/log/snowflake-server/snowflake-server.log | perl -ane '$F[7] =~ m#^(.*)/(.*)$#; printf("%s %s %8d/%-8d %6.2f%%\n", $F[0], $F[1], $1, $2, 100 * ($2 == 0 ? 0.0 : $1/$2));'
2022/06/25 17:53:16 194793/195854 99.46%
2022/06/26 17:53:16 194526/195608 99.45%
2022/06/27 17:53:16 185234/187519 98.78%
2022/06/28 17:53:16 172629/183925 93.86%
2022/06/29 17:53:16 169512/180011 94.17%
2022/06/30 17:53:16 162445/172427 94.21%
```
<details>
<summary>Full percentage log</summary>
```
2022/04/07 17:53:16 4/4 100.00%
2022/04/08 17:53:16 0/0 0.00%
2022/04/09 17:53:16 0/0 0.00%
2022/04/10 17:53:16 0/0 0.00%
2022/04/11 17:53:16 14755/14846 99.39%
2022/04/12 17:53:16 155044/156119 99.31%
2022/04/13 17:53:16 151189/152289 99.28%
2022/04/14 17:53:16 151366/152522 99.24%
2022/04/15 17:53:16 159661/160890 99.24%
2022/04/16 17:53:16 165119/166343 99.26%
2022/04/17 17:53:16 170390/171655 99.26%
2022/04/18 17:53:16 158072/159215 99.28%
2022/04/19 17:53:16 155255/156348 99.30%
2022/04/20 17:53:16 153641/154624 99.36%
2022/04/21 17:53:16 154686/155618 99.40%
2022/04/22 17:53:16 166803/167743 99.44%
2022/04/23 17:53:16 170969/172005 99.40%
2022/04/24 17:53:16 163772/164752 99.41%
2022/04/25 17:53:16 164409/165359 99.43%
2022/04/26 17:53:16 161246/162234 99.39%
2022/04/27 17:53:16 161217/162123 99.44%
2022/04/28 17:53:16 161105/162029 99.43%
2022/04/29 17:53:16 167518/168391 99.48%
2022/04/30 17:53:16 174056/174942 99.49%
2022/05/01 17:53:16 172758/173729 99.44%
2022/05/02 17:53:16 171274/172289 99.41%
2022/05/03 17:53:16 174248/175357 99.37%
2022/05/04 17:53:16 177377/178406 99.42%
2022/05/05 17:53:16 174350/175339 99.44%
2022/05/06 17:53:16 179096/180160 99.41%
2022/05/07 17:53:16 181913/182982 99.42%
2022/05/08 17:53:16 175225/176240 99.42%
2022/05/09 17:53:16 190505/191629 99.41%
2022/05/10 17:53:16 189615/190739 99.41%
2022/05/11 17:53:16 180694/181761 99.41%
2022/05/12 17:53:16 171556/172619 99.38%
2022/05/13 17:53:16 172153/173163 99.42%
2022/05/14 17:53:16 180281/181201 99.49%
2022/05/15 17:53:16 176023/177015 99.44%
2022/05/16 17:53:16 172430/173301 99.50%
2022/05/17 17:53:16 171169/172126 99.44%
2022/05/18 17:53:16 175336/176319 99.44%
2022/05/19 17:53:16 177217/178199 99.45%
2022/05/20 17:53:16 176879/177677 99.55%
2022/05/21 17:53:16 187563/188484 99.51%
2022/05/22 17:53:16 190738/191674 99.51%
2022/05/23 17:53:16 180016/180821 99.55%
2022/05/24 17:53:16 184586/185499 99.51%
2022/05/25 17:53:16 182584/183455 99.53%
2022/05/26 17:53:16 185874/186754 99.53%
2022/05/27 17:53:16 185319/186194 99.53%
2022/05/28 17:53:16 190375/191119 99.61%
2022/05/29 17:53:16 186398/187169 99.59%
2022/05/30 17:53:16 184392/185282 99.52%
2022/05/31 17:53:16 181668/182598 99.49%
2022/06/01 17:53:16 182936/183759 99.55%
2022/06/02 17:53:16 189689/190490 99.58%
2022/06/03 17:53:16 193557/194353 99.59%
2022/06/04 17:53:16 194960/195770 99.59%
2022/06/05 17:53:16 194091/194822 99.62%
2022/06/06 17:53:16 191104/191859 99.61%
2022/06/07 17:53:16 191891/192723 99.57%
2022/06/08 17:53:16 194276/194989 99.63%
2022/06/09 17:53:16 196756/197772 99.49%
2022/06/10 17:53:16 188985/189979 99.48%
2022/06/11 17:53:16 189916/190888 99.49%
2022/06/12 17:53:16 185694/186730 99.45%
2022/06/13 17:53:16 206684/207989 99.37%
2022/06/14 17:53:16 207371/208717 99.36%
2022/06/15 17:53:16 199041/200315 99.36%
2022/06/16 17:53:16 189537/190608 99.44%
2022/06/17 17:53:16 185730/186857 99.40%
2022/06/18 17:53:16 198655/199636 99.51%
2022/06/19 17:53:16 192534/193580 99.46%
2022/06/20 17:53:16 196991/198095 99.44%
2022/06/21 17:53:16 184941/185865 99.50%
2022/06/22 17:53:16 183693/184676 99.47%
2022/06/23 17:53:16 181582/182624 99.43%
2022/06/24 17:53:16 184876/185957 99.42%
2022/06/25 17:53:16 194793/195854 99.46%
2022/06/26 17:53:16 194526/195608 99.45%
2022/06/27 17:53:16 185234/187519 98.78%
2022/06/28 17:53:16 172629/183925 93.86%
2022/06/29 17:53:16 169512/180011 94.17%
2022/06/30 17:53:16 162445/172427 94.21%
2022/07/01 17:53:16 166284/176521 94.20%
2022/07/02 17:53:16 169610/179069 94.72%
2022/07/03 17:53:16 176635/186127 94.90%
2022/07/04 17:53:16 168869/179203 94.23%
2022/07/05 17:53:16 163400/173719 94.06%
2022/07/06 17:53:16 171810/182838 93.97%
2022/07/07 17:53:16 170560/180939 94.26%
2022/07/08 17:53:16 169999/180116 94.38%
2022/07/09 17:53:16 170241/179759 94.71%
2022/07/10 17:53:16 167606/176432 95.00%
2022/07/11 17:53:16 168153/178160 94.38%
2022/07/12 17:53:16 170696/180506 94.57%
2022/07/13 17:53:16 169343/179592 94.29%
2022/07/14 17:53:16 171230/181636 94.27%
2022/07/15 17:53:16 184074/195135 94.33%
2022/07/16 17:53:16 192010/202251 94.94%
2022/07/17 17:53:16 196606/206267 95.32%
2022/07/18 17:53:16 200790/212053 94.69%
2022/07/19 17:53:16 197761/209221 94.52%
2022/07/20 17:53:16 196473/207642 94.62%
2022/07/21 17:53:16 204488/215694 94.80%
2022/07/22 17:53:16 220073/232393 94.70%
2022/07/23 17:53:16 219360/230541 95.15%
2022/07/24 17:53:16 213575/224549 95.11%
2022/07/25 17:53:16 205768/219132 93.90%
2022/07/26 17:53:16 211755/227503 93.08%
2022/07/27 17:53:16 215612/229109 94.11%
2022/07/28 17:53:16 224365/238228 94.18%
2022/07/29 17:53:16 222637/237054 93.92%
2022/07/30 17:53:16 216264/228013 94.85%
2022/07/31 17:53:16 220990/233336 94.71%
2022/08/01 17:53:16 221660/237294 93.41%
2022/08/02 17:53:16 216055/234745 92.04%
2022/08/03 17:53:16 222963/244030 91.37%
2022/08/04 17:53:16 224417/245564 91.39%
2022/08/05 17:53:16 228494/247388 92.36%
2022/08/06 17:53:16 223448/238546 93.67%
2022/08/07 17:53:16 233452/248800 93.83%
2022/08/08 17:53:16 228015/244361 93.31%
2022/08/09 17:53:16 226362/242047 93.52%
2022/08/10 17:53:16 224323/240293 93.35%
2022/08/11 17:53:16 223786/239403 93.48%
2022/08/12 17:53:16 229123/245179 93.45%
2022/08/13 17:53:16 228316/242586 94.12%
2022/08/14 17:53:16 226738/241042 94.07%
2022/08/15 17:53:16 228205/243680 93.65%
2022/08/16 17:53:16 229760/246054 93.38%
2022/08/17 17:53:16 227155/243094 93.44%
2022/08/18 17:53:16 227517/243172 93.56%
2022/08/19 17:53:16 227941/243997 93.42%
2022/08/20 17:53:16 228990/243900 93.89%
2022/08/21 17:53:16 230668/245426 93.99%
2022/08/22 17:53:16 224561/240036 93.55%
2022/08/23 17:53:16 219832/235655 93.29%
2022/08/24 17:53:16 222124/238295 93.21%
2022/08/25 17:53:16 223891/240074 93.26%
2022/08/26 17:53:16 223819/240527 93.05%
2022/08/27 17:53:16 228663/243568 93.88%
2022/08/28 17:53:16 225569/239817 94.06%
2022/08/29 17:53:16 224327/240134 93.42%
2022/08/30 17:53:16 228673/245654 93.09%
2022/08/31 17:53:16 226735/243293 93.19%
2022/09/01 17:53:16 225639/241630 93.38%
2022/09/02 17:53:16 236206/252896 93.40%
2022/09/03 17:53:16 233258/248503 93.87%
2022/09/04 17:53:16 232155/247386 93.84%
2022/09/05 17:53:16 234814/251423 93.39%
2022/09/06 17:53:16 232808/249994 93.13%
2022/09/07 17:53:16 240543/257957 93.25%
2022/09/08 17:53:16 231430/248035 93.31%
2022/09/09 17:53:16 242153/259413 93.35%
2022/09/10 17:53:16 243891/260512 93.62%
2022/09/11 17:53:16 248028/265232 93.51%
2022/09/12 17:53:16 242159/260327 93.02%
2022/09/13 17:53:16 239257/257526 92.91%
2022/09/14 17:53:16 239152/257146 93.00%
2022/09/15 17:53:16 231770/249919 92.74%
2022/09/16 17:53:16 234916/254471 92.32%
2022/09/17 17:53:16 226918/245014 92.61%
2022/09/18 17:53:16 227741/245597 92.73%
2022/09/19 17:53:16 216576/233794 92.64%
2022/09/20 17:53:16 216902/234266 92.59%
2022/09/21 17:53:16 221773/239192 92.72%
2022/09/25 21:34:29 1121561/1260527 88.98%
2022/09/28 02:02:49 1381811/1532420 90.17%
2022/10/03 10:07:21 3700252/3968879 93.23%
2022/10/04 10:07:21 2701111/3187521 84.74%
2022/10/06 01:10:13 878980/1149771 76.45%
2022/10/07 19:51:05 687493/936395 73.42%
2022/10/08 19:51:05 557700/769207 72.50%
2022/10/09 19:51:05 652399/856848 76.14%
2022/10/10 19:51:05 645157/862235 74.82%
2022/10/11 19:51:05 665194/882964 75.34%
2022/10/12 19:51:05 452181/651005 69.46%
2022/10/13 19:51:05 615747/844214 72.94%
2022/10/14 19:51:05 543988/760253 71.55%
2022/10/15 19:51:05 419633/597319 70.25%
2022/10/16 19:51:05 532650/731640 72.80%
2022/10/17 19:51:05 597205/786802 75.90%
2022/10/18 19:51:05 545984/715618 76.30%
2022/10/20 07:31:25 574485/676737 84.89%
2022/10/21 07:31:25 554463/675893 82.03%
2022/10/22 07:31:25 518909/671584 77.27%
2022/10/23 07:31:25 541626/691770 78.30%
2022/10/24 07:31:25 538160/689136 78.09%
2022/10/25 07:31:25 510459/656238 77.79%
2022/10/27 07:00:29 518697/614072 84.47%
2022/10/28 07:00:29 486259/608324 79.93%
2022/10/29 07:00:29 491426/630300 77.97%
2022/10/30 07:00:29 472230/597290 79.06%
2022/10/31 07:00:29 491910/624836 78.73%
2022/11/01 07:00:29 475916/639245 74.45%
2022/11/02 07:00:29 481206/646550 74.43%
2022/11/03 07:00:29 599719/776774 77.21%
2022/11/04 07:00:29 737956/925233 79.76%
2022/11/05 07:00:29 873922/1093662 79.91%
2022/11/06 07:00:29 879774/1077541 81.65%
2022/11/07 07:00:29 905093/1113103 81.31%
2022/11/08 07:00:29 935191/1153421 81.08%
2022/11/09 07:00:29 1074279/1308056 82.13%
2022/11/10 07:00:29 966623/1181925 81.78%
2022/11/11 07:00:29 1031706/1266995 81.43%
2022/11/12 07:00:29 1069843/1285152 83.25%
2022/11/13 07:00:29 999539/1248679 80.05%
2022/11/14 07:00:29 988052/1222193 80.84%
2022/11/15 07:00:29 951821/1191297 79.90%
2022/11/16 07:00:29 1110988/1329583 83.56%
2022/11/17 07:00:29 1117195/1324894 84.32%
2022/11/18 07:00:29 1173611/1379627 85.07%
2022/11/19 07:00:29 1205208/1414808 85.19%
2022/11/20 07:00:29 1189146/1398402 85.04%
2022/11/21 07:00:29 1186982/1495221 79.39%
2022/11/22 07:00:29 1081288/1385520 78.04%
2022/11/23 07:00:29 1244621/1587082 78.42%
2022/11/24 07:00:29 1341024/1674905 80.07%
2022/11/25 07:00:29 1421500/1757062 80.90%
2022/11/26 07:00:29 1560783/1904455 81.95%
2022/11/27 07:00:29 1465706/1671865 87.67%
2022/11/28 07:00:29 1429704/1643762 86.98%
2022/11/29 07:00:29 1457300/1700892 85.68%
2022/11/30 07:00:29 1487931/1720901 86.46%
2022/12/01 07:00:29 1506246/1740213 86.56%
2022/12/02 07:00:29 1460951/1683721 86.77%
2022/12/03 07:00:29 1485571/1703312 87.22%
2022/12/04 07:00:29 1438832/1627932 88.38%
2022/12/05 07:00:29 1323146/1522810 86.89%
2022/12/06 07:00:29 1382024/1609148 85.89%
2022/12/07 07:00:29 1442358/1645869 87.64%
2022/12/08 07:00:29 1736377/1930441 89.95%
2022/12/09 07:00:29 1870385/2047066 91.37%
2022/12/10 07:00:29 1807694/1985861 91.03%
2022/12/11 07:00:29 1798714/1958147 91.86%
2022/12/12 07:00:29 1788181/1951137 91.65%
2022/12/13 07:00:29 1831817/2011966 91.05%
2022/12/14 07:00:29 1963352/2149952 91.32%
2022/12/15 07:00:29 1957450/2146125 91.21%
2022/12/16 07:00:29 2008079/2193701 91.54%
2022/12/17 07:00:29 2405453/2583256 93.12%
2022/12/18 07:00:29 2652740/2823784 93.94%
2022/12/19 07:00:29 2464515/2634705 93.54%
2022/12/20 07:00:29 2589562/2776041 93.28%
2022/12/21 07:00:29 2556656/2727805 93.73%
2022/12/22 07:00:29 2621775/2790903 93.94%
2022/12/23 07:00:29 2612666/2786059 93.78%
2022/12/24 07:00:29 2702727/2863852 94.37%
2022/12/25 07:00:29 2699413/2825448 95.54%
2022/12/26 07:00:29 2695487/2811637 95.87%
2022/12/27 07:00:29 2716205/2840448 95.63%
2022/12/28 23:36:14 2914171/3045553 95.69%
2022/12/29 23:36:14 2898755/3043905 95.23%
2022/12/30 23:36:14 2929119/3093004 94.70%
2022/12/31 23:36:14 2963652/3115697 95.12%
2023/01/01 23:36:14 3019037/3183507 94.83%
2023/01/02 23:36:14 2980591/3177055 93.82%
2023/01/03 23:36:14 3001436/3207975 93.56%
2023/01/05 13:39:26 3078681/3250828 94.70%
2023/01/06 13:39:26 3012651/3191445 94.40%
2023/01/07 13:39:26 3039471/3206161 94.80%
2023/01/08 13:39:26 2982919/3147639 94.77%
2023/01/09 13:39:26 2998712/3176934 94.39%
2023/01/10 13:39:26 2994631/3180844 94.15%
2023/01/13 02:12:23 3159307/3318531 95.20%
2023/01/14 02:12:23 3021483/3193402 94.62%
2023/01/15 02:12:23 3010165/3175348 94.80%
2023/01/16 02:12:23 3010367/3178627 94.71%
2023/01/17 02:12:23 2876650/3059836 94.01%
2023/01/18 02:12:23 2676927/2846637 94.04%
2023/01/19 02:12:23 2689500/2840457 94.69%
2023/01/20 02:12:23 2593768/2734524 94.85%
2023/01/21 02:12:23 2625426/2779702 94.45%
2023/01/22 02:12:23 2730913/2874371 95.01%
2023/01/23 02:12:23 2713914/2856540 95.01%
2023/01/24 02:12:23 2712147/2862818 94.74%
2023/01/25 02:12:23 2764751/2916888 94.78%
2023/01/26 02:12:23 2823173/2977070 94.83%
2023/01/27 02:12:23 2803899/2963868 94.60%
2023/01/28 02:12:23 2752513/2911332 94.54%
2023/01/29 02:12:23 2856561/3005275 95.05%
2023/01/30 02:12:23 2885433/3041352 94.87%
2023/01/31 02:12:23 2861213/3032729 94.34%
2023/02/01 02:12:23 2874864/3039998 94.57%
2023/02/02 02:12:23 2889754/3049058 94.78%
2023/02/03 02:12:23 2914695/3062367 95.18%
2023/02/04 02:12:23 2847896/2998863 94.97%
2023/02/05 02:12:23 2811298/2946670 95.41%
2023/02/06 02:12:23 2761140/2902851 95.12%
2023/02/07 02:12:23 2737250/2890104 94.71%
2023/02/08 02:12:23 2724894/2874121 94.81%
2023/02/09 02:12:23 2755421/2887012 95.44%
2023/02/10 02:12:23 2707386/2840938 95.30%
2023/02/11 02:12:23 2718711/2849491 95.41%
2023/02/12 02:12:23 2605462/2724865 95.62%
2023/02/13 02:12:23 2625173/2756602 95.23%
2023/02/14 02:12:23 2580373/2714888 95.05%
2023/02/15 02:12:23 2603552/2734293 95.22%
2023/02/16 02:12:23 2597875/2736427 94.94%
2023/02/17 02:12:23 2603329/2731863 95.30%
2023/02/18 03:06:58 2783076/2909641 95.65%
2023/02/19 03:06:58 2727879/2848886 95.75%
2023/02/20 03:06:58 2560371/2677627 95.62%
2023/02/21 03:06:58 2580350/2712691 95.12%
2023/02/22 03:06:58 2388451/2546924 93.78%
2023/02/23 03:06:58 2083081/2257625 92.27%
2023/02/24 03:06:58 2146725/2330121 92.13%
2023/02/25 03:06:58 2202918/2391555 92.11%
2023/02/26 03:06:58 2164685/2335185 92.70%
2023/02/27 03:06:58 2136689/2326754 91.83%
2023/02/28 03:06:58 2093819/2302734 90.93%
2023/03/01 03:06:58 2079724/2292969 90.70%
2023/03/02 03:06:58 2024533/2237584 90.48%
2023/03/03 03:06:58 1969091/2164831 90.96%
2023/03/04 03:06:58 1944731/2111440 92.10%
2023/03/05 03:06:58 1911220/2063561 92.62%
2023/03/06 03:06:58 1837312/1997782 91.97%
2023/03/07 03:06:58 1726116/1894914 91.09%
2023/03/08 03:06:58 1739224/1911262 91.00%
2023/03/09 03:06:58 1753215/1913511 91.62%
2023/03/10 03:06:58 1789072/1970817 90.78%
2023/03/11 03:06:58 1761480/1941561 90.72%
2023/03/12 03:06:58 1757541/1924281 91.33%
2023/03/13 03:06:58 1696244/1868432 90.78%
2023/03/14 20:17:54 752394/834628 90.15%
2023/03/15 20:17:54 756717/841520 89.92%
2023/03/16 20:17:54 758070/841276 90.11%
2023/03/17 20:17:54 752392/831973 90.43%
2023/03/18 20:17:54 763324/838603 91.02%
2023/03/19 20:17:54 755424/832243 90.77%
2023/03/20 20:17:54 757856/835657 90.69%
2023/03/21 20:17:54 858242/948627 90.47%
2023/03/23 03:25:05 866883/961001 90.21%
2023/03/24 03:25:05 854870/950286 89.96%
2023/03/25 03:25:05 857745/946211 90.65%
2023/03/26 03:25:05 876268/964114 90.89%
2023/03/27 03:25:05 892848/983992 90.74%
2023/03/28 03:25:05 872174/976084 89.35%
2023/03/29 03:25:05 871535/971938 89.67%
```
</details>
Note that this is not the same as ["no address in clientID-to-IP map"](tpo/anti-censorship/pluggable-transports/snowflake#40084):
that warning is logged when a `client_ip` was present,
but the server could not cache it long enough to associate it with a session.
This is different: `client_ip` is just not present.David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/lox/-/issues/3Improve Lox integration with rdsys2024-02-15T17:38:19ZonyinyangImprove Lox integration with rdsysThe current Lox distributor parses and handles resources from rdsys in a very naive way that does not match with the expected distributor behaviour. Currently, Lox will continue adding all new resources to the Lox database, assuming they...The current Lox distributor parses and handles resources from rdsys in a very naive way that does not match with the expected distributor behaviour. Currently, Lox will continue adding all new resources to the Lox database, assuming they are in fact `new`. In rdsys' implementation, all bridges in the database are `new resources` and are re-sent to distributors at regular intervals to ensure the bridge distributor's database is synced. Since Lox sorts bridges into buckets that are meant to persist until the bridges are blocked, syncing the Lox bridgetable with rdsys' `new resources` will require some care.
This consists of 2 major subtasks.
1. Syncing the Lox bridgetable with rdsys (being tracked in #8)
2. Sorting `new` resources into buckets in a reasonable way (a later issue)onyinyangonyinyanghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/webtunnel/-/issues/18Create a container image for webtunnel2023-04-19T14:33:45Zmeskiomeskio@torproject.orgCreate a container image for webtunnelWebtunnels bridge operators might want to be able to use a container to run their bridge.
We should produce some documentation on how to run the container.Webtunnels bridge operators might want to be able to use a container to run their bridge.
We should produce some documentation on how to run the container.Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetshelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/40066BridgeDB requires outdated packages with known CVEs2023-10-12T14:24:01Zmeskiomeskio@torproject.orgBridgeDB requires outdated packages with known CVEsSelected vulnerable packages:
* Twisted 21.7.0
* Mechanize 0.4.5
* Pillow 8.2.0
* Werkzeug 2.2.2Selected vulnerable packages:
* Twisted 21.7.0
* Mechanize 0.4.5
* Pillow 8.2.0
* Werkzeug 2.2.2meskiomeskio@torproject.orgmeskiomeskio@torproject.org2023-05-31https://gitlab.torproject.org/tpo/anti-censorship/lox/-/issues/10Populate wiki with documentation2023-10-03T18:38:55ZCecylia BocovichPopulate wiki with documentationLet's use this overview project as a way to aggregate issues and documentation since Lox is made of many different pieces. A good start would be:
- There's some high level docs written up at https://gitlab.torproject.org/cohosh/lox/-/wi...Let's use this overview project as a way to aggregate issues and documentation since Lox is made of many different pieces. A good start would be:
- There's some high level docs written up at https://gitlab.torproject.org/cohosh/lox/-/wikis/Lox-Overview that should be moved here and also checked to see if they are accurate
- @onyinyang made some cool graphics at https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/116#note_2884107Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetonyinyangonyinyanghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40267Improve bug discovery process2023-06-20T20:59:52ZitchyonionImprove bug discovery processCreating the issue as a follow up to the meeting on March 16th, 2023 about **snowflake-server buffer reuse bug postmortem https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40260**
(The title of th...Creating the issue as a follow up to the meeting on March 16th, 2023 about **snowflake-server buffer reuse bug postmortem https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40260**
(The title of this ticket could be improved as well. Feel free to do so)
> The harm to users was minor, but incidents like this are a good opportunity to reflect on our process, to make similar things less likely in the future.
>
> The bug (#40199) might have been caught, but was not, at multiple points:
> - Code understanding and review by the initial committer
> - Code review on the merge request
> - Automated tests / CI
> - End user reports or logs
> - Logs or instrumentation at the bridge
>
> **Which of these processes, if any, should we change, to decrease the chance of mistakes?**
>
> **Brainstorming during the meeting:**
>
> - Initial merge request should have included a test to prove the assumption that buffers were not reused.
> - The reviewer might have requested that such a test be added.
> - Any such anomalies, if detected at the client, should be logged in such a way that they show up in the tor log.
> - dcf's private branch that logs KCP's internal error counters:
> - https://gitlab.torproject.org/dcf/snowflake/-/commit/9f43843b59b9753686be836f2c55f209ba29c1e9
> - https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40262#note_2886018
> - The fix this week made the "KCPInErrors" counter go to zero:
> - https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40262#note_2886032
> - We should log whenever KCPInErrors is non-zero, at least.
> - We are missing integration testing as part of CI. We have unit testing, but nothing where all the pieces are working together as in production.
> - shelikhoo's setup for distributed snowflake server testing: https://github.com/xiaokangwang/snowflake-mu-docker/blob/master/docker-compose.yaml
> - Should we have another more verbose level of log (debug/trace) so that it takes less effort to debug things in general? (no need to modify code then rebuilt like hazae41 did it https://hackerone.com/reports/1880610)https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/webtunnel/-/issues/17improve descriptor arguments published2023-10-24T08:17:20Zmeskiomeskio@torproject.orgimprove descriptor arguments publishedThis is the extra-info line I see for the bridge documented in the README:
```
transport webtunnel 127.0.0.1:11000 domainname=n97jlbvjwy3iotetc0axrtu3mm7gq1gajqv7-80-78-24-44.sslip.io,path=939e42a2-8f3f-4645-9d7b-8b9be28b157e,port=443,pu...This is the extra-info line I see for the bridge documented in the README:
```
transport webtunnel 127.0.0.1:11000 domainname=n97jlbvjwy3iotetc0axrtu3mm7gq1gajqv7-80-78-24-44.sslip.io,path=939e42a2-8f3f-4645-9d7b-8b9be28b157e,port=443,publicVersion=0.0.1
```
Currently webtunnel is publishing the localhost IP and port where is listening for connections from the http proxy, we might want to publish a different IP address and port there as they will be used to build the bridgeline. The README uses 192.0.3.2:1, but AFAIK we can't use the same IP-port combination for several bridges as tor will consider is the same bridge and ignore the others. We could generate a random IP address from a private range, but maybe is better to put there the public IP address of the domain we are using as front. Any ideas?
I see in the extra-info line `domainname` and `path`, where AFAIK the client expects `url`. I think this is not coming from the code itself, but from `ServerTransportOptions` in torrc. Isn't it? Maybe we can improve the README to document how that is configured to include the `url` option (and fix our public tunnel).
I have some doubts about `publicVersion`, I think is weird to keep that in the bridgeline. rdsys could strip it out, but is pretty hacky. I would love to see https://gitlab.torproject.org/tpo/core/tor/-/issues/11101 solving that problem instead andSponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetshelikhooshelikhoohttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake-webext/-/issues/81Snowflake is Off / WebRTC feature is not detected.2023-04-23T10:27:21ZcypherpunksSnowflake is Off / WebRTC feature is not detected.I've just installed Snowflake via Chrome and it's not working. Can you confirm the process has been followed correctly? What have I done wrong - or not done at all? Thanks.I've just installed Snowflake via Chrome and it's not working. Can you confirm the process has been followed correctly? What have I done wrong - or not done at all? Thanks.https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/122Update x/net and x/crypto to latest version in Obfs4proxy and Snowflake2023-04-20T08:36:45ZtlaUpdate x/net and x/crypto to latest version in Obfs4proxy and SnowflakeSomebody sent me an issue about this:
https://github.com/tladesignz/IPtProxy/issues/45
> For security purposes, golang.org/x/crypto and golang.org/x/net should be updated to latest versions. This also requires the obfs4 and snowflake s...Somebody sent me an issue about this:
https://github.com/tladesignz/IPtProxy/issues/45
> For security purposes, golang.org/x/crypto and golang.org/x/net should be updated to latest versions. This also requires the obfs4 and snowflake submodules to do the same thing, so there needs to be some coordination between all three projects for this to happen.
Please let me know, if this is valid, and if there's any problems with that request! Thank you!meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/121migrate our images outside docker hub2024-03-04T09:50:16Zmeskiomeskio@torproject.orgmigrate our images outside docker hubWe got an email saying that April 14th our [thetorproject](https://hub.docker.com/u/thetorproject) organization will be close. There is a [FAQ](https://web.docker.com/rs/790-SSB-375/images/privatereposfaq.pdf) hidden in their website wit...We got an email saying that April 14th our [thetorproject](https://hub.docker.com/u/thetorproject) organization will be close. There is a [FAQ](https://web.docker.com/rs/790-SSB-375/images/privatereposfaq.pdf) hidden in their website with some information about it.
If we don't do anything our images will deleted April 14th, making our 900k+ unable to update them or to install them. But docker promises that *your organization’s namespace will not be released, so other users can’t “squat” your images*.
We can ask to be included in the [Docker-Sponsored Open Source Program](https://www.docker.com/community/open-source/application/), in that case they will *defer any organization suspension or deletion while DSOS application is under review, and give organizations at least 30 days before we suspend the organization if the application is ultimately rejected*. We should qualify for it, but there are many stories in internet of people being rejected from it.
We can move to another registry, there are few options:
* TPA plans to add a registry (https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/89)
* [quay.io](https://quay.io/)
* [github](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry)
* An individual docker hub account, will require changing the namemeskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40265Standalone proxy - error polling probe - 'certificate is not standards compli...2023-03-16T17:40:24ZMarkCStandalone proxy - error polling probe - 'certificate is not standards compliant'@meskio As requested.
Yesterday I updated one of my proxies after !136 completed. When I run the proxy it starts, then begins probetest, then immediately (and repeatedly) returns an error:
2023/03/16 00:45:47 error polling probe: x509:...@meskio As requested.
Yesterday I updated one of my proxies after !136 completed. When I run the proxy it starts, then begins probetest, then immediately (and repeatedly) returns an error:
2023/03/16 00:45:47 error polling probe: x509: “snowflake-broker.torproject.net” certificate is not standards compliant
2023/03/16 00:45:47 NAT type: unknown
2023/03/16 00:45:48 error polling broker: x509: “snowflake-broker.torproject.net” certificate is not standards compliant
2023/03/16 00:45:48 Error reading broker response: unexpected end of JSON input
2023/03/16 00:45:48 body:
2023/03/16 00:45:48 bad offer from broker
Here’s the log of the start up (note that the git pull in this sequence says 'already up to date' because I ran it a second time as a double check).
[proxy_error_polling_broker_-_03_15_23.txt](/uploads/e8d343dc26e784d66e59880530374774/proxy_error_polling_broker_-_03_15_23.txt)
On the previous build everything was functioning properly.
@itchyonion Might this result be caused by the bug you mentioned in #40108 yesterday?itchyonionitchyonionhttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40264Add option to configure private snowflake proxy2023-03-28T18:36:08ZRendezvousPointAdd option to configure private snowflake proxyJust like obfs4 private bridges.Just like obfs4 private bridges.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40263proxy: option to set IPv4 and IPv6 bind addresses2023-03-28T18:36:39Zbennyproxy: option to set IPv4 and IPv6 bind addressesfeature request:
On a system with multiple IPv4 & IPv6 addresses and services it would be very helpful to set one (or more) IPv4 and one (or more) IPv6 address for the tcp/udp sockets by proxy.
Different to --outbound-address it should...feature request:
On a system with multiple IPv4 & IPv6 addresses and services it would be very helpful to set one (or more) IPv4 and one (or more) IPv6 address for the tcp/udp sockets by proxy.
Different to --outbound-address it should not be a priority - it should be a fixed IP assignment.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40262Deploy snowflake-server for QueuePacketConn buffer reuse fix (#40260)2023-07-18T00:47:30ZDavid Fifielddcf@torproject.orgDeploy snowflake-server for QueuePacketConn buffer reuse fix (#40260)#40260 describes a buffer reuse error in `QueuePacketConn` resulting from #40199.
!140 promises to fix it.
After merging !140, deploy to the bridges and restart.
- [x] snowflake-01
- [x] snowflake-02
/cc @cohosh @linus#40260 describes a buffer reuse error in `QueuePacketConn` resulting from #40199.
!140 promises to fix it.
After merging !140, deploy to the bridges and restart.
- [x] snowflake-01
- [x] snowflake-02
/cc @cohosh @linusDavid Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40261Standalone proxy: use standard notation of "inbound" and "outbound"2023-06-28T09:21:33ZitchyonionStandalone proxy: use standard notation of "inbound" and "outbound"Right now, the standalone proxy increment the "outbound" count when it [receives messages from SF client](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/main/proxy/lib/snowflake.go#L414) and incre...Right now, the standalone proxy increment the "outbound" count when it [receives messages from SF client](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/main/proxy/lib/snowflake.go#L414) and increment the "inbound" count when it [writes to the relay](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/main/proxy/lib/snowflake.go#L296). It will be better to use the standard "inbound" and "outbound" notation to include everything regardless of traffic logic because:
1. proxy runners might have a limit of inbound/outbound traffic imposed by their ISP or hosting service provider, and they will be using the standard way to count
2. it's less confusing and more consistent with everything else
3. traffic logic shouldn't matter to proxy runnershttps://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/156Missing Authentication at Resource Registration2023-10-12T14:26:44Zmeskiomeskio@torproject.orgMissing Authentication at Resource RegistrationWhile auditing the source code of the Rdsys, it is noticed that the Rdsys backend has missing authentication for the resource registration endpoint. This allows an adversary to register arbitrary malicious resources which will be distrib...While auditing the source code of the Rdsys, it is noticed that the Rdsys backend has missing authentication for the resource registration endpoint. This allows an adversary to register arbitrary malicious resources which will be distributed to the users.
The following snippet of code shows the affected code where the resource registration endpoint does not have any kind of authentication checks as compared to the other endpoints.
Affected file:
rdsys/internal/backend.go
Affected code:
```go
func (b *BackendContext) resourcesHandler(w http.ResponseWriter, r *http.Request) {
switch r.Method {
[...]
case http.MethodPost:
if r.URL.Path == b.Config.Backend.ResourcesEndpoint {
b.postResourcesHandler(w, r)
[...]
}
func (b *BackendContext) postResourcesHandler(w http.ResponseWriter, req *http.Request) {
body, err := ioutil.ReadAll(req.Body)
[...]
rTypes := map[string]struct{}{}
for _, r := range rs {
b.Resources.Add(r)
rTypes[r.Type()] = struct{}{}
log.Printf("Added %s's %q resource to collection.", req.RemoteAddr, r.Type())
}
for rType := range rTypes {
b.rStore.Save(rType)
}
[...]
}
```
The aforementioned issue can be reproduced by executing the following cURL request.
PoC:
```
curl http://localhost:7100/resources -i --data '[{"type": "obfs2", "address": "1.2.3.2", "port": 1235, "fingerprint":"10282810115283F99ADE5CFE42D49644F45D715D"}]' -XPOST
# Other endpoints which are missing authentication
curl -i -s -k -X $'GET' \
-H $'Host: localhost:7100' \
$'http://localhost:7100/status?id=10282810115283F99ADE5CFE42D49644F45D715D'
curl -i -s -k -X $'GET' \
-H $'Host: localhost:7100' \
$'http://localhost:7100/rdsys-backend-metrics'
```
To address the issue, it is strongly advised to implement robust authentication mechanisms for all endpoints, with particular attention paid to the registration of resources. This will help to ensure that only authorized users are able to register resources, thereby reducing the risk of unauthorized access.meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40260Weird KCP packets received by the client2023-09-25T19:36:14ZCecylia BocovichWeird KCP packets received by the clientWe were notified by @hazae41 of some suspicious looking kcp traffic traffic arriving at the client. This traffic looks like it belongs to KCP sessions that do not correspond to the client's own KCP sessions.
I'm in the process of trying...We were notified by @hazae41 of some suspicious looking kcp traffic traffic arriving at the client. This traffic looks like it belongs to KCP sessions that do not correspond to the client's own KCP sessions.
I'm in the process of trying to reproduce this. Here is the patch they used to see the contents of the KCP packets:
<details>
<summary> kcp library logging patch </summary>
```diff
diff --git a/readloop.go b/readloop.go
index 697395a..1d8e17b 100644
--- a/readloop.go
+++ b/readloop.go
@@ -1,6 +1,7 @@
package kcp
import (
+ "log"
"sync/atomic"
"github.com/pkg/errors"
@@ -11,6 +12,7 @@ func (s *UDPSession) defaultReadLoop() {
var src string
for {
if n, addr, err := s.conn.ReadFrom(buf); err == nil {
+ log.Println("kcp <- ", buf)
// make sure the packet is from the same source
if src == "" { // set source address
src = addr.String()
```
</details>
and the following smux patch:
<details>
<summary> smux library logging patch </summary>
```diff
diff --git a/session.go b/session.go
index bc56066..1fe1122 100644
--- a/session.go
+++ b/session.go
@@ -96,7 +96,7 @@ func newSession(config *Config, conn io.ReadWriteCloser, client bool) *Session {
s.chProtoError = make(chan struct{})
if client {
- s.nextStreamID = 1
+ s.nextStreamID = 5
} else {
s.nextStreamID = 0
}
```
</details>
They report seeing smux headers that have a streamid of 3 regardless of the above change that starts stream ids at 5, and seeing kcp headers that do not contain recognizable kcp conv ids followed by a stream id of 3 and a TLS application packet (despite the fact that new KCP sessions should always contain a TLS handshake first since they represent a new OR conn).Cecylia BocovichCecylia Bocovich