Sudden reduction in snowflake-01 bridge bandwidth, 2022-10-04 17:15

At around 2022-10-04 17:15 (about 2.75 hours ago), bandwidth on the bridge suddenly dropped from about 3.2/2.4 Gbps TX/RX to about 1.0/0.8 Gbps TX/RX.

There is normally a nightly drop in usage, but this drop is several hours too early for that. It's also unusually flat.

/cc @linus

assigned to @dcf

I think I found the cause. The snowflake-server process was running out of file descriptors. I am not sure why that should have caused such a drastic reduction in throughput, but the log messages around the time of the drop are clear:

2022/10/04 17:13:59 http: Accept error: accept tcp [scrubbed]: accept4: too many open files; retrying in 5ms
2022/10/04 17:13:59 http: Accept error: accept tcp [scrubbed]: accept4: too many open files; retrying in 5ms
2022/10/04 17:13:59 http: Accept error: accept tcp [scrubbed]: accept4: too many open files; retrying in 10ms
2022/10/04 17:13:59 handleConn: failed to connect to ORPort: dial tcp [scrubbed]->[scrubbed]: socket: too many open files
2022/10/04 17:13:59 http: Accept error: accept tcp [scrubbed]: accept4: too many open files; retrying in 5ms
2022/10/04 17:13:59 http: Accept error: accept tcp [scrubbed]: accept4: too many open files; retrying in 5ms
2022/10/04 17:13:59 http: Accept error: accept tcp [scrubbed]: accept4: too many open files; retrying in 10ms
2022/10/04 17:13:59 handleConn: failed to connect to ORPort: error reading TOR_PT_AUTH_COOKIE_FILE "/etc/extor-static-cookie/static_extended_orport_auth_cookie": open /etc/extor-static-cookie/static_extended_orport_auth_cookie: too many open files
2022/10/04 17:13:59 http: Accept error: accept tcp [scrubbed]: accept4: too many open files; retrying in 5ms
2022/10/04 17:13:59 http: Accept error: accept tcp [scrubbed]: accept4: too many open files; retrying in 5ms
2022/10/04 17:13:59 http: Accept error: accept tcp [scrubbed]: accept4: too many open files; retrying in 10ms
2022/10/04 17:13:59 handleConn: failed to connect to ORPort: error reading TOR_PT_AUTH_COOKIE_FILE "/etc/extor-static-cookie/static_extended_orport_auth_cookie": open /etc/extor-static-cookie/static_extended_orport_auth_cookie: too many open files
2022/10/04 17:13:59 http: Accept error: accept tcp [scrubbed]: accept4: too many open files; retrying in 5ms
2022/10/04 17:13:59 http: Accept error: accept tcp [scrubbed]: accept4: too many open files; retrying in 5ms
2022/10/04 17:13:59 handleConn: failed to connect to ORPort: error reading TOR_PT_AUTH_COOKIE_FILE "/etc/extor-static-cookie/static_extended_orport_auth_cookie": open /etc/extor-static-cookie/static_extended_orport_auth_cookie: too many open files
2022/10/04 17:13:59 handleConn: failed to connect to ORPort: error reading TOR_PT_AUTH_COOKIE_FILE "/etc/extor-static-cookie/static_extended_orport_auth_cookie": open /etc/extor-static-cookie/static_extended_orport_auth_cookie: too many open files
2022/10/04 17:14:01 http: TLS handshake error from [scrubbed]: EOF
2022/10/04 17:14:03 http: TLS handshake error from [scrubbed]: read tcp [scrubbed]->[scrubbed]: read: connection reset by peer
2022/10/04 17:14:04 http: TLS handshake error from [scrubbed]: read tcp [scrubbed]->[scrubbed]: i/o timeout
2022/10/04 17:14:07 reading token: websocket: close 1006 (abnormal closure): unexpected EOF
2022/10/04 17:14:13 error copying ORPort to WebSocket EOF
2022/10/04 17:14:13 reading token: websocket: close 1001 (going away)
2022/10/04 17:14:15 reading token: websocket: close 1006 (abnormal closure): unexpected EOF
2022/10/04 17:14:16 http: TLS handshake error from [scrubbed]: EOF
2022/10/04 17:14:16 reading token: websocket: close 1006 (abnormal closure): unexpected EOF
2022/10/04 17:14:18 http: TLS handshake error from [scrubbed]: read tcp [scrubbed]->[scrubbed]: i/o timeout
2022/10/04 17:14:20 reading token: read tcp [scrubbed]->[scrubbed]: read: connection reset by peer
2022/10/04 17:14:23 no address in clientID-to-IP map (capacity 98304)
2022/10/04 17:14:24 http: TLS handshake error from [scrubbed]: read tcp [scrubbed]->[scrubbed]: i/o timeout
2022/10/04 17:14:25 reading token: websocket: close 1006 (abnormal closure): unexpected EOF
2022/10/04 17:14:25 http: TLS handshake error from [scrubbed]: read tcp [scrubbed]->[scrubbed]: i/o timeout
2022/10/04 17:14:28 http: TLS handshake error from [scrubbed]: read tcp [scrubbed]->[scrubbed]: read: connection reset by peer
2022/10/04 17:14:30 http: TLS handshake error from [scrubbed]: EOF
2022/10/04 17:14:31 reading token: websocket: close 1001 (going away)
2022/10/04 17:14:31 no address in clientID-to-IP map (capacity 98304)
2022/10/04 17:14:32 reading token: websocket: close 1001 (going away)
2022/10/04 17:14:35 reading token: websocket: close 1006 (abnormal closure): unexpected EOF
2022/10/04 17:14:35 reading token: websocket: close 1006 (abnormal closure): unexpected EOF
2022/10/04 17:14:35 reading token: websocket: close 1006 (abnormal closure): unexpected EOF

I edited /etc/systemd/system/snowflake-server.service to increase LimitNOFILE from 131072 to 1310720 (factor of 10) and restarted at 2022-10-05 01:10:13. Now I see:

# cat /proc/49677/limits
Limit                     Soft Limit           Hard Limit           Units
...
Max open files            1048576              1048576              files
...

I guess the requested value gets rounded to a power of 2.

It looks like it's working again, though bandwidth is being slow to recover, so I will watch it for a while longer before closing the issue.

Previous instance of file descriptor exhaustion in snowflake-server: #40095 (comment 2772325).

mentioned in commit tpo/network-health/metrics/timeline@40303008

Traffic has at 10:20 still not recovered beyond 1.08 Gbps.

We are getting reports of snowflake failing to connect in Iran, might be related (or at least constitute some part of the drop): tpo/anti-censorship/team#96 (comment 2840481)

Or could be a consequence of the drop in bandwidth, rather than a cause.

Traffic has at 10:20 still not recovered beyond 1.08 Gbps.

You're right. There must be something else going on. And it may be beyond what we have control of on the host. I ran two speedtest downloads, one from Gothenburg, Sweden (ping 10 ms), and one from Dallas, USA (ping 136 ms). The one from Dallas was about 5% as fast (0.129 Gbps) as the one from Gothenburg (2.5 Gbps).

# curl --output /dev/null http://speedtest.tele2.net/10GB.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10.0G  100 10.0G    0     0   308M      0  0:00:33  0:00:33 --:--:--  318M

# curl --output /dev/null http://speedtest.dallas.linode.com/garbage.php?ckSize=1000
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1000M    0 1000M    0     0  16.1M      0 --:--:--  0:01:01 --:--:-- 21.2M

The speed to/from Dallas is the same in both directions, as I tested with some netcat transfers from a server located there.

I compared the two most recent sets of stats files published by the bridge, the division between which conveniently is within 15 minutes of the dropoff:

• 2022-10-04 17:01
• 2022-10-05 17:01

Below are the top 20 countries according to dirreq-v3-reqs on 2022-10-04, and how much they changed 24 hours later. If there's a pattern, it's that countries in or near Europe increased (de, gb, by, fr, nl), while other countries remained about the same (ru, cn, in) or decreased (ir, us, mu, tn).

country	2022-10-04 17:01:14	2022-10-05 17:01:14	increase/decrease
ir	52408	15680	−70%
??	12232	8208	−33%
us	10560	4480	−58%
ru	5232	5056	−3%
cn	656	600	−9%
mu	648	384	−41%
tn	472	200	−58%
de	408	624	+53%
ma	256	112	−56%
gb	208	272	+31%
eg	200	168	−16%
za	192	128	−33%
ng	184	112	−39%
by	168	200	+19%
zm	160	48	−70%
fr	152	272	+79%
nl	136	168	+24%
au	120	88	−27%
sd	120	80	−33%
in	112	104	−7%

There's a similar pattern in bridge-ips. Some countries in or near Europe increased (de, fr), but also, by the measure of bridge-ips, some countries in Europe did not increase very much (gb, by). Other countries remained about the same (ru, cn) or decreased (ir, us, mu, tn, eg).

country	2022-10-04 17:01:23	2022-10-05 17:01:23	increase/decrease
ir	93224	25824	−72%
us	20576	6552	−68%
ru	8008	7600	−5%
mu	1360	600	−56%
tn	896	328	−63%
cn	888	824	−7%
de	672	784	+17%
??	544	496	−9%
eg	440	232	−47%
ma	416	176	−58%
ng	344	160	−53%
za	344	200	−42%
gb	328	352	+7%
by	304	320	+5%
zm	296	80	−73%
fr	224	280	+25%
sd	216	104	−52%
in	192	208	+8%
ci	176	56	−68%
au	168	96	−43%

Source code and data

bridge-stats-end 2022-10-04 17:01:23 (86400 s)
bridge-ips ir=93224,us=20576,ru=8008,mu=1360,tn=896,cn=888,de=672,??=544,eg=440,ma=416,ng=344,za=344,gb=328,by=304,zm=296,fr=224,sd=216,in=192,ci=176,au=168,nl=168,ke=160,ug=152,ua=136,bf=120,ca=120,br=112,tr=88,it=80,jp=72,ae=64,es=64,sa=64,fi=56,mw=56,se=56,ga=48,kz=48,mx=48,pl=48,ro=48,at=40,ch=40,hk=40,id=40,ly=40,mg=40,be=32,cz=32,kr=32,pk=32,ap=24,co=24,dk=24,gh=24,il=24,my=24,nz=24,pt=24,sg=24,sl=24,af=16,am=16,bd=16,bw=16,cg=16,cl=16,cm=16,ee=16,gr=16,hu=16,ie=16,lt=16,lv=16,md=16,no=16,ph=16,re=16,sk=16,tg=16,th=16,tw=16,vn=16,zw=16,al=8,ao=8,ar=8,az=8,ba=8,bg=8,bh=8,bj=8,bo=8,bs=8,bz=8,cd=8,cr=8,cu=8,cv=8,cw=8,cy=8,do=8,dz=8,ec=8,eu=8,ge=8,gm=8,gn=8,gt=8,gw=8,hr=8,iq=8,is=8,jo=8,kg=8,kh=8,km=8,la=8,lb=8,lk=8,lu=8,ml=8,mm=8,mn=8,mo=8,mp=8,mt=8,mv=8,ne=8,ni=8,np=8,om=8,pe=8,pf=8,pg=8,pr=8,ps=8,py=8,qa=8,rs=8,rw=8,si=8,sn=8,so=8,ss=8,sv=8,sy=8,td=8,tj=8,tt=8,tz=8,uz=8,ve=8,vu=8,ye=8
bridge-ip-versions v4=126792,v6=5568
bridge-ip-transports <OR>=8,snowflake=132360

dirreq-stats-end 2022-10-04 17:01:14 (86400 s)
dirreq-v3-ips ir=38552,us=8072,ru=3824,mu=480,cn=456,??=384,tn=352,de=320,ma=176,eg=168,gb=160,za=144,by=136,ng=136,zm=136,fr=96,nl=88,sd=88,au=80,in=80,ca=72,ci=72,ua=64,ug=64,br=56,ke=56,bf=48,it=48,jp=40,tr=40,fi=32,ro=32,ae=24,ch=24,es=24,ga=24,kz=24,mg=24,mw=24,pl=24,sa=24,se=24,ap=16,be=16,cz=16,dk=16,ee=16,hk=16,id=16,il=16,kr=16,lt=16,lv=16,ly=16,mx=16,my=16,ph=16,pk=16,pt=16,sg=16,sl=16,th=16,af=8,al=8,am=8,ao=8,ar=8,at=8,az=8,ba=8,bd=8,bg=8,bh=8,bj=8,bo=8,bw=8,bz=8,cd=8,cg=8,cl=8,cm=8,co=8,cr=8,cu=8,cv=8,cy=8,do=8,dz=8,ec=8,eu=8,ge=8,gh=8,gn=8,gr=8,gt=8,hr=8,hu=8,ie=8,iq=8,kg=8,kh=8,km=8,la=8,lb=8,lk=8,lu=8,md=8,ml=8,mn=8,mp=8,mt=8,mv=8,no=8,nz=8,om=8,pe=8,ps=8,qa=8,re=8,rs=8,rw=8,si=8,sk=8,sn=8,so=8,td=8,tg=8,tj=8,tw=8,uz=8,vn=8,vu=8,zw=8
dirreq-v3-reqs ir=52408,??=12232,us=10560,ru=5232,cn=656,mu=648,tn=472,de=408,ma=256,gb=208,eg=200,za=192,ng=184,by=168,zm=160,fr=152,nl=136,au=120,sd=120,in=112,ca=96,ci=88,ug=88,ua=80,ke=72,br=64,fi=64,bf=56,it=56,jp=48,tr=48,ro=40,ch=32,es=32,ga=32,hk=32,kr=32,kz=32,pl=32,ae=24,id=24,lb=24,mg=24,mw=24,mx=24,sa=24,se=24,am=16,ap=16,ar=16,at=16,be=16,cl=16,co=16,cz=16,dk=16,dz=16,ee=16,gh=16,il=16,lt=16,lv=16,ly=16,my=16,no=16,nz=16,ph=16,pk=16,pt=16,re=16,sg=16,sl=16,th=16,tw=16,af=8,al=8,ao=8,az=8,ba=8,bd=8,bg=8,bh=8,bj=8,bo=8,bw=8,bz=8,cd=8,cg=8,cm=8,cr=8,cu=8,cv=8,cy=8,do=8,ec=8,eu=8,ge=8,gn=8,gr=8,gt=8,hr=8,hu=8,ie=8,iq=8,kg=8,kh=8,km=8,la=8,lk=8,lu=8,md=8,ml=8,mn=8,mp=8,mt=8,mv=8,om=8,pe=8,ps=8,qa=8,rs=8,rw=8,si=8,sk=8,sn=8,so=8,td=8,tg=8,tj=8,uz=8,vn=8,vu=8,zw=8
dirreq-v3-resp ok=86168,not-enough-sigs=0,unavailable=0,not-found=0,not-modified=3600,busy=0
dirreq-v3-direct-dl complete=0,timeout=0,running=0
dirreq-v3-tunneled-dl complete=68184,timeout=17812,running=176,min=787,d1=43086,d2=88499,q1=123628,d3=172560,d4=359875,md=1459414,d6=5401125,d7=11829058,q3=15114750,d8=18410666,d9=29651000,max=118079000

bridge-stats-end 2022-10-05 17:01:23 (86400 s)
bridge-ips ir=25824,ru=7600,us=6552,cn=824,de=784,mu=600,??=496,gb=352,tn=328,by=320,fr=280,eg=232,nl=216,in=208,za=200,ma=176,ng=160,ca=144,br=136,ua=128,sd=104,ug=104,au=96,ke=88,tr=88,zm=80,es=72,ae=64,fi=64,it=64,pl=64,ro=64,ci=56,jp=56,mx=56,sa=56,se=56,ch=48,bf=40,at=32,be=32,cl=32,cz=32,dk=32,id=32,kz=32,mg=32,pk=32,sg=32,ar=24,hk=24,kr=24,mw=24,no=24,ph=24,pt=24,az=16,bg=16,ga=16,ge=16,gr=16,hu=16,ie=16,il=16,lt=16,lv=16,ly=16,my=16,nz=16,th=16,uz=16,af=8,al=8,am=8,ao=8,ap=8,ba=8,bd=8,bj=8,bn=8,bo=8,bw=8,bz=8,cd=8,cg=8,cm=8,co=8,cr=8,cu=8,cw=8,cy=8,do=8,dz=8,ec=8,ee=8,et=8,eu=8,gh=8,gm=8,gn=8,gq=8,gt=8,hr=8,iq=8,is=8,jo=8,kg=8,km=8,kw=8,ky=8,la=8,lb=8,lk=8,lr=8,lu=8,md=8,me=8,ml=8,mm=8,mn=8,mt=8,mv=8,nc=8,ne=8,np=8,om=8,pa=8,pe=8,pr=8,ps=8,qa=8,re=8,rs=8,rw=8,sc=8,si=8,sk=8,sl=8,sn=8,ss=8,sv=8,sy=8,td=8,tg=8,tj=8,tm=8,tt=8,tw=8,tz=8,uy=8,ve=8,vn=8,ye=8,zw=8
bridge-ip-versions v4=45312,v6=2336
bridge-ip-transports <OR>=8,snowflake=47640

dirreq-stats-end 2022-10-05 17:01:14 (86400 s)
dirreq-v3-ips ir=11736,ru=3736,us=3160,de=440,cn=408,??=352,mu=288,gb=168,by=160,fr=160,tn=160,nl=120,eg=112,za=96,ca=80,in=80,ma=80,au=72,ng=72,ua=64,br=56,sd=48,tr=48,ug=48,es=40,fi=40,it=40,ke=40,pl=40,zm=40,jp=32,ro=32,se=32,ae=24,be=24,bf=24,ch=24,ci=24,cl=24,dk=24,mx=24,sa=24,sg=24,ar=16,at=16,cz=16,hk=16,id=16,kz=16,lv=16,mg=16,mw=16,no=16,ph=16,pk=16,af=8,al=8,am=8,ao=8,ap=8,az=8,bd=8,bg=8,bn=8,bo=8,bw=8,bz=8,cd=8,cg=8,cm=8,co=8,cr=8,cu=8,cw=8,cy=8,do=8,dz=8,ee=8,eu=8,ga=8,ge=8,gh=8,gn=8,gr=8,gt=8,hr=8,hu=8,ie=8,il=8,iq=8,is=8,jo=8,kg=8,km=8,kr=8,kw=8,ky=8,la=8,lb=8,lk=8,lr=8,lt=8,lu=8,ly=8,md=8,me=8,ml=8,mm=8,mn=8,mt=8,mv=8,my=8,ne=8,nz=8,om=8,pe=8,pr=8,ps=8,pt=8,qa=8,re=8,rs=8,sc=8,si=8,sk=8,sl=8,sn=8,ss=8,sv=8,td=8,tg=8,th=8,tj=8,tt=8,tw=8,tz=8,uy=8,uz=8,vn=8,ye=8,zw=8
dirreq-v3-reqs ir=15680,??=8208,ru=5056,us=4480,de=624,cn=600,mu=384,fr=272,gb=272,by=200,tn=200,eg=168,nl=168,za=128,fi=120,ma=112,ng=112,in=104,ca=96,au=88,ua=88,it=80,sd=80,ug=80,br=72,tr=64,pl=56,es=48,ke=48,ro=48,zm=48,jp=40,ae=32,bf=32,ch=32,ci=32,dk=32,mx=32,my=32,se=32,sg=32,be=24,cl=24,id=24,kz=24,lu=24,mg=24,mw=24,ph=24,sa=24,ar=16,at=16,cz=16,eu=16,ga=16,gr=16,hk=16,hr=16,hu=16,kr=16,lb=16,lt=16,lv=16,no=16,pk=16,pt=16,re=16,th=16,uz=16,af=8,al=8,am=8,ao=8,ap=8,az=8,bd=8,bg=8,bn=8,bo=8,bw=8,bz=8,cd=8,cg=8,cm=8,co=8,cr=8,cu=8,cw=8,cy=8,do=8,dz=8,ee=8,ge=8,gh=8,gn=8,gt=8,ie=8,il=8,iq=8,is=8,jo=8,kg=8,km=8,kw=8,ky=8,la=8,lk=8,lr=8,ly=8,md=8,me=8,ml=8,mm=8,mn=8,mt=8,mv=8,ne=8,nz=8,om=8,pe=8,pr=8,ps=8,qa=8,rs=8,sc=8,si=8,sk=8,sl=8,sn=8,ss=8,sv=8,td=8,tg=8,tj=8,tt=8,tw=8,tz=8,uy=8,vn=8,ye=8,zw=8
dirreq-v3-resp ok=38544,not-enough-sigs=0,unavailable=0,not-found=0,not-modified=1512,busy=0
dirreq-v3-direct-dl complete=0,timeout=0,running=0
dirreq-v3-tunneled-dl complete=24816,timeout=13604,running=124,min=593,d1=33865,d2=81693,q1=123623,d3=192486,d4=672636,md=15895500,d6=26288500,d7=29706000,q3=31428000,d8=34294000,d9=54214000,max=121812000

library("tidyverse")

parse <- function(s) {
	x <- str_split_fixed(unlist(strsplit(s, ",")), "=", 2)
	tibble(country = x[,1], n = as.integer(x[,2]))
}

dirreq_v3_reqs <- bind_rows(
	parse("ir=52408,??=12232,us=10560,ru=5232,cn=656,mu=648,tn=472,de=408,ma=256,gb=208,eg=200,za=192,ng=184,by=168,zm=160,fr=152,nl=136,au=120,sd=120,in=112,ca=96,ci=88,ug=88,ua=80,ke=72,br=64,fi=64,bf=56,it=56,jp=48,tr=48,ro=40,ch=32,es=32,ga=32,hk=32,kr=32,kz=32,pl=32,ae=24,id=24,lb=24,mg=24,mw=24,mx=24,sa=24,se=24,am=16,ap=16,ar=16,at=16,be=16,cl=16,co=16,cz=16,dk=16,dz=16,ee=16,gh=16,il=16,lt=16,lv=16,ly=16,my=16,no=16,nz=16,ph=16,pk=16,pt=16,re=16,sg=16,sl=16,th=16,tw=16,af=8,al=8,ao=8,az=8,ba=8,bd=8,bg=8,bh=8,bj=8,bo=8,bw=8,bz=8,cd=8,cg=8,cm=8,cr=8,cu=8,cv=8,cy=8,do=8,ec=8,eu=8,ge=8,gn=8,gr=8,gt=8,hr=8,hu=8,ie=8,iq=8,kg=8,kh=8,km=8,la=8,lk=8,lu=8,md=8,ml=8,mn=8,mp=8,mt=8,mv=8,om=8,pe=8,ps=8,qa=8,rs=8,rw=8,si=8,sk=8,sn=8,so=8,td=8,tg=8,tj=8,uz=8,vn=8,vu=8,zw=8") %>% mutate(end = "2022-10-04 17:01:14"),
	parse("ir=15680,??=8208,ru=5056,us=4480,de=624,cn=600,mu=384,fr=272,gb=272,by=200,tn=200,eg=168,nl=168,za=128,fi=120,ma=112,ng=112,in=104,ca=96,au=88,ua=88,it=80,sd=80,ug=80,br=72,tr=64,pl=56,es=48,ke=48,ro=48,zm=48,jp=40,ae=32,bf=32,ch=32,ci=32,dk=32,mx=32,my=32,se=32,sg=32,be=24,cl=24,id=24,kz=24,lu=24,mg=24,mw=24,ph=24,sa=24,ar=16,at=16,cz=16,eu=16,ga=16,gr=16,hk=16,hr=16,hu=16,kr=16,lb=16,lt=16,lv=16,no=16,pk=16,pt=16,re=16,th=16,uz=16,af=8,al=8,am=8,ao=8,ap=8,az=8,bd=8,bg=8,bn=8,bo=8,bw=8,bz=8,cd=8,cg=8,cm=8,co=8,cr=8,cu=8,cw=8,cy=8,do=8,dz=8,ee=8,ge=8,gh=8,gn=8,gt=8,ie=8,il=8,iq=8,is=8,jo=8,kg=8,km=8,kw=8,ky=8,la=8,lk=8,lr=8,ly=8,md=8,me=8,ml=8,mm=8,mn=8,mt=8,mv=8,ne=8,nz=8,om=8,pe=8,pr=8,ps=8,qa=8,rs=8,sc=8,si=8,sk=8,sl=8,sn=8,ss=8,sv=8,td=8,tg=8,tj=8,tt=8,tw=8,tz=8,uy=8,vn=8,ye=8,zw=8") %>% mutate(end = "2022-10-05 17:01:14")
)

bridge_ips <- bind_rows(
	parse("ir=93224,us=20576,ru=8008,mu=1360,tn=896,cn=888,de=672,??=544,eg=440,ma=416,ng=344,za=344,gb=328,by=304,zm=296,fr=224,sd=216,in=192,ci=176,au=168,nl=168,ke=160,ug=152,ua=136,bf=120,ca=120,br=112,tr=88,it=80,jp=72,ae=64,es=64,sa=64,fi=56,mw=56,se=56,ga=48,kz=48,mx=48,pl=48,ro=48,at=40,ch=40,hk=40,id=40,ly=40,mg=40,be=32,cz=32,kr=32,pk=32,ap=24,co=24,dk=24,gh=24,il=24,my=24,nz=24,pt=24,sg=24,sl=24,af=16,am=16,bd=16,bw=16,cg=16,cl=16,cm=16,ee=16,gr=16,hu=16,ie=16,lt=16,lv=16,md=16,no=16,ph=16,re=16,sk=16,tg=16,th=16,tw=16,vn=16,zw=16,al=8,ao=8,ar=8,az=8,ba=8,bg=8,bh=8,bj=8,bo=8,bs=8,bz=8,cd=8,cr=8,cu=8,cv=8,cw=8,cy=8,do=8,dz=8,ec=8,eu=8,ge=8,gm=8,gn=8,gt=8,gw=8,hr=8,iq=8,is=8,jo=8,kg=8,kh=8,km=8,la=8,lb=8,lk=8,lu=8,ml=8,mm=8,mn=8,mo=8,mp=8,mt=8,mv=8,ne=8,ni=8,np=8,om=8,pe=8,pf=8,pg=8,pr=8,ps=8,py=8,qa=8,rs=8,rw=8,si=8,sn=8,so=8,ss=8,sv=8,sy=8,td=8,tj=8,tt=8,tz=8,uz=8,ve=8,vu=8,ye=8") %>% mutate(end = "2022-10-04 17:01:23"),
	parse("ir=25824,ru=7600,us=6552,cn=824,de=784,mu=600,??=496,gb=352,tn=328,by=320,fr=280,eg=232,nl=216,in=208,za=200,ma=176,ng=160,ca=144,br=136,ua=128,sd=104,ug=104,au=96,ke=88,tr=88,zm=80,es=72,ae=64,fi=64,it=64,pl=64,ro=64,ci=56,jp=56,mx=56,sa=56,se=56,ch=48,bf=40,at=32,be=32,cl=32,cz=32,dk=32,id=32,kz=32,mg=32,pk=32,sg=32,ar=24,hk=24,kr=24,mw=24,no=24,ph=24,pt=24,az=16,bg=16,ga=16,ge=16,gr=16,hu=16,ie=16,il=16,lt=16,lv=16,ly=16,my=16,nz=16,th=16,uz=16,af=8,al=8,am=8,ao=8,ap=8,ba=8,bd=8,bj=8,bn=8,bo=8,bw=8,bz=8,cd=8,cg=8,cm=8,co=8,cr=8,cu=8,cw=8,cy=8,do=8,dz=8,ec=8,ee=8,et=8,eu=8,gh=8,gm=8,gn=8,gq=8,gt=8,hr=8,iq=8,is=8,jo=8,kg=8,km=8,kw=8,ky=8,la=8,lb=8,lk=8,lr=8,lu=8,md=8,me=8,ml=8,mm=8,mn=8,mt=8,mv=8,nc=8,ne=8,np=8,om=8,pa=8,pe=8,pr=8,ps=8,qa=8,re=8,rs=8,rw=8,sc=8,si=8,sk=8,sl=8,sn=8,ss=8,sv=8,sy=8,td=8,tg=8,tj=8,tm=8,tt=8,tw=8,tz=8,uy=8,ve=8,vn=8,ye=8,zw=8") %>% mutate(end = "2022-10-05 17:01:23")
)

x <- dirreq_v3_reqs %>%
	spread(end, n) %>%
	arrange(desc(`2022-10-04 17:01:14`)) %>%
	mutate(rel = (`2022-10-05 17:01:14` - `2022-10-04 17:01:14`) / `2022-10-04 17:01:14`)
cat("|country|2022-10-04 17:01:14|2022-10-05 17:01:14|increase/decrease|\n")
cat("|-------|------------------:|------------------:|----------------:|\n")
for (i in 1:20) {
	cat(paste(c("", x[[i, "country"]], x[[i, "2022-10-04 17:01:14"]], x[[i, "2022-10-05 17:01:14"]], str_replace(sprintf("%+.0f%%", x[[i, "rel"]] * 100), "-", "−"), ""), collapse = "|"), "\n")
}

x <- bridge_ips %>%
	spread(end, n) %>%
	arrange(desc(`2022-10-04 17:01:23`)) %>%
	mutate(rel = (`2022-10-05 17:01:23` - `2022-10-04 17:01:23`) / `2022-10-04 17:01:23`)
cat("|country|2022-10-04 17:01:23|2022-10-05 17:01:23|increase/decrease|\n")
cat("|-------|------------------:|------------------:|----------------:|\n")
for (i in 1:20) {
	cat(paste(c("", x[[i, "country"]], x[[i, "2022-10-04 17:01:23"]], x[[i, "2022-10-05 17:01:23"]], str_replace(sprintf("%+.0f%%", x[[i, "rel"]] * 100), "-", "−"), ""), collapse = "|"), "\n")
}

To give an idea of typical day-to-day variation, here is a comparison from one day earlier:

• 2022-10-03 17:01
• 2022-10-04 17:01

dirreq-v3-reqs:

country	2022-10-03 17:01:14	2022-10-04 17:01:14	increase/decrease
ir	55888	52408	−6%
us	10576	10560	−0%
??	9216	12232	+33%
ru	5504	5232	−5%
mu	752	648	−14%
cn	680	656	−4%
tn	472	472	+0%
de	464	408	−12%
ma	304	256	−16%
eg	200	200	+0%
gb	200	208	+4%
by	192	168	−12%
za	192	192	+0%
ng	184	184	+0%
fr	160	152	−5%
zm	152	160	+5%
nl	136	136	+0%
in	112	112	+0%
ua	112	80	−29%
sd	104	120	+15%

bridge-ips:

country	2022-10-03 17:01:23	2022-10-04 17:01:23	increase/decrease
ir	98376	93224	−5%
us	20880	20576	−1%
ru	8320	8008	−4%
mu	1496	1360	−9%
tn	984	896	−9%
cn	968	888	−8%
de	736	672	−9%
??	536	544	+1%
eg	496	440	−11%
ma	472	416	−12%
za	368	344	−7%
gb	360	328	−9%
by	352	304	−14%
ng	336	344	+2%
zm	288	296	+3%
in	264	192	−27%
fr	256	224	−12%
sd	248	216	−13%
nl	176	168	−5%
au	168	168	+0%

Source code and data

@type bridge-extra-info 1.3
extra-info flakey1 5481936581E23D2D178105D44DB6915AB06BFB7F
master-key-ed25519 cefgVl0wPLbQew8dIXs79HPheEaiETs24HPp0FXNWAI
published 2022-10-04 09:01:23
transport snowflake
write-history 2022-10-04 03:48:24 (86400 s) 1314522086400,1182190550016,1432269388800,1780690229248,1860857608192
read-history 2022-10-04 03:48:24 (86400 s) 1297107302400,1282063420416,1415193378816,1753156814848,1836411276288
ipv6-write-history 2022-10-04 03:48:24 (86400 s) 0,0,0,3072,14336
dirreq-write-history 2022-10-04 03:48:24 (86400 s) 34875878400,24556262400,33994770432,43069770752,40381877248
dirreq-read-history 2022-10-04 03:48:24 (86400 s) 3176064000,1817423872,3024691200,2936916992,2718101504
geoip-db-digest B3A9770171D3060502B7F13C0618BE109B92DF6C
geoip6-db-digest 410B70763FC675B3622264FAA0FC67B78FDE30C2
dirreq-stats-end 2022-10-03 17:01:14 (86400 s)
dirreq-v3-ips ir=39640,us=7888,ru=3968,mu=560,cn=464,??=392,tn=352,de=328,ma=192,eg=160,gb=160,by=152,za=144,ng=136,fr=112,zm=112,nl=96,in=88,au=80,sd=80,ca=72,ua=72,br=64,ke=56,ug=56,ci=48,bf=40,it=40,tr=40,ae=32,fi=32,jp=32,kz=32,at=24,ch=24,es=24,ga=24,hk=24,id=24,kr=24,mg=24,mw=24,mx=24,pl=24,ro=24,sa=24,se=24,sg=24,be=16,cm=16,co=16,cz=16,dk=16,gh=16,il=16,lv=16,ly=16,my=16,no=16,nz=16,pk=16,th=16,tw=16,af=8,al=8,am=8,ao=8,ap=8,ar=8,az=8,ba=8,bd=8,bg=8,bi=8,bj=8,bo=8,bw=8,cg=8,cl=8,cr=8,cu=8,cy=8,do=8,dz=8,ec=8,ee=8,eu=8,ge=8,gm=8,gn=8,gr=8,hr=8,hu=8,ie=8,iq=8,is=8,jo=8,kg=8,kh=8,kw=8,la=8,lb=8,lk=8,lt=8,lu=8,md=8,ml=8,mm=8,mn=8,mo=8,mt=8,mv=8,nc=8,ne=8,np=8,om=8,pe=8,ph=8,ps=8,pt=8,re=8,rs=8,si=8,sk=8,sl=8,sn=8,ss=8,sy=8,sz=8,tg=8,tj=8,tm=8,tz=8,uz=8,ve=8,vn=8,ye=8,zw=8
dirreq-v3-reqs ir=55888,us=10576,??=9216,ru=5504,mu=752,cn=680,tn=472,de=464,ma=304,eg=200,gb=200,by=192,za=192,ng=184,fr=160,zm=152,nl=136,in=112,ua=112,sd=104,au=96,br=96,ca=96,ke=96,ug=72,it=64,ci=56,tw=56,ae=48,bf=48,dk=48,fi=48,jp=48,tr=48,es=40,kz=40,be=32,ch=32,ga=32,hk=32,kr=32,mg=32,mw=32,mx=32,ro=32,sa=32,se=32,ar=24,at=24,cz=24,id=24,nz=24,pl=24,sg=24,ap=16,cl=16,cm=16,co=16,ee=16,gh=16,hu=16,il=16,lb=16,lv=16,ly=16,my=16,no=16,pk=16,sl=16,tg=16,th=16,zw=16,af=8,al=8,am=8,ao=8,az=8,ba=8,bd=8,bg=8,bi=8,bj=8,bo=8,bw=8,cg=8,cr=8,cu=8,cy=8,do=8,dz=8,ec=8,eu=8,ge=8,gm=8,gn=8,gr=8,hr=8,ie=8,iq=8,is=8,jo=8,kg=8,kh=8,kw=8,la=8,lk=8,lt=8,lu=8,md=8,ml=8,mm=8,mn=8,mo=8,mt=8,mv=8,nc=8,ne=8,np=8,om=8,pe=8,ph=8,ps=8,pt=8,re=8,rs=8,si=8,sk=8,sn=8,ss=8,sy=8,sz=8,tj=8,tm=8,tz=8,uz=8,ve=8,vn=8,ye=8
dirreq-v3-resp ok=87360,not-enough-sigs=0,unavailable=0,not-found=0,not-modified=2240,busy=0
dirreq-v3-direct-dl complete=0,timeout=0,running=0
dirreq-v3-tunneled-dl complete=70176,timeout=17068,running=112,min=368,d1=34083,d2=63946,q1=81922,d3=103112,d4=168587,md=312166,d6=966645,d7=4802416,q3=8997166,d8=13667500,d9=27050000,max=110119000
hidserv-stats-end 2022-10-03 17:01:14 (86400 s)
hidserv-rend-relayed-cells -7450 delta_f=2048 epsilon=0.30 bin_size=1024
hidserv-dir-onions-seen -23 delta_f=8 epsilon=0.30 bin_size=8
padding-counts 2022-10-03 17:01:23 (86400 s) bin-size=10000 write-drop=0 write-pad=4210000 write-total=3681150000 read-drop=0 read-pad=27340000 read-total=3627170000 enabled-read-pad=22590000 enabled-read-total=3157830000 enabled-write-pad=4010000 enabled-write-total=1465640000 max-chanpad-timers=2470
bridge-stats-end 2022-10-03 17:01:23 (86400 s)
bridge-ips ir=98376,us=20880,ru=8320,mu=1496,tn=984,cn=968,de=736,??=536,eg=496,ma=472,za=368,gb=360,by=352,ng=336,zm=288,in=264,fr=256,sd=248,nl=176,au=168,ke=168,ug=168,ci=152,br=144,ua=144,ca=136,bf=96,it=88,tr=88,ae=80,es=72,sa=72,jp=64,kz=64,ch=56,mx=56,pl=56,ro=56,se=56,fi=48,ga=48,hk=48,id=48,mg=48,mw=48,at=40,gh=40,be=32,cm=32,cz=32,dk=32,kr=32,ly=32,my=32,nz=32,pk=32,sg=32,ap=24,bw=24,co=24,no=24,ph=24,sl=24,th=24,tw=24,ar=16,az=16,bd=16,cg=16,cl=16,dz=16,ee=16,ge=16,gr=16,hu=16,ie=16,il=16,iq=16,lt=16,lv=16,pt=16,sk=16,tg=16,tz=16,vn=16,zw=16,af=8,al=8,am=8,ao=8,ba=8,bg=8,bh=8,bi=8,bj=8,bo=8,bz=8,cd=8,cr=8,cu=8,cv=8,cw=8,cy=8,do=8,ec=8,eu=8,gm=8,gn=8,gu=8,hr=8,is=8,jm=8,jo=8,kg=8,kh=8,km=8,kw=8,la=8,lb=8,li=8,lk=8,lu=8,md=8,ml=8,mm=8,mn=8,mo=8,mt=8,mv=8,mz=8,nc=8,ne=8,np=8,om=8,pa=8,pe=8,pf=8,pg=8,ps=8,qa=8,re=8,rs=8,rw=8,si=8,sn=8,so=8,ss=8,sy=8,sz=8,td=8,tj=8,tm=8,uz=8,ve=8,ye=8
bridge-ip-versions v4=133880,v6=5176
bridge-ip-transports <OR>=8,snowflake=139056
router-digest-sha256 hgDgW2OuA7p68V7KYd7csT888j6wk1K1msAunzcnYW8
router-digest 449C9D7A288F1E837587069655A8C7CD0A05103F

library("tidyverse")

parse <- function(s) {
	x <- str_split_fixed(unlist(strsplit(s, ",")), "=", 2)
	tibble(country = x[,1], n = as.integer(x[,2]))
}

print_table <- function(x, before_col, after_col) {
	x <- arrange(x, desc(.data[[before_col]]))
	cat(sprintf("|country|%s|%s|increase/decrease|\n", before_col, after_col))
	cat("|-------|------------------:|------------------:|----------------:|\n")
	for (i in 1:20) {
		cat(paste(c("", x[[i, "country"]], x[[i, before_col]], x[[i, after_col]], str_replace(sprintf("%+.0f%%", (x[[i, after_col]] - x[[i, before_col]]) / x[[i, before_col]] * 100), "-", "−"), ""), collapse = "|"), "\n")
	}
}

dirreq_v3_reqs <- bind_rows(
	parse("ir=55888,us=10576,??=9216,ru=5504,mu=752,cn=680,tn=472,de=464,ma=304,eg=200,gb=200,by=192,za=192,ng=184,fr=160,zm=152,nl=136,in=112,ua=112,sd=104,au=96,br=96,ca=96,ke=96,ug=72,it=64,ci=56,tw=56,ae=48,bf=48,dk=48,fi=48,jp=48,tr=48,es=40,kz=40,be=32,ch=32,ga=32,hk=32,kr=32,mg=32,mw=32,mx=32,ro=32,sa=32,se=32,ar=24,at=24,cz=24,id=24,nz=24,pl=24,sg=24,ap=16,cl=16,cm=16,co=16,ee=16,gh=16,hu=16,il=16,lb=16,lv=16,ly=16,my=16,no=16,pk=16,sl=16,tg=16,th=16,zw=16,af=8,al=8,am=8,ao=8,az=8,ba=8,bd=8,bg=8,bi=8,bj=8,bo=8,bw=8,cg=8,cr=8,cu=8,cy=8,do=8,dz=8,ec=8,eu=8,ge=8,gm=8,gn=8,gr=8,hr=8,ie=8,iq=8,is=8,jo=8,kg=8,kh=8,kw=8,la=8,lk=8,lt=8,lu=8,md=8,ml=8,mm=8,mn=8,mo=8,mt=8,mv=8,nc=8,ne=8,np=8,om=8,pe=8,ph=8,ps=8,pt=8,re=8,rs=8,si=8,sk=8,sn=8,ss=8,sy=8,sz=8,tj=8,tm=8,tz=8,uz=8,ve=8,vn=8,ye=8") %>% mutate(end = "2022-10-03 17:01:14"),
	parse("ir=52408,??=12232,us=10560,ru=5232,cn=656,mu=648,tn=472,de=408,ma=256,gb=208,eg=200,za=192,ng=184,by=168,zm=160,fr=152,nl=136,au=120,sd=120,in=112,ca=96,ci=88,ug=88,ua=80,ke=72,br=64,fi=64,bf=56,it=56,jp=48,tr=48,ro=40,ch=32,es=32,ga=32,hk=32,kr=32,kz=32,pl=32,ae=24,id=24,lb=24,mg=24,mw=24,mx=24,sa=24,se=24,am=16,ap=16,ar=16,at=16,be=16,cl=16,co=16,cz=16,dk=16,dz=16,ee=16,gh=16,il=16,lt=16,lv=16,ly=16,my=16,no=16,nz=16,ph=16,pk=16,pt=16,re=16,sg=16,sl=16,th=16,tw=16,af=8,al=8,ao=8,az=8,ba=8,bd=8,bg=8,bh=8,bj=8,bo=8,bw=8,bz=8,cd=8,cg=8,cm=8,cr=8,cu=8,cv=8,cy=8,do=8,ec=8,eu=8,ge=8,gn=8,gr=8,gt=8,hr=8,hu=8,ie=8,iq=8,kg=8,kh=8,km=8,la=8,lk=8,lu=8,md=8,ml=8,mn=8,mp=8,mt=8,mv=8,om=8,pe=8,ps=8,qa=8,rs=8,rw=8,si=8,sk=8,sn=8,so=8,td=8,tg=8,tj=8,uz=8,vn=8,vu=8,zw=8") %>% mutate(end = "2022-10-04 17:01:14"),
	parse("ir=15680,??=8208,ru=5056,us=4480,de=624,cn=600,mu=384,fr=272,gb=272,by=200,tn=200,eg=168,nl=168,za=128,fi=120,ma=112,ng=112,in=104,ca=96,au=88,ua=88,it=80,sd=80,ug=80,br=72,tr=64,pl=56,es=48,ke=48,ro=48,zm=48,jp=40,ae=32,bf=32,ch=32,ci=32,dk=32,mx=32,my=32,se=32,sg=32,be=24,cl=24,id=24,kz=24,lu=24,mg=24,mw=24,ph=24,sa=24,ar=16,at=16,cz=16,eu=16,ga=16,gr=16,hk=16,hr=16,hu=16,kr=16,lb=16,lt=16,lv=16,no=16,pk=16,pt=16,re=16,th=16,uz=16,af=8,al=8,am=8,ao=8,ap=8,az=8,bd=8,bg=8,bn=8,bo=8,bw=8,bz=8,cd=8,cg=8,cm=8,co=8,cr=8,cu=8,cw=8,cy=8,do=8,dz=8,ee=8,ge=8,gh=8,gn=8,gt=8,ie=8,il=8,iq=8,is=8,jo=8,kg=8,km=8,kw=8,ky=8,la=8,lk=8,lr=8,ly=8,md=8,me=8,ml=8,mm=8,mn=8,mt=8,mv=8,ne=8,nz=8,om=8,pe=8,pr=8,ps=8,qa=8,rs=8,sc=8,si=8,sk=8,sl=8,sn=8,ss=8,sv=8,td=8,tg=8,tj=8,tt=8,tw=8,tz=8,uy=8,vn=8,ye=8,zw=8") %>% mutate(end = "2022-10-05 17:01:14")
)

bridge_ips <- bind_rows(
	parse("ir=98376,us=20880,ru=8320,mu=1496,tn=984,cn=968,de=736,??=536,eg=496,ma=472,za=368,gb=360,by=352,ng=336,zm=288,in=264,fr=256,sd=248,nl=176,au=168,ke=168,ug=168,ci=152,br=144,ua=144,ca=136,bf=96,it=88,tr=88,ae=80,es=72,sa=72,jp=64,kz=64,ch=56,mx=56,pl=56,ro=56,se=56,fi=48,ga=48,hk=48,id=48,mg=48,mw=48,at=40,gh=40,be=32,cm=32,cz=32,dk=32,kr=32,ly=32,my=32,nz=32,pk=32,sg=32,ap=24,bw=24,co=24,no=24,ph=24,sl=24,th=24,tw=24,ar=16,az=16,bd=16,cg=16,cl=16,dz=16,ee=16,ge=16,gr=16,hu=16,ie=16,il=16,iq=16,lt=16,lv=16,pt=16,sk=16,tg=16,tz=16,vn=16,zw=16,af=8,al=8,am=8,ao=8,ba=8,bg=8,bh=8,bi=8,bj=8,bo=8,bz=8,cd=8,cr=8,cu=8,cv=8,cw=8,cy=8,do=8,ec=8,eu=8,gm=8,gn=8,gu=8,hr=8,is=8,jm=8,jo=8,kg=8,kh=8,km=8,kw=8,la=8,lb=8,li=8,lk=8,lu=8,md=8,ml=8,mm=8,mn=8,mo=8,mt=8,mv=8,mz=8,nc=8,ne=8,np=8,om=8,pa=8,pe=8,pf=8,pg=8,ps=8,qa=8,re=8,rs=8,rw=8,si=8,sn=8,so=8,ss=8,sy=8,sz=8,td=8,tj=8,tm=8,uz=8,ve=8,ye=8") %>% mutate(end = "2022-10-03 17:01:23"),
	parse("ir=93224,us=20576,ru=8008,mu=1360,tn=896,cn=888,de=672,??=544,eg=440,ma=416,ng=344,za=344,gb=328,by=304,zm=296,fr=224,sd=216,in=192,ci=176,au=168,nl=168,ke=160,ug=152,ua=136,bf=120,ca=120,br=112,tr=88,it=80,jp=72,ae=64,es=64,sa=64,fi=56,mw=56,se=56,ga=48,kz=48,mx=48,pl=48,ro=48,at=40,ch=40,hk=40,id=40,ly=40,mg=40,be=32,cz=32,kr=32,pk=32,ap=24,co=24,dk=24,gh=24,il=24,my=24,nz=24,pt=24,sg=24,sl=24,af=16,am=16,bd=16,bw=16,cg=16,cl=16,cm=16,ee=16,gr=16,hu=16,ie=16,lt=16,lv=16,md=16,no=16,ph=16,re=16,sk=16,tg=16,th=16,tw=16,vn=16,zw=16,al=8,ao=8,ar=8,az=8,ba=8,bg=8,bh=8,bj=8,bo=8,bs=8,bz=8,cd=8,cr=8,cu=8,cv=8,cw=8,cy=8,do=8,dz=8,ec=8,eu=8,ge=8,gm=8,gn=8,gt=8,gw=8,hr=8,iq=8,is=8,jo=8,kg=8,kh=8,km=8,la=8,lb=8,lk=8,lu=8,ml=8,mm=8,mn=8,mo=8,mp=8,mt=8,mv=8,ne=8,ni=8,np=8,om=8,pe=8,pf=8,pg=8,pr=8,ps=8,py=8,qa=8,rs=8,rw=8,si=8,sn=8,so=8,ss=8,sv=8,sy=8,td=8,tj=8,tt=8,tz=8,uz=8,ve=8,vu=8,ye=8") %>% mutate(end = "2022-10-04 17:01:23"),
	parse("ir=25824,ru=7600,us=6552,cn=824,de=784,mu=600,??=496,gb=352,tn=328,by=320,fr=280,eg=232,nl=216,in=208,za=200,ma=176,ng=160,ca=144,br=136,ua=128,sd=104,ug=104,au=96,ke=88,tr=88,zm=80,es=72,ae=64,fi=64,it=64,pl=64,ro=64,ci=56,jp=56,mx=56,sa=56,se=56,ch=48,bf=40,at=32,be=32,cl=32,cz=32,dk=32,id=32,kz=32,mg=32,pk=32,sg=32,ar=24,hk=24,kr=24,mw=24,no=24,ph=24,pt=24,az=16,bg=16,ga=16,ge=16,gr=16,hu=16,ie=16,il=16,lt=16,lv=16,ly=16,my=16,nz=16,th=16,uz=16,af=8,al=8,am=8,ao=8,ap=8,ba=8,bd=8,bj=8,bn=8,bo=8,bw=8,bz=8,cd=8,cg=8,cm=8,co=8,cr=8,cu=8,cw=8,cy=8,do=8,dz=8,ec=8,ee=8,et=8,eu=8,gh=8,gm=8,gn=8,gq=8,gt=8,hr=8,iq=8,is=8,jo=8,kg=8,km=8,kw=8,ky=8,la=8,lb=8,lk=8,lr=8,lu=8,md=8,me=8,ml=8,mm=8,mn=8,mt=8,mv=8,nc=8,ne=8,np=8,om=8,pa=8,pe=8,pr=8,ps=8,qa=8,re=8,rs=8,rw=8,sc=8,si=8,sk=8,sl=8,sn=8,ss=8,sv=8,sy=8,td=8,tg=8,tj=8,tm=8,tt=8,tw=8,tz=8,uy=8,ve=8,vn=8,ye=8,zw=8") %>% mutate(end = "2022-10-05 17:01:23")
)

x <- dirreq_v3_reqs %>% spread(end, n)
print_table(x, "2022-10-03 17:01:14", "2022-10-04 17:01:14")
print_table(x, "2022-10-04 17:01:14", "2022-10-05 17:01:14")

x <- bridge_ips %>% spread(end, n)
print_table(x, "2022-10-03 17:01:23", "2022-10-04 17:01:23")
print_table(x, "2022-10-04 17:01:23", "2022-10-05 17:01:23")

I am just now reflecting that these country-specific metrics are not really meaningful if we are considering the problem of the bandwidth drop from a routing perspective. That's because the countries of are those of the end users, not of the Snowflake proxies, which are what connect directly to the bridge.

However the metrics do show that the cause is not a country-wide block of broker rendezvous.

Looking at broker metrics, we also see a sudden drop in client polls from yesterday at around 17:15 utc.

To me, this drop seems too sharp to be caused by performance. The drop happens in 5 minutes or less.

Offhand, I can't think of why lowered bandwidth at the bridge should cause reduced proxy polls. It's possible the causation is the other way: fewer proxy polls cause less bandwidth to be used. But then what could cause a sudden drop in proxy polls? The most obvious explanation is the broker deployment of #40193 (closed), but the timing doesn't match up: 2022-10-03 12:50 versus 2022-10-04 17:15.

Version 0.6.2 of the WebExtension was released at about the right time (the commit is from 2022-10-04 16:09), but is it plausible it would cause such a massive drop all at once? I would not expect all browser users to upgrade simultaneous, nor the Firefox and Chrome stores to make the updates available at the same time.

Oh I hadn't thought about that. I would expect a difficulty with proxies to cause client polls to spike, not suddenly drop. But this is maybe too complex to intuit.

But then what could cause a sudden drop in proxy polls?

My mistake, the graph shows client polls, not proxy polls. Massive and immediate blocking of broker rendezvous channels could have that effect. The obvious suspect is Iran, but #40207 (comment 2840696) shows before-and-after large drops in other countries as well.

A rate limiting issue with the domain front or the broker could cause it to affect everywhere roughly equally.

Is there a way to check if there have been any changes in the CDN account, or anyone we could talk to to find out?

I don't have access, maybe @tpo/tpa?

Edit: for context, this would be our account with fastly

i'll take a glance

i don't see anything that pops out on the traffic graphs... which endpoint is this, snowflake-broker or moat?

snowflake-broker

LGTM no?

We'd be looking specifically for an issue or sudden drop in traffic starting around 2022-10-04 17:15

Yeah it looks good, we can see the effects there of the problem but not the cause. Thanks!

i'll sign off here, ping me again if you need anything

@cohosh what are the units on the vertical axis of the client requests graphs? If it said 75K, is that per second, per hour?

@shelikhoo I am seeing a lot more "rejected relay pattern from proxy" messages in the broker log. In fact, they seem to occur more frequently than SYNs to port 443, which should be impossible? Try this to see how fast they are coming in:

tail -F /var/log/snowflake-broker/current | head -n 1000

I measured the time to see 10000 "rejected relay pattern" logs and compared it to the time to see 10000 SYN packets. I measure about 345 "rejected relay pattern" per second, and only about 67 SYN per second. I may be doing something wrong with the measurement, but can you help explain it?

# time tail -F /var/log/snowflake-broker/current  | grep rejected | head -n 10000 > /dev/null
real    0m28.973s
user    0m0.243s
sys     0m0.778s
# time tcpdump -n 'dst port 443 and ((ip and tcp[tcpflags] & tcp-syn != 0) or (ip6 and (ip6[40+13] & tcp-syn) != 0))' | head -n 10000 > /dev/null
real    2m28.874s
user    0m0.001s
sys     0m0.124s

The graph is generated by the following prometheus query:

sum by(nat) (increase(snowflake_rounded_client_poll_total[5m]))

where snowflake_rounded_client_poll_total is a monotonically increasing counter.

sum by(nat) just means that we've grouped all client polls by NAT type.

According to the prometheus query documentation, increase() shows by how much the counter has increased over the last 5 minutes. This value is given every second, to show a smooth curve.

If you want, I can show the results of a different query that might be more useful to interpret directly.

I am seeing a lot more "rejected relay pattern from proxy" messages in the broker log. In fact, they seem to occur more frequently than SYNs to port 443, which should be impossible?

Rejected proxies re-poll more frequently than proxies that are waiting for clients. We saw this effect in the prometheus metrics for the broker when support for these proxies was turned off last Monday:

My intuition would be that re-polling proxies are re-using their TCP sessions.

My intuition would be that re-polling proxies are re-using their TCP sessions.

Yes, you are probably right about that.

Here's a better screenshot of the prometheus metrics for total client polls

changed the description

I updated the graph in the description to show the dynamics over the past 2 days of diminished bandwidth. It's not flat or capped at 1 Gbps, but still has roughly the expected diurnal pattern. The few little spikes are bandwidth tests being run from the bridge.

Rebooted the bridge at 2022-10-06 19:48:48. It was operational again by 2022-10-06 19:51:18 when I was able to log in with ssh. I'm not seeing any increase in bandwidth, but it was a long shot anyway.

I've put out an appeal for testers and logs:

• Unexplained drop in Snowflake client polls and bandwidth, testers wanted

I contacted Greenhost (eclips.is) support, and they say they are not aware of any network changes around 2022-10-04 17:15.

added Doing label

However the metrics do show that the cause is not a country-wide block of broker rendezvous.

I have a new working hypothesis: the sudden decline is caused by a partial block of the broker in Iran. As for why countries other than Iran are seemingly affected, my best guess is that they are geolocation errors: IP addresses in Iran wrongly being attributed to other countries (chiefly us and ??).

Below are the top 20 countries according to dirreq-v3-reqs on 2022-10-04, and how much they changed 24 hours later.

Here's that information with more context.

snowflake-01-dirreqs-20221014.zip

You can see the overall massive decline between 04 Oct and 05 Oct. The change from 52408 to 15680matches what I posted in the first table at #40207 (comment 2840696). Unlike typical Tor Metrics graphs, this visualization does not sum the separate tor instances into a single point, and it does not linearly distribute the counts over 24-hour periods, instead showing the exact timestamp dirreq-stats-end associated with dirreq-v3-reqs in the descriptor. The apparent decline in ru at the left is illusory, caused by increasing the number of tor instances (#40173 (closed), #40176 (closed); more instances ⇒ less traffic per descriptor).

What I want to call out is: See how us and ?? increase/decrease along with fluctuations in ir. It is almost as if those series are scaled-down versions of the ir series. If we assume that the only significant change at the time was in ir, then we can say the value of us (the number of users that geolocate to the US, correctly or incorrectly) is

us = P(geo(us)|¬ir)×N(¬ir) + P(geo(us)|ir)×N(ir)

where P(X|Y) is the probability that geolocation guesses X when the true country is in Y, and N(X) is the unknown true number of users from countries in X. When N(ir) decreases, the left-hand term stays the same, but the right-hand side decreases, and I think this is what is happening in the graph.

If this working hypothesis is correct, the question becomes, what is the nature of the block? To this question we do not yet have a clear answer.

• One guess is that it has to do with the Golang TLS fingerprint in currently shipped version of snowflake-client. That arose out of this comment that said Tor Browser for desktop worked and Orbot did not work, on the same network.

  • In my own test, Orbot for Android had the same fingerprint as Tor Browser for Linux. However, in #tor-anticensorship, @trinity-1686a found otherwise:

    some difference I see between TB and Orbot handhakes: they use the same cipher-suites, but not in the same order. The TLS extension supported_version list only 1.2 and 1.3 for TB, but also 1.1 and 1.0 for Orbot.

    So I consider TLS fingerprinting to remain a possibility.

  • tpo/applications/tor-browser-build#40629 (closed) will upgrade snowflake-client to be a version that supports utls in a future version of Tor Browser. I don't know about the release plan for Orbot.

• Another report showed that all DNS lookups failed while trying to use Snowflake on a mobile network. I wasn't able to get more information about whether it was only Snowflake-related domains that were affected (or if all DNS was blocked for the user, or if it was a misconfiguration).

to be more precise about what I observed:

TorBrowser on Linux and TorBrowser on Android share the same cipher-suite order, however Orbot has a separate order (but same cipher-suite set). This translate to a different JA3 hash (commonly used for TLS fingerprinting in IDS): 3fed133de60c35724739b913924b6c24 for TBs, 049f44ae40ab2cab555bdfee22e7d7cb for Orbot.

TorBrowser on Linux and Orbot both claims to support TLS 1.0-1.3, whereas TorBrowser on Android support only TLS 1.2-1.3. This does not change the JA3 fingerprint as it's sensitive to list and order of extensions, but not their size or content.

below the TLS sessions I looked at:

• orbot.pcap (note, looks mostly identical to what got shared on net4people bbs
• tb-android.pcap
• tb-linux.pcap

Thank you, that's very helpful. The fingerprint of your orbot.pcap is indeed identical to my one. In tlsfingerprint.io terms, it is adfe55afa6f23950.

This fingerprint adfe55afa6f23950 differs only minorly from 750e3f0f585283bd, which was observed in https://github.com/net4people/bbs/issues/139#issuecomment-1280057679 to be produced by a Go program running on Raspberry Pi.

The critical thing in native Go crypto/tls fingerprints since go1.17 is that the order of ciphersuites depends on whether the platform has support for accelerated AES-GCM. See how there are two versions of everything: cipherSuitesPreferenceOrder and cipherSuitesPreferenceOrderNoAES; defaultCipherSuitesTLS13 and defaultCipherSuitesTLS13NoAES. This explains the two different ciphersuite orderings you observed. The choice of which to use happens at runtime:

hasGCMAsmAMD64 = cpu.X86.HasAES && cpu.X86.HasPCLMULQDQ
hasGCMAsmARM64 = cpu.ARM64.HasAES && cpu.ARM64.HasPMULL
hasGCMAsmS390X = cpu.S390X.HasAES && cpu.S390X.HasAESCBC && cpu.S390X.HasAESCTR &&
	(cpu.S390X.HasGHASH || cpu.S390X.HasAESGCM)
hasAESGCMHardwareSupport = runtime.GOARCH == "amd64" && hasGCMAsmAMD64 ||
	runtime.GOARCH == "arm64" && hasGCMAsmARM64 ||
	runtime.GOARCH == "s390x" && hasGCMAsmS390X

Your tb-android.pcap has fingerprint 14062e58336049c2 and your tb-linux.pcap has fingerprint 99c75b317dc7a5a1. The two are very similar as you said, the only difference being the Supported Versions field. I am guessing, based on the fact that your Android fingerprint prioritized AES-GCM ciphersuites, that the hardware of whatever device you tested it on satisfies the cpu.ARM64.HasAES && cpu.ARM64.HasPMULL test.

My guess as to why I saw the fingerprint adfe55afa6f23950, which de-prioritizes AES-GCM, in the amd64 desktop version of Tor Browser, is that I was running it in a QEMU VM, which may not have signaled the needed CPU features. (I just checked, and indeed, inside the VM, cat /proc/cpuinfo does not have the flags aes and pclmulqdq.)

The upshot of all this is that we can likely get snowflake-client to work again in Iran by shipping utls support. The critical platform for this to happen is Android, because the native Go crypto/tls fingerprint on amd64 happens not to be blocked.

It should be noted that AES acceleration support is more likely to be found on high-end smartphone, while, more affordable phone that use 32-bit version of arm usually don't have AES acceleration support.

TorBrowser on Linux and Orbot both claims to support TLS 1.0-1.3, whereas TorBrowser on Android support only TLS 1.2-1.3.

This is another aspect that apparently is important, which is investigated at tpo/anti-censorship/team#96 (comment 2845607). The Go crypto/tls fingerprint changed in go1.18 just as you describe: it raised the default minimum supported version in Client Hellos from TLS 1.0 to TLS 1.2. It appears that only the min = TLS 1.0 (go1.17) fingerprint is on the current Iran blocklist.

Tor Browser for Android + Snowflake [11.5.4 without any bridge line changes] works, but Orbot (iOS and Android) with snowflake is not working.

Tor Browser 11.5.4 for Android uses go1.18.7, while Tor Browser 11.5.4 for desktop uses go1.17.13. (That's right, the Tor Browser build uses different versions of the Go compiler depending on the target.) Since neither of these two configurations is blocked, we surmise that both conditions have to be true in order to get blocked: you need to be using the non-accelerated AES ciphersuite list (more common on mobile), and you need to have a snowflake-client compiled by a pre-go1.18 version of the compiler (minimum TLS version = 1.0).

Of course, it is probably easy for the censors to add the go1.18 fingerprint to their blocklist, so proper camouflage in the form of uTLS is the right answer. But recompiling client programs with go1.18 could buy a few days' time.

mentioned in issue tpo/anti-censorship/team#96 (closed)

mentioned in merge request tpo/applications/tor-browser-build!540 (closed)

marked this issue as related to tpo/anti-censorship/team#96 (closed)

mentioned in issue tpo/network-health/metrics/analysis#40013 (closed)

I think the analysis is finished. In summary, we believe that the sudden drop in users and bandwidth on 2022-10-04 was caused by blocking of a specific TLS fingerprint in Iran. The TLS fingerprint was the one used by Snowflake in Orbot on most types of mobile devices (and likely other Go-based tools). Closely related fingerprints (different because of accidents of hardware AES support or compiler version) used by Tor Browser on desktop and Android were not blocked. The fact that certain other country codes (notably us and ??) experienced a simultaneous decrease is attributed to geolocation errors (IP addresses actually in Iran being geolocated elsewhere).

closed

mentioned in issue tpo/anti-censorship/team#113 (closed)

mentioned in issue tpo/anti-censorship/team#115 (closed)

mentioned in commit tpo/network-health/metrics/timeline@3cbdb765

mentioned in issue tpo/anti-censorship/censorship-analysis#40038 (closed)

mentioned in issue tpo/applications/tor-browser-build#40740 (closed)

mentioned in commit linus/ac-team.wiki@fb2a23a1