Meek metrics says 12 users at best, tor regularly sees 5000+

Right. It seems we actually had those several thousand users before that huge spike at the begin of October. @hiro: looks like a metrics bug?

I have been thinking what the spikes were about, and I have had pending the task to parse the descriptors manually for the past few months and see.

I was checking onionoo and it looks like we are not recording clients there either: https://onionoo.torproject.org/clients?lookup=B1AA2EF6435720FB710B9F94872691A2DCD49E78

Plotted the number of estimated clients computed manually vs computed on metrics.tpo:

I am not sure the bug is on metrics side.

Interesting. @HackerNCoder: where are you getting the 5000+ number from? Maybe @meskio has a clue as to what is going on here...

Dec 05 20:09:57 tor-meek-bridge Tor[12180]: Heartbeat: Since last heartbeat message, I have seen 8526 unique clients.
Dec 06 02:09:57 tor-meek-bridge Tor[12180]: Heartbeat: Since last heartbeat message, I have seen 5537 unique clients.
Dec 06 08:09:57 tor-meek-bridge Tor[12180]: Heartbeat: Since last heartbeat message, I have seen 6869 unique clients.

In the published extra-info it looks like there are thousands of connections:

@type bridge-extra-info 1.3
extra-info meek B1AA2EF6435720FB710B9F94872691A2DCD49E78
...
bridge-ips cn=3968,ir=3840,us=3640,ru=1624,tm=936,de=856,nl=664,gb=632,fr=544,by=424,hk=360,ca=312,in=304,br=192,ua=192,eg=168,sg=168,jp=160,au=152,es=128,tr=120,tw=120,ye=120,it=112,za=112,pl=104,se=104,ae=96,fi=96,kr=88,ch=80,mx=80,sa=72,at=56,id=56,ap=48,ar=48,ro=48,cl=40,cz=40,ng=40,no=40,ph=40,be=32,bg=32,co=32,dz=32,il=32,kz=32,lu=32,pk=32,pt=32,bd=24,dk=24,ie=24,iq=24,ke=24,ma=24,my=24,th=24,vn=24,az=16,cr=16,eu=16,gr=16,hr=16,hu=16,jo=16,lt=16,lv=16,md=16,mm=16,nz=16,pe=16,rs=16,sk=16,tn=16,uz=16,ve=16,zw=16,??=8,ad=8,af=8,al=8,am=8,ao=8,ba=8,bb=8,bh=8,bj=8,bo=8,bw=8,cd=8,ci=8,cm=8,cu=8,cv=8,cy=8,do=8,ec=8,ee=8,et=8,ge=8,gf=8,gh=8,gn=8,gp=8,gt=8,hn=8,ht=8,is=8,jm=8,kg=8,kh=8,kw=8,la=8,lb=8,lk=8,ls=8,ly=8,me=8,mn=8,mo=8,mr=8,mt=8,mu=8,mv=8,mz=8,nc=8,ni=8,np=8,om=8,pa=8,pr=8,ps=8,py=8,qa=8,re=8,sc=8,sd=8,si=8,sn=8,so=8,sv=8,sy=8,tt=8,tz=8,ug=8,uy=8,zm=8
bridge-ip-versions v4=21784,v6=8
bridge-ip-transports <OR>=4168,meek=21784
...

Isn't metrics.tpo plotting the numbers from the extra-info?

Yes but not exactly. There is an estimation step. I wonder if some descriptors are being excluded.

I guess metrics takes its info from bridge-ips?

@type bridge-extra-info 1.3
extra-info meek B1AA2EF6435720FB710B9F94872691A2DCD49E78
master-key-ed25519 szOJ8LUSKcuyowSG72Q3Ge6Xu7bMOyeyb+tCCBgD/zc
published 2022-12-03 17:35:45
...
bridge-ips 
bridge-ip-versions 
bridge-ip-transports meek=22464
...

The answer of how the metrics are calculated is published here: https://metrics.torproject.org/reproducible-metrics.html#bridge-users

I have noticed the meek bridge extra info doesn't always contain the dirreq-stats-end line which is checked in this condition:

if desc.dir_stats_end is not None \
        and desc.published - desc.dir_stats_end < datetime.timedelta(days = 7) \
        and datetime.timedelta(seconds = desc.dir_stats_interval) < datetime.timedelta(seconds = 90000):
        resp_ok = max(0, desc.dir_v3_responses[stem.descriptor.extrainfo_descriptor.DirResponse.OK] - 4)

The dirreq_stats_end field is checked to make sure that the contained statistics end time is more than 1 week older than the descriptor publication time in the "published" line.

Also statistics with an interval length other than 1 day are skipped. In metrics the we set the threshold higher, because some descriptors have an interval a few seconds larger than 86400.

So if the dirreq_stats_end line is not parsed, the descriptor is discarded. That's probably why we are estimating way less users.

Why isn't the line included?

assigned to @hiro

It's going up! Now metrics says 3000: https://metrics.torproject.org/userstats-bridge-transport.html?start=2022-11-01&end=2022-12-09&transport=meek

@HackerNCoder I am going to look at the descriptors again and check if anything has changed. Meanwhile could you let me know if anything has changed on your side?

The only thing that changed was that I installed tor-geoipdb (or whatever the package is called) since it was complaining about that.

...

On 12/12/22 16:08, Hiro (@hiro) wrote:

Hiro commented:

@HackerNCoder I am going to look at the descriptors again and check if anything has changed. Meanwhile could you let me know if anything has changed on your side?

-- HackerNCoder Translating queer games at the Queerscriptors Contracting for the Tor Project to work on anti-censorship

@HackerNCoder I can confirm the line: dirreq-stats-end 2022-12-09 19:08:32 (86400 s) is now present.

Unless this is going to be fixed as Metrics bug - I guess it isn't and dip will persist - there should be a timeline event.

added Doing Q4 labels

Added a timeline entry timeline@6103e7ad

closed

mentioned in commit timeline@40149597

Meek metrics says 12 users at best, tor regularly sees 5000+

Designs

Child items 0

Activity