Trac issueshttps://gitlab.torproject.org/legacy/trac/-/issues2020-06-13T18:10:57Zhttps://gitlab.torproject.org/legacy/trac/-/issues/9523DocTor doesn't track that tor26 is a bandwidth authority2020-06-13T18:10:57ZDamian JohnsonDocTor doesn't track that tor26 is a bandwidth authorityHi Karsten. While adding DocTor's BandwidthScannerResultsMissing check I realized that tor26 includes measured bandwidths. Probably just need to add it to the enumeration at...
https://gitweb.torproject.org/doctor.git/blob/HEAD:/src/org...Hi Karsten. While adding DocTor's BandwidthScannerResultsMissing check I realized that tor26 includes measured bandwidths. Probably just need to add it to the enumeration at...
https://gitweb.torproject.org/doctor.git/blob/HEAD:/src/org/torproject/doctor/Checker.java#l397
For my part I'm having my scanner both report missing and extra authorities...
https://gitweb.torproject.org/atagar/tor-utils.git/commitdiff/8c097717223dbcd8294771f8845bf17b8f503384
Cheers! -Damianhttps://gitlab.torproject.org/legacy/trac/-/issues/9197Make consensus-health message about missing signatures clearer2020-06-13T18:10:57ZKarsten LoesingMake consensus-health message about missing signatures clearerweasel was rightly confused about the message: "NOTICE: The consensuses downloaded from the following authorities are missing signatures from previously voting authorities: Faravahar, gabelmoo, maatuska, moria1, tor26, turtles"
This mes...weasel was rightly confused about the message: "NOTICE: The consensuses downloaded from the following authorities are missing signatures from previously voting authorities: Faravahar, gabelmoo, maatuska, moria1, tor26, turtles"
This message was because urras didn't sign the consensus, not because one of the listed authorities did something wrong. Rephrase this.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/8462Implement new bridge user counting algorithm (was Why don't .ir bridge users ...2020-06-13T18:13:12ZRoger DingledineImplement new bridge user counting algorithm (was Why don't .ir bridge users fall off when Tor gets censored by DPI?)https://metrics.torproject.org/users.html?graph=direct-users&start=2012-12-01&end=2013-03-13&country=ir&events=points#direct-users sees a pretty clear fall-off. But https://metrics.torproject.org/users.html?graph=bridge-users&start=2012-...https://metrics.torproject.org/users.html?graph=direct-users&start=2012-12-01&end=2013-03-13&country=ir&events=points#direct-users sees a pretty clear fall-off. But https://metrics.torproject.org/users.html?graph=bridge-users&start=2012-12-01&end=2013-03-13&country=ir#bridge-users doesn't show any sort of drop-off.
Similarly, https://metrics.torproject.org/users.html?graph=direct-users&start=2012-12-01&end=2013-03-13&country=sy&events=points#direct-users sees very clear blocking of Tor users, and I assume it's by DPI also (actually I've been assuming it's all SSL), but then the bridge graph doesn't look at all like this: https://metrics.torproject.org/users.html?graph=bridge-users&start=2012-12-01&end=2013-03-13&country=sy#bridge-users
Is our new bridge user counting algorithm wrong in some way? Or are the connections arriving at the bridges, and they're counting them, even if the SSL handshake doesn't finish? Or does it finish but just not proceed past that?
At this point I'm avoiding showing the bridge graphs to funders, since they make no sense to me.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/8393Don't warn when dirauths run a version that's too new2020-06-13T18:10:56ZSebastian HahnDon't warn when dirauths run a version that's too newI'm running master on gabelmoo currently, and consensus-health is complaining about an unrecommended version. The check should probably be adapted to only list versions which are too old, like Tor does when deciding whether to warn or ju...I'm running master on gabelmoo currently, and consensus-health is complaining about an unrecommended version. The check should probably be adapted to only list versions which are too old, like Tor does when deciding whether to warn or just give a notice. I'd prefer not to be notified at all in such a case, tho.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/6989maatuska wrongly reported as missing from the consensus2020-06-13T18:10:55ZLinus Nordberglinus@torproject.orgmaatuska wrongly reported as missing from the consensusmaatuska changed keys yesterday and has since then been reported as
not part of consensus a couple of times.maatuska changed keys yesterday and has since then been reported as
not part of consensus a couple of times.https://gitlab.torproject.org/legacy/trac/-/issues/6904consensus-health message for ‘conflicting or invalid consensus parameters’ is...2020-06-13T18:10:55ZRobert Ransomconsensus-health message for ‘conflicting or invalid consensus parameters’ is unclear```
2012-09-20 20:02:14 <nsa> ^Bor^B: [consensus-health] NOTICE: The following directory authorities set conflicting or invalid consensus parameters: maatuska CircuitPriorityHalflifeMsec=30000 UseOptimisticData=1 bwauthpid=1, moria1 Circ...```
2012-09-20 20:02:14 <nsa> ^Bor^B: [consensus-health] NOTICE: The following directory authorities set conflicting or invalid consensus parameters: maatuska CircuitPriorityHalflifeMsec=30000 UseOptimisticData=1 bwauthpid=1, moria1 CircuitPriorityHalflifeMsec=30000 UseOptimisticData=1 bwauthpid=1, turtles CircuitPriorityHalflifeMsec=30000 UseOptimisticData=1 bwauthpid=1
```
In this message, the consensus health checker is complaining because it thinks the ‘`UseOptimisticData`’ parameter is invalid.
This type of message should be split into two messages:
* one for an ‘invalid’ parameter (usually what this actually means is that consensus-health hasn't been told that a new parameter is valid yet), and
* one for an actual conflict between parameter values (in which case, it should show the name of the parameter with conflicting values, the values specified by each dirauth which votes on it, and the value produced in the consensus).Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/6471Design file format and Python/Java library for multiple GeoIP or AS databases2020-06-13T18:10:54ZKarsten LoesingDesign file format and Python/Java library for multiple GeoIP or AS databasesWhenever we look up relay IP addresses in GeoIP or AS databases covering more than a few months, we can't just use a single database. There's a reason why these databases change over time. An IP address that is resolved to a certain co...Whenever we look up relay IP addresses in GeoIP or AS databases covering more than a few months, we can't just use a single database. There's a reason why these databases change over time. An IP address that is resolved to a certain country or AS may have belonged to a different country or AS a year ago. What we should really do is use multiple databases and look up the IP address in the database that was most recent at the given time.
How about we design a file format for multiple GeoIP or AS databases and write a Python library for using it? The library should allow us to:
- Convert a given GeoIP or AS database from Maxmind or using a similar format to our format.
- Merge a new single GeoIP or AS database into our format.
- Look up the country code/name or AS number/name of an IP address on a given date.
I'd say that non-functional requirements are as follows, from most important to least important: lookup speed, lookup speed, lookup speed, memory consumption, file size, library code complexity.
I found these archives of GeoIP and AS databases:
http://geolite.maxmind.com/download/geoip/database/GeoLiteCity_CSV/
http://www.maxmind.com/download/geoip/database/asnum/old/Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/6329Write script to aggregate relay weights by country, AS, or relay family2020-06-13T18:10:53ZKarsten LoesingWrite script to aggregate relay weights by country, AS, or relay familylunar is interested in adding pie charts about relays/capacity per AS network and similar statistics to Atlas. In order to do this, Onionoo should provide these data in a convenient way. These data would be based on the very last conse...lunar is interested in adding pie charts about relays/capacity per AS network and similar statistics to Atlas. In order to do this, Onionoo should provide these data in a convenient way. These data would be based on the very last consensus and would not incorporate any history.
We should probably wait for the Python Onionoo to implement this and not extend the Java Onionoo. Setting priority to minor.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/6261consensus-health should learn to diff the recommended version sets2020-06-13T18:10:52ZKarsten Loesingconsensus-health should learn to diff the recommended version sets```
18:15:05 < rransom> karsten: consensus-health should learn to diff the recommended version sets.
```
I agree.```
18:15:05 < rransom> karsten: consensus-health should learn to diff the recommended version sets.
```
I agree.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/6163Add graph links to the posts in tor-censorship-events2020-06-13T18:10:52ZGeorge KadianakisAdd graph links to the posts in tor-censorship-eventsAdd links to the graphs in metrics.tpo for every censorship event in the tor-censorship-events mailing list.Add links to the graphs in metrics.tpo for every censorship event in the tor-censorship-events mailing list.https://gitlab.torproject.org/legacy/trac/-/issues/6159Censorship detector script should output full country names2020-06-13T18:10:51ZRuna SandvikCensorship detector script should output full country namesThe censorship detector script should output full country names, not just the country codes.The censorship detector script should output full country names, not just the country codes.George KadianakisGeorge Kadianakishttps://gitlab.torproject.org/legacy/trac/-/issues/5651Annotation header with descriptor types2020-06-13T18:10:51ZDamian JohnsonAnnotation header with descriptor typesIt would be helpful to parsers for the first line of metrics descriptors to contain the descriptor type and version. Moving this discussion from an email thread...
> Speaking of versioning and such, I would really appreciate it if all o...It would be helpful to parsers for the first line of metrics descriptors to contain the descriptor type and version. Moving this discussion from an email thread...
> Speaking of versioning and such, I would really appreciate it if all of these documents came with a header saying what the document type and version is. At present I need to guess based on the filename and first line of content which is both hacky and will break when new versions roll out. Actually, to differentiate between bridge and relay descriptors I need to rely on the '10.x.x.x' scrubbing... :/
Thoughts on how we should format it?
Cheers! -DamianKarsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5637Ignore carriage returns when parsing descriptors2020-06-13T18:10:50ZKarsten LoesingIgnore carriage returns when parsing descriptorsThere are some (very old) server descriptors containing carriage returns in their contact line, more precisely in a full PGP key block. I'm attaching one such descriptor to this ticket.
It's unclear if carriage returns are permitted by...There are some (very old) server descriptors containing carriage returns in their contact line, more precisely in a full PGP key block. I'm attaching one such descriptor to this ticket.
It's unclear if carriage returns are permitted by dir-spec.txt. It does say `"ArgumentChar ::= any printing ASCII character except NL."`, but we know that Tor accepts non-ASCII characters in contact or platform lines, too. So, I guess the only non-permitted character is NL.
In metrics-lib, we parse descriptors using `BufferedReader.readLine()` which treats `\n`, `\r`, and `\r\n` all the same. We may have to write our own `readLine()` replacement that only accepts `\n` as line end.
This may also affect stem's descriptor parser.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4982Extend the metrics descriptor library2020-06-13T18:10:49ZKarsten LoesingExtend the metrics descriptor libraryAndrew and I agreed on making "Extend the metrics descriptor library to parse all metrics data formats and prepare it for use in the metrics data processor and metrics website" a contract deliverable for June 2012. Adding as sponsor Z ...Andrew and I agreed on making "Extend the metrics descriptor library to parse all metrics data formats and prepare it for use in the metrics data processor and metrics website" a contract deliverable for June 2012. Adding as sponsor Z deliverable, because it's not directly funded. Aiming for March 31, 2012.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4878Indicate severities of consensus-health messages2020-06-13T18:10:49ZRobert RansomIndicate severities of consensus-health messages```
2012-01-07 18:03:08 <nsa> \^Bor\^B: [consensus-health] The consensuses downloaded from the following authorities are missing signatures from previously voting authorities: dizum, gabelmoo, maatuska, moria1, tor26, urras
2012-01-07 18...```
2012-01-07 18:03:08 <nsa> \^Bor\^B: [consensus-health] The consensuses downloaded from the following authorities are missing signatures from previously voting authorities: dizum, gabelmoo, maatuska, moria1, tor26, urras
2012-01-07 18:03:08 <nsa> \^Bor\^B: [consensus-health] The following directory authorities are not reporting bandwidth scanner results: ides
2012-01-07 18:03:09 <nsa> \^Bor\^B: [consensus-health] The following directory authorities did not return a consensus within a timeout of 60 seconds: dannenberg, ides
2012-01-08 00:11:05 <nsa> \^Bor\^B: [consensus-health] The consensuses downloaded from the following authorities are missing signatures from previously voting authorities: dizum, gabelmoo, moria1, urras
2012-01-08 00:11:05 <nsa> \^Bor\^B: [consensus-health] The consensuses published by the following directory authorities are more than 1 hour old and therefore not fresh anymore: dizum, gabelmoo, moria1, urras
2012-01-08 00:11:07 <nsa> \^Bor\^B: [consensus-health] The following directory authorities did not return a consensus within a timeout of 60 seconds: dannenberg, ides, maatuska, tor26
2012-01-08 18:03:10 <nsa> \^Bor\^B: [consensus-health] The consensuses downloaded from the following authorities are missing signatures from previously voting authorities: dizum, gabelmoo, moria1, tor26, urras
2012-01-08 18:03:10 <nsa> \^Bor\^B: [consensus-health] The following directory authorities did not return a consensus within a timeout of 60 seconds: dannenberg, ides, maatuska
2012-01-08 18:03:11 <nsa> \^Bor\^B: [consensus-health] The following directory authorities set conflicting or invalid consensus parameters: gabelmoo CircuitPriorityHalflifeMsec=30000 circwindow=1000 refuseunknownexits=1, moria1 CircuitPriorityHalflifeMsec=30000 cbtnummodes=3 cbtquantile=80 circwindow=1000 refuseunknownexits=1
2012-01-08 18:03:13 <nsa> \^Bor\^B: [consensus-health] We're missing votes from the following directory authorities: dannenberg, ides
```
Quick! Which of these sets of messages tells us that the network is about to go down in flames?Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4713Review new consensus-health check implementation2020-06-13T18:10:48ZKarsten LoesingReview new consensus-health check implementationIn the past few weeks I rewrote almost the entire consensus-health check code. The download and parsing logic is now part of metrics-lib. The new code focuses mainly on two things: create an HTML output and write warnings to files.
I ...In the past few weeks I rewrote almost the entire consensus-health check code. The download and parsing logic is now part of metrics-lib. The new code focuses mainly on two things: create an HTML output and write warnings to files.
I think the code is now ready for a review. I would very much appreciate any comments on code style, documentation fixes, refactoring suggestions, potential bugs, real bugs, etc. A patch with comments that I can work off would be great!
The code is here: https://gitweb.torproject.org/doctor.git
Thanks!Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4661When metrics fetches a consensus from each authority, remember fetch duration2020-06-13T18:10:48ZRoger DingledineWhen metrics fetches a consensus from each authority, remember fetch durationOn #4250 it looks like "the consensus-health checker now downloads the consensus from all authorities". Great. Next we should remember statistics about how long it took to fetch from each authority, and see if the numbers are consistentl...On #4250 it looks like "the consensus-health checker now downloads the consensus from all authorities". Great. Next we should remember statistics about how long it took to fetch from each authority, and see if the numbers are consistently (or inconsistently) uglier for some authorities than others.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4250metrics consensus health should check the consensus from each authority2020-06-13T18:10:47ZRoger Dingledinemetrics consensus health should check the consensus from each authorityWhile doing a training yesterday, my poor os x user started tbb and it fetched a consensus that had two signatures on it -- one from gabelmoo-legacy, and one from moria1-legacy.
I didn't catch which authority gave it to her.
Several mi...While doing a training yesterday, my poor os x user started tbb and it fetched a consensus that had two signatures on it -- one from gabelmoo-legacy, and one from moria1-legacy.
I didn't catch which authority gave it to her.
Several minutes later, Tor tried another authority and bootstrapped from there.
While we're checking network health, we should fetch a consensus from each authority and see if it is recent enough and has enough signatures on it.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/3233Make the VisiTor scripts accept more web log formats2020-06-13T18:10:47ZKarsten LoesingMake the VisiTor scripts accept more web log formatsThe [VisiTor](https://metrics.torproject.org/tools.html#visitor) scripts (both Java and Python) parse a web server log and the exit list archives to tell how many of the requests come from Tor users. They currently require Apache's comb...The [VisiTor](https://metrics.torproject.org/tools.html#visitor) scripts (both Java and Python) parse a web server log and the exit list archives to tell how many of the requests come from Tor users. They currently require Apache's combined log format (`"%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\""`) as input. If users have a different format, they need to transform their log to said format before passing it to VisiTor.
We should also support Apache's common log format (`"%h %l %u %t \"%r\" %>s %b"`). This format doesn't have the User-agent string and can therefore not detect Torbutton users. But knowing whether users connect via the Tor network, with or without Torbutton, can still be a useful result.
Are there further commonly used web log formats that we should support?
Setting priority to minor, because I haven't heard of many VisiTor users lately.https://gitlab.torproject.org/legacy/trac/-/issues/2928Mark SVN repos for archive-related tools as moved to Git2020-06-13T18:10:46ZRobert RansomMark SVN repos for archive-related tools as moved to GitThe [Subversion repository for Tor archive-related tools](https://svn.torproject.org/svn/projects/archives/trunk/) contains old versions of the bridge descriptor sanitizer and ExoneraTor. These directories' contents should be removed an...The [Subversion repository for Tor archive-related tools](https://svn.torproject.org/svn/projects/archives/trunk/) contains old versions of the bridge descriptor sanitizer and ExoneraTor. These directories' contents should be removed and replaced with a README file telling users where to find the Git repository they live in now.Karsten LoesingKarsten Loesing