Trac issueshttps://gitlab.torproject.org/legacy/trac/-/issues2020-06-13T18:12:44Zhttps://gitlab.torproject.org/legacy/trac/-/issues/3889Analyze metrics database configuration for possible performance bottleneck2020-06-13T18:12:44ZKarsten LoesingAnalyze metrics database configuration for possible performance bottleneckThe metrics database on yatei is much slower than a freshly set up database on Sebastian's laptop. To some extent this is understandable, because things deteriorate over time. But relay-search queries that take seconds or less on Sebas...The metrics database on yatei is much slower than a freshly set up database on Sebastian's laptop. To some extent this is understandable, because things deteriorate over time. But relay-search queries that take seconds or less on Sebastian's laptop shouldn't take minutes to complete on yatei. We should take a look if we can tweak yatei's database configuration to remove some bottlenecks.
Possible candidates for tweaks are auto-vacuuming intervals, indexes, backups, etc.
How can we compare database configurations? Can we put up yatei's PostgreSQL configuration anywhere public? Also, what PostgreSQL statistics should we compare?Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/3977metrics website package graphs stopped updating in August2020-06-13T17:49:59ZAndrew Lewmanmetrics website package graphs stopped updating in Augusthttps://metrics.torproject.org/packages.html The graphs for overall and each individual country appear to have stopped in late August.https://metrics.torproject.org/packages.html The graphs for overall and each individual country appear to have stopped in late August.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4005Test what happens when we delete a version from Trac2011-09-12T14:57:21ZKarsten LoesingTest what happens when we delete a version from TracThis is a test ticket to see what happens when we delete a really old version from Trac.
Setting version to `"Tor: 0.0.8.1"` and deleting that version afterwards. Let's see what that does to this ticket.
(Sorry for the noise.)This is a test ticket to see what happens when we delete a really old version from Trac.
Setting version to `"Tor: 0.0.8.1"` and deleting that version afterwards. Let's see what that does to this ticket.
(Sorry for the noise.)Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4030Analyze whether we can detect that a bridge is blocked by looking at the stat...2020-06-13T17:46:57ZKarsten LoesingAnalyze whether we can detect that a bridge is blocked by looking at the stats the bridge reportsWe promised this to sponsor E for today. I'm creating a ticket, so that I have a ticket number for metrics-tasks.git.We promised this to sponsor E for today. I'm creating a ticket, so that I have a ticket number for metrics-tasks.git.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4086Compare performance of TokenBucketRefillInterval params in simulated network2020-06-13T17:47:00ZRoger DingledineCompare performance of TokenBucketRefillInterval params in simulated networkOnce we merge #3630, we will start refilling our token buckets 100 times a second rather than once a second.
In theory, this feature should magnify the effect of the CircuitPriorityHalflife config option, as well as generally smoothing ...Once we merge #3630, we will start refilling our token buckets 100 times a second rather than once a second.
In theory, this feature should magnify the effect of the CircuitPriorityHalflife config option, as well as generally smoothing our TCP flows between relays.
It could have surprising adverse effects though, such as making all of our network writes be one cell long (plus IP, TCP, TLS, and MTU overhead).
So the task here is to run simulated Tor networks with various combinations of these config options, at various bandwidth rates, and see if there are sweet spots, surprising bad combinations, etc.
For example, should the TokenBucketRefillInterval value be a function of our available bandwidth?Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4250metrics consensus health should check the consensus from each authority2020-06-13T18:10:47ZRoger Dingledinemetrics consensus health should check the consensus from each authorityWhile doing a training yesterday, my poor os x user started tbb and it fetched a consensus that had two signatures on it -- one from gabelmoo-legacy, and one from moria1-legacy.
I didn't catch which authority gave it to her.
Several mi...While doing a training yesterday, my poor os x user started tbb and it fetched a consensus that had two signatures on it -- one from gabelmoo-legacy, and one from moria1-legacy.
I didn't catch which authority gave it to her.
Several minutes later, Tor tried another authority and bootstrapped from there.
While we're checking network health, we should fetch a consensus from each authority and see if it is recent enough and has enough signatures on it.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4255Simulate how BridgeDB can identify stable bridges2020-06-13T17:47:03ZKarsten LoesingSimulate how BridgeDB can identify stable bridgesWhile working on the Guard/Stable flag analysis (#2911), I came up with a new notion of bridge stability: "A stable bridge relay should be running on the same IP address a few days after a client learns about the bridge and should be ava...While working on the Guard/Stable flag analysis (#2911), I came up with a new notion of bridge stability: "A stable bridge relay should be running on the same IP address a few days after a client learns about the bridge and should be available most of the time in the upcoming weeks."
The idea is that there are two cases when a bridge that BridgeDB told us becomes useless: a) when the bridge isn't around much or b) when the bridge changes its IP address or port.
I suggest to make Tonga calculate two metrics for all bridges, similar to how the directory authorities calculate them for relays: a) Weighted Fractional Uptime (WFU) and b) Mean Time Between Address Change (MTBAC). Tonga then assigns a StableBridge flag to bridges with a WFU at least as high as the median WFU and a MTBAC at least as high as the median MTBAC. BridgeDB would give out at least one bridge with the StableBridge flag (instead of doing so for the Stable flag).
I'm going to evaluate this new flag by looking at the sanitized bridge descriptor archive of, say, the first half of 2011. I should be able to reconstruct when a bridge failed (dropped out of the bridge network status) and when it changed its address or port. The latter part may be difficult, because we're changing the key of the keyed hash function once per month. The result is going to be sets of bridges that would have gotten the StableBridge flag at any given time in the analysis interval. Then I'm going to look up the future WFU and the time until next address change (TUNAC). Ideally, most of the bridges with the StableBridge would have a WFU of 90% or more and a TUNAC of a few days or more.
Setting priority to major, because this is a deliverable for November 1, 2011.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4314Make relay-search accept fingerprints with the $ prefix2020-06-13T18:12:44ZKarsten LoesingMake relay-search accept fingerprints with the $ prefixThe current response to a query for a fingerprint with the $ prefix is "Sorry, I didn't understand your query." We should accept parameters with the $ prefix as fingerprints. Suggested by velope.The current response to a query for a fingerprint with the $ prefix is "Sorry, I didn't understand your query." We should accept parameters with the $ prefix as fingerprints. Suggested by velope.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4387How much GB does a useful relay push per week?2020-06-13T17:47:04ZRuna SandvikHow much GB does a useful relay push per week?As part of #3574, we estimated that a useful bridge should push around 10GB a week. Do you have similar numbers for a relay?
This is related to the Tor Cloud, so I'm choosing that as the component. Feel free to change it to something else.As part of #3574, we estimated that a useful bridge should push around 10GB a week. Do you have similar numbers for a relay?
This is related to the Tor Cloud, so I'm choosing that as the component. Feel free to change it to something else.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4406DB errors from relay.html2020-06-13T18:12:45ZDamian JohnsonDB errors from relay.htmlMoving this to a ticket so it doens't get lost in the irc backlog.
12:52 < atagar> karsten: "Warning: We experienced an unknown database problem while looking up the relay with fingerprint 29290506633292FE318DDA9EA93DE8F452B50C17. If th...Moving this to a ticket so it doens't get lost in the irc backlog.
12:52 < atagar> karsten: "Warning: We experienced an unknown database problem while looking up the relay with fingerprint 29290506633292FE318DDA9EA93DE8F452B50C17. If this problem persists, please let us know!"
12:52 < atagar> from https://metrics.torproject.org/relay.html?fingerprint=29290506633292FE318DDA9EA93DE8F452B50C17
13:00 < armadev> atagar: i also reported the same issue to karsten yesterday. it has to do with his db overload issues.
13:00 < armadev> atagar: current plan is that there is no fix. maybe he should fix relay.html to not make you think it'll work. or maybe that's not worth it.
13:03 < atagar> at the very least he should change the message unless he wants to keep hearing about it ;)Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4440Attempt an implementation of the relay-search database using MongoDB or CouchDB2020-06-13T17:47:05ZKarsten LoesingAttempt an implementation of the relay-search database using MongoDB or CouchDBOur current relay-search function takes forever to return results. There's #2922 for improving the database schema to better support searching for single relays. That ticket assumes that we'll continue to use PostgreSQL.
Today I looke...Our current relay-search function takes forever to return results. There's #2922 for improving the database schema to better support searching for single relays. That ticket assumes that we'll continue to use PostgreSQL.
Today I looked into MongoDB for the web server log analysis, and I wonder if MongoDB or CouchDB might be an alternative for implementing the relay-search database.
Every consensus or descriptor could be represented as a document with references to other documents. Indexes could make typical search queries fast. We don't need complicated Map/Reduce functions, because we're only searching and looking up data, not aggregating anything. (That's also the reason why I think this is worth trying out---replacing the metrics database that aggregates statistics with MongoDB/CouchDB may not make as much sense.) Maybe we should run a simple comparison of the new PostgreSQL database that ExoneraTor uses, and that is highly optimized for searches, to an implementation using MongoDB or CouchDB.
I don't know if such a solution will perform better than a PostgreSQL-based solution. I think we should try to find out.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4458Make sanitized web server logs available2020-06-13T17:49:59ZKarsten LoesingMake sanitized web server logs availableWe have been discussing sanitizing and publishing our web server logs for quite a while. The idea is to remove all potentially sensitive parts from the logs, publish them in monthly tarballs on the metrics website, and analyze them for ...We have been discussing sanitizing and publishing our web server logs for quite a while. The idea is to remove all potentially sensitive parts from the logs, publish them in monthly tarballs on the metrics website, and analyze them for top visited pages, top downloaded packages, etc. See #1641 and #2489 for details.
As of October 18, 2011, the sanitized logs for all of 2010 are available for download:
https://lists.torproject.org/pipermail/tor-dev/2011-October/002981.html
The final step will be to sanitize logs whenever the web server sync them to a metrics machine and to automate making tarballs.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4499Investigate scaling points to handle more bridges2020-06-13T17:47:11ZRuna SandvikInvestigate scaling points to handle more bridgesThe current bridge infrastructure relies on a central bridge authority to collect, distribute, and publish bridge relay descriptors. We believe the current infrastructure can handle up to 10,000 bridges.
The scaling points involve the d...The current bridge infrastructure relies on a central bridge authority to collect, distribute, and publish bridge relay descriptors. We believe the current infrastructure can handle up to 10,000 bridges.
The scaling points involve the database of descriptors, the metrics portal and its ability to handle this many descriptors for analysis, and the reachability testing part of the code for the bridge authority. We should investigate scaling points to handle more than 10,000 bridge descriptors.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4507Identify main problems with the bridge infrastructure wrt scaling to 10k bridges2020-06-13T17:47:13ZRuna SandvikIdentify main problems with the bridge infrastructure wrt scaling to 10k bridgesWe are going to investigate scaling points to handle more bridges (#4499). Before we can do that, we need to identify the main problem areas with the current infrastructure.
Karsten has previously mentioned the he is mostly worried abo...We are going to investigate scaling points to handle more bridges (#4499). Before we can do that, we need to identify the main problem areas with the current infrastructure.
Karsten has previously mentioned the he is mostly worried about the bridge authority, but that the BridgeDB and metrics would need to be extended as well.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4612Put top-10 relays by bandwidth history back on metrics website2020-06-13T18:12:46ZKarsten LoesingPut top-10 relays by bandwidth history back on metrics websiteI just removed the table listing the "top-10 relays by bandwidth history" from the [metrics website](https://metrics.torproject.org/network.html) in commit [c7675d0](https://gitweb.torproject.org/metrics-web.git/commit/c7675d04f7cb861f65...I just removed the table listing the "top-10 relays by bandwidth history" from the [metrics website](https://metrics.torproject.org/network.html) in commit [c7675d0](https://gitweb.torproject.org/metrics-web.git/commit/c7675d04f7cb861f65877d9162da9572eade7242). It was a list of relay fingerprints and written bytes in a given time frame. The list for September 1 to November 30, 2011 was:
Fingerprint Bandwidth history
24b1f63f7df9f85d711864811cc401be5eb5fb9a 103.92 TiB
7dcb5313b9541dd29c94bfde0adf91dc91d2cfe9 102.18 TiB
db8c6d8e0d51a42bdda81a9b8a735b41b2cf95d1 101.05 TiB
253dff1838a2b7782be7735f74e50090d46ca1bc 99.65 TiB
b93dcc053d7f0472bb17a4514e06fe76d9fb714b 98.17 TiB
6daa8317ba9ff4b4cad33588ad780ed65ba8af94 94.37 TiB
ff986a4573185a6912a0ff28335210cc05ce00de 74.14 TiB
2624ae0466bd02afaf3f263d4361d79abe0e7e05 72.84 TiB
4186509c707e96b77b51a76f8294d7e22ff52c61 62.36 TiB
2ffcceace86d8ca306d069386a65359bcee8d67a 61.50 TiB
The fingerprints were links to the relay pages showing more details. Also, there were input boxes for start and end date to customize the table content.
The reason why I took this list out is that updating this table for a new time interval took 2--5 minutes. That's unacceptable. This happened at least four times a day when the cached table data got stale or whenever a user entered a different start or end date of the time interval. This doesn't just delay page load times, but also keeps the database busy and affects other users.
If we want this table back, we'll have to pre-calculate some values.
I'm setting the priority to minor, because I don't think many people looked at this table at all.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4661When metrics fetches a consensus from each authority, remember fetch duration2020-06-13T18:10:48ZRoger DingledineWhen metrics fetches a consensus from each authority, remember fetch durationOn #4250 it looks like "the consensus-health checker now downloads the consensus from all authorities". Great. Next we should remember statistics about how long it took to fetch from each authority, and see if the numbers are consistentl...On #4250 it looks like "the consensus-health checker now downloads the consensus from all authorities". Great. Next we should remember statistics about how long it took to fetch from each authority, and see if the numbers are consistently (or inconsistently) uglier for some authorities than others.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4673Partition the 67,000,000 row statusentry table in the metrics database2020-06-13T18:12:46ZKarsten LoesingPartition the 67,000,000 row statusentry table in the metrics databaseThomas Termin suggested splitting the huge statusentry table in the metrics database into multiple tables to solve some of our metrics website/database performance problems. Tim Wilde chimed in saying they're already doing that for larg...Thomas Termin suggested splitting the huge statusentry table in the metrics database into multiple tables to solve some of our metrics website/database performance problems. Tim Wilde chimed in saying they're already doing that for large tables down to the hour level.
There was some more discussion about splitting the whole table covering 3+ years of data into 36+ month tables and adding a new month table every month. Another suggestion was to move old data into a history table of some kind using a cron-job-like stored procedure. I later saw Tim explain something about using a year table, month tables, etc. down to hour tables and deciding in the application which table(s) to query. Basically, there was some discussion whether to do the splitting and merging in the database or in the application.
A good next step would be to look at the PostgreSQL documentation for [partitioning tables](http://www.postgresql.org/docs/8.4/static/ddl-partitioning.html). I'm also going to look more at the schema and Java code to find the places which are affected by such a change.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4686Decide whether we want to archive consensus download times2020-06-13T17:50:00ZKarsten LoesingDecide whether we want to archive consensus download timesThe consensus-health checker now memorizes consensus download times and displays the last 7 days in a [table](https://metrics.torproject.org/consensus-health.html#downloadstats).
Internally, we're just appending raw download times to a ...The consensus-health checker now memorizes consensus download times and displays the last 7 days in a [table](https://metrics.torproject.org/consensus-health.html#downloadstats).
Internally, we're just appending raw download times to a file. If we want to archive them and make them publicly available, we should define some data format, specify/describe the format on the [Data Formats](https://metrics.torproject.org/formats.html) page, and put up (monthly) tarballs containing the download times.
Let's revisit this task in a few months (e.g., February 2012?) when we have some data. We might even run a quick analysis on the data first to see if there are interesting trends.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4687Provide metrics data from the last 3--7 days via rsync2020-06-13T17:50:00ZKarsten LoesingProvide metrics data from the last 3--7 days via rsyncmetrics-db provides compressed tarballs of its collected and sanitized files via rsync, so that others can copy them for their analyses or services. But these tarballs are only updated once per day, and updating them more often has perf...metrics-db provides compressed tarballs of its collected and sanitized files via rsync, so that others can copy them for their analyses or services. But these tarballs are only updated once per day, and updating them more often has performance implications on the metrics-db host. Also, having others update their services from potentially very large monthly tarballs has performance issues, too.
Can we make the collected and sanitized descriptors from the last, say, 3 to 7 days available via rsync? For a point of reference, in the first 8.5 days of December, metrics-db collected and sanitized 178,000 files being 2.4G in size. Are those numbers totally crazy for rsync?
We could implement this by having a script copy new descriptors to a directory that is then made available via rsync. The same script would also delete files that are older than 3--7 days.
This ticket is part of decoupling metrics-db from metrics-web and other metrics services. Right now, metrics-web uses the metrics-db output using a symbolic link the file system. That means metrics-db and metrics-web rely on running on the same host. We should change that to allow others to run their own metrics-web or or similar service.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4695scumbag tor2011-12-11T15:48:09Zcypherpunksscumbag tory u no make tor fast?
y u make tor slow?
https://metrics.torproject.org/performance.html makes keanu sad and forever alone
most interesting guy in the world says, 'I don't always use tor, but when I do, I plan to lose an entire day to...y u no make tor fast?
y u make tor slow?
https://metrics.torproject.org/performance.html makes keanu sad and forever alone
most interesting guy in the world says, 'I don't always use tor, but when I do, I plan to lose an entire day to browse one site'
u mad bro?Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4713Review new consensus-health check implementation2020-06-13T18:10:48ZKarsten LoesingReview new consensus-health check implementationIn the past few weeks I rewrote almost the entire consensus-health check code. The download and parsing logic is now part of metrics-lib. The new code focuses mainly on two things: create an HTML output and write warnings to files.
I ...In the past few weeks I rewrote almost the entire consensus-health check code. The download and parsing logic is now part of metrics-lib. The new code focuses mainly on two things: create an HTML output and write warnings to files.
I think the code is now ready for a review. I would very much appreciate any comments on code style, documentation fixes, refactoring suggestions, potential bugs, real bugs, etc. A patch with comments that I can work off would be great!
The code is here: https://gitweb.torproject.org/doctor.git
Thanks!Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4776useful metrics network page graph: cdf of advertised bandwidth2020-06-13T18:12:47ZRoger Dingledineuseful metrics network page graph: cdf of advertised bandwidthThe median advertised bandwidth (minimum of the three published values in the descriptor) used to be 50KB, and now it's 100KB. What do the sides look like?
I think that's one useful way to characterize the Tor network: a cdf with advert...The median advertised bandwidth (minimum of the three published values in the descriptor) used to be 50KB, and now it's 100KB. What do the sides look like?
I think that's one useful way to characterize the Tor network: a cdf with advertised bandwidth on the x axis and "number of relays with that bandwidth or less" on the y axis.
Bonus points if you can pick a date and generate a cdf for it.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4809use less political country names on metrics pages2020-06-13T18:14:04ZRoger Dingledineuse less political country names on metrics pagesSome of the country names we use on the metrics pages make us sound very political.
Top examples include "Islamic Republic of Iran", "Democratic Republic of the Congo", "Democratic People's Republic of Korea".
We should replace these n...Some of the country names we use on the metrics pages make us sound very political.
Top examples include "Islamic Republic of Iran", "Democratic Republic of the Congo", "Democratic People's Republic of Korea".
We should replace these names with the simpler names that many people expect, in the case of Iran.
I'm not sure if "North Korea" is less political though, so there are still some landmines to navigate here.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4878Indicate severities of consensus-health messages2020-06-13T18:10:49ZRobert RansomIndicate severities of consensus-health messages```
2012-01-07 18:03:08 <nsa> \^Bor\^B: [consensus-health] The consensuses downloaded from the following authorities are missing signatures from previously voting authorities: dizum, gabelmoo, maatuska, moria1, tor26, urras
2012-01-07 18...```
2012-01-07 18:03:08 <nsa> \^Bor\^B: [consensus-health] The consensuses downloaded from the following authorities are missing signatures from previously voting authorities: dizum, gabelmoo, maatuska, moria1, tor26, urras
2012-01-07 18:03:08 <nsa> \^Bor\^B: [consensus-health] The following directory authorities are not reporting bandwidth scanner results: ides
2012-01-07 18:03:09 <nsa> \^Bor\^B: [consensus-health] The following directory authorities did not return a consensus within a timeout of 60 seconds: dannenberg, ides
2012-01-08 00:11:05 <nsa> \^Bor\^B: [consensus-health] The consensuses downloaded from the following authorities are missing signatures from previously voting authorities: dizum, gabelmoo, moria1, urras
2012-01-08 00:11:05 <nsa> \^Bor\^B: [consensus-health] The consensuses published by the following directory authorities are more than 1 hour old and therefore not fresh anymore: dizum, gabelmoo, moria1, urras
2012-01-08 00:11:07 <nsa> \^Bor\^B: [consensus-health] The following directory authorities did not return a consensus within a timeout of 60 seconds: dannenberg, ides, maatuska, tor26
2012-01-08 18:03:10 <nsa> \^Bor\^B: [consensus-health] The consensuses downloaded from the following authorities are missing signatures from previously voting authorities: dizum, gabelmoo, moria1, tor26, urras
2012-01-08 18:03:10 <nsa> \^Bor\^B: [consensus-health] The following directory authorities did not return a consensus within a timeout of 60 seconds: dannenberg, ides, maatuska
2012-01-08 18:03:11 <nsa> \^Bor\^B: [consensus-health] The following directory authorities set conflicting or invalid consensus parameters: gabelmoo CircuitPriorityHalflifeMsec=30000 circwindow=1000 refuseunknownexits=1, moria1 CircuitPriorityHalflifeMsec=30000 cbtnummodes=3 cbtquantile=80 circwindow=1000 refuseunknownexits=1
2012-01-08 18:03:13 <nsa> \^Bor\^B: [consensus-health] We're missing votes from the following directory authorities: dannenberg, ides
```
Quick! Which of these sets of messages tells us that the network is about to go down in flames?Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4887encoding problem with country list2020-06-13T18:12:48ZRoger Dingledineencoding problem with country listGo to https://metrics.torproject.org/users.html?graph=direct-users&country=st#direct-users and look at the name of the country in the drop-down box.
The same problem also happens with Ivory Coast.Go to https://metrics.torproject.org/users.html?graph=direct-users&country=st#direct-users and look at the name of the country in the drop-down box.
The same problem also happens with Ivory Coast.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4895relay-search should tell users data goes back to 2007-10 only2020-06-13T18:12:48ZSebastian Hahnrelay-search should tell users data goes back to 2007-10 onlyWhen looking at these two, it appears that moria1 started being around in October 2007. We should indicate that that's not quite true.
https://metrics.torproject.org/relay-search.html?search=moria1+2007-09
https://metrics.torproject.org...When looking at these two, it appears that moria1 started being around in October 2007. We should indicate that that's not quite true.
https://metrics.torproject.org/relay-search.html?search=moria1+2007-09
https://metrics.torproject.org/relay-search.html?search=moria1+2007-10Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4957Decide how to sanitize pluggable transport lines in bridge descriptors2020-06-13T17:50:01ZKarsten LoesingDecide how to sanitize pluggable transport lines in bridge descriptorsWe're providing [sanitized versions](https://metrics.torproject.org/formats.html#bridgedesc) of bridge descriptors in almost real-time on the [metrics website](https://metrics.torproject.org/data.html#bridgedesc) and via rsync.
Once we ...We're providing [sanitized versions](https://metrics.torproject.org/formats.html#bridgedesc) of bridge descriptors in almost real-time on the [metrics website](https://metrics.torproject.org/data.html#bridgedesc) and via rsync.
Once we enable bridges to include pluggable transport information in their server and/or extra-info descriptors, we need to come up with a way to sanitize the sensitive parts. We'll want to remove any keys contained in pluggable transport lines, okay. But maybe the fact that a bridge offers a specific pluggable transport is already sensitive? Maybe the fact that it offers _any_ pluggable transport is sensitive?
This problem came up in #3589. Bridge clients aren't supposed to learn about pluggable transports contained in the bridge's extra-info descriptor. Neither the bridge authority nor the bridge gives out extra-info descriptors. But once the client knows the bridge's server descriptor it can easily look up the _sanitized_ extra-info descriptor from the metrics archives. If we don't want the client to learn about the bridge's transports, we need to take that into account. If it helps, we can define new sanitizing rules for each pluggable transport there is.
So, there's a trade-off between revealing too much information and being able to analyze pluggable transport deployment. We'll probably want to run some analyses on pluggable transport deployment. We can only do that if the information is contained in the sanitized versions of bridge descriptors (because we don't use the original descriptors for analysis at all).Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4976"active metrics tickets" trac report broken2020-06-13T18:12:50ZRoger Dingledine"active metrics tickets" trac report brokenreport 21, aka https://trac.torproject.org/projects/tor/report/21 as linked from https://trac.torproject.org/projects/tor/report, shows no results.
If nobody uses it, maybe it should be taken out of the list? If somebody uses it, what for?report 21, aka https://trac.torproject.org/projects/tor/report/21 as linked from https://trac.torproject.org/projects/tor/report, shows no results.
If nobody uses it, maybe it should be taken out of the list? If somebody uses it, what for?Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4982Extend the metrics descriptor library2020-06-13T18:10:49ZKarsten LoesingExtend the metrics descriptor libraryAndrew and I agreed on making "Extend the metrics descriptor library to parse all metrics data formats and prepare it for use in the metrics data processor and metrics website" a contract deliverable for June 2012. Adding as sponsor Z ...Andrew and I agreed on making "Extend the metrics descriptor library to parse all metrics data formats and prepare it for use in the metrics data processor and metrics website" a contract deliverable for June 2012. Adding as sponsor Z deliverable, because it's not directly funded. Aiming for March 31, 2012.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/4988csv file containing average daily bridge users2020-06-13T18:12:51ZTraccsv file containing average daily bridge usersHi,
would it be possible to put up a seperate .csv file for average daily bridge-only users? The graph is already there, but the cvs on the users page has (only) direct+bridge user combined.
**Trac**:
**Username**: skepHi,
would it be possible to put up a seperate .csv file for average daily bridge-only users? The graph is already there, but the cvs on the users page has (only) direct+bridge user combined.
**Trac**:
**Username**: skepKarsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5047Implement basic usage statistics in obfsproxy2020-06-13T18:34:05ZKarsten LoesingImplement basic usage statistics in obfsproxyWe should implement some basic usage statistics in obfsproxy to learn about usage as long as Tor doesn't have support for obfsproxy statistics (#5040). Once Tor supports these statistics, the implementation in obfsproxy can be removed. ...We should implement some basic usage statistics in obfsproxy to learn about usage as long as Tor doesn't have support for obfsproxy statistics (#5040). Once Tor supports these statistics, the implementation in obfsproxy can be removed. Both Tor's and obfsproxy's statistics should be equivalent or at least easily comparable.
The idea is to have obfsproxy log incoming connections in a privacy-aware way and provide a simple script to convert these logs into a format that can be published without issues. Bridge operators can periodically run the script and send the output to the Tor developers who publish and analyze them. The implementation in obfsproxy should be quite simple in order not to break too much stuff. The conversion script should be dead simple, so that bridge operators can understand what's going on.
Here's a possible approach:
We want to count daily connections by country and daily unique IP addresses by country. Similar to other statistics in Tor, we want to aggregate data over 24-hour periods, resolve IP addresses to country codes, and round up frequencies to multiples of 8.
1. When obfsproxy starts, it does three things: a) generate a secret string S that it only keeps in memory; b) note the timestamp TS when it started; c) create a buffer B with a capacity of 100 log messages.
2. Whenever obfsproxy receives a client connection, it runs steps 3 to 5:
3. It checks whether at least 24 hours have passed since TS. If so, it flushes all log messages from buffer B, shuffles them, and appends them to a file on disk. It also increments TS in 24-hour steps until TS is not more than 24 hours in the past.
4. It checks whether B is full, i.e., contains 100 messages. If so, it flushes B and appends messages to a file on disk in random order.
5. It creates a new log message containing a) timestamp TS (which is NOT the current timestamp!), b) the country code of the connecting IP as resolved by a GeoIP database, c) the hashed IP address using secret S, i.e., `H(IP || S)` with a cryptographic hash function of the implementor's choice. An example log message would be `"2012-02-07 14:01:04 de 1234567890123456789012345678901234567890"`.
6. When obfsproxy stops, it does NOT flush the contents of B to disk. It forgets about S, possibly in a cryptographically secure manner.
The buffer has two functions here. First, it removes the original order of connections, which may still be meaningful if it contains connections from countries with few connections. Second, the buffer protects the timing of single client connections that occur when obfsproxy is terminated and restarted shortly after a 24-hour interval ends. The buffer size of 100 was arbitrarily chosen to avoid memory problems on heavily used bridges. Higher numbers are preferred, but if that makes things more complicated, 100 should be a large enough number.
The log messages still reveal too much information to be published. They shouldn't contain IP hashes, and frequencies still need to be rounded up to the next multiple of 8. The following bash script, which probably requires a lot more comments, converts a log message file into a format that can be published by bridge operators.
```
echo "Daily rounded total requests by country"
cut -d" " -f1-3 data | sort | uniq -c | \
awk '{printf "%s %s %s %d\n", $2, $3, $4, 8*(int(($1+7)/8))}'
echo "Daily rounded unique IPs by country"
sort data | uniq | cut -d" " -f1-3 | uniq -c | \
awk '{printf "%s %s %s %d\n", $2, $3, $4, 8*(int(($1+7)/8))}'
```
Note that the approach taken here was designed to keep the changes to obfsproxy small. Of course, we could implement everything in obfsproxy and write nice files that bridge operators can mail to the Tor devs directly. That would be an implementation similar to what Tor does for the various statistics. The buffered logging approach seemed to be a good compromise between not logging sensitive data and not adding too much code. Whether that is true is a question for the obfsproxy developers.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5188Move onionoo to tor's git2020-06-13T00:56:37ZDamian JohnsonMove onionoo to tor's gitSpoke with Karsten and it would be nice to start moving onionoo to tor's infrastructure (and yes, it's kinda related to #5187). Assigning to Karsten first for approval and a description for gitweb.
https://github.com/kloesing/OnionooSpoke with Karsten and it would be nice to start moving onionoo to tor's infrastructure (and yes, it's kinda related to #5187). Assigning to Karsten first for approval and a description for gitweb.
https://github.com/kloesing/OnionooKarsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5245Support multiple OR addresses2020-06-13T17:59:21ZKarsten LoesingSupport multiple OR addressesThe data formats allow including more than one OR address, but the server doesn't implement that yet. It doesn't need to, because neither the bridge authority nor the directory authority publish a relay's second OR address yet. But onc...The data formats allow including more than one OR address, but the server doesn't implement that yet. It doesn't need to, because neither the bridge authority nor the directory authority publish a relay's second OR address yet. But once that changes we should include all OR addresses in the documents.
As a side-effect, OR ports may then be port lists. Using an int to represent an OR port will not work anymore.
Once we support IPv6 OR addresses, we'll also want to support searching for IPv6 addresses.
(This was issue 5 in my GitHub repository.)Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5246Look into caching using HTTP headers2020-06-13T17:59:21ZKarsten LoesingLook into caching using HTTP headersThe current caching strategy is non-existent. We should find out which HTTP headers are most commonly used to tell clients and proxies how to cache our JSON responses. The caching strategy may rely on information that the server has ab...The current caching strategy is non-existent. We should find out which HTTP headers are most commonly used to tell clients and proxies how to cache our JSON responses. The caching strategy may rely on information that the server has about new data becoming available.
Arturo says there are two caching approaches: first, the server says how long a document will be valid; second, a client may check whether its copy of a document is still the most recent document there is. We should probably implement both approaches.
(This was issue 6 in my GitHub repository.)Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5247Include reverse DNS lookup results in details2020-06-13T17:59:22ZKarsten LoesingInclude reverse DNS lookup results in detailsWe should run reverse DNS lookups and include their results in details documents. What's the best way to run these lookups in Java? Also, do we have to run them every hour for every relay?
I wrote a simple Java application that looks up...We should run reverse DNS lookups and include their results in details documents. What's the best way to run these lookups in Java? Also, do we have to run them every hour for every relay?
I wrote a simple Java application that looks up host names using the following code line:
```
InetAddress.getByName(address).getHostName()
```
The application also measures how long each lookup took. I ran it for the first 1000 relays in the consensus published on 2012-02-18 at 03:00:00. Here are some simple statistics:
```
Min. 1st Qu. Median Mean 3rd Qu. Max.
114.0 688.8 1032.0 1906.0 1628.0 81120.0
```
So, looking up all 2759 relays in the consensus would have taken about 1.5 hours. There's no way for sequentially looking up reverse DNS entries for all relays in a consensus every hour. We'll need to make some optimizations before even starting. Questions are:
- Is there a faster way to look up reverse DNS entries than the one used in this simple Java application?
- Can we group multiple lookups and make a single request for them?
- How often do we need to refresh a reverse DNS lookup result? In theory we could cache results for an arbitrary time, but would they still be accurate after 3, 6, 12, 24 hours?
- How many requests can we make in parallel using Java threads? The Java side is easy and probably doesn't eat too much CPU time, but would we trigger some mechanism at our ISP when we make 100 requests at a time?
Here are some comments after talking to George and Damian:
- An average lookup time of 1.9 seconds per request isn't that unlikely.
- Using a thread pool with 5 lookup threads should be a fine start.
- Caching results for 12 hours should work fine. It's much more likely that a relay IP address changes than that the host name changes. We could also keep some simple statistics how often host names actually change when looking them up; if the fraction is higher than we'd like it to be, we can still reduce the caching period to 6 hours or less. We should document in protocol.html how often host names are looked up.
- Performing multiple lookups per request would be cool, but is probably not supported by Java libraries.
- I re-ran the analysis above, but this time with the `host` tool instead of Java. Results are much lower, so there must be something going on in Java which slows down the lookup. More research needed.
```
Min. 1st Qu. Median Mean 3rd Qu. Max.
0.0320 0.1800 0.3780 0.4252 0.5420 12.0300
```
(This was issue 7 in my GitHub repository.)Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5248Support searches for more than 1 search term2020-06-13T17:59:22ZKarsten LoesingSupport searches for more than 1 search termSeparating search terms by spaces doesn't work, because Tomcat cannot parse spaces in URLs other than in parameters. But maybe we can use some other character to separate search terms than space. Also redefine the search semantics. Right...Separating search terms by spaces doesn't work, because Tomcat cannot parse spaces in URLs other than in parameters. But maybe we can use some other character to separate search terms than space. Also redefine the search semantics. Right now, it's "beginning of nickname, fingerprint, etc."
(This was issue 9 in my GitHub repository.)Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5249Add orport parameter to Onionoo2020-06-13T17:59:22ZKarsten LoesingAdd orport parameter to OnionooWe could support searches for ip:port combinations. That might be useful for machines running more than 1 relay. In theory, there shouldn't be many relays per IP, but it wouldn't hurt to support this search.
The only question is whether...We could support searches for ip:port combinations. That might be useful for machines running more than 1 relay. In theory, there shouldn't be many relays per IP, but it wouldn't hurt to support this search.
The only question is whether Tomcat supports ":" in URLs. ;)
When implementing this we can't just look at summary documents anymore to see which relays match. We'll have to look at details documents, too. Shouldn't be too difficult, though.
(This was issue 11 in my GitHub repository.)Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5250Include country/region/city names in details documents2020-06-13T17:59:23ZKarsten LoesingInclude country/region/city names in details documentsDamian suggests to include more information from the GeoIP database in details documents. We discussed the region and the city name of a relay, but it may also be useful to add the country name written out. These information are trivia...Damian suggests to include more information from the GeoIP database in details documents. We discussed the region and the city name of a relay, but it may also be useful to add the country name written out. These information are trivial to look up: `location.countryName`, `regionName.regionNameByCode(location.countryCode, location.region)`, and `location.city`.
I started an initial implementation of this feature, but ran into problems with encoding Java's Unicode strings as UTF-8. It's just not pretty to write Nürnberg as N?rnberg.
(This was issue 12 in my GitHub repository.)Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5251Ponder extending summary documents to include exit addresses in "a" field2020-06-13T17:59:23ZKarsten LoesingPonder extending summary documents to include exit addresses in "a" fieldThe "a" field in summary documents currently contains an "Array of IPv4 or IPv6 addresses where the relay accepts onion-routing connections." It does not contain addresses that the relay uses to exit to the Internet. Should exit addresse...The "a" field in summary documents currently contains an "Array of IPv4 or IPv6 addresses where the relay accepts onion-routing connections." It does not contain addresses that the relay uses to exit to the Internet. Should exit addresses be included if they differ from onion-routing addresses? This affects both the search functionality in the servlet (which is based on the summary document) as well as clients using the summary document to run their own search.
(This was issue 13 in my GitHub repository.)Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5252Include AS information in relay details documents2020-06-13T17:59:23ZKarsten LoesingInclude AS information in relay details documentsDamian and I discussed including AS numbers and names in relay detail documents. Maxmind has "a [free database](http://www.maxmind.com/download/geoip/database/asnum/) that maps IPv4 addresses to ASN, including the name of the ASN." The...Damian and I discussed including AS numbers and names in relay detail documents. Maxmind has "a [free database](http://www.maxmind.com/download/geoip/database/asnum/) that maps IPv4 addresses to ASN, including the name of the ASN." The first four of 181049 lines in their CSV file are:
```
16778240,16779263,"AS56203 Big Red Group"
16781312,16782335,"AS2519 VECTANT Ltd."
16783616,16785407,"AS2519 VECTANT Ltd."
16793600,16809983,"AS18144 Energia Communications,Inc."
```
We could add two new fields "as_number" and "as_name" to relay details documents containing these information for any given relay. If we don't find a relay in their database, we'd just leave out these fields.
(This was issue 14 in my GitHub repository.)Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5253Include consensus bandwidth weights history in Onionoo details files2020-06-13T17:59:23ZKarsten LoesingInclude consensus bandwidth weights history in Onionoo details filesConsensus bandwidth weights are the most real-time bandwidth information that we have about relays. Bandwidth histories are around 9 hours old on average (with a descriptor re-publication interval of 18 hours). It might be useful to have...Consensus bandwidth weights are the most real-time bandwidth information that we have about relays. Bandwidth histories are around 9 hours old on average (with a descriptor re-publication interval of 18 hours). It might be useful to have a consensus bandwidth weight history of the last 24 hours (or more?) in details documents, so that clients can visualize how fast relays are, even though weights are no exact bandwidth values.
(This was issue 16 in my GitHub repository.)Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5254Extend search to country code or other fields in details documents2020-06-13T17:59:24ZKarsten LoesingExtend search to country code or other fields in details documentsWe should consider extending the search URLs to country codes. This isn't trivial to implement, because we'd have to look at all details documents in order to know which relay summaries/details/bandwidth documents to return. Maybe we'd n...We should consider extending the search URLs to country codes. This isn't trivial to implement, because we'd have to look at all details documents in order to know which relay summaries/details/bandwidth documents to return. Maybe we'd need to cache this information in the servlet. Also requires a protocol change, e.g., new URLs like /summary/search-details/:searchterm which may look at other fields in details documents as well.
(This was issue 18 in my GitHub repository.)Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5255Support search via Tor node contact / email address2020-06-13T17:59:24ZKarsten LoesingSupport search via Tor node contact / email addressdjon3s says:
> I'd like to be able to search for Tor nodes by the contact details left on the node. Say I wanted to see every server that Torservers.net runs. Currently, only servers that will return are those with Torservers in the titl...djon3s says:
> I'd like to be able to search for Tor nodes by the contact details left on the node. Say I wanted to see every server that Torservers.net runs. Currently, only servers that will return are those with Torservers in the title. It will miss servers with another name (e.g. 'Chomsky) for which @torservers.net is linked as a contact.
>
> Thanks :-)
Sounds good. This is also related to #5254 (former GitHub issue 18).
(This was issue 19 in my GitHub repository.)Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5354404 for onionoo.torproject.org2020-06-13T17:59:25ZDamian Johnson404 for onionoo.torproject.orgOnionoo's project page links to 'http://onionoo.torproject.org/' but onionoo evidently isn't living there yet.Onionoo's project page links to 'http://onionoo.torproject.org/' but onionoo evidently isn't living there yet.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5368Accept hashed relay fingerprints and hashed hashed bridge fingerprints in sea...2020-06-13T18:05:46ZKarsten LoesingAccept hashed relay fingerprints and hashed hashed bridge fingerprints in search and lookup URLSWhile thinking about how Atlas could [add bridge search support](https://github.com/hellais/TorStatus/issues/8), I came up with the following problem in the [Onionoo protocol](http://onionoo.torproject.org/).
Both the */lookup/ and */se...While thinking about how Atlas could [add bridge search support](https://github.com/hellais/TorStatus/issues/8), I came up with the following problem in the [Onionoo protocol](http://onionoo.torproject.org/).
Both the */lookup/ and */search/ URLs currently require a non-hashed fingerprint to look up a relay and a hashed fingerprint to look up a bridge. This works fine if a) users know how to hash fingerprints and b) the client knows whether the user provided a hashed or non-hashed fingerprint. But what if either a) or b) is not the case? If the client sees a 40-character fingerprint it shouldn't simply submit it to the Onionoo server. It might be a non-hashed bridge fingerprint.
The suggested change is to make */lookup/ and */search/ URLs accept hashed relay fingerprints and hashed hashed bridge fingerprints. Clients would then be advised to always hash any 40-character hex input they receive from users and include that in their request to the Onionoo server. If the user provided a non-hashed fingerprint of a bridge the client wouldn't reveal the non-hashed fingerprint in the URL. If the user provided a hashed fingerprint, looking up the hash of the hash would still work and return the bridge the user was looking for.
The following text would be added to the protocol description:
"GET */search/:searchtext [...] Full fingerprints should always be hashed using SHA-1, regardless of searching for a relay or bridge, in order to not accidentally leak non-hashed bridge fingerprints in the URL."
"GET */lookup/:fingerprint [...] Fingerprints should always be hashed using SHA-1, regardless of looking up a relay or bridge, in order to not accidentally leak non-hashed bridge fingerprints in the URL."
This ticket doesn't solve the problem when the user provides only the first, say, 9 to 39 hex characters of a relay or bridge fingerprint. The client can only hash a full 40-character fingerprint, so there's no way how the client can protect the user from herself here. But I think the most common cases are that users type in the first, say, 8 characters of a fingerprint or paste the full 40-character fingerprint. The first 8 characters alone don't reveal too much information, and the pasted 40-character fingerprints will be protected by the change suggested in this ticket.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5405we need a place to keep track of our tech reports2020-06-13T17:19:57ZRoger Dingledinewe need a place to keep track of our tech reportsWe produced a bunch of pdfs for the mid March deadline, and I think we've had a bunch of pdfs for previous things too. They're all scattered, and soon they'll be lost to the sands of time.
We should gather a tech report page to track th...We produced a bunch of pdfs for the mid March deadline, and I think we've had a bunch of pdfs for previous things too. They're all scattered, and soon they'll be lost to the sands of time.
We should gather a tech report page to track them.
Assigning to me so people aren't like "wtf close it", but do feel free to jump in and do it first if you like. :)Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5573Ensure that metrics can handle ‘ides’ becoming ‘turtles’2020-06-13T17:50:02ZRobert RansomEnsure that metrics can handle ‘ides’ becoming ‘turtles’See #5569 for details.See #5569 for details.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5607Bridge descriptor sanitizer calculates wrong descriptor digest2020-06-13T17:50:02ZKarsten LoesingBridge descriptor sanitizer calculates wrong descriptor digestWhen [determining SHA-1 hashes of sanitized bridge descriptors in metrics-db](https://gitweb.torproject.org/metrics-db.git/blob/9d6c353:/src/org/torproject/ernie/db/SanitizedBridgesWriter.java#l796) we don't include the "router-signature...When [determining SHA-1 hashes of sanitized bridge descriptors in metrics-db](https://gitweb.torproject.org/metrics-db.git/blob/9d6c353:/src/org/torproject/ernie/db/SanitizedBridgesWriter.java#l796) we don't include the "router-signature" line. That means we store sanitized bridge descriptors under the wrong file name and reference them from bridge network statuses using the wrong descriptor identifier. The same applies to [extra-info digests](https://gitweb.torproject.org/metrics-db.git/blob/9d6c353:/src/org/torproject/ernie/db/SanitizedBridgesWriter.java#l1014).
This bug isn't problematic as long as one uses descriptor file names to navigate between network statuses, server descriptors, and extra-info descriptors. But navigating doesn't work anymore when using metrics-lib's getServerDescriptorDigest() or getExtraInfoDigest() methods which calculate the SHA-1 digest correctly. We should fix this bug and sanitize all bridge descriptors again.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5608Order of sanitizing bridge descriptor tarballs matters even though it shouldn't2020-06-13T17:50:03ZKarsten LoesingOrder of sanitizing bridge descriptor tarballs matters even though it shouldn'tThere's a bug in how metrics-db tries to repair references between bridge network statuses, server descriptors, and extra-info descriptors. In theory, the order of processed tarballs shouldn't matter, because we have a mapping file with...There's a bug in how metrics-db tries to repair references between bridge network statuses, server descriptors, and extra-info descriptors. In theory, the order of processed tarballs shouldn't matter, because we have a mapping file with hashed bridge fingerprint, descriptor publication time, server descriptor digest, and extra-info digest. But apparently this approach is buggy. I just tried sanitizing 1 week of tarballs in forward and in reverse order. The result was that sanitized descriptors differed. I suspect the problem is that bridges can publish more than 1 descriptor in a single second, but I'm not sure yet. I'm also not sure whether this leads to data loss or not. More analysis required. We might have to sanitize all bridge descriptors again.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5629How many EC2 bridges do we have?2020-06-13T18:12:53ZRuna SandvikHow many EC2 bridges do we have?We are encouraging people to set up bridges in the Amazon cloud, so it would be nice if we could also count the number of EC2 bridges we have.
Last time aagbsn counted EC2 bridges, the number was ~256. The time before it was ~300, and b...We are encouraging people to set up bridges in the Amazon cloud, so it would be nice if we could also count the number of EC2 bridges we have.
Last time aagbsn counted EC2 bridges, the number was ~256. The time before it was ~300, and before that it was ~600. When Roger counted, it was ~90. Either something's up with BridgeDB, our Tor Cloud images, or someone's counting bridges wrong. Which one is it?Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5632Make Torperf log timestamps after every 10% of received bytes2012-04-19T08:13:17ZKarsten LoesingMake Torperf log timestamps after every 10% of received bytesRob suggests to add timestamps between receiving the first byte and completing the request. He suggests timestamps for every completed 10% or 20% of received bytes.
We can easily add timestamps to Torperf's .data files. I wrote a patc...Rob suggests to add timestamps between receiving the first byte and completing the request. He suggests timestamps for every completed 10% or 20% of received bytes.
We can easily add timestamps to Torperf's .data files. I wrote a patch to trivsocks-client.c that adds timestamps for 10, 20, ..., 90% of read bytes. I'll push the new branch in a minute when this ticket has a number.
Once this patch is reviewed and merged, we'll want to see how metrics-db and metrics-web can handle the extended .data files. I don't expect any major problems.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5637Ignore carriage returns when parsing descriptors2020-06-13T18:10:50ZKarsten LoesingIgnore carriage returns when parsing descriptorsThere are some (very old) server descriptors containing carriage returns in their contact line, more precisely in a full PGP key block. I'm attaching one such descriptor to this ticket.
It's unclear if carriage returns are permitted by...There are some (very old) server descriptors containing carriage returns in their contact line, more precisely in a full PGP key block. I'm attaching one such descriptor to this ticket.
It's unclear if carriage returns are permitted by dir-spec.txt. It does say `"ArgumentChar ::= any printing ASCII character except NL."`, but we know that Tor accepts non-ASCII characters in contact or platform lines, too. So, I guess the only non-permitted character is NL.
In metrics-lib, we parse descriptors using `BufferedReader.readLine()` which treats `\n`, `\r`, and `\r\n` all the same. We may have to write our own `readLine()` replacement that only accepts `\n` as line end.
This may also affect stem's descriptor parser.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5651Annotation header with descriptor types2020-06-13T18:10:51ZDamian JohnsonAnnotation header with descriptor typesIt would be helpful to parsers for the first line of metrics descriptors to contain the descriptor type and version. Moving this discussion from an email thread...
> Speaking of versioning and such, I would really appreciate it if all o...It would be helpful to parsers for the first line of metrics descriptors to contain the descriptor type and version. Moving this discussion from an email thread...
> Speaking of versioning and such, I would really appreciate it if all of these documents came with a header saying what the document type and version is. At present I need to guess based on the filename and first line of content which is both hacky and will break when new versions roll out. Actually, to differentiate between bridge and relay descriptors I need to rely on the '10.x.x.x' scrubbing... :/
Thoughts on how we should format it?
Cheers! -DamianKarsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5755Atlas could show "fraction of Tor network by weight" graphs over time?2020-06-13T18:05:41ZRoger DingledineAtlas could show "fraction of Tor network by weight" graphs over time?Atlas currently has "bandwidth carried by the relay" over time graphs, which are great.
How about a "fraction of the weight for that relay compared to total weights" over time graph?
I expect most relays will be ~0%, but some of the bi...Atlas currently has "bandwidth carried by the relay" over time graphs, which are great.
How about a "fraction of the weight for that relay compared to total weights" over time graph?
I expect most relays will be ~0%, but some of the big ones will be 1% or 3% or something. And whether they go from 1% to 3% or back could help to debug issues like the one Andy asks about in http://archives.seul.org/tor/relays/May-2012/msg00001.htmlKarsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5792Fix metrics-db segfault which might be related to parsing assignment*.gz files2020-06-13T17:50:04ZKarsten LoesingFix metrics-db segfault which might be related to parsing assignment*.gz filesSince May 6, 10:08 CEST, metrics-db's JVM sometimes segfaults with this output:
```
[java] Java Result: 134
[java] #
[java] # A fatal error has been detected by the Java Runtime Environment:
[java] #
[java] # S...Since May 6, 10:08 CEST, metrics-db's JVM sometimes segfaults with this output:
```
[java] Java Result: 134
[java] #
[java] # A fatal error has been detected by the Java Runtime Environment:
[java] #
[java] # SIGSEGV (0xb) at pc=0x00007febde77bdec, pid=6579, tid=140651047905024
[java] #
[java] # JRE version: 6.0_18-b18
[java] # Java VM: OpenJDK 64-Bit Server VM (14.0-b16 mixed mode linux-amd64 )
[java] # Derivative: IcedTea6 1.8.13
[java] # Distribution: Debian GNU/Linux 6.0.4 (squeeze), package 6b18-1.8.13-0+squeeze1
[java] # Problematic frame:
[java] # V [libjvm.so+0x3c3dec]
[java] #
[java] # An error report file with more information is saved as:
[java] # /srv/metrics.torproject.org/db/hs_err_pid6579.log
[java] #
[java] # If you would like to submit a bug report, please include
[java] # instructions how to reproduce the bug and visit:
[java] # http://icedtea.classpath.org/bugzilla
[java] #
```
A quick analysis shows that the segfault happens while processing .gz-compressed bridge pool assignment files. Attempts to reproduce this problem by reading the .gz file in question 100 times in a row and by setting up a separate metrics-db instance were not successful. Switched from commons-compress-1.0.jar to commons-compress-1.4.jar, even though the change log doesn't say anything about a possible segfault.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5805Compare anguilla's tarballs to yatei's and maybe merge them2020-06-13T17:50:04ZKarsten LoesingCompare anguilla's tarballs to yatei's and maybe merge themweasel was running his directory-archive script until a week or two ago. I want to compare anguilla's tarballs to yatei's to figure out if yatei is missing some descriptors and why, and to merge missing descriptors into yatei's tarballs...weasel was running his directory-archive script until a week or two ago. I want to compare anguilla's tarballs to yatei's to figure out if yatei is missing some descriptors and why, and to merge missing descriptors into yatei's tarballs.
This ticket is mostly here to note down the endless hours that I already worked on this task, mostly because I need to write new comparison scripts and investigate differences between single descriptors manually before identifying a pattern. As of now, I spent 9 points on this task, and I'm not done. I think another 3 points remain. The task looked so tiny when I decided to do it, but it's also important enough to spend the remaining points.
Current insights from the comparison, which might turn into new tasks, are:
- Quite a few of the consensuses collected by yatei have missing or extraneous signatures as compared to anguilla's. This has to do with authorities serving consensuses that don't have all signatures. I don't really care, so I'm probably leaving this alone.
- Quite often, missing a consensus automatically means missing all votes. We might switch to downloading votes by all known authorities, not only by the ones contained in a consensus (which we're missing in these cases). Not super important, but probably worth doing.
- We have quite a few files in yatei's tarballs that are empty or truncated. We need to try parsing descriptors with metrics-lib (which is not yet used by metrics-db) and only store valid descriptors to disk.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5806Investigate why metrics-db sometimes takes longer than 5 minutes to run2020-06-13T17:50:05ZKarsten LoesingInvestigate why metrics-db sometimes takes longer than 5 minutes to runmetrics-db runs once every hour and shouldn't take longer than a few minutes. Ideally, we should run it twice per hour to catch consensuses published at :30 of an hour (see #5504). That's only possible if it reliably finishes in under ...metrics-db runs once every hour and shouldn't take longer than a few minutes. Ideally, we should run it twice per hour to catch consensuses published at :30 of an hour (see #5504). That's only possible if it reliably finishes in under 30 minutes, ideally under 5 minutes.
My current idea why metrics-db takes so long is that it implements a quite crappy algorithm to provide the last 3 days of data via rsync: enumerate all files in the current rsync/ directory and store their names and last modified times in memory, iterate over our various output directories, copy files that are missing in rsync/, and delete the ones in rsync/ that are older than 3 days. The problem is the step where we iterate over output directories, because files are not deleted automatically after a given time. Any change needs to be implemented for relay descriptors, bridge descriptors, bridge pool assignments, exit lists, etc.
This task took 1.5 points so far and may take another 2.5 points for a real fix.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5807Propose better bridge usage statistics2020-06-13T17:47:29ZKarsten LoesingPropose better bridge usage statisticsIn #3261 I found that the way how bridges report their statistics to the bridge authority is not the reason why our bridge usage numbers are so unreliable. It's the way how we derive user numbers from unique IP addresses which is totall...In #3261 I found that the way how bridges report their statistics to the bridge authority is not the reason why our bridge usage numbers are so unreliable. It's the way how we derive user numbers from unique IP addresses which is totally broken. We should try an approach that's similar to how we count directory requests on directory mirrors.
There are a few substeps necessary to come up with better bridge usage statistics:
- Write a proposal for new bridge statistics based on counting directory requests per day and country. (4 points)
- Implement the proposal, get it in mergeable state, and test it on three own bridges that don't publish the new statistics to the bridge authority yet. (4 points)
- Evaluate whether the new statistics can improve our user number estimates, and if so, enable reporting to the bridge authority and prepare deployment on all new bridges. (6 points)Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5808Add a parse history for assignments in metrics-db2020-06-13T17:50:05ZKarsten LoesingAdd a parse history for assignments in metrics-dbSince we moved BridgeDB from byblos to ponticum, there are more than just one assignment.log files, some of which are .gz-compressed. This is good, because it solves the problem that the file grows indefinitely. But we should add a par...Since we moved BridgeDB from byblos to ponticum, there are more than just one assignment.log files, some of which are .gz-compressed. This is good, because it solves the problem that the file grows indefinitely. But we should add a parse history file to avoid parsing a file that hasn't changed again. Worth a point if done right.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5812Look into downloading votes by all known authorities2020-06-13T17:50:06ZKarsten LoesingLook into downloading votes by all known authoritiesmetrics-db currently looks at the downloaded consensus to decide which votes to download. This approach fails, as we found in #5805, when metrics-db is missing a consensus. We should look into downloading votes published by all known a...metrics-db currently looks at the downloaded consensus to decide which votes to download. This approach fails, as we found in #5805, when metrics-db is missing a consensus. We should look into downloading votes published by all known authorities, not just the ones contained in the consensus.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5813Sanity-check descriptors using metrics-lib before writing them to disk2020-06-13T17:50:06ZKarsten LoesingSanity-check descriptors using metrics-lib before writing them to diskIn #5805 we found that we have quite a few files that are either empty or truncated. We should parse all descriptors with metrics-lib and only store valid descriptors to disk.In #5805 we found that we have quite a few files that are either empty or truncated. We should parse all descriptors with metrics-lib and only store valid descriptors to disk.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5823Remove dirreq-v2-* and dirreq-v?-share statistics2020-06-13T14:19:30ZKarsten LoesingRemove dirreq-v2-* and dirreq-v?-share statisticsThe v2 directory protocol is (almost?) dead, and we stopped using dirreq-v?-share lines, because they were highly inaccurate. Should we stop including these lines in extra-info descriptors? If so, I can prepare a patch.The v2 directory protocol is (almost?) dead, and we stopped using dirreq-v?-share lines, because they were highly inaccurate. Should we stop including these lines in extra-info descriptors? If so, I can prepare a patch.Tor: 0.2.4.x-finalKarsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5824Stop reporting relay-only statistics as a bridge2020-06-13T14:19:30ZKarsten LoesingStop reporting relay-only statistics as a bridgeSome bridges report statistics that were originally designed for relays only. We throw these statistics away in the sanitizing process, so they don't end up in the tarballs. But the cleaner approach would be to not report them at all.
...Some bridges report statistics that were originally designed for relays only. We throw these statistics away in the sanitizing process, so they don't end up in the tarballs. But the cleaner approach would be to not report them at all.
Here's how the patch could look like; I can make a proper patch with changes file later:
```
diff --git a/src/or/config.c b/src/or/config.c
index ab4f160..2179117 100644
--- a/src/or/config.c
+++ b/src/or/config.c
@@ -1680,8 +1680,9 @@ options_act(const or_options_t *old_options)
time_t now = time(NULL);
int print_notice = 0;
- /* If we aren't acting as a server, we can't collect stats anyway. */
- if (!server_mode(options)) {
+ /* If we aren't acting as a server or if we are a bridge, we can't
+ * collect relay-only stats. */
+ if (!server_mode(options) || options->BridgeRelay) {
options->CellStatistics = 0;
options->DirReqStatistics = 0;
options->EntryStatistics = 0;
```
Is this something for 0.2.3.x or 0.2.4.x? Or do we not care at all?Tor: 0.2.5.x-finalKarsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/5960Advanced searches in onionoo2020-06-13T17:59:26ZArturo FilastòAdvanced searches in onionooGeorge suggested that it would be nice to be able to add support for country based searches in Atlas. This would require an API on onionoo's side to support advanced searches.
We could create something like:
Maybe it would be good to c...George suggested that it would be nice to be able to add support for country based searches in Atlas. This would require an API on onionoo's side to support advanced searches.
We could create something like:
Maybe it would be good to change summary/search/:searchtext to something like summary/search/<relay_paramter (e.x. country_name)>/:searchtext.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/6064Bridge usage statistics on metrics website are broken2020-06-13T18:12:54ZKarsten LoesingBridge usage statistics on metrics website are brokenThe graph on [bridge users from all countries](https://metrics.torproject.org/users.html#bridge-users) recently went up from 10,000 to 50,000. There was no event that could explain this increase, so I looked for a possible bug.
Here's ...The graph on [bridge users from all countries](https://metrics.torproject.org/users.html#bridge-users) recently went up from 10,000 to 50,000. There was no event that could explain this increase, so I looked for a possible bug.
Here's the bug: when we aggregate bridge users per day, we write single observations to a file with lines like this:
```
bridge,date,time,??,a1,a2,...,all
0007BC3A0CFC768DB2FA1E3EB6FB4ABF4EBE2D13,2012-05-24,07:12:18,NA,1.12,NA,...,30.55
```
In the next step we aggregate these lines by summing up all observations of a given day.
Turns out the file with single observations was truncated and we didn't notice. When adding lines to that file, it is read to memory, new observations are added, and the file is written to disk. The file is always kept ordered by bridge fingerprint. Here's the distribution of bridge fingerprints in the file:
```
0 24567
1 24623
2 11687
3 1526
4 1124
5 825
6 1352
7 1422
8 1271
9 1287
A 1336
B 1048
C 1525
D 1227
E 1497
F 994
```
We would expect roughly the same number of bridges in each bucket. Looks like the file was truncated after writing half of the fingerprints starting with 2. This could have happened due to Java running out of memory, the server being restarted while writing the file, etc.
The quick fix is to aggregate bridge usage statistics again and replace the single-observations file on yatei. I'm going to do that now.
The next fix is to avoid truncating the file by writing to a temp file and replacing the original file with it once we're done writing. I'll look into that next.
The real fix is to stop using flat files for something that requires a database. That's going to take me quite a bit longer.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/6070Download times in Torperf graphs have always been 1 second too high2020-06-13T18:12:54ZKarsten LoesingDownload times in Torperf graphs have always been 1 second too highWe recently changed metrics-web to use the new Torperf data file format .tpf instead of Torperf's .data files. This seemed to work fine, but a side-effect of this change was that download times magically went down by about 1 second. Th...We recently changed metrics-web to use the new Torperf data file format .tpf instead of Torperf's .data files. This seemed to work fine, but a side-effect of this change was that download times magically went down by about 1 second. This is most visible in the [50 KiB graphs](https://metrics.torproject.org/performance.html?graph=torperf&start=2012-03-07&end=2012-06-05&source=all&filesize=50kb&dpi=72#torperf).
I found the problem: download times in the graphs have always been 1 second too high! Here's the problematic old code that now got replaced:
```
- long completeMillis = Long.parseLong(parts[16])
- * 1000L + Long.parseLong(parts[17]) / 1000L
- - Long.parseLong(parts[0]) * 1000L
- + Long.parseLong(parts[1]) / 1000L;
```
`parts[16]` and `parts[17]` are the seconds and microseconds when a download was completed, and `parts[0]` and `parts[1]` was when the request was started. Note how there should be parentheses around the last two lines? Instead of subtracting the microseconds when the request was started, we added them. Assume that this number was 500,000 in the mean, so that the overall result was by 1,000,000 microseconds or 1 second too high in the mean. Oops.
This problem is already fixed in the code with the move to metrics-lib and the new .tpf format. The new numbers since early June 2012 are correct.
I'll have to re-do the existing Torperf statistics. This bug was present since we first added Torperf graphs to the metrics website in January 2010.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/6120visualize torperf results in layer graph2020-06-13T17:47:31ZRoger Dingledinevisualize torperf results in layer graphOur torperf graphs show mean over time. Great.
But how much of the time is spent in which phase of downloading? Torperf collects that data, but our metrics graphs don't use it.
Seems to me there are 3 main phases to a fetch:
1) Receive...Our torperf graphs show mean over time. Great.
But how much of the time is spent in which phase of downloading? Torperf collects that data, but our metrics graphs don't use it.
Seems to me there are 3 main phases to a fetch:
1) Receive the connected cell.
2) Receive the first byte of the response.
3) Receive the last byte of the response.
Is it easy to show the torperf graphs as a layered graph of these three components?
This topic comes up because once we do #3890, I expect the middle phase to basically disappear.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/6170Metrics graphs glitches2020-06-13T18:12:55ZGeorge KadianakisMetrics graphs glitchesIt seems that some graphs in metrics.tpo have display glitches.
Examples:
https://metrics.torproject.org/users.html?graph=direct-users&start=2012-03-14&end=2012-06-04&country=gq&dpi=72#direct-users
https://metrics.torproject.org/users...It seems that some graphs in metrics.tpo have display glitches.
Examples:
https://metrics.torproject.org/users.html?graph=direct-users&start=2012-03-14&end=2012-06-04&country=gq&dpi=72#direct-users
https://metrics.torproject.org/users.html?graph=direct-users&start=2012-03-14&end=2012-06-04&country=sy&events=on&dpi=72#direct-users
(the last one is related to the censorship events upscores/downscores graphs)Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/6261consensus-health should learn to diff the recommended version sets2020-06-13T18:10:52ZKarsten Loesingconsensus-health should learn to diff the recommended version sets```
18:15:05 < rransom> karsten: consensus-health should learn to diff the recommended version sets.
```
I agree.```
18:15:05 < rransom> karsten: consensus-health should learn to diff the recommended version sets.
```
I agree.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/6318Compare relay IPs between maxmind db and blockfinder db2020-06-13T17:47:36ZRoger DingledineCompare relay IPs between maxmind db and blockfinder dbMaxmind's geoip db is rotting (#6266). I hear Jake's blockfinder produces similar results. We should check how similar they are in practice. The first question is: do the two databases agree about the IPs in the consensus?Maxmind's geoip db is rotting (#6266). I hear Jake's blockfinder produces similar results. We should check how similar they are in practice. The first question is: do the two databases agree about the IPs in the consensus?Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/6329Write script to aggregate relay weights by country, AS, or relay family2020-06-13T18:10:53ZKarsten LoesingWrite script to aggregate relay weights by country, AS, or relay familylunar is interested in adding pie charts about relays/capacity per AS network and similar statistics to Atlas. In order to do this, Onionoo should provide these data in a convenient way. These data would be based on the very last conse...lunar is interested in adding pie charts about relays/capacity per AS network and similar statistics to Atlas. In order to do this, Onionoo should provide these data in a convenient way. These data would be based on the very last consensus and would not incorporate any history.
We should probably wait for the Python Onionoo to implement this and not extend the Java Onionoo. Setting priority to minor.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/6365Use keywords like SponsorG20120930 for sponsor deliverables instead of milest...2020-06-13T16:45:47ZKarsten LoesingUse keywords like SponsorG20120930 for sponsor deliverables instead of milestonesWe're currently using Trac's milestone field for two things: version milestone and sponsor milestone. It would be good to assign a ticket to both a version like Tor 0.2.4.x-final and to a sponsor milestone like Sponsor G, September 30, ...We're currently using Trac's milestone field for two things: version milestone and sponsor milestone. It would be good to assign a ticket to both a version like Tor 0.2.4.x-final and to a sponsor milestone like Sponsor G, September 30, 2012. Mike and Nick suggested to use Trac keywords for this purpose.
Two questions:
- Is SponsorG20120930 a good format for such keywords?
- Is there an easy way to modify existing tickets in bulk? I don't mind so much doing it manually, but it's going to generate a lot of emails.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/6380Create research.tpo/techreports.html and research.tpo/techreports/ for Tor te...2020-06-13T16:45:50ZKarsten LoesingCreate research.tpo/techreports.html and research.tpo/techreports/ for Tor tech reportsIn #5405 we discuss creating a place to keep track of our tech reports. I'd like to keep the sources in a Git repo (#6378) and put the PDFs on the Tor website. Having an HTML page www.tpo/techreports.html (or similar) for links to tech...In #5405 we discuss creating a place to keep track of our tech reports. I'd like to keep the sources in a Git repo (#6378) and put the PDFs on the Tor website. Having an HTML page www.tpo/techreports.html (or similar) for links to tech reports and a directory www.tpo/techreports/ (or similar) for the tech report PDFs would be very useful. How would I upload PDFs to the new directory without adding them to SVN? Thanks!Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/6463Some metrics colors are hard to distinguish with color-deficient vision2020-06-13T18:12:56ZDavid Fifielddcf@torproject.orgSome metrics colors are hard to distinguish with color-deficient visionWith deuteranomalous color blindness (the most common kind, 6% of males), some of the colors used on metrics graphs are hard to tell apart.
In https://metrics.torproject.org/network.html#relayflags, (reddish?) "Guard" and (brownish?) "F...With deuteranomalous color blindness (the most common kind, 6% of males), some of the colors used on metrics graphs are hard to tell apart.
In https://metrics.torproject.org/network.html#relayflags, (reddish?) "Guard" and (brownish?) "Fast" are almost the same, and (greenish?) "Running" is close to those two. "Exit" and "Stable" are also similar, but not too hard to distinguish.
In https://metrics.torproject.org/network.html#platforms, "Linux" and "Darwin" look the same at a glance.
In https://metrics.torproject.org/network.html#bwhist-flags, "Guard only" and "Middle only" are similar, likewise "Exit only" and "Guard & Exit."
I attached before and after images from http://www.vischeck.com/vischeck/vischeckImage.php of one of the graphs. I find the simulation to be accurate, in that the colors before and after look the same.
You can see how all the graphs look at once by going to
http://vischeck.com/vischeck/vischeckURL.php, entering `https://metrics.torproject.org/network.html`, submitting the form, and then clicking "Deuteranope simulation." It generated me a page at http://vischeck.homeip.net/uploads/134317381427149/, but that might be cached temporarily.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/6471Design file format and Python/Java library for multiple GeoIP or AS databases2020-06-13T18:10:54ZKarsten LoesingDesign file format and Python/Java library for multiple GeoIP or AS databasesWhenever we look up relay IP addresses in GeoIP or AS databases covering more than a few months, we can't just use a single database. There's a reason why these databases change over time. An IP address that is resolved to a certain co...Whenever we look up relay IP addresses in GeoIP or AS databases covering more than a few months, we can't just use a single database. There's a reason why these databases change over time. An IP address that is resolved to a certain country or AS may have belonged to a different country or AS a year ago. What we should really do is use multiple databases and look up the IP address in the database that was most recent at the given time.
How about we design a file format for multiple GeoIP or AS databases and write a Python library for using it? The library should allow us to:
- Convert a given GeoIP or AS database from Maxmind or using a similar format to our format.
- Merge a new single GeoIP or AS database into our format.
- Look up the country code/name or AS number/name of an IP address on a given date.
I'd say that non-functional requirements are as follows, from most important to least important: lookup speed, lookup speed, lookup speed, memory consumption, file size, library code complexity.
I found these archives of GeoIP and AS databases:
http://geolite.maxmind.com/download/geoip/database/GeoLiteCity_CSV/
http://www.maxmind.com/download/geoip/database/asnum/old/Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/6671network advertised bandwidth/history graph by Exit / Guard flag?2020-06-13T18:12:59ZRoger Dingledinenetwork advertised bandwidth/history graph by Exit / Guard flag?https://metrics.torproject.org/network.html#bandwidth is great but doesn't let me see the trends for the nodes with the Exit flag (or with the Guard flag).
(I can't do it with https://metrics.torproject.org/network.html#bwhist-flags bec...https://metrics.torproject.org/network.html#bandwidth is great but doesn't let me see the trends for the nodes with the Exit flag (or with the Guard flag).
(I can't do it with https://metrics.torproject.org/network.html#bwhist-flags because a) that's only bandwidth history, not advertised bandwidth, and b) it doesn't given me the sum of Exit flags or the sum of Guard flags. And https://metrics.torproject.org/network.html#relayflags is totally what I want but it gives number-of-relays, not either bandwidth metric.)Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/6680certain relays are missing "country" information -> columns are shifted (CC =...2020-06-13T17:53:05Zcypherpunkscertain relays are missing "country" information -> columns are shifted (CC = AS number, AS number= AS name, ...)When you inspect the complete relay list in compass [1] you will notice that in some rare occasions the 'CC' column actually contains the AS number and the 'AS number' column contains the AS name.
This is true for:
StopkaConsulting02
Jam...When you inspect the complete relay list in compass [1] you will notice that in some rare occasions the 'CC' column actually contains the AS number and the 'AS number' column contains the AS name.
This is true for:
StopkaConsulting02
JamesGamble
TorLand2
DksTorRelay
wii
TorLand1
homerelay
phx1
... (and others)
After manually inspecting the results for:
https://onionoo.torproject.org/details?search=phx1
I noticed that the
```
"country"
```
entry is missing. That is the reason for this "column shifting".
[1] https://compass.torproject.org/result?top=3000Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/6723Vidalia's milestones in Trac are outdated2020-06-13T16:46:02ZKarsten LoesingVidalia's milestones in Trac are outdatedThere's an open milestone "Vidalia: 0.2.15" in Trac, even though I'm using 0.2.20. Should I close that milestone and create a new milestone "Vidalia: 0.2.x"?There's an open milestone "Vidalia: 0.2.15" in Trac, even though I'm using 0.2.20. Should I close that milestone and create a new milestone "Vidalia: 0.2.x"?Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/6904consensus-health message for ‘conflicting or invalid consensus parameters’ is...2020-06-13T18:10:55ZRobert Ransomconsensus-health message for ‘conflicting or invalid consensus parameters’ is unclear```
2012-09-20 20:02:14 <nsa> ^Bor^B: [consensus-health] NOTICE: The following directory authorities set conflicting or invalid consensus parameters: maatuska CircuitPriorityHalflifeMsec=30000 UseOptimisticData=1 bwauthpid=1, moria1 Circ...```
2012-09-20 20:02:14 <nsa> ^Bor^B: [consensus-health] NOTICE: The following directory authorities set conflicting or invalid consensus parameters: maatuska CircuitPriorityHalflifeMsec=30000 UseOptimisticData=1 bwauthpid=1, moria1 CircuitPriorityHalflifeMsec=30000 UseOptimisticData=1 bwauthpid=1, turtles CircuitPriorityHalflifeMsec=30000 UseOptimisticData=1 bwauthpid=1
```
In this message, the consensus health checker is complaining because it thinks the ‘`UseOptimisticData`’ parameter is invalid.
This type of message should be split into two messages:
* one for an ‘invalid’ parameter (usually what this actually means is that consensus-health hasn't been told that a new parameter is valid yet), and
* one for an actual conflict between parameter values (in which case, it should show the name of the parameter with conflicting values, the values specified by each dirauth which votes on it, and the value produced in the consensus).Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/6947Allow filtering relays by version ranges2020-06-13T18:02:49ZRobert RansomAllow filtering relays by version rangesIt would be much easier to find out how many relays are still vulnerable to bugs like #6811 if there were an easy way to search for relays running versions of Tor in a specified range.It would be much easier to find out how many relays are still vulnerable to bugs like #6811 if there were an easy way to search for relays running versions of Tor in a specified range.Onionoo 1.18.0Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/6966metrics user graphs have huge std dev spikes2020-06-13T18:13:00ZRoger Dingledinemetrics user graphs have huge std dev spikeshttps://metrics.torproject.org/users.html?graph=direct-users&start=2010-12-01&end=2012-09-24&country=sy&events=on&dpi=300#direct-users for example has a huge spike in the std dev graph, making the actual black line not so obvious, and th...https://metrics.torproject.org/users.html?graph=direct-users&start=2010-12-01&end=2012-09-24&country=sy&events=on&dpi=300#direct-users for example has a huge spike in the std dev graph, making the actual black line not so obvious, and the graph not so useful for presentations.
What's the fix? Do we chop off the graph at a certain point?Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/7050metrics graphs in vector format2020-06-13T18:13:01ZZack Weinbergmetrics graphs in vector formatIt would be nice if it were possible to download PDF and/or SVG versions of the graphs from the metrics website. R has generic support for this, all you need to do is add PDF and/or SVG options to the "resolution" dropdowns, map those t...It would be nice if it were possible to download PDF and/or SVG versions of the graphs from the metrics website. R has generic support for this, all you need to do is add PDF and/or SVG options to the "resolution" dropdowns, map those to URLs with the appropriate file extension (`https://metrics.torproject.org/direct-users.pdf?...` for instance) and map _those_ to calls to the existing R scripting.` ggsave()` will do the rest.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/7057Metrics graph glitches v32020-06-13T18:13:01ZGeorge KadianakisMetrics graph glitches v3This is similar to #6170:
https://metrics.torproject.org/users.html?graph=bridge-users&start=2011-07-08&end=2012-10-06&country=tz&dpi=72#bridge-usersThis is similar to #6170:
https://metrics.torproject.org/users.html?graph=bridge-users&start=2011-07-08&end=2012-10-06&country=tz&dpi=72#bridge-usersKarsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/7452almostfastexits graph broken2020-06-13T18:13:03ZRoger Dingledinealmostfastexits graph brokenhttps://metrics.torproject.org/fast-exits.html#almostfastexits has changed so the top graph has the purple line at 1.0 and the orange line at 0.0, and the bottom graph has the purple line at 100 and the orange line at 0.
I'm not sure wh...https://metrics.torproject.org/fast-exits.html#almostfastexits has changed so the top graph has the purple line at 1.0 and the orange line at 0.0, and the bottom graph has the purple line at 100 and the orange line at 0.
I'm not sure what changed, but it's clearly not right currently.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/7587atlas not creating all graphs for a certain node2020-06-13T17:59:33ZTracatlas not creating all graphs for a certain nodetraffic graphs generation for node with id 0b37323298ff98cd86ed404895bb27b7426e8ae1 seem to have some problems since August. The node is constantly up and running for many months and the graphs about advertised bw and exit/guard probabil...traffic graphs generation for node with id 0b37323298ff98cd86ed404895bb27b7426e8ae1 seem to have some problems since August. The node is constantly up and running for many months and the graphs about advertised bw and exit/guard probability seem to work fine for that period.
https://atlas.torproject.org/#details/0b37323298ff98cd86ed404895bb27b7426e8ae1
could it have something to do with the local node configuration ?
**Trac**:
**Username**: kargigKarsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/7652Can we remove relay flags graph on hourly detail?2020-06-13T18:13:03ZKarsten LoesingCan we remove relay flags graph on hourly detail?All metrics graphs except the relay flags graph have a detail level of 1 day. The relay flags graph further provides 1 hour detail.
I'm planning to remove the option to display the relay flags graph on hourly detail. The reason is tha...All metrics graphs except the relay flags graph have a detail level of 1 day. The relay flags graph further provides 1 hour detail.
I'm planning to remove the option to display the relay flags graph on hourly detail. The reason is that I want to remove data on this level of detail from the database, making metrics-web scale better, making it easier to add new graphs in the future. This graph is blocking me there.
How sad will people be if the graph on hourly detail goes away? Are there _important_ reasons to keep it? If not, I'll disable that option in a week from now.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/7701onionoo generates invalid JSON objects2020-06-13T17:59:34ZArturo Filastòonionoo generates invalid JSON objectsWhile reviewing #5307 I noticed the onionoo generates invalid JSON data for certain items.
In particular it adds some extraneous ``,`` in the ``running`` dict key.
To reproduce do:
```
curl -s https://onionoo.torproject.org/details | ...While reviewing #5307 I noticed the onionoo generates invalid JSON data for certain items.
In particular it adds some extraneous ``,`` in the ``running`` dict key.
To reproduce do:
```
curl -s https://onionoo.torproject.org/details | grep ',,' -B 5
```Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/7828Run descriptor parser over all prior descriptors2020-06-13T02:21:45ZDamian JohnsonRun descriptor parser over all prior descriptorsStem's descriptor parser has gotten a pretty good workout, being exercised every time we do a run of the integ tests. However, I've only done spot checks for historical data.
We should talk with Karsten about running a small stem script...Stem's descriptor parser has gotten a pretty good workout, being exercised every time we do a run of the integ tests. However, I've only done spot checks for historical data.
We should talk with Karsten about running a small stem script on one of the metrics hosts that attempts to parse all of the historical descriptors. The script would be trivial to write, and given a week or so we'd know either that stem can handle all historical descriptor content, or where the issues lay.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/8028Decide how to sanitize ntor-onion-key lines in bridge descriptors2020-06-13T17:50:08ZTracDecide how to sanitize ntor-onion-key lines in bridge descriptorsSince upgrading to Tor 0.2.4.9-alpha, my bridges do not appear with correct information in https://onionoo.torproject.org/details?type=bridge:
- platform is either not present or still appears as "Tor 0.2.4.7-alpha on Linux". Note that, ...Since upgrading to Tor 0.2.4.9-alpha, my bridges do not appear with correct information in https://onionoo.torproject.org/details?type=bridge:
- platform is either not present or still appears as "Tor 0.2.4.7-alpha on Linux". Note that, globally, no bridge is reported as running version 0.2.4.9, which is quite unlikely.
- advertised_bandwidth is either not present or 0,
- last_restarted is either not present or still the previous time from when previous version 0.2.4.7 was last restarted.
When downgrading to 0.2.4.7 (thanks to Debian snapshot), advertised_bandwidth moves away from 0, last_restarted and platform are correct again.
**Trac**:
**Username**: torvlnt33rKarsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/8164Validate flag-thresholds in consensus-health checker2020-06-13T16:47:12ZKarsten LoesingValidate flag-thresholds in consensus-health checkerAs of yesterday, votes may contain `flag-thresholds` lines like this one from moria1 (line breaks only added here):
```
flag-thresholds stable-uptime=693369
stable-mtbf=153249
fast-speed=40960
...As of yesterday, votes may contain `flag-thresholds` lines like this one from moria1 (line breaks only added here):
```
flag-thresholds stable-uptime=693369
stable-mtbf=153249
fast-speed=40960
guard-wfu=94.669%
guard-tk=691200
guard-bw-inc-exits=174080
guard-bw-exc-exits=184320
enough-mtbf=1
```
The consensus-health checker can look at these values and warn if one of them gets too high or too low. What values should it consider normal, and when should it begin warning?Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/8393Don't warn when dirauths run a version that's too new2020-06-13T18:10:56ZSebastian HahnDon't warn when dirauths run a version that's too newI'm running master on gabelmoo currently, and consensus-health is complaining about an unrecommended version. The check should probably be adapted to only list versions which are too old, like Tor does when deciding whether to warn or ju...I'm running master on gabelmoo currently, and consensus-health is complaining about an unrecommended version. The check should probably be adapted to only list versions which are too old, like Tor does when deciding whether to warn or just give a notice. I'd prefer not to be notified at all in such a case, tho.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/8462Implement new bridge user counting algorithm (was Why don't .ir bridge users ...2020-06-13T18:13:12ZRoger DingledineImplement new bridge user counting algorithm (was Why don't .ir bridge users fall off when Tor gets censored by DPI?)https://metrics.torproject.org/users.html?graph=direct-users&start=2012-12-01&end=2013-03-13&country=ir&events=points#direct-users sees a pretty clear fall-off. But https://metrics.torproject.org/users.html?graph=bridge-users&start=2012-...https://metrics.torproject.org/users.html?graph=direct-users&start=2012-12-01&end=2013-03-13&country=ir&events=points#direct-users sees a pretty clear fall-off. But https://metrics.torproject.org/users.html?graph=bridge-users&start=2012-12-01&end=2013-03-13&country=ir#bridge-users doesn't show any sort of drop-off.
Similarly, https://metrics.torproject.org/users.html?graph=direct-users&start=2012-12-01&end=2013-03-13&country=sy&events=points#direct-users sees very clear blocking of Tor users, and I assume it's by DPI also (actually I've been assuming it's all SSL), but then the bridge graph doesn't look at all like this: https://metrics.torproject.org/users.html?graph=bridge-users&start=2012-12-01&end=2013-03-13&country=sy#bridge-users
Is our new bridge user counting algorithm wrong in some way? Or are the connections arriving at the bridges, and they're counting them, even if the SSL handshake doesn't finish? Or does it finish but just not proceed past that?
At this point I'm avoiding showing the bridge graphs to funders, since they make no sense to me.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/8497TorBirdy tickets ownership2020-06-13T16:46:24ZSukhbir SinghTorBirdy tickets ownershipIt would be good if I can be set as the co-owner of the TorBirdy component (and henceforth the tickets associated with it) as I don't get notifications for new tickets. ioerror is the current component owner.It would be good if I can be set as the co-owner of the TorBirdy component (and henceforth the tickets associated with it) as I don't get notifications for new tickets. ioerror is the current component owner.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/8899Move "Overlap" over "Relay flags" on Consensus Health2020-06-13T18:13:10ZbastikMove "Overlap" over "Relay flags" on Consensus HealthMy suggestion is to move up
https://metrics.torproject.org/consensus-health.html#overlap
over
https://metrics.torproject.org/consensus-health.html#relayflags
I didn't know that there was a summary on how often an authority did or did no...My suggestion is to move up
https://metrics.torproject.org/consensus-health.html#overlap
over
https://metrics.torproject.org/consensus-health.html#relayflags
I didn't know that there was a summary on how often an authority did or did not vote as the other authorities.
I assumed the Relay flags would be the last part of the Consensus Health and I would have never figured out that you list the overlap of votes if turtles wouldn't vote so funny things.
Previously I never scrolled down the whole list, just looked into some mismatches. Turtles was voting so strange that I looked at the votes it made and figured out there was a summary at the bottom.
The summary is very, very useful.
(your turtles may need water as some turtles are just slow on land, but amazingly fast in water)Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/8903Teach consensus-health about UseNTorHandshake2020-06-13T18:13:10ZSebastian HahnTeach consensus-health about UseNTorHandshakeA majority of dirauths now votes for that parameter.A majority of dirauths now votes for that parameter.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/9183Avoid parsing server descriptors more than once2020-06-13T17:59:36ZKarsten LoesingAvoid parsing server descriptors more than onceIn the past 48 hours, Onionoo's hourly cronjob have taken between 6 and 61 minutes. The latter number is particularly problematic, because two cronjobs must not overlap, in theory.
An analysis of substeps shows that I/O-heavy steps hav...In the past 48 hours, Onionoo's hourly cronjob have taken between 6 and 61 minutes. The latter number is particularly problematic, because two cronjobs must not overlap, in theory.
An analysis of substeps shows that I/O-heavy steps have highest variance. For example, relay and bridge server descriptors are parsed in three places in the code, which takes between 0:16 and 18:18 minutes, between 0:08 and 22:34 minutes, and between 0:11 and 14:30 minutes.
We can save some time here by avoiding to parse server descriptors more than once. In the mentioned cases, we simply parse all server descriptors published in the last 72 hours. What we should do instead is keep a parse history to only parse those descriptors published in the last hour, and read contents of older descriptors from our own state files.
Start with WeightsDataWriter and then tweak DetailsDataWriter.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/9197Make consensus-health message about missing signatures clearer2020-06-13T18:10:57ZKarsten LoesingMake consensus-health message about missing signatures clearerweasel was rightly confused about the message: "NOTICE: The consensuses downloaded from the following authorities are missing signatures from previously voting authorities: Faravahar, gabelmoo, maatuska, moria1, tor26, turtles"
This mes...weasel was rightly confused about the message: "NOTICE: The consensuses downloaded from the following authorities are missing signatures from previously voting authorities: Faravahar, gabelmoo, maatuska, moria1, tor26, turtles"
This message was because urras didn't sign the consensus, not because one of the listed authorities did something wrong. Rephrase this.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/9198Fix "running" fields in bridge summary and bridge details documents2020-06-13T17:59:37ZKarsten LoesingFix "running" fields in bridge summary and bridge details documentsWe say that a bridge is "running" if it's contained in the bridge network status, but it also needs the "Running" flag.
This worked fine for relays, because only running relays are contained in the consensus, but bridge network statuses...We say that a bridge is "running" if it's contained in the bridge network status, but it also needs the "Running" flag.
This worked fine for relays, because only running relays are contained in the consensus, but bridge network statuses may contain non-running bridges.
We should require the "Running" flag for both relays and bridges, just in case we fall back to a consensus method that contains non-running relays.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/9235Remove advertised bandwidth fraction graph data from Onionoo2020-06-13T17:59:37ZKarsten LoesingRemove advertised bandwidth fraction graph data from OnionooHow sad would people be if we removed the data that feeds the advertised bandwidth fraction graph, e.g., on:
https://atlas.torproject.org/#details/5498AFA3EFC5984DBE9F1F63E72EB291048393F9 (yellow line)
http://makepanic.github.io/emberj...How sad would people be if we removed the data that feeds the advertised bandwidth fraction graph, e.g., on:
https://atlas.torproject.org/#details/5498AFA3EFC5984DBE9F1F63E72EB291048393F9 (yellow line)
http://makepanic.github.io/emberjs-tor-onionoo/#/relay/5498AFA3EFC5984DBE9F1F63E72EB291048393F9 (blue line)
As one can see, that graph is already broken since July 7, which was not on purpose. I know the reason why it's broken, but it's non-trivial to fix this.
But this made me wonder: can we live without that graph line? The description says: "If there were no bandwidth authorities, this fraction would be a very rough approximation of the probability of this relay to be selected by clients." But there _are_ bandwidth authorities, and the idea behind Onionoo is to provide actual network status data, not all sorts of other data for research purposes. If we want to do research on this, we can always look at the descriptor tarballs.
We could replace this graph line with the history of the relay to be selected for the middle position. That data is already available in weights documents.
I'm asking, because providing these data is kinda painful if we also want the Onionoo cronjob to run faster. Not saying it's impossible to provide these data, but messy. I'd like to avoid the mess if this graph line isn't essential for people.
Feedback requested until a week from now, so July 17. Thanks!Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/legacy/trac/-/issues/10018Create a Trac component for DocTor2020-06-13T16:47:11ZKarsten LoesingCreate a Trac component for DocTorDamian, should we create a Trac component for your consensus-health checker DocTor? I'd like to move #8164, #8797, and #9103 to it. Any preference how to spell the component (like, is "DocTor" okay)? Want to become the component owner?Damian, should we create a Trac component for your consensus-health checker DocTor? I'd like to move #8164, #8797, and #9103 to it. Any preference how to spell the component (like, is "DocTor" okay)? Want to become the component owner?Karsten LoesingKarsten Loesing