The Tor Project issueshttps://gitlab.torproject.org/groups/tpo/-/issues2020-06-27T14:23:39Zhttps://gitlab.torproject.org/tpo/network-health/metrics/library/-/issues/20525Be more careful when deleting extraneous local descriptor files2020-06-27T14:23:39ZKarsten LoesingBe more careful when deleting extraneous local descriptor filesAs found in [this ticket](https://trac.torproject.org/projects/tor/ticket/18910#comment:95), `DescriptorIndexCollector` deletes descriptor files from a previous or concurrent collect run if it doesn't collect those files itself. This is...As found in [this ticket](https://trac.torproject.org/projects/tor/ticket/18910#comment:95), `DescriptorIndexCollector` deletes descriptor files from a previous or concurrent collect run if it doesn't collect those files itself. This is unexpected behavior and differs from what `DescriptorCollectorImpl` does. Let's fix this.
Here's a minimal example (taken from that other ticket):
```
DescriptorCollector dc =
DescriptorSourceFactory.createDescriptorCollector();
dc.collectDescriptors("https://collector.torproject.org",
new String[] { "/recent/torperf/" }, 0L, new File("sync"), true);
dc.collectDescriptors("https://collector.torproject.org",
new String[] { "/recent/exit-lists/" }, 0L, new File("sync"), true);
```
This is the (annotated) output of running this code (1) with the default `DescriptorIndexCollector`, (2) with `DescriptorCollectorImpl`, and (3) a fixed version of `DescriptorIndexCollector`:
```
$ java -cp bin/:lib/descriptor-1.5.0.jar:lib/slf4j-api-1.7.7.jar:lib/commons-compress-1.9.jar:lib/gson-2.2.4.jar Main
$ du -hs sync/recent/*
14M sync/recent/exit-lists
0B sync/recent/torperf # <- deleted, bad!
$ rm -rf sync/
$ java -cp bin/:lib/descriptor-1.5.0.jar:lib/slf4j-api-1.7.7.jar:lib/commons-compress-1.9.jar:lib/gson-2.2.4.jar -Ddescriptor.collector=org.torproject.descriptor.impl.DescriptorCollectorImpl Main
$ du -hs sync/recent/*
14M sync/recent/exit-lists
3.0M sync/recent/torperf # <- not deleted, good!
$ java -cp bin/:lib/descriptor-1.5.0-dev.jar:lib/slf4j-api-1.7.7.jar:lib/commons-compress-1.9.jar:lib/gson-2.2.4.jar Main
$ du -hs sync/recent/*
14M sync/recent/exit-lists
3.0M sync/recent/torperf # <- not deleted, good!
```
Possible patch used in (3) above:
```
diff --git a/src/main/java/org/torproject/descriptor/index/DescriptorIndexCollector.java b/src/main/java/org/torproject/descriptor/index/DescriptorIndexCollector.java
index d32f811..daef379 100644
--- a/src/main/java/org/torproject/descriptor/index/DescriptorIndexCollector.java
+++ b/src/main/java/org/torproject/descriptor/index/DescriptorIndexCollector.java
@@ -70,7 +70,8 @@ public class DescriptorIndexCollector implements DescriptorCollector {
this.fetchRemoteFiles(index.path, remoteFiles, minLastModified,
localDirectory, localFiles);
if (deleteExtraneousLocalFiles) {
- this.deleteExtraneousLocalFiles(remoteFiles, localDirectory, localFiles);
+ deleteExtraneousLocalFiles(remoteDirectories, remoteFiles, localDirectory,
+ localFiles);
}
}
@@ -108,12 +109,16 @@ public class DescriptorIndexCollector implements DescriptorCollector {
}
}
- static void deleteExtraneousLocalFiles(
+ static void deleteExtraneousLocalFiles(String[] remoteDirectories,
SortedMap<String, FileNode> remoteFiles,
File localDir, SortedMap<String, Long> locals) {
for (String localPath : locals.keySet()) {
- if (!remoteFiles.containsKey(localPath)) {
- new File(localDir, localPath).delete();
+ for (String remoteDirectory : remoteDirectories) {
+ if (localPath.startsWith(remoteDirectory)) {
+ if (!remoteFiles.containsKey(localPath)) {
+ new File(localDir, localPath).delete();
+ }
+ }
}
}
}
```
I briefly thought about an alternative fix where we look at the actual `index.json` contents to decide whether a local descriptor file that we didn't care about in the current collect run would be safe to delete or not. After all, we have that information, so we could as well use it.
The two arguments against it are (1) that we'd change the previous behavior of `DescriptorCollector` in a somewhat backward-incompatible way and (2) that this code might be smarter than developers might think it is and hence confuse them in a bad way.
I think I'd rather go with something like the patch above.metrics-lib 1.6.0Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/tpo/network-health/metrics/library/-/issues/20521Deprecate `DescriptorReader.setExcludeFiles()` and add two separate methods f...2020-06-27T14:23:39ZKarsten LoesingDeprecate `DescriptorReader.setExcludeFiles()` and add two separate methods for loading and saving a history fileThe history file implementation in `DescriptorReader` has a design bug for much too long that I'd finally want to fix.
The issue is that `DescriptorReader` writes the history file passed in `setExcludeFiles()` immediately after reading ...The history file implementation in `DescriptorReader` has a design bug for much too long that I'd finally want to fix.
The issue is that `DescriptorReader` writes the history file passed in `setExcludeFiles()` immediately after reading and parsing the last descriptor and putting it into the queue, regardless of whether the application has finished processing those descriptors.
If the application fails after the history file is written, it may not be able to process descriptors in the next execution that have still been in the queue at the time of failing.
One way to reduce that gap would be to have `BlockingIteratorImpl` tell `DescriptorReader` when the application has first learned that `hasNext()` returned `false` or that `next()` threw a `NoSuchElementException`. The benefit of that solution would be that no interface change would be required. The downside would be that the application might not have fully cleaned up at the time of learning that there are no further descriptors available. In one example, the application would close a database import file, perform the database import, and only write the history file after learning that the database import has succeeded.
A better solution would be to decouple saving the history file to disk from completing the descriptor read operation. We would replace the `setExcludeFiles()` method by a `setHistoryFile()` and a `saveHistoryFile()` method. Applications would use `setHistoryFile()` before starting to read descriptors, process all descriptors, perform any cleaning up, and then call `saveHistoryFile()`.
Here's an example of the code that this would save in applications that currently work around this known design bug by loading and saving history files themselves:
```
diff --git a/modules/hidserv/src/org/torproject/metrics/hidserv/Parser.java b/modules/hidserv/src/org/torproject/metrics/hidserv/Parser.java
index 2ef404e..215b0c9 100644
--- a/modules/hidserv/src/org/torproject/metrics/hidserv/Parser.java
+++ b/modules/hidserv/src/org/torproject/metrics/hidserv/Parser.java
@@ -96,33 +96,7 @@ public class Parser {
public void readParseHistory() {
if (this.parseHistoryFile.exists()
&& this.parseHistoryFile.isFile()) {
- SortedMap<String, Long> excludedFiles =
- new TreeMap<String, Long>();
- try {
- BufferedReader br = new BufferedReader(new FileReader(
- this.parseHistoryFile));
- String line;
- while ((line = br.readLine()) != null) {
- try {
- /* Each line is supposed to contain the last-modified time and
- * absolute path of a descriptor file. */
- String[] parts = line.split(" ", 2);
- excludedFiles.put(parts[1], Long.parseLong(parts[0]));
- } catch (NumberFormatException e) {
- System.err.printf("Illegal line '%s' in parse history. "
- + "Skipping line.%n", line);
- }
- }
- br.close();
- } catch (IOException e) {
- System.err.printf("Could not read history file '%s'. Not "
- + "excluding descriptors in this execution.",
- this.parseHistoryFile.getAbsolutePath());
- }
-
- /* Tell the descriptor reader to exclude the files contained in the
- * parse history file. */
- this.descriptorReader.setExcludedFiles(excludedFiles);
+ this.descriptorReader.setHistoryFile(this.parseHistoryFile);
}
}
@@ -130,33 +104,7 @@ public class Parser {
* and absolute paths to the parse history file to avoid parsing these
* files again, unless they change until the next execution. */
public void writeParseHistory() {
-
- /* Obtain the list of descriptor files that were either parsed now or
- * that were skipped in this execution from the descriptor reader. */
- SortedMap<String, Long> excludedAndParsedFiles =
- new TreeMap<String, Long>();
- excludedAndParsedFiles.putAll(
- this.descriptorReader.getExcludedFiles());
- excludedAndParsedFiles.putAll(this.descriptorReader.getParsedFiles());
- try {
- this.parseHistoryFile.getParentFile().mkdirs();
- BufferedWriter bw = new BufferedWriter(new FileWriter(
- this.parseHistoryFile));
- for (Map.Entry<String, Long> e
- : excludedAndParsedFiles.entrySet()) {
- /* Each line starts with the last-modified time of the descriptor
- * file, followed by its absolute path. */
- String absolutePath = e.getKey();
- long lastModifiedMillis = e.getValue();
- bw.write(String.valueOf(lastModifiedMillis) + " " + absolutePath
- + "\n");
- }
- bw.close();
- } catch (IOException e) {
- System.err.printf("Could not write history file '%s'. Not "
- + "excluding descriptors in next execution.",
- this.parseHistoryFile.getAbsolutePath());
- }
+ this.descriptorReader.saveHistoryFile(this.parseHistoryFile);
}
/** Set of all reported hidden-service statistics.
```
I'll push a branch to my repository in a moment.metrics-lib 1.6.0Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/tpo/network-health/metrics/library/-/issues/20440endless wait in BlockingIteratorImpl2020-06-27T14:23:39Ziwakehendless wait in BlockingIteratorImplAs reported by David:
Onionoo updater didn't stop after supposedly successful run.
Here's the relevant thread dump part:
```
"Thread-0" #8 prio=5 os_prio=0 tid=0x00007f0d602f4000 nid=0x21a7 in Object.wait() [0x00007f0d435e0000]
jav...As reported by David:
Onionoo updater didn't stop after supposedly successful run.
Here's the relevant thread dump part:
```
"Thread-0" #8 prio=5 os_prio=0 tid=0x00007f0d602f4000 nid=0x21a7 in Object.wait() [0x00007f0d435e0000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007157e4ee0> (a org.torproject.descriptor.impl.BlockingIteratorImpl)
at java.lang.Object.wait(Object.java:502)
at org.torproject.descriptor.impl.BlockingIteratorImpl.add(BlockingIteratorImpl.java:40)
- locked <0x00000007157e4ee0> (a org.torproject.descriptor.impl.BlockingIteratorImpl)
at org.torproject.descriptor.impl.DescriptorReaderImpl$DescriptorReaderRunnable.readDescriptors(DescriptorReaderImpl.java:291)
at org.torproject.descriptor.impl.DescriptorReaderImpl$DescriptorReaderRunnable.run(DescriptorReaderImpl.java:188)
at java.lang.Thread.run(Thread.java:745)
```
This is the while loop in BlockingIteratorImpl.add
```
while (this.queue.size() >= this.maxQueueSize) {
try {
wait();
} catch (InterruptedException e) {
/* nothing to be done */
}
}
```
As there are no consumers anymore and the queue is full the while-loop doesn't stop.metrics-lib 1.6.0iwakehiwakehhttps://gitlab.torproject.org/tpo/network-health/metrics/library/-/issues/20430Define common log levels2021-11-29T14:17:03ZiwakehDefine common log levelsLogging for metrics-lib was introduced a while ago, what's still missing is defining appropriate log levels.
This ticket should be for discussion of a log-level scheme and also collecting examples for useful and less useful log level ch...Logging for metrics-lib was introduced a while ago, what's still missing is defining appropriate log levels.
This ticket should be for discussion of a log-level scheme and also collecting examples for useful and less useful log level choices.https://gitlab.torproject.org/tpo/network-health/metrics/library/-/issues/20405create metrics-tools with all of the index.json processing code as first content2020-06-27T14:23:40Ziwakehcreate metrics-tools with all of the index.json processing code as first contentcf. legacy/trac#20039 and legacy/trac#19934
Any better ideas to resolve this issue?cf. legacy/trac#20039 and legacy/trac#19934
Any better ideas to resolve this issue?Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/tpo/network-health/metrics/library/-/issues/20404DescriptorIndexCollector should be default implementation2020-06-27T14:23:40ZiwakehDescriptorIndexCollector should be default implementationDescriptorIndexCollector should be default implementation as a basis for CollecTor 1.1.0DescriptorIndexCollector should be default implementation as a basis for CollecTor 1.1.0metrics-lib 1.5.0iwakehiwakehhttps://gitlab.torproject.org/tpo/network-health/metrics/library/-/issues/20395Add capability to handle large descriptor files2022-04-11T16:42:22ZiwakehAdd capability to handle large descriptor filesDiscovered OOM problem for large descriptor files in legacy/trac#20335, see comment 12 there for how to reproduce.Discovered OOM problem for large descriptor files in legacy/trac#20335, see comment 12 there for how to reproduce.https://gitlab.torproject.org/tpo/network-health/metrics/library/-/issues/20334explicitly log, which implementation is served by DescriptorSourceFactory2020-06-27T14:23:40Ziwakehexplicitly log, which implementation is served by DescriptorSourceFactoryThe implementation that is served should be logged upon creation on info level.
This will facilitate troubleshooting in future.The implementation that is served should be logged upon creation on info level.
This will facilitate troubleshooting in future.metrics-lib 1.5.0iwakehiwakehhttps://gitlab.torproject.org/tpo/network-health/metrics/library/-/issues/20333add descriptor digest to vote and streamline method name2020-06-27T14:23:40Ziwakehadd descriptor digest to vote and streamline method nameThere is currently not vote descriptor digest (so CollecTor calculates it on its own). Most other descriptor types provide a digest.
In addition, it might be nice to provide a `getDigest` method for the different descriptor digests ins...There is currently not vote descriptor digest (so CollecTor calculates it on its own). Most other descriptor types provide a digest.
In addition, it might be nice to provide a `getDigest` method for the different descriptor digests instead of `XyzDescriptor.getXyzDigest`.metrics-lib 1.7.0Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/tpo/network-health/metrics/library/-/issues/20323avoid httpurl connection and use more robust approach2020-06-27T14:23:40Ziwakehavoid httpurl connection and use more robust approachWhile testing sync the following happened:
during the downloading of `recent` from the main collector using the old, i.e. DescriptorCollectorImpl, method the download just hung at some point. I killed it after 30min, restarted and the c...While testing sync the following happened:
during the downloading of `recent` from the main collector using the old, i.e. DescriptorCollectorImpl, method the download just hung at some point. I killed it after 30min, restarted and the collection worked.
When the hang-up happened again, I stopped the process immediately and re-started it using the new method, i.e. DescriptorIndexCollector. Now, FileNotFound warnings were logged and the download finished without problems. The files that were not available anymore had just expired, that is CollecTor just cleaned them out of its recent folder, which would explain the hang-up of the old method, too.
=== Why can one approach cope with disappearing files?
The two ways differ in the trace below FileInputStream (marked below; needs a wide window to be visible)
==== thread dump old method
```
"CollecTor-Scheduled-Thread-1" #9 daemon prio=5 os_prio=0 tid=0x00007f49d43c0800 nid=0x25fd runnable [0x00007f49b0f06000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:593)
at sun.security.ssl.InputRecord.read(InputRecord.java:532)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
- locked <0x000000071dc8b160> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked <0x000000071dcbe658> (a sun.security.ssl.AppInputStream)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) <------------------------------------+
- locked <0x000000071dd26f98> (a java.io.BufferedInputStream) |
at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:552) |
at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:609) | Difference
at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:696) |
- locked <0x000000071dd2ae48> (a sun.net.www.http.ChunkedInputStream) |
at java.io.FilterInputStream.read(FilterInputStream.java:133) <-------------------------------------+
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3336)
at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:238)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x000000071dd2d138> (a java.io.BufferedInputStream)
at org.torproject.descriptor.impl.DescriptorCollectorImpl.fetchRemoteFile(DescriptorCollectorImpl.java:225)
...
```
==== thread dump new method
```
"CollecTor-Scheduled-Thread-1" #9 daemon prio=5 os_prio=0 tid=0x00007f19fc3a9800 nid=0x4060 runnable [0x00007f19e4a3f000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:593)
at sun.security.ssl.InputRecord.read(InputRecord.java:532)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
- locked <0x0000000734394300> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked <0x00000007343963d0> (a sun.security.ssl.AppInputStream)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) <-------------------------------------+
- locked <0x000000071df8b3b8> (a java.io.BufferedInputStream) |
at sun.net.www.MeteredStream.read(MeteredStream.java:134) | Difference
- locked <0x000000071df8f618> (a sun.net.www.http.KeepAliveStream) <------- (*) |
at java.io.FilterInputStream.read(FilterInputStream.java:133) <--------------------------------------+
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3336)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3329)
at java.nio.file.Files.copy(Files.java:2908)
at java.nio.file.Files.copy(Files.java:3027)
at org.torproject.descriptor.index.DescriptorIndexCollector.fetchRemoteFiles(DescriptorIndexCollector.java:88)
at org.torproject.descriptor.index.DescriptorIndexCollector.collectDescriptors(DescriptorIndexCollector.java:61)
...
```
(*) see [KeepAliveStream.java](http://pag-www.gtisc.gatech.edu/chord/examples/jdk/sun/net/www/http/KeepAliveStream.java.html)
example code from DescriptorIndexCollector ([cf. here](https://gitweb.torproject.org/metrics-lib.git/tree/src/main/java/org/torproject/descriptor/index/DescriptorIndexCollector.java?id=38b18e3520ac0adfc7ea4f15332a66fce8f21e5c#n86))
```
try (InputStream is = new URL(baseUrl + "/" + filepathname)
.openStream()) {
Files.copy(is, tempDestinationFile.toPath());
...
```
This is also a little shorter than the [older approach](https://gitweb.torproject.org/metrics-lib.git/tree/src/main/java/org/torproject/descriptor/impl/DescriptorCollectorImpl.java?id=38b18e3520ac0adfc7ea4f15332a66fce8f21e5c#n123)
```
try {
URL url = new URL(urlString);
huc = (HttpURLConnection) url.openConnection();
huc.setRequestMethod("GET");
huc.connect();
int responseCode = huc.getResponseCode();
if (responseCode == 200) {
BufferedReader br = new BufferedReader(new InputStreamReader(
huc.getInputStream()));
String line;
while ((line = br.readLine()) != null) {
sb.append(line).append("\n");
}
br.close();
}
...
```
=== next steps
* Try to device a test that triggers the above problem.
* And/Or analyze the underlying sources for the reason.
* Replace the HttpURLConnection code with the shorter Files.copy of InputStream.
This would probably important for the following tickets, too:
legacy/trac#8799, legacy/trac#16151metrics-lib 1.6.0Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/tpo/network-health/metrics/library/-/issues/20320Avoid running into an IOException and logging a warning for it2020-06-27T14:23:40ZKarsten LoesingAvoid running into an IOException and logging a warning for itWhen we recently switched from System.err printing to slf4j logging, we started logging an `IOException` that we shouldn't be running into and that we simply ignored before. This exception gets thrown when `DescriptorReaderImpl` attempt...When we recently switched from System.err printing to slf4j logging, we started logging an `IOException` that we shouldn't be running into and that we simply ignored before. This exception gets thrown when `DescriptorReaderImpl` attempts to read a parse history file that doesn't exist (yet). We should simply check whether that files exists before attempting to read it. Trivial patch:
```
diff --git a/src/main/java/org/torproject/descriptor/impl/DescriptorReaderImpl.java b/src/main/java/org/torproject/descriptor/impl/DescriptorReaderImpl.java
index fac9475..020cdd7 100644
--- a/src/main/java/org/torproject/descriptor/impl/DescriptorReaderImpl.java
+++ b/src/main/java/org/torproject/descriptor/impl/DescriptorReaderImpl.java
@@ -200,7 +200,7 @@ public class DescriptorReaderImpl implements DescriptorReader {
}
private void readOldHistory() {
- if (this.historyFile == null) {
+ if (this.historyFile == null || !this.historyFile.exists()) {
return;
}
try {
```
I'd say this is a minor issue, because it prints a warning message that might confuse users, but only on the first run. No need to put out a (point) release just for this, but it would be good to fix this in master.metrics-lib 1.5.0Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/tpo/network-health/metrics/library/-/issues/20252Identify changes to dir-spec.txt from proposal 2642020-06-27T14:23:40ZKarsten LoesingIdentify changes to dir-spec.txt from proposal 264I learned about proposal 264 this morning by finding the following warnings in CollecTor's logs:
```
2016-09-26 20:09:03,174 WARN o.t.c.b.SanitizedBridgesWriter:885 Unrecognized line 'proto Cons=1-2 Desc=1-2 DirCache=1 HSDir=1 HSIntro=3...I learned about proposal 264 this morning by finding the following warnings in CollecTor's logs:
```
2016-09-26 20:09:03,174 WARN o.t.c.b.SanitizedBridgesWriter:885 Unrecognized line 'proto Cons=1-2 Desc=1-2 DirCache=1 HSDir=1 HSIntro=3 HSRend=1-2 Link=1-4 LinkAuth=1 Microdesc=1-2 Relay=1-2'. Skipping.
```
Tor's ChangeLog indicates that this line is related to proposal 264. Here's a bridge descriptor that contains this line:
```
router Unnamed 127.0.0.1 443 0 0
identity-ed25519
-----BEGIN ED25519 CERT-----
[...]
-----END ED25519 CERT-----
master-key-ed25519 [...]
platform Tor 0.2.9.3-alpha-dev on Linux
proto Cons=1-2 Desc=1-2 DirCache=1 HSDir=1 HSIntro=3 HSRend=1-2 Link=1-4 LinkAuth=1 Microdesc=1-2 Relay=1-2
published 2016-09-27 [...]
fingerprint [...]
[...]
```
We should identify the exact changes made in proposal 264 and adapt CollecTor's bridgedescs module ASAP. Right now, we're skipping all bridge descriptors containing such a line, and we'll have to reprocess them once the patch is merged.
Once we did that, we should consider updating metrics-lib to handle these lines in both relay and bridge descriptors.metrics-lib 1.6.0Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/tpo/network-health/metrics/library/-/issues/20039integrate `DescriptorIndexCollector` in a fully backward-compatible way2020-06-27T14:23:41Ziwakehintegrate `DescriptorIndexCollector` in a fully backward-compatible wayAdapt the alpha implementation from release 1.4.0, ticket legacy/trac#19791.
See comments 10 to 12 in legacy/trac#19791 for background.Adapt the alpha implementation from release 1.4.0, ticket legacy/trac#19791.
See comments 10 to 12 in legacy/trac#19791 for background.metrics-lib 1.5.0iwakehiwakehhttps://gitlab.torproject.org/tpo/network-health/metrics/library/-/issues/19893Include resources in tarball2020-06-27T14:23:41ZKarsten LoesingInclude resources in tarballWhile putting out the first CollecTor release I noticed that the release tarball of metrics-lib does not include resource files `src/main/resources/*` and `src/test/resources/*`. We should include them in the next release tarball.While putting out the first CollecTor release I noticed that the release tarball of metrics-lib does not include resource files `src/main/resources/*` and `src/test/resources/*`. We should include them in the next release tarball.metrics-lib 1.4.0Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/tpo/network-health/metrics/library/-/issues/19842offer a `LenientParser` option with metrics-lib2020-06-27T14:23:41Ziwakehoffer a `LenientParser` option with metrics-libProvide another parser option, quote from legacy/trac#19170#comment:7
make use of the `descriptor.parser` and `descriptor.reader` properties and supply a different non-ascci-accepting parser, let's call it `LenientParser`, as well a...Provide another parser option, quote from legacy/trac#19170#comment:7
make use of the `descriptor.parser` and `descriptor.reader` properties and supply a different non-ascci-accepting parser, let's call it `LenientParser`, as well as a LenientReader.
* Necessary change in CollecTor would be to set the `descriptor.parser` and `descriptor.reader` properties to the LenientParser class.
* Necessary change in metrics-lib would be the addition of the LenientParser, which consist mostly in providing additional ParserHelper methods that accept non-ascii and calling these in the appropriate places; most of the code will be the same as in the current, stricter implementation. Also a LenientReader would have to be supplied.
That way we could switch between implementations.
Users of metrics-lib would also have another option.Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/tpo/network-health/metrics/library/-/issues/19791Use CollecTor's index.json for download ('''Alpha''' version); adapt current ...2020-06-27T14:23:41ZKarsten LoesingUse CollecTor's index.json for download ('''Alpha''' version); adapt current download to use new date formatWe're parsing dates from the CollecTor's Apache directory listings to decide whether or not to fetch a remote file. The CollecTor host was recently upgraded from wheezy to jessie, which apparently changed the date format from dd-MMM-yyy...We're parsing dates from the CollecTor's Apache directory listings to decide whether or not to fetch a remote file. The CollecTor host was recently upgraded from wheezy to jessie, which apparently changed the date format from dd-MMM-yyyy to yyyy-MM-dd. Adapt to this change.
Obviously, parsing dates like this is very fragile. We should soon switch to using CollecTor's index.json file instead, ideally before the next release.metrics-lib 1.4.0iwakehiwakehhttps://gitlab.torproject.org/tpo/network-health/metrics/library/-/issues/19643use slf4j in metrics-lib (aka DescripTor) instead of System.err or printStack...2020-06-27T14:23:41Ziwakehuse slf4j in metrics-lib (aka DescripTor) instead of System.err or printStackTraceThis would be very helpful for all metrics-lib clients/users.This would be very helpful for all metrics-lib clients/users.metrics-lib 1.4.0iwakehiwakehhttps://gitlab.torproject.org/tpo/network-health/metrics/library/-/issues/19640Review and improve interface hierarchy2020-06-27T14:23:41ZiwakehReview and improve interface hierarchysee legacy/trac#19398 (comment 10)see legacy/trac#19398 (comment 10)iwakehiwakehhttps://gitlab.torproject.org/tpo/network-health/metrics/library/-/issues/19634support shared randomness lines2020-06-27T14:23:42Ziwakehsupport shared randomness linesAs mentioned in legacy/trac#19398 by atagar:
support [Shared randomness](https://gitweb.torproject.org/torspec.git/commit/?id=9949f64) ([stem change](https://gitweb.torproject.org/stem.git/commit/?id=bccda333d40ba6a17131725475f9fdb025c4...As mentioned in legacy/trac#19398 by atagar:
support [Shared randomness](https://gitweb.torproject.org/torspec.git/commit/?id=9949f64) ([stem change](https://gitweb.torproject.org/stem.git/commit/?id=bccda333d40ba6a17131725475f9fdb025c4690a))metrics-lib 1.6.0Karsten LoesingKarsten Loesinghttps://gitlab.torproject.org/tpo/network-health/metrics/library/-/issues/19622use java 8 in DescripTor2020-06-27T14:23:42Ziwakehuse java 8 in DescripTorthis should only be finished when all depending Metrics projects use java 8.this should only be finished when all depending Metrics projects use java 8.metrics-lib 2.1.0