Fix up remaining issues when parsing consensus entries
Here is what we are currently doing when parsing consensus entries:
NetworkStatusEntry entry = e.getValue();
String flavor = "unflavored";
if (desc.getConsensusFlavor() == "microdesc") {
flavor = "microdesc";
}
preparedStatusStatement.setString(1, entry.getNickname());
preparedStatusStatement.setString(2, entry.getFingerprint());
String entryDigest = descUtils.calculateDigestSha256Base64(
entry.getStatusEntryBytes());
preparedStatusStatement.setString(3, entryDigest);
preparedStatusStatement.setTimestamp(4,
new Timestamp(entry.getPublishedMillis()));
preparedStatusStatement.setString(5, entry.getAddress());
preparedStatusStatement.setInt(6, entry.getOrPort());
preparedStatusStatement.setInt(7, entry.getDirPort());
preparedStatusStatement.setString(8,
entry.getOrAddresses().toString());
preparedStatusStatement.setString(9,
descUtils.fieldAsString(entry.getFlags()));
preparedStatusStatement.setString(10, entry.getVersion());
preparedStatusStatement.setBoolean(11, entry.getUnmeasured());
preparedStatusStatement.setLong(12, entry.getBandwidth());
preparedStatusStatement.setString(13,
descUtils.fieldAsString(entry.getProtocols()));
preparedStatusStatement.setString(14, entry.getDefaultPolicy());
preparedStatusStatement.setString(15, entry.getPortList());
preparedStatusStatement.setString(16, flavor);
preparedStatusStatement.setString(17, descUtils.fieldAsString(
entry.getMicrodescriptorDigestsSha256Base64()));
preparedStatusStatement.setString(18, descUtils.fieldAsString(
entry.getStats()));
preparedStatusStatement.setString(19, entry.getMasterKeyEd25519());
preparedStatusStatement.setString(20, digest);
. For dezember
in the regular consensus on 2024-10-15 18:00:00 this results in:
| dezember | FD79B7F9230858B48E75F389661840C702511078 | dhU9OQOTip+J9LNeLFHmoiC4b/+UUWkFH/VbnSNctOc | 2038-01-01 00:00:00 | 194.55.13.50 | 9001 | 0 | [[2a03:4000:31:141::1337]:9001] | ["Fast","Guard","HSDir","Running","Stable","V2Dir","Valid"] | Tor 0.4.8.12 | False | 57000 | {"Conflux":[1],"Cons":[1,2],"Desc":[1,2],"DirCache":[2],"FlowCtrl":[1,2],"HSDir":[2],"HSIntro":[4,5],"HSRend":[1,2],"Link":[1,2,3,4,5],"LinkAuth":[1,3],"Microdesc":[1,2],"Padding":[2],"Relay":[1,2,3,4]} | None | None | unflavored | ["THQH2P1qVCzoGYaeQiqduABhP8AFS5sPrCeM2ASbGlA"] | null | None |
There are a bunch of things wrong in this entry:
-
digest
is calculated over the consensus entry, yet it has to refer to the server descriptor that made it into the consensus -
published
is set to 2038-01-01 00:00:00. However, looking at the actual consensus it should be 2024-10-15 17:16:21. It seems the published time from the microdesc consensus has sneaked in here. -
p
is parsed toNone
,None
, which should bereject
and1-65536
instead. Again, this seems like the microdesc version sneaked in - There is no
m
line in the regular consensus entry, hence we should have no respective SHA-256 micro-desc digest here -
stats
is vote-only -
id
is vote-only
Additionally, I don't get microdesc consensus entries at all, trying to retrieve the microdesc flavor directly with:
"md": "SELECT nickname, fingerprint, network_status_entry.digest, published, ip, or_port, dir_port, " \
"or_addresses, flags, version, bandwidth_unmeasured, " \
"bandwidth_weight, proto, policy, port_list, flavor, microdescriptor_digest, " \
"stats, master_key_ed25519 " \
"FROM network_status_entry INNER JOIN network_status " \
"ON network_status_entry.network_status = network_status.digest " \
"WHERE valid_after = '2024-10-15 18:00:00' " \
"AND fresh_until = '2024-10-15 19:00:00' " \
"AND flavor = 'microdesc' "
in my Python script.
Edited by Georg Koppen