Fix up remaining issues when parsing consensus entries

Here is what we are currently doing when parsing consensus entries:

            NetworkStatusEntry entry = e.getValue();
            String flavor = "unflavored";
            if (desc.getConsensusFlavor() == "microdesc") {
              flavor = "microdesc";
            }
            preparedStatusStatement.setString(1, entry.getNickname());
            preparedStatusStatement.setString(2, entry.getFingerprint());
            String entryDigest = descUtils.calculateDigestSha256Base64(
                entry.getStatusEntryBytes());
            preparedStatusStatement.setString(3, entryDigest);
            preparedStatusStatement.setTimestamp(4,
                new Timestamp(entry.getPublishedMillis()));
            preparedStatusStatement.setString(5, entry.getAddress());
            preparedStatusStatement.setInt(6, entry.getOrPort());
            preparedStatusStatement.setInt(7, entry.getDirPort());
            preparedStatusStatement.setString(8,
                entry.getOrAddresses().toString());
            preparedStatusStatement.setString(9,
                descUtils.fieldAsString(entry.getFlags()));
            preparedStatusStatement.setString(10, entry.getVersion());
            preparedStatusStatement.setBoolean(11, entry.getUnmeasured());
            preparedStatusStatement.setLong(12, entry.getBandwidth());
            preparedStatusStatement.setString(13,
                descUtils.fieldAsString(entry.getProtocols()));
            preparedStatusStatement.setString(14, entry.getDefaultPolicy());
            preparedStatusStatement.setString(15, entry.getPortList());
            preparedStatusStatement.setString(16, flavor);
            preparedStatusStatement.setString(17, descUtils.fieldAsString(
                entry.getMicrodescriptorDigestsSha256Base64()));
            preparedStatusStatement.setString(18, descUtils.fieldAsString(
                entry.getStats()));
            preparedStatusStatement.setString(19, entry.getMasterKeyEd25519());
            preparedStatusStatement.setString(20, digest);

. For dezember in the regular consensus on 2024-10-15 18:00:00 this results in:

| dezember                                 | FD79B7F9230858B48E75F389661840C702511078 | dhU9OQOTip+J9LNeLFHmoiC4b/+UUWkFH/VbnSNctOc | 2038-01-01 00:00:00                      | 194.55.13.50                             | 9001                                     | 0                                        | [[2a03:4000:31:141::1337]:9001]          | ["Fast","Guard","HSDir","Running","Stable","V2Dir","Valid"] | Tor 0.4.8.12    | False           | 57000           | {"Conflux":[1],"Cons":[1,2],"Desc":[1,2],"DirCache":[2],"FlowCtrl":[1,2],"HSDir":[2],"HSIntro":[4,5],"HSRend":[1,2],"Link":[1,2,3,4,5],"LinkAuth":[1,3],"Microdesc":[1,2],"Padding":[2],"Relay":[1,2,3,4]} | None            | None            | unflavored      | ["THQH2P1qVCzoGYaeQiqduABhP8AFS5sPrCeM2ASbGlA"] | null            | None            |

There are a bunch of things wrong in this entry:

  1. digest is calculated over the consensus entry, yet it has to refer to the server descriptor that made it into the consensus
  2. published is set to 2038-01-01 00:00:00. However, looking at the actual consensus it should be 2024-10-15 17:16:21. It seems the published time from the microdesc consensus has sneaked in here.
  3. p is parsed to None, None, which should be reject and 1-65536 instead. Again, this seems like the microdesc version sneaked in
  4. There is no m line in the regular consensus entry, hence we should have no respective SHA-256 micro-desc digest here
  5. stats is vote-only
  6. id is vote-only

Additionally, I don't get microdesc consensus entries at all, trying to retrieve the microdesc flavor directly with:

    "md": "SELECT nickname, fingerprint, network_status_entry.digest, published, ip, or_port, dir_port, " \
          "or_addresses, flags, version, bandwidth_unmeasured, " \
          "bandwidth_weight, proto, policy, port_list, flavor, microdescriptor_digest, " \
          "stats, master_key_ed25519 " \
          "FROM network_status_entry INNER JOIN network_status " \
          "ON network_status_entry.network_status = network_status.digest " \
          "WHERE valid_after = '2024-10-15 18:00:00' " \
          "AND fresh_until = '2024-10-15 19:00:00' " \
          "AND flavor = 'microdesc' "

in my Python script.

Edited by Georg Koppen