Create server statuses

The first feature of aggregator.rs should be that of creating server statuses in the server_status table.

The server status is creating starting from the latest consensus or network status for bridges:

# Fetch the latest descriptors from the database
relayConsensusList = database.fetchLatestRelayNetworkStatusConsensus()
bridgeStatusList = database.fetchLatestBridgeNetworkStatus()

# Process RelayNetworkStatusConsensus entries
for each relayDesc in relayConsensusList:
    buildRelay(relayDesc, millis, conn)

# Process BridgeNetworkStatus entries
for each bridgeDesc in bridgeStatusList:
    buildBridge(bridgeDesc, millis, conn, versions)

The building logic is contained in the RouterStatusBuilder class.

The first thing we do is initialize the status with a bunch of information. Some of this might be duplicated or there is a better way to build it. We start from the first network status entry.

We start from the nickname, the fingerprint, the latest consensus entry (or network status entry for bridges) and the latest consensus (or network status for bridges):

    String fingerprint = entry.getFingerprint();
    String nickname = entry.getNickname();
    String lastNetworkStatusDigest = calculateDigestSha256Base64(
          consensus.getRawDescriptorBytes());
    String lastNetworkStatusEntryDigest = calculateDigestSha256Base64(
          entry.getStatusEntryBytes());
    
    String flags = entry.getFlags();
    String version = entry.getVersion();
    String orAddressesAndPorts =
        entry.getOrAddresses())
        + "," + entry.getAddress()
        + "," + entry.getOrPort();
    String dirPort = entry.getDirPort();
    String orAddresses = entry.getOrAddresses();
    String address = entry.getAddress();
    /* We do not know at this point if the address has changed compared to
     * the previous saved status, so we set the last changed address or port
     * timestamp to the valid after time in milliseconds from the consensus.
     *
     * We will check later on if the address is the same as the previous status
     * and if this is the case we will reset this timestamp to the previous
     * saved value.
     */
    lastChangedAddressesMillis = consensus.getValidAfterMillis();
    String versions = String.join(", ", desc.getRecommendedServerVersions());
    String[] recommendedVersions = computeRecommendedVersions(
        versions);
    Long lastSeenMillis = consensus.getValidAfterMillis();
    Long bandwidth(entry.getBandwidth();

See initializeRelayStatus

Then we want to check when was the first time this node was in a consensus. The first network status is always retrieved because we might have ingested some prior data in the network that we didn't have available at that time

There is a view in the db that is holding that first statuses for all the relays (there is another view for bridges):

SELECT * FROM first_network_status_view WHERE fingerprint=? LIMIT 1;

SELECT * FROM first_bridge_network_status_view WHERE fingerprint=? AND nickname=? LIMIT 1;

For bridges we select the nickname because there is a legit case where we have different bridges with the same fingeprint and different nicknames (like for snowflake).

Then:

family = database.fetchEffectiveFamily()
familyIds = database.fetchFamilyIds() // Right now we create a family digest which is a hash of the family string but this will be replaced by the Happy Family mechanism.
routerStatus = processRunningStatus() // Check the logic to determine if the router is online

// From the consensus entry we get the latest server descriptor digest

String serverDescDigest = entry.getDescriptor();

// Then we check if the server descriptor exists in the DB by querying for it, if it exists we select the info that we need, if not we set the status as stale.

// If a descriptor exists we get the extra info docuemnt

Object extrainfoDescriptor = database.processLastExtraInfoDescriptor();

// Then we compute the geoip information with the LoopupResults object

// Finally we fetch the lastest networkstatus weights and network status totals from their tables