Rebuild fallbackdir list January 2021

!281 (closed)

Seems like updateFallbackDirs.py requires relays to have a DirPort because that's how stem downloads consensuses; and the script uses the consensus download as a form of speedcheck.

Because of the latest attacks, lots of relays have removed their DirPort which means that the new list does not contain a bunch of relays that we might want in.

assigned to @asn

added Doing Emergency labels

Hm. Should we try to get stem to use BEGIN_DIR? Or is the plan something else?

Sounds like "change the script so it doesn't require a dirport to be a fallbackdir" is the clear answer yes.

That said...do we actually have fewer fallbackdirs in the proposed patch? Looks like no, it's plenty either way? Are the new ones 'worse' in some way, like, we would have chosen the others as better but we passed them over because no dirport?

I think we passed over a bunch of good fallbackdirs because of the list of dirport, like Sebastian's. Also, I noticed that one of the two relays from !272 (closed) is missing from the new list (maybe the operator also removed their dirport).

added Backlog label and removed Doing label

changed milestone to %Tor: 0.4.6.x-freeze

mentioned in merge request !331 (merged)

removed milestone %Tor: 0.4.6.x-freeze

Let's try to get this done before the next stable. If we want to re-run the old process, we should start now. If we want to change the old process, we should start doing that now too. :)

changed milestone to %Tor: 0.4.6.x-stable

Yes absolute priority here to make a new simpler fallbackdir script using arti and get this done!

Ok, I'm almost done with a new SIMPLER script that just takes 200 random Stable and Fast relays: https://gitlab.torproject.org/dgoulet/fallbackdir

Also, it is an arti based script so +10 points.

I'm pondering if we should test a one-hop to the relay upon choosing it but this might be skewed by where the script is ran or network conditions of the client... Thus, I think we should just trust Stable and Fast and be agile to changing the list regularly.

mentioned in issue #40351 (closed)

This should check whether the relay is a DirCache as well.

Also, do we know how long a randomly chosen "stable+fast" relay remains on the network on average? I think our old logic was (at least in part) meant to avoid picking relays that hadn't been around for long enough.

This should check whether the relay is a DirCache as well.

The script does this.

Also, do we know how long a randomly chosen "stable+fast" relay remains on the network on average?

No idea. This is a data point we don't have. We'll discuss this with asn tomorrow in a call.

Also, do we know how long a randomly chosen "stable+fast" relay remains on the network on average? I think our old logic was (at least in part) meant to avoid picking relays that hadn't been around for long enough.

From dir-spec.txt:

   "Stable" -- A router is 'Stable' if it is active, and either its Weighted
   MTBF is at least the median for known active routers or its Weighted MTBF
   corresponds to at least 7 days.

This means that Stable flag does not offer the same guarantee of stability that the old fallbackdir script offered. I see that the way the old fallbadir script was doing stability checks was doing an onionoo query https://gitlab.torproject.org/tpo/core/fallback-scripts/-/blob/master/updateFallbackDirs.py#L593

I hacked our rust fallbackdir script to do the same: dgoulet/fallbackdir!1 (merged)

I've informed tor-relays@ about this upcoming change: https://lists.torproject.org/pipermail/tor-relays/2021-April/019568.html

We'll leave couple days to simmer on the list before re-generating a new list in case we missed some important bits that relay operator(s) could point to us.

Also, https://gitlab.torproject.org/tpo/core/fallback-scripts has now the new stuff. It is a rough Rust script but it does the job. Anyone wanting to improve it, please do! :)

mentioned in merge request !360 (closed)

Here is the 035 MR (with changes file): !362 (merged)

Fully merged. We are done here.

closed