Regenerate fallback list for 0.2.9

changed milestone to %Tor: 0.3.0.x-final in legacy/trac

Trac:
Description: We need to regenerate the fallback directory mirror list in 0.2.9 in case any of the 0.2.8 fallbacks have changed details or gone down.

This should not require another opt-in mailout, as we had ~70 additional fallbacks in 0.2.8 that were suitable but not selected.

We should also:

check the bandwdith range in the script's generated C comments
check the IP version, netblock, port, and Exit flag proportions in the script's stderr output

Over the longer term, we could:

reconsider whether to allow 2 fallbacks per operator (contact, family), while keeping 1 per IP
decide whether to change to an opt-out system, where we includes fallbacks unless operators specifically opt-out

to

We need to regenerate the fallback directory mirror list in 0.2.9 in case any of the 0.2.8 fallbacks have changed details or gone down.

This should not require another opt-in mailout, as we had ~70 additional fallbacks in 0.2.8 that were suitable but not selected.

We should also:

restore the 120 day stability period and 99% uptime requirement that were reduced in 0.2.8 due to legacy/trac#18050 (moved)
check the bandwdith range in the script's generated C comments
check the IP version, netblock, port, and Exit flag proportions in the script's stderr output

Over the longer term, we could:

reconsider whether to allow 2 fallbacks per operator (contact, family), while keeping 1 per IP
decide whether to change to an opt-out system, where we includes fallbacks unless operators specifically opt-out

agreed; we should do this about once per release series. (Also perhaps you can recruit somebody else to do it together with you this time, so that two people have experience doing it?)

Trac:
Keywords: 029-proposed deleted, 029-accepted added
Milestone: Tor: 0.2.??? to Tor: 0.2.9.x-final

Replying to nickm:

agreed; we should do this about once per release series. (Also perhaps you can recruit somebody else to do it together with you this time, so that two people have experience doing it?)

Sure. It would also help if that person reviewed legacy/trac#17158 (moved), legacy/trac#17905 (moved), and legacy/trac#18749 (moved) in 0.2.8.

I've started a branch with some more fallback whitelist / blacklist changes: fallbacks-201605 on https://github/com/teor2345/tor.git

We should merge it when we next regenerate the hard-coded list.

Trac:
Points: medium to 3

The latest fallback whitelist / blacklist changes are based on maint-0.2.8 (shortly after 0.2.8.4-rc) and are in fallbacks-201606 on https://github/com/teor2345/tor.git

I've merged the changes in the above branch into bug19071 (targeted at 0.2.8), because there's no way I'm keeping two concurrent whitelist/blacklist versions.

So there's nothing extra that needs to be merged into 0.2.9 at this point in time.

I'm now collecting new fallback whitelist entries on my branch fallbacks-201607 on https://github.com/teor2345/tor.git

We need to rebuild the fallback list entirely in 0.2.9 to make sure we've fixed legacy/trac#19163 (moved) - as of July 2016, every recommended tor version supports ntor.

Since legacy/trac#19610 (moved) causes IPv6-only clients to ask 15 IPv6 fallback directories for microdescriptors, we should increase the fallback numbers to 200 or 300, so we have a reasonable number of IPv6 fallbacks.

What I'd like to do in August is:

generate a draft list of 200 fallbacks,
find relays that are not whitelisted, but have high enough bandwidth and stability to have been included in the list, and
if the operators of these relays have not been contacted already, email them an opt-in request

What I'd like to do in September is:

add any new entries to the whitelist/blacklist,
generate an alpha list of 200 fallbacks,

What I'd like to do as our alphas start to stabilise (December 2016?) is:

check each fallback on the existing list to see if it's still working (same key, IP, and delivers a consensus)
comment-out the ones that aren't working

Trac:
Keywords: TorCoreTeam201609 deleted, TorCoreTeam201608 added

It's likely that legacy/trac#19989 (moved) applies to IPv6-only clients fetching microdescriptors from fallback directories (legacy/trac#19608 (moved)). So we should make sure there are some non-Exit IPv6-capable relays in the set.

I've updated the fallbacks-201607 branch on https://github.com/teor2345/tor.git based on legacy/trac#20010 (moved) and an operator email.

[fallbacks-201607 e7ed8bb] Update fallback addresses based on operator emails and tickets

We have extended the 0.2.9 release deadline, and so I can afford to do some of this in September.

I also want to:

make a wiki page so someone else can update fallbacks if needed (and so I don't forget)
change the fallback script so it has an "exclude existing" mode (exclude both the whitelist and blacklist), to make finding new potential fallbacks easier

Trac:
Keywords: TorCoreTeam201608 deleted, TorCoreTeam201609 added

Trac:
Status: new to assigned

I wrote a wiki page here: https://trac.torproject.org/projects/tor/wiki/doc/UpdatingFallbackDirectoryMirrors

Trac:
Parent: N/A to legacy/trac#20172 (moved)

And I made an updated whitelist and blacklist branch: fallbacks-029 on my github. The changes to the 0.2.8 list are in legacy/trac#20170 (moved).

I am up to step 2 of https://trac.torproject.org/projects/tor/wiki/doc/UpdatingFallbackDirectoryMirrors , but I can't seem to get a good OnionOO uptime document (legacy/trac#20193 (moved)), so I'm kind of stuck.

Still stuck? Or legacy/trac#20193 (moved) is not-a-bug so it's ok, and the remaining issue is to find time to proceed?

Replying to arma:

Still stuck? Or legacy/trac#20193 (moved) is not-a-bug so it's ok, and the remaining issue is to find time to proceed?

Finding time is one issue, because I have to update the script, do a mass email, collate responses, run the script again, and then have the list backported to 0.2.8 onwards.

Also, the current list is working fine, so there's not much urgency here. I would like to get an update in 0.2.9 though.

Trac:
potential_extra_fallbacks_2016-12-04

relays for the opt-in request 2016-12-04

Trac:
potential_extra_fallbacks_2016-12-04.log

log for selection of relays for the opt-in request 2016-12-04

My branch fallbacks-029-v2 on github https://github.com/teor2345/ has the changes I used to generate the lists I just attached.

I will send out an email to tor-relays asking operators on the list to opt-in, wait a week or two, collate the responses, and generate the final list.

Trac:
Status: assigned to needs_information

I still need to fix legacy/trac#20539 (moved), we don't need to exclude the bad 0.2.9 alpha versions until the final list.

Ok, it's done. https://lists.torproject.org/pipermail/tor-relays/2016-December/011113.html

legacy/trac#20539 (moved) is done, and I am contacting operators based on the results.

My draft branch for this is fallbacks-201612-v2 on github. https://github.com/teor2345/tor/tree/fallbacks-201612-v2

Trac:
Status: needs_information to needs_revision
Keywords: TorCoreTeam201609 deleted, TorCoreTeam201612 added

restore the 120 day stability period and 99% uptime requirement that were reduced in 0.2.8 due to legacy/trac#18050 (moved)

I initially tried 183 days stability and 99% uptime, but that excluded too many good relays. Instead, I am switching it back to 120 days (4 months) and 98%.

120 days is a compromise between the 6-monthly major tor release cycle, and actual relay stability.

98% still gives us (0.98)^3 = 94% of clients bootstrapping in the first 5 seconds, before contacting an authority. 8/(200*10 + 8) * 3 = 1.2% of clients try an authority in their first 3 attempts anyway, as the authorities are also in the fallback list (but with lower weight).

This works with 0.2.8, which has: Fallback 0, 1, 5, 16, ... Authority 6, 17, ... https://gitweb.torproject.org/tor.git/tree/src/or/config.c#n550 (The schedules are delays, which are additive.)

And 0.2.9 and later, which have: Fallback 0, 2, 5, 13, 33, ... Authority 6, 11, 27, ... https://trac.torproject.org/projects/tor/ticket/20534#comment:19

Replying to teor:

restore the 120 day stability period and 99% uptime requirement that were reduced in 0.2.8 due to legacy/trac#18050 (moved)

I initially tried 183 days stability and 99% uptime, but that excluded too many good relays. Instead, I am switching it back to 120 days (4 months) and 98%.

120 days is a compromise between the 6-monthly major tor release cycle, and actual relay stability.

98% still gives us (0.98)^3 = 94% of clients bootstrapping in the first 5 seconds, before contacting an authority. 8/(200*10 + 8) * 3 = 1.2% of clients try an authority in their first 3 attempts anyway, as the authorities are also in the fallback list (but with lower weight).

We can't require this level of stability, because so many relays are being excluded due to the recommended versions and legacy/trac#20539 (moved).

Instead, I chose 7 days and 90%, which means at least 73% of fallbacks can bootstrap without contacting an authority.

We can merge the latest version of my github fallbacks-201612-* branch containing the script, whitelist, and blacklist to master in this ticket.

The current draft branch used to build the list in legacy/trac#20170 (moved) is fallbacks-201612-v3 on my github: https://github.com/teor2345/tor/tree/fallbacks-201612-v3

If someone likes reading python, I'd love a review.

Trac:
Status: needs_revision to needs_review

Updated the branch: fallbacks-201612-v4 https://github.com/teor2345/tor/tree/fallbacks-201612-v4

Link not found?

Fixed

Trac:
draft_fallback_dirs_20161210_1121_2594b7b

Draft fallback list 10 Dec 2016 generated from commit 2594b7b

Trac:
draft_fallback_dirs_20161210_1121_2594b7b.log

Selection log for draft fallback list 10 Dec 2016 generated from commit 2594b7b

I sent an email to tor-relays and to all the relay operators on the draft list. I'll leave it another week, then create the final list. Details in legacy/trac#20173 (moved).

The final script and fallback whitelist and blacklist are in fallbacks-20161219 commit fcf19f8b.

The final list is in legacy/trac#20170 (moved).

Trac:
Milestone: Tor: 0.2.9.x-final to Tor: 0.3.0.x-final
Actualpoints: N/A to 5

Trac:
fallback_dirs_final_201612190411_fcf19f8b545a30a6cc9e6adde66e8ca2cb6b3bca.log

Log file for final fallback list generation

Hi! Is there a master branch I can merge here to get all the fallback directory script changes into mainline Tor?

Branch fallbacks-20161219 on my github. (Sorry, I thought I'd posted that somewhere already.)

All the child tickets can close as well.

Merged! Thanks!

All the child tickets can close as well.

Done!

Trac:
Resolution: N/A to fixed
Status: needs_review to closed

closed

changed time estimate to 24h

Regenerate fallback list for 0.2.9

Child items ...

Activity