Add IPv4 Fallback Directory List to tor, active by default

changed milestone to %Tor: 0.2.8.x-final in legacy/trac

Trac:
Status: new to needs_information

Previous work on this in legacy/trac#8374 (moved) wanted IPv6 directory mirrors, but the sample file provided doesn't contain any.

Trac:
Parent: N/A to legacy/trac#8374 (moved)

legacy/trac#15228 (moved) appears to be previous work on this, or a potential duplicate/parent.

Trac:
Parent: legacy/trac#8374 (moved) to legacy/trac#15228 (moved)

Do you want to contact the operator of each directory mirror for permission?

Questions from legacy/trac#15774 (moved):

Do we need to wait until a majority of clients update to versions with this change, controlled by a consensus parameter? (Otherwise, using any entry in the file itself would allow clients to effectively be partitioned from the rest of the network by their behaviour.)

While I'm making a list, do we need to modify the existing proposal which describes fallback directories?

Is this change proposed for 0.2.7?
Or all currently supported releases?

Do we need a new configuration option to give the location of the Fallback Directories file(s)?
How should this interact with the existing FallbackDir option?
I propose multiple files and options are all cumulative, but de-duplicated.

Suggestions for the script: have it also emit (in comments) the name and contact info for each such directory?

Replying to nickm:

Suggestions for the script: have it also emit (in comments) the name and contact info for each such directory?

Updated. Sample output at https://www.palfrader.org/volatile/fallback-dir/out

Thanks, weasel.

I am going to proceed with developing this with the following assumptions:

We don't want to include IPv6 fallback dirs until legacy/trac#6027 (moved) (IPv6 only bootstrap) is fixed
We want to distribute the sample output as a separate file, like the geoip files
- This will need a new config option, say FallbackDirFile
- We won't hardcode any FallbackDirs in strings in the tor binary (we hardcode the directory authorities in strings in the binary)
We want the list of fallback directories to be the union of:
- All FallbackDir lines in the FallbackDirFile file
- All FallbackDir lines in the config file
- With duplicates handled the way current FallbackDir duplicates are handled (ignored? not handled?)

I assume we want this in 0.2.7 from the details in legacy/trac#15228 (moved) (which is for fallback directories generally), which I've just copied down to this ticket.

Edit: speling

Trac:
Summary: Add Fallback Directory List to add_default_fallback_dir_servers() to Add IPv4 Fallback Directory List to add_default_fallback_dir_servers()
Milestone: Tor: 0.2.8.x-final to Tor: 0.2.7.x-final
Priority: normal to major
Status: needs_information to accepted
Owner: N/A to teor
Keywords: N/A deleted, tor-dist, 027-triaged-1-in added

Apologies for my presumption/assumptions, Nick:

Would you like the fallback directories in a file (easier to change on a regular basis), or in a string in the code (easier to implement without any extra code apart from the array of strings)?

I think that having it in a file would make sense to me; it is almost exactly why torrc.defaults exists. Are there any obstacles to that?

No, just a small amount of file loading code, which will work much like torrc.defaults or the geoip files. So I can use them as my example. Then I can access the configured list in add_default_fallback_dir_servers.

I think that the priority should be: (lowest to highest)

torrc.defaults FallbackDir lines
default FallbackDirFile
torrc (including any custom FallbackDirFile)
command-line arguments

Generally, I'd expect a union of all those lists, except when it comes to plusses and minuses and the weird option overriding interactions I don't quite understand. I'll trust the config code to do the right thing here.

In a test network, we'll also need to disable the default FallbackDirFile, otherwise test networks could fallback to the public tor network. Which would be weird and unhelpful.

Most test networks are configured to just override the authorities, and I wouldn't want to break that with this change.

nickm said on IRC:

I don't think it needs a new file torrc.defaults should be fine, I think. Right? or rather, I will believe it needs a new file if I know why torrc.defaults won't work. :)

I replied:

how do we stop a regression to legacy/trac#15642 (moved) - how do we know whether the FallbackDir lines have come from the [public tor] defaults list, or a list supplied with a test network? Or do we just disable [all] fallback directories in a test network?

Trac:
Status: accepted to needs_information

nickm and I spoke on IRC, and he said that we might actually want to test the fallback directories feature in a test network sometime. That would be prudent.

So, I suggest that we:

Stick the default fallback directories in torrc.defaults
- The final set of FallbackDirs will then be the union of torrc.defaults, torrc, and command-line
Create a new FallbackDirsMirrorDefaultDirAuths option set to true by default
Only use FallbackDirs if the current authories satisfy FallbackDirsMirrorDefaultDirAuths, that is:
- If FallbackDirsMirrorDefaultDirAuths is true, and the default directory authorities are being used, use the configured FallbackDirs
- If FallbackDirsMirrorDefaultDirAuths is false, and custom directory authorities are being used, use the configured FallbackDirs
- Otherwise, the FallbackDirs don't mirror the authorities, so don't use any FallbackDirs.

Then the public network, and test networks with no fallback dirs will continue to work as expected. [Edit: clarify "works as expected" cases]

Anyone wanting to test FallbackDirs in a test network will need to:

Set FallbackDirsMirrorDefaultDirAuths to false
Disable the default fallback directories in torrc.defaults, or, more likely, disable the use of torrc.defaults entirely on the command-line

I think it's the simplest way to avoid a regression to legacy/trac#15642 (moved). Otherwise test networks could fallback to the public tor network.

This avoids an additional FallbackDirFile, and also avoids weird special-casing in the code. It puts an additional configuration burden on test network operators who want to use FallbackDirs (a tiny group), and gets it right by default for everyone else.

Trac:
Status: needs_information to accepted

Modify title to reflect goal (not implementation) Add answers to questions asked in enhancement summary.

Trac:
Description: weasel writes on tor-dev:

Tor has included a feature to fetch the initial consensus from nodes other than the authorities for a while now. We just haven't shipped a list of alternate locations for clients to go to yet.

Reasons why we might want to ship tor with a list of additional places where clients can find the consensus is that it makes authority reachability and BW less important.

At the last Tor dev meeting we came up with a list of arbitrary requirements that nodes should meet to be included in this list.

We want them to have been around and using their current key, address, and port for a while now (120 days), and have been running, a guard, and a v2 directory mirror for most of that time.

I have written a script to come up with a list of notes that match our criteria. It's currently at https://www.palfrader.org/volatile/fallback-dir/get-fallback-dir-candidates

It currently produces https://www.palfrader.org/volatile/2015-04-17-VjBkc8DWV8c/list

See https://lists.torproject.org/pipermail/tor-dev/2015-April/008674.html

This file current has 329 entries, and takes up approximately 32kB. If we hard-coded it in the binary like the authorities, it would increase the binary size by approximately 2% on my platform.

Do we expect this in by 0.2.7?

Do we want to work on a signed file first (legacy/trac#15774 (moved))? (A signed file needs a well-defined threat model and signature verification has to work without access to the authorities or fallback directories.)

to

weasel writes on tor-dev:

Tor has included a feature to fetch the initial consensus from nodes other than the authorities for a while now. We just haven't shipped a list of alternate locations for clients to go to yet.

Reasons why we might want to ship tor with a list of additional places where clients can find the consensus is that it makes authority reachability and BW less important.

At the last Tor dev meeting we came up with a list of arbitrary requirements that nodes should meet to be included in this list.

We want them to have been around and using their current key, address, and port for a while now (120 days), and have been running, a guard, and a v2 directory mirror for most of that time.

I have written a script to come up with a list of notes that match our criteria. It's currently at https://www.palfrader.org/volatile/fallback-dir/get-fallback-dir-candidates

It currently produces https://www.palfrader.org/volatile/2015-04-17-VjBkc8DWV8c/list

See https://lists.torproject.org/pipermail/tor-dev/2015-April/008674.html

This file current has 329 entries, and takes up approximately 32kB. If we hard-coded it in the binary like the authorities, it would increase the binary size by approximately 2% on my platform.

Edit: nickm favours putting it in torrc.defaults

Do we expect this in by 0.2.7?

Edit: Yes

Do we want to work on a signed file first (legacy/trac#15774 (moved))? (A signed file needs a well-defined threat model and signature verification has to work without access to the authorities or fallback directories.)

Edit: No clear threat model, defer.
Summary: Add IPv4 Fallback Directory List to add_default_fallback_dir_servers() to Add IPv4 Fallback Directory List to tor, active by default

weasel and I spoke on IRC and torrc.defaults is for package maintainers, not upstream (us). So I'll create a patch to do the IPv4 portion of this in code, using a script to generate an array of C strings, which is then included in the existing empty array in add_default_fallback_dir_servers().

I have made the following changes to weasel's script:

Changed output to C comments and strings
removed potentially harmful characters to avoid injection attacks
Remove relays that have ever had a BadExit flag
No FallbackDirs currently have a BadExit flag
Remove relays that are not running a recommended_version (22/353)
every FallbackDir reports a version
Added IPv6 for legacy/trac#8374 (moved)
44/353 FallbackDirs have at least one IPv6 address
It's not clear what additional work needs to be done for IPv6 directory fetches (legacy/trac#6027 (moved)), but the ipv6 lines can easily be removed if we decide they're better left out. I think they'd make getting legacy/trac#6027 (moved) working easier if they were left in.
Implemented stable sorting of secondary orport addresses, so that we always choose the same IP addresses, even if OnionOO changes the order it returns them in (the secondary address order is documented as arbitrary)
This affects IPv6 addresses (44/353), and IPv4 addresses where we fall back to a secondary ORPort address because the IP addresses of the DirPort and primary ORPort address don't match (none at present).
Implemented last-modified-date to reduce load on OnionOO
cache the files on the local filesystem (100MB) and only re-download them when they change

This leaves 331 default FallbackDirs. Is this about what we were expecting?

I'll attach the modified script and the candidate include file to this task after this comment goes through. The patch will include them both.

It doesn't want FallbackDir at the start of the strings. Fixed.

Trac:
Description: weasel writes on tor-dev:

Tor has included a feature to fetch the initial consensus from nodes other than the authorities for a while now. We just haven't shipped a list of alternate locations for clients to go to yet.

Reasons why we might want to ship tor with a list of additional places where clients can find the consensus is that it makes authority reachability and BW less important.

At the last Tor dev meeting we came up with a list of arbitrary requirements that nodes should meet to be included in this list.

We want them to have been around and using their current key, address, and port for a while now (120 days), and have been running, a guard, and a v2 directory mirror for most of that time.

I have written a script to come up with a list of notes that match our criteria. It's currently at https://www.palfrader.org/volatile/fallback-dir/get-fallback-dir-candidates

It currently produces https://www.palfrader.org/volatile/2015-04-17-VjBkc8DWV8c/list

See https://lists.torproject.org/pipermail/tor-dev/2015-April/008674.html

This file current has 329 entries, and takes up approximately 32kB. If we hard-coded it in the binary like the authorities, it would increase the binary size by approximately 2% on my platform.

Edit: nickm favours putting it in torrc.defaults

Do we expect this in by 0.2.7?

Edit: Yes

Do we want to work on a signed file first (legacy/trac#15774 (moved))? (A signed file needs a well-defined threat model and signature verification has to work without access to the authorities or fallback directories.)

Edit: No clear threat model, defer.

to

weasel writes on tor-dev:

Tor has included a feature to fetch the initial consensus from nodes other than the authorities for a while now. We just haven't shipped a list of alternate locations for clients to go to yet.

Reasons why we might want to ship tor with a list of additional places where clients can find the consensus is that it makes authority reachability and BW less important.

At the last Tor dev meeting we came up with a list of arbitrary requirements that nodes should meet to be included in this list.

We want them to have been around and using their current key, address, and port for a while now (120 days), and have been running, a guard, and a v2 directory mirror for most of that time.

I have written a script to come up with a list of notes that match our criteria. It's currently at https://www.palfrader.org/volatile/fallback-dir/get-fallback-dir-candidates

It currently produces https://www.palfrader.org/volatile/2015-04-17-VjBkc8DWV8c/list

See https://lists.torproject.org/pipermail/tor-dev/2015-April/008674.html

This file current has 329 entries, and takes up approximately 32kB. If we hard-coded it in the binary like the authorities, it would increase the binary size by approximately 2% on my platform.

Edit: nickm favours putting it in torrc.defaults Edit 2: weasel notes torrc.defaults is for package maintainers. Putting it in a list of strings in the code. Much like the authorities.

Do we expect this in by 0.2.7?

Edit: Yes

Do we want to work on a signed file first (legacy/trac#15774 (moved))? (A signed file needs a well-defined threat model and signature verification has to work without access to the authorities or fallback directories.)

Edit: No clear threat model, defer.

I've created a branch with these changes:

Branch: feature15775-fallback Repository: https://github.com/teor2345/tor.git

This branch includes:

scripts/maint/updateFallbackDirs.py, which generates
src/or/fallback_dirs.inc, which is included by
src/or/config.c (the implementation) and
src/test/test_config.c (a unit test) It also includes the standard changes file.

Trac:
Status: accepted to needs_review

I have just tested bootstrap with all the authorities blocked in ExcludeNodes, and without any of the authorities blocked. In both cases, bootstrap works, and the first directory contacted is one of the fallbacks.

Next step: contact operators for opt-in?

This change came through in 0.2.7.1-alpha, do we need to do this for ~300 FallbackDirs as well?

 Add DirAuthority lines for default directory authorities to the
 output of the "GETINFO config/defaults" command if not already
 present. Implements ticket 14840.

It would help to record the relays_published date of each Onionoo document we download, otherwise it will be impossible to trace back from the generated list of fallback directories to the source documents / consensuses.

If different tor developers use documents with the same relays_published date from the same Onionoo instance, they should get the same list of fallback directories. This allows us to cross-check the list of fallback directories.

This fix is now in the branch feature15775-fallback on teor2345 github.

Further fixes in feature15775-fallback-update

For a list of tasks related to IPv4 and IPv6 fallback and bootstrap testing, see: https://lists.torproject.org/pipermail/tor-relays/2015-June/007109.html

teor's TODO list for this task: do prep work for contacting operators of fallback candidates for opt-in:

[18:04] <teor> Based on https://trac.torproject.org/projects/tor/ticket/15775#comment:20
[18:04] <teor> I think we wanted to contact operators for opt-in
[18:05] <nickm> hm. I think you're right.  Did we have somebody offer to do that?
[18:05] <teor> Does it need to be someone @torproject.org
[18:05] <teor> For the officialness of it all?
[18:08] <teor> I can imagine splitting it into: 1. Demangle emails; 2. Draft letter; 3. Send emails (BCC, of course)
[18:08] <nickm> Yeah, #3 should be from a @tp.o address.
[18:08] <teor> And I could do 1 & 2 at least
[18:08] <nickm> Thanks, that would be awesome
[18:09] <nickm> the letter should explain that it's opt-in, explain what we're asking for, let the know that it means that their IP address shouldn't be very unstable, etc
[18:09] <teor> and the purpose of fallbacks in circumventing auth blocks and spreading load
...
[18:20] <nickm> teor: I'm afraid that the tangle exists in my head as well as in the code.  I think it's fine to mention IPv6 addresses in the email we send out...

Trac:
Status: needs_review to needs_information

Alternately, weasel suggests mailing tor-relays:

[18:38] <weasel> I think a mail to tor-relays and a note in the next tor weekly news that links to the mail ought to suffice.
[18:38] <weasel> basically saying "we intent to list all these relays and all future ones that qualify as fallback" yadada.
[18:39] <weasel> if we wanted to be extra cautious, we could try finding a small number, say 10 or so, and try with those as a first step.
[18:39] <weasel> that could be opt-in without the mails to public places
[18:39] <weasel> and also it'd scale saner.

Updated script in branch feature15775-fallback-update:

sort by bandwidth weight (previous sort was by key)
take the TOP_N_BY_WEIGHT relays, currently set to 512 as a sanity check

Possible opt-out mechanisms:

Manual: operator contacts tor-relays, relay key is added to exclusion list (not yet implemented)
Automated: operator configures relay to disable DirPort

To enable opt-in/opt-out, I need to add a whitelist/blacklist feature with IP addresses and/or relay IDs.

In the whitelist, I'd prefer to list both and only include if they are both present in the descriptor.

In the blacklist, I'd prefer to list both and exclude if either is present.

I have updated branch feature15775-fallback-update with gitignore lines for the files generated by the fallback directory update script.

Trac:
torrc.block_authorities

Torrc which blocks authorities for testing fallback directories

We want to start with opt-in via email to tor-relays (with proposed list) and operator contact info. This could be drafted by anyone, but needs to be sent from a torproject.org email address.

Operators who opt-in would have their relay fingerprints and IPs added to the list, and be included on the list if both the fingerprint and IP match.

Other considerations, from IRC discussions:

There is a tradeoff between a small list, where a single operator learns about more clients, and a large list, which may become unmanageable. (And it might slow down bootstrap if too many unreliable relays are included.)

Every release containing a list of fallback directories will forever contact those directories on initial bootstrap. We'll need to ensure that the list is large enough that:

no one relay / IP gets overwhelming traffic
there is always an unblocked relay which can provide an updated consensus (in most cases, the authorities will do this, as long as they are unblocked)

This situation will improve once legacy/trac#15942 (moved) (exponential backoff) and legacy/trac#15935 (moved) (advisory-only old client disabling) are implemented.

I need to check what tor currently does if a fallback directory fails. The ideal behaviour is for it to stop contacting failed (fallback) directories, whether tor is running or not.

But I assume it just uses the current directory code.

dgoulet
do you plan to keep a "state file" of those hardcoded IPs meaning if they fail too many times in time period X, you note it down as invalid for Y days?
or it's just, no DirAuth, I'm trying each of them non stop until one replies?

teor
I don't know what the current implementation does with respect to failed fallback directories
I will make a note and check

Trac:
Keywords: N/A deleted, TorCoreTeam201507 added

Not going to happen in July

Trac:
Keywords: TorCoreTeam201507 deleted, TorCoreTeam201508 added

Directory Failures

Directory failures are handled in directory_get_from_dirserver:

When all the dirservers are down, we call(back) directory_all_unreachable and wait for an application connection before we try any more dirservers. (See legacy/trac#16762 (moved) for the most recent change to this code.)

The some-dirservers-are-down behaviour seems to work just as well - we ignore them until we've tried all the other ones.

I still need to:

Add the whitelist/blacklist feature to the fallback directories list
Write a wiki page explaining fallback directories
Draft an email requesting that relay operators opt-in their ID and IP as a fallback directory
Convince someone @ torproject.org to send the email to tor-relays
someone @ torproject.org collates responses and adds them to the whitelist
Once we have enough whitelist entries, we merge the whitelisted fallback directories into the code

Open Questions:

How many relays should we have in the fallback directory list? 100?
- Modify the code to report the count at the top of the file
- Modify the code to #error out if we fall below a minimum relay count (100?)
Should we adjust weights so no relay has more than 1% (?) of the weight on the list?
- Modify the code to report the total weight, min, and max proportions at the top of the file
- Modify the code to adjust weights so that no relay gets more than a maximum proportion of clients (5%?, or 5/(minimum relay count)?) Dir Auth vs Fallback Weights
The dir auths don't have set weights, but the fallback dirs do. Therefore, the auths get weighted 1.0 (the default weight). The fallback weights range from 300K to 300, so the dir auths will hardly ever get contacted (about 1 in 2 million clients, or 1 per day). Is this the desired behaviour? (Edit: fallback directories are prioritised before the authorities. This paragraph is incorrect.)

I need guidance on whether the numbers above are sensible. But the first step is reporting the actual numbers at the top of the output file.

Do we also want to add these to GETINFO config/defaults if not already present, or to a separate GETINFO config/fallback_directories?

Split off into legacy/trac#16774 (moved). (If only I'd waited another ticket, the numerology would have been perfect...)

Currently, our top fallback directories are weighted around 2%-3% of the total eligible fallback directory weight. It's possible to reduce that to around 1.6% by redistributing some of the consensus weight across the rest of the relays (or, equivalently, reducing the consensus weight of those relays to some maximum). But making the maximum less than about 1.5% tends to warp the distribution of weights too much.

The updated code is in my teor2345 github branch feature15775-fallback-update-v2. I still need to implement whitelisting and blacklisting of fallback directories in the script.

Is it OK if a fallback directory sees 1 in 50 client bootstraps (2%)? (The final figure may change depending on how many relays opt-in to be fallback directories.) We'd need to widen eligibility criteria a bit to get it down to 1%.

Trac:
Status: needs_information to needs_revision

Trac:
Keywords: N/A deleted, DoS, SponsorU added

Trac:
Keywords: TorCoreTeam201508 deleted, TorCoreTeam201509 added

Trac:
Milestone: Tor: 0.2.7.x-final to Tor: 0.2.8.x-final

Trac:
Keywords: TorCoreTeam201509 deleted, TorCoreTeam201510 added

Trac:
Actualpoints: N/A to 20
Keywords: N/A deleted, 0.2.8.x-triage added
Sponsor: N/A to U

Trac:
Keywords: N/A deleted, 028-triaged added

Trac:
Sponsor: U to SponsorU

Trac:
Points: N/A to medium/large

The current behaviour of Fallback Directories is:

tor contacts fallback directories before the directory authorities
tor will try 2 fallback directories, then wait for a minute or two before trying another fallback directory

Do we want to:

increase the number of fallbacks that are tried quickly to 3?
try the authorities before the fallback directories?
- for the trial?
- for the feature release?
try N fallback directories quickly, then use the authorities?

Add IPv4 Fallback Directory List to tor, active by default

Child items ...

Activity