Add IPv4 Fallback Directory List to tor, active by default

changed milestone to %Tor: 0.2.8.x-final

added 0.2.8.x-triage 027-triaged-1-in 028-triaged DoS MikePerry201512R SponsorU TorCoreTeam201512 actualpoints::20 component::core tor/tor milestone::Tor: 0.2.8.x-final owner::teor parent::17158 points::medium/large priority::high resolution::fixed severity::major sponsor::U status::closed tor-dist tor-dos type::enhancement version::tor 0.2.4.7-alpha labels

Trac:
Status: new to needs_information

Previous work on this in #8374 (moved) wanted IPv6 directory mirrors, but the sample file provided doesn't contain any.

Trac:
Parent: N/A to #8374 (moved)

#15228 (moved) appears to be previous work on this, or a potential duplicate/parent.

Trac:
Parent: #8374 (moved) to #15228 (moved)

Do you want to contact the operator of each directory mirror for permission?

Questions from #15774 (moved):

Do we need to wait until a majority of clients update to versions with this change, controlled by a consensus parameter? (Otherwise, using any entry in the file itself would allow clients to effectively be partitioned from the rest of the network by their behaviour.)

While I'm making a list, do we need to modify the existing proposal which describes fallback directories?

Is this change proposed for 0.2.7?
Or all currently supported releases?

Do we need a new configuration option to give the location of the Fallback Directories file(s)?
How should this interact with the existing FallbackDir option?
I propose multiple files and options are all cumulative, but de-duplicated.

Suggestions for the script: have it also emit (in comments) the name and contact info for each such directory?

Replying to nickm:

Suggestions for the script: have it also emit (in comments) the name and contact info for each such directory?

Updated. Sample output at https://www.palfrader.org/volatile/fallback-dir/out

Thanks, weasel.

I am going to proceed with developing this with the following assumptions:

We don't want to include IPv6 fallback dirs until #6027 (moved) (IPv6 only bootstrap) is fixed
We want to distribute the sample output as a separate file, like the geoip files
- This will need a new config option, say FallbackDirFile
- We won't hardcode any FallbackDirs in strings in the tor binary (we hardcode the directory authorities in strings in the binary)
We want the list of fallback directories to be the union of:
- All FallbackDir lines in the FallbackDirFile file
- All FallbackDir lines in the config file
- With duplicates handled the way current FallbackDir duplicates are handled (ignored? not handled?)

I assume we want this in 0.2.7 from the details in #15228 (moved) (which is for fallback directories generally), which I've just copied down to this ticket.

Edit: speling

Trac:
Summary: Add Fallback Directory List to add_default_fallback_dir_servers() to Add IPv4 Fallback Directory List to add_default_fallback_dir_servers()
Milestone: Tor: 0.2.8.x-final to Tor: 0.2.7.x-final
Priority: normal to major
Status: needs_information to accepted
Owner: N/A to teor
Keywords: N/A deleted, tor-dist, 027-triaged-1-in added

Apologies for my presumption/assumptions, Nick:

Would you like the fallback directories in a file (easier to change on a regular basis), or in a string in the code (easier to implement without any extra code apart from the array of strings)?

I think that having it in a file would make sense to me; it is almost exactly why torrc.defaults exists. Are there any obstacles to that?

No, just a small amount of file loading code, which will work much like torrc.defaults or the geoip files. So I can use them as my example. Then I can access the configured list in add_default_fallback_dir_servers.

I think that the priority should be: (lowest to highest)

torrc.defaults FallbackDir lines
default FallbackDirFile
torrc (including any custom FallbackDirFile)
command-line arguments

Generally, I'd expect a union of all those lists, except when it comes to plusses and minuses and the weird option overriding interactions I don't quite understand. I'll trust the config code to do the right thing here.

In a test network, we'll also need to disable the default FallbackDirFile, otherwise test networks could fallback to the public tor network. Which would be weird and unhelpful.

Most test networks are configured to just override the authorities, and I wouldn't want to break that with this change.

nickm said on IRC:

I don't think it needs a new file torrc.defaults should be fine, I think. Right? or rather, I will believe it needs a new file if I know why torrc.defaults won't work. :)

I replied:

how do we stop a regression to #15642 (moved) - how do we know whether the FallbackDir lines have come from the [public tor] defaults list, or a list supplied with a test network? Or do we just disable [all] fallback directories in a test network?

Trac:
Status: accepted to needs_information

nickm and I spoke on IRC, and he said that we might actually want to test the fallback directories feature in a test network sometime. That would be prudent.

So, I suggest that we:

Stick the default fallback directories in torrc.defaults
- The final set of FallbackDirs will then be the union of torrc.defaults, torrc, and command-line
Create a new FallbackDirsMirrorDefaultDirAuths option set to true by default
Only use FallbackDirs if the current authories satisfy FallbackDirsMirrorDefaultDirAuths, that is:
- If FallbackDirsMirrorDefaultDirAuths is true, and the default directory authorities are being used, use the configured FallbackDirs
- If FallbackDirsMirrorDefaultDirAuths is false, and custom directory authorities are being used, use the configured FallbackDirs
- Otherwise, the FallbackDirs don't mirror the authorities, so don't use any FallbackDirs.

Then the public network, and test networks with no fallback dirs will continue to work as expected. [Edit: clarify "works as expected" cases]

Anyone wanting to test FallbackDirs in a test network will need to:

Set FallbackDirsMirrorDefaultDirAuths to false
Disable the default fallback directories in torrc.defaults, or, more likely, disable the use of torrc.defaults entirely on the command-line

I think it's the simplest way to avoid a regression to #15642 (moved). Otherwise test networks could fallback to the public tor network.

This avoids an additional FallbackDirFile, and also avoids weird special-casing in the code. It puts an additional configuration burden on test network operators who want to use FallbackDirs (a tiny group), and gets it right by default for everyone else.

Trac:
Status: needs_information to accepted

Modify title to reflect goal (not implementation) Add answers to questions asked in enhancement summary.

Trac:
Description: weasel writes on tor-dev:

Tor has included a feature to fetch the initial consensus from nodes other than the authorities for a while now. We just haven't shipped a list of alternate locations for clients to go to yet.

Reasons why we might want to ship tor with a list of additional places where clients can find the consensus is that it makes authority reachability and BW less important.

At the last Tor dev meeting we came up with a list of arbitrary requirements that nodes should meet to be included in this list.

We want them to have been around and using their current key, address, and port for a while now (120 days), and have been running, a guard, and a v2 directory mirror for most of that time.

I have written a script to come up with a list of notes that match our criteria. It's currently at https://www.palfrader.org/volatile/fallback-dir/get-fallback-dir-candidates

It currently produces https://www.palfrader.org/volatile/2015-04-17-VjBkc8DWV8c/list

See https://lists.torproject.org/pipermail/tor-dev/2015-April/008674.html

This file current has 329 entries, and takes up approximately 32kB. If we hard-coded it in the binary like the authorities, it would increase the binary size by approximately 2% on my platform.

Do we expect this in by 0.2.7?

Do we want to work on a signed file first (#15774 (moved))? (A signed file needs a well-defined threat model and signature verification has to work without access to the authorities or fallback directories.)

to

weasel writes on tor-dev:

Tor has included a feature to fetch the initial consensus from nodes other than the authorities for a while now. We just haven't shipped a list of alternate locations for clients to go to yet.

Reasons why we might want to ship tor with a list of additional places where clients can find the consensus is that it makes authority reachability and BW less important.

At the last Tor dev meeting we came up with a list of arbitrary requirements that nodes should meet to be included in this list.

We want them to have been around and using their current key, address, and port for a while now (120 days), and have been running, a guard, and a v2 directory mirror for most of that time.

I have written a script to come up with a list of notes that match our criteria. It's currently at https://www.palfrader.org/volatile/fallback-dir/get-fallback-dir-candidates

It currently produces https://www.palfrader.org/volatile/2015-04-17-VjBkc8DWV8c/list

See https://lists.torproject.org/pipermail/tor-dev/2015-April/008674.html

This file current has 329 entries, and takes up approximately 32kB. If we hard-coded it in the binary like the authorities, it would increase the binary size by approximately 2% on my platform.

Edit: nickm favours putting it in torrc.defaults

Do we expect this in by 0.2.7?

Edit: Yes

Do we want to work on a signed file first (#15774 (moved))? (A signed file needs a well-defined threat model and signature verification has to work without access to the authorities or fallback directories.)

Edit: No clear threat model, defer.
Summary: Add IPv4 Fallback Directory List to add_default_fallback_dir_servers() to Add IPv4 Fallback Directory List to tor, active by default

weasel and I spoke on IRC and torrc.defaults is for package maintainers, not upstream (us). So I'll create a patch to do the IPv4 portion of this in code, using a script to generate an array of C strings, which is then included in the existing empty array in add_default_fallback_dir_servers().

I have made the following changes to weasel's script:

Changed output to C comments and strings
removed potentially harmful characters to avoid injection attacks
Remove relays that have ever had a BadExit flag
No FallbackDirs currently have a BadExit flag
Remove relays that are not running a recommended_version (22/353)
every FallbackDir reports a version
Added IPv6 for #8374 (moved)
44/353 FallbackDirs have at least one IPv6 address
It's not clear what additional work needs to be done for IPv6 directory fetches (#6027 (moved)), but the ipv6 lines can easily be removed if we decide they're better left out. I think they'd make getting #6027 (moved) working easier if they were left in.
Implemented stable sorting of secondary orport addresses, so that we always choose the same IP addresses, even if OnionOO changes the order it returns them in (the secondary address order is documented as arbitrary)
This affects IPv6 addresses (44/353), and IPv4 addresses where we fall back to a secondary ORPort address because the IP addresses of the DirPort and primary ORPort address don't match (none at present).
Implemented last-modified-date to reduce load on OnionOO
cache the files on the local filesystem (100MB) and only re-download them when they change

This leaves 331 default FallbackDirs. Is this about what we were expecting?

I'll attach the modified script and the candidate include file to this task after this comment goes through. The patch will include them both.

It doesn't want FallbackDir at the start of the strings. Fixed.

Trac:
Description: weasel writes on tor-dev:

Tor has included a feature to fetch the initial consensus from nodes other than the authorities for a while now. We just haven't shipped a list of alternate locations for clients to go to yet.

Reasons why we might want to ship tor with a list of additional places where clients can find the consensus is that it makes authority reachability and BW less important.

At the last Tor dev meeting we came up with a list of arbitrary requirements that nodes should meet to be included in this list.

We want them to have been around and using their current key, address, and port for a while now (120 days), and have been running, a guard, and a v2 directory mirror for most of that time.

I have written a script to come up with a list of notes that match our criteria. It's currently at https://www.palfrader.org/volatile/fallback-dir/get-fallback-dir-candidates

It currently produces https://www.palfrader.org/volatile/2015-04-17-VjBkc8DWV8c/list

See https://lists.torproject.org/pipermail/tor-dev/2015-April/008674.html

This file current has 329 entries, and takes up approximately 32kB. If we hard-coded it in the binary like the authorities, it would increase the binary size by approximately 2% on my platform.

Edit: nickm favours putting it in torrc.defaults

Do we expect this in by 0.2.7?

Edit: Yes

Do we want to work on a signed file first (#15774 (moved))? (A signed file needs a well-defined threat model and signature verification has to work without access to the authorities or fallback directories.)

Edit: No clear threat model, defer.

to

weasel writes on tor-dev:

Tor has included a feature to fetch the initial consensus from nodes other than the authorities for a while now. We just haven't shipped a list of alternate locations for clients to go to yet.

Reasons why we might want to ship tor with a list of additional places where clients can find the consensus is that it makes authority reachability and BW less important.

At the last Tor dev meeting we came up with a list of arbitrary requirements that nodes should meet to be included in this list.

We want them to have been around and using their current key, address, and port for a while now (120 days), and have been running, a guard, and a v2 directory mirror for most of that time.

I have written a script to come up with a list of notes that match our criteria. It's currently at https://www.palfrader.org/volatile/fallback-dir/get-fallback-dir-candidates

It currently produces https://www.palfrader.org/volatile/2015-04-17-VjBkc8DWV8c/list

See https://lists.torproject.org/pipermail/tor-dev/2015-April/008674.html

This file current has 329 entries, and takes up approximately 32kB. If we hard-coded it in the binary like the authorities, it would increase the binary size by approximately 2% on my platform.

Edit: nickm favours putting it in torrc.defaults Edit 2: weasel notes torrc.defaults is for package maintainers. Putting it in a list of strings in the code. Much like the authorities.

Do we expect this in by 0.2.7?

Edit: Yes

Do we want to work on a signed file first (#15774 (moved))? (A signed file needs a well-defined threat model and signature verification has to work without access to the authorities or fallback directories.)

Edit: No clear threat model, defer.

Add IPv4 Fallback Directory List to tor, active by default

Child items ...

Activity