Formalize toggle override for non-Tor applications that follow RFC 7686

changed the description

@rhatto this is probably something for you to think of here?

added Icebox label

I am against this proposed change.

I do not believe that an environment variable is an appropriate mechanism for changing the name resolution behaviour of an application (or applications)
c-ares is "an asynchronous resolver library"; it must respond with NXDOMAIN if not tor-aware.

I'm not sure that there is a good solution here - adding an environment variable at this point is likely to make the situation more confusing rather than less; only some applications that implement RFC7686 will implement an addition at this stage.

I'm sure that the project considered this before RFC7686 was put forward and am interested in hearing what you have to say on the matter.

mentioned in issue tpo/onion-services/onion-support#53

Re-frame and discuss

Thanks @omni for pointing this change in c-ares and it's impact on curl.

Like @Kangie, I don't think environment variables are the way to recommend for every case, since this mechanism is not part of any internet standard as far as I know. It's a standard for some family of operating systems, so there's no guarantee that it would be ubiquitously supported.

The problem here seems not application compliance to the RFC, but applications not being aware that they are running in a Tor-enabled environment.

I can't say right now if there's an issue with the RFC, like if it didn't predict this use case, and I don't know if that was discussed back then.

I would then re-frame this issue in two levels:

How an application or library should discover that it's Tor-enabled, or part of a Tor-enabled environment?
Where to recommend/enforce this behavior? RFCs usually have greater adherence than Tor Specs, but amending an RFC does not seem to be an easy task.

Another approach would be advocating some switch/toggle functionality with each vendor, without having a spec for that, but maybe that would be sub-optimal.

In summary, I think more discussion is needed to understand the options here.

Just to clarify, this impact on curl through c-ares is since a few years but regardless since curl 8.1.0. RFC 7686 compliance is different between the two.

1. How an application or library should discover that it's Tor-enabled, or part of a Tor-enabled environment?

I'm not sure how an application would discover that it is running in a Tor-enabled environment without risk of leaking that there is a Tor interest, by resolving a .onion address or check.torproject.org or possibly other eaqually reliable means.

I understand that environment variables are only standard for some family of operating systems, as you say, and would be part of a broader suggestion on how to handle this. I've limited myself to this family of operating systems and have no suggestions outside of them.

As an example, it would be convenient to be able to do

ALLOW_DOT_ONION=1 git clone http://eweiibe6tdjsdprb4px6rqrzzcsi22m4koia44kc5pcjr7nec2rlxyad.onion/tpo/core/arti`

in an environment where traffic is transparrently proxied through Tor, where SOCKS proxying is not available.

2. Where to recommend/enforce this behavior?

It is my impression that there is no need to amend RFC 7686, as the key words SHOULD and SHOULD NOT are used for applications (see RFC 2119), but some sort of formalized guidance may be in order. Perhaps Tor Specs is not the place but something official from the Tor Project.

Recent discussions at github/curl:

Older discussions:

mentioned in issue tpo/applications/vpn#45 (closed)

marked this issue as related to tpo/onion-services/onion-support#53

imho, the curl solution is not ideal.

the curl solution

refuse to resolve any onion domain, regardless of environment.

https://github.com/curl/curl/discussions/11125#discussioncomment-5926463

This patch breaks systems with superior security to people running Tor on localhost, and it is important to let those of us with such systems opt out, anything otherwise is an unacceptable compromise to security.

in my case, the curl solution breaks torsocks, which is really unexpected:

$ torsocks curl http://2gzyxa5ihm7nsggfxnu52rck2vv4rvmdlkiu3zzui5du4xyclen53wid.onion/
curl: (6) Not resolving .onion address (RFC 7686)

$ torsocks git clone http://2gzyxa5ihm7nsggfxnu52rck2vv4rvmdlkiu3zzui5du4xyclen53wid.onion/
fatal: unable to access 'http://2gzyxa5ihm7nsggfxnu52rck2vv4rvmdlkiu3zzui5du4xyclen53wid.onion/': Not resolving .onion address (RFC 7686)

RFC 7686:

Application Software: Applications (including proxies) that implement the Tor protocol MUST recognize .onion names as special by either accessing them directly or using a proxy (e.g., SOCKS [RFC1928]) to do so. Applications that do not implement the Tor protocol SHOULD generate an error upon the use of .onion and SHOULD NOT perform a DNS lookup.

alternative solution: separate DNS system

normalize a separate DNS system on port 9053, using /etc/resolv.onion.conf to find DNS servers.

see also: man resolv.conf

alternative solution: local DNS server / DNS proxy / DNS firewall

normalize "DNS firewalls" in operating systems. a DNS firewall is a local DNS server: all DNS requests from apps go to localhost. some requests are blocked, other requests are forwarded to other DNS servers.

This could be based on dnsmasq:

dnsmasq --server=/onion/127.0.0.1#9053

https://dnsmasq.org/docs/dnsmasq-man.html

-S, --local, --server=[/[]/[domain/]][[#]][@][@[#]]

Specify upstream servers directly.

Setting this flag does not suppress reading of /etc/resolv.conf, use --no-resolv to do that.

If one or more optional domains are given, that server is used only for those domains and they are queried only using the specified server. This is intended for private nameservers: if you have a nameserver on your network which deals with names of the form xxx.internal.thekelleys.org.uk at 192.168.1.1 then giving the flag --server=/internal.thekelleys.org.uk/192.168.1.1 will send all queries for internal machines to that nameserver, everything else will go to the servers in /etc/resolv.conf.

A non-standard port may be specified as part of the IP address using a # character.

now the problem is shifted to the DNS firewall, but this is better than an environment variable, because a DNS firewall can have dynamic behavior = it can monitor the live status of a "Tor-enabled environment".

in the example with dnsmasq: DNS requests for *.onion are forwarded to the DNS server 127.0.0.1:9053. if there is no such DNS server, the DNS request fails. optionally, dnsmasq can be configured to use a DNS server in the local network, for example dnsmasq --server=/onion/192.168.0.1#9053.

see also: Linux DNS Resolver for Onions.txt

in my case, the curl solution breaks torsocks, which is really unexpected

But in line with RFC7686 (not saying it's "right", just a somewhat predictable consequence of fixing a reported bug [lack of RFC compliance] in cURL).

alternative solution: separate DNS system

I think you'll find that we're still going to run afoul of RFC 7686 here; how can a client application know that's Tor DNS and not an arbitrary service listening on 9053?

alternative solution: local DNS server / DNS proxy / DNS firewall

Same comment as above

The issue here seems to be less about 'find a workaround for .onion resolution' and more

How can client applications (safely):

discover that they're in a Tor-enabled environment
resolve onion services only via Tor in that circumstance
not leak .onion resolution attempts at all

Right now, not making these requests in the first place is the safest (and correct) thing to do, however inconvenient it may be.

Rather than immediately trying to come up with a band-aid approach a sane mechanism needs to be implemented to:

Prevent each application from coming up with their own solution
Prevent inconsistency in .onion resolution (i.e. no "oh it only leaks if DO_ONION_RESOLUTION is set")
Provide a standardised mechanism for applications that want to be Tor aware to discover that they're in a Tor-enabled environment.

On this last point it may be worth looking into compile-time (rather than runtime) toggles - I don't think it's too much overhead to recompile something for your transproxy (or otherwise special) environment. Just a thought, but saner than accidentally leaking information because an environment got polluted (or someone sneakily set a variable...)

Edit:

Something conceptually like the following pseudo-curl code:


#ifdef HAVE_TOR
#include <tor/tor-daemon.h>
#endif

. . .
dns_resolution_function() {
. . .
	/* onion resolution */
	if(hostname_len >= 7 &&
	(curl_strequal(&hostname[hostname_len-6], ".onion") ||
	curl_strequal(&hostname[hostname_len-7], ".onion."))) {
#ifdef HAVE_TOR
		/* Check if Tor is running in our environment */
		if (tor_are_you_there() == false) {
			failf(data, "Not resolving .onion address; Tor isn't ready");
			return CURLRESOLV_ERROR;
		}
#else
		/* Refuse because RFC 7686 */
		failf(data, "Not resolving .onion address (RFC 7686)");
		return CURLRESOLV_ERROR;
#endif
	}
. . .
	do_dns_resolution_stuff();

. . .

}

This has its own issues (in that there's no sane way to check whether or not tor is listening in a transproxied environment).

This brings me to another question that I have: Is transproxying the recommended / sane approach for what these users want to do?

I see a lot of claims to better security from individuals with "interesting" network configurations, but no evidence that it's the better / preferred approach (I'm also not digging into it myself right now...).

Finally, once a mechanism is selected (and discussed, and has any rough edges rounded off) there needs to be some way to get this standard out into the wider community - perhaps via another RFC.

This is not going to be a fast solution, however the cURL onion resolution patch is only a few lines - in the interim feel free to patch your local sources!

how can a client application know that's Tor DNS and not an arbitrary service listening on 9053?

the same trust-problem exists for the tor socks5 proxy, by default on 127.0.0.1:9050, or for torsocks on TORSOCKS_TOR_ADDRESS:TORSOCKS_TOR_PORT.

with a separate DNS system for tor, torsocks would add two options: TORSOCKS_TOR_DNS_ADDRESS:TORSOCKS_TOR_DNS_PORT.

separate DNS system for tor

for .onion domains, but also for all other domains, to allow anonymous resolution of domains.

How can client applications (safely) discover that they're in a Tor-enabled environment

trust the environment variables, and maybe verify the configuration by sending test-requests to tor's socks5 and DNS servers. such test-requests would give weak security, because obviously, such test-requests could be tricked by malicious servers.

but when localhost is compromised by malicious configurations, or by malicious software, the problem should be out-of-scope for tor.

it may be worth looking into compile-time (rather than runtime) toggles - I don't think it's too much overhead to recompile something for your transproxy (or otherwise special) environment.

using a patched curl is expensive on nixos linux and guix linux, where all reverse-dependencies of curl have to be recompiled.

How can client applications (safely) discover that they're in a Tor-enabled environment

i still believe "the user knows best" about the security status of his system. so curl should

read a config file like /etc/curl.conf with allow_dot_onion=1
read an environment variable like CURL_ALLOW_DOT_ONION=1
accept a CLI flag like --allow-dot-onion=1
accept an API option like curl_allow_dot_onion(1)

sure, all these options can be tricked by an attacker, but when an attacker has this kind of access, it's the user's fault, not curl's fault.

Prevent inconsistency in .onion resolution (i.e. no "oh it only leaks if DO_ONION_RESOLUTION is set")

this is why this issue should be fixed in curl, and not in downstream projects like linux distributions.

I'm working on a separate patch that teaches libcurl about a general concept of blocked domains at runtime: https://git.hardenedbsd.org/shawn.webb/curl/-/compare/39c883560a3b9ac7ddbec78f7a6a8cc13930f219...shawn.webb%2Ffeatures%2Ftld_blocklist?from_project_id=127&straight=false

This approach would enable users to opt-in to the prohibition on .onion resolution rather than force the prohibition on everyone everywhere in every case with no option for opting out.

From my perspective as a developer of Whonix, re-compilation is best avoided. Adds a lot maintenance effort for something as simple as this.

Tor and Whonix

I wonder if/how Tails and Whonix handles this issue.

tails:

https://gitlab.tails.boum.org/tails/tails/-/merge_requests/1123 - Make curl use Tor
- ALL_PROXY="socks5://${random_username}:0@127.0.0.1:9050"
https://gitlab.tails.boum.org/tails/tails/-/issues/19488 - Make curl work out of the box
- echo 'proxy=socks5://$RANDOM:$RANDOM@127.0.0.1:9050' | tee -a ~/.curlrc
- random usernames are used for tor circuit isolation

whonix:

https://github.com/Kicksecure/usability-misc/blob/8f722bbbc7b7f2f3a35619a5a1cdeee337a68a78/usr/bin/dist-installer-cli#L2459C22-L2459C22
- curl --proxy socks5h://$proxy
instead of --socks5-hostname, use more modern --proxy + user:password@ip:port syntax for curl for better stream isolation https://phabricator.whonix.org/T126
- curl --proxy socks5h://user:password@$GATEWAY_IP:$SOCKS_PORT_TBB_DOWNLOAD
- https://github.com/Kicksecure/tb-updater/commit/d040c12085a527f4d39cb1751f2e514642c7ebaa
- https://github.com/Kicksecure/sdwdate/commit/5724d83b258a469b7a9a7bbc65153987f986369b

all these fixes require knowledge of a socks5 server address and port

all these fixes require knowledge of a socks5 server address and port

And all of these workarounds force a technology (socks5) that might not be desired in certain environments.

For applications developed under the Kicksecure or Whonix umbrella, specifically setting socks5 proxy settings including socks user name for stream isolation (IsolateSOCKSAuth) is alright.

However for "plain" invocations on the command line this doesn't work great. As for Whonix 17, invocation of for example curl in essence results in torsocks curl. (uwt)

torsocks is configured with:

IsolatePID 1 (stream isolation)
AllowOutboundLocalhost 1 (allow connecting to local services such as a locally running a web server for testing purposes)
AllowInbound 1 (for applications that open server ports).

All of this breaks if torsocks can no longer be used. Configuration of a socks5 proxy also isn't supported by all applications. Hence a way for the environment to express "torified" (allow onions) is desirable.

mentioned in issue torsocks#40016 (closed)

I wonder if the RFC is just plain wrong in its recommendation on rejecting .onion resolution. Some applications, and even entire devices, may not be aware they're sitting behind Tor. I have a fully Tor-ified home network, complete with a wireless access point, that serves as a research network. All devices behind that network have their traffic automatically re-routed through Tor: https://git.hardenedbsd.org/shawn.webb/articles/-/blob/master/infosec/tor/2017-01-14_torified_home/article.md

I would prefer it if applications and libraries didn't treat .onion resolutions as anything special. Just do the normal thing.

solution 1

I'd prefer that part of RFC 7686 to be rescinded.

If that isn't happening the RFC could be amended. Meanwhile, an official statement by Tor Project would provide guidance to curl, etc.

solution 2

At curl it was suggested.

/etc/resolv.conf:

option allow-domains *.onion

But I am not sure how realistic that is.

solution 3:

How about applications and libraries could check for the existence of a file in the filesystem such as /etc/torified? (And their equivalents for non-Linux based operating systems.)

solution 4:

An environment variable, while not available for some operating systems, would still be useful in supplement of solution 3. This is because then applications such as torsocks could set it because then at least torsocks could be unbroken. The current state of even torsocks breaking is really bad.

An analysis

Meanwhile, an official statement by Tor Project would provide guidance to curl, etc.

Recently, Matt Jolly asked for advice in the tor-dev mailing list.

I have answered it as best as I could. Right now I can't say it counts as an official statement, but could help anyway in the discussion people are having with curl maintainers.

added Onion Services label

Prompted by recent discussions I had over DNS in Tor, here's my thoughts on a solution.

I think the concern here is that if clients attempt to resolve .onion address by default this leaks what onion sites someone is accessing to their DNS server - less than ideal.

Discovering information about a DNS resolver's environment has been of recent interest in the IETF, mostly pushed by the move to encrypt all the things. Two RFCs come to mind that would help us here:

All of these use specially crafted and locally served DNS records to signal something to the client about its current environment. RFC 9606 is almost exactly what we want.

Building on these I propose that we standardize a RESINFO key to signal that a resolver, or its environment, can handle .onion domains, and that a client application should leave .onion resolution to its environment. We could even standardize keys to signal to a client where to find the Tor daemon SOCKS port in its environment, etc.

Formalize toggle override for non-Tor applications that follow RFC 7686

Child items 0

Activity

Re-frame and discuss

Tor and Whonix

An analysis