PT spec changes for better interoperability with other projects

I spoke with the i2p guys today and here are some of their suggestions for the PT spec. These would make it easier for them (and future other projects) to use Tor's PTs.

Major improvements:

• better spec documentation
  • less Tor jargon, split Tor-specific information into separate sections (e.g. Tor env vars)
  • some guidelines for non-Tor programs to use PTs
• better handling of per-endpoint config params such as pubkeys, instead of current hack via SOCKS auth params

Smaller enhancements, "good to have":

• possibility of letting a single process to act as both a client (outgoing) and server (incoming).
• flashproxy must allow client-specific remote endpoints (already as #10196 (closed))
• don't trust the entire localhost machine to make outgoing connections, e.g. if one users wants to run his own instance. two options here:
  • SSL connection with user/pass to the SOCKS transport client
  • use unix domain sockets. This also frees up ports, which is extra-useful in PT composition. Doesn't work on windows, though.

changed milestone to %Tor: unspecified

added SponsorS component::circumvention/pluggable transport milestone::Tor: unspecified owner::yawning priority::medium pt-ng pt-spec pt-v2 severity::normal status::new tor-pt type::enhancement labels

Trac:
Owner: N/A to infinity0
Status: new to assigned

This is pretty excellent; it will be good to drill down into these requirements more.

In my mind the biggest issue to consider here is backward compatibility: we want all the old PTs to keep working, and I2P probably wants to work with most (all?) of the old PTs, so we should make sure that any design decisions we make allow that.

With that in mind...

Replying to infinity0:

• better spec documentation

• less Tor jargon, split Tor-specific information into separate sections (e.g. Tor env vars)

Good idea. In particular, we should describe which of the TOR_* variables are actually Tor-specific, and which are just using our namespace.

• some guidelines for non-Tor programs to use PTs

Right.

• possibility of letting a single process to act as both a client (outgoing) and server (incoming).

That shouldn't be so hard.

[...]

• better handling of per-endpoint config params, such as pubkeys (instead of current hack via SOCKS auth params)

This is one of the cases where we need to think about compatibility. I2P would need to support the SOCKS hack already if it wants to work with any existing PTs that use it, and PTs will need to support the SOCKS hack in the future if they want to work with existing versions of Tor.

It's okay to add a new message-passing framework, or to clean up the existing protocols into something cleaner, but as we do so we need to let old PTs continue to work, and we should make sure that we're actually creating less work for implementors of Tor, I2P, and PTs in the long run, rather than more.

• SSL connection with user/pass to SOCKS client; currently we assume cleartext

This is possible as an extension, though IMO you should never run this on a remote system, so cleartext is fine.

• (my own suggestion) allow unix domain sockets

Doable.

I'm moving this into the "Tor" milestone since we manage specs that describe what Tor does (and in the Tor spec) in terms of the Tor component.

Trac:
Component: Pluggable transport to Tor

Restructured the description to make it clear which ones are important vs optional.

Trac:
Description: I spoke with the i2p guys today and here are some of their suggestions for the PT spec. These would make it easier for them (and future other projects) to use Tor's PTs.

• better spec documentation
  • less Tor jargon, split Tor-specific information into separate sections (e.g. Tor env vars)
  • some guidelines for non-Tor programs to use PTs
• possibility of letting a single process to act as both a client (outgoing) and server (incoming).
• flashproxy must allow client-specific remote endpoints (already as #10196 (closed))
• better handling of per-endpoint config params, such as pubkeys (instead of current hack via SOCKS auth params)
• SSL connection with user/pass to SOCKS client; currently we assume cleartext
  • (my own suggestion) allow unix domain sockets

related: #7153 (moved)

to

I spoke with the i2p guys today and here are some of their suggestions for the PT spec. These would make it easier for them (and future other projects) to use Tor's PTs.

Major improvements:

• better spec documentation
  • less Tor jargon, split Tor-specific information into separate sections (e.g. Tor env vars)
  • some guidelines for non-Tor programs to use PTs
• better handling of per-endpoint config params such as pubkeys, instead of current hack via SOCKS auth params

Smaller enhancements, "good to have":

• possibility of letting a single process to act as both a client (outgoing) and server (incoming).
• flashproxy must allow client-specific remote endpoints (already as #10196 (closed))
• don't trust the entire localhost machine, e.g. if one users wants to run his own instance. two options here:
  • SSL connection with user/pass to SOCKS client
  • use unix domain sockets. This also frees up ports, which is extra-useful in PT composition.

Trac:
Description: I spoke with the i2p guys today and here are some of their suggestions for the PT spec. These would make it easier for them (and future other projects) to use Tor's PTs.

Major improvements:

• better spec documentation
  • less Tor jargon, split Tor-specific information into separate sections (e.g. Tor env vars)
  • some guidelines for non-Tor programs to use PTs
• better handling of per-endpoint config params such as pubkeys, instead of current hack via SOCKS auth params

Smaller enhancements, "good to have":

• possibility of letting a single process to act as both a client (outgoing) and server (incoming).
• flashproxy must allow client-specific remote endpoints (already as #10196 (closed))
• don't trust the entire localhost machine, e.g. if one users wants to run his own instance. two options here:
  • SSL connection with user/pass to SOCKS client
  • use unix domain sockets. This also frees up ports, which is extra-useful in PT composition.

to

I spoke with the i2p guys today and here are some of their suggestions for the PT spec. These would make it easier for them (and future other projects) to use Tor's PTs.

Major improvements:

• better spec documentation
  • less Tor jargon, split Tor-specific information into separate sections (e.g. Tor env vars)
  • some guidelines for non-Tor programs to use PTs
• better handling of per-endpoint config params such as pubkeys, instead of current hack via SOCKS auth params

Smaller enhancements, "good to have":

• possibility of letting a single process to act as both a client (outgoing) and server (incoming).
• flashproxy must allow client-specific remote endpoints (already as #10196 (closed))
• don't trust the entire localhost machine to make outgoing connections, e.g. if one users wants to run his own instance. two options here:
  • SSL connection with user/pass to the SOCKS transport client
  • use unix domain sockets. This also frees up ports, which is extra-useful in PT composition. Doesn't work on windows, though.

I clarified the SSL SOCKs connection with the i2p devs. The intention isn't to use PT clients remotely, but rather to avoid giving access to other non-privileged users on the localhost machine.

So, for example, plaintext authentication to the PT client would work as well, assuming that non-priv users can't sniff traffic on localhost interfaces. (Not sure if this clashes with the current use of SOCKS auth for other things though.)

And to re-state, this is a minor point and is not intended to block them integrating or working with PTs. But it's something to keep in mind during this design work.

Replying to infinity0:

I clarified the SSL SOCKs connection with the i2p devs. The intention isn't to use PT clients remotely, but rather to avoid giving access to other non-privileged users on the localhost machine.

So, for example, plaintext authentication to the PT client would work as well, assuming that non-priv users can't sniff traffic on localhost interfaces. (Not sure if this clashes with the current use of SOCKS auth for other things though.)

That's the case for every common OS out there (If someone makes tcpdump suid root, they get what they deserve), including Windows. I assume this is more of a defense in depth thing, since the traffic they pass over a PT should be encrypted anyway....

This might be diverting from "help other projects adopt PTs" to "how to improve the PT design", but here are some more things that should be improved:

• Logging support. It's not clear how obfsproxy/etc. opreators are supposed to do logging atm, and it's not clear where the log file should be written. Solutions here include passing the log messages to Tor somehow, or forcing PTs to implement syslog support (or log in pt_state), or something like that.

• It might also be worth standardizing the PT CLI interface. So for example, all PTs should use --log-file to specify a logfile and --log-min-severity to specify the logging severity (those were example names: it's what obfsproxy uses and they are awful). This might help in adding some sort of torrc switch which enables debug logging for all pluggable transports, for example.

Grouping in #10047 (moved).

Replying to asn:

• Logging support. It's not clear how obfsproxy/etc. opreators are supposed to do logging atm, and it's not clear where the log file should be written. Solutions here include passing the log messages to Tor somehow, or forcing PTs to implement syslog support (or log in pt_state), or something like that.

Apart from the question of where to write log files, it would be very nice if transports could tell tor to log something (so that it will end up in the log that Tor Launcher copies to the clipboard, for example).

tor keeps the stdout of each plugin process open; how about we add commands like

LOG DEBUG Received SOCKS connection.
LOG WARN Unable to find any foobars, this transport probably won't work.

tor could then copy those lines into its own log, prefixed by the transport name.

BTW, if anybody's going to be at the hack day in Iceland next week, it would be cool to have a ~40 minute meeting to go through all this stuff. Can somebody agree to lead it, if so?

Replying to nickm:

BTW, if anybody's going to be at the hack day in Iceland next week, it would be cool to have a ~40 minute meeting to go through all this stuff. Can somebody agree to lead it, if so?

Yeah, I can do that.

Trac:
Milestone: N/A to Tor: 0.2.???

Trac:
Parent: N/A to #16755 (moved)

Trac:
Component: Tor to Pluggable transport
Keywords: N/A deleted, SponsorS pt-spec added
Owner: infinity0 to yawning

Trac:
Username: thomasfr
Cc: N/A to thomasf@jossystem.se

Milestone renamed

Trac:
Milestone: Tor: 0.2.??? to Tor: 0.3.???

Finally admitting that 0.3.??? was a euphemism for Tor: unspecified all along.

Trac:
Keywords: SponsorS pt-spec deleted, tor-03-unspecified-201612, SponsorS, pt-spec added
Milestone: Tor: 0.3.??? to Tor: unspecified

Remove an old triaging keyword.

Trac:
Keywords: tor-03-unspecified-201612 deleted, N/A added

Trac:
Sponsor: N/A to N/A
Severity: N/A to Normal
Reviewer: N/A to N/A
Keywords: N/A deleted, pt-v2, tor-pt, pt-ng added

Trac:
Parent: #16755 (moved) to N/A
Status: assigned to new

Trac:
Cc: thomasf@jossystem.se to thomasf@jossystem.se, dmr

mentioned in issue #10671 (moved)

mentioned in issue #11211 (moved)

mentioned in issue #12434 (moved)

mentioned in issue #16755 (moved)

mentioned in issue tpo/core/tor#12434 (closed)

moved to tpo/anti-censorship/pluggable-transports/trac#10629 (closed)