analyze thunderbird HTTP proxy behaviour
I think the HTTP proxy is only used for HTTP related things - the
question is - what happens when it has no HTTP proxy? Do HTTP
related things simply leak out?Activity
Hi,
[TorBirdy + TBTracer]
- user has no HTTP proxy set at all
With TorBirdy, I disabled the HTTP/SSL proxies. Thunderbird was not leaking anything -- everything was being routed through Tor.
Addons - worked. Updates - worked.
I could see mostly HTTPS connections, however, I do see some plain HTTP GET requests being made. Say this:
[HTTP] [Sunday, September 23, 2012 6:09:57 PM] GET /contribute/ HTTP/1.1 Host: www.mozilla.org Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip, deflate Connection: keep-alive [HTTP] [Sunday, September 23, 2012 6:09:59 PM] HTTP/1.1 302 Found Server: Apache X-Backend-Server: bedrock4.webapp.scl3.mozilla.com Content-Type: text/html; charset=iso-8859-1 Date: Sun, 23 Sep 2012 18:06:12 GMT
This happens (infrequently) when the addons page (and with specific addons) is being accessed. So any thoughts about this?
- user has a HTTP proxy set but there is no proxy running
(the earlier approach)
Addons - not working. Updates - not working.
I have pushed the new changes (no HTTP/SSL proxies).
Trac:
Cc: ioerror to ioerror, sukhbir.in@gmail.comOk, so I just tested with Enigmail.
HTTP proxy (fail closed): Key fetching/ searching doesn't work. No HTTP proxy: Enigmail leaks.
It seems from #2846 (closed) that we can't get gpg to use a SOCKS proxy, so there's that.
Otherwise, I guess we can try to integrate Moxie's crazy HTTP proxy hack: https://github.com/moxie0/Convergence/blob/master/client/components/LocalProxy.js
I've also mailed the gnupg-devel mailing list and it should show up in the archive soon: http://lists.gnupg.org/pipermail/gnupg-devel/2012-September/date.html
Replying to sukhbir:
Replying to ioerror:
I propose that we call gpg with usewithtor in Enigmail as a temporary work around for those with torsocks/usewithtor - I wonder how to make Enigmail do this?
We can't do this through Enigmail.
Are you sure? Enigmail creates the process - why can't it have a command prefix?
I understand that currently we can't do it - what if we extend Enigmail to be Tor aware?
Replying to ioerror:
Replying to sukhbir:
We can't do this through Enigmail.
Are you sure? Enigmail creates the process - why can't it have a command prefix?
I understand that currently we can't do it - what if we extend Enigmail to be Tor aware?
Yes, I meant currently :) If we were to extend Enigmail, sure, it is possible. But then it depends on which route do we want to take -- get this done through the gpg folks or through Enigmail or with Moxie's solution. Let's see...
Replying to sukhbir:
Replying to ioerror:
Replying to sukhbir:
We can't do this through Enigmail.
Are you sure? Enigmail creates the process - why can't it have a command prefix?
I understand that currently we can't do it - what if we extend Enigmail to be Tor aware?
Yes, I meant currently :) If we were to extend Enigmail, sure, it is possible. But then it depends on which route do we want to take -- get this done through the gpg folks or through Enigmail or with Moxie's solution. Let's see...
I think it would be easy to ask for a command line prefix as a feature - unrelated to anything else - it would even be a good idea.
Replying to ioerror:
At the moment, I think we should set the proxy in Enigmail and let it fail as it has always been failing.
Done.
Hopefully we can find a solution such as using Moxie's code later. This should allow us to unset the HTTP proxy for the rest of Thunderbird and things should be OK, more functional, even.
Yup.
Replying to sukhbir:
Replying to ioerror:
At the moment, I think we should set the proxy in Enigmail and let it fail as it has always been failing.
Done.
Hopefully we can find a solution such as using Moxie's code later. This should allow us to unset the HTTP proxy for the rest of Thunderbird and things should be OK, more functional, even.
Yup.
Moxie is a hero and thinks that the local JavaScript proxy hack is actually possible.
I've opened a ticket about the HTTP proxy in javascript: #6958 (closed)
Replying to ioerror:
I've opened a ticket about the HTTP proxy in javascript: #6958 (closed)
I agree that Moxie needs mad props for bending XPCOM to his will with that JS HTTP proxy thing. However, I think a direct SOCKS patch for GPG (#2846 (closed)) has less vulnerability surface and should be simpler code in total.
The reason I think that the direct SOCKS patch reduces the vulnerability surface is nuanced. It probably doesn't matter unless we actually disable jsctypes in our own builds of Thunderbird (#6152 (moved)).
So for now, my recommendation is:
- Disable GPG network activity by setting the proxy to garbage for now.
- Try Moxie's code from #6958 (closed). If you can get it to work out of the box within an hour, use it for now
- If Moxie's code takes more than an hour to get it to work, you should try to provide a patch for SOCKS support to GPG (#2846 (closed)). It's possible someone may have already written one already, somewhere...
Replying to mikeperry:
Replying to ioerror:
I've opened a ticket about the HTTP proxy in javascript: #6958 (closed)
I agree that Moxie needs mad props for bending XPCOM to his will with that JS HTTP proxy thing. However, I think a direct SOCKS patch for GPG (#2846 (closed)) has less vulnerability surface and should be simpler code in total.
We don't ship GPG as part of TorBirdy - we have to go with what we have and that is an older gpg, without a currently non-existent SOCKS patch or SOCKS support.
The reason I think that the direct SOCKS patch reduces the vulnerability surface is nuanced. It probably doesn't matter unless we actually disable jsctypes in our own builds of Thunderbird (#6152 (moved)).
So for now, my recommendation is:
- Disable GPG network activity by setting the proxy to garbage for now.
That will break Enigmail for everyone. We're setting it to what should be a Torified HTTP proxy and if it fails, it fails. We won't break it for everyone.
- Try Moxie's code from #6958 (closed). If you can get it to work out of the box within an hour, use it for now
That is the goal.
- If Moxie's code takes more than an hour to get it to work, you should try to provide a patch for SOCKS support to GPG (#2846 (closed)). It's possible someone may have already written one already, somewhere...
Such a patch is worthwhile but it will not help us until it is tested, merged, and then deployed to everyone. Thus, we'll still be waiting for a long long time. We should do both but first, we'll make a local HTTP proxy somehow.
Replying to ioerror:
I've also mailed the gnupg-devel mailing list and it should show up in the archive soon: http://lists.gnupg.org/pipermail/gnupg-devel/2012-September/date.html
http://lists.gnupg.org/pipermail/gnupg-devel/2012-September/026923.html
Looks promising unfortunately an OS specific thing.. Did anyone try it?
gpg --keyserver-options http-proxy=socks5://127.0.0.1:9050 --search jacob@appelbaum.net [...] gpgkeys: HTTP search error 7: Unable to receive initial SOCKS5 response. [...]Tor SocksPort is running on 9050. curl version is > 7.21.7.
Is it working for you? (I should probably subscribe to gnupg-devel)
The output with debug
gpg --keyserver-options http-proxy=socks5://127.0.0.1:9050,debug,verbose --search jacob@appelbaum.netwas not much more helpful.
I tried with a SOCKS port of 9050 (Tor):
gpg --keyserver-options http-proxy=socks5://127.0.0.1:9050,debug,verbose --search jacob@appelbaum.net gpg: searching for "jacob@appelbaum.net" from hkp server pool.sks-keyservers.net gpgkeys: curl version = GnuPG curl-shim gpgkeys: search type is 0, and key is "jacob@appelbaum.net" * HTTP proxy is "socks5://127.0.0.1:9050" * HTTP URL is "http://pool.sks-keyservers.net:11371/pks/lookup?op=index&options=mr&search=jacob%40appelbaum%2Enet" * HTTP auth is "null" * HTTP method is GET ?: invalid HTTP proxy (socks5://127.0.0.1:9050): unsupported URI gpgkeys: HTTP search error 7: couldn't connect: Success gpg: key "jacob@appelbaum.net" not found on keyserver gpg: keyserver internal error gpg: keyserver search failed: keyserver errorI'm not clear that it is supposed to work that way - I think rather we'd need to write a patch to add a SOCKS5 proxy option. Is that incorrect?
gpg2 on Ubuntu seems to support things a little better but it still doesn't work:
gpg2 --keyserver-options http-proxy=socks5://127.0.0.1:9050,debug,verbose --search jacob@appelbaum.net gpg: searching for "jacob@appelbaum.net" from hkp server pool.sks-keyservers.net gpgkeys: curl version = libcurl/7.21.3 GnuTLS/2.8.6 zlib/1.2.3.4 libidn/1.18 gpgkeys: search type is 0, and key is "jacob@appelbaum.net" * About to connect() to proxy 127.0.0.1 port 9050 (#0) * Trying 127.0.0.1... * connected * Connected to 127.0.0.1 (127.0.0.1) port 9050 (#0) > GET http://pool.sks-keyservers.net:11371/pks/lookup?op=index&options=mr&search=jacob%40appelbaum.net HTTP/1.1 Host: pool.sks-keyservers.net:11371 Accept: */* Proxy-Connection: Keep-Alive Pragma: no-cache Cache-Control: no-cache * HTTP 1.0, assume close after body < HTTP/1.0 501 Tor is not an HTTP Proxy < Content-Type: text/html; charset=iso-8859-1 < * Closing connection #0 gpg: key "jacob@appelbaum.net" not found on keyserverReplying to tagnaq:
No, there is no patch needed.
The following works fine for me: {{{ gpg --keyserver-options http-proxy=socks5://127.0.0.1:,debug,verbose --search jacob@appelbaum.net }}}
Make sure you are using a recent curl version as David pointed out.
There are a few things at play here - a major one is that gpg must be actually using curl (!) and when it does, it must be the right version. Sadly, curl appears to see the right proxy with gpg2 but then it doesn't do the right thing at all.
So if anything, I think TorBirdy should set:
--keyserver-options http-proxy=socks5-hostname://127.0.0.1:9050,debug,verboseOr
--keyserver-options http-proxy=socks5-hostname://127.0.0.1:9050,debug,verboseThe full command would look like this:
gpg --keyserver-options http-proxy=socks5-hostname://127.0.0.1:9050 --search jacob@appelbaum.netI think this ticket was done with Sukhbirs anwers (results). Now we are basically a duplicate of #2846 (closed) - lets continue there? </ side note>
Ok - well, I guess we have a partial solution here.
What can we about Windows user?
Replying to ioerror:
So if anything, I think TorBirdy should set: {{{ --keyserver-options http-proxy=socks5-hostname://127.0.0.1:9050,debug,verbose }}}
Or {{{ --keyserver-options http-proxy=socks5-hostname://127.0.0.1:9050,debug,verbose }}}
I don't see any difference in the first and second options? Do you think we need debug and verbose too? I suppose we shouldn't use them statically there.
I committed without ",debug,verbose" - sorry: https://github.com/ioerror/torbirdy/commit/cc6082247c4bdc6bb8ba8a846fb1c8ec4d90e522
I've put the details into #2846 (closed) - lets close this bug.
Trac:
Resolution: N/A to fixed
Status: reopened to closedmentioned in issue #6958 (closed)