Failures when rewriting request URLs generated by other extensions

When I install a userscript that depends on a javascript library whose url HTTPS Everywhere rewrites, Scriptish can't download the dependency. With the Scriptish master branch and the development version of HTTPS Everywhere, I get:

Error loading dependency: http://ajax.googleapis.com/ajax/libs/jquery/1.3/jquery.min.js
Error! Server returned: nothing (timed out)

Greasemonkey 0.9.3 also fails, but silently.

First reported here: https://github.com/erikvold/scriptish/issues/282 The download code is over there: https://github.com/erikvold/scriptish/blob/master/extension/modules/script/scriptdownloader.js

Trac:
Username: Tobu

added component::https everywhere/eff-https everywhere owner::pde priority::very high reporter::Tobu resolution::fixed status::closed type::defect labels

Trac:
Username: Tobu
Summary: Breaks download of userscript dependencies by Scriptish to Breaks download of userscript dependencies by Greasemonkey and Scriptish

Anyone up for wading into this bug? I would start with the JavaScript debugger, as well as setting LogLevel to 1 or 2 in about:config and trying to work out where things are going wrong...

Trac:
Status: new to accepted

This basically blocks the browser at least on Mac Snow Leopard/Scriptish Latest and the browser needs to be force killed. The priority for this should be raised.

Trac:
Username: effuser

See https://github.com/scriptish/scriptish/issues/441 for more details on Scriptish end as the original 'First reported here' page has disappeared.

Trac:
Username: effuser

Even the more recent Scriptish bug report link is broken for me now.

The problem here is that we have to replace the channel that Scriptish / Greasemonkey is working with, but S/G doesn't realise the channel has been replaced out from underneath it.

Until Firefox has a sane, consistent request rewriting policy, I'm not sure whether there is an elegant solution to this problem. There is, however, an inelegant solution that S/G could implement, which was built by jsamuel for RequestPolicy:

https://www.requestpolicy.com/dev/ticket/115

Trac:
Keywords: N/A deleted, wont-fix-here added

s/The problem here/I believe the problem here/

(this is still just a hypothesis)

Trac:
Keywords: wont-fix-here deleted, wont-fix-here workaround-available added
Summary: Breaks download of userscript dependencies by Greasemonkey and Scriptish to Failures when rewriting request URLs generated by other extensions
Priority: normal to major

Several likely duplicates of this bug:

https://trac.torproject.org/projects/tor/ticket/1574 https://trac.torproject.org/projects/tor/ticket/2295 https://trac.torproject.org/projects/tor/ticket/2281

Trac:
Priority: major to critical

A note for any other Extension developers that may be reading here:

The "https-everywhere-uri-rewrite" event that we currently send lists the (before,after) URLs from the rewrite. We could probably replace that with an event that sends the post-rewrite nsIHTTPChannel, which is much more elegant if you were trying to obtain data from the channel :).

If anyone lets us know that they would like such code from us, we can include it and push it to our userbase fairly quickly.

I think we have a fix in the works for this bug. Can someone give me some examples of UserScripts for Scriptish/Greasemonkey that try to fetch jquery from Google over http, so I can test it?

I have had issues with XMLHttpRequests from Greasemonkey with HTTPS Everywhere. My script https://www.userscripts.org/scripts/review/100840 did not work until I put the "https" into the gdata API URL. Feel free to test with that.

Additionally, I had issues with the NewsFox RSS reader extension (not fetching new articles anymore on some feeds) with HTTPS Everywhere enabled.

Replying to pde:

I think we have a fix in the works for this bug. Can someone give me some examples of UserScripts for Scriptish/Greasemonkey that try to fetch jquery from Google over http, so I can test it?

pde: The fix the one Giorgio mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=677643#c75 right? Ie, we basically expect this bug to get knocked out by #4175 (moved), pending a new NoScript release, yes?

Replying to mikeperry:

pde: The fix the one Giorgio mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=677643#c75 right? Ie, we basically expect this bug to get knocked out by #4175 (moved), pending a new NoScript release, yes?

Yes, and I have that committed in a private repo but I just wanted to confirm that it actually fixes the bug before pushing it.

Thanks for the test case cypherpunks. Sadly, what it's telling me is that Giorgio's patch doesn't fix this bug (or possibly, doesn't fix all instances of this bug) :( :( :(

I just tried to reproduce this bug with either LastPass or StumbleUpon (to see if Giorgio's patch might fix either of those cases), but I just can't reproduce. Perhaps the bug reports caused these extensions to switch over to HTTPS themselves?

A version of HTTPS Everywhere with Giorgio's patch is in the extension-compatibility branch of my remote, git://gitweb.torproject.org/pde/https-everywhere.git

(https://gitweb.torproject.org/pde/https-everywhere.git/shortlog/refs/heads/extension-compatibility )

I can also get a clean reproduction with NewsFox, subscribing to an RSS feed like http://www.economist.com/feeds/print-sections/68/letters.xml with the Economist ruleset enabled.

Giorgio's IOUtils patch isn't helping in that case either :(

Replying to pde:

A version of HTTPS Everywhere with Giorgio's patch is in the extension-compatibility branch of my remote, git://gitweb.torproject.org/pde/https-everywhere.git

(https://gitweb.torproject.org/pde/https-everywhere.git/shortlog/refs/heads/extension-compatibility )

Hrmm.. We probably want to just merge with NoScript 2.1.5.. 2.1.5 involves a file rename, and some other refactoring as well, so it might be nice to try to keep it as simple as possible to deal with. I believe it also includes this fix.

This commit is an workaround for much of the breakage experienced by Greasemonkey and Scriptish users.  However it leaves users installing these scripts over HTTP and I wonder whether we shouldn't just disable the ruleset altogether until we can get a more secure workaround into those extensions :(.

We hope that a lot of the extension incompatibility bugs will be fixed if/when Mozilla accepts our patches for this ticket: https://bugzilla.mozilla.org/show_bug.cgi?id=765934

At least, extensions that use XMLHTTPRequest or nsIHTTPChannel.asyncOpen() should be fixed that way... nsIHTTPChannel.open() will require more work.

The next TBB 2.3.x-alpha should ship with that Firefox patch already applied, and it is on the HTTPS-Everywhere 3.0 dev channel as well. Might be a useful early testbed...

What is hopefully a fix for this bug has landed in mozilla-central and shipped in today's Firefox nightly build. We now need to test whether each of the sub-instances of this bug has been resolved from today's build of FF 21 onwards.

Trac:
Keywords: wont-fix-here workaround-available deleted, N/A added

I just did a test using 32-bit Firefox Nightly 2013-02-01 built from http://hg.mozilla.org/mozilla-central/rev/50cf5bbcb180, with NewsFox 1.0.8.4.1.1 and HTTPS Everywhere 4.0development.5.

I subscribed to the insecure version of the Google News feed at http://news.google.com/news?pz=1&cf=all&ned=us&hl=en&topic=h&num=3&output=rss and refreshed the feed. The feed was successfully populated with articles, and Wireshark showed that the connection was being made over HTTPS, not HTTP. (I also checked that this URL would have been rewritten by HTTPS Everywhere, and that the rewritten and non-rewritten versions both contain RSS content.)

Looks good!

We got the fix into Firefox 20 (currently Aurora) but it seems like we won't get to Firefox 19 or earlier, including the ESR release.

I put my test case for this back online: http://reworld.org/old/bugs/5477-tst.html

I agree that we're good now. Time to close all of these bugs!

With this bug closing and FF 20 released, I think we could consider experimenting with reenabling rulesets for Scriptish, Greasemonkey, MyWOT, StumbleUpon and soforth. At least in the development branch...

Ding dong.

Trac:
Status: accepted to closed
Resolution: N/A to fixed

closed

mentioned in issue #4122 (moved)

mentioned in issue #4278 (moved)

mentioned in issue #4408 (moved)

mentioned in issue #5350 (moved)

mentioned in issue #5754 (moved)

mentioned in issue #5912 (moved)

mentioned in issue #5942 (moved)

mentioned in issue #6199 (moved)

mentioned in issue #6499 (moved)

mentioned in issue #6592 (moved)

mentioned in issue #6688 (moved)

mentioned in issue #6688 (moved)

mentioned in issue #7099 (moved)

mentioned in issue #7423 (moved)

mentioned in issue #7769 (moved)

moved to tpo/applications/https-everywhere-eff#3190 (closed)

mentioned in issue tpo/applications/https-everywhere-eff#6199 (closed)