Relocating the Tor Browser directory is broken with Tor Browser 8

added TorBrowserTeam201910R actualpoints::3 component::applications/tor browser owner::tbb-team points::0.5 priority::high resolution::fixed severity::major status::closed tbb-8.0-issues tbb-8.0.1-can tbb-regression type::defect labels

Marking for 8.0.1 can.

Trac:
Keywords: N/A deleted, tbb-8.0.1-can added

Kathy and I cannot reproduce the crash, but any action that causes the browser profile to be renamed or moved breaks things. The specific problem which we observed on both macOS and Linux is that Torbutton and Tor Launcher are not loaded after a rename or move.

A workaround is to delete the extensions.json file, which contains full paths. Probably that file is supposed to be recreated automatically when the profile is moved. Webextensions do not seem to suffer the same fate even though extensions.json contains their paths as well (which are outdated after the profile move).

#28199 (moved) is a duplicate.

FWIW, this happens on Windows as well it seems, see: https://blog.torproject.org/comment/279768#comment-279768

Trac:
Summary: Relocating the Tor Browser directory is broken with Tor Browser 8 on Linux to Relocating the Tor Browser directory is broken with Tor Browser 8

Closed #27574 (moved) and #29711 (moved) as duplicates.

Trac:
Cc: N/A to user6969

win x64, tb 8.0.8 and still... the same...

Trac:
Cc: user6969 to user6969, catalyst

why isn't this fixed already? my tor browser is up to date and if i move the folder somewhere else than the location where i first installed it and try to launch the program i can't get it to work. i thought tor was supposed to be portable

Trac:
Username: ayylmao

Replying to ayylmao:

why isn't this fixed already? my tor browser is up to date and if i move the folder somewhere else than the location where i first installed it and try to launch the program i can't get it to work. i thought tor was supposed to be portable

It isn't fixed yet because nobody has worked on it so far. But it could be you to solve this, helping us out! It would be really appreciated.

A more permanent workaround for me was to delete extensions.json, create an empty 0 KB file in its place, and mark it as read-only. That forced the browser to use a temporary extensions.json for each session, and kept me from having to delete the file every time the directory changed.

Trac:
Username: teconmoon

Replying to teconmoon:

A more permanent workaround for me was to delete extensions.json, create an empty 0 KB file in its place, and mark it as read-only. That forced the browser to use a temporary extensions.json for each session, and kept me from having to delete the file every time the directory changed.

Wow, that's painful. I guess what we could do is patching XPIProviderUtils.js's parseDB() and add a sanity check making sure the path of the extensions in the DB exists. If not we need to recreate the DB. I suspect that would be something Mozilla would take if we upstreamed it.

We could as well just delete the json file during browser shutdown. However, that seems to be a bit too excessive.

Trac:
Keywords: N/A deleted, TorBrowserTeam201906 added

FWIW, John Haller from PortableApps applied a fix (see https://portableapps.com/node/60299 , and comment 9), but I'm not entirely sure what he did, but also had an additional issue (see https://portableapps.com/node/60496 ) with addonStartup.json.lz4 (edit: which was added in FF67).

Let me know if you would like me to reach out to John Halter and ask what he did, but I think you know what needs to be done already, and getting this done upstream is the best solution IMO.

Update: https://portableapps.com/comment/240579#comment-240579 -> John's reply

I hadn't seen John's reply - I also reached out to him on Twitter:

https://twitter.com/JohnTHaller/status/1142630151873859586 https://twitter.com/JohnTHaller/status/1142630329028612096

usable upstream bug(s)?

Moving tickets to July

Trac:
Keywords: TorBrowserTeam201906 deleted, TorBrowserTeam201907 added

#31894 (moved) is a duplicate.

Trac:
Cc: user6969, catalyst to user6969, catalyst, salah

So, this was broken in 8.x in a way that makes the whole experience failing hard which is kind of good. I wonder how the breakage will look like in 9.0 given that Tor Launcher and Torbutton are now no separate extensions anymore. The fear here is that the user retains now a somewhat usable Tor Browser but broken in subtle and dangerous ways. We should double-check that and fix it if necessary.

Trac:
Points: N/A to 0.5
Keywords: TorBrowserTeam201907 deleted, GeorgKoppen201910, TorBrowserTeam201910 added

mcs and brade are picking that one up.

Trac:
Keywords: GeorgKoppen201910 deleted, N/A added
Cc: user6969, catalyst, salah to user6969, catalyst, salah, mcs, brade

This problem still exists with Tor Browser 9.0a8. After renaming the profile directory, HTTPS-E and NoScript are both broken. On macOS we had to remove the following three files in order to make things work again (I assume this is also true for Windows and Linux, although the file paths are different):

TorBrowser-Data/Browser/XXX.default/extensions.json
TorBrowser-Data/Browser/Caches/XXX.default/startupCache/webext.sc.lz4
TorBrowser-Data/Browser/XXX.default/addonStartup.json.lz4

Looking at the code, each is handled in a different place.

For extensions.json, the solution described here should work: https://bugzilla.mozilla.org/show_bug.cgi?id=1429838#c11

For webext.sc.lz4, we could use a similar solution: we could modify _readData() inside components/extensions/ExtensionParent.jsm to check extension paths and ignore the cached data if one or more files pointed to by the paths no longer exist.

For addonStartup.json.lz4, we could again use a similar solution: we could modify loadExtensionState() inside toolkit/mozapps/extensions/internal/XPIProvider.jsm to check extension paths and ignore the cached data if paths have changed.

All of the above said, it is too risky to fix this in Tor Browser 9.0. Maybe something for 9.5alpha?

Replying to mcs:

This problem still exists with Tor Browser 9.0a8. After renaming the profile directory, HTTPS-E and NoScript are both broken. On macOS we had to remove the following three files in order to make things work again

What does that mean? Is the browser usable now but not as expected because NoScript and HTTPS-E are missing?

Replying to gk:

Replying to mcs:

This problem still exists with Tor Browser 9.0a8. After renaming the profile directory, HTTPS-E and NoScript are both broken. On macOS we had to remove the following three files in order to make things work again

What does that mean? Is the browser usable now but not as expected because NoScript and HTTPS-E are missing?

Sorry for not providing more detail. The browser is usable but HTTPS-E and NoScript are not working correctly. For example, JS is executed on regular (non-SSL) pages even when the security level is set to Safer. I am 98% sure that the extension code is not loaded at all because this is visible on the browser console after startup:

Error while loading 'jar:file:///Users/XXX/Desktop/Test1/TorBrowser-Data/Browser/l98sbeb6.default/extensions/https-everywhere-eff@eff.org.xpi!/manifest.json' (NS_ERROR_FILE_NOT_FOUND) Extension.jsm:513

(/Users/XXX/Desktop/Test1 is the original folder name, prior to renaming).

The Tor Launcher and the Torbutton functionality (such as circuit display) are not broken in any obvious way.

Other things: after the rename, about:addons shows both HTTPS-E and NoScript but the extension icons show up as broken images. The same broken image are shown on about:debugging, and debugging does not work (no surprise). And if I add one or both of the HTTPS-E/NoScript icons to the browser toolbar before renaming the directory that contains TorBrowser-Data, the toolbar icons are gone after the rename.

Replying to mcs:

Replying to gk:

Replying to mcs:

This problem still exists with Tor Browser 9.0a8. After renaming the profile directory, HTTPS-E and NoScript are both broken. On macOS we had to remove the following three files in order to make things work again

What does that mean? Is the browser usable now but not as expected because NoScript and HTTPS-E are missing?

Sorry for not providing more detail. The browser is usable but HTTPS-E and NoScript are not working correctly. For example, JS is executed on regular (non-SSL) pages even when the security level is set to Safer. I am 98% sure that the extension code is not loaded at all because this is visible on the browser console after startup: {{{ Error while loading 'jar:file:///Users/XXX/Desktop/Test1/TorBrowser-Data/Browser/l98sbeb6.default/extensions/https-everywhere-eff@eff.org.xpi!/manifest.json' (NS_ERROR_FILE_NOT_FOUND) Extension.jsm:513 }}}

(/Users/XXX/Desktop/Test1 is the original folder name, prior to renaming).

The Tor Launcher and the Torbutton functionality (such as circuit display) are not broken in any obvious way.

Other things: after the rename, about:addons shows both HTTPS-E and NoScript but the extension icons show up as broken images. The same broken image are shown on about:debugging, and debugging does not work (no surprise). And if I add one or both of the HTTPS-E/NoScript icons to the browser toolbar before renaming the directory that contains TorBrowser-Data, the toolbar icons are gone after the rename.

Okay, that's what I feared, thanks. Could you pick this up for our alpha series?

Trac:
Severity: Normal to Major
Priority: Medium to High

Added brade/mcs points so far (0.6).

Trac:
Actualpoints: N/A to 0.6

I think that another problem is the torrc file where there is:

DataDirectory C:\XXX\Tor Browser\Browser\TorBrowser\Data\Tor
GeoIPFile C:\XXX\Tor Browser\Browser\TorBrowser\Data\Tor\geoip
GeoIPv6File C:\XXX\Tor Browser\Browser\TorBrowser\Data\Tor\geoip6

When Relocating the Tor Browser directory, this are not updated.

Trac:
Username: morar

Replying to morar:

I think that another problem is...

I was going to post something about this. With the release of ESR68 based stable, and since I need to keep various releases around (FP tests, regressions etc), I decided to cleanup my folder naming convention for the ESR60 based stable -> e.g. to TB60, TB60-32bit. Once I did this, the browser opens but Tor doesn't. It doesn't even try to connect. Not even a failed message or dialog or anything.

I guess morar has nailed the root of the issue

Replying to Thorin:

I was going to post something about this. With the release of ESR68 based stable, and since I need to keep various releases around (FP tests, regressions etc), I decided to cleanup my folder naming convention for the ESR60 based stable -> e.g. to TB60, TB60-32bit. Once I did this, the browser opens but Tor doesn't. It doesn't even try to connect. Not even a failed message or dialog or anything.

I guess morar has nailed the root of the issue

Are you sure? Tor Launcher passes values for those torrc parameters in the arg list / command line when it starts tor, which means they should automatically be corrected the next time you open the browser. Did you delete extensions.json? In your ESR60-based browsers, Tor Launcher is possibly not being loaded at all.

I made some tests:

Relocating the Tor Browser 7.5.6, restarted and loaded Tor Browser 7.5.6, torrc not updated.
Relocating the Tor Browser 8.5.5 after delete extensions.json, restarted and loaded Tor Browser 8.5.5, torrc not updated.
Relocating the Tor Browser 9.0, restarted and loaded Tor Browser 9.0, torrc not updated.

In all three cases

DataDirectory C:\XXX\Tor Browser\Browser\TorBrowser\Data\Tor
GeoIPFile C:\XXX\Tor Browser\Browser\TorBrowser\Data\Tor\geoip
GeoIPv6File C:\XXX\Tor Browser\Browser\TorBrowser\Data\Tor\geoip6

are not updated with the new folder.

Trac:
Username: morar

Replying to morar:

I made some tests: ... In all three cases

{{{ DataDirectory C:\XXX\Tor Browser\Browser\TorBrowser\Data\Tor GeoIPFile C:\XXX\Tor Browser\Browser\TorBrowser\Data\Tor\geoip GeoIPv6File C:\XXX\Tor Browser\Browser\TorBrowser\Data\Tor\geoip6 }}}

are not updated with the new folder.

Thanks for testing. In those cases, did Tor Browser open and work correctly (after you applied workarounds including deleting the extensions.json file)? I realized yesterday that even though new values for config parameters such as DataDirectory are passed to tor via args (and therefore used), the torrc file will not be overwritten unless you make other config changes. In other words, tor doesn't write out an updated torrc in response to new command line args.

Trac:
Username: morar

With Tor Browser 7.5.6 and Tor Browser 8.5.5 i can surf the internet. But i think that some command don't work like ExitNodes {DE}.

Today, after replacing Tor Browser 9.0, i have a critical error. When Tor Browser starts, i only see a window with a gray background and error:

XML Parsing Error: undefined entity
Location: chrome://browser/content/browser.xul
Line Number 60, Column 1:

<window id="main-window"
^

See attachment.

If I put the folder back in the original place, the error disappears.

Trac:
Username: morar

#32302 (moved) is a duplicate

Replying to mcs:

Are you sure? ... Did you delete extensions.json?...

I didn't delete or do anything special, just renamed the dir. After the rename, I tried several times but it just continued to skip the tor connection part/dialog completely. I ended up just deleting those TB instances, and set up two (64/32bit) brand new legacy 8.5.5's with updates disabled. I also set up two version 9 stables, and I have my two alphas.

FWIW, on all six, I cannot replicate: make a copy, rename dir, start, and I always get success (i.e tor connection dialog etc, browsing works). Maybe I'm weird, because the extensions (NS, HTTPS-E) also load and work the first time.

So IDK what went on, and I no longer have the originals which were initially clean v8.0 setups with updates. Chalk it up to experience :)

Patch for review: https://github.com/acatarineu/tor-browser/commit/27604

I think there are (at least) three different issues here:

One is the problem of extensions being broken that mcs mentioned, which I think was introduced by https://bugzilla.mozilla.org/show_bug.cgi?id=1512436. More concretely, the problem seems to be the new rootURI field in addonStartup.json which is absolute, unlike the path one which is serialized as relative: https://searchfox.org/mozilla-esr68/rev/22bae08f58d48ff86e02d5bbd12e5630af148d6f/toolkit/mozapps/extensions/internal/XPIProvider.jsm#555. I think this was needed to have built-in addons with re[/](/) rootURIlikere[/gre/modules/themes/default/.](/gre/modules/themes/default/.) Fixed this by always overriding it here if this.file exists (should not happen with addons with re[/](/`) rootURIs).

The other is some special treatment that langpacks get due to https://bugzilla.mozilla.org/show_bug.cgi?id=1492459. These are flagged as missing early here (currentModifiedTime is set here only for langpacks) if the (old) path of the extension xpi does not exist. If that's the case the langpack is removed in https://searchfox.org/mozilla-esr68/rev/715f10032bb8be971ff0e9846e12be58afad4950/toolkit/mozapps/extensions/internal/XPIDatabase.jsm#3143. That seems to fallback to English, but on browser restart it's completely broken, showing the same error message mentioned in comment:29 and in bugzilla. I'm not so sure what's the best way to fix this, in the proposed patch I'm checking for the (new) extension file existing before flagging it as missing.

Third is the issue of extensions.json (and possibly webext.sc.lz4) paths not being updated, but I'm not so sure this has functionality impact if the two previous issues are fixed. In any case, I think making scanForChanges return true when the path of some addon location has changed will do the trick here and trigger an update of extensions.json. I verified that webext.sc.lz4 paths are also updated, although I did not investigate what exactly in the code is updating the latter.

With the patch I cannot reproduce the issues anymore, verified on Linux and Windows. I also verified that a profile which is in the bad state of comment:29 is fixed.

I think it's worth filing bugzillas or updating the existing ones and try to get this fixed in Firefox (and also get some feedback on the suggested patches).

Trac:
Keywords: TorBrowserTeam201910 deleted, TorBrowserTeam201910R added
Status: new to needs_review

Replying to Thorin:

Replying to mcs:

Are you sure? ... Did you delete extensions.json?...

I didn't delete or do anything special, just renamed the dir. After the rename, I tried several times but it just continued to skip the tor connection part/dialog completely. I ended up just deleting those TB instances, and set up two (64/32bit) brand new legacy 8.5.5's with updates disabled. I also set up two version 9 stables, and I have my two alphas.

FWIW, on all six, I cannot replicate: make a copy, rename dir, start, and I always get success (i.e tor connection dialog etc, browsing works). Maybe I'm weird, because the extensions (NS, HTTPS-E) also load and work the first time.

So IDK what went on, and I no longer have the originals which were initially clean v8.0 setups with updates. Chalk it up to experience :)

These are the steps I followed to reproduce in Linux.

Extract tor-browser 9 to a new folder, so that tor-browser_{locale} is the only one there.
Run tor-browser.
Close it.
Rename tor-browser_{locale} to something else.
Run again.
Check that the extension icons in Customize... are gone (and the other issues mcs mentioned in comment:21).

Can you confirm whether you still cannot reproduce with these steps?

Trac:
Actualpoints: 0.6 to 3

Replying to acat:

Rename tor-browser_{locale} to something else.

Can you confirm whether you still cannot reproduce with these steps?

I wasn't thinking in my re-test, it helps to not leave the original dir lying around - on Windows with 9.0 (and I did not delete any files)

First run after dir rename

the extension icons are gone in customize and about:addons
console errors eg Error while loading 'jar:file:///D:/TBTest/TB1/Browser/TorBrowser/Data/Browser/profile.default/extensions/...
torcc file shows the original path TB1, not the new path TB2

Second (and subsequent) run after dir rename:

NoScript seems fine: no console errors, icons present, test with safest setting did not execute JS
HTTPS-E is not: same console error, missing icon in customize and about:addons
torcc file updated to new path TB2

see below the first line is still looking for TB1

Error while loading 'jar:file:///D:/TBTest/TB1/Browser/TorBrowser/Data/Browser/profile.default/extensions/https-everywhere-eff@eff.org.xpi!/manifest.json' (NS_ERROR_FILE_NOT_FOUND) Extension.jsm:513
    readJSON resource://gre/modules/Extension.jsm:513
    onStopRequest resource://gre/modules/NetUtil.jsm:128
    _openNetworkSettings jar:file:///D:/TBTest/TB2/Browser/browser/omni.ja!/chrome/torlauncher/components/tl-process.js:751
    _controlTor jar:file:///D:/TBTest/TB2/Browser/browser/omni.ja!/chrome/torlauncher/components/tl-process.js:586
    TorStartAndControlTor jar:file:///D:/TBTest/TB2/Browser/browser/omni.ja!/chrome/torlauncher/components/tl-process.js:333
    observe jar:file:///D:/TBTest/TB2/Browser/browser/omni.ja!/chrome/torlauncher/components/tl-process.js:135

Replying to acat:

Patch for review: https://github.com/acatarineu/tor-browser/commit/27604

I think there are (at least) three different issues here:

One is the problem of extensions being broken that mcs mentioned, which I think was introduced by https://bugzilla.mozilla.org/show_bug.cgi?id=1512436. More concretely, the problem seems to be the new rootURI field in addonStartup.json which is absolute, unlike the path one which is serialized as relative: https://searchfox.org/mozilla-esr68/rev/22bae08f58d48ff86e02d5bbd12e5630af148d6f/toolkit/mozapps/extensions/internal/XPIProvider.jsm#555. I think this was needed to have built-in addons with re[/](/) rootURIlikere[/gre/modules/themes/default/.](/gre/modules/themes/default/.) Fixed this by always overriding it here if this.file exists (should not happen with addons with re[/](/`) rootURIs).

The other is some special treatment that langpacks get due to https://bugzilla.mozilla.org/show_bug.cgi?id=1492459. These are flagged as missing early here (currentModifiedTime is set here only for langpacks) if the (old) path of the extension xpi does not exist. If that's the case the langpack is removed in https://searchfox.org/mozilla-esr68/rev/715f10032bb8be971ff0e9846e12be58afad4950/toolkit/mozapps/extensions/internal/XPIDatabase.jsm#3143. That seems to fallback to English, but on browser restart it's completely broken, showing the same error message mentioned in comment:29 and in bugzilla. I'm not so sure what's the best way to fix this, in the proposed patch I'm checking for the (new) extension file existing before flagging it as missing.

Third is the issue of extensions.json (and possibly webext.sc.lz4) paths not being updated, but I'm not so sure this has functionality impact if the two previous issues are fixed. In any case, I think making scanForChanges return true when the path of some addon location has changed will do the trick here and trigger an update of extensions.json. I verified that webext.sc.lz4 paths are also updated, although I did not investigate what exactly in the code is updating the latter.

With the patch I cannot reproduce the issues anymore, verified on Linux and Windows. I also verified that a profile which is in the bad state of comment:29 is fixed.

I think it's worth filing bugzillas or updating the existing ones and try to get this fixed in Firefox (and also get some feedback on the suggested patches).

Yes, please do update https://bugzilla.mozilla.org/show_bug.cgi?id=1429838 so we can get some feedback there as well.

That said this looks pretty good. One thing we should fix is getting

JavaScript error: re[/gre/modules/ExtensionCommon.jsm,](/gre/modules/ExtensionCommon.jsm,) line 2318: Error: listener not re-registered
JavaScript error: re[/gre/modules/ExtensionCommon.jsm,](/gre/modules/ExtensionCommon.jsm,) line 2318: Error: listener not re-registered
JavaScript error: re[/gre/modules/ExtensionCommon.jsm,](/gre/modules/ExtensionCommon.jsm,) line 2318: Error: listener not re-registered
JavaScript error: re[/gre/modules/ExtensionCommon.jsm,](/gre/modules/ExtensionCommon.jsm,) line 2318: Error: listener not re-registered
JavaScript error: re[/gre/modules/ExtensionCommon.jsm,](/gre/modules/ExtensionCommon.jsm,) line 2318: Error: listener not re-registered
JavaScript error: re[/gre/modules/ExtensionCommon.jsm,](/gre/modules/ExtensionCommon.jsm,) line 2318: Error: listener not re-registered
JavaScript error: re[/gre/modules/ExtensionCommon.jsm,](/gre/modules/ExtensionCommon.jsm,) line 2318: Error: listener not re-registered
JavaScript error: re[/gre/modules/ExtensionCommon.jsm,](/gre/modules/ExtensionCommon.jsm,) line 2318: Error: listener not re-registered

after applying your patches. (I blew the startup cache in the profile directory away before I started a Tor Browser modified with your patches) Re-locating does not matter I start to get those JS errors even if I keep the current location.

Trac:
Status: needs_review to needs_revision
Keywords: TorBrowserTeam201910R deleted, TorBrowserTeam201910 added

I can't reproduce... Which exact files did you remove to get these?

Replying to acat:

I can't reproduce... Which exact files did you remove to get these?

Hrm. It seems I can't reproduce that anymore with clean bundles, sorry for the noise. I wonder what I did do differently before, though... Anyway, nice work! I cherry-picked your patch onto tor-browser-68.2.0esr-9.5-1 (commit 7d5f879bba1e81ee64576e882f9d8916a2bc471e).

It will be interesting to see what Mozilla folks are saying. One thing we should not forget when upstreaming is including a test that makes sure this scenario is still working. It seems over time more and more functionality crept in that broke the relocation scenario.

We miiiight want to backport the changes at some point but not sure. This will probably be sysrqb's decision.

Trac:
Resolution: N/A to fixed
Keywords: TorBrowserTeam201910 deleted, tbb-backport, TorBrowserTeam201910R added
Status: needs_revision to closed

#32989 (moved) is a duplicate.

Trac:
Cc: user6969, catalyst, salah, mcs, brade to user6969, catalyst, salah, mcs, brade, pudim

No backport for those tickets/fixes. They will emerge with 9.5.

Trac:
Keywords: tbb-backport deleted, N/A added

closed

changed time estimate to 4h

added 24h of time spent

mentioned in issue #28199 (moved)

mentioned in issue #29711 (moved)

mentioned in issue #30718 (moved)

mentioned in issue #31894 (moved)

Relocating the Tor Browser directory is broken with Tor Browser 8

Child items 0

Activity