Bug legacy/trac#10819 (closed): Add a pref, "privacy.thirdparty.isolate", to allow the activation or deactivation of isolating DOM storage and image caching by first party URI.
There are also a couple of regression test patches that I think may be broken:
Trac: Description: We should verify that if we revert any of our Tor Browser patches that have a corresponding regression test, then the test will fail.
Patches with regression tests:
We have regression tests for the following patches:
Bug legacy/trac#10819 (closed): Add a pref, "privacy.thirdparty.isolate", to allow the activation or deactivation of isolating DOM storage and image caching by first party URI.
There are also a couple of regression test patches that I think may be broken:
Bug legacy/trac#10819 (closed): Add a pref, "privacy.thirdparty.isolate", to allow the activation or deactivation of isolating DOM storage and image caching by first party URI.
There are also a couple of regression test patches that I think may be broken:
There are two more patches (currently under revision in legacy/trac#16429 (moved)) with regression tests for blob URI and mediasource URI cache isolation. I have confirmed that the regression tests fail when our isolation patches are reverted.
In order to check if our regression tests are failing without their corresponding patches, I used my repo with split branches and split them more to put the tests in a separate branch, to be able to push them separately to Mozilla Try.
I did that first using the version of the patches based on 38.0.1esr and here are the results:
38.0.1esr/2874-Block-Components.interfaces-from-content: * 75e3f67964df Regression tests for #2874: Block Components.interfaces from content * 6d4165108321 Bug #2874: Block Components.interfaces from content * d919ec4dedbe Enable tbb-tests/mochitest.ini * 52459e0c7dd3 Bug #12620: TorBrowser regression tests folder https://treeherder.mozilla.org/#/jobs?repo=try&revision=18cfee0eae1938.0.1esr/2874-Block-Components.interfaces-from-content.test: * 0801b13aea17 Regression tests for #2874: Block Components.interfaces from content * d919ec4dedbe Enable tbb-tests/mochitest.ini * 52459e0c7dd3 Bug #12620: TorBrowser regression tests folder https://treeherder.mozilla.org/#/jobs?repo=try&revision=3b8e8a64ccdd busted: X testfailed: M(5), M(bc1)38.0.1esr/2875-Limit-device-and-system-specific-CSS-Media-Queries: * 0721c0ce16d6 Regression tests for #2875: Limit device and system specific CSS Media Queries. * 09149b1417db Bug #2875: Limit device and system specific CSS Media Queries. * d919ec4dedbe Enable tbb-tests/mochitest.ini * 52459e0c7dd3 Bug #12620: TorBrowser regression tests folder https://treeherder.mozilla.org/#/jobs?repo=try&revision=fac4b042d797 testfailed: M(4), M(5), M(bc2)38.0.1esr/2875-Limit-device-and-system-specific-CSS-Media-Queries.test: * e7df7918a148 Regression tests for #2875: Limit device and system specific CSS Media Queries. * d919ec4dedbe Enable tbb-tests/mochitest.ini * 52459e0c7dd3 Bug #12620: TorBrowser regression tests folder https://treeherder.mozilla.org/#/jobs?repo=try&revision=ec37fdbad233 busted: X testfailed: M(5), M(bc1)38.0.1esr/2950-Make-Permissions-Manager-memory-only: * fd15a9ebb804 Regression tests for Bug #2950: Make Permissions Manager memory-only * 79ef2a1f2bde Bug #2950: Make Permissions Manager memory-only * 6d16271a374e Enable tbb-tests/browser.ini * 52459e0c7dd3 Bug #12620: TorBrowser regression tests folder https://treeherder.mozilla.org/#/jobs?repo=try&revision=0fcd667c057038.0.1esr/4755-Return-client-window-coordinates-for-mouse-event-screenX-Y: * bebd5ea65b12 Regression tests for #4755: Return client window coordinates for mouse event screenX/Y (for dragend, 0,0 is returned). * 931ba1ed3956 Bug #4755: Return client window coordinates for mouse event screenX/Y (for dragend, 0,0 is returned). * d919ec4dedbe Enable tbb-tests/mochitest.ini * 52459e0c7dd3 Bug #12620: TorBrowser regression tests folder https://treeherder.mozilla.org/#/jobs?repo=try&revision=b6663db2aa8e testfailed: M(2), M(5)38.0.1esr/4755-Return-client-window-coordinates-for-mouse-event-screenX-Y.test: * 8b70cce28921 Regression tests for #4755: Return client window coordinates for mouse event screenX/Y (for dragend, 0,0 is returned). * d919ec4dedbe Enable tbb-tests/mochitest.ini * 52459e0c7dd3 Bug #12620: TorBrowser regression tests folder https://treeherder.mozilla.org/#/jobs?repo=try&revision=c97b19954a9b busted: X testfailed: M(5), M(bc1)38.0.1esr/5856-Do-not-expose-physical-screen-info-via-window-window.screen: * a555a43c9424 Bug 13025: Lie about screen orientation. * 3048f766632b Regression tests for #5856: Do not expose physical screen info via window & window.screen. * 5fbc32940b32 fixup! Bug #5856: Do not expose physical screen info via window & window.screen. * 292b0cd6d6e5 Bug #5856: Do not expose physical screen info via window & window.screen. * d919ec4dedbe Enable tbb-tests/mochitest.ini * 52459e0c7dd3 Bug #12620: TorBrowser regression tests folder https://treeherder.mozilla.org/#/jobs?repo=try&revision=61782741cc52 busted: M(3), M(bc1) testfailed: M(1), M(2), M(4), M(5), Mn38.0.1esr/5856-Do-not-expose-physical-screen-info-via-window-window.screen.test: * 71ecb463b046 Regression tests for #5856: Do not expose physical screen info via window & window.screen. * d919ec4dedbe Enable tbb-tests/mochitest.ini * 52459e0c7dd3 Bug #12620: TorBrowser regression tests folder https://treeherder.mozilla.org/#/jobs?repo=try&revision=91ea5927a49b busted: X testfailed: M(5), M(bc1)38.0.1esr/isolation: * c1e2dfc12244 fixup! Bug 13670.2: Isolate OCSP requests by first party domain * 3ce0db6951ca Bug 16300: Isolate Broadcast Channels to first party. * c054d08392e0 fixup! Bug #13749.2: Regression tests for first-party isolation of cache * afa4c0776abe Bug 13439: No canvas prompt for content-callers. * 935edc4208d6 Bug 13021: Prompt before allowing Canvas isPointIn*() calls. * 5c38cf9e2658 Bug #6253: Add canvas image extraction prompt. * e424bc280f64 Bug 13900: Remove 3rd party HTTP auth tokens. * 4f1f4b8aed26 Bug #15502, Part 2: Regression tests for blob URL isolation * d383dc1d54f1 Bug #15502. Isolate blob URLs to first party; no blobURLs in Web Workers * 25ea17272c4a fixup! Bug 13670.2: Isolate OCSP requests by first party domain * 1f36ee3d68ce Bug 13670.2: Isolate OCSP requests by first party domain * 935c0e104682 Bug #13749.1: regression tests for first party isolation of localStorage * f4b3b48f1901 Bug #13749.2: Regression tests for first-party isolation of cache * 6a99e486c6ac Bug #6564: Isolate DOM storage to first party URI. * fed97227fac5 Bug #6539: Isolate the Image Cache per url bar domain. * f730e6fb0dbf Bug 13742: Isolate cache to URL bar domain. * 553369720c84 Bug #10819: Add a pref, "privacy.thirdparty.isolate", to allow the activation or deactivation of isolating DOM storage and image caching by first party URI. * ff065c6cb9f6 Bug #5742: API allows you to get the url bar URI for a channel or nsIDocument. * 5745c7c53605 Bug #3455.2. Allow RFC1929 authentication (username/password) to SOCKS servers. https://treeherder.mozilla.org/#/jobs?repo=try&revision=f04617e1e11b busted: M(1), M(3), M(4), M(5), X testfailed: M(2), M(bc1), M(bc2), M(bc3), Mn, R(C), R(R1), R(R2), R(R3), R(R4)38.0.1esr/isolation.test1: * 73e313df2b5e Bug #13749.2: Regression tests for first-party isolation of cache https://treeherder.mozilla.org/#/jobs?repo=try&revision=515168b9ab50 busted: X testfailed: M(bc1), M(bc2)38.0.1esr/isolation.test2: * 1bd90b7aad33 Bug #13749.1: regression tests for first party isolation of localStorage https://treeherder.mozilla.org/#/jobs?repo=try&revision=19d8c28c177c busted: X testfailed: M(4), M(bc1)38.0.1esr/isolation.test3: * 8fa4c6df059f Bug #15502, Part 2: Regression tests for blob URL isolation https://treeherder.mozilla.org/#/jobs?repo=try&revision=9b0d05c0bda9 busted: S, X testfailed: M(1), M(bc1)
OK: Bug #5856: Do not expose physical screen info via window & window.screen.
As expected, tbb-tests/test_tor_bug5856.html is failing without the corresponding patches. With the patches applied, we have other failures but not tbb-tests/test_tor_bug5856.html.
OK: Bug #2875: Limit device and system specific CSS Media Queries.
As expected, tbb-tests/test_tor_bug2875.html is failing without the patches. With the patches applied, we have other failures but not tbb-tests/test_tor_bug2875.html.
Unknown: Bug #2950: Make Permissions Manager memory-only
With the patches applied, no failure. I forgot this one and did not try yet the test alone.
OK: Bug #4755: Return client window coordinates for mouse event screenX/Y (for dragend, 0,0 is returned).
As expected, tbb-tests/test_tor_bug4755.html is failing without the patches. With the patches applied, we have other failures, but not tbb-tests/test_tor_bug4755.html.
OK: Bug #2874: Block Components.interfaces from content
As expected, tbb-tests/test_tor_bug2874.html is failing without the patches. With the patches applied, nothing failed.
OK: Bug #13749.1: regression tests for first party isolation of localStorage
As expected, dom/tests/mochitest/localstorage/test_localStorageByFirstParty.html is failing without the patches. With the patches, we have many other tests failing, but not dom/tests/mochitest/localstorage/test_localStorageByFirstParty.html.
NOT OK: Bug #13749.2: Regression tests for first-party isolation of cache
There seems to be something wrong with this one. As expected, netwerk/test/browser/browser_cacheFirstParty.js is failing without the patches. With the patches, we have many other tests failing, but netwerk/test/browser/browser_cacheFirstParty.js is still failing too, although the error is different in both cases (in bc2):
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f04617e1e11b
400 INFO TEST-UNEXPECTED-FAIL | netwerk/test/browser/browser_cacheFirstParty.js | Cache entries expected for iframe.html: 2 -407 INFO TEST-UNEXPECTED-FAIL | netwerk/test/browser/browser_cacheFirstParty.js | Cache entries expected for audio.ogg: 2 -408 INFO TEST-UNEXPECTED-FAIL | netwerk/test/browser/browser_cacheFirstParty.js | Cache entries expected for video.ogv: 2 -431 INFO TEST-UNEXPECTED-FAIL | netwerk/test/browser/browser_cacheFirstParty.js | Cache entries expected for iframe.html: 1 -
NOT OK: Bug #15502, Part 2: Regression tests for blob URL isolation
There seems to be something wrong to with this one too. As expected, dom/base/test/test_tor_bug15502.html is failing without the patches. But it is also failing with the patches applied, with the same error in both cases:
2816 INFO TEST-UNEXPECTED-FAIL | dom/base/test/test_tor_bug15502.html | Test timed out. - expected PASS
Or maybe I forgot a patch in this branch (38.0.1esr/isolation) to make this test pass ?
I also updated my repo to use the latest version of the patches based on 38.1.0esr and push them to Mozilla Try. However I now have a strange error with anything that I push to Try, even with the 38.1.0esr commit without any of our patches:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=648171333847
I will need to push the new patches once this problem with Try is fixed.
I now have run the tests with the latest version of the patches and here are the results:
38.1.0esr/2874-Block-Components.interfaces-from-content: * 2fedd6401715 Bug #2874: Block Components.interfaces from content * 0584bc69c579 Regression tests for #2874: Block Components.interfaces from content * 8bb1871d1662 Enable tbb-tests/mochitest.ini * eb1338e59d18 Bug #12620: TorBrowser regression tests folder https://treeherder.mozilla.org/#/jobs?repo=try&revision=5667a746a7a238.1.0esr/2874-Block-Components.interfaces-from-content.test: * 0584bc69c579 Regression tests for #2874: Block Components.interfaces from content * 8bb1871d1662 Enable tbb-tests/mochitest.ini * eb1338e59d18 Bug #12620: TorBrowser regression tests folder https://treeherder.mozilla.org/#/jobs?repo=try&revision=047068b6189f testfailed: M(5)38.1.0esr/2875-Limit-device-and-system-specific-CSS-Media-Queries: * 5e42a79885da Bug #16315: Test spoofed media queries in picture elements * 62b39799f1f1 Bug #2875: Limit device and system specific CSS Media Queries. * 4b90d3dfa101 Regression tests for #2875: Limit device and system specific CSS Media Queries. * 8bb1871d1662 Enable tbb-tests/mochitest.ini * eb1338e59d18 Bug #12620: TorBrowser regression tests folder https://treeherder.mozilla.org/#/jobs?repo=try&revision=70c7054dba91 testfailed: M(4), M(5), M(bc2)38.1.0esr/2875-Limit-device-and-system-specific-CSS-Media-Queries.test: * 5e42a79885da Bug #16315: Test spoofed media queries in picture elements * 4b90d3dfa101 Regression tests for #2875: Limit device and system specific CSS Media Queries. * 8bb1871d1662 Enable tbb-tests/mochitest.ini * eb1338e59d18 Bug #12620: TorBrowser regression tests folder https://treeherder.mozilla.org/#/jobs?repo=try&revision=bbc8d4f4cebd testfailed: M(5)38.1.0esr/2950-Make-Permissions-Manager-memory-only: * 82c34192f0c9 Bug #2950: Make Permissions Manager memory-only * 7ad308d447b3 Regression tests for Bug #2950: Make Permissions Manager memory-only * d6cc69fcce53 Enable tbb-tests/browser.ini * eb1338e59d18 Bug #12620: TorBrowser regression tests folder https://treeherder.mozilla.org/#/jobs?repo=try&revision=25963bdedf75 busted: B38.1.0esr/2950-Make-Permissions-Manager-memory-only.test: * 7ad308d447b3 Regression tests for Bug #2950: Make Permissions Manager memory-only * d6cc69fcce53 Enable tbb-tests/browser.ini * eb1338e59d18 Bug #12620: TorBrowser regression tests folder https://treeherder.mozilla.org/#/jobs?repo=try&revision=571d0773359638.1.0esr/4755-Return-client-window-coordinates-for-mouse-event-screenX-Y: * 0c22fb2b808d Bug #4755: Return client window coordinates for mouse event screenX/Y (for dragend, 0,0 is returned). * 1b331b328be9 Regression tests for #4755: Return client window coordinates for mouse event screenX/Y (for dragend, 0,0 is returned). * 8bb1871d1662 Enable tbb-tests/mochitest.ini * eb1338e59d18 Bug #12620: TorBrowser regression tests folder https://treeherder.mozilla.org/#/jobs?repo=try&revision=9ce27edb14e1 testfailed: M(2), M(5)38.1.0esr/4755-Return-client-window-coordinates-for-mouse-event-screenX-Y.test: * 1b331b328be9 Regression tests for #4755: Return client window coordinates for mouse event screenX/Y (for dragend, 0,0 is returned). * 8bb1871d1662 Enable tbb-tests/mochitest.ini * eb1338e59d18 Bug #12620: TorBrowser regression tests folder https://treeherder.mozilla.org/#/jobs?repo=try&revision=b718f0dc9bf3 busted: B testfailed: M(5)38.1.0esr/5856-Do-not-expose-physical-screen-info-via-window-window.screen: * 88e847987255 Bug 13025: Lie about screen orientation. * 4bfaa29b87a7 Bug #5856: Do not expose physical screen info via window & window.screen. * bdd2050eb184 Regression tests for #5856: Do not expose physical screen info via window & window.screen. * 8bb1871d1662 Enable tbb-tests/mochitest.ini * eb1338e59d18 Bug #12620: TorBrowser regression tests folder https://treeherder.mozilla.org/#/jobs?repo=try&revision=b815207b716e busted: M(3), M(bc1) testfailed: M(1), M(2), M(4), M(5), Mn38.1.0esr/5856-Do-not-expose-physical-screen-info-via-window-window.screen.test: * bdd2050eb184 Regression tests for #5856: Do not expose physical screen info via window & window.screen. * 8bb1871d1662 Enable tbb-tests/mochitest.ini * eb1338e59d18 Bug #12620: TorBrowser regression tests folder https://treeherder.mozilla.org/#/jobs?repo=try&revision=c3bec8053cee testfailed: M(5)38.1.0esr/isolation: * 35057ce8253e Bug #15703: Regression tests for isolation of mediasource URI * 22fd25196518 Bug #15502. Isolate blob, mediasource & mediastream URLs to first party * efffaa79dfa2 Revert "Bug #15502. Isolate blob URLs to first party; no blobURLs in Web Workers" * f570cc96b05c fixup! Bug #15502, Part 2: Regression tests for blob URL isolation * e29928e5cedf fixup! Bug #15502, Part 2: Regression tests for blob URL isolation * 68cb7039717d Bug 1078657 - Add SpawnTask.js for async tasks in mochitests. r=jmaher * 6c86e0b3ba76 Bug #13670.1: Isolate favicon requests by first party * 8b41e3270339 Bug #13749.2: Regression tests for first-party isolation of cache * e60ceb745790 Bug 16300: Isolate Broadcast Channels to first party. * b9192e10b1cb Bug 13439: No canvas prompt for content-callers. * 0a95cb37783a Bug 13021: Prompt before allowing Canvas isPointIn*() calls. * 9b5c88b50e26 Bug #6253: Add canvas image extraction prompt. * d9bda72ed926 Bug 13900: Remove 3rd party HTTP auth tokens. * 202535212160 Bug #15502. Isolate blob URLs to first party; no blobURLs in Web Workers * eda348f52c66 Bug 13670.2: Isolate OCSP requests by first party domain * df4af5429551 Bug #15502, Part 2: Regression tests for blob URL isolation * bade4109a82c Bug #13749.1: regression tests for first party isolation of localStorage * c00ad2e604ee Bug #6564: Isolate DOM storage to first party URI. * 6e563493a0bf Bug #6539: Isolate the Image Cache per url bar domain. * 5b9f7f92979d Bug 13742: Isolate cache to URL bar domain. * 0077a5265d73 Bug #10819: Add a pref, "privacy.thirdparty.isolate", to allow the activation or deactivation of isolating DOM storage and image caching by first party URI. * 42886d8b7758 Bug #5742: API allows you to get the url bar URI for a channel or nsIDocument. * 6ee15fb44586 Bug #3455.2. Allow RFC1929 authentication (username/password) to SOCKS servers. https://treeherder.mozilla.org/#/jobs?repo=try&revision=5bcf5e5ee1aa busted: B, S38.1.0esr/isolation.test1: * 8b41e3270339 Bug #13749.2: Regression tests for first-party isolation of cache https://treeherder.mozilla.org/#/jobs?repo=try&revision=bac1e7d8ad00 testfailed: M(bc2)38.1.0esr/isolation.test2: * bade4109a82c Bug #13749.1: regression tests for first party isolation of localStorage https://treeherder.mozilla.org/#/jobs?repo=try&revision=57aa6742679c testfailed: M(4)38.1.0esr/isolation.test3: * f570cc96b05c fixup! Bug #15502, Part 2: Regression tests for blob URL isolation * e29928e5cedf fixup! Bug #15502, Part 2: Regression tests for blob URL isolation * 68cb7039717d Bug 1078657 - Add SpawnTask.js for async tasks in mochitests. r=jmaher * df4af5429551 Bug #15502, Part 2: Regression tests for blob URL isolation https://treeherder.mozilla.org/#/jobs?repo=try&revision=c252627855f4 running: M(1) testfailed: M(1)38.1.0esr/isolation.test4: * 35057ce8253e Bug #15703: Regression tests for isolation of mediasource URI * 68cb7039717d Bug 1078657 - Add SpawnTask.js for async tasks in mochitests. r=jmaher https://treeherder.mozilla.org/#/jobs?repo=try&revision=5ad074472859 running: M(1) testfailed: M(1)
Summary of the results:
OK: Bug #5856: Do not expose physical screen info via window & window.screen.
As expected, tbb-tests/test_tor_bug5856.html is failing without the corresponding patches. With the patches applied, we have other failures but not tbb-tests/test_tor_bug5856.html.
OK: Bug #2875: Limit device and system specific CSS Media Queries.
As expected, tbb-tests/test_tor_bug2875.html is failing without the patches. With the patches applied, we have other failures but not tbb-tests/test_tor_bug2875.html.
NOT OK: Bug #2950: Make Permissions Manager memory-only
OK: Bug #4755: Return client window coordinates for mouse event screenX/Y (for dragend, 0,0 is returned).
As expected, tbb-tests/test_tor_bug4755.html is failing without the patches. With the patches applied, we have other failures, but not tbb-tests/test_tor_bug4755.html.
OK: Bug #2874: Block Components.interfaces from content
As expected, tbb-tests/test_tor_bug2874.html is failing without the patches. With the patches applied, nothing failed.
Unknown: isolation patches
The isolation.test* branches have test failures as expected. However the isolation branch fails to build because of a -Werror=unused-variable error, so I have not been able to check yet that the patch make the tests not fail:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5bcf5e5ee1aa
Seems like there may be some sponsor 131 adjacent work here. It does make sense that building just a Mozilla tag should result in all of our tor-browser specific tests failing.