tor-browser-bundle-testsuite issueshttps://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/-/issues2022-07-09T21:51:25Zhttps://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/-/issues/40060Write test for spoofed navigator* values in workers2022-07-09T21:51:25ZGeorg KoppenWrite test for spoofed navigator* values in workerslegacy/trac#13027 fixed the problem with navigator* values not being spoofed. Back then we could not write a Mozmill test as Mozmill itself was affected by the underlying issue (https://bugzilla.mozilla.org/show_bug.cgi?id=1062920 togeth...legacy/trac#13027 fixed the problem with navigator* values not being spoofed. Back then we could not write a Mozmill test as Mozmill itself was affected by the underlying issue (https://bugzilla.mozilla.org/show_bug.cgi?id=1062920 together with https://bugzilla.mozilla.org/show_bug.cgi?id=1078163).https://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/-/issues/40059Investigate tbb-test for #23104 (line height compensation)2022-12-08T15:15:25ZAlex CatarineuInvestigate tbb-test for #23104 (line height compensation)After the changes from tor-browser#30832 we set `layout.css.line-height.normal-as-resolved-value.enabled = false` in order to measure the `line-height` correctly for the arabic test case (see https://gitlab.torproject.org/tpo/application...After the changes from tor-browser#30832 we set `layout.css.line-height.normal-as-resolved-value.enabled = false` in order to measure the `line-height` correctly for the arabic test case (see https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/30832#note_2664115). We should try to investigate and possibly fix the test case so that we don't have to set that pref.https://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/-/issues/40058Add support for Fenix into existing Tor Browser regression and integration tests2022-07-08T23:09:57ZMatthew FinkelAdd support for Fenix into existing Tor Browser regression and integration testsIntegrate Fenix into test suiteIntegrate Fenix into test suitehttps://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/-/issues/40057add automated test to ensure OS-provided certificates are not used2022-07-08T21:22:02ZMark Smithadd automated test to ensure OS-provided certificates are not usedFrom tor-browser#33534: Firefox 75 can use OS-provided certificates. This fingerprinting vector is disabled by default but we should add an automated test to confirm that it stays off during future rebasings.
https://blog.mozilla.org/se...From tor-browser#33534: Firefox 75 can use OS-provided certificates. This fingerprinting vector is disabled by default but we should add an automated test to confirm that it stays off during future rebasings.
https://blog.mozilla.org/security/2020/04/14/expanding-client-certificates-in-firefox-75/https://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/-/issues/40056mochitest failures for #23104 patch2022-06-23T19:11:59ZArthur Edelsteinmochitest failures for #23104 patchI ran the mochitests for the legacy/trac#23104 patch on our tor-browser-52.6.0esr-8.0-2 branch on Debian GNU/Linux 9.1 and I ran into the following failures:
```
Passed: 3
Failed: 3
Todo: 0
failed | Line Height validation - got 20.4, ex...I ran the mochitests for the legacy/trac#23104 patch on our tor-browser-52.6.0esr-8.0-2 branch on Debian GNU/Linux 9.1 and I ran into the following failures:
```
Passed: 3
Failed: 3
Todo: 0
failed | Line Height validation - got 20.4, expected 19.8
failed | Line Height validation - got 20.4, expected 19.8
failed | Line Height validation - got 20.4, expected 19.8
passed | Line Height validation
passed | Line Height validation
passed | Line Height validation
```https://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/-/issues/40055Unit test for js locale2022-08-04T21:45:08ZArthur EdelsteinUnit test for js localeLet's introduce an automated regression test to ensure that the `javascript.use_us_english_locale` pref is applied to
* members of `window.Date` objects
* Intl API
* toLocaleString
* `DateTimeFormat.formatToParts`
I can work on this at ...Let's introduce an automated regression test to ensure that the `javascript.use_us_english_locale` pref is applied to
* members of `window.Date` objects
* Intl API
* toLocaleString
* `DateTimeFormat.formatToParts`
I can work on this at some point. The problem is that we need to set the LANG variable to de_DE or another non-en_US locale before starting Firefox. Maybe boklm's testing framework can do this? Or maybe it's possible in Mozilla's automated testing system, but I don't know a way yet.https://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/-/issues/40054Regression tests for canvas image extraction prompt2022-08-04T21:45:09ZArthur EdelsteinRegression tests for canvas image extraction promptIt would be nice to have regression tests for our Canvas Image Extraction patch (legacy/trac#6253).
mcs and brade developed some tests here: https://people.torproject.org/~brade/tests/canvasTest.html. Perhaps we could port these to a mo...It would be nice to have regression tests for our Canvas Image Extraction patch (legacy/trac#6253).
mcs and brade developed some tests here: https://people.torproject.org/~brade/tests/canvasTest.html. Perhaps we could port these to a mochitest.https://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/-/issues/40053Test cases for OCSP isolation2022-08-04T21:45:10ZArthur EdelsteinTest cases for OCSP isolationWe need some sort of regression tests for OCSP isolation, which was implemented in legacy/trac#13670.2.We need some sort of regression tests for OCSP isolation, which was implemented in legacy/trac#13670.2.https://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/-/issues/40052Unit tests for font whitelisting patch2022-06-17T20:13:46ZArthur EdelsteinUnit tests for font whitelisting patchWe need unit tests for the font whitelisting patch (legacy/trac#13313).We need unit tests for the font whitelisting patch (legacy/trac#13313).https://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/-/issues/40051Regression tests for Bug #2950 (Make Permissions Manager memory-only) never f...2022-06-16T03:55:20ZboklmRegression tests for Bug #2950 (Make Permissions Manager memory-only) never failsThe regression test for bug legacy/trac#2950 does not fail when used on 38.2.0esr without patch:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a81cdc18350bThe regression test for bug legacy/trac#2950 does not fail when used on 38.2.0esr without patch:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a81cdc18350bhttps://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/-/issues/40049Unit tests to confirm content pref service doesn't write to disk in PBM2022-06-16T00:21:49ZArthur EdelsteinUnit tests to confirm content pref service doesn't write to disk in PBMIf we can confirm that the Content Pref Service doesn't write to disk in private browsing mode, then we can drop our patch for legacy/trac#3229. See the discussion at https://bugzilla.mozilla.org/show_bug.cgi?id=967809
Probably the most...If we can confirm that the Content Pref Service doesn't write to disk in private browsing mode, then we can drop our patch for legacy/trac#3229. See the discussion at https://bugzilla.mozilla.org/show_bug.cgi?id=967809
Probably the most secure way to do this is to write unit tests.https://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/-/issues/40048add automated tests for svg.in-content.enabled2022-06-16T00:14:53ZMark Smithadd automated tests for svg.in-content.enabledAs a followup to legacy/trac#12827 and legacy/trac#15794, we should (as Mike suggested) add a bunch of tests for various SVG-related scenarios including things such as use of <style> and <img> tags within <svg> blocks. Kathy and I teste...As a followup to legacy/trac#12827 and legacy/trac#15794, we should (as Mike suggested) add a bunch of tests for various SVG-related scenarios including things such as use of <style> and <img> tags within <svg> blocks. Kathy and I tested quite a few scenarios when we did the work for legacy/trac#12827 but obviously we missed at least one.
I am opening this new, separate ticket so we do not lose track of this work item.https://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/-/issues/40047Regression tests for #5926 patch2022-06-15T02:45:11ZArthur EdelsteinRegression tests for #5926 patchThis regression test should cover all problems reported in legacy/trac#5926, legacy/trac#10284, and legacy/trac#13019.This regression test should cover all problems reported in legacy/trac#5926, legacy/trac#10284, and legacy/trac#13019.https://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/-/issues/40044Create a test for proxy usage in Moat requests2022-08-04T21:45:05ZPier Angelo VendrameCreate a test for proxy usage in Moat requestsIn addition to checking that Tor works over proxies (#40043) we should add tests that:
1. Moat requests can be done through a proxy as well (this can make finding the location harder for S96-related work, but it may be compulsory for so...In addition to checking that Tor works over proxies (#40043) we should add tests that:
1. Moat requests can be done through a proxy as well (this can make finding the location harder for S96-related work, but it may be compulsory for some networks)
2. the proxy settings that are set in the advanced preferences panel are used also for thishttps://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/-/issues/40030Fix onion_client_auth2022-02-04T21:42:42ZMatthew FinkelFix onion_client_auth```
mozversion application_buildid: 20210810010101
mozversion application_display_name: Tor Browser ...```
mozversion application_buildid: 20210810010101
mozversion application_display_name: Tor Browser
mozversion application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
mozversion application_name: Firefox
mozversion application_remotingname: firefox
mozversion application_vendor: Mozilla
mozversion application_version: 78.12.0
mozversion platform_buildid: 20210810010101
mozversion platform_version: 78.12.0
Application command: /home/sysrqb/tor-browser-bundle-testsuite/tmp/FjDegO9ORB/tor-browser_en-US/Browser/ff_wrapper -no-remote -marionette -profile /home/sysrqb/tor-browser-bundle-testsuite/reports/r/tbb.
nightly.2021.08.10_onion_client_auth/results-tor-browser-linux64-tbb-nightly.2021.08.10_en-US.tar.xz/onion_client_auth_ws/tmpZWwYGi.profile.default
Profile path is /home/sysrqb/tor-browser-bundle-testsuite/reports/r/tbb.nightly.2021.08.10_onion_client_auth/results-tor-browser-linux64-tbb-nightly.2021.08.10_en-US.tar.xz/onion_client_auth_ws/tmpZWwYGi
.profile.default
Starting fixture servers
Fixture server listening on http://127.0.0.1:33281/
Fixture server listening on https://127.0.0.1:35923/
e10s is enabled
mozversion application_buildid: 20210810010101
mozversion application_display_name: Tor Browser
mozversion application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
mozversion application_name: Firefox
mozversion application_remotingname: firefox
mozversion application_vendor: Mozilla
mozversion application_version: 78.12.0
mozversion platform_buildid: 20210810010101
mozversion platform_version: 78.12.0
SUITE-START | Running 1 tests
TEST-START | ../../../../marionette/tor_browser_tests/test_onion_client_auth.py Test.test_client_auth
Application command: /home/sysrqb/tor-browser-bundle-testsuite/tmp/FjDegO9ORB/tor-browser_en-US/Browser/ff_wrapper -no-remote -marionette -profile /home/sysrqb/tor-browser-bundle-testsuite/reports/r/tbb.
nightly.2021.08.10_onion_client_auth/results-tor-browser-linux64-tbb-nightly.2021.08.10_en-US.tar.xz/onion_client_auth_ws/tmpZWwYGi.profile.default
TEST-UNEXPECTED-ERROR | ../../../../marionette/tor_browser_tests/test_onion_client_auth.py Test.test_client_auth | NoSuchElementException: Unable to locate element: enumeration
stacktrace:
WebDriverError@chrome://marionette/content/error.js:175:5
NoSuchElementError@chrome://marionette/content/error.js:387:5
element.find/</<@chrome://marionette/content/element.js:330:16
Traceback (most recent call last):
File "/home/sysrqb/tor-browser-bundle-testsuite/virtualenv-marionette-5.0.0/local/lib/python2.7/site-packages/marionette_harness-5.0.0-py2.7.egg/marionette_harness/marionette_test/testcases.py", line 1
59, in run
testMethod()
File "/home/sysrqb/tor-browser-bundle-testsuite/marionette/tor_browser_tests/test_onion_client_auth.py", line 183, in test_client_auth
m.find_element('id', 'enumeration')
File "/home/sysrqb/tor-browser-bundle-testsuite/virtualenv-marionette-5.0.0/local/lib/python2.7/site-packages/marionette_driver-3.0.0-py2.7.egg/marionette_driver/marionette.py", line 1694, in find_elem
ent
body, key="value")
File "/home/sysrqb/tor-browser-bundle-testsuite/virtualenv-marionette-5.0.0/local/lib/python2.7/site-packages/marionette_driver-3.0.0-py2.7.egg/marionette_driver/decorators.py", line 26, in _
return func(*args, **kwargs)
File "/home/sysrqb/tor-browser-bundle-testsuite/virtualenv-marionette-5.0.0/local/lib/python2.7/site-packages/marionette_driver-3.0.0-py2.7.egg/marionette_driver/marionette.py", line 598, in _send_mess
age
self._handle_error(err)
File "/home/sysrqb/tor-browser-bundle-testsuite/virtualenv-marionette-5.0.0/local/lib/python2.7/site-packages/marionette_driver-3.0.0-py2.7.egg/marionette_driver/marionette.py", line 618, in _handle_er
ror
raise errors.lookup(error)(message, stacktrace=stacktrace)
TEST-INFO took 61639ms
```Tor Browser: 11.0 Issues with previous releasehttps://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/-/issues/40028Fix noscript test2022-07-26T16:41:06ZMatthew FinkelFix noscript test```
mozversion application_buildid: 20210810010101
mozversion application_display_name: Tor Browser
mozversion application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
mozversion application_name: Firefox
mozversion application_remotingnam...```
mozversion application_buildid: 20210810010101
mozversion application_display_name: Tor Browser
mozversion application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
mozversion application_name: Firefox
mozversion application_remotingname: firefox
mozversion application_vendor: Mozilla
mozversion application_version: 78.12.0
mozversion platform_buildid: 20210810010101
mozversion platform_version: 78.12.0
Application command: /home/sysrqb/tor-browser-bundle-testsuite/tmp/tqma2o0O7h/tor-browser_en-US/Browser/ff_wrapper -no-remote -marionette -profile /home/sysrqb/tor-browser-bundle-testsuite/reports/r/tbb.
nightly.2021.08.10/results-tor-browser-linux64-tbb-nightly.2021.08.10_en-US.tar.xz/noscript_ws/tmpX3SFXr.profile.default
Profile path is /home/sysrqb/tor-browser-bundle-testsuite/reports/r/tbb.nightly.2021.08.10/results-tor-browser-linux64-tbb-nightly.2021.08.10_en-US.tar.xz/noscript_ws/tmpX3SFXr.profile.default
Starting fixture servers
Fixture server listening on http://127.0.0.1:44557/
Fixture server listening on https://127.0.0.1:32975/
e10s is enabled
mozversion application_buildid: 20210810010101
mozversion application_display_name: Tor Browser
mozversion application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
mozversion application_name: Firefox
mozversion application_remotingname: firefox
mozversion application_vendor: Mozilla
mozversion application_version: 78.12.0
mozversion platform_buildid: 20210810010101
mozversion platform_version: 78.12.0
SUITE-START | Running 1 tests
TEST-START | ../../../../marionette/tor_browser_tests/test_noscript.py Test.test_noscript
TEST-UNEXPECTED-ERROR | ../../../../marionette/tor_browser_tests/test_noscript.py Test.test_noscript | UnknownException: Reached error page: about:neterror?e=dnsNotFound&u=http%3A//test-data.tbb.mars-att
acks.org/noscript//http_src.html&c=UTF-8&d=We%20can%E2%80%99t%20connect%20to%20the%20server%20at%20test-data.tbb.mars-attacks.org.
stacktrace:
WebDriverError@chrome://marionette/content/error.js:175:5
UnknownError@chrome://marionette/content/error.js:482:5
handleReadyState@chrome://marionette/content/listener.js:297:13
handleEvent@chrome://marionette/content/listener.js:263:14
Traceback (most recent call last):
File "/home/sysrqb/tor-browser-bundle-testsuite/virtualenv-marionette-5.0.0/local/lib/python2.7/site-packages/marionette_harness-5.0.0-py2.7.egg/marionette_harness/marionette_test/testcases.py", line 1
59, in run
testMethod()
File "/home/sysrqb/tor-browser-bundle-testsuite/marionette/tor_browser_tests/test_noscript.py", line 25, in test_noscript
self.marionette.navigate("%s/http_src.html" % self.http_url)
File "/home/sysrqb/tor-browser-bundle-testsuite/virtualenv-marionette-5.0.0/local/lib/python2.7/site-packages/marionette_driver-3.0.0-py2.7.egg/marionette_driver/marionette.py", line 1467, in navigate
{"url": url})
File "/home/sysrqb/tor-browser-bundle-testsuite/virtualenv-marionette-5.0.0/local/lib/python2.7/site-packages/marionette_driver-3.0.0-py2.7.egg/marionette_driver/decorators.py", line 26, in _
return func(*args, **kwargs)
File "/home/sysrqb/tor-browser-bundle-testsuite/virtualenv-marionette-5.0.0/local/lib/python2.7/site-packages/marionette_driver-3.0.0-py2.7.egg/marionette_driver/marionette.py", line 598, in _send_mess
age
self._handle_error(err)
File "/home/sysrqb/tor-browser-bundle-testsuite/virtualenv-marionette-5.0.0/local/lib/python2.7/site-packages/marionette_driver-3.0.0-py2.7.egg/marionette_driver/marionette.py", line 618, in _handle_er
ror
raise errors.lookup(error)(message, stacktrace=stacktrace)
TEST-INFO took 14969ms
```https://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/-/issues/40006Migrate marionette tests because of deprecation2022-02-04T20:53:04ZAlex CatarineuMigrate marionette tests because of deprecation*At some point* we should migrate to something else, since `marionette_harness` and `marionette_driver` are deprecated: https://groups.google.com/forum/#!topic/mozilla.tools.marionette/s6_03Qt2ubc (also https://lists.mozilla.org/pipermai...*At some point* we should migrate to something else, since `marionette_harness` and `marionette_driver` are deprecated: https://groups.google.com/forum/#!topic/mozilla.tools.marionette/s6_03Qt2ubc (also https://lists.mozilla.org/pipermail/tools-marionette/2019-January/000060.html). Given the effort required, I would try to postpone this as much as possible, so I would keep current `marionette` as long as tests are reasonably in good shape.
If at some point tests start failing without good reason, we could try to use `marionette_driver` and `marionette_harness` from the in-tree gecko repository, since these will be more up-to-date than the pypi packages we currently use. Then, if that doesn't solve the issues, we should decide what's the best alternative and port the tests.Tor Browser: 11.0 Issues with previous releasehttps://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/-/issues/8558Re-verify app-launching defenses on Windows2021-07-02T15:03:43ZMike PerryRe-verify app-launching defenses on WindowsRsnake claims that some stuff he did 3 years ago still works on TBB. We certainly fixed the two vectors he mentioned (itms and smb) with Torbutton, but it is possible that one or more random things have been broken/undone by FF17. We sho...Rsnake claims that some stuff he did 3 years ago still works on TBB. We certainly fixed the two vectors he mentioned (itms and smb) with Torbutton, but it is possible that one or more random things have been broken/undone by FF17. We should retest as many of them as we can, especially on Windows. Especially since Rsnake seems insistent on being as unhelpful as possible :/. Gotta love timewasters....
Most decloaking attacks are based on plugins, which are disabled by a Firefox patch and also by Firefox settings, but the following two decloak.net attacks should be retested:
1. "When the iTunes is installed, it registers the itms:// protocol handler. This protocol handler will open iTunes and do a direct connection to the specified URL. There are some restrictions on the URL you can pass, but we found a nice way around them :-)"
2. "When Microsoft Office is installed and configured to automatically open documents, a file can be returned which automatically downloads an image from the internet. This can bypass proxy settings and expose the real DNS servers of the user."
Unfortunately, decloak.net is now down, so the exact itms url it used is unavailable (unless the source is still around somewhere).
Also, this test should be verified on Windows:
http://pseudo-flaw.net/tor/torbutton/ipleak-dotnet-assistant.html
I think the .NET assistant addon might need to be explicitly installed these days. It used to auto-install with some piece of .NET but then Mozilla blacklisted it. They may have removed the blacklist, though...
Also, we should try some SMB urls on windows. Native Firefox SMB handling appears to be unimplemented still, but it may be possible to shove something in the registry that enables an external handler:
http://kb.mozillazine.org/Register_protocol#Windows
http://msdn.microsoft.com/en-us/library/aa767914.aspx
Such external handlers *should* still be blocked by Torbutton, though. They certainly are on MacOS and Linux...https://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/-/issues/6585Automated testing framework for TBB2021-07-02T14:46:45ZMike PerryAutomated testing framework for TBBMozilla has like 13 automated testing frameworks for Firefox. We should write a wrapper script that calls them all, and also performs some additional integration tests that we write.
https://developer.mozilla.org/en/Mozilla_automated_te...Mozilla has like 13 automated testing frameworks for Firefox. We should write a wrapper script that calls them all, and also performs some additional integration tests that we write.
https://developer.mozilla.org/en/Mozilla_automated_testing
Ideally, we'd want to get this framework to the point where it can easily be run by volunteers on their obscure/rare OS+AV software combinations, so we can crowdsource that testing rather than keep an instance of everything in house. There might be some additional cleanup we need to do for this, though. It looks like the Mozilla frameworks are not easy for end users to run:
https://bugzilla.mozilla.org/show_bug.cgi?id=514067https://gitlab.torproject.org/tpo/applications/tor-browser-bundle-testsuite/-/issues/5291Re-test TBB on dual-stacked IPv6 machine2021-06-14T13:50:52ZMike PerryRe-test TBB on dual-stacked IPv6 machineThis sure looks like a scary bug: https://bugzilla.mozilla.org/show_bug.cgi?id=684893
We should find someone on a dual-stack to verify that whatever the hell "Happy Eyeballs autodetection" means, it doesn't end up allowing non-proxied ...This sure looks like a scary bug: https://bugzilla.mozilla.org/show_bug.cgi?id=684893
We should find someone on a dual-stack to verify that whatever the hell "Happy Eyeballs autodetection" means, it doesn't end up allowing non-proxied IPv6 traffic. The feature sounds like some kind of testing mechanism to probe for IPv6 connectivity and fallback to IPv4 otherwise.
Since Tor uses SOCKS hostnames and only proxies IPv4 traffic right now, we probably can test this with an html page with some img tags in it with IPv6 IP addresses, and over a few different schemes for good measure.
We should then keep this page in some sort of test archive, because I imagine we'll want to keep checking on this situation periodically in the future. We might want to add in some ipv6 and 6+4 host names in the test for this reason.