Review RemoteSettings for ESR78

added 1 deleted label

List of requests that I have seen in browser console (with security.remote_settings.intermediates.enable = false from #30682 (closed)):

https://firefox.settings.services.mozilla.com/v1/buckets/monitor/collections/changes/records?
https://firefox.settings.services.mozilla.com/v1/buckets/blocklists/collections/addons/changeset?_expected=1595258518841&_since=%221592386739136%22
https://content-signature-2.cdn.mozilla.net/chains/remote-settings.content-signature.mozilla.org-2020-09-04-17-16-15.chain
https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/search-config/changeset?_expected=1595254832070&_since=%221593025575021%22
https://content-signature-2.cdn.mozilla.net/chains/remote-settings.content-signature.mozilla.org-2020-09-04-17-16-15.chain
https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/search-default-override-allowlist/changeset?_expected=1595254618540
https://content-signature-2.cdn.mozilla.net/chains/remote-settings.content-signature.mozilla.org-2020-09-04-17-16-15.chain
https://firefox.settings.services.mozilla.com/v1/buckets/security-state/collections/cert-revocations/changeset?_expected=1594941182549
https://content-signature-2.cdn.mozilla.net/chains/onecrl.content-signature.mozilla.org-2020-09-04-17-16-07.chain
https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/url-classifier-skip-urls/changeset?_expected=1594765026508&_since=%221590784809268%22
https://content-signature-2.cdn.mozilla.net/chains/remote-settings.content-signature.mozilla.org-2020-09-04-17-16-15.chain
https://firefox.settings.services.mozilla.com/v1/buckets/blocklists/collections/plugins/changeset?_expected=1594658825158&_since=%221584379093300%22
https://content-signature-2.cdn.mozilla.net/chains/remote-settings.content-signature.mozilla.org-2020-09-04-17-16-15.chain
https://firefox.settings.services.mozilla.com/v1/buckets/security-state/collections/intermediates?_expected=1591908420097
https://firefox.settings.services.mozilla.com/v1/buckets/security-state/collections/onecrl?_expected=1591722356162
https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/public-suffix-list/changeset?_expected=1575468539758
https://content-signature-2.cdn.mozilla.net/chains/remote-settings.content-signature.mozilla.org-2020-09-04-17-16-15.chain
https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/language-dictionaries?_expected=1569410800356
https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/anti-tracking-url-decoration?_expected=1564511755134
https://content-signature-2.cdn.mozilla.net/chains/remote-settings.content-signature.mozilla.org-2020-09-04-17-16-15.chain
https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/sites-classification?_expected=1544035467383
https://firefox.settings.services.mozilla.com/v1/buckets/pinning/collections/pins/changeset?_expected=1485794868067
https://content-signature-2.cdn.mozilla.net/chains/pinning-preload.content-signature.mozilla.org-2020-09-04-17-16-12.chain
https://firefox.settings.services.mozilla.com/v1/buckets/blocklists/collections/gfx?_expected=1480349135384
https://content-signature-2.cdn.mozilla.net/chains/remote-settings.content-signature.mozilla.org-2020-09-04-17-16-15.chain

mentioned in merge request !8 (merged)

We could also use this to think whether it's ok for some cases (e.g. like in #30682 (closed)) to allow them and just disable the syncing part, but still use the dumps that the browser ships (e.g. services/settings/dumps/security-state/intermediates.json). At the time of writing this, the dump for 78.0.1esr is the same as the remote state.

I mean, the intermediates.json dump.

added Needs Review label

added Doing label and removed Needs Review label

assigned to @acat

mentioned in issue #40048 (closed)

Making RemoteSettings instances work with just the local JSON dumps seems not so simple, as these are only loaded on first time, so even if in later releases we update these JSON dumps the local DBs will not keep up to date with them. Of course we can patch the implementation to change this, but I'm not sure if it's worth the effort. Given that we need at least some of the instances (e.g. blocklists), the privacy gain of avoiding some collections to be kept up to date with remote is not so big (we still have to ping Firefox servers every 24 hours, but we can avoid one additional request per RemoteSettings instance that we do not need). This was already mentioned in #31740 (closed).

Something I missed in the previous review is that even if a RemoteSettings instance is not explicitly created, a client for it may be automatically be created in https://searchfox.org/mozilla-esr78/rev/67637e7e1d647453974cb4d970cef5f5dd087d69/services/settings/remote-settings.js#134 (when called from https://searchfox.org/mozilla-esr78/rev/67637e7e1d647453974cb4d970cef5f5dd087d69/services/settings/remote-settings.js#302, which after first run is then scheduled every 24 hours via https://searchfox.org/mozilla-esr78/rev/67637e7e1d647453974cb4d970cef5f5dd087d69/services/settings/servicesSettings.manifest#7). That will happen if the collection is in the "main" bucket and its local database exists or a JSON dump for it exists. I think we can patch this and avoid creating these instances.

With that patch, this is the list of remaining active RemoteSettings clients (bucket/collection).

toolkit/mozapps/extensions/Blocklist.jsm:
- blocklists/addons
- blocklists/gfx
- blocklists/plugins
  - Those are already in 68, I think we can keep them.
toolkit/components/antitracking/URLDecorationAnnotationsService.jsm:
- main/anti-tracking-url-decoration:
  - This is new, I don't think we need it -> patch.
browser/components/aboutlogins/LoginBreaches.jsm, browser/components/fxmonitor/FirefoxMonitor.jsm:
- main/fxmonitor-breaches
  - This keeps a list of sites with breaches, and displays a doorhanger warning if the user visits that site. In principle I would think this can be good for users, but it does use the "Firefox Monitor" brand and points to monitor.firefox.com, so I'm not sure if we should keep it. But I guess is more about deciding whether to remove the whole feature rather than just remove the RemoteSettings instance. Maybe create a ticket for this?
toolkit/components/search/SearchUtils.jsm (already patched), browser/modules/HomePage.jsm (new):
- main/hijack-blocklists
  - The first callsite is already addressed in the current patch. The second one is new, apparently resets the homepage url if the value is in a blocklist: https://bugzilla.mozilla.org/show_bug.cgi?id=1535049. I'm not sure this is highly important, so I would also patch.
netwerk/dns/PublicSuffixList.jsm
- main/public-suffix-list
  - We can address this in #40073 (closed).
security/manager/ssl/RemoteSecuritySettings.jsm
- pinning/pins
- security-state/cert-revocations
- security-state/intermediates
- security-state/onecrl
  - cert-revocations is new, crlite. I think we should check the value of security.remote_settings.crlite_filters.enabled and not create the RemoteSettings instance if this is false. And similarly with security-state/intermediates.
  - The other two, pinning/pins and security-state/onecrl were already in 68, so we can keep them.

Ok, I looked a bit more exhaustively and I was missing some instances. There's no easy way of finding out this list, since many of these instances are created lazily some time after the browser has started.

That said, I made a patch that (apart from removing a couple more instances):

Clears the remote-settings indexeddb taking advantage of the v2 -> v3 migration (for a fresh start).
Stops packaging some JSON dumps for collections we should not need.

I did this because for collections in the "main" bucket, even if there's not explicit RemoteSettings(collection) call, they will still be updated/persisted if there is a JSON dump for them or there's data already stored in the local DB. With this and #40048 (closed), #40073 (closed), #30682 (closed) fixed (in principle) in ESR78 these should be the only active/updated/persisted RemoteSettings instances:

main/search-default-override-allowlist
main/language-dictionaries
main/hijack-blocklists (I think we can actually keep this one, given that the use case is similar to blocklists/addons)
blocklists/addons
blocklists/gfx
blocklists/plugins
security-state/onecrl
pinning/pins

These would be active if it was not for the reason mentioned in each one:

main/anti-tracking-url-decoration: patch
main/fxmonitor-breaches: extensions.fxmonitor.enabled = false, signon.management.page.breach-alerts.enabled = false(#40048 (closed))
main/public-suffix-list: #40073 (closed)
main/url-classifier-skip-urls: patch
main/search-config: browser.search.modernConfig = false
blocklists/addons-bloomfilters: extensions.blocklist.useMLBF = false
security-state/cert-revocations: patch + security.remote_settings.crlite_filters.enabled = false
security-state/intermediates: patch + security.remote_settings.intermediates.enabled = false (#30682 (closed))

These are not active because of the newtab/activity-stream patch:

main/cfr
main/cfr-fxa
main/cfr-srg
main/message-groups
main/messaging-experiments
main/ms-language-packs
main/sites-classification
main/tippytop
main/whats-new-panel

And these because of Normandy being disabled:

main/normandy-recipes
main/normandy-recipes-capabilities
main/personality-provider-models
main/personality-provider-recipe

mentioned in merge request !26 (closed)

mentioned in merge request !33 (merged)

closed via merge request !33 (merged)

mentioned in issue #40099

Review RemoteSettings for ESR78

Designs

Child items ...

Activity