navigator.storage "best effort" + "persistent" leak partitionSize/totalSpace entropy

changed the description

added Backlog ~95597 labels

changed milestone to %Sponsor 131 - Phase 3 - Major ESR 102 Migration

changed the description

ugh, it's "estimate" not "quota", feel free to fix the title

https://hg.mozilla.org/mozilla-central/rev/b5d3ace93ec2 is where's it's all happening

changed title from harden ESR102+ storage quota with RFP upstream to harden ESR102+ storage estimate with RFP upstream

I had TBA 96.x installed on a 64 GB Android device, and it showed 2,147,483,648 as you expected. I then updated the browser and it shows 5641604300 now.

Also see - https://github.com/arkenfox/user.js/issues/1512 - OP here has been updated with new data. As said elsewhere, this looks like we can surmise that 5 estimates are 64gb (or lower?), but the fact that it's not stable is a bit worrying for real-time cross-site linkage

If you want to add it: I booted an Ubuntu VM with 15GB of storage, and it shows 1521166745 on FF102+.

2934867968 : Debian XCFE 2glops 650gb spare from 1TB is weird why is it around 3 when everyone else with 100gb+ spare disk space is a 10

changed the description

changed title from harden ESR102+ storage estimate with RFP upstream to harden ESR102+ storage estimate with RFP upstream [upstream 1781277]

changed the description

added Sponsor 131 label

added Uplift label

removed FF102-esr label

changed milestone to %Sponsor 131 - Phase 2 - Privacy Browser

added Bug label

removed Backlog label

added Roadmap::Future label

added All Platforms label

mentioned in issue #25438 (closed)

cypherpunks1 said

Is there a benefit of setting a default value for dom.quotaManager.temporaryStorage.fixedLimit? Using 52428800 would mean a storage estimate of 10737418240 on all platforms.

for testing just whack this into your console

navigator.storage.estimate().then(estimate => {console.log(estimate.quota)})

note: you need to restart the browser between tests for the new value to take effect, or in the case of android, a force stop

Tests

windows: I have TBs of room
      -1 : 10737418240 (to be expected = max limit)
52428800 : 10737418240 (value we want)
26214400 :  5368709120 (to see if a lower value is respected)

android
      -1 :  5512729396
52428800 : 10737418240 (to see if a higher value is respected)

does what it says on the can fixedLimit - but IDK if that pref will stay around, and wold simply prefer we hardcode the value under RFP

for upstream (TB doesn't prompt for persistent data permission), this of course isn't robust enough if permission is granted

Would temporary and persistent storage require different estimates when the value is hard-coded?

not sure if I follow the terms correctly

in FF

if you go to https://arkenfox.github.io/TZP/tzp.html#storage quota will detect e.g. 10GB (as a default with no permissions granted). If you then hit the click for manager you are prompted for permission - this is a permanent. Click Allow, and this then reveals a different figure for manager. If you rerun the section you will get the same answer for both: for me it is 300009650176 bytes)
if you restart FF and load the test, it will report e.g. 300009650176 bytes for quota (and manager if clicked) - this is to be expected, we granted permission
if you revoke the permission, via the urlbar, the permission is not revoked until a new session - e.g. I still return 300009650176 bytes in a test reload, even in a new tab with the old one closed: in a PB window it returns the original 10GB

in TB, the permission is blocked, users are never prompted - IDK where we did that. So for TB, at this stage all we care about is the hardcoded value for estimate. We don't want to use the pref for an upper limit, because the problem is those who fall under that.

IDK if, when permission is granted, the new value has entropy. If we wanted to relax that permission, it would at least be session only (site permissions are memory due to permissions.memory_only = true. But we should decide if we want this (allowed the gated permission) and if it holds entropy. I honestly do not know what benefit it holds for TB users (downloading 100GB files from mega? no idea)

IDK where we did that

anyone? @pierov

I am considering a merge request to return a hard-coded value for navigator.storage.estimate() without actually changing the limits. I'm not sure which one would be preferable. I thought this would also cover the persistent storage but that may not be the case.

in my comment at #41065 (comment 2883802) I tested dom.quotaManager.temporaryStorage.fixedLimit and this seems to work, but in costa rica @tom and I talked about it and I think this only sets an upper limit .. edit: I am now confused and like everything all the time everywhere, it all needs to be retested :)

https://bugzilla.mozilla.org/show_bug.cgi?id=1781277#c3 (emphasis mine)

so in effect we are really only disguising things for people who have at least cap free space

If a user allows a per-site permission (e.g. in FF), then we should respect that - it's not a universal and likely only limited to one or two sites, and it unbreaks that site/functionality. For TB, we can suppress the gated permission, lock the default permission, and we still only allow per session permissions at worst.

So lets get testing with dom.quotaManager.temporaryStorage.fixedLimit, I suggest 5GB, locked pref because we love locked prefs. Not a fan of relying on prefs, but this would solve the issue (tests permitting) in a single flip and we wouldn't have to worry about upstream (or waiting for them) or rebasing patches or dealing with users fiddling in about:config (lock those prefs!!)

https://bugzilla.mozilla.org/show_bug.cgi?id=1781277#c3

Given this, I think what I'm leaning towards would be to hardcode the limit when RFP is enabled, similar to the override pref. This would affect all origins (that is, an exempted origin wouldn't get a different quota) but that's okay.

I think that also means when permission is granted we still return our hardcoded value? Not a fan of this if IIUIC, see previous comment - permission should be respected, e.g. same as a canvas exception

As I understand it, the 10 GB group limit is there to avoid a single group using up too much disk space. What if we hard-coded a very big number for the estimate to cover both the temporary and persistent storage? This would not change the actual group limits.

What if we hard-coded a very big number

IIUIC ... so two things. One is sites via user gestures can request permission and thus the real value is returned, so e.g. you don't try and save a 100gb file on your 2gb spare space. This means it is possible for users to download very large files

The second is what tom alluded to - the default (no permission), i.e we would allow much larger files than the smaller drives would allow

My worry is how will the browser behave if the user's free disk space is very low, say 100MB, but we report there is more space available?

So I think the answer is to report as small a size as practical to limit the exposure of those under that limit (hence I said 5GB)

I presume the website will be able to detect this somehow

I personally don't think scripts are going to try and fill or test your HDD space, it's super unfriendly and probably costly

I think 10 GB would also be fine. A reported 5 GB storage estimate equals 25 GB of available disk space because the group limit is capped to 20% of the temporary storage.

I don't have a preference - the higher value means less breakage , the lower value means less possible exposure .. I think

equals 25 GB of available disk space because the group limit is capped

I don't follow, what is a "group limit"? If we normally report 10GB and now report 5GB, or 20GB or whatever, that doesn't change any underlying hardware - we can report what we like - we're breaking the spec/code, not abiding by it

Storage estimate for temporary storage doesn't reflect the available disk space but rather the group limit which is capped by 10 GB or 20% of the temporary storage, whichever is lower.

IDK where we did that

anyone? @pierov

I don't know, either. Isn't it a Firefox thing?

Or, if it's a preference thing, it must have a non-easy to guess name .

we wouldn't have to worry about upstream (or waiting for them) or rebasing patches

+1 for the pref, if possible.

dom.quotaManager.temporaryStorage.fixedLimit is in kB, judging from the code (it's a signed 32-bit integer, by default -1 because any negative value is ignored).

I don't follow, what is a "group limit"?

eTLD+1, see https://developer.mozilla.org/en-US/docs/Web/API/Storage_API/Storage_quotas_and_eviction_criteria#firefox.

But it doesn't apply to persistent mode, only to the best-effort mode:

Origins for which persistent storage has been granted can store up to 50% of the total disk size, capped at 8 TiB, and are not subject to the eTLD+1 group limit.

It also says:

Note that it might not actually be possible for the origin to reach its quota because it is calculated based on the hard drive total size, not the currently available disk space. This is done for security reasons, to avoid fingerprinting.

Since Firefox already says doesn't report the actual size for fingerprinting reasons, sites should expect failures and we could always report a certain big size (50GB, 100GB, 200GB, whatever, even the 8TiB cap) and call it a day. I don't know how to properly choose a number. The only survey I can come up with is Steam's, which is probably biased. Maybe Moz has some telemetry data (ew!) we can have a look at. I know people constantly having a handful of free GBs on a 500GB drive, we could mask all our users as them .

Having a user accepting to download a big file just to see when it fails is an expensive way to fingerprint, I'd say. On the other hand, having the user happily download a big file over Tor (not possible, right now) is a UX win.

Of course, this holds if the documentation is correct. I haven't checked the code. But it seems to be, at least for me (it reports about 425-430GB for me, and my profile is in a 875GB partition, with the usual caveat of the space only for root on Linux).

locked pref because we love locked prefs. (lock those prefs!!)

In Firefox, the maximum storage space an origin can use in best-effort mode is whichever is the smaller of:

10% of the total disk size where the profile of the user is stored.

Or 10 GiB, which is the group limit that Firefox applies to all origins that are part of the same eTLD+1 domain.

Not great for Tails and live systems in general. We should test also there. 10% is easily less than 2GB.

Tails is already fingerprintable because of uB0 enabled by default (more or less, my guess is that many TBB users install uB0, maybe it's actually time we ship it officially), but having buckets based on the memory size wouldn't be great. I hope this is already covered.

Finally, removing Uplift, as we don't even have a patch for us, yet. I'm also doubtful of Bug, it seems more like an enhancement, or a Feature, since we currently stop the permission request by default.

but having buckets based on the memory size wouldn't be great. I hope this is already covered.

8GB VM:

[ click ] storage manager	0 of 416260915 bytes
[ i ]  storage quota	0.3GB [416260915 bytes]

16GB VM:

[ click ] storage manager	0 of 838415974 bytes
[ i ]  storage quota	0.7GB [838415974 bytes]

Funk 11GB VM:

[ click ] storage manager	0 of 574176051 bytes
[ i ]  storage quota	0.5GB [574176051 bytes]

Yikes.

Changing my idea about the Bug label: it definitely is one, and probably we'll need to Backport.

Oh, wait: min(10GB, 0.1 * totalSpace) means that all users that have less than 100GB can be fingerprinted.

Many, many VM users, very likely! Possibly including Qubes users, too. And that'd be scarier, since VM image sizes are less standardized than memory amounts (in physical machines).

EDIT: as expected, my Windows VM with 99.3GB of storage:

[ click ] storage manager	0 of 10671495987 bytes
[ i ]  storage quota	9.9GB [10671495987 bytes]

One of my Linux VMs, with 105087164416 bytes (105GB, 97.87GiB):

[ click ] storage manager	0 of 10508716441 bytes
[ i ]  storage quota	9.7GB [10508716441 bytes]

(what's the 5B difference? ).

Anyway, not great, and not fixable through preferences.

EDIT 2: oh dear, this is a catastrophe for Android. Low-end and old models with less than 100GB are a lot.

And Arkenfox warned us.

Sorry @thorin for not jumping earlier on this.

I have more, but not ticketed ... and at least as bad, and the solution is not 100% but does improve things - I mentioned it 3 years ago upstream but nothing happened

Finally, removing Uplift , as we don't ...

there is action happening upstream ... catch it and give it some more momentum

Okay, thanks again, I wasn't on CC on the upstream Bug. (The label for these cases is Upstream, not Uplift ).

As I've written there, I don't think a precise measure could benefit any users, only the "b'stards", as you call them. So, even users without the cap should have their size rounded, IMHO (then, of course, RFP users normalized to only one).

Would if (StaticPrefs::privacy_resistFingerprinting()) at ActorsParent.cpp#L2680 solve this issue?

looks right to me - see my earlier comment #41065 (comment 2822483)

Not sure how to phrase the RFP check, as it's being plumbed for fine grained control upstream, and we also want to check what to do when a user grants persistent-storage permission for FF and maybe Privacy Browser users - currently TB doesn't even prompt so not sure where/when that patch was done

side note: in FF at least, if you grant then revoke the permission, the permitted value is still returned for the duration of the session (I think there's a bugzilla on that)

mentioned in merge request !690 (closed)

removed Uplift label

added Upstream label

changed title from harden ESR102+ storage estimate with RFP upstream [upstream 1781277] to navigator.storage "best effort" + "persistent" leak partitionSize/totalSpace entropy

mentioned in issue #42235

mentioned in issue #42289 (closed)

@fkilic this might also be an interesting one for you.

As per our IRC conversation, upstream would prefer to close it as a wontfix:

2025-01-09 16:49:08	PieroV	fkilic[m]: since you're here :) Does https://phabricator.services.mozilla.com/D223842 have any problem?
2025-01-09 16:49:31	PieroV	I see it's missing an approval from tjr, but I can't find comments about why
2025-01-09 16:49:36	PieroV	But maybe you remember something about it
2025-01-09 16:50:45	PieroV	The comment says it's for nsRFPService::GetTemporaryStorageLimit()
2025-01-09 16:56:42	fkilic[m]	we talked with tjr about that patch today actually. we don't really like the current patch. I mean if tor browser is okay with this approach it is fine, but we would like to just lie to get storage estimate call instead of limiting the whole storage to 50gbs, but that didn't get approval from storage folks. In its current state, it would also limit persistent storage (the one that requires permission grant) to 50gbs which is why we
2025-01-09 16:56:42	fkilic[m]	dont want it. Tom suggested I write up a patch on ESR branch and just lie to getStorageEstimate calls for Tor and close this bugzilla issue as wontfix
2025-01-09 16:58:30	PieroV	fkilic[m]: we don't really care of persistent storage, as it's only non-PBM and we don't support it
2025-01-09 16:59:29	PieroV	Too bad Firefox won't have a protection for this though :(
2025-01-09 17:00:02	fkilic[m]	I mean then I guess tor browser can just set dom.quotaManager.temporaryStorage.fixedLimit and lock it?
2025-01-09 17:00:19	fkilic[m]	PieroV: maybe one day lol
2025-01-09 17:03:02	PieroV	fkilic[m]: so, my idea is that at a certain point we could just take https://phabricator.services.mozilla.com/D223842 as a downstream patch
2025-01-09 17:05:53	PieroV	Or yep, use fixedLimit and lock it as you suggested 
2025-01-09 17:08:29	fkilic[m]	yeah since it is going to be only a tor only patch anyway, then dom.quotaManager.temporaryStorage.fixedLimit should be easier (and probably less complex)
2025-01-09 17:16:24	PieroV	Either ways, I need to check it's actually okay also with the temporary storage
2025-01-09 17:16:34	PieroV	Permanent storage isn't a problem because we don't have a notification
2025-01-09 17:16:40	PieroV	But we need to hardcode also the other one
2025-01-09 17:16:47	PieroV	For Tails
2025-01-09 17:22:07	fkilic[m]	> But we need to hardcode also the other one
2025-01-09 17:22:07	fkilic[m]	not sure what you mean by this. but also just to let you know, persistent/temporary storage is weirdly named. within firefox core, I think (? not fully sure) any storage available to websites (be it localstorage, indexeddb etc.) is called temporary **even if the page has persistent storage permission**. It is temporary but with higher limits. the permanent storage within firefox core I think refers to internal/browser data and not
2025-01-09 17:22:07	fkilic[m]	available to websites directly. so for fingerprinting purposes, I think all that matters is temporary storage limit (which is also used as persistent storage limit to websites)
2025-01-09 17:25:10	PieroV	fkilic[m]: `navigator.storage.estimate()` is also problematic for us
2025-01-09 17:25:25	PieroV	As one of our use cases (Tails) runs as a live system
2025-01-09 17:25:38	PieroV	So, it has RAM as a storage, which usually is much less than 100GB
2025-01-09 17:25:49	PieroV	Therefore, the 10% rule apply
2025-01-09 17:26:26	PieroV	However, it seems that the pref works also for this use case
2025-01-09 17:30:15	fkilic[m]	yes correct it should work for it too. I assume you wouldn't have any data about tails users right? Like what is a safe number for Tor or tails users? 8GB * 0.1?
2025-01-09 17:35:44	PieroV	I think 1GB would be fine
2025-01-09 17:40:17	PieroV	Okay, so it seems we take the minimum between the pref / 5 and 10GB. So, if we want it to be 1GB, we need to do 5GB in KiB
2025-01-09 17:40:19	PieroV	GiB
2025-01-09 17:40:33	PieroV	So 5242880

Also, no need to lock, if I understand things correctly (nothing else will override it).

Notice that we take for granted we keep the current behavior (automatically reject the notification as a consequence of being in PBM).

If this is a problem for the future, we might implement a notification with a user choice for bucketed sizes (e.g., start with a kinda low minimum, like 50GB, and let users choose a bigger maximum for big downloads).

However, I don't think UX would be happy with this solution .