intl.accept_languages gets into a stuck modifed state

added Fingerprinting FF115-esr labels

changed title from language change doesn't properly set languages to language change doesn't properly set languages or locale

changed the description

OK, the UI is a red herring ... in my TB-alpha-lang [1], where I have been testing all the languages, I have every single language in the UI - so as long as we are controlling navigator.language and navigator.languages and the HTTP header then the UI doesn't really matter?

[1] I run a separate TB-alpha always set to en-US which is the one I used in OP

pic

changed title from language change doesn't properly set languages or locale to language change doesn't always properly set languages and/or locale

added UX label

removed UX label

added Bug label

@thorin I'm not sure I'm following.

Do you have clear STR?

I've tried to add Italian, and I had it-IT added to the accepted languages with spoofing English off.

Then I've added Spanish, but it wasn't added again to the list (the behavior I didn't like of 102).

I'm very confused, and I don't think I got STR right.

But I think the easiest solution would be to dump the locale list (get that button out of about:preferences), and:

if you have spoof English, do everything in en-US
if you don't have spoof English, just use the first language you have

We could hack the preferences page to do so and automatically reset the users to this behavior once.

Then, if they preferred the old one (why?) and they mess up with about:config, let them be, and fix it the next time they change the language with the preferences.

The alternative is to force the list for accepted locales at each restart, but I wonder if forcing something eventually ever works.

added UX label

added Desktop label

I can confirm that the remove button stays disabled.

The logic to allow removing a language is:

!packagedLocales.has(code)

But this hasn't changed from 12.5.1, you can only add the packaged locales, but never remove them. The delete button in the UI language is inconsistent, but with the appropriate order unpleasant surprises can be avoided, so we don't have to care about this and let upstream solve it or discuss it on Bugzilla anyway (for completeness, Firefox has a property for last fallback and it's always en-US: we could check that instead of packagedLocales.has).

Do we have an issue about removing the choose the languages for display already? If we do, I'll remove UX from this issue, and keep it for other nonsense we might find here instead, even though I think it would be the ultimate solution to these problems.

Well, I think add alternatives could be removed - it's basically a duplicate of the dropdown, no?

That might sound confusing, but it might help with incomplete translations.

E.g., you know French or Spanish better than English, you can set your mother language as first, French/Spanish as second and English as third.

As long as this doesn't influence anything noticeable (accepted-language, for example) we're fine with it.

but adding and changing anything but the top item doesn't change our fingerprint - see my massive pic of all languages added in comment1

Yes.

And it's good. We don't want to expose users' settings for the UI. So, not modifying the web page languages if you change a language except the first one works for me.

Letting the first language change the fingerprint is also okay, because it happens only when you don't enable spoof English. It's what a user wants, eventually.

Not being able to remove a UI language kinda sucks, but it isn't a big deal either. Upstream might be doing something, because I can't choose my language at all in Nightly. If they aren't changing anything, we can move to Bugzilla, solve it upstream and then port downstream.

I don't follow. The add alternatives in TB only lets you add languages that we cover (38 of them). And it doesn't (bugs, if any, aside) so anything more than using the combobox next to add alternatives

wait ... the web content one is below that Choose your preferred language for displaying pages - my bad - I mixed app vs content - so the add alternatives is for the app .. got it (still don't understand why anyone would want mixed languages in the app - is this really a thing, I guess so, FF has it)

still doesn't explain the pic in OP

Letting the first language change the fingerprint is also okay, because it happens only when you don't enable spoof English. It's what a user wants, eventually.

noooo. that's just crazy talk ... users shouldn't be allowed to change anything

Agreed. In the application's add alternatives, sure, if the first one is Spanish you would probably want the app to match (except where they opted into spoof english). This is what threw me, I was using the app change to change the prefers languages (because I do not ever use spoof english)

that also explains the mixed languages in the UI in one of my other issues: re burmese

still don't understand why anyone would want mixed languages in the app - is this really a thing, I guess so, FF has it

In case of incomplete languages:

E.g., you know French or Spanish better than English, you can set your mother language as first, French/Spanish as second and English as third.

By re-reading this I admit I wasn't clear enough.

I was thinking to the case of countries that were French colonies at a certain point of their history, so, even though they have a first language that isn't French now, French is still the second language (I expect to be more than one).

I don't know if we support any in Tor Browser, but I guess Firefox does.

changed title from language change doesn't always properly set languages and/or locale to app language change somehow changed webcontent locale ???

removed UX label

added Needs Information label

removed Bug label

I think the only way to change fingerprint is the second button, the one for the web languages.

Maybe you got confused. It's a known bad UX after all.

If we don't have STR I'd close this issue, and reopen if we have them.

I got confused because (yes the UX is bad), but mainly because I have trained myself (because spoof english is always off) to use the app language UI to flip content languages - i.e to reset them as per a new install for testing

still miffed I can't repro - it changed locale but not language (100% sure I was on en-US and the restart and apply was because it was asking to be set to danish), so weird

OTOH, I couldn't find the issue about deleting the confusing and risky button. I found #41339 (closed) and I made it a Meta, because I think it hasn't an action to do, but many interesting information.

I think the only way to change fingerprint is the second button, the one for the web languages

you mean change the FP from expected (all 38 of them), because clearly I can flip content language (excluding situation in op) via app language when spoof english is off. But I know what you mean ...

that second button is a nightmare - we have tickets for tightening all this up

ooh you found a ticket I mentioned

changed title from app language change somehow changed webcontent locale ??? to app language change (no spoof english) missed changing content languages but got locale

This just KEEPS happening, but every time I try to repeat it with some reliable steps, I waste an hour with no result

I suggest, if you want to reproduce, just do what I'm doing (with spoof english prompt already set as no, and no use_english pref either)

changing app language, apply and restart, test
rinse, repeat

support

config (I have say no and this has been set for ages)

fubar

added 13.0 stable label

added Bug label

removed 13.0 stable label

added 13.5 stable label

removed Needs Information label

added Roadmap::Future label

removed FF115-esr label

mentioned in issue #42084 (closed)

@thorin since you managed to consistently reproduce it, could you please verify with 13.0a4 or later or a nightly if the problem has gone with #42084 (closed) solved?

mentioned in issue #42289 (closed)

changed title from app language change (no spoof english) missed changing content languages but got locale to intl.accept_languages gets into a stuck modifed state

OK, I have narrowed this down

TEST: https://arkenfox.github.io/TZP/tzp.html#region

My OS is effectively english, my new pristine alpha13.5a2 when it created a new profile auto chose english and I left it at that - never prompted to spoof english (which is a red herring)

app lang = en-US
intl.accept_languages = en-US, en (not modified)
languages = en-US, en + locale = en-US =

change app lang to e.g. french + apply and restart

app lang = french
test: when asked on first HTTP visit say Non
- javascript.use_us_english_locale = hidden (reset)
- privacy.spoof_english = 1
intl.accept_languages = fr, fr-fr, en-us, en
languages = fr, fr-fr, en-us, en + locale = fr =

check request english for websites and click OK

note: whilst the dialog is open
- the accept languages list does not change
- if checked the list and buttons will be disabled (greyed out)
app lang = fr (no change)
don't restart (you are not prompted to, not that it matters)
javascript.use_us_english_locale = true
privacy.spoof_english = 2
intl.accept_languages = en-US, en (modified)
- feel free to reopen the dialog and see it has updated
languages = en-US, en + locale = en-US =

uncheck request english for websites and click OK

app lang = fr (no change)
don't restart (you are not prompted to, not that it matters)
javascript.use_us_english_locale = hidden (reset)
privacy.spoof_english = 1
intl.accept_languages = en-US, en (modified)
- feel free to reopen the dialog and see it hasn't updated
languages = en-US, en + locale = fr =

and repeat yourself for any app language and intl.accept_languages will never change and thus becomes STUCK and will be at odds with the locale should that not be en-US

to unstick reset intl.accept_languages

you don't even need to test anything - just either check the intl.accept_languages pref value or reopen the accept languages dialog

@pierov

super reduced STR - modify intl.accept_languages and now try and change app lang + locale to match

here I was using german, went into languages and changed the order - this had the effect of making the pref modified - then I switched app lang to french (and restarted) and accept languages failed to follow suit

I was starting to think it might be related to english only .. but nope .. so even more

even more

Come on baby, light my fire
Try to set the night on fire, yeah

Took me a while, but I think I got my ideas sorted.

Maybe you already knew about this pref, but I didn't so let me just write something for anyone that is in my same condition .

The truth behind `intl.accept_languages`

The main problem is that intl.accept_languages isn't a normal pref, it's "complex value", and its actual value is set by reading chrome://global/locale/intl.properties.

chrome://.../locale/something.properties are special URLs that automatically lead to the translated versions of the various .properties file. It's the magic behind the creation of string bundles.

However, about:config will just display it as a normal string.

If you check modules/libpref/init/all.js, at least another 3 prefs behave like it:

intl.menuitems.alwaysappendaccesskeys
intl.menuitems.insertseparatorbeforeaccesskeys
font.language.group

And if you extend the search to the whole codebase, you also find pluralRule in intl/locale/PluralForm.sys.mjs.

For it-IT, for example, these are exactly the prefs that are in that file.

Upstream (not RFP)

I think this is a feature for Firefox, not a bug . When you set a display language, you also want everything to be in that language.

If you don't care of leaking data (again, no RFP), losing users' prefs is worse than forcing the correct value.

If I understand the code correctly, intl.accept_languages is never reset!

When a user changes language, if their pref is still the default, it'll automatically load the values to use from the new localized file. If they changed intl.accept_languages, they just keep their changed value.

It's somehow an elegant way to handle a difficult problem.

RFP Firefox

Upstream

There's a very easy approach: if RFP is on, reset also intl.accept_languages whenever the language is changed (unless you have spoof English).

I think we could easily open a Bug to ask for that. Then we could cherry-pick it back to our codebase if upstream is happy with that, or just patch our browsers.

We also have #42084 (closed) (!767 (merged)) to possibly upstream that is also dealing with intl.accept_languages (/cc @ma1).

Bonus for us

We should keep only the spoof English checkbox, without the actual form to change the languages.

An even cleaner solution would be to remove the modal, and replace the button to open it with the checkbox.

i knew the pref was "special" but had no idea what the sauce was. When spoof english is enabled, the pref also cannot be reset !!

re remove dialog - trying to find where this was already suggested (by me of course) - where we talked about the UI being a bit of a mess, etc - and I mentioned (in many places) that users can alter their language order (we have a ticket somewhere about a user changing from en-US, en to en, en-US) and that we should "lock it down" - not the pref, but harden it. Not only the order but they can add/remove languages and of course this also affects the passive header FP. I think I suggested resetting values

So I think we mentioned that removing the dialog and cleaning up the language UI section

LANGUAGE

Choose your preferred language for displaying pages and used to display menus, messages, and notifications from Tor Browser
[ English (en-US) ]

[ ] Request English versions of web pages for enhanced privacy (display if not en-US)

Choose additional languages for displaying pages and used to display menus, messages, and notifications from Tor Browser
[ Set Alternatives ]

this needs work

whatever we do, we will need a watershed to reset users - i.e if not spoofing english, reset - or we could just add that to startup somewhere forever - users can still edit the pref

Generally speaking, we don't fight against users that want to restore prefs. We run the migration run and then never again.

If we're fine with that, we can just do it once in the migration function, without a watershed (_migrateUITBB, in browser/components/BrowserGlue.sys.mjs).

Otherwise, we can just add a patch to continuously do it at startup. The patch ma1 created a while ago works in a similar way, IIRC.

https://bugzilla.mozilla.org/show_bug.cgi?id=1869821 - that's going to idle for years, it needs MOAR cowbell (and peepers on it)

If you think about it, we're in a kind of special situation: while we use Firefox's multi-lingual utilities, upstream ships packages in a lot of languages.

So, users can switch only between the downloaded lang and English, unless they download additional languages.

If you have to add RFP to that, well, it makes quite low priority for upstream for sure. Also, the line between a defect and enhancement is blurry, and I felt I couldn't make this into a defect.

So, I think we should tackle it downstream.

Clearing any user pref when changing language is easy. Hiding the modal and getting the checkbox out is also easy, but user-facing, so I need UX approval (but we don't have UX time until January ). We need more UX people! One UX peer for each dev .

Well currently upstream has no protection with RFP (well, prior it only protected en-US) - javascript.use_us_english_locale pref has been removed so spoof_english won't do much (probably protects some app messages, e.g. xml) - so locale is not protected (to match language) - e.g. see my case where I use en-US app language, and accept languages, but all my resolved options are en-mycountry because en-mycountry is a "variant/derivative" of en

IDK what or when they plan to tackle this - something about JS realms and using one locale per set (e.g. en-US for all on en-CA, en-GB, .. there's 100+ en* locales, 60+ es* ones etc and you get what your OS is)

I've also noticed during my testing, that when I am spoofing (and changing app language + restart as a quick way to always spoof but flip app languages) .. when I select en-US, the spood english checkbox stays checked, so (and I would need to recheck I suppose .. so much going on here) the pref is not reset which could cause issues/differences with spoof_english patches

this may or may not be a symptom of the larger issue of a stuck intl pref state but is a footgun IMO

class, discuss!

mentioned in issue #42326

assigned to @thorin

marked this issue as related to #42327

assigned to @pierov and unassigned @thorin

mentioned in issue #42349 (closed)

A few days ago I was about to fix this, but then I saw an upstream comment that says this is done on purpose.

Personally, I think this is a bad decision. So, should we yolo it and reset the pref?

FF59: I think it has something to do with https://bugzilla.mozilla.org/show_bug.cgi?id=1039069#c31

listen to the pref change event ...

FF61: Ahhh found it using one of my secret lists - https://bugzilla.mozilla.org/show_bug.cgi?id=1447592 : Don't reset privacy.spoof_english when privacy.resistFingerprinting is flipped back to false

OK, so that doesn't really makes sense (at least not today). 0 = prompt, 1 = not on, 2 = on. So if we reset to 1 you won't get prompts anyway (or that's what the patch fixed? IDK and IDFC)

Anywhoooo ... I think that was a bad decision as well. And we should upstream and fix it. Note: upstream spoof english is a mess because use_english was removed as a pref (which I think is correct, the key is to set the locale) and setting en-US as locale (all the resolved options in Intl) is not handled on non en-US locales that are still en* (e.g. en-GB, en-CA, and a hundred others). Haven't tried with a VM setup in e.g. ja, de etc but I think it works there.

But of course, RFP is not meant to be flipped on and off all the time, and at TB we lock it. Do spoof english patches rely on RFP being on or just spoof english being on (this might help upstream)?

So I'm thinking of upstream

user turns on RFP
user spoofs english
user disables RFP - what happens?
- quite frankly this should (if spoof is 2) reset to 1 AND ALSO reset their languages
- ^ so yeah, I don't have any problems with that
user with RFP disables spoof
- reset to 1 AND ALSO reset their languages

edit: so disable spoof, disable RFP should both do the same thing (sorry forgot to flesh out all the options/matrix)

Let's do it

Class, discuss!

Do spoof english patches rely on RFP being on or just spoof english being on (this might help upstream)?

If I understand where Mozilla is going, Spoof English theoretically should not be accessed directly, but through RFPTarget::JSLocale, which will then check the preference only when RFP is active.

But it seems that the pref is still checked, and sometimes without checking if l10n is allowed in the document:

But I think it doesn't make sense to use spoof English without RFP. I think the pref should be used only for UI purposes.

user disables RFP - what happens?

We stop listening to the changes of the spoof English preference. The user stay with what they have in intl.accept_languages (en-US if they had spoof English). I don't care of this.

Finally, I'd be up for resetting intl.accept_languages also every time you change your language, if you don't have spoof English.

So, quick question (I could probably figure out by myself, but I prefer keeping it here to show I thought of it ).

Does intl.accept_languages influence the formatting API?

In case, having it to en-US but then getting your actual locale in the formatting API is more entropy than resetting intl.accept_languages first...

I don't care of this

Oh yeah, upstream will use RFPTargets (which is not RFP) to toggle per domain - that's an upstream problem - it's such a mess. I guess shouldRFP or whatever it is doesn't care what turned it on (RFP or +ALLTargets) but I'm sure they could engineer a check

So for upstreaming our change IDK if/how we can do that to make it acceptable - but for us it seems fairly straightforward

resetting intl.accept_languages also every time you change your language, if you don't have spoof English

I said this somewhere about hardening JS languages (probably #41339 (closed)) where we need to protect against the user not only adding additional languages, but also the order (one issue here in the past showed a user with en, en-US instead of en-US, en). So:

reset it all the time
remove the Choose your preferred language for displaying pages>Choose dialog
- move the spoof english checkbox into the main settings
- we have spoken about this before: time to do it: this should also close #41339 (closed)

So, quick question ... Does intl.accept_languages influence the formatting API

So AFAICT, in the past yes/maybe. We used to have a javascript.use_us_english_locale pref but it's redundant (and deprecated next ESR). And in the past we maybe had issues with to*String functions - but today this is all unified under the hood with Intl. And we control Intl.*.resolvedOptions via setting the locale. We should probably check with Zibi Braniecki (and/or André Bargull)

So using this bug to get a different locale vs language - the locale changed every metric concerned - cogito ergo sum, the language has no bearing (although the TZP tests are not 100% finished, they do touch on most of it including to*String)

just to be clear, in that diff above, a bunch of info is hidden under a locale hash (e.g. intl locale, toString, resolvedOptions): here's what those three hashes look like in detail. And all those datetime+formatting ones will end up the same in future. Combined they all create the max entropy you can get out all Intl (basically you can determine the language without asking for it - great for detecting if anything is affected, e.g. OS formatting leaking in etc)

so, most individual metrics change, overall they change, and look at the previous pic, they all have green ticks for intl and/or locale which means they are all matching fr (or en-US) vs undefined = the locale is fully controlling all of these

so I think I've answered this question, but if you want I can check with Zibi

click me for more insight and if you want to learn

reset it all the time

After #42084 (closed) we ~~reset~~ set intl.accept_languages en-US at every startup when we have spoof English. This should help the spoof English case.

So, I agree resetting intl.accept_languages when spoof English changes is the (first) missing piece here, which is what you meant if I understand correctly.

Unless...

remove the Choose your preferred language for displaying pages>Choose dialog

move the spoof english checkbox into the main settings

we have spoken about this before: time to do it: this should also close #41339 (closed)

I think that before doing this, we'll have to reset intl.accept_languages also when a user changes their language.

Was this included in your "reset it all the time"?

Or in general was it: at the startup, every time RFP is on, either set it to en-US, en (what we're doing) + clear any user customization?

So, quick question ... Does intl.accept_languages influence the formatting API

So AFAICT, in the past yes/maybe. We used to have a javascript.use_us_english_locale pref but it's redundant (and deprecated next ESR). And in the past we maybe had issues with to*String functions - but today this is all unified under the hood with Intl. And we control Intl.*.resolvedOptions via setting the locale. We should probably check with Zibi Braniecki (and/or André Bargull)

I think we could make this second part a 128 problem at this point, since we'll have to deal with javascript.use_us_english_locale anyway.

Was this included in your "reset it all the time"?

probably, I just assume it's covered by the listener? But yeah, we want robustness. If they change it via about:config that's on them (not sure if the listener will catch this) - and we plan to make about:config scary'ish and enforce the warning each time - so out of scope

so if that's kinda out of scope, then only the UI is left, so sure .. I included that, so I am smart

mentioned in issue #42552 (closed)

@pierov is there anything we should+can do for 13.5 here?

Good question, I forgot about this issue.

I think for 13.5 we could reset intl.accept_languages every time you disable spoof English, and do more (potentially remove parts of the UI) with 14.0, with more time to test, since we'll have to deal with l10n protections anyway.

wfm

added Needs Information label

removed Roadmap::Future label

added Next label

added 14.0 stable label

added Doing label and removed Next label

mentioned in issue mullvad-browser#240 (closed)

mentioned in merge request !1016 (merged)

removed 14.0 stable label

Gaudeamus, we've killed the accepted languages UI!

closed

removed Doing label

intl.accept_languages gets into a stuck modifed state

Designs

Child items ...

Activity

The truth behind `intl.accept_languages`

Upstream (not RFP)

RFP Firefox

Upstream

Bonus for us

intl.accept_languages gets into a stuck modifed state

Activity

The truth behind intl.accept_languages

Upstream (not RFP)

RFP Firefox

Upstream

Bonus for us

The truth behind `intl.accept_languages`