Bug 1749522 - When plain text encoding speculation fails, restart the plaintext mode of the tokenizer. r=smaug a=dmeehan

Differential Revision: https://phabricator.services.mozilla.com/D135830

Differential Revision: https://phabricator.services.mozilla.com/D135096

Differential Revision: https://phabricator.services.mozilla.com/D134951

To avoid copying in the common case, expat points directly into the
(internal) input buffer for strings referenced from the tag stack. When
expat unwinds back to the caller, it attempts to call storeRawNames() to
walk the tag stack and copy any such strings into persistent memory
before the caller potentially invokes XML_Parse() again and shuffles the
input buffer, thereby invalidating these references.

Unfortunately, it doesn't do it in all the right places. Because
different parsing states set |processor| to different callbacks (so that
parsing can resume in the right context), there are a number of
non-obvious entry points. In this case, the input stream was chunked so
that parsing paused in middle of processing an internal entity [1], so
|processor| was set to internalEntityProcessor(), which invokes
doContent() but does not call storeRawNames(). The doContent() call then
parsed some nested tags from the entity. Tags within an entity are
generally required to be balanced, but a host callback returned an error
code to interrupt parsing midway through. This caused Expat to return to
the caller with a tag stack still referencing the input buffer, which
got clobbered in the next call to XML_Parse, causing a tag mismatch on
the next close tag.

Conceptually, this optimization should be managed by doContent(), and I
believe the only reason that isn't the case is that doContent() has so
many return paths (and we don't have RAII in C). We can fix this by
wrapping doContent() in a helper.

[1] &certerror.expiredCert.whatCanYouDoAboutIt2;

Differential Revision: https://phabricator.services.mozilla.com/D134878

Differential Revision: https://phabricator.services.mozilla.com/D134506

Differential Revision: https://phabricator.services.mozilla.com/D134506

Differential Revision: https://phabricator.services.mozilla.com/D134653

Differential Revision: https://phabricator.services.mozilla.com/D134670

Bug 1747514 - Ensure the expat sandbox is large enough to hold the base URI. r=shravanrn,deian, a=dsmith

Differential Revision: https://phabricator.services.mozilla.com/D134653

Differential Revision: https://phabricator.services.mozilla.com/D134670

Differential Revision: https://phabricator.services.mozilla.com/D134415

Differential Revision: https://phabricator.services.mozilla.com/D133996

Differential Revision: https://phabricator.services.mozilla.com/D134320

Differential Revision: https://phabricator.services.mozilla.com/D134021

Differential Revision: https://phabricator.services.mozilla.com/D134142

Differential Revision: https://phabricator.services.mozilla.com/D134135

Bug 1736248 - Update the charset source if non-ASCII is seen after the first detector guess but the encoding does not change. r=smaug

Differential Revision: https://phabricator.services.mozilla.com/D133731

Bug 1741665 - Align nsCString's public size_type better with other C++ APIs, r=mccr8,geckoview-reviewers,agi

Differential Revision: https://phabricator.services.mozilla.com/D131422

Differential Revision: https://phabricator.services.mozilla.com/D134320

Bug 1745139 - Check for termination even after the first flush loop in CommitToInternalEncoding. r=smaug

Differential Revision: https://phabricator.services.mozilla.com/D133342

Depends on D133009

Differential Revision: https://phabricator.services.mozilla.com/D133158

Backed out 2 changesets (bug 1744460) for causing build bustages at RLBoxSandboxPool.cpp. CLOSED TREE

Backed out changeset 582101d582a0 (bug 1744460)
Backed out changeset dba7b7c19b2f (bug 1744460)

Differential Revision: https://phabricator.services.mozilla.com/D133158

Depends on D133009

Differential Revision: https://phabricator.services.mozilla.com/D133158

Implements https://github.com/whatwg/html/issues/6962 . Improves performance
when <meta charset> occurs in head but after the first kilobyte and aligns
behavior better with WebKit and Blink.

The main change is to avoid reloads when meta appears within head but
after the first kilobyte. Prior to this change, Gecko reloaded in that
case (in compliance with the spec!) even though WebKit and Blink did not.

Differences from WebKit and Blink:

* WebKit and Blink honor <meta charset> in <noscript>. This implementation
  does not.
* WebKit and Blink look for meta as if the tree builder was unaware of
  foreign content. This implementation is foreign content-aware. This
  makes a difference for CDATA sections that contain a > before the meta
  as well as style and script elements within foreign content. This could
  happen if the CDATA section that has mysteriously been introduced around
  a what looks like a meta tag also contains another prior tag-looking
  run of text.
* This implementation processes rel=preload and speculative loads that are
  seen before <meta charset> has been seen. WebKit and Blink instead first
  look for the meta and rewind before starting speculative parsing.
* Unlike WebKit, if there is neither an honored meta nor syntax resembling
  an XML declaration, detection from content takes place (as in Blink).
* Unlike Blink, if there is neither an honored meta nor syntax resembling
  an XML declaration, the detection from content is not dependent of network
  buffer boundaries.
* Unlike Blink, detection from content can trigger a reload at the end of
  the stream if the guess made at that point differs from the first guess.
  (See below for the definition of the input to the first guess.)

Differences from the old spec and Gecko previously:

* Meta inside script and RCDATA elements is no longer honored.
* Late meta is now ignored and no longer triggers a reload.
* Later meta counts as early enough meta: In addition to the previous
  meta within the first 1024 bytes, now a meta that started within the first
  1024 bytes counts as early enough. Additionally, if by then there hasn't
  been a template start tag and head hasn't ended, meta occurring before the
  earlier of the end of the head or a template start tag counts as early
  enough.
* Meta now counts as not-late even if the encoding label has numeric
  character reference escapes.
* Syntax resembling an XML declaration longer than a kilobyte is honored if
  there is no honored meta.
* If there is neither an honored meta nor syntax resembling an XML declaration,
  the initial chardetng scan is potentially longer than before: the first 1024
  bytes, the token spanning the 1024-byte boundary if there is such a token,
  and, if by then head hasn't ended and there hasn't been a template start tag
  until the end of the template start tag or the end of the token that causes
  head to end, ever comes first. However, if the token implying the end of the
  head is a text token, bytes only to the end of the previous non-text token is
  considered. (This definition avoids depending on network buffer boundaries.)
* XML View Source now uses the code for syntax resembling an XML declaration
  instead of expat for extracting the internal encoding label.

Reftest are added as both WPT and Gecko reftests in order to test both http:
and file: URL scenarios. The Gecko tests retain the WPT <link> tags in order
to use the exact same bytes.

An encoding declaration has been added to a number of old tests that didn't
intend to test the new speculation behavior especially in the context of
https://bugzilla.mozilla.org/show_bug.cgi?id=1727750 .

Differential Revision: https://phabricator.services.mozilla.com/D125808

Backed out changeset 1778ca2ab291 (bug 1744425) for bc failures on browser_xpcom_graph_wait.js. CLOSED TREE

Bug 1723674 added a new nsID::GenerateUUID() static factory function to generate UUIDs without the overhead of querying and instantiating an nsIUUIDGenerator object. nsContentUtils::GenerateUUID() is a utility function that amortizes that overhead by holding an nsIUUIDGenerator singleton. That's no longer necessary because code that calls nsContentUtils::GenerateUUID() can now just call nsID::GenerateUUID(). No nsIUUDGenerator is needed.

Differential Revision: https://phabricator.services.mozilla.com/D132866

Backed out changeset 3dfd3c94a105 (bug 1701828) for causing mochitest failures on browser_hsts_host.js CLOSED TREE

Implements https://github.com/whatwg/html/issues/6962 . Improves performance
when <meta charset> occurs in head but after the first kilobyte and aligns
behavior better with WebKit and Blink.

The main change is to avoid reloads when meta appears within head but
after the first kilobyte. Prior to this change, Gecko reloaded in that
case (in compliance with the spec!) even though WebKit and Blink did not.

Differences from WebKit and Blink:

* WebKit and Blink honor <meta charset> in <noscript>. This implementation
  does not.
* WebKit and Blink look for meta as if the tree builder was unaware of
  foreign content. This implementation is foreign content-aware. This
  makes a difference for CDATA sections that contain a > before the meta
  as well as style and script elements within foreign content. This could
  happen if the CDATA section that has mysteriously been introduced around
  a what looks like a meta tag also contains another prior tag-looking
  run of text.
* This implementation processes rel=preload and speculative loads that are
  seen before <meta charset> has been seen. WebKit and Blink instead first
  look for the meta and rewind before starting speculative parsing.
* Unlike WebKit, if there is neither an honored meta nor syntax resembling
  an XML declaration, detection from content takes place (as in Blink).
* Unlike Blink, if there is neither an honored meta nor syntax resembling
  an XML declaration, the detection from content is not dependent of network
  buffer boundaries.
* Unlike Blink, detection from content can trigger a reload at the end of
  the stream if the guess made at that point differs from the first guess.
  (See below for the definition of the input to the first guess.)

Differences from the old spec and Gecko previously:

* Meta inside script and RCDATA elements is no longer honored.
* Late meta is now ignored and no longer triggers a reload.
* Later meta counts as early enough meta: In addition to the previous
  meta within the first 1024 bytes, now a meta that started within the first
  1024 bytes counts as early enough. Additionally, if by then there hasn't
  been a template start tag and head hasn't ended, meta occurring before the
  earlier of the end of the head or a template start tag counts as early
  enough.
* Meta now counts as not-late even if the encoding label has numeric
  character reference escapes.
* Syntax resembling an XML declaration longer than a kilobyte is honored if
  there is no honored meta.
* If there is neither an honored meta nor syntax resembling an XML declaration,
  the initial chardetng scan is potentially longer than before: the first 1024
  bytes, the token spanning the 1024-byte boundary if there is such a token,
  and, if by then head hasn't ended and there hasn't been a template start tag
  until the end of the template start tag or the end of the token that causes
  head to end, ever comes first. However, if the token implying the end of the
  head is a text token, bytes only to the end of the previous non-text token is
  considered. (This definition avoids depending on network buffer boundaries.)
* XML View Source now uses the code for syntax resembling an XML declaration
  instead of expat for extracting the internal encoding label.

Reftest are added as both WPT and Gecko reftests in order to test both http:
and file: URL scenarios. The Gecko tests retain the WPT <link> tags in order
to use the exact same bytes.

An encoding declaration has been added to a number of old tests that didn't
intend to test the new speculation behavior especially in the context of
https://bugzilla.mozilla.org/show_bug.cgi?id=1727750 .

Differential Revision: https://phabricator.services.mozilla.com/D125808

This patchs adds new error messages which are extending existing ones,
providing extra information to the user.
A webconsole mochitest is added in the following patch of this stack.

Differential Revision: https://phabricator.services.mozilla.com/D131889

-Wshadow warnings are not enabled globally, so these -Wno-shadow suppressions have no effect. I had intended to enable -Wshadow globally along with these suppressions in some directories (in bug 1272513), but that was blocked by other issues.

There are too many -Wshadow warnings (now over 2000) to realistically fix them all. We should remove all these unnecessary -Wno-shadow flags cluttering many moz.build files.

Differential Revision: https://phabricator.services.mozilla.com/D132289

Differential Revision: https://phabricator.services.mozilla.com/D126435

Backed out changeset d486edc7499b (bug 1732201) for causing web-platform-tests failures on header-totalsfntsize-001.xht CLOSED TREE

Differential Revision: https://phabricator.services.mozilla.com/D126435

Differential Revision: https://phabricator.services.mozilla.com/D132171

Differential Revision: https://phabricator.services.mozilla.com/D132113

Backed out changeset 524df7136a1f (bug 1742914) for causing assertion failures on htmlparser/nsExpatDriver.cpp. CLOSED TREE

Differential Revision: https://phabricator.services.mozilla.com/D132113

Differential Revision: https://phabricator.services.mozilla.com/D104658