www.torproject.org https cert is failing, because pinned to old key

Starting an hour or so ago, our main website (www.torproject.org) now fails to load, with the message

An error occurred during a connection to www.torproject.org. The server uses key pinning (HPKP) but no trusted certificate chain could be constructed that matches the pinset. Key pinning violations cannot be overridden.

Error code: MOZILLA_PKIX_ERROR_KEY_PINNING_FAILURE

The new cert appears to be Valid Not Before Mon, 08 Jul 2024 00:33:24 GMT, i.e. we just got it.

The theory is that we are pinned to some earlier LE cert that is no longer used in today's chain.

Some more hints:

<PieroV> issuer=C=US, O=Let's Encrypt, CN=R11
<PieroV> It seems something not on the hardcoded list
<Peng> Let's Encrypt started using new intermediate certificates June 6 or so
<Peng> Same root certificates, new intermediates

PieroV further points to https://searchfox.org/mozilla-central/source/security/manager/ssl/StaticHPKPins.h#324-334 in that we prepared for this situation and pinned a backup cert too.

Suggested short term workaround is to move back to yesterday's cert, because it should still be valid for some weeks, and then you can breathe more easily while figuring out what the new cert ought to actually be.