Differential Revision: https://phabricator.services.mozilla.com/D136880

Currently, the bootstrap code from configure gets bootstrapped task info
from the taskgraph, and then invokes `mach artifact toolchain` which...
does the same again. That cumulatively wastes a noticeable amount of
time, especially on Windows.

So to avoid repeating this work that the configure side already did,
we do a little more on the configure side (resolution from index to
task-id), and just give a (task-id, artifact path) pair to `mach
artifact toolchain`. The added code to `mach artifact toolchain` is
actually ironically very similar to the code that was removed in
bug 1687594.

And now that mach bootstrap uses the configure code, it benefits from
this change as well.

Differential Revision: https://phabricator.services.mozilla.com/D135311

Depends on D135274

Differential Revision: https://phabricator.services.mozilla.com/D135275

The python-test infra runs each test file in a separate process for
parallelization.  This is expedient, but it means that when a test fails, the
output is often buried 1000s of lines in scrollback and difficult to find.

In the future, we could look into using 'pytest-async' rather than rolling our
own parallelization, this way Pytest would output the failures again at the end
of the run.

But for now, it's easier to simply defer outputting failed tests until the very
end. This is done in a try/finally block so even if there was a
KeyboardInterrupt or other exception, we'll still print the failed tests.

Differential Revision: https://phabricator.services.mozilla.com/D135274

MSYS2 has a slightly different directory structure (binaries are
now under "/usr/bin/" instead of "/bin/"), and we're now plonking
it in `%MOZILLABUILD%\msys2` instead of `%MOZILLABUILD%\msys` so that
MSYS1 files don't interfere with MSYS2 after a pave-over install.

Speaking of pave-over installs: if both `msys2` and `msys` are available,
this patch prefers `msys2`. This is because MozillaBuild installations
with MSYS2 are going to _assume_ they're using MSYS2, and therefore
be most compatible with the versions of packages shipped with MSYS2.

Differential Revision: https://phabricator.services.mozilla.com/D133549

Assuming that the `MOZILLABUILD` environment variable is set, allow
invoking Mach from non-MozillaBuild terminals.

Note that MozillaBuild still needs to be installed, and the
`MOZILLABUILD` environment variable will have to be set.

For future reference: when I tried setting this up with Windows
Store's Python 3.9, I encountered issues when running binaries installed
via `pip`: it would fail with `abort: failed to load Python DLL
python3x.dll`.

Differential Revision: https://phabricator.services.mozilla.com/D133936

As part of this, the shell-script part of `./mach` can be removed,
making it pure Python.

There's a change in `--profile-command` behaviour, though: it now only
profiles the specific command, rather than all of Mach.
This is because _so much of Mach_ has already been run before
CLI arguments are parsed in the Python process.

If a developer wants to profile Mach itself, they can manually run
`python3 -m cProfile -o <file> ./mach ...`

Differential Revision: https://phabricator.services.mozilla.com/D133928

Depends on D132592

Differential Revision: https://phabricator.services.mozilla.com/D132593

Bug 1748383 - In configure, make e.g. os.path.join mean the same thing whether importing os or not. r=firefox-build-system-reviewers,mhentges

Currently, using any of the functions defined in
`ConfigureSandbox.OS.path` in a `@depends` function will have a different
behavior if the function has an `@imports("os")` or not. In the former
case, we get plain `os.path.$function`, while in the latter we get the
function from `ConfigureSandbox.OS.path`, which handles path separators
differently, which makes a significant difference on Windows.

With this change, we now consistently use the versions from
`ConfigureSandbox.OS.path`.

Differential Revision: https://phabricator.services.mozilla.com/D135003

Backed out changeset a6eabd95e31d (bug 1740123)
Backed out changeset bfd5211060b4 (bug 1725895)

Bug 1748391 - Remove advertisement for the Visual Studio backend. r=firefox-build-system-reviewers,mhentges

The VS backend is always built when building on Windows for Windows, so
in practice, the message isn't printed... except when cross-compiling on
Windows, in which case the VS backend doesn't work.

Differential Revision: https://phabricator.services.mozilla.com/D135015

Bug 1647542 - Handle host libraries properly when cross-compiling on Windows. r=firefox-build-system-reviewers,mhentges

Differential Revision: https://phabricator.services.mozilla.com/D135013

Bug 1748383 - In configure, make e.g. os.path.join mean the same thing whether importing os or not. r=firefox-build-system-reviewers,mhentges

Currently, using any of the functions defined in
`ConfigureSandbox.OS.path` in a `@depends` function will have a different
behavior if the function has an `@imports("os")` or not. In the former
case, we get plain `os.path.$function`, while in the latter we get the
function from `ConfigureSandbox.OS.path`, which handles path separators
differently, which makes a significant difference on Windows.

With this change, we now consistently use the versions from
`ConfigureSandbox.OS.path`.

Differential Revision: https://phabricator.services.mozilla.com/D135003

Bug 1748385 - Use the --dont-define-prefix flag of pkg-config when supported. r=firefox-build-system-reviewers,mhentges

Differential Revision: https://phabricator.services.mozilla.com/D135012

Bug 1748385 - Add --shared flag when using pkgconf >= 1.7.4. r=firefox-build-system-reviewers,mhentges

Differential Revision: https://phabricator.services.mozilla.com/D135011

pkgconf is an alternative implementation of pkg-config that is more
cross-platform. It has also become the default on Fedora, so it's not
some random project.

Differential Revision: https://phabricator.services.mozilla.com/D135009

Backed out changeset bc23b3cfb583 (bug 1748385)
Backed out changeset 53f3c4e3e600 (bug 1748385)
Backed out changeset 5c744edba720 (bug 1748385)
Backed out changeset ad23d3af4df6 (bug 1748385)
Backed out changeset 54e3431d9152 (bug 513617)

Bug 1748385 - Use the --dont-define-prefix flag of pkg-config when supported. r=firefox-build-system-reviewers,mhentges

Differential Revision: https://phabricator.services.mozilla.com/D135012

Bug 1748385 - Add --shared flag when using pkgconf >= 1.7.4. r=firefox-build-system-reviewers,mhentges

Differential Revision: https://phabricator.services.mozilla.com/D135011

pkgconf is an alternative implementation of pkg-config that is more
cross-platform. It has also become the default on Fedora, so it's not
some random project.

Differential Revision: https://phabricator.services.mozilla.com/D135009

As part of this, the shell-script part of `./mach` can be removed,
making it pure Python.

There's a change in `--profile-command` behaviour, though: it now only
profiles the specific command, rather than all of Mach.
This is because _so much of Mach_ has already been run before
CLI arguments are parsed in the Python process.

If a developer wants to profile Mach itself, they can manually run
`python3 -m cProfile -o <file> ./mach ...`

Differential Revision: https://phabricator.services.mozilla.com/D133928

Depends on D129534

Differential Revision: https://phabricator.services.mozilla.com/D129535

Bug 1743688: Add a new type of source hosting for angle to look for the upstream branch on chrome beta r=jgilbert,jewilde

Depends on D129533

Differential Revision: https://phabricator.services.mozilla.com/D129534

It occurs to me that because tracking is a general vendoring need
(not just for updatebot) that it should live under vendoring.

Depends on D129437

Differential Revision: https://phabricator.services.mozilla.com/D129533

Depends on D129436

Differential Revision: https://phabricator.services.mozilla.com/D129437

Depends on D129434

Differential Revision: https://phabricator.services.mozilla.com/D129436

Differential Revision: https://phabricator.services.mozilla.com/D129434

Bug 1741896 Accept/prevent first load prompts for CNN during automated recording r=perftest-reviewers,kimberlythegeek

Differential Revision: https://phabricator.services.mozilla.com/D131508

- Avoid the flag selection silently not enabling the new pass manager
  when --enable-new-pass-manager is passed explicitly.
- Avoid adding the -fexperimental-new-pass-manager to clang >= 13, which
  has it enabled by default. Likewise for the linker flags.
- Remove the force-enable of the new pass manager with clang < 12 on
  automation, since we're using version 13 anyways.
- Account for the fact that both lld and ld64 can pass the
  -import-hot-multiplier flag to the LTO plugin on mac builds, which
  effectively will set it for the first time on mac, and might improve
  performance.

Differential Revision: https://phabricator.services.mozilla.com/D134860

`pip install`'s standard behaviour is to satisfy the new requirement,
even if it conflicts with existing packages. If a conflict _does_
occur, `pip` will simply warn about it after the installation has
already damaged the environment.

By using the `--constraint` feature, we force `pip` to consider
already-installed packages before installing new ones. If a conflict
is detected, an error is raised and `pip` stops without damaging
the environment.

Since we're capturing system packages (when needed) with this feature,
we can now allow virtualenvs that do pip-installations to safely work
with system-using Mach environments.

Differential Revision: https://phabricator.services.mozilla.com/D126925

Bug 1730712: Avoid redundant venv rebuilds in nested Mach calls r=perftest-reviewers,ahal,AlexandruIonescu

Port some `sys.path` modifications to the centralized system.
Not only is this cleaner (ad-hoc global modifications are spicy), but
this also avoids "reordering" issues that can creep in during nested
calls, causing virtualenvs to be unnecessarily recreated.

Differential Revision: https://phabricator.services.mozilla.com/D134481

NSS gyp files use -l$lib, and while OS_LIBS accepts this form and passes
it through, it's not actually a recognized way to link libraries on
clang-cl builds. So, re-normalize the values to not include -l, which
will add it back in the backend when appropriate, or switch to $lib.lib
on platforms that use this form.

Differential Revision: https://phabricator.services.mozilla.com/D134736

This patch resolves cases like the following:
1. The system Python has `zstandard`.
2. `MOZ_AUTOMATION=1 ./mach python --virtualenv psutil <script>`
   is run, adding `psutil` to the import scope.
3. `<script>` runs Mach a _second_ time, and this time Mach needs to
   be able to import `zstandard` (in this case, it should be able
   to fetch it from the system Python's site-packages).

The previous behaviour would add the "site-packages" of the //invoking//
Python interpreter, but ancestor packages would get dropped.

To rectify this issue, this patch changes "import inheritance" to keep
more of the `sys.path`, rather than just
`<external-python>.all_site_packages_dirs()`.

Note: the original implementation of this patch passed forward *all*
of the `sys.path` when creating virtualenvs. However, this caused issues
when Mach ran `pip`, because `pip` was no longer able to discover the
"standard library" (it was failing because it assumed all paths defined
in a virtualenv's site were non-standard-library paths, and the original
implementation broke that assumption).

As part of this, a distinction was defined between the "current" Python
interpreter (external_python) and the top-level Python interpreter
(original_python). This was needed to enable discovering which paths
are "standard library paths".

Differential Revision: https://phabricator.services.mozilla.com/D134201

Bug 1747745 - Fix git worktree support in `./mach bootstrap`. r=firefox-build-system-reviewers,mhentges

Differential Revision: https://phabricator.services.mozilla.com/D134719

We clean up our Gecko clone between tasks via `hg robustcheckout --purge`, which runs `hg purge`. This is very effective, *but* it doesn't detect or clean up any nested clones.

Because we run cross-channel on Gecko workers, and because we clone `comm/` in cross-channel and haven't cleaned it up, and because `hg purge` doesn't detect or clean up nested clones, and because our current virtualenv setup code traverses the tree and can error out on `comm/` clones, let's clean up `comm/` after running cross-channel.

We'll be moving TB cross-channel to different tasks/workers in bug 1742711, and ideally we can update robustcheckout and/or `hg purge` to be able to detect and/or clean up nested clones.

Differential Revision: https://phabricator.services.mozilla.com/D134582

Bug 1747347 - Use configure autobootstrap code to install toolchains in mach bootstrap. r=firefox-build-system-reviewers,andi

This uses the same trick as bug 1743832 under the hood. We could go the
other way around, extracting the configure code to a separate module,
but the longer term goal here is to have configure figure out which
things to bootstrap for the selected build type.

As a side effect, mach bootstrap will stop re-bootstrapping things that
are already up-to-date, at least for things using
install_toolchain_artifact, excluding those that don't follow the
convention wrt the extracted directory path.

Differential Revision: https://phabricator.services.mozilla.com/D134595

Bug 1747387 - Fix MozillaBuildBootstrapper after bug 1747216. r=firefox-build-system-reviewers,glandium

Differential Revision: https://phabricator.services.mozilla.com/D134608