Bug 1861273 - Add some profile markers for sub-phases of configure. r=firefox-build-system-reviewers,ahochheiden

Differential Revision: https://phabricator.services.mozilla.com/D191910

Not necessary because pyparsing is no longer a dependency of packaging, so it is no longer vendored.

Differential Revision: https://phabricator.services.mozilla.com/D186411

Bug 1840537 - Enable value checking in moz.build for CONFIG variables derived from target.*. r=firefox-build-system-reviewers,andi,sergesanspaille

This makes the types from mozbuild.configure.constants repr()-able, and
repr()-ed in config.status, which, when processing moz.build, translates
into value checking for comparison tests involving the variables.

To make them pickable, though, we replace the use of EnumString.subclass
with actual subclassing, which is a little less convenient, but avoids
having to figure out how to make the classes EnumString.subclass creates
pickable.

This caught some mismatches in media/libpng and third_party/libsrtp.

This also means we don't need to normalize the config before dumping it
in config.status, because the only types that this was actually useful
for are these (historically, we'd also turn byte strings into unicode
strings but that hasn't been a thing for 4 years ; the special treatment
of dicts and iterables was there to apply the normalization recursively,
not to normalize dicts and iterables themselves). We still normalization
before passing values to gyp, though.

Differential Revision: https://phabricator.services.mozilla.com/D182141

Bug 1840537 - Enable value checking in moz.build for CONFIG variables derived from target.*. r=firefox-build-system-reviewers,andi,sergesanspaille

This makes the types from mozbuild.configure.constants repr()-able, and
repr()-ed in config.status, which, when processing moz.build, translates
into value checking for comparison tests involving the variables.

To make them pickable, though, we replace the use of EnumString.subclass
with actual subclassing, which is a little less convenient, but avoids
having to figure out how to make the classes EnumString.subclass creates
pickable.

This caught a mismatch in media/libpng.

Differential Revision: https://phabricator.services.mozilla.com/D182141

Backed out changeset 80f21c20d24d (bug 1840537) for causing problems when running e.g. mach lint CLOSED TREE

Backed out changeset 80f21c20d24d (bug 1840537) for causing problems when running e.g. mach lint CLOSED TREE

Bug 1840537 - Enable value checking in moz.build for CONFIG variables derived from target.*. r=firefox-build-system-reviewers,andi

This makes the types from mozbuild.configure.constants repr()-able, and
repr()-ed in config.status, which, when processing moz.build, translates
into value checking for comparison tests involving the variables.

This caught a mismatch in media/libpng.

Differential Revision: https://phabricator.services.mozilla.com/D182141

Bug 1838763 - Move the location of site/command virtualenvs out of the objdir and into the state dir r=firefox-build-system-reviewers,glandium

This is necessary for activating the site/command virtualenvs earlier,
since we can't determine what the objdir will be very early on in the
mach process intialization. We do know where the state dir is, and how
to get to the state dir for a specific topsrcdir, so we can use that
instead. This is already where the `mach` virtualenv lives anyway.

Differential Revision: https://phabricator.services.mozilla.com/D181029

Bug 1838763 - Move the location of site/command virtualenvs out of the objdir and into the state dir r=firefox-build-system-reviewers,glandium

This is necessary for activating the site/command virtualenvs earlier,
since we can't determine what the objdir will be very early on in the
mach process intialization. We do know where the state dir is, and how
to get to the state dir for a specific topsrcdir, so we can use that
instead. This is already where the `mach` virtualenv lives anyway.

Differential Revision: https://phabricator.services.mozilla.com/D181029

Its implementation in python3 is just wrapping pprint.pprint, and
there's only one use.

The python2 implementation is obviously dead code.

Differential Revision: https://phabricator.services.mozilla.com/D179179

Bug 1801836 - Remove no longer necessary 'from __future__' imports. r=linter-reviewers,glandium,webdriver-reviewers,perftest-reviewers,geckoview-reviewers,jld,ahal,owlish,afinder DONTBUILD

Differential Revision: https://phabricator.services.mozilla.com/D165395

Bug 1790816 - Reformat client.py and configure.py with isort. r=linter-reviewers,sylvestre DONTBUILD

# ignore-this-changeset

Differential Revision: https://phabricator.services.mozilla.com/D162671

Differential Revision: https://phabricator.services.mozilla.com/D155919

We also avoid creating the virtualenv before those checks, and remove
the long useless check for build residue in the source directory (these
files will not have been created because builds that would create them
have been forbidden for years)

Differential Revision: https://phabricator.services.mozilla.com/D134584

Bug 1747332 - Avoid creating a virtualenv when running configure.py --help. r=firefox-build-system-reviewers,andi

Differential Revision: https://phabricator.services.mozilla.com/D134583

Differential Revision: https://phabricator.services.mozilla.com/D132190

Bug 1743031 - Add missing path for mozfile when bootstrapping configure.py. r=firefox-build-system-reviewers,mhentges

Differential Revision: https://phabricator.services.mozilla.com/D132189

Mach's import scope includes:
* Its `pth` entries
* Its pip packages, which is either:
    * The Mach virtualenv site-packages, if packages were "pip
      installed" over the internet.
    * The system environment's site-packages, if installing packages
      over the internet is disabled and Mach is grabbing packages from
      the system Python instead.

Command virtualenvs _already_ had this import scope when they were
dynamically activated from an existing Mach process. However, when
used directly (e.g. by `<venv>/bin/python <script>`), they would be
missing this import scope, which was a confusing inconsistency.

However, resolving this inconsistency adds a new risk: when Mach is
using the system Python, the system packages now populate the same
context as command virtualenv packages - and they hadn't been checked
for compatibility. So, this patch also
includes behaviour to verify system<=>command-venv compatibility at
activation-time.

A few notes about this system-package-verification:
* It happens at virtualenv activation-time instead of build-time because
  system packages may change after the virtualenv is built.
* It takes roughly 1.5s, which is significant, but it should mainly occur
  in CI, where it's acceptable. Devs using `MACH_USE_SYSTEM_PACKAGES`
  should unset the variable to avoid the time delay.
* The algorithm works by asserting top-level requirements (e.g.
  `psutil>=5.4.2,<=5.8.0`), then runs `pip check` over the combined set
  of packages that would be in-scope.
* Note that, in this patch, system packages are *not* asserted to be
  the same version as their vendored counterparts. This is because, until
  we parse `third_party/python/requirements.in`, we don't know which
  packages we depend on directly (and whose APIs we care about if they
  change). Since leaning on system packages is essentially only used in
  CI (which we have strong control on), this downside seemed acceptable.

Differential Revision: https://phabricator.services.mozilla.com/D126288

The command site manager needs to be able to do ad-hoc pip
installations, while the Mach site manager needs to manage
the system `sys.path` and conditionally create an on-disk
virtualenv.

By splitting the class into two, we can now give each use case the
attention it deserves.

Differential Revision: https://phabricator.services.mozilla.com/D129529

The existing terminology had two issues:
* `VirtualenvManager` wasn't always associated with an on-disk
  `virtualenv`: for example, when running in automation, Mach
  "activates" a `VirtualenvManager`, updating its import scope,
  but without ever creating an on-disk `virtualenv`.
* An upcoming patch splits the `VirtualenvManager` class, pulling
  "on-disk virtualenv-handling functions" from the project-wide
  interface for managing Python's import scope.

After some good discussion with Ahal, I think we've struck
the terminology that handles this distinction well: we'll call
the "import scope"-handling part the "site", and we'll continue
to call on-disk virtualenvs (and their representative classes)
as, well, virtualenvs.

Differential Revision: https://phabricator.services.mozilla.com/D130391

`js/src` hasn't been rescursively configured from toplevel since bug
1520394.

Differential Revision: https://phabricator.services.mozilla.com/D130509

It's possible for the `PYTHON3` config to point to a different Python
than that that is executing the configure scripts themselves.

This flexibility was needed for the Python 2->3 migration, but now that
it's complete, we can remove the extra configuration and just lean on
the Python interpreter used to run configure.

Differential Revision: https://phabricator.services.mozilla.com/D129863

Bug 1718878 - Updates to the build system to enable compatibility with Python 3.10 (Mainly regarding collections.abc) r=mhentges,webdriver-reviewers,whimboo

Previously, collections.abc was available from the collections import. In Python
3.10, they were explicitly moved to collections.abc. As such, any references to
collections abstract base classes need to import collections.abc in Python 3.10.

More reading here: https://docs.python.org/3/library/collections.abc.html

Differential Revision: https://phabricator.services.mozilla.com/D128838

There's some trade-offs in play here: the major issue is that we can't
pin `psutil`'s because it's pre-installed on some CI workers with a
different version (5.4.2).

Additionally, we can't "just" bump that version, because CI workers jump
between different revisions to do work, so a specific pinned version
won't work when we try to update such a package.

One option is that we could avoid validating package versions in CI, but
then that will cause surprises (heck, I didn't know we were still using
`psutil==5.4.2` instead of `5.8.0` until now). By doing validation, we
make it more explicit and avoid accidentally depending on behaviour of
too new of such a package.

However, in most cases, we manage the installation environment and can
pin dependencies. So, I've made the top-level Mach virtualenv the _only_
one that is able to use requirement specifiers other than "==".

Differential Revision: https://phabricator.services.mozilla.com/D122889

Rather than re-implementing it as `search_path()`, use the existing
`MachEnvRequirements` tool to parse `mach_virtualenv_requirements.txt`

Differential Revision: https://phabricator.services.mozilla.com/D126280

Backed out 10 changesets (bug 1712151, bug 1724279, bug 1730712, bug 1717051, bug 1723031, bug 1731145) for causing failures on test_yaml.py
Backed out changeset 7f64d538701b (bug 1723031)
Backed out changeset 394152994966 (bug 1723031)
Backed out changeset 9bfeb01bcc9a (bug 1723031)
Backed out changeset 3d283616a57d (bug 1730712)
Backed out changeset bc677b409650 (bug 1724279)
Backed out changeset 784c94c2f528 (bug 1723031)
Backed out changeset 6e1bde40e3b4 (bug 1723031)
Backed out changeset 7adf7e2136a3 (bug 1712151)
Backed out changeset 2aef162b9a1b (bug 1717051)
Backed out changeset 9beeb6d3d95b (bug 1731145)

There's some trade-offs in play here: the major issue is that we can't
pin `psutil`'s because it's pre-installed on some CI workers with a
different version (5.4.2).

Additionally, we can't "just" bump that version, because CI workers jump
between different revisions to do work, so a specific pinned version
won't work when we try to update such a package.

One option is that we could avoid validating package versions in CI, but
then that will cause surprises (heck, I didn't know we were still using
`psutil==5.4.2` instead of `5.8.0` until now). By doing validation, we
make it more explicit and avoid accidentally depending on behaviour of
too new of such a package.

However, in most cases, we manage the installation environment and can
pin dependencies. So, I've made the top-level Mach virtualenv the _only_
one that is able to use requirement specifiers other than "==".

Differential Revision: https://phabricator.services.mozilla.com/D122889

Rather than re-implementing it as `search_path()`, use the existing
`MachEnvRequirements` tool to parse `mach_virtualenv_requirements.txt`

Differential Revision: https://phabricator.services.mozilla.com/D126280

Now that we use Python 3.6+, this code is obsolete.

Differential Revision: https://phabricator.services.mozilla.com/D119929

Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.

To produce this patch I did all of the following:

1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.

2. Run ./mach lint --linter black --fix

3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.

4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.

5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).

# ignore-this-changeset

Differential Revision: https://phabricator.services.mozilla.com/D94045

Backed out 10 changesets (bug 1654103, bug 1672023, bug 1518999) for PanZoomControllerTest.touchEventForResult gv-junit failures CLOSED TREE

Backed out changeset ff3fb0b4a512 (bug 1672023)
Backed out changeset e7834b600201 (bug 1654103)
Backed out changeset 807893ca8069 (bug 1518999)
Backed out changeset 13e6b92440e9 (bug 1518999)
Backed out changeset 8b2ac5a6c98a (bug 1518999)
Backed out changeset 575748295752 (bug 1518999)
Backed out changeset 65f07ce7b39b (bug 1518999)
Backed out changeset 4bb80556158d (bug 1518999)
Backed out changeset 8ac8461d7bd7 (bug 1518999)
Backed out changeset e8ba13ee17f5 (bug 1518999)

Bug 1654103: Standardize on Black for Python code in `mozilla-central`. r=remote-protocol-reviewers,marionette-reviewers,webdriver-reviewers,perftest-reviewers,devtools-backward-compat-reviewers,jgilbert,preferences-reviewers,sylvestre,maja_zf,webcompat-reviewers,denschub,ntim,whimboo,sparky

Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.

To produce this patch I did all of the following:

1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.

2. Run ./mach lint --linter black --fix

3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.

4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.

5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).

# ignore-this-changeset

Differential Revision: https://phabricator.services.mozilla.com/D94045

Bug 1654103: Standardize on Black for Python code in `mozilla-central`. r=remote-protocol-reviewers,marionette-reviewers,webdriver-reviewers,perftest-reviewers,devtools-backward-compat-reviewers,jgilbert,preferences-reviewers,sylvestre,maja_zf,webcompat-reviewers,denschub,ntim,whimboo,sparky

Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.

To produce this patch I did all of the following:

1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.

2. Run ./mach lint --linter black --fix

3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.

4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).

# ignore-this-changeset

Differential Revision: https://phabricator.services.mozilla.com/D94045

Bug 1671424 - Move configure execution from client.mk to `mach configure`. r=firefox-build-system-reviewers,rstewart

`mach configure` currently runs the equivalent to `make -f client.mk`.
This is history, and essentially does the following:
- Create `configure` and `js/src/configure` from `configure.in` and
`js/src/configure.in` respectively.
- Create the objdir.
- Run `configure` from the objdir.

The `configure` script is, nowadays, only really used as a means to set
OLD_CONFIGURE (and also for people who want to run `configure`,
literally, as in the `configure; make` workflow). `mach configure`
actually doesn't need it. Neither does recursing into `js/src` require
`js/src/configure`, since bug 1520340 (and now as of bug 1669633, we
don't even recurse).

Because configure.py can actually derive OLD_CONFIGURE on its own
(except for `js/src/configure`, but `mach configure` doesn't run that),
we don't really need `configure` for `mach configure`.

So all in all, we're at a point in history where it's straightforward to
just initiate configure.py from mach configure, so we just do that.

And in the hypothetical case where the `mach configure` code is somehow
running in python2, we get the mach virtualenv python3 and use it to
execute `configure.py`.

Differential Revision: https://phabricator.services.mozilla.com/D93741

Bug 1671424 - Move configure execution from client.mk to `mach configure`. r=firefox-build-system-reviewers,rstewart

`mach configure` currently runs the equivalent to `make -f client.mk`.
This is history, and essentially does the following:
- Create `configure` and `js/src/configure` from `configure.in` and
`js/src/configure.in` respectively.
- Create the objdir.
- Run `configure` from the objdir.

The `configure` script is, nowadays, only really used as a means to set
OLD_CONFIGURE (and also for people who want to run `configure`,
literally, as in the `configure; make` workflow). `mach configure`
actually doesn't need it. Neither does recursing into `js/src` require
`js/src/configure`, since bug 1520340 (and now as of bug 1669633, we
don't even recurse).

Because configure.py can actually derive OLD_CONFIGURE on its own
(except for `js/src/configure`, but `mach configure` doesn't run that),
we don't really need `configure` for `mach configure`.

So all in all, we're at a point in history where it's straightforward to
just initiate configure.py from mach configure, so we just do that.

And in the hypothetical case where the `mach configure` code is somehow
running in python2, we get the mach virtualenv python3 and use it to
execute `configure.py`.

Differential Revision: https://phabricator.services.mozilla.com/D93741

Bug 1669633 - Don't recurse into js/src for the python-part of configure. r=firefox-build-system-reviewers,rstewart

Instead, we now run js/src/old-configure from the top-level configure
after having run old-configure and extracted a few variables to inherit
from it.

Because we're now running from the top-level, $_objdir is always the
top-level objdir, which simplifies some things. The topobjdir in
js/src/config.status, however, needs to stay in js/src because of the
build frontend expecting it there.

When running js/src/old-configure, we used to need some special
treatment for a large number of variables for historic reasons, where
we'd take values from the assigned values before running old-configure
for some, or from AC_SUBSTs after running old-configure.

Now that both old-configure and js/src/old-configure get the same
assignments from old-configure.vars, we don't need anything special for
the former. And only a few remaining variables still need manual work
for the latter.

One notable difference, though, is that the new code doesn't try to
avoid running js subconfigure, which added complexity, and was actually
error-prone.

Differential Revision: https://phabricator.services.mozilla.com/D92725