Bug 1734594 - Terminate the sccache server in cases where submakes end in an error. r=firefox-build-system-reviewers,andi

Differential Revision: https://phabricator.services.mozilla.com/D127919

Historically, client.mk was not invoked with -jn because it would create
race conditions, but that was actually mostly solved by the addition of
`.NOTPARALLEL` in bug 422986, although the mechanism of adding -jn via
`MOZ_MAKE_FLAGS` or `MOZ_PARALLEL_BUILD` has continued well past that.

Nowadays, client.mk is only invoked by mach (it will even bail out if
that's not the case) and only has one target (`build`) and no
dependencies.

This means we don't need to rely on `MOZ_PARALLEL_BUILD` to pass `-jn` in
some cases, and can just always invoke `make -f client.mk` with `-jn`, even
when we just want no parallelism, in which case we can use `-j1`.

This, in turn, allows to remove the extra allow_parallel argument to
`_run_make`, and only rely on `num_jobs`, and to remove some of the
multiple ways the `n` in `-jn` could be set.

Differential Revision: https://phabricator.services.mozilla.com/D124729

It was moved in bug 1495798 for rusttests, because they didn't use
client.mk, but as of bug 1683797, they do. And it turns out that when
sccache is really started as originally intended, inheriting the make
jobserver, the build is dead-locked until sccache quits (because sccache
still has a jobserver token for some reason). But sccache never quits
when we make it stop outside client.mk.

Differential Revision: https://phabricator.services.mozilla.com/D124728

Bug 1683797: Removes unnecessary lines from client.mk r=sheehan,firefox-build-system-reviewers,glandium

CWD and BUILDSTATUS are never used.

Differential Revision: https://phabricator.services.mozilla.com/D102661

Backed out changeset b3654ee48ff4 (bug 1683797) for causing frequent build timeouts as bug 1411358. CLOSED TREE

Bug 1683797: Removes unnecessary lines from client.mk r=sheehan,firefox-build-system-reviewers,glandium

CWD and BUILDSTATUS are never used, and core-detection happens within
`mach`.

Differential Revision: https://phabricator.services.mozilla.com/D102661

Bug 1671424 - Move configure execution from client.mk to `mach configure`. r=firefox-build-system-reviewers,rstewart

`mach configure` currently runs the equivalent to `make -f client.mk`.
This is history, and essentially does the following:
- Create `configure` and `js/src/configure` from `configure.in` and
`js/src/configure.in` respectively.
- Create the objdir.
- Run `configure` from the objdir.

The `configure` script is, nowadays, only really used as a means to set
OLD_CONFIGURE (and also for people who want to run `configure`,
literally, as in the `configure; make` workflow). `mach configure`
actually doesn't need it. Neither does recursing into `js/src` require
`js/src/configure`, since bug 1520340 (and now as of bug 1669633, we
don't even recurse).

Because configure.py can actually derive OLD_CONFIGURE on its own
(except for `js/src/configure`, but `mach configure` doesn't run that),
we don't really need `configure` for `mach configure`.

So all in all, we're at a point in history where it's straightforward to
just initiate configure.py from mach configure, so we just do that.

And in the hypothetical case where the `mach configure` code is somehow
running in python2, we get the mach virtualenv python3 and use it to
execute `configure.py`.

Differential Revision: https://phabricator.services.mozilla.com/D93741

Bug 1671424 - Move configure execution from client.mk to `mach configure`. r=firefox-build-system-reviewers,rstewart

`mach configure` currently runs the equivalent to `make -f client.mk`.
This is history, and essentially does the following:
- Create `configure` and `js/src/configure` from `configure.in` and
`js/src/configure.in` respectively.
- Create the objdir.
- Run `configure` from the objdir.

The `configure` script is, nowadays, only really used as a means to set
OLD_CONFIGURE (and also for people who want to run `configure`,
literally, as in the `configure; make` workflow). `mach configure`
actually doesn't need it. Neither does recursing into `js/src` require
`js/src/configure`, since bug 1520340 (and now as of bug 1669633, we
don't even recurse).

Because configure.py can actually derive OLD_CONFIGURE on its own
(except for `js/src/configure`, but `mach configure` doesn't run that),
we don't really need `configure` for `mach configure`.

So all in all, we're at a point in history where it's straightforward to
just initiate configure.py from mach configure, so we just do that.

And in the hypothetical case where the `mach configure` code is somehow
running in python2, we get the mach virtualenv python3 and use it to
execute `configure.py`.

Differential Revision: https://phabricator.services.mozilla.com/D93741

Bug 1660614 - Upgrade `sccache` to pick up more resilient behavior in the presence of cache read failures r=froydnj

This avoids a set of intermittent issues related to `zstd` decompression failures, which in the absence of these changes break the entire build.

This also requires [updating an environment variable](https://github.com/mozilla/sccache/pull/822), which we do in `client.mk` as well as documentation.

Differential Revision: https://phabricator.services.mozilla.com/D88184

Differential Revision: https://phabricator.services.mozilla.com/D72479

Backed out 6 changesets (bug 1632916, bug 1599658, bug 1633037, bug 1633039, bug 1633016, bug 1632920) for SA bustages CLOSED TREE

Backed out changeset 332ce0963b4e (bug 1633039)
Backed out changeset a9904cbc40d9 (bug 1633037)
Backed out changeset d06b0ec349f8 (bug 1599658)
Backed out changeset 8fd300cad80f (bug 1633016)
Backed out changeset f8820941c703 (bug 1632916)
Backed out changeset ac9c2c8746ed (bug 1632920)

Differential Revision: https://phabricator.services.mozilla.com/D72479

Bug 1498031 - Merge code paths for running configure between Tup and Make based backends. r=firefox-build-system-reviewers,mshal

This addresses a related issue along the way: a build that results in running
configure would not update the value of self.config_environment (and therefore
self.substs) as seen from the build driver, so out of date values would have
been used.

The changes to Makefile.in and client.mk made exploit the assumption that by
he time anything in the Make build is running, config.status is up to date.
Users running "make" without the benefit of "mach" will need to manually run
configure when necessary in order to take this into account.

Differential Revision: https://phabricator.services.mozilla.com/D8473

--HG--
extra : moz-landing-system : lando

bug 1495798 - shut down sccache server in mozharness after all build steps. r=firefox-build-system-reviewers,gps

In builds that use sccache, the sccache server would be shut down at the end
of the build step in client.mk. Unfortunately in rusttest builds the check
step winds up compiling more Rust code which restarts the sccache server but
without the proper configuration. This patch moves sccache shutdown to a
post-script step in mozharness so that the server will stay running through
the check step.

Differential Revision: https://phabricator.services.mozilla.com/D7651

--HG--
extra : moz-landing-system : lando

MozReview-Commit-ID: 8lPom2Sgmp

Differential Revision: https://phabricator.services.mozilla.com/D7625

--HG--
extra : moz-landing-system : lando

bug 1490325 - write sccache log directly to artifact directory, make logs slightly more verbose. r=mshal

This patch makes it so we write the sccache log directly to the artifact
directory, so that it will be uploaded even if the build fails. It also makes
the log slightly more verbose. Both of these should help with diagnosing
sccache failures in CI.

The sccache log will no longer be explicitly gzip compressed, but some
Taskcluster client implementations will store it as gzip compressed.

Differential Revision: https://phabricator.services.mozilla.com/D6187

--HG--
extra : moz-landing-system : lando

Bug 1461836 - Write out complete configure dependencies from configure for consumption by make and non-make backends. r=mshal

MozReview-Commit-ID: 792seCZ2rs1

--HG--
extra : rebase_source : e6450cd2947e2fd55085a2af469535421bc07bfb

Unfortunately, I wasn't able to figure out a way to make firefox build & run in
the intermediate stages of these commits. Because of this, I am going to just
delete most of the code which I am deleting in the first patch, as I figure that
those are somewhat uninteresting changes, and then make the other changes in the
following patches.

In total, the following things are deleted:
1. All of xpcom/typelib, except for `xpt/tools` - this directory is being
subsumed entirely into xpcom/reflect/xptinfo.
2. Most of the code in xpcom/reflect/xptinfo, it is being rewritten to avoid
allocating and contain all of the necessary data structures.
3. idl-parser's typelib.py XPT generator, as it will be replaced.
4. Most includes of files which have been deleted.

NOTE: xpcom/typelib/xpt/tools/xpt.py was not removed, as it is used by bundling
code & bundling tests, which we don't want to remove yet.

In order to determine if we need to re-run configure, we write
a JSON file representing the evaluated mozconfig. If this JSON
file changes, configure (and config.status for that matter) is
out of data and it is re-executed.

This commit moves the generation of that JSON file to Python.

MozReview-Commit-ID: 636rpSY7gOm

--HG--
extra : rebase_source : ee1defd74decfd64ffb66a45b053dada58de04fb

This is a pretty straightforward port of the logic. But we
even go a step further: we delete the file in the objdir if there is
no source mozconfig!

MozReview-Commit-ID: AHFFzy6mXRY

--HG--
extra : rebase_source : 1b9387bd72f5a8e9bf8274f5764b0db0176fdba2
extra : source : 0cab9a382d817e6fbab9daa37db0f23e7f73e71f

This target isn't referenced elsewhere in client.mk. It appears to be
dead.

When this target became orphaned, I don't know. I suspect at some point
some included .mk file attempted to "include autoconf.mk." But I'm not
sure what file that would have been or when it would have changed.

FWIW, autoconf.mk is generated by config.status, courtesy of it being
listed in CONFIGURE_SUBST_FILES.

MozReview-Commit-ID: AcPrihAou11

--HG--
extra : rebase_source : 68b1b19428d0754884515c42250353bd75407784

The file is a filtered version of the make file that we previously
started generating for client.mk. Why there is special casing for
UPLOAD_EXTRA_FILES, I'm not sure. This smells fishy and is something
I'd like to take a look at once all code is ported out of client.mk.

The removal of the logic from client.mk meant that we could remove
a bunch of code from client.mk related to loading mozconfig files.
We can now simply include the auto-generated make file directly and
be done with it.

MozReview-Commit-ID: 4M5NElQA7iR

--HG--
extra : rebase_source : 87ed98fa62513007c6fdd2df00eb871a5a29a146
extra : source : 247617a64b7c438528f97d10c86e2f7b8cb72237

We write the file that client.mk is printing from Python. We can
also log it from there pretty easily. So do that.

MozReview-Commit-ID: 7eeugdOJR5b

--HG--
extra : rebase_source : 308826e948fa20684bbc40c806322f802689627e

Now that we require mach to run client.mk and mach only invokes
client.mk for full builds (the "build" target) and configure
(the "configure" target), we should no longer have anyone using
client.mk as a proxy to make targets in the build backend.

So remove that feature.

MozReview-Commit-ID: BbaMdZHnRXy

--HG--
extra : rebase_source : 570e81a62e35cbe4ea27e011883a306f23f3024c

This was added before mach AFAICT. If people need this functionality,
we can add a mach command.

MozReview-Commit-ID: HL52nJE2SMQ

--HG--
extra : rebase_source : cdf67a997731e7f1f94e16aa36c3f806c09779e8

The clobber logic is already written in Python. Now that we
always use mach in front of client.mk, we can check the clobber
state before we execute client.mk.

Since we always check the clobber state, we can remove the
CLOBBER files from various dependencies in client.mk. The
clobberer code should ensure everything is in a good state.

The refactor of the clobber Python code required some changes to
its testing. We drop some support for verifying output strings.
But testing this correctly would require a bit of effort. I don't
think it is worth it.

MozReview-Commit-ID: 69CoImCgtNm

--HG--
extra : rebase_source : c925bb49fd54fe6a5abaa4ac9dc0833e139c6a57

I'm pretty sure this was related to the now removed feature for
building multiple projects.

MozReview-Commit-ID: 4AG7lsk6BLS

--HG--
extra : rebase_source : b677a141161255ffeea57358d4f497fc05c8fb1c

Now that mach is being used to invoke client.mk, we can perform
objdir directory creation there.

Removing the use of mkdir_deps meant that we could remove some
included make files which AFAICT were only used to provide
$(mkdir_deps).

MozReview-Commit-ID: 4ZRToz8xqZy

--HG--
extra : rebase_source : 8d0d2430b33863e1dec8cee84e72178307d1c6e0
extra : source : d223afd90123eba035714288d5da9394b2dbb8d8

The auto-generated make file that we include (and the `mach environment`
output that we included before that) should not contain comment lines.
I think it is safe to remove the code that filters them out.

It is possible a multi-line value in mozconfigs could contain lines
looking like comments and this may cause problems. I'm inclined to
believe that this scenario doesn't exist. If someone complains and
we need to bring back support, we could certainly do that.

MozReview-Commit-ID: 8kKw91HH4ms

--HG--
extra : rebase_source : 3fab8af7f713c91632f3d5172664832af10b555f

This should have the same net result.

TBH, I'm not 100% convinced we need this export. It is only needed
to send variables to sub-makes. And the only make file reading
FOUND_MOZCONFIG is client.mk. Since the code that evals the
auto-generated make file is always executed in client.mk, we
shouldn't need this export.

All this code is going away soon anyway. So I'm inclined to cargo
cult this just in case.

MozReview-Commit-ID: DqF1BU702A

--HG--
extra : rebase_source : 31859c0d4bb6ceb06367bf0ca554d79d57d2d0c3

Currently, client.mk calls `mach environment` to obtain a make file
to be evaluated in the context of client.mk. The reason it is
implemented this way is because client.mk could be an entrypoint to
the build system.

With recent changes that require the use of mach to use client.mk,
we are now guaranteed to have Python code running before client.mk
is invoked. This means we don't need to invoke `mach` from client.mk.

This commit ports the code for generating a client.mk suitable make
file from `mach environment` to the build dispatcher. We now write out
a new .mozconfig-client-mk file in the objdir. client.mk is changed
to cat this file and to include it as a native make file.

The OBJDIR environment variable is also set so client.mk knows where
to read the auto-generated file from.

This commit should be backwards compatible.

Hopefully it is obvious, but this new make file is only temporary.
As soon as the remaining mozconfig logic is moved out of client.mk,
we should be able to simplify down to a single "include" in client.mk.

MozReview-Commit-ID: BEfWo76Z1qA

--HG--
extra : rebase_source : 752df93f816b95bda108a3c787d7f4941b136bbb

Instead of evaluating config.guess in client.mk, we evaluate it
in Python. The Python code also looks for CONFIG_GUESS in the
mozconfig. This still happens in client.mk courtesy of evaling
the mozconfig's relevant parts.

MozReview-Commit-ID: 87NmQiB2ccX

--HG--
extra : rebase_source : 368bc7bf1375a3943ce62fbb77458c40091a7092

MozbuildObject._run_make() also invokes baseconfig.mk as part of
validating the make binary that will be executed.

Since mach calls this Python code and mach is now required to
invoke client.mk, the validation in client.mk is redundant and can
be removed.

autoconf.mk includes baseconfig.mk. So even once we eliminate
client.mk, the make backend itself will still validate make.
We should probably move this validation to moz.configure (if it
isn't already there). But that can be done another time.

MozReview-Commit-ID: DPCBdz253S6

--HG--
extra : rebase_source : 8588738c517bd636039555299095981c71722600

MozReview-Commit-ID: B5TfneZRan7

--HG--
extra : rebase_source : 60f70e7ea656d58658fc5b609a36ada0b3fbbc02

AFAICT this is unused.

There are some mozharness consumers that evaluate an echo-variable-*
target. However, I believe they are hitting the target from debugmake.mk
and not going through client.mk.

MozReview-Commit-ID: 9XjykFMRpsT

--HG--
extra : rebase_source : edc9be971ddb0384408652fd04f550d7d8e51aa6

The last use of PERL was removed recently.

The last use of comma was removed by c8276e89a513 in 2008.

The last use of SH was... never AFAICT. The variable was
defined but unused in the initial import of client.mk from CVS
in 1998.

MozReview-Commit-ID: LFqcVRf36wf

--HG--
extra : rebase_source : 5a32c1969870c29065c80a94ba3f20ba3873569f

After removing sccache.mk, there are no more references to these
variables. Let's nuke them.

MozReview-Commit-ID: LH1oHm59SnU

--HG--
extra : rebase_source : 5138ddd703350caa5f54fe8941a2e0dcb9dc2c17

sccache.mk is the only thing that uses MOZ_PREFLIGHT_ALL and
MOZ_POSTFLIGHT_ALL. We're trying to kill client.mk and these
variables are next on the chopping block.

This commit essentially moves the logic of sccache.mk inline into
client.mk. Initially, I wanted to move the management of the
sccache daemon to Python as part of what `mach build` invokes.
However, the sccache daemon needs access to the make jobserver.
Since we don't have an active make process in `mach`, this is
obviously problematic. The sccache daemon is also used by
configure, which is launched from client.mk. So the best we
can do right now is move the sccache daemon logic into client.mk.

As part of the port, we pass the path to the sccache binary via
an environment variable. This feels slightly better than hardcoding
the path that automation uses.

MozReview-Commit-ID: zcOYR4I1OG

--HG--
extra : rebase_source : 305a237dc9f5bd96e4aa83d491ac2498fe3c3ba0

This was mainly used to support Universal MacOS builds, which were
removed several months ago.

In theory, someone could be using this feature to build multiple
applications with one build system invocation. But given that client.mk
is no longer the preferred interface to the build system and multiple
applications can be built by running `mach build` with different
mozconfigs, I don't think support for this feature is worth keeping.

This commit removes support for MOZ_BUILD_PROJECTS and related
functionality from client.mk. Support for recognizing
MOZ_CURRENT_PROJECT in configure and mozconfig evaluation has also
been removed. This includes support for the ac_add_app_options
mozconfig function.

Good riddance.

MozReview-Commit-ID: 7xI2jYxDFFr

--HG--
extra : rebase_source : 91068f3b8ae32fbcda4defb5d4bb086a04387598