This change makes us upload an `$(PKG_BASENAME).generated-files.tar.gz` archive
alongside other build artifacts which contains all the generated source files
from the build. A change after this will introduce an `upload-generated-sources`
task to take this artifact and upload the individual files to an S3 bucket.

This will be used to provide links to generated source files when they appear
in stack traces in crash reports.

MozReview-Commit-ID: 6yQAdlZ5q3O

--HG--
extra : rebase_source : d92fb17ae737d1360e9724997f6688e29bedef12
extra : source : 14d18d7cf454c4c3d0f6d49d1d01660e06e4be4b

The --track flag provides a more accurate accounting of what files were
installed by the manifest, so they can be appropriately removed. For
example, test files are now removed from _tests if an entry in a test
file is deleted.

The --no-remove flag is removed as an alternative, and the --track flag
is now mandatory.

MozReview-Commit-ID: Wiup4Gzwkb

--HG--
extra : rebase_source : 4a44c7fe066ba9b5f1e37ec682464f7f4f6cb2cf

The SDK installation code was removed in bug 1333826.

MozReview-Commit-ID: 3h63nFSKOfi

--HG--
extra : rebase_source : 4a99f56e62c749e8991831f1a86cb4d98a58d31e

MozReview-Commit-ID: Gy4YrUtaB5H

--HG--
extra : rebase_source : baa1aa44becab40e1305105b4ab821043ceab9ae

Backed out 4 changesets (bug 1259832) for apparently scheduling nightlies on every push/platform a=backout

Backed out changeset 7781a37a4db0 (bug 1259832)
Backed out changeset 73bf88110b38 (bug 1259832)
Backed out changeset 14d18d7cf454 (bug 1259832)
Backed out changeset a02695cbf576 (bug 1259832)

MozReview-Commit-ID: 4Tf6jBcvKLq

This change makes us upload an `$(PKG_BASENAME).generated-files.tar.gz` archive
alongside other build artifacts which contains all the generated source files
from the build. A change after this will introduce an `upload-generated-sources`
task to take this artifact and upload the individual files to an S3 bucket.

This will be used to provide links to generated source files when they appear
in stack traces in crash reports.

MozReview-Commit-ID: 6yQAdlZ5q3O

--HG--
extra : rebase_source : 3f6ef734c062e0f5e9c2ca433ffad51fdf14b1ad

Note this will only enable it on try, autoland and mozilla-inbound,
which are the only branches where sccache are enabled at the moment.
Enabling on more builds (or not) is the subject of bug 1373460.

Also note that bug 1181040, that ensured PGO builds weren't using
sccache mentions that back then, link times went up when using sccache
(with -Z7) vs. without, but that was presumably with MSVC 2013. Try
suggests link times are the same using sccache now (still using -Z7,
pending bug 1318370).

--HG--
extra : rebase_source : 9f9d87714f3c841b790eb7d692ea0968b1755b70

When investigating why a particular build got a particularly high number
of cache misses, it is useful to know exactly *what* had cache misses,
and you can't get the missing information after the fact.

We used to have some stats stored in a log file uploaded as artifact
with the old python sccache per bug 1005495 (and before that inline in
the build log), and we actually still have remains of that in the form
of a EXTRA_UPLOAD_FILES in build/mozconfig.cache.

Let's reuse that, and get some useful logs from sccache. Ideally sccache
would give better logs, but those are sufficient for now. We'll figure
things out eventually. https://github.com/mozilla/sccache/issues/151

--HG--
extra : rebase_source : a2e7093697a5b720ab28f155560211d6c0a4347e

Thunderbird is still using MOZ_PKG_PRETTYNAMES which has spaces in the package
name, which gets mangled by abspath.

MozReview-Commit-ID: 23ngx6gIESI

--HG--
extra : rebase_source : 9038f295e582bec5872898a5e8f052bfacf8f85a

MozReview-Commit-ID: BIKtzjfu2SF

--HG--
extra : rebase_source : ac53d184a28a7364955cb47f6ecf0d29113801ce

MozReview-Commit-ID: 542dZflb00G

--HG--
extra : rebase_source : 6b942e687517a3053b349a20475de7cff956327a

MozReview-Commit-ID: SKwzZ7l8CS

--HG--
extra : rebase_source : 9e5765df89a966edfbe054b4f555ef347a3fd7e3

This commit makes sccache dump JSON stats at the end of the build, and then
reads them in `BuildScript.generate_build_stats` and adds them to the
build_metrics we submit to Perfherder. The stats dumping is done in
Makefile.in where we currently dump verbose sccache stats because sccache
doesn't persist stats to disk right now and it will also shut down its server
process after 5 minutes, so when the post-build automation steps take more
than 5 minutes the server shuts down and the stats are lost.

Currently it's collecting:
* Cache hit rate
* Cache write errors
* Non-cacheable requests (compiler invocations that sccache can't cache)

We can always grow this list later.

MozReview-Commit-ID: J9CwU7XB05I

--HG--
extra : rebase_source : 084b09c3b0621330ac331a99b1bca9a15cf833b7

This commit moves symbol dumping to the compile tier, to be run via "syms"
targets. Tracking files are used for the sake of incremental builds, because
dump_syms may genearate multiple outputs whose paths are not known ahead of
time.

Minimal changes to symbolstore.py are made here. More extensive
simplifications will be made in a future commit on the basis of symbolstore.py
handling one file at a time.

MozReview-Commit-ID: 3mOP8A6Y7iM

--HG--
extra : rebase_source : bfe97afcdfc05b9e79f01577701c83e8b00eb4e9

In bug 1320194 we want to generate a TestManifest backend without first
invoking configure. However, we still need a way to update the backend,
which is normally handled by the top-level Makefile. We can split this
logic out into a separate file, rebuild-backend.mk, which can be invoked
directly as long as the appropriate environment variables are set
(BUILD_BACKEND_FILES, BACKEND_GENERATION_SCRIPT, and PYTHON).

--HG--
extra : rebase_source : b6a486db3e3c97b406b11074fda052bd63fe1b8f

Bug 1330421 - Update artifact code to populate generated test support files from the tests archive. r=gps

This updates the client artifact code to locate test support files in the
common test archive and populate the objdir with these files appropriately.

MozReview-Commit-ID: GuXjwUtsl

--HG--
extra : rebase_source : 3560efee22533f60be1e7394fa0841005991cca6

These automation steps are no longer relevant, since the last use of
MOZ_AUTOMATION_PRETTY was removed along with non-unified builds in bug
1121000.

MozReview-Commit-ID: 11VDu8pIs1p

--HG--
extra : rebase_source : a82b52400b73224e06e124dd10d6646d4b21a7fc

Bug 1330421 - Update artifact code to populate generated test support files from the tests archive. r=gps

This updates the client artifact code to locate test support files in the
common test archive and populate the objdir with these files appropriately.

MozReview-Commit-ID: GuXjwUtsl

--HG--
extra : rebase_source : 3560efee22533f60be1e7394fa0841005991cca6

This replaces the 'run-tests-deps' make target with a python function that will directly
read moz.build files, emit them with TestManifestEmitter, then consume them with
TestManifestBackend. Because the TestResolver is the only place that actually reads the
test metadata files, we can remove this logic from the CommonBackend as well.

MozReview-Commit-ID: DXgMoeH5dKf



MozReview-Commit-ID: HstZ57qkqf2

--HG--
extra : rebase_source : f377fa6863ef66d3adb86ed64f844e346686862f

MozReview-Commit-ID: HPKIqgZK0Kj

--HG--
extra : rebase_source : 621afc1b1f60f79aeaaa37db41cb602cdc527848

This patch does a few things:
1) Change all the in-tree tooltool manifests to contain sccache2 instead of the existing Python sccache
2) Change mozconfig.cache to point at sccache.
3) Lightly tweak the --with-cccache configure option to support sccache, and detect whether we're using ccache or sccache and set an option appropriately.
4) Add a MOZ_SCCACHE_VERBOSE_STATS option, and add a target in the top-level Makefile to make sccache spit out its stats at the end of the build. This is useful to see the cache hits/errors until we get something better.
5) Add MOZ_USING_SCCACHE to the build telemetry. Not that I think it will be immediately useful, but for future use.

MozReview-Commit-ID: 9lrdLwNj5Bm

--HG--
extra : rebase_source : d323457df10d0ee0ac5811940e518d9422a7e070

Each archive takes 10+ seconds to generate. Split the archiving
part into separate make targets so this completes faster.

I'm not super thrilled about adding yet more make targets to the
root Makefile.in. We can refactor this later. For now, let's
take the quick win.

MozReview-Commit-ID: 2Oeq1rX6yLw

--HG--
extra : rebase_source : 55282a1dacf65adcaf559c178c61ae42591b29ea

Backed out changeset f82827622145 (bug 1304042) for making the build fail on existing failures a=backout

MozReview-Commit-ID: LteC4jGF3FD

--HG--
extra : rebase_source : 2465a5fd04a45d73660cf7557e706631855f24d5

MozReview-Commit-ID: HoDUuKkmv2l

--HG--
extra : rebase_source : 3d4acbcf639976d769a51eacd0d04beb87a40bf3

MozReview-Commit-ID: CPYA1OJUfo8

--HG--
extra : rebase_source : ed8f4b193ec37f3015277a4d39f19a4904b04d9a

MozReview-Commit-ID: L10fff3MbX1

--HG--
extra : rebase_source : 7445541346ea4c84b314a8e2ebdaec324a5e0e72

MozReview-Commit-ID: zyojbOFLLn

--HG--
extra : rebase_source : 17ebd8701bf1dc1a45559a3620a3f6703f950070

This removes the UNIFY_DIST and UNIFIED_BUILD variables, as well as the
--unify flag from the packager and UnifiedBuildFinder from mozpack. As a
result the STAGEPATH variable is never defined anymore, so its uses can
be removed as well.

test_unify.py is currently the only mozbuild/mozpack test that fails
without running configure first, and there isn't much point in fixing
tests for things that we don't actually use anymore.

MozReview-Commit-ID: F5q1FPW3Did

--HG--
extra : rebase_source : cadbd237f51c23ea1983135294521d628d16f0df

MozReview-Commit-ID: DWwxQweCMze

--HG--
extra : rebase_source : eb6ed55c6147ff496f6d15aff037ca276885a171

MozReview-Commit-ID: LNsbghIGJvH

--HG--
extra : rebase_source : 475bfbb846910095ae3d8f4b9d3412bd22b0eab5

This also fixes the issue of processing the artifacts twice in some
situations (bug 1275673). Note that the artifact download no longer
happens when a specific target is passed to 'mach build'.


MozReview-Commit-ID: Ktys6u3r1kG

Bug 1311462 - Dump symbols from */dist/bin instead of dist/universal for Mac universal builds. r=ted

--HG--
extra : rebase_source : 43f8a83a616f7bb825e5e2d2cddf67e868b287ad

These includes make out-of-tree builds of SpiderMonkey depend on the
installer.

On my i7-6700K time and size for the full zip archive for a Windows
debug build is as follows:

          Time       Size
Before:   25.4s    278,708,278
After:    13.0s    280,623,521

Anything lower than level 5 and the archive size starts increasing by
several megabytes. Level 5 is definitely the sweet spot.

MozReview-Commit-ID: 9lW2W1htq93

--HG--
extra : rebase_source : 26458386e4032dfd85eddd937bf6e6bff057d99b

These includes make out-of-tree builds of SpiderMonkey depend on the
installer.

Bug 1242051 - Install test files to the objdir lazily rather than with each invocation of mach. r=gps

This moves test installation for test files out of the monolithic install
manifest for $objdir/_tests, and determines the test and support files
to install based on the object derived from all-tests.json. Additionally,
the files resulting from TEST_HARNESS_FILES are installed, as some tests
will depend on them.

As a result, the time to install tests when invoking the test runner will
scale with the number of tests requested to run rather than the entire set
of tests in the tree, resulting in significantly less overhead.

MozReview-Commit-ID: LeIrUVh1yD4

MozReview-Commit-ID: 4fyzQufrXbO

--HG--
extra : rebase_source : a739e0c0004df6d3d49d3ec9687a25010b024dfe