Commit 60e0566e authored by Axel Hecht's avatar Axel Hecht
Browse files

Bug 1601249, update l10n-related build documentation, r=flod

This fixes quite a bit of historical baggage, and also goes
into a bit more details of what the l10n repacks actually do.

Differential Revision: https://phabricator.services.mozilla.com/D55807

--HG--
extra : moz-landing-system : lando
parent c134c198
......@@ -4,21 +4,26 @@
Localized Builds
================
Single-locale language repacks
==============================
Localization repacks
====================
To save on build time, the build system and automation collaborate to allow
downloading a packaged en-US Firefox, performing some locale-specific
post-processing, and re-packaging a locale-specific Firefox. Such artifacts
are termed "single-locale language repacks". There is another concept of a
post-processing, and re-packaging a locale-specific Firefox. Such artifacts
are termed "single-locale language repacks". There is another concept of a
"multi-locale language build", which is more like a regular build and less
like a re-packaging post-processing step.
.. note::
These builds rely on make targets that don't work for
`artifact builds <https://bugzilla.mozilla.org/show_bug.cgi?id=1387485>`_.
Instructions for single-locale repacks for developers
-----------------------------------------------------
This assumes that ``$AB_CD`` is the locale you want to repack with; I tested
with "ar" and "en-GB".
This assumes that ``$AB_CD`` is the locale you want to repack with; you
find the available localizations on `l10n-central <https://hg.mozilla.org/l10n-central/>`_.
#. You must have a built and packaged object directory, or a pre-built
``en-US`` package.
......@@ -48,6 +53,31 @@ can point to the directory via
ac_add_options --with-l10n-base=/make/this/a/absolute/path
This build also packages a language pack.
Instructions for language packs
-------------------------------
Language packs are extensions that contain just the localized resources. Building
them doesn't require an actual build, but they're only compatible with the
``mozilla-central`` source they're built with.
.. code-block:: shell
./mach build langpack-$AB_CD
This target shares much of the logic of the ``installers-$AB_CD`` target above,
and does the check-out of the localization repository etc. It doesn't require
a package or a build, though. The generated language pack is in
``OBJDIR/dist/$(MOZ_PKG_PLATFORM)/xpi/``.
.. note::
Despite the platform-dependent location in the build directory, language packs
are platform independent, and the content that goes into them needs to be
built in a platform-independent way.
Instructions for multi-locale builds
------------------------------------
......@@ -75,16 +105,42 @@ If you want to create a single build with multiple locales, you will do
AB_CD=multi ./mach package
This `currently <https://bugzilla.mozilla.org/show_bug.cgi?id=1362496>`_ only
works for Firefox for Android.
General flow of repacks
-----------------------
The general flow of the locale repacks is controlled by
``$MOZ_BUILD_APP/locales/Makefile.in`` and ``toolkit/locales/l10n.mk``, plus
the packaging build system. The three main entry points above all trigger
related build flows:
#. Get the localization repository, if needed
#. Run l10n-merge with a prior clobber of the merge dir
#. Copy l10n files to ``dist``, with minor differences here between ``libs-%`` and ``chrome-%``
#. Repackage and package
Details on l10n-merge are described in its own section below.
The copying of files is mainly controlled by ``jar.mn``, in the few source
directories that include localizable files. ``libs-%`` is used for repacks,
``chrome-%`` for multi-locale packages. The repackaging is dedicated
Python code in ``toolkit/mozapps/installer/l10n-repack.py``, using an existing
package. It strips existing ``chrome`` l10n resources, and adds localizations
and metadata.
Language packs don't require repackaging. The windows installers are generated
by merely packaging an existing repackaged zip into to an installer.
Exposing strings
================
Localizers only handle a few file formats in well-known locations in the
The localization flow handles a few file formats in well-known locations in the
source tree.
The locations are specified by TOML files. They're part of the bigger
Alongside being built by including the directory in ``$MOZ_BUILD_APP/locales/Makefile.in``
and respective entries in a ``jar.mn``, we also have configuration files tailored
to localization tools and infrastructure. They're also controlling which
files l10n-merge handles, and how.
These configurations are TOML files. They're part of the bigger
localization ecosystem at Mozilla, and `the documentation about the
file format <http://moz-l10n-config.readthedocs.io/en/latest/fileformat.html>`_
explains how to set them up, and what the entries mean. In short, you find
......@@ -126,10 +182,12 @@ An example looks like this
[includes]
toolkit = toolkit/locales/l10n.ini
This tells the l10n infrastructure three things: Resolve the paths against the
directory two levels up, include files in :file:`browser/locales/en-US` and
:file:`browser/branding/official/locales/en-US`, and load more data from
:file:`toolkit/locales/l10n.ini`.
This tells the l10n infrastructure three things:
* resolve the paths against the directory two levels up
* include files in :file:`browser/locales/en-US` and
:file:`browser/branding/official/locales/en-US`
* load more data from :file:`toolkit/locales/l10n.ini`
For projects like Thunderbird and SeaMonkey in ``comm-central``, additional
data needs to be provided when including an ``l10n.ini`` from a different
......@@ -143,7 +201,7 @@ repository:
repo = https://hg.mozilla.org/
l10n.ini = toolkit/locales/l10n.ini
This tells the l10n pieces where to find the repository, and where inside
This tells the l10n infrastructure where to find the repository, and where inside
that repository the ``l10n.ini`` file is. This is needed because for local
builds, :file:`mail/locales/l10n.ini` references
:file:`mozilla/toolkit/locales/l10n.ini`, which is where the comm-central
......@@ -166,9 +224,9 @@ Properties
`plural support <https://developer.mozilla.org/docs/Mozilla/Localization/Localization_and_Plurals>`_.
ini
Used by the crashreporter and updater, avoid if possible.
foo.defines
Used during builds, for example to create :file:`install.rdf` for
language packs.
inc
Used during builds, for example to create metadata for
language packs or bookmarks.
Adding new formats involves changing various different tools, and is strongly
discouraged.
......@@ -177,10 +235,28 @@ Exceptions
----------
Generally, anything that exists in ``en-US`` needs a one-to-one mapping in
all localizations. There are a few cases where that's not wanted, notably
around search settings and spell-checking dictionaries.
around locale configuration and locale-dependent metadata.
For optional strings and files, l10n-merge won't add ``en-US`` content if
the localization doesn't have that content.
To enable tools to adjust to those exceptions, there's a python-coded
:py:mod:`filter.py`, implementing :py:func:`test`, with the following
For the TOML files, the
`[[filters]] documentation <https://moz-l10n-config.readthedocs.io/en/latest/fileformat.html#filters>`_
is a good reference. In short, filters match the localized source code, optionally
a ``key``, and an action. An example like
.. code-block:: toml
[[filters]]
path = "{l}browser/defines.inc"
key = "MOZ_LANGPACK_CONTRIBUTORS"
action = "ignore"
indicates that the ``MOZ_LANGPACK_CONTRIBUTORS`` in ``browser/defines.inc``
is optional.
For the legacy ini configuration files, there's a Python module
``filter.py`` next to the main ``l10n.ini``, implementing :py:func:`test`, with the following
signature
.. code-block:: python
......@@ -204,19 +280,25 @@ in the en-US file.
l10n-merge
==========
Gecko doesn't support fallback from a localization to ``en-US`` at runtime.
The chrome registry in Gecko doesn't support fallback from a localization to ``en-US`` at runtime.
Thus, the build needs to ensure that the localization as it's built into
the package has all required strings, and that the strings don't contain
errors. To ensure that, we're *merging* the localization and ``en-US``
at build time, nick-named :term:`l10n-merge`.
at build time, nick-named l10n-merge.
For Fluent, we're also removing erroneous messages. For many errors in Fluent,
that's cosmetic, but when a localization has different values or attributes
on a message, that's actually important so that the DOM bindings of Fluent
can apply the translation without having to load the ``en-US`` source to
compare against.
The process can be manually triggered via
.. code-block:: bash
$> ./mach build merge-de
$> ./mach build merge-$AB_CD
It creates another directory in the object dir, :file:`merge-dir/ab-CD`, in
It creates another directory in the object dir, :file:`browser/locales/merge-dir/$AB_CD`, in
which the modified files are stored. The actual repackaging process looks for
the localized files in the merge dir first, then the localized file, and then
in ``en-US``. Thus, for the ``de`` localization of
......@@ -230,7 +312,9 @@ and will include the first of those files it finds.
l10n-merge modifies a file if it supports the particular file type, and there
are missing strings which are not filtered out, or if an existing string
shows an error. See the Checks section below for details.
shows an error. See the Checks section below for details. If the files are
not modified, l10n-merge copies them over to the respective location in the
merge dir.
Checks
------
......@@ -239,8 +323,7 @@ As part of the build and other localization tool chains, we run a variety
of source-based checks. Think of them as linters.
The suite of checks is usually determined by file type, i.e., there's a
suite of checks for DTD files and one for properties files, etc. An exception
are Android-specific checks.
suite of checks for DTD files and one for properties files, etc.
Localizations
-------------
......@@ -248,7 +331,7 @@ Localizations
Now that we talked in-depth about how to expose content to localizers,
where are the localizations?
We host a mercurial repository per locale and per branch. All of our
We host a mercurial repository per locale. All of our
localizations can be found on https://hg.mozilla.org/l10n-central/.
You can search inside our localized files on
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment