Bug 1773223 - Generate webrtc moz.builds for all platforms at once....

# Setup a Virtual Machine running OpenBSD to generate OpenBSD build files for libwebrtc.

1. Download the OpenBSD 7.0 ISO [here](https://cdn.openbsd.org/pub/OpenBSD/7.0/amd64/install70.iso)

2. Start by setting up a new VM. These steps reference VMWare fusion options,

   but they should translate to other VM hosts.  During the VM creation steps,

   using the following settings:

   - VMWare - Other Linux 5.x kernel 64-bit

   - VMWare - Legacy BIOS

   - VMWare - Customize settings

   - VMWare - Processors & Memory - 2 cores, 2048MB Memory

   - VMWare - Hard Disk - 50.00 GB

3. Start VM to begin the OpenBSD install process.

   In general, choose the default options for everything unless listed below:

   - select ```(I)nstall```

   - allow root ssh - ```yes```

   - disk - for more info see [here](https://azcrumpty.weebly.com/journal/easy-openbsd-partition-scheme)

       - ```c``` # for custom

       - ```z``` # for delete all

       - ```a a``` # for add 'a' partition

       - enter forward slash ('/') for the mount point

       - ```q``` # to quit editor

   - directory does not contain SHA256.sig. Continue without verification? ```yes```

4. Upgrade to OpenBSD -current

   Upgrading to -current is required for the minimum rust version.  For more

   info, see [this](https://unix.stackexchange.com/questions/406870/how-to-follow-openbsd-current).

   - login as root

           ftp -o /bsd.rd https://cloudflare.cdn.openbsd.org/pub/OpenBSD/snapshots/amd64/bsd.rd

           reboot

   - at the bootloader prompt _quickly_ type

           boot bsd.rd

   - select ```U(Upgrade)``` at first prompt question, default to all other questions

   - after upgrade completes allow reboot

5. Upgrade and install necessary packages, add non-root user

   - login as root

         pkg_add -u

         pkg_add sudo-- git-- bash-- mercurial-- m4-- py3-pip--

         ln -sf /usr/local/bin/pip3.9 /usr/local/bin/pip3

         ln -sf /usr/local/bin/python3 /usr/local/bin/python

         visudo # uncomment line with '%wheel        ALL=(ALL) NOPASSWD: SETENV: ALL'

         export NEW_USER={pick-a-user-name}

         useradd -b /home -m -G wheel $NEW_USER

         usermod -s /usr/local/bin/bash $NEW_USER

         passwd $NEW_USER

   - logout

6. Clone mozilla-central repository and bootstrap

   - login as {NEW_USER}

           hg clone https://hg.mozilla.org/mozilla-central

           (cd mozilla-central && ./mach bootstrap)

   - for bootstrap:

       - ignoring errors around glean-sdk

       - select option ```1``` for all pkg_add prompts

7. Continue with step 4 in README.md

# Generate new gn json files and moz.build files for building libwebrtc in our tree

1. If generating on macOS for Apple Silicon (cross-compiling), make sure to have

   at least [Xcode 12.2](https://download.developer.apple.com/Developer_Tools/Xcode_12.2/Xcode_12.2.xip).

/!\ This is only supported on Linux and macOS. If you are on Windows, you can run

the script under [WSL](https://docs.microsoft.com/en-us/windows/wsl/install).

   The aarch64 Rust target will need to be installed via:

   `rustup target add aarch64-apple-darwin`

2. If generating on Windows 10, Visual Studio 2019 is required.  Please follow

   the install instructions from [here](https://firefox-source-docs.mozilla.org/setup/windows_build.html)

   In addition, the following options must be selected in the VS2019 installer:

   - C++ ATL for latest v142 build tools (ARM64)

   - Windows 10 SDK (10.0.19041.0)

   - MSVC v142 - VS2019 C++ ARM64 build tools

   "Debugging Tools for Windows" is also required.

   - Under `Settings -> Apps` search for "Windows Software Development Kit" with

     the version number 10.0.19041.685.

   - select Modify (and allow the installer to modify)

   - select Change and then click Next

   - select "Debugging Tools for Windows" and then click Change.

   The aarch64 Rust target will need to be installed via:

   `rustup target add aarch64-pc-windows-msvc`

3. Download a version of the `gn` executable that corresponds to

1. Download a version of the `gn` executable that corresponds to

    `Thu Nov 19 14:14:00 2020`.  In our case, that is version `1889 (8fe02009)`.

   - [Win](https://chrome-infra-packages.appspot.com/p/gn/gn/windows-amd64/+/e_UmTHedzuu4zJ2gdpW8jrFFNnzIhThljx3jn3RMlVsC)

   - [Linux](https://chrome-infra-packages.appspot.com/p/gn/gn/linux-amd64/+/bvBFKgehaepiKy_YhFnbiOpF38CK26N2OyE1R1jXof0C)

   - [macOS](https://chrome-infra-packages.appspot.com/p/gn/gn/mac-amd64/+/nXvMRpyJhLhisAcnRmU5s9UZqovzMAhKAvWjax-swioC)

   executable:

        unzip gn-mac-amd64.zip && export GN=`pwd`/gn

        unzip gn-windows-amd64.zip && export GN=`pwd`/gn.exe

        unzip gn-linux-amd64.zip && export GN=`pwd`/gn

   On platforms that don't have pre-built `gn` executables, `ninja` and `gn` can

   be easily built:

        git clone https://github.com/ninja-build/ninja.git

        git clone https://gn.googlesource.com/gn

        (cd gn && git checkout 8fe02009)

        (cd ninja && ./configure.py --bootstrap)

        (export NINJA=`pwd`/ninja/ninja ; cd gn && python build/gen.py && $NINJA -C out)

        export GN=`pwd`/gn/out/gn

   On OpenBSD, a slightly newer version of `gn` is needed in order to build:

        (cd gn && git checkout 31f2bba8)

4. It is time to generate the build files.  The script should be run from the

2. It is time to generate the build files.  The script should be run from the

   top directory of our firefox tree.

        bash ./dom/media/webrtc/third_party_build/gn-configs/generate-gn-build-files.sh

   setting the DEBUG_GEN environment variable to a non-empty value. This will 

   print everything that the script executes.

5. Checkin all the generated/modified files and try your build!

3. Checkin all the generated/modified files and try your build!

# Adding new configurations to the build

- Each new platform/architecture will require 2 new mozconfig files, one for the

  debug build and one for the non-debug build.  The filenames follow the same

  pattern as the generated json files, `a-b-c-d.mozconfig` where:

  - a = generating cpu (example: x64)

  - b = debug (True / False)

  - c = target cpu (example: x64 / arm64)

  - d = target platform (mac/linux)

- Each mozconfig file defines, in addition to debug/non-debug, the output

  directory (MOZ_OBJDIR) and any architecture definition.

- The new configs must be added to the appropriate platform section in

  `generate-gn-build-files.sh`.

**Note:** when adding new mozconfig files, especially for linux/android configs,

it is important to include the `ac_add_options --enable-bootstrap`.  This

ensures switching archtectures for "cross-compiled" generation works properly.

For example, when generating `x86` or `arm64` linux json files, it would be

necessary to install additional libraries in order for the configure step to

complete.

Edit the `GnConfigGenBackend` class in `python/mozbuild/mozbuild/gn_processor.py` file.

    # Remove references to CR_XCODE_VERSION and the output directory of the gn process.

    # This change does not cause a change in the generated moz.build files.

    file = re.sub(r' *"CR_XCODE_VERSION=[0-9]+",\n', r"", file)

    file = re.sub(r' *"CR_XCODE_VERSION=([0-9.]+)",\n', r"", file)

    file = re.sub(r' *"CR_SYSROOT_HASH=[0-9a-f]+",\n', r"", file)

    file = re.sub(r',\n *"(.\:)?/.*/third_party/libwebrtc/gn-output/gen/"', r"", file)

    # machines, we only need remove the reference to osx sdk.  Removing it doesn't change

    # the generated moz.build files and makes diffs much easier to see.

    file = re.sub(

        r' *"-isysroot",\n *"[\./]*/Applications/Xcode\.app/Contents'

        "/Developer/Platforms/MacOSX\.platform/Developer/SDKs/"

        'MacOSX([0-9][0-9]\.[0-9])?\.sdk",\n',

        r' *"-isysroot",\n *".*/Contents/Developer/Platforms/MacOSX\.platform/Developer/SDKs/'

        'MacOSX([0-9]+\.[0-9]+)?\.sdk",\n',

        r"",

        file,

    )

  exit 1

fi

IS_WIN=0

IS_DARWIN=0

IS_LINUX=0

# Erase content of third_party/libwebrtc/moz.build to help with generating

# files that may conflict with current moz.build files.  Each config calls

# ./mach configure.  If there are conflicts in any moz.build files under

# third_party/libwebrtc, it would keep our configure step from completing

# successfully.  Since this file will be regenerated at the end of this

# process, we can make it an empty file to avoid conflicts during the

# configure step.

echo "" > third_party/libwebrtc/moz.build

# For now, only macOS, Windows, and Linux (including Android builds) are supported here.

if [ "x$SYS_NAME" = "xDarwin" ]; then

  CONFIGS="x64_False_arm64_mac x64_True_arm64_mac x64_False_x64_mac x64_True_x64_mac"

  IS_DARWIN=1

elif [ "x$SYS_NAME" = "xMINGW32_NT-6.2" ]; then

  unset ANSICON

  CONFIGS="x64_True_arm64_win x64_False_arm64_win"

  CONFIGS="$CONFIGS x64_True_x64_win x64_False_x64_win"

  CONFIGS="$CONFIGS x64_True_x86_win x64_False_x86_win"

  IS_WIN=1

elif [ "x$SYS_NAME" = "xOpenBSD" ]; then

  CONFIGS="x64_False_x64_openbsd x64_True_x64_openbsd"

else

  # Ensure rust has the correct targets for building x86 and arm64.  These

  # operations succeed quickly if previously completed.

  rustup target add aarch64-unknown-linux-gnu

  rustup target add i686-unknown-linux-gnu

  CONFIGS="x64_False_x64_linux_False x64_True_x64_linux_False"

  CONFIGS="$CONFIGS x64_False_x64_linux_True x64_True_x64_linux_True"

  CONFIGS="$CONFIGS x64_False_x86_linux_False x64_True_x86_linux_False"

  CONFIGS="$CONFIGS x64_False_x86_linux_True x64_True_x86_linux_True"

  CONFIGS="$CONFIGS x64_False_arm64_linux_False x64_True_arm64_linux_False"

  CONFIGS="$CONFIGS x64_False_arm64_linux_True x64_True_arm64_linux_True"

  CONFIGS="$CONFIGS x64_False_arm_android x64_True_arm_android"

  CONFIGS="$CONFIGS x64_False_x64_android x64_True_x64_android"

  CONFIGS="$CONFIGS x64_False_x86_android x64_True_x86_android"

  CONFIGS="$CONFIGS x64_False_arm64_android x64_True_arm64_android"

  IS_LINUX=1

fi

CONFIG_DIR=dom/media/webrtc/third_party_build/gn-configs

echo "CONFIG_DIR is $CONFIG_DIR"

# Each mozconfig is for a particular "platform" and defines debug/non-debug and a

# MOZ_OBJDIR based on the name of the mozconfig to make it easier to know where to

# find the generated gn json file.

# The output of this step is a series of gn produced json files with a name format

# that follows the form a-b-c-d.json where:

#   a = generating cpu (example: x64)

#   b = debug (True / False)

#   c = target cpu (example: x64 / arm64)

#   d = target platform (mac/linux)

for THIS_BUILD in $CONFIGS

do

  echo "Building gn json file for $THIS_BUILD"

  export MOZCONFIG=$CONFIG_DIR/$THIS_BUILD.mozconfig

  echo "Using MOZCONFIG=$MOZCONFIG"

  ./mach configure | tee $THIS_BUILD.configure.log

  if [ ! -d obj-$THIS_BUILD ]; then

    echo "Expected build output directory obj-$THIS_BUILD is missing, ensure this is set in $MOZCONFIG"

export MOZ_OBJDIR=$(mktemp -d -p . obj-XXXXXXXXXX)

./mach configure

if [ ! -d $MOZ_OBJDIR ]; then

  echo "Expected build output directory $MOZ_OBJDIR is missing"

  exit 1

fi

  ./mach build-backend -b GnConfigGen --verbose | tee $THIS_BUILD.build-backend.log

  cp obj-$THIS_BUILD/third_party/libwebrtc/gn-output/$THIS_BUILD.json $CONFIG_DIR

done

./mach build-backend -b GnConfigGen --verbose | tee $MOZ_OBJDIR/build-backend.log

cp $MOZ_OBJDIR/third_party/libwebrtc/gn-output/*.json $CONFIG_DIR

# run some fixup (mostly removing dev-machine dependent info) on json files

for THIS_CONFIG in $CONFIGS

for THIS_CONFIG in $CONFIG_DIR/*.json

do

  echo "fixup file: $CONFIG_DIR/$THIS_CONFIG.json"

  ./$CONFIG_DIR/fixup_json.py $CONFIG_DIR/$THIS_CONFIG.json

  if [ "$IS_WIN" == 1 ]; then

    # Use Linux/UNIX line endings

    dos2unix $CONFIG_DIR/$THIS_CONFIG.json

  fi

  echo "fixup file: $THIS_CONFIG"

  ./$CONFIG_DIR/fixup_json.py $THIS_CONFIG

done

#  The symlinks are no longer needed after generating the .json files.

echo "Building moz.build files from gn json files"

./mach build-backend -b GnMozbuildWriter --verbose

# Make sure all the moz.build files have unix line endings if generated

# on Windows.

if [ "$IS_WIN" == 1 ]; then

  MODIFIED_BUILD_FILES=`hg status --modified --added --no-status --include '**/moz.build'`

  for BUILD_FILE in $MODIFIED_BUILD_FILES

  do

    dos2unix $BUILD_FILE

  done

fi

rm -rf $MOZ_OBJDIR

echo

echo "Done generating gn build files. You should now be able to build with ./mach build"

# Use the same Java binary that was specified in bootstrap. This way, if the default system

# Java is different than what Firefox needs, users should just need to override it (with

# $JAVA_HOME) when running bootstrap, rather than when interacting with the build.

ac_add_options --with-java-bin-path=/usr/lib/jvm/java-8-openjdk-amd64/bin

# Build GeckoView/Firefox for Android:

ac_add_options --enable-application=mobile/android

# Targeting the following architecture.

# For regular phones, no --target is needed.

# For x86 emulators (and x86 devices, which are uncommon):

# ac_add_options --target=i686

# For newer phones.

ac_add_options --target=aarch64

# For x86_64 emulators (and x86_64 devices, which are even less common):

# ac_add_options --target=x86_64

ac_add_options --enable-bootstrap

mk_add_options MOZ_OBJDIR=obj-x64_False_arm64_android