GitLab

This script can be used to start/stop containers during rbm builds.

In ce6b9136 we added caching of the exec template function,
however we didn't take the project into account.

Optimize the config function by reducing the number of calls to
config_p.

We also change the behavior of the function: we stop looking after the
first definition of a target. This means for example that if a target is
defined as an array in rbm.conf, and also defined as a hash in the
project's config, then the definition from rbm.conf will be ignored.

Exit with an error message if target, target_prepend or target_append
defined in input_files is not an array.

When using the build command, step gets initialized to 'build'. However
this was not done in some of the other non-build commands where the
value of step was left to rbm_init.

Instead, use the sha256file functon, which keeps the sha256sum of
files to avoid computing it twice. The function can now take a HASH as a
second argument, which can contain the key remove_cache indicating that
the sha256sum of a file should be cleared: we use this when we
find that a file has the wrong checksum and needs to be downloaded
again.

In the build_pkg call in input_files, we were setting the output_dir
option to the project's directory, causing this directory to be
incorrectly used as the output directory in some cases. But there is no
need to set output_dir at this point, so we can just remove it.

We also introduce a new feature allowing to set options for the current
project but not for the child projects listed in `input_files`. By
default the options are applied recursively to all projects. If some
options are defined under `norec/` then they are only applied on the
current project.

We use this feature in two places where we were setting the output_dir,
which removes the need to reset it in two places.

At the same time, we fix the `urlget()` call in input_files which was
incorrectly ignoring options.

See https://wiki.debian.org/ReproducibleBuilds/TimestampsInGzipHeaders

Thanks to boklm for helping with this patch.

When checking the signature on a tag, we also need to check that the tag
is really the expected tag in order to avoid rollback attacks.

Thanks to Santiago Torres-Arias and Keving Gallagher from NYU for
reporting and helping to fix this issue.

The reason we don't run |git checkout --detach| in all cases is that it
fails when we cloned a repository without a master branch. To avoid
running it in this case, we were checking if HEAD is pointing to a
branch which has a corresponding file in the .git directory.

However it seems there are other cases where HEAD is pointing to a
branch which does not have a corresponding file in the .git directory.
Instead we now check that |git rev-parse --verify HEAD| does not return
an error, in which case we assume that |git checkout --detach| will not
produce an error too.

Set DEBIAN_FRONTEND=noninteractive when using apt-get in install_package.
Also use the -q (quiet) argument.

We replace a call to:
  git submodule update --init
with calls to:
  git submodule init
  git submodule sync
  git submodule update

The call to `git submodule sync` is solving the issue that could occur
when a submodule URL is changing.

To create an archive containing all submodules, we were creating a
temporary archive of each submodule before appending them to the main
archive. We were using the submodule path in the temporary archive
filename, which was failing if the submodule is in a subdirectory.
To avoid that, we are removing the path from the temporary archive's
filename and directly appending it to the main archive.

By default gpg will from time to time update its trust database. When
this happens it will also modify the keyring files to add some trust
information. To avoid this we add the --no-auto-check-trustdb option.

As we don't use the Web of Trust when we use a keyring file, we also
disable it using `--trust-model always`.

Print stderr from the git/hg command if the checkout failed.

This script will create a tarball containing the website and print
commands that can be used to upload it.

Fix a regression introduced by #24361.

When a build script contains some wide character, rbm fails with an
error:
  Wide character in syswrite at /usr/share/perl5/File/Slurp.pm line 506.

This is because write_file from File::Slurp which we use to write the
build scripts is encoding files to latin-1 by default. While fixing this,
switching to Path::Tiny instead of File::Slurp seems like a good idea:
http://blogs.perl.org/users/leon_timmermans/2015/08/fileslurp-is-broken-and-wrong.html

Switching to Path::Tiny is also improving performance:

  When using File::Slurp:

    $ time ./rbm/rbm showconf --target alpha --target torbrowser-linux-x86_64 tor-browser var/build_id
    82b8cb

    real    0m28.543s
    user    0m24.448s
    sys     0m3.956s

  When using Path::Tiny:

    $ time ~/rbm/rbm showconf --target alpha --target torbrowser-linux-x86_64 tor-browser var/build_id
    82b8cb

    real    0m15.745s
    user    0m13.940s
    sys     0m1.940s

The path function in lib/RBM.pm is conflicting with the path constructor
from Path::Tiny, so we rename it to rbm_path.

The sha256_hex function gives an error if a string contains wide
characters. To avoid this we first encode the string to utf8 (which
transforms it to a sequence of octets), before passing it to sha256_hex.

When an input file is already present (downloaded during a previous
build), but is not matching the expected sha256sum, we download it
again.

To do that we rename the input_file_id_need_dl function to
input_file_need_dl, and make it work for all types of actions (instead
of only 'input_files_id' before). In input_file_need_dl we ignore an
the file if it is not matching the expected sha256sum.

We were looking at the $remote_tmp_src and $remote_tmp_dst variables to
decide if running remote_finish is needed. However, those variables are
not defined if there was an error in remote_start, and we should still
run remote_finish in that case. Instead we look at the remote_exec
option to decide if we need to run remote_finish.