Gitlab runner for applications team
Due to the size of the firefox repository, and the needs of the Applications team, their CI runs cause gitlab being blocked or unacceptably slow for all gitlab users (see this issue).
Because of this problem, the apps team's CI development is stalled. The problem is expected to get more severe as they ramp up their QA automation work.
A more comprehensive solution will take some time (I understand from @anarcat that one approach would be to pull apart the different gitlab components and isolate gitaly into its own service that can be optimized and not cause these issues). Because the Apps team work is blocked on this, the discussed solution is to use the idle time on two of the TB build machines to run an apps team group specific gitlab runner which has the firefox source code checked out on a daily basis via cron, and made available into the runner as a mount point.
We believe that this solution will allow us to use existing under-utilized resources, unblock the Apps team's QA work, and not cause gitlab service interruptions for the rest of Tor.
So the request is for TPA to do the following:
-
Install gitlab-runner on
tb-build-02.torproject.organdtb-build-03.torproject.org -
register that runner as a group runner under the applications project (https://gitlab.torproject.org/groups/tpo/applications/-/runners/new) setting the tag
firefoxso it will only pick up jobs that have been tagged that way -
add to the gitlab-runner's
/etc/gitlab-runner/config.tomla read-only mount path from the host (I've picked/srv/tb, if you want to change that, then be sure to change the cronjob below to match a different path. Whatever path is picked, it should need to have at least 20gb available, preferably more):concurrent = 30 [runners.docker] volumes = ["/srv/tb/bundles:/srv:ro"] -
setup the following cronjob to run daily at 00:00UTC
#!/bin/bash
# Exit on error, undefined variable, and errors in a pipeline
set -euo pipefail
# Retry limit for Git operations
RETRY_LIMIT=3
RETRY_DELAY=5 # in seconds
REPOSITORY="https://gitlab.torproject.org/tpo/applications/tor-browser.git"
CLONE_PATH="/srv/tb/bundles" # This folder needs to be mounted to the containers of Gitlab jobs.
BUNDLE_FILE="tor-browser-latest.bundle"
retry() {
local n=0
until [ "$n" -ge "$RETRY_LIMIT" ]; do
"$@" && break
n=$((n+1))
echo "Attempt $n failed! Retrying in $RETRY_DELAY seconds."
sleep "$RETRY_DELAY"
done
if [ "$n" -ge "$RETRY_LIMIT" ]; then
echo "Error: $1 failed after $RETRY_LIMIT attempts. Exiting."
exit 1
fi
}
if [ ! -d "$CLONE_PATH/.git" ]; then
echo "Repository not found in $CLONE_PATH. Initializing."
mkdir -p "$CLONE_PATH"
git -C "$CLONE_PATH" init || {
echo "Failed to initialize repository"
exit 1
}
git -C "$CLONE_PATH" remote add origin "$REPOSITORY" || {
echo "Failed to add remote $REPOSITORY"
exit 1
}
fi
cd "$CLONE_PATH"
DEFAULT_BRANCH_NAME=$(git ls-remote --symref origin HEAD | awk '/^ref:/ {gsub("refs/heads/", "", $2); print $2}') || {
echo "Error: Failed to retrieve default branch name."
exit 1
}
echo "Detected default branch: $DEFAULT_BRANCH_NAME"
echo "Fetching latest changes from $DEFAULT_BRANCH_NAME."
retry git fetch origin "$DEFAULT_BRANCH_NAME"
echo "Updating references."
LATEST_COMMIT_SHA=$(git show --no-patch FETCH_HEAD --format=%H)
echo "$LATEST_COMMIT_SHA" > .git/refs/heads/HEAD || {
echo "Error: Failed to update reference for HEAD"
exit 1
}
echo "$LATEST_COMMIT_SHA" > ".git/refs/heads/$DEFAULT_BRANCH_NAME" || {
echo "Error: Failed to update reference for $DEFAULT_BRANCH_NAME"
exit 1
}