Automated ISO/IMG builds and tests on Jenkins

⚠️ This service will change during policy/tpa-rfc-73-tails-infra-merge-roadmap and this page should be updated when that happens.

• Automated ISO/IMG builds and tests on Jenkins
  • Configuration
  • Upgrade policy
  • Upgrade steps
  • Agent to controller connections
  • Generating jobs
  • Passing parameters through jobs
  • Builds
  • Tests

Configuration

• master:

  • jenkins class

  • tails::jenkins::master class

  • a few Jenkins plugins installed with jenkins::plugin

  • YAML jobs configuration lives in a dedicated Git repository; Jenkins Job Builder uses it to configure Jenkins

• slaves:

  • tails::profile::jenkins::isoworker, tails::iso_builder, tails::jenkins::slave, tails::jenkins::slave::iso_builder, tails::jenkins::slave::iso_tester, and tails::tester classes

• web server:

  • tails::jenkins::reverse_proxy class

Upgrade policy

Here are some guidelines to triage security vulnerabilities in Jenkins and the plugins we have installed:

1. Protecting our infra from folks who have access to Jenkins

   → Upgrading quarterly is sufficient.

2. Protecting our infra from attacks against folks who have access to Jenkins

   For example, XSS that could lead a legitimate user to perform unintended actions with Jenkins credentials (i.e. root in practice).

   → We should stay on top of security advisories and react more quickly than "in less than 3 months".

3. Protecting our infra from other 3rd-parties that affect Jenkins' security

   For example, say some Jenkins plugin, that connects to remote services, has a TLS certificate checking bug. This could potentially allow a MitM to run arbitrary code with Jenkins orchestrator or workers permissions, i.e. root.

   → We should stay on top of security advisories and react more quickly than "in less than 3 months".

Upgrade steps

• Preparation:

  • Go through the changelog, paying attention to changes on how agents connect to controller, config changes that may need update, important changes in plugins, etc.

• Deployment:

  • Take note of currently running builds before starting the upgrades.

  • Deploy Jenkins upgrade to latest version available using Puppet.

  • Generate a list of up-to-date plugins by running [this Groovy script](jenkins/1.generate-plugins-list.groovy] in the Jenkins Script Console. Make sure to update the initial list containing actively used plugins if there were changes.

  • Generate updated Puppet code for tails::jenkins::master using this Python3 script and the output of the above script.

  • Deploy plugin upgrades using the code generated above.

  • Restart all agents.

  • Manually run the Update jobs script (may be needed so XML is valid with current Jenkins):

    sudo -u jenkins /usr/local/sbin/deploy_jenkins_jobs update

• Wrap up:

  • Go through warnings in Jenkins interface.

  • Manually remove unneeded plugins from /var/lib/jenkins/plugins.

  • Restart builds that were interrupted by Jenkins restart.

  • Update the Jenkins upgrade steps documentation in case there were changes.

  • Schedule next update.

Agent to controller connections

These are the steps a Jenkins agent does when connecting to the controller:

1. Fetch connection info from http://jenkins.dragon:8080 (see the tails::jenkins::slave Puppet class).

2. Receive the connection URL https://jenkins.tails.net ("Jenkins URL", manually configured in Configure System).

3. Resolve jenkins.tails.net to 192.168.122.1 (because of libvirt config).

4. Connect using HTTPS to jenkins.tails.net:443.

5. Learn about port 42585 (fixed "TCP port for inbound agents", manually configured in Configure Global Security).

6. Finally, connect using HTTP to jenkins.tails.net:42585.

Generating jobs

We generate automatically a set of Jenkins jobs for branches that are active in the Tails main Git repository.

The first brick extracts the list of active branches and output the needed information:

• config/chroot_local-includes/usr/lib/python3/dist-packages/tailslib/git.py
• config/chroot_local-includes/usr/lib/python3/dist-packages/tailslib/jenkins.py

This list is parsed by the generate_tails_iso_jobs script run by a cronjob and deployed by our puppet-tails tails::jenkins::iso_jobs_generator manifest.

This script output YAML files compatible with jenkins-job-builder. It creates one project for each active branch, which in turn uses several JJB job templates to create jobs for each branch:

• build_Tails_ISO_*
• reproducibly_build_Tails_ISO_*
• test_Tails_ISO_*

This changes are pushed to our jenkins-jobs git repo by the cronjob, and thanks to their automatic deployment in our tails::jenkins::master and tails::gitolite::hooks::jenkins_jobs manifests in our puppet-tails repo, these new changes are applied to our Jenkins instance.

Passing parameters through jobs

We pass information from build job to follow-up jobs (reproducibility testing, test suite) via two means:

• the Parameterized Trigger plugin, whenever it's sufficient

• the EnvInject plugin, for more complex cases:

  • In the build job, a script collects the needed information and writes it to a file that's saved as a build artifact.

  • This file is used by the build job itself, to setup the variables it needs (currently only $NOTIFY_TO).

  • Follow-up jobs imported this file in the workspace along with the build artifacts, then use an EnvInject pre-build step to load it and set up variables accordingly.

Builds

See jenkins/automated-builds-in-jenkins.

Tests

See jenkins/automated-tests-in-jenkins.