Unverified Commit a2f56564 authored by boklm's avatar boklm
Browse files

Bug 25817: add ansible roles for tbb-nightly setup

parent 88f2fc7c
...@@ -138,43 +138,18 @@ Automated builds using tbb-testsuite ...@@ -138,43 +138,18 @@ Automated builds using tbb-testsuite
------------------------------------ ------------------------------------
The Tor Browser testsuite scripts can also be used to do nightly builds The Tor Browser testsuite scripts can also be used to do nightly builds
and publish the build logs. and publish the build logs. The recommended way to do that is to use
the ansible roles from the tools/ansible directory. See next section
for details.
If you want to do that, start by cloning the git repository:
$ git clone https://git.torproject.org/boklm/tor-browser-bundle-testsuite.git Using ansible to set up a nightly build machine
-----------------------------------------------
Install some dependencies: The directory tools/ansible contains some ansible roles to set up a
nightly build machine. You can look at the playbook defined in
# apt-get install -y libdata-dump-perl libfile-slurp-perl \ boklm-tbb-nightly-build.yml and variables in group_vars/boklm-tbb-nightly/
libio-captureoutput-perl perlmagick libjson-perl \ for an example of how it can be used.
libwww-perl liblwp-protocol-https-perl libtemplate-perl \
libyaml-syck-perl libdatetime-perl \
libemail-sender-perl libemail-simple-perl libfile-type-perl \
libipc-run-perl libxml-libxml-perl
Copy the config/tor-browser_build-boklm file and edit it:
$ cd tor-browser-bundle-testsuite
$ cp config/tor-browser_build-boklm config/tor-browser_build-$user
$ vim config/tor-browser_build-$user
Change the publish_dir and publish_url options. The publish_dir option is
the local directory where the builds will be stored. The publish_url
option is the public URL where the builds will be available.
Copy the tools/tor-browser-builds-boklm file and edit it to change the
--config= option:
$ cp tools/tor-browser-builds-boklm tools/tor-browser-builds-$user
$ vim tools/tor-browser-builds-$user
You can now run ./tools/tor-browser-builds-$user to start the build, and
add it to your crontab.
The html build reports will be available in the reports/ directory, and
the build files in the tor-browser-builds/ directory (unless you changed
the publish_dir option).
Signing builds Signing builds
......
...@@ -3,3 +3,6 @@ ansible-tbb-build: ...@@ -3,3 +3,6 @@ ansible-tbb-build:
fpcentral: fpcentral:
ANSIBLE_CONFIG='$(@D)/ansible-fpcentral.cfg' ansible-playbook -i inventory --ask-become-pass fpcentral.yml ANSIBLE_CONFIG='$(@D)/ansible-fpcentral.cfg' ansible-playbook -i inventory --ask-become-pass fpcentral.yml
boklm-tbb-nightly-build:
ansible-playbook -i inventory boklm-tbb-nightly-build.yml
...@@ -16,3 +16,9 @@ fpcentral: ...@@ -16,3 +16,9 @@ fpcentral:
you need to be in the fpcentral tpo ldap group. Your ldap password will you need to be in the fpcentral tpo ldap group. Your ldap password will
be asked, to sudo to the fpcentral user. be asked, to sudo to the fpcentral user.
boklm-tbb-nightly-build:
This target is used by boklm to deploy a nightly build machine. If
anybody else wants to set up their own nightly builds, it is possible to
do it by adding a new host to the inventory file and making copies of
group_vars/boklm-tbb-nightly/ and boklm-tbb-nightly-build.yml.
---
- hosts: boklm-tbb-nightly-build
roles:
- role: tbb-builder
- role: tbb-nightly-build
- role: unattended-upgrades
---
nightly_build_hostname: f4amtbsowhix7rrf.onion
nightly_build_url: 'http://{{ nightly_build_hostname }}'
build-sunet-a ansible_ssh_user=root ansible_ssh_host=build-sunet-a.torproject.net build-sunet-a ansible_ssh_user=root ansible_ssh_host=build-sunet-a.torproject.net
fpcentral ansible_become=True ansible_become_method=sudo ansible_become_user=fpcentral ansible_ssh_host=forrestii.torproject.org allow_world_readable_tmpfiles=True fpcentral ansible_become=True ansible_become_method=sudo ansible_become_user=fpcentral ansible_ssh_host=forrestii.torproject.org allow_world_readable_tmpfiles=True
boklm-tbb-nightly-build ansible_ssh_user=root ansible_become_method=su
[tbb-build] [tbb-build]
build-sunet-a build-sunet-a
[fpcentral] [fpcentral]
fpcentral fpcentral
[boklm-tbb-nightly]
boklm-tbb-nightly-build
---
nightly_build_user: tbb-nightly
nightly_build_cron_hour: 2
nightly_build_cron_minute: 20
nightly_build_keep_builds: 2
testsuite_dir: "/home/{{ nightly_build_user }}/tbb-testsuite"
testsuite_git_url: https://git.torproject.org/boklm/tor-browser-bundle-testsuite.git
testsuite_git_commit: 348ad855711382089c4fbf1badfec58e31a6c148
nightly_build_wwwdir: "/home/{{ nightly_build_user }}/www"
nightly_build_nginx_enable: true
nightly_build_nginx_listen: 127.0.0.1:80
---
- name: restart nginx
service:
name: nginx
state: restarted
---
- name: create tbb-nightly user
user:
name: "{{ nightly_build_user }}"
comment: "Tor Browser Nightly Builds"
groups: tbb-build
createhome: yes
home: "/home/{{ nightly_build_user }}"
- name: clone tor browser testsuite
become: yes
become_user: "{{ nightly_build_user }}"
git:
repo: "{{ testsuite_git_url }}"
dest: "{{ testsuite_dir }}"
version: "{{ testsuite_git_commit }}"
- name: install testsuite dependencies
command: "{{ testsuite_dir }}/install-deps"
- name: add testsuite config
template:
src: testsuite-config
dest: "{{ testsuite_dir }}/config/tbb-nightly"
mode: 0644
owner: "{{ nightly_build_user }}"
- name: create rbm-config directory
file:
path: '{{ testsuite_dir }}/rbm-config/'
state: directory
owner: '{{ nightly_build_user }}'
mode: 0755
- name: add rbm config
template:
src: rbm.local.conf
dest: "{{ testsuite_dir }}/rbm-config/tbb-nightly.rbm.local.conf"
mode: 0644
owner: "{{ nightly_build_user }}"
- name: add start-tbb-nightly script
template:
src: start-tbb-nightly
dest: "/home/{{ nightly_build_user }}/start-tbb-nightly"
mode: 0755
owner: "{{ nightly_build_user }}"
- name: add cron to start nighly build
cron:
name: tbb-nightly-build
user: "{{ nightly_build_user }}"
hour: "{{ nightly_build_cron_hour }}"
minute: "{{ nightly_build_cron_minute }}"
job: "/home/{{ nightly_build_user }}/start-tbb-nightly"
- name: create wwwdir
file:
path: '{{ nightly_build_wwwdir }}'
state: directory
owner: '{{ nightly_build_user }}'
mode: 0755
- name: add index page
template:
src: www-index.html
dest: "{{ nightly_build_wwwdir }}/index.html"
mode: 0644
owner: "{{ nightly_build_user }}"
- name: install nginx
apt:
name: nginx
state: present
when: nightly_build_nginx_enable
- name: add nginx config
template:
src: nginx.conf
dest: /etc/nginx/sites-enabled/tbb-nightly-build
mode: 0644
notify:
- restart nginx
when: nightly_build_nginx_enable
server {
listen {{ nightly_build_nginx_listen }};
server_name {{ nightly_build_hostname }};
root {{ nightly_build_wwwdir }};
index index.html;
location / {
try_files $uri $uri/ =404;
}
location /reports {
alias {{ testsuite_dir }}/reports;
}
location /tor-browser-builds {
alias {{ testsuite_dir }}/tor-browser-builds;
autoindex on;
}
}
---
### This file is used to override options from rbm.conf to adapt them
### to your local setup.
###
### Copy this file as rbm.local.conf to enable it, and uncomment the
### options you want to modify.
### The tmp_dir option defines where temporary files are stored. The
### builds are made from this directory, so using a directory on a fast
### disk can improve build time. By default we are using a tmp directory
### under the tor-browser-build directory.
#tmp_dir: /tmp
### The debug option defines whether a debugging shell should be opened
### automatically in the build directory/container in case of build
### failure. If you are doing automated builds, you might want to disable
### this.
#debug: 0
### The build_log option defines in which file the build logs of each
### component are stored. If you set it to '-' the logs are output on
### stdout and stderr.
#build_log: '-'
### By default, the logs from previous builds are kept in the log files.
### If you set build_log_append to 0, then previous logs are cleaned
### when starting a new build.
#build_log_append: 0
buildconf:
buildconf: 1
### The buildconf/num_procs option can be used to select the number of
### build processes to run simultaneously. You can also use the
### RBM_NUM_PROCS environment variable. The default is 4.
#num_procs: 8
### The buildconf/git_signtag_opt option is useful when you tag a release.
### It is used to set the 'git tag' argument to select the keyid for
### signing the tag.
#git_signtag_opt: '-u keyid'
var:
local_conf: 1
### The var/sign_build option defines if you want to sign the
### sha256sums-unsigned-build.txt file with gpg.
#sign_build: 1
### The var/sign_build_gpg_opts option can be used to define some gpg
### options to select the key to use to sign the sha256sums-unsigned-build.txt
### file.
#sign_build_gpg_opts: '--local-user XXXXXXXX'
### The clean configuration is used by the cleaning script to find the
### branches and build targets you are using, to compute the list of
### files that should be kept.
###
### If you only do alpha builds for all platforms, you can use the
### following configuration:
clean:
HEAD:
- project: release
target:
- alpha
- torbrowser-all
- project: release
target:
- nightly
- torbrowser-all
#
### If you are doing 'release' builds in the maint-7.0 branch and
### 'alpha' builds in the master branch, you can use the following
### configuration:
#clean:
# master:
# - project: release
# target:
# - alpha
# - torbrowser-all
# maint-7.0:
# - project: release
# target:
# - release
# - torbrowser-all
targets:
### testbuild is based on alpha by default. Uncomment this if you want it
### to be based on nightly.
#torbrowser-testbuild:
# - testbuild
# - nightly
testbuild:
var:
testbuild: 1
### Uncomment this if you want to create mar files in your test build.
#build_mar: 1
nightly:
### By default 'fetch' is set to 1 for nightly builds, meaning that new
### commits will be fetched automatically during each build. You can
### disable this during development if you want to do rebuilds to test
### a specific change, but don't want rebuilds caused by unrelated
### changes, or if you want to decide at which point new commits are
### fetched. When 'fetch' is set to 'if_needed', new commits will only
### be fetched if the selected commit (or branch, or tag) is not present,
### which means that existing branches won't be updated. In that case
### you can fetch new commits by running 'make fetch'.
fetch: 'if_needed'
# vim: filetype=yaml sw=2
#!/bin/sh
cd {{ testsuite_dir }}
export RBM_NO_DEBUG=1
./tbb-testsuite --config=tbb-nightly "$@"
./tools/prune-old-builds/prune-old-builds --prefix '' --days {{ nightly_build_keep_builds }} ./tor-browser-builds
# vim: filetype=perl expandtab
use strict;
use FindBin;
use DateTime;
use TBBTestSuite::TestSuite::TorBrowserBuild;
my $date = DateTime->now->ymd;
my $name = "tor-browser-$date";
if (-d "$options->{'reports-dir'}/r/$name") {
print "Doing nothing: $name already done\n";
return ( args => [] );
}
my $testsuite = TBBTestSuite::TestSuite::TorBrowserBuild->new({
publish_dir => "$FindBin::Bin/tor-browser-builds/$date",
publish_url => "{{ nightly_build_url }}/tor-browser-builds/$date",
rbm_local_conf => "$FindBin::Bin/rbm-config/tbb-nightly.rbm.local.conf",
make_clean => 1,
});
my %res = (
name => $name,
args => [ $testsuite ],
tags => [ 'nightly' ],
);
%res;
<html>
<head>
<title>Tor Browser Nightly Builds</title>
</head>
<body>
<h1>Tor Browser Nightly Builds</title>
<a href="reports/"><h3>Build Reports</h3></a>
<a href="tor-browser-builds/"><h3>Build files</h3></a>
</body>
</html>
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment