Commit cd15766f authored by Wander Lairson Costa's avatar Wander Lairson Costa
Browse files

Bug 1333167: Add extra try options to taskcluster. r=dustin a=jmaher 

We add the following command line options to Taskcluster try syntax:

--spsProfile: enable profile mode.
--rebuild-talos <N>: retrigger talos tests N times.
--setenv <VAR>=<val>: add extra environments variables.
--tag <TAG>: run tests only the tag TAG.
--no-retry: doesn't retry failed jobs.

We have a chicken-egg problem, as we first generate the full task graph
and then parse the try message. But the graph generation step needs to
know the try message to process the aforementioned options. The
solution is to parse the message before graph generation and then
pass the command line options to the transforms. Then, each transform
can look at the option that interests it and process it accordingly.

The message parse function is configured in kind.yml, which gives some
flexibility for future implementations of alternative syntaxes.

MozReview-Commit-ID: EQlE6q5E8z7

--HG--
extra : rebase_source : 4b7323cd915e8ef9820816015b4b45524811eaf1
parent add1db1b

taskcluster/ci/build/kind.yml

+2 −0
Original line number Diff line number Diff line
@@ -15,3 +15,5 @@ jobs-from: 
    - linux.yml

    - macosx.yml

    - windows.yml



parse-commit: taskgraph.try_option_syntax:parse_message

taskcluster/ci/test/kind.yml

+2 −0
Original line number Diff line number Diff line
@@ -6,3 +6,5 @@ kind-dependencies: 
transforms:

   - taskgraph.transforms.tests:transforms

   - taskgraph.transforms.task:transforms



parse-commit: taskgraph.try_option_syntax:parse_message

taskcluster/docs/loading.rst

+9 −0
Original line number Diff line number Diff line
@@ -29,3 +29,12 @@ on tasks that already exist. You can see a nice example of this behavior in 


For more information on how all of this works, consult the docstrings and

comments in the source code itself.



Try option syntax

-----------------



The ``parse-commit`` optional field specified in ``kind.yml`` links to a

function to parse the command line options in the ``--message`` mach parameter.

Currently, the only valid value is ``taskgraph.try_option_syntax:parse_message``.

The parsed arguments are stored in ``config.config['args']``, it corresponds

to the same object returned by ``parse_args`` from ``argparse`` Python module.

taskcluster/taskgraph/generator.py

+10 −1
Original line number Diff line number Diff line
@@ -6,6 +6,7 @@ from __future__ import absolute_import, print_function, unicode_literals 
import logging

import os

import yaml

import copy



from . import filter_tasks

from .graph import Graph
@@ -38,7 +39,15 @@ class Kind(object): 


    def load_tasks(self, parameters, loaded_tasks):

        impl_class = self._get_impl_class()

        return impl_class.load_tasks(self.name, self.path, self.config,

        config = copy.deepcopy(self.config)



        if 'parse-commit' in self.config:

            parse_commit = find_object(config['parse-commit'])

            config['args'] = parse_commit(parameters['message'])

        else:

            config['args'] = None



        return impl_class.load_tasks(self.name, self.path, config,

                                     parameters, loaded_tasks)

taskcluster/taskgraph/target_tasks.py

+26 −6
Original line number Diff line number Diff line
@@ -42,13 +42,33 @@ def target_tasks_try_option_syntax(full_task_graph, parameters): 
    target_tasks_labels = [t.label for t in full_task_graph.tasks.itervalues()

                           if options.task_matches(t.attributes)]



    # If the developer wants test jobs to be rebuilt N times we add that value here

    if int(options.trigger_tests) > 1:

    attributes = {

        k: getattr(options, k) for k in [

            'env',

            'no_retry',

            'tag',

        ]

    }



    for l in target_tasks_labels:

        task = full_task_graph[l]

        if 'unittest_suite' in task.attributes:

            task.attributes['task_duplicates'] = options.trigger_tests



    for l in target_tasks_labels:

        task = full_task_graph[l]

        # If the developer wants test jobs to be rebuilt N times we add that value here

        if options.trigger_tests > 1 and 'unittest_suite' in task.attributes:

            task.attributes['task_duplicates'] = options.trigger_tests

            task.attributes['profile'] = False



        # If the developer wants test talos jobs to be rebuilt N times we add that value here

        if options.talos_trigger_tests > 1 and 'talos_suite' in task.attributes:

            task.attributes['task_duplicates'] = options.talos_trigger_tests

            task.attributes['profile'] = options.profile



        task.attributes.update(attributes)



    # Add notifications here as well

    if options.notifications:

        for task in full_task_graph: