Fabric Tasks

This is a collection of Fabric tasks aimed at automating tasks that cannot be done through Puppet easily. Those are often ad-hoc, bootstrap jobs like server retirement or installation, but it can be anything, really.

Program overview

Here are the known objects here:

• fabric_tpa: our growing pile of code built around Fabric
  • typically a library for other tools in the top level directory, but many functions can be called directly
  • see fab -l after installing dependencies (below)
• find-crontabs.py: a prototype experimenting with Mitogen as a Fabric replacement
• ganeti: a bunch of tools for managing Ganeti clusters
• gitlab-hooks.py: a tool to check and modify web hooks, see below
• handle-abuse.py: pipe an abuse message from Hetzner in there to unsubscribe users from CiviCRM (among, hopefully, other things)
• install: Fabric-based installer
• installer: a lot of legacy installers, some of which are still in use
• reboot: a script to reboot or halt servers reliably, built with Fabric
• retire: a script to (partially) retire a server, built with Fabric

Installation

To run the scripts and programs here, you need to have python 3, fabric, ssh and some other utilities like dh-python. The bootstrap.py script will do that for you, currently for Debian only.

See the setup.cfg file for runtime dependencies. To install dependencies, pip should work, but it's preferable to install them from your distribution.

Programs here are designed to be run directly from the Git repository or a copy of the files, without installation other than the dependencies. Fabric can run from the top-level directory, where the fabfile.py is, otherwise it will look for the file in sys.path.

Code here is developed on Debian stable which currently ships with Python 3.11. It is known to fail on Python 3.7 because of newer type annotations, at least.

What belongs here

If you have a small script that is very minimal and won't grow much more, and that it's designed to run from your workstation (and not the remote servers) it might be a good place for it. Ideally, it would be made into a Fabric task under fabric_tpa, but it's not absolutely mandatory.

If, on the other hand, it's a script that should be run directly on servers managed by Puppet, it might be better to deploy that script through the tor-puppet.git repository.

Note that we're considering changing this dichotomy, as this repository might be deployed on servers by Puppet, see tpo/tpa/team#41484.

Hooks configuration management

The gitlab-hooks.py script is ran on a schedule from GitLab CI, with a group-level secret stored in the tpo/tpa group variables.

It ensures that all repositories in the tpo/tpa namespace have at least some webhook configured, to send IRC notifications, specifically.

Right now, it runs gitlab-tools audit to fail the pipeline when no hook is configured, but it could eventually be improved to automatically configure those hooks based on a configuration file instead. This job expects a GITLAB_PRIVATE_TOKEN variable from CI/CD, a token defined in the tpo/tpa group as gitlab-tools-webhook-audit. See also tpo/tpa/team#41510.

When a pipeline fails because a project doesn't have a hook configured, you can go in the web interface to configure one, or make one using the gitlab-hooks command. You will need two secrets:

• GITLAB_PRIVATE_TOKEN: a personal access tokens, ideally a temporary one for your user, with read/write API access

• HTTP_KGB_TOKEN: a password to store in the web hook so it can authenticate with the bot, that password is available on chives, with:

   grep http_x_gitlab_token /etc/nginx/sites-available/kgb-bot.torproject.org.conf

Then the following command would create a web hook for issues, merge requests, push, tag and pipeline events on the renovate-cron project:

export CI_SERVER_URL=https://gitlab.torproject.org/
export GITLAB_PRIVATE_TOKEN=[REDACTED]
./gitlab-hooks.py -p tpo/tpa/renovate-cron create --token $HTTP_KGB_TOKEN \
    --issues-events --merge-requests-events --pipeline-events --push-events \
    --tag-push-events --url https://kgb-bot.torproject.org/webhook/?channel=tor-admin

Unit tests and linting

Please do write tests for new code as much as possible. It can be hard to write tests for the kind of tasks we do here, but at least try to write small functions that can be unit tested individually and test those.

Code is checked with mypy and black as well, and all those checks are wrapped in tox. To run those tests, call:

tox --current-env

This assumes the following packages are installed:

• tox
• tox-current-env: uses the local packages instead of pip install AKA curl | sh
• all dependencies specified in setup.cfg, including the test group

Those can be installed with the bootstrap.py command, which also deploys the lint.sh wrapper as a pre-commit hook.

Authors

Most of the code here was written by anarcat for the Tor Project.

History

This project used to be called tsa-misc on the Gitolite repository, but was renamed fabric-tasks after migrating to GitLab, partly to get rid of the old "TSA" acronym (which has negative connotations) but also to encourage the use of Fabric for future tasks.