puppet.md

TPA uses [Puppet](https://puppet.com/) to manage all servers it operates. It handles
most of the configuration management of the base operating system and
some services. It is *not* designed to handle ad-hoc tasks, for which
we favor the use of [[fabric]].

[[!toc levels=3]]

# Tutorial

This page has a lot of stuff! There's a `Reference` section that
explains how everything is setup, then a few `How-to guides` that show
how to do more specific things. But this first section hopes to get
you running with a simple task that you will get you to do *something*
correctly quickly.

In this tutorial, we will add an IP address to the global allow list,
on all firewalls on all machines. This is a big deal! It will allow
that IP address to access the SSH servers on all boxes and more. This
should be an **static** IP address on a trusted network.

If you have never used Puppet before or if you are nervous at all
about making such a change, it's a good idea to have a more
experienced sysadmin nearby to help you or to ask for help. They can
also confirm this tutorial is what you actually need to do.

 1. To any change on the Puppet server, you will first need to clone
    the git repository:

        git clone pauli.torproject.org:/srv/puppet.torproject.org/git/tor-puppet

    This needs to be only done once.

 2. The firewall rules are defined in the `ferm` module, which lives
    in `modules/ferm`. The file you specifically need to change is
    `modules/ferm/templates/defs.conf.erb`, so open that in your
    editor of choice:

        $EDITOR modules/ferm/templates/defs.conf.erb

 3. The code you are looking for is `ADMIN_IPS`. Add a `@def` for your
    IP address and add the new macro to the `ADMIN_IPS` macro. When
    you exit your editor, git should show you a diff that looks
    something like this:

        --- a/modules/ferm/templates/defs.conf.erb
        +++ b/modules/ferm/templates/defs.conf.erb
        @@ -77,7 +77,10 @@ def $TPO_NET = (<%= networks.join(' ') %>);
         @def $linus   = ();
         @def $linus   = ($linus 193.10.5.2/32); # kcmp@adbc
         @def $linus   = ($linus 2001:6b0:8::2/128); # kcmp@adbc
        -@def $ADMIN_IPS = ($weasel $linus);
        +@def $anarcat = ();
        +@def $anarcat = ($anarcat 203.0.113.1/32); # home IP
        +@def $anarcat = ($anarcat 2001:DB8::DEAD/128 2001:DB8:F00F::/56); # home IPv6
        +@def $ADMIN_IPS = ($weasel $linus $anarcat);


         @def $BASE_SSH_ALLOWED = ();

 4. Then you can commit this and *push*:

        git commit -m'add my home address to the allow list' && git push

 5. Then you should login to one of the hosts and make sure the code
    applies correctly:

        ssh -tt perdulce.torproject.org sudo puppet agent -t

Puppet shows colorful messages. If nothing is red and it returns
correctly, you are done. If that doesn't work, go back to step 2. If
that doesn't work, ask for help from your colleague in the Tor
sysadmin team.

If this works, congratulations, you have made your first change across
the entire Puppet infrastructure! You might want to look at the rest
of the documentation to learn more about how to do different tasks and
how things are setup. A key Howto we recommend is the `Progressive
deployment` section below, which will teach you how to make a change
like the above while making sure you don't break anything even if it
affects a lot of machines.

# How to guides

## Modifying an existing configuration

For new deployments, this is *NOT* the prefered method. For example,
if you are deploying new software that is not already in use in our
infrastructure, do *not* follow this guide and instead follow the
`Adding a new module` guide below.

If you are touching an *existing* configuration, things are much
simpler however: you simply go to the module where the code already
exists and make changes. You `git commit` and `git push` the code,
then immediately run `puppet agent -t` on the affected node.

Look at the `File layout` section above to find the right piece of
code to modify. If you are making changes that potentially affect more
than one host, you should also definitely look at the `Progressive
deployment` section below.

## Adding a new module

This is a broad topic, but let's take the Prometheus monitoring system
as an example which followed the [role/profile/module][]
pattern.

First, the [Prometheus modules on the Puppet forge][] were evaluated
for quality and popularity. There was a clear winner there: the
[Prometheus module][] from [Vox Populi][] had hundreds of thousands
more downloads than the [next option][], which was deprecated.

[next option]: https://forge.puppet.com/brutus777/prometheus
[Vox Populi]: https://voxpupuli.org/
[Prometheus module]: https://forge.puppet.com/puppet/prometheus
[Prometheus modules on the Puppet forge]: https://forge.puppet.com/modules?q=prometheus

Next, the module was added to the Puppetfile (in
`3rdparty/Puppetfile`):

    mod 'puppet-prometheus', '6.4.0'

... and librarian was ran:

    librarian-puppet install

This fetched a lot of code from the Puppet forge: the stdlib, archive
and system modules were all installed or updated. All those modules
were audited manually, by reading each file and looking for obvious
security flaws or backdoors. Then the code was committed into git:

    git add 3rdparty
    git commit -m'install prometheus module after audit'

Then the module was configured in a profile, in `modules/profile/manifests/prometheus/server.pp`:

    class profile::prometheus::server {
      class {
        'prometheus::server':
          # follow prom2 defaults
          localstorage        => '/var/lib/prometheus/metrics2',
          storage_retention   => '15d',
      }
    }

The above contains our local configuration for the upstream
`prometheus::server` class installed in the `3rdparty` directory. In
particular, it sets a retention period and a different path for the
metrics, so that they follow the new Prometheus 2.x defaults.

Then this profile was added to a *role*, in
`modules/roles/manifests/monitoring.pp`:

    # the monitoring server
    class roles::monitoring {
      include profile::prometheus::server
    }

Notice how the role does not refer to any implementation detail, like
that the monitoring server uses Prometheus. It looks like a trivial,
useless, class but it can actually grow to include *multiple*
profiles.

Then that role is added to the Hiera configuration of the monitoring
server, in `hiera/nodes/hetzner-nbg1-01.torproject.org.yaml`:

    classes:
      - roles::monitoring

And Puppet was ran on the host, with:

    puppet --enable ; puppet agent -t --noop ; puppet --disable "testing prometheus deployment"

This led to some problems as the upstream module doesn't support
installing from Debian packages. Support for Debian was added to the
code in `3rdparty/modules/prometheus`, and committed into git:

    emacs 3rdparty/modules/prometheus/manifests/*.pp # magic happens
    git commit -m'implement all the missing stuff' 3rdparty
    git push

And the above puppet commandline was ran again, continuing that loop
until things were good.

If you need to deploy the code to multiple hosts, see the `Progressive
deployment` section below. To contribute changes back upstream (and
you should do so), see the section right below.

## Contributing changes back upstream

For simple changes, the above workflow works well, but eventually it
is preferable to actually fork the upstream repo and operate on our
fork until the changes are merged upstream.

First, the modified module is moved out of the way:

    mv 3rdparty/modules/prometheus{,.orig}

The module is then forked on GitHub or wherever it is hosted, and then
added to the Puppetfile:

    mod 'puppet-prometheus',
        :git => 'https://github.com/anarcat/puppet-prometheus.git',
        :branch => 'deploy'

Then Librarian is ran again to fetch that code:

    librarian-puppet install

Because Librarian is a little dumb, it might checkout your module in
"detached head" mode, in which case you will want to fix the checkout:

    cd 3rdparty/modules/prometheus
    git checkout deploy
    git reset --hard origin/deploy
    git pull

Note that the `deploy` branch here is a merge of all the different
branches proposed upstream in different pull requests, but it could
also be the `master` branch or a single branch if only a single pull
request was sent.

Since you now have a clone of the upstream repository, you can push
and pull normally with upstream. When you make a change, however, you
need to commit (and push) the change *both* in the sub-repo and the
main repository:

    cd 3rdparty/modules/prometheus
    $EDITOR manifests/init.pp # more magic stuff
    git commit -m'change the frobatz to a argblu'
    git push
    cd ..
    git commit -m'change the frobatz to a argblu'
    git push

Often, I make commits directly in our main Puppet repository, without
pushing to the third party fork, until I am happy with the code, and
then I craft a nice pretty commit that can be pushed upstream,
reversing that process:

    $EDITOR 3rdparty/prometheus/manifests/init.pp # dirty magic stuff
    git commit -m'change the frobatz to a quuxblah'
    git push
    # see if that works, generally not
    git commit -m'rah. wanted a quuxblutz'
    git push
    # now we are good, update our pull request
    cd 3rdparty/modules/prometheus
    git commit -m'change the frobatz to a quuxblutz'
    git push

It's annoying to double-commit things, but I haven't found a best way
to do so just yet. This problem is further discussed in [ticket #29387][].

Also note that when you update code like this, the `Puppetfile` does
not change, but the `Puppetfile.lock` file *does* change. The `GIT.sha`
parameter needs to be updated. This can be done by hand, but since
that is error-prone, you might want to simply run this to update
modules:

    librarian-puppet update

This will *also* update dependencies so make sure you audit those
changes before committing and pushing.

## Running tests

Ideally, Puppet modules have a test suite. This is done with
[rspec-puppet](https://rspec-puppet.com/) and [rspec-puppet-facts](https://github.com/mcanevet/rspec-puppet-facts). This is not very well
documented upstream, but it's apparently part of the [Puppet
Development Kit](https://puppet.com/docs/pdk/1.x/pdk.html) (PDK). Anyways: assuming tests exists, you will
want to run some tests before pushing your code upstream, or at least
upstream might ask you for this before accepting your changes. Here's
how to get setup:

    sudo apt install ruby-rspec-puppet ruby-puppetlabs-spec-helper ruby-bundler
    bundle install --path vendor/bundle

This installs some basic libraries, system-wide (Ruby bundler and the
rspec stuff). Unfortunately, required Ruby code is rarely all present
in Debian and you still need to install extra gems. In this case we
set it up within the `vendor/bundle` directory to isolate them from
the global search path.

Finally, to run the tests, you need to wrap your invocation with
`bundle exec`, like so:

    bundle exec rake test

## Listing all hosts under puppet

This will list all active hosts known to the Puppet master:

    ssh -t pauli.torproject.org 'sudo -u postgres psql puppetdb -P pager=off -A -t -c "SELECT c.certname FROM certnames c WHERE c.deactivated IS NULL"'

The following will list all hosts under Puppet and their `virtual`
value:

    ssh -t pauli.torproject.org "sudo -u postgres psql puppetdb -P pager=off -F',' -A -t -c \"SELECT c.certname, value_string FROM factsets fs INNER JOIN facts f ON f.factset_id = fs.id INNER JOIN fact_values fv ON fv.id = f.fact_value_id INNER JOIN fact_paths fp ON fp.id = f.fact_path_id INNER JOIN certnames c ON c.certname = fs.certname WHERE fp.name = 'virtual' AND c.deactivated IS NULL\""  | tee hosts.csv

The resulting file is a Comma-Seperated Value (CSV) file which can be
used for other purposes later.

Possible values of the `virtual` field can be obtain with a similar
query:

    ssh -t pauli.torproject.org "sudo -u postgres psql puppetdb -P pager=off -A -t -c \"SELECT DISTINCT value_string FROM factsets fs INNER JOIN facts f ON f.factset_id = fs.id INNER JOIN fact_values fv ON fv.id = f.fact_value_id INNER JOIN fact_paths fp ON fp.id = f.fact_path_id WHERE fp.name = 'virtual';\""

The currently known values are: `kvm`, `physical`, and `xenu`.

As a bonus, this query will show the number of hosts running each os release:

    SELECT COUNT(c.certname), value_string FROM factsets fs INNER JOIN facts f ON f.factset_id = fs.id INNER JOIN fact_values fv ON fv.id = f.fact_value_id INNER JOIN fact_paths fp ON fp.id = f.fact_path_id INNER JOIN certnames c ON c.certname = fs.certname WHERE fp.name = 'lsbdistcodename' AND c.deactivated IS NULL GROUP BY value_string;

### Other ways of extracting a host list

 * Using the [PuppetDB API][]:

        curl -s -G http://localhost:8080/pdb/query/v4/facts  | jq -r ".[].certname"

   The [fact API][] is quite extensive and allows for very complex
   queries. For example, this shows all hosts with the `apache2` fact
   set to `true`:

        curl -s -G http://localhost:8080/pdb/query/v4/facts --data-urlencode 'query=["and", ["=", "name", "apache2"], ["=", "value", true]]' | jq -r ".[].certname"

   This will list all hosts sorted by their report date, older first,
   followed by the timestamp, space-separated:

        curl -s -G http://localhost:8080/pdb/query/v4/nodes  | jq -r 'sort_by(.report_timestamp) | .[] | "\(.certname) \(.report_timestamp)"' | column -s\  -t

   This will list all hosts with the `roles::static_mirror` class:

        curl -s -G http://localhost:8080/pdb/query/v4 --data-urlencode 'query=inventory[certname] { resources { type = "Class" and title = "Roles::Static_mirror" }} ' | jq .[].certname

   This will show all hosts running Debian buster:

        curl -s -G http://localhost:8080/pdb/query/v4 --data-urlencode 'query=nodes { facts { name = "lsbdistcodename" and value = "buster" }}' | jq .[].certname

 * Using [[cumin]]

 * Using LDAP:

        HOSTS=$(ssh alberti.torproject.org 'ldapsearch -h db.torproject.org -x -ZZ -b dc=torproject,dc=org -LLL "hostname=*.torproject.org" hostname | awk "\$1 == \"hostname:\" {print \$2}" | sort')
        for i in `echo $HOSTS`; do mkdir hosts/x-$i 2>/dev/null || continue; echo $i; ssh $i ' ...'; done

    the mkdir is so that I can run the same command in many terminal
    windows and each host gets only one once

 [PuppetDB API]: https://puppet.com/docs/puppetdb/4.3/api/index.html
 [fact API]: https://puppet.com/docs/puppetdb/4.3/api/query/v4/facts.html

## Batch jobs on all hosts

With that trick, a job can be ran on all hosts with
[parallel-ssh][], for example, check the uptime:

    cut -d, -f1 hosts.hsv | parallel-ssh -i -h /dev/stdin uptime

This would do the same, but only on physical servers:

    grep 'physical$' hosts.hsv | cut -d -f1 | parallel-ssh -i -h /dev/stdin uptime

This would fetch the `/etc/motd` on all machines:

    cut -d -f1 hosts.csv | parallel-slurp -h /dev/stdin -L motd /etc/motd motd

To run batch commands through sudo that requires a password, you will need to fool both sudo and ssh a little more:

    cut -d -f1 hosts.csv | parallel-ssh -P -I -i -x -tt -h /dev/stdin -o pvs sudo pvs

You should then type your password then Control-d. Warning: this will
show your password on your terminal and probably in the logs as well.

Batch jobs can also be ran on all Puppet hosts with Cumin:

    ssh -N -L8080:localhost:8080 pauli.torproject.org &
    cumin '*' uptime

See [[cumin]] for more examples.

 [parallel-ssh]: https://parallel-ssh.org/

## Progressive deployment

If you are making a major change to the infrastructure, you may want
to deploy it progressively. A good way to do so is to include the new
class manually in the node configuration, say in
`hiera/nodes/$fqdn.yaml`:

    classes:
      - my_new_class

Then you can check the effect of the class on the host with the
`--noop` mode. Make sure you disable Puppet so that automatic runs do
not actually execute the code, with:

    puppet agent --disable "testing my_new_class deployment"

Then the new manifest can be simulated with this command:

    puppet agent --enable ; puppet agent -t --noop ; puppet agent --disable "testing my_new_class deployment"

Examine the output and, once you are satisfied, you can re-enable the
agent and actually run the manifest with:

    puppet agent --enable ; puppet agent -t

If the change is *inside* an existing class, that change can be
enclosed in a class parameter and that parameter can be passed as an
argument from Hiera. This is how the transition to a managed
`/etc/apt/sources.list` file was done:

 1. first, a parameter was added to the class that would remove the
    file, defaulting to `false`:

        class torproject_org(
          Boolean $manage_sources_list = false,
        ) {
          if $manage_sources_list {
            # the above repositories overlap with most default sources.list
            file {
              '/etc/apt/sources.list':
                ensure => absent,
            }
          }
        }

 2. then that parameter was enabled on one host, say in
    `hiera/nodes/brulloi.torproject.org.yaml`:

        torproject_org::manage_sources_list: true

 3. Puppet was run on that host using the simulation mode:

        puppet agent --enable ; puppet agent -t --noop ; puppet agent --disable "testing my_new_class deployment"

 4. when satisfied, the real operation was done:

        puppet agent --enable ; puppet agent -t --noop

 5. then this was added to two other hosts, and Puppet was ran there

 6. finally, all hosts were checked to see if the file was present on
    hosts and had any content, with [[cumin]] (see above for
    alternative way of running a command on all hosts):

        cumin '*' 'du /etc/apt/sources.list'

 7. since it was missing everywhere, the parameter was set to `true`
    by default and the custom configuration removed from the three
    test nodes

 8. then Puppet was ran by hand everywhere, using Cumin, with a batch
    of 5 hosts at a time:

        cumin -o txt -b 5 '*' 'puppet agent -t'

    because Puppet returns a non-zero value when changes are made,
    this will above when any one host in a batch of 5 will actually
    operate a change. You can then examine the output and see if the
    change is legitimate or abort the configuration change.

## Debugging things

When a Puppet manifest is not behaving as it should, the first step is
to run it by hand on the host:

    puppet agent -t

If that doesn't yield enough information, you can see pretty much
everything that Puppet does with the `--debug` flag. This will, for
example, include `Exec` resources `onlyif` commands and allow you to
see why they do not work correctly (a common problem):

    puppet agent -t --debug

Finally, some errors show up only on the Puppetmaster: you can look in
`/var/log/daemon.log` there for errors that will only show up there.

Connecting to the PuppetDB database itself can sometimes be easier
than trying to operate the API. There you can inspect the entire thing
as a normal SQL database, use this to connect:

    sudo -u postgres psql puppetdb

It's possible exported resources do surprising things sometimes. It is
useful to look at the actual PuppetDB to figure out which tags
exported resources have. For example, this query lists all exported
resources with `troodi` in the name:

    SELECT certname_id,type,title,file,line,tags FROM catalog_resources WHERE exported = 't' AND title LIKE '%troodi%';

Keep in mind that there are [automatic tags](https://puppet.com/docs/puppet/6.4/lang_tags.html) in exported resources
which can complicate things.

## Password management

If you need to set a password in a manifest, there are special
functions to handle this. We do not want to store passwords directly
in Puppet source code, for various reasons: it is hard to erase
because code is stored in git, but also, ultimately, we want to
publish that source code publicly.

We have two mechanisms on how to do this now: a HKDF to generate
passwords by hashing a common secret, and Trocla, which is currently
[in testing][].

[in testing]: https://bugs.torproject.org/30009

### hkdf

Most passwords in Puppet are managed through a [Key Derivation
Function][] (KDF), more specifically a [hash-based KDF][] that takes a
secret stored on the Puppet master (in `/etc/puppet/secret`)
concatenates this with a unique token picked by the caller, and
generates a secret unique to that token. An example:

[hash-based KDF]: https://en.wikipedia.org/wiki/HKDF
[Key Derivation Function]: https://en.wikipedia.org/wiki/Key_derivation_function

    $secret = hkdf('/etc/puppet/secret', "dip-${::hostname}-base-secret")

This generates a unique passwords for the given token. The password is
then used, in clear text, by the puppet client as appropriate.

The function is an implementation of [RFC5869][], a [SHA256][]-based
HKDF taken from an earlier version of [John Downey's Rubygems
implementation][].

[John Downey's Rubygems implementation]: https://rubygems.org/gems/hkdf
[RFC5869]: https://tools.ietf.org/html/rfc5869
[SHA256]: https://en.wikipedia.org/wiki/SHA-2

### Trocla

[Trocla][] is another password-management solution that takes another
approach. With Trocla, each password is generated on the fly from a
secure entropy source ([Ruby's SecureRandom module][]) and stored
inside a state file (in `/var/lib/trocla/trocla_data.yml`, configured
`/etc/puppet/troclarc.yaml`) on the Puppet master.

Trocla can return "hashed" versions of the passwords, so that the
plain text password is never visible from the client. The plain text
can still be stored on the Puppet master, or it can be deleted once
it's been transmitted to the user or another password manager. This
makes it possible to have Trocla not keep any secret at all.

[Ruby's SecureRandom module]: https://ruby-doc.org/stdlib-1.9.3/libdoc/securerandom/rdoc/SecureRandom.html
[Trocla]: https://github.com/duritong/trocla

This piece of code will generate a [bcrypt][]-hashed password for the
Grafana admin, for example:

    $grafana_admin_password = trocla('grafana_admin_password', 'bcrypt')

The plaintext for that password will never leave the Puppet master. it
will still be stored on the Puppet master, and you can see the value
with:

    trocla get grafana_admin_password plain

... on the commandline.

[bcrypt]: https://en.wikipedia.org/wiki/Bcrypt

A password can also be set with this command:

    trocla set grafana_guest_password plain

Note that this might *erase* other formats for this password, although
those will get regenerated as needed.

Also note that `trocla get` will fail if the particular password or
format requested does not exist. For example, say you generate a
plaintext password with and then get the `bcrypt` version:

    trocla create test plain
    trocla get test bcrypt

This will return the empty string instead of the hashed
version. Instead, use `trocla create` to generate that password. In
general, it's safe to use `trocla create` as it will reuse existing
password. It's actually how the `trocla()` function behaves in Puppet
as well.


## Revoking and generating a new certificate for a host

Revocation procedures problems were discussed in [33587][] and [33446][].

[33587]: https://bugs.torproject.org/33587
[33446]: https://gitlab.torproject.org/legacy/trac/-/issues/33446#note_2349434

 1. Clean the certificate on the master

        puppet cert clean host.torproject.org

 2. Clean the certificate on the client:

        find /var/lib/puppet/ssl -name host.torproject.org.pem -delete

 3. Then run the bootstrap script on the client from
    `tsa-misc/installer/puppet-bootstrap-client ` and get a new checksum

 4. Run `tpa-puppet-sign-client` on the master and pass the checksum

 5. Run `puppet agent -t` to have puppet running on the client again.

# Reference

This documents generally how things are setup.

## Before it all starts

- `puppet.tpo` is currently being run on `pauli.tpo`
- This is where the tor-puppet git repo lives
- The repo has hooks to populate `/etc/puppet` with its contents, most
  notably the modules directory.
- All paths in this document are relative to the root of this
  repository.

## File layout

- `3rdparty/modules` include modules that are shared publicly and do
  not contain any TPO-specific configuration. There is a `Puppetfile`
  there that documents where each module comes from and that can be
  maintained with [r10k][] or [librarian][].

  [librarian]: https://librarian-puppet.com/
  [r10k]: https://github.com/puppetlabs/r10k/

- `modules` includes roles, profiles, and classes that make the bulk
  of our configuration.

- in there, the `roles` class (`modules/roles/manifests/init.pp`) maps
  services to roles, using the `$nodeinfo` variable.

- The `torproject_org` module
  (`modules/torproject_org/manifests/init.pp`) performs basic host
  initialisation, like configuring Debian mirrors and APT sources,
  installing a base set of packages, configuring puppet and timezone,
  setting up a bunch of rc-files and running ud-replicate.

- In there, `local.yaml` (`modules/torproject_org/misc/local.yaml`)
  defines services and list which host(s) supply each
  service. `local.yaml` is read by the `roles` class above for setting
  up the `$localinfo` and `$nodeinfo` variables. It also defines the
  `$roles` parameter and defines `ferm` macros.

- There is also the `hoster.yaml` file
  (`modules/torproject_org/misc/hoster.yaml`) which defines hosting
  providers and specifies things like which network blocks they use,
  if they have a DNS resolver or a Debian mirror. `hoster.yaml` is read
  by
  - the `nodeinfo()` function
    (`modules/puppetmaster/lib/puppet/parser/functions/nodeinfo.rb`),
    used for setting up the `$nodeinfo` variable
  - ferm's `def.conf` template (`modules/ferm/templates/defs.conf.erb`)
  - the `entropy provider`
    (`modules/puppetmaster/lib/puppet/parser/functions/entropy_provider.rb`)
    TODO

- The root of definitions and execution is in Pupept is found in
  the `manifests/site.pp` file, but this file is now mostly empty, in
  favor of Hiera.

Note that the above is the current state of the file hierachy. As part
of the transition to Hiera, a lot of the above architecture will
change in favor of the more standard [role/profile/module][]
pattern. See [ticket #29387][] for an in-depth discussion.

[role/profile/module]: https://puppet.com/docs/pe/2017.2/r_n_p_intro.html
[ticket #29387]: https://bugs.torproject.org/29387

## Custom facts

`modules/torproject_org/lib/facter/software.rb` defines our custom
facts, making it possible to get answer to questions like "Is this
host running apache2?" byt simply looking at a puppet variable.

## Style guide

Puppet manifests should generally follow the [Puppet style
guide][]. This can be easily done with [Flycheck][] in Emacs,
[vim-puppet][], or a similar plugin in your favorite text editor.

Many files do not *currently* follow the style guide, as they
*predate* the creation of said guide. Files should *not* be completely
reformatted unless there's a good reason. For example, if a
conditional covering a large part of a file is removed and the file
needs to be reindented, it's a good opportunity to fix style in the
file. Same if a file is split in two components or for some other
reason completely rewritten.

Otherwise the style already in use in the file should be followed.

[Puppet style guide]: https://puppet.com/docs/puppet/4.8/style_guide.html
[Flycheck]: http://flycheck.org/
[vim-puppet]: https://github.com/rodjek/vim-puppet

## Hiera

[Hiera][] is a "key/value lookup tool for configuration data" which
Puppet uses to look up values for class parameters and node
configuration in General.

We are in the process of transitionning over to this mecanism from our
previous set of custom YAML lookup system. This documents the way we
currently use Hiera.

[Hiera]: https://puppet.com/docs/hiera/3.2/

### Classes definitions

Each host declares which class it should include through a `classes`
parameter. For example, this is what configures a Prometheus server:

    classes:
      - roles::monitoring

Roles should be *abstract* and *not* implementation specific. Each
role includes a set of profiles which *are* implementation
specific. For example, the `monitoring` role includes
`profile::prometheus::server` and `profile::grafana`. Do *not* include
profiles directly from Hiera.

As a temporary exception to this rule, old modules can be included as
we transition from the `has_role` mechanism to Hiera, but eventually
those should be ported to shared modules from the Puppet forge, with
our glue built into a profile on top of the third-party module. The
role `roles::monitoring` follows that pattern correctly.

### Node configuration

On top of the host configuration, some node-specific configuration can
be performed from Hiera. This should be avoided as much as possible,
but sometimes there is just no other way. A good example was the
`build-arm-*` nodes which included the following configuration:

    bacula::client::ensure: "absent"

This disables backups on those machines, which are normally configured
everywhere. This is done because they are behind a firewall and
therefore not reachable, an unusual condition in the network. Another
example is `nutans` which sits behind a NAT so it doesn't know its own
IP address. To export proper firewall rules, the allow address has
been overriden as such:

    bind::secondary::allow_address: 89.45.235.22

Those types of parameters are normally automatically guess inside
modules' classes, but they are overridable from Hiera.

Note: eventually *all* host configuration will be done here, but there
are currently still some configurations hardcoded in individual
modules. For example, the Bacula director is hardcoded in the `bacula`
base class (in `modules/bacula/manifests/init.pp`). That should be
moved into a class parameter, probably in `common.yaml`.

## Cron and scheduling

The Puppet agent is *not* running as a daemon, it's running through
good old `cron`.

Puppet runs on each node every four hour, although with a random 2h
jitter, so the actual frequency is somewhere between 4 and 6
hours.

This configuration is in `/etc/cron.d/puppet-crontab` and deployed by
Puppet itself, currently as part of the `torproject_org` module.