puppet.md

If you see an error like:

    Check last node runs from PuppetDB WARNING - cupani.torproject.org did not update since 2020-05-11T04:38:54.512Z

It can also be eventually accompanied with the puppet server reporting
the same problem:

    Subject: ** PROBLEM Service Alert: pauli/puppet - all catalog runs is WARNING **
    [...]
    Check last node runs from PuppetDB WARNING - cupani.torproject.org did not update since 2020-05-11T04:38:54.512Z

One of the following is happening, in decreasing likeliness:

 1. the node's Puppet manifest has an error of some sort that makes it
    impossible to run the catalog
 2. the node is down and has failed to report since the last time
    specified
 3. the Puppet **server** is down and **all** nodes will fail to
    report in the same way (in which case a lot more warnings will
    show up, and other warnings about the server will come in)

The first situation will usually happen after someone pushed a commit
introducing the error. We try to keep all manifests compiling all the
time and such errors should be immediately fixed. Look at the history
of the Puppet source tree and try to identify the faulty
commit. Reverting such a commit is acceptable to restore the service.

The second situation can happen if a node is in maintenance for an
extended duration. Normally, the node will recover when it goes back
online. If a node is to be permanently retired, it should be removed
from Puppet, using the [host retirement procedures](howto/retire-a-host).

Finally, if the main Puppet **server** is down, it should definitely
be brought back up. See disaster recovery, below.

In any case, running the Puppet agent on the affected node should give
more information:

    ssh NODE puppet agent -t

### Problems pushing to the Puppet server

Normally, when you push new commits to the Puppet server, a hook runs
and updates the working copy. But sometimes this fails with an error
like:

    remote: error: unable to unlink old 'modules/ipsec/misc/config.yaml': Permission denied.

The problem, in such cases, is that the files in the `/etc/puppet/`
checkout are not writable by your user. It could also happen that the
repository itself (in `/srv/puppet.torproject.org/git/tor-puppet`)
could have permission issues.

This problem is described in [issue 29663][] and is due to someone
not pushing properly before you. To fix the permissions, try:

    sudo chown -R root:adm /etc/puppet
    sudo chown :puppet /etc/puppet/secret
    sudo chmod -R g+rw /etc/puppet
    sudo chmod g-w /etc/puppet/secret

[issue 29663]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/29663

A similar recipe could be applied to the git repository, as
needed. Hopefully this will be resolved when we start deploying with a
role account instead.

## Disaster recovery

Ideally, the main Puppet server would be deployable from Puppet
bootstrap code and the [main installer](new-machine). But in practice, much of
its configuration was done manually over the years and it MUST be
restored from [backups](backup) in case of failure.

This probably includes a restore of the [PostgreSQL](postgresql) database
backing the PuppetDB server as well. It's *possible* this step *could*
be skipped in an emergency, because most of the information in
PuppetDB is a cache of exported resources, reports and facts. But it
could also break hosts and make converging the infrastructure
impossible, as there might be dependency loops in exported resources.

In particular, the Puppet server needs access to the LDAP server, and
that is configured in Puppet. So if the Puppet server needs to be
rebuilt from scratch, it will need to be manually allowed access to
the LDAP server to compile its manifest.

So it is strongly encouraged to restore the PuppetDB server database
as well in case of disaster.

This also applies in case of an IP address change of the Puppet
server, in which case access to the LDAP server needs to be manually
granted before the configuration can run and converge. This is a known
bootstrapping issue with the Puppet server and is further discussed in
the [design section](#ldap-integration).

# Reference

This documents generally how things are setup.

## Installation

Setting up a new Puppet server from scratch is not supported, or, to
be more accurate, would be somewhat difficult. The server expects
various external services to populate it with data, in particular:

 * it [fetches data from LDAP](#ldap-integration)
 * [Nagios generates the NRPE configuration](#nagios-integration)
 * the [letsencrypt repository manages the TLS certificates](#lets-encrypt-tls-certificates)

The auto-ca component is also deployed manual, and so are the git
hooks, repositories and permissions.

This needs to be documented, automated and improved. Ideally, it
should be possible to install a new Puppet server from scratch using
nothing but a Puppet bootstrap manifest, see [issue 30770][] and
[issue 29387][], along with [discussion about those improvements in
this page](#proposed-solution), for details.

[issue 30770]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/30770

## SLA

No formal SLA is defined. Puppet runs on a fairly slow `cron` job so
doesn't have to be highly available right now. This could change in
the future if we rely more on it for deployments.

## Design

The Puppet server and PuppetDB currently live on `pauli`. That server
was setup in 2011 by weasel. It follows the configuration of the
Debian Sysadmin (DSA) Puppet server, which has its source code
available in the [dsa-puppet repository](https://salsa.debian.org/dsa-team/mirror/dsa-puppet/). 

The service is maintained by TPA and manages *all* TPA-operated
machines. Ideally, all services are managed by Puppet, but
historically, only basic services were configured through Puppet,
leaving service admins responsible for deploying their services on top
of it. That tendency has shifted recently (~2020) with the deployment
of the [GitLab](gitlab) service through Puppet, for example.

The source code to the Puppet manifests (see below for a Glossary) is
managed through git on a repository hosted directly on the Puppet
server. Agents are deployed as part of the [install process](new-machine), and
talk to the central server using a Puppet-specific certificate
authority (CA).

As mentioned in the [installation section](#installation), the Puppet server
assumes a few components (namely [LDAP](ldap), [Nagios](nagios), [Let's
Encrypt](tls) and auto-ca) feed information into it. This is also
detailed in the sections below. In particular, Puppet acts as a
duplicate "source of truth" for some information about servers. For
example, LDAP has a "purpose" field describing what a server is for,
but Puppet also has the concept of a role, attributed through Hiera
(see [issue 30273][]). A similar problem exists with IP addresses and
user access control, in general.

[issue 30273]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/30273

Puppet is generally considered stable, but the code base is somewhat
showing its age and has accumulated some technical debt.

For example, much of the Puppet code deployed is specific to Tor (and
DSA, to a certain extent) and therefore is only maintained by a
handful of people. It would be preferable to migrate to third-party,
externally maintained modules (e.g. [systemd](https://gitlab.torproject.org/tpo/tpa/team/-/issues/33449), but also many
others, see [issue 29387][] for details). A similar problem exists
with custom Ruby code implemented for various functions, which is
being replaced with Hiera ([issue 30020][]).

The Puppet infrastructure being kept up to date with the latest
versions in Debian but will require some work to port to Puppet 6, as
the current deployment system ("puppetmaster") has been removed in
that new release (see [issue 33588][]).

[issue 33588]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/33588

### Glossary

This is a subset of the [Puppet glossary](https://puppet.com/docs/puppet/latest/glossary.html) to quickly get you
started with the vocabulary used in this document.

 * **Puppet node**: a machine (virtual or physical) running Puppet
 * **Manifest**: Puppet source code
 * **Catalog**: a set of compiled of Puppet source which gets applied
   on a **node** by a **Puppet agent**
 * **Puppet agents**: the Puppet program that runs on all nodes to
   apply manifests
 * **Puppet server**: the server which all **agents** connect to to
   fetch their **catalog**, also known as a **Puppet master** in older
   Puppet versions (pre-6)
 * **Facts**: information collected by Puppet agents on nodes, and
   exported to the Puppet server
 * **Reports**: log of changes done on nodes recorded by the Puppet
   server
 * **[PuppetDB](https://puppet.com/docs/puppetdb/) server**: an application server on top of a PostgreSQL
   database providing an [API](https://puppet.com/docs/puppetdb/5.2/api/index.html) to query various resources like node
   names, facts, reports and so on

### File layout

The Puppet server and PuppetDB server run on
`pauli.torproject.org`. That is where the main git repository
(`tor-puppet`) lives, in
`/srv/puppet.torproject.org/git/tor-puppet`. That repository has hooks
to populate `/etc/puppet` which is the live checkout from which the
Puppet server compiles its catalogs.
  
All paths below are relative to the root of that git repository.

- `3rdparty/modules` include modules that are shared publicly and do
  not contain any TPO-specific configuration. There is a `Puppetfile`
  there that documents where each module comes from and that can be
  maintained with [r10k][] or [librarian][].

  [librarian]: https://librarian-puppet.com/
  [r10k]: https://github.com/puppetlabs/r10k/

- `modules` includes roles, profiles, and classes that make the bulk
  of our configuration.

- each node is assigned a "role" through the ENC, in
  `hiera-enc/nodes/$FQDN.yaml`

  To be more accurate, the ENC assigns top-scope `$role` variable to
  each node, which is in turn used to include a `role::$rolename`
  class on each node. This occurs in the default node definition in
  `manifests/site.pp`.

  Some nodes include a list of classes, inherited from the previous
  Hiera-based setup, but we're in the process of transitioning all
  nodes to single role classes, see [issue 40030][] for progress on
  this work.

[issue 40030]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40030

- The `torproject_org` module
  (`modules/torproject_org/manifests/init.pp`) performs basic host
  initialisation, like configuring Debian mirrors and APT sources,
  installing a base set of packages, configuring puppet and timezone,
  setting up a bunch of configuration files and running `ud-replicate`.

- There is also the `hoster.yaml` file
  (`modules/torproject_org/misc/hoster.yaml`) which defines hosting
  providers and specifies things like which network blocks they use,
  if they have a DNS resolver or a Debian mirror. `hoster.yaml` is read
  by
  - the `nodeinfo()` function
    (`modules/puppetmaster/lib/puppet/parser/functions/nodeinfo.rb`),
    used for setting up the `$nodeinfo` variable
  - `ferm`'s `def.conf` template (`modules/ferm/templates/defs.conf.erb`)

- The root of definitions and execution is in Puppet is found in
  the `manifests/site.pp` file. Its purpose is to include a role class
  for the node as well as a number of other classes which are common
  for all nodes.

Note that the above is the current state of the file hierarchy. As
part Hiera transition ([issue 30020][]), a lot of the above
architecture will change in favor of the more standard
[role/profile/module][] pattern.

Note that this layout might also change in the future with the
introduction of a role account ([issue 29663][]) and when/if the
repository is made public (which requires changing the layout).

See [ticket #29387][] for an in-depth discussion.

[issue 29387]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/29387
[role/profile/module]: https://puppet.com/docs/pe/2017.2/r_n_p_intro.html
[ticket #29387]: https://bugs.torproject.org/29387
[issue 30020]: https://bugs.torproject.org/30020

### Installed packages facts

The `modules/torproject_org/lib/facter/software.rb` file defines our
custom facts, making it possible to get answer to questions like "Is
this host running `apache2`?" by simply looking at a puppet
variable. 

Those facts are deprecated and we should instead install packages
through Puppet instead of manually installing packages on hosts.

### Style guide

Puppet manifests should generally follow the [Puppet style
guide][]. This can be easily done with [Flycheck][] in Emacs,
[vim-puppet][], or a similar plugin in your favorite text editor.

Many files do not *currently* follow the style guide, as they
*predate* the creation of said guide. Files should *not* be completely
reformatted unless there's a good reason. For example, if a
conditional covering a large part of a file is removed and the file
needs to be re-indented, it's a good opportunity to fix style in the
file. Same if a file is split in two components or for some other
reason completely rewritten.

Otherwise the style already in use in the file should be followed.

[Puppet style guide]: https://puppet.com/docs/puppet/4.8/style_guide.html
[Flycheck]: http://flycheck.org/
[vim-puppet]: https://github.com/rodjek/vim-puppet

### External Node Classifier (ENC)

We use an External Node Classifier (or ENC for short) to classify
nodes in different roles but also assign them environments and other
variables. The way the ENC works is that the Puppet server requests
information from the ENC about a node before compiling its catalog.

The Puppet server pulls three elements about nodes from the ENC:

 * `environment` is the standard way to assign nodes to a Puppet
   environment. The default is `production` which is the only
   environment currently deployed.

 * `parameters` is a hash where each key is made available as a
   top-scope variable in a node's manifests. We use this assign a
   unique "role" to each node. The way this works is, for a given role
   `foo`, a class `role::foo` will be included. That class should only
   consist of a set of profile classes.

 * `classes` is an array of class names which Puppet includes on the
   target node. We are currently transitioning from this method of
   including classes on nodes (previously in Hiera) to the `role`
   parameter and unique role classes.

For a given node named `$fqdn`, these elements are defined in
`tor-puppet.git/hiera-enc/nodes/$fqdn.yaml`. Defaults can also be set
in `tor-puppet.git/hiera-enc/nodes/default.yaml`.

#### Role classes

Each host defined in the ENC declares which unique role it should be
attributed through the `parameter` hash. For example, this is what
configures a GitLab runner:

    parameters:
      - role: gitlab::runner

Roles should be *abstract* and *not* implementation specific. Each
role class includes a set of profiles which *are* implementation
specific. For example, the `monitoring` role includes
`profile::prometheus::server` and `profile::grafana`.

As a temporary exception to this rule, old modules can be included as
we transition from the Hiera mechanism, but eventually those should
be ported to shared modules from the Puppet forge, with our glue built
into a profile on top of the third-party module. The role
`role::gitlab` follows that pattern correctly. See [issue 40030][] for
progress on that work.

### Hiera

[Hiera][] is a "key/value lookup tool for configuration data" which
Puppet uses to look up values for class parameters and node
configuration in General.

We are in the process of transitioning over to this mechanism from our
previous set of custom YAML lookup system. This documents the way we
currently use Hiera.

[Hiera]: https://puppet.com/docs/hiera/

#### Common configuration

Class parameters which are common across several or all roles can be
defined in `hiera/common.yaml` to avoid duplication at the role level.

However, unless this parameter can be expected to change or evolve over
time, it's sometimes preferable to hardcode some parameters directly in
profile classes in order to keep this dataset from growing too much,
which can impact performance of the Puppet server and degrade its
readability. In other words, it's OK to place site-specific data in
profile manifests, as long as it may never or very rarely change.

These parameters can be override by role and node configurations.

#### Role configuration

Class parameters specific to a certain node role are defined in
`hiera/roles/${::role}.yaml`. This is the principal method by which we
configure the various profiles, thus shaping each of the roles we
maintain.

These parameters can be override by node-specific configurations.

#### Node configuration

On top of the role configuration, some node-specific configuration can
be performed from Hiera. This should be avoided as much as possible,
but sometimes there is just no other way. A good example was the
`build-arm-*` nodes which included the following configuration:

    bacula::client::ensure: "absent"

This disables backups on those machines, which are normally configured
everywhere. This is done because they are behind a firewall and
therefore not reachable, an unusual condition in the network. Another
example is `nutans` which sits behind a NAT so it doesn't know its own
IP address. To export proper firewall rules, the allow address has
been overridden as such:

    bind::secondary::allow_address: 89.45.235.22

Those types of parameters are normally automatically guess inside
modules' classes, but they are overriddable from Hiera.

Note: eventually *all* host configuration will be done here, but there
are currently still some configurations hardcoded in individual
modules. For example, the Bacula director is hardcoded in the `bacula`
base class (in `modules/bacula/manifests/init.pp`). That should be
moved into a class parameter, probably in `common.yaml`.

### Cron and scheduling

The Puppet agent is *not* running as a daemon, it's running through
good old `cron`.

Puppet runs on each node every four hour, although with a random 2h
jitter, so the actual frequency is somewhere between 4 and 6
hours.

This configuration is in `/etc/cron.d/puppet-crontab` and deployed by
Puppet itself, currently as part of the `torproject_org` module.

### LDAP integration

The Puppet is configured to talk with Puppet through a few custom
functions defined in
`modules/puppetmaster/lib/puppet/parser/functions`. The main plumbing
function is called `ldapinfo()` and connects to the LDAP server
through `db.torproject.org` over TLS on port 636. It takes a hostname
as an argument and will load all hosts matching that pattern under the
`ou=hosts,dc=torproject,dc=org` subtree. If the specified hostname is
the `*` wildcard, the result will be a hash of `host => hash` entries,
otherwise only the `hash` describing the provided host will be
returned.

The `nodeinfo()` function uses that function to populate the global
`$nodeinfo` hash available globally, or, more specifically, the
`$nodeinfo['ldap']` component. It also loads the `$nodeinfo['hoster']`
value from the `whohosts()` function. That function, in turn, tries to
match the IP address of the host against the "hosters" defined in the
`hoster.yaml` file.

The `allnodeinfo()` function does a similar task as `nodeinfo()`,
except that it loads *all* nodes from LDAP, into a single hash. It
does *not* include the "hoster" and is therefore equivalent to calling
`nodeinfo()` on each host and extracting only the `ldap` member hash
(although it is not implemented that way).

Puppet does not require any special credentials to access the LDAP
server. It accesses the LDAP database anonymously, although there is a
firewall rule (defined in Puppet) that grants it access to the LDAP
server. 

There is a bootstrapping problem there: if one would be to rebuild the
Puppet server, it would actually fail to compile its catalog because
it would not be able to connect to the LDAP server to fetch its
catalog, unless the LDAP server has been manually configured to let
the Puppet server through.

NOTE: much (if not all?) of this is being moved into Hiera, in
particular the YAML files. See [issue 30020](https://trac.torproject.org/projects/tor/ticket/30020) for details. Moving
the host information into Hiera would resolve the bootstrapping
issues, but would require, in turn some more work to resolve questions
like how users get granted access to individual hosts, which is
currently managed by `ud-ldap`. We cannot, therefore, simply move host
information from LDAP into Hiera without creating a duplicate source
of truth without rebuilding or tweaking the user distribution
system. See also the [LDAP design document](ldap#Design) for more information
about how LDAP works.

### Nagios integration

Nagios (which is really Icinga, but let's call it Nagios because
that's how it's called everywhere in the source) is hooked into Puppet
through an external sync system. Our [Nagios deployment](nagios) operates
through Git hooks which run a special `Makefile` that compiles and
deploys the Icinga configuration, but also compiles the client-side
NRPE configuration.

The NRPE configuration is generated on the Nagios server and then
pushed to the Puppet server with `rsync` over SSH, using a public key
distributed by Puppet from the `roles::puppetmaster` class. That key
has a restricted `command` field which limits access to the Puppet
manifest, in this single file:

    /etc/puppet/modules/nagios/files/tor-nagios/generated/nrpe_tor.cfg

This file then gets distributed to all nodes through the
`nagios::client` class using a simple `File` resource.

So when a Nagios check is added or changed, Puppet needs to run on all
the affected host for the check to take affect, on top of, of course,
adding the check into the Nagios git repository.

### Let's Encrypt TLS certificates

Public TLS certificates, as issued by Let's Encrypted, are distributed
by Puppet. Those certificates are generated by the "letsencrypt" Git
repository (see the [TLS documentation](tls) for details on that
workflow). The relevant part, as far as Puppet is concerned, is that
certificates magically end up in the following directory when a
certificate is issued or (automatically) renewed:

    /srv/puppet.torproject.org/from-letsencrypt

See also the [TLS deployment docs](tls#lets-encrypt-workflow) for how that directory gets
populated.

Normally, those files would not be available from the Puppet
manifests, but the `ssl` Puppet module uses a special trick whereby
those files are read by Puppet `.erb` templates. For example, this is
how `.crt` files get generated on the Puppet master, in
`modules/ssl/templates/crt.erb`:

    <%=
      fn = "/srv/puppet.torproject.org/from-letsencrypt/#{@name}.crt"
      out = File.read(fn)
      out
    %>

Similar templates exist for the other files.

Those certificates should not be confused with the "auto-ca" TLS
certificates in use internally and which are deployed directly in
`/etc/puppet/modules/ssl/files/`, see below.

### Internal auto-ca TLS certificates

The Puppet server also manages an internal CA which we informally call
"auto-ca". Those certificates are internal in that they are used to
authenticate nodes to each other, not to the public. They are used, for
example, to encrypt connections between mail servers (in Postfix) and
[backup servers](backup) (in Bacula).

The auto-ca deploys those certificates directly inside the Puppet
server checkout, in `/etc/puppet/modules/ssl/files/certs/` and
`.../clientcerts/`. Details of that system are available in the [TLS documentation](tls#internal-auto-ca).

## Issues

There is no issue tracker specifically for this project, [File][] or
[search][] for issues in the [team issue tracker][search] component.

 [File]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/new
 [search]: https://gitlab.torproject.org/tpo/tpa/team/-/issues

## Monitoring and testing

Puppet is hooked into Nagios in two ways:

 * one job runs on the Puppetmaster and checks PuppetDB for
   reports. this was done with a [patched](https://github.com/evgeni/check_puppetdb_nodes/pull/14) version of the
   [check_puppetdb_nodes](https://github.com/evgeni/check_puppetdb_nodes/) Nagios check, now packaged inside the
   `tor-nagios-checks` Debian package
 * the same job actually runs twice; once to check all manifests, and
   another to check each host individually and assign the result to
   the right how

The twin checks are present so that we can find stray Puppet hosts,
for example if a host was retired from Nagios but not retired from
Puppet, or added to Nagios but not Puppet.

Note that we exclude some errors from the logs because we've been
having intermittent failures with PuppetDB since the Debian 10.12
"buster" point release on March 26 (see [issue
tpo/tpa/team#40699](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40699)). We speculate this issue will go away when the
PuppetDB package is fixed ([tpo/tpa/team#40707](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40707)).

The `check_puppetdb_nodes` was originally [deployed in March
2019](https://gitlab.torproject.org/tpo/tpa/team/-/issues/29676). An alternative check was the [check_puppet_agent](https://github.com/aswen/nagios-plugins/blob/master/check_puppet_agent) Nagios
check which has also recently (2022) been added to the
`tor-nagios.git` repository, but never actually used, as the puppetdb
check seems sufficient. It could, however, be used to replace the
above (to a certain extent) if we (for example) need to get rid of
PuppetDB for some reason.

An alternative implementation [using Prometheus](https://forge.puppet.com/puppet/prometheus_reporter) was considered but
[Prometheus still hasn't replaced Nagios](https://gitlab.torproject.org/tpo/tpa/team/-/issues/29864) at the time of writing.

There are no validation checks and *a priori* no peer review of code:
code is directly pushed to the Puppet server without validation. Work
is being done to [implement automated checks](https://gitlab.torproject.org/tpo/tpa/team/-/issues/31226) but that is only
being deployed on some clients for now.

## Logs and metrics

PuppetDB itself holds performance information about the Puppet agent
runs, which are called "reports". Those reports contain information
about changes operated on each server, how long the agent runs take
and so on. Those metrics could be made more visible by using a
dashboard, but that has not been implemented yet (see [issue
31969][]).

[issue 31969]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/31969

The Puppet server, Puppet agents and PuppetDB keep logs of their
operations. The latter keeps its logs in `/var/log/puppetdb/` for a
maximum of 90 days or 1GB, whichever comes first (configured in
`/etc/puppetdb/request-logging.xml` and
`/etc/puppetdb/logback.xml`). The other logs are sent to `syslog`, and
usually end up in `daemon.log`.

Puppet should hold minimal personally identifiable information, like
user names, user public keys and project names.

## Other documentation

 * [Latest Puppet docs](https://puppet.com/docs/puppet/latest/puppet_index.html) - might be too new, see also the [Puppet
   5.5 docs](https://puppet.com/docs/puppet/5.5/puppet_index.html)
   * [Function reference](https://puppet.com/docs/puppet/latest/function.html)
   * [Type reference](https://puppet.com/docs/puppet/latest/type.html)
 * [Mapping between versions of Puppet Entreprise, Facter, Hiera, Agent, etc](https://puppet.com/docs/pe/2019.0/component_versions_in_recent_pe_releases.html)

# Discussion

This section goes more in depth into how Puppet is setup, why it was
setup the way it was, and how it could be improved.

## Overview

Our Puppet setup dates back from 2011, according to the git history,
and was probably based off the [Debian System Administrator's Puppet
codebase](https://salsa.debian.org/dsa-team/mirror/dsa-puppet) which dates back to 2009.

## Goals

The general goal of Puppet is to provide basic automation across the
architecture, so that software installation and configuration, file
distribution, user and some service management is done from a central
location, managed in a git repository. This approach is often called
[Infrastructure as code](https://en.wikipedia.org/wiki/Infrastructure_as_Code).

This section also documents possible improvements to our Puppet
configuration that we are considering.

### Must have

 * **secure**: only sysadmins should have access to push configuration,
   whatever happens. this includes deploying only audited and verified
   Puppet code into production.
 * **code review**: changes on servers should be verifiable by our peers,
   through a git commit log
 * **fix permissions issues**: deployment system should allow all admins
   to push code to the puppet server without having to constantly fix
   permissions (e.g. trough a [role account](https://gitlab.torproject.org/tpo/tpa/team/-/issues/29663))
 * **secrets handling**: there are some secrets in Puppet. those
   should remain secret.

We mostly have this now, although there are concerns about permissions
being wrong sometimes, which a role account could fix.

### Nice to have

Those are mostly issues with the current architecture we'd like to fix:

 * **Continuous Integration**: before deployment, code should be vetted by
   a peer and, ideally, automatically checked for errors and tested
 * **single source of truth**: when we add/remove nodes, we should not
   have to talk to multiple services (see also the [install automation
   ticket](https://gitlab.torproject.org/tpo/tpa/team/-/issues/31239) and the [new-machine discussion](new-machine#discussion)
 * **collaboration** with other sysadmins outside of TPA, for which we
   would need to...
 * ... **publicize our code** (see [ticket 29387](https://gitlab.torproject.org/tpo/tpa/team/-/issues/29387))
 * **no manual changes**: every change on every server should be committed
   to version control somewhere
 * **bare-metal recovery**: it should be possible to recover a service's
   *configuration* from a bare Debian install with Puppet (and with
   data from the [backup](backup) service of course...)
 * **one commit only**: we shouldn't have to commit "twice" to get
   changes propagated (once in a submodule, once in the parent module,
   for example)

### Non-Goals

 * **ad hoc changes** to the infrastructure. one-off jobs should be
   handled by [fabric](fabric), Cumin, or straight SSH.

## Approvals required

TPA should approve policy changes as per [tpa-rfc-1](/policy/tpa-rfc-1-policy).

## Proposed Solution

To improve on the above "Goals", I would suggest the following
configuration.

TL;DR:

 1. Use a control repository
 2. Get rid of 3rdparty
 3. Deploy with g10k
 4. Authenticate with checksums
 5. Deploy to branch-specific environments
 6. Rename the default branch "production"
 7. Push directly on the Puppet server
 8. Use a role account
 9. Use local test environments
 10. Develop a test suite
 11. Hook into CI
 12. OpenPGP verification and web hook

Steps 1-8 could be implemented without too much difficulty and should
be a mid term objective. Steps 9 to 12 require significantly more work
and could be implemented once the new infrastructure stabilizes.

What follows is an explanation and justification of each step.

### Use a control repository

The base of the infrastructure is a [control-repo](https://puppet.com/docs/pe/latest/control_repo.html) ([example](https://github.com/puppetlabs/control-repo),
[another more complex example](https://github.com/example42/psick))
which chain-loads all the other modules. This implies turning all our
"modules" into "profiles" and moving "real" modules (which are fit for
public consumption) "outside", into public repositories (see also
[issue 29387: publish our puppet repository](https://gitlab.torproject.org/tpo/tpa/team/-/issues/29387)).

Note that the control repository *could* also be public: we could
simply have the private data inside of Hiera or some other private
repository.

The control repository concept is specific to the proprietary version
of Puppet (Puppet Enterprise or PE) but its logic should be usable
with the open source Puppet release as well.

### Get rid of 3rdparty

The control repo's core configuration file is the `Puppetfile`. We
already use a Puppetfile, but only to manage modules inside of the
`3rdparty` directory. Now it would manage *all* modules, or, more
specifically, `3rdparty` would become the default `modules` directory
which would, incidentally, encourage us to upstream our modules and
publish them to the world.

Our current `modules` directory would move into `site-modules`, which
is the designated location for "roles, profiles, and custom
modules". This has been suggested before in [issue 29387: publish our
puppet repository](https://gitlab.torproject.org/tpo/tpa/team/-/issues/29387)) and is important for the `Puppetfile` to do its
job.

### Deploy with g10k

It seems clear that everyone is converging over the use of a
`Puppetfile` to deploy code. While there are still monorepos out
there, but they do make our life harder, especially when we need to
operate on non-custom modules.

Instead, we should converge towards *not* following upstream modules
in our git repository. Modules managed by the `Puppetfile` would *not*
be managed in our git monorepo and, instead, would be deployed by
`r10k` or `g10k` (most likely the latter because of its support for
checksums).

Note that neither `r10k` or `g10k` resolve dependencies in a
`Puppetfile`. We therefore also need a tool to verify the file
correctly lists all required modules. The following solutions need to
be validated but could address that issue:

 * [generate-puppetfile](https://github.com/rnelson0/puppet-generate-puppetfile): take a `Puppetfile` and walk the
   dependency tree, generating a new `Puppetfile` (see also [this
   introduction to the project](https://rnelson0.com/2015/11/06/introducing-generate-puppetfile-or-creating-a-ruby-program-to-update-your-puppetfile-and-fixtures-yml/))
 * [Puppetfile-updater](https://github.com/camptocamp/puppetfile-updater): read the `Puppetfile` and fetch new releases
 * [ra10ke](https://github.com/voxpupuli/ra10ke): a bunch of Rake tasks to validate a `Puppetfile`
   * `r10k:syntax`: syntax check, see also `r10k puppetfile check`
   * `r10k:dependencies`: check for out of date dependencies
   * `r10k:solve_dependencies`: check for **missing** dependencies
   * `r10k:install`: wrapper around `r10k` to install with some
     caveats
   * `r10k:validate`: make sure modules are accessible
   * `r10k:duplicates`: look for duplicate declarations
 * [lp2r10k](https://github.com/dharmabruce/lp2r10k/): convert "librarian" `Puppetfile` (missing
   dependencies) into a "r10k" `Puppetfile` (with dependencies)

Note that this list comes from the [updating your Puppetfile](https://github.com/puppetlabs/r10k/blob/master/doc/updating-your-puppetfile.mkd#automatic-updates)
documentation in the r10k project, which is also relevant here.

### Authenticate code with checksums

This part is the main problem with moving away from a monorepo. By
using a monorepo, we can audit the code we push into production. But
if we offload this to `r10k`, it can download code from wherever the
`Puppetfile` says, effectively shifting our trust path from OpenSSH
to HTTPS, the Puppet Forge, git and whatever remote gets added to the
`Puppetfile`.

There is no obvious solution for this right now, surprisingly. Here
are two possible alternatives:

 1. [g10k](https://github.com/xorpaul/g10k/) supports using a `:sha256sum` parameter to checksum
    modules, but that only works for Forge modules. Maybe we could
    pair this with using an explicit `sha1` reference for git
    repository, ensuring those are checksummed as well. The downside
    of that approach is that it leaves checked out git repositories in
    a "detached head" state.

 2. `r10k` has a [pending pull request](https://github.com/puppetlabs/r10k/pull/823) to add a `filter_command`
    directive which could run after a git checkout has been
    performed. it could presumably be used to verify OpenPGP
    signatures on git commits, although this would work only on
    modules we sign commits on (and therefore not third party)

It seems the best approach would be to use g10k for now with checksums
on both git commit and forge modules.

A validation hook running *before* g10k COULD validate that all `mod`
lines have a `checksum` of some sort...

Note that this approach does *NOT* solve the "double-commit" problem
identified in the Goals. It is believed that only a "monorepo" would
fix that problem and that approach comes in direct conflict with the
"collaboration" requirement. We chose the latter.

This could be implemented as a patch to `ra10ke`.

### Deploy to branch-specific environments

A key feature of r10k (and, of course, g10k) is that they are capable
of deploying code to new environments depending on the branch we're
working on. We would enable that feature to allow testing some large
changes to critical code paths without affecting all servers.

### Rename the default branch "production"

In accordance with Puppet's best practices, the control repository's
default branch would be called "production" and not "master".

Also: Black Lives Matter.

### Push directly on the Puppet server

Because we are worried about the GitLab attack surface, we could still
keep on pushing to the Puppet server for now. The control repository
could be mirrored to GitLab using a deploy key. All other repositories
would be published on GitLab anyways, and there the attack surface
would not matter because of the checksums in the control repository.

### Use a role account

To avoid permission issues, use a role account (say `git`) to accept
pushes and enforce git hooks.

### Use local test environments

It should eventually be possible to test changes locally before
pushing to production. This would involve radically simplifying the
Puppet server configuration and probably either getting rid of the
LDAP integration or at least making it optional so that changes can be
tested without it.

This would involve "puppetizing" the Puppet server configuration so
that a Puppet server and test agent(s) could be bootstrapped
automatically. Operators would run "smoke tests" (running Puppet by
hand and looking at the result) to make sure their code works before
pushing to production.

### Develop a test suite

The next step is to start working on a test suite for services, at
least for new deployments, so that code can be tested without running
things by hand. Plenty of Puppet modules have such test suite,
generally using [rspec-puppet](https://rspec-puppet.com/) and [rspec-puppet-facts](https://github.com/mcanevet/rspec-puppet-facts), and we
already have a few modules in `3rdparty` that have such tests. The
idea would be to have those tests on a per-role or per-profile basis.

The Foreman people have published [their test infrastructure](https://github.com/theforeman/foreman-infra/tree/master/puppet) which
could be useful as inspiration for our purposes here.

### Hook into continuous integration

Once tests are functional, the last step is to move the control
repository into GitLab directly and start running CI against the
Puppet code base. This would probably not happen until GitLab CI is
deployed, and would require lots of work to get there, but would
eventually be worth it.

The GitLab CI would be indicative: an operator would need to push to a
topic branch there first to confirm tests pass but would still push
directly to the Puppet server for production.

Note that we are working on (client-side) validation hooks for now,
see [issue 31226][].

[issue 31226]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/31226

### OpenPGP verification and web hook

To stop pushing directly to the Puppet server, we could implement
OpenPGP verification on the control repository. If a hook checks that
commits are signed by a trusted party, it does not matter where the
code is hosted.

A good reference for OpenPGP verification is [this guix article](https://guix.gnu.org/blog/2020/securing-updates/)
which covers a few scenarios and establishes a pretty solid
verification workflow. There's also a larger project-wide discussion
in [GitLab](howto/gitlab) [issue 81](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/81).

We could use the [webhook](https://github.com/voxpupuli/puppet_webhook) system to have GitLab notify the Puppet
server to pull code.

## Cost

N/A.

## Alternatives considered

Ansible was considered for managing [GitLab](gitlab) for a while, but
this was eventually abandoned in favor of using Puppet and the
"Omnibus" package.

For ad hoc jobs, [fabric](fabric) is being used.

For code management, I have done a more extensive review of possible
alternatives. [This talk](https://www.youtube.com/watch?v=RdIyStATgFE) is a good introduction for git submodule,
librarian and r10k. Based on that talk and [these slide](https://arlimus.github.io/slides/librarian.and.r10k/), I've made
the following observations:

### monorepo

This is our current approach, which is that all code is committed in
one monolithic repository. This effectively makes it impossible to
share code outside of the repository with anyone else because there is
private data inside, but also because it doesn't follow the standard
role/profile/modules separation that makes collaboration possible at
all. To work around that, I designed a workflow where we locally clone
subrepos as needed, but this is clunky as it requires to commit every
change twice: one for the subrepo, one for the parent.

Our giant monorepo also mixes all changes together which can be an pro
*and* a con: on the one hand it's easy to see and audit all changes at
once, but on the other hand, it can be overwhelming and confusing.

But it does allow us to integrate with librarian right now and is a
good stopgap solution. A better solution would need to solve the
"double-commit" problem and still allow us to have smaller
repositories that we can collaborate on outside of our main tree.

### submodules

The talk partially covers how difficult `git submodules` work and how
hard they are to deal with. I say partially because submodules are
even harder to deal with than the examples she gives. She shows how
submodules are hard to add and remove, because the metadata is stored
in stored in multiple locations (`.gitsubmodules`, `.git/config`,
`.git/modules/` and the submodule repository itself).

She also mentions submodules don't know about dependencies and it's
likely you will break your setup if you forget one step. (See [this
post](https://web.archive.org/web/20171101202911/http://somethingsinistral.net/blog/git-submodules-are-probably-not-the-answer/) for more examples.)

In my experience, the biggest annoyance with submodules is the
"double-commit" problem: you need to make commits in the submodule,
then *redo* the commits in the parent repository to chase the head of
that submodule. This does not improve on our current situation, which
is that we need to do those two commits anyways in our giant monorepo.

One advantage with submodules is that they're mostly standard:
everyone knows about them, even if they're not familiar and their
knowledge is reusable outside of Puppet.

### librarian

Librarian is written in ruby. It's built on top of [another library
called librarian](https://github.com/applicationsonline/librarian) that is used by Ruby's [bundler](https://gembundler.com/). At the time
of the talk, was "pretty active" but unfortunately, librarian now
seems to be [abandoned](https://github.com/voxpupuli/librarian-puppet/issues/48) so we might be forced to use r10k in the
future, which has a quite different workflow.

One problem with librarian right now is that `librarian update` clears
any existing git subrepo and re-clones it from scratch. If you have
temporary branches that were not pushed remotely, all of those are
lost forever. That's really bad and annoying! it's by design: it
"takes over your modules directory", as she explains in the talk and
everything comes from the Puppetfile.

Librarian does resolve dependencies recursively and store the decided
versions in a lockfile which allow us to "see" what happens when you
update from a Puppetfile.

But there's no cryptographic chain of trust between the repository
where the Puppetfile is and the modules that are checked out. Unless
the module is checked out from git (which isn't the default), only
version range specifiers constrain which code is checked out, which
gives a huge surface area for arbitrary code injection in the entire
puppet infrastructure (e.g. MITM, forge compromise, hostile upstream
attacks)

### r10k

r10k was written because librarian was too slow for large
deployments. But it covers more than just managing code: it also
manages environments and is designed to run on the Puppet master. It
doesn't have dependency resolution or a `Puppetfile.lock`,
however. See [this ticket](https://github.com/puppetlabs/r10k/issues/38), closed in favor of [that one](https://tickets.puppetlabs.com/browse/RK-3).

r10k is more complex and very opiniated: it requires lots of
configuration including its own YAML file, hooks into the Puppetmaster
and can [take a while to deploy](http://garylarizza.com/blog/2014/02/18/puppet-workflow-part-3/). r10k is still in [active