... | ... | @@ -74,7 +74,7 @@ sysadmin team. |
|
|
If this works, congratulations, you have made your first change across
|
|
|
the entire Puppet infrastructure! You might want to look at the rest
|
|
|
of the documentation to learn more about how to do different tasks and
|
|
|
how things are setup. A key Howto we recommend is the `Progressive
|
|
|
how things are setup. A key "How to" we recommend is the `Progressive
|
|
|
deployment` section below, which will teach you how to make a change
|
|
|
like the above while making sure you don't break anything even if it
|
|
|
affects a lot of machines.
|
... | ... | @@ -83,7 +83,7 @@ affects a lot of machines. |
|
|
|
|
|
## Modifying an existing configuration
|
|
|
|
|
|
For new deployments, this is *NOT* the prefered method. For example,
|
|
|
For new deployments, this is *NOT* the preferred method. For example,
|
|
|
if you are deploying new software that is not already in use in our
|
|
|
infrastructure, do *not* follow this guide and instead follow the
|
|
|
`Adding a new module` guide below.
|
... | ... | @@ -126,7 +126,7 @@ Next, the module was added to the Puppetfile (in |
|
|
This fetched a lot of code from the Puppet forge: the stdlib, archive
|
|
|
and system modules were all installed or updated. All those modules
|
|
|
were audited manually, by reading each file and looking for obvious
|
|
|
security flaws or backdoors. Then the code was committed into git:
|
|
|
security flaws or back doors. Then the code was committed into git:
|
|
|
|
|
|
git add 3rdparty
|
|
|
git commit -m'install prometheus module after audit'
|
... | ... | @@ -178,7 +178,7 @@ code in `3rdparty/modules/prometheus`, and committed into git: |
|
|
git commit -m'implement all the missing stuff' 3rdparty
|
|
|
git push
|
|
|
|
|
|
And the above puppet commandline was ran again, continuing that loop
|
|
|
And the above puppet command-line was ran again, continuing that loop
|
|
|
until things were good.
|
|
|
|
|
|
If you need to deploy the code to multiple hosts, see the `Progressive
|
... | ... | @@ -188,7 +188,7 @@ you should do so), see the section right below. |
|
|
## Contributing changes back upstream
|
|
|
|
|
|
For simple changes, the above workflow works well, but eventually it
|
|
|
is preferable to actually fork the upstream repo and operate on our
|
|
|
is preferable to actually fork the upstream repository and operate on our
|
|
|
fork until the changes are merged upstream.
|
|
|
|
|
|
First, the modified module is moved out of the way:
|
... | ... | @@ -221,7 +221,7 @@ request was sent. |
|
|
|
|
|
Since you now have a clone of the upstream repository, you can push
|
|
|
and pull normally with upstream. When you make a change, however, you
|
|
|
need to commit (and push) the change *both* in the sub-repo and the
|
|
|
need to commit (and push) the change *both* in the sub-repository and the
|
|
|
main repository:
|
|
|
|
|
|
cd 3rdparty/modules/prometheus
|
... | ... | @@ -297,7 +297,7 @@ value: |
|
|
|
|
|
ssh -t pauli.torproject.org "sudo -u postgres psql puppetdb -P pager=off -F',' -A -t -c \"SELECT c.certname, value_string FROM factsets fs INNER JOIN facts f ON f.factset_id = fs.id INNER JOIN fact_values fv ON fv.id = f.fact_value_id INNER JOIN fact_paths fp ON fp.id = f.fact_path_id INNER JOIN certnames c ON c.certname = fs.certname WHERE fp.name = 'virtual' AND c.deactivated IS NULL\"" | tee hosts.csv
|
|
|
|
|
|
The resulting file is a Comma-Seperated Value (CSV) file which can be
|
|
|
The resulting file is a Comma-Separated Value (CSV) file which can be
|
|
|
used for other purposes later.
|
|
|
|
|
|
Possible values of the `virtual` field can be obtain with a similar
|
... | ... | @@ -307,7 +307,7 @@ query: |
|
|
|
|
|
The currently known values are: `kvm`, `physical`, and `xenu`.
|
|
|
|
|
|
As a bonus, this query will show the number of hosts running each os release:
|
|
|
As a bonus, this query will show the number of hosts running each release:
|
|
|
|
|
|
SELECT COUNT(c.certname), value_string FROM factsets fs INNER JOIN facts f ON f.factset_id = fs.id INNER JOIN fact_values fv ON fv.id = f.fact_value_id INNER JOIN fact_paths fp ON fp.id = f.fact_path_id INNER JOIN certnames c ON c.certname = fs.certname WHERE fp.name = 'lsbdistcodename' AND c.deactivated IS NULL GROUP BY value_string;
|
|
|
|
... | ... | @@ -343,7 +343,7 @@ As a bonus, this query will show the number of hosts running each os release: |
|
|
HOSTS=$(ssh alberti.torproject.org 'ldapsearch -h db.torproject.org -x -ZZ -b dc=torproject,dc=org -LLL "hostname=*.torproject.org" hostname | awk "\$1 == \"hostname:\" {print \$2}" | sort')
|
|
|
for i in `echo $HOSTS`; do mkdir hosts/x-$i 2>/dev/null || continue; echo $i; ssh $i ' ...'; done
|
|
|
|
|
|
the mkdir is so that I can run the same command in many terminal
|
|
|
the `mkdir` is so that I can run the same command in many terminal
|
|
|
windows and each host gets only one once
|
|
|
|
|
|
[PuppetDB API]: https://puppet.com/docs/puppetdb/4.3/api/index.html
|
... | ... | @@ -352,7 +352,7 @@ As a bonus, this query will show the number of hosts running each os release: |
|
|
## Batch jobs on all hosts
|
|
|
|
|
|
With that trick, a job can be ran on all hosts with
|
|
|
[parallel-ssh][], for example, check the uptime:
|
|
|
[parallel-ssh][], for example, check the `uptime`:
|
|
|
|
|
|
cut -d, -f1 hosts.hsv | parallel-ssh -i -h /dev/stdin uptime
|
|
|
|
... | ... | @@ -364,7 +364,7 @@ This would fetch the `/etc/motd` on all machines: |
|
|
|
|
|
cut -d -f1 hosts.csv | parallel-slurp -h /dev/stdin -L motd /etc/motd motd
|
|
|
|
|
|
To run batch commands through sudo that requires a password, you will need to fool both sudo and ssh a little more:
|
|
|
To run batch commands through `sudo` that requires a password, you will need to fool both `sudo` and ssh a little more:
|
|
|
|
|
|
cut -d -f1 hosts.csv | parallel-ssh -P -I -i -x -tt -h /dev/stdin -o pvs sudo pvs
|
|
|
|
... | ... | @@ -474,7 +474,7 @@ see why they do not work correctly (a common problem): |
|
|
|
|
|
puppet agent -t --debug
|
|
|
|
|
|
Finally, some errors show up only on the Puppetmaster: you can look in
|
|
|
Finally, some errors show up only on the Puppet server: you can look in
|
|
|
`/var/log/daemon.log` there for errors that will only show up there.
|
|
|
|
|
|
Connecting to the PuppetDB database itself can sometimes be easier
|
... | ... | @@ -503,7 +503,7 @@ publish that source code publicly. |
|
|
|
|
|
We have two mechanisms on how to do this now: a HKDF to generate
|
|
|
passwords by hashing a common secret, and Trocla, which generates
|
|
|
random passwords and stores the hash or, if necessary, the cleartext
|
|
|
random passwords and stores the hash or, if necessary, the clear-text
|
|
|
in a YAML file.. The HKDF function is deprecated and should be
|
|
|
[replaced by Trocla][trocla-migration] eventually.
|
|
|
|
... | ... | @@ -558,13 +558,13 @@ Grafana admin, for example: |
|
|
|
|
|
$grafana_admin_password = trocla('grafana_admin_password', 'bcrypt')
|
|
|
|
|
|
The plaintext for that password will never leave the Puppet master. it
|
|
|
The plain-text for that password will never leave the Puppet master. it
|
|
|
will still be stored on the Puppet master, and you can see the value
|
|
|
with:
|
|
|
|
|
|
trocla get grafana_admin_password plain
|
|
|
|
|
|
... on the commandline.
|
|
|
... on the command-line.
|
|
|
|
|
|
[bcrypt]: https://en.wikipedia.org/wiki/Bcrypt
|
|
|
|
... | ... | @@ -577,7 +577,7 @@ those will get regenerated as needed. |
|
|
|
|
|
Also note that `trocla get` will fail if the particular password or
|
|
|
format requested does not exist. For example, say you generate a
|
|
|
plaintext password with and then get the `bcrypt` version:
|
|
|
plain-text password with and then get the `bcrypt` version:
|
|
|
|
|
|
trocla create test plain
|
|
|
trocla get test bcrypt
|
... | ... | @@ -656,7 +656,7 @@ LDAP especially need to be documented. |
|
|
|
|
|
## SLA
|
|
|
|
|
|
No formal SLA is defined. Puppet runs on a fairly slow cron job so
|
|
|
No formal SLA is defined. Puppet runs on a fairly slow `cron` job so
|
|
|
doesn't have to be highly available right now. This could change in
|
|
|
the future if we rely more on it for deployments.
|
|
|
|
... | ... | @@ -673,8 +673,8 @@ the future if we rely more on it for deployments. |
|
|
### Before it all starts
|
|
|
|
|
|
- `puppet.tpo` is currently being run on `pauli.tpo`
|
|
|
- This is where the tor-puppet git repo lives
|
|
|
- The repo has hooks to populate `/etc/puppet` with its contents, most
|
|
|
- This is where the tor-puppet git repository lives
|
|
|
- The repository has hooks to populate `/etc/puppet` with its contents, most
|
|
|
notably the modules directory.
|
|
|
- All paths in this document are relative to the root of this
|
|
|
repository.
|
... | ... | @@ -699,7 +699,7 @@ the future if we rely more on it for deployments. |
|
|
(`modules/torproject_org/manifests/init.pp`) performs basic host
|
|
|
initialisation, like configuring Debian mirrors and APT sources,
|
|
|
installing a base set of packages, configuring puppet and timezone,
|
|
|
setting up a bunch of rc-files and running ud-replicate.
|
|
|
setting up a bunch of configuration files and running `ud-replicate`.
|
|
|
|
|
|
- In there, `local.yaml` (`modules/torproject_org/misc/local.yaml`)
|
|
|
defines services and list which host(s) supply each
|
... | ... | @@ -715,16 +715,16 @@ the future if we rely more on it for deployments. |
|
|
- the `nodeinfo()` function
|
|
|
(`modules/puppetmaster/lib/puppet/parser/functions/nodeinfo.rb`),
|
|
|
used for setting up the `$nodeinfo` variable
|
|
|
- ferm's `def.conf` template (`modules/ferm/templates/defs.conf.erb`)
|
|
|
- the `entropy provider`
|
|
|
(`modules/puppetmaster/lib/puppet/parser/functions/entropy_provider.rb`)
|
|
|
TODO
|
|
|
- `ferm`'s `def.conf` template (`modules/ferm/templates/defs.conf.erb`)
|
|
|
|
|
|
- The root of definitions and execution is in Pupept is found in
|
|
|
- The root of definitions and execution is in Puppet is found in
|
|
|
the `manifests/site.pp` file, but this file is now mostly empty, in
|
|
|
favor of Hiera.
|
|
|
|
|
|
Note that the above is the current state of the file hierachy. As part
|
|
|
Note that the above is the current state of the file hierarchy. As part
|
|
|
of the transition to Hiera, a lot of the above architecture will
|
|
|
change in favor of the more standard [role/profile/module][]
|
|
|
pattern. See [ticket #29387][] for an in-depth discussion.
|
... | ... | @@ -736,7 +736,7 @@ pattern. See [ticket #29387][] for an in-depth discussion. |
|
|
|
|
|
`modules/torproject_org/lib/facter/software.rb` defines our custom
|
|
|
facts, making it possible to get answer to questions like "Is this
|
|
|
host running apache2?" byt simply looking at a puppet variable.
|
|
|
host running `apache2`?" by simply looking at a puppet variable.
|
|
|
|
|
|
### Style guide
|
|
|
|
... | ... | @@ -748,7 +748,7 @@ Many files do not *currently* follow the style guide, as they |
|
|
*predate* the creation of said guide. Files should *not* be completely
|
|
|
reformatted unless there's a good reason. For example, if a
|
|
|
conditional covering a large part of a file is removed and the file
|
|
|
needs to be reindented, it's a good opportunity to fix style in the
|
|
|
needs to be re-indented, it's a good opportunity to fix style in the
|
|
|
file. Same if a file is split in two components or for some other
|
|
|
reason completely rewritten.
|
|
|
|
... | ... | @@ -764,7 +764,7 @@ Otherwise the style already in use in the file should be followed. |
|
|
Puppet uses to look up values for class parameters and node
|
|
|
configuration in General.
|
|
|
|
|
|
We are in the process of transitionning over to this mecanism from our
|
|
|
We are in the process of transitioning over to this mechanism from our
|
|
|
previous set of custom YAML lookup system. This documents the way we
|
|
|
currently use Hiera.
|
|
|
|
... | ... | @@ -804,12 +804,12 @@ everywhere. This is done because they are behind a firewall and |
|
|
therefore not reachable, an unusual condition in the network. Another
|
|
|
example is `nutans` which sits behind a NAT so it doesn't know its own
|
|
|
IP address. To export proper firewall rules, the allow address has
|
|
|
been overriden as such:
|
|
|
been overridden as such:
|
|
|
|
|
|
bind::secondary::allow_address: 89.45.235.22
|
|
|
|
|
|
Those types of parameters are normally automatically guess inside
|
|
|
modules' classes, but they are overridable from Hiera.
|
|
|
modules' classes, but they are overriddable from Hiera.
|
|
|
|
|
|
Note: eventually *all* host configuration will be done here, but there
|
|
|
are currently still some configurations hardcoded in individual
|
... | ... | @@ -855,7 +855,7 @@ This was [implemented in March 2019](https://gitlab.torproject.org/tpo/tpa/team/ |
|
|
replaced Nagios](https://gitlab.torproject.org/tpo/tpa/team/-/issues/29864) at the time of writing.
|
|
|
|
|
|
There are no validation checks and *a priori* no peer review of code:
|
|
|
code is directly pushed to the puppetmaster without validation. Work
|
|
|
code is directly pushed to the Puppet server without validation. Work
|
|
|
is being done to [implement automated checks](https://gitlab.torproject.org/tpo/tpa/team/-/issues/31226) but that is only
|
|
|
being deployed on some clients for now.
|
|
|
|
... | ... | |