Skip to content
Snippets Groups Projects
Commit a508e13d authored by zen's avatar zen
Browse files

update Puppet doc to account for removal of the 3rdparty/ dir 

refs team#41974
parent 66312579
No related branches found
No related tags found
No related merge requests found
Pipeline #255032 failed
Stage: build
Stage: test
Hide whitespace changes
Inline Side-by-side
howto/new-machine-hetzner-robot.md
+ 1
− 1
View file @ a508e13d
......@@ -271,7 +271,7 @@ which we'll use below.
fi
TODO: add the above as a post-hook. possibly merge with
`tor-puppet/3rdparty/modules/ganeti/files/instance-debootstrap/hooks/gnt-debian-interfaces`
`tor-puppet/modules/ganeti/files/instance-debootstrap/hooks/gnt-debian-interfaces`
TODO: add IPv6 address configuration. look at how
`tor-install-generate-ldap` guesses as well.
......
howto/puppet.md
+ 40
− 97
View file @ a508e13d
......@@ -223,21 +223,29 @@ more downloads than the [next option][], which was deprecated.
[Prometheus modules on the Puppet forge]: https://forge.puppet.com/modules?q=prometheus
Next, the module was added to the Puppetfile (in
`3rdparty/Puppetfile`):
`./Puppetfile`):
mod 'puppet-prometheus', '6.4.0'
mod 'puppet/prometheus', # 12.5.0
:git => 'https://github.com/voxpupuli/puppet-prometheus.git',
:commit => '25dd701b489fc32c892390fd464e765ebd6f513a' # tag: v12.5.0
... and librarian was ran:
Note that:
librarian-puppet install
- Since tpo/tpa/team#41974 we don't import 3rd-party code into our repo and
instead deploy the modules dynamicaly in the server.
- Because of that, modules in the `Puppetfile` should always be pinned to a Git
repo and commit, as that's currently the simplest way to avoid some MITM
issues.
- We currently don't have an automated way of managing module dependencies, so
you'll have to manually and recursively add dependencies to the `Puppetfile`.
Sorry!
- Make sure to manually audit the code for each module, by reading each file
and looking for obvious security flaws or back doors.
This fetched a lot of code from the Puppet forge: the stdlib, archive
and system modules were all installed or updated. All those modules
were audited manually, by reading each file and looking for obvious
security flaws or back doors. Then the code was committed into git:
Then the code was committed into git:
git add 3rdparty
git commit -m'install prometheus module after audit'
git add Puppetfile
git commit -m'install prometheus module and its dependencies after audit'
Then the module was configured in a profile, in `modules/profile/manifests/prometheus/server.pp`:
......@@ -251,7 +259,7 @@ Then the module was configured in a profile, in `modules/profile/manifests/prome
}
The above contains our local configuration for the upstream
`prometheus::server` class installed in the `3rdparty` directory. In
`prometheus::server` class. In
particular, it sets a retention period and a different path for the
metrics, so that they follow the new Prometheus 2.x defaults.
......@@ -278,97 +286,31 @@ And Puppet was ran on the host, with:
puppet --enable ; puppet agent -t --noop ; puppet --disable "testing prometheus deployment"
This led to some problems as the upstream module doesn't support
installing from Debian packages. Support for Debian was added to the
code in `3rdparty/modules/prometheus`, and committed into git:
emacs 3rdparty/modules/prometheus/manifests/*.pp # magic happens
git commit -m'implement all the missing stuff' 3rdparty
git push
And the above puppet command-line was ran again, continuing that loop
until things were good.
If you need to deploy the code to multiple hosts, see the `Progressive
deployment` section below. To contribute changes back upstream (and
you should do so), see the section right below.
### Contributing changes back upstream
For simple changes, the above workflow works well, but eventually it
is preferable to actually fork the upstream repository and operate on our
fork until the changes are merged upstream.
First, the modified module is moved out of the way:
mv 3rdparty/modules/prometheus{,.orig}
Fork the upstream repository and operate on your fork until the changes are
eventually merged upstream.
The module is then forked on GitHub or wherever it is hosted, and then
added to the Puppetfile:
Then, update the `Puppetfile`, for example:
The module is then forked on GitHub or wherever it is hosted
mod 'puppet-prometheus',
:git => 'https://github.com/anarcat/puppet-prometheus.git',
:branch => 'deploy'
Then Librarian is ran again to fetch that code:
librarian-puppet install
Because Librarian is a little dumb, it might checkout your module in
"detached head" mode, in which case you will want to fix the checkout:
cd 3rdparty/modules/prometheus
git checkout deploy
git reset --hard origin/deploy
git pull
:commit => '(...)'
Note that the `deploy` branch here is a merge of all the different
branches proposed upstream in different pull requests, but it could
also be the `master` branch or a single branch if only a single pull
request was sent.
Since you now have a clone of the upstream repository, you can push
and pull normally with upstream. When you make a change, however, you
need to commit (and push) the change *both* in the sub-repository and the
main repository:
cd 3rdparty/modules/prometheus
$EDITOR manifests/init.pp # more magic stuff
git commit -m'change the frobatz to a argblu'
git push
cd ..
git commit -m'change the frobatz to a argblu'
git push
Often, I make commits directly in our main Puppet repository, without
pushing to the third party fork, until I am happy with the code, and
then I craft a nice pretty commit that can be pushed upstream,
reversing that process:
$EDITOR 3rdparty/prometheus/manifests/init.pp # dirty magic stuff
git commit -m'change the frobatz to a quuxblah'
git push
# see if that works, generally not
git commit -m'rah. wanted a quuxblutz'
git push
# now we are good, update our pull request
cd 3rdparty/modules/prometheus
git commit -m'change the frobatz to a quuxblutz'
git push
It's annoying to double-commit things, but I haven't found a best way
to do so just yet. This problem is further discussed in [ticket #29387][].
Also note that when you update code like this, the `Puppetfile` does
not change, but the `Puppetfile.lock` file *does* change. The `GIT.sha`
parameter needs to be updated. This can be done by hand, but since
that is error-prone, you might want to simply run this to update
modules:
librarian-puppet update
This will *also* update dependencies so make sure you audit those
changes before committing and pushing.
You'll have to keep a clone of the upstream repository somewhere outside of the
`tor-puppet` work tree, from which you can push and pull normally with
upstream. When you make a change, you need to commit (and push) the change in
your external clone and update the `Puppetfile` in the repository.
### Running tests
......@@ -1621,7 +1563,8 @@ classes, see [issue 40030][] for progress on this work.
Environments on the Puppet Server are managed using `tor-puppet.git` which is
our "control repository". Each branch on this repo is mapped to an environment
on the server which takes the name of the branch.
on the server which takes the name of the branch, with every non `\W` character
replaced by an underscore.
This deployment is orchestrated using a git `pre-receive` hook that's managed
via the `profile::puppet::server` class and the `puppet` module.
......@@ -1644,7 +1587,7 @@ all shared among all nodes regadless of their assigned environment.
The environments themselves are structured as follows. All paths are relative
to the root of that git repository.
- `3rdparty/modules` include modules that are shared publicly and do
- `modules` include modules that are shared publicly and do
not contain any TPO-specific configuration. There is a `Puppetfile`
there that documents where each module comes from and that can be
maintained with [r10k][] or [librarian][].
......@@ -1652,17 +1595,17 @@ to the root of that git repository.
[librarian]: https://librarian-puppet.com/
[r10k]: https://github.com/puppetlabs/r10k/
- `modules` includes roles, profiles, and classes that make the bulk
- `site` includes roles, profiles, and classes that make the bulk
of our configuration.
- The `torproject_org` module
(`modules/torproject_org/manifests/init.pp`) performs basic host
(`legacy/torproject_org/manifests/init.pp`) performs basic host
initialisation, like configuring Debian mirrors and APT sources,
installing a base set of packages, configuring puppet and timezone,
setting up a bunch of configuration files and running `ud-replicate`.
- There is also the `hoster.yaml` file
(`modules/torproject_org/misc/hoster.yaml`) which defines hosting
(`legacy/torproject_org/misc/hoster.yaml`) which defines hosting
providers and specifies things like which network blocks they use,
if they have a DNS resolver or a Debian mirror. `hoster.yaml` is read
by
......@@ -2174,11 +2117,11 @@ In other words, this is the checklist:
requires creating `roles` for each machine (more or less) --
effectively done as far as this issue is concerned
* [ ] sanitize repository (tpo/tpa/team#29387)
* [ ] rename `hiera/` to `data/`
* [ ] add `site/` and `legacy/` to modulepaths environment config
* [ ] move `modules/profile/` and `modules/role/` modules into `site/`
* [ ] move remaining modules in `modules/` into `legacy/`
* [ ] move `3rdparty/*` into environment root
* [x] rename `hiera/` to `data/`
* [x] add `site/` and `legacy/` to modulepaths environment config
* [x] move `modules/profile/` and `modules/role/` modules into `site/`
* [x] move remaining modules in `modules/` into `legacy/`
* [x] move `3rdparty/*` into environment root
Once this is done, our Puppet environment would look like this:
......@@ -2325,7 +2268,7 @@ The next step is to start working on a test suite for services, at
least for new deployments, so that code can be tested without running
things by hand. Plenty of Puppet modules have such test suite,
generally using [rspec-puppet](https://rspec-puppet.com/) and [rspec-puppet-facts](https://github.com/mcanevet/rspec-puppet-facts), and we
already have a few modules in `3rdparty` that have such tests. The
already have a few modules in `modules` that have such tests. The
idea would be to have those tests on a per-role or per-profile basis.
The Foreman people have published [their test infrastructure](https://github.com/theforeman/foreman-infra/tree/master/puppet) which
......
policy/tpa-rfc-77-puppet-merge.md
+ 3
− 3
View file @ a508e13d
......@@ -60,18 +60,18 @@ Tails:
- (1.6) Rename all profiles from `tails::profile` to `profile::tails`
- (1.7) Ensure all exported resources' tags are prefixed with tails_
- (1.8) Upgrade 3rdparty modules to match TPA versions
- (1.8) Upgrade 3rd-party modules to match TPA versions
Tor:
- (1.9) Install all `3rdparty` modules that are used by Tails but not by Tor
- (1.9) Install all 3rd-party modules that are used by Tails but not by Tor
- (1.10) Isolate all exported resources and collectors using tags
- (1.11) Move default configuration to a new `profile::tpa` class and include it in all nodes
- (1.12) Enforce signed commits
- (1.13) Ensure all private data is moved to Trocla and publish the repo ([tpo/tpa/team#29387](https://gitlab.torproject.org/tpo/tpa/team/-/issues/29387))
- (1.14) Import the `tails::profile::puppet::eyaml` profile into TPA's `profile::puppet::server`
- (1.15) Copy the EYAML keys from the Tails to the Tor puppet server, and adapt `hiera.yaml` to use them
- (1.16) Upgrade 3rdparty modules to match Tails versions
- (1.16) Upgrade 3rd-party modules to match Tails versions
When we say "upgrade", we don't mean to upgrade to the latest upstream
version of a module, but to the latest release that is highest version
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment