refs #41974

howto/puppet.md

@@ -223,21 +223,29 @@ more downloads than the [next option][], which was deprecated.
 [Prometheus modules on the Puppet forge]: https://forge.puppet.com/modules?q=prometheus
 
 Next, the module was added to the Puppetfile (in
-`3rdparty/Puppetfile`):
+`./Puppetfile`):
 
-    mod 'puppet-prometheus', '6.4.0'
+    mod 'puppet/prometheus', # 12.5.0
+      :git => 'https://github.com/voxpupuli/puppet-prometheus.git',
+      :commit => '25dd701b489fc32c892390fd464e765ebd6f513a' # tag: v12.5.0
 
-... and librarian was ran:
+Note that:
 
-    librarian-puppet install
+- Since tpo/tpa/team#41974 we don't import 3rd-party code into our repo and
+  instead deploy the modules dynamicaly in the server.
+- Because of that, modules in the `Puppetfile` should always be pinned to a Git
+  repo and commit, as that's currently the simplest way to avoid some MITM
+  issues.
+- We currently don't have an automated way of managing module dependencies, so
+  you'll have to manually and recursively add dependencies to the `Puppetfile`.
+  Sorry!
+- Make sure to manually audit the code for each module, by reading each file
+  and looking for obvious security flaws or back doors.
 
-This fetched a lot of code from the Puppet forge: the stdlib, archive
-and system modules were all installed or updated. All those modules
-were audited manually, by reading each file and looking for obvious
-security flaws or back doors. Then the code was committed into git:
+Then the code was committed into git:
 
-    git add 3rdparty
-    git commit -m'install prometheus module after audit'
+    git add Puppetfile
+    git commit -m'install prometheus module and its dependencies after audit'
 
 Then the module was configured in a profile, in `modules/profile/manifests/prometheus/server.pp`:
 
@@ -251,7 +259,7 @@ Then the module was configured in a profile, in `modules/profile/manifests/prome
     }
 
 The above contains our local configuration for the upstream
-`prometheus::server` class installed in the `3rdparty` directory. In
+`prometheus::server` class. In
 particular, it sets a retention period and a different path for the
 metrics, so that they follow the new Prometheus 2.x defaults.
 
@@ -278,97 +286,31 @@ And Puppet was ran on the host, with:
 
     puppet --enable ; puppet agent -t --noop ; puppet --disable "testing prometheus deployment"
 
-This led to some problems as the upstream module doesn't support
-installing from Debian packages. Support for Debian was added to the
-code in `3rdparty/modules/prometheus`, and committed into git:
-
-    emacs 3rdparty/modules/prometheus/manifests/*.pp # magic happens
-    git commit -m'implement all the missing stuff' 3rdparty
-    git push
-
-And the above puppet command-line was ran again, continuing that loop
-until things were good.
-
 If you need to deploy the code to multiple hosts, see the `Progressive
 deployment` section below. To contribute changes back upstream (and
 you should do so), see the section right below.
 
 ### Contributing changes back upstream
 
-For simple changes, the above workflow works well, but eventually it
-is preferable to actually fork the upstream repository and operate on our
-fork until the changes are merged upstream.
-
-First, the modified module is moved out of the way:
-
-    mv 3rdparty/modules/prometheus{,.orig}
+Fork the upstream repository and operate on your fork until the changes are
+eventually merged upstream.
 
-The module is then forked on GitHub or wherever it is hosted, and then
-added to the Puppetfile:
+Then, update the `Puppetfile`, for example:
 
+The module is then forked on GitHub or wherever it is hosted
     mod 'puppet-prometheus',
         :git => 'https://github.com/anarcat/puppet-prometheus.git',
-        :branch => 'deploy'
-
-Then Librarian is ran again to fetch that code:
-
-    librarian-puppet install
-
-Because Librarian is a little dumb, it might checkout your module in
-"detached head" mode, in which case you will want to fix the checkout:
-
-    cd 3rdparty/modules/prometheus
-    git checkout deploy
-    git reset --hard origin/deploy
-    git pull
+        :commit => '(...)'
 
 Note that the `deploy` branch here is a merge of all the different
 branches proposed upstream in different pull requests, but it could
 also be the `master` branch or a single branch if only a single pull
 request was sent.
 
-Since you now have a clone of the upstream repository, you can push
-and pull normally with upstream. When you make a change, however, you
-need to commit (and push) the change *both* in the sub-repository and the
-main repository:
-
-    cd 3rdparty/modules/prometheus
-    $EDITOR manifests/init.pp # more magic stuff
-    git commit -m'change the frobatz to a argblu'
-    git push
-    cd ..
-    git commit -m'change the frobatz to a argblu'
-    git push
-
-Often, I make commits directly in our main Puppet repository, without
-pushing to the third party fork, until I am happy with the code, and
-then I craft a nice pretty commit that can be pushed upstream,
-reversing that process:
-
-    $EDITOR 3rdparty/prometheus/manifests/init.pp # dirty magic stuff
-    git commit -m'change the frobatz to a quuxblah'
-    git push
-    # see if that works, generally not
-    git commit -m'rah. wanted a quuxblutz'
-    git push
-    # now we are good, update our pull request
-    cd 3rdparty/modules/prometheus
-    git commit -m'change the frobatz to a quuxblutz'
-    git push
-
-It's annoying to double-commit things, but I haven't found a best way
-to do so just yet. This problem is further discussed in [ticket #29387][].
-
-Also note that when you update code like this, the `Puppetfile` does
-not change, but the `Puppetfile.lock` file *does* change. The `GIT.sha`
-parameter needs to be updated. This can be done by hand, but since
-that is error-prone, you might want to simply run this to update
-modules:
-
-    librarian-puppet update
-
-This will *also* update dependencies so make sure you audit those
-changes before committing and pushing.
+You'll have to keep a clone of the upstream repository somewhere outside of the
+`tor-puppet` work tree, from which you can push and pull normally with
+upstream. When you make a change, you need to commit (and push) the change in
+your external clone and update the `Puppetfile` in the repository.
 
 ### Running tests
 
@@ -1621,7 +1563,8 @@ classes, see [issue 40030][] for progress on this work.
 
 Environments on the Puppet Server are managed using `tor-puppet.git` which is
 our "control repository". Each branch on this repo is mapped to an environment
-on the server which takes the name of the branch.
+on the server which takes the name of the branch, with every non `\W` character
+replaced by an underscore.
 
 This deployment is orchestrated using a git `pre-receive` hook that's managed
 via the `profile::puppet::server` class and the `puppet` module.
@@ -1644,7 +1587,7 @@ all shared among all nodes regadless of their assigned environment.
 The environments themselves are structured as follows. All paths are relative
 to the root of that git repository.
 
-- `3rdparty/modules` include modules that are shared publicly and do
+- `modules` include modules that are shared publicly and do
   not contain any TPO-specific configuration. There is a `Puppetfile`
   there that documents where each module comes from and that can be
   maintained with [r10k][] or [librarian][].
@@ -1652,17 +1595,17 @@ to the root of that git repository.
   [librarian]: https://librarian-puppet.com/
   [r10k]: https://github.com/puppetlabs/r10k/
 
-- `modules` includes roles, profiles, and classes that make the bulk
+- `site` includes roles, profiles, and classes that make the bulk
   of our configuration.
 
 - The `torproject_org` module
-  (`modules/torproject_org/manifests/init.pp`) performs basic host
+  (`legacy/torproject_org/manifests/init.pp`) performs basic host
   initialisation, like configuring Debian mirrors and APT sources,
   installing a base set of packages, configuring puppet and timezone,
   setting up a bunch of configuration files and running `ud-replicate`.
 
 - There is also the `hoster.yaml` file
-  (`modules/torproject_org/misc/hoster.yaml`) which defines hosting
+  (`legacy/torproject_org/misc/hoster.yaml`) which defines hosting
   providers and specifies things like which network blocks they use,
   if they have a DNS resolver or a Debian mirror. `hoster.yaml` is read
   by
@@ -2174,11 +2117,11 @@ In other words, this is the checklist:
        requires creating `roles` for each machine (more or less) --
        effectively done as far as this issue is concerned
  * [ ] sanitize repository (tpo/tpa/team#29387)
- * [ ] rename `hiera/` to `data/`
- * [ ] add `site/` and `legacy/` to modulepaths environment config
- * [ ] move `modules/profile/` and `modules/role/` modules into `site/`
- * [ ] move remaining modules in `modules/` into `legacy/`
- * [ ] move `3rdparty/*` into environment root
+ * [x] rename `hiera/` to `data/`
+ * [x] add `site/` and `legacy/` to modulepaths environment config
+ * [x] move `modules/profile/` and `modules/role/` modules into `site/`
+ * [x] move remaining modules in `modules/` into `legacy/`
+ * [x] move `3rdparty/*` into environment root
 
 Once this is done, our Puppet environment would look like this:
 
@@ -2325,7 +2268,7 @@ The next step is to start working on a test suite for services, at
 least for new deployments, so that code can be tested without running
 things by hand. Plenty of Puppet modules have such test suite,
 generally using [rspec-puppet](https://rspec-puppet.com/) and [rspec-puppet-facts](https://github.com/mcanevet/rspec-puppet-facts), and we
-already have a few modules in `3rdparty` that have such tests. The
+already have a few modules in `modules` that have such tests. The
 idea would be to have those tests on a per-role or per-profile basis.
 
 The Foreman people have published [their test infrastructure](https://github.com/theforeman/foreman-infra/tree/master/puppet) which