migrate to puppetserver and Puppet agent 7 before EOL
Our current "puppetmaster" configuration ("apache + passenger") is deprecated and will be removed in Puppet 6. we need to switch to the alternative, which is "puppetserver", a daemon written in Clojure especially for that purpose.
the tool is not yet in Debian, so this can wait until then. otherwise we could also use the upstream puppet debian repositories.
our "old" passenger configuration lead to at least one security issue (#33587 (closed)) which was due to how complex that configuration is.
puppet 5, as a whole, is EOL in november 2020, so we should consider an upgrade path to Puppet 6 by then. the packaging work is happening in bts #950182.
update: it might be desirable to jump directly to Puppet 7 if we're going to make such a switch, since Puppet 6 itself is going to be EOL by February 2023 (source).
update: we most definitely will want to jump to puppet 7 for both the server and agent components, as the 6.x series is missing support for Ruby 3.0 which is what bookworm ships currently.
note: a puppet cluster upgrade normally goes through those two stages:
- upgrade puppetdb, because its consumers are typically more tolerant of API changes
- upgrade the server
- upgrade the agents
In other words, the server is backwards compatible with older clients, but not the reverse. and the server and puppet db need to be upgraded together. How far back the server is compatible with the clients is unclear. this document seems to state, surprisingly, that Puppetserver 7 is backwards compatible with Puppet 4 agents. But this other document states the opposite, that even Puppet agent 5.x is not supported by 7.x. So that's still unclear.
update: bastelfreak stated that Puppet agent 5.5 should work with Server 7. one concern they had was with the "intermediate cert" created by Puppet server 7, but as long as we keep the existing cert, we should be fine.
This is the state of the Puppet packages in Debian and upstream:
| Package | buster | bullseye | testing | unstable | experimental | upstream |
|---|---|---|---|---|---|---|
| Puppet agent | 5.5.10-4 | 5.5.22-2 | 5.5.22-4 | 5.5.22-4 | 7.16.0-2 | 6.27.0/7.17.0 |
| Puppet master | 5.5.10-4 | 5.5.22-2 | 5.5.22-4 | 5.5.22-4 | N/A | N/A |
| Puppetserver | N/A | N/A | N/A | N/A | 6.16.0-1 | 6.19.0/7.8.0 |
| PuppetDB | 6.2.0-3 | N/A | N/A | 6.2.0-5 | N/A | 6.21.0/7.10.1 |
| Facter | 3.11.0-2+deb10u2 | 3.14.12-1 | 3.14.12-1.1 | 3.14.12-1.1 | N/A | 3.14.23/4.2.10 |
This is the different EOL dates:
- Puppet 5: November 2020 (source?)
- Puppet 6: February 2023 (source)
- Puppet 7: unannounced?
this implies that we could do this:
- upgrade pauli to bullseye, keeping the old puppetmaster 5 configuration (or rebuild with bullseye)
- upgrade to puppet server 6 (probably using upstream packages, because the ITP is still not resolved, details in this post too)
- eventually, upgrade all agents to puppet agent 6 (see this bug report, fixed in experimental?), possibly part of the bookworm upgrade?
- upgrade to puppet server 7, possibly with the Bookworm upgrade (see %Debian 12 bookworm upgrade) with Debian packages if jruby gets their thing together
it should be noted that puppetdb is also in bad shape. it's at version 6.2.0 (while upstream is 6.20), with at least three RC bugs which kept it from shipping in bullseye and keep it from bookworm right now as well. see #40707 (closed) for that package specifically.
so that's another thing that should be fixed. according to @pollo, that is simpler to fix than the puppetserver SNAFU, fortunately. it seems it could be upgraded separately from the other as well.
there's a coding sprint coming in the clojure team to address some of the underlying issues in clojure Debian packaging that directly affect Puppet packaging, see https://wiki.debian.org/Sprints/2022/ClojureTeam. I think we should participate in this sprint to help with this effort.
finally, there's also facter to worry about. Puppet 7 seems to mention Facter 4 in its upgrade notes and specifically trouble with upgrades from 6 to 7 that need to go through 6.22+ on the server side. Ugh. Update: on that front, we seem safe enough: people are apparently running Puppet 7 with Facter 3 without problems.
update: there's been some progress recently in packaging puppet agent 7 (almost ready) and puppetdb (some deps still missing). as for puppetserver its still blocking on jruby, so the outlook is uncertain currently. this ongoing work is being tracked in more detail on this Debian wiki page.