replace the puppet agent job with a systemd timer

our current cron job is a little overly complicated: it has timeouts, a manual splay, and so on. in fact, it's really complicated. plus it doesn't retry at all, which is a problem with our recent puppetdb stuff (#40699 (closed)).

the task here is to convert the cron job into something more robust, maybe a script or, more likely, simply a systemd timer. the latter would have the advantage of showing failures in nagios's systemd monitoring. i think the requirements are:

• progressive deployment so we only break one host at a time :p
• not a daemon
• "splay": don't run all jobs at once on all hosts...
• ... yet still have a somewhat predictable frequency (e.g. "every four to six hours" is the current schedule, i think
• handle timeouts?
• look at the cron job to make sure we didn't miss anything else

added Next Puppet labels

assigned to @lavamind

changed the description

marked this issue as related to #40699 (closed)

added Doing label and removed Next label

One thing that the cron job does and is not straightforward when switching to a systemd timer is mailing the output. The current setup even has a mechanism to filter a few uninteresting lines from the output before mailing it out. Is that something we want to keep, puppet logs by mail?

With a systemd service we'd still have access to the output of the puppet runs in the regular system logs.

Jérôme Charaoui commented:

One thing that the cron job does and is not straightforward when switching to a systemd timer is mailing the output. The current setup even has a mechanism to filter a few uninteresting lines from the output before mailing it out.

There are tricks to do this in systemd:

https://wiki.archlinux.org/title/Systemd/Timers#MAILTO

Is that something we want to keep, puppet logs by mail?

I don't think so. I don't actually remember seeing puppet logs by email, and, in general, we get way too many emails already. One less is better, not worse.

With a systemd service we'd still have access to the output of the puppet runs in the regular system logs.

Yeah, and we have monitoring to alert us of problems.

...

On 2022-04-27 15:52:24, Jérôme Charaoui (@lavamind) wrote:

marked the checklist item progressive deployment so we only break one host at a time :p as completed

marked the checklist item not a daemon as completed

marked the checklist item "splay": don't run all jobs at once on all hosts... as completed

marked the checklist item ... yet still have a somewhat predictable frequency (e.g. "every four to six hours" is the current schedule, i think as completed

marked the checklist item handle timeouts? as completed

marked the checklist item look at the cron job to make sure we didn't miss anything else as completed

I've deployed the new systemd service/timer unit pair in a new class profile::puppet.

The puppet-run.timer unit will trigger puppet-run.service every 4 hours plus a random delay from 0 to 4 hours. The previous cron job also fired every 4 hours but with only a 2 hour random delay, for some reason.

FixedRandomDelay=true, which is supported on systemd in bullseye and later, ensures that the random delay is predictable between restarts of the timer unit.

I've checked with cumin '*' "systemctl list-timers puppet-run.timer --no-legend" and at a glance the runs are nicely spread out, with only two clusters of 2 and 3 nodes each scheduled to fire at the same time, which I think is fine.

The service unit now has automatic restart in case of failures, up to 3 times which should be sufficient to deal with the transient PuppetDB errors. It doesn't have any output filtering or timeout like the previous cron job. I don't think a timeout is useful to have here since systemd will prevent the service unit starting while it's still running. And if a puppet run is really stuck indefinitely, Nagios will emit a warning about that.

closed