add extra tricks for progressive puppet deployment

howto/puppet.md

@@ -488,6 +488,25 @@ argument from Hiera. This is how the transition to a managed
     operate a change. You can then examine the output and see if the
     change is legitimate or abort the configuration change.
 
+Once the Puppet agent is disabled on all nodes, it's possible to enable
+it and run the agent only on nodes that still have the agent disabled.
+This way it's possible to "resume" a deployment when a problem or
+change causes the `cumin` run to abort.
+
+    cumin -b 5 '*' 'if test -f /var/lib/puppet/state/agent_disabled.lock; then puppet agent --enable ; puppet agent -t ; fi'
+
+Because the output `cumin` produces groups together nodes that return
+identical output, and because `puppet agent -t` outputs unique
+strings like catalog serial number and runtime in fractions of a
+second, it can be useful to pipe the Puppet agent output to `sed` to
+remove this noise. This way, all nodes on which the agent makes no
+modification are grouped in the report printed by `cumin`.
+
+    set -o pipefail ; puppet agent -t | sed -r "s/^\x1b\[[0-9;]*m//;s/^(Info: (Caching catalog|Applying configuration)).*/\1/;s/^(Notice: Applied catalog )in [0-9\.]+ seconds/\1/"
+
+Here `set -o pipefail` is used so that the return code from `sed`
+doesn't clobber the one from `puppet agent -t`
+
 ## Adding/removing a global admin
 
 To add a new sysadmin, you need to add their SSH key to the root