expand on prometheus/puppet monitoring, compare to icinga

howto/puppet.md

@@ -1855,7 +1855,31 @@ label.
 
 ## Monitoring and testing
 
-Puppet is monitored [using Prometheus](https://forge.puppet.com/puppet/prometheus_reporter).
+Puppet is monitored using Prometheus through the [Prometheus
+reporter](https://forge.puppet.com/puppet/prometheus_reporter). This is a small Ruby module that ingests reports posted
+by Puppet agent to the Puppet server and writes metrics to the
+Prometheus node exporter textfile collector, in
+`/var/lib/prometheus/node-exporter`.
+
+We were previously checking Puppet *twice* when we were running
+Icinga:
+
+ * One job ran on the Puppetmaster and checked PuppetDB for
+   reports. This was done with a [patched](https://github.com/evgeni/check_puppetdb_nodes/pull/14) version of the
+   [check_puppetdb_nodes](https://github.com/evgeni/check_puppetdb_nodes/) Nagios check, shipped inside the
+   `tor-nagios-checks` Debian package
+ * The same job actually runs twice; once to check all manifests, and
+   another to check each host individually and assign the result to
+   the right host.
+
+The twin checks were present so that we could find stray Puppet hosts.
+For example, if a host was retired from Icinga but not retired from
+Puppet, or added to Icinga but not Puppet, we would notice. This was
+necessary because the Icinga setup was not Puppetized: the twin check
+now seems superfluous and we only check reports on the server.
+
+Note that we *could* check agents individually with the [puppet agent
+exporter](https://github.com/retailnext/puppet-agent-exporter).
 
 There are no validation checks and *a priori* no peer review of code:
 code is directly pushed to the Puppet server without validation. Work