more prom docs

howto/prometheus.md

@@ -121,10 +121,51 @@ See also the [targets.d documentation in the git repository](https://gitlab.torp
 
 ## Adding targets on the internal server
 
-TODO: talk about `scrape_jobs` for in-puppet configurations.
+Normally, services configured in Puppet SHOULD automatically be
+scraped by Prometheus (see below). If, however, you need to manually
+configure a service, you *may* define extra jobs in the
+`$scrape_configs` array, in the
+`profile::prometheus::server::internal` Puppet class.
+
+For example, because the GitLab Prometheus setup is not managed by
+Puppet ([tpo/tpa/gitlab#20](https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/20)), we cannot use this automatic setup,
+so manual scrape targets are defined like this:
+
+      $scrape_configs =
+      [
+        {
+          'job_name'       => 'gitaly',
+          'static_configs' => [
+            {
+              'targets' => [
+                'gitlab-02.torproject.org:9236',
+              ],
+              'labels'  => {
+                'alias' => 'Gitaly-Exporter',
+              },
+            },
+          ],
+        },
+        [...]
+      ]
+
+But ideally those would be configured with automatic targets, below.
+
+## Automatic targets on the internal server
+
+Metrics for the internal server are scraped automatically if the
+exporter is configured by the [puppet-prometheus][] module. This is
+done almost automatically, apart from the need to open a firewall port
+in our configuration. 
+
+To take the `apache_exporter`, as an example, in
+`profile::prometheus::apache_exporter`, include the
+`prometheus::apache_exporter` class from the upstream Puppet module,
+then we open the port to the Prometheus server on the exporter, with:
+
+    Ferm::Rule <<| tag == 'profile::prometheus::server-apache-exporter' |>>
 
-TODO: show how to hook a custom scrape job, and on where server to put
-it.
+Those rules are declared on the server, in `prometheus::prometheus::server::internal`.
 
 ## Web dashboard access
 
@@ -723,13 +764,14 @@ but it was [salvaged](https://github.com/prometheus-community/bind_exporter/issu
 
 Another important layer is the large amount of Puppet code that is
 used to deploy Prometheus and its components. This is all part of a
-big Puppet module, [puppet-prometheus](https://github.com/voxpupuli/puppet-prometheus/), managed by the [voxpupuli
+big Puppet module, [puppet-prometheus][], managed by the [voxpupuli
 collective](https://github.com/voxpupuli). Our integration with the module is not yet complete:
 we have a lot of glue code on top of it to correctly make it work with
 Debian packages. A lot of work has been done to complete that work by
 anarcat, but work still remains, see [upstream issue 32](https://github.com/voxpupuli/puppet-prometheus/issues/32) for
 details.
 
+[puppet-prometheus]: https://github.com/voxpupuli/puppet-prometheus/
 ## Monitoring and testing
 
 Prometheus doesn't have specific tests, but there *is* a test suite in