document how to add a blackbox target (#41655) authored by anarcat's avatar anarcat
This was already documented, but buried in the puppet section.
...@@ -296,63 +296,6 @@ That rule, in turn, is defined with the ...@@ -296,63 +296,6 @@ That rule, in turn, is defined with the
# [...] # [...]
} }
In another example, to configure the ssh scrape jobs (in
`modules/profile/manifests/ssh.pp`), the scrape job is created with:
@@prometheus::scrape_job { "blackbox_ssh_banner_${facts['networking']['fqdn']}":
job_name => 'blackbox_ssh_banner',
targets => [ "${facts['networking']['fqdn']}:22" ],
labels => {
'alias' => $facts['networking']['fqdn'],
'team' => 'TPA',
},
}
But because this is a `blackbox_exporter`, the `scrape_configs`
configuration is more involved, as it needs to define the
`relabel_configs` element that make the `blackbox_exporter` work:
- job_name: 'blackbox_ssh_banner'
metrics_path: '/probe'
params:
module:
- 'ssh_banner'
relabel_configs:
- source_labels:
- '__address__'
target_label: '__param_target'
- source_labels:
- '__param_target'
target_label: 'instance'
- target_label: '__address__'
replacement: 'localhost:9115'
Scrape jobs for non-TPA services are defined in Hiera under keys named
`scrape_configs` in `hiera/common/prometheus.yaml`. Here's one example of such a
scrape job definition:
profile::prometheus::server::external::scrape_configs:
# generic blackbox exporters from any team
- job_name: blackbox
metrics_path: "/probe"
params:
module:
- http_2xx
file_sd_configs:
- files:
- "/etc/prometheus-alerts/targets.d/blackbox_*.yaml"
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: localhost:9115
Some scrape jobs can be simpler and not require the relabeling part. In the
above case, the relabeling is done since the exporter runs on the Prometheus
server itself instead of the actual target.
Targets for scrape jobs defined in Hiera are however not managed by Targets for scrape jobs defined in Hiera are however not managed by
puppet. They are defined through files in the [`prometheus-alerts.git` puppet. They are defined through files in the [`prometheus-alerts.git`
repository][]. See the section below for more details on how things repository][]. See the section below for more details on how things
...@@ -416,6 +359,68 @@ Those rules are declared on the server, in `prometheus::prometheus::server::inte ...@@ -416,6 +359,68 @@ Those rules are declared on the server, in `prometheus::prometheus::server::inte
[`gitlab#20`]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/20 [`gitlab#20`]: https://gitlab.torproject.org/tpo/tpa/gitlab/-/issues/20
### Adding a `blackbox` target
Adding targets to the `blackbox` exporter is rather more involved and
complicated than a normal target.
For example, this is how the ssh scrape jobs (in
`modules/profile/manifests/ssh.pp`) are created:
@@prometheus::scrape_job { "blackbox_ssh_banner_${facts['networking']['fqdn']}":
job_name => 'blackbox_ssh_banner',
targets => [ "${facts['networking']['fqdn']}:22" ],
labels => {
'alias' => $facts['networking']['fqdn'],
'team' => 'TPA',
},
}
But because this is a `blackbox` configuration, the `scrape_configs`
configuration is more involved, as it needs to define the
`relabel_configs` element that make the `blackbox_exporter` work:
- job_name: 'blackbox_ssh_banner'
metrics_path: '/probe'
params:
module:
- 'ssh_banner'
relabel_configs:
- source_labels:
- '__address__'
target_label: '__param_target'
- source_labels:
- '__param_target'
target_label: 'instance'
- target_label: '__address__'
replacement: 'localhost:9115'
Scrape jobs for non-TPA services are defined in Hiera under keys named
`scrape_configs` in `hiera/common/prometheus.yaml`. Here's one example of such a
scrape job definition:
profile::prometheus::server::external::scrape_configs:
# generic blackbox exporters from any team
- job_name: blackbox
metrics_path: "/probe"
params:
module:
- http_2xx
file_sd_configs:
- files:
- "/etc/prometheus-alerts/targets.d/blackbox_*.yaml"
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: localhost:9115
The `blackbox_exporter` is rather peculiar and counter-intuitive, see
the [`blackbox_exporter` reference section](#blackbox-exporter) for
more information.
## Writing an alert ## Writing an alert
Now that you have [metrics in your application][] and those are Now that you have [metrics in your application][] and those are
... ...
......