Verified Commit b7b66e17 authored by lelutin's avatar lelutin
Browse files

Move reference doc for blackbox exporter out of installation 

I wanted to place this further down but failed to see the same section
names were also present in the section Installation.
parent 3448442a

howto/prometheus.md

+28 −28
Original line number Diff line number Diff line
@@ -1252,34 +1252,6 @@ that the gateway script will run directly on `prometheus2` to avoid 
setting up authentication and/or firewall rules, but this could be

changed.



### Blackbox exporter



Most exporters are pretty straightforward: a service binds to a port and exposes

metrics through HTTP requests on that port, generally on the `/metrics` URL.



The blackbox exporter, however, is a little bit more contrived. The exporter can

be configured to run a bunch of different tests (e.g. tcp connections, http

requests, ICMP ping, etc) for a list of targets of its own. So the prometheus

server has one target, the host with the port for the blackbox exporter, but

that exporter in turn is set to check other hosts.



The [upstream documentation](https://github.com/prometheus/blackbox_exporter)

has some details that can help. We also have examples

[above](#configuring-alert-rules-in-prometheus) for how to configure it in our

setup.



One thing that's nice to know in addition to how it's configured is how you can

debug it. You can query the exporter from localhost in order to get more

information. If you are using this method for debugging, you'll most probably

want to include debugging output. For example, to run an ICMP test on host

pauli.torproject.org:



    curl http://localhost:9115/probe?target=pauli.torproject.org&module=icmp&debug=true



Note that the above trick can be used for _any_ target, not just for ones

currently configured in the blackbox exporter. So you can also use this to test

things before creating the final configuration for the target.



### Alertmanager



The [Alertmanager][] is configured on the external Prometheus server
@@ -1449,6 +1421,34 @@ would otherwise be around long enough for Prometheus to scrape their 
metrics. We use it as a workaround to bridge Metrics data with

Prometheus/Grafana.



### Blackbox exporter



Most exporters are pretty straightforward: a service binds to a port and exposes

metrics through HTTP requests on that port, generally on the `/metrics` URL.



The blackbox exporter, however, is a little bit more contrived. The exporter can

be configured to run a bunch of different tests (e.g. tcp connections, http

requests, ICMP ping, etc) for a list of targets of its own. So the prometheus

server has one target, the host with the port for the blackbox exporter, but

that exporter in turn is set to check other hosts.



The [upstream documentation](https://github.com/prometheus/blackbox_exporter)

has some details that can help. We also have examples

[above](#configuring-alert-rules-in-prometheus) for how to configure it in our

setup.



One thing that's nice to know in addition to how it's configured is how you can

debug it. You can query the exporter from localhost in order to get more

information. If you are using this method for debugging, you'll most probably

want to include debugging output. For example, to run an ICMP test on host

pauli.torproject.org:



    curl http://localhost:9115/probe?target=pauli.torproject.org&module=icmp&debug=true



Note that the above trick can be used for _any_ target, not just for ones

currently configured in the blackbox exporter. So you can also use this to test

things before creating the final configuration for the target.



## Alertmanager



The [Alertmanager][] is a separate program that receives notifications