service/prometheus.md

@@ -1504,6 +1504,49 @@ IRC relay:
 
 [default route errors]: #default-route-errors
 
+## Checking for targets changes
+
+If you are making significant changes to the way targets are
+discovered by Prometheus, you might want to make sure you are not
+missing anything.
+
+There used to be a [targets](https://prometheus.torproject.org/classic/targets) web interface but it might be broken
+([1108095](https://bugs.debian.org/1108095)) or even retired altogether ([tpo/tpa/team#41790](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41790))
+and besides, visually checking for this is error-prone.
+
+It's better to do a stricter check. For that, you can use the API
+endpoint and `diff` the resulting JSON, after some filtering. Here's
+an example.
+
+ 1. fetch the targets before the change:
+
+        curl localhost:9090/api/v1/targets > before.json
+
+ 2. make the change (typically by running Puppet):
+
+        pat
+
+ 3. fetch the targets after the change:
+
+        curl localhost:9090/api/v1/targets > after.json
+
+ 4. diff the two, you'll notice this is way too noisy because the
+    scrape times have changed. you might also get changed paths that
+    you should ignore:
+
+        diff -u before.json after.json
+
+    Files might be sorted differently as well.
+
+ 5. so instead, created a filtered and sorted JSON file:
+
+        jq -S '.data.activeTargets| sort_by(.scrapeUrl)' < before.json  | grep -v -e lastScrape -e 'meta_filepath' > before-subset.json
+        jq -S '.data.activeTargets| sort_by(.scrapeUrl)' < after.json  | grep -v -e lastScrape -e 'meta_filepath' > after-subset.json
+
+ 6. then diff the filtered views:
+
+        diff -u before-subset.json after-subset.json
+
 ## Metric relabeling
 
 The [blackbox target documentation](#adding-a-blackbox-target) uses a technique called