diff --git a/howto/prometheus.md b/howto/prometheus.md index 4098c33a6b474feb4bbdaa5058f94c60ef1967c0..fb943bd92c38ea3da6c84e5091fe24946c6d72d0 100644 --- a/howto/prometheus.md +++ b/howto/prometheus.md @@ -406,6 +406,41 @@ configured, inside Puppet, in `profile::prometheus::server::external`. Note that it's [not possible to push timestamps](https://github.com/prometheus/pushgateway#about-timestamps) into the Pushgateway, so it's not useful to ingest past historical data. +## Deleting metrics + +Deleting metrics can be done through the Admin API. That first needs +to be enabled in `/etc/default/prometheus`, by adding +`--web.enable-admin-api` to the `ARGS` list, then Prometheus needs to +be restarted: + + service prometheus restart + +WARNING: make sure there is authentication in front of Prometheus +because this could expose the server to more destruction. + +Then you need to issue a special query through the API. This, for +example, will wipe all metrics associated with the given instance: + + curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="gitlab-02.torproject.org:9101"}' + +The same, but only for about an hour, good for testing that only the +wanted metrics are destroyed: + + curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="gitlab-02.torproject.org:9101"}&start=2021-10-25T19:00:00Z&end=2021-10-25T20:00:00Z' + +To match only a job on a specific instance: + + curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="gitlab-02.torproject.org:9101"}&match[]={job="gitlab"}' + +Deleted metrics are not necessarily immediately removed from disk but +are "eligible for compaction". Changes *should* show up immediately +however. The "Clean Tombstones" should be used to remove samples from +disk, if that's absolutely necessary: + + curl -XPOST http://localhost:9090/api/v1/admin/tsdb/clean_tombstones + +Make sure to disable the Admin API when done. + ## Pager playbook TBD.