@@ -406,6 +406,41 @@ configured, inside Puppet, in `profile::prometheus::server::external`.
Note that it's [not possible to push timestamps](https://github.com/prometheus/pushgateway#about-timestamps) into the
Pushgateway, so it's not useful to ingest past historical data.
## Deleting metrics
Deleting metrics can be done through the Admin API. That first needs
to be enabled in `/etc/default/prometheus`, by adding
`--web.enable-admin-api` to the `ARGS` list, then Prometheus needs to
be restarted:
service prometheus restart
WARNING: make sure there is authentication in front of Prometheus
because this could expose the server to more destruction.
Then you need to issue a special query through the API. This, for
example, will wipe all metrics associated with the given instance:
curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="gitlab-02.torproject.org:9101"}'
The same, but only for about an hour, good for testing that only the
wanted metrics are destroyed:
curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="gitlab-02.torproject.org:9101"}&start=2021-10-25T19:00:00Z&end=2021-10-25T20:00:00Z'
To match only a job on a specific instance:
curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="gitlab-02.torproject.org:9101"}&match[]={job="gitlab"}'
Deleted metrics are not necessarily immediately removed from disk but
are "eligible for compaction". Changes *should* show up immediately
however. The "Clean Tombstones" should be used to remove samples from
