... | ... | @@ -406,6 +406,41 @@ configured, inside Puppet, in `profile::prometheus::server::external`. |
|
|
Note that it's [not possible to push timestamps](https://github.com/prometheus/pushgateway#about-timestamps) into the
|
|
|
Pushgateway, so it's not useful to ingest past historical data.
|
|
|
|
|
|
## Deleting metrics
|
|
|
|
|
|
Deleting metrics can be done through the Admin API. That first needs
|
|
|
to be enabled in `/etc/default/prometheus`, by adding
|
|
|
`--web.enable-admin-api` to the `ARGS` list, then Prometheus needs to
|
|
|
be restarted:
|
|
|
|
|
|
service prometheus restart
|
|
|
|
|
|
WARNING: make sure there is authentication in front of Prometheus
|
|
|
because this could expose the server to more destruction.
|
|
|
|
|
|
Then you need to issue a special query through the API. This, for
|
|
|
example, will wipe all metrics associated with the given instance:
|
|
|
|
|
|
curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="gitlab-02.torproject.org:9101"}'
|
|
|
|
|
|
The same, but only for about an hour, good for testing that only the
|
|
|
wanted metrics are destroyed:
|
|
|
|
|
|
curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="gitlab-02.torproject.org:9101"}&start=2021-10-25T19:00:00Z&end=2021-10-25T20:00:00Z'
|
|
|
|
|
|
To match only a job on a specific instance:
|
|
|
|
|
|
curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="gitlab-02.torproject.org:9101"}&match[]={job="gitlab"}'
|
|
|
|
|
|
Deleted metrics are not necessarily immediately removed from disk but
|
|
|
are "eligible for compaction". Changes *should* show up immediately
|
|
|
however. The "Clean Tombstones" should be used to remove samples from
|
|
|
disk, if that's absolutely necessary:
|
|
|
|
|
|
curl -XPOST http://localhost:9090/api/v1/admin/tsdb/clean_tombstones
|
|
|
|
|
|
Make sure to disable the Admin API when done.
|
|
|
|
|
|
## Pager playbook
|
|
|
|
|
|
TBD.
|
... | ... | |