... | ... | @@ -484,6 +484,28 @@ was incrementing. From there, we reproduced the work the exporter was |
|
|
doing manually and fixed the issue, which involved passing the correct
|
|
|
argument to the exporter.
|
|
|
|
|
|
### Slow startup times
|
|
|
|
|
|
If Prometheus takes a long time to start, and floods logs with lines
|
|
|
like this every second:
|
|
|
|
|
|
Nov 01 19:43:03 hetzner-nbg1-02 prometheus[49182]: level=info ts=2022-11-01T19:43:03.788Z caller=head.go:717 component=tsdb msg="WAL segment loaded" segment=30182 maxSegment=30196
|
|
|
|
|
|
... it's somewhat normal. At the time of writing, Prometheus2 takes
|
|
|
over a minute to start because of this problem. When it's done, it
|
|
|
will show the timing information, which is currently:
|
|
|
|
|
|
Nov 01 19:43:04 hetzner-nbg1-02 prometheus[49182]: level=info ts=2022-11-01T19:43:04.533Z caller=head.go:722 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=314.859946ms wal_replay_duration=1m16.079474672s total_replay_duration=1m16.396139067s
|
|
|
|
|
|
The solution for this is to use the [memory-snapshot-on-shutdown
|
|
|
feature flag](https://prometheus.io/docs/prometheus/latest/feature_flags/#memory-snapshot-on-shutdown), but that is available only from 2.30.0 onward (not
|
|
|
in Debian bullseye), and there are critical bugs in the feature flag
|
|
|
before 2.34 (see [PR 10348](https://github.com/prometheus/prometheus/pull/10348)), so thread carefully.
|
|
|
|
|
|
In other words, this is frustrating, but expected for older releases
|
|
|
of Prometheus. Newer releases may have optimisations for this, but
|
|
|
they need a restart to apply.
|
|
|
|
|
|
### Pushgateway errors
|
|
|
|
|
|
The Pushgateway web interface provides some basic information about
|
... | ... | |