Poking around the alertmanager UI (#41733), I found out that prometheus itself has a similar "modern" UI bundled with it, and stripped out of the Debian package. I have tried it on my home server and it didn't quite work, but visually, it's essentially the same thing as the classic UI:  The problem, however, is that upstream seems to have removed the "classic" dashboard entirely. For now, it looks like Debian still supports it, but a week ago, [a change was introduced in the unstable debian package](https://salsa.debian.org/go-team/packages/prometheus/-/commit/48662fbe7d0ee08db411273b3a22ddabdcb14768) warning that the classic UI could be removed from Debian as well eventually. I've filed an [issue in the Debian package](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1083100.) to make sure that, if that happens, we actually *do* get the react app packaged, which is not *that* far fetched because react now *is* in Debian, amazingly, and has been since bullseye! https://packages.debian.org/sid/node-react In any case, it seems pretty important to make sure we deal with that deprecation. We do use parts of the prometheus dashboards, particularly for debugging alerts and we have tons of places pointing at stuff like https://prometheus.torproject.org/classic/graph?g0.range_input=1h&g0.expr=sum(needrestart_processes_with_outdated_libraries)+by+(alias)+%3E+0&g0.tab=1 (which we might want to stop doing as well). Here's a list of features we currently use in the classic web UI, and possible replacements we have, keep updating this as we find stuff: - [ ] the [alerts listing](https://prometheus.torproject.org/classic/alerts), in particular: - [x] *firing* alerts (karma is okay for this) - [x] *pending* alerts (grafana is somewhat okay for this, in the [availability dashboard](https://grafana.torproject.org/d/adwbl8mxnaneoc/availability?orgId=1)) - [ ] *inactive* alerts (no replacement, shows the currently parsed alerts and links to their expressions) - [x] [targets list](https://prometheus.torproject.org/classic/targets), especially useful to diagnose scraping failures (the https://prometheus.torproject.org/api/v1/targets provides a JSON view, [this script turns it into a table](https://gist.github.com/akunzai/da9f5a0d638fe53f1cb811245530a065), wiki updated to show how to parse the JSON) - [x] [ad-hoc queries](https://prometheus.torproject.org/classic/graph) ([grafana explore](https://grafana.torproject.org/explore) is okay, but not accessible to all users, we have good tooling in fabric for that now (prometheus.query*) ) - [ ] query links from Karma (would need to replace with grafana explore), in general we link to prometheus.tpo all over the place and would need redirections - [ ] [service discovery information](https://prometheus.torproject.org/classic/service-discovery) (rarely used, shows which labels are attached to which scrape targets) - [ ] [rules listing](https://prometheus.torproject.org/classic/rules), which includes timing for alerting rules evaluation, but also recording rules - [ ] [configuration dump](https://prometheus.torproject.org/classic/config), [command-line flags](https://prometheus.torproject.org/classic/flags) (mostly unused) - [ ] [run-time and build information](https://prometheus.torproject.org/classic/status) (unused, but i just noticed it shows important stats like high cardinality labels and so on, cardinality alternatives documented in https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/prometheus#running-out-of-disk-space In general, the solution probably will be to use the web interface less and replace it by scripts and tools that talk with the API directly.