gitaly exporter unreachable from prometheus
while doing the reboot and investigating routing issues with the alertmanager (#41667 (closed)), i noticed the gitaly exporter doesn't seem reachable from prometheus at all, and there's actually an alert firing about its job being down:
root@hetzner-nbg1-01:~# curl gitlab-02.torproject.org:9236
curl: (7) Failed to connect to gitlab-02.torproject.org port 9236 after 8 ms: Couldn't connect to server
it indeed looks like it's listening only on localhost:
root@gitlab-02:~# lsof -p 1134 | grep :9
gitaly 1134 git 10u IPv4 27400 0t0 TCP localhost:9236 (LISTEN)
gitaly 1134 git 11u IPv4 27400 0t0 TCP localhost:9236 (LISTEN)
so either this is a new thing gitlab did to harden that service, or it never worked. either way, this should be fixed, either by removing the target from our configuration, or by somehow allowing prometheus to pull those metrics.