Expose gettor updater metrics
Gettor updater is producing metrics, but those are not being exposed outside rdsys-frontend-01 neither being consumed by prometheus. In the same machine we are exposing the gettor distributor metrics in https://rdsys-gettor.torproject.org/metrics.
The gettor updater is publishing the metrics in http://localhost:7800/metrics. Those should be publicly reachable. Should we configure them to be in https://rdsys-gettor-updater.torproject.org/metrics? or https://rdsys-gettor.torproject.org/updater-metrics?
And after is public can tpa configure prometheus to fetch those?
Activity
i guess the short answer is "yes", but devil is, again, in the details.
it's mostly how you prefer it... it's generally easier to not alter the
/metricsbit because that means a new clause in the Prometheus configuration... we're likely going to need to change the proxy config anyway, so might as well make that https://rdsys-gettor-updater.torproject.org/metrics or something like that...is there another use for that domain name? is the updater supposed to be exposed on the internet other than for metrics?
if not, i think we could also consider hardlinking a file in
/var/lib/prometheus/node_exporter, but that ends up in the primary prometheus server, not the secondary... hmmm...it's as you wish, really.
No, there is no other usage of rdsys-gettor.tpo or rdsys-gettor-updater.tpo beside exposing metrics. None of those services publish anything else over http.
If I understand correctly using the node exporter will also make the metrics not public. Making the metrics endpoint public to the world is not a hard requirement, but is nice to have as our grafana is not public some people in the community can consume the metrics directly.
We need those metrics to end up in grafana2.tpo, but I don't mind one or other way.
added Next Prometheus labels
assigned to @anarcat
-
hmmm... looking at this config, i forgot i would also need to add an entry to DNS, then get a cert, and then add a vhost...
why can't this be bundled in the metrics already provided by gettor? it's a separate daemon? what about just exposing port
7800to the Big Bad Internet? i could just add a rule to punch a hole in the firewall to let the prometheus server scrape it...sorry, we're still trying to figure out this monitoring thing, i don't actually know what the best solution is for those things in the long run.
i do feel like a TLS cert only for metrics seems a bit overkill, and i'm not sure tweaking the
/metricsside is a much better fix... after all, exporters are mostly managed by port, not name, traditionnally, why are we taking a different route in the first place here?Edited by anarcat -
why can't this be bundled in the metrics already provided by gettor? it's a separate daemon?
Yes, is a separated daemon. There are two daemons for gettor:
- distributor. The one that we have already metrics for. Takes care of replying to emails
- updater. The one I'm asking to expose it's metrics. Checks if there are new versions of TB and uploads the installers to the providers if there is a new version.
what about just exposing port
7800to the Big Bad Internet? i could just add a rule to punch a hole in the firewall to let the prometheus server scrape it...I'm ok with that, as long as those metrics end up in prometheus I'm happy
i do feel like a TLS cert only for metrics seems a bit overkill, and i'm not sure tweaking the
/metricsside is a much better fix... after all, exporters are mostly managed by port, not name, traditionnally, why are we taking a different route in the first place here?I tend to always say, let's do TLS even if we don't know why, maybe we'll find a why some time in the future (see apt...). But yes, I don't have any good excuse to do TLS here I'm happy to switch to use ports to handle that. We should figure out a tor standard for exposing metrics and use it everywhere, and all my requirements are to make the metrics publicly reachable, I'm happy with whatever makes the life of TPA easier.
-
At this point, we're just trying out different things and seeing what sticks.
I think the easiest for me would be to just add the 7800 port ... That would mean just opening the firewall here to expose the port to the prometheus2 server.
Then you would make a PR on the prometheus-alerts thing to get the target added.
...
On 2023-07-24 15:55:48, meskio wrote:
meskio commented on a discussion: #41265 (comment 2924548)
why can't this be bundled in the metrics already provided by gettor? it's a separate daemon?
Yes, is a separated daemon. There are two daemons for gettor:
- distributor. The one that we have already metrics for. Takes care of replying to emails
- updater. The one I'm asking to expose it's metrics. Checks if there are new versions of TB and uploads the installers to the providers if there is a new version.
what about just exposing port
7800to the Big Bad Internet? i could just add a rule to punch a hole in the firewall to let the prometheus server scrape it...I'm ok with that, as long as those metrics end up in prometheus I'm happy
i do feel like a TLS cert only for metrics seems a bit overkill, and i'm not sure tweaking the
/metricsside is a much better fix... after all, exporters are mostly managed by port, not name, traditionnally, why are we taking a different route in the first place here?I tend to always say, let's do TLS even if we don't know why, maybe we'll find a why some time in the future (see apt...). But yes, I don't have any good excuse to do TLS here I'm happy to switch to use ports to handle that. We should figure out a tor standard for exposing metrics and use it everywhere, and all my requirements are to make the metrics publicly reachable, I'm happy with whatever makes the life of TPA easier.
-- Antoine Beaupré torproject.org system administration
-
It's as you wish, really: if you want the whole internet to access the exporter metrics directly, we can do that, but typically, we restrict that attack surface to a minimum.
...
On 2023-07-24 16:56:24, meskio wrote:
meskio commented on a discussion: #41265 (comment 2924577)
Sounds good. Jsut a correction when you say
expose the port to the prometheus2 server, can we expose it to the whole internet and not just the prometheus2 server?-- Antoine Beaupré torproject.org system administration
-
We make all anti-censorship exporters publicly reachable, we don't collect metrics that can't be published. Is a pity that we can't give access to our grafana so people can see our metrics. But at least making them public others can point their prometheus to our metrics, not sure how much this is actually happening.
I think the attack surface is minimum, they will need to find a security issue in the go HTTP or prometheus implementation. If that exist all anti-censorship services will be affected.
added Needs Information label and removed Doing label
-
@lavamind can you take this ticket, i won't be able to tend to it before next week otherwise...
i think the next step is just to open the firewall to said port in puppet.
@meskio i think for now let's play it safe and let only the prometheus scrape jobs talk to the exporters. then the policy of what to do with the collected metrics can be established separately (and properly).
-
@meskio i think for now let's play it safe and let only the prometheus scrape jobs talk to the exporters. then the policy of what to do with the collected metrics can be established separately (and properly).
I can live with that for now, and lets make that policy so we have a clear way to publish metrics.
-
lets make that policy so we have a clear way to publish metrics
opened #41280 to track that!
added Next label and removed Needs Information label
marked this issue as related to #41280
mentioned in issue #41280
@meskio I've added a firewall exception for port
7800onrdsys-frontend-01.You will probably need to bind the exporter to
0.0.0.0before the metrics are actually accessible.-
Changed the bind and create a merge request to add de target to prometheus: prometheus-alerts!33 (merged)
added Needs Review label and removed Doing label
added Doing label and removed Needs Review label