Nginx/GitLab Prometheus exporter should not be hardcoded
Hi!
I have noticed the services were "degraded" on gitlab-02
this morning. After digging a little, it seemed the prometheus-nginx-exporter
server was failing:
Nov 3 15:41:16 gitlab-02/gitlab-02 prometheus-nginx-exporter[2129]: 2020/11/03 15:41:16 Could not create Nginx Client: Failed to create NginxClient: failed to parse response body "<!DOCTYPE html>\n<html class=\"devise-layout-html\">\n<head prefix=\"og: http://ogp.me/ns#\">\n<meta charset=\"utf-8\">\n<link as=\"style\" href=\"http://127.0.0.1:8080/assets/application-bf1ba5d5d3395adc5bad6f17cc3cb21b3fb29d3e3471a5b260e0bc5ec7a57bc4.css\" rel=\"preload\">\n<link as=\"style\" href
... basically, it was hitting the GitLab frontpage instead of the stub_status
page.
To clear the Nagios warning, I have added a Nginx location
block in a new arbitrary server
that does only that service. But then I realized that nothing was actually scraping the exporter: the main prometheus server does not have data on the nginx server running on Gitlab-02.
Maybe I missed something, but it seems to me this data should be scraped on the prometheus server. As @hiro pointed out in the comments, the target is being correctly scraped. But it's done by hardcoding the configuration on the Prometheus server, instead of using exported resources. We should still fix this and contribute our precious puppet code back upstream to the prometheus module, where it belongs...
The way this is typically done is by including a profile
(e.g. profile::prometheus::apache_exporter
) which configures a class from the 3rdparty Prometheus module (e.g. prometheus::apache_exporter
). Except in this case, the third-party prometheus module doesn't support the nginx exporter we're using: it only supports the vts
one.
So one of two things:
- write our own prometheus exporter wrapper in Puppet for the exporter we're using (and, of course, ideally, contribute it back upstream), or;
- just hack something together in
profile::prometheus::nginx_exporter
(which is basically the same, except we can't rely on sharing this with the community in the future)
We could start with hacking something together real quick (option 2) and move the code to the 3rdparty module (option 1) once we're happy with the result. In any case, we need something better than the current situation, because as things are, the prometheus nginx exporter just does nothing. It just sits there waiting for a prometheus scraper that never comes. It's pretty harmless, but then we don't get precious nginx metrics that could help us diagnose performance issues in the future.
assigned to @hiro because she built this, and it (working with prometheus puppet upstream) would be an important thing to learn. let me know if you want me to do it or need help with this!