gitlab monitoring

we need to have gitlab metrics in grafana. we also need to make sure things work (nagios).

this implies monitoring the webserver (nginx i guess?) and it would also be nice to have internal gitlab metrics. there's a builtin prometheus exporter in gitlab so we should be able to reuse that. see:

https://docs.gitlab.com/ee/administration/monitoring/prometheus/ https://docs.gitlab.com/ee/administration/monitoring/prometheus/gitlab_metrics.html

Trac:
Parent Ticket: legacy/trac#29400 (moved)

added component::internal services/services admin team in Legacy / Trac owner::hiro in Legacy / Trac parent::29400 in Legacy / Trac priority::low in Legacy / Trac severity::normal in Legacy / Trac status::assigned in Legacy / Trac tpa-roadmap-june in Legacy / Trac type::task in Legacy / Trac labels

Trac:
Keywords: N/A deleted, tpa-roadmap-april added
Status: new to assigned
Owner: N/A to hiro

Trac:
Keywords: tpa-roadmap-april deleted, tpa-roadmap-may added

i deprioritized that work because we already have alerting setup. this is about hooking up gitlab's Prometheus endpoint into prometheus, so it's less a priority than other work we have.

Trac:
Priority: Medium to Low

This should start enabling gitlab exporters and connecting them to our prometheus server.

https://share.riseup.net/#66tKNKzMwngju3kXXjLfIg

Currently WIP.

+        monitoring_whitelist      => [
+          '195.201.139.202',
+        ],

this should have a # XXX MAGIC-IP-ADDRESS comment over it, in the short term. but even better, it should not be hardcoded at all and somehow guessed correctly. weasel had some tricks to fetch the IP address of another node, but I forgot how he pulled it off.

@@ -38,6 +38,10 @@ class profile::prometheus::server::internal (
     { 'job_name' => 'postfix' },
     { 'job_name' => 'postgres' },
     { 'job_name' => 'mtail' },
+    { 'job_name' => 'gitlab_exporter' },
+    { 'job_name' => 'redis_exporter' },
+    { 'job_name' => 'gitaly' },
+    { 'job_name' => 'gitlab_workhorse' },
   ]
   class { 'profile::prometheus::server::common':
     vhost_name          => $vhost_name,

that, in itself, won't be sufficient for the server to talk to the exporters. you'd need to export (as in @@rule in Puppet) the right resource for that to work. see how profile::prometheus::blackbox_exporter (and, really, whatever is going on in prometheus::blackbox_exporter) for an example on how to do this.

@@ -57,5 +61,9 @@ class profile::prometheus::server::internal (
     'postgres': port => 9187;
     'bind': port => 9119;
     'mtail': port => 3903;
+    'gitlab_exporter': port => 9168;
+    'redis_exporter': port => 9121;
+    'gitaly': port => 9236;
+    'gitlab_workhorse': port => 9229;
   }
 }

inversely, this will not do anything either unless you create a rule like the one in prometheus::blackbox_exporter, e.g.

  # realize the allow rules defined on the prometheus server(s)
  Ferm::Rule <<| tag == 'profile::prometheus::server-blackbox-exporter' |>>

I should have had specified that my first comment wasn't a request for a review, but rather me logging how I was going about solving this. As a matter of fact I did not see your comments till today when I came to update this ticket with the grafana dashboard.

As you mentioned there are many things that should be improved. Not hard coding prometheus IP in gitlab config is one of them.

Anyways, this is the first iteration. Dashboards available at:

Gitlab dashboard: https://grafana.torproject.org/d/QrDJktiMz/gitlab-omnibus?orgId=1&refresh=5m

Gitaly dashboard: https://grafana.torproject.org/d/x6Z50y-iz/gitlab-gitaly?orgId=1&refresh=1m

Gitlab Node dashboard: https://grafana.torproject.org/d/Z7T7Cfemz/node-exporter-full?orgId=1&var-job=gitlab&var-node=gitlab-02.torproject.org&var-port=9101

Anyways, this is the first iteration. Dashboards available at:

That looks awesome, thanks! :)

As a matter of fact I did not see your comments till today when I came to update this ticket with the grafana dashboard.

as an aside, I'm a bit concerned by that... don't you get email notification on ticket updates? :) or is there another way i should notify you when i update tickets?

There are a few things missing yet on this ticket.

Firstly the prometheus configuration should be cleaned up. We shouldn't have the prometheus server IP hardcoded in gitlab.rb. Also nginx isn't currently monitored.

Trac:
Keywords: tpa-roadmap-may deleted, tpa-roadmap-june added

mentioned in issue legacy/trac#29400 (moved)

moved from legacy/trac#33921 (moved)

mentioned in issue gitlab#20 (moved)

added Backlog label

added Task label and removed 1 deleted label

added Service Admin label and removed 1 deleted label

mentioned in issue #29400 (closed)

moved to gitlab#49 (closed)