monitor individual gitlab runners
the only monitoring we have of GitLab CI right now is how many jobs are pending or running, it's useful but not enough.
i believe the runners themselves provide more information through a prometheus exporter. see how that works and try to tap into that, to answer questions like:
-
#41032 (comment 2872402): "In the meantime, how can i check the status of
chi-node-14-verylarge? I have another job waiting for 20 hours on it right now." - "what is the average wait time on runners"
- etc? what else?
Activity
mentioned in issue #41032 (closed)
marked this issue as related to #41032 (closed)
I've checked out gitlab-ci's
/metricsendpoint. No metrics really caught my attention as being really useful (joined a dump so people can checkout without setting up a runner).Gitlab has an API endpoint that would help with checking what a runner is doing (and has done recently). Though it may be privileged for shared runners (needs Reporter+ for project runners)
"what is the average wait time on runners"
the best answer that can be obtained from Gitlab would only include already finished jobs, and not jobs in queue, as a job is assigned to a runner only when that runner says it has a slot for it.
marked this issue as related to #41295 (closed)
mentioned in issue #41295 (closed)
in #41295 (closed) we found it would be useful to monitor fail rate when changing stuff in the runner's configuration.
so i'll take a look at this again, https://docs.gitlab.com/runner/monitoring/index.html#available-metrics looks much more promising than the metrics sample @trinity-1686a provided here...
assigned to @anarcat
-
i opened the port as per this documentation and there's actually a bunch of interesting stuff.
for example, here are the stats on the podman runner right now:
root@ci-runner-x86-02:~# curl -s localhost:9252/metrics | grep ^gitlab_runner_ | grep -v -e _api_ -e autoscal gitlab_runner_concurrent 8 gitlab_runner_configuration_loaded_total 1 gitlab_runner_configuration_loading_error_total 0 gitlab_runner_configuration_saved_total 0 gitlab_runner_configuration_saving_error_total 0 gitlab_runner_errors_total{level="error"} 0 gitlab_runner_errors_total{level="fatal"} 0 gitlab_runner_errors_total{level="panic"} 0 gitlab_runner_errors_total{level="warning"} 2 gitlab_runner_job_duration_seconds_bucket{runner="__hc2zXq",system_id="s_39a8ec4bc83a",le="30"} 3 gitlab_runner_job_duration_seconds_bucket{runner="__hc2zXq",system_id="s_39a8ec4bc83a",le="60"} 3 gitlab_runner_job_duration_seconds_bucket{runner="__hc2zXq",system_id="s_39a8ec4bc83a",le="300"} 3 gitlab_runner_job_duration_seconds_bucket{runner="__hc2zXq",system_id="s_39a8ec4bc83a",le="600"} 3 gitlab_runner_job_duration_seconds_bucket{runner="__hc2zXq",system_id="s_39a8ec4bc83a",le="1800"} 3 gitlab_runner_job_duration_seconds_bucket{runner="__hc2zXq",system_id="s_39a8ec4bc83a",le="3600"} 3 gitlab_runner_job_duration_seconds_bucket{runner="__hc2zXq",system_id="s_39a8ec4bc83a",le="7200"} 3 gitlab_runner_job_duration_seconds_bucket{runner="__hc2zXq",system_id="s_39a8ec4bc83a",le="10800"} 3 gitlab_runner_job_duration_seconds_bucket{runner="__hc2zXq",system_id="s_39a8ec4bc83a",le="18000"} 3 gitlab_runner_job_duration_seconds_bucket{runner="__hc2zXq",system_id="s_39a8ec4bc83a",le="36000"} 3 gitlab_runner_job_duration_seconds_bucket{runner="__hc2zXq",system_id="s_39a8ec4bc83a",le="+Inf"} 3 gitlab_runner_job_duration_seconds_sum{runner="__hc2zXq",system_id="s_39a8ec4bc83a"} 43.833719257 gitlab_runner_job_duration_seconds_count{runner="__hc2zXq",system_id="s_39a8ec4bc83a"} 3 gitlab_runner_jobs{executor_stage="idle",runner="__hc2zXq",stage="idle",state="idle",system_id="s_39a8ec4bc83a"} 0 gitlab_runner_jobs_total{runner="__hc2zXq",system_id="s_39a8ec4bc83a"} 3 gitlab_runner_limit{runner="__hc2zXq",system_id="s_39a8ec4bc83a"} 0 gitlab_runner_request_concurrency{runner="__hc2zXq",system_id="s_39a8ec4bc83a"} 0 gitlab_runner_request_concurrency_exceeded_total{runner="__hc2zXq",system_id="s_39a8ec4bc83a"} 0 gitlab_runner_version_info{architecture="amd64",branch="16-2-stable",built_at="2023-08-08T00:08:26+0000",go_version="go1.20.5",name="gitlab-runner",os="linux",revision="674e0e29",version="16.2.1"} 1 gitlab_runner_worker_feed_failures_total{runner="__hc2zXq",runner_name="ci-runner-x86-02-main",system_id="s_39a8ec4bc83a"} 0 gitlab_runner_worker_feeds_total{runner="__hc2zXq",runner_name="ci-runner-x86-02-main",system_id="s_39a8ec4bc83a"} 278 gitlab_runner_worker_health_check_failures_total{runner="__hc2zXq",runner_name="ci-runner-x86-02-main",system_id="s_39a8ec4bc83a"} 0 gitlab_runner_worker_processing_failures_total{failure_type="no_free_executor",runner="__hc2zXq",runner_name="ci-runner-x86-02-main",system_id="s_39a8ec4bc83a"} 0 gitlab_runner_worker_processing_failures_total{failure_type="other",runner="__hc2zXq",runner_name="ci-runner-x86-02-main",system_id="s_39a8ec4bc83a"} 0 gitlab_runner_worker_slot_operations_total{operation="started"} 8 gitlab_runner_worker_slot_operations_total{operation="stopped"} 0 gitlab_runner_worker_slots_number 8note how i redacted a bunch of stuff. interesting things:
- concurrency setting (8) vs current concurrency (0), and also
gitlab_runner_request_concurrency_exceeded_total -
gitlab_runner_job_duration_seconds_buckethistogram can give us runtime percentiles and heat maps -
gitlab_runner_job_duration_seconds_countcounts the number of jobs ran (but, strangely, also doesgitlab_runner_jobs_total)
that runner didn't fail a job so far, more interesting things happened on ci-runner-x86-01, which did fail a job:
root@ci-runner-x86-01:~# curl -s localhost:9252/metrics | grep ^gitlab_runner_ | grep -v -e _api_ -e autoscal gitlab_runner_concurrent 8 gitlab_runner_configuration_loaded_total 1 gitlab_runner_configuration_loading_error_total 0 gitlab_runner_configuration_saved_total 0 gitlab_runner_configuration_saving_error_total 0 gitlab_runner_errors_total{level="error"} 0 gitlab_runner_errors_total{level="fatal"} 0 gitlab_runner_errors_total{level="panic"} 0 gitlab_runner_errors_total{level="warning"} 3 gitlab_runner_failed_jobs_total{failure_reason="",runner="68uhRMR2"} 1 gitlab_runner_job_duration_seconds_bucket{runner="68uhRMR2",system_id="s_e6f7404972f2",le="30"} 2 gitlab_runner_job_duration_seconds_bucket{runner="68uhRMR2",system_id="s_e6f7404972f2",le="60"} 2 gitlab_runner_job_duration_seconds_bucket{runner="68uhRMR2",system_id="s_e6f7404972f2",le="300"} 3 gitlab_runner_job_duration_seconds_bucket{runner="68uhRMR2",system_id="s_e6f7404972f2",le="600"} 3 gitlab_runner_job_duration_seconds_bucket{runner="68uhRMR2",system_id="s_e6f7404972f2",le="1800"} 3 gitlab_runner_job_duration_seconds_bucket{runner="68uhRMR2",system_id="s_e6f7404972f2",le="3600"} 3 gitlab_runner_job_duration_seconds_bucket{runner="68uhRMR2",system_id="s_e6f7404972f2",le="7200"} 3 gitlab_runner_job_duration_seconds_bucket{runner="68uhRMR2",system_id="s_e6f7404972f2",le="10800"} 3 gitlab_runner_job_duration_seconds_bucket{runner="68uhRMR2",system_id="s_e6f7404972f2",le="18000"} 3 gitlab_runner_job_duration_seconds_bucket{runner="68uhRMR2",system_id="s_e6f7404972f2",le="36000"} 3 gitlab_runner_job_duration_seconds_bucket{runner="68uhRMR2",system_id="s_e6f7404972f2",le="+Inf"} 3 gitlab_runner_job_duration_seconds_sum{runner="68uhRMR2",system_id="s_e6f7404972f2"} 263.173719729 gitlab_runner_job_duration_seconds_count{runner="68uhRMR2",system_id="s_e6f7404972f2"} 3 gitlab_runner_job_duration_seconds_bucket{runner="yNovyE3y",system_id="s_e6f7404972f2",le="30"} 1 gitlab_runner_job_duration_seconds_bucket{runner="yNovyE3y",system_id="s_e6f7404972f2",le="60"} 1 gitlab_runner_job_duration_seconds_bucket{runner="yNovyE3y",system_id="s_e6f7404972f2",le="300"} 1 gitlab_runner_job_duration_seconds_bucket{runner="yNovyE3y",system_id="s_e6f7404972f2",le="600"} 1 gitlab_runner_job_duration_seconds_bucket{runner="yNovyE3y",system_id="s_e6f7404972f2",le="1800"} 1 gitlab_runner_job_duration_seconds_bucket{runner="yNovyE3y",system_id="s_e6f7404972f2",le="3600"} 1 gitlab_runner_job_duration_seconds_bucket{runner="yNovyE3y",system_id="s_e6f7404972f2",le="7200"} 1 gitlab_runner_job_duration_seconds_bucket{runner="yNovyE3y",system_id="s_e6f7404972f2",le="10800"} 1 gitlab_runner_job_duration_seconds_bucket{runner="yNovyE3y",system_id="s_e6f7404972f2",le="18000"} 1 gitlab_runner_job_duration_seconds_bucket{runner="yNovyE3y",system_id="s_e6f7404972f2",le="36000"} 1 gitlab_runner_job_duration_seconds_bucket{runner="yNovyE3y",system_id="s_e6f7404972f2",le="+Inf"} 1 gitlab_runner_job_duration_seconds_sum{runner="yNovyE3y",system_id="s_e6f7404972f2"} 11.457837355 gitlab_runner_job_duration_seconds_count{runner="yNovyE3y",system_id="s_e6f7404972f2"} 1 gitlab_runner_jobs{executor_stage="idle",runner="68uhRMR2",stage="idle",state="idle",system_id="s_e6f7404972f2"} 0 gitlab_runner_jobs{executor_stage="idle",runner="yNovyE3y",stage="idle",state="idle",system_id="s_e6f7404972f2"} 0 gitlab_runner_jobs_total{runner="68uhRMR2",system_id="s_e6f7404972f2"} 3 gitlab_runner_jobs_total{runner="yNovyE3y",system_id="s_e6f7404972f2"} 1 gitlab_runner_limit{runner="68uhRMR2",system_id="s_e6f7404972f2"} 0 gitlab_runner_limit{runner="yNovyE3y",system_id="s_e6f7404972f2"} 1 gitlab_runner_request_concurrency{runner="68uhRMR2",system_id="s_e6f7404972f2"} 0 gitlab_runner_request_concurrency{runner="yNovyE3y",system_id="s_e6f7404972f2"} 1 gitlab_runner_request_concurrency_exceeded_total{runner="68uhRMR2",system_id="s_e6f7404972f2"} 0 gitlab_runner_request_concurrency_exceeded_total{runner="yNovyE3y",system_id="s_e6f7404972f2"} 0 gitlab_runner_version_info{architecture="amd64",branch="16-2-stable",built_at="2023-08-08T00:08:26+0000",go_version="go1.20.5",name="gitlab-runner",os="linux",revision="674e0e29",version="16.2.1"} 1 gitlab_runner_worker_feed_failures_total{runner="68uhRMR2",runner_name="ci-runner-x86-01-main",system_id="s_e6f7404972f2"} 0 gitlab_runner_worker_feed_failures_total{runner="yNovyE3y",runner_name="ci-runner-x86-01-large",system_id="s_e6f7404972f2"} 0 gitlab_runner_worker_feeds_total{runner="68uhRMR2",runner_name="ci-runner-x86-01-main",system_id="s_e6f7404972f2"} 290 gitlab_runner_worker_feeds_total{runner="yNovyE3y",runner_name="ci-runner-x86-01-large",system_id="s_e6f7404972f2"} 290 gitlab_runner_worker_health_check_failures_total{runner="68uhRMR2",runner_name="ci-runner-x86-01-main",system_id="s_e6f7404972f2"} 0 gitlab_runner_worker_health_check_failures_total{runner="yNovyE3y",runner_name="ci-runner-x86-01-large",system_id="s_e6f7404972f2"} 0 gitlab_runner_worker_processing_failures_total{failure_type="no_free_executor",runner="68uhRMR2",runner_name="ci-runner-x86-01-main",system_id="s_e6f7404972f2"} 0 gitlab_runner_worker_processing_failures_total{failure_type="no_free_executor",runner="yNovyE3y",runner_name="ci-runner-x86-01-large",system_id="s_e6f7404972f2"} 0 gitlab_runner_worker_processing_failures_total{failure_type="other",runner="68uhRMR2",runner_name="ci-runner-x86-01-main",system_id="s_e6f7404972f2"} 1 gitlab_runner_worker_processing_failures_total{failure_type="other",runner="yNovyE3y",runner_name="ci-runner-x86-01-large",system_id="s_e6f7404972f2"} 0 gitlab_runner_worker_slot_operations_total{operation="started"} 8 gitlab_runner_worker_slot_operations_total{operation="stopped"} 0 gitlab_runner_worker_slots_number 8it has exactly the metric we were looking for:
gitlab_runner_failed_jobs_total{failure_reason="",runner="68uhRMR2"} 1the
reasonlabel is a bit scary, but anyway, that looks promising. i'll work on throwing this in puppet and scraping this so we have better metrics of our runners. the metrics documentation is here:https://docs.gitlab.com/runner/fleet_scaling/index.html#monitoring-runners
gitlab.com also has grafana dashboards to monitor runners and configuration tips we should follow:
https://docs.gitlab.com/runner/fleet_scaling/index.html#grafana-dashboard-configuration-tips
- concurrency setting (8) vs current concurrency (0), and also
-
origin/master 1da5b2422f4445eb9c8a4f01d9b322f6580f7923 Author: Antoine Beaupré <anarcat@debian.org> Date: Thu Aug 17 15:25:08 2023 -0400 monitor gitlab runner prometheus exporter (tpo/tpa/team#41042) This is mainly to figure out if failure rates are higher in the podman runner than the docker one, but could also be useful for other things. One question this doesn't answer is the queue wait time, but that should be available on the GitLab rails exporter itself, it's not visible from the runners. 2 files changed, 15 insertions(+) modules/profile/manifests/gitlab/runner.pp | 14 ++++++++++++++ modules/profile/manifests/prometheus/server/internal.pp | 1 + modified modules/profile/manifests/gitlab/runner.pp @@ -192,6 +192,20 @@ class profile::gitlab::runner( Ferm::Rule::Simple <<| tag == 'gitlab-app-to-ci-runner-session-server' |>> } + file_line { 'gitlab-runner metrics listen_address': + path => '/etc/gitlab-runner/config.toml', + after => '^shutdown_timeout', + line => $facts['networking']['fqdn'], + notify => Service[$gitlab_ci_runner::package_name], + } + # grant Prometheus access to the exporter + Ferm::Rule <<| tag == 'profile::prometheus::server-gitlab-runner-exporter' |>> + @@prometheus::scrape_job { "gitlab-runner_${facts['networking']['fqdn']}_9252": + job_name => 'gitlab_runner', + targets => [ "${facts['networking']['fqdn']}:9252" ], + labels => { 'alias' => $facts['networking']['fqdn'] }, + } + # prevent Puppet from ever restarting gitlab-runner Service <| tag == 'gitlab_ci_runner::service' |> { hasrestart => true, modified modules/profile/manifests/prometheus/server/internal.pp @@ -180,6 +180,7 @@ class profile::prometheus::server::internal ( 'nginx': port => 9113; 'gitlab': port => 9178; 'gitlab-redis': port => 9121; + 'gitlab-runner': port => 9252; 'gitaly': port => 9236; 'gitlab-workhorse': port => 9229; 'process': port => 9256;"what is the average wait time on runners"
that's not something the runner exporters can answer, but we should be able to answer questions like:
- is the podman runner failure rate higher than the docker one? (#41295 (closed))
- are we exceeding the concurrency limits on our runners? GitLab.com calls that runner capacity: "View the number of jobs being executed divided by the value of limit or concurrent."
- how long do jobs take (in percentiles, etc)
more metrics could be eventually be available directly in GitLab through the epic Runner Fleet: Queue visibility and observability.
next step is to make a grafana dashboard for this.
-
inspired by gitlab's metrics in https://gitlab.com/gitlab-com/runbooks/-/blob/master/libsonnet/stage-groups/verify-runner/job_graphs.libsonnet, i made this :
https://grafana.torproject.org/d/fd0b2fb2-88d0-4f85-bc86-16164c083b51/gitlab-ci-overview?orgId=1
a bunch of panels are not working, but i suspect that's because the runners are kind of idle and have been so since the exporter started. maybe once a bigger build runs we'll see better numbers, particularly on the "increase" or failed jobs dashboards.
for now, i'll call this done.
mentioned in commit wiki-replica@87cf79f9