monitor individual gitlab runners

the only monitoring we have of GitLab CI right now is how many jobs are pending or running, it's useful but not enough.

i believe the runners themselves provide more information through a prometheus exporter. see how that works and try to tap into that, to answer questions like:

  • #41032 (comment 2872402): "In the meantime, how can i check the status of chi-node-14-verylarge? I have another job waiting for 20 hours on it right now."
  • "what is the average wait time on runners"
  • etc? what else?