unexplained increase in sidekiq / redis jobs on gitlab-02
In #41597 (closed) we found that gitlab was suffering from severe performance problems. While those people have largely been handled, we have stumbled upon an anomaly that should probably be investigated.
This is a screenshot of the "background jobs" section of admin interface:
In there you can see that we typically handle around 100k jobs per day (an increase from november 2023, where the average seemed to be 50k). But for the past 3 days now, this amount tripled, to 350k.
Wait a few days and see if that is still happening: it could be just a one-off thing (but that would be quite unlikely given how steady this has been in the past 6 months).
Also considering adding this to a GitLab dashboard and alerting.