gitlab-02 is slow again

gitlab is so slow as to be considered down, couldn't load issues this morning, blocked traffic to users without cookies.

First diagnostic

Logs and dumps, click to expand

times utc-4 below:

09:45:51 -ALERTOR3:#tor-alerts- JobDown [firing] Exporter job "gitlab-rails" on gitlab-02.torproject.org:443 is down
09:46:10 -ALERTOR3:#tor-alerts- HTTPSUnreachable [firing] Website https://gitlab.torproject.org/tpo/tpa/team/ is unreachable via HTTPS CRITICAL!
09:50:11 -ALERTOR4:#tor-alerts- HTTPSFlapping [firing] Website https://gitlab.torproject.org/tpo/tpa/team/ is flapping via HTTPS
09:52:20 <@anarcat> i got two HTTPSUnreachable by email here
09:56:21 -ALERTOR4:#tor-alerts- JobDown [resolved] Exporter job "gitlab-rails" on gitlab-02.torproject.org:443 is down
10:00:51 -ALERTOR3:#tor-alerts- JobDown [firing] Exporter job "gitlab-rails" on gitlab-02.torproject.org:443 is down
10:05:51 -ALERTOR3:#tor-alerts- JobDown [resolved] Exporter job "gitlab-rails" on gitlab-02.torproject.org:443 is down

Current status

blocked an AS, restored cookie-less access, situation seems to have stabilized as of 2026-04-27 14:33UTC

Roles

  • Lead: unless otherwise noted, the issue assignee
  • Operations:
  • Communications: @lelutin
  • Planning:

Next steps

  • monitor performance
  • post-mortem
  • close the incident on status.tpo

Dashboards

Post-mortem

Detailed post-mortem to fill in later, click to expand
  • Affected users:
  • Duration:
  • Status page link:
  • Report Status: not started

Timeline

Root cause analysis

What went well?

What could have gone better?

Recommendations and related issues

Edited by anarcat