investigate a tiered approach to gitlab CI runners
We've had multiple cases of users abusing our runners (e.g. #41032 (closed)) which wouldn't be that bad if it wasn't blocking production for our users.
At the Wikimedia foundation, they use a tiered approach to managing their fleeet of runners:
https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner/Security_Evaluation
Analyze this approach and see if it's something we'd want to adopt, and how. This might mean deploying more runners or having a call out for the community to provide some.
Consider that teams are likely to need much more resources in large runners, as the applications team is likely to start using CI as well in the future.
Also consider that some teams will need some reliance on the integrity of the build process for some runners, particularly TPA as we shift away from gitolite for our git hosting. Other teams might make release builds on GitLab as well.