scale up our ci-runner-x86-03 and possibly use ci-runner-x86-14
we seem to be under-using our runners dramatically. just look at those memory usage graphs:
at least -02 has the decency of almost (but not entirely!) using up its RAM for caching (the red area), but on -03, that barely makes a dent in the amazingly large 1.5TB capacity.
ci-runner-x86-14 is just completely unused at the moment as well, look at this CPU usage graph:
clearly, someone needs to use -14 for something. @morgan would the apps team be open to sharing that fancy new server we setup for you, or do you plan on using it soon? :)
and perhaps we could raise concurrency a bit on -02? i guess that depends on whether that would break shadow sims... @jnewsome, do you think we could run more jobs on -03?
memory pressure is decent on all three:
... although it is quite busy on -02 there, and it spiked above our threshold on -03 once today. but nothing alarming.
finally, note that disk I/O is getting close to being an issue on -03:
we're getting close to the SATA 6gbps limit here, which will influence how/if we replace the drives on that machine (#42274 ).
i estimate wee could double concurrency on -03 without ill effects.