graceful shutdown for gitlab runners and other hosts running long jobs
Ideally when something attempts to shut down or restart a gitlab runner (or the machine its hosted on), it should first complete any current tasks (running jobs, uploading artifacts, etc).
Motivating example 1: today anarcat had to remember to ping jnewsome before rebooting chi-node-14. Thankfully he did, since there was a long-running shadow sim in progress. anarcat then needed to manually wait for the job to finish before initiating shutdown. Ideally anarcat could just issue a shutdown command without having to worry about it, and the gitlab runner would prevent the host machine from restarting until it had completed its current tasks.
Motivating example 2: Maybe this would also address or help address #40437 (closed)?
Potentially relevant gitlab runner docs on graceful shutdown: https://docs.gitlab.com/runner/best_practice/index.html#graceful-shutdown