Ensure check-mirrors.rb does not run forever

Originally created by @intrigeri on #17702 (Redmine)

Today on misc.lizard there were 3 instances of that cronjob running: 1 current + 2 that started respectively on April 22 and 23. The 2 oldest ones were all stuck waiting for a wget command to complete.

This might explain some of the leftover temporary directories we’ve seen recently (tails/tails#17679): one hypothesis is that these directories were not leftovers of “old instances of check-mirrors.rb was killed and did not clean up after itself”, but rather a consequence of “weeks-old check-mirrors.rb instances were still running”.

Either way, I think we need 2 things:

• sysadmin: a lock on the cronjob (like we have for most similar cronjobs already) and possibly a timeout (see e.g. timeout(1) which seems to fit the bill), in the cronjob. In case the timeout expires or the lock can’t be taken by a new cronjob instance, I think the mirrors team should be made aware.
• check-mirrors.rb itself: tails/tails#17736

Related issues

• Related to tails/tails#17679