limit IRC notification with an upper boundary
This introduces a maximum timeout for notifying on IRC for builds. I have made a detailed diagram of possible edge cases and I believe this is fairly reliable, but it's possible this completely silences some notifications in some rather very specific corner cases.
Right now, we're getting one message every 2 hours after the 90m minimum when someone forgets a push, and it's quite distracting. Example:
06:13:30 <KGB-TPA> Found a pending deploy-prod job in tpo/web/community, to run job visit https://gitlab.torproject.org/tpo/web/community/-/jobs/558410
08:13:38 <KGB-TPA> Found a pending deploy-prod job in tpo/web/community, to run job visit https://gitlab.torproject.org/tpo/web/community/-/jobs/558410
10:13:36 <KGB-TPA> Found a pending deploy-prod job in tpo/web/community, to run job visit https://gitlab.torproject.org/tpo/web/community/-/jobs/558410
12:13:29 <KGB-TPA> Found a pending deploy-prod job in tpo/web/community, to run job visit https://gitlab.torproject.org/tpo/web/community/-/jobs/558410
14:13:32 <KGB-TPA> Found a pending deploy-prod job in tpo/web/community, to run job visit https://gitlab.torproject.org/tpo/web/community/-/jobs/558410
16:13:31 <KGB-TPA> Found a pending deploy-prod job in tpo/web/community, to run job visit https://gitlab.torproject.org/tpo/web/community/-/jobs/558410In the above scenario, only the following notification would be shown:
06:13:30 <KGB-TPA> Found a pending deploy-prod job in tpo/web/community, to run job visit https://gitlab.torproject.org/tpo/web/community/-/jobs/558410Edited by anarcat