weather-01 is spamming tor-commits

editing for clarification now that the issue has been mitigated. this thread doesn't have a single cohesive list of events, so here's what happened:

on march 31, someone signed up to the weather.tpo service with the email address "tor-commits@lists.torproject.org". the confirmation email was sent from weather-01, via eugeni, to the tor-commits list. you can see it in the archive. i assume this would normally have ended up in the moderation queue, but the tor-commits list allowed all emails from @torproject.org through without moderation (this has since been fixed). after confirming the account, the user then signed up to receive notifications about every event from every relay. this means that any time any event was generated from any relay, tor-commits subscribers each got an email about it.

the current mitigations include removing *@torproject.org from mailing list allow lists, and stopping postfix (and puppet) on weather-01. weather-01's postfix server is still stopped, because starting it back up means the email spam resumes. tor-weather doesn't include a ban functionality, so while we could drop the user from the database directly, anyone can sign back up with that email address. hopefully removing the allow list entry would stop any of the spam from getting through, but that would still fill up the moderation queue (assuming none of the emails slip through anyway).

now back to your regularly scheduled high-severity incident

---

someone subscribed every single relay they could find to weather.tpo, and now weather-01 is spamming emails to eugeni, which is spamming millions of emails to tor-commits.

for the moment i'll be suspending postfix on weather-01, but i'm worried that all the queued mails are already sent to eugeni. i'll also be checking the mailman config on eugeni to see if we can mitigate.

• remove the @torproject.org allow on tor-commits@ mailing list
• remove the spam emails from the retry queue
• move the legitimate emails back to mailman's retry queue
• most tor-commits emails will go to moderation, we need to fix that
• check if our email reputation has taken a serious hit
• make a header check to forbid outgoing mail to lists.torproject.org (@anarcat)
• check if other lists have blanket allow policies @torproject.org email addresses, just for kicks (tor-announce anyone?) (@anarcat)
• re-enable postfix and puppet on weather-01
• work out a plan to mititgate this in the future (moved to tpo/network-health/tor-weather#61)
• document a disaster recovery plan in case this happens again (moved to tpo/network-health/tor-weather#61)