mailman is disabling unrelated emails and failing to deliver to lots of providers

Can confirm. I was confused by this when I was going over email earlier today.

marked this issue as related to #41877 (closed)

assigned to @anarcat

changed milestone to %TPA-RFC-71: emergency email deployments, phase B

have you tried setting the "Admin immed notify" setting to "off"?

In #41872 (closed), that was the problem: mailman was forwarding spam to the list moderators, and then the recipient (e.g. gmail) was bouncing that forward, which in turned disabled the subscription for that user.

added Needs Information label and removed Doing label

The same has happened with anti-censorship-team mailing list today. And it did happen some weeks ago also #41877 (closed)

Where can I see the setting "Admin immed notify"? I've being looking at the list settings but I can't find it.

which list is this?

The same has happened with anti-censorship-team mailing list today

okay, then that's concerning, because I did flip that setting for that list in #41877 (comment 3131280)

now i'm looking at https://lists.torproject.org/mailman3/postorius/lists/anti-censorship-team.lists.torproject.org/settings/automatic_responses and the setting is back to "yes", so I'm not sure what's going on... did another owner change that setting?

I started a discussion with TPA earlier today. From my perspective this affected multiple people. I try to compile a list below.

name	list
@cohosh	anti-censorship-team
@shelikhoo	anti-censorship-team
@trunp	tor-consensus-health
@boklm	tor-qa
@hacim	tor-consensus-health
@andz	tor-consensus-health
@ahf	tor-consensus-health
@gk	tor-consensus-health
@arma	tor-consensus-health

okay, then this is broader than just moderators getting spammed and disabled because of that.

is it possible that spam got through those lists and triggered those bounces?

i guess the next step would be to trace mail logs to see what exactly triggered the bounce detection to see if we can deal with that.

i suspect the solution here would be to add incoming filtering on the mailing list server like we did on the mail exchanger, but that requires a significant overhaul of the puppet code (#40626 (closed)), something @groente has planned, but is unlikely to happen in the short term.

Are there copies of bounces that are saved for users to investigate the cause?

The tor-consensus-health list shouldn't be getting any posts from the outside typically.

...

On 2024-12-02 17:32:32, anarcat (@anarcat) wrote:

anarcat commented on a discussion: #41907 (comment 3136744)

okay, then this is broader than just moderators getting spammed and disabled because of that.

is it possible that spam got through those lists and triggered those bounces?

is it possible that spam got through those lists and triggered those bounces?

I could be wrong, but, I don't think I am sending any bounces. I definitely don't have a set-up like riseup where spam gets autorefused.

So I am currently thinking that it is a mistake somewhere in mailman, and "exceeding the mailing list's bounce_score_threshold" doesn't actually mean there were bounces.

Which means yes, if anybody can produce such a bounce from me I would be interested to see it. :)

i can confirm there's something really weird going on with mailman.

i'm sorry if i blamed the victims here: this doesn't only affect moderators, but basically everybody. we have massive issues sending mail from mailman 3 right now, and it's unclear to me exactly what is going on.

what my investigation shows is that email that seems to be delivered properly (i took meskio as a test sample) nevertheless trigger a bounce rate. it seems like mailman doesn't map properly bounces.

so there's a couple of issues here:

people's addresses get disabled even though their personal mailboxes are not bouncing, e.g. meskio got marked as failed with an error like 4.5.3 Error: too many recipients which seems to have been emitted from icloud.com, not his email provider
we're failing to deliver to icloud.com (like related to a proofpoint blockage documented in #41900 (closed))
we're struggling to deliver to gmail, possibly because of a DKIM configuration issue

stay tuned as we keep investigating this.

added Doing label and removed Needs Information label

changed title from Your subscription for tor-consensus-health mailing list has been disabled to mailman is disabling unrelated emails and failing to deliver to lots of providers

i suspect the solution here would be to add incoming filtering on the mailing list server like we did on the mail exchanger, but that requires a significant overhaul of the puppet code (#40626 (closed)), something @groente has planned, but is unlikely to happen in the short term.

also, for the record, i don't believe this is spam-related after all. as @micah correctly pointed out, this also affects mailing lists which don't receive outside email.

i'll keep digging.

@groente generously offered to look at this the coming morning (in europe), reassigning.

assigned to @groente and unassigned @anarcat

i added a few lines to the postfix configuration so we're now getting copies of the bounce messages, hopefully that will help in figuring out wtf is going on here

to add to the joy, it now seems like microsoft is blocking us: 550 5.7.1 Unfortunately, messages from [49.12.57.148] weren't sent. Please contact your Internet service provider since part of their network is on our block list (S3140). You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors.

at least part of the problem was lists-01 didn't seem to be doing any kind of spamfiltering. i've deployed clamav and rspamd so it now does the same filtering as mx-dal-01.

marked this issue as related to #41853

marked this issue as related to #40471 (closed)

it's pretty hard to figure out what went wrong the last few days, but we're now in a much better place to debug, so @qbi , @gk , @meskio , @ahf : if you notice any new similar disabling of members, please let me know!

i'll try to keep an eye on things in the meantime as well, hopefully the added spamfiltering and fixing of dmarc mitigation settings will be enough for this not to reoccur.

removed Doing label

added Needs Review label

@anarcat mailman3 is sending moderation reminder mails with a 'postmaster@torproject.org' from header. i'm guessing it's using the site_owner value there. how about changing that to postmaster@lists.torproject.org? that way we'll get valid spf and proper dkim signing.

so, another thing that struck me:

tor-announce@lists.torproject.org is owned by listowner@torproject.org
listowner@torproject.org is an alias for listmaster@lists.torproject.org
listmaster@lists.torproject.org is not a mailman list, but an alias for three personal @torproject.org addresses
these personal addresses are aliases for their actual external mailboxes

for the love of god, can we please get rid of this and just make the individual people owners of the list?

Another observation: gmail also rejects moderation mails when admin_immed_notify is set to False. Apparently a reminder with a link to the list admin interface is considered spam as well.

Setting the header from so that we do DKIM signing might help to alleviate this.

i noticed we're still getting 'too many recipients' in mailman3's smtp log, so i added some limitations in the mta section of mm3's config file. let's see if that helps.

even though we now have incoming spamfiltering, there's still an issue with old spam in moderation queues. what happens is:

there's spam in the moderation queue
mailman3 sends a moderation reminder to the list owners/moderators
this reminder mail looks spammy and gets mistreated as spam
the reminder mail bounces
the owners' bounce scores increase
the owners never get the reminder and don't moderate
repeat until delivery gets disabled because the bounce scores are too high

dear listowners, @ahf @arma @meskio @qbi @gk can you please have a look at the moderation queue of your lists and make sure all spam is removed? thank you!

Happened to me again just now!

Date: Fri, 06 Dec 2024 00:08:10 +0000
From: tor-consensus-health-bounces@lists.torproject.org
To: arma@torproject.org
Subject: Your subscription for tor-consensus-health mailing list has been
        disabled

Your subscription has been disabled on the tor-consensus-health@lists.torproject.org mailing list
because it has received a number of bounces indicating that there may
be a problem delivering messages to arma@torproject.org.  You may want to
check with your mail administrator for more help.

If you have any questions or problems, you can contact the mailing
list owner at

    tor-consensus-health-owner@lists.torproject.org

Do we have more debugging in place to explore?

Do we have more debugging in place to explore?

Well yes, but the odd thing is I don't see any bounces to your address today...

I do see you getting disabled on to-consensus-health exactly one week ago, around the same time:

Nov 29 00:06:36 2024 (1206) Member arma@torproject.org on list tor-consensus-health.lists.torproject.org, bounce score 5 >= threshold 5, disabling delivery.

Everything else I see in the logs also seems consistent with the 'spam in the moderation queue' scenario: your spam score increasing daily at pretty much the exact same time, i'm guessing when the moderation reminder is being sent.

I've enabled verp probes, that way mailman should send a probe before disabling based on bounces.

my theory on this bug is that mailman is failing at bounce tracking.

when you send email A to list L, it delivers to recipients X, Y, Z, etc. If any of X, Y, or Z fails (say X fails), all of those users get their bounce score raised, regardless of whether they get delivered correctly. In other words, say i write to tor-relays, mailman tries to deliver to X (foo@example.com) and Y (bar@riseup.net). X bounces, but both X and Y get disabled.

I suspect this happens when they are in the same "batch".

I also wonder if it might be related to the new sender-rewriting stuff, as i am not sure this was happening before we did that change. We might want to double-check whether this affects non-@torproject.org subscriber, as far as i can tell, only forwards have been reported with this problem so far, which could confirm the theory that it's SRS related.

Finally, I should point out that Mailman has two minor releases pending upload in Debian. We're at 3.3.8, and there's 3.3.10 out there, which does seem to fix issues with bounces and disables:

https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/NEWS.html

possibly relevant:

The default setting for bounce_matching_headers is shortened. (Closes #1172)
Task runner now uses the configured dsn_lifetime when deleting bounce events.
When bounce processing disables delivery for a user, the user’s score is reset so it will be zero if delivery is enabled. (Closes #1061)
The task runner now evicts old, processed bounce events. (Closes #1067)
Bounce processing notifications due to a DSN for multiple users now have the DSN attached to each notice. (Closes #1101)

issue 1061 seems particularly relevant here, as it would explain why bounce scores never reset.

issue 1101 might be a good entry point to figure out how bounces are handled for multiple users.

the problem with the debian package is not only that it's out of date, but looking for help: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=998223 there's already a release-critical bug because it breaks in Python 3.13, and apparently keeping up with the django deps is a pain in the back... but if we think it's worth it, i might try to give it a shot, two minor releases, especially if we focus just on the core, might not be so bad...

HTH!

i've enabled verp for all delivery, that should tackle at least the scenario where a bounce on one address is confused for another in the same batch. it'll have some performance impact, but with our relatively low traffic, i'm pretty sure we can afford it.

I just got unsubscribed from anti-censorship-team again on December 8th. Sorry to spam here, but is there a procedure in place to get myself resubscribed? Do you still want us to let you know if it happens for debugging reasons?

Today I got an information that @morgan was unsubscribed from tor-qa and @dgoulet was unsubscribed from tor-onions. Both due to excessive bounces.

from what i can tell, there have been no false bounce score increments since verp was enabled. failure to deliver to many providers has also been fixed, this was mostly microsoft blocking us. i'm gonna call this done.

if there are any new false positives in the bounce handling, feel free to reopen this issue.

closed

Not sure if this should be reopened but I've just gotten some disable notifications over the weekend for two members of the anti-censorship-team mailing list:

[REDACTED]'s subscription has been disabled on anti-censorship-team@lists.torproject.org due to their bounce score
exceeding the mailing list's bounce_score_threshold.

The triggering DSN if available is attached.

if these are protonmail addresses, they're actually legitimately disabled. they're bouncing because they appear to be no longer in use.

Okay yeah, I see three and they are all proton mail addresses.

@groente do you think it's worth digging into those again?

@cohosh could you privately tell us who those two members are?

Not sure how to make a private comment on here, they are both proton mail addresses though.

it's fine, i see them in the logs and can confirm they're all legit bounces. everything's working as expected.

reopened

closed

mailman is disabling unrelated emails and failing to deliver to lots of providers

Designs

Child items ...

Activity