upgrade mailman to mailman 3

Mailman 2 was removed from Debian bullseye, we need to either upgrade to Mailman 3 or get rid of it. This is part of the 2022-Q1/Q2 OKRs and the %Debian 11 bullseye upgrade milestone.

upgrade procedure: https://docs.mailman3.org/en/latest/migration.html

as part of %TPA-RFC-71: emergency email deployments, phase B, we proposed to make a new install on a new VM (mailman-01?).

current status

VM (lists-01) has been installed, mailman 3 setup, all mailing lists are in the progress of being migrated, see below for details.

update: all lists migrated, everything in order. next step is to finish service docs, followup tickets in #41853, #41850, #41852 (closed)

checklist

• install mailman3 through Puppet
• test the site:
  • registration and login (web)
  • create a list (web)
  • create a list (cli)
  • invites (web)
  • subscribe (email)
  • subscribe (web)
  • reply (email)
  • subscribe other users
  • unsubscribe (email)
  • unsubscribe (web)
  • signup (web)
  • password reset (web)
  • archives (not working!)
  • private archives
  • reply from web (or turn off)
  • translations (french not working, not a blocker for launch)
  • delete a list (test2, cli)
  • delete a list (web)
• fix issues found in testing
  • cron job fires garbage to www-data every minute (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1051617)
  • www-data@ alias delivers locally (!?)
• fix schleuder routing to keep sending mail to mta.tails
• redeploy with PostgreSQL? (sqlite is not recommended and we've seen locking issues)
• send reminders to mailing lists
  • tor-project
  • tor-relays (moderated)
  • act (moderated)
  • tor-consensus-health (moderated)
  • tpa-team
  • tor-announce (moderated)
  • tor-dev (moderated)
  • tor-qa (moderated)
  • tor-board
• add notice on status.tpo
• archive the old site:
  • update https://wiki.archiveteam.org/index.php/Mailman/2
  • crawl site starting from last crawl in september, using (1\d{3}|20[0-1]\d|2020) as an exclude, last full job is from 2021, so we crawl up to (and including) 2020, see https://archive.fart.website/archivebot/viewer/job/20211101142707clpzk, last job took about 2 days to run
  • sync a copy of the public mail archives to https://archive.torproject.org/websites/lists.torproject.org/pipermail/
• add rewriting rules from mailman2 on lists-01, for cgi-bin/mailman
• copy over archives and lists
• check lists for readiness (done, emailed list owners for pending requests, digests will be flushed before migration
• convert one test list
• route @lists.tpo to lists-01 for test list
• convert tpa-team, reroute
• remove authentication on lists-01
• confirm tpa-team works properly
• post-testing issues:
  • add DKIM records to DNS
  • add DMARC munging https://gitlab.com/mailman/mailman/-/issues/1181
  • strip incoming DKIM sigs
• schedule a more precise maintenance window
• final migration (maintenance window)
  • convert all lists
    • anti-censorship-alerts
    • anti-censorship-team
    • board-executive
    • board-finance
    • board-legal
    • board-marketing
    • dei
    • dir-auth
    • eng-leads (note: no archives)
    • global-south (733 subscriptions ignored)
    • mailman N/A
    • meeting-planners (7 held messages ignored)
    • membership-advisors (71 held messages ignored)
    • metrics-alerts
    • network-health (1 held message ignored)
    • onion-advisors
    • onionspace-berlin
    • onionspace-seattle
    • ooni-bugs
    • ooni-dev
    • ooni-operators
    • ooni-talk
    • regional-nyc
    • research-response
    • tbb-commits
    • tbb-dev
    • team-leads
    • test
    • tor-access
    • tor-alums
    • tor-announce
    • tor-board (no archive)
    • tor-boardmembers-only (no archive)
    • tor-censorship-events
    • tor-commits (indexer in batch(1), 216717 emails!)
    • tor-community-team
    • tor-consensus-health (indexer in batch(1))
    • tor-dev
    • tor-employees (no archive)
    • tor-gsoc (indexer in batch(1), as well as all other lists below, unless otherwise noted)
    • tor-internal
    • tor-l10n (8 held messages ignored)
    • tor-meeting
    • tor-mirrors
    • tor-network-alerts
    • tor-onions (28 held messages ignored)
    • tor-operations (no archives)
    • tor-packagers
    • tor-project
    • tor-qa
    • tor-relays (large, 5 held messages ignored)
    • tor-relays-universities
    • tor-research-safety (no archives)
    • tor-svninternal
    • tor-team (no archives)
    • tor-test-network (no archives)
    • tor-users
    • tor-vpn
    • tpa-team
    • translation-admin (13 held messages ignored)
    • wtf (no archives)
    • www-team
  • clear out /srv/mailman (mm2 copy) on lists-01 to make room for the rest
  • change lists CNAME record to point to lists-01
  • redirect lists.tpo/pipermail to https://archive.torproject.org/websites/lists.torproject.org/pipermail/ (only effective after DNS gets switched to lists-01)
  • mark maintenance as done on status.tpo
• post-launch:
  • remove mailman2 mailing lists passwords from password manager
  • move postgresql to /srv
  • make sure indexers complete
  • resync archive.torproject.org pipermail archive
  • notify owners about their lost pending messages
  • notify everyone about lost private archives, new user accounts, new features, etc
  • silence warning from daily cron job (INFO Enqueued 29, see https://gitlab.com/mailman/hyperkitty/-/issues/295)
  • silence django exceptions by email (e.g. Subject: [Django] ERROR (EXTERNAL IP): Internal Server Error: /mailman3/postorius/lists/)
  • remove eugeni DKIM record from lists.tpo (make sure the queue is empty of lists messages on eugeni first, delegated to #40987 (closed))
  • delete old lists archives from lists-01
  • write a plugin to replace built-in styles so DMARC mitigation works out of the box (or patch the debian package, see #41853)
  • add missing postgresql -> mailman3 -> mailman3-web service dependency
  • write service docs
  • copy the mbox archives to lists-01, unaccessible
  • copy the public .mbox files to archive-01
  • setup a mailman2.torproject.org alias for people to peruse old settings and approve messages
  • cleanup Lists issues
  • consider ARC signing (delegated to #41852 (closed))
  • french translations not working, even though they are marked at 96% done on weblate and mailman3 should support translation, with regular commits... this is possibly fixed in trixie
  • after a delay, retire mailman from eugeni, (delegated to the eugeni upgrade, #40694 (closed))