Mailman 2 was removed from Debian bullseye, we need to either upgrade to Mailman 3 or get rid of it. This is part of the 2022-Q1/Q2 OKRs and the %"Debian 11 bullseye upgrade" milestone. upgrade procedure: https://docs.mailman3.org/en/latest/migration.html as part of %"TPA-RFC-71: emergency email deployments, phase B", we proposed to make a new install on a new VM (mailman-01?). # current status VM (`lists-01`) has been installed, mailman 3 setup, all mailing lists are in the progress of being migrated, see below for details. update: all lists migrated, everything in order. next step is to finish service docs, followup tickets in #41853, #41850, #41852 # checklist - [x] install mailman3 through Puppet - [x] test the site: - [x] registration and login (web) - [x] create a list (web) - [x] create a list (cli) - [x] invites (web) - [x] subscribe (email) - [x] subscribe (web) - [x] reply (email) - [x] subscribe other users - [x] unsubscribe (email) - [x] unsubscribe (web) - [x] signup (web) - [x] password reset (web) - [x] archives (not working!) - [x] private archives - [x] reply from web (or turn off) - [x] translations (french not working, not a blocker for launch) - [x] delete a list (test2, cli) - [x] delete a list (web) - [x] fix issues found in testing - [x] cron job fires garbage to www-data every minute (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1051617) - [x] www-data@ alias delivers locally (!?) - [x] fix schleuder routing to keep sending mail to mta.tails - [x] redeploy with PostgreSQL? (sqlite is not recommended and we've seen locking issues) - [x] send reminders to mailing lists - [x] tor-project - [x] tor-relays (moderated) - [x] act (moderated) - [x] tor-consensus-health (moderated) - [x] tpa-team - [x] tor-announce (moderated) - [x] tor-dev (moderated) - [x] tor-qa (moderated) - [x] tor-board - [x] add notice on status.tpo - [x] archive the old site: - [x] update https://wiki.archiveteam.org/index.php/Mailman/2 - [x] crawl site starting from last crawl in september, using `(1\d{3}|20[0-1]\d|2020)` as an exclude, last full job is from 2021, so we crawl up to (and including) 2020, see https://archive.fart.website/archivebot/viewer/job/20211101142707clpzk, last job took about 2 days to run - [x] sync a copy of the public mail archives to https://archive.torproject.org/websites/lists.torproject.org/pipermail/ - [x] add rewriting rules from mailman2 on lists-01, for `cgi-bin/mailman` - [x] copy over archives and lists - [x] check lists for readiness (done, emailed list owners for pending requests, digests will be flushed before migration - [x] convert one test list - [x] route @lists.tpo to lists-01 for test list - [x] convert tpa-team, reroute - [x] remove authentication on lists-01 - [x] confirm tpa-team works properly - [x] post-testing issues: - [x] add DKIM records to DNS - [x] add DMARC munging https://gitlab.com/mailman/mailman/-/issues/1181 - [x] strip incoming DKIM sigs - [x] schedule a more precise maintenance window - [x] final migration (maintenance window) - [x] convert all lists - [x] anti-censorship-alerts - [x] anti-censorship-team - [x] board-executive - [x] board-finance - [x] board-legal - [x] board-marketing - [x] dei - [x] dir-auth - [x] eng-leads (note: no archives) - [x] global-south (733 subscriptions ignored) - [x] ~~mailman~~ N/A - [x] meeting-planners (7 held messages ignored) - [x] membership-advisors (71 held messages ignored) - [x] metrics-alerts - [x] network-health (1 held message ignored) - [x] onion-advisors - [x] onionspace-berlin - [x] onionspace-seattle - [x] ooni-bugs - [x] ooni-dev - [x] ooni-operators - [x] ooni-talk - [x] regional-nyc - [x] research-response - [x] tbb-commits - [x] tbb-dev - [x] team-leads - [x] test - [x] tor-access - [x] tor-alums - [x] tor-announce - [x] tor-board (no archive) - [x] tor-boardmembers-only (no archive) - [x] tor-censorship-events - [x] tor-commits (indexer in batch(1), 216717 emails!) - [x] tor-community-team - [x] tor-consensus-health (indexer in batch(1)) - [x] tor-dev - [x] tor-employees (no archive) - [x] tor-gsoc (indexer in batch(1), as well as all other lists below, unless otherwise noted) - [x] tor-internal - [x] tor-l10n (8 held messages ignored) - [x] tor-meeting - [x] tor-mirrors - [x] tor-network-alerts - [x] tor-onions (28 held messages ignored) - [x] tor-operations (no archives) - [x] tor-packagers - [x] tor-project - [x] tor-qa - [x] tor-relays (large, 5 held messages ignored) - [x] tor-relays-universities - [x] tor-research-safety (no archives) - [x] tor-svninternal - [x] tor-team (no archives) - [x] tor-test-network (no archives) - [x] tor-users - [x] tor-vpn - [x] tpa-team - [x] translation-admin (13 held messages ignored) - [x] wtf (no archives) - [x] www-team - [x] clear out `/srv/mailman` (mm2 copy) on `lists-01` to make room for the rest - [x] change `lists` CNAME record to point to `lists-01` - [x] redirect lists.tpo/pipermail to https://archive.torproject.org/websites/lists.torproject.org/pipermail/ (only effective after DNS gets switched to lists-01) - [x] mark maintenance as done on status.tpo - [ ] post-launch: - [x] remove mailman2 mailing lists passwords from password manager - [x] move postgresql to `/srv` - [x] make sure indexers complete - [x] resync archive.torproject.org pipermail archive - [x] notify owners about their lost pending messages - [x] notify everyone about lost private archives, new user accounts, new features, etc - [x] silence warning from daily cron job (`INFO Enqueued 29`, see https://gitlab.com/mailman/hyperkitty/-/issues/295) - [x] silence django exceptions by email (e.g. `Subject: [Django] ERROR (EXTERNAL IP): Internal Server Error: /mailman3/postorius/lists/`) - [x] remove eugeni DKIM record from lists.tpo (~~make sure the queue is empty of lists messages on eugeni first~~, delegated to #40987) - [x] delete old lists archives from lists-01 - [x] write a plugin to replace built-in styles so DMARC mitigation works out of the box (or patch the debian package, see #41853) - [x] add missing postgresql -> mailman3 -> mailman3-web service dependency - [x] write service docs - [x] copy the mbox archives to lists-01, unaccessible - [x] copy the public .mbox files to archive-01 - [x] setup a mailman2.torproject.org alias for people to peruse old settings and approve messages - [x] cleanup ~Lists issues - [x] consider [ARC signing](https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/handlers/docs/arc_sign.html) (delegated to https://gitlab.torproject.org/tpo/tpa/team/-/issues/41852) - [ ] french translations not working, even though they are [marked at 96% done on weblate](https://hosted.weblate.org/projects/gnu-mailman/#languages) and mailman3 *should* [support translation](https://docs.mailman3.org/en/latest/translation.html), with regular commits... this is possibly fixed in trixie - [x] after a delay, retire mailman from eugeni, (delegated to the eugeni upgrade, #40694)