Skip to content

upgrade mailman to mailman 3

Mailman 2 was removed from Debian bullseye, we need to either upgrade to Mailman 3 or get rid of it. This is part of the 2022-Q1/Q2 OKRs and the %Debian 11 bullseye upgrade milestone.

upgrade procedure: https://docs.mailman3.org/en/latest/migration.html

as part of %TPA-RFC-71: emergency email deployments, phase B, we proposed to make a new install on a new VM (mailman-01?).

current status

VM (lists-01) has been installed, mailman 3 setup, all mailing lists are in the progress of being migrated, see below for details.

update: all lists migrated, everything in order. next step is to finish service docs, followup tickets in #41853, #41850, #41852 (closed)

checklist

  • install mailman3 through Puppet
  • test the site:
    • registration and login (web)
    • create a list (web)
    • create a list (cli)
    • invites (web)
    • subscribe (email)
    • subscribe (web)
    • reply (email)
    • subscribe other users
    • unsubscribe (email)
    • unsubscribe (web)
    • signup (web)
    • password reset (web)
    • archives (not working!)
    • private archives
    • reply from web (or turn off)
    • translations (french not working, not a blocker for launch)
    • delete a list (test2, cli)
    • delete a list (web)
  • fix issues found in testing
  • fix schleuder routing to keep sending mail to mta.tails
  • redeploy with PostgreSQL? (sqlite is not recommended and we've seen locking issues)
  • send reminders to mailing lists
    • tor-project
    • tor-relays (moderated)
    • act (moderated)
    • tor-consensus-health (moderated)
    • tpa-team
    • tor-announce (moderated)
    • tor-dev (moderated)
    • tor-qa (moderated)
    • tor-board
  • add notice on status.tpo
  • archive the old site:
  • add rewriting rules from mailman2 on lists-01, for cgi-bin/mailman
  • copy over archives and lists
  • check lists for readiness (done, emailed list owners for pending requests, digests will be flushed before migration
  • convert one test list
  • route @lists.tpo to lists-01 for test list
  • convert tpa-team, reroute
  • remove authentication on lists-01
  • confirm tpa-team works properly
  • post-testing issues:
  • schedule a more precise maintenance window
  • final migration (maintenance window)
    • convert all lists
      • anti-censorship-alerts
      • anti-censorship-team
      • board-executive
      • board-finance
      • board-legal
      • board-marketing
      • dei
      • dir-auth
      • eng-leads (note: no archives)
      • global-south (733 subscriptions ignored)
      • mailman N/A
      • meeting-planners (7 held messages ignored)
      • membership-advisors (71 held messages ignored)
      • metrics-alerts
      • network-health (1 held message ignored)
      • onion-advisors
      • onionspace-berlin
      • onionspace-seattle
      • ooni-bugs
      • ooni-dev
      • ooni-operators
      • ooni-talk
      • regional-nyc
      • research-response
      • tbb-commits
      • tbb-dev
      • team-leads
      • test
      • tor-access
      • tor-alums
      • tor-announce
      • tor-board (no archive)
      • tor-boardmembers-only (no archive)
      • tor-censorship-events
      • tor-commits (indexer in batch(1), 216717 emails!)
      • tor-community-team
      • tor-consensus-health (indexer in batch(1))
      • tor-dev
      • tor-employees (no archive)
      • tor-gsoc (indexer in batch(1), as well as all other lists below, unless otherwise noted)
      • tor-internal
      • tor-l10n (8 held messages ignored)
      • tor-meeting
      • tor-mirrors
      • tor-network-alerts
      • tor-onions (28 held messages ignored)
      • tor-operations (no archives)
      • tor-packagers
      • tor-project
      • tor-qa
      • tor-relays (large, 5 held messages ignored)
      • tor-relays-universities
      • tor-research-safety (no archives)
      • tor-svninternal
      • tor-team (no archives)
      • tor-test-network (no archives)
      • tor-users
      • tor-vpn
      • tpa-team
      • translation-admin (13 held messages ignored)
      • wtf (no archives)
      • www-team
    • clear out /srv/mailman (mm2 copy) on lists-01 to make room for the rest
    • change lists CNAME record to point to lists-01
    • redirect lists.tpo/pipermail to https://archive.torproject.org/websites/lists.torproject.org/pipermail/ (only effective after DNS gets switched to lists-01)
    • mark maintenance as done on status.tpo
  • post-launch:
    • remove mailman2 mailing lists passwords from password manager
    • move postgresql to /srv
    • make sure indexers complete
    • resync archive.torproject.org pipermail archive
    • notify owners about their lost pending messages
    • notify everyone about lost private archives, new user accounts, new features, etc
    • silence warning from daily cron job (INFO Enqueued 29, see https://gitlab.com/mailman/hyperkitty/-/issues/295)
    • silence django exceptions by email (e.g. Subject: [Django] ERROR (EXTERNAL IP): Internal Server Error: /mailman3/postorius/lists/)
    • remove eugeni DKIM record from lists.tpo (make sure the queue is empty of lists messages on eugeni first, delegated to #40987)
    • delete old lists archives from lists-01
    • write a plugin to replace built-in styles so DMARC mitigation works out of the box (or patch the debian package, see #41853)
    • add missing postgresql -> mailman3 -> mailman3-web service dependency
    • write service docs
    • copy the mbox archives to lists-01, unaccessible
    • copy the public .mbox files to archive-01
    • setup a mailman2.torproject.org alias for people to peruse old settings and approve messages
    • cleanup Lists issues
    • consider ARC signing (delegated to #41852 (closed))
    • french translations not working, even though they are marked at 96% done on weblate and mailman3 should support translation, with regular commits... this is possibly fixed in trixie
    • after a delay, retire mailman from eugeni, (delegated to the eugeni upgrade, #40694 (closed))
Edited by anarcat