diff --git a/service/lists.md b/service/lists.md index 250c05c30c638efe7622ee31818449e7a11fcc91..26d051d77d21f425c53fe16c1106c624bc8b6b67 100644 --- a/service/lists.md +++ b/service/lists.md @@ -38,7 +38,9 @@ ssh lists-01.torproject.org mailman-wrapper create LISTNAME If you do not have root access, proceed with the mailman admin password on the [list creation form](https://lists.torproject.org/mailman3/postorius/lists/new/), which is, however, only -accessible to Mailman administrators. +accessible to Mailman administrators. This also allows you to pick a +different [style](https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/styles/docs/styles.html) for the new list, something which is not +available from the commandline before Mailman 3.3.10. Mailman creates the list name with an upper case letter. Usually people like all lower-case more. So log in to the newly created list @@ -67,6 +69,16 @@ because this removes the list without confirmation! ssh lists-01.torproject.org mailman-wrapper remove LISTNAME ``` +## Changing list settings from the CLI + +The [shell subcommand](https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/commands/docs/shell.html) is the equivalent of the old `withlit` +command. By calling: + + mailman-wrapper shell -l LISTNAME + +... you end up in a Python interpreter with the `mlist` object +accessible for modification. + ## What are our most important lists? New to Tor? If so then welcome! Our most important lists are as @@ -165,7 +177,8 @@ their email account. If you were a moderator or admin on a mailing list, simply [sign up for an account](https://lists.torproject.org/mailman3/accounts/signup/?next=/mailman3/postorius/lists/) and you should be able to access the list -moderation facilities. +moderation facilities. See also the [upstream FAQ about this](https://docs.mailman3.org/en/latest/faq-migration.html) and +the [architecture page](https://docs.mailman3.org/en/latest/architecture.html#understanding-user-and-member-accounts). Note that for site-wide administration, there's a different "superuser" concept in the web interface. For this, you need to make a @@ -189,9 +202,11 @@ manual](https://docs.mailman3.org/en/latest/userguide.html) should help you find We upgraded to Mailman 3 because Mailman 2 is unsupported upstream and the Debian machine hosting it was running an unsupported version of -Debian for this reason. See [TPA-RFC-71](https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-71-emergency-email-deployments-round-2) for more background. The +Debian for this reason. See [TPA-RFC-71][] for more background. The [upstream upgrade guide](https://docs.mailman3.org/en/latest/migration.html) also has some reasoning. +[TPA-RFC-71]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/policy/tpa-rfc-71-emergency-email-deployments-round-2 + ## Pager playbook <!-- information about common errors from the monitoring system and --> @@ -327,11 +342,23 @@ through the web interface. TODO: explain a bit the mailman3 architecture, see https://docs.mailman3.org/en/latest/architecture.html +TODO mention zope and django in the same sentence + ## Services <!-- open ports, daemons, cron jobs --> -TODO: document config files https://docs.mailman3.org/en/latest/django-primer.html +As mentioned in the architecture, Mailman is made of different +components who communicate over HTTP, typically. Cron jobs handle +indexing lists for searching. + +All configuration files reside in `/etc/mailman3`, although the +`mailman3-web.py` configuration file has its defaults in +`/usr/share/mailman3-web/settings.py`. Note that this configuration is +actually a Django configuration file, see also the [upstream Django primer](https://docs.mailman3.org/en/latest/django-primer.html). + +The REST API server configuration can be dumped with `mailman-wrapper +conf`, but be careful as it outputs cleartext passwords. ## Storage @@ -538,6 +565,12 @@ label ~Foo. [File]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/new [search]: https://gitlab.torproject.org/tpo/tpa/team/-/issues?label_name%5B%5D=Foo +### Known issues + + - [DMARC mitigations are not enabled by default](https://gitlab.com/mailman/mailman/-/issues/1181) and require + manual modification after a list is created, the fix for this seems + to be to create a [plugin](https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/plugins/docs/intro.html), see [issue #41853](https://gitlab.torproject.org/tpo/tpa/team/-/issues/41853) + ## Maintainer <!-- document who deployed and operates this service, the team and --> @@ -633,11 +666,53 @@ label ~Foo. ## Mailman 2 migration -TODO: document https://gitlab.torproject.org/tpo/tpa/team/-/issues/40471, see also https://docs.mailman3.org/en/latest/migration.html - https://docs.mailman3.org/en/latest/faq-migration.html - https://docs.debops.org/en/stable-2.1/ansible/roles/mailman/mailman2-migration.html - - not documented in debian https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=999861 +The current Mailman 3 server was built from scratch in Puppet, and all +mailing lists were imported from the old Mailman 2 server (`eugeni`) +in [issue 40471](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40471), as part of the broader [TPA-RFC-71][] emergency +email fixes. + +This section documents the upgrade procedure, and is kept for +historical purpose and to help others upgrade. + +### List migration procedure (Fabric) + +We have established a procedure for migrating a single list, derived +from [the upstream migration documentation](https://docs.mailman3.org/en/latest/migration.html#before-you-upgrade) and [Debian bug report +999861](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=999861). The final business logic was written in a [Fabric](howto/fabric) +called `mailman.migrate-mm2-mm3`, see [`fabric_tpa.mailman`](https://gitlab.torproject.org/tpo/tpa/fabric-tasks/-/blob/42d22186a8bb02a4d91670979f44ef9078f3e531/fabric_tpa/mailman.py#L22) for +details. To migrate a list, the following was used: + + fab mailman.migrate-mm2-mm3 tor-relays + +The above assumes a `tpa.mm2_mm3_migration_cleanup` module in the +Python path, currently deployed in Puppet. + +The list owners to contact about issues with pending requests was +generated with: + + sudo -u list /var/lib/mailman/bin/withlist -l -a -r mm2_mm3_migration_cleanup.list_pending_reqs_owners -q + +Others have [suggested the `bounce_info` needs a reset](https://docs.debops.org/en/stable-2.1/ansible/roles/mailman/mailman2-migration.html) but this +has not proven to be necessary in our case. + +Migrating the 60+ lists took the best of a full day of work, with +indexing eventually processed the next day, after the mailing lists +were put online on the Mailman 3 server. + +List migration is CPU bound, spending lots of time in Hyperkitty +import and indexing, about 10 minutes per 10k mails on a two core +VM. It's unclear if this can be parallelized efficiently. + +Interestingly, the new server takes *much* less space than the old +one: the Mailman 2 server had 35G used in `/var/lib/mailman` and the +new one manages to cram everything in 3G of disk. This *might* be +because some lists were discarded in the migration, however. + +### List migration procedure (manual) + +The following procedure was used for the first test list, to figure +out how to do this and help establish the Fabric job. It's kept only +for historical purposes. To check for anomalies in the mailing lists migrations, the following script was used: @@ -658,8 +733,8 @@ and called with for example: sudo -u list /var/lib/mailman/bin/withlist -l -a -r check_list_mm3.check_pending_reqs The `bounce_info` check was done because of a comment found in [this -post](https://docs.debops.org/en/stable-2.1/ansible/roles/mailman/mailman2-migration.html) saying the conversion script had problem with those, it -wasn't checked. +post](https://docs.debops.org/en/stable-2.1/ansible/roles/mailman/mailman2-migration.html) saying the conversion script had problem with those, that +turned out to be unnecessary. The `pending_reqs` check was done because those are not converted by the script. @@ -675,10 +750,8 @@ But it's simpler to just send the actual digest without checking with: This essentially does a `mlist.send_digest_now` so perhaps it would be simpler to just add that to one script. -### List migration procedure - -Procedure for migrating a single list, derived from [the upstream -migration documentation](https://docs.mailman3.org/en/latest/migration.html#before-you-upgrade): +This was the final migration procedure used for the test list and +`tpa-team`: 1. flush digest mbox with: @@ -741,58 +814,3 @@ test-request@lists.torproject.org smtp:lists-01.torproject.org test-bounces@lists.torproject.org smtp:lists-01.torproject.org test-confirm@lists.torproject.org smtp:lists-01.torproject.org ``` - -Note that a [Fabric](howto/fabric) job is being created to automate the above, -currently in `mailman.migrate-mm2-mm3`. - -The list owners to contact were generated with: - - sudo -u list /var/lib/mailman/bin/withlist -l -a -r mm2_mm3_migration_cleanup.list_pending_reqs_owners -q - -The above assumes a `tpa.mm2_mm3_migration_cleanup` module in the -Python path, currently deployed in Puppet. - -TODO: cross ref -https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=999861 - -https://docs.debops.org/en/stable-2.1/ansible/roles/mailman/mailman2-migration.html -not necessary - -critical info to share with users: - -> After running import21 to import a list, those members will -> immediately be able to receive and send mail, and generally -> participate in the list. They will be members at the level of the -> mailman3_core database. However if they choose to manage their -> subscriptions in the web UI, which offers many more options, it will -> require going to “Sign Up†in the upper right corner of the website, -> and creating a new Django account. See further discussion about this -> on the Architecture Page in the section “Understanding User and -> Member Accountsâ€. - -https://docs.mailman3.org/en/latest/migration.html#why-upgrade - -https://docs.mailman3.org/en/latest/faq-migration.html - -TODO: mailman conf, also, it dumps secrets -TODO mention zope and django in the same sentence - -TODO document the [DMARC issue](https://gitlab.com/mailman/mailman/-/issues/1181) - -TODO: document the three list "styles" in the list creation howto, -then also how to create more styles in a separate howto, check out -https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/styles/docs/styles.html -which is a [plugin](https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/plugins/docs/intro.html) - -TODO: document settings, limited: - -TODO: document the new withlist https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/commands/docs/shell.html - -creating a list mailman-wrapper create listname + hack - -deleting a list: mailman-wrapper remove test2@lists.torproject.org -(watch out, no confirm) - -list migration is CPU bound. spending lots of time in hyperkitty -import and indexing, about 10 minutes per 10k mails on a two core VM, -unclear if paralellized.