Skip to content
Snippets Groups Projects

tpa rfc 31 outsource email

Last edited by Antoine Beaupré 
title: TPA-RFC-31: Outsource email services
costs: 2-4 months full time setup, a day a month, TODO hosting
approval: TPA, tor-internal, and TPI
affected users: torproject.org email users
deadline: TODO
status: rejected
discussion: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40798

Summary: outsource as much email as we can to an external provider with IMAP mailboxes and webmail, proper standards and inbound spam filtering. optionally retire Schleuder, Mailman. delegate mass mailings (e.g. CiviCRM) to external provider.

Background

In late 2021, the TPA team adopted the following first Objective and Key Results (OKR):

Improve mail services:

  1. David doesn't complain about "mail getting into spam" anymore
  2. RT is not full of spam
  3. we can deliver and receive mail from state.gov

There were two ways of implementing solutions to this problem. One way was to complete the implementation of email services internally, adding standard tools like DKIM, SPF, and so on to our services and hosting mailboxes. This approach was investigated fully in TPA-RFC-15 but was ultimately rejected as too risky.

Instead, we are looking at the other solution to this problem which is to outsource all or a part of our mail services to some external provider. This proposal aims at clarifying which services we should outsource, and to whom.

Current status

Email has traditionally been completely decentralised at Tor: while we would support forwarding emails @torproject.org to other mailboxes, we have never offered mailboxes directly, nor did we offer ways for users to send emails themselves through our infrastructure.

This situation led to users sending email with @torproject.org email addresses from arbitrary locations on the internet: Gmail, Riseup, and other service providers (including personal mail servers) are typically used to send email for torproject.org users.

This changed at the end of 2021 when the new submission service came online. We still, however, have limited adoption of this service, with only 22 users registered compared to the ~100 users in LDAP (as of 2022-10-31, up ~30%, from 16 in April 2022).

In parallel, we have historically not adopted any modern email standards like SPF, DKIM, or DMARC. But more recently, we added SPF records to both the Mailman and CiviCRM servers (see issue 40347).

We have also been processing DKIM headers on incoming emails on the bridges.torproject.org server, but that is an exception. Finally, we are running Spamassassin on the RT server to try to deal with the large influx of spam on the generic support addresses (support@, info@, etc) that the server processes. We do not process SPF records on incoming mail in any way, which has caused problems with Hetzner (issue 40539).

We do not have any DMARC headers anywhere in DNS, but we do have workarounds setup in Mailman for delivering email correctly when the sender has DMARC records, since September 2021 (see issue 19914).

We do not offer mailboxes, although we do have Dovecot servers deployed for specific purposes. The GitLab and CiviCRM servers, for example, use it for incoming email processing, and the submission server uses it for authentication.

Processing mail servers

Those servers handle their own outgoing email (ie. they do not go through eugeni) and handle incoming email as well, unless otherwise noted:

  • BridgeDB (polyanthum)
  • CiviCRM (crm-int-01, Dovecot)
  • GitLab (gitlab-02, Dovecot)
  • LDAP (alberti)
  • MTA (eugeni)
  • rdsys (rdsys-frontend-01, Dovecot)
  • RT (rude)
  • Submission (submit-01)

This list was generated from Puppet, by grepping for profile::postfix::mail_processing.

Requirements

Those are the requirements the external service provider must fulfill before being considered for this proposal.

Email interfaces

We have users currently using Gmail, Thunderbird, Apple Mail, Outlook, and other email clients. Some people keep their mail on the server, some fetch it once and never keep a copy. Some people read their mail on their phone.

Therefore, the new provider MUST offer IMAP and POP mailboxes, alongside a modern and mobile-friendly Webmail client.

It MUST be possible for users (and machines) to submit emails using a username/password combination through a dedicated SMTP server (also known as a "submission port"). Ideally, this could be done with TLS certificates, especially for client machines.

Some users are unlikely to leave Gmail, and should be able to forward their email there. This will require the hosting provider to implement some sender rewriting scheme to ensure delivery from other providers with a "hard" SPF policy. Inversely, they should be able to send mail from Gmail through a submission address.

Deliverability

Provider SHOULD be able to reliably deliver email to both large service providers like Gmail and Outlook, but also government sites like state.gov or other, smaller mail servers.

Therefore, modern email standards like SPF, DKIM, DMARC, and hopefully ARC SHOULD be implemented by the new provider.

We also often perform mass mailings to announce software releases (through Mailman 2, soon 3) but also larger fundraising mailings through CiviCRM, which contact almost 300,000 users every month. Provisions must therefore be made for those services to keep functioning, possibly through a dedicated mail submission host as well. Servers which currently send regular emails to end users include:

  • CiviCRM: 270,000 emails in June 2021
  • Mailman: 12,600 members on tor-announce
  • RT: support tracker, ~1000 outgoing mails per month
  • GitLab: ~2,000 active users
  • Discourse: ~1,000 monthly active users

Part of our work involves using email to communicate to fundraiser but also people in censored country, so censorship resistance is important. Ideally, a Tor .onion service should be provided for email submission, for example.

Also, we have special email services like gettor.torproject.org which send bridges or download links for accessing the Tor network. Those should also keep functioning properly, but should also be resistant to attacks aiming to list all bridges, for example. This is currently done by checking incoming DKIM signature and limiting the service to certain providers.

Non-mail machines will relay their mail through a new internal relay server that will then submit its mail to the new provider. This will help us automate configuration of "regular" email server to avoid having to create an account in the new provider's control panel every time we setup a new server.

Mailing lists (optional)

We would prefer to outsource our mailing list services. We are currently faced with the prospect of upgrading from Mailman 2 to Mailman 3 and if we're going to outsource email services, it would seem reasonable to avoid such a chore and instead migrate our subscribers and archives to an external service as well.

Spam control

State of the art spam filtering software MUST be deployed to keep the bulk of spam from reaching user's mail boxes, or hopefully triage them in a separate "Spam folder".

Bayesian training MAY be used to improve the accuracy of those filters and the user should be able to train the algorithm to allow certain emails to go through.

Privacy

We are an organisation that takes user privacy seriously. Under absolutely no circumstances should email contents or metadata be used in other fashion than for delivering mail to its destination or aforementioned spam control. Ideally, mail boxes would be encrypted with a user-controlled key so that the provider may not be able to read the contents of mailboxes at all.

Strong log file anonymization is expected or at least aggressive log rotation should be enforced.

Privately identifiable information (e.g. client IP address) MUST NOT leak through email headers.

We strongly believe that free and open source software is the only way to ensure privacy guarantees like these are enforceable. At least the services provided MUST be usable with standard, free software email clients (for example, Thunderbird).

Service level

Considering that email is a critical service at the Tor Project, we want to have some idea of how long problems would take to get resolved.

Availability

We expect the service to be generally available 24/7, with outages limited to one hour or less per month (~99.9% availability).

We also expect the provider to be able to deliver mail to major providers, see the deliverability section, above, for details.

Support

TPI staff should be able to process level 1 support requests about email like password resets, configuration assistance, and training. Ideally, those could be forwarded directly to support staff at the service provider.

We expect same-day response for reported issues, with resolution within a day or a week (business hours), depending on the severity of the problem reported.

Backups

We do not expect users to require mailbox recovery, that will remain the responsibility of users.

But we do expect the provider to set clear RTO (Recovery Time Objective) and PTO (Point-in-Time Objective).

For example, we would hope a full system failure would not lose more than a day of work, and should be restored within less than a week.

Proposal

Progressively migrate all @torproject.org email aliases and forwards to a new, external, email hosting provider.

Retire the "submission service" and destroy of the submit-01 server, after migration of all users to the new provider.

Replacement of all in-house "processing mail servers" with an outsourced counterpart, with some rare exceptions.

Optionally, retirement or migration offsite of Mailman 2.

Scope

This proposal affects the all inbound and outbound email services hosted under torproject.org. Services hosted under torproject.net are not affected.

It also does not address directly phishing and scamming attacks (issue 40596), but it is hoped that stricter enforcement of email standards will reduce those to a certain extent.

Affected users

This affects all users which interact with torproject.org and its subdomains over email. It particularly affects all "tor-internal" users, users with LDAP accounts or forwards under @torproject.org.

The personas section below gives more examples of what exactly will happen to various users and services.

Architecture diagram

Those diagrams detail the infrastructure before and after the changes detailed in this proposal.

Legend:

  • red: legacy hosts, mostly eugeni services, no change
  • orange: hosts that manage and/or send their own email, now relaying except the mail exchanger might be the one relaying the @torproject.org mail to it instead of eugeni
  • green: new hosts, might be multiple replicas
  • purple: new hosting provider
  • rectangles: machines
  • triangle: the user
  • ellipse: the rest of the internet, other mail hosts not managed by tpo

Before

current mail architecture diagram

In the above diagram, we can see how most TPA-managed servers relay email over SMTPS through the eugeni email server, which also hosts Mailman, Schleuder, and incoming email from the rest of the Internet. Users are allowed to send email through TPA infrastructure by using a submission server. There are also mail hosts like GitLab, RT, and CiviCRM who send and receive mail on their own. Finally, the diagram also shows other hosts like Riseup or Gmail who are currently allowed to send mail as torproject.org. Those are called "impersonators".

After

new architecture diagram

In this new diagram, all incoming and outgoing mail with the internet go through the external hosting provider. The only exception is the LDAP server, although it might be possible to work around that problem by using forwarding for inbound email and SMTP authentication for outbound.

In the above diagram, the external hosting provider also handles mailing lists, or we self-host Discourse in which case it behaves like another "mail host".

Also note that in the above diagram, some assumptions are made about the design of the external service provider. This might change during negotiations with the provider, and should be not considered part of the proposal itself.

Actual changes

The actual changes proposed here are broken down in different changes detailed below. A cost estimate of each one is detailed in the costs section.

New mail transfer agent

Configure new "mail transfer agent" server(s) to relay mails from servers that do not send their own email, replacing a part of eugeni.

This host would remain as the last email server in operation by TPA. It is require because we want to avoid the manual overhead of creating accounts for each server on the external mail submission server unless absolutely necessary.

All servers would submit email through this server using mutual TLS authentication the same way eugeni currently does this service. It would then relay those emails to the external service provider.

This server will be called mta-01.torproject.org and could be horizontally scaled up for availability. See also the Naming things challenge below.

Schleuder retirement

Schleuder is likely going to be retired completely from our infrastructure, see TPA-RFC-41: Schleuder retirement.

Mailing lists migration

The new host should be in a position to host mailing lists for us, which probably involves migrating from Mailman 2 to some other software, either Mailman 3 or some other mailing list manager.

Another option here is to self-host a Discourse instance that would replace mailing lists, but that would be done in a separate proposal.

A fallback position would be to keep hosting our mailing lists, which involves upgrading from Mailman 2 to Mailman 3, on a new host. See issue tpo/tpa/team#40471.

User account creation

On-boarding and off-boarding procedures will be modified to add an extra step to create a user account on the external email provider. Ideally, they could delegate authentication for our LDAP server so that the step is optional, but that is not a hard requirement.

For the migration, each user currently in LDAP will have an account created on the external service provider by TPA, and be sent an OpenPGP-encrypted email with their new credentials at their current forwarding address.

Machine account creation

Each mail server not covered by the new transfer agent above will need an account created in the external mail provider.

TPA will handle manually creating an account for each server and configure the server for SMTP-based authenticated submission and incoming IMAP-based spools. Dovecot servers will be retired after the migration, once their folders are confirmed empty and the email communication is confirmed functional.

The following operations will be required, specifically:

Service Server Fate
BridgeDB polyanthum external SMTP/IMAP, IMAP service conversion
CiviCRM crm-int-01, Dovecot external SMTP/IMAP, Dovecot retirement
GitLab gitlab-02, Dovecot external SMTP/IMAP, Dovecot retirement
LDAP alberti added to SPF records, @db.tpo kept active as legacy
MTA eugeni retirement
rdsys rdsys-frontend-01, Dovecot external SMTP/IMAP, Dovecot retirement
RT rude external SMTP/IMAP, forward cleanups
Submission submit-01 retirement

Discourse wouldn't need modification as they handle email themselves in their own domain and mail server. If we would be to self-host, it is assumed Discourse could use an existing SMTP and IMAP configuration as well.

RT is likely to stick around for at least 2023. There are plans to review its performance when compared to the cdr.link instance, but any change will not happen before this proposal needs to be implemented, so we need to support RT for the foreseeable future.

Eugeni retirement

The current, main, mail server (eugeni) deserves a little more attention than the single line above. Its retirement is a complex manner involving many different services and components and is rather risky.

Yet it's a task that is already planned, in some sense, as part of the Debian bullseye upgrade, since we plan to rebuild it in multiple, smaller servers anyways. The main difference here is whether or not some services (mailing lists, mostly) would be rebuilt or not.

The "mail transfer agent" service that eugeni currently operates would still continue operating in a new server, as all mail servers would relay mails through that new host.

See also the mailing lists migration and Schleuder retirement tasks above, since those two services are also hosted on eugeni and would be executed before the retirement.

alberti case

Alberti is a special case because it uses a rather complex piece of software called userdir-ldap (documented in the LDAP service page). It is considered too complicated for us to add an IMAP spool support for that software, so it would still need to accept incoming email directed at @db.torproject.org.

For outgoing mail, however, it could relay mail using the normal mail transfer agent or an account specifically for that service with the external provider, therefore not requiring changes to the SPF, DKIM, or DMARC configurations.

polyanthum case

Polyanthum, AKA https://bridges.torproject.org currently processes incoming email through a forward. It is believe it should be possible to migrate this service to use an incoming IMAP mail spool.

If it is, then it becomes a mail host like CiviCRM or GitLab.

If it isn't possible, then it becomes a special case like alberti.

RT/rude conversion

RT will need special care to be converted to an IMAP based workflow. Postfix could be retained to deal with the SMTP authentication, or that could be delegated to RT itself.

The old queue forwards and the spam filtering system will be retired in favor of a more standard IMAP-based polling and the upstream spam filtering system.

Puppet refactoring

Refactor the mail-related code in Puppet, and reconfigure all servers according to the mail relay server change above, see issue 40626 for details. This should probably happen before or during all the other tasks, not after.

Cost estimates

Summary:

  • setup staff: 35-62 days, 2-4 months full time
  • ongoing staff: unsure, at most a day a month
  • TODO: add summary of hosting costs from below

Staff

This is an estimate of the time it will take to complete this project, based on the tasks established in the actual changes section. The process follows the Kaplan-Moss estimation technique.

Task Estimate Uncertainty Total (days) Note
New mail transfer agent 3 days low 3.3 similar to current submission server
Schleuder retirement 3 days high 6 might require hackery
Mailing list migration 2 weeks high 20 migration or upgrade
User account creation 1 week medium 7.5
Machine account creation
- bridgedb 3 days low 3.3
- CiviCRM 1 day low 1.1
- LDAP 1 day extreme 5
- MTA 1 day extreme 5
- rdsys 1 day low 1.1
- RT 1 day low 1.1
- submission 1 day low 1.1
Puppet refactoring 1 week medium 7.5
Total 35 days ~high 62

Interestingly, the amount of time required to do the migration is in the same magnitude of the estimates behind TPA-RFC-15 (40-80 days), which resulted in us running our own mail infrastructure.

The estimate above could be reduce by postponing mailing list and Schleuder retirements, but this would go against the spirit of this proposal, which requires us to stop hosting our own email...

A large chunk of the estimate (2 weeks, high uncertainty) is around the fate of those two mailing list servers (Mailman and Schleuder, between 13 and 26 days of work, or about a third of the staff costs). Deciding on that fate earlier could help reduce the uncertainty of this proposal.

Ongoing costs

The above doesn't cover ongoing maintenance costs and the overhead of processing incoming questions or complaints and forwarding them upstream, or of creating or removing new accounts for machines or people during on-boarding and retirement.

We can certainly hope this will be much less work than self-hosting our mail services ourselves, however. Let's cap this at one person-day per monnth, which is 12 days of work, or 5,000EUR, per year.

Hosting

TODO: estimate hosting costs

Timeline

TODO: when are we going to do this? and how?

Challenges

Delays

In early June 2021, it became apparent that we were having more problems delivering mail to Gmail, possibly because of the lack of DKIM records (see for example tpo/tpa/team#40765). We may have had time to implement some countermeasures, had TPA-RFC-15 been adopted, but alas, we decided to go with an external provider.

It is unclear, at this point, whether this will speed things up or not. We may take too much time deliberating on the right service provider, or this very specification, or find problems during the migrations, which may make email even more unreliable.

Naming things

In TPA-RFC-15, it became apparent that the difficulty of naming things did not escape those proposals. For example, in TPA-RFC-15, the term "relay" has been used liberally to talk about a new email server processing email for other servers. That terminology, unfortunately, clashes with the term "relay" used extensively in the Tor network to designate "Tor relays", which create circuits that make up the Tor network.

This is the reason why the mta-01 server is named a "MTA" and not a "relay" or "submission" server. The former is reserved for Tor relays, and the latter for "email submission servers" provided by upstream. Technically, the difference between the "MTA" and the "submission" server is that the latter is expected to deliver the email out of the responsibility of the torproject.org domain, to its final destination, while the MTA is allowed to transfer to another MTA or submission server.

Aging Puppet code base and other legacy

This deployment will still need some work on the Puppet code, since we will need to rewire email services on all hosts for email to keep being operational.

We should spend some time to refactor and cleanup that code base before we can do things like SMTP authentication. The work here should be simpler than the one originally involved in TPA-RFC-15, however, so uncertainty around that task's cost was reduced. See also issue 40626 for the original discussion on this issue.

Security and privacy issues

Delegating email services to a third party implies inherent security risks. We would entrust privacy of a sensitive part of our users to a third party provider.

Operators at that provider would likely be in a position to read all of our communications, unless those are encrypted client-side. But given the diminishing interest in OpenPGP inside the team, it seems unlikely we could rely on this for private communications.

Server-side mailbox encryption might mitigate some of those issues, but would require trust in the provider.

On the upside, a well-run service might offer security improvements like two-factor authentication logins, which would have been harder to implement ourselves.

Duplication of services with LDAP

One issue with outsourcing email services is that it will complicate on-boarding and off-boarding processes, because it introduces another authentication system.

As mentioned in the User account creation section, it might be possible for the provider to delegate authentication to our LDAP server, but that would be exceptional.

Upstream support burden

It is unclear how we will handle support requests: will users directly file issues upstream, or will this be handled by TPA first?

How will password resets be done, for example?

Sunk costs

There has been a lot of work done in the current email infrastructure. In particular, anarcat spent a significant amount of time changing the LDAP services to allow the addition of an "email password" and deploying those to a "submission server" to allow people to submit email through the torproject.org infrastructure. The TPA-RFC-15 design and proposal will also go to waste with this proposal.

Partial migrations

With this proposal, we might end up in a "worst case scenario" where we both have the downsides of delegating email hosting (e.g. the privacy issues) and still having deliverability issues (e.g. because we cannot fully outsource all email services, or because the service provider is having its own deliverability issues).

In particular, there is a concern we might have to maintain a significant part of email infrastructure, even after this proposal is implemented. We already have, as part of the spec, a mail transfer agent and the LDAP server as mail servers to maintain, but we might also have to maintain a full mailing list server (Mailman), complete with its (major) Debian bullseye upgrade.

Personas

Here we collect a few "personas" and try to see how the changes will affect them.

We have taken the liberty of creating mostly fictitious personas, but they are somewhat based on real-life people. We do not mean to offend. Any similarity that might seem offensive is an honest mistake on our part which we will be happy to correct. Also note that we might have mixed up people together, or forgot some. If your use case is not mentioned here, please do report it. We don't need to have exactly "you" here, but all your current use cases should be covered by one or many personas.

The personas below reuse the ones from TPA-RFC-15 but, of course, adapted to the new infrastructure.

Ariel, the fundraiser

Ariel does a lot of mailing. From talking to fundraisers through their normal inbox to doing mass newsletters to thousands of people on CiviCRM, they get a lot of shit done and make sure we have bread on the table at the end of the month. They're awesome and we want to make them happy.

Email is absolutely mission critical for them. Sometimes email gets lost and that's a huge problem. They frequently tell partners their personal Gmail account address to workaround those problems. Sometimes they send individual emails through CiviCRM because it doesn't work through Gmail!

Their email is forwarded to Google Mail and they do not have an LDAP account.

TPA will make them an account that forwards to their current Gmail account. This might lead to emails bouncing when sent from domains with a "hard" SPF policy unless the external service provider has some mitigations in place to rewrite the sender. In that case incoming email addresses might be mangled to ensure delivery, which may lead to replies failing.

Ariel will still need an account with the external provider, which will be provided over Signal, IRC, snail mail, or smoke signals. Ariel will promptly change the password upon reception and use it to configure their Gmail account to send email through the external service provider.

Gary, the support guy

Gary is the ticket master. He eats tickets for breakfast, then files 10 more before coffee. A hundred tickets is just a normal day at the office. Tickets come in through email, RT, Discourse, Telegram, Snapchat and soon, TikTok dances.

Email is absolutely mission critical, but some days he wishes there could be slightly less of it. He deals with a lot of spam, and surely something could be done about that.

His mail forwards to Riseup and he reads his mail over Thunderbird and sometimes webmail.

TPA will make an account for Gary and send the credentials in an encrypted email to his Riseup account.

He will need to reconfigure his Thunderbird to use the new email provider. The incoming mail checks from the new provider should, hopefully, improve the spam situation across the board, but especially for services like RT. It might be more difficult, however, for TPA to improve spam filtering capabilities on services like RT since spam filtering will be delegated to the upstream provider.

He will need, however, to abandon Riseup for TPO-related email, since Riseup cannot be configured to relay mail through the external service provider.

John, the external contractor

John is a freelance contractor that's really into privacy. He runs his own relays with some cools hacks on Amazon, automatically deployed with Terraform. He typically run his own infra in the cloud, but for email he just got tired of fighting and moved his stuff to Microsoft's Office 365 and Outlook.

Email is important, but not absolutely mission critical. The submission server doesn't currently work because Outlook doesn't allow you to add just an SMTP server.

John will have to reconfigure his Outlook to send mail through the external service provider server and use the IMAP service as a backend.

Nancy, the fancy sysadmin

Nancy has all the elite skills in the world. She can configure a Postfix server with her left hand while her right hand writes the Puppet manifest for the Dovecot authentication backend. She knows her shit. She browses her mail through a UUCP over SSH tunnel using mutt. She runs her own mail server in her basement since 1996.

Email is a pain in the back and she kind of hates it, but she still believes everyone should be entitled to run their own mail server.

Her email is, of course, hosted on her own mail server, and she has an LDAP account.

She will have to reconfigure her Postfix server to relay mail through the external service provider. To read email, she will need to download email from the IMAP server, although it will still be technically possible to forward her @torproject.org email to her personal server directly, as long as the server is configured to send email through the external service provider.

Mallory, the director

Mallory also does a lot of mailing. She's on about a dozen aliases and mailing lists from accounting to HR and other obscure ones everyone forgot what they're for. She also deals with funders, job applicants, contractors and staff.

Email is absolutely mission critical for her. She often fails to contact funders and critical partners because state.gov blocks our email (or we block theirs!). Sometimes, she gets told through LinkedIn that a job application failed, because mail bounced at Gmail.

She has an LDAP account and it forwards to Gmail. She uses Apple Mail to read their mail.

For her Mac, she'll need to configure the submission server and the IMAP server in Apple Mail. Like Ariel, it is technically possible for her to keep using Gmail, but with the same caveats about forwarded mail from SPF-hardened hosts.

The new mail relay servers should be able to receive mail state.gov properly. Because of the better reputation related to the new SPF/DKIM/DMARC records, mail should bounce less (but still may sometimes end up in spam) at Gmail.

Orpheus, the developer

Orpheus doesn't particular like or dislike email, but sometimes has to use it to talk to people instead of compilers. They sometimes have to talk to funders (#grantlife) and researchers and mailing lists, and that often happens over email. Sometimes email is used to get important things like ticket updates from GitLab or security disclosures from third parties.

They have an LDAP account and it forwards to their self-hosted mail server on a OVH virtual machine.

Email is not mission critical, but it's pretty annoying when it doesn't work.

They will have to reconfigure their mail server to relay mail through the external provider. They should also start using the provider's IMAP server.

Blipblop, the bot

Blipblop is not a real human being, it's a program that receives mails from humans and acts on them. It can send you a list of bridges (bridgedb), or a copy of the Tor program (gettor), when requested. It has a brother bot called Nagios/Icinga who also sends unsolicited mail when things fail. Both of those should continue working properly, but will have to be added to SPF records and an adequate OpenDKIM configuration should be deployed on those hosts as well.

There's also a bot which sends email when commits get pushed to gitolite. That bot is deprecated and is likely to go away.

Most bots will be modified to send and receive email through the external service providers. Some bots will need to be modified to fetch mail over IMAP instead of being pushed mail over SMTP.

Any bot that requests it will be able to get their own account to send and receive email at the external service provider.

Other alternatives

Those are other alternatives than this proposal that were considered while or before writing it.

Hosting our own email

We have first proposed to host our own email completely and properly, through the TPA-RFC-15 proposal. That proposal was rejected

The rationale here is that we prefer to outsource the technical and staff risks to the outside, because the team is already overloaded. It was felt that email was a service too critical to be left to the already overloaded team to improve, and that we should consider external hosting instead for now.

Status quo

The status quo situation is similar (if not worse) than the status quo described in TPA-RFC-15: email services are suffering from major deliverability problems and things are only going to get worse over time, up to a point when no one will use their @torproject.org email address.

The end of email

This is similar to the discussion mentioned in TPA-RFC-15: email is still a vital service and we cannot at the moment consider completely replacing it with other tools.

Generic providers evaluation

TODO: update those numbers, currently taken directly from TPA-RFC-15, without change

Fastmail: 5,000$/year, no mass mailing

Fastmail were not contacted directly but their pricing page says about 5USD/user-month, with a free 30-day trial. This amounts to 500/mth or 6,000$/year.

It's unclear if we could do mass-mailing with this service. Do note that they do not use their own service to send their own newsletter (!?):

In 2018, we closed Listbox, our email marketing service. It no longer fit into our suite of products focused on human-to-human connection. To send our own newsletter, and to give you the best experience reading newsletters, it made sense to move on to one of the many excellent paid email marketing services, as long as customer privacy could be maintained.

So it's quite likely we would have trouble sending mass mailings through Fastmail.

They do not offer mailing lists services.

Gandi: 480-2400/year

Gandi, the DNS provider, also offers mailbox services which are priced at 0.40/user-month (3GB mailboxes) or 2.00/user-month (50GB).

It's unclear if we could do mass-mailing with this service.

They do not offer mailing lists services.

Google: 10,000$/year

Google were not contacted directly, but their promotional site says it's "Free for 14 days, then 7.80$ per user per month", which, for tor-internal (~100 users), would be 780$/month or ~10,000USD/year.

We probably wouldn't be able to do mass mailing with this service. Unclear.

Google offers "Google groups" which could replace our mailing list services.

Greenhost: ~1600€/year, negotiable

We had a quote from Greenhost for 129€/mth for a Zimbra frontend with a VM for mailboxes, DKIM, SPF records and all that jazz. The price includes an office hours SLA.

TODO: check if Greenhost does mailing lists TODO: check if Greenhost could do mass mailings

Mailcow: 480€/year

Mailcow is interesting because they actually are based on a free software stack (based on PHP, Dovecot, Sogo, rspamd, postfix, nginx, redis, memcached, solr, Oley, and Docker containers). They offer a hosted service for 40€/month, with a 100GB disk quota and no mailbox limitations (which, in our case, would mean 1GB/user).

We also get full admin access to the control panel and, given their infrastructure, we could self-host if needed. Integration with our current services would be, however, tricky.

It's unclear if we could do mass-mailing with this service.

Mailfence: 2,500€/year, 1750€ setup

The mailfence business page doesn't have prices but last time we looked at this, it was a 1750€ setup fee with 2.5€ per user-year.

It's unclear if we could do mass-mailing with this service.

Riseup

Riseup already hosts a significant number of email accounts by virtue of being the target of @torproject.org forwards. During the last inventory, we found that, out of 91 active LDAP accounts, 30 were being forwarded to riseup.net, so about 30%.

Riseup supports webmail, IMAP, and, more importantly, encrypted mailboxes. While it's possible that an hostile attacker or staff could modify the code to inspect a mailbox's content, it's leagues ahead of most other providers in terms of privacy.

Riseup's prices are not public, but they are close to "market" prices quoted above.

We might be able to migrate our mailing lists to Riseup, but we'd need to convert our subscribers over to their mailing list software (Sympa) and the domain name of the lists would change (to lists.riseup.net).

We could probably do mass mailings at Riseup, as long as our opt-out correctly work and we ramp up outgoing properly.

Transactional providers evaluation

Those providers specialize in sending mass mailings. Those do not cover all use cases required by our email hosting needs; in particular, they do provide IMAP or Webmail services, any sort of mailboxes, and not manage inbound mail beyond bounce handling.

This list is based on the recommended email providers from Discourse. As a reminder, we send over 250k emails during our mass mailings, with 270,000 sent in June 2021, so the prices below are based on those numbers, roughly.

Mailgun: 200-250$/mth

  • Free plan: 5,000 mails per month, 1$/1,000 mails extra (about 250$/mth)
  • 80$/mth: 100,000 mails per month, 0.80$/1,000 extra (about 200$/mth)

All plans:

  • hosted in EU or US
  • GDPR policy: message bodies kept for 7 days, metadata for 30 days, email addresses fully suppressed after 30 days when unsubscribed
  • sub-processors: Amazon Web Services, Rackspace, Softlayer, and Google Cloud Platform
  • privacy policy: uses google analytics and many more
  • AUP: maybe problematic for Tor, as:

You may not use our platform [...] to engage in, foster, or promote illegal, abusive, or irresponsible behavior, including (but not limited to):

[...]

2b – Any activity intended to withhold or cloak identity or contact information, including the omission, deletion, forgery or misreporting of any transmission or identification information, such as return mailing and IP addresses;

SendGrid: 250$/mth

  • Free plan: 40k mails on a 30 day trial
  • Pro 100k plan: 90$/mth estimated, 190,000 emails per month
  • Pro 300k plan: 250$/mth estimated, 200-700,000 emails per month
  • about 3-6$/1k extra

Details:

  • security policy: security logs kept for a year, metadata kept for 30 days, random content sampling for 61 days, aggregated stats, suppression lists (bounces, unsubscribes), spam reports kept indefinitely
  • privacy policy: Twilio's. Long and hard to read.

Owned by Twilio now.

Mailjet: 225$/mth

Pricing page:

  • Free plan: 6k mails per month
  • Premium: 250,000 mails at 225$/mth
  • around 1$/1k overage

Note, same corporate owner than Mailgun, so similar caveats but, interestingly, no GDPR policy.

Elastic email: 25-500$/mth

https://elasticemail.com/email-marketing-pricing

  • 500$/mth for 250k contacts

https://elasticemail.com/email-api-pricing

  • 0.10$/1000 emails + 0.50$/day, so, roughly, 25$ per mailing

Mailchimp: 1300$/mth

Those guys are kind of funny. When you land on their pricing page, they preset you with 500 contacts and charge you 23$/mth for "Essential", and 410$/mth for "Premium". But when you scroll your contacts up to 250k+, all boxes get greyed out and the "talk to sales" phone number replaces the price. The last quoted price, at 200k contacts, is 1300$USD per month.

References

Comments

    Please register or sign in to add a comment.