---
title: TPA-RFC-31: Outsource email services
---

[[_TOC_]]

Summary: TODO

# Background

In late 2021, the TPA team adopted the following first Objective and
Key Results (OKR):

> [Improve mail services][OKR]:
> 
>  1. David doesn't complain about "mail getting into spam" anymore
>  2. RT is not full of spam
>  3. we can deliver and receive mail from state.gov

There were two ways of implementing solutions to this problem. One way
was to complete the implementation of email services internally,
adding standard tools like DKIM, SPF, and so on to our services and
hosting mailboxes. This approach was investigated fully in
[TPA-RFC-15][] but was ultimately rejected as too risky.

[DMARC]: https://en.wikipedia.org/wiki/DMARC
[DKIM]: https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail
[SPF]: http://www.open-spf.org/

Instead, we are looking at the other solution to this problem which is
to outsource all or a part of our mail services to some external
provider. This proposal aims at clarifying which services we should
outsource, and to whom.

# Proposal

Progressively migrate all `@torproject.org` email aliases and forwards
to the new provider.

Retirement of the "submission service" and destruction of the
`submit-01` server, after migration of all users to the new provider.

## Requirements

Those are the requirements the external service provider must fulfill
before being considered for this proposal.

### Email interfaces

We have users currently using Gmail, Thunderbird, Apple Mail, Outlook,
and other email clients. Some people keep their mail on the server,
some fetch it once and never keep a copy. Some people read their mail
on their phone.

Therefore, the new provider MUST offer IMAP and POP mailboxes,
alongside a modern and mobile-friendly Webmail client.

It MUST be possible for users (and machines) to submit emails using a
username/password combination through a dedicated SMTP server (also
known as a "submission port"). Ideally, this could be done with TLS
certificates, especially for client machines.

Some users are unlikely to leave Gmail, and should be able to forward
their email there. Inversely, they should be able to send mail *from*
Gmail through a submission address.

### Deliverability

Provider SHOULD be able to reliably deliver email to both large
service providers like Gmail and Outlook, but also government sites
like `state.gov` or other, smaller mail servers.

Therefore, modern email standards like SPF, DKIM, DMARC, and hopefully
ARC SHOULD be implemented by the new provider.

We also often perform mass mailings to announce software releases
(through Mailman 2, soon 3) but also larger fundraising mailings
through CiviCRM, which contact almost 300,000 users every
month. Provisions must therefore be made for those services to keep
functioning, possibly through a dedicated mail submission host as
well. Servers which currently send regular emails to end users
include:

 * CiviCRM: 270,000 emails in June 2021
 * Mailman: 12,600 members on tor-announce
 * RT: support tracker, 15,000 registered users?
 * GitLab: ~2,000 active users
 * Discourse: ~1,000 monthly active users

Part of our work involves using email to communicate to fundraiser but
also people in censored country, so censorship resistance is
important. Ideally, a Tor `.onion` service should be provided for
email submission, for example. 

Also, we have special email services like [gettor.torproject.org](https://gettor.torproject.org/)
which send bridges or download links for accessing the Tor
network. Those should also keep functioning properly, but should also
be resistant to attacks aiming to list all bridges, for example. This
is currently done by checking incoming DKIM signature and limiting the
service to certain providers.

Non-mail machines will relay their mail through a new internal relay
server that will then submit its mail to the new provider. This will
help us automate configuration of "regular" email server to avoid
having to create an account in the new provider's control panel every
time we setup a new server.

### Spam control

State of the art spam filtering software MUST be deployed to keep the
bulk of spam from reaching user's mail boxes, or hopefully triage them
in a separate "Spam folder".

Bayesian training MAY be used to improve the accuracy of those filters
and the user should be able to train the algorithm to allow certain
emails to go through.

### Privacy

We are an organisation that takes user privacy seriously. Under
absolutely no circumstances should email contents or metadata be used
in other fashion than for delivering mail to its destination or
aforementioned spam control. Ideally, mail boxes would be encrypted
with a user-controlled key so that the provider may not be able to
read the contents of mailboxes at all.

Strong log file anonymization is expected or at least aggressive log
rotation should be enforced.

Privately identifiable information (e.g. client IP address) MUST NOT
leak through email headers.

We strongly believe that free and open source software is the only way
to ensure privacy guarantees like these are enforceable. At least the
services provided MUST be usable with standard, free software email
clients (for example, Thunderbird).

## Scope

This proposal doesn't address the fate of Schleuder or Mailman (or,
for that matter, Discourse, RT, or other services that may use email
unless explicitly mentioned).

It also does *not* address directly phishing and scamming attacks
([issue 40596][]), but it is hoped that stricter
enforcement of email standards will reduce those to a certain
extent. 

TODO: Those architectural details may change through negotiations with
the new provider.

## Affected users

This affects all users which interact with `torproject.org` and its
subdomains over email. It particularly affects all "tor-internal"
users, users with LDAP accounts or forwards under `@torproject.org`.

TODO: what will actually happen? maybe delegate this to personas?

## Architecture diagram

Those diagrams detail the infrastructure before and after the changes
detailed above.

Legend:

 * red: legacy hosts, mostly eugeni services, no change
 * orange: hosts that manage and/or send their own email, no change
   except the mail exchanger might be the one relaying the
   `@torproject.org` mail to it instead of eugeni
 * green: new hosts, might be multiple replicas
 * rectangles: machines
 * triangle: the user
 * ellipse: the rest of the internet, other mail hosts not managed by tpo

### Before

TODO: taken from TPA-RFC-15, review

![current mail architecture diagram](tpa-rfc-31-outsource-email/architecture-pre.png)

### After

TODO: make new post-proposal architecture diagram

## Actual changes

TODO: task breakdown, this was one subheading per task in TPA-RFC-15

### New mail transfer agent

Configure new "mail transfer agent" server(s) to relay mails from
servers that do not send their own email, replacing a part of
`eugeni`.

This server will be called `mta-01.torproject.org` and could be
horizontally scaled up for availability. See also the [Naming
things](#naming-things) Challenge below.

## Cost estimates

TODO: estimate cost of tasks above

## Timeline

TODO: when are we going to do this?

## Challenges

### Delays

In early June 2021, it became apparent that we were having more
problems delivering mail to Gmail, possibly because of the lack of
DKIM records (see for example [tpo/tpa/team#40765](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40765)). We may have
had time to implement some countermeasures, had TPA-RFC-15 been
adopted, but alas, we decided to go with an external provider.

It is unclear, at this point, whether this will speed things up or
not. We may take too much time deliberating on the right service
provider, or this very specification, or find problems during the
migrations, which may make email even more unreliable.

### Naming things

In TPA-RFC-15, it became apparent that the difficulty of naming things
did not escape those proposals. For example, in TPA-RFC-15, the term
"relay" has been used liberally to talk about a new email server
processing email for other servers. That terminology, unfortunately,
clashes with the term "relay" used extensively in the Tor network to
designate "Tor relays", which create circuits that make up the Tor
network.

This is the reason why the `mta-01` server is named a "MTA" and not a
"relay" or "submission" server. The former is reserved for Tor relays,
and the latter for "email submission servers" provided by
upstream. Technically, the difference between the "MTA" and the
"submission" server is that the latter is expected to deliver the
email out of the responsibility of the `torproject.org` domain, to its
final destination, while the MTA is allowed to transfer to another MTA
or submission server.

### Aging Puppet code base and other legacy

TODO: copy from TPA-RFC-15?

### Security and privacy issues

TODO: 2FA? trust?

### Duplication of services with LDAP

TODO: How will users be authenticated? can we bridge with LDAP?

### Upstream support burden

It is unclear how we will handle support requests: will users directly
file issues upstream, or will this be handled by TPA first?

How will password resets be done, for example?

### Sunk costs

There has been a *lot* of work done in the current email
infrastructure. In particular, anarcat spent a significant amount of
time changing the LDAP services to allow the addition of an "email
password" and deploying those to a "submission server" to allow people
to submit email through the `torproject.org` infrastructure. The
TPA-RFC-15 design and proposal will also go to waste with this
proposal.

# Personas

TODO: what this will mean for everyone, possibly reusing our personas
from TPA-RFC-15

# Other alternatives

## Hosting our own email

TODO: that's TPA-RFC-15, detail why rejected

## Status quo

The status quo situation is similar (if not worse) than the status quo
described in TPA-RFC-15: email services are suffering from major
deliverability problems and things are only going to get worse over
time, up to a point when no one will use their `@torproject.org` email
address.

## The end of email

This is similar to the discussion mentioned in TPA-RFC-15: email is
still a vital service and we cannot at the moment consider completely
replacing it with other tools.

## Providers evaluation

TODO: update those numbers, currently taken directly from TPA-RFC-15, without change

### Fastmail: 6,000$/year

Fastmail were not contacted directly but [their pricing page][] says
about 5$USD/user-month, with a free 30-day trial. This amounts to
500$/mth or 6,000$/year.

It's unclear if we could do mass-mailing with this service.

[their pricing page]: https://www.fastmail.com/pricing/

### Gandi: 480$-2400$/year

Gandi, the DNS provider, also offers [mailbox services][] which are
priced at 0.40$/user-month (3GB mailboxes) or 2.00$/user-month (50GB).

It's unclear if we could do mass-mailing with this service.

[mailbox services]: https://www.gandi.net/en-US/domain/email

### Google: 10,000$/year

Google were not contacted directly, but [their promotional site][]
says it's "Free for 14 days, then 7.80$ per user per month", which,
for tor-internal (~100 users), would be 780$/month or ~10,000USD/year.

We probably wouldn't be able to do mass mailing with this service.

[their promotional site]: https://workspace.google.com/solutions/new-business/

### Greenhost: ~1600€/year, negotiable

We had a quote from Greenhost for 129€/mth for a Zimbra frontend with
a VM for mailboxes, DKIM, SPF records and all that jazz. The price
includes an office hours SLA.

### Mailcow: 480€/year

[Mailcow][] is interesting because they actually are based on a [free
software stack][] (based on PHP, Dovecot, Sogo, rspamd, postfix,
nginx, redis, memcached, solr, Oley, and Docker containers). They
offer a [hosted service][] for 40€/month, with a 100GB disk quota and
no mailbox limitations (which, in our case, would mean 1GB/user).

We also get full admin access to the control panel and, given their
infrastructure, we could self-host if needed. Integration with our
current services would be, however, tricky.

It's there unclear if we could do mass-mailing with this service.

[hosted service]: https://www.servercow.de/mailcow?lang=en#managed
[free software stack]: https://github.com/mailcow/mailcow-dockerized
[Mailcow]: https://mailcow.email/

### Mailfence: 2,500€/year, 1750€ setup

The [mailfence business page][] doesn't have prices but last time we
looked at this, it was a 1750€ setup fee with 2.5€ per user-year.

It's unclear if we could do mass-mailing with this service.

[mailfence business page]: https://mailfence.com/en/secure-business-email.jsp

### Riseup

Riseup already hosts a significant number of email accounts by virtue
of being the target of `@torproject.org` forwards. During the [last
inventory][], we found that, out of 91 active LDAP accounts, 30 were
being forwarded to `riseup.net`, so about 30%.

Riseup supports webmail, IMAP, and, more importantly, encrypted
mailboxes. While it's possible that an hostile attacker or staff could
modify the code to inspect a mailbox's content, it's leagues ahead of
most other providers in terms of privacy.

Riseup's prices are not public, but they are close to "market" prices
quoted below.

[last inventory]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/roadmap/2022

# Approval

This proposal will need the approval of TPA, tor-internal, and TPI,
the latter which will have the final word on the decision.

# Deadline

TODO: establish deadline

# Status

This proposal is currently in the `draft` state.

# References

 * this work is part of the [improve mail services OKR][OKR], part of the
   [2022 roadmap][], Q1/Q2
 * this proposal is specifically discussed in [tpo/tpa/team#40798][]

[tpo/tpa/team#40798]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40798
[OKR]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/4
