---
title: TPA-RFC-15: email services
---

[[_TOC_]]

Summary: deploy incoming and outgoing [SPF][], [DKIM][], [DMARC][],
and (possibly) [ARC][] checks and records on torproject.org
infrastructure. Deploy an IMAP service, alongside enforcement of the
use of the submission server for outgoing mail. Establish end-to-end
deliverability monitoring. Rebuild mail services to get rid of legacy
infrastructure.

[DMARC]: https://en.wikipedia.org/wiki/DMARC
[DKIM]: https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail
[SPF]: http://www.open-spf.org/

# Background

In late 2021, the TPA team adopted the following first Objective and
Key Results (OKR):

> [Improve mail services][OKR]:
> 
>  1. David doesn't complain about "mail getting into spam" anymore
>  2. RT is not full of spam
>  3. we can deliver and receive mail from state.gov

This seemingly simple objective actually involves major changes to the
way email is handled on the `torproject.org` domain. Specifically, we
believe we will need to implement standards like SPF, DKIM, and DMARC
to have our mail properly delivered to large email providers, on top
of keeping hostile parties from falsely impersonating us.

## Current status

Email has traditionally been completely decentralised at Tor: while we
would support forwarding emails `@torproject.org` to other mailboxes,
we have never offered mailboxes directly, nor did we offer ways for
users to send emails themselves through our infrastructure.

This situation led to users sending email with `@torproject.org` email
addresses from arbitrary locations on the internet: Gmail, Riseup, and
other service providers (including personal mail servers) are
typically used to send email for `torproject.org` users.

This changed at the end of 2021 when the new [submission service][]
came online. We still, however, have limited adoption of this service,
with only 16 users registered compared to the ~100 users in LDAP.

In parallel, we have historically not adopted any modern email
standards like SPF, DKIM, or DMARC. But more recently, we added SPF
records to both the Mailman and CiviCRM servers (see [issue
40347][]).

We have also been processing DKIM headers on incoming emails on the
`bridges.torproject.org` server, but that is an exception. Finally, we
are running Spamassassin on the RT server to try to deal with the
large influx of spam on the generic support addresses (`support@`,
`info@`, etc) that the server processes. We do not process SPF records
on incoming mail in any way, which has caused problems with Hetzner
([issue 40539][]).

We do not have any DMARC headers anywhere in DNS, but we do have
workarounds setup in Mailman for delivering email correctly when the
sender has DMARC records, since September 2021 (see [issue 19914][]).

We do not offer mailboxes, although we do have Dovecot servers
deployed for specific purposes. The GitLab and CiviCRM servers, for
example, use it for incoming email processing, and the submission
server uses it for authentication.

[submission service]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/submission
[issue 40347]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40347
[issue 40539]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40539
[issue 19914]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/19914

### Processing mail servers

Those servers handle their own outgoing email (ie. they do *not* go
through `eugeni`) and handle incoming email as well, unless otherwise
noted:

 * BridgeDB (`polyanthum`)
 * CiviCRM (`crm-int-01`, Dovecot)
 * Gettor (`gettor-01`)
 * GitLab (`gitlab-02`)
 * LDAP (`alberti`)
 * MTA (`eugeni`)
 * Nagios/Icinga (`hetzner-hel1-01`, no incoming)
 * Prometheus (`prometheus-02`, no incoming)
 * RT (`rude`)
 * Submission (`submit-01`)

Surprisingly, the Gitolite service (`cupani`) does *not* relay mail
through the MTA (`eugeni`).

## Known issues

The current email infrastructure has many problems.  In general,
people feel like their emails are not being delivered or "getting into
spam". And sometimes, in the other direction, people simply cannot get
mail from certain domains.

Here are the currently documented problems:

 * deliverability issues: [Yahoo][], [state.gov][], [Gmail][], [Gmail
   again][]
 * reception issues: state.gov
 * complaints about lists.tpo lacking SPF/DKIM ([issue 40347][])
 * submission server incompatible with Apple Mail/Outlook (see [issue
   40586][])
 * email infrastructure has multiple single points of failure ([issue
   40604][])

Interlocking issues:

 * outgoing SPF deployment requires everyone to use the submission
   mail server, or at least have their server added to SPF
 * outgoing DKIM deployment requires testing and integration with DNS
   (and therefore possibly ldap)
 * outgoing DMARC deployment requires submission mail server adoption
   as well
 * SPF and DKIM require DMARC to properly function
 * DMARC requires a monitoring system to be effectively enabled

In general, we lack end-to-end deliverability tests to see if any
measures we take have an impact ([issue 40494][]).

[Yahoo]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/34134
[state.gov]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40202
[Gmail]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40170
[Gmail again]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40149
[issue 40586]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40586
[issue 40604]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40604
[issue 40494]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40494

## Previous evaluations

As part of the submission service launch, we did [an evaluation][]
that is complementary to this one. It evaluated the costs of hosting
various levels of our mail from "none at all" to "everything including
mailboxes", before settling on only the submission server as a
compromise.

It did not touch on email standards like this proposal does.

[an evaluation]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/submission#discussion

# Proposal

After a grace period, we progressively add "soft", then "hard" SPF,
DKIM, and DMARC record to the `lists.torproject.org`,
`crm.torproject.org`, `rt.torproject.org`, and, ultimately,
`torproject.org` domains.

This deployment will be paired with end to end deliverability tests
alongside "reports" analysis (from DMARC, mainly).

An IMAP server with a webmail is configured on a new server. A new
mail exchanger and relay are setup.

This assumes that, during the grace period, everyone eventually adopts
the submission server for outgoing email, or stop using their
`@torproject.org` email address for outgoing mail.

## Scope

This proposal affects SPF, DKIM, DMARC, and possibly ARC record for
outgoing mail, on all domains managed by TPA, specifically the domain
`torproject.org` and its subdomains. It explicitly does not cover the
`torproject.net` domain.

It also includes offering small mailboxes with IMAP and webmail
services to our users that desire one, and enforces the use of the
already deployed submission server. Server-side mailbox encryption
(Riseup's TREES or Dovecot's encryption) is out of scope at first.

It also affects incoming email delivery on all `torproject.org`
domains and subdomains, which will be filtered for SPF, DKIM, and
DMARC record alongside spam filtering.

This proposal doesn't address the fate of Schleuder or Mailman (or,
for that matter, Discourse, RT, or other services that may use email
unless explicitly mentioned).

It also does *not* address directly phishing and scamming attacks
([issue 40596][]), but it is hoped that stricter
enforcement of email standards will reduce those to a certain
extent. The rebuild of certain parts of the legacy infrastructure will
also help deal with such attacks in the future.

[issue 40596]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40596

## Affected users

This affects all users which interact with `torproject.org` and its
subdomains over email. It particularly affects all "tor-internal"
users, users with LDAP accounts or forwards under `@torproject.org`.

It especially affects users which send email from their own provider
or another provider than the [submission service][]. Those users will
eventually be unable to send mail with a `torproject.org` email
address.

## Actual changes

The actual changes proposed here are divided in smaller chunks,
described in detail below:

 1. End-to-end deliverability checks
 2. DMARC reports analysis
 3. DKIM and ARC signatures
 4. IMAP deployment
 5. SPF/DMARC records
 6. Incoming mail filtering
 7. New mail exchangers
 8. New mail relays
 9. Puppet refactoring

### End-to-end deliverability checks

End-to-end deliverability monitoring involves:
    
 * actual delivery roundtrips
 * block list checks
 * DMARC/MTA-STS feedback loops (covered below)

This may be implemented as Nagios or Prometheus checks ([issue
40539][]). This also includes evaluating how to monitor metrics
offered by [Google postmaster tools][] and Microsoft ([issue
40168][]).

[issue 40168]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40168
[Google postmaster tools]: https://postmaster.google.com

### DMARC reports analysis

DMARC reports analysis are also covered by [issue 40539][], but are
implemented separately because they are considered to be more complex
(e.g. RBL and e2e delivery checks are already present in Nagios).

This might also include extra work for MTA-STS feedback loops.

### IMAP deployment

This consists of an IMAP and webmail server deployment.

We are currently already using Dovecot in a limited way on some
servers, so we will reuse some of that Puppet code for the IMAP
server. The webmail will likely be deployed with [Roundcube][], alongside
the IMAP server. Both programs are packaged and well supported in
Debian. Alternatives like [Rainloop][] or [Snappymail][] could be considered.

Mail filtering is detailed in another section below.

[Roundcube]: https://roundcube.net/
[Rainloop]: https://www.rainloop.net/
[Snappymail]: https://snappymail.eu/

### Incoming mail filtering

Deploy a tool for inspection of incoming mail for SPF, DKIM, DMARC
records, affecting either "reputation" (e.g. add a marker in mail
headers) or just downright rejection (e.g. rejecting mail before
queue).

We currently use Spamassassin for this purpose, and we could consider
collaborating with the [Debian listmasters][] for the Spamassassin
rules. [rspamd][] should also be evaluated as part of this work to see
if it is a viable alternative.

[Debian listmasters]: https://salsa.debian.org/debian-listmasters/spamassassin_config
[rspamd]: https://rspamd.com/

### New mail exchangers

Configure new "mail exchanger" (MX) server(s) with TLS certificates
signed by a public CA, most likely Let's Encrypt for incoming mail,
replacing a part of `eugeni`.

### New mail relays

Configure new "mail relay" server(s) to relay mails from servers that
do not send their own email, replacing a part of `eugeni`. Those are
temporarily called `submission-tls` but could be named something else,
see the [Naming things](#naming-things) Challenge below.

This is similar to current submission server, except with TLS
authentication instead of password.

### DKIM and ARC signatures

Implement outgoing DKIM signatures, probably with OpenDKIM. This will
actually involve deploying that configuration on *any* server that
produces outgoing email. Each of those servers (listed in "Processing
mail servers" above) will therefore require its own DKIM records and
running a copy of the DKIM configuration.

### SPF/DMARC records

Deploy of SPF and DMARC DNS records to a strict list of allowed
servers. This list should include any email servers that send their
own email (without going through the relay, currently `eugeni`),
listed in the "Processing mail servers" section.

This will impact users not on the submission and IMAP servers. This
includes users with plain forwards and without an LDAP account.

Possible solutions for those users include:

 0. users adopt the submission server for outgoing mail,
 1. or aliases are removed,
 2. or transformed into LDAP accounts,
 3. or forwards can't be used for outgoing mail,
 4. or forwarded emails are rewritten (e.g. [SRS][])
    
This goes in hand with the [email policy problem][] which is basically
the question of what service can be used for (e.g. forwards vs lists
vs RT). In general, email forwarding causes all sorts of problems and
we may want to consider, in the long term, other options for many
aliases, either mailing lists or issue trackers. That question is out
of scope of this proposal for now. See also the broader [End of
Email][] discussion.

[SRS]: https://en.wikipedia.org/wiki/Sender_Rewriting_Scheme
[End of Email]: #the-end-of-email

### Puppet refactoring

Refactor the mail-related code in Puppet, and reconfigure all servers
according to the mail relay server change above, see [issue
40626][] for details. This should probably happen
*before* or *during* all the other tasks.

[issue 40626]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40626

## Architecture diagram

Those diagrams detail the infrastructure before and after the changes
detailed above.

Legend:

 * red: legacy hosts, mostly eugeni services, no change
 * orange: hosts that manage and/or send their own email, no change
   except the mail exchanger might be the one relaying the
   `@torproject.org` mail to it instead of eugeni
 * green: new hosts, might be multiple replicas
 * rectangles: machines
 * triangle: the user
 * ellipse: the rest of the internet, other mail hosts not managed by tpo

### Before

![current mail architecture diagram](tpa-rfc-15-email-services/architecture-pre.png)

### After

![final mail architecture diagram](tpa-rfc-15-email-services/architecture-post.png)

Changes in this diagram:

 * added: `submission-tls`, `mx`, `mailbox`, the hosts defined in steps e, g,
   and h above
 * changed:
   * `eugeni` stops relaying email for all the hosts and stops
     receiving mail for the `torproject.org` domain, but keeps doing
     mailman and schleuder work
   * other TPA hosts: start relaying mail through `relay` instead of
     `eugeni`
   * "impersonators": those are external mail relays like gmail or
     riseup, or individual mail servers operated by TPO personnel
     which previously could send email as `@torproject.org` but will
     likely be unable to. they can still receive forwards for those
     emails, but those will come from the `mx` instead of `eugeni`.
   * users will start submitting email through the `submission` server
     (already possible, now mandatory) and read email through the
     `mailbox` server

## Timeline

The changes will be distributed over a year, and the following is a
per-quarter breakdown, starting from when the proposal is adopted.

Obviously, the deployment will depend on availability of TPA staff and
the collaboration of TPO members. It might also be reordered to
prioritize more urgent problems that come up. The complaints we
received from Hetzner, for example should probably be a priority
([issue 40539][]).

 * 2022 Q2:
   * End-to-end deliverability checks
   * DMARC reports analysis (DMARC record `p=none`)
   * partial incoming mail filtering (bridges, lists, tpo, [issue
     40539][])
   * progressive adoption of submission server
   * Puppet refactoring
 * 2022 Q3:
   * IMAP and webmail server deployment
   * mail exchanger deployment
   * relay server deployment
   * global incoming mail filtering
   * deadline for adoption of the submission server
 * 2022 Q4:
   * DKIM and ARC signatures
   * SPF records, "soft" (`~all`)
 * 2023 Q1:
   * hard DMARC (`p=reject`) and SPF (`-all`) records

## Challenges

### Aging Puppet code base

This deployment will require a lot of work on the Puppet modules,
since our current codebase around email services is a little old and
hard to modify. We will need to spend some time to refactor and
cleanup that codebase before we can move ahead with more complicated
solutions like incoming SPF checks or outgoing DKIM signatures, for
example. See [issue 40626][] for details.

### Incoming filtering implementation

Some research work will need to be done to determine the right tools
to use to deploy the various checks on incoming mail.

For DKIM, OpenDKIM is a well established program and standard used in
many locations, and it is not expected to cause problems in
deployment, software wise.

Our LDAP server already has support for per-user DKIM records, but we
will probably ignore that functionality and setup separate DKIM
records, maintained manually.

It's currently unclear how [ARC][] would be implemented, as the known
implementations ([OpenARC][] and [Fastmail's authentication milter][])
were not packaged in Debian at the time of writing. ARC can help with riseup -> TPO -> riseup
forwarding trips, which can be marked as spam by riseup.

(Update: [OpenARC is now in Debian][].)

[OpenARC is now in Debian]: https://tracker.debian.org/pkg/openarc

Other things to be careful about:

 * watch out for [DKIM replay attacks][]

 * decide key rotation policy (how frequently, should we [publish
   private keys][], see also [this guide][])

[ARC]: http://arc-spec.org/
[OpenARC]: https://github.com/trusteddomainproject/OpenARC
[Fastmail's authentication milter]: https://github.com/fastmail/authentication_milter
[email policy problem]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40404
[publish private keys]: https://blog.cryptographyengineering.com/2020/11/16/ok-google-please-publish-your-dkim-secret-keys/
[DKIM replay attacks]: https://utcc.utoronto.ca/~cks/space/blog/spam/DKIMSpamReplayAttack
[this guide]: https://rya.nc/dkim-privates.html

### Security concerns

The proposed architecture does not offer users two-factor
authentication (2FA) and could therefore be considered less secure
than other commercial alternatives. Implementing 2FA in the context of
our current LDAP service would be a difficult challenge.

Hosting people's email contents adds a new security
concern. Typically, we are not very worried about "leaks" inside TPA
infrastructure, except in rare situations (like bridgedb). Most of the
data we host is public, in other words. If we start hosting mailboxes,
we suddenly have a much higher risk of leaking personal data in case
of compromise. This is a trade-off with the privacy we gain from not
giving that data to a third party.

### Naming things

Throughout this document, the term "relay" has been used liberally to
talk about a new email server processing email for other
servers. That terminology, unfortunately, clashes with the term
"relay" used extensively in the Tor network to designate "Tor relays",
which create circuits that make up the Tor network.

As a stopgap measure, the new relays were called `submission-tls` in
the architecture diagram, but that is also problematic because it
might be confused with the current `submission` server, which serves a
very specific purpose of relaying mail for *users*.

Technically, the `submission` server and the `submission-tls` servers
are both MTA, or a [Message Transfer Agent][MTA]. Maybe that terminology
could be used for the new "relay" servers to disambiguate them from
the submission server, for example the first relay would be called `mta-01.torproject.org`.

Or, inversely, we might want to consider both servers to be the same
and both name them `submission` and have the `submission` service also
accept mail from other TPO servers over TLS. So far that approach has
been discarded to separate those tasks, as it seemed simpler
architecturally.

 [MTA]: https://en.wikipedia.org/wiki/Message_transfer_agent

## Cost estimates

Summary:

 * setup: about four months, about 32,000EUR staff, 200EUR hardware
 * ongoing: unsure, between one day a week or a month, so about
   5,000-20,000EUR/year in staff
 * hardware costs: possibly up to 2200EUR/year

### Staff

This is an estimate of the time it will take to complete this project,
based on the tasks established in the [actual changes section][]. The
process follows the [Kaplan-Moss estimation technique][].

[actual changes section]: #actual-changes
[Kaplan-Moss estimation technique]: https://jacobian.org/2021/may/25/my-estimation-technique/

| Task                       | Estimate    | Uncertainty | Note                                          | Total (days) |
|----------------------------|-------------|-------------|-----------------------------------------------|--------------|
| 1. e2e deliver. checks     | 3 days      | medium      | access to other providers uncertain           | 4.5          |
| 2. DMARC reports           | 1 week      | high        | needs research                                | 10           |
| 3. DKIM signing            | 3 days      | medium      | expiration policy and per-user keys uncertain | 4.5          |
| 4. IMAP deployment         | 2 weeks     | high        | may require training to onboard users         | 20           |
| 5. SPF/DMARC records       | 3 days      | high        | impact on forwards unclear, SRS               | 7            |
| 6. incoming mail filtering | 1 weeks     | high        | needs research                                | 10           |
| 7. new MX                  | 1 weeks     | high        | key part of eugeni, might be hard             | 10           |
| 8. new mail relays         | 3 days      | low         | similar to current submission server          | 3.3          |
| 9. Puppet refactoring      | 1 weeks     | high        |                                               | 10           |
| **Total**                  | **8 weeks** | **high**    |                                               | **80**       |

This amounts to a total estimate time of 80 days, or about 16 weeks or
four months, full time. At 50EUR/hr, that's about 32,000EUR of work.

This estimate doesn't cover for ongoing maintenance costs and support
associated with running the service. So far, the submission server has
yielded little support requests. After a bumpy start requiring patches
to userdir-ldap and a little documentation, things ran rather
smoothly.

It is possible, however, that the remaining 85% of users that do *not*
currently use the submission server might require extra hand-holding,
so that's one variable that is not currently considered. Furthermore,
we do not have any IMAP service now and this will require extra
onboarding, training and documentation

We should consider at least one person-day per month, possibly even
per week, which gives us a range of 12 to 52 days of work, for an
extra cost of 5,000-20,000EUR, per year.

### Hardware

In [the submission service hosting cost evaluation][], the hardware
costs related to mailboxes were evaluated at about 2500EUR/year with a
200EUR setup fee, hardware wise. Those numbers are from 2019, however,
so let's review them.

[the submission service hosting cost evaluation]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/submission#internal-hosting-cost-evaluation

Assumptions are similar:

 * each mailbox is on average, a maximum of 10GB
 * 100 mailboxes maximum at first (so 1TB of storage required)
 * LUKS full disk encryption
 * IMAP and basic webmail (Roundcube or Rainloop)

We account for two new boxes, in the worst case, to cover for the
service:

 * Hetzner px62nvme 2x1TB RAID-1 64GB RAM 74EUR/mth, 888EUR/yr
   (1EUR/mth less)
 * Hetzner px92 2x1TB SSD RAID-1 128GB RAM 109EUR/mth, 1308EUR/yr
   (6EUR/mth less)
 * Total hardware: 2196EUR/yr, ~200EUR setup fee

This assumes hosting the server on a dedicated server at Hetzner.  It
might be possible (and more reliable) to ensure further cost savings
by hosting it on our shared virtualized infrastructure.

# Examples

Here we collect a few "personas" and try to see how the changes will
affect them.

> We have taken the liberty of creating mostly fictitious personas,
> but they are somewhat based on real-life people. We do not mean to
> offend. Any similarity that might seem offensive is an honest
> mistake on our part which we will be happy to correct. Also note
> that we might have mixed up people together, or forgot some. If your
> use case is not mentioned here, please do report it. We don't need
> to have *exactly* "you" here, but all your current use cases should
> be covered by one or many personas.

## Ariel, the fundraiser

Ariel does a lot of mailing. From talking to fundraisers through their
normal inbox to doing mass newsletters to thousands of people on
CiviCRM, they get a lot of shit done and make sure we have bread on
the table at the end of the month. They're awesome and we want to make
them happy.

Email is absolutely mission critical for them. Sometimes email gets
lost and that's a huge problem. They frequently tell partners their
personal Gmail account address to workaround those problems. Sometimes
they send individual emails through CiviCRM because it doesn't work
through Gmail!

Their email is forwarded to Google Mail and they do *not* have an LDAP
account.

They will need to get an LDAP acount, set a mail password, and either
use the Webmail service or configure a mail client like Thunderbird to
access the IMAP server and submit email through the submission server.

Technically, it would also be possible to keep using Gmail to send
email as long as it is configured to relay mail through the submission
server, but that configuration will be unsupported.

## Gary, the support guy

Gary is the ticket master. He eats tickets for breakfast, then files
10 more before coffee. A hundred tickets is just a normal day at the
office. Tickets come in through email, RT, Discourse, Telegram,
Snapchat and soon, TikTok dances.

Email is absolutely mission critical, but some days he wishes there
could be slightly less of it. He deals with a lot of spam, and surely
something could be done about that.

His mail forwards to Riseup and he reads his mail over Thunderbird and
sometimes webmail.

He will need to reconfigure his Thunderbird to use the submission and
IMAP server after setting up an email password. The incoming mail
checks should improve the spam situation. He will need, however, to
abandon Riseup for TPO-related email, since Riseup cannot be
configured to relay mail through the submission server.

## John, the external contractor

John is a freelance contractor that's really into privacy. He runs his
own relays with some cools hacks on Amazon, automatically deployed
with Terraform. He typically run his own infra in the cloud, but
for email he just got tired of fighting and moved his stuff to
Microsoft's Office 365 and Outlook.

Email is important, but not absolutely mission critical. The
submission server doesn't currently work because Outlook doesn't allow
you to add just an SMTP server.

He'll have to reconfigure his Outlook to send mail through the
submission server and use the IMAP service as a backend.

## Nancy, the fancy sysadmin

Nancy has all the elite skills in the world. She can configure a
Postfix server with her left hand while her right hand writes the
Puppet manifest for the Dovecot authentication backend. She knows her
shit. She browses her mail through a UUCP over SSH tunnel using
mutt. She runs her own mail server in her basement since 1996.

Email is a pain in the back and she kind of hates it, but she still
believes everyone should be entitled to run their own mail server.

Her email is, of course, hosted on her own mail server, and she have
an LDAP account.

She will have to reconfigure her Postfix server to relay mail through
the submission or relay servers, if she want to go fancy. To read
email, she will need to download email from the IMAP server, although
it will still be technically possible to forward her `@torproject.org`
email to her personal server directly, as long as the server is
configured to send email through the TPO servers.

## Mallory, the director

Mallory also does a lot of mailing. She's on about a dozen aliases and
mailing lists from accounting to HR and other obscure ones everyone
forgot what they're for. She also deals with funders, job applicants,
contractors and staff.

Email is absolutely mission critical for her. She often fails to
contact funders and critical partners because state.gov blocks our
email (or we block theirs!). Sometimes, she gets told through LinkedIn
that a job application failed, because mail bounced at Gmail.

She has an LDAP account and it forwards to Gmail. She uses Apple Mail
to read their mail.

For her Mac, she'll need to configure the submission server *and* the
IMAP server in Apple Mail. Like Ariel, it is technically possible for
her to keep using Gmail, but that is unsupported.

The new mail relay servers should be able to receive mail state.gov
properly. Because of the better reputation related to the new
SPF/DKIM/DMARC records, mail should bounce less (but still may
sometimes end up in spam) at Gmail.

## Orpheus, the developer

Orpheus doesn't particular like or dislike email, but sometimes has to
use it to talk to people instead of compilers. They sometimes have to
talk to funders (`#grantlife`) and researchers and mailing lists, and
that often happens over email. Sometimes email is used to get
important things like ticket updates from GitLab or security
disclosures from third parties.

They have an LDAP account and it forwards to their self-hosted mail
server on a OVH virtual machine.

Email is not mission critical, but it's pretty annoying when it
doesn't work.

They will have to reconfigure their mail server to relay mail through
the submission server. They will also likely start using the IMAP
server.

## Blipblop, the bot

Blipblop is not a real human being, it's a program that receives mails
from humans and acts on them. It can send you a list of bridges
(bridgedb), or a copy of the Tor program (gettor), when requested. It
has a brother bot called Nagios/Icinga who also sends unsolicited mail
when things fail. Both of those should continue working properly, but
will have to be added to SPF records and an adequate OpenDKIM
configuration should be deployed on those hosts as well.

There's also a bot which sends email when commits get pushed to
gitolite. That bot is deprecated and is likely to go away.

In general, attention will be given to those precious little bots we
have everywhere that send their own email. They will be taken care of,
as much as humanely possible.

# Other alternatives

Those are other alternatives that were considered as part of drafting
this proposal. None of those options is considered truly viable from a
technical perspective, except possibly external hosting, which remains
to be investigated and discussed further.

## No mailboxes

An earlier draft of this proposal considered changing the
infrastructure to add only a mail exchanger and a relay, alongside all
the DNS changes (SPF, DKIM, DMARC).

We realized the IMAP was a requirement requirement because the SPF
records will *require* people to start using the submission server to
send mail. And that, in turn requires an IMAP server because of
clients limitations. For example, it's not possible to configure Apple
mail of Office 365 with a remote SMTP server unless they also provide
an IMAP service, see [issue 40586][] for details.

It's also possible that implementing mailboxes could help improve spam
filtering capabilities, which are after all necessary to ensure good
reputation with hosts we currently relay mail to.

Finally, it's possible that we will not be able to make "hard"
decisions about policies like SPF, DKIM, or DMARC and would be forced
to implement a "rating" system for incoming mail, which would be
difficult to deploy without user mailboxes, especially for feedback
loops.

There's a lot of uncertainty regarding incoming email filtering, but
that is a problem we need to solve in the current setup anyways, so we
don't believe the extra costs of this would be significant. At worst,
training would require extra server resources and staff time for
deployment. User support might require more time than with a plain
forwarding setup, however.

## High availability setup

We have not explicitly designed this proposal for high availability
situations, which have been explicitly requested in [issue
40604][]. The current design is actually more scalable
than the previous legacy setup, because each machine will be setup by
Puppet and highly reproducible, with minimal local state (except for
the IMAP server). So while it *may* be possible to scale up the
service for higher availability in the future, it's not a mandatory
part of the work described here.

In particular, setting up new mail exchanger and submission servers is
somewhat trivial. It consists of setting up new machines in separate
locations and following the install procedure. There is no state
replicated between the servers other than what is already done through
[LDAP][].

The IMAP service is another problem, however. It will potentially have
large storage requirements (terabytes) and will be difficult to
replicate using our current tool set. We may consider setting it up on
bare metal to avoid the performance costs of the Ganeti cluster,
which, in turn, may make it vulnerable to outages. Dovecot provides
some server synchronisation mechanisms which we could consider, but we
may also want to consider filesystem-based replication for a "warm"
spare.

Multi-primary setups would require "sharding" the users across
multiple servers and is definitely considered out of scope.

[LDAP]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/ldap

## Personal SPF/DKIM records and partial external hosting

At Debian.org, it's possible for members to configure their own DKIM
records which allows them to sign their personal, outgoing email with
their own DKIM keys and send signed emails out to the world from their
own email server. We will not support such a configuration, as it is
considered too complex to setup for normal users.

Furthermore, it would not *easily* help people currently hosted by
Gmail or Riseup: while it's technically possible for users to
*individually* delegate their DKIM signatures to those entities, those
keys could change without notice and break delivery.

DMARC has similar problems, particularly with monitoring and error
reporting.

Delegating SPF records might be slightly easier (because delegation is
built into the protocol), but has also been rejected for now. It is
considered risky to grant *all* of Gmail the rights to masquerade as
`torproject.org` (even though that's currently the status quo). And
besides delegating SPF alone wouldn't solve the more general problem
of *partially* allowing third parties to send mail as
`@torproject.org` (because of DKIM and DMARC).

## Status quo

The current status quo is also an option. But it is our belief that it
will lead to further and further problem in deliverability. We already
have a lot of problems delivering mail to various providers, and it's
hard to diagnose issues because anyone can currently send mail
masquerading as us from anywhere.

There might be *other* solutions than the ones proposed here, but we
haven't found any good ways of solving those issues without radically
changing the infrastructure so far.

If anything, if things continue as they are, people are going to use
their `@torproject.org` email address less and less, and we'll
effectively be migrating to external providers, but delegating that
workload to individual volunteers and workers. The mailing list and,
more critically, support and promotional tools (RT and CiviCRM)
services will become less and less effective in actually delivering
emails in people's inbox and, ultimately, this will hurt our capacity
to help our users and raise funds that are critical to the future of
the project.

## The end of email

One might also consider that email is a deprecated technology from
another millennia, and it is not the primary objective of the Tor
Project to continue using it, let alone host the infrastructure.

There are actually many different alternatives to email emerging, many
of which are already in use in the community.

For example, we already have a [Discourse server][] that is generating
great community participation and organisation.

We have also seen a good uptake on the Matrix bridges to our IRC
channels. Many places are seeing increase use of chat tools like Slack
as a replacement for email, and we could adopt Matrix more broadly as
such an alternative.

We also use informal Signal groups to organise certain conversations
as well.

[Nextcloud][] and [Big Blue Button][] also provide us with asynchronous and
synchronous coordination mechanisms.

We may be able to convert many of our uses of email right now to some
other tools:

 * "role forwards" like "accounting" or "job" aliases could be
   converted to RT or cdr.link (which, arguably, are also *primarily*
   email-based, but could be a transition to a web or messaging
   ticketing interface)

 * Mailman could be replaced by Discourse

 * Schleuder could be replaced by Matrix and/or Discourse?

That being said, we doubt *all* of our personas would be in a position
to *abandon* email completely at this point. We suspect many of our
personas, particularly in the fundraising team, would absolutely not
be able to do their work without email. We also do recurring
fundraising campaigns where we send *emails* to *thousands* of users
to raise money.

Note that if we do consider commercial alternatives, we *could* use a
mass-mailing provider service like [Mailchimp][] or [Amazon SES][] for
mass mailings, but this raises questions regarding the privacy of our
users. This is currently considered to be an unacceptable compromise.

There is therefore not a clear alternative to all of those problems
right now, so we consider email to be a mandatory part of our
infrastructure for the time being.

[Discourse server]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40539
[Big Blue Button]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/conference/
[Nextcloud]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/nextcloud
[Mailchimp]: https://mailchimp.com/
[Amazon SES]: https://docs.aws.amazon.com/ses/

## External hosting

Other service providers have been contacted to see if it would be
reasonable to host with them. This section details those options.

All of those service providers come with significant caveats:

 * most of those may not be able to take over *all* of our email
   services. services like RT, GitLab, Mailman, CiviCRM or Discourse
   require their own mail services and may not necessarily be possible
   to outsource, particularly for mass mailings like Mailman or
   CiviCRM

 * there is a privacy concern in hosting our emails elsewhere: unless
   otherwise noted, all email providers keep mail in clear text which
   makes it accessible to hostile or corrupt staff, law enforcement,
   or external attackers

Therefore most of those solutions involve a significant compromise in
terms of privacy.

The costs here also do not take into account the residual maintenance
cost of the email infrastructure that we'll have to deal with if the
provider only offers a partial solution to our problems, so all of
those estimates are under-estimates, unless otherwise noted.

### Greenhost: ~1600€/year, negotiable

We had a quote from Greenhost for 129€/mth for a Zimbra frontend with
a VM for mailboxes, DKIM, SPF records and all that jazz. The price
includes an office hours SLA.

### Riseup

Riseup already hosts a significant number of email accounts by virtue
of being the target of `@torproject.org` forwards. During the [last
inventory][], we found that, out of 91 active LDAP accounts, 30 were
being forwarded to `riseup.net`, so about 30%.

Riseup supports webmail, IMAP, and, more importantly, encrypted
mailboxes. While it's possible that an hostile attacker or staff could
modify the code to inspect a mailbox's content, it's leagues ahead of
most other providers in terms of privacy.

Riseup's prices are not public, but they are close to "market" prices
quoted below.

[last inventory]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/roadmap/2022

### Gandi: 480$-2400$/year

Gandi, the DNS provider, also offers [mailbox services][] which are
priced at 0.40$/user-month (3GB mailboxes) or 2.00$/user-month (50GB).

It's unclear if we could do mass-mailing with this service.

[mailbox services]: https://www.gandi.net/en-US/domain/email

### Google: 10,000$/year

Google were not contacted directly, but [their promotional site][]
says it's "Free for 14 days, then 7.80$ per user per month", which,
for tor-internal (~100 users), would be 780$/month or ~10,000USD/year.

We probably wouldn't be able to do mass mailing with this service.

[their promotional site]: https://workspace.google.com/solutions/new-business/

### Fastmail: 6,000$/year

Fastmail were not contacted directly but [their pricing page][] says
about 5$USD/user-month, with a free 30-day trial. This amounts to
500$/mth or 6,000$/year.

It's unclear if we could do mass-mailing with this service.

[their pricing page]: https://www.fastmail.com/pricing/

### Mailcow: 480€/year

[Mailcow][] is interesting because they actually are based on a [free
software stack][] (based on PHP, Dovecot, Sogo, rspamd, postfix,
nginx, redis, memcached, solr, Oley, and Docker containers). They
offer a [hosted service][] for 40€/month, with a 100GB disk quota and
no mailbox limitations (which, in our case, would mean 1GB/user).

We also get full admin access to the control panel and, given their
infrastructure, we could self-host if needed. Integration with our
current services would be, however, tricky.

It's there unclear if we could do mass-mailing with this service.

[hosted service]: https://www.servercow.de/mailcow?lang=en#managed
[free software stack]: https://github.com/mailcow/mailcow-dockerized
[Mailcow]: https://mailcow.email/

### Mailfence: 2,500€/year, 1750€ setup

The [mailfence business page][] doesn't have prices but last time we
looked at this, it was a 1750€ setup fee with 2.5€ per user-year.

It's unclear if we could do mass-mailing with this service.

[mailfence business page]: https://mailfence.com/en/secure-business-email.jsp

# Deadline

This proposal will be brought up to tor-internal and presented at a
all-hands meeting, and followed by a four-week feedback delay, after
which a decision will be taken.

# Approval

This decision needs the approval of tor-internal, TPA and TPI, the
latter of which will likely make the final call based on input from
the former.

# Status

This proposal is currently in the `rejected` state.

# References

 * this work is part of the [improve mail services OKR][OKR], part of the
   [2022 roadmap][], Q1/Q2
 * specifically, the draft of this proposal was established and
   discussed in [make a plan regarding mail standards (DKIM,SPF,
   DMARC)][]
 * the [submission service][] is the previous major undertaking to fix
   related issues to this project, and has a proposal that touches on
   some of those issues as well

[2022 roadmap]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/roadmap/2022
[OKR]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/4
[make a plan regarding mail standards (DKIM,SPF, DMARC)]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40363

# Appendix

## Other experiences from survey

anarcat did a survey of an informal network he's a part of, and here
are the anonymized feedback. Out 9 surveyed groups, 3 are outsourcing
to either [Mailcow][], Gandi, or Fastmail. Of the remaining 6:

 * filtering:
   * Spamassassin: 3
   * rspamd: 3
 * DMARC: 3
 * outgoing:
   * SPF: 3
   * DKIM: 2
   * DMARC: 3
   * ARC: 1
 * SMTPS: 4
   * Let's Encrypt: 4
   * MTA-STS: 1
   * DANE: 2
 * mailboxes: 4, mostly recommending Dovecot

here's a detailed listing

### Org A

 * Spamassassin: x
 * RBL: x
 * DMARC: x (quarantine, not reject)
 * SMTPS: LE
 * Cyrus: x (but suggests dovecot)

### Org B

 * used to self-host, migrated to 

### Org C

 * SPF: x
 * DKIM: soon
 * Spamassassin: x (also grades SPF, reject on mailman)
 * ClamAV: x
 * SMTPS: LE, tries SMTPS outgoing
 * Dovecot: x

### Org D

 * used to self-host, migrated to Gandi

### Org E

 * SPF, DKIM, DMARC, ARC, outbound and inbound
 * rspamd
 * SMTPS: LE + DANE
 * Dovecot

### Org F

 * SPF, DKIM
 * DMARC on lists
 * Spamassassin
 * SMTPS: LE + DANE (which triggered some outages)
 * MTA-STS
 * Dovecot

### Org G

 * no SPF/DKIM/etc
 * rspamd

### Org H

 * migrated to fastmail

### Org I

 * self-hosted in multiple locations
 * rspamd
 * no SPF/DKIM/DMARC outgoing
