draft requirements for email providers (team#40798)

policy/tpa-rfc-31-outsource-email.md

+---
+title: TPA-RFC-31: Outsource email services
+---
+
+[[_TOC_]]
+
+Summary: TODO
+
+# Background
+
+In late 2021, the TPA team adopted the following first Objective and
+Key Results (OKR):
+
+> [Improve mail services][OKR]:
+> 
+>  1. David doesn't complain about "mail getting into spam" anymore
+>  2. RT is not full of spam
+>  3. we can deliver and receive mail from state.gov
+
+There were two ways of implementing solutions to this problem. One way
+was to complete the implementation of email services internally,
+adding standard tools like DKIM, SPF, and so on to our services and
+hosting mailboxes. This approach was investigated fully in
+[TPA-RFC-15][] but was ultimately rejected as too risky.
+
+[DMARC]: https://en.wikipedia.org/wiki/DMARC
+[DKIM]: https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail
+[SPF]: http://www.open-spf.org/
+
+Instead, we are looking at the other solution to this problem which is
+to outsource all or a part of our mail services to some external
+provider. This proposal aims at clarifying which services we should
+outsource, and to whom.
+
+# Proposal
+
+Progressively migrate all `@torproject.org` email aliases and forwards
+to the new provider.
+
+Retirement of the "submission service" and destruction of the
+`submit-01` server, after migration of all users to the new provider.
+
+## Requirements
+
+Those are the requirements the external service provider must fulfill
+before being considered for this proposal.
+
+### Email interfaces
+
+We have users currently using Gmail, Thunderbird, Apple Mail, Outlook,
+and other email clients. Some people keep their mail on the server,
+some fetch it once and never keep a copy. Some people read their mail
+on their phone.
+
+Therefore, the new provider MUST offer IMAP and POP mailboxes,
+alongside a modern and mobile-friendly Webmail client.
+
+It MUST be possible for users (and machines) to submit emails using a
+username/password combination through a dedicated SMTP server (also
+known as a "submission port"). Ideally, this could be done with TLS
+certificates, especially for client machines.
+
+Some users are unlikely to leave Gmail, and should be able to forward
+their email there. Inversely, they should be able to send mail *from*
+Gmail through a submission address.
+
+### Deliverability
+
+Provider SHOULD be able to reliably deliver email to both large
+service providers like Gmail and Outlook, but also government sites
+like `state.gov` or other, smaller mail servers.
+
+Therefore, modern email standards like SPF, DKIM, DMARC, and hopefully
+ARC SHOULD be implemented by the new provider.
+
+We also often perform mass mailings to announce software releases
+(through Mailman 2, soon 3) but also larger fundraising mailings
+through CiviCRM, which contact almost 300,000 users every
+month. Provisions must therefore be made for those services to keep
+functioning, possibly through a dedicated mail submission host as
+well. Servers which currently send regular emails to end users
+include:
+
+ * CiviCRM: 270,000 emails in June 2021
+ * Mailman: 12,600 members on tor-announce
+ * RT: support tracker, 15,000 registered users?
+ * GitLab: ~2,000 active users
+ * Discourse: ~1,000 monthly active users
+
+Part of our work involves using email to communicate to fundraiser but
+also people in censored country, so censorship resistance is
+important. Ideally, a Tor `.onion` service should be provided for
+email submission, for example. 
+
+Also, we have special email services like [gettor.torproject.org](https://gettor.torproject.org/)
+which send bridges or download links for accessing the Tor
+network. Those should also keep functioning properly, but should also
+be resistant to attacks aiming to list all bridges, for example. This
+is currently done by checking incoming DKIM signature and limiting the
+service to certain providers.
+
+Non-mail machines will relay their mail through a new internal relay
+server that will then submit its mail to the new provider. This will
+help us automate configuration of "regular" email server to avoid
+having to create an account in the new provider's control panel every
+time we setup a new server.
+
+### Spam control
+
+State of the art spam filtering software MUST be deployed to keep the
+bulk of spam from reaching user's mail boxes, or hopefully triage them
+in a separate "Spam folder".
+
+Bayesian training MAY be used to improve the accuracy of those filters
+and the user should be able to train the algorithm to allow certain
+emails to go through.
+
+### Privacy
+
+We are an organisation that takes user privacy seriously. Under
+absolutely no circumstances should email contents or metadata be used
+in other fashion than for delivering mail to its destination or
+aforementioned spam control. Ideally, mail boxes would be encrypted
+with a user-controlled key so that the provider may not be able to
+read the contents of mailboxes at all.
+
+Strong log file anonymization is expected or at least aggressive log
+rotation should be enforced.
+
+Privately identifiable information (e.g. client IP address) MUST NOT
+leak through email headers.
+
+We strongly believe that free and open source software is the only way
+to ensure privacy guarantees like these are enforceable. At least the
+services provided MUST be usable with standard, free software email
+clients (for example, Thunderbird).
+
+## Scope
+
+This proposal doesn't address the fate of Schleuder or Mailman (or,
+for that matter, Discourse, RT, or other services that may use email
+unless explicitly mentioned).
+
+It also does *not* address directly phishing and scamming attacks
+([issue 40596][]), but it is hoped that stricter
+enforcement of email standards will reduce those to a certain
+extent. 
+
+TODO: Those architectural details may change through negotiations with
+the new provider.
+
+## Affected users
+
+This affects all users which interact with `torproject.org` and its
+subdomains over email. It particularly affects all "tor-internal"
+users, users with LDAP accounts or forwards under `@torproject.org`.
+
+TODO: what will actually happen? maybe delegate this to personas?
+
+## Architecture diagram
+
+Those diagrams detail the infrastructure before and after the changes
+detailed above.
+
+Legend:
+
+ * red: legacy hosts, mostly eugeni services, no change
+ * orange: hosts that manage and/or send their own email, no change
+   except the mail exchanger might be the one relaying the
+   `@torproject.org` mail to it instead of eugeni
+ * green: new hosts, might be multiple replicas
+ * rectangles: machines
+ * triangle: the user
+ * ellipse: the rest of the internet, other mail hosts not managed by tpo
+
+### Before
+
+TODO: taken from TPA-RFC-15, review
+
+![current mail architecture diagram](tpa-rfc-31-outsource-email/architecture-pre.png)
+
+### After
+
+TODO: make new post-proposal architecture diagram
+
+## Actual changes
+
+TODO: task breakdown, this was one subheading per task in TPA-RFC-15
+
+### New mail transfer agent
+
+Configure new "mail transfer agent" server(s) to relay mails from
+servers that do not send their own email, replacing a part of
+`eugeni`.
+
+This server will be called `mta-01.torproject.org` and could be
+horizontally scaled up for availability. See also the [Naming
+things](#naming-things) Challenge below.
+
+## Cost estimates
+
+TODO: estimate cost of tasks above
+
+## Timeline
+
+TODO: when are we going to do this?
+
+## Challenges
+
+### Delays
+
+In early June 2021, it became apparent that we were having more
+problems delivering mail to Gmail, possibly because of the lack of
+DKIM records (see for example [tpo/tpa/team#40765](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40765)). We may have
+had time to implement some countermeasures, had TPA-RFC-15 been
+adopted, but alas, we decided to go with an external provider.
+
+It is unclear, at this point, whether this will speed things up or
+not. We may take too much time deliberating on the right service
+provider, or this very specification, or find problems during the
+migrations, which may make email even more unreliable.
+
+### Naming things
+
+In TPA-RFC-15, it became apparent that the difficulty of naming things
+did not escape those proposals. For example, in TPA-RFC-15, the term
+"relay" has been used liberally to talk about a new email server
+processing email for other servers. That terminology, unfortunately,
+clashes with the term "relay" used extensively in the Tor network to
+designate "Tor relays", which create circuits that make up the Tor
+network.
+
+This is the reason why the `mta-01` server is named a "MTA" and not a
+"relay" or "submission" server. The former is reserved for Tor relays,
+and the latter for "email submission servers" provided by
+upstream. Technically, the difference between the "MTA" and the
+"submission" server is that the latter is expected to deliver the
+email out of the responsibility of the `torproject.org` domain, to its
+final destination, while the MTA is allowed to transfer to another MTA
+or submission server.
+
+### Aging Puppet code base and other legacy
+
+TODO: copy from TPA-RFC-15?
+
+### Security and privacy issues
+
+TODO: 2FA? trust?
+
+### Duplication of services with LDAP
+
+TODO: How will users be authenticated? can we bridge with LDAP?
+
+### Upstream support burden
+
+It is unclear how we will handle support requests: will users directly
+file issues upstream, or will this be handled by TPA first?
+
+How will password resets be done, for example?
+
+### Sunk costs
+
+There has been a *lot* of work done in the current email
+infrastructure. In particular, anarcat spent a significant amount of
+time changing the LDAP services to allow the addition of an "email
+password" and deploying those to a "submission server" to allow people
+to submit email through the `torproject.org` infrastructure. The
+TPA-RFC-15 design and proposal will also go to waste with this
+proposal.
+
+# Personas
+
+TODO: what this will mean for everyone, possibly reusing our personas
+from TPA-RFC-15
+
+# Other alternatives
+
+## Hosting our own email
+
+TODO: that's TPA-RFC-15, detail why rejected
+
+## Status quo
+
+The status quo situation is similar (if not worse) than the status quo
+described in TPA-RFC-15: email services are suffering from major
+deliverability problems and things are only going to get worse over
+time, up to a point when no one will use their `@torproject.org` email
+address.
+
+## The end of email
+
+This is similar to the discussion mentioned in TPA-RFC-15: email is
+still a vital service and we cannot at the moment consider completely
+replacing it with other tools.
+
+## Providers evaluation
+
+TODO: update those numbers, currently taken directly from TPA-RFC-15, without change
+
+### Fastmail: 6,000$/year
+
+Fastmail were not contacted directly but [their pricing page][] says
+about 5$USD/user-month, with a free 30-day trial. This amounts to
+500$/mth or 6,000$/year.
+
+It's unclear if we could do mass-mailing with this service.
+
+[their pricing page]: https://www.fastmail.com/pricing/
+
+### Gandi: 480$-2400$/year
+
+Gandi, the DNS provider, also offers [mailbox services][] which are
+priced at 0.40$/user-month (3GB mailboxes) or 2.00$/user-month (50GB).
+
+It's unclear if we could do mass-mailing with this service.
+
+[mailbox services]: https://www.gandi.net/en-US/domain/email
+
+### Google: 10,000$/year
+
+Google were not contacted directly, but [their promotional site][]
+says it's "Free for 14 days, then 7.80$ per user per month", which,
+for tor-internal (~100 users), would be 780$/month or ~10,000USD/year.
+
+We probably wouldn't be able to do mass mailing with this service.
+
+[their promotional site]: https://workspace.google.com/solutions/new-business/
+
+### Greenhost: ~1600€/year, negotiable
+
+We had a quote from Greenhost for 129€/mth for a Zimbra frontend with
+a VM for mailboxes, DKIM, SPF records and all that jazz. The price
+includes an office hours SLA.
+
+### Mailcow: 480€/year
+
+[Mailcow][] is interesting because they actually are based on a [free
+software stack][] (based on PHP, Dovecot, Sogo, rspamd, postfix,
+nginx, redis, memcached, solr, Oley, and Docker containers). They
+offer a [hosted service][] for 40€/month, with a 100GB disk quota and
+no mailbox limitations (which, in our case, would mean 1GB/user).
+
+We also get full admin access to the control panel and, given their
+infrastructure, we could self-host if needed. Integration with our
+current services would be, however, tricky.
+
+It's there unclear if we could do mass-mailing with this service.
+
+[hosted service]: https://www.servercow.de/mailcow?lang=en#managed
+[free software stack]: https://github.com/mailcow/mailcow-dockerized
+[Mailcow]: https://mailcow.email/
+
+### Mailfence: 2,500€/year, 1750€ setup
+
+The [mailfence business page][] doesn't have prices but last time we
+looked at this, it was a 1750€ setup fee with 2.5€ per user-year.
+
+It's unclear if we could do mass-mailing with this service.
+
+[mailfence business page]: https://mailfence.com/en/secure-business-email.jsp
+
+### Riseup
+
+Riseup already hosts a significant number of email accounts by virtue
+of being the target of `@torproject.org` forwards. During the [last
+inventory][], we found that, out of 91 active LDAP accounts, 30 were
+being forwarded to `riseup.net`, so about 30%.
+
+Riseup supports webmail, IMAP, and, more importantly, encrypted
+mailboxes. While it's possible that an hostile attacker or staff could
+modify the code to inspect a mailbox's content, it's leagues ahead of
+most other providers in terms of privacy.
+
+Riseup's prices are not public, but they are close to "market" prices
+quoted below.
+
+[last inventory]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/roadmap/2022
+
+# Approval
+
+This proposal will need the approval of TPA, tor-internal, and TPI,
+the latter which will have the final word on the decision.
+
+# Deadline
+
+TODO: establish deadline
+
+# Status
+
+This proposal is currently in the `draft` state.
+
+# References
+
+ * this work is part of the [improve mail services OKR][OKR], part of the
+   [2022 roadmap][], Q1/Q2
+ * this proposal is specifically discussed in [tpo/tpa/team#40798][]
+
+[tpo/tpa/team#40798]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40798
+[OKR]: https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/4

policy/tpa-rfc-31-outsource-email/architecture-pre.dot

+digraph email {
+        label="Email architecture, torproject.org, April 2022"
+        labelloc=bottom
+        node [shape=rectangle]
+
+        // a user
+
+        user [shape=triangle]
+
+        // all other boxes that don't do anything special with email
+        tpoboxes [label="other TPA hosts"]
+        // hosts that manage and send their own email
+        mailhosts [color="orange" label="mail hosts\ngitlab, RT, CiviCRM"]
+        submission [color=orange]
+        // legacy
+        eugeni [color=red label="eugeni\nmailman, schleuder\nmx, relay"]
+        // external boxes
+        dotdotdot [label="internet\nnon TPO mail hosts", shape=ellipse]
+        // other hosts that currently masquerade as us
+        masquerade [label="impersonators\nriseup, gmail, etc"]
+
+        // all relations
+
+        // eugeni handles basically everything except some rare
+        // exceptions on mailhosts
+        dotdotdot -> { mailhosts, eugeni } [label="SMTP"]
+        // users interact with those
+        user -> submission [label="SMTP"]
+        user -> eugeni [label="HTTP, SMTP"]
+        user -> mailhosts [label="HTTP"]
+        // those all relay mail to the outside
+        { submission, mailhosts, eugeni, masquerade } -> dotdotdot [label="SMTP"]
+        // except those, who still go through the relay
+        tpoboxes -> eugeni [label="SMTPS"]
+        // eugeni also forwards mails to the impersonators
+        eugeni -> masquerade [label="SMTP"]
+}