|
|
---
|
|
|
title: Email submission services
|
|
|
---
|
|
|
|
|
|
Email submission services consist of a server that accepts email using
|
|
|
authenticated SMTP for LDAP users of the `torproject.org` domain.
|
|
|
|
|
|
[[_TOC_]]
|
|
|
|
|
|
# Tutorial
|
|
|
|
|
|
<!-- simple, brainless step-by-step instructions requiring little or -->
|
|
|
<!-- no technical background -->
|
|
|
|
|
|
TODO: should include easy configuration instructions for major
|
|
|
platforms (Thunderbird, Mail app, Outlook and Google mail,
|
|
|
maybe?). pili volunteered to write/test the Mac OS instructions. ggus
|
|
|
says that Access Now! has some docs we could start from. vegas agrees
|
|
|
that this documentation and support will be covered by teams outside
|
|
|
of TPA.
|
|
|
|
|
|
# How-to
|
|
|
|
|
|
<!-- more in-depth procedure that may require interpretation -->
|
|
|
|
|
|
## Glossary
|
|
|
|
|
|
* **SMTP**: Simple Mail Transfer Protocol. The email protocol spoken
|
|
|
between servers to deliver email. Consists of two standards,
|
|
|
[RFC821](https://tools.ietf.org/html/rfc821) and [RFC5321](https://tools.ietf.org/html/rfc5321) which defined SMTP extensions, also
|
|
|
known as **ESMTP**.
|
|
|
* **MTA**: Mail Transport Agent. A generic SMTP server. Eugeni is
|
|
|
such a server.
|
|
|
* **MUA**: Mail User Agent. An "email client", a program used to
|
|
|
receive, manage and send email for users.
|
|
|
* **MSA** : Mail Submission Agent. An SMTP server specifically
|
|
|
designed to only *receive* email.
|
|
|
* **MDA**: Mail Delivery Agent. The email service actually writing
|
|
|
the email to the user's mailbox. Out of scope.
|
|
|
|
|
|
This document describes the implementation of a **MSA**, although the
|
|
|
service will most likely also include a **MTA** functionality in that
|
|
|
it will actually deliver emails to targets.
|
|
|
|
|
|
## Pager playbook
|
|
|
## Disaster recovery
|
|
|
|
|
|
N/A. The server should be rebuildable from scratch using the Puppet
|
|
|
directive and does not have long-term user data. All user data is
|
|
|
stored in DNS or LDAP.
|
|
|
|
|
|
If email delivery starts failing, users are encouraged to go back to
|
|
|
the email providers they were using before this service was deployed.
|
|
|
|
|
|
# Reference
|
|
|
|
|
|
## Installation
|
|
|
<!-- how to setup the service from scratch -->
|
|
|
|
|
|
_submission._tcp.example.com. SRV 0 1 587 mail.example.com.
|
|
|
|
|
|
## SLA
|
|
|
<!-- this describes an acceptable level of service for this service -->
|
|
|
|
|
|
## Design
|
|
|
<!-- how this is built -->
|
|
|
<!-- should reuse and expand on the "proposed solution", it's a -->
|
|
|
<!-- "as-built" documented, whereas the "Proposed solution" is an -->
|
|
|
<!-- "architectural" document, which the final result might differ -->
|
|
|
<!-- from, sometimes significantly -->
|
|
|
|
|
|
## Issues
|
|
|
|
|
|
Project is coordinated in [ticket #30608][].
|
|
|
|
|
|
There is no issue tracker specifically for this project, [File][] or
|
|
|
[search][] for issues in the [generic internal services][search] component.
|
|
|
|
|
|
[File]: https://trac.torproject.org/projects/tor/newticket?component=Internal+Services%2FTor+Sysadmin+Team
|
|
|
[search]: https://trac.torproject.org/projects/tor/query?status=!closed&component=Internal+Services%2FTor+Sysadmin+Team
|
|
|
|
|
|
# Discussion
|
|
|
|
|
|
## Overview
|
|
|
|
|
|
The idea is to create a new server to deal with delivery problems
|
|
|
torproject.org email users are currently seeing. While they can
|
|
|
receive email through their `user@torproject.org` forwards without too
|
|
|
much problem, their emails often get dropped to the floor when
|
|
|
*sending* from that email address.
|
|
|
|
|
|
It is suspected that users are having those problems because the
|
|
|
originating servers are not in the `torproject.org` domain. The hope
|
|
|
is that setting up a new server inside that domain would help with
|
|
|
delivery. There's anecdotal evidence (see [this comment](https://gitlab.torproject.org/legacy/trac/-/issues/30608#note_2325187) for
|
|
|
example) that delivery emails from existing servers (over SSH to
|
|
|
`iranicum`, in that example) improves reliability of email delivery
|
|
|
significantly.
|
|
|
|
|
|
This project came out of [ticket #30608][], which has the launch
|
|
|
checklist.
|
|
|
|
|
|
[ticket #30608]: https://bugs.torproject.org/30608
|
|
|
|
|
|
## Goals
|
|
|
|
|
|
### Must have
|
|
|
|
|
|
* basic compatibility with major clients (Thunderbird, Mail.app,
|
|
|
Outlook, Gmail?)
|
|
|
* delivery over secure (TLS + password) SMTP
|
|
|
* credentials stored in LDAP
|
|
|
|
|
|
### Nice to have
|
|
|
|
|
|
* automatic client configuration
|
|
|
* improved delivery over current federated configuration
|
|
|
* delivery reliability monitoring with major providers (e.g. hotmail,
|
|
|
gmail, yahoo)
|
|
|
* pretty graphs
|
|
|
* formalized SSH-key delivery to avoid storing cleartext passwords on
|
|
|
clients
|
|
|
|
|
|
### Non-Goals
|
|
|
|
|
|
* 100%, infaillable, universal delivery to all providers (ie. emails
|
|
|
will still be lost)
|
|
|
* mailbox management (ie. no incoming email, IMAP, POP, etc)
|
|
|
* spam filtering (ie. we won't check outgoing emails)
|
|
|
* no DKIM, SPF, DMARC, or ARC for now, although maybe a "null" SPF
|
|
|
record if it helps with delivery
|
|
|
|
|
|
## Approvals required
|
|
|
|
|
|
Approved by vegas, requested by network team, agreed with TPA at the
|
|
|
Stockholm meeting.
|
|
|
|
|
|
## Proposed Solution
|
|
|
|
|
|
The proposed design is to setup a new email server in the [howto/ganeti](howto/ganeti)
|
|
|
cluster (currently `gnt-fsn`) with the user list synchronized from
|
|
|
LDAP, using a new password field (named `mailPassword`). The access
|
|
|
would therefore be granted only to LDAP users, and LDAP accounts would
|
|
|
be created as needed. In the short term, LDAP can be used to modify
|
|
|
that password but in the mid-term, it would be modifiable through the
|
|
|
web interface like the `webPassword` or `rtcPassword` fields.
|
|
|
|
|
|
### Current inventory
|
|
|
|
|
|
* active LDAP accounts: 91
|
|
|
* non-LDAP forwards (to real people): 24
|
|
|
* role forwards (to other @torproject.org emails): 76
|
|
|
|
|
|
Forward targets:
|
|
|
|
|
|
* riseup.net: 30
|
|
|
* gmail.com: 21
|
|
|
* other: 93 (only 4 domains have more than one forward)
|
|
|
|
|
|
Delivery rate: SMTP, on eugeni, is around 0.5qps, with a max of 8qps
|
|
|
in the last 7 days (2019-06-06). But that includes mailing lists as
|
|
|
well. During that period, around 27000 emails were delivered to
|
|
|
@torproject.org aliases.
|
|
|
|
|
|
## Cost
|
|
|
|
|
|
Labor and `gnt-fsn` VM costs. To be detailed.
|
|
|
|
|
|
Below is an evaluation of the various Alternatives that were considered.
|
|
|
|
|
|
### External hosting cost evaluation
|
|
|
|
|
|
* Google: 8$/mth/account? (to be verified?)
|
|
|
* riseup.net: anarcat requested price quotation
|
|
|
* koumbit.org: default pricing: 100$/year on shared hosting and 50GB
|
|
|
total, possibly no spam filter. 1TB disk: 500$/year. disk
|
|
|
encryption would need to be implemented, quoted 2000-4000$ setup
|
|
|
fee to implement it in the AlternC opensource control panel.
|
|
|
* self-hosting: ~4000-500EUR setup, 5000EUR-7500EUR/year, liberal
|
|
|
estimate (will probably be less)
|
|
|
* [mailfence](https://mailfence.com/en/secure-business-email.jsp) 1750 setup cost and 2.5 euros per user/year
|
|
|
|
|
|
Note that the self-hosting cost evaluation is for the fully-fledged
|
|
|
service. Option 2, above, of relaying email, has overall negligible
|
|
|
costs although that theory has been questioned by members of the
|
|
|
sysadmin team.
|
|
|
|
|
|
### Internal hosting cost evaluation
|
|
|
|
|
|
This is a back-of-the-napkin calculation of what it would cost to host
|
|
|
actual email services at TPA infrastructure itself. We consider this
|
|
|
to be a “liberal” estimate, ie. costs would probably be less and time
|
|
|
estimates have been padded (doubled) to cover for errors.
|
|
|
|
|
|
Assumptions:
|
|
|
|
|
|
* each mailbox is on average, a maximum of 10GB
|
|
|
* 100 mailboxes maximum at first (so 1TB of storage required)
|
|
|
* LUKS full disk encryption
|
|
|
* IMAP and basic webmail (Roundcube or Rainloop)
|
|
|
* “Trees” mailbox encryption out of scope for now
|
|
|
|
|
|
Hardware:
|
|
|
|
|
|
* Hetzner px62nvme 2x1TB RAID-1 64GB RAM 75EUR/mth, 900EUR/yr
|
|
|
* Hetzner px92 2x1TB SSD RAID-1 128GB RAM 115EUR/mth, 1380EUR/yr
|
|
|
* Total hardware: 2280EUR/yr, ~200EUR setup fee
|
|
|
|
|
|
This assumes hosting the server on a dedicated server at Hetzner. It
|
|
|
might be possible (and more reliable) to ensure further cost savings
|
|
|
by hosting it on our shared virtualized infrastructure. Calculations
|
|
|
for this haven’t been performed by the team, but I would guess we
|
|
|
might save around 25 to 50% of the above costs, depending on the
|
|
|
actual demand and occupancy on the mail servers.
|
|
|
|
|
|
Staff:
|
|
|
|
|
|
* LDAP password segregation: 4 hours*
|
|
|
* Dovecot deployment and LDAP integration: 8 hours
|
|
|
* Dovecot storage optimization: 8 hours
|
|
|
* Postfix mail delivery integration: 8 hours
|
|
|
* Spam filter deployment: 8 hours
|
|
|
* 100% cost overrun estimate: 36 hours
|
|
|
* Total setup costs: 72 hours @ 50EUR/hr: 3600EUR one time
|
|
|
|
|
|
This is the most imprecise evaluation. Most email systems have been
|
|
|
built incrementally. The biggest unknown is the extra labor
|
|
|
associated with running the IMAP server and spam filter. A few
|
|
|
hypothesis:
|
|
|
|
|
|
* 1 hour a week: 52 hours @ 50EUR/hr: 2600EUR/yr
|
|
|
* 2 hours a week: 5200EUR/yr
|
|
|
|
|
|
I would be surprised if the extra work goes beyond one hour a week,
|
|
|
and will probably be less. This also does not include 24/7 response
|
|
|
time, but no service provider evaluated provides that level of service
|
|
|
anyways.
|
|
|
|
|
|
Total:
|
|
|
|
|
|
* One-time setup: 3800EUR (200EUR hardware, 3600EUR staff)
|
|
|
* Recurrent: roughly between 5000EUR and 7500EUR/year, majority in staff
|
|
|
|
|
|
## Alternatives considered
|
|
|
|
|
|
There are three dimensions to our “decision tree”:
|
|
|
|
|
|
1. Hosting mailboxes or only forwards: this means that instead of
|
|
|
just forwarding emails to some other providers, we actually allow
|
|
|
users to store emails on the server. Current situation is we only
|
|
|
do forwards
|
|
|
2. SMTP authentication: this means allowing users to submit email
|
|
|
using a username and password over the standard SMTP (technically
|
|
|
“submission”) port. This is currently not allowed also some have
|
|
|
figured out they can do this over SSH already.
|
|
|
3. Self-hosted or hosted elsewhere: if we host the email service
|
|
|
ourselves right now or not. The current situation is we allow
|
|
|
inbound messages but we do not store them. Mailbox storage is
|
|
|
delegated to each individual choice of email provider, which also
|
|
|
handles SMTP authentication.
|
|
|
|
|
|
Here are is the breakdown of pros and cons of each approach. Note that
|
|
|
there are multiple combinations of those possible, for example we
|
|
|
could continue not having mailboxes but allow SMTP authentication, and
|
|
|
delegate this to a third party. Obviously, some combinations (like no
|
|
|
SMTP authentication and mailboxes) are a little absurd and should be
|
|
|
taken with a grain of salt.
|
|
|
|
|
|
### TP full hosting: mailboxes, SMTP authentication
|
|
|
|
|
|
Pros:
|
|
|
|
|
|
* Easier for TPA to diagnose email problems than if email is hosted
|
|
|
by an undetermined third party
|
|
|
* People’s personal email is not mixed up with Tor email.
|
|
|
* Easier delegation between staff on rotations
|
|
|
* Control over where data is stored and how
|
|
|
* Full control of our infrastructure
|
|
|
* Less trust issues
|
|
|
|
|
|
Cons:
|
|
|
|
|
|
* probably the most expensive option
|
|
|
* requires more skilled staff
|
|
|
* high availability harder to achieve
|
|
|
* high costs
|
|
|
|
|
|
### TP not hosting mailboxes; TP hosting outgoing SMTP authentication server
|
|
|
|
|
|
Pros:
|
|
|
|
|
|
* No data retention issues: TP not responsible for legal issues
|
|
|
surrounding mailboxes contents
|
|
|
* Solves delivery problem and nothing else (minimal solution)
|
|
|
* We’re already running an SMTP server
|
|
|
* SSH tunnels already let our lunatic-fringe do a version of this
|
|
|
* Staff keeps using own mail readers (eg gmail UI) for receiving mail
|
|
|
* Federated solution
|
|
|
* probably the cheapest option
|
|
|
* Work email cannot be accessed by TP staff
|
|
|
|
|
|
Cons:
|
|
|
|
|
|
* SMTP-AUTH password management (admin effort and risk)
|
|
|
* Possible legal requests to record outgoing mail? (SSH
|
|
|
lunatic-fringe already at risk, though)
|
|
|
* DKIM/SPF politics vs “slippery slope”
|
|
|
* Forces people to figure out some good ISP to host their email
|
|
|
* Shifts the support burden to individuals
|
|
|
* Harder to diagnose email problems
|
|
|
* Staff or “role” email accounts cannot be shared
|
|
|
|
|
|
### TP pays third party (riseup, protonmail, mailfence, gmail??) for full service (mailboxes, delivery)
|
|
|
|
|
|
Pros:
|
|
|
|
|
|
* Less admin effort
|
|
|
* Less/no risk to TP infrastructure (legal or technical)
|
|
|
* Third party does not hold email data hostage; only handles outgoing
|
|
|
* We know where data is hosted instead of being spread around
|
|
|
|
|
|
Cons:
|
|
|
|
|
|
* Not a federated solution
|
|
|
* Implicitly accepts email cartel model of “trusted” ISPs
|
|
|
* Varying levels of third party data management trust required
|
|
|
* Some third parties require custom software (protonmail)
|
|
|
* Single point of failure.
|
|
|
* Might force our users to pick a provider they dislike
|
|
|
* All eggs in the same basket
|
|
|
|
|
|
### Status quo (no mailboxes, no authentication)
|
|
|
|
|
|
Pros:
|
|
|
|
|
|
* Easy. Fast. Cheap. Pick three.
|
|
|
|
|
|
Cons:
|
|
|
|
|
|
* Shifts burden of email debugging to users, lack of support
|
|
|
|
|
|
|
|
|
Details of the chosen alternative (SMTP authentication):
|
|
|
|
|
|
* Postfix + offline LDAP authentication (current proposal)
|
|
|
* Postfix + direct LDAP authentication: discarded because it might
|
|
|
fail when the LDAP server goes down. LDAP server is currently not
|
|
|
considered to be critical and can be restarted for maintenance
|
|
|
without affecting the rest of the infrastructure.
|
|
|
* reusing existing field like `webPassword` or `rtcPassword` in LDAP:
|
|
|
considered a semantic violation.
|
|
|
|
|
|
See also internal Nextcloud document.
|
|
|
|
|
|
No benchmark considered necessary. |