Snippets Groups Projects

4 years ago
9e8717a1

fix issue links to point at gitlab · 9e8717a1
anarcat authored 4 years ago

Verified
9e8717a1

History

fix issue links to point at gitlab
anarcat authored 4 years ago

submission.md 12.53 KiB

title: Email submission services

Email submission services consist of a server that accepts email using authenticated SMTP for LDAP users of the torproject.org domain.

Tutorial

TODO: should include easy configuration instructions for major platforms (Thunderbird, Mail app, Outlook and Google mail, maybe?). pili volunteered to write/test the Mac OS instructions. ggus says that Access Now! has some docs we could start from. vegas agrees that this documentation and support will be covered by teams outside of TPA.

How-to

Glossary

SMTP: Simple Mail Transfer Protocol. The email protocol spoken between servers to deliver email. Consists of two standards, RFC821 and RFC5321 which defined SMTP extensions, also known as ESMTP.
MTA: Mail Transport Agent. A generic SMTP server. Eugeni is such a server.
MUA: Mail User Agent. An "email client", a program used to receive, manage and send email for users.
MSA : Mail Submission Agent. An SMTP server specifically designed to only receive email.
MDA: Mail Delivery Agent. The email service actually writing the email to the user's mailbox. Out of scope.

This document describes the implementation of a MSA, although the service will most likely also include a MTA functionality in that it will actually deliver emails to targets.

Pager playbook

Disaster recovery

N/A. The server should be rebuildable from scratch using the Puppet directive and does not have long-term user data. All user data is stored in DNS or LDAP.

If email delivery starts failing, users are encouraged to go back to the email providers they were using before this service was deployed.

Reference

Installation

_submission._tcp.example.com.     SRV 0 1 587 mail.example.com.

SLA

Design

Issues

Project is coordinated in ticket #30608.

There is no issue tracker specifically for this project, File or search for issues in the team issue tracker component.

Discussion

Overview

The idea is to create a new server to deal with delivery problems torproject.org email users are currently seeing. While they can receive email through their user@torproject.org forwards without too much problem, their emails often get dropped to the floor when sending from that email address.

It is suspected that users are having those problems because the originating servers are not in the torproject.org domain. The hope is that setting up a new server inside that domain would help with delivery. There's anecdotal evidence (see this comment for example) that delivery emails from existing servers (over SSH to iranicum, in that example) improves reliability of email delivery significantly.

This project came out of ticket #30608, which has the launch checklist.

Goals

Must have

basic compatibility with major clients (Thunderbird, Mail.app, Outlook, Gmail?)
delivery over secure (TLS + password) SMTP
credentials stored in LDAP

Nice to have

automatic client configuration
improved delivery over current federated configuration
delivery reliability monitoring with major providers (e.g. hotmail, gmail, yahoo)
pretty graphs
formalized SSH-key delivery to avoid storing cleartext passwords on clients

Non-Goals

100%, infaillable, universal delivery to all providers (ie. emails will still be lost)
mailbox management (ie. no incoming email, IMAP, POP, etc)
spam filtering (ie. we won't check outgoing emails)
no DKIM, SPF, DMARC, or ARC for now, although maybe a "null" SPF record if it helps with delivery

Approvals required

Approved by vegas, requested by network team, agreed with TPA at the Stockholm meeting.

Proposed Solution

The proposed design is to setup a new email server in the howto/ganeti cluster (currently gnt-fsn) with the user list synchronized from LDAP, using a new password field (named mailPassword). The access would therefore be granted only to LDAP users, and LDAP accounts would be created as needed. In the short term, LDAP can be used to modify that password but in the mid-term, it would be modifiable through the web interface like the webPassword or rtcPassword fields.

Current inventory

active LDAP accounts: 91
non-LDAP forwards (to real people): 24
role forwards (to other @torproject.org emails): 76

Forward targets:

riseup.net: 30
gmail.com: 21
other: 93 (only 4 domains have more than one forward)

Delivery rate: SMTP, on eugeni, is around 0.5qps, with a max of 8qps in the last 7 days (2019-06-06). But that includes mailing lists as well. During that period, around 27000 emails were delivered to @torproject.org aliases.

Cost

Labor and gnt-fsn VM costs. To be detailed.

Below is an evaluation of the various Alternatives that were considered.

External hosting cost evaluation

Google: 8$/mth/account? (to be verified?)
riseup.net: anarcat requested price quotation
koumbit.org: default pricing: 100/year on shared hosting and 50GB total, possibly no spam filter. 1TB disk: 500/year. disk encryption would need to be implemented, quoted 2000-4000$ setup fee to implement it in the AlternC opensource control panel.
self-hosting: ~4000-500EUR setup, 5000EUR-7500EUR/year, liberal estimate (will probably be less)
mailfence 1750 setup cost and 2.5 euros per user/year

Note that the self-hosting cost evaluation is for the fully-fledged service. Option 2, above, of relaying email, has overall negligible costs although that theory has been questioned by members of the sysadmin team.

Internal hosting cost evaluation

This is a back-of-the-napkin calculation of what it would cost to host actual email services at TPA infrastructure itself. We consider this to be a “liberal” estimate, ie. costs would probably be less and time estimates have been padded (doubled) to cover for errors.

Assumptions:

each mailbox is on average, a maximum of 10GB
100 mailboxes maximum at first (so 1TB of storage required)
LUKS full disk encryption
IMAP and basic webmail (Roundcube or Rainloop)
“Trees” mailbox encryption out of scope for now

Hardware:

Hetzner px62nvme 2x1TB RAID-1 64GB RAM 75EUR/mth, 900EUR/yr
Hetzner px92 2x1TB SSD RAID-1 128GB RAM 115EUR/mth, 1380EUR/yr
Total hardware: 2280EUR/yr, ~200EUR setup fee

This assumes hosting the server on a dedicated server at Hetzner. It might be possible (and more reliable) to ensure further cost savings by hosting it on our shared virtualized infrastructure. Calculations for this haven’t been performed by the team, but I would guess we might save around 25 to 50% of the above costs, depending on the actual demand and occupancy on the mail servers.

Staff:

LDAP password segregation: 4 hours*
Dovecot deployment and LDAP integration: 8 hours
Dovecot storage optimization: 8 hours
Postfix mail delivery integration: 8 hours
Spam filter deployment: 8 hours
100% cost overrun estimate: 36 hours
Total setup costs: 72 hours @ 50EUR/hr: 3600EUR one time

This is the most imprecise evaluation. Most email systems have been built incrementally. The biggest unknown is the extra labor associated with running the IMAP server and spam filter. A few hypothesis:

1 hour a week: 52 hours @ 50EUR/hr: 2600EUR/yr
2 hours a week: 5200EUR/yr

I would be surprised if the extra work goes beyond one hour a week, and will probably be less. This also does not include 24/7 response time, but no service provider evaluated provides that level of service anyways.

Total:

One-time setup: 3800EUR (200EUR hardware, 3600EUR staff)
Recurrent: roughly between 5000EUR and 7500EUR/year, majority in staff

Alternatives considered

There are three dimensions to our “decision tree”:

Hosting mailboxes or only forwards: this means that instead of just forwarding emails to some other providers, we actually allow users to store emails on the server. Current situation is we only do forwards
SMTP authentication: this means allowing users to submit email using a username and password over the standard SMTP (technically “submission”) port. This is currently not allowed also some have figured out they can do this over SSH already.
Self-hosted or hosted elsewhere: if we host the email service ourselves right now or not. The current situation is we allow inbound messages but we do not store them. Mailbox storage is delegated to each individual choice of email provider, which also handles SMTP authentication.

Here are is the breakdown of pros and cons of each approach. Note that there are multiple combinations of those possible, for example we could continue not having mailboxes but allow SMTP authentication, and delegate this to a third party. Obviously, some combinations (like no SMTP authentication and mailboxes) are a little absurd and should be taken with a grain of salt.

TP full hosting: mailboxes, SMTP authentication

Pros:

Easier for TPA to diagnose email problems than if email is hosted by an undetermined third party
People’s personal email is not mixed up with Tor email.
Easier delegation between staff on rotations
Control over where data is stored and how
Full control of our infrastructure
Less trust issues

Cons:

probably the most expensive option
requires more skilled staff
high availability harder to achieve
high costs

TP not hosting mailboxes; TP hosting outgoing SMTP authentication server

Pros:

No data retention issues: TP not responsible for legal issues surrounding mailboxes contents
Solves delivery problem and nothing else (minimal solution)
We’re already running an SMTP server
SSH tunnels already let our lunatic-fringe do a version of this
Staff keeps using own mail readers (eg gmail UI) for receiving mail
Federated solution
probably the cheapest option
Work email cannot be accessed by TP staff

Cons:

SMTP-AUTH password management (admin effort and risk)
Possible legal requests to record outgoing mail? (SSH lunatic-fringe already at risk, though)
DKIM/SPF politics vs “slippery slope”
Forces people to figure out some good ISP to host their email
Shifts the support burden to individuals
Harder to diagnose email problems
Staff or “role” email accounts cannot be shared

TP pays third party (riseup, protonmail, mailfence, gmail??) for full service (mailboxes, delivery)

Pros:

Less admin effort
Less/no risk to TP infrastructure (legal or technical)
Third party does not hold email data hostage; only handles outgoing
We know where data is hosted instead of being spread around

Cons:

Not a federated solution
Implicitly accepts email cartel model of “trusted” ISPs
Varying levels of third party data management trust required
Some third parties require custom software (protonmail)
Single point of failure.
Might force our users to pick a provider they dislike
All eggs in the same basket

Status quo (no mailboxes, no authentication)

Pros:

Easy. Fast. Cheap. Pick three.

Cons:

Shifts burden of email debugging to users, lack of support

Details of the chosen alternative (SMTP authentication):

Postfix + offline LDAP authentication (current proposal)
Postfix + direct LDAP authentication: discarded because it might fail when the LDAP server goes down. LDAP server is currently not considered to be critical and can be restarted for maintenance without affecting the rest of the infrastructure.
reusing existing field like webPassword or rtcPassword in LDAP: considered a semantic violation.

See also internal Nextcloud document.

No benchmark considered necessary.