If you get double OpenDKIM signatures, consider adding
`receive_override_options=no_milters` to the server in `master.cf`, if
already does some content filtering.
It's possible that some mail doesn't get signed when injected from
Mailman, consider changing the `InternalHosts` to:
InternalHosts lists-01.torproject.org,lists.torproject.org,127.0.0.1,::1,localhost # cargo-culted, to remove?
If the signatures come from another server and cause problems (for
example with Mailman), you can strip those with:
# strip incoming sigs, typically from submit-01 or other servers on mailman
RemoveOldSignatures yes
### SPF records
In [tpo/tpa/team#40990](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40990), anarcat deployed "soft" [SPF](https://en.wikipedia.org/wiki/Sender_Policy_Framework) records
for all outgoing mail servers under `torproject.org`. The full
specification of SPF is in [RFC7208](https://www.rfc-editor.org/rfc/rfc7208), here's a condensed
gitlab.torproject.org. 1415 IN CNAME gitlab-02.torproject.org.
gitlab-02.torproject.org. 3496 IN TXT "v=spf1 a ~all"
#### CRM: hard record
Finally, one last example is the CiviCRM records:
```
crm IN A 116.202.120.186 ; crm-int-01
IN AAAA 2a01:4f8:fff0:4f:266:37ff:fe4d:f883
IN TXT "v=spf1 a -all"
IN MX 0 crm
```
Those are similar to the `db.torproject.org` records except they are
"hard" (`-all`) which should, in theory, make other servers completely
reject attempts from servers not in the `A` or `AAAA` record of
`crm.torproject.org`.
#### Debugging SPF
[vamsoft's SPF checker](https://vamsoft.com/support/tools/spf-policy-tester) is good to quickly diagnose issues.
### DMARC records
[DMARC](https://en.wikipedia.org/wiki/DMARC) records glue together SPF and DKIM to tell which *policy*
to apply once the rules defined above check out (or not). It is
defined in [RFC7489](https://www.rfc-editor.org/rfc/rfc7489.html) and has a [friendly homepage](https://dmarc.org/) with a [good
introduction](https://dmarc.org/overview/). Note that DMARC usage has been growing steadily
since 2018 and more steeply since 2021, see the [usage stats](https://dmarc.org/stats/dmarc/). See
also the [Alex top site usage](https://dmarc.org/stats/alexa-top-sites/).
Our current DMARC policy is:
_dmarc IN TXT "v=DMARC1;p=none;pct=100;rua=mailto:postmaster@torproject.org"
That is a "soft" policy (`p=` is `none` instead of `quarantine` or
`reject`) that applies to all email (`pct=100`) and sends reports to
the `postmaster@` address.
Note that this applies to *all* subdomains by default, to change the
subdomain policy, the `sp=` mechanism would be used (same syntax as
`p=`, e.g. `sp=quarantine` would apply the `quarantine` policy to
subdomains, independently of the top domain policy). See [RFC 7489
section 6.6.3](https://www.rfc-editor.org/rfc/rfc7489.html#section-6.6.3) for more details on discovery.
### Submission server
To setup a new submission mail server, create a machine with the
`email::submission` role in Puppet. Ideally, it should be on a network
with a good IP reputation.
In `letsencrypt.git`, add an entry for that host's specific TLS
certificate. For example, the `submit-01.torproject.org` server has a
line like this:
submit-01.torproject.org submit.torproject.org
Those domains are glued together in DNS with:
submission IN CNAME submit-01
_submission._tcp IN SRV 0 1 465 submission
This implies there is only *one*`submission.torproject.org`, because
one cannot have multiple `CNAME` records, of course. But it should
make replacing the server transparent for end-users.
The latter SRV record is actually specified in [RFC6186](https://datatracker.ietf.org/doc/html/rfc6186), but may
not be sufficient for all automatic configuration. We do *not* go
deeper into auto-discovery, because that typically implies IMAP
servers and so on. But if we would, we could consider using [this
software which tries to support all of them](https://github.com/Monogramm/autodiscover-email-settings)(e.g.[Microsoft](https://docs.microsoft.com/en-us/exchange/architecture/client-access/autodiscover?view=exchserver-2019),
[Mozilla](https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Autoconfiguration/FileFormat/HowTo), [Apple](https://developer.apple.com/library/archive/featuredarticles/iPhoneConfigurationProfileRef/index.html)). For now, we'll only stick with the SRV
record.
### Mailman server
See the [mailman documentation](service/lists#installation).
## Upgrades
Upgrades should generally be covered by the normal Debian package
workflow.
## SLA
There is no SLA specific to this service, but mail delivery is
generally considered to be high priority. Complaints about delivery
failure should be filed as [issues in our ticket tracker](#issues) and
addressed.
## Design and architecture
The submission email service allows users to submit mail as if they
were on a `torproject.org` machine. Concretely, it is a Postfix server
which relays email to anywhere once [SASL authentication](https://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer) is
passed.
Most of the code is glue code in Puppet, along with a small set of
patches to ud-ldap which were sent (and mostly accepted) upstream.
### Email password sync flow
This horrid diagram describes the way email passwords are set from
LDAP to the submission server:
## Services
The "submission" port (587) was previously used in the documentation
by default because it is typically *less* blocked by ISP firewalls
than the "smtps" port (465), but both are supported. Lately, the
documentation has been changed for suggest port 465 first instead.
The TLS server is authenticated using the regular Let's Encrypt CA
(see [TLS documentation](howto/tls)).
## Storage
Mail services currently do not involve any sort of storage other than
mail queues (below).
## Queues
Mail servers typically transfer emails into a queue on reception, and
flush them out of the queue when the email is successfully
delivered. Temporary delivery failures are retried for 5 days
([`bounce_queue_lifetime`](https://www.postfix.org/postconf.5.html#bounce_queue_lifetime) and [`maximal_queue_lifetime`](https://www.postfix.org/postconf.5.html#maximal_queue_lifetime)).
We use the Postfix defaults for those settings, which may vary from
the above.
## Interfaces
Most of Postfix and Dovecot operations are done through the
commandline interface.
## Authentication
On the submission server, SASL authentication is delegated to a dummy
Dovecot server which is *only* used for authentication (i.e. it
doesn't provide IMAP or POP storage). Username/password pairs are
deployed by [ud-ldap](howto/ldap) into `/etc/dovecot/private/mail-passwords`.
The LDAP server stores those passwords in a `mailPassword` field and
the web interface is used to modify those passwords. Passwords are
(currently) encrypted with a salted MD5 hash because of compatibility
problems between the Perl/ud-ldap implementation and Dovecot which
haven't been resolved yet.
## Implementation
Most software in this space is written in C (Postfix, Dovecot, OpenDKIM).
## Related services
The submission and mail forwarding services both rely on the [LDAP
service](howto/ldap), for secrets and aliases, respectively.
The [mailing list service](services/lists) and [schleuder](service/schleuder) both depend on basic
email services for their normal operations. The [CiviCRM service](service/crm)
is also a particularly large mail sender.
## Issues
There is no issue tracker specifically for this project, [File][] or
[search][] for issues in the [team issue tracker][search] with the
~Email label.
When reporting email issues, do mind the [reporting email problems](doc/reporting-email-problems)
* TPA-RFC-45: Mail architecture (long term plans from TPA-RFC-44)
## High availability notes
If we do host our own IMAP servers eventually, we would like them to
be highly available, without human intervention. That means having an
"active-active" mirror setup where the failure of one host doesn't
affect users at all and doesn't require human intervention to restore
services.
We already know quite well how to do an active/passive setup: DRBD
allows us to replicate entire disks between machines. It *might* be
possible to do the same with active/active setups in DRBD, in theory,
but in practice this quickly runs into filesystem limitations, as
(e.g.) ext4 is *not* designed to be accessed by multiple machines
simultaneously.
Dovecot has a [replication system called dsync](https://doc.dovecot.org/configuration_manual/replication/) that replicates
mailboxes over a pipe. There are examples for TCP, TLS and SSH. [This
blog post](http://blog.dovecot.org/2012/02/dovecot-clustering-with-dsync-based.html) explains the design as well. A pair of [director](https://doc.dovecot.org/admin_manual/director/dovecotdirector/)
processes could be used to direct users to the right server. [This
tutorial](http://web.archive.org/web/20201111212844/https://blog.le-vert.net/?p=160) seems to have been useful for people.
Dovecot also shows a [HAProxy configuration](https://doc.dovecot.org/configuration_manual/haproxy/). A script called
[poolmon](https://github.com/brandond/poolmon/tree/master) seems to be used by some folks to remove/re-add backends
to the director when the go unhealthy. Dovecot now ships a
[dovemon](https://doc.dovecot.org/configuration_manual/dovemon/) program that works similarly, but it's only available in
the non-free "Pro" version.
There's also a [ceph plugin](https://github.com/ceph-dovecot/dovecot-ceph-plugin) to store emails in a Ceph backend.
It also seems possible to store mailbox and index objects in an
[object storage backend](https://doc.dovecot.org/admin_manual/dovecot_backend/), a configuration documented in the
[Dovecot Cluster Architecture](https://doc.dovecot.org/admin_manual/dovecot_cluster_architecture/). It seems that, unfortunately, this
is part of the "Pro" version of Dovecot, not usable in the free
version (see [mailbox formats](https://doc.dovecot.org/admin_manual/mailbox_formats/)). There's also someone who
implemented a [syncthing backend](https://github.com/fragtion/dovecot-core).
## Submission server proposal
Note: this proposal was discussed inline in the old
[howto/submission](howto/submission) page, before the TPA-RFC process existed. It is
kept here for historical reference.
The idea is to create a new server to deal with delivery problems
torproject.org email users are currently seeing. While they can
receive email through their `user@torproject.org` forwards without too
much problem, their emails often get dropped to the floor when
*sending* from that email address.
It is suspected that users are having those problems because the
originating servers are not in the `torproject.org` domain. The hope
is that setting up a new server inside that domain would help with
delivery. There's anecdotal evidence (see [this comment](https://gitlab.torproject.org/legacy/trac/-/issues/30608#note_2325187) for
example) that delivery emails from existing servers (over SSH to
`iranicum`, in that example) improves reliability of email delivery
significantly.
This project came out of [ticket #30608][], which has the launch
checklist.
[ticket #30608]:https://bugs.torproject.org/30608
Note: [this article](https://news.purelymail.com/posts/blog/2019-06-21-deliverability-for-the-rest-of-us.html) has a good overview of deliverability issues
faced by autonomous providers, which we already face on eugeni, but
might be accentuated by this project.
### Goals
#### Must have
* basic compatibility with major clients (Thunderbird, Mail.app,
Outlook, Gmail?)
* delivery over secure (TLS + password) SMTP
* credentials stored in LDAP
#### Nice to have
* automatic client configuration
* improved delivery over current federated configuration
* delivery reliability monitoring with major providers (e.g. hotmail,
gmail, yahoo)
* pretty graphs
* formalized SSH-key delivery to avoid storing cleartext passwords on
clients
#### Non-Goals
* 100%, infaillable, universal delivery to all providers (ie. emails
will still be lost)
* mailbox management (ie. no incoming email, IMAP, POP, etc)
* spam filtering (ie. we won't check outgoing emails)
* no DKIM, SPF, DMARC, or ARC for now, although maybe a "null" SPF
record if it helps with delivery
### Approvals required
Approved by vegas, requested by network team, agreed with TPA at the
Stockholm meeting.
### Proposed Solution
The proposed design is to setup a new email server in the [howto/ganeti](howto/ganeti)
cluster (currently `gnt-fsn`) with the user list synchronized from
LDAP, using a new password field (named `mailPassword`). The access
would therefore be granted only to LDAP users, and LDAP accounts would
be created as needed. In the short term, LDAP can be used to modify
that password but in the mid-term, it would be modifiable through the
web interface like the `webPassword` or `rtcPassword` fields.
#### Current inventory
* active LDAP accounts: 91
* non-LDAP forwards (to real people): 24
* role forwards (to other @torproject.org emails): 76
Forward targets:
* riseup.net: 30
* gmail.com: 21
* other: 93 (only 4 domains have more than one forward)
Delivery rate: SMTP, on eugeni, is around 0.5qps, with a max of 8qps
in the last 7 days (2019-06-06). But that includes mailing lists as
well. During that period, around 27000 emails were delivered to
@torproject.org aliases.
### Cost
Labor and `gnt-fsn` VM costs. To be detailed.
Below is an evaluation of the various Alternatives that were considered.
#### External hosting cost evaluation
* Google: 8$/mth/account? (to be verified?)
* riseup.net: anarcat requested price quotation
* koumbit.org: default pricing: 100$/year on shared hosting and 50GB
total, possibly no spam filter. 1TB disk: 500$/year. disk
encryption would need to be implemented, quoted 2000-4000$ setup
fee to implement it in the AlternC opensource control panel.
* self-hosting: ~4000-500EUR setup, 5000EUR-7500EUR/year, liberal
estimate (will probably be less)
*[mailfence](https://mailfence.com/en/secure-business-email.jsp) 1750 setup cost and 2.5 euros per user/year
Note that the self-hosting cost evaluation is for the fully-fledged
service. Option 2, above, of relaying email, has overall negligible
costs although that theory has been questioned by members of the
sysadmin team.
#### Internal hosting cost evaluation
This is a back-of-the-napkin calculation of what it would cost to host
actual email services at TPA infrastructure itself. We consider this
to be a “liberal” estimate, ie. costs would probably be less and time
estimates have been padded (doubled) to cover for errors.
Assumptions:
* each mailbox is on average, a maximum of 10GB
* 100 mailboxes maximum at first (so 1TB of storage required)
SRS would classify as 'other TPA mail server' in this picture. It notably *does* send mail to internet non-TPO mail hosts.
### Routing
Our main domain name is `torproject.org`. There are numerous subdomains and domain variants (e.g., `nevii.torproject.org`, `torproject.net`, etc.). These are all alias domains, meaning all addresses will be aliased to their `torproject.org` counterpart.
Lacking mailboxes, a `torproject.org` e-mail address can either be defined as an alias or as a forward.
Aliases are defined in Hiera.
Domain aliases are defined in Hiera and through puppet exported resources.
Forwards are defined in Hiera and in LDAP.
The MX resolves all aliases. It does *not* resolve forwards, but transports them to the SRS server(s). It does *not* deliver mail to internet non-TPO mail servers.
The SRS server resolves all forwards, applies sender rewriting when necessary, and sends the mail out into the world.
### DNS
MX servers need to be part of the `torproject.org` MX record.
SRS servers need to be part of the `srs.torproject.org` MX record.
SRS servers need to be added to the `torproject.org` SPF record.
SRS servers need to have their DKIM keys added to `torproject.org` DNS.
All mail servers need reverse DNS records.
All mail servers need a TLSA record.
### TLS
All mail servers need a valid public CA (Let's Encrypt) certificate.
All mail servers implement DANE (both in terms of publishing a TLSA record and checking for TLSA records when delivering mail).
### DMARC
DMARC checks are performed on the MX servers and Authentication-Results are added as header. DMARC is not strictly enforced, but failure to pass results in penalties to the spam score.
The SRS servers add DKIM signatures for all mail that has a From: header with a `torproject.org` address (header `From`, not envelope From). It does so using rspamd.
### Spam and virus checks
The MX server checks incoming mail for spam and virus using rspamd and clamav.
Viruses and very obvious spam get rejected straight away.
Suspicion of possible spam results in gray listing, with spam results added as headers when the mail does go through.
### SRS
SRS servers perform sender rewriting, changing the envelope `From` to a `torproject.org` address where needed.
The MX servers translate rewritten `torproject.org` addresses back to their original to process bounce mails.
