@@ -514,9 +515,60 @@ and use their personal address instead of user@torproject.org.
# Reference
## Mail servers
## Installation
The new mail server setup is fully Puppetized. See the [design and
architecture section](#design-and-architecture) for more information about the various
components and associated Puppet classes in use.
### Submission server
To setup a new submission mail server, create a machine with the
`email::submission` role in Puppet. Ideally, it should be on a network
with a good IP reputation.
In `letsencrypt.git`, add an entry for that host's specific TLS
certificate. For example, the `submit-01.torproject.org` server has a
line like this:
submit-01.torproject.org submit.torproject.org
Those domains are glued together in DNS with:
submission IN CNAME submit-01
_submission._tcp IN SRV 0 1 465 submission
This implies there is only *one*`submission.torproject.org`, because
one cannot have multiple `CNAME` records, of course. But it should
make replacing the server transparent for end-users.
The latter SRV record is actually specified in [RFC6186](https://datatracker.ietf.org/doc/html/rfc6186), but may
not be sufficient for all automatic configuration. We do *not* go
deeper into auto-discovery, because that typically implies IMAP
servers and so on. But if we would, we could consider using [this
software which tries to support all of them](https://github.com/Monogramm/autodiscover-email-settings)(e.g.[Microsoft](https://docs.microsoft.com/en-us/exchange/architecture/client-access/autodiscover?view=exchserver-2019),
[Mozilla](https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Autoconfiguration/FileFormat/HowTo), [Apple](https://developer.apple.com/library/archive/featuredarticles/iPhoneConfigurationProfileRef/index.html)). For now, we'll only stick with the SRV
record.
### Mailman server
See the [mailman documentation](service/lists#installation).
## Upgrades
### Design & architecture
Upgrades should generally be covered by the normal Debian package
workflow.
## SLA
There is no SLA specific to this service, but mail delivery is
generally considered to be high priority. Complaints about delivery
failure should be filed as [issues in our ticket tracker](#issues) and
addressed.
## Design and architecture
### Mail servers
Our main mail servers (mx-dal-01, srs-dal-01, mta-dal-01, and submit-01) try to fit into the picture presented in TPA-RFC-44:
[DMARC](https://en.wikipedia.org/wiki/DMARC) records glue together SPF and DKIM to tell which *policy*
to apply once the rules defined above check out (or not). It is
defined in [RFC7489](https://www.rfc-editor.org/rfc/rfc7489.html) and has a [friendly homepage](https://dmarc.org/) with a [good
introduction](https://dmarc.org/overview/). Note that DMARC usage has been growing steadily
since 2018 and more steeply since 2021, see the [usage stats](https://dmarc.org/stats/dmarc/). See
also the [Alex top site usage](https://dmarc.org/stats/alexa-top-sites/).
Our current DMARC policy is:
_dmarc IN TXT "v=DMARC1;p=none;pct=100;rua=mailto:postmaster@torproject.org"
That is a "soft" policy (`p=` is `none` instead of `quarantine` or
`reject`) that applies to all email (`pct=100`) and sends reports to
the `postmaster@` address.
Note that this applies to *all* subdomains by default, to change the
subdomain policy, the `sp=` mechanism would be used (same syntax as
`p=`, e.g. `sp=quarantine` would apply the `quarantine` policy to
subdomains, independently of the top domain policy). See [RFC 7489
section 6.6.3](https://www.rfc-editor.org/rfc/rfc7489.html#section-6.6.3) for more details on discovery.
We currently have DMARC policy set to none, but this should be changed.
DKIM signing and verification is done by rspamd. The profile::rspamd::dkimdomain can be set to ensure all mail from those domains are signed. The profile automatically ensures the appropriate DNS record is created.
...
...
@@ -864,460 +937,6 @@ or
/usr/bin/rspamc -h localhost learn_ham
```
## Monitoring
By default, all servers with profile::postfix::independent set to true are monitored by Prometheus.
This only checks that the SMTP port (or optionally whatever you set in profile::postfix::monitor_ports or monitor_tls_ports) is open. We do
not have end to end delivery monitoring just yet, that is part of the
[improve mail services milestone](https://gitlab.torproject.org/groups/tpo/tpa/-/milestones/4), specifically [issue 40494](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40494).
All servers that have `profile::postfix::mtail_monitor` enabled (which is the default) have the `mtail` exporter
(`profile::prometheus::postfix_mtail_exporter`). The [Grafana
dashboard](https://grafana.torproject.org/d/Ds5BxBYGk/postfix-mtail?orgId=1&var-node=submit-01.torproject.org) should provide shiny graphs.
## High availability
We currently have no high availability/redundancy.
Since SMTP conveniently has failover mechanisms built in, it would be easy to add redundancy for our MX, SRS, and MTA servers by simply deploying copies of them.
If we do host our own IMAP servers eventually, we would like them to
be highly available, without human intervention. That means having an
"active-active" mirror setup where the failure of one host doesn't
affect users at all and doesn't require human intervention to restore
services.
We already know quite well how to do an active/passive setup: DRBD
allows us to replicate entire disks between machines. It *might* be
possible to do the same with active/active setups in DRBD, in theory,
but in practice this quickly runs into filesystem limitations, as
(e.g.) ext4 is *not* designed to be accessed by multiple machines
simultaneously.
Dovecot has a [replication system called dsync](https://doc.dovecot.org/configuration_manual/replication/) that replicates
mailboxes over a pipe. There are examples for TCP, TLS and SSH. [This
blog post](http://blog.dovecot.org/2012/02/dovecot-clustering-with-dsync-based.html) explains the design as well. A pair of [director](https://doc.dovecot.org/admin_manual/director/dovecotdirector/)
processes could be used to direct users to the right server. [This
tutorial](http://web.archive.org/web/20201111212844/https://blog.le-vert.net/?p=160) seems to have been useful for people.
Dovecot also shows a [HAProxy configuration](https://doc.dovecot.org/configuration_manual/haproxy/). A script called
[poolmon](https://github.com/brandond/poolmon/tree/master) seems to be used by some folks to remove/re-add backends
to the director when the go unhealthy. Dovecot now ships a
[dovemon](https://doc.dovecot.org/configuration_manual/dovemon/) program that works similarly, but it's only available in
the non-free "Pro" version.
There's also a [ceph plugin](https://github.com/ceph-dovecot/dovecot-ceph-plugin) to store emails in a Ceph backend.
It also seems possible to store mailbox and index objects in an
[object storage backend](https://doc.dovecot.org/admin_manual/dovecot_backend/), a configuration documented in the
[Dovecot Cluster Architecture](https://doc.dovecot.org/admin_manual/dovecot_cluster_architecture/). It seems that, unfortunately, this
is part of the "Pro" version of Dovecot, not usable in the free
version (see [mailbox formats](https://doc.dovecot.org/admin_manual/mailbox_formats/)). There's also someone who
implemented a [syncthing backend](https://github.com/fragtion/dovecot-core).
## Future plans
We went through a number of proposals to improve mail services over
time:
*[TPA-RFC-15: Email services](policy/tpa-rfc-15-email-services)(rejected, replaced with
TPA-RFC-31)
*[TPA-RFC-31: outsource email services](policy/tpa-rfc-31-outsource-email)(rejected as well, in
If you get double OpenDKIM signatures, consider adding
`receive_override_options=no_milters` to the server in `master.cf`, if
already does some content filtering.
It's possible that some mail doesn't get signed when injected from
Mailman, consider changing the `InternalHosts` to:
InternalHosts lists-01.torproject.org,lists.torproject.org,127.0.0.1,::1,localhost # cargo-culted, to remove?
If the signatures come from another server and cause problems (for
example with Mailman), you can strip those with:
# strip incoming sigs, typically from submit-01 or other servers on mailman
RemoveOldSignatures yes
### SPF records
In [tpo/tpa/team#40990](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40990), anarcat deployed "soft" [SPF](https://en.wikipedia.org/wiki/Sender_Policy_Framework) records
for all outgoing mail servers under `torproject.org`. The full
specification of SPF is in [RFC7208](https://www.rfc-editor.org/rfc/rfc7208), here's a condensed
gitlab.torproject.org. 1415 IN CNAME gitlab-02.torproject.org.
gitlab-02.torproject.org. 3496 IN TXT "v=spf1 a ~all"
#### CRM: hard record
Finally, one last example is the CiviCRM records:
```
crm IN A 116.202.120.186 ; crm-int-01
IN AAAA 2a01:4f8:fff0:4f:266:37ff:fe4d:f883
IN TXT "v=spf1 a -all"
IN MX 0 crm
```
Those are similar to the `db.torproject.org` records except they are
"hard" (`-all`) which should, in theory, make other servers completely
reject attempts from servers not in the `A` or `AAAA` record of
`crm.torproject.org`.
#### Debugging SPF
[vamsoft's SPF checker](https://vamsoft.com/support/tools/spf-policy-tester) is good to quickly diagnose issues.
### DMARC records
[DMARC](https://en.wikipedia.org/wiki/DMARC) records glue together SPF and DKIM to tell which *policy*
to apply once the rules defined above check out (or not). It is
defined in [RFC7489](https://www.rfc-editor.org/rfc/rfc7489.html) and has a [friendly homepage](https://dmarc.org/) with a [good
introduction](https://dmarc.org/overview/). Note that DMARC usage has been growing steadily
since 2018 and more steeply since 2021, see the [usage stats](https://dmarc.org/stats/dmarc/). See
also the [Alex top site usage](https://dmarc.org/stats/alexa-top-sites/).
Our current DMARC policy is:
_dmarc IN TXT "v=DMARC1;p=none;pct=100;rua=mailto:postmaster@torproject.org"
That is a "soft" policy (`p=` is `none` instead of `quarantine` or
`reject`) that applies to all email (`pct=100`) and sends reports to
the `postmaster@` address.
Note that this applies to *all* subdomains by default, to change the
subdomain policy, the `sp=` mechanism would be used (same syntax as
`p=`, e.g. `sp=quarantine` would apply the `quarantine` policy to
subdomains, independently of the top domain policy). See [RFC 7489
section 6.6.3](https://www.rfc-editor.org/rfc/rfc7489.html#section-6.6.3) for more details on discovery.
### Submission server
To setup a new submission mail server, create a machine with the
`email::submission` role in Puppet. Ideally, it should be on a network
with a good IP reputation.
In `letsencrypt.git`, add an entry for that host's specific TLS
certificate. For example, the `submit-01.torproject.org` server has a
line like this:
submit-01.torproject.org submit.torproject.org
Those domains are glued together in DNS with:
submission IN CNAME submit-01
_submission._tcp IN SRV 0 1 465 submission
This implies there is only *one*`submission.torproject.org`, because
one cannot have multiple `CNAME` records, of course. But it should
make replacing the server transparent for end-users.
The latter SRV record is actually specified in [RFC6186](https://datatracker.ietf.org/doc/html/rfc6186), but may
not be sufficient for all automatic configuration. We do *not* go
deeper into auto-discovery, because that typically implies IMAP
servers and so on. But if we would, we could consider using [this
software which tries to support all of them](https://github.com/Monogramm/autodiscover-email-settings)(e.g.[Microsoft](https://docs.microsoft.com/en-us/exchange/architecture/client-access/autodiscover?view=exchserver-2019),
[Mozilla](https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Autoconfiguration/FileFormat/HowTo), [Apple](https://developer.apple.com/library/archive/featuredarticles/iPhoneConfigurationProfileRef/index.html)). For now, we'll only stick with the SRV
record.
### Mailman server
See the [mailman documentation](service/lists#installation).
## Upgrades
Upgrades should generally be covered by the normal Debian package
workflow.
## SLA
There is no SLA specific to this service, but mail delivery is
generally considered to be high priority. Complaints about delivery
failure should be filed as [issues in our ticket tracker](#issues) and
addressed.
## Design and architecture
The submission email service allows users to submit mail as if they
were on a `torproject.org` machine. Concretely, it is a Postfix server
which relays email to anywhere once [SASL authentication](https://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer) is
passed.
Most of the code is glue code in Puppet, along with a small set of
patches to ud-ldap which were sent (and mostly accepted) upstream.
### Email password sync flow
This horrid diagram describes the way email passwords are set from
LDAP to the submission server:
## Services
The "submission" port (587) was previously used in the documentation
...
...
@@ -1393,6 +1012,11 @@ The emergency changes to the infrastructure (including DKIM, DMARC,
and SPF records) were done as part of [TPA-RFC-44](policy/tpa-rfc-44-email-emergency-recovery)
SRS would classify as 'other TPA mail server' in this picture. It notably *does* send mail to internet non-TPO mail hosts.
### Routing
Our main domain name is `torproject.org`. There are numerous subdomains and domain variants (e.g., `nevii.torproject.org`, `torproject.net`, etc.). These are all alias domains, meaning all addresses will be aliased to their `torproject.org` counterpart.
Lacking mailboxes, a `torproject.org` e-mail address can either be defined as an alias or as a forward.
Aliases are defined in Hiera.
Domain aliases are defined in Hiera and through puppet exported resources.
Forwards are defined in Hiera and in LDAP.
The MX resolves all aliases. It does *not* resolve forwards, but transports them to the SRS server(s). It does *not* deliver mail to internet non-TPO mail servers.
The SRS server resolves all forwards, applies sender rewriting when necessary, and sends the mail out into the world.
### DNS
MX servers need to be part of the `torproject.org` MX record.
SRS servers need to be part of the `srs.torproject.org` MX record.
SRS servers need to be added to the `torproject.org` SPF record.
SRS servers need to have their DKIM keys added to `torproject.org` DNS.
All mail servers need reverse DNS records.
All mail servers need a TLSA record.
### TLS
All mail servers need a valid public CA (Let's Encrypt) certificate.
All mail servers implement DANE (both in terms of publishing a TLSA record and checking for TLSA records when delivering mail).
### DMARC
DMARC checks are performed on the MX servers and Authentication-Results are added as header. DMARC is not strictly enforced, but failure to pass results in penalties to the spam score.
The SRS servers add DKIM signatures for all mail that has a From: header with a `torproject.org` address (header `From`, not envelope From). It does so using rspamd.
### Spam and virus checks
The MX server checks incoming mail for spam and virus using rspamd and clamav.
Viruses and very obvious spam get rejected straight away.
Suspicion of possible spam results in gray listing, with spam results added as headers when the mail does go through.
### SRS
SRS servers perform sender rewriting, changing the envelope `From` to a `torproject.org` address where needed.
The MX servers translate rewritten `torproject.org` addresses back to their original to process bounce mails.
