To do this, add a `profile::opendkim::domain` key in Hiera, ideally in the
yaml file that provide data for the node's role, eg.
`hiera/roles/email::submission.yaml` for the submission sever. The value must
be specified as an array containing one or more email sender domains. For
example, if a node sends mail as `gettor@torproject.org`, the
`profile::opendkim::domain` must be an array that contains `torproject.org`.
srs-dal-01 handles e-mail forwards to external providers and would classify as 'other TPA mail server' in this picture. It notably *does* send mail to internet non-TPO mail hosts.
Once this configuration is applied on the node, at least one new DNS entry must
be created for external hosts to be able to validate DKIM signatures. For
example, on the `submit-01` host, a `/etc/opendkim/keys/2022-submit-01.txt`
file is automatically generated containing a DNS entry appropriate for
validating mails signed for the `@torproject.org` sender domain:
Our main domain name is `torproject.org`. There are numerous subdomains and domain variants (e.g., `nevii.torproject.org`, `torproject.net`, etc.). These are all alias domains, meaning all addresses will be aliased to their `torproject.org` counterpart.
2022-submit-01._domainkey IN TXT ( "v=DKIM1; h=sha256; k=rsa; "
"H0MB6tE4O41rCjZUjSZ7Ar7BjVID6foCmlbr/3EG7dbzQv6YqH19OX6YgL0UMfG2RhvhWEUNYghS6K88vTelnHx/ShUzIeu05jd6mi9OLCA/Hl2bFRsa0f1ttHKpnzuC+ecn0sWwIDAQAB" ) ; ----- DKIM key 2022 for crm.torproject.org
Domain aliases are defined in Hiera and through puppet exported resources.
The key may now be tested with the command below. Make sure to use the correct
sender domain for the `-d` command line argument:
Forwards are defined in Hiera and in LDAP.
The MX resolves all aliases. It does *not* resolve forwards, but transports them to the SRS server(s). It does *not* deliver mail to internet non-TPO mail servers.
The SRS server resolves all forwards, applies sender rewriting when necessary, and sends the mail out into the world.
The sender address rewriting ensures forwarded mail originating from other domains doesn't break SPF by rewriting the from address to @torproject.org. This only affects the envelope-from address, not the header from.
Anything with a header from @torproject.org will be DKIM signed by the SRS server. This is done by rspamd. The required DNS record is automatically created by puppet.
If you see `record not found`, it's because the DNS record wasn't
found. See if DNS has propagated properly, maybe flush negative
responses with `unbound-control flush-negative`.
### resolving and sending of forwards
The `keys not secure` message means you are not using DNSSEC,
which should *not* happenin our configuration. Investigate if you
do see the warning.
Forwards are defined in two places:
- in `data/common/mail.yaml`, eventually ending up in `/etc/postfix/maps/alias`,
- in LDAP. SRS runs a local LDAP replica which it queries, according to `/etc/postfix/maps/alias_local`
8. stop puppet to keep it from messing with the Puppet config
To test if an LDAP forward is configured properly, you can run:
puppet agent --disable 'rush opendkim deployment by hand, tpo/tpa/team#40988'
9. reload postfix (warning: this will retry the queue)
Our submission server, submit-01, is managed by the profile::submission manifest.
service postfix reload
It provides the following functionality:
10.[test outgoing mail](#testing-outgoing-mail):
- relaying authenticated mail
- DKIM signing
The submission server *only* receives mail on smtps and submission ports and it only accepts authenticated mail.
Submission servers need a letsencrypt certificate for both their fqdn and submission.torproject.org, so be sure to add them to the `letsencrypt-domains` repo as follows:
```
echo "this is a test email" | mail -r postmaster@torproject.org -s 'test email from anarcat' -- check-auth@verifier.port25.com
- an SRV record for _submission._tcp.torproject.org
- an entry in the torproject.org SPF record
and must have a matching PTR record.
If you get double OpenDKIM signatures, consider adding
`receive_override_options=no_milters` to the server in `master.cf`, if
already does some content filtering.
There is currently no easy way to turn this into a highly available / redundant service, we'd have to research how different clients support failover mechanisms.
It's possible that some mail doesn't get signed when injected from
Mailman, consider changing the `InternalHosts` to:
#### Authentication
InternalHosts lists-01.torproject.org,lists.torproject.org,127.0.0.1,::1,localhost # cargo-culted, to remove?
SASL authentication is delegated to a dummy
Dovecot server which is *only* used for authentication (i.e. it
doesn't provide IMAP or POP storage). Username/password pairs are
deployed by [ud-ldap](howto/ldap) into `/etc/dovecot/private/mail-passwords`.
The LDAP server stores those passwords in a `mailPassword` field and
the web interface is used to modify those passwords. Passwords are
(currently) encrypted with a salted MD5 hash because of compatibility
problems between the Perl/ud-ldap implementation and Dovecot which
haven't been resolved yet.
This horrid diagram describes the way email passwords are set from
LDAP to the submission server:
If the signatures come from another server and cause problems (for
example with Mailman), you can strip those with:
#### DKIM signing
# strip incoming sigs, typically from submit-01 or other servers on mailman
RemoveOldSignatures yes
Anything with a header from @torproject.org will be DKIM signed by the submission server. This is done by rspamd. The required DNS record is automatically created by puppet.
### SPF records
### MTA
Our MTA servers, currently only mta-dal-01, are managed by the profile::mta manifest.
It provides the following functionality:
- relaying authenticated mail
- DKIM signing
The submission server *only* receives mail on the submission port from other TPO nodes and it only accepts authenticated mail.
MTA servers need a letsencrypt certificate, so be sure to add them to the `letsencrypt-domains` repo.
MTA servers need to be manually added to:
- the mta.torproject.org MX record
- the torproject.org SPF record
and must have a matching PTR record.
#### Authentication
Other TPO nodes are authenticated using client certificates. Distribution is done through puppet, the fingerprints are exported in the profile::postfix manifest and collected in the profile::mta manifest.
#### DKIM signing
Anything with a header from @torproject.org will be DKIM signed by the submission server. This is done by rspamd. The required DNS record is automatically created by puppet.
## Regular nodes
Regular nodes have no special mail needs and just need to be able to deliver mail. They can be recognised in puppet by having profile::postfix::independent set to false (the default value). They use our MTA servers as relayhost. This is taken care of by the profile::postfix manifest, which is included on all TPO nodes.
Currently regular nodes have no local mail delivery whatsoever, though this is subject to change, see [#42024](https://gitlab.torproject.org/tpo/tpa/team/-/issues/42024).
By default, system users will mail as @hostname.torproject.org . This has two disadvantages:
- Replying will result in mail sent to user@hostname.torproject.org , which is an alias for user@torproject.org. This may cause collisions between mail needs from different servers
- Mails from @hostname.torproject.org do not get any DKIM signature, which may cause them to be rejected by gmail and the likes.
We should ideally ensure an @torproject.org address is used for outgoing mail.
## Independent mailers
Independent mailers are nodes that receive mail on their own subdomain (which should be different from the node's fqdn) and/or deliver mail themselves without using our MTA. They can be recognised in puppet by having profile::postfix::independent set to true.
There are several things to take into consideration when setting up an independent mailer. In nearly all cases you need to make sure to include profile::rspamd.
If your node is going to accept mail, you need to:
- ensure there's an entry in the letsencypt-domains repo
- ensure there's an ssl::service with the appropriate tlsaport notifying Service['postfix']
- add appropriate postfix configuration for handling the incoming mail in profile::postfix::extra_params
- open up firewalling
- potentially adjust the profile::postfix::monitor_ports and monitor_tls_ports
- set an MX record
- ensure there's a PTR record
- add it to [dnswl.org](https://www.dnswl.org/)
If your node is going to deliver its own mail, you need to:
- if you're mailing as something other than @fqdn or @torproject.org, set an MX record (yes, an MX record is needed, it doesn't need to actually receive mail, but other mailers hate receiving mail from domains that don't have any MX)
- set / add to the appropriate SPF records
- set profile::rspamd::dkimdomain
- consider setting profile::rspamd::antispam to false if you're not receiving mail or don't care about spam
Examples of independent mailers are: lists-01.torproject.org, crm-int-01.torproject.org, rt.torproject.org
## DMARC
We currently have DMARC policy set to none, but this should be changed.
DKIM signing and verification is done by rspamd. The profile::rspamd::dkimdomain can be set to ensure all mail from those domains are signed. The profile automatically ensures the appropriate DNS record is created.
SPF verification is done by rspamd. SPF records for all TPO node fqdn's are automatically created in profile::postfix. The records for torproject.org itself and subdomains like rt.torproject.org and lists.torproject.org are managed manually.
In [tpo/tpa/team#40990](https://gitlab.torproject.org/tpo/tpa/team/-/issues/40990), anarcat deployed "soft" [SPF](https://en.wikipedia.org/wiki/Sender_Policy_Framework) records
for all outgoing mail servers under `torproject.org`. The full
specification of SPF is in [RFC7208](https://www.rfc-editor.org/rfc/rfc7208), here's a condensed
interpretation of our (current, 2025) policy:
interpretation of some of our (current, 2025) policies:
#### torproject.org
...
...
@@ -748,48 +792,27 @@ Some servers have a record like that:
db IN A 49.12.57.132 ; alberti
IN AAAA 2a01:4f8:fff0:4f:266:37ff:fea1:4d3 ; alberti
IN MX 0 alberti
IN TXT"v=spf1 a ~all"
IN TXT"v=spf1 a a:alberti.torproject.org ~all"
```
This is also a "soft" record that tells servers to check the `A` or
`AAAA` records (`a`) to see if it matches the connecting server. It
will match only if the connecting server has an IP matching the `A` or
`AAAA` record for `db.torproject.org`.
`AAAA` record for `db.torproject.org` or `alberti.torproject.org`.
#### lists.torproject.org: mx
```
lists IN TXT"v=spf1 mx ~all"
lists IN TXT"v=spf1 mx a:mta.tails.net a:lists-01.torproject.org a ~all"
```
This is also a "soft" record that tells servers to check the Mail
Exchanger record (`MX`) to see if it matches the connecting server.
It might be better to use a `a:` record here to avoid a DNS lookup,
but it might actually be possible that the MX for lists is in a
It also allows the Tails schleuder server to send as lists.torproject.org using the `a:` record.
The `a` and `a:lists-01.torproject.org` are redundant here, but it might actually be possible that the MX for lists is in a
different location than the web interface, for example.
#### gitlab: a and CNAME
Some mail servers have a record like:
```
gitlab IN CNAME gitlab-02
gitlab-02 IN TXT "v=spf1 a ~all" ; no one else than gitlab can send for gitlab
```
This is similar to the `db` record in that it uses a `a` mechanism,
but the actual name is behind a `CNAME` record.
In other words, only that server can send email as
`gitlab-02.torproject.org`. But since there's also a `CNAME` from
`gitlab` to `gitlab-02`, this policy actually also applies to
gitlab.torproject.org. 1415 IN CNAME gitlab-02.torproject.org.
gitlab-02.torproject.org. 3496 IN TXT "v=spf1 a ~all"
#### CRM: hard record
Finally, one last example is the CiviCRM records:
...
...
@@ -804,311 +827,52 @@ crm IN A 116.202.120.186 ; crm-int-01
Those are similar to the `db.torproject.org` records except they are
"hard" (`-all`) which should, in theory, make other servers completely
reject attempts from servers not in the `A` or `AAAA` record of
`crm.torproject.org`.
#### Debugging SPF
[vamsoft's SPF checker](https://vamsoft.com/support/tools/spf-policy-tester) is good to quickly diagnose issues.
### DMARC records
[DMARC](https://en.wikipedia.org/wiki/DMARC) records glue together SPF and DKIM to tell which *policy*
to apply once the rules defined above check out (or not). It is
defined in [RFC7489](https://www.rfc-editor.org/rfc/rfc7489.html) and has a [friendly homepage](https://dmarc.org/) with a [good
introduction](https://dmarc.org/overview/). Note that DMARC usage has been growing steadily
since 2018 and more steeply since 2021, see the [usage stats](https://dmarc.org/stats/dmarc/). See
also the [Alex top site usage](https://dmarc.org/stats/alexa-top-sites/).
Our current DMARC policy is:
_dmarc IN TXT "v=DMARC1;p=none;pct=100;rua=mailto:postmaster@torproject.org"
That is a "soft" policy (`p=` is `none` instead of `quarantine` or
`reject`) that applies to all email (`pct=100`) and sends reports to
the `postmaster@` address.
Note that this applies to *all* subdomains by default, to change the
subdomain policy, the `sp=` mechanism would be used (same syntax as
`p=`, e.g. `sp=quarantine` would apply the `quarantine` policy to
subdomains, independently of the top domain policy). See [RFC 7489
section 6.6.3](https://www.rfc-editor.org/rfc/rfc7489.html#section-6.6.3) for more details on discovery.
### Submission server
To setup a new submission mail server, create a machine with the
`email::submission` role in Puppet. Ideally, it should be on a network
with a good IP reputation.
In `letsencrypt.git`, add an entry for that host's specific TLS
certificate. For example, the `submit-01.torproject.org` server has a
line like this:
submit-01.torproject.org submit.torproject.org
Those domains are glued together in DNS with:
submission IN CNAME submit-01
_submission._tcp IN SRV 0 1 465 submission
This implies there is only *one*`submission.torproject.org`, because
one cannot have multiple `CNAME` records, of course. But it should
make replacing the server transparent for end-users.
The latter SRV record is actually specified in [RFC6186](https://datatracker.ietf.org/doc/html/rfc6186), but may
not be sufficient for all automatic configuration. We do *not* go
deeper into auto-discovery, because that typically implies IMAP
servers and so on. But if we would, we could consider using [this
software which tries to support all of them](https://github.com/Monogramm/autodiscover-email-settings)(e.g.[Microsoft](https://docs.microsoft.com/en-us/exchange/architecture/client-access/autodiscover?view=exchserver-2019),
[Mozilla](https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Autoconfiguration/FileFormat/HowTo), [Apple](https://developer.apple.com/library/archive/featuredarticles/iPhoneConfigurationProfileRef/index.html)). For now, we'll only stick with the SRV
record.
### Mailman server
See the [mailman documentation](service/lists#installation).
## Upgrades
Upgrades should generally be covered by the normal Debian package
workflow.
## SLA
There is no SLA specific to this service, but mail delivery is
generally considered to be high priority. Complaints about delivery
failure should be filed as [issues in our ticket tracker](#issues) and
addressed.
## Design and architecture
The submission email service allows users to submit mail as if they
were on a `torproject.org` machine. Concretely, it is a Postfix server
which relays email to anywhere once [SASL authentication](https://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer) is
passed.
Most of the code is glue code in Puppet, along with a small set of
patches to ud-ldap which were sent (and mostly accepted) upstream.
### Email password sync flow
This horrid diagram describes the way email passwords are set from
LDAP to the submission server:
## Services
`crm.torproject.org`. Note that `-all` is rarely enforced this strictly.
The "submission" port (587) was previously used in the documentation
by default because it is typically *less* blocked by ISP firewalls
than the "smtps" port (465), but both are supported. Lately, the
documentation has been changed for suggest port 465 first instead.
## DANE
The TLS server is authenticated using the regular Let's Encrypt CA
(see [TLS documentation](howto/tls)).
TLSA records are created through puppet using the tlsaport parameter of the ssl::service resource.
## Storage
We enforce DANE on all outgoing connections, except for stanford (what the hell, stanford?). This is defined in the tls_policy map in profile::postfix.
Mail services currently do not involve any sort of storage other than
mail queues (below).
## Spamfiltering
## Queues
We use rspamd and clamav for spamfiltering.
Mail servers typically transfer emails into a queue on reception, and
flush them out of the queue when the email is successfully
delivered. Temporary delivery failures are retried for 5 days
([`bounce_queue_lifetime`](https://www.postfix.org/postconf.5.html#bounce_queue_lifetime) and [`maximal_queue_lifetime`](https://www.postfix.org/postconf.5.html#maximal_queue_lifetime)).
We use the Postfix defaults for those settings, which may vary from
the above.
## Interfaces
Most of Postfix and Dovecot operations are done through the
commandline interface.
## Authentication
On the submission server, SASL authentication is delegated to a dummy
Dovecot server which is *only* used for authentication (i.e. it
doesn't provide IMAP or POP storage). Username/password pairs are
deployed by [ud-ldap](howto/ldap) into `/etc/dovecot/private/mail-passwords`.
The LDAP server stores those passwords in a `mailPassword` field and
the web interface is used to modify those passwords. Passwords are
(currently) encrypted with a salted MD5 hash because of compatibility
problems between the Perl/ud-ldap implementation and Dovecot which
haven't been resolved yet.
## Implementation
Most software in this space is written in C (Postfix, Dovecot, OpenDKIM).
## Related services
The submission and mail forwarding services both rely on the [LDAP
service](howto/ldap), for secrets and aliases, respectively.
The [mailing list service](services/lists) and [schleuder](service/schleuder) both depend on basic
email services for their normal operations. The [CiviCRM service](service/crm)
is also a particularly large mail sender.
## Issues
There is no issue tracker specifically for this project, [File][] or
[search][] for issues in the [team issue tracker][search] with the
~Email label.
When reporting email issues, do mind the [reporting email problems](doc/reporting-email-problems)
* TPA-RFC-45: Mail architecture (long term plans from TPA-RFC-44)
## High availability notes
Since SMTP conveniently has failover mechanisms built in, it would be easy to add redundancy for our MX, SRS, and MTA servers by simply deploying copies of them.
If we do host our own IMAP servers eventually, we would like them to
be highly available, without human intervention. That means having an
...
...
@@ -1144,346 +908,24 @@ is part of the "Pro" version of Dovecot, not usable in the free
version (see [mailbox formats](https://doc.dovecot.org/admin_manual/mailbox_formats/)). There's also someone who
implemented a [syncthing backend](https://github.com/fragtion/dovecot-core).
## Submission server proposal
Note: this proposal was discussed inline in the old
[howto/submission](howto/submission) page, before the TPA-RFC process existed. It is
kept here for historical reference.
The idea is to create a new server to deal with delivery problems
torproject.org email users are currently seeing. While they can
receive email through their `user@torproject.org` forwards without too
much problem, their emails often get dropped to the floor when
*sending* from that email address.
It is suspected that users are having those problems because the
originating servers are not in the `torproject.org` domain. The hope
is that setting up a new server inside that domain would help with
delivery. There's anecdotal evidence (see [this comment](https://gitlab.torproject.org/legacy/trac/-/issues/30608#note_2325187) for
example) that delivery emails from existing servers (over SSH to
`iranicum`, in that example) improves reliability of email delivery
significantly.
This project came out of [ticket #30608][], which has the launch
checklist.
[ticket #30608]:https://bugs.torproject.org/30608
Note: [this article](https://news.purelymail.com/posts/blog/2019-06-21-deliverability-for-the-rest-of-us.html) has a good overview of deliverability issues
faced by autonomous providers, which we already face on eugeni, but
might be accentuated by this project.
### Goals
#### Must have
* basic compatibility with major clients (Thunderbird, Mail.app,
Outlook, Gmail?)
* delivery over secure (TLS + password) SMTP
* credentials stored in LDAP
#### Nice to have
* automatic client configuration
* improved delivery over current federated configuration
* delivery reliability monitoring with major providers (e.g. hotmail,
gmail, yahoo)
* pretty graphs
* formalized SSH-key delivery to avoid storing cleartext passwords on
clients
#### Non-Goals
* 100%, infaillable, universal delivery to all providers (ie. emails
will still be lost)
* mailbox management (ie. no incoming email, IMAP, POP, etc)
* spam filtering (ie. we won't check outgoing emails)
* no DKIM, SPF, DMARC, or ARC for now, although maybe a "null" SPF
record if it helps with delivery
### Approvals required
Approved by vegas, requested by network team, agreed with TPA at the
Stockholm meeting.
### Proposed Solution
The proposed design is to setup a new email server in the [howto/ganeti](howto/ganeti)
cluster (currently `gnt-fsn`) with the user list synchronized from
LDAP, using a new password field (named `mailPassword`). The access
would therefore be granted only to LDAP users, and LDAP accounts would
be created as needed. In the short term, LDAP can be used to modify
that password but in the mid-term, it would be modifiable through the
web interface like the `webPassword` or `rtcPassword` fields.
#### Current inventory
* active LDAP accounts: 91
* non-LDAP forwards (to real people): 24
* role forwards (to other @torproject.org emails): 76
Forward targets:
* riseup.net: 30
* gmail.com: 21
* other: 93 (only 4 domains have more than one forward)
Delivery rate: SMTP, on eugeni, is around 0.5qps, with a max of 8qps
in the last 7 days (2019-06-06). But that includes mailing lists as
well. During that period, around 27000 emails were delivered to
@torproject.org aliases.
### Cost
Labor and `gnt-fsn` VM costs. To be detailed.
Below is an evaluation of the various Alternatives that were considered.
#### External hosting cost evaluation
* Google: 8$/mth/account? (to be verified?)
* riseup.net: anarcat requested price quotation
* koumbit.org: default pricing: 100$/year on shared hosting and 50GB
total, possibly no spam filter. 1TB disk: 500$/year. disk
encryption would need to be implemented, quoted 2000-4000$ setup
fee to implement it in the AlternC opensource control panel.
* self-hosting: ~4000-500EUR setup, 5000EUR-7500EUR/year, liberal
estimate (will probably be less)
*[mailfence](https://mailfence.com/en/secure-business-email.jsp) 1750 setup cost and 2.5 euros per user/year
Note that the self-hosting cost evaluation is for the fully-fledged
service. Option 2, above, of relaying email, has overall negligible
costs although that theory has been questioned by members of the
sysadmin team.
#### Internal hosting cost evaluation
This is a back-of-the-napkin calculation of what it would cost to host
actual email services at TPA infrastructure itself. We consider this
to be a “liberal” estimate, ie. costs would probably be less and time
estimates have been padded (doubled) to cover for errors.
Assumptions:
* each mailbox is on average, a maximum of 10GB
* 100 mailboxes maximum at first (so 1TB of storage required)
SRS would classify as 'other TPA mail server' in this picture. It notably *does* send mail to internet non-TPO mail hosts.
### Routing
Our main domain name is `torproject.org`. There are numerous subdomains and domain variants (e.g., `nevii.torproject.org`, `torproject.net`, etc.). These are all alias domains, meaning all addresses will be aliased to their `torproject.org` counterpart.
Lacking mailboxes, a `torproject.org` e-mail address can either be defined as an alias or as a forward.
Aliases are defined in Hiera.
Domain aliases are defined in Hiera and through puppet exported resources.
Forwards are defined in Hiera and in LDAP.
The MX resolves all aliases. It does *not* resolve forwards, but transports them to the SRS server(s). It does *not* deliver mail to internet non-TPO mail servers.
The SRS server resolves all forwards, applies sender rewriting when necessary, and sends the mail out into the world.
### DNS
MX servers need to be part of the `torproject.org` MX record.
SRS servers need to be part of the `srs.torproject.org` MX record.
SRS servers need to be added to the `torproject.org` SPF record.
SRS servers need to have their DKIM keys added to `torproject.org` DNS.
All mail servers need reverse DNS records.
All mail servers need a TLSA record.
### TLS
All mail servers need a valid public CA (Let's Encrypt) certificate.
All mail servers implement DANE (both in terms of publishing a TLSA record and checking for TLSA records when delivering mail).
### DMARC
DMARC checks are performed on the MX servers and Authentication-Results are added as header. DMARC is not strictly enforced, but failure to pass results in penalties to the spam score.
The SRS servers add DKIM signatures for all mail that has a From: header with a `torproject.org` address (header `From`, not envelope From). It does so using rspamd.
### Spam and virus checks
The MX server checks incoming mail for spam and virus using rspamd and clamav.
Viruses and very obvious spam get rejected straight away.
We went through a number of proposals to improve mail services over
time:
Suspicion of possible spam results in gray listing, with spam results added as headers when the mail does go through.
*[TPA-RFC-15: Email services](policy/tpa-rfc-15-email-services)(rejected, replaced with
TPA-RFC-31)
*[TPA-RFC-31: outsource email services](policy/tpa-rfc-31-outsource-email)(rejected as well, in
