Skip to content
Snippets Groups Projects
Verified Commit f271393e authored by anarcat's avatar anarcat
Browse files

word-wrap, regroup links

parent 31d1f812
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
policy/tpa-rfc-15-email-services.md
+ 89
− 101
View file @ f271393e
......@@ -6,10 +6,10 @@ title: TPA-RFC-15: email services
Summary: deploy incoming and outgoing [SPF][], [DKIM][], [DMARC][],
and (possibly) [ARC][] checks and records on torproject.org
infrastructure. Deploy an IMAP service, alongside
enforcement of the use of the submission server for outgoing
mail. Establish end-to-end deliverability monitoring. Rebuild mail
services to get rid of legacy infrastructure.
infrastructure. Deploy an IMAP service, alongside enforcement of the
use of the submission server for outgoing mail. Establish end-to-end
deliverability monitoring. Rebuild mail services to get rid of legacy
infrastructure.
[DMARC]: https://en.wikipedia.org/wiki/DMARC
[DKIM]: https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail
......@@ -41,23 +41,18 @@ users to send emails themselves through our infrastructure.
This situation led to users sending email with `@torproject.org` email
addresses from arbitrary locations on the internet: Gmail, Riseup, and
other service providers (including personal mail servers)
are typically used to send email for
`torproject.org` users.
other service providers (including personal mail servers) are
typically used to send email for `torproject.org` users.
This changed at the end of 2021 when the new [submission service][]
came online. We still, however, have limited adoption of this service,
with only 16 users registered compared to the ~100 users in LDAP.
[submission service]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/submission
In parallel, we have historically not adopted any modern
email standards like SPF, DKIM, or DMARC. But more recently, we
added SPF records to both the Mailman and CiviCRM servers (see [ticket
In parallel, we have historically not adopted any modern email
standards like SPF, DKIM, or DMARC. But more recently, we added SPF
records to both the Mailman and CiviCRM servers (see [issue
40347][]).
[ticket 40347]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40347
We have also been processing DKIM headers on incoming emails on the
`bridges.torproject.org` server, but that is an exception. Finally, we
are running Spamassassin on the RT server to try to deal with the
......@@ -66,30 +61,31 @@ large influx of spam on the generic support addresses (`support@`,
on incoming mail in any way, which has caused problems with Hetzner
([issue 40539][]).
[issue 40539]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40539
We do not have any DMARC headers anywhere in DNS, but we do have
workarounds setup in Mailman for delivering email correctly when the
sender has DMARC records, since September 2021 (see [ticket 19914][]).
sender has DMARC records, since September 2021 (see [issue 19914][]).
[ticket 19914]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/19914
We do not offer mailboxes, although we do have Dovecot servers
deployed for specific purposes. The GitLab and CiviCRM servers, for
example, use it for incoming email processing, and the submission
server uses it for authentication.
We do not offer mailboxes, although we do have Dovecot servers deployed for
specific purposes. The GitLab and CiviCRM servers, for example, use it
for incoming email processing, and the submission server uses it for
authentication.
[submission service]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/submission
[issue 40347]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40347
[issue 40539]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40539
[issue 19914]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/19914
## Known issues
The current email infrastructure has many problems.
In general, people feel like their
emails are not being delivered or "getting into spam". And sometimes, in
the other direction, people simply cannot get mail from certain
domains.
The current email infrastructure has many problems. In general,
people feel like their emails are not being delivered or "getting into
spam". And sometimes, in the other direction, people simply cannot get
mail from certain domains.
Here are the currently documented problems:
* deliverability issues: [Yahoo][], [state.gov][], [Gmail][], [Gmail again][]
* deliverability issues: [Yahoo][], [state.gov][], [Gmail][], [Gmail
again][]
* reception issues: state.gov
* complaints about lists.tpo lacking SPF/DKIM ([issue 40347][])
* submission server incompatible with Apple Mail/Outlook (see [issue
......@@ -109,16 +105,15 @@ Interlocking issues:
* DMARC requires a monitoring system to be effectively enabled
In general, we lack end-to-end deliverability tests to see if any
measures we take have an impact ([issue tpo/tpa/team#40494][]).
measures we take have an impact ([issue 40494][]).
[Yahoo]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/34134
[state.gov]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40202
[Gmail]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40170
[Gmail again]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40149
[issue 40347]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40347
[issue 40586]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40586
[issue 40604]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40604
[issue tpo/tpa/team#40494]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40494
[issue 40494]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40494
## Previous evaluations
......@@ -134,8 +129,8 @@ It did not touch on email standards like this proposal does.
# Proposal
After a grace period, we progressively add "soft", then
"hard" SPF, DKIM, and DMARC record to the `lists.torproject.org`,
After a grace period, we progressively add "soft", then "hard" SPF,
DKIM, and DMARC record to the `lists.torproject.org`,
`crm.torproject.org`, `rt.torproject.org`, and, ultimately,
`torproject.org` domains.
......@@ -169,12 +164,12 @@ for that matter, Discourse, RT, or other services that may use email
unless explicitly mentioned).
It also does *not* address directly phishing and scamming attacks
([issue tpo/tpa/team#40596][], but it is hoped that stricter
([issue 40596][], but it is hoped that stricter
enforcement of email standards will reduce those to a certain
extent. The rebuild of certain parts of the legacy infrastructure will
also help deal with such attacks in the future.
[issue tpo/tpa/team#40596]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40596
[issue 40596]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40596
## Affected users
......@@ -211,9 +206,9 @@ End-to-end deliverability monitoring involves:
* DMARC/MTA-STS feedback loops (covered below)
This may be implemented as Nagios or Prometheus checks ([issue
40539][]). This also
includes evaluating how to monitor metrics offered by [Google
postmaster tools][] and Microsoft ([issue 40168][]).
40539][]). This also includes evaluating how to monitor metrics
offered by [Google postmaster tools][] and Microsoft ([issue
40168][]).
[issue 40168]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40168
[Google postmaster tools]: https://postmaster.google.com
......@@ -231,16 +226,17 @@ This might also include extra work for MTA-STS feedback loops.
This consists of an IMAP and webmail server deployment.
We are currently already using Dovecot in a limited way on some
servers, so we will reuse some of that Puppet code for the IMAP server. The
webmail will likely be deployed with Roundcube, alongside the IMAP
server. Both programs are packaged and well supported in Debian. Mail
filtering is detailed in another section below.
servers, so we will reuse some of that Puppet code for the IMAP
server. The webmail will likely be deployed with Roundcube, alongside
the IMAP server. Both programs are packaged and well supported in
Debian. Mail filtering is detailed in another section below.
### Incoming mail filtering
Deploy a tool for inspection of incoming mail for SPF, DKIM, DMARC
records, affecting either "reputation" (e.g. add a marker in mail headers)
or just downright rejection (e.g. rejecting mail before queue).
records, affecting either "reputation" (e.g. add a marker in mail
headers) or just downright rejection (e.g. rejecting mail before
queue).
We currently use Spamassassin for this purpose, and we could consider
collaborating with the [Debian listmasters][] for the Spamassassin
......@@ -259,7 +255,7 @@ replacing a part of `eugeni`.
### New mail relays
Configure new "mail relay" server(s) to relay mails from servers that
do not send their own email, replacing a part of `eugeni`.
do not send their own email, replacing a part of `eugeni`.
This is similar to current submission server, except with TLS
authentication instead of password.
......@@ -297,10 +293,10 @@ Email][] discussion.
Refactor the mail-related code in Puppet, and reconfigure all servers
according to the mail relay server change above, see [issue
tpo/tpa/team#40626][] for details. This should probably happen
40626][] for details. This should probably happen
*before* or *during* all the other tasks.
[issue tpo/tpa/team#40626]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40626
[issue 40626]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40626
## Architecture diagram
......@@ -350,7 +346,7 @@ Changes in this diagram:
This timeline is a draft, and will be updated according to when this
proposal is adopted. The changes will be distributed over a year, and
the following is a per-quarter breakdown, starting from when the
proposal is adopted.
proposal is adopted.
Obviously, the deployment will depend on availability of TPA staff and
the collaboration of TPO members. It might also be reordered to
......@@ -386,7 +382,7 @@ since our current codebase around email services is a little old and
hard to modify. We will need to spend some time to refactor and
cleanup that codebase before we can move ahead with more complicated
solutions like incoming SPF checks or outgoing DKIM signatures, for
example. See [issue tpo/tpa/team#40626][] for details.
example. See [issue 40626][] for details.
### Incoming filtering implementation
......@@ -430,10 +426,10 @@ our current LDAP service would be a difficult challenge.
Hosting people's email contents adds a new security
concern. Typically, we are not very worried about "leaks" inside TPA
infrastructure, except in rare situations (like bridgedb). Most of the
data we host is public, in other words. If we start hosting
mailboxes, we suddenly have a much higher risk of leaking personal
data in case of compromise. This is a trade-off with the privacy we
gain from not giving that data to a third party.
data we host is public, in other words. If we start hosting mailboxes,
we suddenly have a much higher risk of leaking personal data in case
of compromise. This is a trade-off with the privacy we gain from not
giving that data to a third party.
## Cost estimates
......@@ -465,9 +461,8 @@ process follows the [Kaplan-Moss estimation technique][].
| 8. new mail relays | 3 days | low | similar to current submission server | 3.3 |
| 9. Puppet refactoring | 1 week | high | | 10 |
This amounts to a total estimate time of 80 days, or about 16 weeks
or four months, full time. At 50EUR/hr, that's about 32,000EUR of
work.
This amounts to a total estimate time of 80 days, or about 16 weeks or
four months, full time. At 50EUR/hr, that's about 32,000EUR of work.
This estimate doesn't cover for ongoing maintenance costs and support
associated with running the service. So far, the submission server has
......@@ -520,14 +515,14 @@ by hosting it on our shared virtualized infrastructure.
Here we collect a few "personas" and try to see how the changes will
affect them.
> We have taken the liberty of creating mostly fictitious personas, but
> they are somewhat based on real-life people. We do not mean to
> We have taken the liberty of creating mostly fictitious personas,
> but they are somewhat based on real-life people. We do not mean to
> offend. Any similarity that might seem offensive is an honest
> mistake on our part which we will be happy to correct. Also note that
> we might have mixed up people together, or forgot some. If your use
> case is not mentioned here, please do report it. We don't
> need to have *exactly* "you" here, but all your current use cases
> should be covered by one or many personas.
> mistake on our part which we will be happy to correct. Also note
> that we might have mixed up people together, or forgot some. If your
> use case is not mentioned here, please do report it. We don't need
> to have *exactly* "you" here, but all your current use cases should
> be covered by one or many personas.
## Ariel, the fundraiser
......@@ -546,10 +541,9 @@ through Gmail!
Their email is forwarded to Google Mail and they do *not* have an LDAP
account.
They will need to get an LDAP acount, set a mail password, and
either use the Webmail service or configure a mail client like
Thunderbird to access the IMAP server and submit email through the
submission server.
They will need to get an LDAP acount, set a mail password, and either
use the Webmail service or configure a mail client like Thunderbird to
access the IMAP server and submit email through the submission server.
Technically, it would also be possible to keep using Gmail to send
email as long as it is configured to relay mail through the submission
......@@ -558,13 +552,13 @@ server, but that configuration will be unsupported.
## Gary, the support guy
Gary is the ticket master. He eats tickets for breakfast, then files
10 more before coffee. A hundred tickets is just a normal day at
the office. Tickets come in through email, RT, Discourse, Telegram,
10 more before coffee. A hundred tickets is just a normal day at the
office. Tickets come in through email, RT, Discourse, Telegram,
Snapchat and soon, TikTok dances.
Email is absolutely mission critical, but some days he wishes there
could be slightly less of it. He deals with a lot of spam, and
surely something could be done about that.
could be slightly less of it. He deals with a lot of spam, and surely
something could be done about that.
His mail forwards to Riseup and he reads his mail over Thunderbird and
sometimes webmail.
......@@ -604,27 +598,27 @@ believes everyone should be entitled to run their own mail server.
Her email is, of course, hosted on her own mail server, and she have
an LDAP account.
She will have to reconfigure her Postfix server to relay mail
through the submission or relay servers, if she want to go fancy. To
read email, she will need to download email from the IMAP server,
although it will still be technically possible to forward her
`@torproject.org` email to her personal server directly, as long as
the server is configured to send email through the TPO servers.
She will have to reconfigure her Postfix server to relay mail through
the submission or relay servers, if she want to go fancy. To read
email, she will need to download email from the IMAP server, although
it will still be technically possible to forward her `@torproject.org`
email to her personal server directly, as long as the server is
configured to send email through the TPO servers.
## Mallory, the director
Mallory also does a lot of mailing. She's on about a dozen aliases and
mailing lists from accounting to HR and other obscure ones everyone
forgot what they're for. She also deals with funders, job
applicants, contractors and staff.
forgot what they're for. She also deals with funders, job applicants,
contractors and staff.
Email is absolutely mission critical for her. She often fails to
contact funders and critical partners because state.gov blocks our
email (or we block theirs!). Sometimes, she gets told through LinkedIn that
a job application failed, because mail bounced at Gmail.
email (or we block theirs!). Sometimes, she gets told through LinkedIn
that a job application failed, because mail bounced at Gmail.
She has an LDAP account and it forwards to Gmail. She uses Apple
Mail to read their mail.
She has an LDAP account and it forwards to Gmail. She uses Apple Mail
to read their mail.
For her Mac, she'll need to configure the submission server *and* the
IMAP server in Apple Mail. Like Ariel, it is technically possible for
......@@ -671,10 +665,8 @@ We realized the IMAP was a requirement requirement because the SPF
records will *require* people to start using the submission server to
send mail. And that, in turn requires an IMAP server because of
clients limitations. For example, it's not possible to configure Apple
mail of Office 365 with a remote SMTP server unless they also provide an
IMAP service, see [issue tpo/tpa/team#40586][] for details.
[issue tpo/tpa/team#40586]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40586
mail of Office 365 with a remote SMTP server unless they also provide
an IMAP service, see [issue 40586][] for details.
It's also possible that implementing mailboxes could help improve spam
filtering capabilities, which are after all necessary to ensure good
......@@ -697,23 +689,19 @@ forwarding setup, however.
We have not explicitly designed this proposal for high availability
situations, which have been explicitly requested in [issue
tpo/tpa/team#40604][]. The current design is actually more scalable
40604][]. The current design is actually more scalable
than the previous legacy setup, because each machine will be setup by
Puppet and highly reproducible, with minimal local state (except for
the IMAP server). So while it *may* be possible to scale up the
service for higher availability in the future, it's not a mandatory
part of the work described here.
[issue tpo/tpa/team#40604]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40604
In particular, setting up new mail exchanger and submission servers is
somewhat trivial. It consists of setting up new machines in separate
locations and following the install procedure. There is no state
replicated between the servers other than what is already done through
[LDAP][].
[LDAP]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/ldap
The IMAP service is another problem, however. It will potentially have
large storage requirements (terabytes) and will be difficult to
replicate using our current tool set. We may consider setting it up on
......@@ -726,6 +714,8 @@ spare.
Multi-primary setups would require "sharding" the users across
multiple servers and is definitely considered out of scope.
[LDAP]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/ldap
## Personal SPF/DKIM records and partial external hosting
At Debian.org, it's possible for members to configure their own DKIM
......@@ -779,13 +769,11 @@ another millennia, and it is not the primary objective of the Tor
Project to continue using it, let alone host the infrastructure.
There are actually many different alternatives to email emerging, many
of which are already in use in the community.
of which are already in use in the community.
For example, we already have a [Discourse server][] that is generating
great community participation and organisation.
[Discourse server]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40539
We have also seen a good uptake on the Matrix bridges to our IRC
channels. Many places are seeing increase use of chat tools like Slack
as a replacement for email, and we could adopt Matrix more broadly as
......@@ -797,9 +785,6 @@ as well.
[Nextcloud][] and [Big Blue Button][] also provide us with asynchronous and
synchronous coordination mechanisms.
[Big Blue Button]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/conference/
[Nextcloud]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/nextcloud
We may be able to convert many of our uses of email right now to some
other tools:
......@@ -824,13 +809,16 @@ mass-mailing provider service like [Mailchimp][] or [Amazon SES][] for
mass mailings, but this raises questions regarding the privacy of our
users. This is currently considered to be an unacceptable compromise.
[Mailchimp]: https://mailchimp.com/
[Amazon SES]: https://docs.aws.amazon.com/ses/
There is therefore not a clear alternative to all of those problems
right now, so we consider email to be a mandatory part of our
infrastructure for the time being.
[Discourse server]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/40539
[Big Blue Button]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/conference/
[Nextcloud]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/service/nextcloud
[Mailchimp]: https://mailchimp.com/
[Amazon SES]: https://docs.aws.amazon.com/ses/
## External hosting
Other service providers have been contacted to see if it would be
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment