TLS is the [Transport Layer Security][] protocol, previously known as
SSL and also known as [HTTPS][] on the web. This page documents how
TLS is used across the TPA infrastructure and specifically how we
manage the related [X.509][] certificates that make this work.

[X.509]: https://en.wikipedia.org/wiki/X.509
[HTTPS]: https://en.wikipedia.org/wiki/HTTPS
[Transport Layer Security]: https://en.wikipedia.org/wiki/Transport_Layer_Security
[[_TOC_]]

# Tutorial

## How to get an X.509 certificate for a domain with Let's Encrypt

 1. If not already done, clone git repos `letsencrypt-domains`:

        git clone ssh://git@git-rw.torproject.org/admin/letsencrypt-domains

 2. Add your domain name and optional alternative names (`SAN`) to the
    `domains` file:
    
        $EDITOR domains

 3. Push the updated domain list to the letsencrypt-domains repo

        git diff domains
        git add domains
        git commit
        git push

The last command will produce output from the `dehydrated` command
which talks with the DNS primary (currently `nevii`) to fetch new keys
and update old ones.

The new keys and certs are being copied to the LDAP host
(currently `pauli`) under
`/srv/puppet.torproject.org/from-letsencrypt/`. Then [howto/Puppet](howto/Puppet) pick
those up in the `ssl` module. Use the `ssl::service` resource to
deploy them.

See the "Design" section below for more information on how that works.

See also [howto/static-component](howto/static-component) for an example of how to deploy an
encrypted virtual host and onion service.

## Renewing a certificate before its expiry date

If a certificate has been revoked, it should be renewed before its
expiry date. To do so, you can drop a special file in the
`per-domain-config` directory to change the expiry date range and run
the script by hand.

Create a file matching the primary domain name of the certificate on
the DNS master:

    cat <<EOF > /srv/letsencrypt.torproject.org/repositories/letsencrypt-domains/per-domain-config/example.torproject.org
    RENEW_DAYS="85"
    EOF

Here we tell the ACME client (dehydrated) to renew the cert if it is
85 days or older (instead of the 30 days period).

Then run the script by hand (or wait for cron to do its thing):

    letsencrypt@nevii:~$ /srv/letsencrypt.torproject.org/bin/dehydrated-wrap --cron
    [...]
    Processing example.torproject.org with alternative names: example.torproject.org
     + Using certificate specific config file!
       + RENEW_DAYS = 85
     + Checking domain name(s) of existing cert... unchanged.
     + Checking expire date of existing cert...
     + Valid till May 18 20:40:45 2020 GMT Certificate will expire
    (Less than 85 days). Renewing!
     + Signing domains...
    [..]

Then remove the file.

# How-to

## Pager playbook

### Digicert validation emails

If you get email from DigiCert Validation, ask the Tor Browser team,
they use it to sign code (see "Design" below for more information
about which CAs are in use)

### Waiting for master to update

If a push to the Let's encrypt repository loops on a warning like:

    remote: Waiting for master to update torproject.net (for _acme-challenge.pages.torproject.net) from 2021012804.  Currently at 2021012804..

It might be because the Let's Encrypt hook is not *really* changing
the zonefile, and *not* incrementing the serial number (as hinted
above). This can happen if you force-push an empty change to the
repository and/or a previous hook failed to get a cert or was
interrupted.

The trick then is to abort the above push, then *manually edit* (yes)
the zonefile in (for the `torproject.net` domain, in the above
example):

    $EDITOR /srv/dns.torproject.org/var/generated/torproject.net

... and remove the ` _acme-challenge` line. Then you should somehow
update the zone with another, unrelated change, to trigger a serial
number change. For example, you could add a random A record:

    ynayMF5xckel8uGpo0GdVEQjM7X9    IN TXT "random record to trigger a zone rebuild, should be removed"

And push *that* change (in `dns/domains.git`). Then the serial number
will change, and the infrastructure will notice the `_acme-challenge`
record is gone. Then you can re-do the certification process and it
should go through.

Don't forget to remove the random `TXT` record created above once
everything is done.

### Challenge is invalid!

If you get an email that looks like:

    Subject: Cron <letsencrypt@nevii> sleep $(( RANDOM % 3600 )) && chronic dehydrated-wrap --cron

    [...]

    Waiting for master to update torproject.org (for _acme-challenge.dip.torproject.org) from 2021021304.  Currently at 2021021305..
    Waiting for secondaries to update to match master at 2021021305..
    Waiting for secondaries to update to match master at 2021021305..
    Waiting for secondaries to update to match master at 2021021305..
    Waiting for secondaries to update to match master at 2021021305..
    Waiting for secondaries to update to match master at 2021021305..
    Waiting for secondaries to update to match master at 2021021305..
    Waiting for secondaries to update to match master at 2021021305..
    Waiting for secondaries to update to match master at 2021021305..
     SOA nevii.torproject.org. hostmaster.torproject.org. 2021021305 10800 3600 1814400 3601 from server 49.12.57.135 in 0 ms.
     SOA nevii.torproject.org. hostmaster.torproject.org. 2021021304 10800 3600 1814400 3601 from server 194.58.198.32 in 11 ms.
     SOA nevii.torproject.org. hostmaster.torproject.org. 2021021305 10800 3600 1814400 3601 from server 95.216.159.212 in 26 ms.
     SOA nevii.torproject.org. hostmaster.torproject.org. 2021021305 10800 3600 1814400 3601 from server 89.45.235.22 in 29 ms.
     SOA nevii.torproject.org. hostmaster.torproject.org. 2021021305 10800 3600 1814400 3601 from server 38.229.72.12 in 220 ms.
    Waiting for secondaries to update to match master at 2021021305..
    Waiting for secondaries to update to match master at 2021021305..
    Waiting for secondaries to update to match master at 2021021305..
    Waiting for secondaries to update to match master at 2021021305..
    Waiting for master to update torproject.org (for _acme-challenge.gitlab.torproject.org) from 2021021304.  Currently at 2021021305..
    Waiting for secondaries to update to match master at 2021021305..
     + Responding to challenge for dip.torproject.org authorization...
     + Cleaning challenge tokens...
     + Challenge validation has failed :(
    ERROR: Challenge is invalid! (returned: invalid) (result: ["type"]	"dns-01"
    ["status"]	"invalid"
    ["error","type"]	"urn:ietf:params:acme:error:dns"
    ["error","detail"]	"During secondary validation: DNS problem: query timed out looking up CAA for torproject.org"
    ["error","status"]	400
    ["error"]	{"type":"urn:ietf:params:acme:error:dns","detail":"During secondary validation: DNS problem: query timed out looking up CAA for torproject.org","status":400}

It's because the DNS challenge took too long to deploy and it was
refused. This is harmless: it will eventually succeed. Ignore the
message, or, if you want to make sure, run the cron job by hand:

    ssh -tt root@nevii.torproject.org sudo -u letsencrypt /srv/letsencrypt.torproject.org/bin/dehydrated-wrap --cron

### db.torproject.org is WARNING: Certificate will expire

This message indicates the upcoming expiration of the OpenLDAP self-signed TLS
certificate.

See [howto/ldap#server-certificate-renewal](howto/ldap#server-certificate-renewal)
for instructions on how to renew it.

## Disaster recovery

No disaster recovery plan yet (TODO).

# Reference

## Installation

There is no documentation on how to deploy this service from
scratch. To deploy a new cert, see the above section and the
`ssl::service` Puppet resource.

## SLA

TLS is critical and should be highly available when relevant. It
should fail closed, that is if it fails a security check, it should
not allow a connexion.

## Design

TLS is one of two major transport security protocols used at TPA (the
other being [howto/ipsec](howto/ipsec)). It is used by web servers (Apache, HA Proxy,
Nginx), bacup servers (Bacula), mail servers (Postfix), and possibly
more.

Certificate generation is done by git hooks for Let's Encrypt or by a
`makefile` and cron job for auto-ca, see below for details.

### Certificate authorities in use at Tor

This documents mostly covers the Let's Encrypt certificates used by
websites and other services managed by TPA.

But there are other certificate authorities in use inside TPA and,
more broadly, at Tor. Here's the list of known CAs in operation at the
time of writing (2020-04-15):

 * [Let's Encrypt][]: automatically issues certificates for most websites
   and domains, managed by TPA
 * [Globalsign][]: used by the [Fastly][] CDN used to distributed
   TBB updates
 * [Digicert][]: used by other teams to sign software releases for
   Windows
 * [howto/Puppet](howto/Puppet): our configuration management infrastructure has its own
   X.509 certificate authority which allows "Puppet agents" to
   authenticate and verify the "Puppet Master", see [our
   documentation](howto/puppet) and [upstream documentation][] for details
 * [howto/ldap](howto/ldap): our OpenLDAP server uses a custom self-signed x.509
   certificate authority that is distributed to clients via Puppet, see
   [the documentation](howto/ldap#server-certificate-renewal) for instructions to
   renew this certificate manually
 * internal "auto-ca": all nodes in Puppet get their own X.509
   certificate signed by a standalone, self-signed X.509 certificate,
   documented below. it is used for backups (Bacula) and mail deliver
   (Postfix)

[upstream documentation]: https://puppet.com/docs/puppet/latest/ssl_certificates.html
[Digicert]: https://www.digicert.com/
[Fastly]: https://www.fastly.com/
[Globalsign]: https://globalsign.com
[Let's Encrypt]: https://letsencrypt.org

See also the [alternative certificate authorities we could consider](#other-certificate-authorities).

### Internal auto-ca

The internal "auto-ca" is a standalone certificate authority running
on the Puppet master (currently `pauli`), in
`/srv/puppet.torproject.org/auto-ca`. 

The CA runs based on a `Makefile` which takes care of creating,
revoking, and distributing certificates to all nodes. Certificates are
valid for a year (365 days, actually). If a certificate is going to
expire in less than 30 days, it gets revoked and removed.

The makefile then iterates over the known hosts (as per
`/var/lib/misc/thishost/ssh_known_hosts`, generated from [howto/ldap](howto/ldap)) to
create (two) certificates for each host. This makes sure certs get
renewed before their expiry. It will also remove certificates from
machines that are *not* known, which is the source of the `revoked
client` emails TPA gets when a machine gets [retired](howto/retire-a-host).

The Makefile then creates *two* certificates per host: a "clientcert"
(in `clientcerts/`) and a "server" (?) cert (in `certs/`). The former
is used by Bacula and Postfix clients to authenticate with the central
servers for backups and mail delivery, respectively. The latter is
used by those servers to authenticate to their clients but is also
used as default HTTPS certificates on new apache hosts.

Once all certs are created, revoked, and/or removed, they gets copied
into Puppet's file hierarchy, in the following locations:

 * `/etc/puppet/modules/ssl/files/certs/`: server certs
 * `/etc/puppet/modules/ssl/files/clientcerts/`: client certs.
 * `/etc/puppet/modules/ssl/files/clientcerts/fingerprints`:
   colon-separated `SHA256` fingerprints of all "client certs", one per
   line
 * `/etc/puppet/modules/ssl/files/certs/ca.crt`: CA's certificate
 * `/etc/puppet/modules/ssl/files/certs/ca.crl`: certificate
   revocation list

This work gets run from cron (in `/etc/cron.daily/local-auto-ca`)
which calls `make -s install` every day.

### Let's encrypt workflow

When you push to the git repository on the `git-rw.torproject.org`
server (currently `cupani`):

 1. a per-repository hook gets called in
    `/srv/git.torproject.org/git-helpers/post-receive-per-repo.d/admin\%letsencrypt-domains/trigger-letsencrypt-server`

 2. this hooks hits the DNS master over SSH (`letsencrypt@nevii`) and
    there the `authorized_keys` file hardcodes the command to
    `/srv/letsencrypt.torproject.org/bin/from-githost`

 3. ... which in turns just calls `bin/update` in the same directory
    (`/srv/letsencrypt.torproject.org`)

 4. ... which in turns pulls the `letsencrypt-domains` repository and
    runs `dehydrated-wrap --cron` with a special `BASE` variable that
    points dehydrated at our configuration, in
    `etc/dehydrated-config`, again in the same directory

 5. Through that special configuration, the dehydrated command is
    configured to call a custom hook (`bin/le-hook`) which implements
    logic around the DNS-01 authentication challenge, notably adding
    challenges, bumping serial numbers in the primary nameserver, and
    waiting for secondaries to sync. Note that there's a configuration
    file for that hook in `/etc/dsa/le-hook.conf`.

 6. The `le-hook` also pushes the changes around. The hook calls the
    `bin/deploy` file which installs the certificates files in
    `var/result`. 

 7. CODE REMOVED: It also generates a Public Key Pin (PKP) hash with the
    `bin/get-pin` command and appends Diffie-Hellman paramets
    (`dh-$size.pem`) to the certificate chain.

 8. It finally calls the `bin/push` command which runs `rsync` to the
    Puppet server, which in turns hardcodes the place where those
    files are dumped (in
    `pauli:/srv/puppet.torproject.org/from-letsencrypt`) through its
    `authorized_keys` file.

 9. Finally, those certificates are collected by Puppet through the
    `ssl` module. Pay close attention to how the
    `tor-puppet/modules/apache2/templates/ssl-key-pins.erb` template
    works: it will not deploy key pinning if the backup `.pin` file is
    missing.

## Issues

There is no issue tracker specifically for this project, [File][] or
[search][] for issues in the [team issue tracker][search] with the
~TLS label.

 [File]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/new
 [search]: https://gitlab.torproject.org/tpo/tpa/team/-/issues?label_name%5B%5D=TLS

## Monitoring and testing

When a HTTPS certificate is configured on a host, it MUST be
(manually) configured in Nagios. This can be done by adding the host
to the `apache-https-host`, `haproxy-https-host`, `nginx-https-hosts`,
depending on the webserver implementation. If the TLS server is
another implementation, a new check SHOULD be written.

All Let's Encrypt certificates are automatically checked for expiry by
Nagios as well, on top of the above checks.

## Logs and metrics

<!-- TODO: where are the logs? how long are they kept? any PII? -->
<!-- what about performance metrics? same questions -->

## Other documentation

TLS and X.509 is a vast application domain with lots of
documentation.

TODO: identify key TLS docs that should be linked to here. RFCs? LE
upstream docs?

The `letsencrypt-domains.git` repository is actually a fork of the
"upstream" project, from Debian System Administrators (DSA), see
[the upstream git repository](https://salsa.debian.org/dsa-team/mirror/letsencrypt-domains) for more information.

# Discussion

## Overview

There are no plans to do major changes to the TLS configuration,
although [review of the cipher suites][] is in progress (as of April
2020). We should have mechanisms to do such audits on a more
regular basis, and facilitate changes of those configurations over the
entire infrastructure.

[review of the cipher suites]: https://bugs.torproject.org/32351

<!-- the Goals, Approvals, Proposed solutions and Cost sections have -->
<!-- been removed from the template because we don't have any big -->
<!-- project on the TLS infra at the moment -->

## Goals

TODO: evaluate alternatives to the current letsencrypt deployment
systems and see if we can reduce the number of CAs.

### Must have

### Nice to have

### Non-Goals

## Approvals required

<!-- for example, legal, "vegas", accounting, current maintainer -->

## Proposed Solution

## Cost

## Alternatives considered

The `auto-ca` machinery could be replaced by Puppet code. We could
also leverage the ACME protocol designed by letsencrypt to run our own
CA instead of just OpenSSL, although that might be overkill. In
general it might be preferable to reuse an existing solution than
maintain our own software in Make.

### Other Certificate Authorities

There are actually a few other ACME-compatible certificate authorities
which issue free certificates. The [https.dev site](https://https.dev/) [lists a few
alternatives](https://docs.https.dev/list-of-acme-servers) which are, at the time of writing:

 * [Let's Encrypt](https://letsencrypt.org/) - currently in use
 * [ZeroSSL](https://zerossl.com/documentation/acme/) - Sectigo reseller
 * [BuyPass](https://www.buypass.com/ssl/products/acme) - Norway CA
 * [Sectigo](https://sectigo.com/resource-library/sectigos-acme-automation) - formerly known as Comodo CA
 * [InCommon](https://support.sectigo.com/Com_KnowledgeDetailPage?Id=kA01N000000bvYj) - also Sectigo?

### HPKP

HPKP used to be used at Tor, but we expired it in March 2020 and
completely stopped sending headers in October 2020. It is generally
considered Deprecated, it has been [disabled in Google Chrome in
2017][] and should generally not be used anymore. See [issue 33592][]
for details, and the history of this page for previous instructions.

[issue 33592]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/33592
[disabled in Google Chrome in 2017]: https://www.zdnet.com/article/google-chrome-is-backing-away-from-public-key-pinning-and-heres-why/
