---
title: DNS procedures
---

[[_TOC_]]

# How to

Most operations on DNS happens in the `admin/dns/domains` repository
(`git@git-rw.torproject.org:admin/dns/domains`). Those zones contains
the master copy of the zone files, stored as (mostly) standard Bind zonefiles
([RFC 1034](https://tools.ietf.org/html/rfc1034)), but notably without a SOA.

Tor's DNS support is fully authenticated with DNSSEC, both to the
outside world but also internally, where all TPO hosts use DNSSEC in
their resolvers.

## Editing a zone

Zone records can be added or modified to a zone in the `domains` git
and a push. DNSSEC records are managed automatically by
`manage-dnssec-keys` in the `dns/dns-helpers` git repository. through
a cron job in the `dnsadm` user on the master DNS server (currently
nevii).

Serial numbers are managed automatically by the git repository hooks.

## Adding a zone

To add a *new* zone to our infrastructure, the following procedure
must be followed:

 1. add zone in `dns/domains.git` repository
 2. add zone in the
    `modules/bind/templates/named.conf.torproject-zones.erb` Puppet
    template for DNS secondaries to pick up the zone
 3. also add IP address ranges (if it's a reverse DNS zone file) to
    `modules/torproject_org/misc/hoster.yaml` in the `tor-puppet.git`
    repository
 4. run puppet on DNS servers: `cumin 'C:roles::dns_primary or C:bind::secondary' 'puppet agent -t'`
 5. add zone to `modules/postfix/files/virtual`, unless it is a
    reverse zonefile
 6. add zone to nagios: copy an existing `DNS SOA sync` block and
    adapt
 7. add zone to external DNS secondaries (currently [Netnod](https://www.netnod.se/))
 8. make sure the zone is delegated by the root servers somehow. for
    normal zones, this involves adding our nameservers in the
    registrar's configuration. for reverse DNS, this involves asking
    our upstreams to delegate the zone to our DNS servers.

Note that this is a somewhat rarer procedure: this happens only when a
completely new [domain name](https://en.wikipedia.org/wiki/Domain_name) (e.g. `torproject.net`) or IP address
space (so reverse DNS, e.g. `38.229.82.0/24` AKA
`82.229.38.in-addr.arpa`) is added to our infrastructure.

## Removing a zone

 * git grep the domain in the `tor-nagios` git repository
 * remove the zone in the `dns/domains.git` repository
 * on nevii, remove the generated zonefiles and keys:

        cd /srv/dns.torproject.org/var/keys/
        mv generated/torproject.fr* OLD-generated/
        mv keys/torproject.fr OLD-KEYS/

 * remove the zone from the secondaries (Netnod and our own
   servers). this means visiting the Netnod web interface for that
   side, and Puppet
   (`modules/bind/templates/named.conf.torproject-zones.erb`) for our
   own
 * the domains will probably be listed in other locations, grep Puppet
   for Apache virtual hosts and email aliases
 * the domains will also probably exist in the `letsencrypt-domains`
   repository

## DS records expiry and renewal

A special case is the rotation of the `DNSKEY` / `DS` records. Those
rotate about once every two years, and require manual operation on the
registrar (currently <https://joker.com>). 

A Nagios hook is in `/srv/dns.torproject.org/bin/dsa-check-and-extend-DS`, and
basically wraps `manage-dnssec-keys` with some Nagios status codes. It
will warn when the key is about to expire and extend it before it
expires (while still flagging a critical warning in Nagios).

To fix this error, you need to [visit joker.com](https://joker.com/) and authenticate
with the password in `hosts-extra-info` in tor-passwords, along with
the 2FA dance. Then:

 1. click on the "modify" button next to the domain affected (was
    first a gear but is now a pen-like icon thing)
 2. find the DNSSEC section
 3. click the "modify" button to edit records
 4. click "more" to add a record

The new key should already be present on the DNS master (currently
`nevii`) in:

    /srv/dns.torproject.org/var/keys/$DOMAIN/dsset

It is in the format (from [rfc4034](https://tools.ietf.org/html/rfc4034)):

    domain IN DS keytag algo type digest

For example:

    torproject.com.  IN DS 28234 8 2 260a11137e3fca013b90da649d50e9c5eb71b814cc1797ea81ee7c91c17b398a; Pub: 2019-05-25 17:40:07;  Act: 2019-05-25 17:40:07;  Inact: 2021-11-16 17:40:07;  Del: 2021-11-16 17:40:07;  Rev: 2021-10-02 17:40:07
    torproject.com.  IN DS 57040 8 2 ebdf81e6b773f243cdee2879f0d12138115d9b14d560276fcd88e9844777d7e3; Pub: 2021-06-13 17:40:07;  Act: 2021-06-13 17:40:07;  Inact: 2023-10-16 17:40:07;  Del: 2023-10-16 17:40:07;  Rev: 2023-09-01 17:40:07

Note that there are *two* keys there: one (the oldest) should already
be in Joker. you need to add the new one.

With the above, you would have the following in Joker:

 * `alg`: 8 ("RSA/SHA-256", [IANA](https://www.iana.org/assignments/dns-sec-alg-numbers/dns-sec-alg-numbers.xhtml), [RFC5702](https://www.rfc-editor.org/rfc/rfc5702.html))
 * `digest`: ebdf81e6b773f243cdee2879f0d12138115d9b14d560276fcd88e9844777d7e3
 * `type`: 2 ("SHA-256", [IANA](https://www.iana.org/assignments/ds-rr-types/ds-rr-types.xhtml), [RFC4509](https://www.rfc-editor.org/rfc/rfc4509.html))
 * `keytag`: 57040

And click "save".

Make sure to update the record in the `tor-puppet.git` repository, in:

    modules/unbound/files/torproject.org.key

Copy the latest `dsset` entry in there. Normally, unbound takes care
of updating that file to chase new versions, but new hosts will need
that new anchor for bootstrapping.

After a little while, you should be able to check if the new DS record
works on [DNSviz.net](http://dnsviz.net/), for example, the [DNSviz.net view of
torproject.net](http://dnsviz.net/d/torproject.net/dnssec/) should be sane.

The changes will take a while (~10 hours?) to trickle out into all
caches, so it might take a while for the Nagios check to return green.

Eventually, Nagios will complain about the old keys, and we can remove
them from the registrar. Make sure to remove the *old* key, not the
new key. Be careful because the web interface might sort the keys in
an unexpected way. Check the keytag and compare with the expiration
specified in the `dsset` file. The Nagios warning that you will see
will look like:

    DNS - security delegations: WARNING: torproject.com (57040,-28234), torproject.net (63619,-53722), torproject.org (33670,-28486)

The `-` entries (e.g. `-28234`) are the ones that should be removed.

Note: this procedure could be automated by talking with the
registrar's API, for example [Joker.com's DMAPI domain modification
API](https://joker.com/faq/content/27/24/en/domain_modify.html) (see also [those docs](https://dmapi.joker.com/docs/DMAPI-ext.txt)). There are also proposals at the
IETF to allow delegation from the parent zone to allow the child zone
to perform those updates on its own.

Further, puppet ships trust anchors for some of our zones to our unbounds.  If
you updated the DS for one of those, update the corresponding file in
`tsa-puppet/modules/unbound/files`.  Existing machines don't need that (since
we do slow, [RFC5011](https://www.rfc-editor.org/rfc/rfc5011.html)-style rolling of KSKs), but new instances will be sad if we
ship them obsolete trust anchors.

### Special case: RFC1918 zones

The above is for public zones, for which we have Nagios checks that
warn us about impeding doom. But we also sign zones about reverse IP
looks, specifically `30.172.in-addr.arpa.` Normally, recursive
nameservers pick new signatures in that zone automatically, thanks to
[rfc 5011](https://tools.ietf.org/html/rfc5011).

But if a new host gets provisionned, it needs to get bootstrapped
somehow. This is done by Puppet, but those records are maintained by
hand and will get out of date. This implies that after a while, you
will start seeing messages like this for hosts that were installed
after the expiration date:

    16:52:39 <nsa> tor-nagios: [submit-01] unbound trust anchors is WARNING: Warning: no valid trust anchors found for 30.172.in-addr.arpa.

The solution is to go on the primary nameserver (currently `nevii`)
and pick the non-revoked DSSET line from this file:

    /srv/dns.torproject.org/var/keys/30.172.in-addr.arpa/dsset

... and inject it in Puppet, in:

    tor-puppet/modules/unbound/files/30.172.in-addr.arpa.key

Then new hosts will get the right key and bootstrap properly. Old
hosts can get the new key by removing the file by hand on the server
and re-running Puppet:

    rm /var/lib/unbound/30.172.in-addr.arpa.key ; puppet agent -t

## Pager playbook

In general, to debug DNS issues, those tools are useful:

 * [DNSviz.net](https://dnsviz.net/), e.g. [a DNSSEC Authentication Chain](https://dnsviz.net/d/82.229.38.in-addr.arpa/dnssec/)
 * `dig`

### unbound trust anchors: Some keys are old

This warning can happen when a host was installed with old keys and
unbound wasn't able to rotate them:

    20:05:39 <nsa> tor-nagios: [chi-node-05] unbound trust anchors is WARNING: Warning: Some keys are old: /var/lib/unbound/torproject.org.key.

The fix is to remove the affected file and rerun Puppet:

    rm /var/lib/unbound/torproject.org.key
    puppet agent --test

### unbound trust anchors: Warning: no valid trust anchors

So this can happen too:

    11:27:49 <nsa> tor-nagios: [chi-node-12] unbound trust anchors is WARNING: Warning: no valid trust anchors found for 30.172.in-addr.arpa.

If this happens on *many* hosts, you will need to update the key, see
the [Special case: RFC1918 zones section](#special-case-rfc1918-zones), above. But if it's a
single host, it's possible it was installed during the window where
the key was expired, and hasn't been properly updated by Puppet
yet. 

Try this:

    rm /var/lib/unbound/30.172.in-addr.arpa.key ; puppet agent -t

Then the warning should have gone away:

    # /usr/lib/nagios/plugins/dsa-check-unbound-anchors
    OK: All keys in /var/lib/unbound recent and valid

If not, see the [Special case: RFC1918 zones section](#special-case-rfc1918-zones) above.

### DNS - zones signed properly is CRITICAL

When adding a new reverse DNS zone, it's possible you get this warning
from Nagios:

    13:31:35 <nsa> tor-nagios: [global] DNS - zones signed properly is CRITICAL: CRITICAL: 82.229.38.in-addr.arpa
    16:30:36 <nsa> tor-nagios: [global] DNS - key coverage is CRITICAL: CRITICAL: 82.229.38.in-addr.arpa

That might be because Nagios thinks this zone should be signed (while
it isn't and cannot). The fix is to add this line to the zonefile:

    ; ds-in-parent = no

And push the change. Nagios should notice and stop caring about the
zone.

In general, this Nagios check provides a good idea of the DNSSEC chain
of a zone:

    $ /usr/lib/nagios/plugins/dsa-check-dnssec-delegation overview 82.229.38.in-addr.arpa
                           zone DNSKEY               DS@parent       DLV dnssec@parent
    --------------------------- -------------------- --------------- --- ----------
         82.229.38.in-addr.arpa                                          no(229.38.in-addr.arpa), no(38.in-addr.arpa), yes(in-addr.arpa), yes(arpa), yes(.)

Notice how the `38.in-addr.arpa` zone is not signed? This zone can
therefore not be signed with DNSSEC.

### DNS - delegation and signature expiry is WARNING

If you get a warning like this:

    13:30:15 <nsa> tor-nagios: [global] DNS - delegation and signature expiry is WARNING: WARN: 1: 82.229.38.in-addr.arpa: OK: 12: unsigned: 0

It might be that the zone is not delegated by upstream. To confirm,
run this command on the Nagios server:

    $ /usr/lib/nagios/plugins/dsa-check-zone-rrsig-expiration  82.229.38.in-addr.arpa
    ZONE WARNING: No RRSIGs found; (0.66s) |time=0.664444s;;;0.000000

On the primary DNS server, you should be able to confirm the zone is
signed:

    dig @nevii  -b 127.0.0.1 82.229.38.in-addr.arpa +dnssec

Check the next DNS server up (use `dig -t NS` to find it) and see if
the zone is delegated:

    dig @ns1.cymru.com 82.229.38.in-addr.arpa +dnssec

If it's not delegated, it's because you forgot step 8 in the zone
addition procedure. Ask your upstream or registrar to delegate the
zone and run the checks again.

### DNS - security delegations is WARNING

This error:

    11:51:19 <nsa> tor-nagios: [global] DNS - security delegations is WARNING: WARNING: torproject.net (63619,-53722), torproject.org (33670,-28486)

... **will** happen after rotating the DNSSEC keys at the
registrar. The trick is then simply to remove those keys, at the
registrar. See [DS records expiry and renewal](#ds-records-expiry-and-renewal) for the procedure.

# Reference

## Design

This needs to be documented better. weasel made a [blog post](https://dsa.debian.org/dsablog/2014/The_Debian_DNS_universe/)
describing parts of the infrastructure on Debian.org, and that is
partly relevant to TPO as well.

TODO: adapt this document to the [service template](template).

## Other documentation

 * [DNSSEC debugger](https://dnssec-debugger.verisignlabs.com/)
 * [DNSviz.net](https://dnsviz.net/)

# Discussion

## Automation

Debian has a [set of scripts](https://salsa.debian.org/dsa-team/mirror/dsa-misc/-/tree/master/scripts/dns-providers) to automate talking to some providers
like Netnod. A YAML file has metadata about the configuration, and
pushing changes is as simple as:

    publish tor-dnsnode.yaml

That config file would look something like:

    ---
      endpoint: https://dnsnodeapi.netnod.se/apiv3/
      base_zone:
        endcustomer: "TorProject"
        masters:
          # nevii.torproject.org
          - ip: "49.12.57.130"
            tsig: "netnod-torproject-20180831."
          - ip: "2a01:4f8:fff0:4f:266:37ff:fee9:5df8"
            tsig: "netnod-torproject-20180831."
        product: "probono-premium-anycast"

This is not currently in use at TPO and changes are operated manually
through the web interface.
