Skip to content
Snippets Groups Projects
Select Git revision
  • master default protected
  • tpa-rfc-80-merge-tor-and-tails-support-policies
  • puppet-merge-costs
  • rfc-77-improv
  • tails-merge-admin-mailing-lists
  • mkdocs
  • new-blog-doc
7 results
Blame Permalink
exit-ops.mdwn 9.58 KiB 
# exit-ops

Exit Scanner, TorDNSEL and Tor Check Operations

[[!toc levels=3]]

# Overview

While the three services described in this document could be implemented as discrete components,
they currently have tight coupling which means they must all be deployed on the same host.

```
                +------------+
                |  Exit      +---------------------------------------+
                |  Scanner   |-----------------------------+         |
                |            +----------> DNS Zone File    |         |
                +--------+---+                 +           |         |
            launches|    ^                     |           |         |
                    v    |gets results         v           v         |
+-------+       +---+--------+            +----+-----+     Bulk      |
| Tor   +^------+   exitmap  |            |  TSA     |     Exit      |
| Client+------v+            |            |  Managed |     List      |
+-------+       +------------+            |  BIND    |               |
              using ipscan module         +----------+               |
                                                                     |
                                                                     v
                         +--------+          +-----------+     Exit List
                         | Tor    +<---------+   Tor     |           +
                         | Client |          |   Check   +<----------+
                         |        +--------->+           |
                         +--------+          +-----+-----+
                                                   ^
                                                   |
                                             +-----+-----+
                                             |  Apache   |
                                             |  Reverse  |
                                             |  Proxy    |
                                             +-----------+
```

## Exit Scanner

The exit scanner performs active measurement of Tor exit relays in order to determine the IP addresses that are used for exit connections.
The active measurement uses an exitmap module, which is wrapped in a script to produce output formatted as an [Exit List](<https://metrics.torproject.org/collector.html#type-tordnsel>).

The exit list results are consumed by CollecTor, [TorDNSEL](tordnsel) and [Tor Check](../check-ops/).
Exit lists and bulk exit lists are also consumed by third-party external applications at the following URLs:

- <https://check.torproject.org/exit-addresses> - Latest exit list
- <https://check.torproject.org/torbulkexitlist> - Latest bulk exit list

Previous scan results are kept in the latest exit list for up to 48 hours since
they were last seen. Scans will run every 40 minutes, or continuously if scans
take longer than 40 minutes to complete.

There is currently no process that will clean up old exit lists from the exit
list server.

## TorDNSEL

TorDNSEL is a DNS list service that behaves in a similar way to [Domain Name System-based Blackhole Lists](https://en.wikipedia.org/wiki/Domain_Name_System-based_Blackhole_List).
IP addresses will give positive results in the event that an address has been found to be used by an exit relay in a recent scan.

As with the exit lists, IP addresses will be removed after not being seen for
48 hours.

A future update to this script may include information from the consensus, but
currently a postitive match must have been found from the exit scanner for an
IP address to be included.

## Tor Check

Tor Check is a website that can be used to determine if a browser is using the Tor network for queries.
It will also check the User-Agent to determine if a user is using Tor Browser.
It is accessed via HTTPS at <https://check.torproject.org/>.

There are also JSON APIs:

- <https://check.torproject.org/api/ip> - Check IP and if it's a Tor IP
- <https://check.torproject.org/api/bulk> - Bulk exit list exporter

# Contacts

The primary contact for this service is the Metrics Team <[metrics-team@lists.torproject.org](mailto:metrics-team@lists.torproject.org)>.
For urgent queries, contact **karsten** or **gaba** in IRC.

# Implementation Specifics

The underlying infrastructure for the exit scanner, TorDNSEL and Tor Check services is provided by the
Tor Sysadmin Team (TSA). All services run on one virtual machine with the hostname `check-01.torproject.org`.

## Exit Scanner

The exitmap module used is a custom module named [ipscan](https://gitweb.torproject.org/metrics-cloud.git/tree/ansible/roles/exit-scanner/files/ipscan.py).

The
[exitscan](https://gitweb.torproject.org/metrics-cloud.git/tree/ansible/roles/exit-scanner/files/exitscan.py)
wrapper runs as a systemd user service under the `tordnsel` user, named
[exitscanner.service](https://gitweb.torproject.org/metrics-cloud.git/tree/ansible/roles/exit-scanner/files/exitscanner.service).

## TorDNSEL

The exitscan wrapper produces a BIND zone file
`/srv/tordnsel.torproject.org/dnsel.torproject.org`. The wrapper also
automatically reloads this zone file in the TPA managed BIND server whenever it
has been updated. The name `dnsel.torproject.org` is delegated to the `check-01`
host with NS records.

weasel had ideas about how we could run this as a hidden master, transfer the
zone to the main domain servers and also have them sign the zone with DNSSEC.
This is an idea for the future.

# Sources

The sources for exitmap are available on GitHub: <https://github.com/NullHypothesis/exitmap>.
The [exitmap wrapper](https://gitweb.torproject.org/metrics-cloud.git/tree/ansible/roles/exit-scanner/files/exitscan.py) and [module](https://gitweb.torproject.org/metrics-cloud.git/tree/ansible/roles/exit-scanner/files/ipscan.py) used by the exit scanner can be found in the metrics-cloud repository.

The wrapper script is also responsible for writing out the zone file to be used by the TorDNSEL service
and triggering a reload of the zone.

The sources for Tor Check are available in our git: <https://gitweb.torproject.org/check.git>.

# Deployment

## Initial deployment

The initial deployment procedure is split into 2 parts:

-   Installing and starting the services

There are no manual steps required to load state, and backups do not need to be
performed for the host running this service.

Everything can be configured from scratch with only the Ansible playbook.

### Development/testing in AWS

For development or testing in AWS, a CloudFormation template is available named [`exit-scanner-dev.yml`](https://gitweb.torproject.org/metrics-cloud.git/plain/cloudformation/exit-scanner-dev.yml).

From the [CloudFormation portal](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks), select your stack and view the outputs.
You will find here the public IP address for the EC2 instance that has been created.
Add this instance to **ansible/dev** in your local copy of metrics-cloud.git under "[exit-scanners]".

You can now setup the machine with Ansible by running:

```
ansible-playbook -i dev exit-scanners-aws.yml
```

Note that the AWS AMI used has passwordless sudo, so no password need be given.

### Fresh machine from TSA

Add the host name of the new instance to **ansible/production** in your local
copy of metrics-cloud.git under "[exit-scanners]" and commit the change.

You can now setup the machine with Ansible by running:

```
ansible-playbook -i production -K exit-scanners.yml
```

## Upgrade

The upstream sources for the applications that make up this service do not have managed releases
which makes this difficult.

To fix a bug in the exit scanner wrapper script, fix this in the metrics-cloud repository and re-run
the deployment playbook.

-   [ ] Can we upgrade exitmap sensibly?
-   [ ] Can we upgrade Tor Check sensibly?

# Diagnostics

## Logs

The exit scanner wrapper logs output to `/srv/tordnsel.torproject.org/exitscanner.log`.

These logs include every measurement ever performed, and this might fill up the disk eventually. See #33475.

# Monitoring

CollecTor gives a warning in the case that the exit lists are too old.

There are no Nagios checks for this service beyond the standard TPA systems checks (and I didn't actually check if we have those yet).

# Disaster Recovery

## Service needs restart

The service can be restarted, as the tordnsel user, with:

```
XDG_RUNTIME_DIR=/run/user/`id -u` systemctl --user restart exitscanner.service
```

## Total Loss

The exit scanner service does not need to maintain any state between runs.
It's nice if it can in order to cope with a relay that happened to be down at the time we tried to measure
it but in the event of a failure it is perfectly acceptable to throw away the old box and provision a new one.
Follow the initial deployment instructions above.

# Standards

The exit scanner service produces exit lists according to the [TorDNSEL exit list format](https://2019.www.torproject.org/tordnsel/exitlist-spec.txt).

# Major Caveats

The exit scanner service does not support IPv6.

Serial numbers for the DNS zones are based on the date. This needs to be better
planned out before 2042, where we will see integer overflows on the serial
number.

Previous exit lists to merge are found using a glob that will break after 23:59
on 31st Dec 2999.

# Bugs

Known bugs can be found in the Tor Project Trac for:

-   [Exit Scanner](https://trac.torproject.org/projects/tor/query?status=!closed&component=Metrics%2FExit%20Scanner)
-   [Tor Check](https://trac.torproject.org/projects/tor/query?status=!closed&component=Applications%2FTor%20Check)

For bugs relating to exitmap, they are found on the GitHub project: <https://github.com/NullHypothesis/exitmap/issues>

New bug reports should be filed in the appropriate tracker and component.