Exit Scanner, TorDNSEL and Tor Check Operations
While the three services described in this document could be implemented as discrete components, they currently have tight coupling which means they must all be deployed on the same host.
+------------+ | Exit +---------------------------------------+ | Scanner |-----------------------------+ | | +----------> DNS Zone File | | +--------+---+ + | | launches| ^ | | | v |gets results v v | +-------+ +---+--------+ +----+-----+ Bulk | | Tor +^------+ exitmap | | TSA | Exit | | Client+------v+ | | Managed | List | +-------+ +------------+ | BIND | | using ipscan module +----------+ | | v +--------+ +-----------+ Exit List | Tor +<---------+ Tor | + | Client | | Check +<----------+ | +--------->+ | +--------+ +-----+-----+ ^ | +-----+-----+ | Apache | | Reverse | | Proxy | +-----------+
The exit scanner performs active measurement of Tor exit relays in order to determine the IP addresses that are used for exit connections. The active measurement uses an exitmap module, which is wrapped in a script to produce output formatted as an Exit List.
- https://check.torproject.org/exit-addresses - Latest exit list
- https://check.torproject.org/torbulkexitlist - Latest bulk exit list
Previous scan results are kept in the latest exit list for up to 48 hours since they were last seen. Scans will run every 40 minutes, or continuously if scans take longer than 40 minutes to complete.
There is currently no process that will clean up old exit lists from the exit list server.
TorDNSEL is a DNS list service that behaves in a similar way to Domain Name System-based Blackhole Lists. IP addresses will give positive results in the event that an address has been found to be used by an exit relay in a recent scan.
As with the exit lists, IP addresses will be removed after not being seen for 48 hours.
A future update to this script may include information from the consensus, but currently a postitive match must have been found from the exit scanner for an IP address to be included.
Tor Check is a website that can be used to determine if a browser is using the Tor network for queries. It will also check the User-Agent to determine if a user is using Tor Browser. It is accessed via HTTPS at https://check.torproject.org/.
There are also JSON APIs:
- https://check.torproject.org/api/ip - Check IP and if it's a Tor IP
- https://check.torproject.org/api/bulk - Bulk exit list exporter
The primary contact for this service is the Metrics Team <email@example.com>. For urgent queries, contact karsten or gaba in IRC.
The underlying infrastructure for the exit scanner, TorDNSEL and Tor Check services is provided by the
Tor Sysadmin Team (TSA). All services run on one virtual machine with the hostname
The exitmap module used is a custom module named ipscan.
The exitscan wrapper produces a BIND zone file
/srv/tordnsel.torproject.org/dnsel.torproject.org. The wrapper also
automatically reloads this zone file in the TPA managed BIND server whenever it
has been updated. The name
dnsel.torproject.org is delegated to the
host with NS records.
weasel had ideas about how we could run this as a hidden master, transfer the zone to the main domain servers and also have them sign the zone with DNSSEC. This is an idea for the future.
The wrapper script is also responsible for writing out the zone file to be used by the TorDNSEL service and triggering a reload of the zone.
The sources for Tor Check are available in our git: https://gitweb.torproject.org/check.git.
The initial deployment procedure is split into 2 parts:
- Installing and starting the services
There are no manual steps required to load state, and backups do not need to be performed for the host running this service.
Everything can be configured from scratch with only the Ansible playbook.
Development/testing in AWS
For development or testing in AWS, a CloudFormation template is available named
From the CloudFormation portal, select your stack and view the outputs. You will find here the public IP address for the EC2 instance that has been created. Add this instance to ansible/dev in your local copy of metrics-cloud.git under "[exit-scanners]".
You can now setup the machine with Ansible by running:
ansible-playbook -i dev exit-scanners-aws.yml
Note that the AWS AMI used has passwordless sudo, so no password need be given.
Fresh machine from TSA
Add the host name of the new instance to ansible/production in your local copy of metrics-cloud.git under "[exit-scanners]" and commit the change.
You can now setup the machine with Ansible by running:
ansible-playbook -i production -K exit-scanners.yml
The upstream sources for the applications that make up this service do not have managed releases which makes this difficult.
To fix a bug in the exit scanner wrapper script, fix this in the metrics-cloud repository and re-run the deployment playbook.
- Can we upgrade exitmap sensibly?
- Can we upgrade Tor Check sensibly?
The exit scanner wrapper logs output to
These logs include every measurement ever performed, and this might fill up the disk eventually. See #33475.
CollecTor gives a warning in the case that the exit lists are too old.
There are no Nagios checks for this service beyond the standard TPA systems checks (and I didn't actually check if we have those yet).
Service needs restart
The service can be restarted, as the tordnsel user, with:
XDG_RUNTIME_DIR=/run/user/`id -u` systemctl --user restart exitscanner.service
The exit scanner service does not need to maintain any state between runs. It's nice if it can in order to cope with a relay that happened to be down at the time we tried to measure it but in the event of a failure it is perfectly acceptable to throw away the old box and provision a new one. Follow the initial deployment instructions above.
The exit scanner service produces exit lists according to the TorDNSEL exit list format.
The exit scanner service does not support IPv6.
Serial numbers for the DNS zones are based on the date. This needs to be better planned out before 2042, where we will see integer overflows on the serial number.
Previous exit lists to merge are found using a glob that will break after 23:59 on 31st Dec 2999.
Known bugs can be found in the Tor Project Trac for:
For bugs relating to exitmap, they are found on the GitHub project: https://github.com/NullHypothesis/exitmap/issues
New bug reports should be filed in the appropriate tracker and component.