This is an old version of this page. You can view the most recent version or browse the history.

Onionoo

☕ CollecTor is a Java application. If you have not yet, refer to the Java survival guide which contains information relevant to all Metrics Java projects.

Service Overview

Onionoo is a web-based protocol to learn about currently running Tor relays and bridges. Onionoo itself was not designed as a service for human beings—at least not directly. Onionoo provides the data for other applications and websites which in turn present Tor network status information to humans.

The primary interface to Onionoo is via HTTPS using the round-robin DNS name onionoo.torproject.org

The underlying infrastructure for the Onionoo service is provided by the Tor Sysadmin Team (TSA). There are a number of HTTP caches (onionoo-frontend-*.torproject.org, currently running varnish) that sit in front of a number of backends (onionoo-backend-*.torproject.org, running the two Java components: the updater and web server).

The frontends are entirely managed by TSA. The frontends communicate with the backends via IPsec tunnels managed by TSA.

The backend hosts are managed by TSA with the Onionoo services being managed by Network Health Team. The Onionoo services get their data from the CollecTor service.

The backends are redundant and the service can survive outages, in those conditions:

shorter than 72 hours: backends can self-heal
longer partial outage: as long as a backend remains, the other backends can be restored from the remaining backend, although that is a manual process.
longer total outage: if all backends go down for more than 72h, data can still be recovered from collector, but that's another, different manual process that still has to be implemented

Note that data is recovered from CollecTor, which has similar self-healing systems that cover 72 hours.

The Disaster Recovery section details how to recover from those situations.

Service Architecture

The Onionoo service consists of two parts: the hourly updater and the web server.

Both parts run on each backend host and the parts run with privilege seperation.

Hourly Updater

The hourly updater is contained in the JAR file, which is built from the sources with:

ant jar

The JAR file is also included in the tarballs made available with releases in the generated/dist/ folder. The filename will look like onionoo-{protocol version}-{software version}.jar and on the backend host should be found in /srv/onionoo.torproject.org/onionoo/.

Web Server

The web server is contained in the WAR file, which is built from the sources with:

ant war

The WAR file is also included in the tarballs made available with releases in the generated/dist/ folder. The filename will look like onionoo-{protocol version}-{software version}.jar and on the backend host should be found in /srv/onionoo.torproject.org/onionoo/.

Sources

Onionoo releases are available from dist.torproject.org with the source code available from Tor Project git.

Setting up the service

Initial deployment

The initial deployment procedure is split into 3 parts:

System setup
Importing history
Installing and starting the service

System setup

If you have a machine from TSA, i.e. this is one of the "official" Onionoo instances, these steps are already taken care of by their provisioning setup in Puppet and you can skip to the next section.

Onionoo uses filesystem storage and creates a lot of small files. It is best to use XFS rather than ext4 for the Onionoo data directory for this reason. To replicate a production setup, or to deploy a new production instance, mount an XFS filesystem at /srv.

You will need to install a JRE. Use AdoptOpenJDK's JRE 8 build.

Create two users for the service, named onionoo and onionoo-unpriv.

Importing history

If you are setting up an instance for development and testing, you might not need the entire history of the Tor network. If that's the case you can skip this section.

Copy the state and out directories from another Onionoo backend to /srv/onionoo.torproject.org/onionoo/ on the new backend.

Installing and starting the service

Start by downloading the latest release from: https://dist.torproject.org/onionoo/VERSION/onionoo-VERSION.tar.gz.

Unpack generated/dist/onionoo-VERSION.{jar,war} from the tarball to /srv/onionoo.torproject.org/onionoo/.

The out directory must exist and be populated before you start the web service, otherwise it will exit with an error. If you have not copied this from another backend, you can prime this one with:

cd /srv/onionoo.torproject.org/onionoo
java -Xmx4g -Dsun.net.client.defaultConnectTimeout=60000 -Dsun.net.client.defaultReadTimeout=60000 -DLOGBASE=logs -cp onionoo-{{ onionoo_version }}.jar org.torproject.metrics.onionoo.cron.Main --single-run

Then you can start the updater:

# As the onionoo user
java -Xmx4g -Dsun.net.client.defaultConnectTimeout=60000 \
          -Dsun.net.client.defaultReadTimeout=60000 -DLOGBASE=logs \
          -cp onionoo-VERSION.jar \
          org.torproject.metrics.onionoo.cron.Main

and web service:

# As the onionoo-unpriv user
java -Xmx2g -DLOGBASE=web-logs -jar onionoo-VERSION.war

Upgrade

Start by downloading the latest release from: https://dist.torproject.org/onionoo/VERSION/onionoo-VERSION.tar.gz.

Unpack generated/dist/onionoo-VERSION.{jar,war} from the tarball to /srv/onionoo.torproject.org/onionoo/.

Run tail -f logs/metrics.log to ensure that the updater is not currently running. If it is currently running, best to let it finish.

Stop the currently running updater with killall java (one day we might have a less heavy handed approach, but this works). The web and updater processes run under different users so each will only be able to kill their own, you'll probably get a permission denied error for the other's process which you can safely ignore.

Then you can restart the updater:

# As the onionoo user
java -Xmx4g -Dsun.net.client.defaultConnectTimeout=60000 \
          -Dsun.net.client.defaultReadTimeout=60000 -DLOGBASE=logs \
          -cp onionoo-VERSION.jar \
          org.torproject.metrics.onionoo.cron.Main

and web service:

# As the onionoo-unpriv user
killall java
java -Xmx2g -DLOGBASE=web-logs -jar onionoo-VERSION.war

To start the new version on the next reboot, update the version number for the JAR in:

~onionoo/bin/start-onionoo
~onionoo/bin/start-onionoo-unpriv

Diagnostics and Monitoring

Logs

Logs for the hourly updater can be found in /srv/onionoo.torproject.org/logs/, and for the web server in /srv/onionoo.torproject.org/web-logs/.

Web Server Direct Connection

It may be necessary to determine if an issue is caused by a frontend or a backend.

To connect directly to an Onionoo backend's web server, you will need to use SSH port forwarding. This is because the web server is not available for incoming connections on its Internet address. For example:

ssh -L8039:localhost:8080 onionoo-backend-01.torproject.org

You'll then be able to connect to localhost:8039 in your web browser.

Nagios Plugin

Onionoo is monitored by the TSA Nagios instance using the tor-check-onionoo plugin. The frontends and backends are each monitored. The check is simple: it looks for the last time that bridge and relay information was updated and alerts if they are too old.

TODO: Check where these alerts go, if anywhere.

TODO: Make this into a Prometheus exporter.

Disaster recovery

Single backend data corruption, no hardware failure

sudo -u onionoo -i bash -c 'systemctl --user stop onionoo'
sudo -u onionoo-unpriv -i bash -c 'systemctl --user stop onionoo-web'
rm -rf /srv/onionoo.torproject.org/onionoo/home/{.,}\*
rm -rf /srv/onionoo.torproject.org/onionoo/home-unpriv/{.,}\*
rm -rf /srv/onionoo.torproject.org/onionoo/onionoo/{.,}\*

Then pretend you are deploying a new backend from the instructions above.

Single backend failure, hardware failure

In the event of a single backend failure, ask TSA to trash it and make a new one. Once Puppet has configured their side of it, pretend you are deploying a new backend from the instructions above.

Total loss

In the event of a total loss, ask TSA to trash all the backends and make new ones. Once Puppet has configured one host, restore the state and out directories from the latest good backup. It may be necessary to refer to the logs to work out when the latest good backup might be, which should also be backed up. Once state and out are in place, pretend you are deploying a new backend from the instructions above.

Total loss including all backups

In the event that the backups have also been lost, it will not be possible to restore history. The data does exist in CollecTor to do this, but there is no code that actually does it.

If no out directory is present on the instance when the Ansible playbook is run to install and start the service, it will perform an initial single run of the updater to bootstrap. This will be where history starts.

Try to avoid this happening.

Links

The Onionoo service implements the Onionoo protocol.