Tor-weather is a web service that alerts relay operators about issues with their relays. It runs on the host weather-01.

Tutorial

How-to

Pager playbook

Disaster recovery

Reference

Installation

the profile::weather class in the tor-puppet repository configures a systemd service to run the tor_weather app with gunicorn, as well as an apache site config to proxy requests to http://localhost:8000. tor-weather handles its own database schema creation, but database and database user creation are still manual.

add the profile::weather class to a node in puppet, then follow the instructions below to configure and deploy the application.

Creating the postgres database

First, follow the postgresql installation howto.

Run sudo -u postgres psql and enter the following commands. Make sure you generate a secure password for the torweather user. The password must be url safe (ASCII alphanumeric characters, -, _, ~) since we'll be using the password in a URI later.

CREATE DATABASE torweather;
CREATE USER torweather;
alter user torweather password '<password>';
GRANT ALL ON DATABASE torweather TO torweather;

Preparing a release

because tor-weather is managed using poetry, there are a few steps necessary to prepare a release before deploying:

clone the tor-weather repo locally
export the dependencies using poetry: poetry export --output requirements.txt
note which dependencies are installable by apt using this list of packages in debian
check out the latest release, and build a wheel: poetry build --format=wheel
scp the wheel and requirements files to the server: scp requirements.txt dist/tor_weather-*.whl weather-01.torproject.org:/home/weather/

Installing on the server

deploy the role::weather Puppet class to the server
create a virtual environment: python3 -m venv tor-weather-venv and source it: . tor-weather-venv/bin/activate
install the remaining dependencies from requirements.txt: pip install -r requirements.txt
enable and start the systemd user service units: tor-weather.service tor-weather-celery.service tor-weather-celerybeat.service
the /home/weather/.tor-weather.env file configures the tor-weather application through environment variables. This file is managed by Puppet.

SMTP_HOST=localhost
SMTP_PORT=25
SMTP_USERNAME=weather@torproject.org
SMTP_PASSWORD=''

SQLALCHEMY_DATABASE_URI='postgresql+psycopg2://torweather:<database password>@localhost:5432/torweather'

BROKER_URL='amqp://torweather:<broker password>@localhost:5672'
API_URL='https://onionoo.torproject.org'
BASE_URL='https://weather.torproject.org'

ONIONOO_JOB_INTERVAL=15

# XXX: change this
# EMAIL_ENCRYPT_PASS is a 32 byte string that has been base64-encoded
EMAIL_ENCRYPT_PASS='Q0hBTkdFTUVDSEFOR0VNRUNIQU5HRU1FQ0hBTkdFTUU='

# XXX: change this
SECRET_KEY='secret'

SQLALCHEMY_TRACK_MODIFICATIONS=
CELERY_BIN=/home/weather/tor-weather-venv/bin/celery
CELERY_APP=tor_weather.celery.celery
CELERYD_NODES=worker1
CELERYD_LOG_FILE=logs/celery/%n%I.log
CELERYD_LOG_LEVEL=info
CELERYD_OPTS=
CELERYBEAT_LOG_FILE=logs/celery/beat.log
CELERYBEAT_LOG_LEVEL=info

Upgrades

activate the tor-weather virtualenv
install the latest tor-weather: pip install tor-weather --index-url https://gitlab.torproject.org/api/v4/projects/1550/packages/pypi/simple --upgrade
restart the service: sudo -u weather env XDG_RUNTIME_DIR=/run/user/$(id -u weather) systemctl --user restart tor-weather.service
restart the celery service: sudo -u weather env XDG_RUNTIME_DIR=/run/user/$(id -u weather) systemctl --user restart tor-weather-celery.service
restart the celery beat service: sudo -u weather env XDG_RUNTIME_DIR=/run/user/$(id -u weather) systemctl --user restart tor-weather-celerybeat.service

Migrating the database schema

After an upgrade or an initial deployment, you'll need to create or migrate the database schema. This script will activate the tor-weather virtual environment, export the tor-weather envvar settings, and then create/migrate the database schema. Note: the flask command might need to get updated dependent on the Python version running.

sudo -u weather bash
cd /home/weather
source tor-weather-venv/bin/activate
set -a
source .tor-weather.env
set +a
flask --app tor_weather.app db upgrade --directory /home/weather/tor-weather-venv/lib/python3.11/site-packages/tor_weather/migrations
exit

SLA

Design and architecture

Services

The tor-weather deployment consists of three main services:

apache: configured in puppet. proxies requests to http://localhost:8000
gunicorn: started by a systemd service file configured in puppet. runs with 5 workers (recommended by gunicorn docs: (2 * nproc) + 1), listens on localhost port 8000
postgres: a base postgres installation with a torweather user and database

Additionally, there are three services related to task scheduling:

rabbitmq: configured in puppet, a message broker (listening on localhost:5672)
celery: task queue, started by a systemd service file configured in puppet
celery beat: scheduler, started by a systemd service file configured in puppet

Storage

tor-weather is backed by a postgres database. the postgres database is configured in the /home/weather/.tor-weather.env file, using a sqlalchemy connection URI.

Queues

Onionoo Update Job

The tor-weather-celerybeat.service file triggers a job every 15 minutes to update tor-weather's onionoo metrics information.

Interfaces

Authentication

tor-weather handles its own user creation and authentication via the web interface.

Implementation

Related services

Postgres

Issues

Issues can be filed on the tor-weather issue tracker.

Maintainer

tor-weather is maintained by the network-health team.

Users

Monitoring and metrics

Tests

Logs

Logs are kept in <working directory>/logs. In the current deployment this is /home/weather/tor-weather/logs.

tor weather