Weather in 2014
This page is temporary. The purpose is to coordinate our efforts and avoid duplicating work. As soon as there's a single code repository and a deployed version, this page may go away.
Sreenatha was accepted as a GSoC 2014 student. He will be working on the weather re-write and should be contacted via tor-dev or IRC for any efforts over this summer. meejah (also on IRC and tor-dev) is his primary mentor as well as Karsten Loesing. See meejah's github repository (linked below) for the code integration point for this project ("next" branch).
- Karsten Loesing email@example.com karsten on IRC and Trac (brainstorm design, do basic code reviews, answer questions about Onionoo interface)
- Abhiram Chintangal firstname.lastname@example.org feverDream on IRC and Trac (for now, implementing the design)
- Norbert Kurz email@example.com nufuk on IRC (SourceCode for the backend, the daily-script [see design])
- Andrew Lewman firstname.lastname@example.org phobos on Trac (comment on infrastructure requirements)
- Oliver Baumann email@example.com baumanno on IRC and Trac (Frontend & UI, backend hacking)
- Roger Dingledine firstname.lastname@example.org armadev on IRC and arma on Trac (comment on requirements)
- Sreenatha Bhatlapenumarthi email@example.com lucyd on IRC and sreenatha on Trac (generic, mostly backend)
- Jonathan D. Baker firstname.lastname@example.org jondbaker on IRC and Trac (frontend + backend)
- meejah email@example.com meejah on IRC and Trac (GSoC primary mentor)
- Lukas Erlacher firstname.lastname@example.org luketheduke on IRC, honestly on Trac (Onion-Py OnionOO API interface)
(Add yourself here if you want to help out, and describe what you're planning to contribute.)
Next meeting: Wednesday, June 25, 2014 at 17:00 UTC in #tor-dev on OFTC. (If we forget to update this, the meeting happens every Wednesday at 17:00 UTC in #tor-dev on OFTC.)
- March 5th 2014
- June 4, 2014 also log from meeting
- June 11, 2014 also log from meeting
- June 25, 2014 also log from meeting
- July 2, 2014 also log from meeting
Integration Plan for Summer 2014
For the duration of the GSoC project, this is how things will work:
- github.com/meejah/tor-weather "next" is "the new code"
- karsten will pull this into his official-repo "next" branch
- only meejah will commit to meejah/tor-weather
- all contributions will be via pull-requests:
- well-documented at github.com using the "fork and pull" model
- make a fork of meejah/tor-weather
- add contributions (either on a branch or not, your preference)
- push them to your fork
- create pull-request via adding a link to your code to the respective Trac-ticket
- meejah reviews; any fixups happen in your fork (and repeat from "push to your fork")
- meejah merges to "next" branch
- karsen pulls into official "next" branch
- github.com/meejah/tor-weather: the re-write will get code merged through this repository, from where Karsten will pull it to the "real" (torproject.org) one.
- weather.git/master: this is the official repository, and it's what is currently deployed on bahri; there are some local changes, but this is the branch that is closest to what's deployed.
- Damian's Weather Repo: TorCtl has been replaced by Stem, needs more testing; this is not deployed anywhere, but it seems like the best thing to start with.
- Karsten's "weather-next" branch
- Karsten's "weather-vagrant" branch: Abhiram started working on a Vagrantfile and puppet provisioning; this branch is based on Damian's; we took Abhiram's branch two weeks ago and tweaked it towards having an almost working Vagrant setup; we shouldn't base other branches on this one, because we're planning to squash things once the Vagrant stuff works.
Wrap-Up: We should merge complete features into a branch that is based on Damian's, ideally the "weather-next" branch.
(Add links to code repositories here, even those that contain only a subset of the new functionality.)
Here's a quick description of what the rewritten Weather should do. (Please edit to make this more accurate!)
Maintain a list of subscriptions, consisting of an email address, a password, a relay identity fingerprint, how soon the user wants to be notified of problems, when it was last notified, etc.
Allow users to create, read, update, and delete subscriptions via a web interface. All these operations should have the usual security features like email address verification, password login, etc.
Allow users to search for relays to subscribe for by relay IP address, relay identity fingerprint, or relay nickname. This search can be done with help of Onionoo's search feature, or by simply adding a link to Atlas or Globe.
Once per hour, download a list from Onionoo that contains relays that have been running in the last week. Check if there are any relays that have been offline for long enough to notify a subscribed user. Send out emails.
Once per day, download bandwidth histories of relays from Onionoo and check whether a relay has been running long enough and fast enough that the operator should be offered a t-shirt. Send out emails, regardless of subscriptions, and ask if operators would want one.
Once per day, send out welcome emails to relays that are new.
https://trac.torproject.org/projects/tor/ticket/9889#comment:2 has some URLs and pseudocode for using Onionoo's bandwidth documents to determine whether a relay operator should have a t-shirt.
https://gitweb.torproject.org/compass.git/blob/HEAD:/compass.py fetches Onionoo's details documents, filters and groups relays, and generally provides the output on https://compass.torproject.org/.
Tickets that are specifically related to the Weather rewrite, including closed tickets:
Tickets that have been opened against the current Weather but that might still be relevant for the rewrite:
(Add thoughts that need discussion, and respond to thoughts. Add your initials to statements here, e.g. "-KL".)
Q: What technologies should we use/avoid?
- We're not running PHP on our infrastructure. It's fraught with security issues. We're not running MySQL due to current licensing questions and corporate ownership. Python, ruby, perl, text files, postgresql seem like fine solutions. [-AL]
Q: Would it also be an option to re-use parts of the existing weather?
- A: In my opinion, doing so will reduce some of the workload and it also makes it the choice of choosing the language and framework clearer. [-AC]
- Agreed. If we can re-use code, great! [-KL]
- Consensus as of January 22 is that we should keep the Python/Django parts of current Weather, rather than re-implementing them. [-KL]
Q:Should we stick with Stem or move to Onionoo?
- A: It seems convenient to use the onionoo's restful interface. Having said that most of the code that is used to interact with tor via stem is already there. [-AC]
- Agreed. Though special focus should be on maintaining Weather in the future, and that means less code is better, even if that means having to change more code now. [-KL]
- I too think so, It leaves us one less to thing to worry about in the future. [-AC]
Q: What are the different parts of the application?
- A: After spending some time with the current weather code-base, the registration,login and the normal usual suspects of user management paired with CRUD of subscription seems like independent part of the application. The data models and Django application skeleton is already there. Do you guys have any ideas about this portion, let me know. [-AC]
- The design suggests PHP for the web interface. Would it also be an option to re-use (parts of) the existing web interface part of Weather? [-KL]
- Both hourly and daily scripts don't really have to archive the JSON files they downloaded from Onionoo. They may of course write them to disk for debugging purposes, but neither script needs to access previous file versions to run. So, just name the files details.json and bandwidth.json (or the like) and overwrite them in every execution. [-KL]
- The hourly script needs a step 5) where it writes to the database that it has sent an email notification, and what type of notification that was (node is down, version is out of date, low bandwidth capacity, eligible for t-shirt; see subscription page of current Weather). The goal is to send out each notification type only once until the situation has been resolved. [-KL]
- Onionoo will soon have a field
recommended_versionwhich is either True or False, so that Weather can easily check when a relay runs an outdated Tor version. [-KL] Added on January 13. [-KL]
- The hourly script also needs to write to the database when a relay that has been offline (or that ran an outdated version, or that was slow) is online (up-to-date/fast) again, so that when it becomes offline (outdated/slow) again, Weather can send another notification email. [-KL]
- The daily script needs to check subscriptions when a relay becomes eligible for receiving a t-shirt. Some relay operators don't put useful contact information into their torrc file, but prefer to subscribe to Weather instead. [-KL]
- The daily script also needs to write to the database when it has sent email about a relay being around long enough for the operator to have a t-shirt, so that it only sends out that email once in the relay's lifetime. [-KL]
Design Document of current Weather as of 2014-01-16 (html), for a good overview of the design process and working of the current weather application. [-AC]