This page holds all information about rewriting [https://weather.torproject.org/ Tor Weather] using [https://onionoo.torproject.org/ Onionoo] as data backend. It originates from [https://lists.torproject.org/pipermail/tor-dev/2014-January/006039.html this email thread].
This page holds all information about rewriting [Tor Weather](https://weather.torproject.org/) using [Onionoo](https://onionoo.torproject.org/) as data backend. It originates from [this email thread](https://lists.torproject.org/pipermail/tor-dev/2014-January/006039.html).
This page is temporary. The purpose is to coordinate our efforts and avoid duplicating work. As soon as there's a single code repository and a deployed version, this page may go away.
This page is temporary. The purpose is to coordinate our efforts and avoid duplicating work. As soon as there's a single code repository and a deployed version, this page may go away.
Sreenatha was accepted as a GSoC 2014 student. He will be working on the weather re-write and should be contacted via tor-dev or IRC for any efforts over this summer. meejah (also on IRC and tor-dev) is his primary mentor as well as Karsten Loesing. See meejah's github repository (linked below) for the code integration point for this project ("next" branch).
Sreenatha was accepted as a GSoC 2014 student. He will be working on the weather re-write and should be contacted via tor-dev or IRC for any efforts over this summer. meejah (also on IRC and tor-dev) is his primary mentor as well as Karsten Loesing. See meejah's github repository (linked below) for the code integration point for this project ("next" branch).
=== Contributors ===
### Contributors
* Karsten Loesing <karsten@torproject.org> karsten on IRC and Trac (brainstorm design, do basic code reviews, answer questions about Onionoo interface)
* Karsten Loesing <karsten@torproject.org> karsten on IRC and Trac (brainstorm design, do basic code reviews, answer questions about Onionoo interface)
* Abhiram Chintangal <abhiram.chintangal@gmail.com> feverDream on IRC and Trac (for now, implementing the design)
* Abhiram Chintangal <abhiram.chintangal@gmail.com> feverDream on IRC and Trac (for now, implementing the design)
* Norbert Kurz <tor@nufuk.com> nufuk on IRC (SourceCode for the backend, the daily-script [see design])
* Norbert Kurz <tor@nufuk.com> nufuk on IRC (SourceCode for the backend, the daily-script [see design])
...
@@ -20,25 +20,25 @@ Sreenatha was accepted as a GSoC 2014 student. He will be working on the weather
...
@@ -20,25 +20,25 @@ Sreenatha was accepted as a GSoC 2014 student. He will be working on the weather
(Add yourself here if you want to help out, and describe what you're planning to contribute.)
(Add yourself here if you want to help out, and describe what you're planning to contribute.)
=== Meetings ===
### Meetings
Next meeting: Wednesday, June 25, 2014 at 17:00 UTC in #tor-dev on OFTC. (If we forget to update this, the meeting happens every Wednesday at 17:00 UTC in #tor-dev on OFTC.)
Next meeting: Wednesday, June 25, 2014 at 17:00 UTC in #tor-dev on OFTC. (If we forget to update this, the meeting happens every Wednesday at 17:00 UTC in #tor-dev on OFTC.)
Minutes:
Minutes:
* [https://lists.torproject.org/pipermail/tor-dev/2014-March/006400.html March 5th 2014]
* [http://meetbot.debian.net/tor-dev/2014/tor-dev.2014-06-04-17.13.html June 4, 2014] also [http://meetbot.debian.net/tor-dev/2014/tor-dev.2014-06-04-17.13.log.html log from meeting]
*[June 4, 2014](http://meetbot.debian.net/tor-dev/2014/tor-dev.2014-06-04-17.13.html) also [log from meeting](http://meetbot.debian.net/tor-dev/2014/tor-dev.2014-06-04-17.13.log.html)
* [http://meetbot.debian.net/tor-dev/2014/tor-dev.2014-06-11-16.58.html June 11, 2014] also [http://meetbot.debian.net/tor-dev/2014/tor-dev.2014-06-11-16.58.log.html log from meeting]
*[June 11, 2014](http://meetbot.debian.net/tor-dev/2014/tor-dev.2014-06-11-16.58.html) also [log from meeting](http://meetbot.debian.net/tor-dev/2014/tor-dev.2014-06-11-16.58.log.html)
* [http://meetbot.debian.net/tor-dev/2014/tor-dev.2014-06-25-16.59.html June 25, 2014] also [http://meetbot.debian.net/tor-dev/2014/tor-dev.2014-06-25-16.59.log.html log from meeting]
*[June 25, 2014](http://meetbot.debian.net/tor-dev/2014/tor-dev.2014-06-25-16.59.html) also [log from meeting](http://meetbot.debian.net/tor-dev/2014/tor-dev.2014-06-25-16.59.log.html)
* [http://meetbot.debian.net/tor-dev/2014/tor-dev.2014-07-02-16.59.html July 2, 2014] also [http://meetbot.debian.net/tor-dev/2014/tor-dev.2014-07-02-16.59.log.html log from meeting]
*[July 2, 2014](http://meetbot.debian.net/tor-dev/2014/tor-dev.2014-07-02-16.59.html) also [log from meeting](http://meetbot.debian.net/tor-dev/2014/tor-dev.2014-07-02-16.59.log.html)
=== Integration Plan for Summer 2014 ===
### Integration Plan for Summer 2014
For the duration of the GSoC project, this is how things will work:
For the duration of the GSoC project, this is how things will work:
* github.com/meejah/tor-weather "next" is "the new code"
* github.com/meejah/tor-weather "next" is "the new code"
* karsten will pull this into his official-repo "next" branch
* karsten will pull this into his official-repo "next" branch
* only meejah will commit to meejah/tor-weather
* only meejah will commit to meejah/tor-weather
* all contributions will be via pull-requests:
* all contributions will be via pull-requests:
* well-documented [https://help.github.com/articles/using-pull-requests at github.com] using the "fork and pull" model
* well-documented [at github.com](https://help.github.com/articles/using-pull-requests) using the "fork and pull" model
* make a fork of meejah/tor-weather
* make a fork of meejah/tor-weather
* add contributions (either on a branch or not, your preference)
* add contributions (either on a branch or not, your preference)
* push them to your fork
* push them to your fork
...
@@ -47,25 +47,25 @@ For the duration of the GSoC project, this is how things will work:
...
@@ -47,25 +47,25 @@ For the duration of the GSoC project, this is how things will work:
* meejah merges to "next" branch
* meejah merges to "next" branch
* karsen pulls into official "next" branch
* karsen pulls into official "next" branch
=== Code repositories ===
### Code repositories
* [https://github.com/meejah/tor-weather github.com/meejah/tor-weather]: the re-write will get code merged through this repository, from where Karsten will pull it to the "real" (torproject.org) one.
*[github.com/meejah/tor-weather](https://github.com/meejah/tor-weather): the re-write will get code merged through this repository, from where Karsten will pull it to the "real" (torproject.org) one.
* [https://gitweb.torproject.org/weather.git weather.git/master]: this is the official repository, and it's what is currently deployed on bahri; there are some local changes, but this is the branch that is closest to what's deployed.
*[weather.git/master](https://gitweb.torproject.org/weather.git): this is the official repository, and it's what is currently deployed on bahri; there are some local changes, but this is the branch that is closest to what's deployed.
* Damian's [https://gitweb.torproject.org/user/atagar/weather.git/ Weather Repo]: TorCtl has been replaced by Stem, needs more testing; this is not deployed anywhere, but it seems like the best thing to start with.
* Damian's [Weather Repo](https://gitweb.torproject.org/user/atagar/weather.git/): TorCtl has been replaced by Stem, needs more testing; this is not deployed anywhere, but it seems like the best thing to start with.
* [https://gitweb.torproject.org/user/karsten/weather.git/shortlog/refs/heads/vagrant Karsten's "weather-vagrant" branch]: Abhiram started working on a Vagrantfile and puppet provisioning; this branch is based on Damian's; we took Abhiram's branch two weeks ago and tweaked it towards having an almost working Vagrant setup; we shouldn't base other branches on this one, because we're planning to squash things once the Vagrant stuff works.
*[Karsten's "weather-vagrant" branch](https://gitweb.torproject.org/user/karsten/weather.git/shortlog/refs/heads/vagrant): Abhiram started working on a Vagrantfile and puppet provisioning; this branch is based on Damian's; we took Abhiram's branch two weeks ago and tweaked it towards having an almost working Vagrant setup; we shouldn't base other branches on this one, because we're planning to squash things once the Vagrant stuff works.
**Wrap-Up:** We should merge complete features into a branch that is based on Damian's, ideally the "weather-next" branch.
**Wrap-Up:** We should merge complete features into a branch that is based on Damian's, ideally the "weather-next" branch.
(Add links to code repositories here, even those that contain only a subset of the new functionality.)
(Add links to code repositories here, even those that contain only a subset of the new functionality.)
=== Requirements ===
### Requirements
Here's a quick description of what the rewritten Weather should do. (Please edit to make this more accurate!)
Here's a quick description of what the rewritten Weather should do. (Please edit to make this more accurate!)
* Maintain a list of subscriptions, consisting of an email address, a password, a relay identity fingerprint, how soon the user wants to be notified of problems, when it was last notified, etc.
* Maintain a list of subscriptions, consisting of an email address, a password, a relay identity fingerprint, how soon the user wants to be notified of problems, when it was last notified, etc.
* Allow users to create, read, update, and delete subscriptions via a web interface. All these operations should have the usual security features like email address verification, password login, etc.
* Allow users to create, read, update, and delete subscriptions via a web interface. All these operations should have the usual security features like email address verification, password login, etc.
* Allow users to search for relays to subscribe for by relay IP address, relay identity fingerprint, or relay nickname. This search can be done with help of Onionoo's search feature, or by simply adding a link to [https://atlas.torproject.org/ Atlas] or [https://globe.torproject.org/ Globe].
* Allow users to search for relays to subscribe for by relay IP address, relay identity fingerprint, or relay nickname. This search can be done with help of Onionoo's search feature, or by simply adding a link to [Atlas](https://atlas.torproject.org/) or [Globe](https://globe.torproject.org/).
* Once per hour, download a list from Onionoo that contains relays that have been running in the last week. Check if there are any relays that have been offline for long enough to notify a subscribed user. Send out emails.
* Once per hour, download a list from Onionoo that contains relays that have been running in the last week. Check if there are any relays that have been offline for long enough to notify a subscribed user. Send out emails.
...
@@ -73,12 +73,12 @@ Here's a quick description of what the rewritten Weather should do. (Please edi
...
@@ -73,12 +73,12 @@ Here's a quick description of what the rewritten Weather should do. (Please edi
* Once per day, send out welcome emails to relays that are new.
* Once per day, send out welcome emails to relays that are new.
=== Starting points ===
### Starting points
* https://trac.torproject.org/projects/tor/ticket/9889#comment:2 has some URLs and pseudocode for using Onionoo's bandwidth documents to determine whether a relay operator should have a t-shirt.
* https://trac.torproject.org/projects/tor/ticket/9889#comment:2 has some URLs and pseudocode for using Onionoo's bandwidth documents to determine whether a relay operator should have a t-shirt.
* https://gitweb.torproject.org/compass.git/blob/HEAD:/compass.py fetches Onionoo's details documents, filters and groups relays, and generally provides the output on https://compass.torproject.org/.
* https://gitweb.torproject.org/compass.git/blob/HEAD:/compass.py fetches Onionoo's details documents, filters and groups relays, and generally provides the output on https://compass.torproject.org/.
=== Tickets ===
### Tickets
Tickets that are specifically related to the Weather rewrite, including closed tickets:
Tickets that are specifically related to the Weather rewrite, including closed tickets:
(Add thoughts that need discussion, and respond to thoughts. Add your initials to statements here, e.g. "-KL".)
(Add thoughts that need discussion, and respond to thoughts. Add your initials to statements here, e.g. "-KL".)
* Q: What technologies should we use/avoid?
* Q: What technologies should we use/avoid?
...
@@ -107,8 +107,8 @@ Tickets that have been opened against the current Weather but that might still b
...
@@ -107,8 +107,8 @@ Tickets that have been opened against the current Weather but that might still b
* Q: What are the different parts of the application?
* Q: What are the different parts of the application?
* A: After spending some time with the current weather code-base, the registration,login and the normal usual suspects of user management paired with CRUD of subscription seems like independent part of the application. The data models and Django application skeleton is already there. Do you guys have any ideas about this portion, let me know. [-AC]
* A: After spending some time with the current weather code-base, the registration,login and the normal usual suspects of user management paired with CRUD of subscription seems like independent part of the application. The data models and Django application skeleton is already there. Do you guys have any ideas about this portion, let me know. [-AC]
* The design suggests PHP for the web interface. Would it also be an option to re-use (parts of) the existing web interface part of Weather? [-KL]
* The design suggests PHP for the web interface. Would it also be an option to re-use (parts of) the existing web interface part of Weather? [-KL]
* Both hourly and daily scripts don't really have to archive the JSON files they downloaded from Onionoo. They may of course write them to disk for debugging purposes, but neither script needs to access previous file versions to run. So, just name the files details.json and bandwidth.json (or the like) and overwrite them in every execution. [-KL]
* Both hourly and daily scripts don't really have to archive the JSON files they downloaded from Onionoo. They may of course write them to disk for debugging purposes, but neither script needs to access previous file versions to run. So, just name the files details.json and bandwidth.json (or the like) and overwrite them in every execution. [-KL]
* The hourly script needs a step 5) where it writes to the database that it has sent an email notification, and what type of notification that was (node is down, version is out of date, low bandwidth capacity, eligible for t-shirt; see subscription page of current Weather). The goal is to send out each notification type only once until the situation has been resolved. [-KL]
* The hourly script needs a step 5) where it writes to the database that it has sent an email notification, and what type of notification that was (node is down, version is out of date, low bandwidth capacity, eligible for t-shirt; see subscription page of current Weather). The goal is to send out each notification type only once until the situation has been resolved. [-KL]
...
@@ -117,8 +117,8 @@ Tickets that have been opened against the current Weather but that might still b
...
@@ -117,8 +117,8 @@ Tickets that have been opened against the current Weather but that might still b
* The daily script needs to check subscriptions when a relay becomes eligible for receiving a t-shirt. Some relay operators don't put useful contact information into their torrc file, but prefer to subscribe to Weather instead. [-KL]
* The daily script needs to check subscriptions when a relay becomes eligible for receiving a t-shirt. Some relay operators don't put useful contact information into their torrc file, but prefer to subscribe to Weather instead. [-KL]
* The daily script also needs to write to the database when it has sent email about a relay being around long enough for the operator to have a t-shirt, so that it only sends out that email once in the relay's lifetime. [-KL]
* The daily script also needs to write to the database when it has sent email about a relay being around long enough for the operator to have a t-shirt, so that it only sends out that email once in the relay's lifetime. [-KL]
* Design Weather V0.2 from 2014-01-13 ([PNG](https://trac.torproject.org/projects/tor/attachment/wiki/doc/weather-in-2014/Weather_Overall_V0.2.png), [DIA](https://trac.torproject.org/projects/tor/attachment/wiki/doc/weather-in-2014/Weather_Overall_V0.2.dia)) [-NK]
* UML Diagram from current Weather as of 2014-01-16 generated by [https://github.com/django-extensions/django-extensions django-extensions] ([https://trac.torproject.org/projects/tor/attachment/wiki/doc/weather-in-2014/mymodels.png PNG]). [-AC]
* UML Diagram from current Weather as of 2014-01-16 generated by [django-extensions](https://github.com/django-extensions/django-extensions)([PNG](https://trac.torproject.org/projects/tor/attachment/wiki/doc/weather-in-2014/mymodels.png)). [-AC]
* Design Document of current Weather as of 2014-01-16 ([https://trac.torproject.org/projects/tor/attachment/wiki/doc/weather-in-2014/design.html html]), for a good overview of the design process and working of the current weather application. [-AC]
* Design Document of current Weather as of 2014-01-16 ([html](https://trac.torproject.org/projects/tor/attachment/wiki/doc/weather-in-2014/design.html)), for a good overview of the design process and working of the current weather application. [-AC]
