Changes

Page history
Apply conversion script to all *.md files. authored by Alexander Hansen Færøy's avatar Alexander Hansen Færøy
Show whitespace changes
Inline Side-by-side
org/roadmaps/GetTor/design.md
View page @ a1a4b621
[[TOC]]
== Design ==
The easiest way to understand how !GetTor works is by enumerating the ## Design
The easiest way to understand how GetTor works is by enumerating the
different steps involved in the problem we want to solve. Consider the different steps involved in the problem we want to solve. Consider the
following situations: following situations:
...@@ -11,11 +11,11 @@ following situations: ...@@ -11,11 +11,11 @@ following situations:
3. Construct a response according to the information extracted. 3. Construct a response according to the information extracted.
4. Create an anti-flood mechanism that allows to blacklist specific users. 4. Create an anti-flood mechanism that allows to blacklist specific users.
5. Verify that the source address/user of the request is not permanently or temporarily blacklisted. 5. Verify that the source address/user of the request is not permanently or temporarily blacklisted.
6. Send back a reply with the links to download Tor Browser from some popular ''non-blocked'' cloud service. 6. Send back a reply with the links to download Tor Browser from some popular _non-blocked_ cloud service.
7. Keep track of the number of requests received by !GetTor. 7. Keep track of the number of requests received by GetTor.
8. Upload Tor Browser to popular ''non-blocked'' cloud services. 8. Upload Tor Browser to popular _non-blocked_ cloud services.
The current design of !GetTor consists of a series of modules, each one The current design of GetTor consists of a series of modules, each one
intended for a specific task. There are two big groups: the main modules, intended for a specific task. There are two big groups: the main modules,
and the service modules. The main modules are Core, and the service modules. The main modules are Core,
Blacklist and Database, aimed to cover the points 3), 4), 5) and 7). Blacklist and Database, aimed to cover the points 3), 4), 5) and 7).
...@@ -35,7 +35,7 @@ In both cases, the Core module increases the number of requests received ...@@ -35,7 +35,7 @@ In both cases, the Core module increases the number of requests received
in the database. A very simple diagram of the modules interaction looks in the database. A very simple diagram of the modules interaction looks
like this: like this:
{{{ ```
...@@ -55,54 +55,54 @@ like this: ...@@ -55,54 +55,54 @@ like this:
\---------->| Other Services | \---------->| Other Services |
---------------- ----------------
}}} ```
There is one of the points enumerated before that is not covered by the There is one of the points enumerated before that is not covered by the
previous modules, which is uploading Tor Browser to popular ''non-blocked'' previous modules, which is uploading Tor Browser to popular _non-blocked_
cloud services. This is handled by a series of scripts, one for each cloud cloud services. This is handled by a series of scripts, one for each cloud
service supported. Currently, there are scripts for Dropbox and Google service supported. Currently, there are scripts for Dropbox and Google
Drive.\ Drive.\
\ \
Below you will find a more detailed description of each one of the modules Below you will find a more detailed description of each one of the modules
and scripts of !GetTor. and scripts of GetTor.
=== Core === Core
As its name suggests, this is the core module of !GetTor, and its main As its name suggests, this is the core module of GetTor, and its main
purpose is to provide a simple and robust interface for obtaining the links purpose is to provide a simple and robust interface for obtaining the links
to download the Tor Browser. The design of this module is based on one to download the Tor Browser. The design of this module is based on one
main concept: storing the links on files. The idea consists on having main concept: storing the links on files. The idea consists on having
one file for each cloud service or provider, where each file follows the one file for each cloud service or provider, where each file follows the
Python PEP8 format for configuration files, which means that the data is Python PEP8 format for configuration files, which means that the data is
categorized under sections and accesible by keys. Every ''links file'' categorized under sections and accesible by keys. Every _links file_
must have the following five sections: must have the following five sections:
'''[provider]''': Contains only one key, the name of the cloud service/provider. **[provider]**: Contains only one key, the name of the cloud service/provider.
'''[key]''': Contains only one key, the fingerprint of the PGP key used to sign **[key]**: Contains only one key, the fingerprint of the PGP key used to sign
the Tor Browser packages. the Tor Browser packages.
'''[linux]''': Contains all the links for the Linux operating system, with one **[linux]**: Contains all the links for the Linux operating system, with one
key for each locale available. Every locale should have no more than six key for each locale available. Every locale should have no more than six
lines. There is one line for the Tor Browser link, other for the ASC lines. There is one line for the Tor Browser link, other for the ASC
signature link, and other for the sha256 of Tor Browser. There is one signature link, and other for the sha256 of Tor Browser. There is one
set of three lines for 32-bit and other for 64-bit (six lines in total). set of three lines for 32-bit and other for 64-bit (six lines in total).
'''[windows]''': Contains all the links for the Windows operating system, with **[windows]**: Contains all the links for the Windows operating system, with
one key for each locale available. Every locale should have no more than one key for each locale available. Every locale should have no more than
three lines. There is one line for the Tor Browser link, other for the three lines. There is one line for the Tor Browser link, other for the
ASC signature link, and other for the sha256 of Tor Browser. The windows ASC signature link, and other for the sha256 of Tor Browser. The windows
package of Tor Browser is intended for both 32 and 64 bits. package of Tor Browser is intended for both 32 and 64 bits.
'''[osx]''': Contains all the links for the Mac OSX operating system, with one **[osx]**: Contains all the links for the Mac OSX operating system, with one
key for each locale available. Every locale should have no more than six key for each locale available. Every locale should have no more than six
lines. There is one line for the Tor Browser link, other for the ASC lines. There is one line for the Tor Browser link, other for the ASC
signature link, and other for the sha256 of Tor Browser. There is one signature link, and other for the sha256 of Tor Browser. There is one
set of three lines for 32-bit and other for 64-bit (six lines in total). set of three lines for 32-bit and other for 64-bit (six lines in total).
A sample ''links file'' should look like this: A sample _links file_ should look like this:
{{{ ```
[provider] [provider]
name = Dropbox name = Dropbox
...@@ -128,28 +128,28 @@ es = Package (32-bit): link-to-dropbox-es32 ...@@ -128,28 +128,28 @@ es = Package (32-bit): link-to-dropbox-es32
[osx] [osx]
.... ....
}}} ```
Please note that for the purposes of making things easier, the name of a ''links file'' should Please note that for the purposes of making things easier, the name of a _links file_ should
be ''provider_in_lowercase.links''. All of the above allow us an easy access be _provider_in_lowercase.links_. All of the above allow us an easy access
to the links depending on the operating system and language that we need. The public method to the links depending on the operating system and language that we need. The public method
for doing this is the following: for doing this is the following:
{{{#!python ```
get_links(service, os, lc) get_links(service, os, lc)
}}} ```
This returns a string with the links, where: This returns a string with the links, where:
'''service''': String that identifies the service communicating with the core module. This is for stats purposes only. **service**: String that identifies the service communicating with the core module. This is for stats purposes only.
'''os''': The operating system for which we need the links. There are currently three options: ''windows'', ''linux'', and ''osx''. **os**: The operating system for which we need the links. There are currently three options: _windows_, _linux_, and _osx_.
'''lc''': The locale for which we need the links. There is currently one supported option: ''en'' (for English). **lc**: The locale for which we need the links. There is currently one supported option: _en_ (for English).
Below is a sample script that communicates with the core module: Below is a sample script that communicates with the core module:
{{{#!python ```
#!/usr/bin/python #!/usr/bin/python
import gettor.core import gettor.core
...@@ -158,42 +158,42 @@ core = gettor.core.Core() ...@@ -158,42 +158,42 @@ core = gettor.core.Core()
links = core.get_links('dummy service', 'linux', 'en') links = core.get_links('dummy service', 'linux', 'en')
print links print links
}}} ```
For more details you are welcome to see the implementation on the code repository.\\ For more details you are welcome to see the implementation on the code repository.\\
The secondary purpose of the core module is to provide methods to ease The secondary purpose of the core module is to provide methods to ease
the creation of ''links files'' for cloud services. There are two public the creation of _links files_ for cloud services. There are two public
methods for this: methods for this:
{{{#!python ```
create_links_file(provider, fingerprint) create_links_file(provider, fingerprint)
}}} ```
This creates a ''links file'' with the format ''provider_in_lowecase.links'', where: This creates a _links file_ with the format _provider_in_lowecase.links_, where:
'''provider''': String for the name of the provider/cloud service (e.g. Dropbox) **provider**: String for the name of the provider/cloud service (e.g. Dropbox)
'''fingerprint''': String that represents the fingerprint used to sign the Tor Browser packages. **fingerprint**: String that represents the fingerprint used to sign the Tor Browser packages.
And, And,
{{{#!python ```
add_link(provider, os, lc, link) add_link(provider, os, lc, link)
}}} ```
This adds a link to the ''links file'' of the provider, where: This adds a link to the _links file_ of the provider, where:
'''provider''': Strings that identifies the provider/cloud service. This is also the name of the ''links file''. **provider**: Strings that identifies the provider/cloud service. This is also the name of the _links file_.
'''os''': The operating system for which we intend to add the link. There are currently three options: ''windows'', ''linux'', and ''osx''. **os**: The operating system for which we intend to add the link. There are currently three options: _windows_, _linux_, and _osx_.
'''lc''': Locale for which we intend to add the link. There is currently one supported option: ''en'' (for English). **lc**: Locale for which we intend to add the link. There is currently one supported option: _en_ (for English).
'''link''': String that represents the actual link to be added. **link**: String that represents the actual link to be added.
Below is a sample script to create a ''links file'' and add a couple of links to it: Below is a sample script to create a _links file_ and add a couple of links to it:
{{{#!python ```
#!/usr/bin/python #!/usr/bin/python
import gettor.core import gettor.core
...@@ -209,14 +209,14 @@ core = gettor.core.Core() ...@@ -209,14 +209,14 @@ core = gettor.core.Core()
core.create_links_file('Dropbox', '8738 A680 B84B 3031 A630 F2DB 416F 0610 63FE E659') core.create_links_file('Dropbox', '8738 A680 B84B 3031 A630 F2DB 416F 0610 63FE E659')
core.add_link('Dropbox', 'linux', 'en', link64) core.add_link('Dropbox', 'linux', 'en', link64)
core.add_link('Dropbox', 'linux', 'en', link32) core.add_link('Dropbox', 'linux', 'en', link32)
}}} ```
For more details on these methods please check the code repository and/or see the cloud service scripts section. For more details on these methods please check the code repository and/or see the cloud service scripts section.
=== Distribution Channels === Distribution Channels
Ideally, a user should have various ways to contact !GetTor and receive Ideally, a user should have various ways to contact GetTor and receive
the Tor Browser. This ''distribution channels'' should parse a request, the Tor Browser. This _distribution channels_ should parse a request,
get the user's OS and language, ask for the links to the core module and get the user's OS and language, ask for the links to the core module and
then send this info back to the user. Ideally, each distribution channel then send this info back to the user. Ideally, each distribution channel
should be handled by a separate module. Currently, there is one distribution should be handled by a separate module. Currently, there is one distribution
...@@ -226,7 +226,7 @@ not finished (Twitter). ...@@ -226,7 +226,7 @@ not finished (Twitter).
==== SMTP ==== SMTP
This modules is on charge of receive and reply requests via email. Back This modules is on charge of receive and reply requests via email. Back
in 2008 when !GetTor was conceived, SMTP was the main and only distribution in 2008 when GetTor was conceived, SMTP was the main and only distribution
channel. Requests were answered with the actual bundle as an attachment channel. Requests were answered with the actual bundle as an attachment
instead of links. This approach was good, but the bundles started to get instead of links. This approach was good, but the bundles started to get
larger in size to the point were it was no longer feasible to send it as larger in size to the point were it was no longer feasible to send it as
...@@ -234,33 +234,33 @@ an attachment (the current size of Tor Browser is ~40Mb). ...@@ -234,33 +234,33 @@ an attachment (the current size of Tor Browser is ~40Mb).
There three scenarios involved in sending links via email: There three scenarios involved in sending links via email:
* Listen for users' emails directed to !GetTor robot. * Listen for users' emails directed to GetTor robot.
* Determine the type of request and get the necessary data to reply it. * Determine the type of request and get the necessary data to reply it.
* Send back a reply to the user. * Send back a reply to the user.
The first point is handled by the mail server provided by the Tor Project. The first point is handled by the mail server provided by the Tor Project.
In addition, we use email forwarding to make sure we get all the emails In addition, we use email forwarding to make sure we get all the emails
directed to !GetTor robot. For this a .forward like the following is used: directed to GetTor robot. For this a .forward like the following is used:
{{{#!bash ```
|"python2.7 /path/to/gettor/smtp_process.py" |"python2.7 /path/to/gettor/smtp_process.py"
}}} ```
With this, the only concern of the smtp_process.py script is to receive With this, the only concern of the smtp_process.py script is to receive
emails fron the standard input and talk to the SMTP module to process it. emails fron the standard input and talk to the SMTP module to process it.
The SMTP module has only one public method: The SMTP module has only one public method:
{{{#!python ```
process_email(raw_msg) process_email(raw_msg)
}}} ```
Where: Where:
'''raw_msg''': String that represents the email received. **raw_msg**: String that represents the email received.
A basic script for communicating with the SMTP module should look like this: A basic script for communicating with the SMTP module should look like this:
{{{#!python ```
#!/usr/bin/env python #!/usr/bin/env python
import sys import sys
import gettor.smtp import gettor.smtp
...@@ -268,7 +268,7 @@ import gettor.smtp ...@@ -268,7 +268,7 @@ import gettor.smtp
service = gettor.smtp.SMTP() service = gettor.smtp.SMTP()
incoming = sys.stdin.read() incoming = sys.stdin.read()
service.process_email(incoming) service.process_email(incoming)
}}} ```
The other two points are handled by the SMTP module. The first step after The other two points are handled by the SMTP module. The first step after
receiving a request is determine if the address is blacklisted. See the receiving a request is determine if the address is blacklisted. See the
...@@ -277,7 +277,7 @@ next step is to determine the type of request received.For now, there are ...@@ -277,7 +277,7 @@ next step is to determine the type of request received.For now, there are
only two types of request that could be received: help and links. The decision only two types of request that could be received: help and links. The decision
process to determine what type we have received is the following: process to determine what type we have received is the following:
* Does the body of the message include the words ''windows'', ''linux'', or ''osx''? * Does the body of the message include the words _windows_, _linux_, or _osx_?
If so, we have received a links request. If so, we have received a links request.
* Any other case should be considered as a help request, including blank * Any other case should be considered as a help request, including blank
emails. emails.
...@@ -303,54 +303,54 @@ To be redacted. ...@@ -303,54 +303,54 @@ To be redacted.
=== Database === Database
The database module, as its name suggests, is in charge of interacting The database module, as its name suggests, is in charge of interacting
with the !GetTor database. The current design is quite simple and satisfies with the GetTor database. The current design is quite simple and satisfies
two needs: two needs:
'''Add a request'''. For now it consists only in knowing how many requests we have received so far. No other data is collected. **Add a request**. For now it consists only in knowing how many requests we have received so far. No other data is collected.
'''Add/delete/update a user'''. This allow us to know how many requests a single user has made and thus avoid any type of flood (see Blacklisting). For this purpose we collect the following data: **Add/delete/update a user**. This allow us to know how many requests a single user has made and thus avoid any type of flood (see Blacklisting). For this purpose we collect the following data:
* ''user'': 256 hash of the user address/account. * _user_: 256 hash of the user address/account.
* ''service'': string that represents the distribution method used by the user * _service_: string that represents the distribution method used by the user
(e.g. SMTP). (e.g. SMTP).
* ''times'': number of requests received from the same user. * _times_: number of requests received from the same user.
* ''blocked'': boolean flag to know if user is permanently blacklisted. * _blocked_: boolean flag to know if user is permanently blacklisted.
* ''last_request'': timestamp that represents the last time a given user * _last_request_: timestamp that represents the last time a given user
made a request from the same distribution channel. made a request from the same distribution channel.
The initial design of the database module (during the revamp) considered a The initial design of the database module (during the revamp) considered a
lot of data to be collected (type of request, language, os, etc.), but lot of data to be collected (type of request, language, os, etc.), but
eventually we decided to keep just the necessary data to know how many eventually we decided to keep just the necessary data to know how many
requests !GetTor has received and to avoid flood. The type of database requests GetTor has received and to avoid flood. The type of database
choosen for this purpose was SQLite. You can check a sample database in choosen for this purpose was SQLite. You can check a sample database in
the code repository (gettor.db). the code repository (gettor.db).
=== Blacklisting === Blacklisting
The current blacklisting mechanism is quite simply and it's based on the data The current blacklisting mechanism is quite simply and it's based on the data
collected by the 'users' table specified in !GetTor's database, plus some collected by the 'users' table specified in GetTor's database, plus some
extra parameters defined in blacklist.cfg, which help us to stablish limits extra parameters defined in blacklist.cfg, which help us to stablish limits
to avoid flood. The current mechanism depends on four parameters: to avoid flood. The current mechanism depends on four parameters:
* '''user''': Hashed address/account of the user. It helps to identify malicious users. * **user**: Hashed address/account of the user. It helps to identify malicious users.
* '''service''': Service or distribution channel used by the user trying to contact !GetTor. * **service**: Service or distribution channel used by the user trying to contact GetTor.
* '''max_req''': Maximum number of requests per user ''and'' service allowed at the moment. * **max_req**: Maximum number of requests per user _and_ service allowed at the moment.
* '''wait_time''': Number of minutes a user should wait until she reaches '''max_req'''. * **wait_time**: Number of minutes a user should wait until she reaches **max_req**.
Both the '''user''' and '''service''' parameteres are obtained in real time Both the **user** and **service** parameteres are obtained in real time
when !GetTor receives a request. The other two, '''max_req''' and '''wait_time''' when GetTor receives a request. The other two, **max_req** and **wait_time**
are specified in blacklist.cfg. Each service module (e.g. SMTP) should be are specified in blacklist.cfg. Each service module (e.g. SMTP) should be
in charge of specifying the path to this configuration file and interact in charge of specifying the path to this configuration file and interact
with the !Blacklisting module according to that information. The current with the !Blacklisting module according to that information. The current
mechanism also depends on the '''last_request''', '''times''', and '''blocked''' mechanism also depends on the **last_request**, **times**, and **blocked**
fields of the database for the record associated with '''user''' and '''service'''. fields of the database for the record associated with **user** and **service**.
With all of this, the decision algorithm can be described as follows: With all of this, the decision algorithm can be described as follows:
{{{#!python ```
if blocked: if blocked:
update_user_on_db(user, service, times+1, 1) update_user_on_db(user, service, times+1, 1)
raise BlacklistError("Blocked user") raise BlacklistError("Blocked user")
...@@ -368,15 +368,15 @@ With all of this, the decision algorithm can be described as follows: ...@@ -368,15 +368,15 @@ With all of this, the decision algorithm can be described as follows:
else: else:
# adding up a request for user # adding up a request for user
update_user_on_db(user, service, times+1, 0) update_user_on_db(user, service, times+1, 0)
}}} ```
This simple mechanism helps us avoid malicious users from flooding one or This simple mechanism helps us avoid malicious users from flooding one or
more services/distribution channels with infinite requests. As you may more services/distribution channels with infinite requests. As you may
otice, if a user make a request before the '''wait_time''' has passed, then otice, if a user make a request before the **wait_time** has passed, then
the user must wait another '''wait_time''' to make a request again, and the user must wait another **wait_time** to make a request again, and
if a user make a request after she has reached the maximun number of requests if a user make a request after she has reached the maximun number of requests
and waited '''wait_time''', then the counter of her requests is setted to and waited **wait_time**, then the counter of her requests is setted to
one. You can check the {{{_is_blacklisted}}} method of the SMTP module to one. You can check the `_is_blacklisted` method of the SMTP module to
see how a service should interact with the Blacklisting module. see how a service should interact with the Blacklisting module.
This mechanism could certainly be improved. If you have any ideas/comments This mechanism could certainly be improved. If you have any ideas/comments
...@@ -384,7 +384,7 @@ about it, please tell us (ideally by filling a ticket :) ...@@ -384,7 +384,7 @@ about it, please tell us (ideally by filling a ticket :)
=== Cloud Services === Cloud Services
For each service used by !GetTor to distribute the Tor Browser files there For each service used by GetTor to distribute the Tor Browser files there
should be a script in charge of uploading such files according to the methods should be a script in charge of uploading such files according to the methods
provided by the service used. Each one of these scripts must assume that the provided by the service used. Each one of these scripts must assume that the
latest Tor Browser files has been downloaded (see Other Scripts) and contemplate the latest Tor Browser files has been downloaded (see Other Scripts) and contemplate the
...@@ -402,11 +402,11 @@ You can check the existing scripts for Dropbox and Google Drive to see the curre ...@@ -402,11 +402,11 @@ You can check the existing scripts for Dropbox and Google Drive to see the curre
methods used to do the points listed above, specially 1, 3, 5, and 6. For more details methods used to do the points listed above, specially 1, 3, 5, and 6. For more details
on how the links files are created and how the links are stored, check the documentation on how the links files are created and how the links are stored, check the documentation
about the Core module. Below is a list of the current services/providers integrated about the Core module. Below is a list of the current services/providers integrated
with !GetTor: with GetTor:
* '''Dropbox''': Deployed. In use for a long time. * **Dropbox**: Deployed. In use for a long time.
* '''Google Drive''': Implemented, but not yet deployed. * **Google Drive**: Implemented, but not yet deployed.
* '''Github''': Implemented, but not yet deployed. This one should be especially useful to distribute the Tor Browser * **Github**: Implemented, but not yet deployed. This one should be especially useful to distribute the Tor Browser
in places where Dropbox and Google Drive are blocked (e.g. China). in places where Dropbox and Google Drive are blocked (e.g. China).
If you have an idea for a new service that could be used (even if you don't know how to implement it), If you have an idea for a new service that could be used (even if you don't know how to implement it),
...@@ -416,9 +416,9 @@ please contact us (ideally by filling a ticket :). ...@@ -416,9 +416,9 @@ please contact us (ideally by filling a ticket :).
Below is a list of scripts used for diverse and "smaller" tasks: Below is a list of scripts used for diverse and "smaller" tasks:
* '''blacklist.py''': Handle blacklisting of users. Execute blacklist.py -h for more details. * **blacklist.py**: Handle blacklisting of users. Execute blacklist.py -h for more details.
* '''create_db.py''': Handle the creation of the SQLite database used by !GetTor for managing blacklisting of users and keep track of basic stats. Execute create_db.py -h for more details. * **create_db.py**: Handle the creation of the SQLite database used by GetTor for managing blacklisting of users and keep track of basic stats. Execute create_db.py -h for more details.
* '''stats.py''': Handle basic stats according to the information stored in the SQLite database. Execute stats.py -h for more details. * **stats.py**: Handle basic stats according to the information stored in the SQLite database. Execute stats.py -h for more details.
* '''fetch_latest_torbrowser.py''': Automate the download of Tor Browser files from Tor Project's website and upload of these files to the services used by !GetTor every time a new stable version of Tor Browser is available. Implemented, but not yet deployed. See the source file in the repository for more details. * **fetch_latest_torbrowser.py**: Automate the download of Tor Browser files from Tor Project's website and upload of these files to the services used by GetTor every time a new stable version of Tor Browser is available. Implemented, but not yet deployed. See the source file in the repository for more details.