Commit 150b15fc authored by ilv's avatar ilv
Browse files

Changes related to GetTor meeting of October 10th. Deleted obsolete documentation.

parent f377c24d
gettor is distributed under this license:
GetTor is distributed under this license:
Copyright (c) 2008-2014, The Tor Project, Inc.
......@@ -28,4 +28,4 @@ LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
\ No newline at end of file
gettor
GetTor
======
GetTor Revamp (on development).
Google Summer of Code 2014.
* To get the current repo:
$ git clone https://github.com/ileiva/gettor.git
* To upload bundles to Dropbox and create a links file:
1) Install the Dropbox and GnuPG Python modules (just the first time):
$ pip install dropbox gnupg
2) Change account info in src/dropbox.py (app_key, app_secret, access_token)
3) Specify the path of the PGP key that signed the packages (to include fingerprint).
4) Run the script:
$ cd src/providers/;rm *.links;cd ..; python dropbox.py
If everything works good, you should see a dropbox.links file inside the 'providers' directory. The script will take the files on upload_dir (default to 'upload/') which end up on .xz and .xz.asc respectively. To add more locales for testing do the following (example for german):
$ cd upload; cp tor-browser-linux32-3.6.2_en-EN.tar.xz tor-browser-linux32-3.6.2_de-DE.tar.xz
$ cd upload; cp tor-browser-linux32-3.6.2_en-EN.tar.xz.asc tor-browser-linux32-3.6.2_de-DE.tar.xz.asc
A script for getting the latest bundles is pending.
* To test if the core module is working:
1) Use the dummy script provided:
$ python core_demo.py
* To test the smtp module (without mail server):
1) Set request parameters on smtp/sample/sample-email.eml (by default, 'To: gettor+en@torproject.org' and 'linux' in the body of the message.
2) Run dummy script:
$ python smtp_demo.py < smtp/sample/sample-email.eml
If mail server is configured, then uncomment lines 328-332, 337, 353-359, and comment lines 334-335, 338, 360 on gettor/smtp.py. Also, you should enable e-mail forwarding as specified on https://gitweb.torproject.org/gettor.git/blob/HEAD:/README
* To test the xmpp module
1) Install the SleekXMPP module:
$ pip install sleekxmpp
2) Change user details on xmpp.cfg
3) Run dummy script.
$ python xmpp_demo.py
4) To communicate with the bot using Pidgin click on Friends -> New instant message -> Enter the address used in xmpp.cfg to start comunicating with it. Current commands are as follows:
operating_system locale: it'll get you links for operating system in the locale specified e.g. linux en
help locale: it'll get you help info in locale e.g. help es
Default locale is en, and default response is help.
The xmpp module has been used in the following providers:
* dukgo.com (works)
* riseup.net (works)
* jabber.ccc.de (works)
* jit.si (worked for a while, not any longer)
GetTor development. Project started at the Google Summer of Code 2014, under the
Tor Project organization.
[general]
basedir: /path/to/gettor
basedir: /home/ilv/Proyectos/tor/gettor/src/
db: gettor.db
[links]
dir: providers/
dir: /path/to/providers/
os: linux,windows,osx
locales: es,en
[log]
dir: log/
level: DEBUG
dir: /home/ilv/Proyectos/tor/gettor/src/log/
level: DEBUG
\ No newline at end of file
......@@ -7,9 +7,9 @@ import gettor.core
try:
core = gettor.core.Core()
links = core.get_links('dummy service', 'linux', 'es')
links = core.get_links('dummy service', 'linux', 'en')
print links
except gettor.core.ConfigurationError as e:
except gettor.core.ConfigError as e:
print "Misconfiguration: " + str(e)
except gettor.core.UnsupportedOSError as e:
print "Unsupported OS: " + str(e)
......
[general]
upload_dir: upload
tbb_key: tbb-key.asc
[app]
key: suchkey
secret: suchsecret
access_token: suchtoken
\ No newline at end of file
# -*- coding: utf-8 -*-
#
# This file is part of GetTor, a Tor Browser Bundle distribution system.
# This file is part of GetTor, a Tor Browser distribution system.
#
# :authors: Israel Leiva <ilv@riseup.net>
# see also AUTHORS file
......@@ -14,18 +14,23 @@ import re
import os
import gnupg
import hashlib
import ConfigParser
import dropbox
import gettor.core
def valid_bundle_format(file):
"""
Checks for a valid bundle format
(e.g. tor-browser-linux32-3.6.2_es-ES.tar.xz
def valid_format(file):
"""Check for valid bundle format
Returns True or False if it's valid or not.
"""
Check if the given file has a valid bundle format
(e.g. tor-browser-linux32-3.6.2_es-ES.tar.xz)
:param: file (string) the name of the file.
:return: (boolean) true if the bundle format is valid, false otherwise.
"""
m = re.search(
'tor-browser-(\w+)\d\d-\d\.\d\.\d_(\w\w)-\w+\.tar\.xz',
file)
......@@ -36,27 +41,36 @@ def valid_bundle_format(file):
def get_bundle_info(file):
"""Get the operating system and locale from a bundle string.
"""Get the os, arch and lc from a bundle string.
it raises a ValueError if the bundle doesn't have a valid format
(although you should probably call valid_bundle_format first).
It returns the pair of strings operating system, locale.
:param: file (string) the name of the file.
:raise: ValueError if the bundle doesn't have a valid bundle format.
:return: (list) the os, arch and lc.
"""
m = re.search(
'tor-browser-(\w+)\d\d-\d\.\d\.\d_(\w\w)-\w+\.tar\.xz',
'tor-browser-(\w+)(\d\d)-\d\.\d\.\d_(\w\w)-\w+\.tar\.xz',
file)
if m:
operating_system = m.group(1)
locale = m.group(2)
return operating_system, locale
os = m.group(1)
arch = m.group(2)
lc = m.group(3)
return os, arch, lc
else:
raise ValueError("Bundle invalid format %s" % file)
raise ValueError("Invalid bundle format %s" % file)
def get_file_sha256(file):
"""Get the sha256 of a file."""
# as seen on the internet
"""Get the sha256 of a file.
:param: file (string) the path of the file.
:return: (string) the sha256 hash.
"""
# as seen on the internetz
BLOCKSIZE = 65536
hasher = hashlib.sha256()
with open(file, 'rb') as afile:
......@@ -66,15 +80,21 @@ def get_file_sha256(file):
buf = afile.read(BLOCKSIZE)
return hasher.hexdigest()
def upload_files(basedir, client):
"""Upload files from 'basedir' to Dropbox.
"""Upload files to Dropbox.
It looks for files ending with 'tar.xz' inside 'basedir'. It
raises ValueError in case the given file doesn't have a .asc file.
It raises UploadError if something goes wrong while uploading the
files to Dropbox. All files are uploaded to '/'.
Looks for files ending with 'tar.xz' inside basedir.
Returns a list with the names of the uploaded files.
:param: basedir (string) path of the folder with the files to be
uploaded.
:param: client (object) DropboxClient object.
:raise: ValueError if the .xz file doesn't have an .asc file.
:raise: UploadError if something goes wrong while uploading the
files to Dropbox. All files are uploaded to '/'.
:return: (list) the names of the uploaded files.
"""
files = []
......@@ -83,18 +103,18 @@ def upload_files(basedir, client):
for name in os.listdir(basedir):
path = os.path.abspath(os.path.join(basedir, name))
if os.path.isfile(path) and p.match(path) and valid_bundle_format(name):
if os.path.isfile(path) and p.match(path) and valid_format(name):
files.append(name)
for file in files:
asc = file + '.asc'
asc = "%s.asc" % file
abs_file = os.path.abspath(os.path.join(basedir, file))
abs_asc = os.path.abspath(os.path.join(basedir, asc))
if not os.path.isfile(abs_asc):
raise ValueError("%s doesn't exist!" % asc)
# Chunk upload for big files
# chunk upload for big files
to_upload = open(abs_file, 'rb')
size = os.path.getsize(abs_file)
uploader = client.get_chunked_uploader(to_upload, size)
......@@ -104,22 +124,26 @@ def upload_files(basedir, client):
except rest.ErrorResponse, e:
UploadError("An error ocurred while uploading %s" % abs_file)
uploader.finish(file)
print "Uploading %s" % file
# This should be small, upload it simple
# this should be small, upload it simple
to_upload_asc = open(abs_asc, 'rb')
response = client.put_file(asc, to_upload_asc)
print "Uploading %s" % asc
return files
if __name__ == '__main__':
# to-do: use config file
app_key = ''
app_secret = ''
access_token = ''
upload_dir = 'upload/'
config = ConfigParser.ConfigParser()
config.read('dropbox.cfg')
app_key = config.get('app', 'key')
app_secret = config.get('app', 'secret')
access_token = config.get('app', 'access_token')
upload_dir = config.get('general', 'upload_dir')
# important: this must be the key that signed the packages
tbb_key = 'tbb-key.asc'
# important: this key must be the one that signed the packages
tbb_key = config.get('general', 'tbb_key')
client = dropbox.client.DropboxClient(access_token)
......@@ -136,27 +160,30 @@ if __name__ == '__main__':
try:
uploaded_files = upload_files(upload_dir, client)
# use default config
core = gettor.core.Core()
core = gettor.core.Core('/home/gettor/core.cfg')
# erase old links
core.create_links_file('Dropbox', readable)
for file in uploaded_files:
for file in uploaded_files:
# build file names
asc = file + '.asc'
asc = "%s.asc" % file
abs_file = os.path.abspath(os.path.join(upload_dir, file))
abs_asc = os.path.abspath(os.path.join(upload_dir, asc))
sha_file = get_file_sha256(abs_file)
# build links
link_file = client.share(file)
link_asc = client.share(asc)
link = link_file[u'url'] + ' ' + link_asc[u'url'] + ' ' + sha_file
# add links
operating_system, locale = get_bundle_info(file)
core.add_link('Dropbox', operating_system, locale, link)
link_file = client.share(file, short_url=False)
link_asc = client.share(asc, short_url=False)
osys, arch, lc = get_bundle_info(file)
link = "Package (%s-bit): %s\nASC signature (%s-bit): %s\n"\
"Package SHA256 checksum (%s-bit): %s\n" %\
(arch, link_file[u'url'], arch, link_asc[u'url'],
arch, sha_file)
core.add_link('Dropbox', osys, lc, link)
except (ValueError, RuntimeError) as e:
print str(e)
except dropbox.rest.ErrorResponse as e:
......
# -*- coding: utf-8 -*-
#
# This file is part of GetTor, a Tor Browser Bundle distribution system.
# This file is part of GetTor, a Tor Browser distribution system.
#
# :authors: Israel Leiva <ilv@riseup.net>
# see also AUTHORS file
......@@ -57,38 +57,34 @@ class Blacklist(object):
if cfg is None or not os.path.isfile(cfg):
cfg = DEFAULT_CONFIG_FILE
log.info("Using default configuration")
log.info("Reading configuration file %s" % cfg)
config.read(cfg)
try:
dbname = config.get('general', 'db')
self.db = db.DB(dbname)
except ConfigParser.Error as e:
log.warning("Couldn't read 'db' from 'general' (%s)" % cfg)
raise ConfigurationError("Error with conf. See log file.")
raise ConfigError("Couldn't read 'db' from 'general'")
try:
self.logdir = config.get('log', 'dir')
logdir = config.get('log', 'dir')
logfile = os.path.join(logdir, 'blacklist.log')
except ConfigParser.Error as e:
log.warning("Couldn't read 'dir' from 'log' (%s)" % cfg)
raise ConfigurationError("Error with conf. See log file.")
raise ConfigError("Couldn't read 'dir' from 'log'")
try:
self.loglevel = config.get('log', 'level')
loglevel = config.get('log', 'level')
except ConfigParser.Error as e:
log.warning("Couldn't read 'level' from 'log' (%s)" % cfg)
raise ConfigurationError("Error with conf. See log file.")
raise ConfigError("Couldn't read 'level' from 'log'")
# keep log levels separated
self.log = utils.filter_logging(log, self.logdir, self.loglevel)
self.log.setLevel(logging.getLevelName(self.loglevel))
log.debug('Redirecting logging to %s' % self.logdir)
# establish log level and redirect to log file
log.info('Redirecting logging to %s' % logfile)
logfileh = logging.FileHandler(logfile, mode='a+')
logfileh.setLevel(logging.getLevelName(loglevel))
log.addHandler(logfileh)
# stop logging on stdout from now on
log.propagate = False
self.log.debug("New blacklist object created")
def is_blacklisted(self, user, service, max_req, wait_time):
"""Check if a user is blacklisted.
......@@ -117,7 +113,6 @@ class Blacklist(object):
if r:
# permanently blacklisted
if r['blocked']:
self.log.info("Request from blocked user %s" % user)
self.db.update_user(user, service, r['times']+1, 1)
raise BlacklistError("Blocked user")
# don't be greedy
......@@ -127,17 +122,15 @@ class Blacklist(object):
next = last + datetime.timedelta(minutes=wait_time)
if datetime.datetime.now() < next:
self.log.info("Too many requests from user %s" % user)
# too many requests from the same user
self.db.update_user(user, service, r['times']+1, 0)
raise BlacklistError("Too many requests")
else:
# fresh user again!
self.log.debug("Request after wait time, cleaning up for"
" %s" % user)
self.db.update_user(user, service, 1, 0)
else:
self.log.debug("Adding up a request for %s" % user)
# adding up a request for user
self.db.update_user(user, service, r['times']+1, 0)
else:
self.log.debug("New request for %s" % user)
# new request for user
self.db.add_user(user, service, 0)
# -*- coding: utf-8 -*-
#
# This file is part of GetTor, a Tor Browser Bundle distribution system.
# This file is part of GetTor, a Tor Browser distribution system.
#
# :authors: Israel Leiva <ilv@riseup.net>
# see also AUTHORS file
......@@ -23,7 +23,7 @@ import utils
"""Core module for getting links from providers."""
class ConfigurationError(Exception):
class ConfigError(Exception):
pass
......@@ -62,7 +62,7 @@ class Core(object):
UnsupportedOSError: Request for an unsupported operating system.
UnsupportedLocaleError: Request for an unsupported locale.
ConfigurationError: Something's misconfigured.
ConfigError: Something's misconfigured.
LinkFormatError: The link added doesn't seem legit.
LinkFileError: Error related to the links file of a provider.
InternalError: Something went wrong internally.
......@@ -82,70 +82,61 @@ class Core(object):
logging.basicConfig(format='[%(levelname)s] %(asctime)s - %(message)s',
datefmt="%Y-%m-%d %H:%M:%S")
logger = logging.getLogger(__name__)
log = logging.getLogger(__name__)
config = ConfigParser.ConfigParser()
if cfg is None or not os.path.isfile(cfg):
cfg = DEFAULT_CONFIG_FILE
logger.info("Using default configuration")
logger.info("Reading configuration file %s" % cfg)
config.read(cfg)
try:
self.basedir = config.get('general', 'basedir')
basedir = config.get('general', 'basedir')
except ConfigParser.Error as e:
logger.warning("Couldn't read 'basedir' from 'general' (%s)" % cfg)
raise ConfigurationError("Error with conf. See log file.")
raise ConfigError("Couldn't read 'basedir' from 'general'")
try:
dbname = config.get('general', 'db')
dbname = os.path.join(self.basedir, dbname)
dbname = os.path.join(basedir, dbname)
self.db = db.DB(dbname)
except ConfigParser.Error as e:
logger.warning("Couldn't read 'db' from 'general' (%s)" % cfg)
raise ConfigurationError("Error with conf. See log file.")
raise ConfigError("Couldn't read 'db' from 'general'")
try:
self.linksdir = config.get('links', 'dir')
self.linksdir = os.path.join(self.basedir, self.linksdir)
self.linksdir = os.path.join(basedir, self.linksdir)
except ConfigParser.Error as e:
logger.warning("Couldn't read 'links' from 'dir' (%s)" % cfg)
raise ConfigurationError("Error with conf. See log file.")
raise ConfigError("Couldn't read 'links' from 'dir'")
try:
self.supported_lc = config.get('links', 'locales')
except ConfigParser.Error as e:
logger.warning("Couldn't read 'locales' from 'links' (%s)" % cfg)
raise ConfigurationError("Error with conf. See log file.")
raise ConfigError("Couldn't read 'locales' from 'links'")
try:
self.supported_os = config.get('links', 'os')
except ConfigParser.Error as e:
logger.warning("Couldn't read 'os' from 'links' (%s)" % cfg)
raise ConfigurationError("Error with conf. See log file.")
raise ConfigError("Couldn't read 'os' from 'links'")
try:
self.loglevel = config.get('log', 'level')
loglevel = config.get('log', 'level')
except ConfigParser.Error as e:
logger.warning("Couldn't read 'level' from 'log' (%s)" % cfg)
raise ConfigurationError("Error with conf. See log file.")
raise ConfigError("Couldn't read 'level' from 'log'")
try:
self.logdir = config.get('log', 'dir')
self.logdir = os.path.join(self.basedir, self.logdir)
logdir = config.get('log', 'dir')
logfile = os.path.join(logdir, 'core.log')
except ConfigParser.Error as e:
logger.warning("Couldn't read 'dir' from 'log' %s)" % cfg)
raise ConfigurationError("Error with conf. See log file.")
raise ConfigError("Couldn't read 'dir' from 'log'")
# keep log levels separated
self.log = utils.filter_logging(logger, self.logdir, self.loglevel)
# self.log.setLevel(logging.getLevelName(self.loglevel))
self.log.info('Redirecting logging to %s' % self.logdir)
# establish log level and redirect to log file
log.info('Redirecting logging to %s' % logfile)
logfileh = logging.FileHandler(logfile, mode='a+')
logfileh.setLevel(logging.getLevelName(loglevel))
log.addHandler(logfileh)
# stop logging on stdout from now on
self.log.propagate = False
self.log.debug("New core object created")
log.propagate = False
def get_links(self, service, os, lc):
"""Get links for OS in locale.
......@@ -166,35 +157,28 @@ class Core(object):
"""
# Which module called us and what was asking for?
self.log.info("%s did a request for %s, %s." % (service, os, lc))
if lc not in self.supported_lc:
self.log.warning("Request for unsupported locale: %s" % lc)
raise UnsupportedLocaleError("Locale %s not supported" % lc)
if os not in self.supported_os:
self.log.warning("Request for unsupported OS: %s" % os)
raise UnsupportedOSError("OS %s not supported " % os)
# this could change in the future, let's leave it isolated.
links = self._get_links(os, lc)
if links is None:
self.log.error("Couldn't get the links", exc_info=True)
raise InternalError("Something went wrong internally. See logs for"
" detailed info.")
raise InternalError("Something went wrong internally")
self.log.info("Returning the links")
# thanks for stopping by
return links
def _get_links(self, os, lc):
def _get_links(self, osys, lc):
"""Internal method to get the links.
Looks for the links inside each provider file. This should only be
called from get_links() method.
:param: os (string) the operating system.
:param: osys (string) the operating system.
:param: lc (string) the locale.
:return: (string/None) links on success, None otherwise.
......@@ -218,60 +202,48 @@ class Core(object):
# links were found
providers = {}
self.log.info("Reading links from providers directory")
# reading links from providers directory
for name in links:
self.log.debug("Reading %s" % name)
# We're reading files listed on linksdir, so they must exist!
# we're reading files listed on linksdir, so they must exist!
config = ConfigParser.ConfigParser()
config.read(name)
try:
pname = config.get('provider', 'name')
except ConfigParser.Error as e:
self.log.warning("Couldn't get 'name' from 'provider' (%s)"
% name)
raise InternalError("Error while reading %s links file. See "
"log file" % name)
self.log.debug("Checking if %s has links for %s in %s" %
(pname, os, lc))
raise InternalError("Couldn't get 'name' from 'provider'")
# checking if current provider pname has links for os in lc
try:
providers[pname] = config.get(os, lc)
providers[pname] = config.get(osys, lc)
# avoid showing it all together
providers[pname] = providers[pname].replace(",", "\n")
except ConfigParser.Error as e:
self.log.warning("Couldn't get %s from %s (%s)" %