GitLab is used only for code review, issue tracking and project management. Canonical locations for source code are still https://gitweb.torproject.org/ https://git.torproject.org/ and git-rw.torproject.org.

Commit 74493452 authored by Hiro's avatar Hiro 🏄
Browse files

Update readme

Breakdown strings for translations
Delete unused scripts
parent d3ac24f6
...@@ -32,144 +32,55 @@ Here is a list of the main goals the new GetTor should accomplish: ...@@ -32,144 +32,55 @@ Here is a list of the main goals the new GetTor should accomplish:
Installing GetTor Installing GetTor
================= =================
```sh
WORKON_HOME=${HOME}/venv
export WORKON_HOME
mkdir -p $WORKON_HOME
source $(which virtualenvwrapper.sh)
git clone https://git.torproject.org/gettor.git && cd gettor
mkvirtualenv -a $PWD -r requirements.txt --unzip-setuptools --setuptools gettor
```
From now on, to use GetTor's virtualenv, just do ``$ workon gettor``
(after sourcing virtualenvwrapper.sh, as before). To exit the virtualenv
without exiting the shell, do ``$ deactivate``.
```sh
export PYTHONPATH=$PYTHONPATH:${VIRTUAL_ENV}/lib/python/site-packages
```
```sh
$ ./scripts/create_db
$ ./bin/gettor_service start
```
Running tests
=================
GetTor includes PyTest unit tests. To run the tests, first install some dependencies:
```sh
pip3 install -r .test.requirements.txt
```
Then you can run `pytest` against the `tests/` directory.
```sh
pytest tests/
```
How does the new GetTor works?
==============================
Below are some specifications and core concepts on how the new GetTor works. To install gettor locally please install the following packages (on debian):
*Links files*: Currently links are saved in files with the '.links' extension, python3-coverage
using the ConfigParser format (RFC 882). A sample link file should look like python3-dkim
this: python3-dns
python3-internetarchive
python3-openssl
python3-pytest
python3-requests-oauthlib
python3-service-identity
python3-twisted
sqlite3
--- BEGIN FILE --- The following packages are needed to run a gettor instance:
[provider] internetarchive
name = CoolCloudProvider jq
rclone
[key] Specifically:
fingerprint = AAAA BBBB CCCC DDDD EEEE FFFF GGGG HHHH IIII JJJJ internetarchive is needed to send Tor Browser files via command line to the internet archive.
jq is a json parser that is used to find out about the new tor browser releases.
Both internetarchive and jq are used in: scripts/update_files
[linux] The following packages are instead needed to deploy gettor via ansible:
en = Package (64-bit): https://cool.cloud.link64
ASC signature (64-bit): https://cool.cloud.link64.asc
Package SHA256 checksum (64-bit): superhash64,
Package (32-bit): https://cool.cloud.link32
ASC signature (32-bit): https://cool.cloud.link32.asc
Package SHA256 checksum (32-bit): superhash32
[windows] ansible
... ansible-lint
[osx] Gettor ansible playbooks live at: https://gitweb.torproject.org/admin/services/gettor.git/
...
--- END FILE --- Finally the following package is used store Tor Browser files via git and support large files:
git-lfs
You can also check providers/dropbox.links for a better example.
WORKON_HOME=${HOME}/.virtualenvs
export WORKON_HOME
mkdir -p $WORKON_HOME
source $(which virtualenvwrapper.sh)
*Core*: the heart of GetTor. Receives requests for links for a certain OS and
language and respond accordingly. It also presents an easy way for scripts
to create links file.
*SMTP*: Receives requests via email, process them, contact the core module if
necessary and respond to the user in the specified language. People can send
blank or dummy emails to it to receive a help message describing how to ask
for links. Email forwarding is used to redirect the emails to GetTor.
*XMPP*: Same as above, but via XMPP (account needed). It has been tested with Once gettor is installed you can run it with:
dukgo.com, jabber.ccc.de, riseup.net. It doesn't seem to be able to interact
with gtalk users.
*Twitter*: Receive requests via Twitter direct messages, contact the core module ```
if necessary and respond to the user in the specified language. Unfinished. $ ./bin/gettor_service start
```
*DB*: Store anonymous info about the people that interact with GetTor in order
to keep count of the number of requests per person and avoid malicious users
that try to collapse the service. It also keeps count of how many requests
GetTor has received during its lifetime. A lot of other data was being saved
in the original gsoc project, but it was changed to save the minimum.
*Blacklist*: Provide a mechanism to avoid flood and deny interaction to
malicious users.
*Providers scripts*: every supported provider should have a script to
automatically upload packages to 'the cloud' and create the corresponding
links files. The script should consider the following steps:
* Upload the packages.
* Get the sha256 checksum of the files uploaded.
* Get the PGP key fingerprint that signed the files.
* Check for .asc file for every package uploaded.
* Put all together in a '.link' file (using the core module).
What is the current status of the new GetTor?
=============================================
Deployed and working.
How can I help?
================
If you have ideas to improve GetTor and/or add new providers, please tell us!
I'm currently the lead developer on this, so if you have any comments/doubts/
ideas you can send me an e-mail to ilv _at_ riseup _dot_ net or ping me (ilv),
or sukhe or mrphs at #tor-dev in the OFTC IRC network. For openning tickets you
should use the trac[0] and select the GetTor component. Some neat ideas we
could use are the following:
* Report bugs! Running tests
* Create script for new providers, namely: Google Drive, Github. Check =================
providers.txt
* Create a new module for distributing links. Check distribution_methods.txt
* Finish the Twitter module.
* Propose code/behaviour improvements.
* Update the specs.
GetTor includes PyTest unit tests. To run the tests, first install some dependencies:
References
===========
[0] https://trac.torproject.org/projects/tor/query?status=accepted&status=assigned&status=needs_information&status=needs_review&status=needs_revision&status=new&status=reopened&component=GetTor&col=id&col=summary&col=component&col=status&col=type&col=priority&col=milestone&order=priority ```
$ pytest-3 tests/
```
#!/usr/bin/python
# -*- coding: utf-8 -*-
#
# This file is part of GetTor, a Tor Browser distribution system.
#
# :authors: Israel Leiva <ilv@riseup.net>
# see also AUTHORS file
#
# :copyright: (c) 2008-2014, The Tor Project, Inc.
# (c) 2014, Israel Leiva
#
# :license: This is Free Software. See LICENSE for license information.
import sys
import time
import sqlite3
import argparse
def main():
"""Script for managing blacklisting of users.
See argparse usage for more details.
"""
parser = argparse.ArgumentParser(description='Utility for GetTor'
' blacklisting')
parser.add_argument('database', metavar='database.db', type=str,
help='the database file')
parser.add_argument('-u', '--user', default=None,
help='filter by user hash')
parser.add_argument('-s', '--service', default=None,
help='filter by service')
parser.add_argument('-b', '--blocked', default=None,
help='filter by blocked users')
parser.add_argument('-a', '--add', default=None, nargs=3,
metavar=('USER', 'SERVICE', 'BLOCKED'),
help='add user')
parser.add_argument('-c', '--clean', default=None, const='c', nargs='?',
metavar='user hash',
help='clean table (delete expired blacklistings)')
parser.add_argument('-r', '--requests', default=None,
help='number of requests; everyone with number of'
' requests greather than this will be cleaned up')
args = parser.parse_args()
query = ''
con = sqlite3.connect(args.database)
if args.add:
# add new entry, useful for adding users permanently blocked
query = "INSERT INTO users VALUES('%s', '%s', 1, %s, %s)"\
% (args.add[0], args.add[1], args.add[2], time.time())
with con:
cur = con.cursor()
cur.execute(query)
print "Query execute successfully"
elif args.clean:
if args.clean == 'c':
if args.requests:
# delete by number of times
query = "DELETE FROM users WHERE times > %s" % args.requests
with con:
cur = con.cursor()
cur.execute(query)
print "Query executed successfully."
else:
sys.exit("Number of requests missing. See --help.")
else:
# delete by id
query = "DELETE FROM users WHERE id='%s'" % args.clean
with con:
cur = con.cursor()
cur.execute(query)
print "Query execute succcessfully."
else:
query = "SELECT * FROM users"
has_where = False
# filter
if args.service:
query = "%s %s" % (query, "WHERE service='%s'" % args.service)
has_where = True
if args.user:
if has_where:
query = "%s %s" % (query, "AND id='%s'" % args.user)
else:
query = "%s %s" % (query, "WHERE id='%s'" % args.user)
has_where = True
if args.blocked:
if has_where:
query = "%s %s" % (query, "AND blocked=%s" % args.blocked)
has_where = True
else:
query = "%s %s" % (query, "WHERE blocked=%s" % args.blocked)
with con:
cur = con.cursor()
cur.execute(query)
rows = cur.fetchall()
# show it nice
print "\nNumber of results: %s\n" % len(rows)
cns = [cn[0] for cn in cur.description]
print "%-70s %-10s %-10s %-10s %-s" % (cns[0], cns[1], cns[2],
cns[3], cns[4])
for row in rows:
print "%-70s %-10s %-10s %-10s %s" % (row[0], row[1], row[2],
row[3], row[4])
if __name__ == "__main__":
main()
#!/usr/bin/python
#
# Dummy script to test GetTore's Core module
#
import gettor.core
try:
core = gettor.core.Core()
links = core.get_links('dummy service', 'linux', 'en')
print links
except gettor.core.ConfigError as e:
print "Misconfiguration: " + str(e)
except gettor.core.UnsupportedOSError as e:
print "Unsupported OS: " + str(e)
except gettor.core.UnsupportedLocaleError as e:
print "Unsupported Locale: " + str(e)
except gettor.core.InternalError as e:
print "Internal error: " + str(e)
# -*- coding: utf-8 -*-
#
# This file is part of GetTor.
#
# :authors: Israel Leiva <ilv@torproject.org>
# see also AUTHORS file
#
# :copyright: (c) 2008-2015, The Tor Project, Inc.
# (c) 2015, Israel Leiva
#
# :license: This is Free Software. See LICENSE for license information.
import os
import re
import json
import codecs
import urllib2
import ConfigParser
from time import gmtime, strftime
"""Script to build a static mirror of GetTor RESTful API"""
# currently supported locales for Tor Browser
LC = ['ar', 'de', 'en-US', 'es-ES', 'fa', 'fr', 'it', 'ko', 'nl', 'pl',
'pt-PT', 'ru', 'tr', 'vi', 'zh-CN']
# https://gitweb.tpo/tor-browser-spec.git/tree/processes/VersionNumbers
# does not say anything about operating systems, so it's possible the
# notation might change in the future. We should always use the same three
# strings though: linux, windows, osx.
OS = {
'Linux': 'linux',
'Windows': 'windows',
'MacOS': 'osx'
}
# based on
# https://gitweb.tpo.org/tor-browser-spec.git/tree/processes/VersionNumbers
# except for the first one, which is based on current RecommendedTBBVersions
RE = {
'os': '(.*)-(\w+)',
'alpha': '\d\.\d(\.\d)*a\d+',
'beta': '\d\.\d(\.\d)*b\d+',
'stable': '\d\.\d(\.\d)*'
}
# strings to build names of packages depending on OS.
PKG = {
'windows': 'torbrowser-install-%s_%s.exe',
'linux': 'tor-browser-linux%s-%s_%s.tar.xz',
'osx': 'TorBrowser-%s-osx64_%s.dmg'
}
# bin and asc are used to build the download links for each version, os and lc
URL = {
'version': 'https://gettor.torproject.org/api/latest.json',
'mirrors': 'https://gettor.torproject.org/api/mirrors.json',
'providers': 'https://gettor.torproject.org/api/providers.json'
}
GETTOR_URL = 'http://gettor.torproject.org'
SERVER_URL = 'https://yourprivateserver.com'
API_TREE = '/home/ilv/Proyectos/tor/static_api/'
class ConfigError(Exception):
pass
class InternalError(Exception):
pass
class APIMirror(object):
""" Provide useful resources via RESTful API. """
def __init__(self, cfg=None):
self.url = SERVER_URL
self.tree = API_TREE
def _is_json(self, my_json):
""" Check if json generated is valid.
:param: my_json (string) data to ve verified.
:return: (bool) true if data is json-valid, false otherwise.
"""
try:
json_object = json.loads(my_json)
except ValueError, e:
return False
return True
def _write_json(self, path, content):
"""
"""
try:
with codecs.open(
path,
"w",
"utf-8"
) as jsonfile:
# Make pretty json
json.dump(
content,
jsonfile,
sort_keys=True,
indent=4,
separators=(',', ': '),
encoding="utf-8",
)
except IOError as e:
#logging.error("Couldn't write json: %s" % str(e))
print "Error building %s: %s" % (path, str(e))
print "%s built" % path
def _get_provider_name(self, p):
""" Return simplified version of provider's name.
:param: p (string) provider's name.
:return: (string) provider's name in lowercase and without spaces.
"""
p = p.replace(' ', '-')
return p.lower()
def _load_latest_version(self):
""" Load latest version data from GetTor API. """
response = urllib2.urlopen(URL['version'])
json_response = json.load(response)
self.lv = json_response
def _load_links(self):
""" Load links and providers data. """
response = urllib2.urlopen(URL['providers'])
json_response = json.load(response)
links = {}
for provider in json_response:
if provider != 'updated_at':
provider_url = json_response[provider]
provider_response = urllib2.urlopen("%s.json" % provider_url)
provider_content = json.load(provider_response)
json_response[provider] = provider_url.replace(
GETTOR_URL,
self.url
)
links[provider] = provider_content
self.providers = json_response
self.links = links
def _load_mirrors(self):
""" Load mirrors data from GetTor API. """
response = urllib2.urlopen(URL['mirrors'])
json_response = json.load(response)
self.mirrors = json_response
def _load_resources(self):
""" Load available resources data. """
self.resources = {
'providers': '%s/providers' % self.url,
'mirrors': '%s/mirrors' % self.url,
'latest_version': '%s/latest' % self.url,
'updated_at': strftime("%Y-%m-%d %H:%M:%S", gmtime())
}
def load_data(self):
""" Load all data."""
self._load_links()
self._load_mirrors()
self._load_resources()
self._load_latest_version()
def build(self):
""" Build RESTful API. """
print "Building API mirror"
# resources
self._write_json(
os.path.join(self.tree, 'api'),
self.resources
)
api_path = os.path.join(self.tree, 'api-content')
if not os.path.isdir(api_path):
os.mkdir(api_path)
# providers
self._write_json(
os.path.join(api_path, 'providers'),
self.providers
)
providers_path = os.path.join(api_path, 'providers-content')
if not os.path.isdir(providers_path):
os.mkdir(providers_path)
for provider in self.links:
if provider == 'updated_at':
continue
self._write_json(
os.path.join(providers_path, provider),
self.links[provider]
)
provider_path = os.path.join(
providers_path,
"%s-content" % provider
)
if not os.path.isdir(provider_path):
os.mkdir(provider_path)
for osys in self.links[provider]:
self._write_json(
os.path.join(provider_path, osys),
self.links[provider][osys]
)
provider_os_path = os.path.join(
provider_path, "%s-content" % osys
)
if not os.path.isdir(provider_os_path):
os.mkdir(provider_os_path)
for lc in self.links[provider][osys]:
self._write_json(
os.path.join(provider_os_path, lc),
self.links[provider][osys][lc]
)
# latest version
self._write_json(
os.path.join(api_path, 'latest'),
self.lv
)
lv_path = os.path.join(api_path, 'latest-content')
if not os.path.isdir(lv_path):
os.mkdir(lv_path)
for release in self.lv:
if release == 'updated_at':
continue
self._write_json(
os.path.join(lv_path, release),
self.lv[release]
)
release_path = os.path.join(
lv_path,
"%s-content" % release
)
if not os.path.isdir(release_path):
os.mkdir(release_path)
for osys in self.lv[release]['downloads']:
self._write_json(
os.path.join(release_path, osys),
self.lv[release]['downloads'][osys]
)
release_os_path = os.path.join(
release_path,
"%s-content" % osys
)
if not os.path.isdir(release_os_path):
os.mkdir(release_os_path)
for lc in self.lv[release]['downloads'][osys]:
self._write_json(
os.path.join(release_os_path, lc),
self.lv[release]['downloads'][osys][lc]
)
# mirrors
self._write_json(
os.path.join(api_path, 'mirrors'),
self.mirrors
)
def main():
api = APIMirror()
api.load_data()
api.build()
if __name__ == '__main__':
main()