Unverified Commit dda4bbee authored by Isis Lovecruft's avatar Isis Lovecruft
Browse files

Merge branch 'fix/22871_r1' into develop

parents 35603395 3667a630
......@@ -24,6 +24,7 @@ exclude_lines =
if options[.verbosity.]
def __repr__
if __name__ == .__main__.:
except Exception as impossible:
# Ignore source code which cannot be found:
ignore_errors = True
# Exit with status code 2 if under this percentage is covered:
......
......@@ -420,6 +420,179 @@ e.g. bridges@...) and sent from an ``@riseup.net``, ``@gmail.com``, or
You can email our BridgeDB instance `here <mailto:bridges@torproject.org>`__.
----------------------------
Accessing the Moat Interface
----------------------------
Moat is a bridge distributor for requesting bridges through `Tor Launcher's
<https://gitweb.torproject.org/tor-launcher.git/>`__ user interface.
The following describes the Moat API, version 0.1.0.
The client and server both MUST conform to `JSON-API <http://jsonapi.org/>`__.
The client SHOULD direct all requests via the Meek reflector at ``MEEK_REFECTOR``.
..
XXX meek reflector URL
Requesting Bridges
""""""""""""""""""
The client MUST send a ``POST /meek/moat/fetch`` containing the following JSON::
{
'data': {
'version': '0.1.0',
'type': 'client-transports',
'supported': [ 'TRANSPORT', 'TRANSPORT', ... ],
}
}
where:
* ``TRANSPORT`` is a string identifying a transport, e.g. ``"obfs3"`` or
``"obfs4"``. Currently supported transport identifiers are:
- ``"vanilla"``
- ``"fte"``
- ``"obfs3"``
- ``"obfs4"``
- ``"scramblesuit"``
Receiving a CAPTCHA challenge
"""""""""""""""""""""""""""""
The moat server will respond with ``200 OK``.
If there is no overlap with the transports which BridgeDB supports, the moat
server will respond with the list of transports which is *does* support::
{
'data': {
'version': '0.1.0',
'type': 'moat-transports',
'supported': [ 'TRANSPORT', 'TRANSPORT', ... ],
}
}
If there is an overlap with what BridgeDB supports, the moat server will select
the "best" transport from the list of supported transports, and respond with the
following JSON containing a CAPTCHA challenge::
{
'data': {
'id': 1,
'type': 'moat-challenge',
'version': '0.1.0',
'transport': TRANSPORT,
'image': CAPTCHA,
'challenge': CHALLENGE,
}
}
where:
* ``TRANSPORT`` is the agreed upon transport which will be distributed,
* ``CAPTCHA`` is a base64-encoded, jpeg image that is 400 pixels in
length and 125 pixels in height,
* ``CHALLENGE`` is a base64-encoded CAPTCHA challenge which MUST be
later passed back to the server along with the proposed solution.
The challenge contains an encrypted-then-HMACed timestamp, and
solutions submitted more than 30 minutes after requesting the CAPTCHA
are considered invalid.
Responding to a CAPTCHA challenge
"""""""""""""""""""""""""""""""""
To propose a solution to a CAPTCHA, the client MUST send a request for ``POST
/meek/moat/check``, where the body of the request contains the following JSON::
{
'data': {
'id': 2,
'type': 'moat-solution',
'version': '0.1.0',
'transport': TRANSPORT,
'challenge': CHALLENGE,
'solution': SOLUTION,
'qrcode': BOOLEAN,
}
}
where:
* ``TRANSPORT`` is the agreed upon transport which will be distributed,
* ``CHALLENGE`` is a base64-encoded CAPTCHA challenge which MUST be
later passed back to the server along with the proposed solution.
* ``SOLUTION`` is a valid unicode string, up to 20 bytes in length,
containing the client's answer (i.e. what characters the CAPTCHA
image displayed). The solution is *not* case-sensitive.
* ``BOOLEAN`` is ``'true'`` if the client wants a qrcode containing the bridge
lines to be generated and returned; ``'false'`` otherwise.
Receiving Bridges
"""""""""""""""""
If the ``CHALLENGE`` has already timed out, or if the ``SOLUTION`` was
incorrect, the server SHOULD respond with ``419 No You're A Teapot``.
If the ``SOLUTION`` was successful for the supplied ``CHALLENGE``, the
server responds ``200 OK`` with the following JSON::
{
'data': {
'id': 3,
'type': 'moat-bridges',
'version': '0.1.0',
'bridges': [ 'BRIDGE_LINE', ... ],
'qrcode': QRCODE,
}
}
where:
* ``BRIDGE_LINE`` is a bridge line suitable for configuration in a torrc,
* ``QRCODE`` is a base64-encoded jpeg image of a QRCode containing all the
``BRIDGE_LINE``, if one was requested, otherwise this field will be ``NaN``.
..
XXX do we care to differentiate the errors for "unable to distribute
bridges"? are any of these useful to Tor Launcher?
If the ``SOLUTION`` was successful for the supplied ``CHALLENGE``, but the
server is unable to distribute the requested Bridges, the server responds ``200
OK`` with the following JSON::
{
'error': {
'id': 1,
'code': '404',
'status': 'Not Found',
'title': 'Could not fetch the type of bridges you requested',
'detail': DETAILS,
}
}
where:
* ``DETAILS`` is some string describing the detailed nature of the issue.
Other Responses
"""""""""""""""
If the client requested some page other than ``/meek/moat/fetch``, or
``/meek/moat/check``, the server MUST respond with ``501 Not Implemented``.
If the client attempts any other HTTP method, other than ``POST``, the server
MUST respond ``403 FORBIDDEN``.
=================
Contact & Support
=================
......
......@@ -150,12 +150,6 @@ STATUS_FILE = "networkstatus-bridges"
#
IGNORE_NETWORKSTATUS = True
# Certificate file and private key for the HTTPS Distributor. To create a
# self-signed cert, run ``scripts/make-ssl-cert`` it will create these files
# in your current directory.
HTTPS_CERT_FILE="cert"
HTTPS_KEY_FILE="privkey.pem"
#----------------
# Output Files \ Where to store created data
#------------------------------------------------------------------------------
......@@ -293,16 +287,105 @@ SUPPORTED_TRANSPORTS = {
# menu).
DEFAULT_TRANSPORT = 'obfs4'
#-------------------------------
# Moat Distribution Options \
#------------------------------------------------------------------------------
#
# These options configure the behaviour of a web interface which speaks JSON API
# to a remote application in order to present said application with the
# necessary information for creating a user interface for bridge distribution
# mechanism, similar to the web interface of BridgeDB's HTTPS Distributor. If
# MOAT_DIST is enabled, make sure that the MOAT_CERT_FILE and MOAT_KEY_FILE
# options point to the correct location of your SSL certificate and key!
# ------------------------------------------------------------------------------
# (boolean) True to enable distribution via Moat; False otherwise.
MOAT_DIST = True
# (boolean) True to only allow Moat distribution via a Meek tunnel. False to
# only allow Moat distribution via untunneled HTTP(S).
MOAT_DIST_VIA_MEEK_ONLY = True
# Certificate file and private key for the Moar Distributor. To create a
# self-signed cert, run ``scripts/make-ssl-cert`` it will create these files
# in your current directory.
MOAT_TLS_CERT_FILE="moat-tls.crt"
MOAT_TLS_KEY_FILE="moat-tls.pem"
# (string) The Fully-Qualified Domain Name (FQDN) of the server that the Moat
# and/or HTTPS distributor(s) is/are publicly reachable at.
if MOAT_DIST_VIA_MEEK_ONLY:
MOAT_SERVER_PUBLIC_ROOT = '/meek/moat'
else:
MOAT_SERVER_PUBLIC_ROOT = '/moat'
# How many bridges do we give back in an answer (either HTTP or HTTPS)?
MOAT_BRIDGES_PER_ANSWER = 3
# (list) An ordered list of the preferred transports which moat should
# distribute, in order from most preferable to least preferable.
MOAT_TRANSPORT_PREFERENCE_LIST = ["obfs4", "vanilla"]
# (string or None) The IP address where we listen for HTTPS connections. If
# ``None``, listen on the default interface.
MOAT_HTTPS_IP = '127.0.0.1'
# (integer or None) The port to listen on for incoming HTTPS connections.
MOAT_HTTPS_PORT = 6791
# (string or None) The IP address to listen on for unencrypted HTTP
# connections. Set to ``None`` to disable unencrypted connections to the web
# interface.
MOAT_HTTP_IP = None
# (integer or None) The port to listen on for incoming HTTP connections.
MOAT_HTTP_PORT = None
# If true, there is a trusted proxy relaying incoming messages to us: take
# the *last* entry from its X-Forwarded-For header as the client's IP.
MOAT_USE_IP_FROM_FORWARDED_HEADER = True
# How many clusters do we group IPs in when distributing bridges based on IP?
# Note that if PROXY_LIST_FILES is set (below), what we actually do here
# is use one higher than the number here, and the extra cluster is used
# for answering requests made by IP addresses in the PROXY_LIST_FILES file.
MOAT_N_IP_CLUSTERS = 4
# (string or None) The period at which the available bridges rotates to a
# separate set of bridges. This setting can be used in the form
#
# "COUNT PERIOD" where
# COUNT is an integer
# PERIOD is one of "second", "minute", "hour", "day",
# "week", or "month" (or any plural form).
#
# For example, setting HTTPS_ROTATION_PERIOD = "3 days" will result in the set
# of bridges which are available through the web interface (either HTTP or
# HTTPS) getting rotated once every three days. Setting this to None disables
# rotation entirely.
MOAT_ROTATION_PERIOD = "3 hours"
# The location of the files which store the HMAC secret key and RSA keypair
# (for checking captcha responses):
MOAT_GIMP_CAPTCHA_HMAC_KEYFILE = 'moat_captcha_hmac_key'
MOAT_GIMP_CAPTCHA_RSA_KEYFILE = 'moat_captcha_rsa_key'
#-------------------------------
# HTTP(S) Distribution Options \
#------------------------------------------------------------------------------
#
# These options configure the behaviour of the web interface bridge
# distribution mechanism. If HTTPS_DIST is enabled, make sure that the above
# distribution mechanism. If HTTPS_DIST is enabled, make sure that the
# HTTPS_CERT_FILE and HTTPS_KEY_FILE options point to the correct location of
# your SSL certificate and key!
#------------------------------------------------------------------------------
# Certificate file and private key for the HTTPS Distributor. To create a
# self-signed cert, run ``scripts/make-ssl-cert`` it will create these files
# in your current directory.
HTTPS_CERT_FILE="cert"
HTTPS_KEY_FILE="privkey.pem"
# (string) The Fully-Qualified Domain Name (FQDN) of the server that the HTTP
# and/or HTTPS distributor(s) is/are publicly reachable at.
SERVER_PUBLIC_FQDN = 'bridges.torproject.org'
......@@ -585,14 +668,17 @@ EMAIL_GPG_PASSPHRASE_FILE = None
# Once a bridge is assigned to either of the first two groups, it stays there
# persistently. The bridges are allocated to these groups in a proportion of
#
# ``HTTPS_SHARE`` : ``EMAIL_SHARE`` : ``RESERVED_SHARE``
# ``MOAT_SHARE`` : ``HTTPS_SHARE`` : ``EMAIL_SHARE`` : ``RESERVED_SHARE``
# ------------------------------------------------------------------------------
# The proportion of bridges to allocate to Moat distribution.
MOAT_SHARE = 20
# The proportion of bridges to allocate to HTTP distribution.
HTTPS_SHARE = 10
# The proportion of bridges to allocate to Email distribution.
EMAIL_SHARE = 5
EMAIL_SHARE = 2
# An integer specifying the proportion of bridges which should remain
# unallocated, for backup usage and manual distribution.
......
......@@ -290,7 +290,6 @@ class GimpCaptcha(Captcha):
if hmacIsValid:
try:
answerBlob = secretKey.decrypt(encBlob)
timestamp = answerBlob[:12].lstrip('0')
then = cls.sched.nextIntervalStarts(int(timestamp))
now = int(time.time())
......
......@@ -110,6 +110,7 @@ def loadConfig(configFile=None, configCls=None):
for attr in ["DB_FILE", "DB_LOG_FILE", "MASTER_KEY_FILE", "PIDFILE",
"ASSIGNMENTS_FILE", "HTTPS_CERT_FILE", "HTTPS_KEY_FILE",
"MOAT_CERT_FILE", "MOAT_KEY_FILE",
"LOG_FILE", "COUNTRY_BLOCK_FILE",
"GIMP_CAPTCHA_DIR", "GIMP_CAPTCHA_HMAC_KEYFILE",
"GIMP_CAPTCHA_RSA_KEYFILE", "EMAIL_GPG_HOMEDIR",
......@@ -120,11 +121,18 @@ def loadConfig(configFile=None, configCls=None):
else:
setattr(config, attr, os.path.abspath(os.path.expanduser(setting)))
for attr in ["HTTPS_ROTATION_PERIOD", "EMAIL_ROTATION_PERIOD"]:
for attr in ["MOAT_ROTATION_PERIOD",
"HTTPS_ROTATION_PERIOD",
"EMAIL_ROTATION_PERIOD"]:
setting = getattr(config, attr, None) # Default to None
setattr(config, attr, setting)
for attr in ["IGNORE_NETWORKSTATUS", "CSP_ENABLED", "CSP_REPORT_ONLY",
for attr in ["IGNORE_NETWORKSTATUS",
"MOAT_CSP_ENABLED",
"MOAT_CSP_REPORT_ONLY",
"MOAT_CSP_INCLUDE_SELF",
"CSP_ENABLED",
"CSP_REPORT_ONLY",
"CSP_INCLUDE_SELF"]:
setting = getattr(config, attr, True) # Default to True
setattr(config, attr, setting)
......
"""Methods for distributing bridges."""
import email
import https
#import moat
"""Common resources and utilities for bridge distribution systems."""
# -*- coding: utf-8 ; test-case-name: bridgedb.test.test_distributors_common_http -*-
#
# This file is part of BridgeDB, a Tor bridge distribution system.
#
# :authors: please see included AUTHORS file
# :copyright: (c) 2017, The Tor Project, Inc.
# (c) 2017, Isis Lovecruft
# :license: see LICENSE for licensing information
"""
.. py:module:: bridgedb.distributors.common.http
:synopsis: Common utilities for HTTP-based distributors.
bridgedb.distributors.common.http
==================================
Common utilities for HTTP-based distributors.
"""
import logging
import os
from bridgedb.parse.addr import isIPAddress
#: The fully-qualified domain name for any and all web servers we run.
SERVER_PUBLIC_FQDN = None
def setFQDN(fqdn, https=True):
"""Set the global :data:`SERVER_PUBLIC_FQDN` variable.
:param str fqdn: The public, fully-qualified domain name of the HTTP
server that will serve this resource.
:param bool https: If ``True``, then ``'https://'`` will be prepended to
the FQDN. This is primarily used to create a
``Content-Security-Policy`` header that will only allow resources to
be sourced via HTTPS, otherwise, if ``False``, it allow resources to
be sourced via any transport protocol.
"""
if https:
fqdn = 'https://' + fqdn
logging.info("Setting HTTP server public FQDN to %r" % fqdn)
global SERVER_PUBLIC_FQDN
SERVER_PUBLIC_FQDN = fqdn
def getFQDN():
"""Get the setting for the HTTP server's public FQDN from the global
:data:`SERVER_PUBLIC_FQDN variable.
:rtype: str or None
"""
return SERVER_PUBLIC_FQDN
def getClientIP(request, useForwardedHeader=False):
"""Get the client's IP address from the ``'X-Forwarded-For:'``
header, or from the :api:`request <twisted.web.server.Request>`.
:type request: :api:`twisted.web.http.Request`
:param request: A ``Request`` for a :api:`twisted.web.resource.Resource`.
:param bool useForwardedHeader: If ``True``, attempt to get the client's
IP address from the ``'X-Forwarded-For:'`` header.
:rtype: ``None`` or :any:`str`
:returns: The client's IP address, if it was obtainable.
"""
ip = None
if useForwardedHeader:
header = request.getHeader("X-Forwarded-For")
if header:
ip = header.split(",")[-1].strip()
if not isIPAddress(ip):
logging.warn("Got weird X-Forwarded-For value %r" % header)
ip = None
else:
ip = request.getClientIP()
return ip
"""Servers for BridgeDB's email bridge distributor."""
# -*- coding: utf-8 -*-
#
# This file is part of BridgeDB, a Tor bridge distribution system.
#
# :authors: Isis Lovecruft 0xA3ADB67A2CDB8B35 <isis@torproject.org>
# please also see AUTHORS file
# :copyright: (c) 2012-2017 Isis Lovecruft
# (c) 2007-2017, The Tor Project, Inc.
# (c) 2007-2017, all entities within the AUTHORS file
# :license: 3-clause BSD, see included LICENSE for information
'''Package containing modules for BridgeDB's email bridge distributor.
.. py:module:: bridgedb.distributors.email
:synopsis: Package containing modules for BridgeDB's email bridge
distributor.
'''
import autoresponder
import distributor
......
......@@ -228,3 +228,6 @@ class EmailDistributor(Distributor):
# Since prepopulateRings is called every half hour when the bridge
# descriptors are re-parsed, we should clean the database then.
self.cleanDatabase()
logging.info("Bridges allotted for %s distribution: %d"
% (self.name, len(self.hashring)))
"""Servers for BridgeDB's HTTPS bridge distributor."""
import distributor
import request
import server
#import distributor
#import request
#import server
......@@ -262,6 +262,17 @@ class HTTPSDistributor(Distributor):
self.hashring.addRing(ring, filters, byFilters(filters),
populate_from=self.hashring.bridges)
logging.info("Bridges allotted for %s distribution: %d"
% (self.name, len(self.hashring)))
logging.info("\tNum bridges:\tFilter set:")
for (ringname, (filterFn, subring)) in self.hashring.filterRings.items():
filterSet = ' '.join(self.hashring.extractFilterNames(ringname))
logging.info("\t%2d bridges\t%s" % (len(subring), filterSet))
logging.info("Total subrings for %s: %d"
% (self.name, len(self.hashring.filterRings)))
def insert(self, bridge):
"""Assign a bridge to this distributor."""
self.hashring.insert(bridge)
......
......@@ -50,6 +50,9 @@ from bridgedb import crypto
from bridgedb import strings
from bridgedb import translations
from bridgedb import txrecaptcha
from bridgedb.distributors.common.http import setFQDN
from bridgedb.distributors.common.http import getFQDN
from bridgedb.distributors.common.http import getClientIP
from bridgedb.distributors.https.request import HTTPSBridgeRequest
from bridgedb.parse import headers
from bridgedb.parse.addr import isIPAddress
......@@ -60,8 +63,9 @@ from bridgedb.schedule import ScheduledInterval
from bridgedb.util import replaceControlChars
#: The path to the HTTPS distributor's web templates. (Should be the
#: "templates" directory in the same directory as this file.)
TEMPLATE_DIR = os.path.join(os.path.dirname(__file__), 'templates')
rtl_langs = ('ar', 'he', 'fa', 'gu_IN', 'ku')
# Setting `filesystem_checks` to False is recommended for production servers,
# due to potential speed increases. This means that the atimes of the Mako
......@@ -76,61 +80,9 @@ lookup = TemplateLookup(directories=[TEMPLATE_DIR],
collection_size=500)
logging.debug("Set template root to %s" % TEMPLATE_DIR)
#: This server's public, fully-qualified domain name.
SERVER_PUBLIC_FQDN = None
def setFQDN(fqdn, https=True):
"""Set the global :data:`SERVER_PUBLIC_FQDN` variable.
:param str fqdn: The public, fully-qualified domain name of the HTTP
server that will serve this resource.
:param bool https: If ``True``, then ``'https://'`` will be prepended to
the FQDN. This is primarily used to create a
``Content-Security-Policy`` header that will only allow resources to
be sourced via HTTPS, otherwise, if ``False``, it allow resources to
be sourced via any transport protocol.
"""
if https:
fqdn = 'https://' + fqdn
logging.info("Setting HTTP server public FQDN to %r" % fqdn)
global SERVER_PUBLIC_FQDN
SERVER_PUBLIC_FQDN = fqdn
def getFQDN():
"""Get the setting for the HTTP server's public FQDN from the global
:data:`SERVER_PUBLIC_FQDN variable.
:rtype: str or None
"""
return SERVER_PUBLIC_FQDN
def getClientIP(request, useForwardedHeader=False):
"""Get the client's IP address from the ``'X-Forwarded-For:'``
header, or from the :api:`request <twisted.web.server.Request>`.
:type request: :api:`twisted.web.http.Request`
:param request: A ``Request`` for a :api:`twisted.web.resource.Resource`.
:param bool useForwardedHeader: If ``True``, attempt to get the client's
IP address from the ``'X-Forwarded-For:'`` header.
:rtype: ``None`` or :any:`str`
:returns: The client's IP address, if it was obtainable.
"""
ip = None
if useForwardedHeader:
header = request.getHeader("X-Forwarded-For")
if header:
ip = header.split(",")[-1].strip()
if not isIPAddress(ip):
logging.warn("Got weird X-Forwarded-For value %r" % header)
ip = None
else:
ip = request.getClientIP()
#: Localisations which BridgeDB supports which should be rendered right-to-left.
rtl_langs = ('ar', 'he', 'fa', 'gu_IN', 'ku')
return ip
def replaceErrorPage(request, error, template_name=None, html=True):
"""Create a general error page for displaying in place of tracebacks.
......@@ -334,6 +286,7 @@ class ErrorResource(CSPResource):
render_POST = render_GET
resource404 = ErrorResource('error-404.html', code=404)
resource500 = ErrorResource('error-500.html', code=500)
maintenance = ErrorResource('error-503.html', code=503)
......
"""Servers for BridgeDB's Moat bridge distributor."""
#import distributor
#import server
# -*- coding: utf-8 ; test-case-name: bridgedb.test.test_moat_distributor -*-
#
# This file is part of BridgeDB, a Tor bridge distribution system.
#
# :authors: Nick Mathewson
# Isis Lovecruft 0xA3ADB67A2CDB8B35 <isis@torproject.org>
# Matthew Finkel 0x017DD169EA793BE2 <sysrqb@torproject.org>
# :copyright: (c) 2013-2017, Isis Lovecruft
# (c) 2013-2017, Matthew Finkel
# (c) 2007-2017, The Tor Project, Inc.
# :license: see LICENSE for licensing information
"""
bridgedb.distributors.moat.distributor
==========================
A Distributor that hands out bridges through a web interface.
.. inheritance-diagram:: MoatDistributor
:parts: 1