Unverified Commit 2ccf7ef9 authored by Philipp Winter's avatar Philipp Winter
Browse files

Merge branch 'release-0.10.0'

parents 9a7ae57c f9a9a9cd
......@@ -18,7 +18,7 @@ Babel==2.8.0
beautifulsoup4==4.8.2
Mako==1.1.1
pycryptodome==3.9.6
Twisted==19.10.0
Twisted==20.3.0
coverage==5.0.3
coveralls==1.10.0
gnupg==2.3.1
......
Changes in version 0.10.0 - 2020-04-01
* FIXES https://bugs.torproject.org/30317
Update our "howto" box, which explains how one adds bridges to Tor
Browser. In addition to updating the instructions, this patch also
links to instructions for Android.
* FIXES https://bugs.torproject.org/33631
So far, BridgeDB remembered only the first distribution mechanism it
ever learned for a given bridge. That means that if a bridge would
change its mind and re-configure its distribution mechanism using
BridgeDistribution, BridgeDB would ignore it. This patch changes this
behavior, so bridges can actually change their distribution mechanism.
* FIXES https://bugs.torproject.org/31967
Use a CSPRNG for selecting cached CAPTCHAs.
* FIXES https://bugs.torproject.org/33008
Add an info page, available at bridges.torproject.org/info. Relay
Search links to this info page to explain to bridge operators what their
bridge distribution mechanism means.
Changes in version 0.9.4 - 2020-02-19
* FIXES https://bugs.torproject.org/30946
......
......@@ -406,14 +406,14 @@ The client SHOULD direct all requests via the Meek reflector at ``MEEK_REFECTOR`
Requesting Bridges
""""""""""""""""""
The client MUST send a ``POST /meek/moat/fetch`` containing the following JSON::
The client MUST send a ``POST /moat/fetch`` containing the following JSON::
{
"data": {
"data": [{
"version": "0.1.0",
"type": "client-transports",
"supported": [ "TRANSPORT", "TRANSPORT", ... ],
}
}]
}
where:
......@@ -437,14 +437,14 @@ the "best" transport from the list of supported transports, and respond with the
following JSON containing a CAPTCHA challenge::
{
"data": {
"data": [{
"id": "1",
"type": "moat-challenge",
"version": "0.1.0",
"transport": "TRANSPORT",
"image": "CAPTCHA",
"challenge": "CHALLENGE",
}
}]
}
where:
......@@ -463,14 +463,14 @@ If there is no overlap with the transports which BridgeDB supports, the moat
server will respond with the list of transports which is *does* support::
{
"data": {
"data": [{
"id": "1",
"type": "moat-challenge",
"version": "0.1.0",
"transport": [ "TRANSPORT", "TRANSPORT", ... ],
"image": "CAPTCHA",
"challenge": "CHALLENGE",
}
}]
}
......@@ -478,10 +478,10 @@ Responding to a CAPTCHA challenge
"""""""""""""""""""""""""""""""""
To propose a solution to a CAPTCHA, the client MUST send a request for ``POST
/meek/moat/check``, where the body of the request contains the following JSON::
/moat/check``, where the body of the request contains the following JSON::
{
"data": {
"data": [{
"id": "2",
"type": "moat-solution",
"version": "0.1.0",
......@@ -489,7 +489,7 @@ To propose a solution to a CAPTCHA, the client MUST send a request for ``POST
"challenge": "CHALLENGE",
"solution": "SOLUTION",
"qrcode": "BOOLEAN",
}
}]
}
......@@ -515,13 +515,13 @@ If the ``SOLUTION`` was successful for the supplied ``CHALLENGE``, the
server responds ``200 OK`` with the following JSON::
{
"data": {
"data": [{
"id": "3",
"type": "moat-bridges",
"version": "0.1.0",
"bridges": [ "BRIDGE_LINE", ... ],
"qrcode": "QRCODE",
}
}]
}
where:
......@@ -557,8 +557,8 @@ where:
Other Responses
"""""""""""""""
If the client requested some page other than ``/meek/moat/fetch``, or
``/meek/moat/check``, the server MUST respond with ``501 Not Implemented``.
If the client requested some page other than ``/moat/fetch``, or
``/moat/check``, the server MUST respond with ``501 Not Implemented``.
If the client attempts any other HTTP method, other than ``POST``, the server
MUST respond ``403 FORBIDDEN``.
......
......@@ -180,6 +180,14 @@ class BridgeRing(object):
for tp, val, count, subring in self.subrings:
subring.clear()
def remove(self, bridge):
"""Remove a **bridge** from this hashring."""
for tp, val, _, subring in self.subrings:
subring.remove(bridge)
pos = self.hmac(bridge.identity)
if pos in self.bridges:
del self.bridges[pos]
def insert(self, bridge):
"""Add a **bridge** to this hashring.
......@@ -364,46 +372,6 @@ class BridgeRing(object):
f.write("%s %s\n" % (b.fingerprint, " ".join(desc).strip()))
class FixedBridgeSplitter(object):
"""Splits bridges up based on an HMAC and assigns them to one of several
subhashrings with equal probability.
"""
def __init__(self, key, rings):
self.hmac = getHMACFunc(key, hex=True)
self.rings = rings[:]
def insert(self, bridge):
# Grab the first 4 bytes
digest = self.hmac(bridge.identity)
pos = int( digest[:8], 16 )
which = pos % len(self.rings)
self.rings[which].insert(bridge)
def clear(self):
"""Clear all bridges from every ring in ``rings``."""
for r in self.rings:
r.clear()
def __len__(self):
"""Returns the total number of bridges in all ``rings``."""
total = 0
for ring in self.rings:
total += len(ring)
return total
def dumpAssignments(self, filename, description=""):
"""Write all bridges assigned to this hashring to ``filename``.
:param string description: If given, include a description next to the
index number of the ring from :attr:`FilteredBridgeSplitter.rings`
the following bridges were assigned to. For example, if the
description is ``"IPv6 obfs2 bridges"`` the line would read:
``"IPv6 obfs2 bridges ring=3"``.
"""
for index, ring in zip(range(len(self.rings)), self.rings):
ring.dumpAssignments(filename, "%s ring=%s" % (description, index))
class UnallocatedHolder(object):
"""A pseudo-bridgeholder that ignores its bridges and leaves them
unassigned.
......@@ -411,8 +379,17 @@ class UnallocatedHolder(object):
def __init__(self):
self.fingerprints = []
def remove(self, bridge):
logging.debug("Removing %s from unallocated" % bridge.fingerprint)
i = -1
try:
i = self.fingerprints.index(bridge.fingerprint)
except ValueError:
return
del self.fingerprints[i]
def insert(self, bridge):
logging.debug("Leaving %s unallocated", bridge.fingerprint)
logging.debug("Leaving %s unallocated" % bridge.fingerprint)
if not bridge.fingerprint in self.fingerprints:
self.fingerprints.append(bridge.fingerprint)
......@@ -488,43 +465,52 @@ class BridgeSplitter(object):
return
validRings = self.rings
distribution_method = None
distribution_method = orig_method = None
# If the bridge already has a distributor, use that.
with bridgedb.Storage.getDB() as db:
distribution_method = db.getBridgeDistributor(bridge, validRings)
orig_method = db.getBridgeDistributor(bridge, validRings)
if orig_method is not None:
distribution_method = orig_method
logging.info("So far, bridge %s was in hashring %s" %
(bridge, orig_method))
# Check if the bridge requested a distribution method and if so, try to
# use it.
if bridge.distribution_request:
distribution_method = bridge.distribution_request
logging.info("Bridge %s requested placement in hashring %s" %
(bridge, distribution_method))
# Is the bridge requesting a distribution method that's different
# from the one we have on record? If so, we have to delete it from
# its previous ring.
if orig_method is not None and \
orig_method != distribution_method and \
distribution_method in (validRings + ["none"]):
logging.info("Bridge %s is in %s but wants to be in %s." %
(bridge, orig_method, distribution_method))
prevRing = self.ringsByName.get(orig_method)
prevRing.remove(bridge)
# If they requested not to be distributed, honor the request:
if distribution_method == "none":
logging.info("Bridge %s requested to not be distributed." % bridge)
return
if distribution_method:
logging.info("%s bridge %s was already in hashring %s" %
(self.__class__.__name__, bridge, distribution_method))
else:
# Check if the bridge requested a specific distribution method.
if bridge.distribution_request:
distribution_method = bridge.distribution_request
logging.info("%s bridge %s requested placement in hashring %s"
% (self.__class__.__name__, bridge,
distribution_method))
# If they requested not to be distributed, honor the request:
if distribution_method == "none":
logging.info("%s bridge %s requested to not be distributed."
% (self.__class__.__name__, bridge))
return
# If we didn't know what they are talking about, or they requested
# "any" distribution method, and we've never seen this bridge
# before, then determine where to place it.
if ((distribution_method not in validRings) or
(distribution_method == "any")):
pos = self.hmac(bridge.identity)
n = int(pos[:8], 16) % self.totalP
pos = bisect.bisect_right(self.pValues, n) - 1
assert 0 <= pos < len(self.rings)
distribution_method = self.rings[pos]
logging.info(("%s placing bridge %s into hashring %s (via n=%s,"
" pos=%s).") % (self.__class__.__name__, bridge,
distribution_method, n, pos))
# If we didn't know what they are talking about, or they requested
# "any" distribution method, and we've never seen this bridge
# before, then deterministically determine where to place it.
if ((distribution_method not in validRings) or
(distribution_method == "any")):
pos = self.hmac(bridge.identity)
n = int(pos[:8], 16) % self.totalP
pos = bisect.bisect_right(self.pValues, n) - 1
assert 0 <= pos < len(self.rings)
distribution_method = self.rings[pos]
logging.info(("%s placing bridge %s into hashring %s (via n=%s,"
" pos=%s).") % (self.__class__.__name__, bridge,
distribution_method, n, pos))
with bridgedb.Storage.getDB() as db:
ringname = db.insertBridgeAndGetRing(bridge, distribution_method,
......@@ -567,7 +553,8 @@ class FilteredBridgeSplitter(object):
I-guess-it-passes-for-some-sort-of-hashring classes in this
module.
:ivar hmac: DOCDOC
:ivar bridges: DOCDOC
:ivar bridges: A dictionary mapping a bridge's fingerprint to its
:class:`~bridgedb.bridges.Bridge` object.
:type distributorName: str
:ivar distributorName: The name of this splitter's distributor. See
:meth:`~bridgedb.distributors.https.distributor.HTTPSDistributor.setDistributorName`.
......@@ -575,7 +562,7 @@ class FilteredBridgeSplitter(object):
self.key = key
self.filterRings = {}
self.hmac = getHMACFunc(key, hex=True)
self.bridges = []
self.bridges = {}
self.distributorName = ''
#XXX: unused
......@@ -585,9 +572,30 @@ class FilteredBridgeSplitter(object):
return len(self.bridges)
def clear(self):
self.bridges = []
self.bridges = {}
self.filterRings = {}
def remove(self, bridge):
"""Remove a bridge from all appropriate sub-hashrings.
:type bridge: :class:`~bridgedb.bridges.Bridge`
:param bridge: The bridge to remove.
"""
logging.debug("Removing %s from hashring..." % bridge)
try:
del self.bridges[bridge.fingerprint]
except KeyError:
logging.warn("Was asked to remove non-existant bridge %s "
"from ring." % bridge)
return
for ringname, (filterFn, subring) in self.filterRings.items():
if filterFn(bridge):
subring.remove(bridge)
logging.debug("Removed bridge %s from %s subhashring." %
(bridge, ringname))
def insert(self, bridge):
"""Insert a bridge into all appropriate sub-hashrings.
......@@ -603,15 +611,9 @@ class FilteredBridgeSplitter(object):
"bridge: %s") % bridge)
return
index = 0
logging.debug("Inserting %s into hashring..." % bridge)
for old_bridge in self.bridges[:]:
if bridge.fingerprint == old_bridge.fingerprint:
self.bridges[index] = bridge
break
index += 1
else:
self.bridges.append(bridge)
self.bridges[bridge.fingerprint] = bridge
for ringname, (filterFn, subring) in self.filterRings.items():
if filterFn(bridge):
subring.insert(bridge)
......@@ -695,7 +697,7 @@ class FilteredBridgeSplitter(object):
if populate_from:
inserted = 0
for bridge in populate_from:
for bridge in populate_from.values():
if isinstance(bridge, Bridge) and filterFn(bridge):
subring.insert(bridge)
inserted += 1
......@@ -709,7 +711,7 @@ class FilteredBridgeSplitter(object):
# bridges may be present in multiple filter sets
# only one line should be dumped per bridge
for b in self.bridges:
for b in self.bridges.values():
# gather all the filter descriptions
desc = []
for n,(g,r) in self.filterRings.items():
......
......@@ -166,9 +166,8 @@ class Database(object):
def insertBridgeAndGetRing(self, bridge, setRing, seenAt, validRings,
defaultPool="unallocated"):
'''Updates info about bridge, setting ring to setRing if none was set.
Also sets distributor to `defaultPool' if the bridge was found in
the database, but its distributor isn't valid anymore.
'''Updates info about bridge, setting ring to setRing. Also sets
distributor to `defaultPool' if setRing isn't a valid ring.
Returns the name of the distributor the bridge is assigned to.
'''
......@@ -178,26 +177,22 @@ class Database(object):
h = bridge.fingerprint
assert len(h) == HEX_ID_LEN
cur.execute("SELECT id, distributor "
"FROM Bridges WHERE hex_key = ?", (h,))
# Check if this is currently a valid ring name. If not, move into
# default pool.
if setRing not in validRings:
setRing = defaultPool
cur.execute("SELECT id FROM Bridges WHERE hex_key = ?", (h,))
v = cur.fetchone()
if v is not None:
i, ring = v
# Check if this is currently a valid ring name. If not, move back
# into default pool.
if ring not in validRings:
ring = defaultPool
bridgeId = v[0]
# Update last_seen, address, port and (possibly) distributor.
cur.execute("UPDATE Bridges SET address = ?, or_port = ?, "
"distributor = ?, last_seen = ? WHERE id = ?",
(str(bridge.address), bridge.orPort, ring,
timeToStr(seenAt), i))
return ring
(str(bridge.address), bridge.orPort, setRing,
timeToStr(seenAt), bridgeId))
return setRing
else:
# Check if this is currently a valid ring name. If not, move back
# into default pool.
if setRing not in validRings:
setRing = defaultPool
# Insert it.
cur.execute("INSERT INTO Bridges (hex_key, address, or_port, "
"distributor, first_seen, last_seen) "
......
......@@ -389,7 +389,7 @@ class GimpCaptcha(Captcha):
and a challenge string (used for checking the client's solution).
"""
try:
imageFilename = random.choice(os.listdir(self.cacheDir))
imageFilename = random.SystemRandom().choice(os.listdir(self.cacheDir))
imagePath = os.path.join(self.cacheDir, imageFilename)
with open(imagePath, 'rb') as imageFile:
self.image = imageFile.read()
......
......@@ -96,14 +96,9 @@ def addHowto(template):
"""
howToTBB = template.gettext(strings.HOWTO_TBB[1]) % strings.EMAIL_SPRINTF["HOWTO_TBB1"]
howToTBB += u'\n\n'
howToTBB += template.gettext(strings.HOWTO_TBB[2])
howToTBB += u'\n\n'
howToTBB += u'\n'.join(["> {0}".format(ln) for ln in
template.gettext(strings.HOWTO_TBB[3]).split('\n')])
howToTBB += u'\n\n'
howToTBB += template.gettext(strings.HOWTO_TBB[4])
howToTBB += u'\n\n'
howToTBB += strings.EMAIL_REFERENCE_LINKS.get("HOWTO_TBB1")
howToTBB += strings.EMAIL_REFERENCE_LINKS.get("HOWTO_TBB2")
howToTBB += strings.EMAIL_REFERENCE_LINKS.get("HOWTO_TBB3")
howToTBB += u'\n\n'
return howToTBB
......
......@@ -389,13 +389,14 @@ class TranslatedTemplateResource(CustomErrorHandlingResource, CSPResource):
"""
isLeaf = True
def __init__(self, template=None):
def __init__(self, template=None, showFaq=True):
"""Create a new :api:`Resource <twisted.web.resource.Resource>` for a
Mako-templated webpage.
"""
gettext.install("bridgedb")
CSPResource.__init__(self)
self.template = template
self.showFaq = showFaq
def render_GET(self, request):
self.setCSPHeader(request)
......@@ -409,7 +410,8 @@ class TranslatedTemplateResource(CustomErrorHandlingResource, CSPResource):
getSortedLangList(),
rtl=rtl,
lang=langs[0],
langOverride=translations.isLangOverridden(request))
langOverride=translations.isLangOverridden(request),
showFaq=self.showFaq)
except Exception as err: # pragma: no cover
rendered = replaceErrorPage(request, err)
request.setHeader("Content-Type", "text/html; charset=utf-8")
......@@ -435,6 +437,11 @@ class OptionsResource(TranslatedTemplateResource):
TranslatedTemplateResource.__init__(self, 'options.html')
class InfoResource(TranslatedTemplateResource):
def __init__(self):
TranslatedTemplateResource.__init__(self, 'info.html', showFaq=False)
class HowtoResource(TranslatedTemplateResource):
"""A resource which explains how to use bridges."""
......@@ -1133,6 +1140,7 @@ def addWebServer(config, distributor):
index = IndexResource()
options = OptionsResource()
howto = HowtoResource()
info = InfoResource()
robots = static.File(os.path.join(TEMPLATE_DIR, 'robots.txt'))
assets = static.File(os.path.join(TEMPLATE_DIR, 'assets/'))
keys = static.Data(strings.BRIDGEDB_OPENPGP_KEY.encode('utf-8'), 'text/plain')
......@@ -1148,6 +1156,7 @@ def addWebServer(config, distributor):
root.putChild(b'assets', assets)
root.putChild(b'options', options)
root.putChild(b'howto', howto)
root.putChild(b'info', info)
root.putChild(b'maintenance', maintenance)
root.putChild(b'error', resource500)
root.putChild(CSPResource.reportURI, csp)
......
## -*- coding: utf-8 -*-
<%namespace name="base" file="base.html" inheritable="True"/>
<%page args="strings, langs, rtl=False, lang='en', langOverride=False, **kwargs"/>
<%page args="strings, langs, rtl=False, lang='en', langOverride=False, showFaq=True, **kwargs"/>
<!DOCTYPE html>
<html lang="${lang}">
......@@ -59,6 +59,7 @@
${next.body(strings, langs, rtl=rtl, lang=lang, langOverride=langOverride, **kwargs)}
% if showFaq:
<div class="faq">
<div class="row-fluid marketing">
......@@ -84,6 +85,7 @@ ${next.body(strings, langs, rtl=rtl, lang=lang, langOverride=langOverride, **kwa
</p>
</div>
</div> <!-- end faq -->
% endif
<div class="footer footer-small">
<hr>
......
......@@ -91,21 +91,15 @@ ${_("""This QRCode contains your bridge lines. Scan it with a QRCode """ \
<div class="container-fluid" id="howto">
<p>
${_(strings.HOWTO_TBB[1]) % \
("""<a href="https://www.torproject.org/projects/torbrowser.html"
("""<a href="https://www.torproject.org/download/"
target="_blank">""",
"""</a>""",
"""<a href="https://tb-manual.torproject.org/bridges/#entering-bridge-addresses"
target="_blank">""",
"""</a>""",
"""<a href="https://tb-manual.torproject.org/mobile-tor/#circumvention"
target="_blank">""",
"""</a>""")}
${_(strings.HOWTO_TBB[2])}
</p>
<br />
<div class="bs-component">
<blockquote>
<p>
${_(strings.HOWTO_TBB[3])}
</p>
</blockquote>
</div>
<p>
${_(strings.HOWTO_TBB[4])}
</p>
</div>
</div>
......
......@@ -15,21 +15,15 @@
<div class="container-fluid" id="howto">
<p>
${_(strings.HOWTO_TBB[1]) % \
("""<a href="https://www.torproject.org/projects/torbrowser.html"
("""<a href="https://www.torproject.org/download/"
target="_blank">""",
"""</a>""",
"""<a href="https://tb-manual.torproject.org/bridges/#entering-bridge-addresses"
target="_blank">""",
"""</a>""",
"""<a href="https://tb-manual.torproject.org/mobile-tor/#circumvention"
target="_blank">""",
"""</a>""")}
${_(strings.HOWTO_TBB[2])}
</p>
<br />
<div class="bs-component">
<blockquote>
<p>
${_(strings.HOWTO_TBB[3])}
</p>
</blockquote>
</div>
<p>
${_(strings.HOWTO_TBB[4])}
</p>
</div>
</div>
......
## -*- coding: utf-8 -*-
<%inherit file="base.html"/>
<%page args="strings, langs, rtl=False, lang='en', langOverride=False, **kwargs"/>
<div class="container-fluid container-fluid-outer-96">
<div class="container-fluid container-fluid-inner">
<p>
<h3>${_(strings.BRIDGEDB_INFO[0])}</h3>
<p>${_(strings.BRIDGEDB_INFO[1]) % \
("""<a href="https://metrics.torproject.org/bridgedb-distributor.html">""",
"""</a>""")}</p>
<div class="row-fluid marketing">
<h4><a name="https">HTTPS</a></h4>
<p>${_(strings.BRIDGEDB_INFO[2]) % \
("""<a href="https://bridges.torproject.org/options">""",