BridgeDB issueshttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues2023-10-12T14:24:01Zhttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/40066BridgeDB requires outdated packages with known CVEs2023-10-12T14:24:01Zmeskiomeskio@torproject.orgBridgeDB requires outdated packages with known CVEsSelected vulnerable packages:
* Twisted 21.7.0
* Mechanize 0.4.5
* Pillow 8.2.0
* Werkzeug 2.2.2Selected vulnerable packages:
* Twisted 21.7.0
* Mechanize 0.4.5
* Pillow 8.2.0
* Werkzeug 2.2.2meskiomeskio@torproject.orgmeskiomeskio@torproject.org2023-05-31https://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/40061Refresh captchas2022-10-14T06:28:29Zmeskiomeskio@torproject.orgRefresh captchasIt looks like there is a spike of usage, let's refresh the captchas.It looks like there is a spike of usage, let's refresh the captchas.Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetmeskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/40050Change the 'select all' button for a 'copy all' when giving the bridges2022-08-17T19:29:57ZemmapeelChange the 'select all' button for a 'copy all' when giving the bridgesThis change has been suggested by a translator.
It will be even more comfortable to have the bridges copied when clicking the button, than just selected and then you have to do the copying. Lets save users a step, and copy the bridges f...This change has been suggested by a translator.
It will be even more comfortable to have the bridges copied when clicking the button, than just selected and then you have to do the copying. Lets save users a step, and copy the bridges for them:
![copyall](/uploads/1745238b654be298dfa507734a4352bd/copyall.png)Sponsor 30 - Objective 2.2https://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/40046add description of settings and telegram distribution mechanisms2022-05-04T11:28:58Zmeskiomeskio@torproject.orgadd description of settings and telegram distribution mechanismshttps://bridges.torproject.org/info is missing a description of the new distribution mechanisms, and the metrics relay search points to it.https://bridges.torproject.org/info is missing a description of the new distribution mechanisms, and the metrics relay search points to it.Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetmeskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/40042filtered ring names are not readable2022-01-13T15:00:10Zmeskiomeskio@torproject.orgfiltered ring names are not readableSince !23 got merged ring names look like `frozenset({<function bySubring.<locals>._bySubring at 0x7ba09d305c10>, <function byIPv.<locals>._byIPv at 0x7ba0a63d1dc0>})`. Before the merge they were the ring name: moat, https, email, ...
S...Since !23 got merged ring names look like `frozenset({<function bySubring.<locals>._bySubring at 0x7ba09d305c10>, <function byIPv.<locals>._byIPv at 0x7ba0a63d1dc0>})`. Before the merge they were the ring name: moat, https, email, ...
Somewhere the name is being messed up.Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetmeskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/40038regnerate the captchas2022-06-07T08:42:53Zmeskiomeskio@torproject.orgregnerate the captchasThe existing captchas are over [a year old](https://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/24607#note_2599604), we might make the life of the censor harder regenerating them.The existing captchas are over [a year old](https://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/24607#note_2599604), we might make the life of the censor harder regenerating them.meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/40036get back the country blocking mechanims2022-05-12T16:38:59Zmeskiomeskio@torproject.orgget back the country blocking mechanimsIt was remove by the move to rdsys and is useful to block bridges by country:
https://gitlab.torproject.org/meskio/bridgedb/-/commit/6eac9c00b0809a3277ce6abe019514be17b0cf13It was remove by the move to rdsys and is useful to block bridges by country:
https://gitlab.torproject.org/meskio/bridgedb/-/commit/6eac9c00b0809a3277ce6abe019514be17b0cf13Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetmeskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/40027Use leftmost address when parsing `X-Forwarded-For` header for client IP2021-08-16T23:33:27ZCecylia BocovichUse leftmost address when parsing `X-Forwarded-For` header for client IPWhen a client passes through multiple proxies, each subsequent address is appended to the X-Forwarded-For header, resulting in a comma-separated list of IP addresses:
```
X-Forwarded-For: <client>, <proxy1>, <proxy2>
```
Right now Bridg...When a client passes through multiple proxies, each subsequent address is appended to the X-Forwarded-For header, resulting in a comma-separated list of IP addresses:
```
X-Forwarded-For: <client>, <proxy1>, <proxy2>
```
Right now BridgeDB only looks for the client's IP in the rightmost address
```
if useForwardedHeader:
header = request.getHeader("X-Forwarded-For")
if header:
index = -1
ip = header.split(",")[index].strip()
if skipLoopback:
logging.info(("Parsing X-Forwarded-For again, ignoring "
"loopback addresses..."))
while isLoopback(ip):
index -= 1
ip = header.split(",")[index].strip()
if not skipLoopback and isLoopback(ip):
logging.warn("Accepting loopback address: %s" % ip)
else:
if not isIPAddress(ip):
logging.warn("Got weird X-Forwarded-For value %r" % header)
ip = None
```
This causes trouble with our Moat and Apache ProxyPass setup, which results in X-Forwarded-For headers like the following:
```
X-Forwarded-For: <client>, ... <proxies> ... <local address>
```
I think we should modify this to parse the addresses from left to right, ignoring loopback/internal addresses, until we find a valid address for the client.
This is a follow-up modification for #32276 and a prerequisite for #40025.Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/40014HTML injection vulnerability with lang parameter2021-06-01T13:56:41ZCecylia BocovichHTML injection vulnerability with lang parameterWe just got the following email through the hackerone:
```
Hlo Sir,
I want to report the vulnerability and possible bypass methods ...i
found on your site https://torproject.org
This vulnerability is much more html injection and possi...We just got the following email through the hackerone:
```
Hlo Sir,
I want to report the vulnerability and possible bypass methods ...i
found on your site https://torproject.org
This vulnerability is much more html injection and possible xss that may
be used by hackers in order to harm others
for phising purpose...
URL: https://bridges.torproject.org/options?lang=anonyks
this is the vulnerable url where i got my vulnerability,
the parameter is lang= (any string) [here is used AnonyKs]
then after payload is used is : "><h1>Giveaway:-P <a
href="//evil.am">CLICK ME</a> </h1>
so the url become:
URL:
https://bridges.torproject.org/options?lang=anonyks"><h1>Giveaway:-P <a
href="//evil.com">CLICK ME</a> </h1>
copy the url and paste it in the browser
and click on CLICK ME [ there 'click me' is show in three different
places and each redirect to evil.com]
You may fix the upper vulnerability but still there can be other ways
that hacker can use
so...
Now regarding bypass/other possible ways :
URL:
https://bridges.torproject.org/options?lang=anonyks"><h1>Giveaway:-P <a
href="%2f%2fevil.com">CLICK ME</a> </h1>
URL:
https://bridges.torproject.org/options?lang=anonyks"><h1>Giveaway:-P <a
href="http://evil.com">CLICK ME</a> </h1>
now econding,
URL:
https://bridges.torproject.org/options?lang=anonyks%22%3E%3Ch1%3EGiveaway:P%20%3CA%20HREF=%22http://evil.com/%22%3EClickMe%3C/A%3E%20%3C/h1%3E
URL:
https://bridges.torproject.org/options?lang=anonyks%22%3E%3Ch1%3EGiveaway:P%20%3CA%20HREF=%22//evil.com/%22%3EClickMe%3C/A%3E%20%3C/h1%3E
URL:
https://bridges.torproject.org/options?lang=anonyks%22%3E%3Ch1%3EGiveaway:P%20%3CA%20HREF=%22%2f%2fevil.com/%22%3EClickMe%3C/A%3E%20%3C/h1%3E
There are more ways , now they are without <h1> html tag
URL: https://bridges.torproject.org/options?lang=anonyks"><a
href="http://evil.com">Click Me</a>
URL: https://bridges.torproject.org/options?lang=anonyks"><a
href="//evil.com">Click Me</a>
now again encoding them,
URL:
https://bridges.torproject.org/options?lang=anonyks%22%3E%3CA%20HREF=%22http://evil.com/%22%3EClickMe%3C/A%3E
URL:
https://bridges.torproject.org/options?lang=anonyks%22%3E%3CA%20HREF=%22//evil.com/%22%3EClickMe%3C/A%3E
Impact
Alogethee Today i submitted 10 vulnerability with possible bypass or
possible method that an attactr can use for crime purpose..
So in sum up all urls are:
Url: https://bridges.torproject.org/options?lang=anonyks"><a
href="//evil.com">Click Me</a>
Url: https://bridges.torproject.org/options?lang=anonyks"><a
href="http://evil.com">Click Me</a>
Url:
https://bridges.torproject.org/options?lang=anonyks"><h1>Giveaway:-P <a
href="%2f%2fevil.com">CLICK ME</a> </h1>
Url:
https://bridges.torproject.org/options?lang=anonyks"><h1>Giveaway:-P <a
href="//evil.com">CLICK ME</a> </h1>
Url:
https://bridges.torproject.org/options?lang=anonyks"><h1>Giveaway:-P <a
href="http://evil.com">CLICK ME</a> </h1>
Url:
https://bridges.torproject.org/options?lang=anonyks%22%3E%3CA%20HREF=%22//evil.com/%22%3EClickMe%3C/A%3E
Url:
https://bridges.torproject.org/options?lang=anonyks%22%3E%3CA%20HREF=%22http://evil.com/%22%3EClickMe%3C/A%3E
Url:
https://bridges.torproject.org/options?lang=anonyks%22%3E%3Ch1%3EGiveaway:P%20%3CA%20HREF=%22//evil.com/%22%3EClickMe%3C/A%3E%20%3C/h1%3E
Url:
https://bridges.torproject.org/options?lang=anonyks%22%3E%3Ch1%3EGiveaway:P%20%3CA%20HREF=%22http://evil.com/%22%3EClickMe%3C/A%3E%20%3C/h1%3E
Url:
https://bridges.torproject.org/options?lang=anonyks%22%3E%3Ch1%3EGiveaway:P%20%3CA%20HREF=%22%2f%2fevil.com/%22%3EClickMe%3C/A%3E%20%3C/h1%3E
Impact:
The vulnerability allow a malicious user to inject html tags and execute
Javascript which could lead to steal user's session, peform CSRF attacks
or open a phishing page.
Broadly,
When the input fields are not properly sanitized over in a webpage, thus
sometimes this HTML Injection vulnerability might lead us to Cross-Site
Scripting(XSS) or Server-Side Request Forgery(SSRF) attacks
```Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/40013Lost translations on the last release2021-07-16T10:14:43Zmeskiomeskio@torproject.orgLost translations on the last release`bridgedb.test.test_https_server.HowtoResourceTests.test_HowtoResource_render_GET_lang_ru` is [failing in the main branch](https://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/jobs/20874). It did start failing after we [updated t...`bridgedb.test.test_https_server.HowtoResourceTests.test_HowtoResource_render_GET_lang_ru` is [failing in the main branch](https://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/jobs/20874). It did start failing after we [updated the translated strings](https://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/commit/ff2cb23bacc6630122b4b753999347285c285613). It looks like the [rusian translation got deleted](https://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/commit/ff2cb23bacc6630122b4b753999347285c285613#d22d6f103948abadb3babee394cc2fcde50b7cbe).Sponsor 30 - Objective 2.4meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/40006Most bridges don't have a transport type2021-03-02T15:43:03ZCecylia BocovichMost bridges don't have a transport typeLooking at the recent bridge [pool assignment stats](https://metrics.torproject.org/collector.html#bridge-pool-assignments), the vast majority of bridges aren't reporting a transport type.
```
$ grep "obfs4" Downloads/2021-02-07-23-30-2...Looking at the recent bridge [pool assignment stats](https://metrics.torproject.org/collector.html#bridge-pool-assignments), the vast majority of bridges aren't reporting a transport type.
```
$ grep "obfs4" Downloads/2021-02-07-23-30-26 | wc -l
181
$ wc -l Downloads/2021-02-07-23-30-26
1547 Downloads/2021-02-07-23-30-26
```
I noticed this when I used the advance options to try requesting an obfs4 bridge with IPv6 and got a message saying there were no bridges in that ring. If I set transport type to `none` I get a vanilla bridge but no obfs4 bridges. I asked @phw about this the other day and he said that almost all of our bridges should be obfs4. If that's the case, there seems to be a bug somewhere.Sponsor 30 - Objective 2.2Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/40025Make available our censorship snapshot over a domain-fronted URL2022-03-25T10:35:40ZPhilipp Winterphw@torproject.orgMake available our censorship snapshot over a domain-fronted URLOver at tpo/community/outreach#28531, we're working on a [censorship snapshot](https://gitlab.torproject.org/tpo/anti-censorship/state-of-censorship/) that summarises where and how Tor is blocked. The idea is that Tor Browser will use th...Over at tpo/community/outreach#28531, we're working on a [censorship snapshot](https://gitlab.torproject.org/tpo/anti-censorship/state-of-censorship/) that summarises where and how Tor is blocked. The idea is that Tor Browser will use this snapshot to help the user figure out what circumvention method works. The question is: how does Tor Browser get this snapshot? It's not a good idea to hard-code it into Tor Browser because it will take a long time to update it. It's probably better for Tor Browser to fetch the snapshot each time it starts.
To guarantee that Tor Browser can always get the snapshot, even in the face of censorship, we should serve it over a domain-fronted endpoint, like we do for moat. There are several ways in which we could serve the snapshot:
* Polyanthum, the host that runs BridgeDB and rdsys, runs an Apache reverse proxy. We could serve the snapshot directly over Apache, which is probably the simplest solution.
* We could also teach rdsys how to serve the snapshot. Technically, the snapshot is a resource and rdsys is our resource distribution system.
Whatever solution we go with, we should make sure that we can easily and quickly update the snapshot if we learn of a new censorship event.Sponsor 96: Rapid Expansion of Access to the Uncensored Internet through Tor in China, Hong Kong, & Tibetmeskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/40001BridgeDB pretends to reload descriptors but actually doesn't2020-07-07T15:58:28ZPhilipp Winterphw@torproject.orgBridgeDB pretends to reload descriptors but actually doesn'tBridgeDB is supposed to reload its bridges descriptors every 30 minutes. The following cron job takes care of that (you can see it by running `crontab -l` as the bridgedb user on polyanthum):
```
*/30 * * * * bin/reload-bridgedb >/dev/n...BridgeDB is supposed to reload its bridges descriptors every 30 minutes. The following cron job takes care of that (you can see it by running `crontab -l` as the bridgedb user on polyanthum):
```
*/30 * * * * bin/reload-bridgedb >/dev/null 2>&1
```
Among other things, the script `reload-bridgedb` does the following:
```
[...]
bridgedb -c ${BRIDGEDB_CONFIG} --reload
[...]
```
BridgeDB implements the `--reload` command line switch but the switch doesn't actually reload anything. In fact, it does nothing at all. So when the cron job runs bridgedb with the `--reload` switch every 30 minutes, it starts a separate (!) process that (re)loads the bridge descriptors and then exits because it cannot bind to the ports that the original BridgeDB is already bound to. Unfortunately, the new BridgeDB process writes to our log file, and makes it look like the original process is reloading bridge descriptors while it actually is our volatile, new process. In other words: the way it's set up now, BridgeDB never actually reloads bridges.
Fortunately, the fix is relatively simple: we just have to send the process a SIGHUP. BridgeDB has a SIGHUP handler that – unlike the `--reload` switch – actually works. We should also remove the `--reload` switch.Philipp Winterphw@torproject.orgPhilipp Winterphw@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/34322Make BridgeDB's web interface look like torproject.org2022-08-24T15:20:18ZPhilipp Winterphw@torproject.orgMake BridgeDB's web interface look like torproject.orgBridgeDB's web interface at bridges.torproject.org uses bootstrap, with its own look and feel. We should make it look like torproject.org. Antonela suggested that this may be as simple as loading torproject.org's CSS on top of the existi...BridgeDB's web interface at bridges.torproject.org uses bootstrap, with its own look and feel. We should make it look like torproject.org. Antonela suggested that this may be as simple as loading torproject.org's CSS on top of the existing CSS files.Sponsor 30 - Objective 2.2meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/34318BridgeDB doesn't like non-UTF8 encoded requests2022-07-09T04:22:44ZPhilipp Winterphw@torproject.orgBridgeDB doesn't like non-UTF8 encoded requestsI stumbled upon the following exception in BridgeDB's log:
```
Traceback (most recent call last):
File "/home/bridgedb/virtualenvs/bridgedb/lib/python3.7/site-packages/twisted/web/http.py", line 1755, in dataReceived
finishCallbac...I stumbled upon the following exception in BridgeDB's log:
```
Traceback (most recent call last):
File "/home/bridgedb/virtualenvs/bridgedb/lib/python3.7/site-packages/twisted/web/http.py", line 1755, in dataReceived
finishCallback(data[contentLength:])
File "/home/bridgedb/virtualenvs/bridgedb/lib/python3.7/site-packages/twisted/web/http.py", line 2171, in _finishRequestBody
self.allContentReceived()
File "/home/bridgedb/virtualenvs/bridgedb/lib/python3.7/site-packages/twisted/web/http.py", line 2284, in allContentReceived
req.requestReceived(command, path, version)
File "/home/bridgedb/virtualenvs/bridgedb/lib/python3.7/site-packages/twisted/web/http.py", line 946, in requestReceived
self.process()
--- <exception caught here> ---
File "/home/bridgedb/virtualenvs/bridgedb/lib/python3.7/site-packages/twisted/web/server.py", line 235, in process
self.render(resrc)
File "/home/bridgedb/virtualenvs/bridgedb/lib/python3.7/site-packages/twisted/web/server.py", line 302, in render
body = resrc.render(self)
File "/home/bridgedb/virtualenvs/bridgedb/lib/python3.7/site-packages/twisted/web/resource.py", line 265, in render
return m(request)
File "/home/bridgedb/virtualenvs/bridgedb/lib/python3.7/site-packages/bridgedb-0.10.0+34.ga6eb0d1c.dirty-py3.7.egg/bridgedb/distributors/https/server.py", line 722, in render_POST
return CaptchaProtectedResource.render_POST(self, request)
File "/home/bridgedb/virtualenvs/bridgedb/lib/python3.7/site-packages/bridgedb-0.10.0+34.ga6eb0d1c.dirty-py3.7.egg/bridgedb/distributors/https/server.py", line 573, in render_POST
request.args = stringifyRequestArgs(request.args)
File "/home/bridgedb/virtualenvs/bridgedb/lib/python3.7/site-packages/bridgedb-0.10.0+34.ga6eb0d1c.dirty-py3.7.egg/bridgedb/distributors/https/server.py", line 109, in stringifyRequestArgs
arg = arg if isinstance(arg, str) else arg.decode("utf-8")
builtins.UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 1: invalid continuation byte
```Armin HuremagicArmin Huremagichttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/34260Make BridgeDB take into account its BlockedBridges table2022-07-24T17:07:58ZPhilipp Winterphw@torproject.orgMake BridgeDB take into account its BlockedBridges tableBridgeDB has a table called BlockedBridges in its SQLite database. The table is empty and currently unused but we are extending it over at legacy/trac#34154.
BridgeDB already has code to deal with bridge blocking (e.g., [here](https://g...BridgeDB has a table called BlockedBridges in its SQLite database. The table is empty and currently unused but we are extending it over at legacy/trac#34154.
BridgeDB already has code to deal with bridge blocking (e.g., [here](https://gitweb.torproject.org/bridgedb.git/tree/bridgedb/bridges.py?id=2ccf7ef9dee07c0d645a5dfed5b099678c243fc5#n936)) but we still have to make it process BlockedBridges and take it into account when handing out bridges.Sponsor 30 - Objective 2.3Philipp Winterphw@torproject.orgPhilipp Winterphw@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/32900Reimplement and generalise BridgeDB?2020-09-02T18:05:21ZPhilipp Winterphw@torproject.orgReimplement and generalise BridgeDB?Over at legacy/trac#30946, I spent several hours trying to port BridgeDB to Python 3 but progress has been frustratingly slow. Given the port and our in-progress grant application proposing a social bridge distributor, I started thinking...Over at legacy/trac#30946, I spent several hours trying to port BridgeDB to Python 3 but progress has been frustratingly slow. Given the port and our in-progress grant application proposing a social bridge distributor, I started thinking about a rewrite of BridgeDB. Here's how I see the (dis)advantages breaking down:
Disadvantages:
* Re-implementations are never as smooth and straightforward as anticipated. It will take a lot of time.
* We won't (easily) be able to use Stem to parse bridge descriptors. We could extend [zoossh](https://gitweb.torproject.org/user/phw/zoossh.git/) though.
Advantages:
* We could re-implement the service in golang because the anti-censorship team is comfortable in the language.
* We could generalise the concept of BridgeDB: What goes in should be an abstract type of proxy (be it bridge descriptors, snowflake-style proxy registrations, or maybe even Lantern proxies) and distributors (as we already have them in BridgeDB) determine who gets these proxies.
* We would design the service with redundancy and "containerisation" in mind.
* It would make it easier to add new features, especially significant ones, like a new distributor.
* A re-implementation may be a return on investment and save us time in the long run.
Note that we don't need to abandon BridgeDB and then redirect our focus to its re-implementation. I would instead like to spend some hours experimenting with a new design in parallel to maintaining BridgeDB. We also don't need to aim for a feature-complete replacement of BridgeDB. For example, we don't need to PGP-sign emails. If all of the above proves fruitful, we can gradually transition to the new design.
Thoughts? Any other features or architectural changes we should make in a re-implementation?Sponsor 30 - Objective 2.3Philipp Winterphw@torproject.orgPhilipp Winterphw@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/32276Help BridgeDB see client IP addresses of moat requests2021-08-16T20:11:10ZPhilipp Winterphw@torproject.orgHelp BridgeDB see client IP addresses of moat requestsBridgeDB sees the following HTTP headers for incoming moat requests:
```
Content-Length: ['504']
Via: ['1.1 bridges.torproject.org']
X-Forwarded...BridgeDB sees the following HTTP headers for incoming moat requests:
```
Content-Length: ['504']
Via: ['1.1 bridges.torproject.org']
X-Forwarded-Host: ['bridges.torproject.org']
X-Forwarded-For: ['127.0.0.1']
Connection: ['Keep-Alive']
Host: ['127.0.0.1:3881']
X-Forwarded-Server: ['bridges.torproject.org']
Content-Type: ['application/vnd.api+json']
```
As part of our BridgeDB metrics (legacy/trac#9316) it would be useful to see the IP address of incoming requests.
Here's an IRC conversation in which dcf explains the big picture and proposes a way to fix this issue:
```
dcf1│ phw: there are 2 layers of HTTP in Moat -- one is the CDN-traversal (meek) layer, and one is the end-to-end tunnelled HTTP to the web
server itself.
dcf1│ The CDN will set XFF on the outer meek layer, but not on the inner layer which is what BridgeDB sees (and it couldn't touch the inner
layer because it's HTTPS, which is the whole reason for the complicated proxypass setup).
dcf1│ phw: in other words, XFF is being set on the connection to port 2000, but then that layer is stripped off (only meek-server sees it)
and the tunneled contents go to port 3881.
dcf1│ Even though BridgeDB is an HTTP service, we go to the trouble of tunnelling a whole independent HTTP+TLS stream *inside* the
domain-fronted layer, just to prevent the CDN from tampering with end-to-end traffic.
dcf1│ That's why there are 2 proxypass: the first terminates the meek CDN layer, and the second terminates the tunnelled actual BridgeDB
exchange.
dcf1│ One way to solve this would be yet another shim that understands ExtORPort, parses out the USERADDR, and inserts it into XFF into an
HTTP header. As if the pipeline weren't long enough...
phw│ dcf1: gotcha. i'll create a ticket for this. do you mind if i quote your above explanation?
dcf1│ go for it
phw│ is meek-server exposed to USERADDR? i don't think i follow
dcf1│ meek-server can provide USERADDR to whatever port it forwards to. Currently I'm sure it's set up not to do that, to just forward the
contents without any prefix.
dcf1│ meek-server looks at Meek-IP, X-Forwarded-For, or request source IP address, and uses those to construct a USERADDR, which normally it
would provide to tor on tor's ExtORPort.
dcf1│ The purpose of ExtORPort, as opposed to ordinary ORPort, is to allow passing extra metadata like that, before the actual stream data.
dcf1│ So currently meek-server in Moat is set up to treat the local HTTPS server as its ORPort (not ExtORPort, because Apache wouldn't
understand that).
phw│ dcf1: ah, i understand. thanks for elaborating
dcf1│ Just thought of a research idea: see if any bridges are wrongly exposing their ExtORPort, which would conceivably permit manipulating
statistics.
phw│ we should call this new shim rube-goldberg-machine ;)
dcf1│ Take a subset of bridges, then port-scan them to see if any other port understands ExtORPort-prefixed Tor connections.
dcf1│ Moat is a concrete case where pluggable transports fall short of what you might want them to do. We're trying to "plug" meek-server
into another pipeline, but it's difficult because it's talking to something other than Tor. It's possible, but it quickly turns into a
big pile of shims and adapters.
```Sponsor 30 - Objective 2.1Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/32117Understand and document BridgeDB bot scraping attempts2020-10-29T17:40:19ZCecylia BocovichUnderstand and document BridgeDB bot scraping attemptsWe are aware of automated attempts to enumerate bridges in BridgeDB, but lack a more rigorous understanding of the problem.
We have detected bot requests from bridgeDB's web interface and deployed some defences by forbidding requests wi...We are aware of automated attempts to enumerate bridges in BridgeDB, but lack a more rigorous understanding of the problem.
We have detected bot requests from bridgeDB's web interface and deployed some defences by forbidding requests with headers that are commonly associated with bots, and handing out fake bridges to suspected bot requests (legacy/trac#31252), and
We also suspect that these bots are solving our CAPTCHAs more accurately than users (legacy/trac#24607).
After a recent campaign to get more volunteer bridges, we set up an experiment to test the reachability of a subset of these new bridges from a probe site in Beijing and found all new bridges in our sample to be blocked (most were blocked from the very start of the experiment): legacy/trac#31701
This ticket is for documenting bot behaviour and brainstorming ways to detect and analyze the automatic scraping of BridgeDB from censor-owned bots.hanneloresxhanneloresxhttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/31422Make BridgeDB report internal metrics2020-07-08T20:58:35ZPhilipp Winterphw@torproject.orgMake BridgeDB report internal metricsWe're done with legacy/trac#9316, which means that we have code in place that allows BridgeDB to export metrics. So far, all metrics are user-centric, meaning that they focus on how BridgeDB users interact with the system. BridgeDB-centr...We're done with legacy/trac#9316, which means that we have code in place that allows BridgeDB to export metrics. So far, all metrics are user-centric, meaning that they focus on how BridgeDB users interact with the system. BridgeDB-centric metrics would help us debug and understand BridgeDB. The following come to mind:
* Number of bridges per distribution ring.
* Number of bridges per transport, similar to assignments.log (originally proposed in legacy/trac#14453)
* Number of requests for which we had no bridges.
We could also incorporate bridge assignments in our metrics, so we don't have to report them separately in the assignments.log file (see legacy/trac#29480). Let's not forget to update BridgeDB's metrics specification.
[In the context of #31274 O2.1 - Create an evaluation framework and collect data to better monitor and evaluate current bridge selection and distribution processes.]Sponsor 30 - Objective 2.1Philipp Winterphw@torproject.orgPhilipp Winterphw@torproject.org