Anti-censorship issueshttps://gitlab.torproject.org/groups/tpo/anti-censorship/-/issues2021-09-09T14:26:41Zhttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/12547Get analysed data from bridge reachability tests to tor-devs2021-09-09T14:26:41ZArturo FilastòGet analysed data from bridge reachability tests to tor-devsThis means setting up a web server on the post processing machine (the one running the collector) with some access control so that tor devs can read the reports.This means setting up a web server on the post processing machine (the one running the collector) with some access control so that tor devs can read the reports.https://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/12506Separate BridgeDB databases from distributors2021-09-09T14:25:11ZIsis LovecruftSeparate BridgeDB databases from distributorsWhat was originally called "BridgeDB" is essentially two very different systems, which should be kept separate, and should be runnable on separate machines:
1. BridgeDB descriptor parsers, databases, and hashrings.
This part of B...What was originally called "BridgeDB" is essentially two very different systems, which should be kept separate, and should be runnable on separate machines:
1. BridgeDB descriptor parsers, databases, and hashrings.
This part of BridgeDB's system takes the bridge descriptor files received from the BridgeAuthority (Tonga), parses them, shoves them into databases, and assigns them to the hashrings of the various bridge distributors.
2. BridgeDB distributors.
This part of BridgeDB's system distributes the bridges to clients, using various mechanisms. This would currently be the `bridgedb.Dist.EmailDistributor` which replies to email requests for bridges, and the `bridgedb.Dist.IPBasedDistributor` which runs the [https://bridges.torproject.org web interface]. Each distributor listens for clients' requests and responds with bridges from that distributor's hashring, as appropriate.https://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/12031Create a Key-Value database system for simple/flat datatypes in BridgeDB2021-09-09T14:22:51ZIsis LovecruftCreate a Key-Value database system for simple/flat datatypes in BridgeDBBridgeDB will likely need some rather high-level code for interacting with Redis through Twisted in order to complete [Section 2 of proposal #226](https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/226-bridgedb-database-impro...BridgeDB will likely need some rather high-level code for interacting with Redis through Twisted in order to complete [Section 2 of proposal #226](https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/226-bridgedb-database-improvements.txt#l143).
We'll probably want to start with storing the `(emailAddress,timestamp)` pairs for the `bridgedb.Storage.getEmailTime` and `bridgedb.Storage.getWarnedEmail` inside Redis. Other simple datatypes which can be stored in Redis should eventually get listed here (and eventually in [bridgedb-spec.txt](https://gitweb.torproject.org/torspec.git/blob/HEAD:/bridgedb-spec.txt).https://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/12030Create a DatabaseManager for interacting with BridgeDB's database backends2021-09-09T14:22:37ZIsis LovecruftCreate a DatabaseManager for interacting with BridgeDB's database backendsWe need a `DatabaseManager` which will handle receiving `BridgeRequest`s from BridgeDB's distributors, and will [queue these requests](https://twitter.com/BigDataBorat/status/360850476150956032) as transactions with the backend databases...We need a `DatabaseManager` which will handle receiving `BridgeRequest`s from BridgeDB's distributors, and will [queue these requests](https://twitter.com/BigDataBorat/status/360850476150956032) as transactions with the backend databases.
The distributors are going to use a common method (just call it `getBridgesFromDBManager()` for now) to request `Bridge`s from the `DatabaseManager`, and they will expect to receive some relatively well-supported, simple, serialised format (probably JSON) in return.
1. **The Easy Way** The _easier_ way to do this would be to not really actually truly make a for-reals ORM, and simply expect to be interacting with a NOSQLly CouchDB backend [as described in proposal #226](https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/226-bridgedb-database-improvements.txt#l82). CouchDB documents are JSON anyway, so this is super easy. If we go this route, we'll need some code to convert the old SQLly stuff to the new NOSQLly format.
2. **The Masochistic Way** The harder, but possibly better in the long run (should we ever decide to stop using NOSQL/OODBMSs), way to do this would be to write a true ORM/RDBMS which can work with either system.
This databases which this system will interact with will be used for storing complex/referential/self-referential/recursive datatypes, such as `bridgedb.Bridges.Bridge`s and structures for storing data about blocking events for bridges (including timestamps, discovery method, and country code of the block, etc.) as defined [in Section 1.b. of proposal #226](https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/226-bridgedb-database-improvements.txt#l112).https://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/9332Implement whitelisting of (email_address, gpg_key_id) pairs for encrypted, au...2021-09-09T14:20:45ZGeorge KadianakisImplement whitelisting of (email_address, gpg_key_id) pairs for encrypted, automated email bridge distributionRoger told me that BridgeDB used to send bridges to a list of emails. It got those bridges from the reserved pool, and sent some of them to the members of the mailing list every so often.
This feature seems to be disabled now (for some ...Roger told me that BridgeDB used to send bridges to a list of emails. It got those bridges from the reserved pool, and sent some of them to the members of the mailing list every so often.
This feature seems to be disabled now (for some reason), but it might be a good idea to revive it.https://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/31875BridgeDB should consider a user's location2021-09-09T14:18:39ZPhilipp Winterphw@torproject.orgBridgeDB should consider a user's locationbridges.torproject.org knows the user's physical location and should consider it when returning bridges. For example, if somebody in country X asks for an obfsN bridge but BridgeDB knows that only obfsN+1 works in X (e.g., by consulting ...bridges.torproject.org knows the user's physical location and should consider it when returning bridges. For example, if somebody in country X asks for an obfsN bridge but BridgeDB knows that only obfsN+1 works in X (e.g., by consulting the result of tpo/community/outreach#28531), it should return obfsN+1.
Ideally, we would do this for all of BridgeDB's distribution mechanisms. We could also do it for email – if the user emailed bridges+CC@bridges.torproject.org. As I understand it, we cannot do it for moat because BridgeDB doesn't get to see the user's IP address in this case.Sponsor 30 - Objective 3.3meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/meek/-/issues/40002meek-client tests failing2021-09-03T21:58:42Zmax-bmeek-client tests failingI think for one, it looks like [copyPublicFields](https://gitweb.torproject.org/pluggable-transports/meek.git/tree/meek-client/utls_test.go#n17) never got added and/or the `TestCopyPublicFieldsHTTPTransport` test is no longer necessary?
...I think for one, it looks like [copyPublicFields](https://gitweb.torproject.org/pluggable-transports/meek.git/tree/meek-client/utls_test.go#n17) never got added and/or the `TestCopyPublicFieldsHTTPTransport` test is no longer necessary?
Additionally, the `TestProxyHTTPSCONNECT` seems to be failing with this output:
```
❯ go test . -v
=== RUN TestMakeProxySpec
--- PASS: TestMakeProxySpec (0.00s)
=== RUN TestAddrForDial
--- PASS: TestAddrForDial (0.00s)
=== RUN TestProxyHTTPCONNECT
--- PASS: TestProxyHTTPCONNECT (0.00s)
=== RUN TestProxyHTTPProxyAuthorization
--- PASS: TestProxyHTTPProxyAuthorization (0.00s)
=== RUN TestProxyHTTPSCONNECT
proxy_test.go:95: local error: tls: unexpected message
--- FAIL: TestProxyHTTPSCONNECT (0.01s)
panic: remote error: tls: unexpected message [recovered]
panic: remote error: tls: unexpected message
goroutine 18 [running]:
testing.tRunner.func1.2(0x7d9200, 0xc0000203c0)
/usr/local/go/src/testing/testing.go:1144 +0x332
testing.tRunner.func1(0xc00008a300)
/usr/local/go/src/testing/testing.go:1147 +0x4b6
panic(0x7d9200, 0xc0000203c0)
/usr/local/go/src/runtime/panic.go:965 +0x1b9
git.torproject.org/pluggable-transports/meek.git/meek-client.TestProxyHTTPSCONNECT(0xc00008a300)
/home/maxb/work/tor/meek/meek-client/proxy_test.go:232 +0x3ec
testing.tRunner(0xc00008a300, 0x847030)
/usr/local/go/src/testing/testing.go:1194 +0xef
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:1239 +0x2b3
FAIL git.torproject.org/pluggable-transports/meek.git/meek-client 0.021s
FAIL
```https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/44document what gettor does and how is implemented2021-09-02T15:16:30Zmeskiomeskio@torproject.orgdocument what gettor does and how is implementedLet's add some documentation about gettor.Let's add some documentation about gettor.Sponsor 30 - Objective 2.3meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/17About more powerful pluggable transport2021-09-02T14:38:55ZTracAbout more powerful pluggable transportI found a more powerful proxy protocol.
It's "shadowsocks".
https://github.com/shadowsocks
This protocol was created to avoid GFW.
If you can use shadowsocks for pluggable transport, I think we can use Tor more easily and faster in China...I found a more powerful proxy protocol.
It's "shadowsocks".
https://github.com/shadowsocks
This protocol was created to avoid GFW.
If you can use shadowsocks for pluggable transport, I think we can use Tor more easily and faster in China.
p.s. I'm Japanese so I understand English a little.
Since there is a possibility that there is a mistake in the translation, I will post the original.
日本語の原文
私はより強力なプロキシープロトコルを見つけました。
それは「shadowsocks」です。
https://github.com/shadowsocks
このプロトコルはGFWを回避するために作られました。
私はshadowsocksをpluggable transportとして利用できれば、中国でより簡単かつ高速にTorを利用できるようになると思います。
p.s. 私は日本人なので少ししか英語を理解出来ません。
翻訳に間違いがある可能性があるため、原文を載せておきます。
**Trac**:
**Username**: Anon8101919https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/9Implement conjure as a pluggable transport2021-09-02T14:28:37ZGabagaba@torproject.orgImplement conjure as a pluggable transportGet [the refraction networking client](https://github.com/refraction-networking/gotapdance)'s code into a pluggable transport for Tor.
- [ ] get familiar with the Go code https://github.com/refraction-networking/gotapdance
- [x] create...Get [the refraction networking client](https://github.com/refraction-networking/gotapdance)'s code into a pluggable transport for Tor.
- [ ] get familiar with the Go code https://github.com/refraction-networking/gotapdance
- [x] create conjure repo in https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports for this new PT
- https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/conjure
Any new issue to convert it into a PT will live in the new repository:
- Decide how we're going to do signaling -- maybe by reusing our Fastly domain fronting thing, if we don't like the signaling component Conjure uses now.
- Integrate into Tor Browser (including reproducible build)
- Verify that obfs4+conjure actually works
- Deploy bridge somewhere to receive traffic from already setup infrastructure
- Make sure we can still get our extorport-style metrics: if the conjure infra de-obfs4's the traffic, do we lose the features of the extorport, like passing along the country of the original user? If yes fix it somehow.Sponsor 30 - Objective 2.3Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/25Add parser for bridge descriptors2021-09-02T14:21:08ZPhilipp Winterphw@torproject.orgAdd parser for bridge descriptorsRdsys currently only parses extrainfo documents and therefore only knows about pluggable transports. We need to add a parser for bridge descriptors, giving rdsys the ability to also learn about vanilla bridges.Rdsys currently only parses extrainfo documents and therefore only knows about pluggable transports. We need to add a parser for bridge descriptors, giving rdsys the ability to also learn about vanilla bridges.https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/50GetTor should set In-Reply-To when responding to email2021-08-31T10:43:02ZPhilipp Winterphw@torproject.orgGetTor should set In-Reply-To when responding to emailGetTor currently doesn't set the `In-Reply-To` header when responding to an email. That breaks threading in the user's mailbox and it also makes it slightly more difficult to test our autoresponder.
It's not high priority but let's add ...GetTor currently doesn't set the `In-Reply-To` header when responding to an email. That breaks threading in the user's mailbox and it also makes it slightly more difficult to test our autoresponder.
It's not high priority but let's add the `In-Reply-To` header at some point.meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/58run tests in the gitlab CI2021-08-30T16:04:04Zmeskiomeskio@torproject.orgrun tests in the gitlab CISponsor 30 - Objective 2.3meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/docker-obfs4-bridge/-/issues/2Offer obfs4 docker image for additional architectures2021-08-26T16:41:53ZPhilipp Winterphw@torproject.orgOffer obfs4 docker image for additional architecturesA bridge operator asked us to cross-compile our docker image for arm64. This sounds like an easy-ish fix that would make the lifes of our bridge operators easier. Let's figure out what it takes to support ARM and potentially other archit...A bridge operator asked us to cross-compile our docker image for arm64. This sounds like an easy-ish fix that would make the lifes of our bridge operators easier. Let's figure out what it takes to support ARM and potentially other architectures.https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/33800Remove uniuri dependency2021-08-17T03:50:46ZDavid Fifielddcf@torproject.orgRemove uniuri dependencyuniuri is only used in a minor way, to generate a random string for local identification of a snowflake client.uniuri is only used in a minor way, to generate a random string for local identification of a snowflake client.David Fifielddcf@torproject.orgDavid Fifielddcf@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/40027Use leftmost address when parsing `X-Forwarded-For` header for client IP2021-08-16T23:33:27ZCecylia BocovichUse leftmost address when parsing `X-Forwarded-For` header for client IPWhen a client passes through multiple proxies, each subsequent address is appended to the X-Forwarded-For header, resulting in a comma-separated list of IP addresses:
```
X-Forwarded-For: <client>, <proxy1>, <proxy2>
```
Right now Bridg...When a client passes through multiple proxies, each subsequent address is appended to the X-Forwarded-For header, resulting in a comma-separated list of IP addresses:
```
X-Forwarded-For: <client>, <proxy1>, <proxy2>
```
Right now BridgeDB only looks for the client's IP in the rightmost address
```
if useForwardedHeader:
header = request.getHeader("X-Forwarded-For")
if header:
index = -1
ip = header.split(",")[index].strip()
if skipLoopback:
logging.info(("Parsing X-Forwarded-For again, ignoring "
"loopback addresses..."))
while isLoopback(ip):
index -= 1
ip = header.split(",")[index].strip()
if not skipLoopback and isLoopback(ip):
logging.warn("Accepting loopback address: %s" % ip)
else:
if not isIPAddress(ip):
logging.warn("Got weird X-Forwarded-For value %r" % header)
ip = None
```
This causes trouble with our Moat and Apache ProxyPass setup, which results in X-Forwarded-For headers like the following:
```
X-Forwarded-For: <client>, ... <proxies> ... <local address>
```
I think we should modify this to parse the addresses from left to right, ignoring loopback/internal addresses, until we find a valid address for the client.
This is a follow-up modification for #32276 and a prerequisite for #40025.Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/issues/32276Help BridgeDB see client IP addresses of moat requests2021-08-16T20:11:10ZPhilipp Winterphw@torproject.orgHelp BridgeDB see client IP addresses of moat requestsBridgeDB sees the following HTTP headers for incoming moat requests:
```
Content-Length: ['504']
Via: ['1.1 bridges.torproject.org']
X-Forwarded...BridgeDB sees the following HTTP headers for incoming moat requests:
```
Content-Length: ['504']
Via: ['1.1 bridges.torproject.org']
X-Forwarded-Host: ['bridges.torproject.org']
X-Forwarded-For: ['127.0.0.1']
Connection: ['Keep-Alive']
Host: ['127.0.0.1:3881']
X-Forwarded-Server: ['bridges.torproject.org']
Content-Type: ['application/vnd.api+json']
```
As part of our BridgeDB metrics (legacy/trac#9316) it would be useful to see the IP address of incoming requests.
Here's an IRC conversation in which dcf explains the big picture and proposes a way to fix this issue:
```
dcf1│ phw: there are 2 layers of HTTP in Moat -- one is the CDN-traversal (meek) layer, and one is the end-to-end tunnelled HTTP to the web
server itself.
dcf1│ The CDN will set XFF on the outer meek layer, but not on the inner layer which is what BridgeDB sees (and it couldn't touch the inner
layer because it's HTTPS, which is the whole reason for the complicated proxypass setup).
dcf1│ phw: in other words, XFF is being set on the connection to port 2000, but then that layer is stripped off (only meek-server sees it)
and the tunneled contents go to port 3881.
dcf1│ Even though BridgeDB is an HTTP service, we go to the trouble of tunnelling a whole independent HTTP+TLS stream *inside* the
domain-fronted layer, just to prevent the CDN from tampering with end-to-end traffic.
dcf1│ That's why there are 2 proxypass: the first terminates the meek CDN layer, and the second terminates the tunnelled actual BridgeDB
exchange.
dcf1│ One way to solve this would be yet another shim that understands ExtORPort, parses out the USERADDR, and inserts it into XFF into an
HTTP header. As if the pipeline weren't long enough...
phw│ dcf1: gotcha. i'll create a ticket for this. do you mind if i quote your above explanation?
dcf1│ go for it
phw│ is meek-server exposed to USERADDR? i don't think i follow
dcf1│ meek-server can provide USERADDR to whatever port it forwards to. Currently I'm sure it's set up not to do that, to just forward the
contents without any prefix.
dcf1│ meek-server looks at Meek-IP, X-Forwarded-For, or request source IP address, and uses those to construct a USERADDR, which normally it
would provide to tor on tor's ExtORPort.
dcf1│ The purpose of ExtORPort, as opposed to ordinary ORPort, is to allow passing extra metadata like that, before the actual stream data.
dcf1│ So currently meek-server in Moat is set up to treat the local HTTPS server as its ORPort (not ExtORPort, because Apache wouldn't
understand that).
phw│ dcf1: ah, i understand. thanks for elaborating
dcf1│ Just thought of a research idea: see if any bridges are wrongly exposing their ExtORPort, which would conceivably permit manipulating
statistics.
phw│ we should call this new shim rube-goldberg-machine ;)
dcf1│ Take a subset of bridges, then port-scan them to see if any other port understands ExtORPort-prefixed Tor connections.
dcf1│ Moat is a concrete case where pluggable transports fall short of what you might want them to do. We're trying to "plug" meek-server
into another pipeline, but it's difficult because it's talking to something other than Tor. It's possible, but it quickly turns into a
big pile of shims and adapters.
```Sponsor 30 - Objective 2.1Cecylia BocovichCecylia Bocovichhttps://gitlab.torproject.org/tpo/anti-censorship/bridgestrap/-/issues/20review CollecTor metrics2021-08-16T07:51:45Zmeskiomeskio@torproject.orgreview CollecTor metricsThe bridgestrap metrics still look broken and I didn't seem to fix them with #18. Two things look weird to me:
* In the last few days no bridge appear as functional
* The same bridge appears many times, I believe bridge tests happen once...The bridgestrap metrics still look broken and I didn't seem to fix them with #18. Two things look weird to me:
* In the last few days no bridge appear as functional
* The same bridge appears many times, I believe bridge tests happen once every 18 hours and recurrent manual tests should hit the cache and don't produce entries on the metrics.Sponsor 30 - Objective 2.3meskiomeskio@torproject.orgmeskiomeskio@torproject.orghttps://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/55Allow to get Tor bundles and bridges via Tox2021-08-12T14:28:42ZcypherpunksAllow to get Tor bundles and bridges via ToxTox is a protocol of decentralyzed crypto-messenger.Tox is a protocol of decentralyzed crypto-messenger.https://gitlab.torproject.org/tpo/anti-censorship/rdsys/-/issues/54Implementing a GetTor FaceBook Messenger Bot2021-08-12T14:26:50ZTracImplementing a GetTor FaceBook Messenger BotI believe a FaceBook messenger bot would be a great addition to the GetTor service and purpose we begin researching into if a FB bot would be suitable for distributing GetTor and how the community feels about it. I am happy to take quest...I believe a FaceBook messenger bot would be a great addition to the GetTor service and purpose we begin researching into if a FB bot would be suitable for distributing GetTor and how the community feels about it. I am happy to take questions and gather the answers required as I am the one purposing this.
Bot can be implemented in Python for ease of integrating into GetTor.
There are a few FB policies covering the use of bots and what they can and cannot do, I do not see how the current GetTor bot would fall foul of these policies.
There are limits for api requests which vary depending on which of the two FB api is used for the bot, both of which are very high
A verified (via mobile phone or cc) FB account would be required for making the Bot public along with a review of the bot by FB itself.
I believe this could be a viable and beneficial project and welcome any comments or criticisms.
**Trac**:
**Username**: iooryx