Limit access to Moat's /meek endpoint to a trusted CDN

The Moat setup has both a /meek and a /moat endpoint. External requests are supposed to arrive at /meek, which adds an X-Forwarded-For header via moat-shim (#69 (closed)), and is then forwarded to /moat. /moat can trust the X-Forwarded-For header because it was last updated by moat-shim from /meek.

But anonymous11 points out that because /meek is exposed externally, an adversary can simulate control over many IP addresses by writing its own X-Forwarded-For header and thereby get access to more of the pool.

https://ntc.party/t/moat/1604/4

Но Moat (в отличии от HTTPS) делает сбор совсем простым. Можно подключаться напрямую к Meek серверу (/meek), Moat дистрибьютеру (/moat) с ложными заголовками Meek-IP (изображая CDN), X-Forwarded-For (изображая moat-shim).

But Moat (as opposed to HTTPS) makes gathering quite simple. You can connect directly to a Meek server (/meek), a Moat distributor (/moat) with false headers Meek-IP (impersonating CDN), X-Forwarded-For (impersonating moat-shim).

So

1. We can block direct access to /moat (allowing only localhost access from moat-shim), prohibiting X-Forwarded-For spoofing from outside.
2. We can try to ensure that access to /meek comes only from a CDN we trust to add X-Forwarded-For correctly.

This may work as a quick fix:

<Location /meek>
        Require ip 127.0.0.1/8
</Location>

https://httpd.apache.org/docs/2.4/howto/access.html#host

Ooof this is a thorny problem. My read of it is that it's not really about whether these paths are public (more on why they need to be later), but rather that we're not validating that the X-Forwarded-For header we're trusting was added by a legit domain fronting CDN (that we also presumably trust).

Just a quick update on how moat is designed to work (after the shim was introduced):

1. A user sends a request from the meek client through the front to the meek server. This server is accessible from two places:

• https://bridges.torproject.org/meek
• https://moat.torproject.org

2. After stripping off the meek layer, instead of sending tor traffic to a tor OR port, this meek server sends the (http) traffic to the shim
3. The shim does some TLS stripping and X-Forwarded-For alterations and then sends the traffic to the bridgedb Apache frontend

Before the shim was introduced, meek traffic was sent directly the Apache frontend.

Both of these need to be publicly facing because the domain fronting CDN accounts are configured to forward traffic to them. The meek-azure account was configured to forward traffic to https://bridges.torproject.org/meek and the fastly account is configured to forward traffic to https://moat.torproject.org

I don't see a way to make the meek server endpoint private, unless it's by doing something like restricting access to only fastly (or meek or whatever) IP addresses? (We'd have to enumerate them)

The fundamental problem here is that we trust that the X-Forwarded-For header contains the client's real IP address. We could do some additional validation (it would have to be by the meek-server) to make sure that the source IP is coming from fastly. Then, we could ensure BridgeDB uses the most recently appended X-Forwarded-For value. We had recently changed this because of the way the shim worked, it was appending polyanthum's (the server BridgeDB is deployed on) external IP to the X-Forwarded-For list but it's not difficult to revert that change. This would, hopefully reduce the problem to IP spoofing which I'm assuming is relatively hard to get away with.

The other endpoint of concern is https://bridges.torproject.org/moat which arguably does not need to be public, except for the fact that the meek server is set up to connect directly to (now the moat-shim and then to) the Apache HTTP frontend. So really you can make any BridgeDB web requests through moat if you want, it's just that we've configured Tor Browser to send all the requests to https://bridges.torproject.org/moat

One thing we might be able do here is have the moat shim talk directly to the moat distributor (http://127.0.0.1:3881/moat/) since it's already stripping the underlying TLS anyway in order to add in X-Forwarded-For headers. However, I don't really see a harm in making https://bridges.torproject.org/moat public since hiding it won't prevent the attack mentioned above.

(sorry made a few edits for readability)

This is the best existing documentation on how moat infrastructure is set up: https://gitlab.torproject.org/tpo/anti-censorship/bridgedb/-/wikis/Moat

This is the documentation on how to use Moat: https://gitlab.torproject.org/tpo/anti-censorship/bridgedb#id18

Wait, I may have things backwards. It's actually /moat we want to block from the outside, not /meek. /meek needs to be exposed because that's the point of entry for domain-fronted client requests. /moat is the thing that needs to trust the IP address seen be /meek. Is that right?

That's correct, though, making it public won't necessarily solve the issue. Because even if we make /moat accessible only from the moat-shim (or from the meek server or whatever), a user could also use the meek server without the front and fake their X-Forwarded-For header.

Oh, but I think I see your point now. Even if /moat is inaccessible, anyone may connect to /meek as if they were a CDN and give whatever IP address they want.

Maybe the thing to do is to change the /meek endpoint into a long unguessable path, which is stored only at the CDN?

Looks like it's also possible to configure the CDN to use a client certificate.

https://docs.fastly.com/en/guides/working-with-hosts#specifying-a-tls-client-certificate-and-key

What we have to do is vet that the request is coming from a valid front domain so that we can trust the X-Forwarded-For header wasn't manually spoofed

Maybe the thing to do is to change the /meek endpoint into a long unguessable path, which is stored only at the CDN?

Oh interesting, that's a reasonable idea. We might also log the IPs that connect (they should just be CDN IPs) so we can detect abuse if it doesn't work.

Okay so it seems like we want to do a few things:

• restrict access to https://bridges.torproject.org/moat to just local requests
• change the meek endpoint so that only the CDN can connect to it (whether by using an unguessable path or client certs)
• make sure we use the most recently appended X-Forwarded-For value that isn't a local IP address

I apologize for dropping all this all at once. I posted before I really understood what was going on.

changed title from Do not expose the /meek endpoint externally to Limit access to Moat's /meek endpoint to a trusted CDN

changed the description

Oh no worries, the blind trust of the X-Forwarded-For should be fixed.

FWIW, the IP address of the client isn't even used as an enumeration defence. See our bridgedb configuration. It's only used for metrics right now. So, while it would be nice to safeguard against attacks on our metrics, this issue isn't actually allowing for the easy enumeration of moat bridges.

We likely weren't using it because before the moat shim existed, all valid client requests appeared to be coming from the same (local) IP. So we should consider this issue a prerequisite for finally enabling that feature

added Backlog label

mentioned in issue #69 (closed)

added Sponsor 96 label

added Roadmap::Future label and removed Backlog label

Adam Pritchard's article on X-Forwarded-For links to this issue:

Tor is an anonymity network. They have recently realized that they have a control server that is both directly connected to the internet and behind a reverse proxy and are using XFF to give them trustworthy IPs, so they’re vulnerable to spoofing. It looks like they’re working towards limiting the public-ness of the interface, or doing better verification of CDN connections, or both.

The Conclusions section gives Adam's recommendations for inferring the client IP address.

@pierov points out that the new Tor Browser alpha is using the passed-through IP address, to do the geoip resolve to help the user guess what country they're in.

But I don't think that we are relying on the IP address in a new security sense, so we are still ok wrt this ticket.

But also, it remains the case that we shouldn't lose track of this ticket, because it is (as the legal metaphor goes) a rusty rake lying in the yard waiting for somebody to hurt themselves. :)