Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
  • M meek
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Issues 13
    • Issues 13
    • List
    • Boards
    • Service Desk
    • Milestones
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
  • Wiki
    • Wiki
  • Activity
  • Create a new issue
  • Jobs
  • Issue Boards
Collapse sidebar
  • The Tor Project
  • Anti-censorship
  • Pluggable Transports
  • meek
  • Issues
  • #12857
Closed
Open
Issue created Aug 13, 2014 by David Fifield@dcfOwner

Use streaming downloads

How meek works now is, the client reads a small chunk of data from tor (up to 64 KB) and sends it in the body of a POST request. The server receives it, reads a small chunk of data from tor (up to 64 KB), and sends it back in the response. The client doesn't make another request until it has received the response to the first one, in order to keep all the chunks of data in order.

Here's what would be better. The client sends a small chunk of data. The server sends a response header and any data it has pending, and leaves the response channel open. The server can use the chunked transfer-encoding to send a body of indeterminate length. The server sends downstream data over the existing streaming response channel until it receives another request. At that point, it closes the response channel and opens up a new one (which will be the response to the just-received request).

The advantages are mainly about performance: there's no client polling; there's less HTTP header overhead (see legacy/trac#12778 (moved)); and the client can send data (send a request) whenever it feels like it, without waiting for the server's most recent response, if the underlying HTTP library supports pipelining.

Psiphon has already implemented something like this. There's a bit of difficulty in that the golang HTTP server doesn't notify you of new requests while you're still sending a response on the same keep-alive channel. Their workaround (and I think it is a good one) is to put a timeout on the download streaming, so that a long response won't block upstream data forever.

  • https://bitbucket.org/psiphon/psiphon-circumvention-system/diff/go/meek-server/meek-server.go?diff1=0ca24558e14e772a5b0e3dfb7b166914682a8fc0&diff2=692f9187147c&at=meek-performance

We'll need to overhaul the web browser extensions, because they currently assume requests and responses with sizes known in advance.

This approach won't work with Google App Engine, because App Engine doesn't support streaming downloads. But it should work with CloudFront. See legacy/trac#12428 (moved) for how to improve performance with App Engine.

Assignee
Assign to
Time tracking