Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
  • S Snowflake
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 92
    • Issues 92
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 8
    • Merge requests 8
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • The Tor Project
  • Anti-censorship
  • Pluggable Transports
  • Snowflake
  • Issues
  • #40198
Closed
Open
Issue created Sep 30, 2022 by David Fifield@dcfOwner

Out of localhost ephemeral ports ("cannot assign requested address") on link between snowflake-server and haproxy

Since 2022-09-28 10:57:26 (53 hours ago) the snowflake-server log is full of:

# grep 'cannot assign requested address$' /var/log/snowflake-server/snowflake-server.log | head
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
# grep 'cannot assign requested address$' /var/log/snowflake-server/snowflake-server.log | tail
2022/09/30 15:44:17 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/30 15:44:18 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/30 15:44:18 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/30 15:44:18 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/30 15:44:18 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address

The error message means that the kernel could not allocate an ephemeral port number for a localhost TCP connection. In this case it for the connection between snowflake-server and haproxy.

My analysis at https://lists.torproject.org/pipermail/anti-censorship-team/2022-September/000265.html was incomplete: the total number of localhost sockets does not matter for the purpose of counting 4-tuples, but it does matter for the purpose of allocating ephemeral ports. There are currently only 21712 sockets open between snowflake-server and haproxy, which means the remainder of the ephemeral port range is taken up by other localhost communication.

My immediate plan for this is to move haproxy to 127.0.0.2:10000 rather than 127.0.0.1:10000, and maybe similarly with the individual tor instances if necessary.

/cc @linus

Assignee
Assign to
Time tracking