Out of localhost ephemeral ports ("cannot assign requested address") on link between snowflake-server and haproxy

Since 2022-09-28 10:57:26 (53 hours ago) the snowflake-server log is full of:

# grep 'cannot assign requested address$' /var/log/snowflake-server/snowflake-server.log | head
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/28 10:57:26 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
# grep 'cannot assign requested address$' /var/log/snowflake-server/snowflake-server.log | tail
2022/09/30 15:44:17 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/30 15:44:18 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/30 15:44:18 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/30 15:44:18 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address
2022/09/30 15:44:18 handleConn: failed to connect to ORPort: dial tcp [scrubbed]: connect: cannot assign requested address

The error message means that the kernel could not allocate an ephemeral port number for a localhost TCP connection. In this case it for the connection between snowflake-server and haproxy.

My analysis at https://lists.torproject.org/pipermail/anti-censorship-team/2022-September/000265.html was incomplete: the total number of localhost sockets does not matter for the purpose of counting 4-tuples, but it does matter for the purpose of allocating ephemeral ports. There are currently only 21712 sockets open between snowflake-server and haproxy, which means the remainder of the ephemeral port range is taken up by other localhost communication.

My immediate plan for this is to move haproxy to 127.0.0.2:10000 rather than 127.0.0.1:10000, and maybe similarly with the individual tor instances if necessary.

/cc @linus