Out of ephemeral ports on link between haproxy and extor-static-cookie
The multi-KCP optimization parallelization of #40200 (closed) has enabled a large increase in bandwidth, currently above 3.5 Gbps outgoing / 2.5 Gbps incoming. But now we are experiencing another ephemeral port exhaustion issue like #40198 (closed). This time it's on the link between haproxy and extor-static-cookie. The snowflake-server log says:
2022/10/02 05:06:47 handleConn: failed to connect to ORPort: EOF
The haproxy log says:
Oct 2 05:06:53 snowflake-01 haproxy[9134]: Connect() failed for backend tor-instances: no free ports.
For context, there are about 38000 live connections between snowflake-server and haproxy.
# ss -f inet -n state established | grep '127.0.0.1:10000\s*$' | wc -l
37918
/cc @linus