Deploy further snowflake-server performance improvements 2022-09-24
The snowflake-01 bridge is getting close to its memory limits again after yesterday's deployments (#40173 (closed), #40175 (closed)). The little dip marked in the graph below, I investigated it, and it occurred during a time when the memory was almost 100% full. You can also tell the server is reaching its limits because the send and recv traces become a little bit decorrelated. For comparison, currently, at the right edge of the graph, RAM use is 85%.
Profiling yesterday revealed two major contributors to memory usage:
io.Copy
buffers and client send queues, each of which accounts for about 25%
of the total memory used by the process.
I have a branch that is meant to resolve the io.Copy
buffer one.
It exposes the io.WriterTo
interface of smux.Stream
through SnowflakeClientConn
,
which will prevent io.Copy
from allocating a buffer per client.
I also put in a minor optimization to ClientMap.SendQueue
,
the benefit of which turns out to be tiny,
but there's no reason not to do it.
dcf/snowflake@a16133ff...4d4fad30
/cc @linus