Consider refactoring flow control code to optionally use flow control logic
Follow-up from !3054 (comment 3221665).
We must support two forms of flow control for streams: sendme-window flow control and xon/xoff flow control. These are both implemented differently in different parts of the code.
- SENDME flow control: implemented in circuit reactor and
StreamReceiver
- XON/XOFF flow control: implemented in circuit reactor,
DataReader
, andDataWriter
The current code is designed such that both SENDME flow control and XON/XOFF flow control are always functional, and the circuit reactor decides which to use. For example the rate limit update stream which notifies the DataWriter
when the circuit reactor receives an XON/XOFF message will always be initialized. If SENDME flow control is used for the stream, the circuit reactor simply drops the sending side so that a rate limit update can never be sent.
We should instead consider making the SENDME flow control and XON/XOFF flow control functionality optional, for example by wrapping related objects in an Option
or storing a bool
. For example if SENDME flow control is in use, we should never construct the rate limit update stream and should store a None
instead. Or if XON/XOFF flow control is in use, the StreamReceiver
should store a false
so that it knows not to send a CtrlMsg::SendSendme
to the circuit reactor.
It's unclear which approach is better, but we should experiment with both.