Draft: Circuit reactor: keep queued stream messages in their streams
Obsoletes !2092 (closed)
This is progress towards prop340 (packing and fragmentation).
The previous code relies on the assumption that a message needs at most a SENDME window of 1. It would break if a message was pulled from the stream before we actually had sufficient stream-level SENDME window to send it. Making the stream receiver peekable lets us peek at the waiting message and determine whether there is sufficient stream-level window.
It would have also been difficult to do opportunistic packing in the previous code. e.g. the only messages available to inspect for potential packing were the ones in the hop-level outbound queue, which we expect would typically be empty. Now that the streams' mpsc channels are peekable, we can check whether any stream has a message ready to send and whether it will fit into the padding of a cell being packed before pulling it out of the stream's mpsc channel.
The previous code implemented fairness of circuit-level SENDME window usage over streams by always pulling a message from all ready streams when any one stream had a message ready, and queueing any that couldn't actually be sent due to circuit-level SENDME window on the hop-level outbound queue. Now that we no longer queue stream messages there, we instead implement fairness by keeping an indexset of streams we know to have a message ready to send. When circuit-level capacity is available, we use that indexset to serve the streams that have been waiting the longest first.