XON/XOFF stream flow control questions

This isn't really an issue, but I don't know a better place to track these questions. These are the first round of questions I have which I think I need to answer before I can start implementing stream XON/XOFF flow control for congestion control.

https://spec.torproject.org/proposals/324-rtt-congestion-control.html

Section "4.1. Stream Flow Control Without Windows" discusses sending XONs, but not receiving them. What happens when an XON is received? What does it do with the ewma rate? I'm assuming we need to reduce the send rate of the stream, but it's not specified. Do we implement this rate limit with a token bucket? What should the initial rate be before an XON is received, unlimited?
Once the queue is expected to empty, [...]

What does this mean? When the queue is empty? When the queue has fewer than L bytes? When the measured drain rate predicts that the queue will be empty in fewer than T milliseconds?
Once the queue is expected to empty, [...]

Arti doesn't have a single outgoing queue. There's the byte queue in DataReaderImpl for partial/leftover cells, the mpsc queue of cells in StreamReader, and maybe others? Which queue should we consider? Do we need to consider all of them? If so, how can we do that?
This XON also indicates the average rate of queue drain since the XOFF

Is this the EWMA rate using cc_xon_rate and N_EWMA like in the "XON_ADVISORY" section, or just the plain average (mean) rate? Does this mean that we're always computing the EWMA rate, not just when we're in a "XON advisory" state?
Advisory XON cells are also sent whenever [...]

Is there any difference between the first XON and advisory XONs?
In order to measure the drain rate of a connection, we need to measure the time it took between flushing N bytes on the socket

What does "flushing" mean, does this just mean writing N bytes to the socket?
For example, lets say we just wrote 100 bytes on the socket at time t = 0sec and at time t = 2sec the socket becomes writeable again, we then estimate that the rate of the socket is 100 / 2sec thus 50B/sec

This doesn't make sense to me. If you write 100 bytes to the socket, and then 2 sec later the socket becomes available for writing again, but only has enough room to accept 10 bytes, your drain rate should be 10 bytes / 2 sec.
In order to measure the drain rate of a connection, we need to measure the time it took between flushing N bytes on the socket and when the socket is available for writing again

This doesn't really follow how Arti is designed. Streams in Arti don't have access to the socket. Do we really need to concern ourselves with the socket? I think we only need to care about when data is taken from the async stream object? So I think this becomes "the time it took between N bytes becoming queued on the stream and when something reads >1 byte from the stream"?
In order to measure the drain rate of a connection, we need to measure the time it took between flushing N bytes on the socket and when the socket is available for writing again

The existing window-based stream flow control is handled in the StreamReader, which deals with reading cells rather than bytes. I think to deal with bytes for XON/XOFF-based flow control, we need to copy the StreamTarget into the DataReader or DataReaderImpl (I'm not yet sure what the distinction is) as well, which will communicate with the circuit reactor to send XON/XOFF cells. So stream SENDMEs will be handled within the StreamReader and stream XON/XOFFs will be handled within the DataReader.
cc_xon_rate - Specifies how many full packed cells of bytes must arrive before we can compute a rate, [...]

What do you do when receiving a non-full cell?
To make such measurement, we start the timer by recording a timestamp as soon as data begins to accumulate in an edge connection's outbuf, currently 16KB (32 cells). We use such value for now because Tor write up to 32 cells at once on a connection outbuf and so we use this burst of data as an indicator that bytes are starting to accumulate.

This doesn't sound like it applies to Arti. Can we just take the timestamp as soon as the buffer goes from empty to non-empty?
In order to measure the drain rate of a connection, we need to measure the time it took between flushing N bytes on the socket and when [...]
[...]
To make such measurement, we start the timer by recording a timestamp as soon as data begins to accumulate in an edge connection's outbuf

These are saying two different things. The first says we start the timer when writing N bytes to the socket, and the second says we start the timer as data begins to accumulate in the outbuf. Which one is correct?
After 'cc_xon_rate' cells worth of stream data, we use N_EWMA to [...]

After 'cc_xon_rate' cells worth of stream data are queued? Or after 'cc_xon_rate' cells worth of stream data are written to the socket?
After 'cc_xon_rate' cells worth of stream data, [...]

This paragraph is confusing to me. It switches between counting bytes and cells. Does this really track both the number of bytes and the number of cells? Or is it estimating the number of bytes based on an assumption that all cells are full? What also makes it confusing is phrases like "Because the EWMA rate is only updated every 'cc_xon_rate' cells worth of bytes, [...]", so I don't really understand what's actually going on here.
If the buffers are non-zero, [...]

I'm assuming this means non-empty?
When the outbuf completely drains to 0, and has been 0 for 'cc_xon_rate' cells worth of data, we double the EWMA rate

How can the outbuf drain to 0 bytes, and maintain a length of 0 bytes even after receiving 'cc_xon_rate' cells worth of data? Doesn't this data get put on the outbuf, meaning it doesn't stay at a length of 0 bytes?
The measurement timestamp is also set back to 0

What does this mean? I'm assuming this doesn't mean the unix epoch, so maybe this means "we unset the measurement timestamp" (like as a rust None)? But if that's the case, I don't understand why.
If a clock stall or jump is detected by [CLOCK_HEURISTICS], we also clear the fields, but do not record them in ewma

I'm assuming this means to clear the timestamp and byte count, but not the running ewma value?

/cc @dgoulet, @mikeperry

Edited Apr 22, 2025 by opara