sched: DESTROY cell on a circuit bypasses the scheduler
If you look at
circuitmux_append_destroy_cell(), it is the one appending a DESTROY cell to the cmux queue and then calls
channel_flush_from_first_active_circuit() if no writes are pending that is if the outbuf is empty (also looks at the out queue but that is always empty #23709 (moved)).
In the case the flush is triggered, the cell is immediately put in the outbuf and written to kernel by libevent which completely bypasses the scheduler. Maybe it is what we want that is go as fast as we can in destroying a circuit? Don't know but it has this effect on the scheduler where the channel is scheduled with a "wants_to_write" event from the connection subsystem and ultimately the channel gets scheduled with nothing in the queue because it is already on the outbuf. For KIST, this is not ideal because KIST should control the flow of data to the kernel.
It seems there are two places we queue cells into a cmux queue:
append_cell_to_circuit_queue(). The latter triggers a "has waiting cells" for the scheduler which is what we want but the former just bypasses it.
I think it should simply trigger that notify to the scheduler instead of flushing it by itself.