Refactor our checking of whether we should be reading/writing on a connection to use a set of reason-flags
There are plenty of things that can block reading or writing on a connection: being out of bandwidth, not having anything more to say, having an internal buffer get too full, etc.
There are too many places where one of the things that would block read/write becomes false, and so we check all of the other potential reasons not to read/write before we re-enable reading/writing.
There's a much better pattern we could use: have a flags variable for "read_blocked" and a flags variable for "write_blocked", each bit of which represents a reason not to be reading or writing. Instead of calling connection_{stop,start}{reading,writing} directly, we would call {un,}block{reading,writing}(conn, reason), which would set or clear a bit corresponding to 'reason', and block/unblock reading/writing accordingly.
I haven't yet verified that this is a win; we should audit all the uses of connection_{stop,start}_{reading,writing} reading to see.