fix: stale connection not closing on timeout sometimes

When no messages from the client were ever sent