connection_consider_empty_{read,write}_buckets should not affect cpuworker connections

On IRC, skruffy points out that connection_consider_empty_{read,write}_buckets seem to affect local connections to cpuworkers. That can't possibly be right.