client: ControlPort set doesn't mean you're a client
Splitting from #19665 (moved) to avoid confusion.
This ticket is solely to make a patch so that when the ControlPort is set (Unix or not), we do not consider it as a client with any_client_port_set().