Refrain from listener rebinding when address families differ

In retry_listener_ports, we check if port matches exactly and one of the addresses is a wildcard address:

2795         const int may_need_rebind =
2796           port_matches_exact && bool_neq(tor_addr_is_null(&wanted->addr),
2797                                          tor_addr_is_null(&conn->addr));

We should also check if address family is the same between old and new listener. If they differ, we don't need to do rebinding. See discussion on #17873 (moved).