In a2bb0bfd we started using a separate client identity key. When we are
in "public server mode" (that means not a bridge) we will use the same
key. Reusing the key without doing the proper refcounting leads to a
segfault on cleanup during shutdown. Fix that.

Also introduce an assert that triggers if our refcount falls below 0.
That should never happen.