I've made an exception for cases where I'm sure that users can't
influence the inputs.  This is likely to cause a slowdown somewhere,
but it's safer to siphash everything and *then* look for cases to
optimize.

This patch doesn't actually get us any _benefit_ from siphash yet,
since we don't really randomize the key at any point.