Remove some needless refs and slicing

Prompted by nightly's clippy (which has some false positives, so is currently disabled).

This is a follow-up to !323 (merged)