test_util: remove directory special case

This was apparently a workaround for #40141 (closed), but should not be necessary with !164 (merged) merged.

cc @Jigsaw52