* This assertion fails when executing the whole suite, but not when executing
  this test by itself
* Ideally I'd prefer starting with a guaranteed empty directory, but it's not
  very important in this case as non-existence of other paths is being checked
  explicitly