Make the behavior for using _SC_OPEN_MAX vaguely sensible.

This code is one big logic error. See proposed fix.