In #26913 we solved a bug where CacheDirectoryGroupReadable would
override DataDirectoryGroupReadable when the two directories are the
same.  We never did the same for KeyDirectory, though, because
that's a rare setting.

Now that I'm testing this code, though, fixing this issue seems
fine.  Fixes bug #27992; bugfix on 0.3.3.1-alpha.