Config testing - default config, examples, test cases, reuse - open questions
We would like to improve the way we handle config test cases, particularly with respect to the example config file.
This was prompted by @nickm's remarks !744 (comment 2842467)
We had a discussion on irc with some ideas. These need consideration and refinement.
19:00 <+Diziet> I have a question [about that comment]
19:00 <+Diziet> Do you mean thw way it couples the example config file to the
test case ?
19:01 <+Diziet> The advantage of doing it this way is that we test that the
nontrivial (nondefault) examples in the default config are
correct.
19:01 <+Diziet> (And we do want some such nontrivial examples in the file, I
think, given the complexity of bridge line syntax)
19:02 <+nickm> Diziet: yes, that's what I mean
19:02 <+nickm> The disadvantage is that if we change the nontrivial nondefault
examples (such as by adding a 4th line) we break the test
19:03 <+nickm> or if we format them differently
19:03 <+Diziet> Strictly, the things the test case hardwires about the default
config are 1. That the example start with "# For example:" 2.
That there's a ''' and a [ ] example that mean the same thing
3. That the example has 3 bridge lines
19:04 <+Diziet> The formatting for the ''' and [ ] are the only sensible
formatting I think, but you're right about the 4th example.
19:04 <+nickm> Also, there's the issue that these three requirements are
undocumented
19:04 <+Diziet> I wanted to add a test that it wasn't zero and decided
asserting 3 would be OK but maybe you are right that that is
too brittle.
19:04 <+Diziet> nickm: undocumented> That is a good point.
19:05 <+Diziet> We can't really document them in the example file, but I could
put commentary in the test case explaining how to fix it when
it breaks
19:05 <+nickm> I think that would be a good start
19:05 <+Diziet> I'm open to alternative suggestions.
19:06 <+nickm> I still think we will eventually need a better suite of config
tests, but this is probably good enough for now
19:06 <+Diziet> I'm not sure if you're arguing in favour of not trying to test
parse the nondefault examples at all
19:06 <+Diziet> If you're arguing that there ought to be an additional test
(perhaps not right away) I'm not opposed to that.
19:06 <+nickm> I don't know exactly what I'm arguing for, but that's okay since
I don't think we should do it now
19:07 <+Diziet> Hah. OK :-)
19:07 <+Diziet> Are you mostly concerned that the coverage might be poor, or
that the test is brittle ?
19:07 <+nickm> So, I think that we should build our configuration testing logic
out of general rules and special cases, and this test is
annoyingly in-between: it is a test specific to a single option
that requires specialization.
19:07 <+Diziet> Or something else ?
19:08 <+Diziet> I am imagining that other nondefault options we want to write
examples for we would handle with the ExampleSectionLines thing
19:08 <+nickm> To be clear , this is NOT what I am arguing for doing now, but I
think that 1) we could have a general way to specify a
non-default option that should be tested for well-formedness.
19:08 <+Diziet> If we don't want the examples in the file then the test case
needs to be in cfg.rs
19:08 <+nickm> (in the config file)
19:08 <+nickm> and 2) we could have a general way to specify two alternative
formats of the same option that should be checked for equivalence
19:09 <+Diziet> What, some kind of little markup dropping in the example file ?
19:09 <+nickm> yeah. Or have a standardized piece of text like "Other
examples:"
19:10 <+Diziet> I'm not against that in principle but there then starts to be
more tension between making the file useful to users and making
it useful as part of our tests.
19:10 <+Diziet> Another option would be to write a thing to filter out these
droppings and provide a version of the file without them. Not
sure I like that.
19:11 <+nickm> It could be worse.
19:11 <+nickm> yet another partial option would be to distinguish tests that
are about the ocnfig file from tests that are about the
configuration code.
19:12 <+Diziet> nickm: distinguish tests> Yes. Although the example config
file can make a good source of test cases, and I think it's
nice to be able to use the same text for both