Changes to default ruleset state need to work even if the rule was previously present

This was something we have worked around until now. But when we fix #8774 (moved) we don't want to have to re-name all of the affected rulesets.