We should be smarter about fetching all missing votes
If soemthing has gone quite wrong, and we as an authority have no votes, we'll try to fetch every vote from every other authority. That's quite a lot of data! We ran into trouble with this as legacy/trac#14261 (moved) , and increased the limit, but the base scenario here isn't so great.