Compare anguilla's tarballs to yatei's and maybe merge them
weasel was running his directory-archive script until a week or two ago. I want to compare anguilla's tarballs to yatei's to figure out if yatei is missing some descriptors and why, and to merge missing descriptors into yatei's tarballs.
This ticket is mostly here to note down the endless hours that I already worked on this task, mostly because I need to write new comparison scripts and investigate differences between single descriptors manually before identifying a pattern. As of now, I spent 9 points on this task, and I'm not done. I think another 3 points remain. The task looked so tiny when I decided to do it, but it's also important enough to spend the remaining points.
Current insights from the comparison, which might turn into new tasks, are:
Quite a few of the consensuses collected by yatei have missing or extraneous signatures as compared to anguilla's. This has to do with authorities serving consensuses that don't have all signatures. I don't really care, so I'm probably leaving this alone.
Quite often, missing a consensus automatically means missing all votes. We might switch to downloading votes by all known authorities, not only by the ones contained in a consensus (which we're missing in these cases). Not super important, but probably worth doing.
We have quite a few files in yatei's tarballs that are empty or truncated. We need to try parsing descriptors with metrics-lib (which is not yet used by metrics-db) and only store valid descriptors to disk.