Change Store::microdescs to return an Iterator? Or to parse incrementally?
The microdescs
method in SqliteStore returns a string-to-descriptor map; that means that it copies a whole pile of data out of the database before that data is parsed. It would be better for memory usage if we parsed the data as we read it, rather than allocating megabytes at a time.
(Showed up in profiles for #87)
I can think of a few ways to do this:
- we could have
microdescs()
return an iterator of (MdDigest, String). - we could call
microdescs()
with smaller inputs. - we could parse the results of
microdescs()
as we go along, perhaps by having it take a closure that tells it how to interpret its strings?