High memory usage from parse_ns_body

From earlier today...

09:16 < atagar> Figured out my memory issue. It's from calling TorCtl's get_network_status function which consumes 3.5 MB of memory at once. 09:16 < atagar> I immediately discard those results and the memory is freed, however because of how large objects are allocated in python 'ps' is never aware that the memory's been released: 09:16 < atagar> http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete-a-large-object.htm 09:16 < atagar> Fix: either ignore it (it's misreported memory usage) or implement my own less memory intensive version of get_network_status. 09:20 < mikeperry> atagar: does python never release said memory? in otherwords, do you experience this as a leak, or just a consistent 3.5MB of unneeded overhead? 09:20 < mikeperry> s/release/reuse 09:22 < atagar> I'm not quite sure how to tell the difference. If I call get_network_status and discard the results (ie, don't get a reference to the nslist object it returns) then ps reports that memory as being used for the life of the python process. 09:23 < atagar> though I'd imagine this memory would be reused by python for other activity when needed 09:25 < mikeperry> hopefully. there is definitely a leak in the bw authorities that is proving ridiculously hard to track down. we're down to it either being a python leak, an sqlite leak, or a python-sqlite bindings leak 09:26 < atagar> side note: the memory all comes from the NetworkStatus objects generated on line 1246 (ie, the problem is definitely because we have a list with all that memory allocated at one time, rather than the incremental results contributing to it) 09:27 < atagar> ie, if we were able to have an iterator pattern here then there would be no issue 09:28 < mikeperry> if you want to write an intermediate funtion that only gives you them one at a time, you can do that. just don't change the external behavior of the current function. 09:28 < mikeperry> I'd merge that 09:29 < atagar> Will do. I'm a little worried about a future change accidently making a get_network_status call, but I'll edit the comment to warn of this. 09:29 < atagar> btw, do you have any idea when you'll be able to look at the five torctl patches I sent on Monday? 09:30 * nickm is going to be semi- offline for most of today afternoon doing errands writing reviews and hacking. Hoping this is a good idea. :) 09:34 < atagar> mikeperry: I'll also fix update_consensus so it doesn't encounter this issue (no external change but will avoid new consensus events from triggering huge memory usage)