Index to better support downloaders
Hi Karsten, as discussed at the dev meeting it would be sweeeeeet for Stem to have a CollecTorDownloader class that supports getting and processing descriptors from CollecTor.
Presently you read and process the Apache index html, but it would it could be a lot nicer to have a machine readable index.json file instead. This is the ticket for it.
To support future expansion it would be nice to be able to include additional metadata, so I propose a 'contents' hash, such as...
https://collector.torproject.org/recent/index.json
{
'resource': 'https://collector.torproject.org/recent/index.json',
'contents': [
{
'name': 'bridge-descriptors',
'type': 'directory',
'last_modified': '31-May-2014 09:49', # or a unix timestam?
'url': 'https://collector.torproject.org/recent/bridge-descriptors/'
},
{
'name': 'exit-lists',
'type': 'directory',
'last_modified': '11-Oct-2015 15:02',
'url': 'https://collector.torproject.org/recent/exit-lists/'
},
{
'name': 'example-file.txt',
'type': 'file',
'last_modified': '31-May-2014 09:50',
'size': 42901,
'url': 'https://collector.torproject.org/recent/example-file.txt'
}
]
}