Add research idea to Run some onion services to observe crawling trends
We know some research groups that are doing full crawling of onion services. We also know that Ahmia et al are doing it. I keep hearing these days about big security companies selling "onion intelligence" or the like.
What are the characteristics of these crawls? Are many of them one level deep, or k levels deep, or full crawls? Do they obey robots.txt? Do they identify themselves by their user agent? Do they visit urls that are embedded in html comments that humans would never find? Do they de-obfuscate urls and visit those? Do they get suckered by web tarpits that produce infinite pages? Are the crawling trends going up quickly or slowly?
We should consider running a couple of onion services with various characteristics, and monitor their usage and see if we learn anything.