Onionprobe Triage

I did some brainstorming on fixes and improvements that needs to be evaluated and broken into separate issues:

• Descriptor:
  • Option --save_descriptor that dumps descriptors into .desc files.
  • Add a --descriptor-only to just get the descriptor, bypassing the connection tests.
• Probes:
  • Create a probes folder and base class.
  • HTTP, TLS, Certificates, Onion-Location etc as probes.
  • Probes can declare it's dependencies and can be enabled/disabled by global or per-endpoint config.
  • Probes declare their own metrics, which are evaluated upon startup.
  • In the future: support for more service discovery probes.
  • Add support for pluggable probes (Roadmap: Future).
• HTTP probe:
  • Get all HTTP headers.
• CI/CD:
  • Schedule regular jobs for pages, slides, debian, configs, python.
  • Debian: check/report lintian errors.
  • Tests:
    • Tor's GitLab Onion Service.
    • Sample Onion Services with many errors (descriptors not available, self-signed/expired/mismatched certs, misc status codes etc).
    • Test with different locales?
    • Command line tests: ./onionprobe -e bbcnewsd73hkzno2ini43t4gblxvycyac5aw4gnv7t2rccijh7745uqd.onion ./onionprobe -e www.bbcnewsd73hkzno2ini43t4gblxvycyac5aw4gnv7t2rccijh7745uqd.onion ./onionprobe -e www.bbcnewsd73hkzno2ini43t4gblxvycyac5aw4gnv7t2rccijh7745uqd.onion:443 ./onionprobe -e https://bbcnewsd73hkzno2ini43t4gblxvycyac5aw4gnv7t2rccijh7745uqd.onion ./onionprobe -e https://www.bbcnewsd73hkzno2ini43t4gblxvycyac5aw4gnv7t2rccijh7745uqd.onion ./onionprobe -e https://www.bbcnewsd73hkzno2ini43t4gblxvycyac5aw4gnv7t2rccijh7745uqd.onion:443 ./onionprobe -e https://www.bbcnewsd73hkzno2ini43t4gblxvycyac5aw4gnv7t2rccijh7745uqd.onion:443/ ./onionprobe -e https://www.bbcnewsd73hkzno2ini43t4gblxvycyac5aw4gnv7t2rccijh7745uqd.onion:443/index.html
• Command line:
  • If no .onion is provided, connect to the site using Tor, look for Onion-Location and then test the service.
  • Support for multiple configuration files (merging endpoints and options; options found in later configs overrides the previous ones).
• Logging:
  • Stack tracing on errors when log level is debug using the traceback module.
  • Add more debug messages.
  • Add a silent log level.
• Docs:
  • Developing:
    • Procedure when adding new options.