Dropping the threshold and adding Colin as requested on...

  https://trac.torproject.org/projects/tor/ticket/29094

Happy new year! Bumping the dates for 2019...

  % find . -type f -iname '*.py' -exec sed -i 's/-2018/-2019/g' "{}" +;
  % grep -R "# Copyright 2018," ./*

Only moria1 votes on the new flag. Treating it in the same way as BadExit...

  https://lists.torproject.org/pipermail/tor-consensus-health/2018-December/009451.html

https://gitweb.torproject.org/nyx.git/commit/?id=f14cf44
https://gitweb.torproject.org/stem.git/commit/?id=0d09943

Great point from Iain that we should be keeping an eye on the source url rather
than individual packages.

Heard back from Iain. He indicated that python2 packages are being removed for
Debian's next release, but txtorcon's python3 package is still around.

Our wiki now links to different pages. Scanning the same for version
information.

Unfortunately the page line with the version don't have any anchors to key off,
but in practice this seems fine for the moment.

The wiki no longer points toward the 'wip' page so we can now include it like
our other packages.

The page we link to has been gone for about a week. A few days ago I reached
out to Iain to see if the package was renamed but haven't yet heard back.

Our PACKAGES constant match our wiki's layout so there's no need to have a
separate dictionary with static indices. In the process also learned that we
weren't tracking Nyx's Fedora package.

Txtorcon also now has a NetBSD package so tracking that as well.

Added new packages to the wiki so gotta adjust this script as well. Yeah, yeah,
stinky I do all this index stuff but the point of this script is to be an
effort saving daemon for me so meh.

The debian package link no longer works. Reached out to Arturo and Iain to see
if that's intentional, and if the package has possibly been renamed.

Correction thanks to juga and nusenu.

This has been notifying everyone for weeks due to breaching the 25% thresold.
Four days ago I reached out to teor and Nick to see what we want to do with
this but haven't heard back.

DocTor notices need to spur action or they're just noise. Bumping the threshold
to 50% so this no longer emails us.

Requested by teor on 8/21 (yeah, I'm really late in getting around to this).

All these entries have now expired. Dropping them from the tracked list.

FreeBSD's torsocks package changed its tarball name. It's a good change (prior
one was odd), but means we need to adjust our regex a tad to match.

Requested by nusenu.

Oops, need to be a tad more defensive here...

	Traceback (most recent call last):
  	File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 976, in <module>
    	main()
	  File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 258, in main
  	  issues += run_checks(consensuses, votes)
	  File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 352, in run_checks
  	  issues = checker(latest_consensus, consensuses, votes)
	  File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 640, in has_authority_flag
  	  seen_authorities.remove('tor26')
	KeyError: 'tor26'

The whole point of our fallback directory daemon is to tell us how many
fallbacks are unusable, but now that one of their DirPorts is misbehaving
it's resulting in uncaught exceptions. Interestingly, we're getting quite a
range...

  Traceback (most recent call last):
    File "fallback_directories.py", line 82, in <module>
      main()
    File "fallback_directories.py", line 57, in main
      downloader.get_consensus(endpoints = [(relay.address, relay.dir_port)]).run()
    File "/home/atagar/Desktop/tor/doctor/stem/descriptor/remote.py", line 445, in run
      return list(self._run(suppress))
    File "/home/atagar/Desktop/tor/doctor/stem/descriptor/remote.py", line 456, in _run
      raise self.error
  HTTPError: HTTP Error 404: Not Found

... as well as...

  HTTPError: HTTP Error 503: Directory busy, try again later

... or...

  CertificateError: hostname '85.214.62.48' doesn't match either of 'andamur.com', 'www.andamur.com'

Our DescriptorDownloader raises an unpleasantly large array of exceptions so
unfortunately requires a catch-all here (maybe I'll change that at some point
in stem...).

Stem no longer provides this information. Hardcoding it in DocTor.

Using stem's new stem.directory module rather than the deprecated aliases.

This adds the fingerprint entries from approved-routers.conf. 3 months
expiry date has been set for those.

Closes #26055

Signed-off-by: David Goulet <dgoulet@torproject.org>

This only adds the IP address from bad.conf. 3 months expiry date has
been set for those.

The fingerprints from approved-routers.conf are coming in a second
following commit.

Part of #26055

Signed-off-by: David Goulet <dgoulet@torproject.org>

Oops, since we iterate it suppresses errors resulting in a pretty unhelpful
notice...

  Traceback (most recent call last):
    File "/srv/doctor.torproject.org/doctor/relay_check.py", line 63, in <module>
      main()
    File "/srv/doctor.torproject.org/doctor/relay_check.py", line 32, in main
      ))[0]
  IndexError: list index out of range

Now that Stem can download descriptors from an ORPort this health check can be
quite a bit nicer.

Interesting! Roger reported an issue where he isn't getting notified about
moria1's bwauth scanner being down. In taking a look at the logs I'm indeed
seeing something funky...

  04/21/2018 21:05:46 [DEBUG] NOTICE: The following directory authorities are not reporting bandwidth scanner results: gabelmoo
  04/21/2018 21:05:47 [INFO] Suppressing The_following_directory_authorities_are_not_reporting_bandwidth_scanner_results:_gabelmoo, time remaining is 1
  3 hours
  04/21/2018 21:05:47 [INFO] All 1 issues were suppressed. Not sending a notification.
  04/21/2018 21:05:47 [DEBUG] Checks finished, runtime was 46.84 seconds
  04/21/2018 22:05:45 [DEBUG] ERROR: The following directory authorities are not reporting bandwidth scanner results: gabelmoo, moria1
  04/21/2018 22:05:46 [INFO] All 1 issues were suppressed. Not sending a notification.
  04/21/2018 22:05:46 [DEBUG] Checks finished, runtime was 45.56 seconds

I suspect what's happening is this...

  * When only gabelmoo is down this is a NOTICE runlevel notification, which
    works.

  * When moria1 is down as well it's an ERROR runlevel notice which should
    generate an email every hour, but isn't. I think this is due to a bug where
    ERROR notices in specific are getting incorrectly suppressed.

Lets give this a try...

Great first use of our new ORPort capabilities. Using it to provide a health
check when my relay's down. If this goes well I'l add it to our Stem tutorials.

Yeah, yeah, I know. Not dirauth related. But hey, there's no reason DocTor
can't provide outage notifications to me as well! :P

Strange. It doesn't commonly happen but as reported in...

  https://trac.torproject.org/projects/tor/ticket/25789

... sometimes we list a relay twice. My only guess is that this arises if a
particular address/port is a relay in the consensus twice. Adjusting the code
in a way that should deduplicate these.

Oops. The message included a float that changes each time, causing us to send a
notice every hour rather than daily as intended for notice runlevel messages.

Roger had the great suggestion that we should check for clock skew...

  https://trac.torproject.org/projects/tor/ticket/25768

Expanded stem a bit along with this to surface response headers.

Oops, forgot about a second validation script that's spamming me.

To cut down on abuse tor26 rejects DirPort requests without a '.z' suffix
(technically it throttles them, but I've never seen one work so effectively
it's blocked). Stem no longer appends this suffix, using headers to indicate
the compression it wants instead.

When four authorites have the BadExit flag and one does not we only need to
notify minority part (and same vice versa). Noticed thanks to Linus.

Stem's integ tests ran into an issue where tor26 lacked moria1's server
descriptor. Turns out this is a longstanding issue due to tor26 holding
on to an outdated ed25519 key. Providing a notice for this.

When an authority first starts up it doesn't vote on the Running flag. Smarter
thing would be to check the known-flags in the vote to see if Running is there,
but for now just dropping the false alarms...

  https://lists.torproject.org/pipermail/tor-consensus-health/2018-February/008580.html

The counts vary every hour so time based suppressions didn't work.

Suppression requested by Roger...

  https://lists.torproject.org/pipermail/tor-dev/2018-February/012943.html