Happy new year! Bumping the dates for 2019...

  % find . -type f -iname '*.py' -exec sed -i 's/-2018/-2019/g' "{}" +;
  % grep -R "# Copyright 2018," ./*

Only moria1 votes on the new flag. Treating it in the same way as BadExit...

  https://lists.torproject.org/pipermail/tor-consensus-health/2018-December/009451.html

Correction thanks to juga and nusenu.

Oops, need to be a tad more defensive here...

	Traceback (most recent call last):
  	File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 976, in <module>
    	main()
	  File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 258, in main
  	  issues += run_checks(consensuses, votes)
	  File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 352, in run_checks
  	  issues = checker(latest_consensus, consensuses, votes)
	  File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 640, in has_authority_flag
  	  seen_authorities.remove('tor26')
	KeyError: 'tor26'

Stem no longer provides this information. Hardcoding it in DocTor.

Using stem's new stem.directory module rather than the deprecated aliases.

Interesting! Roger reported an issue where he isn't getting notified about
moria1's bwauth scanner being down. In taking a look at the logs I'm indeed
seeing something funky...

  04/21/2018 21:05:46 [DEBUG] NOTICE: The following directory authorities are not reporting bandwidth scanner results: gabelmoo
  04/21/2018 21:05:47 [INFO] Suppressing The_following_directory_authorities_are_not_reporting_bandwidth_scanner_results:_gabelmoo, time remaining is 1
  3 hours
  04/21/2018 21:05:47 [INFO] All 1 issues were suppressed. Not sending a notification.
  04/21/2018 21:05:47 [DEBUG] Checks finished, runtime was 46.84 seconds
  04/21/2018 22:05:45 [DEBUG] ERROR: The following directory authorities are not reporting bandwidth scanner results: gabelmoo, moria1
  04/21/2018 22:05:46 [INFO] All 1 issues were suppressed. Not sending a notification.
  04/21/2018 22:05:46 [DEBUG] Checks finished, runtime was 45.56 seconds

I suspect what's happening is this...

  * When only gabelmoo is down this is a NOTICE runlevel notification, which
    works.

  * When moria1 is down as well it's an ERROR runlevel notice which should
    generate an email every hour, but isn't. I think this is due to a bug where
    ERROR notices in specific are getting incorrectly suppressed.

Lets give this a try...

Oops. The message included a float that changes each time, causing us to send a
notice every hour rather than daily as intended for notice runlevel messages.

Roger had the great suggestion that we should check for clock skew...

  https://trac.torproject.org/projects/tor/ticket/25768

Expanded stem a bit along with this to surface response headers.

To cut down on abuse tor26 rejects DirPort requests without a '.z' suffix
(technically it throttles them, but I've never seen one work so effectively
it's blocked). Stem no longer appends this suffix, using headers to indicate
the compression it wants instead.

When four authorites have the BadExit flag and one does not we only need to
notify minority part (and same vice versa). Noticed thanks to Linus.

Stem's integ tests ran into an issue where tor26 lacked moria1's server
descriptor. Turns out this is a longstanding issue due to tor26 holding
on to an outdated ed25519 key. Providing a notice for this.

When an authority first starts up it doesn't vote on the Running flag. Smarter
thing would be to check the known-flags in the vote to see if Running is there,
but for now just dropping the false alarms...

  https://lists.torproject.org/pipermail/tor-consensus-health/2018-February/008580.html

The counts vary every hour so time based suppressions didn't work.

Suppression requested by Roger...

  https://lists.torproject.org/pipermail/tor-dev/2018-February/012943.html

Recently dannenberg stopped issuing the Exit flag. Check Requested by nusenu...

https://lists.torproject.org/pipermail/tor-dev/2018-February/012918.html

Disabling this per Roger's request.

Silly I suppose to do right before new years, but just noticed they still said
2016.

Earlier gabelmoo's bwauth had issues and after a while Sebastian disabled it.
During it we disabled this check so he woulnd't spam for having *or* omitting
bwauth measurements. Sebastian's now running a bwauth again so dropping this
exception.

Requested by Sebastian.

I need to be more careful about code I write before my morning coffee. :P

Stupid me, missing a bracket. Would be nice if we had a test suite for these... :/

Addressing a simple stacktrace I see on occasion...

  Traceback (most recent call last):
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 885, in <module>
      main()
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 244, in main
      issues += run_checks(consensuses, votes)
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 336, in run_checks
      issues = checker(latest_consensus, consensuses, votes)
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 790, in shared_random_commit_partitioning
      if commitment.commit != self_commitments[commitment.identity]:
  KeyError: u'D586D18309DED4CD6D57C18FDB97EFA96D330566'

Dropping the runlevel so it provides daily notifications rather than every four
hours. This is because maatuska's is down and there's no ETA for it.

Not entirely sure what's up yet but got the following stacktrace on 1/26/17...

  Traceback (most recent call last):
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 879, in <module>
      main()
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 244, in main
      issues += run_checks(consensuses, votes)
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 336, in run_checks
      issues = checker(latest_consensus, consensuses, votes)
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 810, in shared_random_reveal_partitioning
      our_reveal = [c.reveal for c in vote.directory_authorities[0].shared_randomness_commitments if c.identity == our_v3ident][0]
  IndexError: list index out of range

Third check for https://trac.torproject.org/projects/tor/ticket/17434

Second check requested on https://trac.torproject.org/projects/tor/ticket/17434

Shared randomness does different things during different hours so need to add
special handling.

First of four shared random checks requested on...

  https://trac.torproject.org/projects/tor/ticket/17434

The descriptor checker (which only notifies me) already runs the descriptor
validation checks. No point in having them block consensus health notices.
Also, this lets our checks run a lot faster and with less memory usage.

Just addressing an interesting edge case that's currently cropping up because the consensus is invalid.

  Traceback (most recent call last):
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 802, in <module>
      main()
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 239, in main
      consensuses, consensus_fetching_issues = get_consensuses()
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 755, in get_consensuses
      return _get_documents('consensus', '/tor/status-vote/current/consensus.z')
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 790, in _get_documents
      median_time = sorted(times_taken.values())[len(times_taken) / 2]
  IndexError: list index out of range

Lets give this a try. Roger's tweaked moria1's torrc to get in sync with other
auths. This reverts commit c3f7557b.

This was disabled to avoid noise while removing urras. It's been long enough
that we can re-enable this.

Authorities have been out of sync for months with no sign it'll be fixed.
Disabling the check.

Reverting commit 10193a86 now that Tonga has been fully replaced by Bifroest.

Oops, tried to disable notices to Tonga but inadvertantly caused...

  NOTICE: The following authorities were not expected in the consensus: Tonga

Disabling this part of the check until Tonga's gone.

If the only disagreement about a BadExit is due to lacking a vote then don't
send a notice. There's already other notices about lacking a vote.

The directory authorities are in the process of removing urras so disabling
this as per Roger's request...

  https://lists.torproject.org/pipermail/tor-consensus-health/2016-July/007211.html

New notifications requested by teor that provides a summary if more than 25% of
the fallback directories are unreachable or slow. Notices are sent to teor,
nickm, me, and #tor-bots...

  https://trac.torproject.org/projects/tor/ticket/18177

Simple check that was requested at the Berlin dev meeting to see if the
authority's ORPort is reachable...

  https://trac.torproject.org/projects/tor/ticket/18176

This has been pending IPv6 support. Now that DocTor's running on carinatum this
can become a thing!

  https://trac.torproject.org/projects/tor/ticket/17298