al asked for disabling DocTor checks as he shut down tor temporarily
due to load issues and time to cope with them.

Good catch from starlight on ticket #32842 that these require a comma so the
constants are tuples rather than strings.

The last commit added unused DIRAUTH_SKIP_REACHABLE and DIRAUTH_SKIP_SEEN
constants. If we have use for this at some point then happy to reintroduce
them, but for the moment dead code is unhelpful.

Starlight authored 5 years ago and Damian Johnson committed 5 years ago

replace hard coded exclusion of authority tor26 with three
lists that during checks determine which authorities to not
a) check reachability for
b) check presence in consensus for
c) request and validate documents from

correct defect where router-line IPv4 OR address was not
checked for reachability

eliminate Python 3 exception by truncating array index
expression result to integer (further 2to3 revisions
applied during testing but not committed)

This reverts commit 3f1c5f67 and 7b121dc3.

Dannenberg will be unable to sign the consensus until November due to lack of
the present consensus in Tor's LTS release (and by extension openbsd's).

Dizum moved...

  https://trac.torproject.org/projects/tor/ticket/31406

Alex has requested monitoring of its old address, which will hopefully forward
until clients that reference it get deprecated.

Happy new year! Bumping the dates for 2019...

  % find . -type f -iname '*.py' -exec sed -i 's/-2018/-2019/g' "{}" +;
  % grep -R "# Copyright 2018," ./*

Only moria1 votes on the new flag. Treating it in the same way as BadExit...

  https://lists.torproject.org/pipermail/tor-consensus-health/2018-December/009451.html

Correction thanks to juga and nusenu.

Oops, need to be a tad more defensive here...

	Traceback (most recent call last):
  	File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 976, in <module>
    	main()
	  File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 258, in main
  	  issues += run_checks(consensuses, votes)
	  File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 352, in run_checks
  	  issues = checker(latest_consensus, consensuses, votes)
	  File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 640, in has_authority_flag
  	  seen_authorities.remove('tor26')
	KeyError: 'tor26'

Stem no longer provides this information. Hardcoding it in DocTor.

Using stem's new stem.directory module rather than the deprecated aliases.

Interesting! Roger reported an issue where he isn't getting notified about
moria1's bwauth scanner being down. In taking a look at the logs I'm indeed
seeing something funky...

  04/21/2018 21:05:46 [DEBUG] NOTICE: The following directory authorities are not reporting bandwidth scanner results: gabelmoo
  04/21/2018 21:05:47 [INFO] Suppressing The_following_directory_authorities_are_not_reporting_bandwidth_scanner_results:_gabelmoo, time remaining is 1
  3 hours
  04/21/2018 21:05:47 [INFO] All 1 issues were suppressed. Not sending a notification.
  04/21/2018 21:05:47 [DEBUG] Checks finished, runtime was 46.84 seconds
  04/21/2018 22:05:45 [DEBUG] ERROR: The following directory authorities are not reporting bandwidth scanner results: gabelmoo, moria1
  04/21/2018 22:05:46 [INFO] All 1 issues were suppressed. Not sending a notification.
  04/21/2018 22:05:46 [DEBUG] Checks finished, runtime was 45.56 seconds

I suspect what's happening is this...

  * When only gabelmoo is down this is a NOTICE runlevel notification, which
    works.

  * When moria1 is down as well it's an ERROR runlevel notice which should
    generate an email every hour, but isn't. I think this is due to a bug where
    ERROR notices in specific are getting incorrectly suppressed.

Lets give this a try...

Oops. The message included a float that changes each time, causing us to send a
notice every hour rather than daily as intended for notice runlevel messages.

Roger had the great suggestion that we should check for clock skew...

  https://trac.torproject.org/projects/tor/ticket/25768

Expanded stem a bit along with this to surface response headers.

To cut down on abuse tor26 rejects DirPort requests without a '.z' suffix
(technically it throttles them, but I've never seen one work so effectively
it's blocked). Stem no longer appends this suffix, using headers to indicate
the compression it wants instead.

When four authorites have the BadExit flag and one does not we only need to
notify minority part (and same vice versa). Noticed thanks to Linus.

Stem's integ tests ran into an issue where tor26 lacked moria1's server
descriptor. Turns out this is a longstanding issue due to tor26 holding
on to an outdated ed25519 key. Providing a notice for this.

When an authority first starts up it doesn't vote on the Running flag. Smarter
thing would be to check the known-flags in the vote to see if Running is there,
but for now just dropping the false alarms...

  https://lists.torproject.org/pipermail/tor-consensus-health/2018-February/008580.html

The counts vary every hour so time based suppressions didn't work.

Suppression requested by Roger...

  https://lists.torproject.org/pipermail/tor-dev/2018-February/012943.html

Recently dannenberg stopped issuing the Exit flag. Check Requested by nusenu...

https://lists.torproject.org/pipermail/tor-dev/2018-February/012918.html

Disabling this per Roger's request.

Silly I suppose to do right before new years, but just noticed they still said
2016.

Earlier gabelmoo's bwauth had issues and after a while Sebastian disabled it.
During it we disabled this check so he woulnd't spam for having *or* omitting
bwauth measurements. Sebastian's now running a bwauth again so dropping this
exception.

Requested by Sebastian.

I need to be more careful about code I write before my morning coffee. :P

Stupid me, missing a bracket. Would be nice if we had a test suite for these... :/

Addressing a simple stacktrace I see on occasion...

  Traceback (most recent call last):
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 885, in <module>
      main()
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 244, in main
      issues += run_checks(consensuses, votes)
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 336, in run_checks
      issues = checker(latest_consensus, consensuses, votes)
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 790, in shared_random_commit_partitioning
      if commitment.commit != self_commitments[commitment.identity]:
  KeyError: u'D586D18309DED4CD6D57C18FDB97EFA96D330566'

Dropping the runlevel so it provides daily notifications rather than every four
hours. This is because maatuska's is down and there's no ETA for it.

Not entirely sure what's up yet but got the following stacktrace on 1/26/17...

  Traceback (most recent call last):
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 879, in <module>
      main()
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 244, in main
      issues += run_checks(consensuses, votes)
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 336, in run_checks
      issues = checker(latest_consensus, consensuses, votes)
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 810, in shared_random_reveal_partitioning
      our_reveal = [c.reveal for c in vote.directory_authorities[0].shared_randomness_commitments if c.identity == our_v3ident][0]
  IndexError: list index out of range

Third check for https://trac.torproject.org/projects/tor/ticket/17434

Second check requested on https://trac.torproject.org/projects/tor/ticket/17434

Shared randomness does different things during different hours so need to add
special handling.

First of four shared random checks requested on...

  https://trac.torproject.org/projects/tor/ticket/17434

The descriptor checker (which only notifies me) already runs the descriptor
validation checks. No point in having them block consensus health notices.
Also, this lets our checks run a lot faster and with less memory usage.

Just addressing an interesting edge case that's currently cropping up because the consensus is invalid.

  Traceback (most recent call last):
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 802, in <module>
      main()
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 239, in main
      consensuses, consensus_fetching_issues = get_consensuses()
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 755, in get_consensuses
      return _get_documents('consensus', '/tor/status-vote/current/consensus.z')
    File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 790, in _get_documents
      median_time = sorted(times_taken.values())[len(times_taken) / 2]
  IndexError: list index out of range

Lets give this a try. Roger's tweaked moria1's torrc to get in sync with other
auths. This reverts commit c3f7557b.

This was disabled to avoid noise while removing urras. It's been long enough
that we can re-enable this.