GitLab

Closes #40023Signed-off-by: David Goulet <dgoulet@torproject.org>

dannenberg needs to get a handle on the overload situation first.

DIRAUTH_SKIP_CHECKS prevented us from downloading (and by extension checking)
a dirauth's consensus documents, but it did not suppress other checks (presence
in votes, ORPort reachability, etc). This expands the suppression to cover
everything that references the DIRECTORY_AUTHORITIES global, which should
hopefully be everything.

------------------------------------------------------------
DocTor Output Before
------------------------------------------------------------

WARNING: moria1 is missing the server descriptor of dannenberg
WARNING: moria1 is missing the server descriptor of Faravahar
WARNING: dizum is missing the server descriptor of dannenberg
WARNING: dizum is missing the server descriptor of Faravahar
WARNING: gabelmoo is missing the server descriptor of dannenberg
WARNING: gabelmoo is missing the server descriptor of Faravahar
WARNING: maatuska is missing the server descriptor of dannenberg
WARNING: maatuska is missing the server descriptor of Faravahar
WARNING: Faravahar is missing the server descriptor of dannenberg
WARNING: longclaw is missing the server descriptor of dannenberg
WARNING: longclaw is missing the server descriptor of Faravahar
WARNING: bastet is missing the server descriptor of dannenberg
WARNING: bastet is missing the server descriptor of Faravahar
WARNING: The following directory authorities do not support the consensus method that the consensus uses: Faravahar
NOTICE: Consensus fetched from dizum was missing the following authority signatures: dannenberg, Faravahar
NOTICE: Consensus fetched from gabelmoo was missing the following authority signatures: dannenberg, Faravahar
NOTICE: Consensus fetched from maatuska was missing the following authority signatures: dannenberg, Faravahar
NOTICE: Consensus fetched from Faravahar was missing the following authority signatures: dannenberg, Faravahar
NOTICE: Consensus fetched from longclaw was missing the following authority signatures: dannenberg, Faravahar
NOTICE: Consensus fetched from bastet was missing the following authority signatures: dannenberg, Faravahar
NOTICE: The following directory authorities are not reporting bandwidth scanner results: Faravahar
WARNING: The following authorities are missing from the consensus: dannenberg, Faravahar
NOTICE: Authorities disagree about the BadExit flag for 867982ADCFBE55EC1C00A5B34C8991FF87605906 (with flag: longclaw, gabelmoo, moria1, without flag: Faravahar)
NOTICE: Authorities disagree about the BadExit flag for B9324F1BEDB1DF62AEE219B0CC41507CE710BA3C (with flag: longclaw, gabelmoo, moria1, without flag: Faravahar)
NOTICE: Authorities disagree about the BadExit flag for 81F59E8936D0B2105182E4B8200AC07D9274961F (with flag: longclaw, gabelmoo, moria1, without flag: Faravahar)
NOTICE: Authorities disagree about the BadExit flag for 8A3606F17B1DDAC4F8BFA0C99E656FFA11CC747D (with flag: longclaw, gabelmoo, moria1, without flag: Faravahar)
NOTICE: Authorities disagree about the BadExit flag for 35449EB3D025CC24601FB43884F9699367D677CF (with flag: longclaw, gabelmoo, without flag: Faravahar, moria1)

------------------------------------------------------------
DocTor Output After
------------------------------------------------------------

WARNING: moria1 is missing the server descriptor of Faravahar
WARNING: dizum is missing the server descriptor of Faravahar
WARNING: gabelmoo is missing the server descriptor of Faravahar
WARNING: maatuska is missing the server descriptor of Faravahar
WARNING: longclaw is missing the server descriptor of Faravahar
WARNING: bastet is missing the server descriptor of Faravahar
WARNING: The following directory authorities do not support the consensus method that the consensus uses: Faravahar
NOTICE: Consensus fetched from moria1 was missing the following authority signatures: Faravahar
NOTICE: Consensus fetched from dizum was missing the following authority signatures: Faravahar
NOTICE: Consensus fetched from gabelmoo was missing the following authority signatures: Faravahar
NOTICE: Consensus fetched from maatuska was missing the following authority signatures: Faravahar
NOTICE: Consensus fetched from Faravahar was missing the following authority signatures: Faravahar
NOTICE: Consensus fetched from longclaw was missing the following authority signatures: Faravahar
NOTICE: Consensus fetched from bastet was missing the following authority signatures: Faravahar
NOTICE: The following directory authorities are not reporting bandwidth scanner results: Faravahar
WARNING: The following authorities are missing from the consensus: Faravahar
NOTICE: Authorities disagree about the BadExit flag for 81F59E8936D0B2105182E4B8200AC07D9274961F (with flag: gabelmoo, moria1, longclaw, without flag: Faravahar)
NOTICE: Authorities disagree about the BadExit flag for 35449EB3D025CC24601FB43884F9699367D677CF (with flag: gabelmoo, longclaw, without flag: Faravahar, moria1)
NOTICE: Authorities disagree about the BadExit flag for B9324F1BEDB1DF62AEE219B0CC41507CE710BA3C (with flag: gabelmoo, moria1, longclaw, without flag: Faravahar)
NOTICE: Authorities disagree about the BadExit flag for 8A3606F17B1DDAC4F8BFA0C99E656FFA11CC747D (with flag: gabelmoo, moria1, longclaw, without flag: Faravahar)
NOTICE: Authorities disagree about the BadExit flag for 867982ADCFBE55EC1C00A5B34C8991FF87605906 (with flag: gabelmoo, moria1, longclaw, without flag: Faravahar)

This got caught by the 2to3 tool.

danneberg temporarily migrated to FreeBSD to fix the LibreSSL issue
(see: tpo/core/tor#40128). Andreas agreed, too, to remove the previous
special treatment for dannenberg which got implemented earlier this
year while the directory authorities were under heavy load.

Bug 40005: Discard dannenberg from missing_authorities

See merge request !1

Until tor#40128 is solved there is not much the operator of dannenberg
can do.

`UseNTorHandshake` and `Support022HiddenServices` are obsolete. Adding
`ExtendByEd25519ID`.

al asked for disabling DocTor checks as he shut down tor temporarily
due to load issues and time to cope with them.

Oops! Georg shouldn't get these notifications. This script is a health check
for my personal relay.

Georg has requested ownership of this service so changing all administrative
notifications to him.

That said, I'll keep package_versions.py since it is unrelated to network
health.

Empty commit to hopefully trigger mirroring into GitHub...

  https://github.com/torproject/doctor

Good catch from starlight on ticket #32842 that these require a comma so the
constants are tuples rather than strings.

The last commit added unused DIRAUTH_SKIP_REACHABLE and DIRAUTH_SKIP_SEEN
constants. If we have use for this at some point then happy to reintroduce
them, but for the moment dead code is unhelpful.

replace hard coded exclusion of authority tor26 with three
lists that during checks determine which authorities to not
a) check reachability for
b) check presence in consensus for
c) request and validate documents from

correct defect where router-line IPv4 OR address was not
checked for reachability

eliminate Python 3 exception by truncating array index
expression result to integer (further 2to3 revisions
applied during testing but not committed)

This reverts commit 3f1c5f67 and 7b121dc3.

Oops, I also get notifications for this daemon as well.

Dannenberg will be unable to sign the consensus until November due to lack of
the present consensus in Tor's LTS release (and by extension openbsd's).

Dizum moved...

  https://trac.torproject.org/projects/tor/ticket/31406

Alex has requested monitoring of its old address, which will hopefully forward
until clients that reference it get deprecated.

Correcting a regression with the earlier change...

  Traceback (most recent call last):
    File "/srv/doctor.torproject.org/doctor/fallback_directories.py", line 92, in <module>
      main()
    File "/srv/doctor.torproject.org/doctor/fallback_directories.py", line 87, in main
      util.send('Announce or', body = body, to = ['tor-misc@commit.noreply.org'])
  NameError: global name 'body' is not defined

Adjusting the irc notifications per teor's request...

  https://trac.torproject.org/projects/tor/ticket/30940

In particiular notifications now begin with a count summary (like emails), and
present at most four relays. For example...

  [fallback-directories] 157/157 (100%) fallback directories have become slow or unresponsive...
  [fallback-directories] 8FA37B93397015B2BC5A525C908485260BE9F422 => ORPort is unreachable (81.7.11.96:9001)
  [fallback-directories] 36B9E7AC1E36B62A9D6F330ABEB6012BA7F0D400 => ORPort is unreachable (37.187.22.87:9001)
  [fallback-directories] 230A8B2A8BA861210D9B4BA97745AEC217A94207 => ORPort is unreachable (163.172.176.167:443)
  [fallback-directories] 823AA81E277F366505545522CEDC2F529CE4DC3F => ORPort is unreachable (192.160.102.164:9001)
  [fallback-directories] ... and 153 more

Adjusting notificaiton recipients per Colin's request.

Recently netbsd changed its tor version to "0.3.5.8nb2"...

  http://pkgsrc.se/net/tor

Clearly the 'nb2' suffix isn't part of our upstream version. I'm not entirely
sure what it means (maybe a re-packaging of 0.3.5.8?). Adjusting the version
parsing I use for netbsd to ignore it.

Dropping the threshold and adding Colin as requested on...

  https://trac.torproject.org/projects/tor/ticket/29094

Happy new year! Bumping the dates for 2019...

  % find . -type f -iname '*.py' -exec sed -i 's/-2018/-2019/g' "{}" +;
  % grep -R "# Copyright 2018," ./*

Only moria1 votes on the new flag. Treating it in the same way as BadExit...

  https://lists.torproject.org/pipermail/tor-consensus-health/2018-December/009451.html

https://gitweb.torproject.org/nyx.git/commit/?id=f14cf44
https://gitweb.torproject.org/stem.git/commit/?id=0d09943

Great point from Iain that we should be keeping an eye on the source url rather
than individual packages.

Heard back from Iain. He indicated that python2 packages are being removed for
Debian's next release, but txtorcon's python3 package is still around.

Our wiki now links to different pages. Scanning the same for version
information.

Unfortunately the page line with the version don't have any anchors to key off,
but in practice this seems fine for the moment.

The wiki no longer points toward the 'wip' page so we can now include it like
our other packages.

The page we link to has been gone for about a week. A few days ago I reached
out to Iain to see if the package was renamed but haven't yet heard back.

Our PACKAGES constant match our wiki's layout so there's no need to have a
separate dictionary with static indices. In the process also learned that we
weren't tracking Nyx's Fedora package.

Txtorcon also now has a NetBSD package so tracking that as well.

Added new packages to the wiki so gotta adjust this script as well. Yeah, yeah,
stinky I do all this index stuff but the point of this script is to be an
effort saving daemon for me so meh.