Functionally teor's patch is fine as test/__init__.py is only imported
in testing contexts. That said, as a general practice it's better to
invoke functional code (like signal listeners) when a module is invoked
rather than imported.

All testing is invoked through run_tests.py's main method, so we can
register our signal handlers there instead.

teor authored 6 years ago and Damian Johnson committed 6 years ago

SIGUSR1: log stack trace and continue
SIGABRT: log stack trace and exit(-1)

Closes ticket 30012.

Yikes. Illia's patch is phenominal, really the only really worth adding is the
changelog entry.

On reflection, process initialization time is cachable whereas uptime is not.
Adding a get_start_time() method, not only for its own usefulness but because
it inherrantly lets us make get_uptime() a cached method.

Taking advantage of our new getinfo option for determining the tor process
runtime.

  https://trac.torproject.org/projects/tor/ticket/25132
  https://gitweb.torproject.org/torspec.git/commit/?id=e9ef624

hannelorestetx authored 6 years ago and Damian Johnson committed 6 years ago

https://trac.torproject.org/projects/tor/ticket/25075

Tor recently added HSFETCH support for v3 services...

  https://trac.torproject.org/projects/tor/ticket/25417
  https://gitweb.torproject.org/torspec.git/commit/?id=34518e1

Leveraging this new capability takes nothing more than
providing the longer hidden service v3 address to the
command. This is neat since it means all we need to do
on our end is stop rejecting v3 addresses.

Our only use of is_valid_hidden_service_address() is
get_hidden_service_descriptor(), so simply adjusting our
helper to recognize both v2 and v3 addresses.

Good point from juga that the bandwidth file spec requires the 'version' header
to be in the second position (unless the document is version 1.0.0, in which
case the version attribute is not present at all).

  https://trac.torproject.org/projects/tor/ticket/29539

Ensuring the descriptors we create conform with this, and if the user parses a
bandwidth file with 'validate = True' then check this.

Couple small tweaks suggested by wagon:

  https://trac.torproject.org/projects/tor/ticket/28300#comment:11

Sadly I forget where it was pointed out, but invoking the control port via
shell is a *lot* faster than stem...

  #!/bin/bash -e

  cmd="$@"
  pass="ControlPortPassword"

  function test_tor() {
      echo "$1" >&3
      sed "/^250 OK\r$/q" <&3
      echo QUIT >&3
      exec 3<&-
  }

  exec 3<>/dev/tcp/127.0.0.1/9051
  echo AUTHENTICATE \"$pass\" >&3
  read -u 3
  test_tor "$cmd"

  ====================

  atagar@morrigan:~$ time ./bench.sh 'GETINFO version' 1>/dev/null

  real	0m0.007s
  user	0m0.004s
  sys	0m0.003s

  ====================

  atagar@morrigan:~$ time tor-prompt --run 'GETINFO version' 1>/dev/null

  real	0m0.186s
  user	0m0.072s
  sys	0m0.030s

Generally speaking this is expected. Spinning up an interpreter takes time. But
in doing a quick investigation realized this is quite a bit slower than it
needs to be...

  total tor-prompt runtime          0.186 seconds
  --------------------------------------------------------
  python interpreter startup        0.016 seconds (9%)
  import statements                 0.079 seconds (42%)
  check if tor is running           0.014 seconds (8%)
  connect to tor                    0.009 seconds (5%)
  autocompete setup                 0.065 seconds (34%)
  invoke tor controller command     0.003 seconds (2%)

Autocompletion is only relevant when the user is presented with an interactive
interpreter. If we're merely invoking a command it's pointless.

So TL;DR: tor-prompt is now ~34% faster when used to invoke controller commands.

Wonderful demo for authenticating using SAFECOOKIE by hand (thanks wagon!).

  https://trac.torproject.org/projects/tor/ticket/28300#comment:9

Listing a couple projects I've been meaning to take a peek at. Only took a
cursory glance but seems Mike and Iain did fantastic jobs.

I also hoped to list SelekTOR [1] somewhere. I don't know much about it, but
sounds like an interesting GUI project. However, it's java (and by extension
doesn't use stem) so I can't list it on the examples page. I have a separate
section for alternate language libraries but it's not that either, so guess I
don't have a spot to plop it. Oh well.

[1] https://www.dazzleships.net/selektor-for-linux/

Oops, forgot to re-run tests with python 3.x. Recent change had a bytes/unicode
regression slip in...

  https://trac.torproject.org/projects/tor/ticket/29332

  ======================================================================
  ERROR: test_format_v1_2
  ----------------------------------------------------------------------
  Traceback (most recent call last):
    File "/home/atagar/Desktop/stem/test/unit/descriptor/bandwidth_file.py", line 103, in test_format_v1_2
      self.assertEqual('1.2.0', desc.version)
    File "/home/atagar/Desktop/stem/stem/descriptor/__init__.py", line 1041, in __getattr__
      parsing_function(self, self._entries)
    File "/home/atagar/Desktop/stem/stem/descriptor/bandwidth_file.py", line 116, in _parse_header
      elif not header and 'node_id=' in line:
  TypeError: a bytes-like object is required, not 'str'

Nick formally documented the 'tor --version' output we rely on (thanks!)...

  https://gitweb.torproject.org/tor.git/commit/?id=0e6e902

Stem asserted the version is the last line of the output but that is not
specified, and I can see scenarios where we might add more text. Loosening our
parser to look for the version in any line of the output and expanding our
tests.

Tor adjusted its spec to make the flavor optional, and explicitly state that if
it is not present it should default to 'ns'...

  https://gitweb.torproject.org/torspec.git/commit/?id=d97f8d980be01bda3f6122fc6220406f4592a717

Good point from starlight that we should default to downloading compressed
rather than plaintext descriptors...

  https://trac.torproject.org/projects/tor/ticket/29186

I purposefully chose a plaintext default because when I added Stem
compression support tor had relability issues with it...

  https://trac.torproject.org/projects/tor/ticket/9379

... but that hasn't been the case for years. DocTor has been downloading
compressed descriptors since 2015 so it's well past time we took some
confidence in changing the default.

  https://gitweb.torproject.org/doctor.git/commit/?id=a8e954f

Thanks to Karsten we've settled on @type annotations for several new document
types...

  https://trac.torproject.org/projects/tor/ticket/28615

Doing the needful on Stem's end to take advantage of these.

Great catch from starlight that if descriptors downloaded via an ORPort
exceeding a certain size get cropped...

  https://trac.torproject.org/projects/tor/ticket/28961

There were a couple issues here...

  * Our socket handling was pretty screwed up. Calling socket's recv() provides
    data available at the time, so if we haven't received the a full cell we
    should pull more.

  * Descriptors can be composed of multiple RELAY cells (the descriptor was
    cropped because we stopped reading after the first). Directory requests
    now keep reading cells we receive the END signal.

Noting both our stem and tor git commit in our test output when available...

  ======================================================================
                               INITIALISING
  ======================================================================

    stem version...                            1.7.1-dev (commit 2e86567b)
    tor version...                             0.4.0.1-alpha-dev (commit 8b254c1c)
    python version...                          2.7.12
    operating system...                        Linux (Ubuntu 16.04)
    cryptography version...                    2.0.3
    ... etc...

Oops. I added a timeout attribute but completely forgot to catch the resulting
exceptions. Caught thanks to starlight...

  https://trac.torproject.org/projects/tor/ticket/28961

  Traceback (most recent call last):
    File "../download_descriptor.py", line 133, in <module>
      main()
    File "../download_descriptor.py", line 115, in main
      endpoints = [args.download_from],
    File "/usr/local/lib/python3.7/site-packages/stem/descriptor/remote.py", line 485, in run
      return list(self._run(suppress))
    File "/usr/local/lib/python3.7/site-packages/stem/descriptor/remote.py", line 496, in _run
      raise self.error
    File "/usr/local/lib/python3.7/site-packages/stem/descriptor/remote.py", line 561, in _download_descriptors
      self.content, self.reply_headers = _download_from_orport(endpoint, self.compression, self.resource)
    File "/usr/local/lib/python3.7/site-packages/stem/descriptor/remote.py", line 962, in _download_from_orport
      with relay.create_circuit() as circ:
    File "/usr/local/lib/python3.7/site-packages/stem/client/__init__.py", line 207, in create_circuit
      for cell in self._msg(create_fast_cell):
    File "/usr/local/lib/python3.7/site-packages/stem/client/__init__.py", line 160, in _msg
      response = self._orport.recv(timeout = 1)
    File "/usr/local/lib/python3.7/site-packages/stem/socket.py", line 416, in recv
      return self._recv(wrapped_recv)
    File "/usr/local/lib/python3.7/site-packages/stem/socket.py", line 274, in _recv
      return handler(my_socket, my_socket_file)
    File "/usr/local/lib/python3.7/site-packages/stem/socket.py", line 410, in wrapped_recv
      return s.recv()
    File "/usr/local/lib/python3.7/ssl.py", line 1037, in recv
      return self.read(buflen)
    File "/usr/local/lib/python3.7/ssl.py", line 913, in read
      return self._sslobj.read(len)
  socket.timeout: The read operation timed out

Through experimentation I also managed to get the following too...

  Traceback (most recent call last):
    File "download_descriptor.py", line 131, in <module>
      main()
    File "download_descriptor.py", line 113, in main
      endpoints = [args.download_from],
    File "/home/atagar/Desktop/stem/stem/descriptor/remote.py", line 485, in run
      return list(self._run(suppress))
    File "/home/atagar/Desktop/stem/stem/descriptor/remote.py", line 496, in _run
      raise self.error
  ssl.SSLError: ('The read operation timed out',)

Unfortunately since this is an SSL socket (rather than a standard one) receive
calls are liable to raise exceptions from both modules making this a bit of a
whack-a-mole game (socket's docs are particularly bad about describing the
array of exceptions that might get raised).

Teor adjusted the bandwidth file spec so it is now at the discretion of parsers
for how we behave when bandwidth files list a relay multiple times. My other
descriptor parsers bias toward strictness so doing so here too. Doing so will
hopefully help uncover if we ever get sbws regressions, and we can always
adjust this back to something looser later.

Code review issues caught by teor (thanks!)...

  https://github.com/torproject/stem/commit/9cac9085504230e036ff65754d88a349ad88d549
  https://trac.torproject.org/projects/tor/ticket/29056#comment:16

Most notably sbws originally used four character header dividers. Tor expected
five so sbws changed, but for backward compatability both are allowed.

Huh. Guess I don't test via python3 much. Commit 5488849b changed our annotation
values to unicode. Unsure that was intentional in that commit but I think
that's preferable so adjusting our tests.

  ======================================================================
  FAIL: test_annotations
  ----------------------------------------------------------------------
  Traceback (most recent call last):
    File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched
      return func(*args, **keywargs)
    File "/home/atagar/Desktop/stem/test/unit/descriptor/server_descriptor.py", line 739, in test_annotations
      self.assertEqual(b'@pepperjack very tasty', desc.get_annotation_lines()[0])
  AssertionError: b'@pepperjack very tasty' != '@pepperjack very tasty'

  ----------------------------------------------------------------------

Though these passed for me a few times for me undefinied ordering finally
caught up with me. Simple to fix. Correcting some pydocs too.

  ======================================================================
  FAIL: test_new_header_attribute
  ----------------------------------------------------------------------
  Traceback (most recent call last):
    File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched
      return func(*args, **keywargs)
    File "/home/atagar/Desktop/stem/test/unit/descriptor/bandwidth_file.py", line 162, in test_new_header_attribute
      self.assertEqual(EXPECTED_NEW_HEADER_CONTENT, str(desc))
  AssertionError: '1410723598\nversion=1.1.0\nnew_header=neat stuff\n=====' != '1410723598\nnew_header=neat stuff\nversion=1.1.0\n====='
    1410723598
  + new_header=neat stuff
    version=1.1.0
  - new_header=neat stuff
    =====

  ----------------------------------------------------------------------

Support for bandwidth authority metrics...

  https://trac.torproject.org/projects/tor/ticket/29056
  https://gitweb.torproject.org/torspec.git/tree/bandwidth-file-spec.txt

Just the regular bytes vs unicode chore.

String parsing was a great spot to start, but highly memory inefficient.
Parsing internally created multiple copies of our bandwidth file content
as processed rather than working from a single copy of the bytes.

Huh. That was easy. Our spec is pretty sparse on what our body includes,
mandating that each line is a series of key=value pairs and includes a
'node_id' but not much beyond that.

Minimal specificity limits what our parser can provide, but also grants sbws
flexability and makes my work dead easy. Body content is vended to users as a
mapping of relay fingerprints to measurement metadata without any additional
processing (no mandatory fields, type casting, etc).

Nothing fancy, just implementing Descriptor's create() and content() method.

Juga and teor indicated they'd like for us to standardize on the name of
'BandwidthFile'. Tracking the spec update on...

  https://trac.torproject.org/projects/tor/ticket/29137

Reading all the header metadata fields. Honestly I'm a bit fuzzy on
the definition of 'eligible', but that's fine. Taking my best guess
at describing these.

  00:08 <+atagar> juga: Minor thing that is crossing my mind while implementing
        stem's bandwidth-file parser is "how does the 'eligible' count differ
        from the number of relay measurements we list?". In other words, why
        does the header provide a 'consensus, eligible, and mininum eligible'
        count rather than a 'consensus, measured, and minimum measured' count.
        'Eligible' is defined as 'having enough measurements' but not
        necessarily having a metric included, so I'm guessing we chose this
        term because there are relays that have metrics (making it 'eligible')
        but are excluded from the results, and need to count toward the
        'minimum percent of the consensus'.

  00:09 <+atagar> Anywho, unimportant to parsing. Just a spot in the spec that
        might benefit from a little clarification.

Just stubbing out an initial class and test.

Unit test data provided by juga on the following ticket, cropped to a
reasonable size.

  https://trac.torproject.org/projects/tor/ticket/29056#comment:11

Unfortunately in this case the cure is worse than the disease. When dannenberg
went down this caused stacktraces in DocTor's emails which gives the impression
the service is having problems...

  ERROR: Unable to retrieve the consensus from dannenberg (http://193.23.244.244:80/tor/status-vote/current/consensus): <urlopen error <urlopen error [Errno 111] Connection refused>

  Original traceback:
    File "/srv/doctor.torproject.org/doctor/stem/descriptor/remote.py", line 594, in _download_descriptors
      self.content, self.reply_headers = _download_from_dirport(self.download_url, self.compression, timeout)
    File "/srv/doctor.torproject.org/doctor/stem/descriptor/remote.py", line 1043, in _download_from_dirport
      timeout = timeout,
    File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
      return opener.open(url, data, timeout)
    File "/usr/lib/python2.7/urllib2.py", line 429, in open
      response = self._open(req, data)
    File "/usr/lib/python2.7/urllib2.py", line 447, in _open
      '_open', req)
    File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
      result = func(*args)
    File "/usr/lib/python2.7/urllib2.py", line 1228, in http_open
      return self.do_open(httplib.HTTPConnection, req)
    File "/usr/lib/python2.7/urllib2.py", line 1198, in do_open
      raise URLError(err)
  URLError: <urlopen error [Errno 111] Connection refused>

I'm kinda torn on this change. On one hand it's usful information for unusual
errors, but on the other it's troubling looking bloat for mundane things like
this.

Encoding a stacktrace within an encepiton message was definitely an icky hack
so simply reverting this for now. Most likely we'll simply need to live without
underlying stacktraces until we drop python 2.x support.

This reverts commit cc43a6ca.

Oops, commit cc43a6ca both broke our unit tests and is incorrect in that it
changes the Query's 'error' attribute to a completely different type.

The error attribute is public so we cannot change it. Simply storing exception
metadata as a separate private attribute for the time being. In stem 2.x we'll
be able to drop this whole thing because without requiring python 2.x
compatibility we can re-raise exceptions while retaining their stacktrace.

When reading ORPort data that exceeded a hardcode (and arbitrary) buffer size
we cropped the content. This was caught by starlight when attempting to use
one of our demo scripts...

  https://trac.torproject.org/projects/tor/ticket/28961
  https://stem.torproject.org/tutorials/examples/download_descriptor.html

  Original traceback:
    File "/home/atagar/Desktop/stem/stem/descriptor/remote.py", line 589, in _download_descriptors
      self.content, self.reply_headers = _download_from_orport(endpoint, self.compression, self.resource)
    File "/home/atagar/Desktop/stem/stem/descriptor/remote.py", line 998, in _download_from_orport
      response = b''.join([cell.data for cell in circ.send(RelayCommand.DATA, request, stream_id = 1)])
    File "/home/atagar/Desktop/stem/stem/client/__init__.py", line 268, in send
      decrypted_cell, backward_key, backward_digest = stem.client.cell.RelayCell.decrypt(self.relay.link_protocol, encrypted_cell, self.backward_key, self.backward_digest)
    File "/home/atagar/Desktop/stem/stem/client/cell.py", line 412, in decrypt
      raise stem.ProtocolError('RELAY cells should be %i bytes, but received %i' % (link_protocol.fixed_cell_length, len(content)))
  ProtocolError: RELAY cells should be 512 bytes, but received 464

I'm unhappy with this approach, but after three days of chewing on this it's
the least bad approach I've come up with and seems to work. Patches welcome if
there's a smarter way of going about this.

Happy new year! Bumping the dates for 2019...

  % find . -type f -iname '*.py' -exec sed -i 's/-2018/-2019/g' "{}" +;
  % grep -R "# Copyright 2018," ./*