Skip to content
Snippets Groups Projects
  1. Nov 05, 2017
  2. Nov 04, 2017
    • Damian Johnson's avatar
      Stacktrace when calling get_circuits() · 778689d3
      Damian Johnson authored
      Running the ONLINE integ target uncovered another bug...
      
        ======================================================================
        ERROR: test_get_circuits
        ----------------------------------------------------------------------
        Traceback (most recent call last):
          File "/home/atagar/Desktop/stem/test/require.py", line 58, in wrapped
            return func(self, *args, **kwargs)
          File "/home/atagar/Desktop/stem/test/require.py", line 58, in wrapped
            return func(self, *args, **kwargs)
          File "/home/atagar/Desktop/stem/test/require.py", line 58, in wrapped
            return func(self, *args, **kwargs)
          File "/home/atagar/Desktop/stem/test/integ/control/controller.py", line 1339, in test_get_circuits
            circuits = controller.get_circuits()
          File "/home/atagar/Desktop/stem/stem/control.py", line 482, in wrapped
            return func(self, *args, **kwargs)
          File "/home/atagar/Desktop/stem/stem/control.py", line 3336, in get_circuits
            circ_message = stem.socket.recv_message(io.BytesIO(stem.util.str_tools._to_unicode('650 CIRC %s\r\n' % circ)))
        TypeError: 'unicode' does not have the buffer interface
      778689d3
    • Damian Johnson's avatar
      Update cached manual information · 48c7a4f4
      Damian Johnson authored
      Ran with a small hack to strip brackets due to malformed manual content...
      
        https://trac.torproject.org/projects/tor/ticket/24147
      48c7a4f4
    • Damian Johnson's avatar
      Bytes/unicode normalization for tor_tools didn't work for python3 · a31af967
      Damian Johnson authored
      We added unit tests to check that tor_tools functions like
      is_valid_fingerprint() worked for both bytes and unicode, but these tests
      actually only passes for python 2.x. With python3 they only accepted unicode.
      Normalizing the inputs.
      a31af967
    • Damian Johnson's avatar
      Stacktrace when calling is_running with python3 · 4052c498
      Damian Johnson authored
      Recent change broke is_running for python3...
      
        ======================================================================
        ERROR: test_is_running_by_name
        ----------------------------------------------------------------------
        Traceback (most recent call last):
          File "/home/atagar/Desktop/stem/test/require.py", line 58, in wrapped
            return func(self, *args, **kwargs)
          File "/home/atagar/Desktop/stem/test/integ/util/system.py", line 144, in test_is_running_by_name
            self.assertTrue(stem.util.system.is_running(tor_cmd) or stem.util.system.is_running('tor.real'))
          File "/home/atagar/Desktop/stem/stem/util/system.py", line 449, in is_running
            if isinstance(command, (bytes, unicode)):
        NameError: global name 'unicode' is not defined
      4052c498
  3. Nov 03, 2017
  4. Nov 02, 2017
    • Damian Johnson's avatar
      Don't check 'UseMicrodescriptors 1' when calling get_network_statuses() · 136537c2
      Damian Johnson authored
      So much confusion around this...
      
        https://trac.torproject.org/projects/tor/ticket/24110
      
      It turns out in practice these methods essentially never provided
      microdescriptor flavored router status entries because 'UseMicrodescriptors'
      defaults to 'auto' (and it's rare to set it to '1' since that's the default
      anyway). However, if that was done we stacktraced...
      
        >>> desc = next(controller.get_network_statuses())
        Traceback (most recent call last):
          File "<console>", line 1, in <module>
          File "/home/atagar/Desktop/stem/stem/control.py", line 494, in wrapped
            for val in func(self, *args, **kwargs):
          File "/home/atagar/Desktop/stem/stem/control.py", line 1993, in get_network_statuses
            for desc in desc_iterator:
          File "/home/atagar/Desktop/stem/stem/descriptor/router_status_entry.py", line 104, in _parse_file
            yield entry_class(desc_content, validate, *extra_args)
          File "/home/atagar/Desktop/stem/stem/descriptor/router_status_entry.py", line 446, in __init__
            raise ValueError("%s must have a '%s' line:\n%s" % (self._name(True), keyword, str(self)))
        ValueError: Router status entries (micro v3) must have a 'm' line:
        r networkofthesmoker AAaX5kASI9/xzl9Qp7aFdPOSnrQ aoqBoLrUWKssJAKnO8r6GpZs5p8 2017-11-02 13:51:59 128.72.180.74 443 0
        s Fast Guard HSDir Running Stable V2Dir Valid
        w Bandwidth=4150
      
      From what I can tell nowadays UseMicrodescriptors doesn't change the 'GETINFO
      ns/all' response. As such reverting...
      
        https://gitweb.torproject.org/stem.git/commit/?id=d70fa9cd
      136537c2
  5. Oct 31, 2017
  6. Oct 30, 2017
    • Damian Johnson's avatar
      Controller's get_ports method frequently failed to provide local ports · 8772dbe2
      Damian Johnson authored
      Oops, sometimes tor provides 127.0.0.1 for localhost and other times 0.0.0.0...
      
        >>> GETINFO net/listeners/or
        250-net/listeners/or="0.0.0.0:1443"
        250 OK
      
        >>> GETINFO net/listeners/control
        250-net/listeners/control="127.0.0.1:9051"
        250 OK
      8772dbe2
    • Damian Johnson's avatar
      Limit number of events buffered by interpreter · 66243c41
      Damian Johnson authored
      Sweet jesus that was a pita to track down. Nyx has a memory leak, holding onto
      events tor emits indefinitely. Spent all day headscratching and finally found
      it. Our control interpreter intercepts events and enqueues them without bound.
      Finally caught thanks to the gc module...
      
        to_ignore = [ locals(), globals(), gc.garbage ]
        log.warn("refs: %s" % [inspect.getframeinfo(r) for r in gc.get_referrers(event_message) if r not in to_ignore])
      
      ... which gave...
      
       06:44:50 [NYX_WARNING] refs: [Traceback(filename='/home/ubuntu/nyx/stem/interpreter/commands.py', lineno=135, function='handle_event_wrapper',
         code_context=['      handle_event_real(event_message)\n'], index=0), Traceback(filename='/home/ubuntu/nyx/stem/control.py', lineno=3812,
         function='_handle_event', code_context=['    log.warn("refs: %s" % [inspect.getframeinfo(r) for r in gc.get_referrers(event_message) if r not in
         to_ignore])\n'], index=0), Traceback(filename='/home/ubuntu/nyx/stem/control.py', lineno=975, function='_event_loop', code_context=['
         self._handle_event(event_message)\n'], index=0)] [998 duplicates hidden]
      66243c41
    • Damian Johnson's avatar
      Cache string representation of messages · d6956762
      Damian Johnson authored
      Controller messages frequently reference their string representation, and it's
      silly to reconstruct it each time.
      d6956762
  7. Oct 29, 2017
  8. Oct 27, 2017
    • Damian Johnson's avatar
      Fix issue causing tests to be unable to run · 10fb8a63
      Damian Johnson authored
      Oops, python 2.6 fixes added a regression breaking everything *except* python
      2.6. Oops. :P
      
        Traceback (most recent call last):
          File "./run_tests.py", line 20, in <module>
            RUN_ASYNC_TESTS = true
        NameError: name 'true' is not defined
      10fb8a63
    • Damian Johnson's avatar
      Loosen regex metching for python 2.6 assertions · e69d22fd
      Damian Johnson authored
      Python 2.6 lacks assertRaisesRegexp so we made our own handler for it. However,
      the assertions it did were more strict, causing failures for tests that pass
      with other python versions...
      
        ======================================================================
        FAIL: test_get_info_address_caching
        ----------------------------------------------------------------------
        Traceback (most recent call last):
          File "/usr/local/lib/python2.6/dist-packages/mock.py", line 1201, in patched
            return func(*args, **keywargs)
          File "/home/atagar/Desktop/stem/test/unit/control/controller.py", line 61, in test_get_info_address_caching
            self.assertRaisesRegexp(stem.ProtocolError, 'Address unknown', self.controller.get_info, 'address')
          File "/home/atagar/Desktop/stem/stem/util/test_tools.py", line 270, in assertRaisesRegexp
            self.assertTrue(re.match(exc_msg, str(exc)))
        AssertionError
      e69d22fd
    • Damian Johnson's avatar
      Skip running async tests under python 2.6 · 70b49df2
      Damian Johnson authored
      Turns out importlib was added in python 2.7...
      
        Traceback (most recent call last):
          File "./run_tests.py", line 9, in <module>
            import importlib
        ImportError: No module named importlib
      
      This means we're losing quite a bit of test coverage under python 2.6 but...
      screw it. Guido rolled his eyes when I mentioned we even offered 2.6 support.
      It's been dead a long time.
      
      We'll continue to support python 2.6 until stem 2.0, but not gonna bend over
      backwards for it. This change will let us invoke our tests again.
      70b49df2
    • Damian Johnson's avatar
      Fix python3 unit test error · a021aae5
      Damian Johnson authored
      Oops, python3 regression slipped in with prior work...
      
        ======================================================================
        ERROR: test_connections
        ----------------------------------------------------------------------
        Traceback (most recent call last):
          File "/usr/local/lib/python3.2/dist-packages/mock/mock.py", line 1305, in patched
            return func(*args, **keywargs)
          File "/home/atagar/Desktop/stem/test/unit/util/proc.py", line 215, in test_connections
            tcp = b'%s\n 0: 11111111:1111 22222222:2222 01 44444444:44444444 55:55555555 66666666 1111        8 99999999' % TITLE_LINE
        TypeError: unsupported operand type(s) for %: 'bytes' and 'bytes'
      a021aae5
  9. Oct 25, 2017
  10. Oct 22, 2017
    • Damian Johnson's avatar
      Avoid accidently clearing caches · 73f9c234
      Damian Johnson authored
      Oops, a recent addition in commit 82a3cc05 can cause us to clear the cache when
      called without any parameters. What we actually meant there was to allow
      callers to explicitly clear, so checking for None instead.
      73f9c234
    • Damian Johnson's avatar
      Cache 'GETINFO fingerprint' when not a relay · b730a0b5
      Damian Johnson authored
      Probably the most requested GETINFO parameter is our fingerprint. We cache
      fingerprint values, but not exceptions causing any request when not a relay to
      go to tor.
      
      This is unnecessary. We can detect if we become a relay so caching the 'not a
      relay' response to avoid having callers (like Nyx) accidently hammer tor.
      b730a0b5
  11. Oct 20, 2017
  12. Oct 17, 2017
    • Damian Johnson's avatar
      Minor simplification for cache fetches · e134dc59
      Damian Johnson authored
      Good spot for map to simplify this a tad.
      e134dc59
    • Damian Johnson's avatar
      Python3 errors due to list modificaion during iteration · 49c084d8
      Damian Johnson authored
      When iterating over keys python 2.x makes a shallow clone of lists whereas
      python 3.x doesn't. Good for performance, but means we need to be careful about
      modifying a list during iteration. Caught thanks to our integ tests...
      
        ======================================================================
        ERROR: test_without_ephemeral_hidden_services
        ----------------------------------------------------------------------
        Traceback (most recent call last):
          File "/home/atagar/Desktop/stem/test/require.py", line 58, in wrapped
            return func(self, *args, **kwargs)
          File "/home/atagar/Desktop/stem/test/require.py", line 58, in wrapped
            return func(self, *args, **kwargs)
          File "/home/atagar/Desktop/stem/test/integ/control/controller.py", line 560, in test_without_ephemeral_hidden_services
            self.assertEqual([], controller.list_ephemeral_hidden_services())
          File "/home/atagar/Desktop/stem/stem/control.py", line 475, in wrapped
            return func(self, *args, **kwargs)
          File "/home/atagar/Desktop/stem/stem/control.py", line 2774, in list_ephemeral_hidden_services
            result += self.get_info('onions/current').split('\n')
          File "/home/atagar/Desktop/stem/stem/control.py", line 475, in wrapped
            return func(self, *args, **kwargs)
          File "/home/atagar/Desktop/stem/stem/control.py", line 1191, in get_info
            self._set_cache(to_cache, 'getinfo')
          File "/home/atagar/Desktop/stem/stem/control.py", line 3112, in _set_cache
            for cache_key in self._request_cache.keys():
        RuntimeError: dictionary changed size during iteration
      49c084d8
    • Damian Johnson's avatar
      Regression for python3 with stem.util.proc · 820f3e9b
      Damian Johnson authored
      Fixing bytes normalization issues that broke our stem.util.proc unit tests for
      python3.
      820f3e9b
  13. Oct 15, 2017
    • Damian Johnson's avatar
      Skip proc connections with a blank address or port · 5942687b
      Damian Johnson authored
      Can't think of any reason we'd want connections with an undefined address or
      port, so skipping those entries. This doesn't come up in practice but when I
      hack my proc contents a bit I get...
      
        File "/usr/lib/python2.7/stem/util/lru_cache.py", line 135, in wrapper
          result = user_function(*args, **kwds)
        File "/usr/lib/python2.7/stem/exit_policy.py", line 289, in can_exit_to
          if rule.is_match(address, port, strict):
        File "/usr/lib/python2.7/stem/exit_policy.py", line 759, in is_match
          raise ValueError("'%s' isn't a valid port" % port)
        ValueError: '0' isn't a valid port
      5942687b
  14. Oct 11, 2017
    • Damian Johnson's avatar
      Integ failure for HiddenServiceNonAnonymousMode with older tor versions · a3aceae0
      Damian Johnson authored
      Tor's HiddenServiceNonAnonymousMode option is rather new. Nice that we have an
      integ test for it, but we should account for versions without it too...
      
        ======================================================================
        FAIL: test_unanonymous_hidden_service_config_must_match
        ----------------------------------------------------------------------
        Traceback (most recent call last):
          File "/home/atagar/Desktop/stem/stem/util/test_tools.py", line 150, in <lambda>
            self.method = lambda test: self.result(test)  # method that can be mixed into TestCases
          File "/home/atagar/Desktop/stem/stem/util/test_tools.py", line 212, in result
            test.fail(self._result.msg)
        AssertionError: Expected 'Process terminated: HiddenServiceNonAnonymousMode does not provide any server anonymity. It must be used with HiddenServiceSingleHopMode set to 1.' but was 'Process terminated: Unknown option 'HiddenServiceNonAnonymousMode'.  Failing.'
      a3aceae0
  15. Oct 10, 2017
  16. Oct 08, 2017
  17. Oct 04, 2017
  18. Sep 26, 2017
    • Damian Johnson's avatar
      Parse proc connections by index · 8da78a9f
      Damian Johnson authored
      Proc's /proc/net/tcp commonly has thousands of lines on even moderately busy
      relays. For me it takes about half a second to process, pegging the cpu during
      that time.
      
      After lots of experimenting finally found something to greatly speed this up.
      Parsing the proc content via index rather than spit makes a large difference.
      More delicate so this might require some adjustment to accomidate how other
      platforms format their proc.
      
      This is about a 30% improvement, but along with the earlier changes this in
      total accounts for a 5x speedup in proc conneciton resolution, going from
      0.38s to 0.07s for my relay's proc contents.
      8da78a9f
  19. Sep 24, 2017
Loading