Skip to content
Snippets Groups Projects
  1. Oct 14, 2013
    • Damian Johnson's avatar
      Stem release 1.1.0 · 2ef49246
      Damian Johnson authored
      1.1.0
      2ef49246
    • Damian Johnson's avatar
      Correcting test issues with python 2.6 and 3.x · 216937c0
      Damian Johnson authored
      Generally I just work with python 2.7, so other interpretor versions just get
      love prior to releases. Most of this is issues with our tests, but @lru_cache
      introduced a lot of legitimate regressions for python 3.x.
      216937c0
    • Damian Johnson's avatar
      Python 3 compatability for mapaddress test · 6c81b101
      Damian Johnson authored
      Quite a few byte vs unicode gotchas. Also, the behavior of indexing into byte
      strings changed in python 3 (it provides ints rather than characters).
      
      ======================================================================
      ERROR: test_mapaddress
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "/home/atagar/Desktop/stem/test/data/python3/test/integ/control/controller.py", line 813, in test_mapaddress
          test.network.negotiate_socks(s, '1.2.1.2', 80)
        File "/home/atagar/Desktop/stem/test/data/python3/test/network.py", line 321, in negotiate_socks
          request = "\x04\x01" + struct.pack("!H", port) + "\x00\x00\x00\x01" + "\x00" + host + "\x00"
      TypeError: Can't convert 'bytes' object to str implicitly
      6c81b101
  2. Oct 13, 2013
    • Damian Johnson's avatar
      Fixing AttributeError in use_directory_mirrors() · 992c94bc
      Damian Johnson authored
      Correcting a couple issues, one in the tests and another in the remote
      descriptor module, causing integ failures.
      
      ======================================================================
      ERROR: test_use_directory_mirrors
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "/home/atagar/Desktop/stem/test/integ/descriptor/remote.py", line 62, in test_use_directory_mirrors
          downloader.use_directory_mirrors()
        File "/home/atagar/Desktop/stem/stem/descriptor/remote.py", line 419, in use_directory_mirrors
          if Flag.V2DIR in desc.flags:
      AttributeError: 'str' object has no attribute 'flags'
      
      ======================================================================
      ERROR: test_using_authorities
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "/home/atagar/Desktop/stem/test/integ/descriptor/remote.py", line 34, in test_using_authorities
          for authority, (address, dirport) in stem.descriptor.remote.DIRECTORY_AUTHORITIES.items():
      992c94bc
    • Damian Johnson's avatar
      Fixing python 3 testing issue with key certs · bfd5a667
      Damian Johnson authored
      Test file wasn't being opened in bytes mode, causing...
      
      ======================================================================
      ERROR: test_metrics_cert
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "/home/atagar/Desktop/stem/test/data/python3/test/integ/descriptor/networkstatus.py", line 178, in test_metrics_cert
          cert = next(stem.descriptor.parse_file(cert_file))
        File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/__init__.py", line 200, in parse_file
          for desc in file_parser(descriptor_file):
        File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/__init__.py", line 241, in _parse_metrics_file
          for desc in stem.descriptor.networkstatus._parse_file_key_certs(descriptor_file, validate = validate, **kwargs):
        File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/networkstatus.py", line 252, in _parse_file_key_certs
          yield stem.descriptor.networkstatus.KeyCertificate(bytes.join(b"", keycert_content), validate = validate)
      TypeError: sequence item 0: expected bytes, str found
      bfd5a667
    • Damian Johnson's avatar
      Fixing python 3 compatability for remote descriptor unit tests · a303b137
      Damian Johnson authored
      ======================================================================
      ERROR: test_query_with_timeout
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "/opt/python3.3/lib/python3.3/unittest/mock.py", line 1079, in patched
          arg = patching.__enter__()
        File "/opt/python3.3/lib/python3.3/unittest/mock.py", line 1132, in __enter__
          self.target = self.getter()
        File "/opt/python3.3/lib/python3.3/unittest/mock.py", line 1294, in <lambda>
          getter = lambda: _importer(target)
        File "/opt/python3.3/lib/python3.3/unittest/mock.py", line 988, in _importer
          thing = __import__(import_path)
      ImportError: No module named 'urllib2'
      a303b137
    • Damian Johnson's avatar
      Fixing python 3 issue with orphaned annotation exceptions · ab86475d
      Damian Johnson authored
      ======================================================================
      ERROR: test_annotations
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "/opt/python3.3/lib/python3.3/unittest/mock.py", line 1087, in patched
          return func(*args, **keywargs)
        File "/home/atagar/Desktop/stem/test/data/python3/test/unit/descriptor/server_descriptor.py", line 224, in test_annotations
          self.assertRaises(ValueError, list, desc_iter)
        File "/opt/python3.3/lib/python3.3/unittest/case.py", line 570, in assertRaises
          return context.handle('assertRaises', callableObj, args, kwargs)
        File "/opt/python3.3/lib/python3.3/unittest/case.py", line 135, in handle
          callable_obj(*args, **kwargs)
        File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/server_descriptor.py", line 157, in _parse_file
          raise ValueError('Content conform to being a server descriptor:\n%s' % '\n'.join(annotations))
      TypeError: sequence item 0: expected str instance, bytes found
      
      ----------------------------------------------------------------------
      Ran 31 tests in 0.028s
      
      FAILED (errors=1)
      ab86475d
    • Damian Johnson's avatar
      Hash functions for descriptor classes · 1ed6e2b7
      Damian Johnson authored
      Ok, finally found out what this python 2 vs 3 hashing oddity is. If you define
      an __eq__() method in python3 then it doesn't inherit its parent's __hash__().
      This makes some sense, though it's unintuitive as hell. Oh well...
      
      http://stackoverflow.com/a/1608882/1067192
      1ed6e2b7
    • Damian Johnson's avatar
      Fixing python 3 compatablity for tordnsel test · eaa89a67
      Damian Johnson authored
      ======================================================================
      ERROR: test_parse_file
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "/home/atagar/Desktop/stem/test/data/python3/test/unit/descriptor/tordnsel.py", line 55, in test_parse_file
          descriptors = list(_parse_file(io.BytesIO(TEST_DESC)))
      TypeError: 'str' does not support the buffer interface
      
      ----------------------------------------------------------------------
      Ran 1 test in 0.001s
      eaa89a67
    • Damian Johnson's avatar
      Fixing a couple python 3 network status document TypeErrors · e1bf7cba
      Damian Johnson authored
      ======================================================================
      ERROR: test_duplicate_fields
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "/home/atagar/Desktop/stem/test/data/python3/test/unit/descriptor/networkstatus/document_v3.py", line 247, in test_duplicate_fields
          if not is_consensus and lines[index].startswith('dir-source'):
      TypeError: startswith first arg must be bytes or a tuple of bytes, not str
      
      ======================================================================
      ERROR: test_misordered_fields
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "/home/atagar/Desktop/stem/test/data/python3/test/unit/descriptor/networkstatus/document_v3.py", line 225, in test_misordered_fields
          elif not is_consensus and lines[index].startswith('dir-source'):
      TypeError: startswith first arg must be bytes or a tuple of bytes, not str
      
      ----------------------------------------------------------------------
      Ran 35 tests in 0.234s
      e1bf7cba
    • Damian Johnson's avatar
      Caching hash values · efe5fbdd
      Damian Johnson authored
      Well, even if we can't use @lru_cache we can still do it the old fashioned way. :P
      efe5fbdd
    • Damian Johnson's avatar
      Dropping @lru_cache for Version hashing · d32d4ab4
      Damian Johnson authored
      The @lru_cache calls hash() so if ever called this would infinitely recurse.
      d32d4ab4
    • Damian Johnson's avatar
      Fixing python 3 issue concerning exit policy compression · 208c03cd
      Damian Johnson authored
      Oops, forgot about the byte vs unicode differences when doing compression for
      the ExitPolicy.
      208c03cd
    • Damian Johnson's avatar
      Adding hash functions to exit policy classes · 4ef5ee1e
      Damian Johnson authored
      Huh, didn't expect that. To work the @lru_cache requires that all arguments
      (including the class in the ase of methods) are hashable. This makes perfectly
      sense, after all the cache is a 'argument => cached value' dictionary.
      
      Under python 2.x all seemed to be well, but under python 3.x the @lru_cache
      complained that our classes weren't hashable (which... well, they aren't). Why,
      then, did it work under python 2.x? Turns out there's a subtle difference where
      our object parent provides a __hash__ method for our id, but in python 3.x it's
      a little different (object still does, but for reasons I haven't been able to
      figure out it doesn't for our classes).
      
      The ExitPolicyRule can certainly be hashable. The ExitPolicy, however, is a lot
      trickier due to its lazy loading of rules. For now just opting for the same
      behavior as python 2.x and using the address.
      4ef5ee1e
    • Damian Johnson's avatar
      Python 3 lru_cache isn't a collections annotation · 677ee157
      Damian Johnson authored
      Oops, it's in functools, not collections.
      677ee157
    • Damian Johnson's avatar
      Using @lru_cache for exit policy's str() · 1dfa353e
      Damian Johnson authored
      Oops, missed one of our instances of manual caching.
      1dfa353e
    • Damian Johnson's avatar
      Using python3's unittest.mock module when available · e89c829f
      Damian Johnson authored
      Guess it's been a while since I last tried to test under python3. From python
      3.3 on up the Mock library is built into the unittest module. Accounting for
      this so we can test under python3 again.
      e89c829f
    • Damian Johnson's avatar
      Fixing python3 issue related to unicode exit policies · d09bd8b3
      Damian Johnson authored
      Our recent exit policy caching caused a regression under python3...
      
      Traceback (most recent call last):
        File "./test/data/python3/run_tests.py", line 25, in <module>
          import test.runner
        File "/home/atagar/Desktop/stem/test/data/python3/test/runner.py", line 51, in <module>
          import stem.connection
        File "/home/atagar/Desktop/stem/test/data/python3/stem/connection.py", line 109, in <module>
          import stem.control
        File "/home/atagar/Desktop/stem/test/data/python3/stem/control.py", line 145, in <module>
          import stem.descriptor.microdescriptor
        File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/__init__.py", line 546, in <module>
          import stem.descriptor.server_descriptor
        File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/server_descriptor.py", line 91, in <module>
          REJECT_ALL_POLICY = stem.exit_policy.ExitPolicy("reject *:*")
        File "/home/atagar/Desktop/stem/test/data/python3/stem/exit_policy.py", line 164, in __init__
          self._input_rules = zlib.compress(','.join(rules))
      TypeError: 'str' does not support the buffer interface
      d09bd8b3
    • Damian Johnson's avatar
      Fixing python 2.6 compatability issue with lru_cache · c10eef25
      Damian Johnson authored
      The recepie is using set syntax only available in python 2.7 and above...
      
      Traceback (most recent call last):
        File "./run_tests.py", line 18, in <module>
          import stem.prereq
        File "/home/atagar/Desktop/stem/stem/__init__.py", line 420, in <module>
          import stem.util.enum
        File "/home/atagar/Desktop/stem/stem/util/enum.py", line 43, in <module>
          import stem.util.str_tools
        File "/home/atagar/Desktop/stem/stem/util/str_tools.py", line 21, in <module>
          import stem.prereq
        File "/home/atagar/Desktop/stem/stem/prereq.py", line 29, in <module>
          from stem.util.lru_cache import lru_cache
        File "/home/atagar/Desktop/stem/stem/util/lru_cache.py", line 37
          fasttypes = {int, str, frozenset, type(None)},
                          ^
      SyntaxError: invalid syntax
      c10eef25
  3. Oct 12, 2013
  4. Oct 10, 2013
  5. Oct 09, 2013
  6. Oct 08, 2013
    • Damian Johnson's avatar
      Chaching version comparison and hash function · 281db911
      Damian Johnson authored
      Versions are intended to be read-only objects, so we might as well add caching
      for these methods.
      281db911
    • Damian Johnson's avatar
      Reducing memory usage of descriptor content · ba9799e2
      Damian Johnson authored
      Handful of improvements to reduce the memory usage of stem's descriptor content
      (with a special focus on server descriptors). This drops memory usage by
      roughly 20% without having a sizable impact on runtime.
      
      Script used for testing:
      
        import resource
        import time
      
        from stem.descriptor import reader
      
        print "memory (initial): %s" % resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
      
        start_time = time.time()
      
        with reader.DescriptorReader(['/home/atagar/.tor/cached-descriptors']) as descriptor_reader:
          server_descriptors = list(descriptor_reader)
      
        print "memory (after read): %s" % resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
        print "read time: %0.2f seconds" % (time.time() - start_time)
      
      Before:
      
        % python scratch.py
        memory (initial): 7468
        memory (after read): 50572
        read time: 6.40 seconds
      
      After:
      
        % python scratch.py
        memory (initial): 7532
        memory (after read): 40816
        read time: 6.31 seconds
      ba9799e2
    • Damian Johnson's avatar
      Compressing unparsed exit policy content · 1920baff
      Damian Johnson authored
      Exit policies constitute a rather substantial chunch of server descriptors. We
      already optimize our runtime by lazily parsing policies on demand. On
      reflection though, we can improve our memory usage quite a bit too by
      compressing the unparsed content. This drops our memory usage by roughly 6%
      without impacting runtime.
      1920baff
    • Damian Johnson's avatar
      Caching parsed tor versions · 2ce003ef
      Damian Johnson authored
      Reusing parsed tor versions saves us an impressive 3.7% memory usage without
      any noticable impact on parsing times.
      2ce003ef
    • Damian Johnson's avatar
      Referencing a single reject-all policy · dbee38d1
      Damian Johnson authored
      Oddly, the only exit policy I can find with greater than five occurances is the
      reject-all policy. Maybe a scripting bug. Reguardless, referencing a single
      reject-all instance saves us a little memory (1.3 MB with the present
      consensus, which is 1.6%).
      dbee38d1
    • Damian Johnson's avatar
      Storing a single IPv6 default exit policy · a0ce8664
      Damian Johnson authored
      Each server descriptor constructed its own default value for the IPv6 exit
      policy, which was foolish. ExitPolicy instances are read-only instances, so all
      descriptors might as well use the same default. This saves roughly 2.3 MB of
      memory when downloading all of the current server descriptors (or roughly 3%).
      a0ce8664
    • Damian Johnson's avatar
      Using @lru_cache for exit policies · 19cca3a0
      Damian Johnson authored
      The ExitPolicy and ExitPolicyRule classes are a perfect fit for @lru_cache.
      They're read-only classes that already do a fair bit of caching. The annotation
      lets us avoid doing this ourselves.
      19cca3a0
  7. Oct 07, 2013
  8. Oct 02, 2013
    • Damian Johnson's avatar
      Tests can fail with TypeError when unable to start tor process · e60a0339
      Damian Johnson authored
      Traceback (most recent call last):
        File "./run_tests.py", line 404, in <module>
          main()
        File "./run_tests.py", line 245, in main
          println(exc, ERROR)
        File "/home/atagar/Desktop/stem/test/output.py", line 59, in println
          msg = term.format(msg, *attr)
        File "/home/atagar/Desktop/stem/stem/util/term.py", line 96, in format
          return (CSI % ";".join(encodings)) + msg + RESET
      TypeError: cannot concatenate 'str' and 'UnsupportedOperation' objects
      e60a0339
    • Damian Johnson's avatar
      Adding a hash method to our Version class · bcd849f0
      Damian Johnson authored
      Our Version class lacked a __hash__() method, causing it to behave
      unintuitively in dictionaries and sets...
      
        >>> from stem.version import Version
        >>> foo = set([Version('0.2.4.9-alpha')])
        >>> bar = set([Version('0.2.4.9-alpha')])
        >>> foo.difference(bar)
        set([<stem.version.Version object at 0xb71fae0c>])
      
      This is because python opts for the object's identity when a hash method does
      not exist.
      bcd849f0
  9. Sep 30, 2013
    • Damian Johnson's avatar
      Halo for windrose icon · a5f9e9cd
      Damian Johnson authored
      Adding a blue halo for the center of the windrose. This doesn't address the
      main thing I dislike about the icon, which is the pixelation of the black
      lines, but still it makes it look a bit nicer. I've tried several things to
      address the black line pixilation (blur, thickening, etc) but this is the best
      improvement I've yet found.
      a5f9e9cd
    • Damian Johnson's avatar
      Adding utilities tutorial to the menu · 4866bd33
      Damian Johnson authored
      Oops, forgot one of the spots we need to add new pages...
      4866bd33
    • Damian Johnson's avatar
      Adding connection resolution tutorial · b0be16b7
      Damian Johnson authored
      Introducing a new tutorial section called 'East of the Sun & West of the Moon'
      (... I love that fairytale) that introduces users to our utility modules.
      Presently this just has connection resolution, a spiffy new feature I added
      last weekend.
      b0be16b7
  10. Sep 28, 2013
  11. Sep 23, 2013
Loading