- Oct 14, 2013
-
-
Damian Johnson authored
-
Damian Johnson authored
Generally I just work with python 2.7, so other interpretor versions just get love prior to releases. Most of this is issues with our tests, but @lru_cache introduced a lot of legitimate regressions for python 3.x.
-
Damian Johnson authored
Quite a few byte vs unicode gotchas. Also, the behavior of indexing into byte strings changed in python 3 (it provides ints rather than characters). ====================================================================== ERROR: test_mapaddress ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/atagar/Desktop/stem/test/data/python3/test/integ/control/controller.py", line 813, in test_mapaddress test.network.negotiate_socks(s, '1.2.1.2', 80) File "/home/atagar/Desktop/stem/test/data/python3/test/network.py", line 321, in negotiate_socks request = "\x04\x01" + struct.pack("!H", port) + "\x00\x00\x00\x01" + "\x00" + host + "\x00" TypeError: Can't convert 'bytes' object to str implicitly
-
- Oct 13, 2013
-
-
Damian Johnson authored
Correcting a couple issues, one in the tests and another in the remote descriptor module, causing integ failures. ====================================================================== ERROR: test_use_directory_mirrors ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/atagar/Desktop/stem/test/integ/descriptor/remote.py", line 62, in test_use_directory_mirrors downloader.use_directory_mirrors() File "/home/atagar/Desktop/stem/stem/descriptor/remote.py", line 419, in use_directory_mirrors if Flag.V2DIR in desc.flags: AttributeError: 'str' object has no attribute 'flags' ====================================================================== ERROR: test_using_authorities ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/atagar/Desktop/stem/test/integ/descriptor/remote.py", line 34, in test_using_authorities for authority, (address, dirport) in stem.descriptor.remote.DIRECTORY_AUTHORITIES.items():
-
Damian Johnson authored
Test file wasn't being opened in bytes mode, causing... ====================================================================== ERROR: test_metrics_cert ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/atagar/Desktop/stem/test/data/python3/test/integ/descriptor/networkstatus.py", line 178, in test_metrics_cert cert = next(stem.descriptor.parse_file(cert_file)) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/__init__.py", line 200, in parse_file for desc in file_parser(descriptor_file): File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/__init__.py", line 241, in _parse_metrics_file for desc in stem.descriptor.networkstatus._parse_file_key_certs(descriptor_file, validate = validate, **kwargs): File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/networkstatus.py", line 252, in _parse_file_key_certs yield stem.descriptor.networkstatus.KeyCertificate(bytes.join(b"", keycert_content), validate = validate) TypeError: sequence item 0: expected bytes, str found
-
Damian Johnson authored
====================================================================== ERROR: test_query_with_timeout ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/python3.3/lib/python3.3/unittest/mock.py", line 1079, in patched arg = patching.__enter__() File "/opt/python3.3/lib/python3.3/unittest/mock.py", line 1132, in __enter__ self.target = self.getter() File "/opt/python3.3/lib/python3.3/unittest/mock.py", line 1294, in <lambda> getter = lambda: _importer(target) File "/opt/python3.3/lib/python3.3/unittest/mock.py", line 988, in _importer thing = __import__(import_path) ImportError: No module named 'urllib2'
-
Damian Johnson authored
====================================================================== ERROR: test_annotations ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/python3.3/lib/python3.3/unittest/mock.py", line 1087, in patched return func(*args, **keywargs) File "/home/atagar/Desktop/stem/test/data/python3/test/unit/descriptor/server_descriptor.py", line 224, in test_annotations self.assertRaises(ValueError, list, desc_iter) File "/opt/python3.3/lib/python3.3/unittest/case.py", line 570, in assertRaises return context.handle('assertRaises', callableObj, args, kwargs) File "/opt/python3.3/lib/python3.3/unittest/case.py", line 135, in handle callable_obj(*args, **kwargs) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/server_descriptor.py", line 157, in _parse_file raise ValueError('Content conform to being a server descriptor:\n%s' % '\n'.join(annotations)) TypeError: sequence item 0: expected str instance, bytes found ---------------------------------------------------------------------- Ran 31 tests in 0.028s FAILED (errors=1)
-
Damian Johnson authored
Ok, finally found out what this python 2 vs 3 hashing oddity is. If you define an __eq__() method in python3 then it doesn't inherit its parent's __hash__(). This makes some sense, though it's unintuitive as hell. Oh well... http://stackoverflow.com/a/1608882/1067192
-
Damian Johnson authored
====================================================================== ERROR: test_parse_file ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/atagar/Desktop/stem/test/data/python3/test/unit/descriptor/tordnsel.py", line 55, in test_parse_file descriptors = list(_parse_file(io.BytesIO(TEST_DESC))) TypeError: 'str' does not support the buffer interface ---------------------------------------------------------------------- Ran 1 test in 0.001s
-
Damian Johnson authored
====================================================================== ERROR: test_duplicate_fields ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/atagar/Desktop/stem/test/data/python3/test/unit/descriptor/networkstatus/document_v3.py", line 247, in test_duplicate_fields if not is_consensus and lines[index].startswith('dir-source'): TypeError: startswith first arg must be bytes or a tuple of bytes, not str ====================================================================== ERROR: test_misordered_fields ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/atagar/Desktop/stem/test/data/python3/test/unit/descriptor/networkstatus/document_v3.py", line 225, in test_misordered_fields elif not is_consensus and lines[index].startswith('dir-source'): TypeError: startswith first arg must be bytes or a tuple of bytes, not str ---------------------------------------------------------------------- Ran 35 tests in 0.234s
-
Damian Johnson authored
Well, even if we can't use @lru_cache we can still do it the old fashioned way. :P
-
Damian Johnson authored
The @lru_cache calls hash() so if ever called this would infinitely recurse.
-
Damian Johnson authored
Oops, forgot about the byte vs unicode differences when doing compression for the ExitPolicy.
-
Damian Johnson authored
Huh, didn't expect that. To work the @lru_cache requires that all arguments (including the class in the ase of methods) are hashable. This makes perfectly sense, after all the cache is a 'argument => cached value' dictionary. Under python 2.x all seemed to be well, but under python 3.x the @lru_cache complained that our classes weren't hashable (which... well, they aren't). Why, then, did it work under python 2.x? Turns out there's a subtle difference where our object parent provides a __hash__ method for our id, but in python 3.x it's a little different (object still does, but for reasons I haven't been able to figure out it doesn't for our classes). The ExitPolicyRule can certainly be hashable. The ExitPolicy, however, is a lot trickier due to its lazy loading of rules. For now just opting for the same behavior as python 2.x and using the address.
-
Damian Johnson authored
Oops, it's in functools, not collections.
-
Damian Johnson authored
Oops, missed one of our instances of manual caching.
-
Damian Johnson authored
Guess it's been a while since I last tried to test under python3. From python 3.3 on up the Mock library is built into the unittest module. Accounting for this so we can test under python3 again.
-
Damian Johnson authored
Our recent exit policy caching caused a regression under python3... Traceback (most recent call last): File "./test/data/python3/run_tests.py", line 25, in <module> import test.runner File "/home/atagar/Desktop/stem/test/data/python3/test/runner.py", line 51, in <module> import stem.connection File "/home/atagar/Desktop/stem/test/data/python3/stem/connection.py", line 109, in <module> import stem.control File "/home/atagar/Desktop/stem/test/data/python3/stem/control.py", line 145, in <module> import stem.descriptor.microdescriptor File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/__init__.py", line 546, in <module> import stem.descriptor.server_descriptor File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/server_descriptor.py", line 91, in <module> REJECT_ALL_POLICY = stem.exit_policy.ExitPolicy("reject *:*") File "/home/atagar/Desktop/stem/test/data/python3/stem/exit_policy.py", line 164, in __init__ self._input_rules = zlib.compress(','.join(rules)) TypeError: 'str' does not support the buffer interface
-
Damian Johnson authored
The recepie is using set syntax only available in python 2.7 and above... Traceback (most recent call last): File "./run_tests.py", line 18, in <module> import stem.prereq File "/home/atagar/Desktop/stem/stem/__init__.py", line 420, in <module> import stem.util.enum File "/home/atagar/Desktop/stem/stem/util/enum.py", line 43, in <module> import stem.util.str_tools File "/home/atagar/Desktop/stem/stem/util/str_tools.py", line 21, in <module> import stem.prereq File "/home/atagar/Desktop/stem/stem/prereq.py", line 29, in <module> from stem.util.lru_cache import lru_cache File "/home/atagar/Desktop/stem/stem/util/lru_cache.py", line 37 fasttypes = {int, str, frozenset, type(None)}, ^ SyntaxError: invalid syntax
-
- Oct 12, 2013
-
-
Damian Johnson authored
Replacing instances where we do... def get_stuff(): if self._stuff is None: self._stuff = ... calculated stuff... return self._stuff ... with a @lru_cache().
-
- Oct 10, 2013
-
-
Damian Johnson authored
Wow, shame on me for not even running unit tests. This causes version comparisons to infinitely recurse - hot.
-
- Oct 09, 2013
-
-
Damian Johnson authored
-
- Oct 08, 2013
-
-
Damian Johnson authored
Versions are intended to be read-only objects, so we might as well add caching for these methods.
-
Damian Johnson authored
Handful of improvements to reduce the memory usage of stem's descriptor content (with a special focus on server descriptors). This drops memory usage by roughly 20% without having a sizable impact on runtime. Script used for testing: import resource import time from stem.descriptor import reader print "memory (initial): %s" % resource.getrusage(resource.RUSAGE_SELF).ru_maxrss start_time = time.time() with reader.DescriptorReader(['/home/atagar/.tor/cached-descriptors']) as descriptor_reader: server_descriptors = list(descriptor_reader) print "memory (after read): %s" % resource.getrusage(resource.RUSAGE_SELF).ru_maxrss print "read time: %0.2f seconds" % (time.time() - start_time) Before: % python scratch.py memory (initial): 7468 memory (after read): 50572 read time: 6.40 seconds After: % python scratch.py memory (initial): 7532 memory (after read): 40816 read time: 6.31 seconds
-
Damian Johnson authored
Exit policies constitute a rather substantial chunch of server descriptors. We already optimize our runtime by lazily parsing policies on demand. On reflection though, we can improve our memory usage quite a bit too by compressing the unparsed content. This drops our memory usage by roughly 6% without impacting runtime.
-
Damian Johnson authored
Reusing parsed tor versions saves us an impressive 3.7% memory usage without any noticable impact on parsing times.
-
Damian Johnson authored
Oddly, the only exit policy I can find with greater than five occurances is the reject-all policy. Maybe a scripting bug. Reguardless, referencing a single reject-all instance saves us a little memory (1.3 MB with the present consensus, which is 1.6%).
-
Damian Johnson authored
Each server descriptor constructed its own default value for the IPv6 exit policy, which was foolish. ExitPolicy instances are read-only instances, so all descriptors might as well use the same default. This saves roughly 2.3 MB of memory when downloading all of the current server descriptors (or roughly 3%).
-
Damian Johnson authored
The ExitPolicy and ExitPolicyRule classes are a perfect fit for @lru_cache. They're read-only classes that already do a fair bit of caching. The annotation lets us avoid doing this ourselves.
-
- Oct 07, 2013
-
-
Damian Johnson authored
Now that we have SSL support there's little reason to point to the http endpoint.
-
Damian Johnson authored
Python 3.2 added a memoization annotation to python's functools module. This is very, very handy, allowing us to avoid the caching boilerplate I do way too often... def get_foo(): if self._foo is None: ... stuff to calculate self._foo... return self._foo With a memoization function this becomes... @lru_cache() def get_foo(): ... stuff to calculate self._foo... This is a MIT licensed backport from... http://code.activestate.com/recipes/578078-py26-and-py30-backport-of-python-33s-lru-cache/ Looking forward to when we require python 3.2 so we can use the builtin!
-
- Oct 02, 2013
-
-
Damian Johnson authored
Traceback (most recent call last): File "./run_tests.py", line 404, in <module> main() File "./run_tests.py", line 245, in main println(exc, ERROR) File "/home/atagar/Desktop/stem/test/output.py", line 59, in println msg = term.format(msg, *attr) File "/home/atagar/Desktop/stem/stem/util/term.py", line 96, in format return (CSI % ";".join(encodings)) + msg + RESET TypeError: cannot concatenate 'str' and 'UnsupportedOperation' objects
-
Damian Johnson authored
Our Version class lacked a __hash__() method, causing it to behave unintuitively in dictionaries and sets... >>> from stem.version import Version >>> foo = set([Version('0.2.4.9-alpha')]) >>> bar = set([Version('0.2.4.9-alpha')]) >>> foo.difference(bar) set([<stem.version.Version object at 0xb71fae0c>]) This is because python opts for the object's identity when a hash method does not exist.
-
- Sep 30, 2013
-
-
Damian Johnson authored
Adding a blue halo for the center of the windrose. This doesn't address the main thing I dislike about the icon, which is the pixelation of the black lines, but still it makes it look a bit nicer. I've tried several things to address the black line pixilation (blur, thickening, etc) but this is the best improvement I've yet found.
-
Damian Johnson authored
Oops, forgot one of the spots we need to add new pages...
-
Damian Johnson authored
Introducing a new tutorial section called 'East of the Sun & West of the Moon' (... I love that fairytale) that introduces users to our utility modules. Presently this just has connection resolution, a spiffy new feature I added last weekend.
-
- Sep 28, 2013
-
-
Damian Johnson authored
Oops, our example page was still linking to my tor-utils repository rather than DocTor's.
-
- Sep 23, 2013
-
-
Damian Johnson authored
Having get_system_resolvers() check that resolvers are in our PATH before returning them.
-
Damian Johnson authored
Oops, our unit tests didn't account for proc now providing the protocol. Caught by our jenkins tests.
-
Damian Johnson authored
Adding long overdue support for process connection resolution... https://trac.torproject.org/7910 This is a highly popular capability of arm, and stem's counterpart for it is quite a bit cleaner (with tests!). That said, this still doesn't get around tor's annoying DisableDebuggerAttachment feature which screws up proc permissions. That's something I'll need to figure out before our next arm or stem release...
-