- Nov 05, 2017
-
-
Damian Johnson authored
-
- Nov 04, 2017
-
-
Damian Johnson authored
Running the ONLINE integ target uncovered another bug... ====================================================================== ERROR: test_get_circuits ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/atagar/Desktop/stem/test/require.py", line 58, in wrapped return func(self, *args, **kwargs) File "/home/atagar/Desktop/stem/test/require.py", line 58, in wrapped return func(self, *args, **kwargs) File "/home/atagar/Desktop/stem/test/require.py", line 58, in wrapped return func(self, *args, **kwargs) File "/home/atagar/Desktop/stem/test/integ/control/controller.py", line 1339, in test_get_circuits circuits = controller.get_circuits() File "/home/atagar/Desktop/stem/stem/control.py", line 482, in wrapped return func(self, *args, **kwargs) File "/home/atagar/Desktop/stem/stem/control.py", line 3336, in get_circuits circ_message = stem.socket.recv_message(io.BytesIO(stem.util.str_tools._to_unicode('650 CIRC %s\r\n' % circ))) TypeError: 'unicode' does not have the buffer interface
-
Damian Johnson authored
Ran with a small hack to strip brackets due to malformed manual content... https://trac.torproject.org/projects/tor/ticket/24147
-
Damian Johnson authored
We added unit tests to check that tor_tools functions like is_valid_fingerprint() worked for both bytes and unicode, but these tests actually only passes for python 2.x. With python3 they only accepted unicode. Normalizing the inputs.
-
Damian Johnson authored
Recent change broke is_running for python3... ====================================================================== ERROR: test_is_running_by_name ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/atagar/Desktop/stem/test/require.py", line 58, in wrapped return func(self, *args, **kwargs) File "/home/atagar/Desktop/stem/test/integ/util/system.py", line 144, in test_is_running_by_name self.assertTrue(stem.util.system.is_running(tor_cmd) or stem.util.system.is_running('tor.real')) File "/home/atagar/Desktop/stem/stem/util/system.py", line 449, in is_running if isinstance(command, (bytes, unicode)): NameError: global name 'unicode' is not defined
-
- Nov 03, 2017
-
-
Damian Johnson authored
Though sqlite3 is usually built in, it's an optional module that's commonly missing on FreeBSD and Gentoo... https://lists.torproject.org/pipermail/tor-relays/2017-October/013433.html https://lists.torproject.org/pipermail/tor-relays/2017-October/013440.html Like the cryptography module making this a soft dependency that's only required if it's used.
-
- Nov 02, 2017
-
-
Damian Johnson authored
So much confusion around this... https://trac.torproject.org/projects/tor/ticket/24110 It turns out in practice these methods essentially never provided microdescriptor flavored router status entries because 'UseMicrodescriptors' defaults to 'auto' (and it's rare to set it to '1' since that's the default anyway). However, if that was done we stacktraced... >>> desc = next(controller.get_network_statuses()) Traceback (most recent call last): File "<console>", line 1, in <module> File "/home/atagar/Desktop/stem/stem/control.py", line 494, in wrapped for val in func(self, *args, **kwargs): File "/home/atagar/Desktop/stem/stem/control.py", line 1993, in get_network_statuses for desc in desc_iterator: File "/home/atagar/Desktop/stem/stem/descriptor/router_status_entry.py", line 104, in _parse_file yield entry_class(desc_content, validate, *extra_args) File "/home/atagar/Desktop/stem/stem/descriptor/router_status_entry.py", line 446, in __init__ raise ValueError("%s must have a '%s' line:\n%s" % (self._name(True), keyword, str(self))) ValueError: Router status entries (micro v3) must have a 'm' line: r networkofthesmoker AAaX5kASI9/xzl9Qp7aFdPOSnrQ aoqBoLrUWKssJAKnO8r6GpZs5p8 2017-11-02 13:51:59 128.72.180.74 443 0 s Fast Guard HSDir Running Stable V2Dir Valid w Bandwidth=4150 From what I can tell nowadays UseMicrodescriptors doesn't change the 'GETINFO ns/all' response. As such reverting... https://gitweb.torproject.org/stem.git/commit/?id=d70fa9cd
-
- Oct 31, 2017
-
-
Damian Johnson authored
FreeBSD ps output lists the full tor path rather than just the process name. Recognizing its default location so our connect() method provides better messaging when unable to connect.
-
Damian Johnson authored
Tim and Sebastian pointed out that we're still not doing correct localhost detection... https://trac.torproject.org/projects/tor/ticket/24085
-
- Oct 30, 2017
-
-
Damian Johnson authored
Oops, sometimes tor provides 127.0.0.1 for localhost and other times 0.0.0.0... >>> GETINFO net/listeners/or 250-net/listeners/or="0.0.0.0:1443" 250 OK >>> GETINFO net/listeners/control 250-net/listeners/control="127.0.0.1:9051" 250 OK
-
Damian Johnson authored
Sweet jesus that was a pita to track down. Nyx has a memory leak, holding onto events tor emits indefinitely. Spent all day headscratching and finally found it. Our control interpreter intercepts events and enqueues them without bound. Finally caught thanks to the gc module... to_ignore = [ locals(), globals(), gc.garbage ] log.warn("refs: %s" % [inspect.getframeinfo(r) for r in gc.get_referrers(event_message) if r not in to_ignore]) ... which gave... 06:44:50 [NYX_WARNING] refs: [Traceback(filename='/home/ubuntu/nyx/stem/interpreter/commands.py', lineno=135, function='handle_event_wrapper', code_context=[' handle_event_real(event_message)\n'], index=0), Traceback(filename='/home/ubuntu/nyx/stem/control.py', lineno=3812, function='_handle_event', code_context=[' log.warn("refs: %s" % [inspect.getframeinfo(r) for r in gc.get_referrers(event_message) if r not in to_ignore])\n'], index=0), Traceback(filename='/home/ubuntu/nyx/stem/control.py', lineno=975, function='_event_loop', code_context=[' self._handle_event(event_message)\n'], index=0)] [998 duplicates hidden]
-
Damian Johnson authored
Controller messages frequently reference their string representation, and it's silly to reconstruct it each time.
-
- Oct 29, 2017
-
-
Damian Johnson authored
Oops, documented it but didn't set it.
-
Damian Johnson authored
Parsing the whole consensus can be burdensome. Skip doing this upfront so the caller can decide if they want a parsed version or not.
-
Damian Johnson authored
Oops, it's optional with other descriptor types but not router status entries.
-
Damian Johnson authored
Dropping validation when we receive it from the control port. This makes a particularly big impact in performance for NEWCONSENSUS events.
-
Damian Johnson authored
This change broke the ability for get_info() to accept unicode inputs (or bytes under python3)... >>> controller.get_network_status(u'0881E47137764C907E729C9303C094BD270CC27F') Traceback (most recent call last): File "<console>", line 1, in <module> File "/home/ubuntu/stem/stem/control.py", line 482, in wrapped return func(self, *args, **kwargs) File "/home/ubuntu/stem/stem/control.py", line 1928, in get_network_status desc_content = self.get_info(query, get_bytes = True) File "/home/ubuntu/stem/stem/control.py", line 482, in wrapped return func(self, *args, **kwargs) File "/home/ubuntu/stem/stem/control.py", line 1165, in get_info cached_results = self._get_cache_map(map(str.lower, params), 'getinfo') TypeError: descriptor 'lower' requires a 'str' object but received a 'unicode' This reverts commit e134dc59.
-
Damian Johnson authored
Thought these might be having an issue but turns out that's not the case. Good tests none the less.
-
- Oct 27, 2017
-
-
Damian Johnson authored
Oops, python 2.6 fixes added a regression breaking everything *except* python 2.6. Oops. :P Traceback (most recent call last): File "./run_tests.py", line 20, in <module> RUN_ASYNC_TESTS = true NameError: name 'true' is not defined
-
Damian Johnson authored
Python 2.6 lacks assertRaisesRegexp so we made our own handler for it. However, the assertions it did were more strict, causing failures for tests that pass with other python versions... ====================================================================== FAIL: test_get_info_address_caching ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.6/dist-packages/mock.py", line 1201, in patched return func(*args, **keywargs) File "/home/atagar/Desktop/stem/test/unit/control/controller.py", line 61, in test_get_info_address_caching self.assertRaisesRegexp(stem.ProtocolError, 'Address unknown', self.controller.get_info, 'address') File "/home/atagar/Desktop/stem/stem/util/test_tools.py", line 270, in assertRaisesRegexp self.assertTrue(re.match(exc_msg, str(exc))) AssertionError
-
Damian Johnson authored
Turns out importlib was added in python 2.7... Traceback (most recent call last): File "./run_tests.py", line 9, in <module> import importlib ImportError: No module named importlib This means we're losing quite a bit of test coverage under python 2.6 but... screw it. Guido rolled his eyes when I mentioned we even offered 2.6 support. It's been dead a long time. We'll continue to support python 2.6 until stem 2.0, but not gonna bend over backwards for it. This change will let us invoke our tests again.
-
Damian Johnson authored
Oops, python3 regression slipped in with prior work... ====================================================================== ERROR: test_connections ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.2/dist-packages/mock/mock.py", line 1305, in patched return func(*args, **keywargs) File "/home/atagar/Desktop/stem/test/unit/util/proc.py", line 215, in test_connections tcp = b'%s\n 0: 11111111:1111 22222222:2222 01 44444444:44444444 55:55555555 66666666 1111 8 99999999' % TITLE_LINE TypeError: unsupported operand type(s) for %: 'bytes' and 'bytes'
-
- Oct 25, 2017
-
-
Damian Johnson authored
This is a mutable parameter, but it's both reasonably static and highly requested. Caching it for a second by default.
-
- Oct 22, 2017
-
-
Damian Johnson authored
Oops, a recent addition in commit 82a3cc05 can cause us to clear the cache when called without any parameters. What we actually meant there was to allow callers to explicitly clear, so checking for None instead.
-
Damian Johnson authored
Probably the most requested GETINFO parameter is our fingerprint. We cache fingerprint values, but not exceptions causing any request when not a relay to go to tor. This is unnecessary. We can detect if we become a relay so caching the 'not a relay' response to avoid having callers (like Nyx) accidently hammer tor.
-
- Oct 20, 2017
-
-
Damian Johnson authored
-
- Oct 17, 2017
-
-
Damian Johnson authored
Good spot for map to simplify this a tad.
-
Damian Johnson authored
When iterating over keys python 2.x makes a shallow clone of lists whereas python 3.x doesn't. Good for performance, but means we need to be careful about modifying a list during iteration. Caught thanks to our integ tests... ====================================================================== ERROR: test_without_ephemeral_hidden_services ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/atagar/Desktop/stem/test/require.py", line 58, in wrapped return func(self, *args, **kwargs) File "/home/atagar/Desktop/stem/test/require.py", line 58, in wrapped return func(self, *args, **kwargs) File "/home/atagar/Desktop/stem/test/integ/control/controller.py", line 560, in test_without_ephemeral_hidden_services self.assertEqual([], controller.list_ephemeral_hidden_services()) File "/home/atagar/Desktop/stem/stem/control.py", line 475, in wrapped return func(self, *args, **kwargs) File "/home/atagar/Desktop/stem/stem/control.py", line 2774, in list_ephemeral_hidden_services result += self.get_info('onions/current').split('\n') File "/home/atagar/Desktop/stem/stem/control.py", line 475, in wrapped return func(self, *args, **kwargs) File "/home/atagar/Desktop/stem/stem/control.py", line 1191, in get_info self._set_cache(to_cache, 'getinfo') File "/home/atagar/Desktop/stem/stem/control.py", line 3112, in _set_cache for cache_key in self._request_cache.keys(): RuntimeError: dictionary changed size during iteration
-
Damian Johnson authored
Fixing bytes normalization issues that broke our stem.util.proc unit tests for python3.
-
- Oct 15, 2017
-
-
Damian Johnson authored
Can't think of any reason we'd want connections with an undefined address or port, so skipping those entries. This doesn't come up in practice but when I hack my proc contents a bit I get... File "/usr/lib/python2.7/stem/util/lru_cache.py", line 135, in wrapper result = user_function(*args, **kwds) File "/usr/lib/python2.7/stem/exit_policy.py", line 289, in can_exit_to if rule.is_match(address, port, strict): File "/usr/lib/python2.7/stem/exit_policy.py", line 759, in is_match raise ValueError("'%s' isn't a valid port" % port) ValueError: '0' isn't a valid port
-
- Oct 11, 2017
-
-
Damian Johnson authored
Tor's HiddenServiceNonAnonymousMode option is rather new. Nice that we have an integ test for it, but we should account for versions without it too... ====================================================================== FAIL: test_unanonymous_hidden_service_config_must_match ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/atagar/Desktop/stem/stem/util/test_tools.py", line 150, in <lambda> self.method = lambda test: self.result(test) # method that can be mixed into TestCases File "/home/atagar/Desktop/stem/stem/util/test_tools.py", line 212, in result test.fail(self._result.msg) AssertionError: Expected 'Process terminated: HiddenServiceNonAnonymousMode does not provide any server anonymity. It must be used with HiddenServiceSingleHopMode set to 1.' but was 'Process terminated: Unknown option 'HiddenServiceNonAnonymousMode'. Failing.'
-
- Oct 10, 2017
-
-
Damian Johnson authored
Bending my usual policy of only adding parsing support once fields are in the spec due to an amusing circular dependency... https://trac.torproject.org/projects/tor/ticket/21177#comment:5
-
- Oct 08, 2017
-
-
- Oct 04, 2017
-
-
Edmund Wong authored
-
- Sep 26, 2017
-
-
Damian Johnson authored
Proc's /proc/net/tcp commonly has thousands of lines on even moderately busy relays. For me it takes about half a second to process, pegging the cpu during that time. After lots of experimenting finally found something to greatly speed this up. Parsing the proc content via index rather than spit makes a large difference. More delicate so this might require some adjustment to accomidate how other platforms format their proc. This is about a 30% improvement, but along with the earlier changes this in total accounts for a 5x speedup in proc conneciton resolution, going from 0.38s to 0.07s for my relay's proc contents.
-
- Sep 24, 2017
-
-
Damian Johnson authored
Still not where I'd care for it to be at but on my relay these drop the average call runtime from 0.1262s to 0.0808s (36% faster).
-
Damian Johnson authored
Decoding addresses is somewhat costly, and repeated quite a bit.
-
Damian Johnson authored
-
Damian Johnson authored
The uid is only used for a simple comparison. Might as well convert what we're looking for to an str rather than every entry to an int.
-
Damian Johnson authored
Silly to do these two on every line of the proc file.
-