- Dec 29, 2019
-
-
Damian Johnson authored
-
Damian Johnson authored
The example we provided did not deduplicate relays, causing us to cite relays repeatedly for each descriptor they published.
-
- Dec 28, 2019
-
-
Damian Johnson authored
-
- Dec 27, 2019
-
-
Damian Johnson authored
When I first wrote this module I played fast and lose with the 'start' and 'end' parameters, guessing relevance purely based on filenames. When Karsten added timestamps to the index he better defined the relevant timestamp to be a descriptor's publication, which everything except microdescriptors contains. Interestingly, archives can contain publications both before and after its filename date. For example... recent/relay-descriptors/server-descriptors/2019-12-27-22-04-59-server-descriptors Old filename derived timestamps: start: 2019-12-27 22:04:59 end: 2019-12-27 23:04:59 Index's publication timpestamps: start: 2019-12-27 20:30:00 end: 2019-12-27 22:45:00 If the file was created at 22:04 how does it contain something published at 22:45? Regardless, now that the index contains publication times for our purposes filenames dates are moot. Our 'start' and 'end' arguments provide the subset of archives that reside within the given publication range. For example, the following downloads descriptors that were published up to two hours ago... recent = datetime.datetime.utcnow() - datetime.timedelta(minutes = 120) descriptors = stem.descriptor.collector.get_server_descriptors(start = recent) If we make this more sophisticated we can demonstrate how many descriptors we pull from each archive... import datetime import stem.descriptor.collector collector = stem.descriptor.collector.get_instance() recent = datetime.datetime.utcnow() - datetime.timedelta(minutes = 120) # This effectively does the same thing as get_server_descriptors(), # but in a way we can also determine the full counts. for f in collector.files('server-descriptor', start = recent): all_desc = list(f.read()) recent_desc = list(f.read(start = recent)) print('%s (%s => %s)' % (f.path, f.start, f.end)) print(' %i of %i descriptors were published recently' % (len(recent_desc), len(all_desc))) # Download them again, but through our more prevalently used # get_server_descriptors() method. print('\nIn total there are %i server descriptors published recently' % len(list(collector.get_server_descriptors(start = recent)))) ---------------------------------------------------------------------- % python demo.py recent/relay-descriptors/server-descriptors/2019-12-27-21-04-59-server-descriptors (2019-12-27 17:59:00 => 2019-12-27 22:13:00) 3 of 817 descriptors were published recently recent/relay-descriptors/server-descriptors/2019-12-27-22-04-59-server-descriptors (2019-12-27 20:30:00 => 2019-12-27 22:45:00) 297 of 776 descriptors were published recently recent/relay-descriptors/server-descriptors/2019-12-27-23-04-59-server-descriptors (2019-12-27 21:49:00 => 2019-12-27 23:01:00) 800 of 800 descriptors were published recently In total there are 1100 server descriptors published recently
-
Damian Johnson authored
Minor fix for three online tests... ====================================================================== FAIL: test_index_plaintext ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/atagar/Desktop/stem/test/require.py", line 43, in wrapped return func(self, *args, **kwargs) File "/home/atagar/Desktop/stem/test/require.py", line 58, in wrapped return func(self, *args, **kwargs) File "/home/atagar/Desktop/stem/test/integ/descriptor/collector.py", line 22, in test_index_plaintext self._test_index(None) File "/home/atagar/Desktop/stem/test/integ/descriptor/collector.py", line 98, in _test_index self.assertEqual(['archive', 'contrib', 'recent'], [entry['path'] for entry in index['directories']]) AssertionError: Lists differ: ['archive', 'contrib', 'recent... != [u'archive', u'recent'] First differing element 1: 'contrib' u'recent' First list contains 1 additional elements. First extra element 2: 'recent' - ['archive', 'contrib', 'recent'] ? ^^^^^^^^^^^ + [u'archive', u'recent'] ? + ^
-
Damian Johnson authored
The manual has changed quite a bit, requiring small parser adjustments.
-
- Dec 25, 2019
-
-
Damian Johnson authored
Adding test coverage for these GETINFO options... https://github.com/torproject/stem/issues/29
-
- Dec 21, 2019
-
-
Damian Johnson authored
Tor is preparing to move to Gitlab. Rather than follow it I'm moving to GitHub. Just finished migraing our tickets so now updating the bug tracker links.
-
- Dec 12, 2019
-
-
-
Damian Johnson authored
Long requested test implemented by hannelores. https://trac.torproject.org/projects/tor/ticket/15701
-
Damian Johnson authored
Functionally the same. Simply avoiding a SETCONF for a fake address (doesn't seem to harm anything in practice, but unnecessary), and adjusting how we split the response.
-
Damian Johnson authored
Stem uses two rather than four space indentations. Also condensing the line wrapped statements (stem doesn't use them, imho they impair readability).
-
-
- Dec 09, 2019
-
-
Damian Johnson authored
Nick's new test passes with flying colors, but does not revert the ORPort back to the value we previously had. This worked because this test coincidently was one of the last. Also fixing minor stylistic issues cited by pycodestyle... STATIC CHECKS * /home/atagar/Desktop/stem/test/integ/control/controller.py line 794 - use single rather than double quotes | controller.set_conf("ORPort", "0") line 797 - use single rather than double quotes | controller.set_options([("UseBridges", "1"), line 798 - use single rather than double quotes | ("Bridge", "127.0.0.1:9999")]) line 801 - use single rather than double quotes | controller.set_conf("UseBridges", "0")
-
This closes Tor ticket 31909.
-
- Dec 04, 2019
-
-
Damian Johnson authored
Oops. I added a string method on the assumption that I'd use it when creating descriptors but didn't. It's unused so simply dropping it so we have the option of adding a different string method later.
-
Damian Johnson authored
Great catch from asn that with python3 our AuthorizedClient construction embeds extra b'', malforming descriptor output... https://trac.torproject.org/projects/tor/ticket/31823#comment:19 Fixing this is a general fashion such that AuthorizedClient both normalizes its field and can be constructed with unsupplied values randomized (so, for example, you can create a AuthorizedClient with only an iv).
-
- Dec 02, 2019
-
-
Damian Johnson authored
One new parameter, and another from an earlier commit I missed. https://gitweb.torproject.org/torspec.git/commit/?id=29a110f
-
- Dec 01, 2019
-
-
Damian Johnson authored
Huh. I lack a repro so unsure how this came about but someone reported a negative reset time... https://trac.torproject.org/projects/tor/ticket/32642 It would be nice to correct the root cause, but cest la vi. Ensuring these counts are non-negative.
-
Damian Johnson authored
Couple fixes for issues caught by teor... https://trac.torproject.org/projects/tor/ticket/32398 I reproed this by issuing a 'killall tor' during our integ tests. This produced a couple errors depending on when the process dies... Traceback (most recent call last): File "run_tests.py", line 468, in <module> main() File "run_tests.py", line 304, in main if not integ_runner.assert_tor_is_running(): File "/home/atagar/Desktop/stem/test/runner.py", line 507, in assert_tor_is_running process_output = (self._tor_process.stdout.read() + '\n\n' + self._tor_process.stderr.read()).strip() TypeError: can't concat bytes to str Traceback (most recent call last): File "run_tests.py", line 468, in <module> main() File "run_tests.py", line 304, in main if not integ_runner.assert_tor_is_running(): File "/home/atagar/Desktop/stem/test/runner.py", line 502, in assert_tor_is_running process_status = self._tor_process.poll() # None if running AttributeError: 'NoneType' object has no attribute 'poll'
-
Damian Johnson authored
Thanks to Karsten CollecTor now provides descriptor types, checksums, and time ranges... https://trac.torproject.org/projects/tor/ticket/31204 As such we no longer need to guess these based on the filename (hazaa!).
-
- Nov 28, 2019
-
-
Damian Johnson authored
George had a great question today about catching event listener exceptions... https://lists.torproject.org/pipermail/tor-dev/2019-November/014092.html Expanding our event listener tutorial to dive a bit deeper into this topic.
-
- Nov 27, 2019
-
-
Damian Johnson authored
Our Event's arrived_at attribute has a couple wrinkes... * This timestamp reflects when the event was **parsed** rather than **received**, so it becomes inaccurate if our event loop gets bogged down. * There's nothing event specific about this attribute. It should apply to all controller messages. As such moving this up to the parent class. I first spotted the bug via the following script... import time from stem.control import EventType, Controller def slow_handler(event): print("processing a BW event that's %0.1f seconds old" % (time.time() - event.arrived_at)) time.sleep(5) with Controller.from_port() as controller: controller.authenticate() controller.add_event_listener(slow_handler, EventType.BW) time.sleep(10) Previously this produced... % python demo.py processing a BW event that's 0.0 seconds old processing a BW event that's 0.0 seconds old processing a BW event that's 0.0 seconds old processing a BW event that's 0.0 seconds old ... and now we get... % python demo.py processing a BW event that's 0.4 seconds old processing a BW event that's 4.4 seconds old processing a BW event that's 8.4 seconds old
-
- Nov 22, 2019
-
-
Damian Johnson authored
Renaming our IntroductionPointV3's create() method to create_from_address, and adding create_for_link_specifiers(). If other constructors are useful we can add using a similar convention.
-
Damian Johnson authored
auth-client are a required field... https://trac.torproject.org/projects/tor/ticket/32563 https://trac.torproject.org/projects/tor/ticket/31823#comment:16 Tor creates sixteen by default, so doing the same unless our caller specifies otherwise. % cat demo.py from stem.descriptor.hidden_service import OuterLayer print(OuterLayer.create()) % python demo.py desc-auth-type x25519 desc-auth-ephemeral-key fhUEgYTR3j4/8MC8aH75WTIKWFOzvKiXURvPHdIXTH4= auth-client 1J4LPB+umNA iOGHkI+vpsWJaugbv6akgg EtisFxHd3buXMcl30uxJ8A auth-client wbmXMy71u+Q aiFlaQQV3hBPrmOaQ8kXDg HP+ZFqEt0z47F5AVBnyzxA auth-client ahHWCcU8INM kQrwT8jZOXAdxiyeVZ/EWw tIBpoOqhvoGhFCOceSLZXA auth-client UrnUpDYoCTs 6VC/C7xcIWd4Dtmrj1nKig 1k2hfoPhGPwX4BORgHeDPw auth-client 5I03RmlXJps rgdrWYmq02EmMF+v7PMoLw lkxQtmEz3+CVniXM0E16vQ auth-client ub0ap7cdghQ gGjmEJTYcw6RGwri2DlPaQ Wz+vnYEc2PmeFxM85lNCcg auth-client TLgL1NsDjIU bKuIgs/blO3mL80mC13JDw 8sjbuwv3o9sQTt0yQbhIxw auth-client aytYGQGGAkE lqg3uBRYIX9DHjWl8aLmxQ Nsb6dvhnykFJtSyeGekUhw auth-client htPYuHAyOsA Wim8CZKo3o2toq+dqeGgMg m6vIoCpQWT0JRR+JgcHIqw auth-client oJJr9IpqKpc gwd5SFc0CWtRMAaZaqWQrQ iA8qoyKVXWs2N6DA9WzqiQ auth-client EEIi97LqhGA K+Vn6P2dfRgFZXHhgBvX9A 9bXjd6UBUXuBY3/aeNCR5w auth-client 1JCG9WKhPKY CgMjZquv+KPLzpqn72uMKQ kC6e5GM/1+2TIvq8kb47Lw auth-client 6MuN+1vob4w Wi1ktyiaiOaG6PaLbneydQ OEigSskYEZb8hGqWZMDKlg auth-client uOoXx8epem4 /VjgpZzC71Gj6WAXuVXzVw Q4dOW6GYc/JoQrKNLUQmbQ auth-client aC9d6RbIr1M FCY56VPOLEktEsQiCDrORQ 0fAhfjriUxnFvbjiTfGDrw auth-client 60vj3crxSwg XSytZMniMFUCDYbhyrdhDg 3P5Q5QaI70AvX/d8Wh7Etw encrypted -----BEGIN MESSAGE----- EqEXDglNZxN+TmE6I7U7fd9DM0ue9ys770n6AnH4ga+fLSJ8AOb0lX4XANG9TuFh 6hyOfGa0joPezoQSeAFf+yk= -----END MESSAGE-----
-
Damian Johnson authored
Great catch from asn on #31823 that we should generate desc-auth-ephemeral-key fields from a key rather than random bytes. Otherwise this can be used as a fingerprint to differentiate our descriptors from tor's.
-
Damian Johnson authored
Thanks to Paul ed25519 blinding is now fully two orders of magnitude faster! https://github.com/pyca/cryptography/issues/5068 Replaced slow_ed25519.py with an optimized implementation from... https://github.com/pyca/ed25519/ This changes the runtime of test_blinding as follows: Python 2.7: 2.25s => 20 ms Python 3.5: 1.83s => 19 ms
-
- Nov 20, 2019
-
-
Damian Johnson authored
Each key blinding takes a couple seconds so I avoided it in our unit tests, but we should perform one instance for coverage. Testing with a static key and fixing the python 3.x normalization issue this surfaced.
-
- Nov 19, 2019
-
-
Damian Johnson authored
Oops, accidently commited a local hack I made while troubleshooting my openssl bindings. This in turn broke our jenkins test runs... ====================================================================== ERROR: test_inner_layer_creation ---------------------------------------------------------------------- Traceback (most recent call last): File "/srv/jenkins-workspace/workspace/stem-tor-ci/test/unit/descriptor/hidden_service_v3.py", line 343, in test_inner_layer_creation IntroductionPointV3.create('1.1.1.1', 9001), File "/srv/jenkins-workspace/workspace/stem-tor-ci/stem/descriptor/hidden_service.py", line 230, in create raise ImportError('Introduction point creation requires the cryptography module ed25519 support') ImportError: Introduction point creation requires the cryptography module ed25519 support ----------------------------------------------------------------------
-
Damian Johnson authored
Correcting the issues spotted by asn at... https://trac.torproject.org/projects/tor/ticket/31823#comment:11
-
Damian Johnson authored
Oops, another nice catch from asn. base64.b64encode() return bytes, which will then include an extra b'' in our encode() method.
-
Damian Johnson authored
We didn't provide a mechanism to exclude an onion-key (create() filled it in with a default), but it was definitely a mistake to include this conditional. Caught by asn on https://trac.torproject.org/projects/tor/ticket/31823
-
Damian Johnson authored
My python 3.5 interpreter lacked ed25519 openssl bindings, preventing me from exercising these code paths. Now that they're working addressing normalization we need.
-
- Nov 18, 2019
-
-
Damian Johnson authored
Thanks to asn we now have an upstream ticket for the ed25519 blinding support we need.
-
Damian Johnson authored
Jenkins has long struggled with this assertion, and I see these failures locally from time to time too... ====================================================================== FAIL: test_query_with_timeout ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/mock/mock.py", line 1305, in patched return func(*args, **keywargs) File "/home/atagar/Desktop/stem/test/unit/descriptor/remote.py", line 387, in test_query_with_timeout self.assertEqual(2, dirport_mock.call_count) AssertionError: 2 != 3 ---------------------------------------------------------------------- No assertion is better than a flaky one, so dropping this last bit.
-
Damian Johnson authored
Oops, turns out the cryptography module doesn't always supply this method... Traceback (most recent call last): File "./run_tests.py", line 36, in <module> import test.runner File "/srv/jenkins-workspace/workspace/stem-tor-ci/test/runner.py", line 44, in <module> import stem.connection File "/srv/jenkins-workspace/workspace/stem-tor-ci/stem/connection.py", line 136, in <module> import stem.control File "/srv/jenkins-workspace/workspace/stem-tor-ci/stem/control.py", line 271, in <module> import stem.descriptor.microdescriptor File "/srv/jenkins-workspace/workspace/stem-tor-ci/stem/descriptor/__init__.py", line 1544, in <module> import stem.descriptor.hidden_service File "/srv/jenkins-workspace/workspace/stem-tor-ci/stem/descriptor/hidden_service.py", line 81, in <module> X25519_AVAILABLE = backend.x25519_supported() AttributeError: 'Backend' object has no attribute 'x25519_supported'
-
- Nov 17, 2019
-
-
Damian Johnson authored
Cryptographically valid support for hidden service creation... https://trac.torproject.org/projects/tor/ticket/31823 HSv3 descriptors consist of three parts: an inner layer, outer layer, and the descriptor itself. Callers of HiddenServiceDescriptorV3's create() and content() methods can supply these to specify that layer's parameters. For example, to supply custom introduction points with random key material simply call... HiddenServiceDescriptorV3.content( inner_layer = InnerLayer.create( introduction_points = [ IntroductionPointV3.create('1.1.1.1', 9001), IntroductionPointV3.create('2.2.2.2', 9001), IntroductionPointV3.create('3.3.3.3', 9001), ], ), )
-
Damian Johnson authored
On reflection, why not simply disable blinding by default? Blinding parameters are unnecessary for the vast majority of use cases, and when needed the caller can simply provide a nonce (and by extension accept the lengthy runtime). We can always re-enable blinding as a default if/when we have a performant implementation.
-
Damian Johnson authored
Resuming our HSv3 tests, with blinding mocked out so we don't negatively impact runtime (without these mocks the unit tests jump from 6s => 51s).
-
Damian Johnson authored
I won't pretend to understand this math. A smarter mind than mine (asn's) came up with this crypto. Just massaging it into a form I find easier to understand.
-