Skip to content
Snippets Groups Projects
  1. Dec 29, 2019
  2. Dec 28, 2019
  3. Dec 27, 2019
    • Damian Johnson's avatar
      CollecTor publication time filtering · 6c07fe55
      Damian Johnson authored
      When I first wrote this module I played fast and lose with the 'start' and
      'end' parameters, guessing relevance purely based on filenames.
      
      When Karsten added timestamps to the index he better defined the relevant
      timestamp to be a descriptor's publication, which everything except
      microdescriptors contains.
      
      Interestingly, archives can contain publications both before and after its
      filename date. For example...
      
        recent/relay-descriptors/server-descriptors/2019-12-27-22-04-59-server-descriptors
      
        Old filename derived timestamps:
          start: 2019-12-27 22:04:59
          end: 2019-12-27 23:04:59
      
        Index's publication timpestamps:
          start: 2019-12-27 20:30:00
          end: 2019-12-27 22:45:00
      
      If the file was created at 22:04 how does it contain something published at
      22:45?
      
      Regardless, now that the index contains publication times for our purposes
      filenames dates are moot. Our 'start' and 'end' arguments provide the subset
      of archives that reside within the given publication range.
      
      For example, the following downloads descriptors that were published up to two
      hours ago...
      
        recent = datetime.datetime.utcnow() - datetime.timedelta(minutes = 120)
        descriptors = stem.descriptor.collector.get_server_descriptors(start = recent)
      
      If we make this more sophisticated we can demonstrate how many
      descriptors we pull from each archive...
      
        import datetime
        import stem.descriptor.collector
      
        collector = stem.descriptor.collector.get_instance()
        recent = datetime.datetime.utcnow() - datetime.timedelta(minutes = 120)
      
        # This effectively does the same thing as get_server_descriptors(),
        # but in a way we can also determine the full counts.
      
        for f in collector.files('server-descriptor', start = recent):
          all_desc = list(f.read())
          recent_desc = list(f.read(start = recent))
      
          print('%s (%s => %s)' % (f.path, f.start, f.end))
          print('  %i of %i descriptors were published recently' % (len(recent_desc), len(all_desc)))
      
        # Download them again, but through our more prevalently used
        # get_server_descriptors() method.
      
        print('\nIn total there are %i server descriptors published recently' % len(list(collector.get_server_descriptors(start = recent))))
      
        ----------------------------------------------------------------------
      
        % python demo.py
      
        recent/relay-descriptors/server-descriptors/2019-12-27-21-04-59-server-descriptors (2019-12-27 17:59:00 => 2019-12-27 22:13:00)
          3 of 817 descriptors were published recently
        recent/relay-descriptors/server-descriptors/2019-12-27-22-04-59-server-descriptors (2019-12-27 20:30:00 => 2019-12-27 22:45:00)
          297 of 776 descriptors were published recently
        recent/relay-descriptors/server-descriptors/2019-12-27-23-04-59-server-descriptors (2019-12-27 21:49:00 => 2019-12-27 23:01:00)
          800 of 800 descriptors were published recently
      
        In total there are 1100 server descriptors published recently
      6c07fe55
    • Damian Johnson's avatar
      Fix collector integ test · bbd6b100
      Damian Johnson authored
      Minor fix for three online tests...
      
        ======================================================================
        FAIL: test_index_plaintext
        ----------------------------------------------------------------------
        Traceback (most recent call last):
          File "/home/atagar/Desktop/stem/test/require.py", line 43, in wrapped
            return func(self, *args, **kwargs)
          File "/home/atagar/Desktop/stem/test/require.py", line 58, in wrapped
            return func(self, *args, **kwargs)
          File "/home/atagar/Desktop/stem/test/integ/descriptor/collector.py", line 22, in test_index_plaintext
            self._test_index(None)
          File "/home/atagar/Desktop/stem/test/integ/descriptor/collector.py", line 98, in _test_index
            self.assertEqual(['archive', 'contrib', 'recent'], [entry['path'] for entry in index['directories']])
        AssertionError: Lists differ: ['archive', 'contrib', 'recent... != [u'archive', u'recent']
      
        First differing element 1:
        'contrib'
        u'recent'
      
        First list contains 1 additional elements.
        First extra element 2:
        'recent'
      
        - ['archive', 'contrib', 'recent']
        ?             ^^^^^^^^^^^
      
        + [u'archive', u'recent']
        ?  +           ^
      bbd6b100
    • Damian Johnson's avatar
      Sync manual and update assertions · f34212fb
      Damian Johnson authored
      The manual has changed quite a bit, requiring small parser adjustments.
      f34212fb
  4. Dec 25, 2019
  5. Dec 21, 2019
    • Damian Johnson's avatar
      Change bug tracker links to github · 7852f528
      Damian Johnson authored
      Tor is preparing to move to Gitlab. Rather than follow it I'm moving to GitHub.
      Just finished migraing our tickets so now updating the bug tracker links.
      7852f528
  6. Dec 12, 2019
  7. Dec 09, 2019
    • Damian Johnson's avatar
      Revert ORPort after new integ test · cb61dfe7
      Damian Johnson authored
      Nick's new test passes with flying colors, but does not revert the ORPort back
      to the value we previously had. This worked because this test coincidently was
      one of the last.
      
      Also fixing minor stylistic issues cited by pycodestyle...
      
        STATIC CHECKS
        * /home/atagar/Desktop/stem/test/integ/control/controller.py
          line 794  - use single rather than double quotes     | controller.set_conf("ORPort", "0")
          line 797  - use single rather than double quotes     | controller.set_options([("UseBridges", "1"),
          line 798  - use single rather than double quotes     | ("Bridge", "127.0.0.1:9999")])
          line 801  - use single rather than double quotes     | controller.set_conf("UseBridges", "0")
      cb61dfe7
    • Nick Mathewson's avatar
      Add a stem test to look for regressions in Tor ticket 31495. · ab44af05
      Nick Mathewson authored and Damian Johnson's avatar Damian Johnson committed
      This closes Tor ticket 31909.
      ab44af05
  8. Dec 04, 2019
  9. Dec 02, 2019
  10. Dec 01, 2019
    • Damian Johnson's avatar
      Ensure accounting reset times are non-negative · 160915ad
      Damian Johnson authored
      Huh. I lack a repro so unsure how this came about but someone reported a
      negative reset time...
      
        https://trac.torproject.org/projects/tor/ticket/32642
      
      It would be nice to correct the root cause, but cest la vi. Ensuring these
      counts are non-negative.
      160915ad
    • Damian Johnson's avatar
      Errors when tor process unexpectedly terminates · 728b71e1
      Damian Johnson authored
      Couple fixes for issues caught by teor...
      
        https://trac.torproject.org/projects/tor/ticket/32398
      
      I reproed this by issuing a 'killall tor' during our integ tests. This produced
      a couple errors depending on when the process dies...
      
        Traceback (most recent call last):
          File "run_tests.py", line 468, in <module>
            main()
          File "run_tests.py", line 304, in main
            if not integ_runner.assert_tor_is_running():
          File "/home/atagar/Desktop/stem/test/runner.py", line 507, in assert_tor_is_running
            process_output = (self._tor_process.stdout.read() + '\n\n' + self._tor_process.stderr.read()).strip()
        TypeError: can't concat bytes to str
      
        Traceback (most recent call last):
          File "run_tests.py", line 468, in <module>
            main()
          File "run_tests.py", line 304, in main
            if not integ_runner.assert_tor_is_running():
          File "/home/atagar/Desktop/stem/test/runner.py", line 502, in assert_tor_is_running
            process_status = self._tor_process.poll()  # None if running
        AttributeError: 'NoneType' object has no attribute 'poll'
      728b71e1
    • Damian Johnson's avatar
      Use new CollecTor index fields · 1543c078
      Damian Johnson authored
      Thanks to Karsten CollecTor now provides descriptor types, checksums, and time
      ranges...
      
        https://trac.torproject.org/projects/tor/ticket/31204
      
      As such we no longer need to guess these based on the filename (hazaa!).
      1543c078
  11. Nov 28, 2019
  12. Nov 27, 2019
    • Damian Johnson's avatar
      Move arrived_at from the Event class to ControlMessage · bd5c8aaa
      Damian Johnson authored
      Our Event's arrived_at attribute has a couple wrinkes...
      
        * This timestamp reflects when the event was **parsed** rather than
          **received**, so it becomes inaccurate if our event loop gets bogged down.
      
        * There's nothing event specific about this attribute. It should apply to all
          controller messages.
      
      As such moving this up to the parent class. I first spotted the bug via the
      following script...
      
        import time
      
        from stem.control import EventType, Controller
      
        def slow_handler(event):
          print("processing a BW event that's %0.1f seconds old" % (time.time() - event.arrived_at))
          time.sleep(5)
      
        with Controller.from_port() as controller:
          controller.authenticate()
          controller.add_event_listener(slow_handler, EventType.BW)
          time.sleep(10)
      
      Previously this produced...
      
        % python demo.py
        processing a BW event that's 0.0 seconds old
        processing a BW event that's 0.0 seconds old
        processing a BW event that's 0.0 seconds old
        processing a BW event that's 0.0 seconds old
      
      ... and now we get...
      
        % python demo.py
        processing a BW event that's 0.4 seconds old
        processing a BW event that's 4.4 seconds old
        processing a BW event that's 8.4 seconds old
      bd5c8aaa
  13. Nov 22, 2019
    • Damian Johnson's avatar
      More flexible IntroductionPointV3 construction · eea41c17
      Damian Johnson authored
      Renaming our IntroductionPointV3's create() method to create_from_address, and
      adding create_for_link_specifiers(). If other constructors are useful we can
      add using a similar convention.
      eea41c17
    • Damian Johnson's avatar
      Include sixteen auth-client lines by default · f5c8c96c
      Damian Johnson authored
      auth-client are a required field...
      
        https://trac.torproject.org/projects/tor/ticket/32563
        https://trac.torproject.org/projects/tor/ticket/31823#comment:16
      
      Tor creates sixteen by default, so doing the same unless our caller specifies
      otherwise.
      
        % cat demo.py
        from stem.descriptor.hidden_service import OuterLayer
      
        print(OuterLayer.create())
      
        % python demo.py
        desc-auth-type x25519
        desc-auth-ephemeral-key fhUEgYTR3j4/8MC8aH75WTIKWFOzvKiXURvPHdIXTH4=
        auth-client 1J4LPB+umNA iOGHkI+vpsWJaugbv6akgg EtisFxHd3buXMcl30uxJ8A
        auth-client wbmXMy71u+Q aiFlaQQV3hBPrmOaQ8kXDg HP+ZFqEt0z47F5AVBnyzxA
        auth-client ahHWCcU8INM kQrwT8jZOXAdxiyeVZ/EWw tIBpoOqhvoGhFCOceSLZXA
        auth-client UrnUpDYoCTs 6VC/C7xcIWd4Dtmrj1nKig 1k2hfoPhGPwX4BORgHeDPw
        auth-client 5I03RmlXJps rgdrWYmq02EmMF+v7PMoLw lkxQtmEz3+CVniXM0E16vQ
        auth-client ub0ap7cdghQ gGjmEJTYcw6RGwri2DlPaQ Wz+vnYEc2PmeFxM85lNCcg
        auth-client TLgL1NsDjIU bKuIgs/blO3mL80mC13JDw 8sjbuwv3o9sQTt0yQbhIxw
        auth-client aytYGQGGAkE lqg3uBRYIX9DHjWl8aLmxQ Nsb6dvhnykFJtSyeGekUhw
        auth-client htPYuHAyOsA Wim8CZKo3o2toq+dqeGgMg m6vIoCpQWT0JRR+JgcHIqw
        auth-client oJJr9IpqKpc gwd5SFc0CWtRMAaZaqWQrQ iA8qoyKVXWs2N6DA9WzqiQ
        auth-client EEIi97LqhGA K+Vn6P2dfRgFZXHhgBvX9A 9bXjd6UBUXuBY3/aeNCR5w
        auth-client 1JCG9WKhPKY CgMjZquv+KPLzpqn72uMKQ kC6e5GM/1+2TIvq8kb47Lw
        auth-client 6MuN+1vob4w Wi1ktyiaiOaG6PaLbneydQ OEigSskYEZb8hGqWZMDKlg
        auth-client uOoXx8epem4 /VjgpZzC71Gj6WAXuVXzVw Q4dOW6GYc/JoQrKNLUQmbQ
        auth-client aC9d6RbIr1M FCY56VPOLEktEsQiCDrORQ 0fAhfjriUxnFvbjiTfGDrw
        auth-client 60vj3crxSwg XSytZMniMFUCDYbhyrdhDg 3P5Q5QaI70AvX/d8Wh7Etw
        encrypted
        -----BEGIN MESSAGE-----
        EqEXDglNZxN+TmE6I7U7fd9DM0ue9ys770n6AnH4ga+fLSJ8AOb0lX4XANG9TuFh
        6hyOfGa0joPezoQSeAFf+yk=
        -----END MESSAGE-----
      f5c8c96c
    • Damian Johnson's avatar
      Generate desc-auth-ephemeral-key from a key · 2526db23
      Damian Johnson authored
      Great catch from asn on #31823 that we should generate desc-auth-ephemeral-key
      fields from a key rather than random bytes. Otherwise this can be used as a
      fingerprint to differentiate our descriptors from tor's.
      2526db23
    • Damian Johnson's avatar
      Faster ed25519 blinding · a1d5d972
      Damian Johnson authored
      Thanks to Paul ed25519 blinding is now fully two orders of magnitude faster!
      
        https://github.com/pyca/cryptography/issues/5068
      
      Replaced slow_ed25519.py with an optimized implementation from...
      
        https://github.com/pyca/ed25519/
      
      This changes the runtime of test_blinding as follows:
      
        Python 2.7: 2.25s => 20 ms
        Python 3.5: 1.83s => 19 ms
      a1d5d972
  14. Nov 20, 2019
    • Damian Johnson's avatar
      Test and fix key blinding with python 3.x · 36a3ca24
      Damian Johnson authored
      Each key blinding takes a couple seconds so I avoided it in our unit tests, but
      we should perform one instance for coverage. Testing with a static key and
      fixing the python 3.x normalization issue this surfaced.
      36a3ca24
  15. Nov 19, 2019
    • Damian Johnson's avatar
      Fix test regression when ed25519 support missing · 68067949
      Damian Johnson authored
      Oops, accidently commited a local hack I made while troubleshooting my openssl
      bindings. This in turn broke our jenkins test runs...
      
        ======================================================================
        ERROR: test_inner_layer_creation
        ----------------------------------------------------------------------
        Traceback (most recent call last):
          File "/srv/jenkins-workspace/workspace/stem-tor-ci/test/unit/descriptor/hidden_service_v3.py", line 343, in test_inner_layer_creation
            IntroductionPointV3.create('1.1.1.1', 9001),
          File "/srv/jenkins-workspace/workspace/stem-tor-ci/stem/descriptor/hidden_service.py", line 230, in create
            raise ImportError('Introduction point creation requires the cryptography module ed25519 support')
        ImportError: Introduction point creation requires the cryptography module ed25519 support
      
        ----------------------------------------------------------------------
      68067949
    • Damian Johnson's avatar
      HSv3 descriptor creation fixes · eccbe01a
      Damian Johnson authored
      Correcting the issues spotted by asn at...
      
        https://trac.torproject.org/projects/tor/ticket/31823#comment:11
      eccbe01a
    • Damian Johnson's avatar
      Ensure onion_key_raw and enc_key_raw are unicode · 35ebded0
      Damian Johnson authored
      Oops, another nice catch from asn. base64.b64encode() return bytes, which will
      then include an extra b'' in our encode() method.
      35ebded0
    • Damian Johnson's avatar
      Always create introduction points with onion-key · b8ac03a2
      Damian Johnson authored
      We didn't provide a mechanism to exclude an onion-key (create() filled it in
      with a default), but it was definitely a mistake to include this conditional.
      
      Caught by asn on https://trac.torproject.org/projects/tor/ticket/31823
      b8ac03a2
    • Damian Johnson's avatar
      Python3 HSv3 descriptor creation fixes · 440cb9b4
      Damian Johnson authored
      My python 3.5 interpreter lacked ed25519 openssl bindings, preventing me from
      exercising these code paths. Now that they're working addressing normalization
      we need.
      440cb9b4
  16. Nov 18, 2019
    • Damian Johnson's avatar
      Cite cryptography blinding ticket · f82040d2
      Damian Johnson authored
      Thanks to asn we now have an upstream ticket for the ed25519 blinding support
      we need.
      f82040d2
    • Damian Johnson's avatar
      Drop flaky test_query_with_timeout assertion · 54f87d30
      Damian Johnson authored
      Jenkins has long struggled with this assertion, and I see these failures
      locally from time to time too...
      
        ======================================================================
        FAIL: test_query_with_timeout
        ----------------------------------------------------------------------
        Traceback (most recent call last):
          File "/usr/local/lib/python2.7/dist-packages/mock/mock.py", line 1305, in patched
            return func(*args, **keywargs)
          File "/home/atagar/Desktop/stem/test/unit/descriptor/remote.py", line 387, in test_query_with_timeout
            self.assertEqual(2, dirport_mock.call_count)
        AssertionError: 2 != 3
      
        ----------------------------------------------------------------------
      
      No assertion is better than a flaky one, so dropping this last bit.
      54f87d30
    • Damian Johnson's avatar
      Fix x25519_supported AttributeError · 95531abf
      Damian Johnson authored
      Oops, turns out the cryptography module doesn't always supply this method...
      
        Traceback (most recent call last):
          File "./run_tests.py", line 36, in <module>
            import test.runner
          File "/srv/jenkins-workspace/workspace/stem-tor-ci/test/runner.py", line 44, in <module>
            import stem.connection
          File "/srv/jenkins-workspace/workspace/stem-tor-ci/stem/connection.py", line 136, in <module>
            import stem.control
          File "/srv/jenkins-workspace/workspace/stem-tor-ci/stem/control.py", line 271, in <module>
            import stem.descriptor.microdescriptor
          File "/srv/jenkins-workspace/workspace/stem-tor-ci/stem/descriptor/__init__.py", line 1544, in <module>
            import stem.descriptor.hidden_service
          File "/srv/jenkins-workspace/workspace/stem-tor-ci/stem/descriptor/hidden_service.py", line 81, in <module>
            X25519_AVAILABLE = backend.x25519_supported()
        AttributeError: 'Backend' object has no attribute 'x25519_supported'
      95531abf
  17. Nov 17, 2019
    • Damian Johnson's avatar
      HSv3 descriptor creation support · 91973b00
      Damian Johnson authored
      Cryptographically valid support for hidden service creation...
      
        https://trac.torproject.org/projects/tor/ticket/31823
      
      HSv3 descriptors consist of three parts: an inner layer, outer layer, and the
      descriptor itself. Callers of HiddenServiceDescriptorV3's create() and
      content() methods can supply these to specify that layer's parameters.
      
      For example, to supply custom introduction points with random key material
      simply call...
      
        HiddenServiceDescriptorV3.content(
          inner_layer = InnerLayer.create(
            introduction_points = [
              IntroductionPointV3.create('1.1.1.1', 9001),
              IntroductionPointV3.create('2.2.2.2', 9001),
              IntroductionPointV3.create('3.3.3.3', 9001),
            ],
          ),
        )
      91973b00
    • Damian Johnson's avatar
      Disable key blinding by default · 860afdb8
      Damian Johnson authored
      On reflection, why not simply disable blinding by default? Blinding parameters
      are unnecessary for the vast majority of use cases, and when needed the caller
      can simply provide a nonce (and by extension accept the lengthy runtime).
      
      We can always re-enable blinding as a default if/when we have a performant
      implementation.
      860afdb8
    • Damian Johnson's avatar
      Resume hidden service tests · 13e166b5
      Damian Johnson authored
      Resuming our HSv3 tests, with blinding mocked out so we don't negatively impact
      runtime (without these mocks the unit tests jump from 6s => 51s).
      13e166b5
    • Damian Johnson's avatar
      Simplify HSv3 blinding · 838f6209
      Damian Johnson authored
      I won't pretend to understand this math. A smarter mind than mine (asn's) came
      up with this crypto. Just massaging it into a form I find easier to understand.
      838f6209
Loading