Skip to content
Snippets Groups Projects
  1. Jun 01, 2023
  2. May 30, 2023
  3. Oct 06, 2022
  4. Oct 03, 2022
  5. Sep 27, 2022
  6. Sep 26, 2022
    • juga's avatar
      Add .gitlab-ci.yml · cea85f60
      juga authored
      to be able to check tests at gitlab.tpo, instead of updating travis
      configuration.
      cea85f60
    • NilacTheGrim's avatar
      Fixup for Python 3.10 · af71483f
      NilacTheGrim authored and juga's avatar juga committed
      Closes issue #109.  Long story short: a few names from collection are
      now moved to collection.abc exclusively starting in Python 3.10. The
      only name this app uses from there that was moved is
      `collections.Iterable`.  Python versions starting from 3.3 support both
      `collections.Iterable` and `collections.abc.Iterable` as the way to refer to
      this class, which Python 3.10 being the first one to drop
      `collections.Iterable`.  So.. we just work around this API quirk
      and always refer ot it as `collections.abc.Iterable`.
      af71483f
    • juga's avatar
      Replace client_auth_v3 arg versionchanged · e599dc01
      juga authored
      from 1.9.0 to 1.8.1, since it was probably an error.
      e599dc01
  7. May 11, 2021
  8. May 06, 2021
  9. May 05, 2021
  10. Dec 29, 2019
  11. Dec 28, 2019
  12. Dec 27, 2019
    • Damian Johnson's avatar
      CollecTor publication time filtering · 6c07fe55
      Damian Johnson authored
      When I first wrote this module I played fast and lose with the 'start' and
      'end' parameters, guessing relevance purely based on filenames.
      
      When Karsten added timestamps to the index he better defined the relevant
      timestamp to be a descriptor's publication, which everything except
      microdescriptors contains.
      
      Interestingly, archives can contain publications both before and after its
      filename date. For example...
      
        recent/relay-descriptors/server-descriptors/2019-12-27-22-04-59-server-descriptors
      
        Old filename derived timestamps:
          start: 2019-12-27 22:04:59
          end: 2019-12-27 23:04:59
      
        Index's publication timpestamps:
          start: 2019-12-27 20:30:00
          end: 2019-12-27 22:45:00
      
      If the file was created at 22:04 how does it contain something published at
      22:45?
      
      Regardless, now that the index contains publication times for our purposes
      filenames dates are moot. Our 'start' and 'end' arguments provide the subset
      of archives that reside within the given publication range.
      
      For example, the following downloads descriptors that were published up to two
      hours ago...
      
        recent = datetime.datetime.utcnow() - datetime.timedelta(minutes = 120)
        descriptors = stem.descriptor.collector.get_server_descriptors(start = recent)
      
      If we make this more sophisticated we can demonstrate how many
      descriptors we pull from each archive...
      
        import datetime
        import stem.descriptor.collector
      
        collector = stem.descriptor.collector.get_instance()
        recent = datetime.datetime.utcnow() - datetime.timedelta(minutes = 120)
      
        # This effectively does the same thing as get_server_descriptors(),
        # but in a way we can also determine the full counts.
      
        for f in collector.files('server-descriptor', start = recent):
          all_desc = list(f.read())
          recent_desc = list(f.read(start = recent))
      
          print('%s (%s => %s)' % (f.path, f.start, f.end))
          print('  %i of %i descriptors were published recently' % (len(recent_desc), len(all_desc)))
      
        # Download them again, but through our more prevalently used
        # get_server_descriptors() method.
      
        print('\nIn total there are %i server descriptors published recently' % len(list(collector.get_server_descriptors(start = recent))))
      
        ----------------------------------------------------------------------
      
        % python demo.py
      
        recent/relay-descriptors/server-descriptors/2019-12-27-21-04-59-server-descriptors (2019-12-27 17:59:00 => 2019-12-27 22:13:00)
          3 of 817 descriptors were published recently
        recent/relay-descriptors/server-descriptors/2019-12-27-22-04-59-server-descriptors (2019-12-27 20:30:00 => 2019-12-27 22:45:00)
          297 of 776 descriptors were published recently
        recent/relay-descriptors/server-descriptors/2019-12-27-23-04-59-server-descriptors (2019-12-27 21:49:00 => 2019-12-27 23:01:00)
          800 of 800 descriptors were published recently
      
        In total there are 1100 server descriptors published recently
      6c07fe55
    • Damian Johnson's avatar
      Fix collector integ test · bbd6b100
      Damian Johnson authored
      Minor fix for three online tests...
      
        ======================================================================
        FAIL: test_index_plaintext
        ----------------------------------------------------------------------
        Traceback (most recent call last):
          File "/home/atagar/Desktop/stem/test/require.py", line 43, in wrapped
            return func(self, *args, **kwargs)
          File "/home/atagar/Desktop/stem/test/require.py", line 58, in wrapped
            return func(self, *args, **kwargs)
          File "/home/atagar/Desktop/stem/test/integ/descriptor/collector.py", line 22, in test_index_plaintext
            self._test_index(None)
          File "/home/atagar/Desktop/stem/test/integ/descriptor/collector.py", line 98, in _test_index
            self.assertEqual(['archive', 'contrib', 'recent'], [entry['path'] for entry in index['directories']])
        AssertionError: Lists differ: ['archive', 'contrib', 'recent... != [u'archive', u'recent']
      
        First differing element 1:
        'contrib'
        u'recent'
      
        First list contains 1 additional elements.
        First extra element 2:
        'recent'
      
        - ['archive', 'contrib', 'recent']
        ?             ^^^^^^^^^^^
      
        + [u'archive', u'recent']
        ?  +           ^
      bbd6b100
Loading