Trac issueshttps://gitlab.torproject.org/legacy/trac/-/issues2022-03-03T15:23:50Zhttps://gitlab.torproject.org/legacy/trac/-/issues/28667Obtain the new descriptors and network status documents by signals instead of...2022-03-03T15:23:50ZjugaObtain the new descriptors and network status documents by signals instead of every 5minThe network status and descriptors are obtained every 5min https://gitweb.torproject.org/sbws.git/tree/sbws/lib/relaylist.py#n144.
Is there any reason for that?.
I think torflow obtain them as they come.
It could be done with stem using:...The network status and descriptors are obtained every 5min https://gitweb.torproject.org/sbws.git/tree/sbws/lib/relaylist.py#n144.
Is there any reason for that?.
I think torflow obtain them as they come.
It could be done with stem using:
```
controller.add_event_listener(
listener_fn, EventType.NS, EventType.NEWDESC)
```
Also, instead overwriting those values, Relay could have a dictionary (key would be the timestamp) with a subset of the attributes of document, in case we might need to calculate median/mean of those attributes
Edit: fix typosbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28905Please install packages for twisted gettor2020-06-13T16:56:14ZIsrael LeivaPlease install packages for twisted gettorA code refactor has been made to gettor using twisted (see #28152). The following packages are needed for it to work:
gcc python2.7 python-dev virtualenv sqlite3 python-pip
Thanks.A code refactor has been made to gettor using twisted (see #28152). The following packages are needed for it to work:
gcc python2.7 python-dev virtualenv sqlite3 python-pip
Thanks.https://gitlab.torproject.org/legacy/trac/-/issues/29726Rename constants, variables, classes, methods, functions2020-06-13T16:15:40ZjugaRename constants, variables, classes, methods, functionsSome ideas are in https://pad.riseup.net/p/rGfvR7ZsvtoZ.
It would be nice to create first a list of all the things to rename and the proposed new names.
It should be done file for file as possible and be reviewed/merged fast to avoid m...Some ideas are in https://pad.riseup.net/p/rGfvR7ZsvtoZ.
It would be nice to create first a list of all the things to rename and the proposed new names.
It should be done file for file as possible and be reviewed/merged fast to avoid many merge conflicts.
Only a part of this was https://github.com/torproject/sbws/issues/202.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/29721Refactor RelayList2020-06-13T16:15:38ZjugaRefactor RelayListRefactor 1:
It should probably have a dictionary of relays instead of a list, so that it is not needed to convert the list to dictionary in resultdump.
It should be then named `Relays` or `RelayDict`
In https://github.com/torproject/sbws...Refactor 1:
It should probably have a dictionary of relays instead of a list, so that it is not needed to convert the list to dictionary in resultdump.
It should be then named `Relays` or `RelayDict`
In https://github.com/torproject/sbws/issues/219 it was proposed to change the list to a set, but that's only used to exclude authorities in the prioritizer, which should also be a method in RelayList.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/29718Include a refactor plan2020-06-13T16:15:36ZjugaInclude a refactor planThe parent ticket have some children tickets and we collected some ideas in https://pad.riseup.net/p/rGfvR7ZsvtoZ, but there are other ideas not collected.The parent ticket have some children tickets and we collected some ideas in https://pad.riseup.net/p/rGfvR7ZsvtoZ, but there are other ideas not collected.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/29717Refactor Relay and RelayList to be able to initialize them without Tor's cont...2020-06-13T16:15:36ZjugaRefactor Relay and RelayList to be able to initialize them without Tor's controllerIt would be easier to create unit tests without the need of having a stem's Controller instance, but only providing consensus and relay descriptor documents.It would be easier to create unit tests without the need of having a stem's Controller instance, but only providing consensus and relay descriptor documents.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/29716Change the way the consensus is received2020-06-13T16:15:35ZjugaChange the way the consensus is receivedCurrently it receives the new consensuses using https://stem.torproject.org/api/control.html#stem.control.Controller.get_network_statuses, which uses the cached-consensus and might not be updated.
It should probably receive the new conse...Currently it receives the new consensuses using https://stem.torproject.org/api/control.html#stem.control.Controller.get_network_statuses, which uses the cached-consensus and might not be updated.
It should probably receive the new consensuses as soon as they arrive using an event listener https://stem.torproject.org/api/control.html#stem.control.Controller.add_event_listener, instead of every x time.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/29057Adapt bandwidth file classes to be compatible with stem (descriptors, etc) do...2020-06-13T16:15:21ZjugaAdapt bandwidth file classes to be compatible with stem (descriptors, etc) documentsso that the code can be moved to stem, as commented in #29056so that the code can be moved to stem, as commented in #29056sbws: unspecifiedjugajugahttps://gitlab.torproject.org/legacy/trac/-/issues/29048Remove unused code2020-06-13T16:15:20ZjugaRemove unused codesbws: unspecifiedjugajugahttps://gitlab.torproject.org/legacy/trac/-/issues/29047Improve code style following PEP8 and PEP2572020-06-13T16:15:19ZjugaImprove code style following PEP8 and PEP257sbws: unspecifiedjugajugahttps://gitlab.torproject.org/legacy/trac/-/issues/28737Redesign sbws torrc option configuration2020-06-13T16:15:03ZteorRedesign sbws torrc option configurationsbws' torrc option handling is broken.
Here's a better scheme:
1. sbws config.default.ini and config.ini contain the following sections:
* tor.options.launch: a minimal set of options that must be configured when tor is launched. The ...sbws' torrc option handling is broken.
Here's a better scheme:
1. sbws config.default.ini and config.ini contain the following sections:
* tor.options.launch: a minimal set of options that must be configured when tor is launched. The minimal set contains the data directory config, control config, and log config. The network is disabled at launch. The initial options are:
```
'DataDirectory': conf.getpath('tor', 'datadir'),
'PidFile': conf.getpath('tor', 'pid'),
# Because we need things from full server descriptors (namely for now: the
# bandwidth line)
'UseMicrodescriptors': '0',
'ControlSocket': conf.getpath('tor', 'control_socket'),
# Easier than password authentication
'CookieAuthentication': '1',
'Log': [
'NOTICE file {}'.format(os.path.join(conf.getpath('tor', 'log'),
'notice.log')),
],
# useful logging options for clients that don't care about anonymity
'SafeLogging': '0',
'LogTimeGranularity': '1',
'ProtocolWarnings': '1',
'DisableNetwork': '1',
```
* tor.options.runtime.(group_name): Zero or more groups of tor options that can be set at runtime. The network is enabled at runtime. The "50-default" group of options is:
```
# We will find out via the ControlPort and not setting something static
# means a lower chance of conflict
'SocksPort': 'auto',
# To avoid path bias warnings
'UseEntryGuards': '0',
c.set_conf('__LeaveStreamsUnattached', '1')
# Things needed to make circuits fail a little faster. We get the
# circuit_timeout as a string instead of an int on purpose: stem only
# accepts strings.
'LearnCircuitBuildTimeout': '0',
'CircuitBuildTimeout': conf['general']['circuit_timeout'],
'DisableNetwork': '0',
```
* Zero or more tor.runtime.ignore_failure.(group name): tor options that are set in groups at runtime, but ignored if they fail. #28692 and #28694 will add options to this list.
Options in config.ini override options with the same name in sbws.default.ini, with + and / having the same meaning as in a torrc file (+ appends, / removes). Launch options are applied first, then runtime options are applied in group name order. Tor will make later runtime options replace earlier runtime options, and launch options.
2. sbws gets its control socket from the launch_options ControlSocket option(s)
3. sbws gets its data directory, pid, log(s), and circuit build timeout using GETCONF
4. For backwards compatibility:
* if tor.extra_lines is present, it should be appended to tor.options.launch. sbws' option merging code never worked, so appending shouldn't cause any more issues than the existing code.
* if any of these sbws options are present in an old config file, synthesise options.launch from the sbws options. If options.launch is also present in the same config file, it overrides the synthetic options. (sbws' option merging never worked for these options.)
```
'DataDirectory': conf.getpath('tor', 'datadir'),
'PidFile': conf.getpath('tor', 'pid'),
'ControlSocket': conf.getpath('tor', 'control_socket'),
'Log': [
'NOTICE file {}'.format(os.path.join(conf.getpath('tor', 'log'),
'notice.log')),
],
# Things needed to make circuits fail a little faster. We get the
# circuit_timeout as a string instead of an int on purpose: stem only
# accepts strings.
'CircuitBuildTimeout': conf['general']['circuit_timeout'],
```
The final option order is:
* sbws creates options.launch, using the last option from:
* synthetic legacy config options from config.default.ini
* options.launch from config.default.ini
* synthetic legacy config options from config.ini
* options.launch from config.ini
* sbws appends extra_lines to options.launch, using the last extra_lines from:
* config.default.ini
* config.ini
* sbws creates options.runtime groups, using the last one of each group from:
* config.default.ini
* config.ini
sbws launches tor with options.launch, then applies each group of options.runtime in order.
This is a new feature, so it should go in sbws 1.1.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28718Simplify configuration options2020-06-13T16:15:01ZjugaSimplify configuration optionsPrograms should allow in this order: cli options, user configuration files passed by cli, environment variables, system configuration files and program defaults.
There seems there's not a common easy way to achieve this in Python.
sbws a...Programs should allow in this order: cli options, user configuration files passed by cli, environment variables, system configuration files and program defaults.
There seems there's not a common easy way to achieve this in Python.
sbws accepts cli and user configuration files in a complicated way.
The program defaults are provided as a configuration file which might not be that convenient seems confuses where to find the defaults.
Maybe an external library like https://github.com/bw2/ConfigArgParse (supported in Debian) or some other design can help to simplify this.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/28282Refactor bandwidth file generation code2020-06-13T16:14:19ZjugaRefactor bandwidth file generation codeInitially, `v3bwfile.py` was just converting measurement `Result`s to a `Bandwidth file` lines format.
Then we started to apply scaling methods and filtering the measurements to satisfy certain restrictions.
To don not have to modify cod...Initially, `v3bwfile.py` was just converting measurement `Result`s to a `Bandwidth file` lines format.
Then we started to apply scaling methods and filtering the measurements to satisfy certain restrictions.
To don not have to modify code in other file/classes, all of that was implemented in `v3bwfile.py` classes/methods, but it should be moved to either an intermediate file/classes that deal with the statistics calculations (and not the format) or modify `Results` to be able to perform those calculations.
Edit: fix typosbws: unspecifiedjugajugahttps://gitlab.torproject.org/legacy/trac/-/issues/27362(sub-)packages outside of core (cli) should not need to know about confs and ...2020-06-13T16:13:51Zjuga(sub-)packages outside of core (cli) should not need to know about confs and argsThat would help to have a more modular design and use other (sub)packages and (sub)modules without the need of creating ConfigParser and ArgumentParser objects.
It also would help to simplify tests configurations.
Additionally, a progra...That would help to have a more modular design and use other (sub)packages and (sub)modules without the need of creating ConfigParser and ArgumentParser objects.
It also would help to simplify tests configurations.
Additionally, a program should take into account in this order:
- cli arguments
- environment variables
- user configuration files
- system configuration files
- program defaults
That is currently almost match. but it would be better if they all can be took into account in a simpler way.
This is not for MVP, but creating the ticket cause i'm creating new code taking this into account, and would be nice to change at some point.
Some tickets, as #27358, happen because of this.sbws: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/26502Stop using the fallback blacklist, and delete it2020-06-13T16:06:29ZteorStop using the fallback blacklist, and delete itWe require relay operators to opt-in to being fallbacks, and we won't ever switch to opt-out. (See #24789.)
So we don't need the fallback blacklist any more.
We should stop loading the blacklist in the script, then delete it.We require relay operators to opt-in to being fallbacks, and we won't ever switch to opt-out. (See #24789.)
So we don't need the fallback blacklist any more.
We should stop loading the blacklist in the script, then delete it.Tor: 0.3.5.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/33956Define and use TOR_ADDRPORT_BUF_LEN2020-06-13T15:53:13ZteorDefine and use TOR_ADDRPORT_BUF_LENIn #33918, we discovered a bug where IPv6 addresses were being truncated in logs.
During the fix, we noticed that we had a TOR_ADDR_BUF_LEN, but no equivalent constant for addresses and ports. The new TOR_ADDRPORT_BUF_LEN should allow s...In #33918, we discovered a bug where IPv6 addresses were being truncated in logs.
During the fix, we noticed that we had a TOR_ADDR_BUF_LEN, but no equivalent constant for addresses and ports. The new TOR_ADDRPORT_BUF_LEN should allow space for:
* TOR_ADDR_BUF_LEN
* IPv6 brackets (2, if not included in TOR_ADDR_BUF_LEN already)
* the port separator (1)
* the port (5)
We should check for other truncation errors while making this change.Tor: 0.4.4.x-finalNeel Chauhanneel@neelc.orgNeel Chauhanneel@neelc.orghttps://gitlab.torproject.org/legacy/trac/-/issues/33739Authority-mode keys and certificates should be owned by dirauth module2020-06-13T15:52:35ZNick MathewsonAuthority-mode keys and certificates should be owned by dirauth moduleRight now these functions are defined in the relay module, when only dirauths need them:
* get_my_v3_authority_cert
* get_my_v3_authority_signing_key
* get_my_v3_legacy_cert
* get_my_v3_legacy_signing_key
We should move them alo...Right now these functions are defined in the relay module, when only dirauths need them:
* get_my_v3_authority_cert
* get_my_v3_authority_signing_key
* get_my_v3_legacy_cert
* get_my_v3_legacy_signing_key
We should move them along with related fields and support functions, to feature/dirauth.Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/32887Remove use of NS() macros to make our code more indentable?2020-06-13T15:49:46ZNick MathewsonRemove use of NS() macros to make our code more indentable?clang-format doesn't do very well with our NS() macros defined in test.h. And we don't actually use those macros very much, and we haven't used them for new tests in a while. Shall we remove them?clang-format doesn't do very well with our NS() macros defined in test.h. And we don't actually use those macros very much, and we haven't used them for new tests in a while. Shall we remove them?Tor: 0.4.3.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/legacy/trac/-/issues/31074Use tor_queue.h macros in config_line_t2020-06-13T15:43:20ZNick MathewsonUse tor_queue.h macros in config_line_tThe config_line_t linked list could be refactored to use the TOR_SLIST macros in tor_queue.hThe config_line_t linked list could be refactored to use the TOR_SLIST macros in tor_queue.hTor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/30968Refactor unit test asserts so they log context2020-06-13T15:42:57ZteorRefactor unit test asserts so they log contextIn #30721, I created some complex macros to preserve line numbers when the unit tests fail.
We could refactor these macros into functions, if the tiny test assertions supported context.
catalyst suggests allowing file and line numbers ...In #30721, I created some complex macros to preserve line numbers when the unit tests fail.
We could refactor these macros into functions, if the tiny test assertions supported context.
catalyst suggests allowing file and line numbers (and functions?):
> I think I see that the problem is buried in the `TT_DECLARE()` macro in `tinytest_macros.h`. I think it's possible to work around it, but it might be nontrivial. (Rough sketch: redefine `TT_DECLARE()` in helper functions to read file and line info from function parameters, and make file and line numbers explicit parameters to helper functions.)
I suggest allowing a format string, which could also print the data that we're testing. (For the tor_addr_port_lookup() tests, the best context is the address string.)Tor: unspecified