The Tor Project issueshttps://gitlab.torproject.org/groups/tpo/-/issues2020-06-27T13:50:05Zhttps://gitlab.torproject.org/tpo/core/tor/-/issues/30769Rename one of the two sendme.h files2020-06-27T13:50:05ZNick MathewsonRename one of the two sendme.h filesWe have a convention that, to avoid confusion, two directories should never have headers with the same name. Our `rectify_include_paths.py` tool takes advantage of this fact to find a canonical path for each header. But in 0.4.1, we no...We have a convention that, to avoid confusion, two directories should never have headers with the same name. Our `rectify_include_paths.py` tool takes advantage of this fact to find a canonical path for each header. But in 0.4.1, we now have two conflicting headers:
```
./src/core/or/sendme.h
./src/trunnel/sendme.h
```
Let's rename one of these.Tor: 0.4.1.x-finalDavid Gouletdgoulet@torproject.orgDavid Gouletdgoulet@torproject.orghttps://gitlab.torproject.org/tpo/core/tor/-/issues/30759Create (or edit) the wiki page for the CI role2020-07-29T04:07:31ZteorCreate (or edit) the wiki page for the CI roleHere is part of an email I sent about the CI role.
I want to turn it into a proposed process, by editing the CI wiki page.
One of the failure modes of this role is that the CI people end up fixing
a lot of failing tests.
But it's be...Here is part of an email I sent about the CI role.
I want to turn it into a proposed process, by editing the CI wiki page.
One of the failure modes of this role is that the CI people end up fixing
a lot of failing tests.
But it's best practice for the original developer to fix the tests that they
wrote: it's more efficient, and people learn from their mistakes.
So I'd like to restrict the scope of this role to "make CI pass, quickly".
Usually that means:
* reverting a failing commit,
* marking a failing job as "allow failures", or
* skipping a failing test.
And then logging a bug for a longer-term fix.
We need a separate process to make sure longer-term fixes happen.
We typically have 3 categories of CI bugs:
* consistent failures from a recent commit,
* intermittent failures, which can be from old commits,
* environmental failures from CI infrastructure changes.
We can assign recent failures to the person who wrote the code.
(Or a paid staff member, if that person is an occasional volunteer.)
I think the CI people should assign the other two categories of
bugs evenly across the team. It's too much for one or two people
to fix all the CI bugs.
If we use this scope, the CI role is similar to the review assigner,
backport decider/merger, and bug triage roles. It's not our job
to fix the bugs, just to triage them, and get CI into a usable state.Tor: unspecifiedhttps://gitlab.torproject.org/tpo/core/tor/-/issues/30758Add stem ONLINE tests to tor's CI2020-07-29T04:07:57ZteorAdd stem ONLINE tests to tor's CIOnce the bootstrapping issues in legacy/trac#30746 are fixed, we should think about adding stem's online tests to tor's CI,Once the bootstrapping issues in legacy/trac#30746 are fixed, we should think about adding stem's online tests to tor's CI,Tor: unspecifiedhttps://gitlab.torproject.org/tpo/core/tor/-/issues/30752Practracker: usability improvements from May retrospective2020-06-27T13:50:05ZNick MathewsonPractracker: usability improvements from May retrospectiveWe talked about a few possible improvements to practracker in our May retrospective a couple of weeks back. Here are the suggestions we came up with.
1. Practracker should have tolerances in its exceptions. For example, if an exceptio...We talked about a few possible improvements to practracker in our May retrospective a couple of weeks back. Here are the suggestions we came up with.
1. Practracker should have tolerances in its exceptions. For example, if an exception allows a function to be 150 lines long, then we should not cause an error until the function is (say) 155 lines long. If the function is 151-154 lines long, that should just be a warning.
These tolerances could be percentage-based, or just flat numbers.
There should be an option that makes all warnings fatal.
2. There should be an option to list unused exceptions, and exceptions that are larger than they need to be.Tor: 0.4.2.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/tpo/core/tor/-/issues/30746Tor relay bootstrap hangs: missing descriptors for 1/2 of our primary entry g...2022-06-24T16:02:35ZDamian JohnsonTor relay bootstrap hangs: missing descriptors for 1/2 of our primary entry guardsHi network team. Stem's integ tests impose a 90 second timeout when bootstrapping. The vast majority of time it succeeds without issue, but sometimes bootstrapping hangs, exceeding this.
I have debug level logs available but it's pretty...Hi network team. Stem's integ tests impose a 90 second timeout when bootstrapping. The vast majority of time it succeeds without issue, but sometimes bootstrapping hangs, exceeding this.
I have debug level logs available but it's pretty big so attaching info level logs instead. If debug logging would be useful just let me know.
```
% du -h ~/Desktop/tor_hang_debug_log.txt
13M /home/atagar/Desktop/tor_hang_debug_log.txt
```
Notice runlevel logs are as follows...
```
Jun 03 18:10:04.000 [notice] Tor 0.4.1.1-alpha-dev (git-180048e013c06ee6) opening new log file.
Jun 03 18:10:04.131 [notice] Tor 0.4.1.1-alpha-dev (git-180048e013c06ee6) running on Linux with Libevent 2.0.21-stable, OpenSSL 1.0.2g, Zlib 1.2.8, Liblzma N/A, and Libzstd N/A.
Jun 03 18:10:04.132 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Jun 03 18:10:04.132 [notice] This version is not a stable Tor release. Expect more bugs than usual.
Jun 03 18:10:04.132 [notice] Read configuration file "/home/atagar/Desktop/stem/test/data/torrc".
Jun 03 18:10:04.136 [notice] Your ContactInfo config option is not set. Please consider setting it, so we can contact you if your server is misconfigured or something else goes wrong.
Jun 03 18:10:04.137 [notice] Based on detected system memory, MaxMemInQueues is set to 5917 MB. You can override this by setting MaxMemInQueues by hand.
Jun 03 18:10:04.137 [warn] ControlPort is open, but no authentication method has been configured. This means that any program on your computer can reconfigure your Tor. That's bad! You should upgrade your Tor controller as soon as possible.
Jun 03 18:10:04.138 [notice] Opening Socks listener on 127.0.0.1:1112
Jun 03 18:10:04.138 [notice] Opened Socks listener on 127.0.0.1:1112
Jun 03 18:10:04.138 [notice] Opening Control listener on 127.0.0.1:1111
Jun 03 18:10:04.139 [notice] Opened Control listener on 127.0.0.1:1111
Jun 03 18:10:04.139 [notice] Opening OR listener on 0.0.0.0:1113
Jun 03 18:10:04.139 [notice] Opened OR listener on 0.0.0.0:1113
Jun 03 18:10:04.139 [warn] Fixing permissions on directory /home/atagar/Desktop/stem/test/data
Jun 03 18:10:04.000 [warn] Your log may contain sensitive information - you're logging more than "notice". Don't log unless it serves an important reason. Overwrite the log afterwards.
Jun 03 18:10:04.000 [notice] Configured to measure directory request statistics, but no GeoIP database found. Please specify a GeoIP database using the GeoIPFile option.
Jun 03 18:10:04.000 [notice] You are running a new relay. Thanks for helping the Tor network! If you wish to know what will happen in the upcoming weeks regarding its usage, have a look at https://blog.torproject.org/blog/lifecycle-of-a-new-relay
Jun 03 18:10:04.000 [notice] It looks like I need to generate and sign a new medium-term signing key, because I don't have one. To do that, I need to load (or create) the permanent master identity key. If the master identity key was not moved or encrypted with a passphrase, this will be done automatically and no further action is required. Otherwise, provide the necessary data using 'tor --keygen' to do it manually.
Jun 03 18:10:04.000 [notice] Your Tor server's identity key fingerprint is 'Unnamed D65E957D19700A27C67D89C314DE822B0CA6CD7D'
Jun 03 18:10:04.000 [notice] Bootstrapped 0% (starting): Starting
Jun 03 18:10:04.000 [notice] Starting with guard context "default"
Jun 03 18:10:04.000 [notice] Guessed our IP address as 71.231.87.208 (source: 86.59.21.38).
Jun 03 18:10:06.000 [notice] I learned some more directory information, but not enough to build a circuit: We have no usable consensus.
Jun 03 18:10:07.000 [notice] The current consensus has no exit nodes. Tor can only build internal paths, such as paths to onion services.
Jun 03 18:10:07.000 [notice] I learned some more directory information, but not enough to build a circuit: We need more microdescriptors: we have 0/6563, and can only build 0% of likely paths. (We have 0% of guards bw, 0% of midpoint bw, and 0% of end bw (no exits in consensus, using mid) = 0% of path bw.)
Jun 03 18:10:07.000 [notice] Bootstrapped 5% (conn): Connecting to a relay
Jun 03 18:10:07.000 [notice] Bootstrapped 10% (conn_done): Connected to a relay
Jun 03 18:10:07.000 [notice] The current consensus contains exit nodes. Tor can build exit and internal paths.
Jun 03 18:10:08.000 [notice] Bootstrapped 14% (handshake): Handshaking with a relay
Jun 03 18:10:08.000 [notice] Bootstrapped 15% (handshake_done): Handshake with a relay done
Jun 03 18:10:08.000 [notice] Bootstrapped 50% (loading_descriptors): Loading relay descriptors
Jun 03 18:10:09.000 [notice] Bootstrapped 55% (loading_descriptors): Loading relay descriptors
Jun 03 18:10:09.000 [notice] Bootstrapped 60% (loading_descriptors): Loading relay descriptors
Jun 03 18:10:09.000 [notice] Bootstrapped 65% (loading_descriptors): Loading relay descriptors
Jun 03 18:10:09.000 [notice] Bootstrapped 70% (loading_descriptors): Loading relay descriptors
Jun 03 18:10:10.000 [warn] Please upgrade! This version of Tor (0.4.1.1-alpha-dev) is not recommended, according to the directory authorities. Recommended versions are: 0.2.9.15,0.2.9.16,0.2.9.17,0.3.4.10,0.3.4.11,0.3.5.7,0.3.5.8,0.4.0.1-alpha,0.4.0.2-alpha,0.4.0.3-alpha,0.4.0.4-rc,0.4.0.5,0.4.0.6,0.4.1.1-alpha,0.4.1.2-alpha
Jun 03 18:10:10.000 [notice] I learned some more directory information, but not enough to build a circuit: We're missing descriptors for 1/2 of our primary entry guards (total microdescriptors: 5662/6563).
[ hangs for 76 more seconds, then stem times out bootstrapping ]
```
Debug log ends with...
```
% tail ~/Desktop/tor_hang_debug_log.txt
Jun 03 18:11:32.000 [debug] conn_read_callback(): socket 116 wants to read.
Jun 03 18:11:32.000 [debug] connection_buf_read_from_socket(): 116: starting, inbuf_datalen 0 (0 pending in tls object). at_most 16448.
Jun 03 18:11:32.000 [debug] connection_buf_read_from_socket(): After TLS read of 514: 543 read, 0 written
Jun 03 18:11:32.000 [debug] connection_or_process_cells_from_inbuf(): 116: starting, inbuf_datalen 514 (0 pending in tls object).
Jun 03 18:11:32.000 [debug] connection_or_process_cells_from_inbuf(): 116: starting, inbuf_datalen 0 (0 pending in tls object).
Jun 03 18:11:33.000 [debug] conn_read_callback(): socket 129 wants to read.
Jun 03 18:11:33.000 [debug] connection_buf_read_from_socket(): 129: starting, inbuf_datalen 0 (0 pending in tls object). at_most 16448.
Jun 03 18:11:33.000 [debug] connection_buf_read_from_socket(): After TLS read of 514: 543 read, 0 written
Jun 03 18:11:33.000 [debug] connection_or_process_cells_from_inbuf(): 129: starting, inbuf_datalen 514 (0 pending in tls object).
Jun 03 18:11:33.000 [debug] connection_or_process_cells_from_inbuf(): 129: starting, inbuf_datalen 0 (0 pending in tls object).
```https://gitlab.torproject.org/tpo/core/tor/-/issues/30745Document disabled CI2021-07-22T16:19:44ZteorDocument disabled CIAdd a section to our CI status page for disabled CI
Add a section to doc/HACKING/ReleasingTor.md reminding releasers to
manually check the status of whatever the disabled CI would have
checked.
CI person should periodically look at the...Add a section to our CI status page for disabled CI
Add a section to doc/HACKING/ReleasingTor.md reminding releasers to
manually check the status of whatever the disabled CI would have
checked.
CI person should periodically look at these jobs.Tor: 0.4.3.x-finalhttps://gitlab.torproject.org/tpo/core/tor/-/issues/30744Allow failures in the Travis test-stem job2020-06-27T13:50:06ZteorAllow failures in the Travis test-stem jobTor: 0.3.5.x-finalteorteorhttps://gitlab.torproject.org/tpo/core/tor/-/issues/30743Write a coccinelle script to catch increment/decrement calls inside log_debug().2020-06-27T13:50:06ZNick MathewsonWrite a coccinelle script to catch increment/decrement calls inside log_debug().See legacy/trac#30628 for motivationSee legacy/trac#30628 for motivationTor: 0.4.2.x-finalNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/tpo/core/tor/-/issues/30742sendme: Better logging overall2020-07-29T04:08:45ZDavid Gouletdgoulet@torproject.orgsendme: Better logging overallThis ticket is for various logging improvements for the SENDME subsystem.This ticket is for various logging improvements for the SENDME subsystem.Tor: unspecifiedhttps://gitlab.torproject.org/tpo/core/tor/-/issues/30737Tor daemon unusable after resuming from suspend on Linux2020-07-29T04:09:12ZTracTor daemon unusable after resuming from suspend on LinuxWhen resuming after having suspended to RAM on Linux (Debian Stretch), the Tor daemon won't build any circuits for minutes. After waiting for some minutes, it works again. Restarting the daemon makes it work immediately again. All newly ...When resuming after having suspended to RAM on Linux (Debian Stretch), the Tor daemon won't build any circuits for minutes. After waiting for some minutes, it works again. Restarting the daemon makes it work immediately again. All newly built circuits fail with "Failed: timeout". Onion Circuits screenshot attached. Also, Tor often switches the bridge relay when this happens, probably because for some reason it cannot reach the bridge.
**Trac**:
**Username**: ParckwartTor: unspecifiedhttps://gitlab.torproject.org/tpo/core/tor/-/issues/30726Missing relay keys in bandwidth file spec2020-06-27T13:50:06ZteorMissing relay keys in bandwidth file specNot in spec or examples:
* relay_in_recent_consensus_count
* consensus_bandwidth
* consensus_bandwidth_is_unmeasured
* desc_bw_avg
* desc_bw_bur
* desc_bw_obs_last
* desc_bw_obs_mean
Not in spec:
* relay_recent_measurement_attempt_count...Not in spec or examples:
* relay_in_recent_consensus_count
* consensus_bandwidth
* consensus_bandwidth_is_unmeasured
* desc_bw_avg
* desc_bw_bur
* desc_bw_obs_last
* desc_bw_obs_mean
Not in spec:
* relay_recent_measurement_attempt_count
* relay_recent_priority_list_count
We need a specification for these keys so we know what they mean.Tor: 0.4.4.x-finalhttps://gitlab.torproject.org/tpo/core/tor/-/issues/30721tor_addr_port_lookup() is overly permissive2021-09-16T14:23:37Zteortor_addr_port_lookup() is overly permissiveLike tor_addr_parse() in legacy/trac#23082, tor_addr_port_lookup() also allows square brackets around IPv4 addresses.
But it's slightly less permissive: it requires a terminating `]`.
For example, this command line should be rejected, ...Like tor_addr_parse() in legacy/trac#23082, tor_addr_port_lookup() also allows square brackets around IPv4 addresses.
But it's slightly less permissive: it requires a terminating `]`.
For example, this command line should be rejected, but it is not:
```
tor ORPort [127.0.0.1]:9001
```Tor: 0.4.2.x-finalteorteorhttps://gitlab.torproject.org/tpo/core/tor/-/issues/30713Disable or allow_fail test_rebind.py in macOS Travis2020-06-27T13:50:07ZTaylor YuDisable or allow_fail test_rebind.py in macOS Travistest_rebind.py is having spurious failures often on macOS. We should either allow_fail the macOS builds, or disable test_rebind.py.test_rebind.py is having spurious failures often on macOS. We should either allow_fail the macOS builds, or disable test_rebind.py.Tor: 0.3.5.x-finalteorteorhttps://gitlab.torproject.org/tpo/core/tor/-/issues/30700Tor's Travis stem job shows debug logs from 10 minutes after the hang2020-07-29T03:54:12ZteorTor's Travis stem job shows debug logs from 10 minutes after the hangBut we need the logs from the time of the hang.
I'm not sure how to solve this issue, maybe we can grep for `date -10m` or something.But we need the logs from the time of the hang.
I'm not sure how to solve this issue, maybe we can grep for `date -10m` or something.Tor: unspecifiedDamian JohnsonDamian Johnsonhttps://gitlab.torproject.org/tpo/core/tor/-/issues/30697test_get_hidden_service_descriptor fails with an unexpected response code2020-06-27T13:50:07Zteortest_get_hidden_service_descriptor fails with an unexpected response code```
======================================================================
ERROR: test_get_hidden_service_descriptor
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/...```
======================================================================
ERROR: test_get_hidden_service_descriptor
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/travis/build/torproject/stem/test/require.py", line 57, in wrapped
return func(self, *args, **kwargs)
File "/home/travis/build/torproject/stem/test/require.py", line 57, in wrapped
return func(self, *args, **kwargs)
File "/home/travis/build/torproject/stem/test/require.py", line 57, in wrapped
return func(self, *args, **kwargs)
File "/home/travis/build/torproject/stem/test/integ/control/controller.py", line 1292, in test_get_hidden_service_descriptor
desc = controller.get_hidden_service_descriptor('3g2upl4pq6kufc4m.onion')
File "/home/travis/build/torproject/stem/stem/control.py", line 487, in wrapped
return func(self, *args, **kwargs)
File "/home/travis/build/torproject/stem/stem/control.py", line 2186, in get_hidden_service_descriptor
raise stem.ProtocolError('HSFETCH returned unexpected response code: %s' % response.code)
ProtocolError: HSFETCH returned unexpected response code: 512
----------------------------------------------------------------------
```
https://travis-ci.org/torproject/stem/jobs/539138294#L1832
I think this might have been a recent tor change?Damian JohnsonDamian Johnsonhttps://gitlab.torproject.org/tpo/core/tor/-/issues/30695Consider running tor's stem CI job in --target ONLINE mode, maybe with chutney2022-06-17T13:33:42ZteorConsider running tor's stem CI job in --target ONLINE mode, maybe with chutneyWe run some of stem's jobs in online mode, we could do the same with tor to increase coverage.
Using chutney for our network could be more reliable, but it would also take more time to implement.
Setting as sponsor 31 can, because we u...We run some of stem's jobs in online mode, we could do the same with tor to increase coverage.
Using chutney for our network could be more reliable, but it would also take more time to implement.
Setting as sponsor 31 can, because we use these jobs to make sure our refactoring works.https://gitlab.torproject.org/tpo/core/tor/-/issues/30694Restrict the tor CI stem job to tests that actually use tor2020-06-27T13:50:07ZteorRestrict the tor CI stem job to tests that actually use torFrom https://trac.torproject.org/projects/tor/ticket/30653?replyto=2#comment:2
> 1. Tests tor/stem interoperability. This is the only thing the network team has cause to care about. For this I'd suggest running python 3.x with...
>
> `...From https://trac.torproject.org/projects/tor/ticket/30653?replyto=2#comment:2
> 1. Tests tor/stem interoperability. This is the only thing the network team has cause to care about. For this I'd suggest running python 3.x with...
>
> `run_tests.py --integ --test control.controller --test control.base_controller --test process`Tor: 0.4.2.x-finalhttps://gitlab.torproject.org/tpo/core/tor/-/issues/30692Write better unit tests for dirserv_load_fingerprint_file()2020-06-27T13:50:07ZteorWrite better unit tests for dirserv_load_fingerprint_file()The existing tests don't catch format string bugs, because they only run on expected inputs:
https://github.com/torproject/tor/pull/970#pullrequestreview-242268332The existing tests don't catch format string bugs, because they only run on expected inputs:
https://github.com/torproject/tor/pull/970#pullrequestreview-242268332Tor: 0.4.2.x-finalhttps://gitlab.torproject.org/tpo/core/tor/-/issues/30691Refactor decoding outside the if expression in dirserv_load_fingerprint_file()2021-09-16T14:23:37ZteorRefactor decoding outside the if expression in dirserv_load_fingerprint_file()There have already been two bugs in this code, so we should refactor it:
https://github.com/torproject/tor/pull/970#pullrequestreview-242268332There have already been two bugs in this code, so we should refactor it:
https://github.com/torproject/tor/pull/970#pullrequestreview-242268332Tor: unspecifiedNeel Chauhanneel@neelc.orgNeel Chauhanneel@neelc.orghttps://gitlab.torproject.org/tpo/core/tor/-/issues/30687Implement a generic counter token bucket2020-06-27T13:50:08ZDavid Gouletdgoulet@torproject.orgImplement a generic counter token bucketWe have the `_raw_t` interface and a `_rw_t` specialized one. Many use cases only need a single counter like for instance the circuit creation DoS defense would use that. Or legacy/trac#15516 introduce2 rate limiting.
After a discussion...We have the `_raw_t` interface and a `_rw_t` specialized one. Many use cases only need a single counter like for instance the circuit creation DoS defense would use that. Or legacy/trac#15516 introduce2 rate limiting.
After a discussion with nickm, it would be wise if we can simply implement a generic counter token bucket:
```
token_bucket_ctr_t
```
... with a single bucket in it and thus could generically be used by many subsystems.Tor: 0.4.2.x-final