Trac issueshttps://gitlab.torproject.org/legacy/trac/-/issues2020-06-13T15:52:08Zhttps://gitlab.torproject.org/legacy/trac/-/issues/34037Make chutney check tor's logs for reachability self-test success2020-06-13T15:52:08ZteorMake chutney check tor's logs for reachability self-test successThis ticket is an alternative to #33582 or #33222.
Instead of fixing bridge descriptor uploads, we can check bridge logs to make sure that reachability self-tests have succeeded.
For consistency, we should also do the same checks for r...This ticket is an alternative to #33582 or #33222.
Instead of fixing bridge descriptor uploads, we can check bridge logs to make sure that reachability self-tests have succeeded.
For consistency, we should also do the same checks for relays.
We can only do these tests on authorities, relays, and bridges that are configured with `AssumeReachable 0`. Chutney's current defaults are:
* directory authorities: 1
* bridge authorities: 1
* relays: 0
* bridges: 0
* clients: clients never perform reachability self-tests
Some custom chutney networks may set `AssumeReachable 1` for relays and bridges. So we should make it easy for them to disable these checks.https://gitlab.torproject.org/legacy/trac/-/issues/33957Unexpected keyword argument 'bufsize' in subprocess.check_output()2020-06-13T13:31:51ZteorUnexpected keyword argument 'bufsize' in subprocess.check_output()On some python versions, chutney's call to subprocess.check_output() results in the following error:
```
Unexpected keyword argument 'bufsize'…
```
https://github.com/torproject/chutney/pull/68#issuecomment-614596946
The check_output() ...On some python versions, chutney's call to subprocess.check_output() results in the following error:
```
Unexpected keyword argument 'bufsize'…
```
https://github.com/torproject/chutney/pull/68#issuecomment-614596946
The check_output() documentation is unclear about which arguments are supported: it simply says that most arguments to Popen() are supported. (And bufsize is an argument to Popen().):
https://docs.python.org/3.5/library/subprocess.html#subprocess.check_output
https://docs.python.org/3.5/library/subprocess.html#subprocess.Popen
I think our best option here is to remove the bufsize argument, and accept the default buffered behaviour.https://gitlab.torproject.org/legacy/trac/-/issues/33918Stop truncating IPv6 addresses in channel logs2020-06-13T15:53:10ZteorStop truncating IPv6 addresses in channel logschannel_tls_get_remote_desc_method() formats IP addresses and ports, but its buffer is only 32 characters. IPv6 addresses can be up to 41 characters long, and the port is an extra 6 characters.channel_tls_get_remote_desc_method() formats IP addresses and ports, but its buffer is only 32 characters. IPv6 addresses can be up to 41 characters long, and the port is an extra 6 characters.Tor: 0.4.4.x-finalteorteorhttps://gitlab.torproject.org/legacy/trac/-/issues/33825Make Environ handle "in" and "get()" like a dict2020-06-13T13:31:49ZteorMake Environ handle "in" and "get()" like a dictSome standard Python dict code doesn't work on chutney's Environ class:
```
is_in_env = 'foo' in self._env
value_or_none = self._env.get('foo')
```
"in" should return a boolean, and "get()" should return the value (or None).
But instea...Some standard Python dict code doesn't work on chutney's Environ class:
```
is_in_env = 'foo' in self._env
value_or_none = self._env.get('foo')
```
"in" should return a boolean, and "get()" should return the value (or None).
But instead, when the key is missing, sometimes they throw a KeyError. (It seems to happen in certain contexts, but not others.)
We should work out if Environ is missing some of the required dict implementation functions. Or if there is some issue with Environ's lookup code.
Then we should implement unit tests, to make sure we don't break Environ in future.https://gitlab.torproject.org/legacy/trac/-/issues/33793Avoid some race conditions when running chutney networks in series2020-06-13T13:31:49ZteorAvoid some race conditions when running chutney networks in seriesWhen chutney runs multiple networks, one after the others, sometimes older tor instances are left running.
To avoid these issues, chutney can:
* wait longer after terminating tor processes
* cleanup pid files, so we don't accidentally t...When chutney runs multiple networks, one after the others, sometimes older tor instances are left running.
To avoid these issues, chutney can:
* wait longer after terminating tor processes
* cleanup pid files, so we don't accidentally terminate unrelated processesteorteorhttps://gitlab.torproject.org/legacy/trac/-/issues/33676Stop waiting a set time for microdescriptors2020-06-13T13:31:48ZteorStop waiting a set time for microdescriptorsYour code in #33675 needs to pass your local "make test-network-all", on master and maint-0.3.5, before you start this ticket.
When we make chutney check for microdescriptors, we can stop waiting a set time for microdescriptors to downl...Your code in #33675 needs to pass your local "make test-network-all", on master and maint-0.3.5, before you start this ticket.
When we make chutney check for microdescriptors, we can stop waiting a set time for microdescriptors to download.
Please read these instructions carefully. Don't leave out any steps!
First, set MIN_START_TIME_LEGACY and NODE_WAIT_FOR_UNCHECKED_DIR_INFO to 0:
https://github.com/torproject/chutney/blob/master/lib/chutney/TorNet.py#L926
https://github.com/torproject/chutney/blob/master/lib/chutney/TorNet.py#L952
Then check that `make test-network-all` still passes on:
* Tor master, and
* Tor maint-0.3.5
You can build a different tor branch using the following commands:
```
git checkout <branch>
make
```
Check the new code on your own machine, and then submit a pull request to run Chutney's CI.
If all those tests pass, you can remove all the code that uses MIN_START_TIME_LEGACY and NODE_WAIT_FOR_UNCHECKED_DIR_INFO:
* remove NODE_WAIT_FOR_UNCHECKED_DIR_INFO
* replace NODE_WAIT_FOR_UNCHECKED_DIR_INFO with 0 in getUncheckedDirInfoWaitTime()
* remove getMinStartTime()
* remove all the constants getMinStartTime() uses
* remove all calls to getMinStartTime()
* remove all the variables, code, and comments that depend on getMinStartTime()
* https://github.com/torproject/chutney/blob/master/lib/chutney/TorNet.py#L2224
* https://github.com/torproject/chutney/blob/master/lib/chutney/TorNet.py#L2273
Then add a commit to your pull request with these changes, and check that they pass CI.
You can ignore macOS hangs ("red ! symbol") on Travis CI. It is a bit unstable right now.https://gitlab.torproject.org/legacy/trac/-/issues/33675Search microdescriptor files for relay ed25519 keys2020-06-13T13:31:47ZteorSearch microdescriptor files for relay ed25519 keysYour code in #33428 needs to pass your local "make test-network-all", before you start this ticket.
We need to enable searching for ed25519 keys in relay microdescriptor files.
There are instructions and a draft search pattern here:
ht...Your code in #33428 needs to pass your local "make test-network-all", before you start this ticket.
We need to enable searching for ed25519 keys in relay microdescriptor files.
There are instructions and a draft search pattern here:
https://github.com/torproject/chutney/blob/master/lib/chutney/TorNet.py#L1325
Please open a new pull request for this ticket. Your branch should be based on the final version of #33428.
Before you push new changes to your pull request, your chutney code should pass:
* "make test-network-all" on tor master
* "make test-network-all" on tor maint-0.3.5
You can build a tor branch using these commands:
```
cd tor
git checkout -b <branch>
make
```
Where <branch> is master or maint-0.3.5https://gitlab.torproject.org/legacy/trac/-/issues/33615Wait for at least 60 seconds for 0.3.5 and earlier to bootstrap2020-06-13T13:31:45ZteorWait for at least 60 seconds for 0.3.5 and earlier to bootstrapThe basic-min network is still a bit unstable, but only for 0.3.5. (And it's much more unrealiable on macOS than Linux.)
It seems like 0.3.5 needs around 60 seconds to bootstrap and sync descriptors. Later versions have microdescriptor ...The basic-min network is still a bit unstable, but only for 0.3.5. (And it's much more unrealiable on macOS than Linux.)
It seems like 0.3.5 needs around 60 seconds to bootstrap and sync descriptors. Later versions have microdescriptor download fixes, and they can bootstrap and verify in just over 20 seconds on basic-min.
So let's make the minimum wait time 70 seconds (3 consensuses), if any nodes in the network are running 0.3.5 (or earlier).teorteorhttps://gitlab.torproject.org/legacy/trac/-/issues/33609Check that onion services have successfully posted descriptors before verifying2020-06-13T13:31:44ZteorCheck that onion services have successfully posted descriptors before verifyingBefore verifying, chutney checks that:
* each relay descriptor is cached at each node
* each relay is in a consensus, cached at each node
* each relay is in a microdesc consensus, cached at each node
* each bridge descriptor is cached at...Before verifying, chutney checks that:
* each relay descriptor is cached at each node
* each relay is in a consensus, cached at each node
* each relay is in a microdesc consensus, cached at each node
* each bridge descriptor is cached at each bridge client
We have other tickets for checking:
* microdescriptors
* cached bridge descriptors at the bridge authority
* the bridge networkstatus
That just leaves onion services.
Onion services are tricky, because they post to some HSDirs in the network, but not all. And those HSDirs don't cache the onion service descriptors in a file.
So here is one possible design for this feature:
* check each onion service log for a successful descriptor post to at least one HSDir
* check v2 and v3 onion services
* call it an extra 200% "bootstrap" stage (because it's a sender log check, not a receiver cached file check)
* require 200% bootstrap for onion servicescchttps://gitlab.torproject.org/legacy/trac/-/issues/33598chutney does not fail on some SOCKS errors2020-06-13T13:31:42Zteorchutney does not fail on some SOCKS errorsWhen tor can't make a connection, and sends back a SOCKS error, chutney keeps on rapidly sending SOCKS requests. Instead, chutney should fail.
I think we introduced this bug when we started using asyncore.
I have worked around the bug...When tor can't make a connection, and sends back a SOCKS error, chutney keeps on rapidly sending SOCKS requests. Instead, chutney should fail.
I think we introduced this bug when we started using asyncore.
I have worked around the bug using a 5 second asyncore timeout, but we should come up with a permanent fix.
I think nickm might be able to help with this issue, because he wrote that code.cchttps://gitlab.torproject.org/legacy/trac/-/issues/33596Fix or disable mixed+hs-v2 for Tor 0.3.52020-06-13T13:31:41ZteorFix or disable mixed+hs-v2 for Tor 0.3.5In #33379, we made some changes that broke mixed+hs-v2 on Tor 0.3.5. Strangely, hs-v2-min works fine.
It looks like it is caused by some microdescriptor download bugs that we fixed in 0.4.0 or 0.4.1.
When we finish the rest of #33232, ...In #33379, we made some changes that broke mixed+hs-v2 on Tor 0.3.5. Strangely, hs-v2-min works fine.
It looks like it is caused by some microdescriptor download bugs that we fixed in 0.4.0 or 0.4.1.
When we finish the rest of #33232, or fix #33428, we should either:
* fix this issue
* require tor or tor-stable to be 0.4.1 or later in mixed+hs-v2Tor: 0.4.4.x-finalhttps://gitlab.torproject.org/legacy/trac/-/issues/33595Stop waiting for unchecked directory info2020-06-13T13:31:40ZteorStop waiting for unchecked directory infoOnce we've fixed #33428 (microdescs), #33582 (bridges), and #33609 (onion services), we might be able to remove WAIT_FOR_UNCHECKED_DIR_INFO in chutney's TorNet.py.
But we might still need to wait for:
* internal tor state changes, in re...Once we've fixed #33428 (microdescs), #33582 (bridges), and #33609 (onion services), we might be able to remove WAIT_FOR_UNCHECKED_DIR_INFO in chutney's TorNet.py.
But we might still need to wait for:
* internal tor state changes, in response to downloading descriptors.
So we will need to make this change, then test it on Tor 0.3.5 and master.https://gitlab.torproject.org/legacy/trac/-/issues/33583Stop setting AssumeReachable on chutney relays and bridges2020-06-13T15:53:49ZteorStop setting AssumeReachable on chutney relays and bridgesWe need to stop setting AssumeReachable on relays and bridges in chutney networks, so they do reachability self-tests.
We should continue to set AssumeReachable on authorities (including the bridge authority), because it disables author...We need to stop setting AssumeReachable on relays and bridges in chutney networks, so they do reachability self-tests.
We should continue to set AssumeReachable on authorities (including the bridge authority), because it disables authority to relay reachability tests. (These tests are still performed on a half-hourly schedule, even in chutney networks. And they are out of scope for Sponsor 55.)
After we implement this change, we should also be able to implement #33581.teorteorhttps://gitlab.torproject.org/legacy/trac/-/issues/33582Make bridges wait until they have bootstrapped, before publishing their descr...2020-06-13T15:52:08ZteorMake bridges wait until they have bootstrapped, before publishing their descriptorInstead of this fix, we can make chutney check tor's logs for reachability self-test successes (#34037), or implement strict self-tests (#33222).
On bridges, there's a race condition when bridges try to publish their descriptor to the b...Instead of this fix, we can make chutney check tor's logs for reachability self-test successes (#34037), or implement strict self-tests (#33222).
On bridges, there's a race condition when bridges try to publish their descriptor to the bridge authority:
* bridges try to publish their descriptors before bootstrapping
* but bridges can't publish their descriptors, because they don't have enough directory info to build a circuit to the bridge authority
Bridges will eventually try to publish their descriptors again, when they become dirty.
We should make bridges wait until they have bootstrapped, before they try to publish their descriptors. (This might be a good change for relays as well: there isn't much point in publishing a relay that can't bootstrap.)
This issue happens regardless of `AssumeReachable`. It is most obvious in chutney networks.
This ticket isn't essential. But the workarounds seem to cause weird race conditions, which are time-consuming to diagnose and fix.Tor: unspecifiedhttps://gitlab.torproject.org/legacy/trac/-/issues/33428Make chutney check for relay microdescriptors before verifying2020-06-13T13:31:37ZteorMake chutney check for relay microdescriptors before verifyingIn #33232, we make chutney check for relay and bridge descriptors, and relays in the consensus, before verifying. But we don't check for relay microdescriptors.
The easiest way to check for relay microdescriptors, is using each relay's ...In #33232, we make chutney check for relay and bridge descriptors, and relays in the consensus, before verifying. But we don't check for relay microdescriptors.
The easiest way to check for relay microdescriptors, is using each relay's ed25519 key. In #30642, we will add an `ed25519-identity` file to each relay's data directory. Chutney can read the contents of this file, and then search for it in `cached-microdescriptors` and `cached-microdescriptors.new`.teorteorhttps://gitlab.torproject.org/legacy/trac/-/issues/33379Make chutney wait for all relays in the consensus before verifying2020-06-13T13:31:35ZteorMake chutney wait for all relays in the consensus before verifyingAs part of #33232, we want chutney to check that all relays are in the consensus, then verify.As part of #33232, we want chutney to check that all relays are in the consensus, then verify.teorteorhttps://gitlab.torproject.org/legacy/trac/-/issues/33378Require chutney node bootstrap before running verify2020-06-13T13:31:34ZteorRequire chutney node bootstrap before running verifyAs part of testing relay reachability self-tests in #33232, we need to make sure that all chutney nodes bootstrap.As part of testing relay reachability self-tests in #33232, we need to make sure that all chutney nodes bootstrap.teorteorhttps://gitlab.torproject.org/legacy/trac/-/issues/33376Update the networks in Chutney's CI to match Tor's new test-network*2020-06-13T13:31:33ZteorUpdate the networks in Chutney's CI to match Tor's new test-network*We need to update the list of chutney networks in chutney's CI, to match the networks we added to Tor's `make test-network` and `make test-network-all` in #33334.We need to update the list of chutney networks in chutney's CI, to match the networks we added to Tor's `make test-network` and `make test-network-all` in #33334.teorteorhttps://gitlab.torproject.org/legacy/trac/-/issues/33358Update dir-spec for consensus voting improvements2020-06-13T15:51:31ZteorUpdate dir-spec for consensus voting improvementsIn #4631, we change how authorities handle late posted votes. We need to update the dir-spec for this change.In #4631, we change how authorities handle late posted votes. We need to update the dir-spec for this change.Tor: 0.4.4.x-finalteorteorhttps://gitlab.torproject.org/legacy/trac/-/issues/33353Split chutney's diagnostics into a new script2020-06-13T13:31:32ZteorSplit chutney's diagnostics into a new scriptChutney's failure diagnostics are currently in the Travis CI config file.
But we want to use them in tor's CI. And maybe chutney users want to use them as well.Chutney's failure diagnostics are currently in the Travis CI config file.
But we want to use them in tor's CI. And maybe chutney users want to use them as well.teorteor