Chutney issueshttps://gitlab.torproject.org/tpo/core/chutney/-/issues2020-06-27T13:19:05Zhttps://gitlab.torproject.org/tpo/core/chutney/-/issues/13980Added new script for warnings in chutney's logs2020-06-27T13:19:05ZTracAdded new script for warnings in chutney's logsIn chutney's repository in the TODO file there was a comment about doing a script for getting warnings in the logs. The script is at https://github.com/huig-/chutney and i made a pull request days ago (see https://trac.torproject.org/pro...In chutney's repository in the TODO file there was a comment about doing a script for getting warnings in the logs. The script is at https://github.com/huig-/chutney and i made a pull request days ago (see https://trac.torproject.org/projects/tor/ticket/13956#ticket) with a different change but now it also includes the script about warnings. The script basically gets warnings int net/nodes/*/info.logs and counts how many times each warning occurs. I have also added an option to get the warnings of only one node (specified by the user) instead of all of them.
The script is called warnings.sh
**Trac**:
**Username**: huigNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/tpo/core/chutney/-/issues/10154Allow a Bridge Authority in Test Networks2020-06-27T13:19:07ZMatthew FinkelAllow a Bridge Authority in Test NetworksCurrently Chutney generates an invalid torrc when the network contains a bridge authority.
```
Nov 13 23:01:36.926 [warn] You cannot set both DirAuthority and Alternate*Authority.
Nov 13 23:01:36.926 [warn] Failed to parse/validate conf...Currently Chutney generates an invalid torrc when the network contains a bridge authority.
```
Nov 13 23:01:36.926 [warn] You cannot set both DirAuthority and Alternate*Authority.
Nov 13 23:01:36.926 [warn] Failed to parse/validate config: Directory authority/fallback line did not parse. See logs for details.
Nov 13 23:01:36.926 [err] Reading config failed--see warnings above.
```Nick MathewsonNick Mathewsonhttps://gitlab.torproject.org/tpo/core/chutney/-/issues/32721Allow chutney users to disable tor's sandbox at runtime2020-07-14T22:24:30ZteorAllow chutney users to disable tor's sandbox at runtimeIn legacy/trac#32240, we discovered that tor's sandbox doesn't work on recent Ubuntu versions. So we need to disable the sandbox on those CI jobs.
One good way to implement this change, is by making it a first-class chutney feature.In legacy/trac#32240, we discovered that tor's sandbox doesn't work on recent Ubuntu versions. So we need to disable the sandbox on those CI jobs.
One good way to implement this change, is by making it a first-class chutney feature.teorteorhttps://gitlab.torproject.org/tpo/core/chutney/-/issues/10155Allow delayed node start2020-07-17T11:02:21ZMatthew FinkelAllow delayed node startIt will be useful to allow nodes to be started after a certain amount of time, especially when interested in its effect on the consensus.It will be useful to allow nodes to be started after a certain amount of time, especially when interested in its effect on the consensus.https://gitlab.torproject.org/tpo/core/chutney/-/issues/18826Allow setting the listen address for all Chutney hosts2020-06-27T13:18:57ZanonymAllow setting the listen address for all Chutney hostsIn Tails' automated test suite we have plans to use Chutney to run a simulated Tor network (for increased performance and, in particular, robustness). Of course, we do not want to run that Chutney instance in the system under testing (i....In Tails' automated test suite we have plans to use Chutney to run a simulated Tor network (for increased performance and, in particular, robustness). Of course, we do not want to run that Chutney instance in the system under testing (i.e. the Tails instance) but on the host orchestrating the tests. In order to avoid having to forward traffic via firewall rules, we'd like it to be possible to make the Chutney hosts listen on some other IP address than localhost.
The attached patch allows setting the address all Chutney hosts will listen on via the `CHUTNEY_LISTEN_ADDRESS` environment variable.anonymanonymhttps://gitlab.torproject.org/tpo/core/chutney/-/issues/18827Allow specifying the Chutney data directory2020-06-27T13:18:57ZanonymAllow specifying the Chutney data directoryWith "Chutney data directory" I am referring to the `net` sub-directory. I find it preferable to be able to store such cruft outside of the source tree. See the attached patch.With "Chutney data directory" I am referring to the `net` sub-directory. I find it preferable to be able to store such cruft outside of the source tree. See the attached patch.anonymanonymhttps://gitlab.torproject.org/tpo/core/chutney/-/issues/19701Allow tor and tor-gencert to be specified on the chutney command-line2020-06-27T13:18:56ZteorAllow tor and tor-gencert to be specified on the chutney command-lineAs well as being specified via the CHUTNEY_TOR and CHUTNEY_TOR_GENCERT environmental variables.As well as being specified via the CHUTNEY_TOR and CHUTNEY_TOR_GENCERT environmental variables.teorteorhttps://gitlab.torproject.org/tpo/core/chutney/-/issues/23503AttributeError: 'NoneType' object has no attribute 'startswith'2020-06-27T13:18:47ZDavid Gouletdgoulet@torproject.orgAttributeError: 'NoneType' object has no attribute 'startswith'Chutney latest master 99b7c5e8e8e063af doesn't work for me and explodes with:
```
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File ...Chutney latest master 99b7c5e8e8e063af doesn't work for me and explodes with:
```
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/dgoulet/git/chutney/lib/chutney/TorNet.py", line 1161, in <module>
sys.exit(main())
File "/home/dgoulet/git/chutney/lib/chutney/TorNet.py", line 1155, in main
result = runConfigFile(args['action'], f)
File "/home/dgoulet/git/chutney/lib/chutney/TorNet.py", line 1131, in runConfigFile
return getattr(network, verb)()
File "/home/dgoulet/git/chutney/lib/chutney/TorNet.py", line 990, in configure
self._checkConfig()
File "/home/dgoulet/git/chutney/lib/chutney/TorNet.py", line 982, in _checkConfig
n.getBuilder().checkConfig(self)
File "/home/dgoulet/git/chutney/lib/chutney/TorNet.py", line 352, in checkConfig
self._createTorrcFile(checkOnly=True)
File "/home/dgoulet/git/chutney/lib/chutney/TorNet.py", line 272, in _createTorrcFile
output = torrc_template.format(self._env)
File "/home/dgoulet/git/chutney/lib/chutney/Templating.py", line 361, in format
v = _BetterTemplate(orig_val).substitute(values)
File "/usr/lib/python2.7/string.py", line 176, in substitute
return self.pattern.sub(convert, self.template)
File "/usr/lib/python2.7/string.py", line 166, in convert
val = mapping[named]
File "/home/dgoulet/git/chutney/lib/chutney/Templating.py", line 111, in __getitem__
return self.lookup(key, self)
File "/home/dgoulet/git/chutney/lib/chutney/Templating.py", line 135, in lookup
return lookup(key, my)
File "/home/dgoulet/git/chutney/lib/chutney/Templating.py", line 120, in lookup
return self._getitem(key, my)
File "/home/dgoulet/git/chutney/lib/chutney/Templating.py", line 216, in _getitem
rv = fn(my)
File "/home/dgoulet/git/chutney/lib/chutney/TorNet.py", line 897, in _get_server_dns_resolv_conf
dns_conf = os.path.abspath(my['dns_conf'])
File "/usr/lib/python2.7/posixpath.py", line 360, in abspath
if not isabs(path):
File "/usr/lib/python2.7/posixpath.py", line 54, in isabs
return s.startswith('/')
AttributeError: 'NoneType' object has no attribute 'startswith'
```https://gitlab.torproject.org/tpo/core/chutney/-/issues/33793Avoid some race conditions when running chutney networks in series2020-06-27T13:18:30ZteorAvoid some race conditions when running chutney networks in seriesWhen chutney runs multiple networks, one after the others, sometimes older tor instances are left running.
To avoid these issues, chutney can:
* wait longer after terminating tor processes
* cleanup pid files, so we don't accidentally t...When chutney runs multiple networks, one after the others, sometimes older tor instances are left running.
To avoid these issues, chutney can:
* wait longer after terminating tor processes
* cleanup pid files, so we don't accidentally terminate unrelated processesSponsor 55: Improving the Tor network’s IPv6 supportteorteorhttps://gitlab.torproject.org/tpo/core/chutney/-/issues/22375Can tor-prompt optionally wait for tor to start on a port, rather than failin...2020-07-23T20:34:18ZteorCan tor-prompt optionally wait for tor to start on a port, rather than failing straight away?When I start a chutney network and want to run tor-prompt on it, I have to wait until the relays are running before I start tor-prompt.
It would be nicer to just start tor-prompt, and have it tell me when the relays are up.When I start a chutney network and want to run tor-prompt on it, I have to wait until the relays are running before I start tor-prompt.
It would be nicer to just start tor-prompt, and have it tell me when the relays are up.https://gitlab.torproject.org/tpo/core/chutney/-/issues/22323Catch and handle socket errors in chutney2020-07-23T20:33:45ZteorCatch and handle socket errors in chutneyIf Tor dies before chutney runs verify, I get several pages of output that end in 'error: [Errno 32] Broken pipe'.
We could be smarter about this. But it's not a bug deal.If Tor dies before chutney runs verify, I get several pages of output that end in 'error: [Errno 32] Broken pipe'.
We could be smarter about this. But it's not a bug deal.https://gitlab.torproject.org/tpo/core/chutney/-/issues/33604Catch common runtime errors in chutney shell scripts2020-07-23T20:31:12ZteorCatch common runtime errors in chutney shell scriptsLet's gradually convert our scripts to use:
```
set -e
set -u
set -o pipefail
```
And maybe:
```
IFS=$'\n\t'
```
These settings help catch common errors in bash scripts at runtime:
http://redsymbol.net/articles/unofficial-bash-strict-m...Let's gradually convert our scripts to use:
```
set -e
set -u
set -o pipefail
```
And maybe:
```
IFS=$'\n\t'
```
These settings help catch common errors in bash scripts at runtime:
http://redsymbol.net/articles/unofficial-bash-strict-mode/
(Shellcheck helps us catch errors while writing scripts, but it can't help with runtime failures.)
Follow-up to legacy/trac#33451.https://gitlab.torproject.org/tpo/core/chutney/-/issues/21552Change the default directory creation permissions globally2020-06-27T13:18:52ZcypherpunksChange the default directory creation permissions globallyThe change in legacy/trac#21464 made it so all calls to mkdir_p now use mode 448. To prevent future problems i suggest changing the default mode in mkdir_p which would also simplify existing calls.The change in legacy/trac#21464 made it so all calls to mkdir_p now use mode 448. To prevent future problems i suggest changing the default mode in mkdir_p which would also simplify existing calls.teorteorhttps://gitlab.torproject.org/tpo/core/chutney/-/issues/27379Check for src/app and src/or in chutney, and fail if they are both present2020-06-27T13:18:45ZteorCheck for src/app and src/or in chutney, and fail if they are both presentOtherwise, chutney can easily choose the wrong tor.Otherwise, chutney can easily choose the wrong tor.teorteorhttps://gitlab.torproject.org/tpo/core/chutney/-/issues/33609Check that onion services have successfully posted descriptors before verifying2020-07-07T15:12:56ZteorCheck that onion services have successfully posted descriptors before verifyingBefore verifying, chutney checks that:
* each relay descriptor is cached at each node
* each relay is in a consensus, cached at each node
* each relay is in a microdesc consensus, cached at each node
* each bridge descriptor is cached at...Before verifying, chutney checks that:
* each relay descriptor is cached at each node
* each relay is in a consensus, cached at each node
* each relay is in a microdesc consensus, cached at each node
* each bridge descriptor is cached at each bridge client
We have other tickets for checking:
* microdescriptors
* cached bridge descriptors at the bridge authority
* the bridge networkstatus
That just leaves onion services.
Onion services are tricky, because they post to some HSDirs in the network, but not all. And those HSDirs don't cache the onion service descriptors in a file.
So here is one possible design for this feature:
* check each onion service log for a successful descriptor post to at least one HSDir
* check v2 and v3 onion services
* call it an extra 200% "bootstrap" stage (because it's a sender log check, not a receiver cached file check)
* require 200% bootstrap for onion servicesNick MathewsonNick Mathewsonhttps://gitlab.torproject.org/tpo/core/chutney/-/issues/32969Chutney 'bootstrap-network.sh' shouldn't overwrite 'CHUTNEY_DATA_DIR'2020-06-27T13:18:35ZoparaChutney 'bootstrap-network.sh' shouldn't overwrite 'CHUTNEY_DATA_DIR'When you set up a tor network with something like:
```
tools/test-network.sh --flavor basic --net-dir /tmp/chutney-net
```
the `tools/test-network.sh` script will set `$CHUTNEY_DATA_DIR` based on the value of `--net-dir` and then call ...When you set up a tor network with something like:
```
tools/test-network.sh --flavor basic --net-dir /tmp/chutney-net
```
the `tools/test-network.sh` script will set `$CHUTNEY_DATA_DIR` based on the value of `--net-dir` and then call `tools/bootstrap-network.sh`. This bootstrapping script will then overwrite the variable `$CHUTNEY_DATA_DIR` if that directory doesn't exist or if the path is relative. This causes unexpected behavior when the user explicitly sets the `--net-dir` option. For example, Chutney works fine if you provide it with a directory that doesn't exist (it will create that directory automatically), so there is no need for the `tools/bootstrap-network.sh` script to unexpectedly change it to `$CHUTNEY_PATH/net`. The `tools/bootstrap-network.sh` also does not need to change it to an absolute path since Chutney [does this automatically](https://github.com/torproject/chutney/blob/master/lib/chutney/TorNet.py#L144).
My suggestion is to remove the following lines from `tools/bootstrap-network.sh`:
```
# Get a working net path
if [ ! -d "$CHUTNEY_DATA_DIR" ]; then
# looks like a broken path: use the chutney path as a base
export CHUTNEY_DATA_DIR="$CHUTNEY_PATH/net"
fi
if [ -d "$PWD/$CHUTNEY_DATA_DIR" ]; then
# looks like a relative path: make chutney path absolute
export CHUTNEY_DATA_DIR="$PWD/$CHUTNEY_DATA_DIR"
fi
```https://gitlab.torproject.org/tpo/core/chutney/-/issues/33023Chutney 'bootstrap-network.sh' starts a different default network flavor than...2020-06-27T13:18:35ZoparaChutney 'bootstrap-network.sh' starts a different default network flavor than listed in the usage commentsSmall documentation problem in Chutney's `bootstrap-network.sh`.
The usage comment says the following:
```
# Usage:
# tools/bootstrap-network.sh [network-flavour]
# network-flavour: one of the files in the networks directory,
# ...Small documentation problem in Chutney's `bootstrap-network.sh`.
The usage comment says the following:
```
# Usage:
# tools/bootstrap-network.sh [network-flavour]
# network-flavour: one of the files in the networks directory,
# (default: 'basic')
```
but the actual code says the following:
```
# Set the variables for the chutney network flavour
export NETWORK_FLAVOUR=${NETWORK_FLAVOUR:-"bridges+hs-v2"}
[ -n "$1" ] && { NETWORK_FLAVOUR=$1; shift; }
export CHUTNEY_NETWORK="$CHUTNEY_PATH/networks/$NETWORK_FLAVOUR"
```
As you can see, the comments say that the default network is 'basic', but the code actually starts 'bridges+hs-v2'. Reference: https://github.com/torproject/chutney/blob/5a9271104fec6c5b9bb23d2574596dc9d7c0a2c4/tools/bootstrap-network.shhttps://gitlab.torproject.org/tpo/core/chutney/-/issues/30058Chutney bootstrap-network script uses the wrong network flavour2020-06-27T13:18:41ZoparaChutney bootstrap-network script uses the wrong network flavourThe 'tools/bootstrap-network.sh' script doesn't actually use its program argument (the network flavour). This means that the incorrect network is started.
The line:
`export CHUTNEY_NETWORK="$CHUTNEY_PATH/networks/$NETWORK_FLAVOUR"`
sh...The 'tools/bootstrap-network.sh' script doesn't actually use its program argument (the network flavour). This means that the incorrect network is started.
The line:
`export CHUTNEY_NETWORK="$CHUTNEY_PATH/networks/$NETWORK_FLAVOUR"`
should be:
`export CHUTNEY_NETWORK="$CHUTNEY_PATH/networks/$flavour"`teorteorhttps://gitlab.torproject.org/tpo/core/chutney/-/issues/40005Chutney Bridges+hs-v23 network is failing in CI2020-07-21T13:39:10ZNick MathewsonChutney Bridges+hs-v23 network is failing in CIIn CI, and on my desktop, chutney bridges+hs-v23 is failing for clients and relays. Apparently, nothing finds that the bridge successfully bootstrapped.
See also related tickets chutney#33581, tor#33582, and tor#33407.
I have a works...In CI, and on my desktop, chutney bridges+hs-v23 is failing for clients and relays. Apparently, nothing finds that the bridge successfully bootstrapped.
See also related tickets chutney#33581, tor#33582, and tor#33407.
I have a works-for-me fix currently in github CI.Nick MathewsonNick Mathewsonhttps://gitlab.torproject.org/tpo/core/chutney/-/issues/25228Chutney configuration to create more non-exits relays than exits relays2020-06-27T13:18:46ZjugaChutney configuration to create more non-exits relays than exits relaysAs commented in [1], the chutney networks/basic-025 configuration [0] creates 16 exits and only 4 non-exits relays (that are actually authorities).
Maybe it should be provided another chutney configuration file, since in the production T...As commented in [1], the chutney networks/basic-025 configuration [0] creates 16 exits and only 4 non-exits relays (that are actually authorities).
Maybe it should be provided another chutney configuration file, since in the production Tor the there are more non-exit relays than exits, or modify the existing one.
The new configuration file could be something like 14 relays and 2 exits? (assuming 1 exit per 7 relays).
Looking other chutney configuration files, it seems like probably networks/middle-025 would be more suitable for bwscanner.
Still that file doesn't totally match Tor production proportions and in bwscanner clients are not needed.
Find attached a possible configuration file for bwscanner.
[0] https://github.com/torproject/chutney/blob/master/networks/basic-025#L6
[1] https://github.com/TheTorProject/bwscanner/issues/95#issuecomment-365117858