os.path and pathlib

There is a mix of os.path.join() calls and simple string concatenation throughout the code. Examples: getTests() versus make_datadir_subdirectory() in TorNet.py (also, just noticed the mixed camelCase and snake_case which might deserve its own style ticket).

I think we want to be consistent with path handling, but has pathlib been taken into consideration to make it more self-documenting where Chutney needs to deal with paths?

Sounds like a good idea to me to unify the use around os.path.join().

Even where we do use os.path.join(), it is called with mixes of individual path components and absolute or relative pathnames across directories, which might cause unexpected results across platforms. Has Chutney been used in Windows environments successfully, for example?

I don't know whether chutney works on windows these days or not, though I think it may once have worked there. I am fine with both pathlib and os.path.

See https://github.com/torproject/chutney/pull/78

added Needs Review label

assigned to @nickm

Assigning to myself for review.

Back to you! I've left a review on gitlab. Also, CI appears to be failing with a syntax issue, likely due to a python version incompatibility?

I'm okay with dropping python 2 support -- it's time -- but if we do so, we need to update our .travis.yml file to use a newer python versions for every test.

I replied that yes, this syntax is incompatible with earlier versions of Python (pre 3.4.3 according to Stack Overflow) but since other parts of the code would cause 2.x tests to fail, we should just drop support.

assigned to @c and unassigned @nickm

added Needs Revision label and removed Needs Review label

ok. our next step here is to get the .travis.yml file fixed up so it's using python 3.

I've started a PR at https://github.com/torproject/chutney/pull/79 for the python3 change on top of this, but it may take a few tries to get right.

Okay, closer and closer. Here's the latest travis results: https://travis-ci.org/github/torproject/chutney/builds/706731931

The first failure seems to be a tor issue (tor#40032 (closed)). The second failure seems to be using the python-2 version of pypy, which we no longer expect to succeed. The third one, though, looks like an actual compatibility issue between your patch and pypy 3. For example, it doesn't like PosixPath as an argument to os.symlink.

Here's the results: https://travis-ci.org/github/torproject/chutney/jobs/706731952

@c, can you have a look at this?

Third issue should be fixed with cc3a467 – it was an oversight on my part

Have a look at the CI at https://github.com/torproject/chutney/pull/79 -- it seems to be failing with this error now:

NOTE: creating PosixPath('/home/travis/build/torproject/chutney/net/nodes.1594643565'), linking to PosixPath('/home/travis/build/torproject/chutney/net/nodes')
<pre>Traceback (most recent call last):
  File "/opt/python/3.8.1/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/python/3.8.1/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/travis/build/torproject/chutney/lib/chutney/TorNet.py", line 2728, in &lt;module&gt;
    sys.exit(main())
  File "/home/travis/build/torproject/chutney/lib/chutney/TorNet.py", line 2722, in main
    result = runConfigFile(args['action'], f.read())
  File "/home/travis/build/torproject/chutney/lib/chutney/TorNet.py", line 2702, in runConfigFile
    return getattr(network, verb)()
  File "/home/travis/build/torproject/chutney/lib/chutney/TorNet.py", line 2297, in configure
    self.create_new_nodes_dir()
  File "/home/travis/build/torproject/chutney/lib/chutney/TorNet.py", line 2269, in create_new_nodes_dir
    newnodesdir.symlink_to(nodeslink)
  File "/opt/python/3.8.1/lib/python3.8/pathlib.py", line 1376, in symlink_to
    self._accessor.symlink(target, self, target_is_directory)
  File "/opt/python/3.8.1/lib/python3.8/pathlib.py", line 444, in symlink
    return os.symlink(a, b)
FileExistsError: [Errno 17] File exists: '/home/travis/build/torproject/chutney/net/nodes' -&gt; '/home/travis/build/torproject/chutney/net/nodes.1594643565'

Any idea what's happening?

Fixed in 7460bf0d, I had Path.symlink_to() arguments mixed up

added Doing label

Much better -- https://travis-ci.org/github/torproject/chutney/builds/709317414 is the latest CI. The OSX failure looks like a fluke, but I want to be more sure about the other one. I've made a new (squashed) version at https://github.com/torproject/chutney/pull/80 to see if we get the same result.

Hm, looks like the failing test (bridges+hs-v23) seems to be failing in master too. I'll let this finish again, merge, then investigate.

mentioned in commit 3c1a744d

Merged at last!

closed

mentioned in issue #40012 (closed)