Tests fail on Gentoo

added component::archived/stem owner::atagar priority::medium resolution::fixed status::closed testing type::defect labels

Interesting! Unit tests should be deterministic, but guess there's a timing issue here.

Thanks! That was definitely a bug. The Reader was supposed to run into malformed descriptor content and did, but rather than raise a ValueError this bug caused it to raise a AttributeError. Fixed...

https://gitweb.torproject.org/stem.git/commit/?id=90bb28f44929bcb33b321eb51a27bb247c4ae6ab

Trac:
Status: new to closed
Resolution: N/A to fixed

Trac:
net-libs_stem-1.2.2_p20141206_20141206-105919.log.gz

test output

so, tests do now finished, however at a hardened Gentoo Linux few tests do fail.

Interesting! This is indicating that a recent-ish change to use the socket module to validate ip addresses is failing on your system for some reason. What happens if you open a python interpretor and do the following?

>>> import socket

>>> socket.inet_pton(socket.AF_INET6, '1::1')
'\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01'

I expect in your case it'll raise a socket.error of some kind. Little curious about that (... these failures seem to indicate your system doesn't know what an IPv6 address looks like).

Pity. Thought we could rely on the python builtins for this but maybe not.

Trac:
Status: closed to reopened
Resolution: fixed to N/A

Just pushed a change that should fix the tests. Please let me know if they pass or not, and thanks for the catch!

https://gitweb.torproject.org/stem.git/commit/?id=91f6c3bb2a5a4f4bb9a822f1280a4c250aeacdcd

Replying to atagar:

Interesting! This is indicating that a recent-ish change to use the socket module to validate ip addresses is failing on your system for some reason. What happens if you open a python interpretor and do the following?
>>> import socket

>>> socket.inet_pton(socket.AF_INET6, '1::1')
'\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01'
}}}

{{{ tfoerste@tor-relay ~ $ python2 Python 2.7.7 (default, Oct 25 2014, 11:03:42) [GCC 4.8.3] on linux2 Type "help", "copyright", "credits" or "license" for more information.

import socket socket.inet_pton(socket.AF_INET6, '1::1') Traceback (most recent call last): File "", line 1, in socket.error: can't use AF_INET6, IPv6 is disabled quit Use quit() or Ctrl-D (i.e. EOF) to exit

Gotcha. And do the tests now pass?

Replying to atagar:

Just pushed a change that should fix the tests. Please let me know if they pass or not, and thanks for the catch!

https://gitweb.torproject.org/stem.git/commit/?id=91f6c3bb2a5a4f4bb9a822f1280a4c250aeacdcd

yes, latest git gives now :

Shutting down tor... done

Static error checking requires pyflakes version 0.7.3 or later. Please install it from ...
  http://pypi.python.org/pypi/pyflakes

Style checks require pep8 version 1.4.2 or later. Please install it from...
  http://pypi.python.org/pypi/pep8

TESTING FAILED (202 seconds)
  [RUN_PTRACE] test_get_connections (test.integ.util.connection.TestConnection) ... ERROR
  [RUN_PTRACE] test_connections (test.integ.util.proc.TestProc) ... ERROR
  [RUN_PTRACE] test_pid_by_port (test.integ.util.system.TestSystem) ... FAIL
  [RUN_PTRACE] test_pid_by_port_lsof (test.integ.util.system.TestSystem) ... ERROR
  [RUN_PTRACE] test_pid_by_port_netstat (test.integ.util.system.TestSystem) ... ERROR
 * ERROR: net-libs/stem-1.2.2_p20141207::toralf failed (test phase):

Great, thanks!

  [RUN_PTRACE] test_get_connections (test.integ.util.connection.TestConnection) ... ERROR
  [RUN_PTRACE] test_connections (test.integ.util.proc.TestProc) ... ERROR
  [RUN_PTRACE] test_pid_by_port (test.integ.util.system.TestSystem) ... FAIL
  [RUN_PTRACE] test_pid_by_port_lsof (test.integ.util.system.TestSystem) ... ERROR
  [RUN_PTRACE] test_pid_by_port_netstat (test.integ.util.system.TestSystem) ... ERROR
 * ERROR: net-libs/stem-1.2.2_p20141207::toralf failed (test phase):

Interesting, but in looking at the earlier output I suppose this shouldn't be surprising...

======================================================================
ERROR: test_get_connections
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/tmp/portage/net-libs/stem-1.2.2_p20141206/work/stem-1.2.2_p20141206/test/integ/util/connection.py", line 27, in test_get_connections
    connections = get_connections(resolver, process_pid = tor_pid)
  File "/var/tmp/portage/net-libs/stem-1.2.2_p20141206/work/stem-1.2.2_p20141206/stem/util/connection.py", line 178, in get_connections
    return [Connection(*conn) for conn in stem.util.proc.connections(process_pid)]
  File "/var/tmp/portage/net-libs/stem-1.2.2_p20141206/work/stem-1.2.2_p20141206/stem/util/proc.py", line 404, in connections
    raise exc
IOError: unable to read '/proc/net/tcp': [Errno 13] Permission denied: '/proc/net/tcp'

Other errors are due to netstat and lsof being broken, which is understandable given the hardening. If you have an 'is this system hardened' check then we can add a clause to skip the tests but otherwise it sounds like there's no further issues. Feel free to reopen if there's anything else!

Trac:
Status: reopened to closed
Resolution: N/A to fixed

Hi toralf, this ticket got me thinking about Gentoo and I was delighted to realize Anthony's packaged it for the distro. Added Gentoo to our download page...

https://stem.torproject.org/download.html

I spotted the Gentoo ticket I suspect you might be looking into...

https://bugs.gentoo.org/show_bug.cgi?id=527072

Interesting. Made a couple more fixes for it...

https://gitweb.torproject.org/stem.git/commit/?id=2ee4e95ab0f2640674c1074ce26e2f09ce9bdf26 https://gitweb.torproject.org/stem.git/commit/?id=4e2473d98c88399d1fd00e4c92ae215c092188b7

However, the PTRACE target failures you mention remain. Probably the best way of addressing those is to simply skip those test on Gentoo. This requires an is_gentoo() method like...

https://gitweb.torproject.org/stem.git/tree/stem/util/system.py#n124

What does platform.system() provide for you? If it's not something distinct then is there something else you'd suggest?

Trac:
Status: closed to reopened
Summary: unit tests do hang to Tests fail on Gentoo
Resolution: fixed to N/A

Trac:
Keywords: N/A deleted, testing added

Hi,

thx for adding Gentoo to the wiki. Gentoo might be detected in this way :

$ cat /etc/gentoo-release Gentoo Base System release 2.2

In this case the sub-architecture is specified via :

$ ls -l /etc/portage/make.profile lrwxrwxrwx 1 root root 47 Oct 24 22:32 /etc/portage/make.profile -> ../../usr/portage/profiles/hardened/linux/amd64

another example would be: $ ls -l /etc/portage/make.profile lrwxrwxrwx 1 root root 61 Jan 19 2013 /etc/portage/make.profile -> ../../usr/portage/profiles/default/linux/x86/13.0/desktop/kde

test results of latest git will follow soon

Trac:
net-libs_stem-1.2.2_p20141208_20141207-110956.log.gz

test results for 4e2473d98c88399d1fd00e4c92ae215c092188b7

$ cat /etc/gentoo-release

Perfect! Thanks.

test results of latest git will follow soon

I wasn't expecting the PTRACE results to change (the fixes I mentioned above were for the Gentoo ticket, which was about test failures due to lacking network access). However, I just pushed some changes for the PTRACE tests.

util.proc... failed (0.01s) util.system... failed (0.38s)

These should now pass. We now check for proc read permissions and skip certain tests on Gentoo.

util.connection... failed (0.01s)

This one, however, is a bit more interesting. You should no longer get that proc error, but I'd like to know if all resolvers are broken on Gentoo or just that one. I changed the test so it'll run each resolver separately.

Would you mind re-running the tests?

Thanks! -Damian

HHm, it hangs here now : {{{ Setting up a test instance... making test directory (/var/tmp/portage/net-libs/stem-1.2.2_p20141210/work/stem-1.2.2_p20141210/test/data)... skipped making control socket directory (/var/tmp/portage/net-libs/stem-1.2.2_p20141210/temp/793314d6-6415-4f55-9e41-249e56e26869)... done configuring logger (/var/tmp/portage/net-libs/stem-1.2.2_p20141210/work/stem-1.2.2_p20141210/test/data/log)... done writing torrc (/var/tmp/portage/net-libs/stem-1.2.2_p20141210/work/stem-1.2.2_p20141210/test/data/torrc)... done # configuration for stem integration tests DataDirectory /var/tmp/portage/net-libs/stem-1.2.2_p20141210/work/stem-1.2.2_p20141210/test/data SocksListenAddress 127.0.0.1:1112 DownloadExtraInfo 1 Log notice stdout Log notice file /var/tmp/portage/net-libs/stem-1.2.2_p20141210/work/stem-1.2.2_p20141210/test/data/tor_log ControlSocket /var/tmp/portage/net-libs/stem-1.2.2_p20141210/temp/793314d6-6415-4f55-9e41-249e56e26869/socket CookieAuthentication 1

Starting tor...

Dec 10 15:42:26.981 [notice] Tor v0.2.5.10 (git-42b42605f8d8eac2) running on Linux with Libevent 2.0.21-stable, OpenSSL 1.0.1j and Zlib 1.2.8. Dec 10 15:42:26.981 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning Dec 10 15:42:26.982 [notice] Read configuration file "/var/tmp/portage/net-libs/stem-1.2.2_p20141210/work/stem-1.2.2_p20141210/test/data/torrc". Dec 10 15:42:26.993 [notice] Opening Socks listener on 127.0.0.1:1112 Dec 10 15:42:26.993 [notice] Opening Control listener on /var/tmp/portage/net-libs/stem-1.2.2_p20141210/temp/793314d6-6415-4f55-9e41-249e56e26869/socket Dec 10 15:42:26.000 [notice] Parsing GEOIP IPv4 file /usr/share/tor/geoip. Dec 10 15:42:27.000 [notice] Parsing GEOIP IPv6 file /usr/share/tor/geoip6. Dec 10 15:42:27.000 [notice] Bootstrapped 0%: Starting Dec 10 15:42:27.000 [notice] Bootstrapped 5%: Connecting to directory server done (0 seconds)

Running tests...

util.conf... success (0.00s) util.connection... success (0.00s) util.proc... success (0.01s) util.system... success (0.37s) descriptor.remote... success (0.00s) descriptor.server_descriptor... success (0.00s) descriptor.extrainfo_descriptor... success (0.00s) descriptor.microdescriptor... success (0.00s) descriptor.networkstatus... success (0.00s) version... success (0.03s) response.protocolinfo... success (0.07s) process... success (0.00s) socket.control_socket... success (0.30s) socket.control_message... success (1.21s) connection.authentication... success (0.14s) connection.connect... success (0.01s) control.base_controller... success (5.21s) control.controller...

}}}and the log file /var/tmp/portage/net-libs/stem-1.2.2_p20141210/work/stem-1.2.2_p20141210/test/data/tor_log has last lines :

{{{ Dec 10 15:42:40.000 [warn] Hidden service ("test_hidden_service_empty/") with no ports configured; ignoring. Dec 10 15:42:40.000 [notice] Tor 0.2.5.10 (git-42b42605f8d8eac2) opening log file. Dec 10 15:42:40.000 [warn] Hidden service ("test_hidden_service_empty/") with no ports configured; ignoring. Dec 10 15:42:40.000 [notice] New control connection opened. Dec 10 15:42:40.000 [notice] New control connection opened. Dec 10 15:42:40.000 [warn] Controller gave us config file that didn't validate: Failed to parse/validate config: While Tor is running, changing DataDirectory ("/var/tmp/portage/net-libs/stem-1.2.2_p20141210/work/stem-1.2.2_p20141210/test/data"->"/var/tmp/portage/net-libs/stem-1.2.2_p20141210/homedir/.tor") is not allowed. Dec 10 15:42:40.000 [warn] Controller gave us config file that didn't validate: Failed to parse/validate config: Unknown option 'Blahblah'. Failing. Dec 10 15:42:40.000 [notice] Tor 0.2.5.10 (git-42b42605f8d8eac2) opening log file. Dec 10 15:42:40.000 [notice] Tor 0.2.5.10 (git-42b42605f8d8eac2) opening log file. Dec 10 15:42:40.000 [notice] Tor 0.2.5.10 (git-42b42605f8d8eac2) opening log file. Dec 10 15:42:40.000 [notice] New control connection opened. Dec 10 15:42:40.000 [notice] New control connection opened. Dec 10 15:42:40.000 [notice] New control connection opened. Dec 10 15:42:40.000 [notice] New control connection opened. Dec 10 15:42:43.000 [notice] New control connection opened. Dec 10 15:42:44.000 [notice] New control connection opened. Dec 10 16:00:46.000 [warn] Received http status code 404 ("Not found") from server '146.0.42.110:25' while fetching "/tor/keys/fp/27B6B5996C426270A5C95488AA5BCEB6BCC86956". }}}

Well, that's unexpected. Does it hang consistently or was this a one-off? Please give run_tests.py the --verbose flag - maybe that'll shed a hint.

right, was a one-time thing, now tests passes with

Shutting down tor... done

Static error checking requires pyflakes version 0.7.3 or later. Please install it from ...
  http://pypi.python.org/pypi/pyflakes

Style checks require pep8 version 1.4.2 or later. Please install it from...
  http://pypi.python.org/pypi/pep8

TESTING FAILED (171 seconds)
  [RUN_PTRACE] test_get_connections_by_lsof (test.integ.util.connection.TestConnection) ... ERROR
  [RUN_PTRACE] test_get_connections_by_netstat (test.integ.util.connection.TestConnection) ... ERROR
  [RUN_PTRACE] test_get_connections_by_proc (test.integ.util.connection.TestConnection) ... ERROR
  [RUN_PTRACE] test_get_connections_by_ss (test.integ.util.connection.TestConnection) ... ERROR
  [RUN_PTRACE] test_connections (test.integ.util.proc.TestProc) ... ERROR
 * ERROR: net-libs/stem-1.2.2_p20141210::toralf failed (test phase):
 *   (no error message)

with kernel 3.17.6-hardened

Well drats. Looks like Gentoo doesn't have any connection resolvers at all then. That's too bad. Made Stem recognize that.

This should fix the test_get_connections_by_* with the exception of test_get_connections_by_proc. That one's indicating that my earlier fix of checking proc permissions isn't working. What are your permissions on /proc/net/tcp and /proc/net/udp? If you run the following what happens?

>>> import os
>>> os.access('/proc/net/tcp', os.R_OK)
True
>>> os.access('/proc/net/udp', os.R_OK)
True

I do get :

tfoerste@tor-relay ~ $ python2
Python 2.7.7 (default, Oct 25 2014, 11:03:42) 
[GCC 4.8.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.access('/proc/net/tcp', os.R_OK)
True
>>> os.access('/proc/net/udp', os.R_OK)
True
>>> 
tfoerste@tor-relay ~ $ ls -l /proc/net/tcp /proc/net/udp
-r--r--r-- 1 root wheel 0 Dec 10 19:37 /proc/net/tcp
-r--r--r-- 1 root wheel 0 Dec 10 19:37 /proc/net/udp
```You might however consider, that it is a hardened Gentoo Linux (kernel is patched with GRsecurity)

Huh. So the filesystem says we have read permissions but actually reading from it fails with permission denied?

IOError: unable to read '/proc/net/tcp': [Errno 13] Permission denied: '/proc/net/tcp'

Sounds like a GRsecurity bug to me. Oh well, I can add a special 'if Gentoo don't believe the file permissions' check. :(

Presently at work. I'll make it so tomorrow morning. Thanks for checking!

Fix pushed, the tests should now pass. Can you confirm?

yeah - good progress, no I just got this :

Shutting down tor... done

Static error checking requires pyflakes version 0.7.3 or later. Please install it from ...
  http://pypi.python.org/pypi/pyflakes

Style checks require pep8 version 1.4.2 or later. Please install it from...
  http://pypi.python.org/pypi/pep8

TESTING FAILED (159 seconds)
  [UNIT TEST] test_system_resolvers (test.unit.util.connection.TestConnection) ... FAIL

which points to this

======================================================================
FAIL: test_system_resolvers
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/mock.py", line 1210, in patched
    return func(*args, **keywargs)
  File "/var/tmp/portage/net-libs/stem-1.2.2_p20141211/work/stem-1.2.2_p20141211/test/unit/util/connection.py", line 121, in test_system_resolvers
    self.assertEqual([Resolver.BSD_SOCKSTAT, Resolver.BSD_PROCSTAT, Resolver.LSOF], stem.util.connection.system_resolvers('FreeBSD'))
AssertionError: Lists differ: ['sockstat (bsd)', 'procstat (... != []

First list contains 3 additional elements.
First extra element 0:
sockstat (bsd)

- ['sockstat (bsd)', 'procstat (bsd)', 'lsof']
+ []

BTW, in the git commit message you shouldn't blame Gentoo. It is the hardened kernel here (grsecurity in this case), which restricts capabilities (I'm convinced, it makes sense to run a hardened kernel, especially if somebody runs a tor relay)

And FWIW a needed a 2nd attempt, the first attempt to run the test failed, b/c the test suite hung at " control.controller..."

yeah - good progress, no I just got this

Oops! Mistake on my part - fixed. Does it now work?

It is the hardened kernel here (grsecurity in this case), which restricts capabilities (I'm convinced, it makes sense to run a hardened kernel, especially if somebody runs a tor relay)

I'd argue it's a bug with grsecurity. If a process lacks read permissions for a resource then it should reflect that in the filesystem permissions. Grsecurity looks to be lying to processes by saying 'You totally have read access for the proc resources! But... you can't read the proc resources.'

This is my first time running into grsecurity which is why it's conflating with Gentoo for me. On reflection these 'is_gentoo()' checks should really be 'is_grsecurity()' - is there an easy method for detecting it?

is there an easy method for detecting it?

I fear not, b/c grsecurity prevents most of those methods :-D

But a quick&dirty test could be -for the moment- : $> uname -a | grep hardened Linux t44 3.17.6-hardened #7 (closed) SMP Fri Dec 12 12:35:58 CET 2014 x86_64 Intel(R) Core(TM) i5-4300U CPU @ 1.90GHz GenuineIntel GNU/Linux

I fear not, b/c grsecurity prevents most of those methods :-D

https://www.youtube.com/watch?v=wRnSnfiUI54

But a quick&dirty test could be -for the moment

Hmmm... this makes me uncomfortable since looking for the string 'hardened' could have false positives, in turn disabling proc resolution erroneously. I'm a little tempted to revert the 'is_gentoo()' checks for proc so when running without grsec Gentoo users will have that feature. But in looking at the other ticket's test output they had the same so guess I'll leave this as-is for now.

Presently connection resolution on Gentoo is disabled and the tests should be passing. Feel free to reopen if we should change anything else.

Thanks for the help!

Trac:
Resolution: N/A to fixed
Status: reopened to closed

Just one question: Was just the test case modified ? Because with the latest installed stem at that system gives now no resolver at all (worked before):

tfoerste@tor-relay ~ $ sudo python
Python 2.7.7 (default, Oct 25 2014, 11:03:42) 
[GCC 4.8.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> from stem.control import Controller
>>> from stem.util.connection import get_connections, system_resolvers
>>> resolvers = system_resolvers()
>>> print len(resolvers)
0
>>> quit
Use quit() or Ctrl-D (i.e. EOF) to exit
>>>

Was just the test case modified ? Because with the latest installed stem at that system gives now no resolver at all (worked before)

Nope, that's the change I made. The tests indicated that every connection resolver was failing on your platform so I changed system_resolvers() to reflect that none are available due to grsecurity (... actually, the check it's doing is 'none on Gentoo' which is why I asked about a better check).

And I made a painful braino b/c Gentoo run the tests under the package manager user id and not under root. But root is needed for a hardened Gentoo, b/c grsecurity restricts a lot of common functionality to just user root).

Oh well, so my delivered results wrt the resolver functionality (starting at #comment 17) are wrong ...

Said this, I run again the tests again the older git status and got this (now the tests run as user root) :

Shutting down tor... done

Static error checking requires pyflakes version 0.7.3 or later. Please install it from ...
  http://pypi.python.org/pypi/pyflakes

Style checks require pep8 version 1.4.2 or later. Please install it from...
  http://pypi.python.org/pypi/pep8

412 TESTS WERE SKIPPED
TESTING PASSED (208 seconds)

>>> Completed testing net-libs/stem-1.2.2_p20141207

tor-relay /usr/local/portage/net-libs/stem # grep ID stem-1.2.2_p20141207.ebuild 
COMMIT_ID="4e2473d98c88399d1fd00e4c92ae215c092188b7"
SRC_URI="https://gitweb.torproject.org/stem.git/snapshot/${COMMIT_ID}.tar.gz -> ${P}.tar.gz"

Ok. How would you care to proceed? As I see it we have three options...

Have system_resolvers() report that no resolvers are available on Gentoo. Advantage is that tests pass, but disadvantage is that it's wrong if you're running as root or without grsec.
Have system_resolvers() treat Gentoo like any other Linux distro as we did earlier. Tests will fail if run under grsecurity.
Change our check to be 'if running under grsecurity and we're not root'. This is the best solution but requires us to have a good method for checking 'are we running with grsecurity?'.

Just talked with guys in #gentoo-hardened and #grsecurity

There are much more security solutions flying around which let a test fail if the test case isn't run under an authorized user.

So in moment 2. would be a robust solution, a perfect solution would require more effort than it would be worth.

Sounds good, made it so.

installed that commit id at the relay - works fine.

thx

Great - thanks for the help!

closed

Tests fail on Gentoo

Child items 0

Activity