Get a fresh set of relay/exit profiles on 0.2.5.5-alpha or later; optimize bottlenecks if found
On legacy/trac#7727 (moved) we had some profile results that we used to identify bottleneck functions in Tor to optimize. We should get some fresh profiles on 0.2.5.4-alpha once it's out (assuming we merge legacy/trac#9841 (moved)), and then see which of those bottlenecks need the most attention.
Activity
changed milestone to %Tor: 0.2.5.x-final in legacy/trac
added 025-triaged in Legacy / Trac component::core tor/tor in Legacy / Trac milestone::Tor: 0.2.5.x-final in Legacy / Trac performance in Legacy / Trac priority::medium in Legacy / Trac resolution::fixed in Legacy / Trac status::closed in Legacy / Trac tor-relay in Legacy / Trac type::defect in Legacy / Trac labels
One thought I had about that last profile, though: SHA1 was at the top of the list. OpenSSL's PRNG uses a bunch of SHA1, and is ridiculously slow for a userspace PRNG. If SHA1 turns up at the top of the list again, we should investigate whether it's our protocols' uses of SHA1, TLS's uses of SHA1, or OpenSSL's PRNG's uses of SHA1 that are most to blame.
Trac:
Keywords: tor-relay performance deleted, tor-relay performance 025-triaged added-
Here are the maximally simple instructions as I understand them.
- Run Linux. Install perf. (On Debian, this is 'apt-get install linux-tools')
- (For call graphs only) Build Tor with -fno-omit-frame-pointer
- Start a Tor server with DisableDebuggerAttachment 0 set
- Wait for the server to get load
- Find the PID of the Tor process; call it PID.
- Run "perf record -p $PID"
- (To benchmark worker threads too) It might be a good idea to do "perf record -a -p $PID"
- (For call graphs only) Instead you might need something like "perf record --call-graph fp -a -g -p $PID"
- After a while (an hour or so?), ctrl-C the 'perf record' command. It will have written its results into perf.data.
- Run "perf report --stdio > results". Gzip that file and send it to nick.
- (For call graphs only) You might need "perf report --call-graph -G --stdio > results".
These instructions probably suck! Please help me improve them.
For the debian package it would probably be like so:
- cd /usr/src
- apt-get source tor
- cd tor-0.2.5*
- vim debian/rules
- append "CFLAGS=-fno-omit-frame-pointer"
- dpkg-buildpackage -B
Packages would then lie in /usr/src.
(Updated, according to legacy/trac#6 (closed))
-
What I most need is information about 0.2.5.4-alpha or later. 0.2.5.4-alpha has some performance improvements that should (I hope) affect the numbers a lot.
[But information about 0.2.5.2-alpha is still useful, because (a) it will let us know what stuff looked like before 0.2.5.4-alpha, and (b) it will help us figure out what the instructions for running perf should say.]
Trac:
Username: esbek-
Replying to nickm: ...
Was this version of Tor built without debugging symbols or something? It was built with this settings: ./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --libdir=/usr/lib64 --disable-silent-rules --disable-dependency-tracking --disable-buf-freelists --enable-asciidoc --docdir=/usr/share/doc/tor-0.2.5.4_alpha-r1 --enable-instrument-downloads --disable-bufferevents --enable-curve25519 --disable-nat-pmp --enable-gcc-hardening --enable-linker-hardening --disable-transparent --enable-threads --disable-upnp --disable-tor2web-mode --disable-unittests --disable-coverage
Trac:
Username: esbek -
Trac:
Username: esbek -
Trac:
Username: esbek -
My exit averages at about 32 Mbit/s. Not the > ~50 Mbit/s as requested on the mailinglist, but perhaps still useful? https://globe.torproject.org/#/relay/94E47A3E0A094C4B544B93D3247038243B61ACBC
I could use some help building with -fno-omit-frame-pointer as the instructions from both #comment:4 and #comment:5 don't seem to work for Ubuntu:
$ cat /etc/issue Ubuntu 12.04.4 LTS \n \l $ sudo apt-get update $ sudo apt-get install linux-tools $ sudo apt-get install build-essential $ wget https://www.torproject.org/dist/tor-0.2.5.4-alpha.tar.gz $ tar xvfz tor-0.2.5.4-alpha.tar.gz $ cd tor-0.2.5.4-alpha $ ./configure -fno-omit-frame-pointer configure: error: unrecognized option: `-fno-omit-frame-pointer' $ ls debian/rules (file not found)
Please advice!
Trac:
Username: yoriz -
Replying to cypherpunks:
Would love to give it a try. Can you clarify specific commands to build Tor? Is it
./configure -fno-omit-frame-pointer; make && make install ?
No; you'd have to try something like
./configure CFLAGS='-Wall -g -O2 -fno-omit-frame-pointer'
I think. But remember, the call-graph stuff is all optional.
-
Trac:
Username: esbek -
Replying to yoriz:
Nickm, thank you for pointing out the correct way to use configure. For those Ubuntu users finding this ticket later; these are the magic spells needed on a Ubuntu machine:
$ cat /etc/issue Ubuntu 12.04.4 LTS \n \l $ sudo apt-get update $ sudo apt-get install linux-tools $ sudo apt-get install build-essential $ sudo apt-get install libevent-dev $ sudo apt-get install libssl-dev $ wget https://www.torproject.org/dist/tor-0.2.5.4-alpha.tar.gz $ tar xvfz tor-0.2.5.4-alpha.tar.gz $ cd tor-0.2.5.4-alpha $ ./configure CFLAGS="-Wall -g -O2 -fno-omit-frame-pointer" $ make
Since I am operating a regular tor exit, I prefer to keep my normal
tor installation intact and keep this experimental build separate.
However, I will reuse the configuration of my regular node:
$ sudo mkdir -p /usr/local/etc/tor $ sudo cp /etc/tor/torrc /usr/local/etc/tor $ sudo ~/tor-0.2.5.4-alpha/src/or/tor $ arm (waiting for traffic to pick up again)
Trac:
Username: yoriz -
Trac:
Username: yoriz Trac:
results.gz-
Nice; I hadn't thought to also try grpof. I wonder if siphash has also started showing up under perf; it didn't show up to high under any of the other perf profiles, but those were before the changes in master that started using siphash for the circid/channel to circuit mappings (legacy/trac#11750 (moved)).
Also, those results won't cover time spent in openssl. Can you get perf results too under current master?
-
I added legacy/trac#12169 (moved) for optimizing memeq and and legacy/trac#12170 (moved) for making a lot of the performance issues (including siphash, I think!) go away.
Replying to nickm:
Nice; I hadn't thought to also try grpof. I wonder if siphash has also started showing up under perf; it didn't show up to high under any of the other perf profiles, but those were before the changes in master that started using siphash for the circid/channel to circuit mappings (legacy/trac#11750 (moved)).
Also, those results won't cover time spent in openssl. Can you get perf results too under current master?
Not without a fair amount of pain-in-the-assery. I think I need different kernel options for it to work.
Replying to nickm:
Nice; I hadn't thought to also try grpof. I wonder if siphash has also started showing up under perf; it didn't show up to high under any of the other perf profiles, but those were before the changes in master that started using siphash for the circid/channel to circuit mappings (legacy/trac#11750 (moved)).
Also, those results won't cover time spent in openssl. Can you get perf results too under current master?
I'd guess the easiest way to get OpenSSL results is to rebuild it with -pg and statically link to it.
-
Okay; I think we have enough info for 0.2.5.4-alpha. In 0.2.5.5-alpha and later, we should confirm that there are no new bottlenecks, and evaluate whether our fixes in legacy/trac#12170 (moved) and legacy/trac#12169 (moved) did any good.
Trac:
Summary: Get a fresh set of relay/exit profiles on 0.2.5.4-alpha or later; optimize accordingly. to Get a fresh set of relay/exit profiles on 0.2.5.5-alpha or later; optimize bottlenecks if found -
Neat. I don't see anything in there that seems like a terrible issue. There are a few one-percent functions in Tor that could probably turn into 0-percent functions, but nothing that looks like it's a bug.
Unless somebody else sees an issue in this profile, I say we call this fixed and open a new ticket to look at profiles in 0.2.6. :)
-
The ticket to do this again in 0.2.6 is legacy/trac#12464 (moved). Thanks, everyone!
Trac:
Status: new to closed
Resolution: N/A to fixed mentioned in issue legacy/trac#12169 (moved)
mentioned in issue legacy/trac#12170 (moved)
mentioned in issue legacy/trac#12464 (moved)