This procedure documents various benchmarking procedures in use inside TPA.
HTTP load testing
Those procedures were quickly established to compare various caching software as part of the cache service setup.
punch a hole in the firewall to allow the test server to access tested server, in case it is not public yet
iptables -I INPUT -s 184.108.40.206 -j ACCEPT ip6tables -I INPUT -s 2a01:4f8:c010:25ff::1 -j ACCEPT
point the test site (e.g.
blog.torproject.org) to the tested server on the test server, in
220.127.116.11 blog.torproject.org 2a01:4f8:fff0:4f:266:37ff:fe26:d6e1 blog.torproject.org
disable Puppet on the test server:
puppet agent --disable 'benchmarking requires /etc/hosts override'
launch the benchmark on the test server
Siege configuration sample:
verbose = false fullurl = true concurrent = 100 time = 2M url = http://www.example.com/ delay = 1 internet = false benchmark = true
Might require this, which might work only with varnish:
proxy-host = 18.104.22.168 proxy-port = 80
Alternative is to hack
ab2 -n 1000 -c 100 -X cache01.torproject.org https://example.com/
-X also doesn't work with ATS, modify
We tested bombardier as an alternative to
go-wrk in previous
benchmarks. The goal of using
go-wrk was that it supported HTTP/2
wrk didn't), but
go-wrk had performance issues, so we went
with the next best (and similar) thing.
Unfortunately, the bombardier package in Debian is not the HTTP benchmarking tool but a commandline game. It's still possible to install it in Debian with:
export GOPATH=$HOME/go apt install golang go get -v github.com/codesenberg/bombardier
Then running the benchmark is as simple as:
./go/bin/bombardier --duration=2m --latencies https://blog.torproject.org/
Note that wrk works similarly to
bombardier, sampled above, and has
the advantage of being already packaged in Debian. Simple cheat sheet:
sudo apt install wrk echo "10.0.0.0 target.example.com" >> /etc/hosts wrk --latency -c 100 --duration 2m https://target.example.com/
The main disadvantage is that it doesn't (seem to) support HTTP/2 or similarly advanced protocols.
Siege has trouble going above ~100 concurrent clients because of its design (and ulimit) limitations. Its interactive features are also limited, here's a set of interesting alternatives:
|ali||golang||HTTP/2||real-time graph, duration, mouse support||unsearchable name||no|
|bombardier||golang||HTTP/2||better performance than siege in my 2017 tests||RFP)|
|boom||Python||HTTP/2||duration||rewrite of apachebench, unsearchable name||no|
|go-wrk||golang||no duration||rewrite of wrk, performance issues in my 2017 tests||no|
|hey||golang||rewrite of apachebench, similar to boom, unsearchable name||ITP|
|Jmeter||Java||interactive, session replay||yes|
|k6.io||JMeter rewrite with "cloud" SaaS||no|
|Locust||distributed, interactive behavior||ITP|
|oha||Rust||TUI||inspired by hey||no|
|wrk||C||multithreaded, epoll, Lua scriptable||yes|
Note that the Proto(col) and Features columns are not exhaustive: a tool might support (say) HTTPS, HTTP/2, or HTTP/3 even if it doesn't explicitly mention it, although it's unlikely.
It should be noted that very few (if any) benchmarking tools seem to
support HTTP/3 (or even QUIC) at this point. Even HTTP/2 support is
spotty: for example, while bombardier supports HTTP/2, it only does so
with the slower
net/http library at the time of writing (2021). It's
unclear how many (if any) other projects to support HTTP/2 as well.