benchmark.md

This procedure documents various benchmarking procedures in use inside TPA.

• HTTP load testing
  • Common procedure
  • Siege
  • apachebench
  • bombardier
  • wrk
  • Other tools

HTTP load testing

Those procedures were quickly established to compare various caching software as part of the cache service setup.

Common procedure

1. punch a hole in the firewall to allow the test server to access tested server, in case it is not public yet

   iptables -I INPUT -s 78.47.61.104 -j ACCEPT
   ip6tables -I INPUT -s 2a01:4f8:c010:25ff::1 -j ACCEPT

2. point the test site (e.g. blog.torproject.org) to the tested server on the test server, in /etc/hosts:

   116.202.120.172	blog.torproject.org
   2a01:4f8:fff0:4f:266:37ff:fe26:d6e1 blog.torproject.org

3. disable Puppet on the test server:

   puppet agent --disable 'benchmarking requires /etc/hosts override'

4. launch the benchmark on the test server

Siege

Siege configuration sample:

verbose = false
fullurl = true
concurrent = 100
time = 2M
url = http://www.example.com/
delay = 1
internet = false
benchmark = true

Might require this, which might work only with varnish:

proxy-host = 209.44.112.101
proxy-port = 80

Alternative is to hack /etc/hosts.

apachebench

Classic commandline:

ab2 -n 1000 -c 100 -X cache01.torproject.org https://example.com/

-X also doesn't work with ATS, modify /etc/hosts instead.

bombardier

We tested bombardier as an alternative to go-wrk in previous benchmarks. The goal of using go-wrk was that it supported HTTP/2 (while wrk didn't), but go-wrk had performance issues, so we went with the next best (and similar) thing.

Unfortunately, the bombardier package in Debian is not the HTTP benchmarking tool but a commandline game. It's still possible to install it in Debian with:

export GOPATH=$HOME/go
apt install golang
go get -v github.com/codesenberg/bombardier

Then running the benchmark is as simple as:

./go/bin/bombardier --duration=2m --latencies https://blog.torproject.org/

wrk

Note that wrk works similarly to bombardier, sampled above, and has the advantage of being already packaged in Debian. Simple cheat sheet:

sudo apt install wrk
echo "10.0.0.0 target.example.com" >> /etc/hosts
wrk --latency -c 100 --duration 2m https://target.example.com/

The main disadvantage is that it doesn't (seem to) support HTTP/2 or similarly advanced protocols.

Other tools

Siege has trouble going above ~100 concurrent clients because of its design (and ulimit) limitations. Its interactive features are also limited, here's a set of interesting alternatives:

Project	Lang	Proto	Features	Notes	Debian
ali	golang	HTTP/2	real-time graph, duration, mouse support	unsearchable name	no
bombardier	golang	HTTP/2		better performance than siege in my 2017 tests	RFP)
boom	Python	HTTP/2	duration	rewrite of apachebench, unsearchable name	no
go-wrk	golang		no duration	rewrite of wrk, performance issues in my 2017 tests	no
hey	golang			rewrite of apachebench, similar to boom, unsearchable name	ITP
Jmeter	Java		interactive, session replay		yes
k6.io				JMeter rewrite with "cloud" SaaS	no
Locust			distributed, interactive behavior		ITP
oha	Rust		TUI	inspired by hey	no
Tsung	Erlang	multi	distributed		yes
wrk	C		multithreaded, epoll, Lua scriptable		yes

Note that the Proto(col) and Features columns are not exhaustive: a tool might support (say) HTTPS, HTTP/2, or HTTP/3 even if it doesn't explicitly mention it, although it's unlikely.