This procedure documents various benchmarking procedures in use inside
TPA.

[[_TOC_]]

# HTTP load testing

Those procedures were quickly established to compare various caching
software as part of the [cache service](howto/cache) setup.

## Common procedure

 1. punch a hole in the firewall to allow the test server to access
    tested server, in case it is not public yet

        iptables -I INPUT -s 78.47.61.104 -j ACCEPT
        ip6tables -I INPUT -s 2a01:4f8:c010:25ff::1 -j ACCEPT

 2. point the test site (e.g. `blog.torproject.org`) to the tested
    server on the test server, in `/etc/hosts`:

        116.202.120.172	blog.torproject.org
        2a01:4f8:fff0:4f:266:37ff:fe26:d6e1 blog.torproject.org

 3. disable Puppet on the test server:
 
        puppet agent --disable 'benchmarking requires /etc/hosts override'

 4. launch the benchmark on the test server

## Siege

Siege configuration sample:

```
verbose = false
fullurl = true
concurrent = 100
time = 2M
url = http://www.example.com/
delay = 1
internet = false
benchmark = true
```

Might require this, which might work only with varnish:

```
proxy-host = 209.44.112.101
proxy-port = 80
```

Alternative is to hack `/etc/hosts`.

## apachebench

Classic commandline:

    ab2 -n 1000 -c 100 -X cache01.torproject.org https://example.com/

`-X` also doesn't work with ATS, modify `/etc/hosts` instead.

## bombardier

We tested bombardier as an alternative to `go-wrk` in previous
benchmarks. The goal of using `go-wrk` was that it supported HTTP/2
(while `wrk` didn't), but `go-wrk` had performance issues, so we went
with the next best (and similar) thing.

Unfortunately, the [bombardier package in Debian](https://tracker.debian.org/pkg/bombardier) is *not* the HTTP
benchmarking tool but a commandline game. It's still possible to
install it in Debian with:

    export GOPATH=$HOME/go
    apt install golang
    go get -v github.com/codesenberg/bombardier

Then running the benchmark is as simple as:

    ./go/bin/bombardier --duration=2m --latencies https://blog.torproject.org/

## wrk

Note that wrk works similarly to `bombardier`, sampled above, and has
the advantage of being already packaged in Debian. Simple cheat sheet:

    sudo apt install wrk
    echo "10.0.0.0 target.example.com" >> /etc/hosts
    wrk --latency -c 100 --duration 2m https://target.example.com/

The main disadvantage is that it doesn't (seem to) support HTTP/2 or
similarly advanced protocols.

## Other tools

Siege has trouble going above ~100 concurrent clients because of its
design (and ulimit) limitations. Its interactive features are also
limited, here's a set of interesting alternatives:

| Project        | Lang   | Proto  | Features                                 | Notes                                                      | Debian   |
|----------------|--------|--------|------------------------------------------|------------------------------------------------------------|----------|
| [ali][]        | golang | HTTP/2 | real-time graph, duration, mouse support | unsearchable name                                          | no       |
| [bombardier][] | golang | HTTP/2 |                                          | better performance than siege in my 2017 tests             | [RFP][]) |
| [boom][]       | Python | HTTP/2 | duration                                 | rewrite of apachebench, unsearchable name                  | no       |
| [go-wrk][]     | golang |        | [no duration][]                          | rewrite of wrk, performance issues in my 2017 tests        | no       |
| [hey][]        | golang |        |                                          | rewrite of apachebench, similar to boom, unsearchable name | [ITP][ITP 943596] |
| [Jmeter][]     | Java   |        | interactive, session replay              |                                                            | yes      |
| [k6.io][]      |        |        |                                          | JMeter rewrite with "cloud" SaaS                           | no       |
| [Locust][]     |        |        | distributed, interactive behavior        |                                                            | [ITP][ITP 982508] |
| [oha][]        | Rust   |        | TUI                                      | inspired by hey                                            | no       |
| [Tsung][]      | Erlang | multi  | distributed                              |                                                            | yes      |
| [wrk][]        | C      |        | multithreaded, epoll, Lua scriptable     |                                                            | yes      |

[ali]: https://github.com/nakabonne/ali
[bombardier]: https://github.com/codesenberg/bombardier
[boom]: https://github.com/tarekziade/boom
[go-wrk]: https://github.com/adjust/go-wrk/
[hey]: https://github.com/rakyll/hey
[ITP 943596]: https://bugs.debian.org/943596
[ITP 982508]: https://bugs.debian.org/982508
[Jmeter]: https://jmeter.apache.org/
[k6.io]: https://k6.io/
[Locust]: https://locust.io/
[no duration]: https://github.com/adjust/go-wrk/issues/2
[oha]: https://github.com/hatoo/oha
[RFP]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=943733
[Tsung]: http://tsung.erlang-projects.org/1/01/about/
[wrk]: https://github.com/wg/wrk/

Note that the Proto(col) and Features columns are not exhaustive: a
tool *might* support (say) HTTPS, HTTP/2, or HTTP/3 even if it doesn't
explicitly mention it, although it's unlikely.

It should be noted that very few (if any) benchmarking tools seem to
support HTTP/3 (or even QUIC) at this point. Even HTTP/2 support is
spotty: for example, while bombardier supports HTTP/2, it only does so
with the slower `net/http` library at the time of writing (2021). It's
unclear how many (if any) other projects to support HTTP/2 as well.
