setup web-chi-03 full mirror in gnt-chi

added Doing label

comment from #40100 (closed), from 16:22UTC:

#40131 (closed) has unblocked enough for me to bootstrap web-chi-03. the web mirror stuff still needs to be done, but the LDAP, puppet, and ganeti parts are complete.

from 21:08UTC:

web-chi-03 is syncing the entire mirror from the static master, which could take a while. i'll look at putting it online next week.

adding setup steps in the issue summary

marked the checklist item iSCSI tests (#40131 (closed)) as completed

marked the checklist item VM setup as completed

marked the checklist item puppet as completed

marked the checklist item ldap as completed

marked the checklist item nagios as completed

marked the checklist item first mirror sync as completed

mentioned in issue #40100 (closed)

changed the description

gnt-fsn benchmarks

note that, at the time of benchmark:

root@fsn-node-01:~# gnt-instance list | grep -e perdulce -e web-fsn
perdulce.torproject.org             kvm        debootstrap+default fsn-node-03.torproject.org running      1.0G
web-fsn-01.torproject.org           kvm        debootstrap+buster  fsn-node-01.torproject.org running      4.0G
web-fsn-02.torproject.org           kvm        debootstrap+buster  fsn-node-04.torproject.org running      4.0G

so no "local" traffic.

/etc/hosts was modified to point to the right host, over IPv4.

web-fsn-02

root@perdulce:~# wrk --latency -c 100 --duration 2m https://www.torproject.org/ 
Running 2m test @ https://www.torproject.org/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    34.02ms   26.61ms 588.66ms   74.98%
    Req/Sec     1.60k   222.64     2.27k    70.25%
  Latency Distribution
     50%   30.59ms
     75%   43.70ms
     90%   63.29ms
     99%  125.53ms
  382610 requests in 2.00m, 6.78GB read
  Socket errors: connect 0, read 40, write 0, timeout 0
Requests/sec:   3187.69
Transfer/sec:     57.87MB

web-fsn-01

root@perdulce:~# wrk --latency -c 100 --duration 2m https://www.torproject.org/ 
Running 2m test @ https://www.torproject.org/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    29.66ms   22.97ms 558.90ms   76.64%
    Req/Sec     1.81k   262.44     2.67k    77.20%
  Latency Distribution
     50%   27.30ms
     75%   36.86ms
     90%   52.09ms
     99%  104.65ms
  432381 requests in 2.00m, 7.67GB read
  Socket errors: connect 0, read 28, write 0, timeout 0
Requests/sec:   3602.74
Transfer/sec:     65.40MB

benchmark gnt-chi

root@chi-node-01:~# gnt-instance list | grep -e web-chi -e test-anarcat
Instance                       Hypervisor OS                 Primary_node               Status  Memory
test-anarcat-01.torproject.org kvm        debootstrap+buster chi-node-04.torproject.org running   8.0G
web-chi-03.torproject.org      kvm        debootstrap+buster chi-node-02.torproject.org running   8.0G

so no "local" traffic either.

/etc/hosts was modified to point to the IPv4 address of web-chi-03.

web-chi-03 benchmark

root@test-anarcat-01:~# wrk --latency -c 100 --duration 2m https://www.torproject.org/
Running 2m test @ https://www.torproject.org/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    62.58ms   40.99ms   1.11s    91.89%
    Req/Sec   828.05    101.69     1.28k    88.21%
  Latency Distribution
     50%   56.90ms
     75%   71.97ms
     90%   91.73ms
     99%  164.85ms
  197382 requests in 2.00m, 3.50GB read
Requests/sec:   1644.09
Transfer/sec:     29.85MB

benchmark on moly

web-cymru-01 bench

interstingly, this seems to be the fastest of all!

root@peninsulare:~# wrk --latency -c 100 --duration 2m https://www.torproject.org/
Running 2m test @ https://www.torproject.org/
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    26.15ms   21.15ms 287.22ms   80.62%
    Req/Sec     2.13k   343.50     3.48k    68.27%
  Latency Distribution
     50%   21.34ms
     75%   33.94ms
     90%   51.38ms
     99%  104.08ms
  507961 requests in 2.00m, 417.18MB read
  Socket errors: connect 0, read 56, write 0, timeout 0
Requests/sec:   4231.11
Transfer/sec:      3.47MB

observations

web-fsn-XX nodes have comparable performance (~3k rps, ~60MB/s, aka ~500mbps)
web-chi-03 has about half that (~1.5k rps, 30MB/s, AKA ~250mbps)
latency is similarly higher
web-cymru-01 is faster than the others, possibly because it doesn't have the limitations of gnt-fsn (openvswitch network, DRBD) or gnt-chi (iSCSI storage backend)

so there's a significantly lower performance here, compared to the other locations. i suspect that iSCSI is at cause. the question is whether we'll be able to survive with this. considering that we're getting about 200-300mbps on the other nodes, I'd be tempted to say "yes, but not much more": the extra node will spread the load around which will mean lower traffic for everyone, but will be less capable of handling load spikes.

so let's just do it.

marked the checklist item full mirror sync: in progress, ETA 2021-03-12 08:00UTC (28GB done out of 248GB in ~1h as of 2021-03-11 22:04UTC) as completed

marked the checklist item benchmarks as completed

staticsync::static_mirror::get_triggered was enabled on web-chi-03, so it should keep up to date with the other nodes in the network. redoing a manual sync to make sure the next ones won't take too long.

pushed DNS change to put it in live rotation. not a fastly backend yet (should it be?)

server now live, i guess we're done here.

closed

mentioned in issue #40775 (closed)

marked this issue as related to #40775 (closed)

note that this machine was setup in a peculiar way, documented here #40131 (comment 2728663)