diff --git a/howto/new-machine-cymru.md b/howto/new-machine-cymru.md index 6a9dec7f7facfeaf322f34d966d8e8bf6a8fe274..fadb8c187cbd7004daf9f96e639cb9667909411d 100644 --- a/howto/new-machine-cymru.md +++ b/howto/new-machine-cymru.md @@ -1441,6 +1441,341 @@ Then the device is available as a unique device in: ... even though there are multiple underlying devices. +## Benchmarks + +Overall, the hardware in the gnt-chi cluster is dated, mainly because +it lacks fast SSD disks. It can still get respectable performance, +because the disks were top of the line when they were setup. In +general, you should expect: + + * local (small) disks: + * read: IOPS=1148, BW=4595KiB/s (4706kB/s) + * write: IOPS=2213, BW=8854KiB/s (9067kB/s) + * iSCSI (network, large) disks: + * read: IOPS=26.9k, BW=105MiB/s (110MB/s) (gigabit network + saturation, probably cached by the SAN) + * write: IOPS=264, BW=1059KiB/s (1085kB/s) + +In other words, local disks can't quite saturate network (far from it: +they don't even saturate a 100mbps link). Network disks seem to be +able to saturate gigabit at first glance, but that's probably a +limitation of the benchmark. Writes are much slower, somewhere around +8mbps. + +Compare this with a more modern setup: + + * NVMe: + * read: IOPS=138k, BW=541MiB/s (567MB/s) + * write: IOPS=115k, BW=448MiB/s (470MB/s) + * SATA: + * read: IOPS=5550, BW=21.7MiB/s (22.7MB/s) + * write: IOPS=199, BW=796KiB/s (815kB/s) + +Notice how the large disk writes are actually *lower* than the iSCSI +store in this case, but this could be a fluke because of the existing +load on the gnt-fsn cluster. + +### Onboard SAS disks, chi-node-01 + + root@chi-node-01:~/bench# fio --name=stressant --group_reporting <(sed /^filename=/d /usr/share/doc/fio/examples/basic-verify.fio) --runtime=1m --filename=test --size=100m + stressant: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 + write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 + fio-3.12 + Starting 2 processes + stressant: Laying out IO file (1 file / 100MiB) + Jobs: 1 (f=1): [_(1),V(1)][94.3%][r=104MiB/s][r=26.6k IOPS][eta 00m:21s] + stressant: (groupid=0, jobs=1): err= 0: pid=13409: Wed Mar 24 17:40:23 2021 + read: IOPS=150k, BW=585MiB/s (613MB/s)(100MiB/171msec) + clat (nsec): min=980, max=1033.1k, avg=6290.36, stdev=46177.07 + lat (nsec): min=1015, max=1033.1k, avg=6329.40, stdev=46177.22 + clat percentiles (nsec): + | 1.00th=[ 1032], 5.00th=[ 1048], 10.00th=[ 1064], 20.00th=[ 1096], + | 30.00th=[ 1128], 40.00th=[ 1144], 50.00th=[ 1176], 60.00th=[ 1192], + | 70.00th=[ 1224], 80.00th=[ 1272], 90.00th=[ 1432], 95.00th=[ 1816], + | 99.00th=[244736], 99.50th=[428032], 99.90th=[618496], 99.95th=[692224], + | 99.99th=[774144] + lat (nsec) : 1000=0.03% + lat (usec) : 2=97.01%, 4=0.84%, 10=0.07%, 20=0.47%, 50=0.13% + lat (usec) : 100=0.12%, 250=0.35%, 500=0.68%, 750=0.29%, 1000=0.01% + lat (msec) : 2=0.01% + cpu : usr=8.82%, sys=27.65%, ctx=372, majf=0, minf=12 + IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% + submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% + complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% + issued rwts: total=25600,0,0,0 short=0,0,0,0 dropped=0,0,0,0 + latency : target=0, window=0, percentile=100.00%, depth=1 + write-and-verify: (groupid=0, jobs=1): err= 0: pid=13410: Wed Mar 24 17:40:23 2021 + read: IOPS=1148, BW=4595KiB/s (4706kB/s)(1024MiB/228181msec) + slat (usec): min=5, max=547, avg=21.60, stdev= 8.70 + clat (usec): min=22, max=767720, avg=13899.92, stdev=26025.10 + lat (usec): min=42, max=767773, avg=13921.96, stdev=26027.93 + clat percentiles (usec): + | 1.00th=[ 42], 5.00th=[ 51], 10.00th=[ 56], 20.00th=[ 65], + | 30.00th=[ 117], 40.00th=[ 200], 50.00th=[ 4146], 60.00th=[ 8029], + | 70.00th=[ 13566], 80.00th=[ 21890], 90.00th=[ 39060], 95.00th=[ 60031], + | 99.00th=[123208], 99.50th=[156238], 99.90th=[244319], 99.95th=[287310], + | 99.99th=[400557] + write: IOPS=2213, BW=8854KiB/s (9067kB/s)(1024MiB/118428msec); 0 zone resets + slat (usec): min=6, max=104014, avg=36.98, stdev=364.05 + clat (usec): min=62, max=887491, avg=7187.20, stdev=7152.34 + lat (usec): min=72, max=887519, avg=7224.67, stdev=7165.15 + clat percentiles (usec): + | 1.00th=[ 157], 5.00th=[ 383], 10.00th=[ 922], 20.00th=[ 1909], + | 30.00th=[ 2606], 40.00th=[ 3261], 50.00th=[ 4146], 60.00th=[ 7111], + | 70.00th=[10421], 80.00th=[13042], 90.00th=[15795], 95.00th=[18220], + | 99.00th=[25822], 99.50th=[32900], 99.90th=[65274], 99.95th=[72877], + | 99.99th=[94897] + bw ( KiB/s): min= 4704, max=95944, per=99.93%, avg=8847.51, stdev=6512.44, samples=237 + iops : min= 1176, max=23986, avg=2211.85, stdev=1628.11, samples=237 + lat (usec) : 50=2.27%, 100=11.27%, 250=10.32%, 500=1.76%, 750=1.19% + lat (usec) : 1000=0.86% + lat (msec) : 2=5.57%, 4=15.85%, 10=17.35%, 20=21.25%, 50=8.72% + lat (msec) : 100=2.72%, 250=0.82%, 500=0.04%, 750=0.01%, 1000=0.01% + cpu : usr=1.67%, sys=4.52%, ctx=296808, majf=0, minf=7562 + IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% + submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% + complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% + issued rwts: total=262144,262144,0,0 short=0,0,0,0 dropped=0,0,0,0 + latency : target=0, window=0, percentile=100.00%, depth=16 + + Run status group 0 (all jobs): + READ: bw=5044KiB/s (5165kB/s), 4595KiB/s-585MiB/s (4706kB/s-613MB/s), io=1124MiB (1179MB), run=171-228181msec + WRITE: bw=8854KiB/s (9067kB/s), 8854KiB/s-8854KiB/s (9067kB/s-9067kB/s), io=1024MiB (1074MB), run=118428-118428msec + + Disk stats (read/write): + dm-1: ios=262548/275002, merge=0/0, ticks=3635324/2162480, in_queue=5799708, util=100.00%, aggrios=262642/276055, aggrmerge=0/0, aggrticks=3640764/2166784, aggrin_queue=5807820, aggrutil=100.00% + dm-0: ios=262642/276055, merge=0/0, ticks=3640764/2166784, in_queue=5807820, util=100.00%, aggrios=262642/267970, aggrmerge=0/8085, aggrticks=3633173/1921094, aggrin_queue=5507676, aggrutil=99.16% + sda: ios=262642/267970, merge=0/8085, ticks=3633173/1921094, in_queue=5507676, util=99.16% + +### iSCSI load testing, chi-node-01 + + root@chi-node-01:/mnt# fio --name=stressant --group_reporting <(sed /^filename=/d /usr/share/doc/fio/examples/basic-verify.fio; echo size=100m) --runtime=1m --filename=test --size=100m + stressant: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 + write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 + fio-3.12 + Starting 2 processes + write-and-verify: Laying out IO file (1 file / 100MiB) + Jobs: 1 (f=0): [_(1),f(1)][100.0%][r=88.9MiB/s][r=22.8k IOPS][eta 00m:00s] + stressant: (groupid=0, jobs=1): err= 0: pid=18332: Wed Mar 24 17:56:02 2021 + read: IOPS=26.9k, BW=105MiB/s (110MB/s)(100MiB/952msec) + clat (nsec): min=1214, max=7423.1k, avg=35799.85, stdev=324182.56 + lat (nsec): min=1252, max=7423.2k, avg=35889.53, stdev=324181.89 + clat percentiles (nsec): + | 1.00th=[ 1400], 5.00th=[ 2128], 10.00th=[ 2288], + | 20.00th=[ 2512], 30.00th=[ 2576], 40.00th=[ 2608], + | 50.00th=[ 2608], 60.00th=[ 2640], 70.00th=[ 2672], + | 80.00th=[ 2704], 90.00th=[ 2800], 95.00th=[ 3440], + | 99.00th=[ 782336], 99.50th=[3391488], 99.90th=[4227072], + | 99.95th=[4358144], 99.99th=[4620288] + bw ( KiB/s): min=105440, max=105440, per=55.81%, avg=105440.00, stdev= 0.00, samples=1 + iops : min=26360, max=26360, avg=26360.00, stdev= 0.00, samples=1 + lat (usec) : 2=3.30%, 4=92.34%, 10=2.05%, 20=0.65%, 50=0.08% + lat (usec) : 100=0.01%, 250=0.11%, 500=0.16%, 750=0.28%, 1000=0.11% + lat (msec) : 2=0.11%, 4=0.67%, 10=0.13% + cpu : usr=4.94%, sys=12.83%, ctx=382, majf=0, minf=12 + IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% + submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% + complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% + issued rwts: total=25600,0,0,0 short=0,0,0,0 dropped=0,0,0,0 + latency : target=0, window=0, percentile=100.00%, depth=1 + write-and-verify: (groupid=0, jobs=1): err= 0: pid=18333: Wed Mar 24 17:56:02 2021 + read: IOPS=23.6k, BW=92.2MiB/s (96.7MB/s)(100MiB/1084msec) + slat (nsec): min=6524, max=66741, avg=15619.91, stdev=6159.27 + clat (usec): min=331, max=52833, avg=658.14, stdev=1305.45 + lat (usec): min=355, max=52852, avg=674.08, stdev=1305.57 + clat percentiles (usec): + | 1.00th=[ 420], 5.00th=[ 469], 10.00th=[ 502], 20.00th=[ 537], + | 30.00th=[ 570], 40.00th=[ 594], 50.00th=[ 619], 60.00th=[ 644], + | 70.00th=[ 676], 80.00th=[ 709], 90.00th=[ 758], 95.00th=[ 799], + | 99.00th=[ 881], 99.50th=[ 914], 99.90th=[ 1188], 99.95th=[52691], + | 99.99th=[52691] + write: IOPS=264, BW=1059KiB/s (1085kB/s)(100MiB/96682msec); 0 zone resets + slat (usec): min=15, max=110293, avg=112.91, stdev=1199.05 + clat (msec): min=3, max=593, avg=60.30, stdev=52.88 + lat (msec): min=3, max=594, avg=60.41, stdev=52.90 + clat percentiles (msec): + | 1.00th=[ 12], 5.00th=[ 15], 10.00th=[ 17], 20.00th=[ 23], + | 30.00th=[ 29], 40.00th=[ 35], 50.00th=[ 44], 60.00th=[ 54], + | 70.00th=[ 68], 80.00th=[ 89], 90.00th=[ 126], 95.00th=[ 165], + | 99.00th=[ 259], 99.50th=[ 300], 99.90th=[ 426], 99.95th=[ 489], + | 99.99th=[ 592] + bw ( KiB/s): min= 176, max= 1328, per=99.67%, avg=1055.51, stdev=127.13, samples=194 + iops : min= 44, max= 332, avg=263.86, stdev=31.78, samples=194 + lat (usec) : 500=4.96%, 750=39.50%, 1000=5.42% + lat (msec) : 2=0.08%, 4=0.01%, 10=0.27%, 20=7.64%, 50=20.38% + lat (msec) : 100=13.81%, 250=7.34%, 500=0.56%, 750=0.02% + cpu : usr=0.88%, sys=3.13%, ctx=34211, majf=0, minf=628 + IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0% + submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% + complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% + issued rwts: total=25600,25600,0,0 short=0,0,0,0 dropped=0,0,0,0 + latency : target=0, window=0, percentile=100.00%, depth=16 + + Run status group 0 (all jobs): + READ: bw=185MiB/s (193MB/s), 92.2MiB/s-105MiB/s (96.7MB/s-110MB/s), io=200MiB (210MB), run=952-1084msec + WRITE: bw=1059KiB/s (1085kB/s), 1059KiB/s-1059KiB/s (1085kB/s-1085kB/s), io=100MiB (105MB), run=96682-96682msec + + Disk stats (read/write): + dm-28: ios=22019/25723, merge=0/1157, ticks=16070/1557068, in_queue=1572636, util=99.98%, aggrios=4341/4288, aggrmerge=0/0, aggrticks=3089/259432, aggrin_queue=262419, aggrutil=99.79% + sdm: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% + sdk: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% + sdi: ios=8686/8573, merge=0/0, ticks=6409/526657, in_queue=532844, util=99.79% + sdl: ios=8683/8576, merge=0/0, ticks=6091/513333, in_queue=519120, util=99.75% + sdj: ios=8678/8580, merge=0/0, ticks=6036/516604, in_queue=522552, util=99.77% + sdh: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% + +### Comparison, NVMe disks, fsn-node-07 + + root@fsn-node-07:~# fio --name=stressant --group_reporting <(sed /^filename=/d /usr/share/doc/fio/examples/basic-verify.fio; echo size=100m) --runtime=1m --filename=test --size=100m + stressant: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 + write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 + fio-3.12 + Starting 2 processes + write-and-verify: Laying out IO file (1 file / 100MiB) + + stressant: (groupid=0, jobs=1): err= 0: pid=31809: Wed Mar 24 17:49:48 2021 + read: IOPS=138k, BW=541MiB/s (567MB/s)(100MiB/185msec) + clat (nsec): min=522, max=2651.8k, avg=6848.59, stdev=57695.32 + lat (nsec): min=539, max=2651.8k, avg=6871.47, stdev=57695.33 + clat percentiles (nsec): + | 1.00th=[ 540], 5.00th=[ 556], 10.00th=[ 572], + | 20.00th=[ 588], 30.00th=[ 596], 40.00th=[ 612], + | 50.00th=[ 628], 60.00th=[ 644], 70.00th=[ 692], + | 80.00th=[ 764], 90.00th=[ 828], 95.00th=[ 996], + | 99.00th=[ 292864], 99.50th=[ 456704], 99.90th=[ 708608], + | 99.95th=[ 864256], 99.99th=[1531904] + lat (nsec) : 750=77.95%, 1000=17.12% + lat (usec) : 2=2.91%, 4=0.09%, 10=0.21%, 20=0.12%, 50=0.09% + lat (usec) : 100=0.04%, 250=0.32%, 500=0.77%, 750=0.28%, 1000=0.06% + lat (msec) : 2=0.03%, 4=0.01% + cpu : usr=10.33%, sys=10.33%, ctx=459, majf=0, minf=11 + IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% + submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% + complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% + issued rwts: total=25600,0,0,0 short=0,0,0,0 dropped=0,0,0,0 + latency : target=0, window=0, percentile=100.00%, depth=1 + write-and-verify: (groupid=0, jobs=1): err= 0: pid=31810: Wed Mar 24 17:49:48 2021 + read: IOPS=145k, BW=565MiB/s (592MB/s)(100MiB/177msec) + slat (usec): min=2, max=153, avg= 3.28, stdev= 1.95 + clat (usec): min=23, max=740, avg=106.23, stdev=44.45 + lat (usec): min=25, max=743, avg=109.56, stdev=44.52 + clat percentiles (usec): + | 1.00th=[ 56], 5.00th=[ 70], 10.00th=[ 73], 20.00th=[ 77], + | 30.00th=[ 82], 40.00th=[ 87], 50.00th=[ 93], 60.00th=[ 101], + | 70.00th=[ 115], 80.00th=[ 130], 90.00th=[ 155], 95.00th=[ 182], + | 99.00th=[ 269], 99.50th=[ 343], 99.90th=[ 486], 99.95th=[ 537], + | 99.99th=[ 717] + write: IOPS=115k, BW=448MiB/s (470MB/s)(100MiB/223msec); 0 zone resets + slat (usec): min=4, max=160, avg= 6.10, stdev= 2.02 + clat (usec): min=31, max=15535, avg=132.13, stdev=232.65 + lat (usec): min=37, max=15546, avg=138.27, stdev=232.65 + clat percentiles (usec): + | 1.00th=[ 76], 5.00th=[ 90], 10.00th=[ 97], 20.00th=[ 102], + | 30.00th=[ 106], 40.00th=[ 113], 50.00th=[ 118], 60.00th=[ 123], + | 70.00th=[ 128], 80.00th=[ 137], 90.00th=[ 161], 95.00th=[ 184], + | 99.00th=[ 243], 99.50th=[ 302], 99.90th=[ 4293], 99.95th=[ 6915], + | 99.99th=[ 6980] + lat (usec) : 50=0.28%, 100=36.99%, 250=61.67%, 500=0.89%, 750=0.04% + lat (usec) : 1000=0.01% + lat (msec) : 2=0.02%, 4=0.03%, 10=0.06%, 20=0.01% + cpu : usr=22.11%, sys=57.79%, ctx=8799, majf=0, minf=623 + IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0% + submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% + complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% + issued rwts: total=25600,25600,0,0 short=0,0,0,0 dropped=0,0,0,0 + latency : target=0, window=0, percentile=100.00%, depth=16 + + Run status group 0 (all jobs): + READ: bw=1081MiB/s (1134MB/s), 541MiB/s-565MiB/s (567MB/s-592MB/s), io=200MiB (210MB), run=177-185msec + WRITE: bw=448MiB/s (470MB/s), 448MiB/s-448MiB/s (470MB/s-470MB/s), io=100MiB (105MB), run=223-223msec + + Disk stats (read/write): + dm-1: ios=25869/25600, merge=0/0, ticks=2856/2388, in_queue=5248, util=80.32%, aggrios=26004/25712, aggrmerge=0/0, aggrticks=2852/2380, aggrin_queue=5228, aggrutil=69.81% + dm-0: ios=26004/25712, merge=0/0, ticks=2852/2380, in_queue=5228, util=69.81%, aggrios=26005/25712, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00% + md1: ios=26005/25712, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=13002/25628, aggrmerge=0/85, aggrticks=1328/1147, aggrin_queue=2752, aggrutil=89.35% + nvme0n1: ios=12671/25628, merge=0/85, ticks=1176/496, in_queue=1896, util=89.35% + nvme1n1: ios=13333/25628, merge=1/85, ticks=1481/1798, in_queue=3608, util=89.35% + +### Comparison, SATA disks, fsn-node-02 + + root@fsn-node-02:/mnt# fio --name=stressant --group_reporting <(sed /^filename=/d /usr/share/doc/fio/examples/basic-verify.fio; echo size=100m) --runtime=1m --filename=test --size=100m + stressant: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 + write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 + fio-3.12 + Starting 2 processes + write-and-verify: Laying out IO file (1 file / 100MiB) + Jobs: 1 (f=0): [_(1),f(1)][100.0%][r=348KiB/s][r=87 IOPS][eta 00m:00s] + stressant: (groupid=0, jobs=1): err= 0: pid=9635: Wed Mar 24 17:50:32 2021 + read: IOPS=5550, BW=21.7MiB/s (22.7MB/s)(100MiB/4612msec) + clat (nsec): min=500, max=273948k, avg=179390.97, stdev=4673600.03 + lat (nsec): min=515, max=273948k, avg=179471.70, stdev=4673600.38 + clat percentiles (nsec): + | 1.00th=[ 524], 5.00th=[ 580], 10.00th=[ 692], + | 20.00th=[ 1240], 30.00th=[ 1496], 40.00th=[ 2320], + | 50.00th=[ 2352], 60.00th=[ 2896], 70.00th=[ 2960], + | 80.00th=[ 3024], 90.00th=[ 3472], 95.00th=[ 3824], + | 99.00th=[ 806912], 99.50th=[ 978944], 99.90th=[ 60030976], + | 99.95th=[110624768], 99.99th=[244318208] + bw ( KiB/s): min= 2048, max=82944, per=100.00%, avg=22296.89, stdev=26433.89, samples=9 + iops : min= 512, max=20736, avg=5574.22, stdev=6608.47, samples=9 + lat (nsec) : 750=11.57%, 1000=3.11% + lat (usec) : 2=23.35%, 4=58.17%, 10=1.90%, 20=0.16%, 50=0.15% + lat (usec) : 100=0.03%, 250=0.03%, 500=0.04%, 750=0.32%, 1000=0.69% + lat (msec) : 2=0.17%, 4=0.04%, 10=0.11%, 20=0.02%, 50=0.02% + lat (msec) : 100=0.05%, 250=0.07%, 500=0.01% + cpu : usr=1.41%, sys=1.52%, ctx=397, majf=0, minf=13 + IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% + submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% + complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% + issued rwts: total=25600,0,0,0 short=0,0,0,0 dropped=0,0,0,0 + latency : target=0, window=0, percentile=100.00%, depth=1 + write-and-verify: (groupid=0, jobs=1): err= 0: pid=9636: Wed Mar 24 17:50:32 2021 + read: IOPS=363, BW=1455KiB/s (1490kB/s)(100MiB/70368msec) + slat (usec): min=2, max=4401, avg=46.08, stdev=38.17 + clat (usec): min=101, max=1002.5k, avg=43920.61, stdev=49423.03 + lat (usec): min=106, max=1002.5k, avg=43967.49, stdev=49419.62 + clat percentiles (usec): + | 1.00th=[ 188], 5.00th=[ 273], 10.00th=[ 383], 20.00th=[ 3752], + | 30.00th=[ 8586], 40.00th=[ 16319], 50.00th=[ 28967], 60.00th=[ 45351], + | 70.00th=[ 62129], 80.00th=[ 80217], 90.00th=[106431], 95.00th=[129500], + | 99.00th=[181404], 99.50th=[200279], 99.90th=[308282], 99.95th=[884999], + | 99.99th=[943719] + write: IOPS=199, BW=796KiB/s (815kB/s)(100MiB/128642msec); 0 zone resets + slat (usec): min=4, max=136984, avg=101.20, stdev=2123.50 + clat (usec): min=561, max=1314.6k, avg=80287.04, stdev=105685.87 + lat (usec): min=574, max=1314.7k, avg=80388.86, stdev=105724.12 + clat percentiles (msec): + | 1.00th=[ 3], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], + | 30.00th=[ 12], 40.00th=[ 45], 50.00th=[ 51], 60.00th=[ 68], + | 70.00th=[ 111], 80.00th=[ 136], 90.00th=[ 167], 95.00th=[ 207], + | 99.00th=[ 460], 99.50th=[ 600], 99.90th=[ 1250], 99.95th=[ 1318], + | 99.99th=[ 1318] + bw ( KiB/s): min= 104, max= 1576, per=100.00%, avg=822.39, stdev=297.05, samples=249 + iops : min= 26, max= 394, avg=205.57, stdev=74.29, samples=249 + lat (usec) : 250=1.95%, 500=4.63%, 750=0.69%, 1000=0.40% + lat (msec) : 2=1.15%, 4=3.47%, 10=18.34%, 20=7.79%, 50=17.56% + lat (msec) : 100=20.45%, 250=21.82%, 500=1.27%, 750=0.23%, 1000=0.10% + cpu : usr=0.60%, sys=1.79%, ctx=46722, majf=0, minf=627 + IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0% + submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% + complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% + issued rwts: total=25600,25600,0,0 short=0,0,0,0 dropped=0,0,0,0 + latency : target=0, window=0, percentile=100.00%, depth=16 + + Run status group 0 (all jobs): + READ: bw=2910KiB/s (2980kB/s), 1455KiB/s-21.7MiB/s (1490kB/s-22.7MB/s), io=200MiB (210MB), run=4612-70368msec + WRITE: bw=796KiB/s (815kB/s), 796KiB/s-796KiB/s (815kB/s-815kB/s), io=100MiB (105MB), run=128642-128642msec + + Disk stats (read/write): + dm-48: ios=26004/27330, merge=0/0, ticks=1132284/2233896, in_queue=3366684, util=100.00%, aggrios=28026/41435, aggrmerge=0/0, aggrticks=1292636/3986932, aggrin_queue=5288484, aggrutil=100.00% + dm-56: ios=28026/41435, merge=0/0, ticks=1292636/3986932, in_queue=5288484, util=100.00%, aggrios=28027/41436, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00% + md125: ios=28027/41436, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=13768/36599, aggrmerge=220/4980, aggrticks=622303/1259843, aggrin_queue=859540, aggrutil=61.10% + sdb: ios=13271/36574, merge=193/5009, ticks=703823/1612782, in_queue=1077576, util=61.10% + sda: ios=14265/36624, merge=248/4951, ticks=540784/906905, in_queue=641504, util=51.08% + +Keep in mind the machine was not idle at the time of testing, quite +the contrary (under about 4-5 load). + ## Glossary * **[SAN][]**: storage area network