Comparing iSCSI vs Fibre Channel Storage Network

As we know, nowadays the network speed has reached 400G at data centers. And, most servers are coming with at least 25G or 50G network adaptors now as a default configuration. This fact brings challenging questions from our customers and we are facing  iSCSI vs Fibre Channel storage network comparisons. Of course, priority is being competitive at the business and catching the future trends.

Before continuing, we assume you are familiar with block storage technologies. However, if you want a brief history of these protocols you can check one of the best technical deep-dive webcast about Fibre Channel and iSCSI comparison by SNIA.

Let’s focus on our main subject here; performance and cost!

First of all, we prepared a testbed to compare apples to apples. This is a very important point to highlight our decision. Because we see lots of non-comparable test scenarios on the internet, I mean apples to oranges. For example, comparing 1G iSCSI vs 8G Fibre Channel. Also, we know Fibre Channel networks are dedicated over SAN switches but somehow iSCSI tests are conducted on a shared backbone or edge switches. Similarly comparing 40G iSCSI vs 16G Fibre Channel is nonsense at all. 

So what we did here is to create a dedicated TCP/IP network with 100G ethernet switches and a 32G Fibre Channel SAN network. And we started to test respectively 40G ethernet vs 32G FC then 100G ethernet vs 64G FC. We chose this because they are dominant current average connectivities at the market, and can be compared as apples to apples both in terms of price and performance.  

First of all, let’s look at the economics;

* Costs are calculated from the online list prices of these components.

While the most important thing is availability we should create our storage infrastructure redundantly. So we need at least 2 components from each to find the minimal cost per server to create redundant storage fabric. Below table shows costs per connectivity type only for a single server. However, you can calculate your own costs depending on your numbers easily.

According to cost tables, creating an FC SAN storage network is ~67% more expensive than an iSCSI network. But before deciding on your future infrastructure, we need performance numbers.

Test Environment Details

To avoid any storage bottlenecks we used the NGX Storage NS200 SCM system.  

Storage: NGX Storage NS200 SCM all-flash

• 2TB DRAM
• 24 Intel® Optane™ SSD DC P5800X 400GB
• 4 x 32G Fibre Channel
• 4 x 100GbE QSFP

Server : Dell R740 / OS Ubuntu 20.04 

• 256GB Memory
• Intel(R) Xeon(R) Gold 6142M CPU @ 2.60GHz
• 2 x 32G Fibre Channel
• 1 x 40G (Mellanox Connect-X 4)
• 1 x 100G (Mellanox Connect-X 5)

We created 8 logical units for test setup and exported them for both FC and iSCSI targets. These tests were conducted via  fio and vdbench which are industry standard benchmark tools. While both results are similar we are sharing fio results here to keep it short. Additionally, we chose an average block size 32k to generate enough throughput and IOPS that can stress this setup. Last but not least we used jumbo frames (MTU 9000) at the network config.

Fio Results:

# FC 32G randomwrite / 32K Block size
fio --rw=randwrite --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=24 --bs=32k --filename=/dev/sda:/dev/sdc:/dev/sdd:/dev/sde:/dev/sdf:/dev/sdg:/dev/sdh:/dev/sdi
random: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 24 processes
Jobs: 24 (f=192): [w(24)][100.0%][w=2899MiB/s][w=92.8k IOPS][eta 00m:00s]
random: (groupid=0, jobs=24): err= 0: pid=13243: Mon Aug 9 14:18:09 2021
write: IOPS=90.1k, BW=2815MiB/s (2951MB/s)(165GiB/60010msec); 0 zone resets
slat (usec): min=3, max=52314, avg=29.97, stdev=97.15
clat (nsec): min=1919, max=279448k, avg=8493051.20, stdev=19921490.32
lat (usec): min=97, max=279488, avg=8523.44, stdev=19921.32
clat percentiles (usec):
| 1.00th=[ 297], 5.00th=[ 709], 10.00th=[ 1020], 20.00th=[ 1450],
| 30.00th=[ 1827], 40.00th=[ 2212], 50.00th=[ 2671], 60.00th=[ 3228],
| 70.00th=[ 4080], 80.00th=[ 5932], 90.00th=[ 14484], 95.00th=[ 49021],
| 99.00th=[108528], 99.50th=[127402], 99.90th=[164627], 99.95th=[179307],
| 99.99th=[212861]
bw ( MiB/s): min= 1872, max= 4049, per=99.99%, avg=2814.28, stdev=15.86, samples=2880
iops : min=59926, max=129579, avg=90056.14, stdev=507.64, samples=2880
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.71%, 500=1.86%, 750=2.93%, 1000=4.11%
lat (msec) : 2=24.79%, 4=34.73%, 10=18.14%, 20=4.24%, 50=3.59%
lat (msec) : 100=3.58%, 250=1.31%, 500=0.01%
cpu : usr=3.47%, sys=11.15%, ctx=4692047, majf=0, minf=11550
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,5405019,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32


Run status group 0 (all jobs):
WRITE: bw=2815MiB/s (2951MB/s), 2815MiB/s-2815MiB/s (2951MB/s-2951MB/s), io=165GiB (177GB), run=60010-60010msec


Disk stats (read/write):
sda: ios=76/672359, merge=0/61, ticks=2971/29410698, in_queue=28056724, util=99.80%
sdc: ios=80/675652, merge=0/2, ticks=85/1729028, in_queue=456664, util=99.82%
sdd: ios=89/675650, merge=0/1, ticks=165/1754645, in_queue=476600, util=99.83%
sde: ios=107/675640, merge=0/2, ticks=163/1736901, in_queue=458316, util=99.87%
sdf: ios=136/675628, merge=0/2, ticks=229/1675202, in_queue=414552, util=99.89%
sdg: ios=143/675626, merge=0/1, ticks=376/1683470, in_queue=415936, util=99.91%
sdh: ios=179/675623, merge=0/0, ticks=224/1720457, in_queue=453484, util=99.94%
sdi: ios=166/675609, merge=0/6, ticks=2374/5740380, in_queue=4325732, util=99.94%


# FC 32G randomread / 32K Block size


fio --rw=randread --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=24 --bs=32k --filename=/dev/sda:/dev/sdc:/dev/sdd:/dev/sde:/dev/sdf:/dev/sdg:/dev/sdh:/dev/sdi
random: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 24 processes
Jobs: 24 (f=192): [r(24)][100.0%][r=2762MiB/s][r=88.4k IOPS][eta 00m:00s]
random: (groupid=0, jobs=24): err= 0: pid=19896: Mon Aug 9 14:20:45 2021
read: IOPS=90.8k, BW=2838MiB/s (2976MB/s)(166GiB/60008msec)
slat (usec): min=3, max=647, avg=15.07, stdev=11.60
clat (usec): min=70, max=215045, avg=8437.26, stdev=11719.66
lat (usec): min=92, max=215057, avg=8452.73, stdev=11719.49
clat percentiles (usec):
| 1.00th=[ 1254], 5.00th=[ 1663], 10.00th=[ 1991], 20.00th=[ 2573],
| 30.00th=[ 3064], 40.00th=[ 3589], 50.00th=[ 4293], 60.00th=[ 5538],
| 70.00th=[ 7439], 80.00th=[ 10814], 90.00th=[ 18482], 95.00th=[ 29492],
| 99.00th=[ 61604], 99.50th=[ 76022], 99.90th=[109577], 99.95th=[125305],
| 99.99th=[162530]
bw ( MiB/s): min= 2042, max= 4014, per=99.99%, avg=2837.96, stdev=17.21, samples=2880
iops : min=65350, max=128472, avg=90814.26, stdev=550.78, samples=2880
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.04%, 1000=0.19%
lat (msec) : 2=9.90%, 4=36.44%, 10=31.46%, 20=12.95%, 50=7.27%
lat (msec) : 100=1.59%, 250=0.15%
cpu : usr=2.56%, sys=7.10%, ctx=5232143, majf=0, minf=15325
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=5450191,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32


Run status group 0 (all jobs):
READ: bw=2838MiB/s (2976MB/s), 2838MiB/s-2838MiB/s (2976MB/s-2976MB/s), io=166GiB (179GB), run=60008-60008msec


Disk stats (read/write):
sda: ios=678997/0, merge=11/0, ticks=7050720/0, in_queue=5582820, util=99.62%
sdc: ios=679011/0, merge=17/0, ticks=7807170/0, in_queue=6350700, util=99.66%
sdd: ios=679034/0, merge=13/0, ticks=6416492/0, in_queue=4946616, util=99.68%
sde: ios=678979/0, merge=7/0, ticks=6162488/0, in_queue=4684856, util=99.88%
sdf: ios=679047/0, merge=8/0, ticks=5285084/0, in_queue=3805036, util=99.72%
sdg: ios=679055/0, merge=9/0, ticks=5711760/0, in_queue=4237272, util=99.76%
sdh: ios=678769/0, merge=11/0, ticks=5581873/0, in_queue=4098172, util=99.78%
sdi: ios=679061/0, merge=2/0, ticks=1766793/0, in_queue=205180, util=99.79%


# ISCSI 100G randomwrite / 32K Block size


fio --rw=randwrite --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=24 --bs=32k --filename=/dev/sda:/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde:/dev/sdf:/dev/sdg:/dev/sdh
random: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 24 processes
Jobs: 24 (f=192): [w(24)][100.0%][w=11.5GiB/s][w=376k IOPS][eta 00m:00s]
random: (groupid=0, jobs=24): err= 0: pid=26320: Mon Aug 9 14:28:52 2021
write: IOPS=369k, BW=11.3GiB/s (12.1GB/s)(675GiB/60004msec); 0 zone resets
slat (usec): min=2, max=22826, avg=37.43, stdev=201.56
clat (usec): min=12, max=158530, avg=2044.13, stdev=2397.96
lat (usec): min=55, max=158551, avg=2081.68, stdev=2450.77
clat percentiles (usec):
| 1.00th=[ 202], 5.00th=[ 277], 10.00th=[ 330], 20.00th=[ 457],
| 30.00th=[ 660], 40.00th=[ 889], 50.00th=[ 1205], 60.00th=[ 1598],
| 70.00th=[ 2147], 80.00th=[ 3064], 90.00th=[ 5014], 95.00th=[ 7111],
| 99.00th=[10159], 99.50th=[11600], 99.90th=[20579], 99.95th=[26084],
| 99.99th=[39584]
bw ( MiB/s): min= 3718, max=12540, per=99.99%, avg=11523.50, stdev=44.09, samples=2880
iops : min=119006, max=401282, avg=368751.24, stdev=1410.96, samples=2880
lat (usec) : 20=0.01%, 50=0.01%, 100=0.04%, 250=3.14%, 500=19.57%
lat (usec) : 750=11.42%, 1000=9.69%
lat (msec) : 2=23.88%, 4=18.27%, 10=12.88%, 20=1.00%, 50=0.10%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=5.62%, sys=19.98%, ctx=11399948, majf=0, minf=20652
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,22128185,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32


Run status group 0 (all jobs):
WRITE: bw=11.3GiB/s (12.1GB/s), 11.3GiB/s-11.3GiB/s (12.1GB/s-12.1GB/s), io=675GiB (725GB), run=60004-60004msec


Disk stats (read/write):
sda: ios=99/2752163, merge=0/3140, ticks=51/4412371, in_queue=1425048, util=99.78%
sdb: ios=67/2750438, merge=0/4785, ticks=54/6071205, in_queue=2471248, util=99.84%
sdc: ios=73/2752430, merge=0/2897, ticks=52/4168589, in_queue=1410376, util=99.88%
sdd: ios=77/2752193, merge=0/3022, ticks=60/4352598, in_queue=1209072, util=99.87%
sde: ios=81/2752607, merge=0/2683, ticks=46/3783654, in_queue=1259132, util=99.92%
sdf: ios=85/2750446, merge=0/4806, ticks=70/5880030, in_queue=2479196, util=99.91%
sdg: ios=96/2752439, merge=0/2640, ticks=55/3764635, in_queue=1259592, util=99.93%
sdh: ios=99/2752171, merge=0/2907, ticks=58/3943191, in_queue=1467624, util=99.98%


# ISCSI 100G randomread / 32K Block size


fio --rw=randread --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=24 --bs=32k --filename=/dev/sda:/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde:/dev/sdf:/dev/sdg:/dev/sdh
random: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 24 processes
Jobs: 24 (f=192): [r(24)][100.0%][r=9.77GiB/s][r=320k IOPS][eta 00m:00s]
random: (groupid=0, jobs=24): err= 0: pid=28942: Mon Aug 9 14:31:01 2021
read: IOPS=311k, BW=9728MiB/s (10.2GB/s)(570GiB/60008msec)
slat (usec): min=2, max=15998, avg=54.88, stdev=311.12
clat (nsec): min=1346, max=117426k, avg=2411243.14, stdev=2988906.27
lat (usec): min=49, max=117431, avg=2466.22, stdev=3088.44
clat percentiles (usec):
| 1.00th=[ 133], 5.00th=[ 227], 10.00th=[ 285], 20.00th=[ 383],
| 30.00th=[ 529], 40.00th=[ 898], 50.00th=[ 1369], 60.00th=[ 1860],
| 70.00th=[ 2573], 80.00th=[ 3720], 90.00th=[ 6194], 95.00th=[ 8848],
| 99.00th=[13042], 99.50th=[14615], 99.90th=[21890], 99.95th=[28705],
| 99.99th=[44303]
bw ( MiB/s): min= 8219, max=11179, per=99.99%, avg=9727.58, stdev=22.70, samples=2880
iops : min=263008, max=357737, avg=311281.48, stdev=726.34, samples=2880
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.39%
lat (usec) : 250=6.44%, 500=21.95%, 750=8.05%, 1000=5.38%
lat (msec) : 2=20.31%, 4=19.10%, 10=14.88%, 20=3.39%, 50=0.12%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=2.24%, sys=13.14%, ctx=7622754, majf=0, minf=28545
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=18680735,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32


Run status group 0 (all jobs):
READ: bw=9728MiB/s (10.2GB/s), 9728MiB/s-9728MiB/s (10.2GB/s-10.2GB/s), io=570GiB (612GB), run=60008-60008msec


Disk stats (read/write):
sda: ios=2325147/0, merge=2153/0, ticks=3577557/0, in_queue=1550356, util=99.81%
sdb: ios=2322967/0, merge=4363/0, ticks=5969166/0, in_queue=3269420, util=99.81%
sdc: ios=2325538/0, merge=1766/0, ticks=3107834/0, in_queue=1333404, util=99.84%
sdd: ios=2324621/0, merge=2602/0, ticks=4137631/0, in_queue=1904632, util=99.84%
sde: ios=2325548/0, merge=1832/0, ticks=3090372/0, in_queue=1324000, util=99.88%
sdf: ios=2323925/0, merge=3299/0, ticks=4792066/0, in_queue=2390652, util=99.88%
sdg: ios=2326375/0, merge=1038/0, ticks=2125565/0, in_queue=680320, util=99.91%
sdh: ios=2325828/0, merge=1374/0, ticks=2568624/0, in_queue=1053704, util=99.94%


# FC 64G randomwrite / 32K Block size


fio --rw=randwrite --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=24 --bs=32k --filename=/dev/sdf:/dev/sdg:/dev/sdh:/dev/sdi:/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde
random: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 24 processes
Jobs: 24 (f=192): [w(24)][100.0%][w=4409MiB/s][w=141k IOPS][eta 00m:00s]
random: (groupid=0, jobs=24): err= 0: pid=17176: Tue Aug 10 09:27:20 2021
write: IOPS=142k, BW=4432MiB/s (4647MB/s)(260GiB/60007msec); 0 zone resets
slat (usec): min=3, max=8465, avg=26.29, stdev=63.42
clat (nsec): min=1377, max=221987k, avg=5386099.00, stdev=10797360.99
lat (usec): min=87, max=222000, avg=5412.75, stdev=10797.10
clat percentiles (usec):
| 1.00th=[ 110], 5.00th=[ 155], 10.00th=[ 221], 20.00th=[ 404],
| 30.00th=[ 676], 40.00th=[ 1037], 50.00th=[ 1516], 60.00th=[ 2245],
| 70.00th=[ 3523], 80.00th=[ 6259], 90.00th=[ 14615], 95.00th=[ 26084],
| 99.00th=[ 55313], 99.50th=[ 67634], 99.90th=[ 93848], 99.95th=[105382],
| 99.99th=[130548]
bw ( MiB/s): min= 2872, max= 7320, per=99.99%, avg=4431.28, stdev=29.57, samples=2880
iops : min=91922, max=234254, avg=141800.09, stdev=946.25, samples=2880
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.31%, 250=11.58%, 500=11.97%, 750=8.44%, 1000=6.83%
lat (msec) : 2=18.01%, 4=15.35%, 10=13.54%, 20=6.83%, 50=5.78%
lat (msec) : 100=1.28%, 250=0.07%
cpu : usr=5.00%, sys=15.40%, ctx=7565473, majf=0, minf=10271
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,8509825,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32


Run status group 0 (all jobs):
WRITE: bw=4432MiB/s (4647MB/s), 4432MiB/s-4432MiB/s (4647MB/s-4647MB/s), io=260GiB (279GB), run=60007-60007msec


Disk stats (read/write):
sdf: ios=77/1063700, merge=0/0, ticks=26/850388, in_queue=57456, util=99.72%
sdg: ios=87/1063711, merge=0/2, ticks=24/858639, in_queue=58676, util=99.77%
sdh: ios=103/1063485, merge=0/23, ticks=377/5969271, in_queue=4023072, util=99.82%
sdi: ios=139/1063555, merge=0/20, ticks=432/7080818, in_queue=5089340, util=99.85%
sdb: ios=158/1063407, merge=0/50, ticks=1337/12471721, in_queue=10439728, util=99.86%
sdc: ios=174/1063586, merge=0/44, ticks=1267/16249379, in_queue=14175540, util=99.90%
sdd: ios=191/1063701, merge=0/2, ticks=77/1015834, in_queue=94512, util=99.93%
sde: ios=190/1063661, merge=0/2, ticks=121/1105839, in_queue=115992, util=99.96%


# FC 64G randomread / 32K Block size


fio --rw=randread --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=24 --bs=32k --filename=/dev/sdf:/dev/sdg:/dev/sdh:/dev/sdi:/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde
random: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 24 processes
Jobs: 24 (f=192): [r(24)][100.0%][r=4922MiB/s][r=157k IOPS][eta 00m:00s]
random: (groupid=0, jobs=24): err= 0: pid=23220: Tue Aug 10 09:29:30 2021
read: IOPS=158k, BW=4924MiB/s (5163MB/s)(289GiB/60010msec)
slat (usec): min=4, max=497, avg=15.23, stdev=11.49
clat (usec): min=2, max=134519, avg=4856.01, stdev=8220.20
lat (usec): min=80, max=134534, avg=4871.60, stdev=8219.90
clat percentiles (usec):
| 1.00th=[ 302], 5.00th=[ 594], 10.00th=[ 775], 20.00th=[ 1057],
| 30.00th=[ 1336], 40.00th=[ 1647], 50.00th=[ 2024], 60.00th=[ 2638],
| 70.00th=[ 3687], 80.00th=[ 5800], 90.00th=[ 11469], 95.00th=[ 19530],
| 99.00th=[ 43254], 99.50th=[ 54264], 99.90th=[ 77071], 99.95th=[ 85459],
| 99.99th=[100140]
bw ( MiB/s): min= 3748, max= 6488, per=100.00%, avg=4923.52, stdev=28.18, samples=2880
iops : min=119944, max=207631, avg=157552.06, stdev=901.61, samples=2880
lat (usec) : 4=0.01%, 20=0.01%, 50=0.01%, 100=0.06%, 250=0.61%
lat (usec) : 500=2.55%, 750=5.95%, 1000=8.69%
lat (msec) : 2=31.66%, 4=22.61%, 10=16.19%, 20=6.81%, 50=4.20%
lat (msec) : 100=0.65%, 250=0.01%
cpu : usr=4.13%, sys=11.76%, ctx=8748698, majf=0, minf=15238
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=9454880,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32


Run status group 0 (all jobs):
READ: bw=4924MiB/s (5163MB/s), 4924MiB/s-4924MiB/s (5163MB/s-5163MB/s), io=289GiB (310GB), run=60010-60010msec


Disk stats (read/write):
sdf: ios=1177865/0, merge=14/0, ticks=3958728/0, in_queue=2039648, util=99.73%
sdg: ios=1177859/0, merge=18/0, ticks=4488655/0, in_queue=2475428, util=99.76%
sdh: ios=1177840/0, merge=21/0, ticks=5190973/0, in_queue=3152964, util=99.81%
sdi: ios=1177860/0, merge=18/0, ticks=5451968/0, in_queue=3471140, util=99.85%
sdb: ios=1177591/0, merge=31/0, ticks=9595141/0, in_queue=7367924, util=99.86%
sdc: ios=1177707/0, merge=16/0, ticks=5389808/0, in_queue=3209780, util=99.90%
sdd: ios=1177868/0, merge=1/0, ticks=1444543/0, in_queue=15944, util=99.91%
sde: ios=1177642/0, merge=28/0, ticks=10157679/0, in_queue=7924056, util=99.93%


# ISCSI 40G randomwrite / 32K Block size


fio --rw=randwrite --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=8 --bs=32k --filename=/dev/sdi:/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde:/dev/sdf:/dev/sdg:/dev/sdh
random: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 8 processes
Jobs: 8 (f=64): [w(8)][100.0%][w=4684MiB/s][w=150k IOPS][eta 00m:00s]
random: (groupid=0, jobs=8): err= 0: pid=36087: Tue Aug 24 16:11:06 2021
write: IOPS=150k, BW=4682MiB/s (4909MB/s)(274GiB/60003msec); 0 zone resets
slat (usec): min=2, max=1007, avg=22.79, stdev=11.97
clat (usec): min=51, max=83870, avg=1684.37, stdev=2338.39
lat (usec): min=87, max=83886, avg=1707.43, stdev=2337.62
clat percentiles (usec):
| 1.00th=[ 225], 5.00th=[ 306], 10.00th=[ 359], 20.00th=[ 433],
| 30.00th=[ 506], 40.00th=[ 603], 50.00th=[ 758], 60.00th=[ 1012],
| 70.00th=[ 1516], 80.00th=[ 2343], 90.00th=[ 4146], 95.00th=[ 6325],
| 99.00th=[11469], 99.50th=[13304], 99.90th=[18482], 99.95th=[23462],
| 99.99th=[38011]
bw ( MiB/s): min= 4549, max= 4796, per=99.98%, avg=4681.12, stdev= 5.73, samples=960
iops : min=145581, max=153500, avg=149795.53, stdev=183.24, samples=960
lat (usec) : 100=0.01%, 250=1.84%, 500=27.41%, 750=20.45%, 1000=9.98%
lat (msec) : 2=16.59%, 4=13.20%, 10=8.91%, 20=1.54%, 50=0.08%
lat (msec) : 100=0.01%
cpu : usr=7.36%, sys=44.99%, ctx=4371222, majf=0, minf=3772
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,8989672,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32


Run status group 0 (all jobs):
WRITE: bw=4682MiB/s (4909MB/s), 4682MiB/s-4682MiB/s (4909MB/s-4909MB/s), io=274GiB (295GB), run=60003-60003msec


Disk stats (read/write):
sdi: ios=102/1120444, merge=0/402, ticks=110/1948704, in_queue=738576, util=99.87%
sdb: ios=108/1120264, merge=0/578, ticks=94/2457704, in_queue=1066512, util=99.89%
sdc: ios=69/1120464, merge=0/390, ticks=55/1932764, in_queue=700948, util=99.90%
sdd: ios=70/1120503, merge=0/351, ticks=56/1817100, in_queue=627440, util=99.90%
sde: ios=69/1120323, merge=0/399, ticks=62/1978459, in_queue=706900, util=99.92%
sdf: ios=67/1120618, merge=0/209, ticks=39/1486035, in_queue=369348, util=99.92%
sdg: ios=70/1120681, merge=0/160, ticks=40/1280505, in_queue=244416, util=99.95%
sdh: ios=68/1120398, merge=0/452, ticks=39/2032347, in_queue=790108, util=99.95%


# ISCSI 40G randomread / 32K Block size


fio --rw=randread --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=8 --bs=32k --filename=/dev/sdi:/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde:/dev/sdf:/dev/sdg:/dev/sdh
random: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 8 processes
Jobs: 8 (f=64): [r(8)][100.0%][r=4612MiB/s][r=148k IOPS][eta 00m:00s]
random: (groupid=0, jobs=8): err= 0: pid=37475: Tue Aug 24 16:13:02 2021
read: IOPS=147k, BW=4609MiB/s (4833MB/s)(270GiB/60005msec)
slat (nsec): min=1945, max=97891k, avg=16815.93, stdev=60543.38
clat (nsec): min=444, max=132128k, avg=1717382.04, stdev=2532988.67
lat (usec): min=78, max=132134, avg=1734.46, stdev=2533.63
clat percentiles (usec):
| 1.00th=[ 139], 5.00th=[ 192], 10.00th=[ 237], 20.00th=[ 326],
| 30.00th=[ 441], 40.00th=[ 594], 50.00th=[ 824], 60.00th=[ 1205],
| 70.00th=[ 1696], 80.00th=[ 2311], 90.00th=[ 4146], 95.00th=[ 6652],
| 99.00th=[11994], 99.50th=[13829], 99.90th=[19268], 99.95th=[25560],
| 99.99th=[44827]
bw ( MiB/s): min= 3921, max= 4780, per=99.99%, avg=4608.40, stdev=13.53, samples=960
iops : min=125500, max=152978, avg=147468.49, stdev=432.91, samples=960
lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.06%, 250=11.47%, 500=22.88%, 750=13.00%, 1000=7.77%
lat (msec) : 2=20.72%, 4=13.69%, 10=8.48%, 20=1.84%, 50=0.08%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=4.34%, sys=35.50%, ctx=4847513, majf=0, minf=4641
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=8849635,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32


Run status group 0 (all jobs):
READ: bw=4609MiB/s (4833MB/s), 4609MiB/s-4609MiB/s (4833MB/s-4833MB/s), io=270GiB (290GB), run=60005-60005msec


Disk stats (read/write):
sdi: ios=1103545/0, merge=43/0, ticks=923489/0, in_queue=40488, util=99.68%
sdb: ios=1103391/0, merge=191/0, ticks=1463725/0, in_queue=318276, util=99.69%
sdc: ios=1103536/0, merge=51/0, ticks=921654/0, in_queue=64784, util=99.70%
sdd: ios=1103493/0, merge=83/0, ticks=1028444/0, in_queue=139716, util=99.86%
sde: ios=1103570/0, merge=14/0, ticks=827315/0, in_queue=1700, util=99.71%
sdf: ios=1102155/0, merge=1265/0, ticks=4257267/0, in_queue=2346016, util=99.72%
sdg: ios=1102327/0, merge=1240/0, ticks=4237243/0, in_queue=2315936, util=99.73%
sdh: ios=1103490/0, merge=96/0, ticks=1157271/0, in_queue=127324, util=99.75%


# ISCSI 40G randomwrite / 4K Block size


fio --rw=randwrite --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=24 --bs=4k --filename=/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde:/dev/sdf:/dev/sdg:/dev/sdi
random: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 24 processes
Jobs: 24 (f=168): [w(24)][100.0%][w=829MiB/s][w=212k IOPS][eta 00m:00s]
random: (groupid=0, jobs=24): err= 0: pid=12665: Mon Sep 6 15:28:41 2021
write: IOPS=198k, BW=775MiB/s (813MB/s)(45.4GiB/60006msec); 0 zone resets
slat (usec): min=2, max=7088, avg=22.03, stdev=61.21
clat (nsec): min=421, max=154679k, avg=3846762.03, stdev=4262129.77
lat (usec): min=53, max=154694, avg=3869.07, stdev=4266.94
clat percentiles (usec):
| 1.00th=[ 155], 5.00th=[ 293], 10.00th=[ 420], 20.00th=[ 635],
| 30.00th=[ 857], 40.00th=[ 1401], 50.00th=[ 2180], 60.00th=[ 3359],
| 70.00th=[ 4883], 80.00th=[ 6915], 90.00th=[ 9765], 95.00th=[11994],
| 99.00th=[16581], 99.50th=[19268], 99.90th=[34866], 99.95th=[44303],
| 99.99th=[64750]
bw ( KiB/s): min=596920, max=957281, per=99.99%, avg=793559.55, stdev=3257.39, samples=2880
iops : min=149230, max=239320, avg=198389.33, stdev=814.35, samples=2880
lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.17%, 250=3.40%, 500=10.14%, 750=11.88%, 1000=8.06%
lat (msec) : 2=14.33%, 4=16.60%, 10=26.04%, 20=8.93%, 50=0.40%
lat (msec) : 100=0.03%, 250=0.01%
cpu : usr=2.55%, sys=17.51%, ctx=8624821, majf=0, minf=3543
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,11905508,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32


Run status group 0 (all jobs):
WRITE: bw=775MiB/s (813MB/s), 775MiB/s-775MiB/s (813MB/s-813MB/s), io=45.4GiB (48.8GB), run=60006-60006msec


# ISCSI 40G randomread / 4K Block size


fio --rw=randread --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=24 --bs=4k --filename=/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde:/dev/sdf:/dev/sdg:/dev/sdi
random: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 24 processes
Jobs: 24 (f=168): [r(24)][100.0%][r=774MiB/s][r=198k IOPS][eta 00m:00s]
random: (groupid=0, jobs=24): err= 0: pid=15458: Mon Sep 6 15:50:47 2021
read: IOPS=191k, BW=748MiB/s (784MB/s)(43.8GiB/60006msec)
slat (nsec): min=1448, max=7725.9k, avg=13066.78, stdev=88385.75
clat (nsec): min=453, max=133793k, avg=3998356.78, stdev=4780447.02
lat (usec): min=41, max=133799, avg=4011.54, stdev=4788.26
clat percentiles (usec):
| 1.00th=[ 101], 5.00th=[ 163], 10.00th=[ 235], 20.00th=[ 429],
| 30.00th=[ 562], 40.00th=[ 889], 50.00th=[ 1827], 60.00th=[ 3326],
| 70.00th=[ 5342], 80.00th=[ 7898], 90.00th=[11076], 95.00th=[13173],
| 99.00th=[16450], 99.50th=[17957], 99.90th=[39584], 99.95th=[49021],
| 99.99th=[69731]
bw ( KiB/s): min=694107, max=869120, per=99.99%, avg=765602.99, stdev=1348.88, samples=2880
iops : min=173526, max=217280, avg=191400.38, stdev=337.23, samples=2880
lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (usec) : 10=0.01%, 20=0.01%, 50=0.03%, 100=0.93%, 250=9.96%
lat (usec) : 500=14.47%, 750=12.39%, 1000=3.50%
lat (msec) : 2=10.14%, 4=12.15%, 10=23.47%, 20=12.57%, 50=0.33%
lat (msec) : 100=0.05%, 250=0.01%
cpu : usr=1.12%, sys=7.28%, ctx=9892062, majf=0, minf=3094
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=11485797,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32


Run status group 0 (all jobs):
READ: bw=748MiB/s (784MB/s), 748MiB/s-748MiB/s (784MB/s-784MB/s), io=43.8GiB (47.0GB), run=60006-60006msec


Disk stats (read/write):
sdb: ios=1636185/0, merge=402/0, ticks=7250108/0, in_queue=4069412, util=99.81%
sdc: ios=1635633/0, merge=792/0, ticks=12912668/0, in_queue=9608452, util=99.83%
sdd: ios=1635621/0, merge=780/0, ticks=13037229/0, in_queue=9730660, util=99.83%
sde: ios=1635806/0, merge=595/0, ticks=10056963/0, in_queue=6780712, util=99.85%
sdf: ios=1636581/0, merge=0/0, ticks=733002/0, in_queue=380, util=99.85%
sdg: ios=1636578/0, merge=1/0, ticks=739285/0, in_queue=224, util=99.86%
sdi: ios=1636575/0, merge=1/0, ticks=747859/0, in_queue=2224, util=99.87%


# 32G FC randomwrite / 4k Block size


fio --rw=randwrite --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=24 --bs=4k --filename=/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde:/dev/sdf:/dev/sdg:/dev/sdi
random: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 24 processes
Jobs: 24 (f=168): [w(24)][100.0%][w=571MiB/s][w=146k IOPS][eta 00m:00s]
random: (groupid=0, jobs=24): err= 0: pid=19103: Mon Sep 6 12:39:37 2021
write: IOPS=149k, BW=583MiB/s (612MB/s)(34.2GiB/60011msec); 0 zone resets
slat (nsec): min=1536, max=8444.5k, avg=18333.39, stdev=12948.59
clat (nsec): min=968, max=206384k, avg=5120638.24, stdev=15466276.36
lat (usec): min=64, max=206398, avg=5139.42, stdev=15466.41
clat percentiles (usec):
| 1.00th=[ 83], 5.00th=[ 96], 10.00th=[ 114], 20.00th=[ 172],
| 30.00th=[ 285], 40.00th=[ 486], 50.00th=[ 791], 60.00th=[ 1221],
| 70.00th=[ 1844], 80.00th=[ 2999], 90.00th=[ 8160], 95.00th=[ 30278],
| 99.00th=[ 86508], 99.50th=[105382], 99.90th=[143655], 99.95th=[154141],
| 99.99th=[175113]
bw ( KiB/s): min=465344, max=753730, per=100.00%, avg=597355.98, stdev=2284.86, samples=2880
iops : min=116336, max=188432, avg=149338.43, stdev=571.22, samples=2880
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=6.28%, 250=21.17%, 500=13.12%, 750=8.19%, 1000=6.45%
lat (msec) : 2=16.69%, 4=12.38%, 10=6.63%, 20=2.48%, 50=3.68%
lat (msec) : 100=2.29%, 250=0.62%
cpu : usr=3.48%, sys=13.75%, ctx=8659903, majf=0, minf=2222
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,8962181,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32


Run status group 0 (all jobs):
WRITE: bw=583MiB/s (612MB/s), 583MiB/s-583MiB/s (612MB/s-612MB/s), io=34.2GiB (36.7GB), run=60011-60011msec


Disk stats (read/write):
sdb: ios=60/1276063, merge=0/10, ticks=229/36668272, in_queue=34103068, util=99.86%
sdc: ios=75/1276708, merge=0/0, ticks=15/410877, in_queue=1128, util=99.91%
sdd: ios=79/1276642, merge=0/1, ticks=179/1678837, in_queue=374496, util=99.93%
sde: ios=96/1276704, merge=0/0, ticks=12/381689, in_queue=1324, util=99.95%
sdf: ios=106/1276699, merge=0/0, ticks=55/1522075, in_queue=281904, util=99.97%
sdg: ios=112/1276685, merge=0/2, ticks=190/3300093, in_queue=1135392, util=99.98%
sdi: ios=134/1276689, merge=0/0, ticks=76/1709044, in_queue=380036, util=99.99%


# 32G FC randomread / 4k Block size


fio --rw=randread --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=24 --bs=4k --filename=/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde:/dev/sdf:/dev/sdg:/dev/sdi
random: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 24 processes
Jobs: 24 (f=168): [r(24)][100.0%][r=711MiB/s][r=182k IOPS][eta 00m:00s]
random: (groupid=0, jobs=24): err= 0: pid=22947: Mon Sep 6 12:41:45 2021
read: IOPS=174k, BW=681MiB/s (714MB/s)(39.9GiB/60008msec)
slat (usec): min=3, max=866, avg=13.71, stdev=16.95
clat (nsec): min=831, max=350900k, avg=4390257.27, stdev=9790207.13
lat (usec): min=54, max=350908, avg=4404.42, stdev=9790.54
clat percentiles (usec):
| 1.00th=[ 71], 5.00th=[ 103], 10.00th=[ 167], 20.00th=[ 314],
| 30.00th=[ 506], 40.00th=[ 758], 50.00th=[ 1123], 60.00th=[ 1762],
| 70.00th=[ 2933], 80.00th=[ 5211], 90.00th=[ 11076], 95.00th=[ 19792],
| 99.00th=[ 49546], 99.50th=[ 63701], 99.90th=[ 94897], 99.95th=[106431],
| 99.99th=[141558]
bw ( KiB/s): min=323440, max=805868, per=99.99%, avg=696941.88, stdev=2159.13, samples=2880
iops : min=80860, max=201467, avg=174234.88, stdev=539.78, samples=2880
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.05%
lat (usec) : 100=4.60%, 250=11.17%, 500=13.87%, 750=10.04%, 1000=7.32%
lat (msec) : 2=15.47%, 4=13.00%, 10=13.32%, 20=6.23%, 50=3.92%
lat (msec) : 100=0.90%, 250=0.07%, 500=0.01%
cpu : usr=3.82%, sys=12.08%, ctx=9466133, majf=0, minf=1922
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=10456193,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32


Run status group 0 (all jobs):
READ: bw=681MiB/s (714MB/s), 681MiB/s-681MiB/s (714MB/s-714MB/s), io=39.9GiB (42.8GB), run=60008-60008msec


Disk stats (read/write):
sdb: ios=1489948/0, merge=10/0, ticks=12879145/0, in_queue=10228592, util=99.85%
sdc: ios=1490311/0, merge=0/0, ticks=922770/0, in_queue=22108, util=99.41%
sdd: ios=1490078/0, merge=7/0, ticks=9530967/0, in_queue=6992564, util=99.42%
sde: ios=1490308/0, merge=0/0, ticks=537624/0, in_queue=8, util=99.48%
sdf: ios=1490288/0, merge=9/0, ticks=10328780/0, in_queue=7750484, util=99.43%
sdg: ios=1490305/0, merge=0/0, ticks=932092/0, in_queue=26688, util=99.45%
sdi: ios=1490132/0, merge=8/0, ticks=10470841/0, in_queue=7889748, util=99.45%