netblogdata

As we know, nowadays the network speed has reached 400G at data centers. And, most servers are coming with at least 25G or 50G network adaptors now as a default configuration. This fact brings challenging questions from our customers and we are facing  iSCSI vs Fibre Channel storage network comparisons. Of course, priority is being competitive at the business and catching the future trends.

Before continuing, we assume you are familiar with block storage technologies. However, if you want a brief history of these protocols you can check one of the best technical deep-dive webcast about Fibre Channel and iSCSI comparison by SNIA.

Let's focus on our main subject here; performance and cost!

First of all, we prepared a testbed to compare apples to apples. This is a very important point to highlight our decision. Because we see lots of non-comparable test scenarios on the internet, I mean apples to oranges. For example, comparing 1G iSCSI vs 8G Fibre Channel. Also, we know Fibre Channel networks are dedicated over SAN switches but somehow iSCSI tests are conducted on a shared backbone or edge switches. Similarly comparing 40G iSCSI vs 16G Fibre Channel is nonsense at all. 

So what we did here is to create a dedicated TCP/IP network with 100G ethernet switches and a 32G Fibre Channel SAN network. And we started to test respectively 40G ethernet vs 32G FC then 100G ethernet vs 64G FC. We chose this because they are dominant current average connectivities at the market, and can be compared as apples to apples both in terms of price and performance.  

First of all, let's look at the economics;

 

iscsivsfceconomics

* Costs are calculated from the online list prices of these components.

 

While the most important thing is availability we should create our storage infrastructure redundantly. So we need at least 2 components from each to find the minimal cost per server to create redundant storage fabric. Below table shows costs per connectivity type only for a single server. However, you can calculate your own costs depending on your numbers easily.

 

iscsivsfceconomics

Expense ratios of the storage networks

 

According to cost tables, creating an FC SAN storage network is ~67% more expensive than an iSCSI network. But before deciding on your future infrastructure, we need performance numbers.

Test Environment Details

To avoid any storage bottlenecks we used the NGX Storage NS200 SCM system.  

Storage: NGX Storage NS200 SCM all-flash

  • 2TB DRAM
  • 24 Intel® Optane™ SSD DC P5800X 400GB
  • 4 x 32G Fibre Channel
  • 4 x 100GbE QSFP

Server : Dell R740 / OS Ubuntu 20.04 

  • 256GB Memory
  • Intel(R) Xeon(R) Gold 6142M CPU @ 2.60GHz
  • 2 x 32G Fibre Channel
  • 1 x 40G (Mellanox Connect-X 4)
  • 1 x 100G (Mellanox Connect-X 5)

 

iscsivsfctestbediSCSI vs FibreChannel Testbed

 

We created 8 logical units for test setup and exported them for both FC and iSCSI targets. These tests were conducted via  fio and vdbench which are industry standard benchmark tools. While both results are similar we are sharing fio results here to keep it short. Additionally, we chose an average block size 32k to generate enough throughput and IOPS that can stress this setup. Last but not least we used jumbo frames (MTU 9000) at the network config.

 

32g_fc_vs_40g_eth

32G FC vs 40G iSCSI

 

64g_fc_vs_100g_eth

64G FC vs 100G iSCSI

 

As we see from these test results iSCSI is as fast as expected. Because network adapters have more bandwidth. So we can say it's not an apple to apple comparison.  However, if we look from a price/performance perspective it definitely is. To clarify the whole picture, let's look below test results which are made with 4K block sizes without saturating the adapter bandwidths. This way we can compare FC vs iSCSI performances as a storage network and the protocols only.

 

32g_fc_vs_40g_eth_4k

32G FC vs 40G iSCSI (4k block)

 

Conclusion

Contrary to popular belief, iSCSI is not slow at all. Actually, it is one of the fastest block storage network today. While NGX Storage supports both FC and iSCSI in its products we can’t take sides between them. Therefore our customers should read and acknowledge this blog post as a technical highlight that helps understanding storage network performance numbers.

 

Notes: 

Networking

You should create a proper storage network while stability,  performance, and reliability are the main subjects. So, just like the SAN network, we need low latency, non-blocking, and lossless packet switching capabilities from the switch network. 

TCP Offload Engine     

Some vendors are providing iSCSI offload with their ToE supported cards. With iSCSI offload, your systems will benefit from low latency, higher IOPS, and lower processor utilization. 

NVMeOF

NVMe is a vital point when we consider future-ready storage fabrics. As a note, both Ethernet and FibreChannel support NVMe over Fabric on top of their network.

 

Reference:

  • https://en.wikipedia.org/wiki/Fibre_Channel
  • https://en.wikipedia.org/wiki/ISCSI
  • https://en.wikipedia.org/wiki/Jumbo_frame
  • https://en.wikipedia.org/wiki/TCP_offload_engine
  • https://fio.readthedocs.io/en/latest/
  • https://www.oracle.com/downloads/server-storage/vdbench-downloads.html

Fio Results:

# FC 32G randomwrite / 32K Block size

fio --rw=randwrite --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=24 --bs=32k --filename=/dev/sda:/dev/sdc:/dev/sdd:/dev/sde:/dev/sdf:/dev/sdg:/dev/sdh:/dev/sdi
random: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 24 processes
Jobs: 24 (f=192): [w(24)][100.0%][w=2899MiB/s][w=92.8k IOPS][eta 00m:00s]
random: (groupid=0, jobs=24): err= 0: pid=13243: Mon Aug 9 14:18:09 2021
write: IOPS=90.1k, BW=2815MiB/s (2951MB/s)(165GiB/60010msec); 0 zone resets
slat (usec): min=3, max=52314, avg=29.97, stdev=97.15
clat (nsec): min=1919, max=279448k, avg=8493051.20, stdev=19921490.32
lat (usec): min=97, max=279488, avg=8523.44, stdev=19921.32
clat percentiles (usec):
| 1.00th=[ 297], 5.00th=[ 709], 10.00th=[ 1020], 20.00th=[ 1450],
| 30.00th=[ 1827], 40.00th=[ 2212], 50.00th=[ 2671], 60.00th=[ 3228],
| 70.00th=[ 4080], 80.00th=[ 5932], 90.00th=[ 14484], 95.00th=[ 49021],
| 99.00th=[108528], 99.50th=[127402], 99.90th=[164627], 99.95th=[179307],
| 99.99th=[212861]
bw ( MiB/s): min= 1872, max= 4049, per=99.99%, avg=2814.28, stdev=15.86, samples=2880
iops : min=59926, max=129579, avg=90056.14, stdev=507.64, samples=2880
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.71%, 500=1.86%, 750=2.93%, 1000=4.11%
lat (msec) : 2=24.79%, 4=34.73%, 10=18.14%, 20=4.24%, 50=3.59%
lat (msec) : 100=3.58%, 250=1.31%, 500=0.01%
cpu : usr=3.47%, sys=11.15%, ctx=4692047, majf=0, minf=11550
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,5405019,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
WRITE: bw=2815MiB/s (2951MB/s), 2815MiB/s-2815MiB/s (2951MB/s-2951MB/s), io=165GiB (177GB), run=60010-60010msec

Disk stats (read/write):
sda: ios=76/672359, merge=0/61, ticks=2971/29410698, in_queue=28056724, util=99.80%
sdc: ios=80/675652, merge=0/2, ticks=85/1729028, in_queue=456664, util=99.82%
sdd: ios=89/675650, merge=0/1, ticks=165/1754645, in_queue=476600, util=99.83%
sde: ios=107/675640, merge=0/2, ticks=163/1736901, in_queue=458316, util=99.87%
sdf: ios=136/675628, merge=0/2, ticks=229/1675202, in_queue=414552, util=99.89%
sdg: ios=143/675626, merge=0/1, ticks=376/1683470, in_queue=415936, util=99.91%
sdh: ios=179/675623, merge=0/0, ticks=224/1720457, in_queue=453484, util=99.94%
sdi: ios=166/675609, merge=0/6, ticks=2374/5740380, in_queue=4325732, util=99.94%

# FC 32G randomread / 32K Block size

fio --rw=randread --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=24 --bs=32k --filename=/dev/sda:/dev/sdc:/dev/sdd:/dev/sde:/dev/sdf:/dev/sdg:/dev/sdh:/dev/sdi
random: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 24 processes
Jobs: 24 (f=192): [r(24)][100.0%][r=2762MiB/s][r=88.4k IOPS][eta 00m:00s]
random: (groupid=0, jobs=24): err= 0: pid=19896: Mon Aug 9 14:20:45 2021
read: IOPS=90.8k, BW=2838MiB/s (2976MB/s)(166GiB/60008msec)
slat (usec): min=3, max=647, avg=15.07, stdev=11.60
clat (usec): min=70, max=215045, avg=8437.26, stdev=11719.66
lat (usec): min=92, max=215057, avg=8452.73, stdev=11719.49
clat percentiles (usec):
| 1.00th=[ 1254], 5.00th=[ 1663], 10.00th=[ 1991], 20.00th=[ 2573],
| 30.00th=[ 3064], 40.00th=[ 3589], 50.00th=[ 4293], 60.00th=[ 5538],
| 70.00th=[ 7439], 80.00th=[ 10814], 90.00th=[ 18482], 95.00th=[ 29492],
| 99.00th=[ 61604], 99.50th=[ 76022], 99.90th=[109577], 99.95th=[125305],
| 99.99th=[162530]
bw ( MiB/s): min= 2042, max= 4014, per=99.99%, avg=2837.96, stdev=17.21, samples=2880
iops : min=65350, max=128472, avg=90814.26, stdev=550.78, samples=2880
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.04%, 1000=0.19%
lat (msec) : 2=9.90%, 4=36.44%, 10=31.46%, 20=12.95%, 50=7.27%
lat (msec) : 100=1.59%, 250=0.15%
cpu : usr=2.56%, sys=7.10%, ctx=5232143, majf=0, minf=15325
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=5450191,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
READ: bw=2838MiB/s (2976MB/s), 2838MiB/s-2838MiB/s (2976MB/s-2976MB/s), io=166GiB (179GB), run=60008-60008msec

Disk stats (read/write):
sda: ios=678997/0, merge=11/0, ticks=7050720/0, in_queue=5582820, util=99.62%
sdc: ios=679011/0, merge=17/0, ticks=7807170/0, in_queue=6350700, util=99.66%
sdd: ios=679034/0, merge=13/0, ticks=6416492/0, in_queue=4946616, util=99.68%
sde: ios=678979/0, merge=7/0, ticks=6162488/0, in_queue=4684856, util=99.88%
sdf: ios=679047/0, merge=8/0, ticks=5285084/0, in_queue=3805036, util=99.72%
sdg: ios=679055/0, merge=9/0, ticks=5711760/0, in_queue=4237272, util=99.76%
sdh: ios=678769/0, merge=11/0, ticks=5581873/0, in_queue=4098172, util=99.78%
sdi: ios=679061/0, merge=2/0, ticks=1766793/0, in_queue=205180, util=99.79%

# ISCSI 100G randomwrite / 32K Block size

fio --rw=randwrite --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=24 --bs=32k --filename=/dev/sda:/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde:/dev/sdf:/dev/sdg:/dev/sdh
random: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 24 processes
Jobs: 24 (f=192): [w(24)][100.0%][w=11.5GiB/s][w=376k IOPS][eta 00m:00s]
random: (groupid=0, jobs=24): err= 0: pid=26320: Mon Aug 9 14:28:52 2021
write: IOPS=369k, BW=11.3GiB/s (12.1GB/s)(675GiB/60004msec); 0 zone resets
slat (usec): min=2, max=22826, avg=37.43, stdev=201.56
clat (usec): min=12, max=158530, avg=2044.13, stdev=2397.96
lat (usec): min=55, max=158551, avg=2081.68, stdev=2450.77
clat percentiles (usec):
| 1.00th=[ 202], 5.00th=[ 277], 10.00th=[ 330], 20.00th=[ 457],
| 30.00th=[ 660], 40.00th=[ 889], 50.00th=[ 1205], 60.00th=[ 1598],
| 70.00th=[ 2147], 80.00th=[ 3064], 90.00th=[ 5014], 95.00th=[ 7111],
| 99.00th=[10159], 99.50th=[11600], 99.90th=[20579], 99.95th=[26084],
| 99.99th=[39584]
bw ( MiB/s): min= 3718, max=12540, per=99.99%, avg=11523.50, stdev=44.09, samples=2880
iops : min=119006, max=401282, avg=368751.24, stdev=1410.96, samples=2880
lat (usec) : 20=0.01%, 50=0.01%, 100=0.04%, 250=3.14%, 500=19.57%
lat (usec) : 750=11.42%, 1000=9.69%
lat (msec) : 2=23.88%, 4=18.27%, 10=12.88%, 20=1.00%, 50=0.10%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=5.62%, sys=19.98%, ctx=11399948, majf=0, minf=20652
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,22128185,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
WRITE: bw=11.3GiB/s (12.1GB/s), 11.3GiB/s-11.3GiB/s (12.1GB/s-12.1GB/s), io=675GiB (725GB), run=60004-60004msec

Disk stats (read/write):
sda: ios=99/2752163, merge=0/3140, ticks=51/4412371, in_queue=1425048, util=99.78%
sdb: ios=67/2750438, merge=0/4785, ticks=54/6071205, in_queue=2471248, util=99.84%
sdc: ios=73/2752430, merge=0/2897, ticks=52/4168589, in_queue=1410376, util=99.88%
sdd: ios=77/2752193, merge=0/3022, ticks=60/4352598, in_queue=1209072, util=99.87%
sde: ios=81/2752607, merge=0/2683, ticks=46/3783654, in_queue=1259132, util=99.92%
sdf: ios=85/2750446, merge=0/4806, ticks=70/5880030, in_queue=2479196, util=99.91%
sdg: ios=96/2752439, merge=0/2640, ticks=55/3764635, in_queue=1259592, util=99.93%
sdh: ios=99/2752171, merge=0/2907, ticks=58/3943191, in_queue=1467624, util=99.98%

# ISCSI 100G randomread / 32K Block size

fio --rw=randread --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=24 --bs=32k --filename=/dev/sda:/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde:/dev/sdf:/dev/sdg:/dev/sdh
random: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 24 processes
Jobs: 24 (f=192): [r(24)][100.0%][r=9.77GiB/s][r=320k IOPS][eta 00m:00s]
random: (groupid=0, jobs=24): err= 0: pid=28942: Mon Aug 9 14:31:01 2021
read: IOPS=311k, BW=9728MiB/s (10.2GB/s)(570GiB/60008msec)
slat (usec): min=2, max=15998, avg=54.88, stdev=311.12
clat (nsec): min=1346, max=117426k, avg=2411243.14, stdev=2988906.27
lat (usec): min=49, max=117431, avg=2466.22, stdev=3088.44
clat percentiles (usec):
| 1.00th=[ 133], 5.00th=[ 227], 10.00th=[ 285], 20.00th=[ 383],
| 30.00th=[ 529], 40.00th=[ 898], 50.00th=[ 1369], 60.00th=[ 1860],
| 70.00th=[ 2573], 80.00th=[ 3720], 90.00th=[ 6194], 95.00th=[ 8848],
| 99.00th=[13042], 99.50th=[14615], 99.90th=[21890], 99.95th=[28705],
| 99.99th=[44303]
bw ( MiB/s): min= 8219, max=11179, per=99.99%, avg=9727.58, stdev=22.70, samples=2880
iops : min=263008, max=357737, avg=311281.48, stdev=726.34, samples=2880
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.39%
lat (usec) : 250=6.44%, 500=21.95%, 750=8.05%, 1000=5.38%
lat (msec) : 2=20.31%, 4=19.10%, 10=14.88%, 20=3.39%, 50=0.12%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=2.24%, sys=13.14%, ctx=7622754, majf=0, minf=28545
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=18680735,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
READ: bw=9728MiB/s (10.2GB/s), 9728MiB/s-9728MiB/s (10.2GB/s-10.2GB/s), io=570GiB (612GB), run=60008-60008msec

Disk stats (read/write):
sda: ios=2325147/0, merge=2153/0, ticks=3577557/0, in_queue=1550356, util=99.81%
sdb: ios=2322967/0, merge=4363/0, ticks=5969166/0, in_queue=3269420, util=99.81%
sdc: ios=2325538/0, merge=1766/0, ticks=3107834/0, in_queue=1333404, util=99.84%
sdd: ios=2324621/0, merge=2602/0, ticks=4137631/0, in_queue=1904632, util=99.84%
sde: ios=2325548/0, merge=1832/0, ticks=3090372/0, in_queue=1324000, util=99.88%
sdf: ios=2323925/0, merge=3299/0, ticks=4792066/0, in_queue=2390652, util=99.88%
sdg: ios=2326375/0, merge=1038/0, ticks=2125565/0, in_queue=680320, util=99.91%
sdh: ios=2325828/0, merge=1374/0, ticks=2568624/0, in_queue=1053704, util=99.94%

# FC 64G randomwrite / 32K Block size

fio --rw=randwrite --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=24 --bs=32k --filename=/dev/sdf:/dev/sdg:/dev/sdh:/dev/sdi:/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde
random: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 24 processes
Jobs: 24 (f=192): [w(24)][100.0%][w=4409MiB/s][w=141k IOPS][eta 00m:00s]
random: (groupid=0, jobs=24): err= 0: pid=17176: Tue Aug 10 09:27:20 2021
write: IOPS=142k, BW=4432MiB/s (4647MB/s)(260GiB/60007msec); 0 zone resets
slat (usec): min=3, max=8465, avg=26.29, stdev=63.42
clat (nsec): min=1377, max=221987k, avg=5386099.00, stdev=10797360.99
lat (usec): min=87, max=222000, avg=5412.75, stdev=10797.10
clat percentiles (usec):
| 1.00th=[ 110], 5.00th=[ 155], 10.00th=[ 221], 20.00th=[ 404],
| 30.00th=[ 676], 40.00th=[ 1037], 50.00th=[ 1516], 60.00th=[ 2245],
| 70.00th=[ 3523], 80.00th=[ 6259], 90.00th=[ 14615], 95.00th=[ 26084],
| 99.00th=[ 55313], 99.50th=[ 67634], 99.90th=[ 93848], 99.95th=[105382],
| 99.99th=[130548]
bw ( MiB/s): min= 2872, max= 7320, per=99.99%, avg=4431.28, stdev=29.57, samples=2880
iops : min=91922, max=234254, avg=141800.09, stdev=946.25, samples=2880
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.31%, 250=11.58%, 500=11.97%, 750=8.44%, 1000=6.83%
lat (msec) : 2=18.01%, 4=15.35%, 10=13.54%, 20=6.83%, 50=5.78%
lat (msec) : 100=1.28%, 250=0.07%
cpu : usr=5.00%, sys=15.40%, ctx=7565473, majf=0, minf=10271
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,8509825,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
WRITE: bw=4432MiB/s (4647MB/s), 4432MiB/s-4432MiB/s (4647MB/s-4647MB/s), io=260GiB (279GB), run=60007-60007msec

Disk stats (read/write):
sdf: ios=77/1063700, merge=0/0, ticks=26/850388, in_queue=57456, util=99.72%
sdg: ios=87/1063711, merge=0/2, ticks=24/858639, in_queue=58676, util=99.77%
sdh: ios=103/1063485, merge=0/23, ticks=377/5969271, in_queue=4023072, util=99.82%
sdi: ios=139/1063555, merge=0/20, ticks=432/7080818, in_queue=5089340, util=99.85%
sdb: ios=158/1063407, merge=0/50, ticks=1337/12471721, in_queue=10439728, util=99.86%
sdc: ios=174/1063586, merge=0/44, ticks=1267/16249379, in_queue=14175540, util=99.90%
sdd: ios=191/1063701, merge=0/2, ticks=77/1015834, in_queue=94512, util=99.93%
sde: ios=190/1063661, merge=0/2, ticks=121/1105839, in_queue=115992, util=99.96%

# FC 64G randomread / 32K Block size

fio --rw=randread --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=24 --bs=32k --filename=/dev/sdf:/dev/sdg:/dev/sdh:/dev/sdi:/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde
random: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 24 processes
Jobs: 24 (f=192): [r(24)][100.0%][r=4922MiB/s][r=157k IOPS][eta 00m:00s]
random: (groupid=0, jobs=24): err= 0: pid=23220: Tue Aug 10 09:29:30 2021
read: IOPS=158k, BW=4924MiB/s (5163MB/s)(289GiB/60010msec)
slat (usec): min=4, max=497, avg=15.23, stdev=11.49
clat (usec): min=2, max=134519, avg=4856.01, stdev=8220.20
lat (usec): min=80, max=134534, avg=4871.60, stdev=8219.90
clat percentiles (usec):
| 1.00th=[ 302], 5.00th=[ 594], 10.00th=[ 775], 20.00th=[ 1057],
| 30.00th=[ 1336], 40.00th=[ 1647], 50.00th=[ 2024], 60.00th=[ 2638],
| 70.00th=[ 3687], 80.00th=[ 5800], 90.00th=[ 11469], 95.00th=[ 19530],
| 99.00th=[ 43254], 99.50th=[ 54264], 99.90th=[ 77071], 99.95th=[ 85459],
| 99.99th=[100140]
bw ( MiB/s): min= 3748, max= 6488, per=100.00%, avg=4923.52, stdev=28.18, samples=2880
iops : min=119944, max=207631, avg=157552.06, stdev=901.61, samples=2880
lat (usec) : 4=0.01%, 20=0.01%, 50=0.01%, 100=0.06%, 250=0.61%
lat (usec) : 500=2.55%, 750=5.95%, 1000=8.69%
lat (msec) : 2=31.66%, 4=22.61%, 10=16.19%, 20=6.81%, 50=4.20%
lat (msec) : 100=0.65%, 250=0.01%
cpu : usr=4.13%, sys=11.76%, ctx=8748698, majf=0, minf=15238
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=9454880,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
READ: bw=4924MiB/s (5163MB/s), 4924MiB/s-4924MiB/s (5163MB/s-5163MB/s), io=289GiB (310GB), run=60010-60010msec

Disk stats (read/write):
sdf: ios=1177865/0, merge=14/0, ticks=3958728/0, in_queue=2039648, util=99.73%
sdg: ios=1177859/0, merge=18/0, ticks=4488655/0, in_queue=2475428, util=99.76%
sdh: ios=1177840/0, merge=21/0, ticks=5190973/0, in_queue=3152964, util=99.81%
sdi: ios=1177860/0, merge=18/0, ticks=5451968/0, in_queue=3471140, util=99.85%
sdb: ios=1177591/0, merge=31/0, ticks=9595141/0, in_queue=7367924, util=99.86%
sdc: ios=1177707/0, merge=16/0, ticks=5389808/0, in_queue=3209780, util=99.90%
sdd: ios=1177868/0, merge=1/0, ticks=1444543/0, in_queue=15944, util=99.91%
sde: ios=1177642/0, merge=28/0, ticks=10157679/0, in_queue=7924056, util=99.93%

# ISCSI 40G randomwrite / 32K Block size

fio --rw=randwrite --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=8 --bs=32k --filename=/dev/sdi:/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde:/dev/sdf:/dev/sdg:/dev/sdh
random: (g=0): rw=randwrite, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 8 processes
Jobs: 8 (f=64): [w(8)][100.0%][w=4684MiB/s][w=150k IOPS][eta 00m:00s]
random: (groupid=0, jobs=8): err= 0: pid=36087: Tue Aug 24 16:11:06 2021
write: IOPS=150k, BW=4682MiB/s (4909MB/s)(274GiB/60003msec); 0 zone resets
slat (usec): min=2, max=1007, avg=22.79, stdev=11.97
clat (usec): min=51, max=83870, avg=1684.37, stdev=2338.39
lat (usec): min=87, max=83886, avg=1707.43, stdev=2337.62
clat percentiles (usec):
| 1.00th=[ 225], 5.00th=[ 306], 10.00th=[ 359], 20.00th=[ 433],
| 30.00th=[ 506], 40.00th=[ 603], 50.00th=[ 758], 60.00th=[ 1012],
| 70.00th=[ 1516], 80.00th=[ 2343], 90.00th=[ 4146], 95.00th=[ 6325],
| 99.00th=[11469], 99.50th=[13304], 99.90th=[18482], 99.95th=[23462],
| 99.99th=[38011]
bw ( MiB/s): min= 4549, max= 4796, per=99.98%, avg=4681.12, stdev= 5.73, samples=960
iops : min=145581, max=153500, avg=149795.53, stdev=183.24, samples=960
lat (usec) : 100=0.01%, 250=1.84%, 500=27.41%, 750=20.45%, 1000=9.98%
lat (msec) : 2=16.59%, 4=13.20%, 10=8.91%, 20=1.54%, 50=0.08%
lat (msec) : 100=0.01%
cpu : usr=7.36%, sys=44.99%, ctx=4371222, majf=0, minf=3772
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,8989672,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
WRITE: bw=4682MiB/s (4909MB/s), 4682MiB/s-4682MiB/s (4909MB/s-4909MB/s), io=274GiB (295GB), run=60003-60003msec

Disk stats (read/write):
sdi: ios=102/1120444, merge=0/402, ticks=110/1948704, in_queue=738576, util=99.87%
sdb: ios=108/1120264, merge=0/578, ticks=94/2457704, in_queue=1066512, util=99.89%
sdc: ios=69/1120464, merge=0/390, ticks=55/1932764, in_queue=700948, util=99.90%
sdd: ios=70/1120503, merge=0/351, ticks=56/1817100, in_queue=627440, util=99.90%
sde: ios=69/1120323, merge=0/399, ticks=62/1978459, in_queue=706900, util=99.92%
sdf: ios=67/1120618, merge=0/209, ticks=39/1486035, in_queue=369348, util=99.92%
sdg: ios=70/1120681, merge=0/160, ticks=40/1280505, in_queue=244416, util=99.95%
sdh: ios=68/1120398, merge=0/452, ticks=39/2032347, in_queue=790108, util=99.95%

# ISCSI 40G randomread / 32K Block size

fio --rw=randread --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=8 --bs=32k --filename=/dev/sdi:/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde:/dev/sdf:/dev/sdg:/dev/sdh
random: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 8 processes
Jobs: 8 (f=64): [r(8)][100.0%][r=4612MiB/s][r=148k IOPS][eta 00m:00s]
random: (groupid=0, jobs=8): err= 0: pid=37475: Tue Aug 24 16:13:02 2021
read: IOPS=147k, BW=4609MiB/s (4833MB/s)(270GiB/60005msec)
slat (nsec): min=1945, max=97891k, avg=16815.93, stdev=60543.38
clat (nsec): min=444, max=132128k, avg=1717382.04, stdev=2532988.67
lat (usec): min=78, max=132134, avg=1734.46, stdev=2533.63
clat percentiles (usec):
| 1.00th=[ 139], 5.00th=[ 192], 10.00th=[ 237], 20.00th=[ 326],
| 30.00th=[ 441], 40.00th=[ 594], 50.00th=[ 824], 60.00th=[ 1205],
| 70.00th=[ 1696], 80.00th=[ 2311], 90.00th=[ 4146], 95.00th=[ 6652],
| 99.00th=[11994], 99.50th=[13829], 99.90th=[19268], 99.95th=[25560],
| 99.99th=[44827]
bw ( MiB/s): min= 3921, max= 4780, per=99.99%, avg=4608.40, stdev=13.53, samples=960
iops : min=125500, max=152978, avg=147468.49, stdev=432.91, samples=960
lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.06%, 250=11.47%, 500=22.88%, 750=13.00%, 1000=7.77%
lat (msec) : 2=20.72%, 4=13.69%, 10=8.48%, 20=1.84%, 50=0.08%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=4.34%, sys=35.50%, ctx=4847513, majf=0, minf=4641
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=8849635,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
READ: bw=4609MiB/s (4833MB/s), 4609MiB/s-4609MiB/s (4833MB/s-4833MB/s), io=270GiB (290GB), run=60005-60005msec

Disk stats (read/write):
sdi: ios=1103545/0, merge=43/0, ticks=923489/0, in_queue=40488, util=99.68%
sdb: ios=1103391/0, merge=191/0, ticks=1463725/0, in_queue=318276, util=99.69%
sdc: ios=1103536/0, merge=51/0, ticks=921654/0, in_queue=64784, util=99.70%
sdd: ios=1103493/0, merge=83/0, ticks=1028444/0, in_queue=139716, util=99.86%
sde: ios=1103570/0, merge=14/0, ticks=827315/0, in_queue=1700, util=99.71%
sdf: ios=1102155/0, merge=1265/0, ticks=4257267/0, in_queue=2346016, util=99.72%
sdg: ios=1102327/0, merge=1240/0, ticks=4237243/0, in_queue=2315936, util=99.73%
sdh: ios=1103490/0, merge=96/0, ticks=1157271/0, in_queue=127324, util=99.75%

# ISCSI 40G randomwrite / 4K Block size

fio --rw=randwrite --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=24 --bs=4k --filename=/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde:/dev/sdf:/dev/sdg:/dev/sdi
random: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 24 processes
Jobs: 24 (f=168): [w(24)][100.0%][w=829MiB/s][w=212k IOPS][eta 00m:00s]
random: (groupid=0, jobs=24): err= 0: pid=12665: Mon Sep 6 15:28:41 2021
write: IOPS=198k, BW=775MiB/s (813MB/s)(45.4GiB/60006msec); 0 zone resets
slat (usec): min=2, max=7088, avg=22.03, stdev=61.21
clat (nsec): min=421, max=154679k, avg=3846762.03, stdev=4262129.77
lat (usec): min=53, max=154694, avg=3869.07, stdev=4266.94
clat percentiles (usec):
| 1.00th=[ 155], 5.00th=[ 293], 10.00th=[ 420], 20.00th=[ 635],
| 30.00th=[ 857], 40.00th=[ 1401], 50.00th=[ 2180], 60.00th=[ 3359],
| 70.00th=[ 4883], 80.00th=[ 6915], 90.00th=[ 9765], 95.00th=[11994],
| 99.00th=[16581], 99.50th=[19268], 99.90th=[34866], 99.95th=[44303],
| 99.99th=[64750]
bw ( KiB/s): min=596920, max=957281, per=99.99%, avg=793559.55, stdev=3257.39, samples=2880
iops : min=149230, max=239320, avg=198389.33, stdev=814.35, samples=2880
lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.17%, 250=3.40%, 500=10.14%, 750=11.88%, 1000=8.06%
lat (msec) : 2=14.33%, 4=16.60%, 10=26.04%, 20=8.93%, 50=0.40%
lat (msec) : 100=0.03%, 250=0.01%
cpu : usr=2.55%, sys=17.51%, ctx=8624821, majf=0, minf=3543
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,11905508,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
WRITE: bw=775MiB/s (813MB/s), 775MiB/s-775MiB/s (813MB/s-813MB/s), io=45.4GiB (48.8GB), run=60006-60006msec

# ISCSI 40G randomread / 4K Block size

fio --rw=randread --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=24 --bs=4k --filename=/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde:/dev/sdf:/dev/sdg:/dev/sdi
random: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 24 processes
Jobs: 24 (f=168): [r(24)][100.0%][r=774MiB/s][r=198k IOPS][eta 00m:00s]
random: (groupid=0, jobs=24): err= 0: pid=15458: Mon Sep 6 15:50:47 2021
read: IOPS=191k, BW=748MiB/s (784MB/s)(43.8GiB/60006msec)
slat (nsec): min=1448, max=7725.9k, avg=13066.78, stdev=88385.75
clat (nsec): min=453, max=133793k, avg=3998356.78, stdev=4780447.02
lat (usec): min=41, max=133799, avg=4011.54, stdev=4788.26
clat percentiles (usec):
| 1.00th=[ 101], 5.00th=[ 163], 10.00th=[ 235], 20.00th=[ 429],
| 30.00th=[ 562], 40.00th=[ 889], 50.00th=[ 1827], 60.00th=[ 3326],
| 70.00th=[ 5342], 80.00th=[ 7898], 90.00th=[11076], 95.00th=[13173],
| 99.00th=[16450], 99.50th=[17957], 99.90th=[39584], 99.95th=[49021],
| 99.99th=[69731]
bw ( KiB/s): min=694107, max=869120, per=99.99%, avg=765602.99, stdev=1348.88, samples=2880
iops : min=173526, max=217280, avg=191400.38, stdev=337.23, samples=2880
lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (usec) : 10=0.01%, 20=0.01%, 50=0.03%, 100=0.93%, 250=9.96%
lat (usec) : 500=14.47%, 750=12.39%, 1000=3.50%
lat (msec) : 2=10.14%, 4=12.15%, 10=23.47%, 20=12.57%, 50=0.33%
lat (msec) : 100=0.05%, 250=0.01%
cpu : usr=1.12%, sys=7.28%, ctx=9892062, majf=0, minf=3094
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=11485797,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
READ: bw=748MiB/s (784MB/s), 748MiB/s-748MiB/s (784MB/s-784MB/s), io=43.8GiB (47.0GB), run=60006-60006msec

Disk stats (read/write):
sdb: ios=1636185/0, merge=402/0, ticks=7250108/0, in_queue=4069412, util=99.81%
sdc: ios=1635633/0, merge=792/0, ticks=12912668/0, in_queue=9608452, util=99.83%
sdd: ios=1635621/0, merge=780/0, ticks=13037229/0, in_queue=9730660, util=99.83%
sde: ios=1635806/0, merge=595/0, ticks=10056963/0, in_queue=6780712, util=99.85%
sdf: ios=1636581/0, merge=0/0, ticks=733002/0, in_queue=380, util=99.85%
sdg: ios=1636578/0, merge=1/0, ticks=739285/0, in_queue=224, util=99.86%
sdi: ios=1636575/0, merge=1/0, ticks=747859/0, in_queue=2224, util=99.87%

# 32G FC randomwrite / 4k Block size

fio --rw=randwrite --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=24 --bs=4k --filename=/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde:/dev/sdf:/dev/sdg:/dev/sdi
random: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 24 processes
Jobs: 24 (f=168): [w(24)][100.0%][w=571MiB/s][w=146k IOPS][eta 00m:00s]
random: (groupid=0, jobs=24): err= 0: pid=19103: Mon Sep 6 12:39:37 2021
write: IOPS=149k, BW=583MiB/s (612MB/s)(34.2GiB/60011msec); 0 zone resets
slat (nsec): min=1536, max=8444.5k, avg=18333.39, stdev=12948.59
clat (nsec): min=968, max=206384k, avg=5120638.24, stdev=15466276.36
lat (usec): min=64, max=206398, avg=5139.42, stdev=15466.41
clat percentiles (usec):
| 1.00th=[ 83], 5.00th=[ 96], 10.00th=[ 114], 20.00th=[ 172],
| 30.00th=[ 285], 40.00th=[ 486], 50.00th=[ 791], 60.00th=[ 1221],
| 70.00th=[ 1844], 80.00th=[ 2999], 90.00th=[ 8160], 95.00th=[ 30278],
| 99.00th=[ 86508], 99.50th=[105382], 99.90th=[143655], 99.95th=[154141],
| 99.99th=[175113]
bw ( KiB/s): min=465344, max=753730, per=100.00%, avg=597355.98, stdev=2284.86, samples=2880
iops : min=116336, max=188432, avg=149338.43, stdev=571.22, samples=2880
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=6.28%, 250=21.17%, 500=13.12%, 750=8.19%, 1000=6.45%
lat (msec) : 2=16.69%, 4=12.38%, 10=6.63%, 20=2.48%, 50=3.68%
lat (msec) : 100=2.29%, 250=0.62%
cpu : usr=3.48%, sys=13.75%, ctx=8659903, majf=0, minf=2222
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,8962181,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
WRITE: bw=583MiB/s (612MB/s), 583MiB/s-583MiB/s (612MB/s-612MB/s), io=34.2GiB (36.7GB), run=60011-60011msec

Disk stats (read/write):
sdb: ios=60/1276063, merge=0/10, ticks=229/36668272, in_queue=34103068, util=99.86%
sdc: ios=75/1276708, merge=0/0, ticks=15/410877, in_queue=1128, util=99.91%
sdd: ios=79/1276642, merge=0/1, ticks=179/1678837, in_queue=374496, util=99.93%
sde: ios=96/1276704, merge=0/0, ticks=12/381689, in_queue=1324, util=99.95%
sdf: ios=106/1276699, merge=0/0, ticks=55/1522075, in_queue=281904, util=99.97%
sdg: ios=112/1276685, merge=0/2, ticks=190/3300093, in_queue=1135392, util=99.98%
sdi: ios=134/1276689, merge=0/0, ticks=76/1709044, in_queue=380036, util=99.99%

# 32G FC randomread / 4k Block size

fio --rw=randread --ioengine=libaio --name=random --size=20g --direct=1 --invalidate=1 --fsync_on_close=1 --norandommap --group_reporting --exitall --runtime=60 --time_based --iodepth=32 --numjobs=24 --bs=4k --filename=/dev/sdb:/dev/sdc:/dev/sdd:/dev/sde:/dev/sdf:/dev/sdg:/dev/sdi
random: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 24 processes
Jobs: 24 (f=168): [r(24)][100.0%][r=711MiB/s][r=182k IOPS][eta 00m:00s]
random: (groupid=0, jobs=24): err= 0: pid=22947: Mon Sep 6 12:41:45 2021
read: IOPS=174k, BW=681MiB/s (714MB/s)(39.9GiB/60008msec)
slat (usec): min=3, max=866, avg=13.71, stdev=16.95
clat (nsec): min=831, max=350900k, avg=4390257.27, stdev=9790207.13
lat (usec): min=54, max=350908, avg=4404.42, stdev=9790.54
clat percentiles (usec):
| 1.00th=[ 71], 5.00th=[ 103], 10.00th=[ 167], 20.00th=[ 314],
| 30.00th=[ 506], 40.00th=[ 758], 50.00th=[ 1123], 60.00th=[ 1762],
| 70.00th=[ 2933], 80.00th=[ 5211], 90.00th=[ 11076], 95.00th=[ 19792],
| 99.00th=[ 49546], 99.50th=[ 63701], 99.90th=[ 94897], 99.95th=[106431],
| 99.99th=[141558]
bw ( KiB/s): min=323440, max=805868, per=99.99%, avg=696941.88, stdev=2159.13, samples=2880
iops : min=80860, max=201467, avg=174234.88, stdev=539.78, samples=2880
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.05%
lat (usec) : 100=4.60%, 250=11.17%, 500=13.87%, 750=10.04%, 1000=7.32%
lat (msec) : 2=15.47%, 4=13.00%, 10=13.32%, 20=6.23%, 50=3.92%
lat (msec) : 100=0.90%, 250=0.07%, 500=0.01%
cpu : usr=3.82%, sys=12.08%, ctx=9466133, majf=0, minf=1922
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=10456193,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
READ: bw=681MiB/s (714MB/s), 681MiB/s-681MiB/s (714MB/s-714MB/s), io=39.9GiB (42.8GB), run=60008-60008msec

Disk stats (read/write):
sdb: ios=1489948/0, merge=10/0, ticks=12879145/0, in_queue=10228592, util=99.85%
sdc: ios=1490311/0, merge=0/0, ticks=922770/0, in_queue=22108, util=99.41%
sdd: ios=1490078/0, merge=7/0, ticks=9530967/0, in_queue=6992564, util=99.42%
sde: ios=1490308/0, merge=0/0, ticks=537624/0, in_queue=8, util=99.48%
sdf: ios=1490288/0, merge=9/0, ticks=10328780/0, in_queue=7750484, util=99.43%
sdg: ios=1490305/0, merge=0/0, ticks=932092/0, in_queue=26688, util=99.45%
sdi: ios=1490132/0, merge=8/0, ticks=10470841/0, in_queue=7889748, util=99.45%