== sanity test 398c: run fio to test AIO ================= 03:06:38 (1747811198)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0863035 s, 486 MB/s
osc.lustre-OST0000-osc-ffff88012b775000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=20961: Wed May 21 03:06:44 2025
  write: IOPS=623, BW=2495KiB/s (2554kB/s)(10.0MiB/4105msec)
    slat (usec): min=14, max=492, avg=56.34, stdev=41.31
    clat (usec): min=1972, max=72519, avg=25440.94, stdev=7275.14
     lat (usec): min=2019, max=72654, avg=25498.66, stdev=7289.88
    clat percentiles (usec):
     |  1.00th=[15533],  5.00th=[17957], 10.00th=[19006], 20.00th=[20317],
     | 30.00th=[21365], 40.00th=[22414], 50.00th=[22938], 60.00th=[23987],
     | 70.00th=[25822], 80.00th=[31327], 90.00th=[37487], 95.00th=[39584],
     | 99.00th=[44827], 99.50th=[48497], 99.90th=[69731], 99.95th=[69731],
     | 99.99th=[72877]
   bw (  KiB/s): min= 1608, max= 3232, per=25.24%, avg=2518.00, stdev=547.50, samples=8
   iops        : min=  402, max=  808, avg=629.50, stdev=136.87, samples=8
  lat (msec)   : 2=0.04%, 4=0.08%, 10=0.04%, 20=16.84%, 50=82.54%
  lat (msec)   : 100=0.47%
  cpu          : usr=0.95%, sys=3.58%, ctx=1616, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=20962: Wed May 21 03:06:44 2025
  write: IOPS=623, BW=2495KiB/s (2554kB/s)(10.0MiB/4105msec)
    slat (usec): min=13, max=598, avg=53.85, stdev=40.85
    clat (usec): min=3324, max=56978, avg=25414.95, stdev=7026.53
     lat (usec): min=3374, max=57154, avg=25470.11, stdev=7041.14
    clat percentiles (usec):
     |  1.00th=[15533],  5.00th=[17957], 10.00th=[19006], 20.00th=[20317],
     | 30.00th=[21365], 40.00th=[22152], 50.00th=[23200], 60.00th=[23987],
     | 70.00th=[25822], 80.00th=[31589], 90.00th=[37487], 95.00th=[39584],
     | 99.00th=[45351], 99.50th=[45876], 99.90th=[56886], 99.95th=[56886],
     | 99.99th=[56886]
   bw (  KiB/s): min= 1584, max= 3200, per=25.14%, avg=2508.00, stdev=547.52, samples=8
   iops        : min=  396, max=  800, avg=627.00, stdev=136.88, samples=8
  lat (msec)   : 4=0.04%, 10=0.12%, 20=16.45%, 50=83.05%, 100=0.35%
  cpu          : usr=0.61%, sys=3.61%, ctx=1449, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=20963: Wed May 21 03:06:44 2025
  write: IOPS=629, BW=2518KiB/s (2578kB/s)(10.0MiB/4067msec)
    slat (usec): min=13, max=551, avg=56.11, stdev=39.76
    clat (usec): min=2681, max=66010, avg=25244.16, stdev=7083.21
     lat (usec): min=2751, max=66171, avg=25301.65, stdev=7097.76
    clat percentiles (usec):
     |  1.00th=[14746],  5.00th=[17695], 10.00th=[18744], 20.00th=[20317],
     | 30.00th=[21103], 40.00th=[22152], 50.00th=[22938], 60.00th=[23725],
     | 70.00th=[25560], 80.00th=[30802], 90.00th=[36963], 95.00th=[39584],
     | 99.00th=[44303], 99.50th=[45876], 99.90th=[56361], 99.95th=[61080],
     | 99.99th=[65799]
   bw (  KiB/s): min= 1596, max= 3201, per=25.34%, avg=2528.62, stdev=550.75, samples=8
   iops        : min=  399, max=  800, avg=632.12, stdev=137.64, samples=8
  lat (msec)   : 4=0.08%, 10=0.23%, 20=17.34%, 50=82.07%, 100=0.27%
  cpu          : usr=0.76%, sys=3.84%, ctx=1600, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=20964: Wed May 21 03:06:44 2025
  write: IOPS=623, BW=2495KiB/s (2554kB/s)(10.0MiB/4105msec)
    slat (usec): min=14, max=407, avg=57.10, stdev=38.38
    clat (usec): min=3539, max=72740, avg=25469.34, stdev=7279.56
     lat (usec): min=3653, max=72776, avg=25527.77, stdev=7292.54
    clat percentiles (usec):
     |  1.00th=[15401],  5.00th=[17957], 10.00th=[19006], 20.00th=[20317],
     | 30.00th=[21365], 40.00th=[22414], 50.00th=[22938], 60.00th=[23987],
     | 70.00th=[25560], 80.00th=[31327], 90.00th=[37487], 95.00th=[40109],
     | 99.00th=[44303], 99.50th=[48497], 99.90th=[69731], 99.95th=[72877],
     | 99.99th=[72877]
   bw (  KiB/s): min= 1608, max= 3208, per=25.24%, avg=2518.00, stdev=546.73, samples=8
   iops        : min=  402, max=  802, avg=629.50, stdev=136.68, samples=8
  lat (msec)   : 4=0.04%, 10=0.08%, 20=15.74%, 50=83.71%, 100=0.43%
  cpu          : usr=0.51%, sys=4.12%, ctx=1762, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=9978KiB/s (10.2MB/s), 2495KiB/s-2518KiB/s (2554kB/s-2578kB/s), io=40.0MiB (41.9MB), run=4067-4105msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=20977: Wed May 21 03:06:47 2025
   read: IOPS=435, BW=1740KiB/s (1782kB/s)(5048KiB/2901msec)
    slat (usec): min=721, max=2210, avg=1167.02, stdev=220.47
    clat (usec): min=9145, max=32300, avg=17177.04, stdev=2305.68
     lat (usec): min=10384, max=33420, avg=18345.76, stdev=2322.10
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13566], 10.00th=[14222], 20.00th=[15139],
     | 30.00th=[15926], 40.00th=[16581], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18482], 80.00th=[19006], 90.00th=[20055], 95.00th=[20841],
     | 99.00th=[22676], 99.50th=[23725], 99.90th=[31065], 99.95th=[32375],
     | 99.99th=[32375]
   bw (  KiB/s): min= 1608, max= 1768, per=24.91%, avg=1721.60, stdev=64.60, samples=5
   iops        : min=  402, max=  442, avg=430.40, stdev=16.15, samples=5
  write: IOPS=447, BW=1790KiB/s (1833kB/s)(5192KiB/2901msec)
    slat (usec): min=14, max=1153, avg=45.01, stdev=37.56
    clat (usec): min=9703, max=30119, avg=17606.08, stdev=1879.15
     lat (usec): min=9767, max=30150, avg=17652.08, stdev=1878.64
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[14746], 10.00th=[15401], 20.00th=[16188],
     | 30.00th=[16581], 40.00th=[17171], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[18482], 80.00th=[19006], 90.00th=[19792], 95.00th=[20579],
     | 99.00th=[22676], 99.50th=[22938], 99.90th=[26346], 99.95th=[30016],
     | 99.99th=[30016]
   bw (  KiB/s): min= 1728, max= 1904, per=25.04%, avg=1801.60, stdev=67.50, samples=5
   iops        : min=  432, max=  476, avg=450.40, stdev=16.88, samples=5
  lat (msec)   : 10=0.12%, 20=90.55%, 50=9.34%
  cpu          : usr=1.00%, sys=6.86%, ctx=2431, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=20978: Wed May 21 03:06:47 2025
   read: IOPS=428, BW=1714KiB/s (1755kB/s)(4976KiB/2903msec)
    slat (usec): min=656, max=2303, avg=1188.59, stdev=229.37
    clat (usec): min=3490, max=26876, avg=17109.49, stdev=2495.25
     lat (usec): min=4426, max=28497, avg=18299.98, stdev=2518.36
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[13304], 10.00th=[13960], 20.00th=[15139],
     | 30.00th=[15926], 40.00th=[16712], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[19006], 90.00th=[20055], 95.00th=[21103],
     | 99.00th=[23462], 99.50th=[24249], 99.90th=[25560], 99.95th=[26870],
     | 99.99th=[26870]
   bw (  KiB/s): min= 1536, max= 1744, per=24.40%, avg=1686.40, stdev=84.93, samples=5
   iops        : min=  384, max=  436, avg=421.60, stdev=21.23, samples=5
  write: IOPS=453, BW=1813KiB/s (1857kB/s)(5264KiB/2903msec)
    slat (usec): min=14, max=283, avg=45.11, stdev=24.76
    clat (usec): min=3433, max=25988, avg=17683.84, stdev=1931.68
     lat (usec): min=3482, max=26038, avg=17730.19, stdev=1929.83
    clat percentiles (usec):
     |  1.00th=[13698],  5.00th=[14746], 10.00th=[15270], 20.00th=[16188],
     | 30.00th=[16712], 40.00th=[17171], 50.00th=[17695], 60.00th=[18220],
     | 70.00th=[18482], 80.00th=[19006], 90.00th=[20055], 95.00th=[20841],
     | 99.00th=[22938], 99.50th=[23987], 99.90th=[25822], 99.95th=[26084],
     | 99.99th=[26084]
   bw (  KiB/s): min= 1776, max= 1912, per=25.28%, avg=1819.20, stdev=57.85, samples=5
   iops        : min=  444, max=  478, avg=454.80, stdev=14.46, samples=5
  lat (msec)   : 4=0.12%, 10=0.35%, 20=89.26%, 50=10.27%
  cpu          : usr=0.55%, sys=7.17%, ctx=2496, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=20979: Wed May 21 03:06:47 2025
   read: IOPS=432, BW=1728KiB/s (1770kB/s)(4984KiB/2884msec)
    slat (usec): min=23, max=3025, avg=1164.13, stdev=247.86
    clat (usec): min=1380, max=36971, avg=17183.62, stdev=2690.49
     lat (usec): min=1414, max=37818, avg=18349.44, stdev=2719.21
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[13304], 10.00th=[14222], 20.00th=[15139],
     | 30.00th=[15926], 40.00th=[16581], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18482], 80.00th=[19006], 90.00th=[20055], 95.00th=[21365],
     | 99.00th=[23462], 99.50th=[24773], 99.90th=[36439], 99.95th=[36963],
     | 99.99th=[36963]
   bw (  KiB/s): min= 1544, max= 1896, per=24.81%, avg=1715.20, stdev=134.30, samples=5
   iops        : min=  386, max=  474, avg=428.80, stdev=33.57, samples=5
  write: IOPS=455, BW=1822KiB/s (1866kB/s)(5256KiB/2884msec)
    slat (usec): min=14, max=310, avg=45.40, stdev=24.63
    clat (usec): min=6561, max=34156, avg=17574.37, stdev=2175.66
     lat (usec): min=6844, max=34200, avg=17620.84, stdev=2174.38
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[14484], 10.00th=[15139], 20.00th=[16057],
     | 30.00th=[16581], 40.00th=[17171], 50.00th=[17695], 60.00th=[17957],
     | 70.00th=[18482], 80.00th=[19006], 90.00th=[20055], 95.00th=[20841],
     | 99.00th=[22938], 99.50th=[24511], 99.90th=[33162], 99.95th=[34341],
     | 99.99th=[34341]
   bw (  KiB/s): min= 1720, max= 1960, per=25.46%, avg=1832.00, stdev=105.98, samples=5
   iops        : min=  430, max=  490, avg=458.00, stdev=26.50, samples=5
  lat (msec)   : 2=0.08%, 10=0.43%, 20=88.75%, 50=10.74%
  cpu          : usr=1.11%, sys=6.76%, ctx=2411, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=20980: Wed May 21 03:06:47 2025
   read: IOPS=437, BW=1748KiB/s (1790kB/s)(5060KiB/2894msec)
    slat (usec): min=700, max=2268, avg=1160.07, stdev=226.51
    clat (usec): min=1586, max=27663, avg=17171.44, stdev=2368.92
     lat (usec): min=2440, max=28840, avg=18333.32, stdev=2380.38
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[13435], 10.00th=[14222], 20.00th=[15270],
     | 30.00th=[16057], 40.00th=[16712], 50.00th=[17171], 60.00th=[17957],
     | 70.00th=[18482], 80.00th=[19006], 90.00th=[20055], 95.00th=[20579],
     | 99.00th=[22938], 99.50th=[23987], 99.90th=[26608], 99.95th=[27657],
     | 99.99th=[27657]
   bw (  KiB/s): min= 1672, max= 1824, per=25.12%, avg=1736.00, stdev=64.75, samples=5
   iops        : min=  418, max=  456, avg=434.00, stdev=16.19, samples=5
  write: IOPS=447, BW=1790KiB/s (1833kB/s)(5180KiB/2894msec)
    slat (usec): min=14, max=1585, avg=47.73, stdev=50.70
    clat (usec): min=9915, max=32504, avg=17588.10, stdev=1999.71
     lat (usec): min=9949, max=32561, avg=17636.77, stdev=1997.71
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[14615], 10.00th=[15270], 20.00th=[16057],
     | 30.00th=[16581], 40.00th=[16909], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[18482], 80.00th=[19006], 90.00th=[20055], 95.00th=[20841],
     | 99.00th=[22676], 99.50th=[23725], 99.90th=[28967], 99.95th=[32375],
     | 99.99th=[32375]
   bw (  KiB/s): min= 1736, max= 1864, per=24.95%, avg=1795.20, stdev=52.34, samples=5
   iops        : min=  434, max=  466, avg=448.80, stdev=13.08, samples=5
  lat (msec)   : 2=0.04%, 10=0.31%, 20=89.69%, 50=9.96%
  cpu          : usr=0.62%, sys=7.29%, ctx=2494, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6913KiB/s (7079kB/s), 1714KiB/s-1748KiB/s (1755kB/s-1790kB/s), io=19.6MiB (20.5MB), run=2884-2903msec
  WRITE: bw=7197KiB/s (7369kB/s), 1790KiB/s-1822KiB/s (1833kB/s-1866kB/s), io=20.4MiB (21.4MB), run=2884-2903msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=20986: Wed May 21 03:06:49 2025
   read: IOPS=10, BW=430MiB/s (451MB/s)(40.0MiB/93msec)
    slat (nsec): min=7824.7k, max=7824.7k, avg=7824688.00, stdev= 0.00
    clat (nsec): min=80334k, max=80334k, avg=80333663.00, stdev= 0.00
     lat (nsec): min=88179k, max=88179k, avg=88179270.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[80217],  5.00th=[80217], 10.00th=[80217], 20.00th=[80217],
     | 30.00th=[80217], 40.00th=[80217], 50.00th=[80217], 60.00th=[80217],
     | 70.00th=[80217], 80.00th=[80217], 90.00th=[80217], 95.00th=[80217],
     | 99.00th=[80217], 99.50th=[80217], 99.90th=[80217], 99.95th=[80217],
     | 99.99th=[80217]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.70%, ctx=3, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=430MiB/s (451MB/s), 430MiB/s-430MiB/s (451MB/s-451MB/s), io=40.0MiB (41.9MB), run=93-93msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout