== sanity test 398c: run fio to test AIO ================= 10:41:45 (1724769705)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0867204 s, 484 MB/s
osc.lustre-OST0000-osc-ffff88012a631000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=10178: Tue Aug 27 10:41:50 2024
  write: IOPS=730, BW=2923KiB/s (2993kB/s)(10.0MiB/3503msec)
    slat (usec): min=12, max=402, avg=52.95, stdev=29.14
    clat (usec): min=2337, max=44885, avg=21761.82, stdev=2932.11
     lat (usec): min=2394, max=44901, avg=21816.12, stdev=2931.70
    clat percentiles (usec):
     |  1.00th=[13698],  5.00th=[17695], 10.00th=[18744], 20.00th=[20055],
     | 30.00th=[20841], 40.00th=[21365], 50.00th=[21890], 60.00th=[22152],
     | 70.00th=[22676], 80.00th=[23462], 90.00th=[24249], 95.00th=[25560],
     | 99.00th=[31327], 99.50th=[35390], 99.90th=[40109], 99.95th=[40109],
     | 99.99th=[44827]
   bw (  KiB/s): min= 2648, max= 3160, per=24.99%, avg=2908.57, stdev=196.01, samples=7
   iops        : min=  662, max=  790, avg=727.14, stdev=49.00, samples=7
  lat (msec)   : 4=0.08%, 10=0.47%, 20=18.16%, 50=81.29%
  cpu          : usr=0.71%, sys=3.97%, ctx=2172, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=10179: Tue Aug 27 10:41:50 2024
  write: IOPS=727, BW=2911KiB/s (2981kB/s)(10.0MiB/3518msec)
    slat (usec): min=12, max=486, avg=49.33, stdev=28.18
    clat (usec): min=4981, max=35185, avg=21823.65, stdev=2455.81
     lat (usec): min=5060, max=35214, avg=21874.21, stdev=2456.18
    clat percentiles (usec):
     |  1.00th=[15401],  5.00th=[17695], 10.00th=[19006], 20.00th=[20317],
     | 30.00th=[20841], 40.00th=[21365], 50.00th=[21890], 60.00th=[22414],
     | 70.00th=[22938], 80.00th=[23462], 90.00th=[24249], 95.00th=[25560],
     | 99.00th=[28443], 99.50th=[32113], 99.90th=[35390], 99.95th=[35390],
     | 99.99th=[35390]
   bw (  KiB/s): min= 2744, max= 3088, per=24.89%, avg=2897.14, stdev=143.66, samples=7
   iops        : min=  686, max=  772, avg=724.29, stdev=35.92, samples=7
  lat (msec)   : 10=0.08%, 20=16.64%, 50=83.28%
  cpu          : usr=0.45%, sys=4.01%, ctx=1783, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=10180: Tue Aug 27 10:41:50 2024
  write: IOPS=728, BW=2914KiB/s (2984kB/s)(10.0MiB/3514msec)
    slat (usec): min=13, max=366, avg=49.63, stdev=29.09
    clat (usec): min=4025, max=36547, avg=21810.82, stdev=2655.58
     lat (usec): min=4079, max=36588, avg=21861.79, stdev=2657.51
    clat percentiles (usec):
     |  1.00th=[15139],  5.00th=[17695], 10.00th=[18744], 20.00th=[20055],
     | 30.00th=[20841], 40.00th=[21365], 50.00th=[21890], 60.00th=[22414],
     | 70.00th=[22938], 80.00th=[23462], 90.00th=[24511], 95.00th=[25560],
     | 99.00th=[28967], 99.50th=[31327], 99.90th=[35390], 99.95th=[35390],
     | 99.99th=[36439]
   bw (  KiB/s): min= 2760, max= 3136, per=24.98%, avg=2907.57, stdev=158.88, samples=7
   iops        : min=  690, max=  784, avg=726.86, stdev=39.68, samples=7
  lat (msec)   : 10=0.66%, 20=18.05%, 50=81.29%
  cpu          : usr=0.71%, sys=3.50%, ctx=1934, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=10181: Tue Aug 27 10:41:50 2024
  write: IOPS=727, BW=2910KiB/s (2980kB/s)(10.0MiB/3519msec)
    slat (usec): min=12, max=393, avg=46.42, stdev=26.94
    clat (usec): min=12494, max=40826, avg=21862.56, stdev=2533.93
     lat (usec): min=12563, max=40862, avg=21910.18, stdev=2536.09
    clat percentiles (usec):
     |  1.00th=[14877],  5.00th=[17957], 10.00th=[19268], 20.00th=[20055],
     | 30.00th=[20841], 40.00th=[21365], 50.00th=[21890], 60.00th=[22414],
     | 70.00th=[22938], 80.00th=[23462], 90.00th=[24249], 95.00th=[25822],
     | 99.00th=[28705], 99.50th=[32113], 99.90th=[34341], 99.95th=[34341],
     | 99.99th=[40633]
   bw (  KiB/s): min= 2752, max= 3184, per=24.98%, avg=2907.43, stdev=170.01, samples=7
   iops        : min=  688, max=  796, avg=726.86, stdev=42.50, samples=7
  lat (msec)   : 20=17.34%, 50=82.66%
  cpu          : usr=0.63%, sys=3.52%, ctx=1666, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=11.4MiB/s (11.9MB/s), 2910KiB/s-2923KiB/s (2980kB/s-2993kB/s), io=40.0MiB (41.9MB), run=3503-3519msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=10193: Tue Aug 27 10:41:54 2024
   read: IOPS=444, BW=1777KiB/s (1820kB/s)(5048KiB/2840msec)
    slat (usec): min=21, max=2698, avg=1131.22, stdev=242.60
    clat (usec): min=1694, max=31215, avg=16867.77, stdev=2514.42
     lat (usec): min=1733, max=32382, avg=18000.95, stdev=2563.38
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[13173], 10.00th=[13960], 20.00th=[15008],
     | 30.00th=[15795], 40.00th=[16450], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[19530], 95.00th=[20579],
     | 99.00th=[22676], 99.50th=[24511], 99.90th=[29754], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 1736, max= 1792, per=25.27%, avg=1772.80, stdev=21.61, samples=5
   iops        : min=  434, max=  448, avg=443.20, stdev= 5.40, samples=5
  write: IOPS=457, BW=1828KiB/s (1872kB/s)(5192KiB/2840msec)
    slat (usec): min=14, max=1342, avg=47.54, stdev=44.54
    clat (usec): min=2516, max=29748, avg=17368.54, stdev=2030.19
     lat (usec): min=2550, max=29776, avg=17417.32, stdev=2031.65
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[14615], 10.00th=[15139], 20.00th=[15926],
     | 30.00th=[16450], 40.00th=[16909], 50.00th=[17433], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[19006], 90.00th=[19792], 95.00th=[20579],
     | 99.00th=[22152], 99.50th=[22938], 99.90th=[27919], 99.95th=[29754],
     | 99.99th=[29754]
   bw (  KiB/s): min= 1760, max= 1920, per=25.37%, avg=1852.80, stdev=62.12, samples=5
   iops        : min=  440, max=  480, avg=463.20, stdev=15.53, samples=5
  lat (msec)   : 2=0.12%, 4=0.08%, 10=0.55%, 20=91.37%, 50=7.89%
  cpu          : usr=0.70%, sys=7.01%, ctx=2974, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=10194: Tue Aug 27 10:41:54 2024
   read: IOPS=434, BW=1740KiB/s (1782kB/s)(4976KiB/2860msec)
    slat (usec): min=680, max=2864, avg=1132.99, stdev=234.28
    clat (usec): min=8742, max=23828, avg=16877.15, stdev=2154.65
     lat (usec): min=9714, max=24819, avg=18011.89, stdev=2172.27
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[13435], 10.00th=[14222], 20.00th=[15008],
     | 30.00th=[15664], 40.00th=[16188], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[19792], 95.00th=[20317],
     | 99.00th=[22414], 99.50th=[22676], 99.90th=[23200], 99.95th=[23725],
     | 99.99th=[23725]
   bw (  KiB/s): min= 1608, max= 1824, per=24.58%, avg=1724.80, stdev=86.46, samples=5
   iops        : min=  402, max=  456, avg=431.20, stdev=21.61, samples=5
  write: IOPS=460, BW=1841KiB/s (1885kB/s)(5264KiB/2860msec)
    slat (usec): min=14, max=381, avg=45.76, stdev=24.96
    clat (usec): min=8623, max=24137, avg=17443.07, stdev=1821.45
     lat (usec): min=8735, max=24178, avg=17490.12, stdev=1820.38
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[14615], 10.00th=[15139], 20.00th=[15926],
     | 30.00th=[16581], 40.00th=[16909], 50.00th=[17433], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[19006], 90.00th=[19792], 95.00th=[20579],
     | 99.00th=[22152], 99.50th=[22676], 99.90th=[23987], 99.95th=[24249],
     | 99.99th=[24249]
   bw (  KiB/s): min= 1768, max= 1936, per=25.43%, avg=1857.60, stdev=67.74, samples=5
   iops        : min=  442, max=  484, avg=464.40, stdev=16.94, samples=5
  lat (msec)   : 10=0.12%, 20=91.76%, 50=8.12%
  cpu          : usr=0.73%, sys=6.96%, ctx=2851, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=10195: Tue Aug 27 10:41:54 2024
   read: IOPS=435, BW=1743KiB/s (1784kB/s)(4984KiB/2860msec)
    slat (usec): min=650, max=2741, avg=1145.68, stdev=233.53
    clat (usec): min=11276, max=31274, avg=16933.69, stdev=2265.54
     lat (usec): min=12404, max=32289, avg=18081.14, stdev=2279.10
    clat percentiles (usec):
     |  1.00th=[12125],  5.00th=[13435], 10.00th=[14091], 20.00th=[15008],
     | 30.00th=[15795], 40.00th=[16319], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[19792], 95.00th=[20579],
     | 99.00th=[22938], 99.50th=[24249], 99.90th=[30278], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 1664, max= 1928, per=24.90%, avg=1747.20, stdev=103.78, samples=5
   iops        : min=  416, max=  482, avg=436.80, stdev=25.95, samples=5
  write: IOPS=459, BW=1838KiB/s (1882kB/s)(5256KiB/2860msec)
    slat (usec): min=13, max=167, avg=46.34, stdev=20.94
    clat (usec): min=11757, max=31149, avg=17385.38, stdev=1905.14
     lat (usec): min=11825, max=31179, avg=17433.00, stdev=1904.52
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[14746], 10.00th=[15139], 20.00th=[15926],
     | 30.00th=[16319], 40.00th=[16909], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[18744], 90.00th=[19530], 95.00th=[20317],
     | 99.00th=[22414], 99.50th=[23725], 99.90th=[31065], 99.95th=[31065],
     | 99.99th=[31065]
   bw (  KiB/s): min= 1736, max= 1968, per=25.26%, avg=1844.80, stdev=108.61, samples=5
   iops        : min=  434, max=  492, avg=461.20, stdev=27.15, samples=5
  lat (msec)   : 20=92.62%, 50=7.38%
  cpu          : usr=0.42%, sys=7.21%, ctx=2921, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=10196: Tue Aug 27 10:41:54 2024
   read: IOPS=442, BW=1771KiB/s (1814kB/s)(5060KiB/2857msec)
    slat (usec): min=711, max=2704, avg=1134.92, stdev=223.09
    clat (usec): min=7871, max=25021, avg=16801.47, stdev=2206.83
     lat (usec): min=8810, max=26246, avg=17938.08, stdev=2218.16
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13304], 10.00th=[13960], 20.00th=[15008],
     | 30.00th=[15533], 40.00th=[16057], 50.00th=[16712], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[19530], 95.00th=[20317],
     | 99.00th=[22152], 99.50th=[22938], 99.90th=[24773], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 1688, max= 1832, per=25.27%, avg=1772.80, stdev=59.22, samples=5
   iops        : min=  422, max=  458, avg=443.20, stdev=14.81, samples=5
  write: IOPS=453, BW=1813KiB/s (1857kB/s)(5180KiB/2857msec)
    slat (usec): min=12, max=374, avg=46.12, stdev=25.06
    clat (usec): min=10753, max=32417, avg=17525.09, stdev=1884.67
     lat (usec): min=10782, max=32455, avg=17572.42, stdev=1885.28
    clat percentiles (usec):
     |  1.00th=[13173],  5.00th=[14615], 10.00th=[15139], 20.00th=[16057],
     | 30.00th=[16581], 40.00th=[16909], 50.00th=[17433], 60.00th=[17957],
     | 70.00th=[18482], 80.00th=[19006], 90.00th=[19792], 95.00th=[20579],
     | 99.00th=[21627], 99.50th=[22676], 99.90th=[28705], 99.95th=[32375],
     | 99.99th=[32375]
   bw (  KiB/s): min= 1744, max= 1880, per=24.82%, avg=1812.80, stdev=63.65, samples=5
   iops        : min=  436, max=  470, avg=453.20, stdev=15.91, samples=5
  lat (msec)   : 10=0.04%, 20=92.23%, 50=7.73%
  cpu          : usr=0.70%, sys=6.97%, ctx=2992, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7017KiB/s (7185kB/s), 1740KiB/s-1777KiB/s (1782kB/s-1820kB/s), io=19.6MiB (20.5MB), run=2840-2860msec
  WRITE: bw=7305KiB/s (7480kB/s), 1813KiB/s-1841KiB/s (1857kB/s-1885kB/s), io=20.4MiB (21.4MB), run=2840-2860msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=10202: Tue Aug 27 10:41:55 2024
   read: IOPS=13, BW=533MiB/s (559MB/s)(40.0MiB/75msec)
    slat (nsec): min=6911.9k, max=6911.9k, avg=6911855.00, stdev= 0.00
    clat (nsec): min=63055k, max=63055k, avg=63055463.00, stdev= 0.00
     lat (nsec): min=69979k, max=69979k, avg=69978914.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[63177],  5.00th=[63177], 10.00th=[63177], 20.00th=[63177],
     | 30.00th=[63177], 40.00th=[63177], 50.00th=[63177], 60.00th=[63177],
     | 70.00th=[63177], 80.00th=[63177], 90.00th=[63177], 95.00th=[63177],
     | 99.00th=[63177], 99.50th=[63177], 99.90th=[63177], 99.95th=[63177],
     | 99.99th=[63177]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.46%, ctx=8, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=533MiB/s (559MB/s), 533MiB/s-533MiB/s (559MB/s-559MB/s), io=40.0MiB (41.9MB), run=75-75msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout