== sanity test 398c: run fio to test AIO ================= 23:41:17 (1708836077)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.085378 s, 491 MB/s
osc.lustre-OST0000-osc-ffff880083fa0800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=7328: Sat Feb 24 23:41:21 2024
  write: IOPS=745, BW=2984KiB/s (3055kB/s)(10.0MiB/3432msec)
    slat (usec): min=12, max=272, avg=46.08, stdev=26.97
    clat (usec): min=3272, max=40734, avg=21292.76, stdev=2715.59
     lat (usec): min=3329, max=40774, avg=21340.20, stdev=2718.76
    clat percentiles (usec):
     |  1.00th=[14615],  5.00th=[17171], 10.00th=[17957], 20.00th=[19268],
     | 30.00th=[20055], 40.00th=[20841], 50.00th=[21627], 60.00th=[22152],
     | 70.00th=[22676], 80.00th=[23200], 90.00th=[23725], 95.00th=[24511],
     | 99.00th=[27132], 99.50th=[36963], 99.90th=[40633], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min= 2792, max= 3344, per=24.97%, avg=2980.00, stdev=205.82, samples=6
   iops        : min=  698, max=  836, avg=745.00, stdev=51.45, samples=6
  lat (msec)   : 4=0.04%, 20=27.77%, 50=72.19%
  cpu          : usr=0.82%, sys=3.88%, ctx=1459, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=7329: Sat Feb 24 23:41:21 2024
  write: IOPS=750, BW=3000KiB/s (3072kB/s)(10.0MiB/3413msec)
    slat (usec): min=11, max=390, avg=48.28, stdev=26.06
    clat (usec): min=2671, max=41334, avg=21180.49, stdev=3032.30
     lat (usec): min=2733, max=41403, avg=21230.15, stdev=3034.93
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[16712], 10.00th=[17957], 20.00th=[19006],
     | 30.00th=[20055], 40.00th=[20841], 50.00th=[21365], 60.00th=[22152],
     | 70.00th=[22676], 80.00th=[23200], 90.00th=[23987], 95.00th=[24249],
     | 99.00th=[32113], 99.50th=[36963], 99.90th=[40633], 99.95th=[40633],
     | 99.99th=[41157]
   bw (  KiB/s): min= 2792, max= 3392, per=25.17%, avg=3004.00, stdev=229.02, samples=6
   iops        : min=  698, max=  848, avg=751.00, stdev=57.25, samples=6
  lat (msec)   : 4=0.08%, 10=0.43%, 20=30.39%, 50=69.10%
  cpu          : usr=0.64%, sys=4.19%, ctx=1732, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=7330: Sat Feb 24 23:41:21 2024
  write: IOPS=747, BW=2991KiB/s (3062kB/s)(10.0MiB/3424msec)
    slat (usec): min=12, max=706, avg=46.29, stdev=27.86
    clat (usec): min=4924, max=40099, avg=21237.00, stdev=2801.01
     lat (usec): min=5007, max=40130, avg=21284.55, stdev=2800.75
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[16909], 10.00th=[17957], 20.00th=[19268],
     | 30.00th=[20055], 40.00th=[20841], 50.00th=[21365], 60.00th=[22152],
     | 70.00th=[22676], 80.00th=[23200], 90.00th=[23987], 95.00th=[24249],
     | 99.00th=[27919], 99.50th=[37487], 99.90th=[39584], 99.95th=[40109],
     | 99.99th=[40109]
   bw (  KiB/s): min= 2816, max= 3336, per=25.07%, avg=2992.00, stdev=210.39, samples=6
   iops        : min=  704, max=  834, avg=748.00, stdev=52.60, samples=6
  lat (msec)   : 10=0.23%, 20=28.55%, 50=71.21%
  cpu          : usr=0.61%, sys=4.15%, ctx=1492, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=7331: Sat Feb 24 23:41:21 2024
  write: IOPS=746, BW=2985KiB/s (3057kB/s)(10.0MiB/3430msec)
    slat (usec): min=11, max=217, avg=44.77, stdev=24.89
    clat (usec): min=3075, max=40558, avg=21279.02, stdev=2898.88
     lat (usec): min=3111, max=40661, avg=21325.11, stdev=2902.26
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[16909], 10.00th=[17957], 20.00th=[19268],
     | 30.00th=[20317], 40.00th=[20841], 50.00th=[21627], 60.00th=[22152],
     | 70.00th=[22414], 80.00th=[23200], 90.00th=[23987], 95.00th=[24511],
     | 99.00th=[27132], 99.50th=[39584], 99.90th=[40633], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min= 2776, max= 3336, per=25.00%, avg=2984.00, stdev=207.07, samples=6
   iops        : min=  694, max=  834, avg=746.00, stdev=51.77, samples=6
  lat (msec)   : 4=0.04%, 10=0.16%, 20=27.93%, 50=71.88%
  cpu          : usr=0.67%, sys=3.65%, ctx=1421, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=11.7MiB/s (12.2MB/s), 2984KiB/s-3000KiB/s (3055kB/s-3072kB/s), io=40.0MiB (41.9MB), run=3413-3432msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=7344: Sat Feb 24 23:41:25 2024
   read: IOPS=402, BW=1612KiB/s (1650kB/s)(5048KiB/3132msec)
    slat (usec): min=22, max=2329, avg=1259.74, stdev=282.70
    clat (usec): min=11112, max=33711, avg=18541.31, stdev=2724.14
     lat (usec): min=12196, max=34826, avg=19803.16, stdev=2776.37
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[14353], 10.00th=[15139], 20.00th=[16188],
     | 30.00th=[17171], 40.00th=[17695], 50.00th=[18482], 60.00th=[19006],
     | 70.00th=[19792], 80.00th=[20841], 90.00th=[22152], 95.00th=[23200],
     | 99.00th=[24773], 99.50th=[26084], 99.90th=[30802], 99.95th=[33817],
     | 99.99th=[33817]
   bw (  KiB/s): min= 1477, max= 1672, per=25.03%, avg=1602.17, stdev=81.19, samples=6
   iops        : min=  369, max=  418, avg=400.50, stdev=20.37, samples=6
  write: IOPS=414, BW=1658KiB/s (1698kB/s)(5192KiB/3132msec)
    slat (usec): min=13, max=198, avg=47.68, stdev=21.03
    clat (usec): min=4817, max=30813, avg=19262.28, stdev=2248.89
     lat (usec): min=4851, max=30869, avg=19311.40, stdev=2249.03
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[15926], 10.00th=[16712], 20.00th=[17695],
     | 30.00th=[18220], 40.00th=[18744], 50.00th=[19268], 60.00th=[19792],
     | 70.00th=[20317], 80.00th=[20841], 90.00th=[21890], 95.00th=[22676],
     | 99.00th=[24511], 99.50th=[25560], 99.90th=[28443], 99.95th=[30802],
     | 99.99th=[30802]
   bw (  KiB/s): min= 1496, max= 1808, per=24.80%, avg=1652.67, stdev=112.75, samples=6
   iops        : min=  374, max=  452, avg=413.17, stdev=28.19, samples=6
  lat (msec)   : 10=0.23%, 20=66.99%, 50=32.77%
  cpu          : usr=0.77%, sys=7.51%, ctx=2371, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=7345: Sat Feb 24 23:41:25 2024
   read: IOPS=396, BW=1588KiB/s (1626kB/s)(4976KiB/3134msec)
    slat (usec): min=13, max=2312, avg=1267.35, stdev=266.76
    clat (usec): min=3072, max=30929, avg=18742.79, stdev=2683.51
     lat (usec): min=4284, max=32316, avg=20012.21, stdev=2708.90
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[14615], 10.00th=[15533], 20.00th=[16450],
     | 30.00th=[17171], 40.00th=[17957], 50.00th=[18744], 60.00th=[19268],
     | 70.00th=[20055], 80.00th=[20841], 90.00th=[22152], 95.00th=[23200],
     | 99.00th=[25560], 99.50th=[27395], 99.90th=[29492], 99.95th=[30802],
     | 99.99th=[30802]
   bw (  KiB/s): min= 1500, max= 1632, per=24.61%, avg=1575.33, stdev=43.11, samples=6
   iops        : min=  375, max=  408, avg=393.83, stdev=10.78, samples=6
  write: IOPS=419, BW=1680KiB/s (1720kB/s)(5264KiB/3134msec)
    slat (usec): min=12, max=698, avg=47.66, stdev=28.23
    clat (usec): min=6891, max=34595, avg=19102.99, stdev=2296.40
     lat (usec): min=6916, max=34648, avg=19151.97, stdev=2296.64
    clat percentiles (usec):
     |  1.00th=[13173],  5.00th=[15664], 10.00th=[16450], 20.00th=[17433],
     | 30.00th=[17957], 40.00th=[18482], 50.00th=[19006], 60.00th=[19792],
     | 70.00th=[20055], 80.00th=[20841], 90.00th=[21890], 95.00th=[22414],
     | 99.00th=[23987], 99.50th=[25297], 99.90th=[32375], 99.95th=[34341],
     | 99.99th=[34341]
   bw (  KiB/s): min= 1556, max= 1832, per=25.18%, avg=1678.00, stdev=118.76, samples=6
   iops        : min=  389, max=  458, avg=419.50, stdev=29.69, samples=6
  lat (msec)   : 4=0.04%, 10=0.12%, 20=68.79%, 50=31.05%
  cpu          : usr=1.18%, sys=7.05%, ctx=2291, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=7346: Sat Feb 24 23:41:25 2024
   read: IOPS=397, BW=1590KiB/s (1628kB/s)(4984KiB/3135msec)
    slat (usec): min=21, max=2329, avg=1250.78, stdev=259.97
    clat (usec): min=11315, max=26633, avg=18645.35, stdev=2590.82
     lat (usec): min=12147, max=28485, avg=19898.31, stdev=2624.56
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[14484], 10.00th=[15401], 20.00th=[16450],
     | 30.00th=[17171], 40.00th=[17957], 50.00th=[18482], 60.00th=[19268],
     | 70.00th=[20055], 80.00th=[20841], 90.00th=[21890], 95.00th=[23200],
     | 99.00th=[25035], 99.50th=[25297], 99.90th=[26346], 99.95th=[26608],
     | 99.99th=[26608]
   bw (  KiB/s): min= 1429, max= 1760, per=24.86%, avg=1591.50, stdev=118.61, samples=6
   iops        : min=  357, max=  440, avg=397.83, stdev=29.72, samples=6
  write: IOPS=419, BW=1677KiB/s (1717kB/s)(5256KiB/3135msec)
    slat (usec): min=12, max=206, avg=47.64, stdev=21.80
    clat (usec): min=5527, max=30470, avg=19224.42, stdev=2237.68
     lat (usec): min=5547, max=30535, avg=19273.48, stdev=2236.77
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[15926], 10.00th=[16909], 20.00th=[17433],
     | 30.00th=[18220], 40.00th=[18744], 50.00th=[19006], 60.00th=[19792],
     | 70.00th=[20317], 80.00th=[20841], 90.00th=[21890], 95.00th=[22676],
     | 99.00th=[24511], 99.50th=[26608], 99.90th=[30540], 99.95th=[30540],
     | 99.99th=[30540]
   bw (  KiB/s): min= 1592, max= 1736, per=24.88%, avg=1658.00, stdev=60.49, samples=6
   iops        : min=  398, max=  434, avg=414.50, stdev=15.12, samples=6
  lat (msec)   : 10=0.08%, 20=67.50%, 50=32.42%
  cpu          : usr=0.70%, sys=7.56%, ctx=2255, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=7347: Sat Feb 24 23:41:25 2024
   read: IOPS=403, BW=1614KiB/s (1653kB/s)(5060KiB/3135msec)
    slat (usec): min=31, max=2799, avg=1257.46, stdev=274.97
    clat (usec): min=3028, max=30466, avg=18698.89, stdev=2618.98
     lat (usec): min=4538, max=31558, avg=19958.31, stdev=2644.89
    clat percentiles (usec):
     |  1.00th=[13042],  5.00th=[14615], 10.00th=[15401], 20.00th=[16581],
     | 30.00th=[17433], 40.00th=[17957], 50.00th=[18744], 60.00th=[19268],
     | 70.00th=[20055], 80.00th=[20841], 90.00th=[21890], 95.00th=[22938],
     | 99.00th=[25560], 99.50th=[26346], 99.90th=[27395], 99.95th=[30540],
     | 99.99th=[30540]
   bw (  KiB/s): min= 1508, max= 1680, per=25.03%, avg=1602.00, stdev=67.59, samples=6
   iops        : min=  377, max=  420, avg=400.50, stdev=16.90, samples=6
  write: IOPS=413, BW=1652KiB/s (1692kB/s)(5180KiB/3135msec)
    slat (usec): min=13, max=1447, avg=48.65, stdev=46.68
    clat (usec): min=9114, max=31620, avg=19148.59, stdev=2284.84
     lat (usec): min=9204, max=31671, avg=19198.49, stdev=2284.28
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[15926], 10.00th=[16581], 20.00th=[17433],
     | 30.00th=[17957], 40.00th=[18482], 50.00th=[19006], 60.00th=[19530],
     | 70.00th=[20055], 80.00th=[20841], 90.00th=[21890], 95.00th=[22676],
     | 99.00th=[25035], 99.50th=[25822], 99.90th=[31589], 99.95th=[31589],
     | 99.99th=[31589]
   bw (  KiB/s): min= 1580, max= 1728, per=24.78%, avg=1651.33, stdev=65.01, samples=6
   iops        : min=  395, max=  432, avg=412.83, stdev=16.25, samples=6
  lat (msec)   : 4=0.04%, 10=0.20%, 20=68.36%, 50=31.41%
  cpu          : usr=0.57%, sys=7.59%, ctx=2285, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6401KiB/s (6555kB/s), 1588KiB/s-1614KiB/s (1626kB/s-1653kB/s), io=19.6MiB (20.5MB), run=3132-3135msec
  WRITE: bw=6664KiB/s (6824kB/s), 1652KiB/s-1680KiB/s (1692kB/s-1720kB/s), io=20.4MiB (21.4MB), run=3132-3135msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=7353: Sat Feb 24 23:41:27 2024
   read: IOPS=10, BW=400MiB/s (419MB/s)(40.0MiB/100msec)
    slat (nsec): min=7746.0k, max=7746.0k, avg=7746047.00, stdev= 0.00
    clat (nsec): min=91032k, max=91032k, avg=91031535.00, stdev= 0.00
     lat (nsec): min=98795k, max=98795k, avg=98794945.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[90702],  5.00th=[90702], 10.00th=[90702], 20.00th=[90702],
     | 30.00th=[90702], 40.00th=[90702], 50.00th=[90702], 60.00th=[90702],
     | 70.00th=[90702], 80.00th=[90702], 90.00th=[90702], 95.00th=[90702],
     | 99.00th=[90702], 99.50th=[90702], 99.90th=[90702], 99.95th=[90702],
     | 99.99th=[90702]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.07%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=400MiB/s (419MB/s), 400MiB/s-400MiB/s (419MB/s-419MB/s), io=40.0MiB (41.9MB), run=100-100msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout