== sanity test 398c: run fio to test AIO ================= 23:40:14 (1745466014)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0845724 s, 496 MB/s
osc.lustre-OST0000-osc-31a044b7-25e0-41f2-a015-f213bc93c5e4.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=2693: Wed Apr 23 23:40:19 2025
  write: IOPS=770, BW=3082KiB/s (3156kB/s)(10.0MiB/3323msec)
    slat (usec): min=12, max=328, avg=47.57, stdev=28.19
    clat (usec): min=3900, max=37079, avg=20636.57, stdev=2550.84
     lat (usec): min=3915, max=37129, avg=20685.14, stdev=2551.72
    clat percentiles (usec):
     |  1.00th=[12780],  5.00th=[16057], 10.00th=[17433], 20.00th=[19268],
     | 30.00th=[20055], 40.00th=[20579], 50.00th=[20841], 60.00th=[21627],
     | 70.00th=[22152], 80.00th=[22676], 90.00th=[23200], 95.00th=[23462],
     | 99.00th=[24511], 99.50th=[25297], 99.90th=[34866], 99.95th=[34866],
     | 99.99th=[36963]
   bw (  KiB/s): min= 2904, max= 3504, per=25.24%, avg=3100.00, stdev=223.01, samples=6
   iops        : min=  726, max=  876, avg=775.00, stdev=55.75, samples=6
  lat (msec)   : 4=0.04%, 10=0.55%, 20=29.41%, 50=70.00%
  cpu          : usr=0.36%, sys=4.30%, ctx=1854, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=2694: Wed Apr 23 23:40:19 2025
  write: IOPS=767, BW=3070KiB/s (3144kB/s)(10.0MiB/3335msec)
    slat (usec): min=12, max=271, avg=48.63, stdev=25.58
    clat (usec): min=4535, max=37455, avg=20680.25, stdev=2450.93
     lat (usec): min=4583, max=37517, avg=20729.92, stdev=2452.32
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[16581], 10.00th=[17433], 20.00th=[19006],
     | 30.00th=[20055], 40.00th=[20317], 50.00th=[20841], 60.00th=[21627],
     | 70.00th=[22152], 80.00th=[22676], 90.00th=[23200], 95.00th=[23462],
     | 99.00th=[24773], 99.50th=[25822], 99.90th=[34866], 99.95th=[35390],
     | 99.99th=[37487]
   bw (  KiB/s): min= 2888, max= 3408, per=25.10%, avg=3082.67, stdev=201.35, samples=6
   iops        : min=  722, max=  852, avg=770.67, stdev=50.34, samples=6
  lat (msec)   : 10=0.66%, 20=29.22%, 50=70.12%
  cpu          : usr=0.63%, sys=3.87%, ctx=2073, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=2695: Wed Apr 23 23:40:19 2025
  write: IOPS=768, BW=3073KiB/s (3147kB/s)(10.0MiB/3332msec)
    slat (usec): min=12, max=243, avg=47.32, stdev=26.06
    clat (usec): min=2125, max=35399, avg=20646.61, stdev=2482.59
     lat (usec): min=2181, max=35458, avg=20695.00, stdev=2485.61
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[16581], 10.00th=[17433], 20.00th=[19268],
     | 30.00th=[19792], 40.00th=[20317], 50.00th=[20841], 60.00th=[21627],
     | 70.00th=[22152], 80.00th=[22676], 90.00th=[23200], 95.00th=[23462],
     | 99.00th=[24511], 99.50th=[25297], 99.90th=[32900], 99.95th=[35390],
     | 99.99th=[35390]
   bw (  KiB/s): min= 2928, max= 3368, per=25.16%, avg=3089.33, stdev=175.81, samples=6
   iops        : min=  732, max=  842, avg=772.33, stdev=43.95, samples=6
  lat (msec)   : 4=0.12%, 10=0.08%, 20=31.17%, 50=68.63%
  cpu          : usr=0.60%, sys=3.96%, ctx=1902, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=2696: Wed Apr 23 23:40:19 2025
  write: IOPS=767, BW=3070KiB/s (3144kB/s)(10.0MiB/3335msec)
    slat (usec): min=12, max=528, avg=48.26, stdev=30.75
    clat (usec): min=2461, max=39779, avg=20699.39, stdev=2443.22
     lat (usec): min=2534, max=39876, avg=20748.67, stdev=2441.85
    clat percentiles (usec):
     |  1.00th=[13829],  5.00th=[16450], 10.00th=[17433], 20.00th=[19006],
     | 30.00th=[20055], 40.00th=[20579], 50.00th=[20841], 60.00th=[21627],
     | 70.00th=[22152], 80.00th=[22676], 90.00th=[23200], 95.00th=[23462],
     | 99.00th=[24511], 99.50th=[25560], 99.90th=[37487], 99.95th=[39584],
     | 99.99th=[39584]
   bw (  KiB/s): min= 2880, max= 3440, per=25.12%, avg=3085.33, stdev=212.36, samples=6
   iops        : min=  720, max=  860, avg=771.33, stdev=53.09, samples=6
  lat (msec)   : 4=0.04%, 10=0.04%, 20=29.65%, 50=70.27%
  cpu          : usr=0.54%, sys=4.17%, ctx=1893, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=11.0MiB/s (12.6MB/s), 3070KiB/s-3082KiB/s (3144kB/s-3156kB/s), io=40.0MiB (41.9MB), run=3323-3335msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=2708: Wed Apr 23 23:40:22 2025
   read: IOPS=473, BW=1896KiB/s (1941kB/s)(5048KiB/2663msec)
    slat (usec): min=613, max=2102, avg=1097.81, stdev=205.74
    clat (usec): min=8845, max=22114, avg=15576.26, stdev=2088.67
     lat (usec): min=9622, max=23075, avg=16676.06, stdev=2094.23
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[12256], 10.00th=[12911], 20.00th=[13698],
     | 30.00th=[14353], 40.00th=[14877], 50.00th=[15533], 60.00th=[16188],
     | 70.00th=[16909], 80.00th=[17433], 90.00th=[18220], 95.00th=[19006],
     | 99.00th=[20317], 99.50th=[20579], 99.90th=[21627], 99.95th=[22152],
     | 99.99th=[22152]
   bw (  KiB/s): min= 1704, max= 2000, per=24.92%, avg=1876.80, stdev=113.08, samples=5
   iops        : min=  426, max=  500, avg=469.20, stdev=28.27, samples=5
  write: IOPS=487, BW=1950KiB/s (1996kB/s)(5192KiB/2663msec)
    slat (usec): min=13, max=821, avg=46.58, stdev=34.93
    clat (usec): min=11012, max=23206, avg=16274.44, stdev=1647.91
     lat (usec): min=11083, max=23243, avg=16321.99, stdev=1648.82
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13435], 10.00th=[14222], 20.00th=[15008],
     | 30.00th=[15401], 40.00th=[15926], 50.00th=[16319], 60.00th=[16712],
     | 70.00th=[17171], 80.00th=[17695], 90.00th=[18482], 95.00th=[19006],
     | 99.00th=[20317], 99.50th=[20579], 99.90th=[21627], 99.95th=[23200],
     | 99.99th=[23200]
   bw (  KiB/s): min= 1832, max= 2048, per=24.86%, avg=1948.80, stdev=77.27, samples=5
   iops        : min=  458, max=  512, avg=487.20, stdev=19.32, samples=5
  lat (msec)   : 10=0.08%, 20=98.75%, 50=1.17%
  cpu          : usr=0.83%, sys=8.04%, ctx=2322, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=2709: Wed Apr 23 23:40:22 2025
   read: IOPS=471, BW=1885KiB/s (1930kB/s)(4976KiB/2640msec)
    slat (usec): min=13, max=1905, avg=1099.86, stdev=215.46
    clat (usec): min=2213, max=26758, avg=15622.27, stdev=2243.01
     lat (usec): min=2777, max=28084, avg=16723.95, stdev=2263.15
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[12256], 10.00th=[12911], 20.00th=[13698],
     | 30.00th=[14484], 40.00th=[15008], 50.00th=[15664], 60.00th=[16188],
     | 70.00th=[16909], 80.00th=[17433], 90.00th=[18220], 95.00th=[19268],
     | 99.00th=[20841], 99.50th=[21365], 99.90th=[25560], 99.95th=[26870],
     | 99.99th=[26870]
   bw (  KiB/s): min= 1796, max= 1960, per=24.81%, avg=1868.00, stdev=84.66, samples=5
   iops        : min=  449, max=  490, avg=467.00, stdev=21.17, samples=5
  write: IOPS=498, BW=1994KiB/s (2042kB/s)(5264KiB/2640msec)
    slat (usec): min=13, max=452, avg=43.24, stdev=24.69
    clat (usec): min=4818, max=26713, avg=16168.57, stdev=1828.61
     lat (usec): min=4876, max=26750, avg=16212.69, stdev=1828.51
    clat percentiles (usec):
     |  1.00th=[10421],  5.00th=[13304], 10.00th=[14091], 20.00th=[14877],
     | 30.00th=[15401], 40.00th=[15926], 50.00th=[16319], 60.00th=[16712],
     | 70.00th=[17171], 80.00th=[17433], 90.00th=[18220], 95.00th=[18744],
     | 99.00th=[20055], 99.50th=[20317], 99.90th=[24511], 99.95th=[26608],
     | 99.99th=[26608]
   bw (  KiB/s): min= 1936, max= 2048, per=25.54%, avg=2002.20, stdev=47.96, samples=5
   iops        : min=  484, max=  512, avg=500.40, stdev=11.87, samples=5
  lat (msec)   : 4=0.12%, 10=0.43%, 20=98.01%, 50=1.45%
  cpu          : usr=0.49%, sys=8.07%, ctx=2306, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=2710: Wed Apr 23 23:40:22 2025
   read: IOPS=468, BW=1874KiB/s (1919kB/s)(4984KiB/2659msec)
    slat (usec): min=690, max=2595, avg=1100.00, stdev=205.26
    clat (usec): min=8062, max=29573, avg=15694.75, stdev=2285.29
     lat (usec): min=9249, max=30669, avg=16796.55, stdev=2288.55
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[12256], 10.00th=[12911], 20.00th=[13698],
     | 30.00th=[14484], 40.00th=[15139], 50.00th=[15664], 60.00th=[16188],
     | 70.00th=[16909], 80.00th=[17433], 90.00th=[18482], 95.00th=[19268],
     | 99.00th=[21627], 99.50th=[22676], 99.90th=[28443], 99.95th=[29492],
     | 99.99th=[29492]
   bw (  KiB/s): min= 1840, max= 1944, per=25.01%, avg=1883.20, stdev=41.03, samples=5
   iops        : min=  460, max=  486, avg=470.80, stdev=10.26, samples=5
  write: IOPS=494, BW=1977KiB/s (2024kB/s)(5256KiB/2659msec)
    slat (usec): min=12, max=1151, avg=47.10, stdev=41.95
    clat (usec): min=10411, max=28388, avg=16205.74, stdev=1714.95
     lat (usec): min=10502, max=28406, avg=16253.78, stdev=1712.66
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[13435], 10.00th=[14091], 20.00th=[14877],
     | 30.00th=[15401], 40.00th=[15926], 50.00th=[16319], 60.00th=[16712],
     | 70.00th=[17171], 80.00th=[17433], 90.00th=[18220], 95.00th=[18744],
     | 99.00th=[20579], 99.50th=[21627], 99.90th=[26608], 99.95th=[28443],
     | 99.99th=[28443]
   bw (  KiB/s): min= 1800, max= 2048, per=24.92%, avg=1953.60, stdev=103.13, samples=5
   iops        : min=  450, max=  512, avg=488.40, stdev=25.78, samples=5
  lat (msec)   : 10=0.20%, 20=97.70%, 50=2.11%
  cpu          : usr=0.94%, sys=7.86%, ctx=2302, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=2711: Wed Apr 23 23:40:22 2025
   read: IOPS=474, BW=1899KiB/s (1944kB/s)(5060KiB/2665msec)
    slat (usec): min=608, max=2483, avg=1101.40, stdev=212.76
    clat (usec): min=4435, max=22264, avg=15663.94, stdev=2216.00
     lat (usec): min=5596, max=23488, avg=16767.05, stdev=2213.65
    clat percentiles (usec):
     |  1.00th=[10814],  5.00th=[12125], 10.00th=[12911], 20.00th=[13698],
     | 30.00th=[14353], 40.00th=[15008], 50.00th=[15664], 60.00th=[16319],
     | 70.00th=[16909], 80.00th=[17433], 90.00th=[18482], 95.00th=[19268],
     | 99.00th=[21103], 99.50th=[21365], 99.90th=[21890], 99.95th=[22152],
     | 99.99th=[22152]
   bw (  KiB/s): min= 1768, max= 1984, per=25.07%, avg=1888.00, stdev=91.04, samples=5
   iops        : min=  442, max=  496, avg=472.00, stdev=22.76, samples=5
  write: IOPS=485, BW=1944KiB/s (1990kB/s)(5180KiB/2665msec)
    slat (usec): min=13, max=1271, avg=47.00, stdev=44.55
    clat (usec): min=9260, max=21134, avg=16263.25, stdev=1734.62
     lat (usec): min=9314, max=21167, avg=16311.23, stdev=1733.87
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[13435], 10.00th=[14091], 20.00th=[14877],
     | 30.00th=[15401], 40.00th=[15926], 50.00th=[16319], 60.00th=[16712],
     | 70.00th=[17171], 80.00th=[17695], 90.00th=[18482], 95.00th=[19006],
     | 99.00th=[20317], 99.50th=[20579], 99.90th=[21103], 99.95th=[21103],
     | 99.99th=[21103]
   bw (  KiB/s): min= 1784, max= 2080, per=24.62%, avg=1929.60, stdev=106.49, samples=5
   iops        : min=  446, max=  520, avg=482.40, stdev=26.62, samples=5
  lat (msec)   : 10=0.23%, 20=97.89%, 50=1.88%
  cpu          : usr=1.01%, sys=7.66%, ctx=2375, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7530KiB/s (7711kB/s), 1874KiB/s-1899KiB/s (1919kB/s-1944kB/s), io=19.6MiB (20.5MB), run=2640-2665msec
  WRITE: bw=7839KiB/s (8028kB/s), 1944KiB/s-1994KiB/s (1990kB/s-2042kB/s), io=20.4MiB (21.4MB), run=2640-2665msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=2717: Wed Apr 23 23:40:24 2025
   read: IOPS=12, BW=494MiB/s (518MB/s)(40.0MiB/81msec)
    slat (nsec): min=7072.5k, max=7072.5k, avg=7072463.00, stdev= 0.00
    clat (nsec): min=69541k, max=69541k, avg=69541440.00, stdev= 0.00
     lat (nsec): min=76637k, max=76637k, avg=76636668.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[69731],  5.00th=[69731], 10.00th=[69731], 20.00th=[69731],
     | 30.00th=[69731], 40.00th=[69731], 50.00th=[69731], 60.00th=[69731],
     | 70.00th=[69731], 80.00th=[69731], 90.00th=[69731], 95.00th=[69731],
     | 99.00th=[69731], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731],
     | 99.99th=[69731]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.50%, ctx=11, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=494MiB/s (518MB/s), 494MiB/s-494MiB/s (518MB/s-518MB/s), io=40.0MiB (41.9MB), run=81-81msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout