== sanity test 398c: run fio to test AIO ================= 18:29:36 (1748471376)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0831752 s, 504 MB/s
osc.lustre-OST0000-osc-ffff8800ac01e000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=26248: Wed May 28 18:29:40 2025
  write: IOPS=786, BW=3146KiB/s (3221kB/s)(10.0MiB/3255msec)
    slat (usec): min=12, max=427, avg=47.65, stdev=32.85
    clat (usec): min=2022, max=33053, avg=20177.74, stdev=3835.11
     lat (usec): min=2118, max=33089, avg=20226.54, stdev=3837.46
    clat percentiles (usec):
     |  1.00th=[ 9896],  5.00th=[14615], 10.00th=[16057], 20.00th=[17171],
     | 30.00th=[18220], 40.00th=[19006], 50.00th=[19792], 60.00th=[20841],
     | 70.00th=[21890], 80.00th=[23200], 90.00th=[24773], 95.00th=[26870],
     | 99.00th=[30016], 99.50th=[30802], 99.90th=[32900], 99.95th=[33162],
     | 99.99th=[33162]
   bw (  KiB/s): min= 2704, max= 3632, per=24.90%, avg=3125.33, stdev=352.79, samples=6
   iops        : min=  676, max=  908, avg=781.33, stdev=88.20, samples=6
  lat (msec)   : 4=0.12%, 10=1.37%, 20=49.88%, 50=48.63%
  cpu          : usr=0.77%, sys=4.24%, ctx=1355, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=26249: Wed May 28 18:29:40 2025
  write: IOPS=785, BW=3140KiB/s (3216kB/s)(10.0MiB/3261msec)
    slat (usec): min=12, max=401, avg=45.82, stdev=30.12
    clat (usec): min=3273, max=33887, avg=20231.34, stdev=3799.01
     lat (usec): min=3316, max=34019, avg=20278.30, stdev=3801.66
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[14353], 10.00th=[16057], 20.00th=[17171],
     | 30.00th=[18220], 40.00th=[19006], 50.00th=[19792], 60.00th=[20841],
     | 70.00th=[22152], 80.00th=[23200], 90.00th=[25035], 95.00th=[26870],
     | 99.00th=[30802], 99.50th=[31065], 99.90th=[32637], 99.95th=[33817],
     | 99.99th=[33817]
   bw (  KiB/s): min= 2712, max= 3624, per=24.85%, avg=3118.67, stdev=351.29, samples=6
   iops        : min=  678, max=  906, avg=779.67, stdev=87.82, samples=6
  lat (msec)   : 4=0.04%, 10=0.74%, 20=50.20%, 50=49.02%
  cpu          : usr=0.58%, sys=4.17%, ctx=1286, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=26250: Wed May 28 18:29:40 2025
  write: IOPS=784, BW=3138KiB/s (3214kB/s)(10.0MiB/3263msec)
    slat (usec): min=11, max=756, avg=46.86, stdev=38.52
    clat (usec): min=1859, max=33835, avg=20258.78, stdev=3710.92
     lat (usec): min=1908, max=33895, avg=20306.77, stdev=3714.56
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[15008], 10.00th=[16188], 20.00th=[17433],
     | 30.00th=[18482], 40.00th=[19006], 50.00th=[20055], 60.00th=[20841],
     | 70.00th=[21890], 80.00th=[23200], 90.00th=[24773], 95.00th=[26608],
     | 99.00th=[30278], 99.50th=[31065], 99.90th=[33817], 99.95th=[33817],
     | 99.99th=[33817]
   bw (  KiB/s): min= 2704, max= 3608, per=24.85%, avg=3118.67, stdev=343.74, samples=6
   iops        : min=  676, max=  902, avg=779.67, stdev=85.93, samples=6
  lat (msec)   : 2=0.08%, 4=0.08%, 10=0.31%, 20=50.16%, 50=49.38%
  cpu          : usr=0.89%, sys=3.92%, ctx=1341, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=26251: Wed May 28 18:29:40 2025
  write: IOPS=787, BW=3152KiB/s (3227kB/s)(10.0MiB/3249msec)
    slat (usec): min=11, max=978, avg=47.59, stdev=39.69
    clat (usec): min=2435, max=44497, avg=20177.17, stdev=4007.50
     lat (usec): min=2509, max=44515, avg=20225.93, stdev=4008.31
    clat percentiles (usec):
     |  1.00th=[ 9765],  5.00th=[14353], 10.00th=[16188], 20.00th=[17171],
     | 30.00th=[18220], 40.00th=[19006], 50.00th=[20055], 60.00th=[20841],
     | 70.00th=[21890], 80.00th=[22938], 90.00th=[25035], 95.00th=[27132],
     | 99.00th=[30278], 99.50th=[32113], 99.90th=[40109], 99.95th=[42730],
     | 99.99th=[44303]
   bw (  KiB/s): min= 2688, max= 3752, per=25.02%, avg=3140.00, stdev=396.39, samples=6
   iops        : min=  672, max=  938, avg=785.00, stdev=99.10, samples=6
  lat (msec)   : 4=0.12%, 10=1.05%, 20=49.45%, 50=49.38%
  cpu          : usr=0.58%, sys=4.34%, ctx=1412, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=12.3MiB/s (12.9MB/s), 3138KiB/s-3152KiB/s (3214kB/s-3227kB/s), io=40.0MiB (41.9MB), run=3249-3263msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=26263: Wed May 28 18:29:44 2025
   read: IOPS=484, BW=1939KiB/s (1986kB/s)(5048KiB/2603msec)
    slat (usec): min=25, max=1991, avg=1060.26, stdev=236.24
    clat (usec): min=1833, max=30736, avg=15529.94, stdev=2545.76
     lat (usec): min=1863, max=31752, avg=16591.69, stdev=2596.75
    clat percentiles (usec):
     |  1.00th=[ 8160],  5.00th=[11863], 10.00th=[12649], 20.00th=[13698],
     | 30.00th=[14353], 40.00th=[15008], 50.00th=[15533], 60.00th=[16188],
     | 70.00th=[16712], 80.00th=[17433], 90.00th=[18482], 95.00th=[19268],
     | 99.00th=[21103], 99.50th=[22152], 99.90th=[29230], 99.95th=[30802],
     | 99.99th=[30802]
   bw (  KiB/s): min= 1856, max= 2032, per=25.43%, avg=1945.60, stdev=64.84, samples=5
   iops        : min=  464, max=  508, avg=486.40, stdev=16.21, samples=5
  write: IOPS=498, BW=1995KiB/s (2042kB/s)(5192KiB/2603msec)
    slat (usec): min=12, max=1095, avg=44.83, stdev=52.42
    clat (usec): min=3572, max=29222, avg=15828.36, stdev=2096.46
     lat (usec): min=3683, max=29270, avg=15874.27, stdev=2095.78
    clat percentiles (usec):
     |  1.00th=[10028],  5.00th=[12780], 10.00th=[13566], 20.00th=[14353],
     | 30.00th=[14877], 40.00th=[15401], 50.00th=[15795], 60.00th=[16188],
     | 70.00th=[16712], 80.00th=[17433], 90.00th=[18220], 95.00th=[19006],
     | 99.00th=[20579], 99.50th=[21103], 99.90th=[26346], 99.95th=[29230],
     | 99.99th=[29230]
   bw (  KiB/s): min= 1856, max= 2136, per=25.03%, avg=1993.60, stdev=110.48, samples=5
   iops        : min=  464, max=  534, avg=498.40, stdev=27.62, samples=5
  lat (msec)   : 2=0.04%, 4=0.23%, 10=0.90%, 20=96.72%, 50=2.11%
  cpu          : usr=1.00%, sys=6.99%, ctx=2531, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=26264: Wed May 28 18:29:44 2025
   read: IOPS=474, BW=1898KiB/s (1943kB/s)(4976KiB/2622msec)
    slat (usec): min=642, max=2259, avg=1064.28, stdev=225.40
    clat (usec): min=5842, max=24334, avg=15573.68, stdev=2298.28
     lat (usec): min=6966, max=25439, avg=16639.54, stdev=2317.89
    clat percentiles (usec):
     |  1.00th=[10814],  5.00th=[11994], 10.00th=[12649], 20.00th=[13566],
     | 30.00th=[14222], 40.00th=[14877], 50.00th=[15533], 60.00th=[16057],
     | 70.00th=[16712], 80.00th=[17433], 90.00th=[18482], 95.00th=[19530],
     | 99.00th=[20841], 99.50th=[21890], 99.90th=[24249], 99.95th=[24249],
     | 99.99th=[24249]
   bw (  KiB/s): min= 1704, max= 1992, per=24.71%, avg=1890.40, stdev=128.36, samples=5
   iops        : min=  426, max=  498, avg=472.60, stdev=32.09, samples=5
  write: IOPS=501, BW=2008KiB/s (2056kB/s)(5264KiB/2622msec)
    slat (usec): min=13, max=165, avg=41.85, stdev=18.64
    clat (usec): min=8881, max=22934, avg=15895.35, stdev=1861.64
     lat (usec): min=8925, max=22977, avg=15938.20, stdev=1860.81
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[13304], 10.00th=[13698], 20.00th=[14353],
     | 30.00th=[14877], 40.00th=[15401], 50.00th=[15795], 60.00th=[16188],
     | 70.00th=[16712], 80.00th=[17433], 90.00th=[18220], 95.00th=[19006],
     | 99.00th=[20841], 99.50th=[21890], 99.90th=[22938], 99.95th=[22938],
     | 99.99th=[22938]
   bw (  KiB/s): min= 1944, max= 2176, per=25.28%, avg=2013.60, stdev=97.14, samples=5
   iops        : min=  486, max=  544, avg=503.40, stdev=24.29, samples=5
  lat (msec)   : 10=0.27%, 20=96.95%, 50=2.77%
  cpu          : usr=0.84%, sys=7.10%, ctx=2430, majf=0, minf=34
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=26265: Wed May 28 18:29:44 2025
   read: IOPS=475, BW=1900KiB/s (1946kB/s)(4984KiB/2623msec)
    slat (usec): min=600, max=2183, avg=1062.77, stdev=220.18
    clat (usec): min=9625, max=22401, avg=15563.69, stdev=2202.07
     lat (usec): min=10607, max=23359, avg=16627.95, stdev=2227.87
    clat percentiles (usec):
     |  1.00th=[10945],  5.00th=[11863], 10.00th=[12649], 20.00th=[13698],
     | 30.00th=[14353], 40.00th=[15008], 50.00th=[15533], 60.00th=[16057],
     | 70.00th=[16712], 80.00th=[17433], 90.00th=[18482], 95.00th=[19268],
     | 99.00th=[20841], 99.50th=[21365], 99.90th=[21890], 99.95th=[22414],
     | 99.99th=[22414]
   bw (  KiB/s): min= 1840, max= 1968, per=24.91%, avg=1905.60, stdev=55.54, samples=5
   iops        : min=  460, max=  492, avg=476.40, stdev=13.89, samples=5
  write: IOPS=500, BW=2004KiB/s (2052kB/s)(5256KiB/2623msec)
    slat (usec): min=12, max=615, avg=41.41, stdev=26.22
    clat (usec): min=9930, max=22887, avg=15883.50, stdev=1766.10
     lat (usec): min=9962, max=22904, avg=15925.88, stdev=1766.25
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[13042], 10.00th=[13566], 20.00th=[14353],
     | 30.00th=[14877], 40.00th=[15401], 50.00th=[15926], 60.00th=[16319],
     | 70.00th=[16909], 80.00th=[17433], 90.00th=[18220], 95.00th=[18744],
     | 99.00th=[20317], 99.50th=[20841], 99.90th=[21627], 99.95th=[22938],
     | 99.99th=[22938]
   bw (  KiB/s): min= 1800, max= 2136, per=24.99%, avg=1990.40, stdev=133.67, samples=5
   iops        : min=  450, max=  534, avg=497.60, stdev=33.42, samples=5
  lat (msec)   : 10=0.16%, 20=97.70%, 50=2.15%
  cpu          : usr=0.92%, sys=6.94%, ctx=2382, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=26266: Wed May 28 18:29:44 2025
   read: IOPS=482, BW=1931KiB/s (1977kB/s)(5060KiB/2621msec)
    slat (usec): min=672, max=2203, avg=1062.85, stdev=209.58
    clat (usec): min=4483, max=23470, avg=15491.33, stdev=2237.47
     lat (usec): min=5511, max=24660, avg=16555.66, stdev=2255.94
    clat percentiles (usec):
     |  1.00th=[10683],  5.00th=[11994], 10.00th=[12649], 20.00th=[13566],
     | 30.00th=[14353], 40.00th=[14877], 50.00th=[15533], 60.00th=[15926],
     | 70.00th=[16581], 80.00th=[17433], 90.00th=[18482], 95.00th=[19268],
     | 99.00th=[21103], 99.50th=[21890], 99.90th=[23200], 99.95th=[23462],
     | 99.99th=[23462]
   bw (  KiB/s): min= 1792, max= 2024, per=25.11%, avg=1920.80, stdev=94.42, samples=5
   iops        : min=  448, max=  506, avg=480.20, stdev=23.61, samples=5
  write: IOPS=494, BW=1976KiB/s (2024kB/s)(5180KiB/2621msec)
    slat (usec): min=12, max=1248, avg=42.81, stdev=48.98
    clat (usec): min=4933, max=29897, avg=15984.44, stdev=1994.41
     lat (usec): min=4956, max=29933, avg=16028.22, stdev=1995.52
    clat percentiles (usec):
     |  1.00th=[11731],  5.00th=[13042], 10.00th=[13698], 20.00th=[14353],
     | 30.00th=[14877], 40.00th=[15401], 50.00th=[15926], 60.00th=[16450],
     | 70.00th=[16909], 80.00th=[17433], 90.00th=[18482], 95.00th=[19268],
     | 99.00th=[20841], 99.50th=[21627], 99.90th=[27395], 99.95th=[30016],
     | 99.99th=[30016]
   bw (  KiB/s): min= 1880, max= 2072, per=24.88%, avg=1981.40, stdev=79.51, samples=5
   iops        : min=  470, max=  518, avg=495.20, stdev=19.83, samples=5
  lat (msec)   : 10=0.43%, 20=97.73%, 50=1.84%
  cpu          : usr=0.95%, sys=6.91%, ctx=2492, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7651KiB/s (7834kB/s), 1898KiB/s-1939KiB/s (1943kB/s-1986kB/s), io=19.6MiB (20.5MB), run=2603-2623msec
  WRITE: bw=7965KiB/s (8156kB/s), 1976KiB/s-2008KiB/s (2024kB/s-2056kB/s), io=20.4MiB (21.4MB), run=2603-2623msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=26272: Wed May 28 18:29:45 2025
   read: IOPS=10, BW=400MiB/s (419MB/s)(40.0MiB/100msec)
    slat (nsec): min=6824.7k, max=6824.7k, avg=6824665.00, stdev= 0.00
    clat (nsec): min=87788k, max=87788k, avg=87787636.00, stdev= 0.00
     lat (nsec): min=94631k, max=94631k, avg=94631381.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[87557],  5.00th=[87557], 10.00th=[87557], 20.00th=[87557],
     | 30.00th=[87557], 40.00th=[87557], 50.00th=[87557], 60.00th=[87557],
     | 70.00th=[87557], 80.00th=[87557], 90.00th=[87557], 95.00th=[87557],
     | 99.00th=[87557], 99.50th=[87557], 99.90th=[87557], 99.95th=[87557],
     | 99.99th=[87557]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.06%, ctx=7, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=400MiB/s (419MB/s), 400MiB/s-400MiB/s (419MB/s-419MB/s), io=40.0MiB (41.9MB), run=100-100msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout