== sanity test 398c: run fio to test AIO ================= 09:54:42 (1741701282)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0838209 s, 500 MB/s
osc.lustre-OST0000-osc-ffff8800b3d90800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=30310: Tue Mar 11 09:54:47 2025
  write: IOPS=743, BW=2976KiB/s (3047kB/s)(10.0MiB/3441msec)
    slat (usec): min=12, max=425, avg=46.68, stdev=24.41
    clat (usec): min=1996, max=44121, avg=21369.26, stdev=2681.39
     lat (usec): min=2049, max=44140, avg=21417.18, stdev=2682.28
    clat percentiles (usec):
     |  1.00th=[12256],  5.00th=[16057], 10.00th=[18744], 20.00th=[20055],
     | 30.00th=[20841], 40.00th=[21365], 50.00th=[21890], 60.00th=[22152],
     | 70.00th=[22676], 80.00th=[22938], 90.00th=[23725], 95.00th=[24249],
     | 99.00th=[25560], 99.50th=[26346], 99.90th=[39060], 99.95th=[41681],
     | 99.99th=[44303]
   bw (  KiB/s): min= 2888, max= 3345, per=25.22%, avg=2992.17, stdev=176.11, samples=6
   iops        : min=  722, max=  836, avg=748.00, stdev=43.93, samples=6
  lat (msec)   : 2=0.04%, 4=0.08%, 10=0.27%, 20=18.20%, 50=81.41%
  cpu          : usr=0.73%, sys=3.90%, ctx=1437, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30311: Tue Mar 11 09:54:47 2025
  write: IOPS=741, BW=2966KiB/s (3038kB/s)(10.0MiB/3452msec)
    slat (usec): min=12, max=451, avg=44.73, stdev=25.53
    clat (usec): min=2457, max=26434, avg=21395.52, stdev=2414.40
     lat (usec): min=2500, max=26464, avg=21441.46, stdev=2416.86
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[16450], 10.00th=[19006], 20.00th=[20055],
     | 30.00th=[20841], 40.00th=[21365], 50.00th=[21890], 60.00th=[22152],
     | 70.00th=[22676], 80.00th=[22938], 90.00th=[23725], 95.00th=[24249],
     | 99.00th=[25297], 99.50th=[25822], 99.90th=[26346], 99.95th=[26346],
     | 99.99th=[26346]
   bw (  KiB/s): min= 2872, max= 3193, per=24.97%, avg=2962.83, stdev=119.65, samples=6
   iops        : min=  718, max=  798, avg=740.67, stdev=29.82, samples=6
  lat (msec)   : 4=0.08%, 10=0.20%, 20=17.73%, 50=81.99%
  cpu          : usr=0.72%, sys=3.65%, ctx=1290, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30312: Tue Mar 11 09:54:47 2025
  write: IOPS=741, BW=2966KiB/s (3038kB/s)(10.0MiB/3452msec)
    slat (usec): min=12, max=250, avg=46.85, stdev=23.08
    clat (usec): min=4893, max=32069, avg=21411.62, stdev=2283.23
     lat (usec): min=4954, max=32126, avg=21459.61, stdev=2285.12
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[16450], 10.00th=[19006], 20.00th=[20055],
     | 30.00th=[20841], 40.00th=[21365], 50.00th=[21890], 60.00th=[22152],
     | 70.00th=[22676], 80.00th=[22938], 90.00th=[23725], 95.00th=[23987],
     | 99.00th=[25560], 99.50th=[25822], 99.90th=[30016], 99.95th=[32113],
     | 99.99th=[32113]
   bw (  KiB/s): min= 2880, max= 3217, per=25.05%, avg=2972.17, stdev=124.61, samples=6
   iops        : min=  720, max=  804, avg=743.00, stdev=31.05, samples=6
  lat (msec)   : 10=0.04%, 20=18.52%, 50=81.45%
  cpu          : usr=0.78%, sys=3.85%, ctx=1484, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30313: Tue Mar 11 09:54:47 2025
  write: IOPS=741, BW=2966KiB/s (3038kB/s)(10.0MiB/3452msec)
    slat (usec): min=12, max=374, avg=46.44, stdev=26.44
    clat (usec): min=2602, max=38865, avg=21429.45, stdev=2369.55
     lat (usec): min=2809, max=38933, avg=21477.09, stdev=2370.41
    clat percentiles (usec):
     |  1.00th=[13173],  5.00th=[16450], 10.00th=[19006], 20.00th=[20055],
     | 30.00th=[20841], 40.00th=[21365], 50.00th=[21890], 60.00th=[22152],
     | 70.00th=[22676], 80.00th=[22938], 90.00th=[23725], 95.00th=[23987],
     | 99.00th=[25560], 99.50th=[25560], 99.90th=[34341], 99.95th=[36963],
     | 99.99th=[39060]
   bw (  KiB/s): min= 2888, max= 3249, per=25.11%, avg=2978.83, stdev=136.13, samples=6
   iops        : min=  722, max=  812, avg=744.67, stdev=33.93, samples=6
  lat (msec)   : 4=0.04%, 20=19.14%, 50=80.82%
  cpu          : usr=0.70%, sys=4.09%, ctx=1389, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=11.6MiB/s (12.1MB/s), 2966KiB/s-2976KiB/s (3038kB/s-3047kB/s), io=40.0MiB (41.9MB), run=3441-3452msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=30326: Tue Mar 11 09:54:51 2025
   read: IOPS=442, BW=1771KiB/s (1813kB/s)(5048KiB/2851msec)
    slat (usec): min=652, max=2708, avg=1137.01, stdev=226.83
    clat (usec): min=2958, max=24978, avg=16776.20, stdev=2347.04
     lat (usec): min=4528, max=26054, avg=17915.14, stdev=2363.68
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[12911], 10.00th=[13960], 20.00th=[14877],
     | 30.00th=[15664], 40.00th=[16188], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[19530], 95.00th=[20579],
     | 99.00th=[21890], 99.50th=[22414], 99.90th=[24773], 99.95th=[25035],
     | 99.99th=[25035]
   bw (  KiB/s): min= 1608, max= 1808, per=24.78%, avg=1744.00, stdev=80.60, samples=5
   iops        : min=  402, max=  452, avg=436.00, stdev=20.15, samples=5
  write: IOPS=455, BW=1821KiB/s (1865kB/s)(5192KiB/2851msec)
    slat (usec): min=14, max=1139, avg=45.20, stdev=37.58
    clat (usec): min=2905, max=24109, avg=17290.81, stdev=1866.37
     lat (usec): min=2950, max=24165, avg=17337.11, stdev=1867.17
    clat percentiles (usec):
     |  1.00th=[13304],  5.00th=[14484], 10.00th=[15139], 20.00th=[15795],
     | 30.00th=[16319], 40.00th=[16909], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[18744], 90.00th=[19530], 95.00th=[20317],
     | 99.00th=[22152], 99.50th=[22414], 99.90th=[22938], 99.95th=[23987],
     | 99.99th=[23987]
   bw (  KiB/s): min= 1728, max= 1904, per=24.94%, avg=1827.20, stdev=71.24, samples=5
   iops        : min=  432, max=  476, avg=456.80, stdev=17.81, samples=5
  lat (msec)   : 4=0.12%, 10=0.23%, 20=92.62%, 50=7.03%
  cpu          : usr=1.09%, sys=7.05%, ctx=2260, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30327: Tue Mar 11 09:54:51 2025
   read: IOPS=438, BW=1752KiB/s (1794kB/s)(4976KiB/2840msec)
    slat (usec): min=655, max=2593, avg=1144.83, stdev=243.65
    clat (usec): min=8762, max=30697, avg=16768.74, stdev=2356.33
     lat (usec): min=10763, max=31838, avg=17915.30, stdev=2375.72
    clat percentiles (usec):
     |  1.00th=[11994],  5.00th=[13304], 10.00th=[13960], 20.00th=[14746],
     | 30.00th=[15401], 40.00th=[16057], 50.00th=[16581], 60.00th=[17171],
     | 70.00th=[17957], 80.00th=[18482], 90.00th=[19792], 95.00th=[20841],
     | 99.00th=[23200], 99.50th=[24511], 99.90th=[28443], 99.95th=[30802],
     | 99.99th=[30802]
   bw (  KiB/s): min= 1632, max= 1848, per=24.71%, avg=1739.20, stdev=90.26, samples=5
   iops        : min=  408, max=  462, avg=434.80, stdev=22.57, samples=5
  write: IOPS=463, BW=1854KiB/s (1898kB/s)(5264KiB/2840msec)
    slat (usec): min=13, max=960, avg=44.45, stdev=32.23
    clat (usec): min=11927, max=30643, avg=17335.99, stdev=1891.71
     lat (usec): min=11964, max=30690, avg=17381.54, stdev=1892.09
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[14353], 10.00th=[15008], 20.00th=[15795],
     | 30.00th=[16319], 40.00th=[16712], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[18744], 90.00th=[19530], 95.00th=[20579],
     | 99.00th=[21890], 99.50th=[23200], 99.90th=[26870], 99.95th=[30540],
     | 99.99th=[30540]
   bw (  KiB/s): min= 1800, max= 1976, per=25.55%, avg=1872.00, stdev=71.55, samples=5
   iops        : min=  450, max=  494, avg=468.00, stdev=17.89, samples=5
  lat (msec)   : 10=0.08%, 20=91.84%, 50=8.09%
  cpu          : usr=1.13%, sys=6.80%, ctx=2325, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30328: Tue Mar 11 09:54:51 2025
   read: IOPS=437, BW=1751KiB/s (1793kB/s)(4984KiB/2847msec)
    slat (usec): min=683, max=2689, avg=1135.52, stdev=236.87
    clat (usec): min=9532, max=26782, avg=16707.29, stdev=2231.21
     lat (usec): min=10718, max=28012, avg=17844.58, stdev=2253.67
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[13042], 10.00th=[13829], 20.00th=[14746],
     | 30.00th=[15401], 40.00th=[16188], 50.00th=[16712], 60.00th=[17433],
     | 70.00th=[17957], 80.00th=[18744], 90.00th=[19530], 95.00th=[20317],
     | 99.00th=[21890], 99.50th=[22414], 99.90th=[24511], 99.95th=[26870],
     | 99.99th=[26870]
   bw (  KiB/s): min= 1656, max= 1960, per=24.85%, avg=1748.80, stdev=123.75, samples=5
   iops        : min=  414, max=  490, avg=437.20, stdev=30.94, samples=5
  write: IOPS=461, BW=1846KiB/s (1890kB/s)(5256KiB/2847msec)
    slat (usec): min=14, max=257, avg=43.95, stdev=20.41
    clat (usec): min=12032, max=30895, avg=17405.08, stdev=1926.04
     lat (usec): min=12065, max=30935, avg=17450.10, stdev=1926.04
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[14615], 10.00th=[15139], 20.00th=[15926],
     | 30.00th=[16450], 40.00th=[16909], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[18744], 90.00th=[19530], 95.00th=[20579],
     | 99.00th=[22676], 99.50th=[23462], 99.90th=[29754], 99.95th=[30802],
     | 99.99th=[30802]
   bw (  KiB/s): min= 1720, max= 2000, per=25.18%, avg=1844.80, stdev=120.61, samples=5
   iops        : min=  430, max=  500, avg=461.20, stdev=30.15, samples=5
  lat (msec)   : 10=0.04%, 20=92.62%, 50=7.34%
  cpu          : usr=0.67%, sys=7.27%, ctx=2359, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30329: Tue Mar 11 09:54:51 2025
   read: IOPS=445, BW=1784KiB/s (1826kB/s)(5060KiB/2837msec)
    slat (usec): min=23, max=2687, avg=1141.65, stdev=249.89
    clat (usec): min=1725, max=28422, avg=16885.85, stdev=2663.50
     lat (usec): min=1799, max=30349, avg=18029.27, stdev=2692.79
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[13042], 10.00th=[13829], 20.00th=[14746],
     | 30.00th=[15533], 40.00th=[16188], 50.00th=[16909], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[19006], 90.00th=[20055], 95.00th=[20841],
     | 99.00th=[23462], 99.50th=[23987], 99.90th=[27132], 99.95th=[28443],
     | 99.99th=[28443]
   bw (  KiB/s): min= 1760, max= 1824, per=25.42%, avg=1788.80, stdev=26.89, samples=5
   iops        : min=  440, max=  456, avg=447.20, stdev= 6.72, samples=5
  write: IOPS=456, BW=1826KiB/s (1870kB/s)(5180KiB/2837msec)
    slat (usec): min=13, max=367, avg=45.07, stdev=24.25
    clat (usec): min=7188, max=30401, avg=17299.15, stdev=1979.77
     lat (usec): min=7243, max=30454, avg=17345.38, stdev=1978.41
    clat percentiles (usec):
     |  1.00th=[13042],  5.00th=[14353], 10.00th=[15008], 20.00th=[15664],
     | 30.00th=[16319], 40.00th=[16909], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[18744], 90.00th=[19530], 95.00th=[20579],
     | 99.00th=[22414], 99.50th=[23200], 99.90th=[30278], 99.95th=[30278],
     | 99.99th=[30278]
   bw (  KiB/s): min= 1744, max= 1928, per=24.89%, avg=1824.00, stdev=74.83, samples=5
   iops        : min=  436, max=  482, avg=456.00, stdev=18.71, samples=5
  lat (msec)   : 2=0.08%, 4=0.16%, 10=0.16%, 20=91.02%, 50=8.59%
  cpu          : usr=0.99%, sys=7.19%, ctx=2335, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=7039KiB/s (7208kB/s), 1751KiB/s-1784KiB/s (1793kB/s-1826kB/s), io=19.6MiB (20.5MB), run=2837-2851msec
  WRITE: bw=7328KiB/s (7504kB/s), 1821KiB/s-1854KiB/s (1865kB/s-1898kB/s), io=20.4MiB (21.4MB), run=2837-2851msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=30335: Tue Mar 11 09:54:52 2025
   read: IOPS=7, BW=310MiB/s (325MB/s)(40.0MiB/129msec)
    slat (nsec): min=8847.2k, max=8847.2k, avg=8847211.00, stdev= 0.00
    clat (nsec): min=115265k, max=115265k, avg=115264872.00, stdev= 0.00
     lat (nsec): min=124150k, max=124150k, avg=124150499.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  115],  5.00th=[  115], 10.00th=[  115], 20.00th=[  115],
     | 30.00th=[  115], 40.00th=[  115], 50.00th=[  115], 60.00th=[  115],
     | 70.00th=[  115], 80.00th=[  115], 90.00th=[  115], 95.00th=[  115],
     | 99.00th=[  115], 99.50th=[  115], 99.90th=[  115], 99.95th=[  115],
     | 99.99th=[  115]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=7.81%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=310MiB/s (325MB/s), 310MiB/s-310MiB/s (325MB/s-325MB/s), io=40.0MiB (41.9MB), run=129-129msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout