== sanity test 398c: run fio to test AIO ================= 02:59:39 (1746169179)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0890165 s, 471 MB/s
osc.lustre-OST0000-osc-ffff8800a98e7000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=979: Fri May  2 02:59:44 2025
  write: IOPS=845, BW=3383KiB/s (3464kB/s)(10.0MiB/3027msec)
    slat (usec): min=11, max=289, avg=36.14, stdev=19.66
    clat (usec): min=5174, max=25736, avg=18748.28, stdev=2483.13
     lat (usec): min=5214, max=25793, avg=18785.42, stdev=2483.18
    clat percentiles (usec):
     |  1.00th=[ 9765],  5.00th=[13173], 10.00th=[15533], 20.00th=[17433],
     | 30.00th=[18220], 40.00th=[18744], 50.00th=[19268], 60.00th=[19530],
     | 70.00th=[20055], 80.00th=[20579], 90.00th=[21103], 95.00th=[21627],
     | 99.00th=[22676], 99.50th=[22676], 99.90th=[22938], 99.95th=[23200],
     | 99.99th=[25822]
   bw (  KiB/s): min= 3176, max= 3968, per=24.92%, avg=3372.00, stdev=311.02, samples=6
   iops        : min=  794, max=  992, avg=843.00, stdev=77.76, samples=6
  lat (msec)   : 10=1.29%, 20=66.29%, 50=32.42%
  cpu          : usr=0.66%, sys=3.54%, ctx=636, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=980: Fri May  2 02:59:44 2025
  write: IOPS=847, BW=3391KiB/s (3472kB/s)(10.0MiB/3020msec)
    slat (usec): min=12, max=293, avg=42.03, stdev=21.02
    clat (usec): min=3203, max=31328, avg=18741.10, stdev=2611.08
     lat (usec): min=3267, max=31372, avg=18784.25, stdev=2612.56
    clat percentiles (usec):
     |  1.00th=[ 9765],  5.00th=[13304], 10.00th=[15139], 20.00th=[17433],
     | 30.00th=[18220], 40.00th=[18744], 50.00th=[19268], 60.00th=[19530],
     | 70.00th=[20055], 80.00th=[20579], 90.00th=[21365], 95.00th=[21627],
     | 99.00th=[22938], 99.50th=[23200], 99.90th=[31327], 99.95th=[31327],
     | 99.99th=[31327]
   bw (  KiB/s): min= 3168, max= 3984, per=25.00%, avg=3382.67, stdev=325.57, samples=6
   iops        : min=  792, max=  996, avg=845.67, stdev=81.39, samples=6
  lat (msec)   : 4=0.12%, 10=1.48%, 20=66.29%, 50=32.11%
  cpu          : usr=0.70%, sys=4.27%, ctx=1163, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=981: Fri May  2 02:59:44 2025
  write: IOPS=846, BW=3385KiB/s (3466kB/s)(10.0MiB/3025msec)
    slat (usec): min=12, max=294, avg=42.84, stdev=22.02
    clat (usec): min=4280, max=35607, avg=18748.63, stdev=2594.89
     lat (usec): min=4341, max=35625, avg=18792.61, stdev=2595.15
    clat percentiles (usec):
     |  1.00th=[ 9896],  5.00th=[13304], 10.00th=[15008], 20.00th=[17433],
     | 30.00th=[18220], 40.00th=[18744], 50.00th=[19268], 60.00th=[19530],
     | 70.00th=[20055], 80.00th=[20579], 90.00th=[21365], 95.00th=[21890],
     | 99.00th=[22676], 99.50th=[22938], 99.90th=[27395], 99.95th=[35390],
     | 99.99th=[35390]
   bw (  KiB/s): min= 3160, max= 3968, per=24.98%, avg=3380.00, stdev=314.62, samples=6
   iops        : min=  790, max=  992, avg=845.00, stdev=78.66, samples=6
  lat (msec)   : 10=1.41%, 20=66.95%, 50=31.64%
  cpu          : usr=0.79%, sys=4.00%, ctx=1186, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=982: Fri May  2 02:59:44 2025
  write: IOPS=849, BW=3400KiB/s (3481kB/s)(10.0MiB/3012msec)
    slat (usec): min=12, max=442, avg=46.43, stdev=25.35
    clat (usec): min=2224, max=35138, avg=18696.59, stdev=2759.93
     lat (usec): min=2293, max=35186, avg=18744.27, stdev=2758.03
    clat percentiles (usec):
     |  1.00th=[ 9896],  5.00th=[12649], 10.00th=[15008], 20.00th=[17171],
     | 30.00th=[18220], 40.00th=[18744], 50.00th=[19268], 60.00th=[19530],
     | 70.00th=[20055], 80.00th=[20579], 90.00th=[21365], 95.00th=[21627],
     | 99.00th=[22938], 99.50th=[23200], 99.90th=[34866], 99.95th=[34866],
     | 99.99th=[35390]
   bw (  KiB/s): min= 3120, max= 4064, per=25.08%, avg=3393.33, stdev=361.56, samples=6
   iops        : min=  780, max= 1016, avg=848.33, stdev=90.39, samples=6
  lat (msec)   : 4=0.08%, 10=1.41%, 20=66.33%, 50=32.19%
  cpu          : usr=0.60%, sys=4.72%, ctx=1530, majf=0, minf=27
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=13.2MiB/s (13.9MB/s), 3383KiB/s-3400KiB/s (3464kB/s-3481kB/s), io=40.0MiB (41.9MB), run=3012-3027msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=994: Fri May  2 02:59:47 2025
   read: IOPS=436, BW=1745KiB/s (1787kB/s)(5048KiB/2893msec)
    slat (usec): min=15, max=3688, avg=1202.66, stdev=270.26
    clat (usec): min=2467, max=29636, avg=17009.25, stdev=2531.08
     lat (usec): min=4530, max=30846, avg=18213.63, stdev=2569.34
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13435], 10.00th=[14222], 20.00th=[15008],
     | 30.00th=[15795], 40.00th=[16319], 50.00th=[16909], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[19006], 90.00th=[20055], 95.00th=[21103],
     | 99.00th=[22938], 99.50th=[24511], 99.90th=[28443], 99.95th=[29754],
     | 99.99th=[29754]
   bw (  KiB/s): min= 1632, max= 1816, per=24.92%, avg=1723.20, stdev=71.24, samples=5
   iops        : min=  408, max=  454, avg=430.80, stdev=17.81, samples=5
  write: IOPS=448, BW=1795KiB/s (1838kB/s)(5192KiB/2893msec)
    slat (usec): min=12, max=766, avg=45.55, stdev=28.08
    clat (usec): min=6644, max=28446, avg=17696.49, stdev=1958.08
     lat (usec): min=6707, max=28485, avg=17743.23, stdev=1957.50
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[14877], 10.00th=[15533], 20.00th=[16057],
     | 30.00th=[16712], 40.00th=[17171], 50.00th=[17695], 60.00th=[18220],
     | 70.00th=[18744], 80.00th=[19268], 90.00th=[20055], 95.00th=[20841],
     | 99.00th=[22938], 99.50th=[23987], 99.90th=[26084], 99.95th=[28443],
     | 99.99th=[28443]
   bw (  KiB/s): min= 1768, max= 1832, per=25.03%, avg=1801.60, stdev=22.91, samples=5
   iops        : min=  442, max=  458, avg=450.40, stdev= 5.73, samples=5
  lat (msec)   : 4=0.12%, 10=0.35%, 20=89.77%, 50=9.77%
  cpu          : usr=0.90%, sys=7.33%, ctx=2216, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=995: Fri May  2 02:59:47 2025
   read: IOPS=428, BW=1715KiB/s (1756kB/s)(4976KiB/2902msec)
    slat (usec): min=20, max=2318, avg=1194.73, stdev=249.44
    clat (usec): min=1501, max=24342, avg=17092.73, stdev=2369.81
     lat (usec): min=2908, max=25441, avg=18289.18, stdev=2391.22
    clat percentiles (usec):
     |  1.00th=[11863],  5.00th=[13566], 10.00th=[14222], 20.00th=[15139],
     | 30.00th=[15926], 40.00th=[16581], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[19006], 90.00th=[20055], 95.00th=[21103],
     | 99.00th=[22152], 99.50th=[22414], 99.90th=[23200], 99.95th=[24249],
     | 99.99th=[24249]
   bw (  KiB/s): min= 1544, max= 1808, per=24.39%, avg=1686.40, stdev=103.13, samples=5
   iops        : min=  386, max=  452, avg=421.60, stdev=25.78, samples=5
  write: IOPS=453, BW=1814KiB/s (1857kB/s)(5264KiB/2902msec)
    slat (usec): min=12, max=176, avg=44.42, stdev=18.69
    clat (usec): min=4723, max=24643, avg=17724.52, stdev=1950.58
     lat (usec): min=4761, max=24670, avg=17770.17, stdev=1950.49
    clat percentiles (usec):
     |  1.00th=[13698],  5.00th=[15008], 10.00th=[15401], 20.00th=[16188],
     | 30.00th=[16712], 40.00th=[17171], 50.00th=[17695], 60.00th=[18220],
     | 70.00th=[18744], 80.00th=[19268], 90.00th=[20055], 95.00th=[21103],
     | 99.00th=[22414], 99.50th=[23200], 99.90th=[23725], 99.95th=[24773],
     | 99.99th=[24773]
   bw (  KiB/s): min= 1760, max= 1856, per=25.20%, avg=1814.40, stdev=41.34, samples=5
   iops        : min=  440, max=  464, avg=453.60, stdev=10.33, samples=5
  lat (msec)   : 2=0.04%, 10=0.43%, 20=88.28%, 50=11.25%
  cpu          : usr=0.86%, sys=7.31%, ctx=2235, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=996: Fri May  2 02:59:47 2025
   read: IOPS=431, BW=1728KiB/s (1769kB/s)(4984KiB/2885msec)
    slat (usec): min=12, max=2214, avg=1195.19, stdev=273.39
    clat (usec): min=2820, max=27742, avg=17082.96, stdev=2614.99
     lat (usec): min=2919, max=29415, avg=18279.82, stdev=2651.52
    clat percentiles (usec):
     |  1.00th=[ 9503],  5.00th=[13435], 10.00th=[14222], 20.00th=[15139],
     | 30.00th=[15795], 40.00th=[16450], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18482], 80.00th=[19268], 90.00th=[20055], 95.00th=[21103],
     | 99.00th=[22938], 99.50th=[24249], 99.90th=[26084], 99.95th=[27657],
     | 99.99th=[27657]
   bw (  KiB/s): min= 1568, max= 1824, per=24.76%, avg=1712.00, stdev=104.92, samples=5
   iops        : min=  392, max=  456, avg=428.00, stdev=26.23, samples=5
  write: IOPS=455, BW=1822KiB/s (1866kB/s)(5256KiB/2885msec)
    slat (usec): min=11, max=242, avg=45.18, stdev=19.82
    clat (usec): min=6049, max=33043, avg=17667.20, stdev=2192.19
     lat (usec): min=6088, max=33062, avg=17713.58, stdev=2191.85
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[14615], 10.00th=[15270], 20.00th=[16057],
     | 30.00th=[16581], 40.00th=[17171], 50.00th=[17695], 60.00th=[18220],
     | 70.00th=[18482], 80.00th=[19006], 90.00th=[20055], 95.00th=[20841],
     | 99.00th=[23725], 99.50th=[25822], 99.90th=[32900], 99.95th=[33162],
     | 99.99th=[33162]
   bw (  KiB/s): min= 1664, max= 2040, per=25.40%, avg=1828.80, stdev=140.35, samples=5
   iops        : min=  416, max=  510, avg=457.20, stdev=35.09, samples=5
  lat (msec)   : 4=0.12%, 10=0.55%, 20=88.48%, 50=10.86%
  cpu          : usr=0.59%, sys=7.52%, ctx=2243, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=997: Fri May  2 02:59:47 2025
   read: IOPS=435, BW=1744KiB/s (1785kB/s)(5060KiB/2902msec)
    slat (usec): min=17, max=2307, avg=1188.53, stdev=248.04
    clat (usec): min=6950, max=24511, avg=17071.21, stdev=2353.22
     lat (usec): min=8287, max=25818, avg=18261.40, stdev=2371.11
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13566], 10.00th=[14091], 20.00th=[14877],
     | 30.00th=[15664], 40.00th=[16319], 50.00th=[16909], 60.00th=[17695],
     | 70.00th=[18220], 80.00th=[19006], 90.00th=[20055], 95.00th=[21103],
     | 99.00th=[22676], 99.50th=[23200], 99.90th=[24511], 99.95th=[24511],
     | 99.99th=[24511]
   bw (  KiB/s): min= 1648, max= 1800, per=24.97%, avg=1726.40, stdev=57.24, samples=5
   iops        : min=  412, max=  450, avg=431.60, stdev=14.31, samples=5
  write: IOPS=446, BW=1785KiB/s (1828kB/s)(5180KiB/2902msec)
    slat (usec): min=13, max=393, avg=45.61, stdev=21.59
    clat (usec): min=5874, max=25450, avg=17699.86, stdev=1918.32
     lat (usec): min=5935, max=25476, avg=17746.63, stdev=1919.46
    clat percentiles (usec):
     |  1.00th=[13173],  5.00th=[14746], 10.00th=[15401], 20.00th=[16188],
     | 30.00th=[16712], 40.00th=[17171], 50.00th=[17695], 60.00th=[17957],
     | 70.00th=[18482], 80.00th=[19268], 90.00th=[20055], 95.00th=[20841],
     | 99.00th=[22414], 99.50th=[22938], 99.90th=[23987], 99.95th=[25560],
     | 99.99th=[25560]
   bw (  KiB/s): min= 1672, max= 1864, per=24.76%, avg=1782.40, stdev=69.37, samples=5
   iops        : min=  418, max=  466, avg=445.60, stdev=17.34, samples=5
  lat (msec)   : 10=0.12%, 20=88.95%, 50=10.94%
  cpu          : usr=1.00%, sys=7.14%, ctx=2263, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6915KiB/s (7081kB/s), 1715KiB/s-1745KiB/s (1756kB/s-1787kB/s), io=19.6MiB (20.5MB), run=2885-2902msec
  WRITE: bw=7199KiB/s (7372kB/s), 1785KiB/s-1822KiB/s (1828kB/s-1866kB/s), io=20.4MiB (21.4MB), run=2885-2902msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=1003: Fri May  2 02:59:49 2025
   read: IOPS=11, BW=444MiB/s (466MB/s)(40.0MiB/90msec)
    slat (nsec): min=1971.1k, max=1971.1k, avg=1971103.00, stdev= 0.00
    clat (nsec): min=83996k, max=83996k, avg=83995795.00, stdev= 0.00
     lat (nsec): min=85977k, max=85977k, avg=85976828.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[84411],  5.00th=[84411], 10.00th=[84411], 20.00th=[84411],
     | 30.00th=[84411], 40.00th=[84411], 50.00th=[84411], 60.00th=[84411],
     | 70.00th=[84411], 80.00th=[84411], 90.00th=[84411], 95.00th=[84411],
     | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411],
     | 99.99th=[84411]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=2.25%, ctx=9, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=444MiB/s (466MB/s), 444MiB/s-444MiB/s (466MB/s-466MB/s), io=40.0MiB (41.9MB), run=90-90msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout