== sanity test 398c: run fio to test AIO ================= 20:54:41 (1777424081)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.115676 s, 363 MB/s
osc.lustre-OST0000-osc-ffff937b0a757000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=189119: Tue Apr 28 20:54:49 2026
  write: IOPS=528, BW=2115KiB/s (2166kB/s)(10.0MiB/4842msec); 0 zone resets
    slat (usec): min=36, max=654, avg=167.82, stdev=86.81
    clat (usec): min=6622, max=54379, avg=30033.86, stdev=3112.95
     lat (usec): min=6781, max=54566, avg=30202.62, stdev=3111.69
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[25822], 10.00th=[26870], 20.00th=[27657],
     | 30.00th=[28705], 40.00th=[29230], 50.00th=[30016], 60.00th=[30540],
     | 70.00th=[31065], 80.00th=[31851], 90.00th=[33424], 95.00th=[34341],
     | 99.00th=[41157], 99.50th=[42206], 99.90th=[54264], 99.95th=[54264],
     | 99.99th=[54264]
   bw (  KiB/s): min= 2016, max= 2208, per=25.10%, avg=2113.78, stdev=58.07, samples=9
   iops        : min=  504, max=  552, avg=528.44, stdev=14.52, samples=9
  lat (msec)   : 10=0.04%, 20=0.51%, 50=99.18%, 100=0.27%
  cpu          : usr=0.68%, sys=6.51%, ctx=2718, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=189120: Tue Apr 28 20:54:49 2026
  write: IOPS=528, BW=2113KiB/s (2164kB/s)(10.0MiB/4846msec); 0 zone resets
    slat (usec): min=38, max=662, avg=174.61, stdev=90.53
    clat (usec): min=19152, max=53005, avg=29959.61, stdev=2717.91
     lat (usec): min=19473, max=53110, avg=30135.20, stdev=2721.32
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[25822], 10.00th=[26870], 20.00th=[27657],
     | 30.00th=[28443], 40.00th=[29230], 50.00th=[30016], 60.00th=[30540],
     | 70.00th=[31065], 80.00th=[32113], 90.00th=[33424], 95.00th=[34341],
     | 99.00th=[36439], 99.50th=[36963], 99.90th=[52691], 99.95th=[52691],
     | 99.99th=[53216]
   bw (  KiB/s): min= 1976, max= 2208, per=25.09%, avg=2112.56, stdev=63.11, samples=9
   iops        : min=  494, max=  552, avg=528.11, stdev=15.78, samples=9
  lat (msec)   : 20=0.23%, 50=99.65%, 100=0.12%
  cpu          : usr=0.58%, sys=6.73%, ctx=2722, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=189121: Tue Apr 28 20:54:49 2026
  write: IOPS=526, BW=2108KiB/s (2158kB/s)(10.0MiB/4858msec); 0 zone resets
    slat (usec): min=50, max=20192, avg=179.15, stdev=406.08
    clat (usec): min=11845, max=50764, avg=29875.86, stdev=2710.09
     lat (usec): min=14827, max=51133, avg=30056.02, stdev=2691.05
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[25560], 10.00th=[26870], 20.00th=[27919],
     | 30.00th=[28443], 40.00th=[29230], 50.00th=[30016], 60.00th=[30540],
     | 70.00th=[31065], 80.00th=[31851], 90.00th=[33162], 95.00th=[34341],
     | 99.00th=[36439], 99.50th=[36963], 99.90th=[45876], 99.95th=[45876],
     | 99.99th=[50594]
   bw (  KiB/s): min= 2008, max= 2224, per=25.00%, avg=2105.33, stdev=66.03, samples=9
   iops        : min=  502, max=  556, avg=526.33, stdev=16.51, samples=9
  lat (msec)   : 20=0.27%, 50=99.69%, 100=0.04%
  cpu          : usr=0.56%, sys=7.14%, ctx=2755, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=189122: Tue Apr 28 20:54:49 2026
  write: IOPS=526, BW=2105KiB/s (2156kB/s)(10.0MiB/4864msec); 0 zone resets
    slat (usec): min=38, max=889, avg=162.48, stdev=89.98
    clat (usec): min=11719, max=38548, avg=29907.60, stdev=2611.88
     lat (usec): min=11860, max=38773, avg=30070.97, stdev=2614.61
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[25822], 10.00th=[27132], 20.00th=[27919],
     | 30.00th=[28443], 40.00th=[29230], 50.00th=[30016], 60.00th=[30540],
     | 70.00th=[31065], 80.00th=[31851], 90.00th=[33162], 95.00th=[34341],
     | 99.00th=[35914], 99.50th=[36963], 99.90th=[37487], 99.95th=[38536],
     | 99.99th=[38536]
   bw (  KiB/s): min= 1968, max= 2208, per=24.90%, avg=2096.89, stdev=81.73, samples=9
   iops        : min=  492, max=  552, avg=524.22, stdev=20.43, samples=9
  lat (msec)   : 20=0.39%, 50=99.61%
  cpu          : usr=0.41%, sys=6.83%, ctx=2746, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8421KiB/s (8623kB/s), 2105KiB/s-2115KiB/s (2156kB/s-2166kB/s), io=40.0MiB (41.9MB), run=4842-4864msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=189133: Tue Apr 28 20:54:58 2026
  read: IOPS=207, BW=830KiB/s (850kB/s)(5048KiB/6083msec)
    slat (usec): min=1015, max=8312, avg=2710.99, stdev=923.95
    clat (usec): min=13811, max=63207, avg=36182.46, stdev=7268.96
     lat (usec): min=14957, max=66388, avg=38895.11, stdev=7635.73
    clat percentiles (usec):
     |  1.00th=[22152],  5.00th=[25822], 10.00th=[27657], 20.00th=[30016],
     | 30.00th=[31589], 40.00th=[33424], 50.00th=[35390], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[41681], 90.00th=[45876], 95.00th=[49546],
     | 99.00th=[56361], 99.50th=[58459], 99.90th=[63177], 99.95th=[63177],
     | 99.99th=[63177]
   bw (  KiB/s): min=  592, max= 1056, per=24.45%, avg=806.45, stdev=121.01, samples=11
   iops        : min=  148, max=  264, avg=201.55, stdev=30.30, samples=11
  write: IOPS=213, BW=854KiB/s (874kB/s)(5192KiB/6083msec); 0 zone resets
    slat (usec): min=56, max=1854, avg=165.97, stdev=92.66
    clat (usec): min=13688, max=62133, avg=36381.52, stdev=6524.94
     lat (usec): min=13805, max=62253, avg=36548.50, stdev=6535.28
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[26870], 10.00th=[28705], 20.00th=[30802],
     | 30.00th=[32637], 40.00th=[34341], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[39060], 80.00th=[41157], 90.00th=[45351], 95.00th=[48497],
     | 99.00th=[55313], 99.50th=[57410], 99.90th=[57934], 99.95th=[62129],
     | 99.99th=[62129]
   bw (  KiB/s): min=  704, max= 1048, per=24.50%, avg=841.36, stdev=121.62, samples=11
   iops        : min=  176, max=  262, avg=210.27, stdev=30.48, samples=11
  lat (msec)   : 20=0.27%, 50=95.62%, 100=4.10%
  cpu          : usr=0.30%, sys=6.49%, ctx=4532, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=189134: Tue Apr 28 20:54:58 2026
  read: IOPS=204, BW=818KiB/s (838kB/s)(4976KiB/6080msec)
    slat (usec): min=1011, max=9705, avg=2739.96, stdev=948.35
    clat (usec): min=9671, max=59547, avg=36265.95, stdev=7290.03
     lat (usec): min=14767, max=62769, avg=39007.69, stdev=7632.30
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[25822], 10.00th=[27657], 20.00th=[29754],
     | 30.00th=[31851], 40.00th=[33424], 50.00th=[35390], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[42206], 90.00th=[46400], 95.00th=[50070],
     | 99.00th=[55313], 99.50th=[57410], 99.90th=[59507], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min=  648, max=  992, per=24.27%, avg=800.55, stdev=119.22, samples=11
   iops        : min=  162, max=  248, avg=200.09, stdev=29.85, samples=11
  write: IOPS=216, BW=866KiB/s (887kB/s)(5264KiB/6080msec); 0 zone resets
    slat (usec): min=57, max=809, avg=166.77, stdev=86.57
    clat (usec): min=22479, max=60659, avg=36487.67, stdev=6455.27
     lat (usec): min=22767, max=60759, avg=36655.82, stdev=6462.42
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[27132], 10.00th=[28705], 20.00th=[31065],
     | 30.00th=[32375], 40.00th=[34341], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[39060], 80.00th=[41681], 90.00th=[45351], 95.00th=[48497],
     | 99.00th=[54789], 99.50th=[57410], 99.90th=[60556], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  616, max= 1048, per=24.89%, avg=854.64, stdev=122.78, samples=11
   iops        : min=  154, max=  262, avg=213.64, stdev=30.71, samples=11
  lat (msec)   : 10=0.04%, 50=95.66%, 100=4.30%
  cpu          : usr=0.26%, sys=6.50%, ctx=4557, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=189135: Tue Apr 28 20:54:58 2026
  read: IOPS=205, BW=822KiB/s (842kB/s)(4984KiB/6063msec)
    slat (usec): min=1099, max=7179, avg=2774.93, stdev=924.84
    clat (usec): min=8796, max=60970, avg=36149.64, stdev=7046.02
     lat (usec): min=11223, max=63021, avg=38926.27, stdev=7395.97
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[26346], 10.00th=[27919], 20.00th=[30278],
     | 30.00th=[31851], 40.00th=[33424], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[39584], 80.00th=[42206], 90.00th=[45351], 95.00th=[49021],
     | 99.00th=[54789], 99.50th=[57410], 99.90th=[60556], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min=  592, max=  968, per=24.48%, avg=807.64, stdev=100.69, samples=11
   iops        : min=  148, max=  242, avg=201.91, stdev=25.17, samples=11
  write: IOPS=216, BW=867KiB/s (888kB/s)(5256KiB/6063msec); 0 zone resets
    slat (usec): min=57, max=666, avg=170.27, stdev=84.37
    clat (usec): min=19835, max=58725, avg=36227.09, stdev=6291.38
     lat (usec): min=19979, max=58828, avg=36398.48, stdev=6300.06
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[27132], 10.00th=[28705], 20.00th=[30540],
     | 30.00th=[32375], 40.00th=[33817], 50.00th=[35390], 60.00th=[37487],
     | 70.00th=[39060], 80.00th=[41157], 90.00th=[44827], 95.00th=[48497],
     | 99.00th=[52691], 99.50th=[54264], 99.90th=[58459], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min=  680, max= 1088, per=24.87%, avg=853.91, stdev=146.97, samples=11
   iops        : min=  170, max=  272, avg=213.45, stdev=36.76, samples=11
  lat (msec)   : 10=0.04%, 20=0.08%, 50=96.52%, 100=3.36%
  cpu          : usr=0.46%, sys=6.42%, ctx=4520, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=189136: Tue Apr 28 20:54:58 2026
  read: IOPS=209, BW=838KiB/s (858kB/s)(5060KiB/6038msec)
    slat (usec): min=109, max=6597, avg=2730.78, stdev=942.15
    clat (usec): min=6702, max=65689, avg=36107.81, stdev=7231.51
     lat (usec): min=6817, max=68783, avg=38840.60, stdev=7605.67
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[26346], 10.00th=[27919], 20.00th=[29754],
     | 30.00th=[31851], 40.00th=[33817], 50.00th=[35390], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[41681], 90.00th=[45876], 95.00th=[48497],
     | 99.00th=[54264], 99.50th=[57934], 99.90th=[64750], 99.95th=[65799],
     | 99.99th=[65799]
   bw (  KiB/s): min=  560, max= 1056, per=25.02%, avg=825.55, stdev=134.02, samples=11
   iops        : min=  140, max=  264, avg=206.36, stdev=33.51, samples=11
  write: IOPS=214, BW=858KiB/s (878kB/s)(5180KiB/6038msec); 0 zone resets
    slat (usec): min=49, max=2523, avg=175.28, stdev=120.62
    clat (usec): min=14550, max=63781, avg=36334.29, stdev=6207.75
     lat (usec): min=14741, max=63983, avg=36510.71, stdev=6213.98
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[27132], 10.00th=[28705], 20.00th=[31065],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[39060], 80.00th=[41157], 90.00th=[44827], 95.00th=[47973],
     | 99.00th=[52167], 99.50th=[54789], 99.90th=[59507], 99.95th=[63701],
     | 99.99th=[63701]
   bw (  KiB/s): min=  752, max= 1024, per=24.65%, avg=846.36, stdev=88.67, samples=11
   iops        : min=  188, max=  256, avg=211.55, stdev=22.20, samples=11
  lat (msec)   : 10=0.16%, 20=0.20%, 50=96.45%, 100=3.20%
  cpu          : usr=0.46%, sys=6.38%, ctx=4609, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3299KiB/s (3378kB/s), 818KiB/s-838KiB/s (838kB/s-858kB/s), io=19.6MiB (20.5MB), run=6038-6083msec
  WRITE: bw=3434KiB/s (3517kB/s), 854KiB/s-867KiB/s (874kB/s-888kB/s), io=20.4MiB (21.4MB), run=6038-6083msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=189141: Tue Apr 28 20:55:01 2026
  read: IOPS=10, BW=400MiB/s (419MB/s)(40.0MiB/100msec)
    slat (nsec): min=10603k, max=10603k, avg=10603320.00, stdev= 0.00
    clat (nsec): min=84651k, max=84651k, avg=84650725.00, stdev= 0.00
     lat (nsec): min=95257k, max=95257k, avg=95257343.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[84411],  5.00th=[84411], 10.00th=[84411], 20.00th=[84411],
     | 30.00th=[84411], 40.00th=[84411], 50.00th=[84411], 60.00th=[84411],
     | 70.00th=[84411], 80.00th=[84411], 90.00th=[84411], 95.00th=[84411],
     | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411],
     | 99.99th=[84411]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=10.10%, ctx=12, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=400MiB/s (419MB/s), 400MiB/s-400MiB/s (419MB/s-419MB/s), io=40.0MiB (41.9MB), run=100-100msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout