== sanity test 398c: run fio to test AIO ================= 10:35:38 (1761143738)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.117746 s, 356 MB/s
osc.lustre-OST0000-osc-ffff90d880d5f800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=192733: Wed Oct 22 10:35:45 2025
  write: IOPS=595, BW=2380KiB/s (2437kB/s)(10.0MiB/4302msec); 0 zone resets
    slat (usec): min=19, max=329, avg=83.98, stdev=39.75
    clat (usec): min=10366, max=57009, avg=26743.89, stdev=3671.82
     lat (usec): min=10536, max=57103, avg=26828.82, stdev=3674.75
    clat percentiles (usec):
     |  1.00th=[13960],  5.00th=[21103], 10.00th=[22938], 20.00th=[24773],
     | 30.00th=[25560], 40.00th=[26346], 50.00th=[27132], 60.00th=[27657],
     | 70.00th=[28181], 80.00th=[28967], 90.00th=[30016], 95.00th=[31589],
     | 99.00th=[35914], 99.50th=[37487], 99.90th=[51119], 99.95th=[56886],
     | 99.99th=[56886]
   bw (  KiB/s): min= 2208, max= 2802, per=25.29%, avg=2377.25, stdev=195.76, samples=8
   iops        : min=  552, max=  700, avg=594.25, stdev=48.78, samples=8
  lat (msec)   : 20=4.26%, 50=95.55%, 100=0.20%
  cpu          : usr=0.49%, sys=4.19%, ctx=2623, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=192734: Wed Oct 22 10:35:45 2025
  write: IOPS=587, BW=2350KiB/s (2407kB/s)(10.0MiB/4357msec); 0 zone resets
    slat (usec): min=28, max=447, avg=86.92, stdev=42.44
    clat (usec): min=11723, max=40352, avg=26840.58, stdev=3249.00
     lat (usec): min=11985, max=40497, avg=26928.48, stdev=3251.64
    clat percentiles (usec):
     |  1.00th=[17171],  5.00th=[21627], 10.00th=[22938], 20.00th=[24773],
     | 30.00th=[25560], 40.00th=[26608], 50.00th=[27132], 60.00th=[27657],
     | 70.00th=[28443], 80.00th=[28967], 90.00th=[30278], 95.00th=[31589],
     | 99.00th=[33817], 99.50th=[36439], 99.90th=[36963], 99.95th=[37487],
     | 99.99th=[40109]
   bw (  KiB/s): min= 2208, max= 2592, per=24.98%, avg=2348.00, stdev=128.71, samples=8
   iops        : min=  552, max=  648, avg=587.00, stdev=32.18, samples=8
  lat (msec)   : 20=3.63%, 50=96.37%
  cpu          : usr=0.46%, sys=4.16%, ctx=2797, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=192735: Wed Oct 22 10:35:45 2025
  write: IOPS=589, BW=2358KiB/s (2415kB/s)(10.0MiB/4342msec); 0 zone resets
    slat (usec): min=27, max=21530, avg=95.11, stdev=425.93
    clat (usec): min=10232, max=52014, avg=26883.84, stdev=3492.46
     lat (usec): min=12121, max=52083, avg=26980.06, stdev=3479.57
    clat percentiles (usec):
     |  1.00th=[15664],  5.00th=[21365], 10.00th=[23200], 20.00th=[24773],
     | 30.00th=[25560], 40.00th=[26608], 50.00th=[27132], 60.00th=[27657],
     | 70.00th=[28181], 80.00th=[28967], 90.00th=[30278], 95.00th=[31851],
     | 99.00th=[36439], 99.50th=[36963], 99.90th=[51643], 99.95th=[52167],
     | 99.99th=[52167]
   bw (  KiB/s): min= 2176, max= 2687, per=25.09%, avg=2358.88, stdev=167.77, samples=8
   iops        : min=  544, max=  671, avg=589.62, stdev=41.73, samples=8
  lat (msec)   : 20=3.75%, 50=96.13%, 100=0.12%
  cpu          : usr=0.55%, sys=4.49%, ctx=2784, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=192736: Wed Oct 22 10:35:45 2025
  write: IOPS=587, BW=2350KiB/s (2407kB/s)(10.0MiB/4357msec); 0 zone resets
    slat (usec): min=23, max=440, avg=85.05, stdev=42.70
    clat (usec): min=4479, max=46052, avg=26874.82, stdev=3480.10
     lat (usec): min=4634, max=46114, avg=26960.88, stdev=3480.27
    clat percentiles (usec):
     |  1.00th=[15926],  5.00th=[21103], 10.00th=[22938], 20.00th=[24773],
     | 30.00th=[25560], 40.00th=[26608], 50.00th=[27395], 60.00th=[27657],
     | 70.00th=[28181], 80.00th=[28967], 90.00th=[30278], 95.00th=[31589],
     | 99.00th=[35914], 99.50th=[37487], 99.90th=[45876], 99.95th=[45876],
     | 99.99th=[45876]
   bw (  KiB/s): min= 2208, max= 2666, per=24.99%, avg=2349.25, stdev=157.92, samples=8
   iops        : min=  552, max=  666, avg=587.25, stdev=39.34, samples=8
  lat (msec)   : 10=0.08%, 20=4.02%, 50=95.90%
  cpu          : usr=0.53%, sys=4.45%, ctx=2801, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=9401KiB/s (9627kB/s), 2350KiB/s-2380KiB/s (2407kB/s-2437kB/s), io=40.0MiB (41.9MB), run=4302-4357msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=192746: Wed Oct 22 10:35:54 2025
  read: IOPS=208, BW=833KiB/s (853kB/s)(5048KiB/6058msec)
    slat (usec): min=1100, max=12752, avg=2765.34, stdev=918.10
    clat (usec): min=7492, max=59628, avg=36242.69, stdev=6517.98
     lat (usec): min=10724, max=64369, avg=39009.52, stdev=6708.36
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[27132], 10.00th=[28967], 20.00th=[31327],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38536], 80.00th=[40633], 90.00th=[44303], 95.00th=[48497],
     | 99.00th=[55837], 99.50th=[57410], 99.90th=[59507], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min=  648, max=  896, per=24.81%, avg=821.73, stdev=77.79, samples=11
   iops        : min=  162, max=  224, avg=205.36, stdev=19.43, samples=11
  write: IOPS=214, BW=857KiB/s (878kB/s)(5192KiB/6058msec); 0 zone resets
    slat (usec): min=27, max=4099, avg=90.24, stdev=124.79
    clat (usec): min=7446, max=59057, avg=36073.44, stdev=5572.92
     lat (usec): min=7571, max=59097, avg=36164.64, stdev=5578.37
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[28967], 10.00th=[30016], 20.00th=[31589],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[40109], 90.00th=[43779], 95.00th=[46400],
     | 99.00th=[52691], 99.50th=[55313], 99.90th=[58983], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min=  744, max= 1024, per=24.83%, avg=856.18, stdev=84.59, samples=11
   iops        : min=  186, max=  256, avg=214.00, stdev=21.18, samples=11
  lat (msec)   : 10=0.12%, 20=0.23%, 50=96.80%, 100=2.85%
  cpu          : usr=0.51%, sys=4.26%, ctx=4579, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=192747: Wed Oct 22 10:35:54 2025
  read: IOPS=206, BW=827KiB/s (847kB/s)(4976KiB/6016msec)
    slat (usec): min=22, max=8589, avg=2791.55, stdev=927.23
    clat (usec): min=6103, max=59733, avg=35946.94, stdev=6086.78
     lat (usec): min=10144, max=62305, avg=38740.10, stdev=6314.90
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[27657], 10.00th=[29230], 20.00th=[31327],
     | 30.00th=[32637], 40.00th=[34341], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[38536], 80.00th=[40109], 90.00th=[43779], 95.00th=[46924],
     | 99.00th=[52167], 99.50th=[54264], 99.90th=[57410], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min=  672, max=  912, per=24.74%, avg=819.55, stdev=78.47, samples=11
   iops        : min=  168, max=  228, avg=204.82, stdev=19.66, samples=11
  write: IOPS=218, BW=875KiB/s (896kB/s)(5264KiB/6016msec); 0 zone resets
    slat (usec): min=23, max=2986, avg=91.04, stdev=90.53
    clat (usec): min=24746, max=57769, avg=36144.77, stdev=5299.60
     lat (usec): min=24892, max=57854, avg=36236.73, stdev=5299.69
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[29230], 10.00th=[30540], 20.00th=[31851],
     | 30.00th=[32900], 40.00th=[33817], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[40109], 90.00th=[43779], 95.00th=[46924],
     | 99.00th=[51119], 99.50th=[54264], 99.90th=[57934], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min=  712, max=  976, per=25.47%, avg=878.09, stdev=73.72, samples=11
   iops        : min=  178, max=  244, avg=219.45, stdev=18.35, samples=11
  lat (msec)   : 10=0.08%, 20=0.23%, 50=97.38%, 100=2.30%
  cpu          : usr=0.43%, sys=4.41%, ctx=4710, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=192748: Wed Oct 22 10:35:54 2025
  read: IOPS=206, BW=825KiB/s (845kB/s)(4984KiB/6042msec)
    slat (usec): min=25, max=8683, avg=2776.20, stdev=883.99
    clat (usec): min=14792, max=59647, avg=36320.19, stdev=6269.39
     lat (usec): min=17793, max=63248, avg=39097.90, stdev=6470.69
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[27132], 10.00th=[28967], 20.00th=[31327],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[38536], 80.00th=[40633], 90.00th=[44827], 95.00th=[48497],
     | 99.00th=[54789], 99.50th=[55837], 99.90th=[59507], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min=  664, max=  936, per=24.88%, avg=824.09, stdev=89.81, samples=11
   iops        : min=  166, max=  234, avg=206.00, stdev=22.43, samples=11
  write: IOPS=217, BW=870KiB/s (891kB/s)(5256KiB/6042msec); 0 zone resets
    slat (usec): min=24, max=457, avg=89.22, stdev=44.52
    clat (usec): min=10603, max=64097, avg=36268.86, stdev=5879.50
     lat (usec): min=10742, max=64229, avg=36359.17, stdev=5879.46
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[28967], 10.00th=[30278], 20.00th=[31851],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[43254], 95.00th=[47449],
     | 99.00th=[55313], 99.50th=[60031], 99.90th=[61604], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  736, max= 1000, per=25.02%, avg=862.73, stdev=81.32, samples=11
   iops        : min=  184, max=  250, avg=215.64, stdev=20.33, samples=11
  lat (msec)   : 20=0.35%, 50=96.60%, 100=3.05%
  cpu          : usr=0.40%, sys=4.40%, ctx=4633, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=192749: Wed Oct 22 10:35:54 2025
  read: IOPS=208, BW=836KiB/s (856kB/s)(5060KiB/6056msec)
    slat (usec): min=1144, max=14176, avg=2748.48, stdev=942.07
    clat (usec): min=19141, max=59767, avg=36035.49, stdev=5831.96
     lat (usec): min=21926, max=64278, avg=38785.45, stdev=6087.13
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[26870], 10.00th=[29230], 20.00th=[31589],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[38536], 80.00th=[40633], 90.00th=[44303], 95.00th=[46400],
     | 99.00th=[52167], 99.50th=[55313], 99.90th=[58459], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min=  696, max=  904, per=25.07%, avg=830.36, stdev=54.06, samples=11
   iops        : min=  174, max=  226, avg=207.55, stdev=13.53, samples=11
  write: IOPS=213, BW=855KiB/s (876kB/s)(5180KiB/6056msec); 0 zone resets
    slat (usec): min=26, max=604, avg=89.61, stdev=51.52
    clat (usec): min=23970, max=64197, avg=36234.21, stdev=5540.56
     lat (usec): min=24035, max=64256, avg=36324.79, stdev=5540.90
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[29230], 10.00th=[30278], 20.00th=[31851],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[40109], 90.00th=[44303], 95.00th=[47449],
     | 99.00th=[52691], 99.50th=[54789], 99.90th=[60556], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  688, max=  976, per=24.70%, avg=851.73, stdev=81.04, samples=11
   iops        : min=  172, max=  244, avg=212.91, stdev=20.27, samples=11
  lat (msec)   : 20=0.08%, 50=97.81%, 100=2.11%
  cpu          : usr=0.55%, sys=4.28%, ctx=4568, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3313KiB/s (3392kB/s), 825KiB/s-836KiB/s (845kB/s-856kB/s), io=19.6MiB (20.5MB), run=6016-6058msec
  WRITE: bw=3449KiB/s (3531kB/s), 855KiB/s-875KiB/s (876kB/s-896kB/s), io=20.4MiB (21.4MB), run=6016-6058msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=192753: Wed Oct 22 10:35:57 2025
  read: IOPS=9, BW=360MiB/s (378MB/s)(40.0MiB/111msec)
    slat (nsec): min=15599k, max=15599k, avg=15598940.00, stdev= 0.00
    clat (nsec): min=90118k, max=90118k, avg=90117865.00, stdev= 0.00
     lat (nsec): min=105720k, max=105720k, avg=105720255.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[89654],  5.00th=[89654], 10.00th=[89654], 20.00th=[89654],
     | 30.00th=[89654], 40.00th=[89654], 50.00th=[89654], 60.00th=[89654],
     | 70.00th=[89654], 80.00th=[89654], 90.00th=[89654], 95.00th=[89654],
     | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654],
     | 99.99th=[89654]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=11.82%, ctx=10, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=360MiB/s (378MB/s), 360MiB/s-360MiB/s (378MB/s-378MB/s), io=40.0MiB (41.9MB), run=111-111msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout