== sanity test 398c: run fio to test AIO ================= 21:35:08 (1787103308)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.13938 s, 301 MB/s
osc.lustre-OST0000-osc-ffff9f05454f4800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=181305: Tue Aug 18 21:35:17 2026
  write: IOPS=478, BW=1915KiB/s (1961kB/s)(10.0MiB/5346msec); 0 zone resets
    slat (usec): min=38, max=3966, avg=193.89, stdev=152.01
    clat (usec): min=5770, max=53782, avg=32882.26, stdev=5405.28
     lat (usec): min=5951, max=53857, avg=33078.51, stdev=5417.20
    clat percentiles (usec):
     |  1.00th=[18744],  5.00th=[25297], 10.00th=[27395], 20.00th=[29230],
     | 30.00th=[30278], 40.00th=[31327], 50.00th=[32375], 60.00th=[33162],
     | 70.00th=[34866], 80.00th=[36963], 90.00th=[39584], 95.00th=[42206],
     | 99.00th=[50594], 99.50th=[51119], 99.90th=[53216], 99.95th=[53740],
     | 99.99th=[53740]
   bw (  KiB/s): min= 1636, max= 2105, per=25.02%, avg=1916.90, stdev=158.77, samples=10
   iops        : min=  409, max=  526, avg=479.20, stdev=39.66, samples=10
  lat (msec)   : 10=0.04%, 20=1.33%, 50=97.50%, 100=1.13%
  cpu          : usr=0.77%, sys=6.79%, ctx=2813, majf=0, minf=7
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=181306: Tue Aug 18 21:35:17 2026
  write: IOPS=482, BW=1930KiB/s (1977kB/s)(10.0MiB/5305msec); 0 zone resets
    slat (usec): min=36, max=4172, avg=182.56, stdev=129.13
    clat (usec): min=13049, max=72639, avg=32910.65, stdev=5653.78
     lat (usec): min=13292, max=72986, avg=33094.31, stdev=5664.40
    clat percentiles (usec):
     |  1.00th=[17433],  5.00th=[24773], 10.00th=[27395], 20.00th=[29230],
     | 30.00th=[30278], 40.00th=[31327], 50.00th=[32375], 60.00th=[33424],
     | 70.00th=[34866], 80.00th=[36963], 90.00th=[39584], 95.00th=[42730],
     | 99.00th=[48497], 99.50th=[51119], 99.90th=[57934], 99.95th=[72877],
     | 99.99th=[72877]
   bw (  KiB/s): min= 1644, max= 2272, per=25.22%, avg=1932.00, stdev=184.58, samples=10
   iops        : min=  411, max=  568, avg=483.00, stdev=46.14, samples=10
  lat (msec)   : 20=2.07%, 50=97.19%, 100=0.74%
  cpu          : usr=0.64%, sys=6.67%, ctx=2814, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=181307: Tue Aug 18 21:35:17 2026
  write: IOPS=482, BW=1929KiB/s (1975kB/s)(10.0MiB/5308msec); 0 zone resets
    slat (usec): min=35, max=2954, avg=198.81, stdev=133.61
    clat (usec): min=8811, max=71634, avg=32852.65, stdev=5860.18
     lat (usec): min=8882, max=71856, avg=33053.48, stdev=5878.43
    clat percentiles (usec):
     |  1.00th=[17171],  5.00th=[24773], 10.00th=[27657], 20.00th=[28967],
     | 30.00th=[30278], 40.00th=[31327], 50.00th=[32375], 60.00th=[33162],
     | 70.00th=[34341], 80.00th=[36439], 90.00th=[39584], 95.00th=[42730],
     | 99.00th=[49021], 99.50th=[52167], 99.90th=[71828], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min= 1644, max= 2272, per=25.24%, avg=1933.50, stdev=193.67, samples=10
   iops        : min=  411, max=  568, avg=483.30, stdev=48.38, samples=10
  lat (msec)   : 10=0.12%, 20=1.95%, 50=97.11%, 100=0.82%
  cpu          : usr=0.66%, sys=7.29%, ctx=2928, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=181308: Tue Aug 18 21:35:17 2026
  write: IOPS=479, BW=1917KiB/s (1963kB/s)(10.0MiB/5343msec); 0 zone resets
    slat (usec): min=53, max=3626, avg=197.96, stdev=127.69
    clat (usec): min=4413, max=53597, avg=32876.60, stdev=5434.55
     lat (usec): min=4547, max=53691, avg=33076.23, stdev=5445.02
    clat percentiles (usec):
     |  1.00th=[18744],  5.00th=[25035], 10.00th=[27395], 20.00th=[29230],
     | 30.00th=[30278], 40.00th=[31327], 50.00th=[32375], 60.00th=[33424],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[39584], 95.00th=[42206],
     | 99.00th=[50070], 99.50th=[51119], 99.90th=[52691], 99.95th=[53216],
     | 99.99th=[53740]
   bw (  KiB/s): min= 1636, max= 2137, per=25.04%, avg=1918.50, stdev=158.28, samples=10
   iops        : min=  409, max=  534, avg=479.60, stdev=39.53, samples=10
  lat (msec)   : 10=0.04%, 20=1.52%, 50=97.58%, 100=0.86%
  cpu          : usr=0.66%, sys=7.75%, ctx=2939, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7662KiB/s (7846kB/s), 1915KiB/s-1930KiB/s (1961kB/s-1977kB/s), io=40.0MiB (41.9MB), run=5305-5346msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=181318: Tue Aug 18 21:35:26 2026
  read: IOPS=211, BW=848KiB/s (868kB/s)(5048KiB/5956msec)
    slat (usec): min=926, max=8518, avg=2512.93, stdev=884.54
    clat (usec): min=8496, max=66092, avg=35322.95, stdev=7162.88
     lat (usec): min=12661, max=69324, avg=37838.02, stdev=7415.87
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[26084], 10.00th=[27657], 20.00th=[29492],
     | 30.00th=[31065], 40.00th=[32637], 50.00th=[34341], 60.00th=[35914],
     | 70.00th=[38011], 80.00th=[40633], 90.00th=[44827], 95.00th=[49021],
     | 99.00th=[56886], 99.50th=[58983], 99.90th=[65274], 99.95th=[66323],
     | 99.99th=[66323]
   bw (  KiB/s): min=  640, max= 1016, per=24.91%, avg=839.27, stdev=127.54, samples=11
   iops        : min=  160, max=  254, avg=209.73, stdev=31.99, samples=11
  write: IOPS=217, BW=872KiB/s (893kB/s)(5192KiB/5956msec); 0 zone resets
    slat (usec): min=56, max=785, avg=158.08, stdev=82.19
    clat (usec): min=16118, max=65171, avg=35930.21, stdev=6719.52
     lat (usec): min=16284, max=65321, avg=36089.50, stdev=6724.91
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[27132], 10.00th=[28705], 20.00th=[30540],
     | 30.00th=[31851], 40.00th=[33162], 50.00th=[34341], 60.00th=[35914],
     | 70.00th=[38011], 80.00th=[41157], 90.00th=[45351], 95.00th=[49546],
     | 99.00th=[55313], 99.50th=[55837], 99.90th=[58459], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  727, max= 1032, per=24.82%, avg=870.27, stdev=104.32, samples=11
   iops        : min=  181, max=  258, avg=217.45, stdev=26.25, samples=11
  lat (msec)   : 10=0.04%, 20=0.20%, 50=95.16%, 100=4.61%
  cpu          : usr=0.50%, sys=6.43%, ctx=4542, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=181319: Tue Aug 18 21:35:26 2026
  read: IOPS=209, BW=836KiB/s (856kB/s)(4976KiB/5952msec)
    slat (usec): min=1020, max=7530, avg=2540.09, stdev=863.42
    clat (usec): min=6995, max=63061, avg=35268.47, stdev=6928.26
     lat (usec): min=9473, max=65288, avg=37810.83, stdev=7225.94
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[25297], 10.00th=[27395], 20.00th=[29754],
     | 30.00th=[31327], 40.00th=[32900], 50.00th=[34341], 60.00th=[35914],
     | 70.00th=[38011], 80.00th=[40633], 90.00th=[44303], 95.00th=[49021],
     | 99.00th=[55313], 99.50th=[56886], 99.90th=[61604], 99.95th=[63177],
     | 99.99th=[63177]
   bw (  KiB/s): min=  582, max=  976, per=24.47%, avg=824.45, stdev=134.01, samples=11
   iops        : min=  145, max=  244, avg=206.00, stdev=33.67, samples=11
  write: IOPS=221, BW=884KiB/s (906kB/s)(5264KiB/5952msec); 0 zone resets
    slat (usec): min=40, max=845, avg=161.98, stdev=83.15
    clat (usec): min=18489, max=58404, avg=35956.41, stdev=6912.42
     lat (usec): min=18589, max=58568, avg=36119.53, stdev=6917.91
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[26870], 10.00th=[28443], 20.00th=[30278],
     | 30.00th=[31851], 40.00th=[33162], 50.00th=[34341], 60.00th=[35914],
     | 70.00th=[38536], 80.00th=[41157], 90.00th=[46400], 95.00th=[50070],
     | 99.00th=[55837], 99.50th=[56886], 99.90th=[58459], 99.95th=[58459],
     | 99.99th=[58459]
   bw (  KiB/s): min=  768, max= 1056, per=25.28%, avg=886.55, stdev=107.03, samples=11
   iops        : min=  192, max=  264, avg=221.55, stdev=26.85, samples=11
  lat (msec)   : 10=0.04%, 20=0.08%, 50=95.43%, 100=4.45%
  cpu          : usr=0.62%, sys=6.30%, ctx=4559, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=181320: Tue Aug 18 21:35:26 2026
  read: IOPS=210, BW=841KiB/s (861kB/s)(4984KiB/5927msec)
    slat (usec): min=1006, max=7273, avg=2553.34, stdev=868.36
    clat (usec): min=7206, max=66891, avg=35323.60, stdev=7184.11
     lat (usec): min=9536, max=70421, avg=37879.26, stdev=7516.23
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[25297], 10.00th=[27132], 20.00th=[29492],
     | 30.00th=[31065], 40.00th=[32900], 50.00th=[34341], 60.00th=[35914],
     | 70.00th=[38011], 80.00th=[41157], 90.00th=[45351], 95.00th=[48497],
     | 99.00th=[55837], 99.50th=[57410], 99.90th=[60556], 99.95th=[66847],
     | 99.99th=[66847]
   bw (  KiB/s): min=  712, max= 1024, per=24.99%, avg=841.91, stdev=97.99, samples=11
   iops        : min=  178, max=  256, avg=210.36, stdev=24.64, samples=11
  write: IOPS=221, BW=887KiB/s (908kB/s)(5256KiB/5927msec); 0 zone resets
    slat (usec): min=49, max=1204, avg=163.30, stdev=82.98
    clat (usec): min=17234, max=66624, avg=35641.92, stdev=6797.09
     lat (usec): min=17390, max=66680, avg=35806.39, stdev=6803.96
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[27395], 10.00th=[28443], 20.00th=[30016],
     | 30.00th=[31589], 40.00th=[32900], 50.00th=[33817], 60.00th=[35390],
     | 70.00th=[37487], 80.00th=[41157], 90.00th=[45876], 95.00th=[49546],
     | 99.00th=[55313], 99.50th=[56886], 99.90th=[62129], 99.95th=[66847],
     | 99.99th=[66847]
   bw (  KiB/s): min=  638, max= 1136, per=25.07%, avg=879.27, stdev=156.72, samples=11
   iops        : min=  159, max=  284, avg=219.73, stdev=39.29, samples=11
  lat (msec)   : 10=0.04%, 20=0.12%, 50=96.02%, 100=3.83%
  cpu          : usr=0.61%, sys=6.50%, ctx=4545, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=181321: Tue Aug 18 21:35:26 2026
  read: IOPS=213, BW=853KiB/s (874kB/s)(5060KiB/5931msec)
    slat (usec): min=83, max=6972, avg=2486.55, stdev=900.02
    clat (usec): min=3670, max=64565, avg=35350.18, stdev=7487.22
     lat (usec): min=5398, max=69193, avg=37838.68, stdev=7824.81
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[25297], 10.00th=[27395], 20.00th=[29492],
     | 30.00th=[31065], 40.00th=[32637], 50.00th=[34341], 60.00th=[35914],
     | 70.00th=[38011], 80.00th=[40633], 90.00th=[45876], 95.00th=[50070],
     | 99.00th=[56361], 99.50th=[58459], 99.90th=[62129], 99.95th=[64750],
     | 99.99th=[64750]
   bw (  KiB/s): min=  672, max=  976, per=25.11%, avg=845.91, stdev=112.84, samples=11
   iops        : min=  168, max=  244, avg=211.36, stdev=28.31, samples=11
  write: IOPS=218, BW=873KiB/s (894kB/s)(5180KiB/5931msec); 0 zone resets
    slat (usec): min=51, max=3379, avg=159.59, stdev=118.60
    clat (usec): min=11539, max=74406, avg=36007.63, stdev=6994.73
     lat (usec): min=11697, max=74588, avg=36168.25, stdev=7004.06
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[27132], 10.00th=[28967], 20.00th=[30540],
     | 30.00th=[31851], 40.00th=[33162], 50.00th=[34341], 60.00th=[35914],
     | 70.00th=[38536], 80.00th=[41157], 90.00th=[45876], 95.00th=[50070],
     | 99.00th=[55837], 99.50th=[58459], 99.90th=[66323], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min=  662, max= 1024, per=24.84%, avg=871.09, stdev=110.44, samples=11
   iops        : min=  165, max=  256, avg=217.73, stdev=27.71, samples=11
  lat (msec)   : 4=0.04%, 10=0.16%, 20=0.27%, 50=94.73%, 100=4.80%
  cpu          : usr=0.66%, sys=6.22%, ctx=4579, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3369KiB/s (3450kB/s), 836KiB/s-853KiB/s (856kB/s-874kB/s), io=19.6MiB (20.5MB), run=5927-5956msec
  WRITE: bw=3508KiB/s (3592kB/s), 872KiB/s-887KiB/s (893kB/s-908kB/s), io=20.4MiB (21.4MB), run=5927-5956msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=181325: Tue Aug 18 21:35:30 2026
  read: IOPS=8, BW=320MiB/s (336MB/s)(40.0MiB/125msec)
    slat (nsec): min=12536k, max=12536k, avg=12535854.00, stdev= 0.00
    clat (nsec): min=107327k, max=107327k, avg=107326503.00, stdev= 0.00
     lat (nsec): min=119868k, max=119868k, avg=119867804.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  108],  5.00th=[  108], 10.00th=[  108], 20.00th=[  108],
     | 30.00th=[  108], 40.00th=[  108], 50.00th=[  108], 60.00th=[  108],
     | 70.00th=[  108], 80.00th=[  108], 90.00th=[  108], 95.00th=[  108],
     | 99.00th=[  108], 99.50th=[  108], 99.90th=[  108], 99.95th=[  108],
     | 99.99th=[  108]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=8.87%, ctx=12, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=320MiB/s (336MB/s), 320MiB/s-320MiB/s (336MB/s-336MB/s), io=40.0MiB (41.9MB), run=125-125msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout