== sanity test 398c: run fio to test AIO ================= 17:07:46 (1771970866)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.137441 s, 305 MB/s
osc.lustre-OST0000-osc-ffffa0d488817800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=148247: Tue Feb 24 17:07:55 2026
  write: IOPS=491, BW=1967KiB/s (2014kB/s)(10.0MiB/5206msec); 0 zone resets
    slat (usec): min=20, max=1174, avg=85.48, stdev=67.39
    clat (usec): min=3014, max=81505, avg=32406.14, stdev=9882.29
     lat (usec): min=3150, max=81569, avg=32492.82, stdev=9886.37
    clat percentiles (usec):
     |  1.00th=[16188],  5.00th=[20317], 10.00th=[22676], 20.00th=[24773],
     | 30.00th=[26608], 40.00th=[28181], 50.00th=[30278], 60.00th=[32637],
     | 70.00th=[35390], 80.00th=[39584], 90.00th=[45876], 95.00th=[50594],
     | 99.00th=[69731], 99.50th=[72877], 99.90th=[74974], 99.95th=[80217],
     | 99.99th=[81265]
   bw (  KiB/s): min= 1253, max= 2552, per=25.08%, avg=1963.90, stdev=417.65, samples=10
   iops        : min=  313, max=  638, avg=490.80, stdev=104.43, samples=10
  lat (msec)   : 4=0.04%, 10=0.12%, 20=4.53%, 50=90.00%, 100=5.31%
  cpu          : usr=0.63%, sys=4.00%, ctx=1231, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=148248: Tue Feb 24 17:07:55 2026
  write: IOPS=489, BW=1959KiB/s (2006kB/s)(10.0MiB/5228msec); 0 zone resets
    slat (usec): min=19, max=1033, avg=86.00, stdev=59.14
    clat (usec): min=7209, max=81628, avg=32523.88, stdev=9767.71
     lat (usec): min=7364, max=81667, avg=32611.17, stdev=9771.47
    clat percentiles (usec):
     |  1.00th=[17957],  5.00th=[20579], 10.00th=[22676], 20.00th=[25035],
     | 30.00th=[26870], 40.00th=[28443], 50.00th=[30540], 60.00th=[32637],
     | 70.00th=[35390], 80.00th=[39060], 90.00th=[45876], 95.00th=[49546],
     | 99.00th=[70779], 99.50th=[73925], 99.90th=[81265], 99.95th=[81265],
     | 99.99th=[81265]
   bw (  KiB/s): min= 1253, max= 2568, per=24.98%, avg=1956.10, stdev=416.31, samples=10
   iops        : min=  313, max=  642, avg=488.90, stdev=104.10, samples=10
  lat (msec)   : 10=0.04%, 20=3.32%, 50=91.64%, 100=5.00%
  cpu          : usr=0.82%, sys=3.88%, ctx=1241, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=148249: Tue Feb 24 17:07:55 2026
  write: IOPS=490, BW=1964KiB/s (2011kB/s)(10.0MiB/5215msec); 0 zone resets
    slat (usec): min=20, max=1219, avg=89.36, stdev=73.17
    clat (usec): min=3001, max=79809, avg=32438.78, stdev=9806.93
     lat (usec): min=3063, max=79958, avg=32529.40, stdev=9810.07
    clat percentiles (usec):
     |  1.00th=[17433],  5.00th=[20317], 10.00th=[22676], 20.00th=[25035],
     | 30.00th=[26608], 40.00th=[28443], 50.00th=[30540], 60.00th=[32375],
     | 70.00th=[35390], 80.00th=[39060], 90.00th=[46400], 95.00th=[50070],
     | 99.00th=[69731], 99.50th=[73925], 99.90th=[74974], 99.95th=[74974],
     | 99.99th=[80217]
   bw (  KiB/s): min= 1237, max= 2536, per=25.04%, avg=1961.10, stdev=424.00, samples=10
   iops        : min=  309, max=  634, avg=490.00, stdev=106.02, samples=10
  lat (msec)   : 4=0.08%, 10=0.08%, 20=3.48%, 50=91.41%, 100=4.96%
  cpu          : usr=0.44%, sys=4.30%, ctx=1287, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=148250: Tue Feb 24 17:07:55 2026
  write: IOPS=489, BW=1958KiB/s (2005kB/s)(10.0MiB/5230msec); 0 zone resets
    slat (usec): min=18, max=1193, avg=83.77, stdev=62.26
    clat (usec): min=12835, max=81863, avg=32518.03, stdev=9664.48
     lat (usec): min=12884, max=82047, avg=32602.99, stdev=9669.54
    clat percentiles (usec):
     |  1.00th=[18220],  5.00th=[20579], 10.00th=[22676], 20.00th=[25035],
     | 30.00th=[26870], 40.00th=[28443], 50.00th=[30278], 60.00th=[32113],
     | 70.00th=[35390], 80.00th=[39060], 90.00th=[45351], 95.00th=[51119],
     | 99.00th=[67634], 99.50th=[72877], 99.90th=[77071], 99.95th=[82314],
     | 99.99th=[82314]
   bw (  KiB/s): min= 1261, max= 2560, per=24.90%, avg=1950.10, stdev=422.43, samples=10
   iops        : min=  315, max=  640, avg=487.30, stdev=105.64, samples=10
  lat (msec)   : 20=3.09%, 50=91.72%, 100=5.20%
  cpu          : usr=0.73%, sys=3.82%, ctx=1223, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7832KiB/s (8020kB/s), 1958KiB/s-1967KiB/s (2005kB/s-2014kB/s), io=40.0MiB (41.9MB), run=5206-5230msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=148260: Tue Feb 24 17:08:02 2026
  read: IOPS=292, BW=1171KiB/s (1199kB/s)(5048KiB/4311msec)
    slat (usec): min=728, max=7589, avg=1573.89, stdev=781.64
    clat (usec): min=11961, max=60399, avg=25769.28, stdev=7682.48
     lat (usec): min=13043, max=63831, avg=27344.99, stdev=7949.55
    clat percentiles (usec):
     |  1.00th=[14615],  5.00th=[16909], 10.00th=[17957], 20.00th=[19792],
     | 30.00th=[21365], 40.00th=[22676], 50.00th=[24249], 60.00th=[25560],
     | 70.00th=[27395], 80.00th=[30016], 90.00th=[35390], 95.00th=[42206],
     | 99.00th=[53740], 99.50th=[56361], 99.90th=[57934], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  696, max= 1424, per=25.01%, avg=1164.38, stdev=217.57, samples=8
   iops        : min=  174, max=  356, avg=291.00, stdev=54.37, samples=8
  write: IOPS=301, BW=1204KiB/s (1233kB/s)(5192KiB/4311msec); 0 zone resets
    slat (usec): min=19, max=1505, avg=78.18, stdev=68.38
    clat (usec): min=4164, max=63476, avg=26214.02, stdev=7715.85
     lat (usec): min=4215, max=63507, avg=26293.21, stdev=7716.13
    clat percentiles (usec):
     |  1.00th=[15139],  5.00th=[17433], 10.00th=[19006], 20.00th=[20579],
     | 30.00th=[21890], 40.00th=[23200], 50.00th=[24511], 60.00th=[25822],
     | 70.00th=[27657], 80.00th=[30278], 90.00th=[35914], 95.00th=[43254],
     | 99.00th=[54264], 99.50th=[56361], 99.90th=[63701], 99.95th=[63701],
     | 99.99th=[63701]
   bw (  KiB/s): min=  800, max= 1413, per=24.75%, avg=1199.62, stdev=188.34, samples=8
   iops        : min=  200, max=  353, avg=299.88, stdev=47.05, samples=8
  lat (msec)   : 10=0.16%, 20=18.63%, 50=79.53%, 100=1.68%
  cpu          : usr=0.81%, sys=6.17%, ctx=2348, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=148261: Tue Feb 24 17:08:02 2026
  read: IOPS=291, BW=1167KiB/s (1195kB/s)(4976KiB/4264msec)
    slat (usec): min=39, max=9632, avg=1584.12, stdev=879.65
    clat (usec): min=3323, max=68942, avg=25672.94, stdev=7898.63
     lat (usec): min=3380, max=70769, avg=27258.88, stdev=8255.69
    clat percentiles (usec):
     |  1.00th=[14353],  5.00th=[16450], 10.00th=[17957], 20.00th=[20055],
     | 30.00th=[21365], 40.00th=[22414], 50.00th=[23987], 60.00th=[25560],
     | 70.00th=[27132], 80.00th=[29754], 90.00th=[35390], 95.00th=[41157],
     | 99.00th=[55837], 99.50th=[57410], 99.90th=[65799], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  752, max= 1357, per=24.95%, avg=1161.25, stdev=192.98, samples=8
   iops        : min=  188, max=  339, avg=290.25, stdev=48.22, samples=8
  write: IOPS=308, BW=1235KiB/s (1264kB/s)(5264KiB/4264msec); 0 zone resets
    slat (usec): min=21, max=721, avg=78.08, stdev=53.62
    clat (usec): min=6405, max=65491, avg=25903.26, stdev=7509.28
     lat (usec): min=6475, max=65568, avg=25982.39, stdev=7513.80
    clat percentiles (usec):
     |  1.00th=[15401],  5.00th=[17433], 10.00th=[18744], 20.00th=[20579],
     | 30.00th=[21890], 40.00th=[23200], 50.00th=[23987], 60.00th=[25297],
     | 70.00th=[27132], 80.00th=[30016], 90.00th=[34866], 95.00th=[41681],
     | 99.00th=[52691], 99.50th=[57410], 99.90th=[63701], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  728, max= 1400, per=25.46%, avg=1233.75, stdev=226.87, samples=8
   iops        : min=  182, max=  350, avg=308.38, stdev=56.68, samples=8
  lat (msec)   : 4=0.08%, 10=0.04%, 20=18.40%, 50=79.69%, 100=1.80%
  cpu          : usr=0.80%, sys=6.12%, ctx=2336, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=148262: Tue Feb 24 17:08:02 2026
  read: IOPS=289, BW=1157KiB/s (1185kB/s)(4984KiB/4306msec)
    slat (usec): min=741, max=8675, avg=1608.80, stdev=838.81
    clat (usec): min=1472, max=72184, avg=25770.57, stdev=8008.93
     lat (usec): min=3170, max=73485, avg=27381.14, stdev=8308.50
    clat percentiles (usec):
     |  1.00th=[14746],  5.00th=[16712], 10.00th=[17957], 20.00th=[19792],
     | 30.00th=[21627], 40.00th=[22676], 50.00th=[23987], 60.00th=[25560],
     | 70.00th=[27395], 80.00th=[30540], 90.00th=[34866], 95.00th=[40633],
     | 99.00th=[57934], 99.50th=[58983], 99.90th=[68682], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  792, max= 1360, per=24.90%, avg=1158.88, stdev=186.85, samples=8
   iops        : min=  198, max=  340, avg=289.62, stdev=46.68, samples=8
  write: IOPS=305, BW=1221KiB/s (1250kB/s)(5256KiB/4306msec); 0 zone resets
    slat (usec): min=21, max=2631, avg=79.19, stdev=90.78
    clat (usec): min=6035, max=62612, avg=26159.26, stdev=7527.94
     lat (usec): min=6109, max=62671, avg=26239.40, stdev=7529.47
    clat percentiles (usec):
     |  1.00th=[15926],  5.00th=[17695], 10.00th=[19006], 20.00th=[20317],
     | 30.00th=[21627], 40.00th=[23200], 50.00th=[24511], 60.00th=[25822],
     | 70.00th=[27657], 80.00th=[30278], 90.00th=[36963], 95.00th=[42206],
     | 99.00th=[54789], 99.50th=[56886], 99.90th=[60556], 99.95th=[62653],
     | 99.99th=[62653]
   bw (  KiB/s): min=  712, max= 1429, per=25.01%, avg=1211.87, stdev=219.55, samples=8
   iops        : min=  178, max=  357, avg=302.88, stdev=54.87, samples=8
  lat (msec)   : 2=0.04%, 10=0.04%, 20=18.24%, 50=79.84%, 100=1.84%
  cpu          : usr=0.65%, sys=6.23%, ctx=2297, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=148263: Tue Feb 24 17:08:02 2026
  read: IOPS=293, BW=1176KiB/s (1204kB/s)(5060KiB/4303msec)
    slat (usec): min=766, max=8090, avg=1588.76, stdev=821.46
    clat (usec): min=2255, max=65121, avg=25437.62, stdev=7300.11
     lat (usec): min=3917, max=66546, avg=27028.00, stdev=7596.12
    clat percentiles (usec):
     |  1.00th=[14222],  5.00th=[17171], 10.00th=[18220], 20.00th=[19792],
     | 30.00th=[21103], 40.00th=[22414], 50.00th=[24249], 60.00th=[25560],
     | 70.00th=[27132], 80.00th=[29754], 90.00th=[34341], 95.00th=[39584],
     | 99.00th=[51119], 99.50th=[55313], 99.90th=[63177], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  672, max= 1344, per=25.03%, avg=1165.38, stdev=214.42, samples=8
   iops        : min=  168, max=  336, avg=291.25, stdev=53.58, samples=8
  write: IOPS=300, BW=1204KiB/s (1233kB/s)(5180KiB/4303msec); 0 zone resets
    slat (usec): min=26, max=1460, avg=81.04, stdev=82.06
    clat (usec): min=13902, max=67468, avg=26448.69, stdev=7935.84
     lat (usec): min=13961, max=67518, avg=26530.73, stdev=7939.49
    clat percentiles (usec):
     |  1.00th=[15401],  5.00th=[17433], 10.00th=[19006], 20.00th=[20841],
     | 30.00th=[22152], 40.00th=[23200], 50.00th=[24511], 60.00th=[25822],
     | 70.00th=[27657], 80.00th=[30278], 90.00th=[36963], 95.00th=[44303],
     | 99.00th=[55837], 99.50th=[57410], 99.90th=[63701], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  856, max= 1437, per=24.84%, avg=1203.88, stdev=178.71, samples=8
   iops        : min=  214, max=  359, avg=300.88, stdev=44.65, samples=8
  lat (msec)   : 4=0.04%, 10=0.04%, 20=17.70%, 50=80.47%, 100=1.76%
  cpu          : usr=0.42%, sys=6.46%, ctx=2375, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=4655KiB/s (4767kB/s), 1157KiB/s-1176KiB/s (1185kB/s-1204kB/s), io=19.6MiB (20.5MB), run=4264-4311msec
  WRITE: bw=4846KiB/s (4963kB/s), 1204KiB/s-1235KiB/s (1233kB/s-1264kB/s), io=20.4MiB (21.4MB), run=4264-4311msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=148267: Tue Feb 24 17:08:06 2026
  read: IOPS=8, BW=354MiB/s (371MB/s)(40.0MiB/113msec)
    slat (nsec): min=12016k, max=12016k, avg=12015625.00, stdev= 0.00
    clat (nsec): min=92124k, max=92124k, avg=92123950.00, stdev= 0.00
     lat (nsec): min=104143k, max=104143k, avg=104143303.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[91751],  5.00th=[91751], 10.00th=[91751], 20.00th=[91751],
     | 30.00th=[91751], 40.00th=[91751], 50.00th=[91751], 60.00th=[91751],
     | 70.00th=[91751], 80.00th=[91751], 90.00th=[91751], 95.00th=[91751],
     | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751],
     | 99.99th=[91751]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=10.71%, ctx=3, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=354MiB/s (371MB/s), 354MiB/s-354MiB/s (371MB/s-371MB/s), io=40.0MiB (41.9MB), run=113-113msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck