== sanity test 398c: run fio to test AIO ================= 19:58:06 (1788911886)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.131792 s, 318 MB/s
osc.lustre-OST0000-osc-ffff8cc486252000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=187082: Tue Sep  8 19:58:15 2026
  write: IOPS=503, BW=2015KiB/s (2063kB/s)(10.0MiB/5082msec); 0 zone resets
    slat (usec): min=24, max=794, avg=87.72, stdev=53.83
    clat (usec): min=4340, max=48321, avg=31363.70, stdev=3721.14
     lat (usec): min=4397, max=48394, avg=31452.36, stdev=3719.39
    clat percentiles (usec):
     |  1.00th=[21365],  5.00th=[26346], 10.00th=[27395], 20.00th=[28705],
     | 30.00th=[29492], 40.00th=[30540], 50.00th=[31065], 60.00th=[31851],
     | 70.00th=[32900], 80.00th=[33817], 90.00th=[35914], 95.00th=[37487],
     | 99.00th=[40109], 99.50th=[47449], 99.90th=[47973], 99.95th=[47973],
     | 99.99th=[48497]
   bw (  KiB/s): min= 1792, max= 2152, per=24.81%, avg=1997.89, stdev=123.90, samples=9
   iops        : min=  448, max=  538, avg=499.44, stdev=31.01, samples=9
  lat (msec)   : 10=0.08%, 20=0.70%, 50=99.22%
  cpu          : usr=0.39%, sys=4.17%, ctx=2667, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=187083: Tue Sep  8 19:58:15 2026
  write: IOPS=506, BW=2027KiB/s (2076kB/s)(10.0MiB/5052msec); 0 zone resets
    slat (usec): min=26, max=1011, avg=94.58, stdev=62.49
    clat (usec): min=14015, max=58276, avg=31410.01, stdev=3790.42
     lat (usec): min=14148, max=58436, avg=31505.57, stdev=3793.19
    clat percentiles (usec):
     |  1.00th=[19006],  5.00th=[26346], 10.00th=[27395], 20.00th=[28967],
     | 30.00th=[29754], 40.00th=[30540], 50.00th=[31327], 60.00th=[31851],
     | 70.00th=[32900], 80.00th=[34341], 90.00th=[35914], 95.00th=[37487],
     | 99.00th=[39584], 99.50th=[42730], 99.90th=[52167], 99.95th=[52167],
     | 99.99th=[58459]
   bw (  KiB/s): min= 1896, max= 2120, per=24.97%, avg=2010.67, stdev=85.88, samples=9
   iops        : min=  474, max=  530, avg=502.67, stdev=21.47, samples=9
  lat (msec)   : 20=1.33%, 50=98.55%, 100=0.12%
  cpu          : usr=0.77%, sys=3.72%, ctx=2770, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=187084: Tue Sep  8 19:58:15 2026
  write: IOPS=503, BW=2013KiB/s (2062kB/s)(10.0MiB/5086msec); 0 zone resets
    slat (usec): min=21, max=523, avg=95.38, stdev=61.82
    clat (usec): min=2631, max=47879, avg=31348.10, stdev=3635.16
     lat (usec): min=2712, max=47992, avg=31444.40, stdev=3635.91
    clat percentiles (usec):
     |  1.00th=[20579],  5.00th=[26084], 10.00th=[27395], 20.00th=[28705],
     | 30.00th=[29492], 40.00th=[30278], 50.00th=[31327], 60.00th=[31851],
     | 70.00th=[32900], 80.00th=[33817], 90.00th=[35914], 95.00th=[37487],
     | 99.00th=[40109], 99.50th=[43779], 99.90th=[47973], 99.95th=[47973],
     | 99.99th=[47973]
   bw (  KiB/s): min= 1789, max= 2152, per=24.72%, avg=1990.78, stdev=119.19, samples=9
   iops        : min=  447, max=  538, avg=497.67, stdev=29.85, samples=9
  lat (msec)   : 4=0.04%, 20=0.43%, 50=99.53%
  cpu          : usr=0.65%, sys=3.83%, ctx=2752, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=187085: Tue Sep  8 19:58:15 2026
  write: IOPS=504, BW=2019KiB/s (2068kB/s)(10.0MiB/5071msec); 0 zone resets
    slat (usec): min=24, max=20357, avg=105.58, stdev=415.74
    clat (usec): min=7427, max=53203, avg=31353.06, stdev=3840.69
     lat (usec): min=7486, max=53247, avg=31459.65, stdev=3824.70
    clat percentiles (usec):
     |  1.00th=[19006],  5.00th=[26084], 10.00th=[27395], 20.00th=[28705],
     | 30.00th=[29754], 40.00th=[30540], 50.00th=[31065], 60.00th=[32113],
     | 70.00th=[32900], 80.00th=[34341], 90.00th=[35914], 95.00th=[37487],
     | 99.00th=[40109], 99.50th=[44303], 99.90th=[53216], 99.95th=[53216],
     | 99.99th=[53216]
   bw (  KiB/s): min= 1864, max= 2128, per=24.87%, avg=2003.11, stdev=99.47, samples=9
   iops        : min=  466, max=  532, avg=500.78, stdev=24.87, samples=9
  lat (msec)   : 10=0.04%, 20=1.17%, 50=98.63%, 100=0.16%
  cpu          : usr=0.49%, sys=4.38%, ctx=2739, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8053KiB/s (8247kB/s), 2013KiB/s-2027KiB/s (2062kB/s-2076kB/s), io=40.0MiB (41.9MB), run=5052-5086msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=187095: Tue Sep  8 19:58:24 2026
  read: IOPS=200, BW=803KiB/s (822kB/s)(5048KiB/6289msec)
    slat (usec): min=953, max=10424, avg=2597.30, stdev=883.40
    clat (usec): min=6803, max=67011, avg=37527.31, stdev=7439.06
     lat (usec): min=9093, max=70868, avg=40126.85, stdev=7694.10
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[26870], 10.00th=[28967], 20.00th=[31327],
     | 30.00th=[33162], 40.00th=[34866], 50.00th=[36439], 60.00th=[38536],
     | 70.00th=[41157], 80.00th=[43779], 90.00th=[47973], 95.00th=[51119],
     | 99.00th=[57934], 99.50th=[61604], 99.90th=[64750], 99.95th=[66847],
     | 99.99th=[66847]
   bw (  KiB/s): min=  654, max=  920, per=25.19%, avg=803.67, stdev=88.55, samples=12
   iops        : min=  163, max=  230, avg=200.83, stdev=22.16, samples=12
  write: IOPS=206, BW=826KiB/s (845kB/s)(5192KiB/6289msec); 0 zone resets
    slat (usec): min=24, max=1701, avg=96.57, stdev=79.50
    clat (usec): min=7422, max=66716, avg=37838.50, stdev=6893.96
     lat (usec): min=7496, max=66838, avg=37936.29, stdev=6896.64
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[28705], 10.00th=[30540], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[36439], 60.00th=[38536],
     | 70.00th=[40109], 80.00th=[42730], 90.00th=[47449], 95.00th=[51119],
     | 99.00th=[57934], 99.50th=[61080], 99.90th=[65799], 99.95th=[66847],
     | 99.99th=[66847]
   bw (  KiB/s): min=  664, max= 1053, per=24.96%, avg=828.83, stdev=133.96, samples=12
   iops        : min=  166, max=  263, avg=207.17, stdev=33.47, samples=12
  lat (msec)   : 10=0.08%, 20=0.04%, 50=93.87%, 100=6.02%
  cpu          : usr=0.54%, sys=4.47%, ctx=4715, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=187096: Tue Sep  8 19:58:24 2026
  read: IOPS=198, BW=794KiB/s (813kB/s)(4976KiB/6270msec)
    slat (usec): min=942, max=13826, avg=2584.72, stdev=932.33
    clat (usec): min=20824, max=70250, avg=37376.41, stdev=7419.54
     lat (usec): min=23480, max=73840, avg=39963.34, stdev=7733.04
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[27132], 10.00th=[28967], 20.00th=[31065],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[36439], 60.00th=[38011],
     | 70.00th=[40633], 80.00th=[43254], 90.00th=[46924], 95.00th=[50070],
     | 99.00th=[61080], 99.50th=[65799], 99.90th=[68682], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min=  629, max=  936, per=25.04%, avg=798.92, stdev=95.57, samples=12
   iops        : min=  157, max=  234, avg=199.67, stdev=23.88, samples=12
  write: IOPS=209, BW=840KiB/s (860kB/s)(5264KiB/6270msec); 0 zone resets
    slat (usec): min=22, max=528, avg=90.54, stdev=49.35
    clat (usec): min=22874, max=71405, avg=37773.01, stdev=6644.37
     lat (usec): min=23067, max=71483, avg=37864.46, stdev=6646.82
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[28705], 10.00th=[30278], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[36963], 60.00th=[38011],
     | 70.00th=[40633], 80.00th=[43254], 90.00th=[46924], 95.00th=[50070],
     | 99.00th=[56886], 99.50th=[61080], 99.90th=[63701], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  688, max= 1053, per=25.41%, avg=843.75, stdev=135.49, samples=12
   iops        : min=  172, max=  263, avg=210.92, stdev=33.84, samples=12
  lat (msec)   : 50=94.73%, 100=5.27%
  cpu          : usr=0.35%, sys=4.66%, ctx=4614, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=187097: Tue Sep  8 19:58:24 2026
  read: IOPS=198, BW=793KiB/s (812kB/s)(4984KiB/6288msec)
    slat (usec): min=1001, max=13780, avg=2578.37, stdev=917.00
    clat (usec): min=19866, max=65812, avg=37187.58, stdev=7234.36
     lat (usec): min=21705, max=68292, avg=39767.78, stdev=7508.96
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[27395], 10.00th=[28967], 20.00th=[31327],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35914], 60.00th=[38011],
     | 70.00th=[40633], 80.00th=[42730], 90.00th=[46400], 95.00th=[49546],
     | 99.00th=[60031], 99.50th=[63177], 99.90th=[65274], 99.95th=[65799],
     | 99.99th=[65799]
   bw (  KiB/s): min=  622, max= 1005, per=25.02%, avg=798.25, stdev=140.51, samples=12
   iops        : min=  155, max=  251, avg=199.50, stdev=35.15, samples=12
  write: IOPS=208, BW=836KiB/s (856kB/s)(5256KiB/6288msec); 0 zone resets
    slat (usec): min=35, max=769, avg=95.46, stdev=66.29
    clat (usec): min=24230, max=66693, avg=38016.75, stdev=6678.41
     lat (usec): min=24380, max=66799, avg=38113.17, stdev=6680.49
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28967], 10.00th=[30278], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[36963], 60.00th=[39060],
     | 70.00th=[41157], 80.00th=[43254], 90.00th=[47449], 95.00th=[50070],
     | 99.00th=[56886], 99.50th=[61080], 99.90th=[66323], 99.95th=[66847],
     | 99.99th=[66847]
   bw (  KiB/s): min=  664, max=  934, per=25.15%, avg=835.17, stdev=95.60, samples=12
   iops        : min=  166, max=  233, avg=208.75, stdev=23.85, samples=12
  lat (msec)   : 20=0.04%, 50=95.31%, 100=4.65%
  cpu          : usr=0.54%, sys=4.41%, ctx=4833, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=187098: Tue Sep  8 19:58:24 2026
  read: IOPS=202, BW=809KiB/s (828kB/s)(5060KiB/6257msec)
    slat (usec): min=47, max=13666, avg=2563.76, stdev=918.87
    clat (usec): min=5630, max=67229, avg=37586.17, stdev=7761.40
     lat (usec): min=5731, max=69987, avg=40152.03, stdev=8029.48
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[27132], 10.00th=[29230], 20.00th=[31589],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[36439], 60.00th=[38011],
     | 70.00th=[40633], 80.00th=[43779], 90.00th=[47449], 95.00th=[51119],
     | 99.00th=[63701], 99.50th=[64226], 99.90th=[65799], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  688, max=  958, per=25.46%, avg=812.25, stdev=96.66, samples=12
   iops        : min=  172, max=  239, avg=203.00, stdev=24.12, samples=12
  write: IOPS=206, BW=828KiB/s (848kB/s)(5180KiB/6257msec); 0 zone resets
    slat (usec): min=33, max=3009, avg=102.31, stdev=110.66
    clat (usec): min=16826, max=75312, avg=37818.62, stdev=6909.10
     lat (usec): min=16935, max=75375, avg=37921.88, stdev=6908.66
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[28443], 10.00th=[30278], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[36963], 60.00th=[38536],
     | 70.00th=[40633], 80.00th=[43254], 90.00th=[46924], 95.00th=[50070],
     | 99.00th=[58983], 99.50th=[60031], 99.90th=[74974], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min=  640, max= 1005, per=25.01%, avg=830.58, stdev=108.28, samples=12
   iops        : min=  160, max=  251, avg=207.58, stdev=27.06, samples=12
  lat (msec)   : 10=0.20%, 20=0.20%, 50=93.79%, 100=5.82%
  cpu          : usr=0.58%, sys=4.43%, ctx=4922, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3191KiB/s (3268kB/s), 793KiB/s-809KiB/s (812kB/s-828kB/s), io=19.6MiB (20.5MB), run=6257-6289msec
  WRITE: bw=3322KiB/s (3402kB/s), 826KiB/s-840KiB/s (845kB/s-860kB/s), io=20.4MiB (21.4MB), run=6257-6289msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=187103: Tue Sep  8 19:58:28 2026
  read: IOPS=9, BW=374MiB/s (392MB/s)(40.0MiB/107msec)
    slat (nsec): min=9269.0k, max=9269.0k, avg=9269989.00, stdev= 0.00
    clat (nsec): min=92249k, max=92249k, avg=92249328.00, stdev= 0.00
     lat (nsec): min=101523k, max=101523k, avg=101522863.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[91751],  5.00th=[91751], 10.00th=[91751], 20.00th=[91751],
     | 30.00th=[91751], 40.00th=[91751], 50.00th=[91751], 60.00th=[91751],
     | 70.00th=[91751], 80.00th=[91751], 90.00th=[91751], 95.00th=[91751],
     | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751],
     | 99.99th=[91751]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.49%, ctx=10, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=374MiB/s (392MB/s), 374MiB/s-374MiB/s (392MB/s-392MB/s), io=40.0MiB (41.9MB), run=107-107msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout