== sanity test 398c: run fio to test AIO ================= 02:01:44 (1789711304)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.125272 s, 335 MB/s
osc.lustre-OST0000-osc-ffff933f6e90d000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=182221: Fri Sep 18 02:01:53 2026
  write: IOPS=476, BW=1907KiB/s (1953kB/s)(10.0MiB/5370msec); 0 zone resets
    slat (usec): min=25, max=501, avg=105.25, stdev=66.43
    clat (usec): min=13438, max=65624, avg=33400.85, stdev=4011.85
     lat (usec): min=13506, max=65863, avg=33507.17, stdev=4015.80
    clat percentiles (usec):
     |  1.00th=[21627],  5.00th=[27657], 10.00th=[29230], 20.00th=[31065],
     | 30.00th=[32113], 40.00th=[32900], 50.00th=[33817], 60.00th=[34341],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[36963], 95.00th=[38011],
     | 99.00th=[41681], 99.50th=[51119], 99.90th=[62129], 99.95th=[62129],
     | 99.99th=[65799]
   bw (  KiB/s): min= 1752, max= 2170, per=25.21%, avg=1908.20, stdev=130.72, samples=10
   iops        : min=  438, max=  542, avg=477.00, stdev=32.57, samples=10
  lat (msec)   : 20=0.62%, 50=98.59%, 100=0.78%
  cpu          : usr=0.34%, sys=4.06%, ctx=2827, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182222: Fri Sep 18 02:01:53 2026
  write: IOPS=475, BW=1903KiB/s (1948kB/s)(10.0MiB/5382msec); 0 zone resets
    slat (usec): min=23, max=482, avg=104.66, stdev=67.80
    clat (usec): min=13209, max=66045, avg=33406.32, stdev=4049.75
     lat (usec): min=13259, max=66124, avg=33512.05, stdev=4055.12
    clat percentiles (usec):
     |  1.00th=[21365],  5.00th=[27657], 10.00th=[28967], 20.00th=[31065],
     | 30.00th=[32113], 40.00th=[32900], 50.00th=[33817], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[35914], 90.00th=[36963], 95.00th=[38011],
     | 99.00th=[50070], 99.50th=[51643], 99.90th=[65799], 99.95th=[65799],
     | 99.99th=[65799]
   bw (  KiB/s): min= 1712, max= 2157, per=25.14%, avg=1902.90, stdev=131.26, samples=10
   iops        : min=  428, max=  539, avg=475.70, stdev=32.76, samples=10
  lat (msec)   : 20=0.62%, 50=98.36%, 100=1.02%
  cpu          : usr=0.58%, sys=3.81%, ctx=2920, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182223: Fri Sep 18 02:01:53 2026
  write: IOPS=473, BW=1893KiB/s (1938kB/s)(10.0MiB/5410msec); 0 zone resets
    slat (usec): min=24, max=500, avg=104.82, stdev=67.68
    clat (usec): min=17645, max=51706, avg=33329.75, stdev=3438.69
     lat (usec): min=17748, max=51806, avg=33435.69, stdev=3444.00
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[27657], 10.00th=[28967], 20.00th=[31065],
     | 30.00th=[31851], 40.00th=[32900], 50.00th=[33817], 60.00th=[34341],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[36963], 95.00th=[38011],
     | 99.00th=[39060], 99.50th=[50594], 99.90th=[51643], 99.95th=[51643],
     | 99.99th=[51643]
   bw (  KiB/s): min= 1744, max= 2048, per=24.96%, avg=1889.40, stdev=102.29, samples=10
   iops        : min=  436, max=  512, avg=472.30, stdev=25.53, samples=10
  lat (msec)   : 20=0.12%, 50=99.30%, 100=0.59%
  cpu          : usr=0.46%, sys=3.96%, ctx=2750, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182224: Fri Sep 18 02:01:53 2026
  write: IOPS=473, BW=1892KiB/s (1938kB/s)(10.0MiB/5411msec); 0 zone resets
    slat (usec): min=29, max=519, avg=101.34, stdev=61.42
    clat (usec): min=11857, max=51800, avg=33396.08, stdev=3431.04
     lat (usec): min=11962, max=51885, avg=33498.52, stdev=3431.78
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[27919], 10.00th=[29492], 20.00th=[31065],
     | 30.00th=[32113], 40.00th=[32900], 50.00th=[33817], 60.00th=[34341],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[36963], 95.00th=[38011],
     | 99.00th=[39060], 99.50th=[51119], 99.90th=[51643], 99.95th=[51643],
     | 99.99th=[51643]
   bw (  KiB/s): min= 1744, max= 2024, per=24.94%, avg=1887.80, stdev=99.48, samples=10
   iops        : min=  436, max=  506, avg=471.90, stdev=24.81, samples=10
  lat (msec)   : 20=0.23%, 50=99.14%, 100=0.62%
  cpu          : usr=0.41%, sys=4.34%, ctx=2797, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7570KiB/s (7751kB/s), 1892KiB/s-1907KiB/s (1938kB/s-1953kB/s), io=40.0MiB (41.9MB), run=5370-5411msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=182235: Fri Sep 18 02:02:03 2026
  read: IOPS=183, BW=734KiB/s (752kB/s)(5048KiB/6876msec)
    slat (usec): min=1068, max=12156, avg=2990.34, stdev=1045.76
    clat (usec): min=4268, max=66106, avg=41023.39, stdev=7483.90
     lat (usec): min=6148, max=69537, avg=44015.31, stdev=7824.88
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[29492], 10.00th=[31589], 20.00th=[34866],
     | 30.00th=[36963], 40.00th=[38536], 50.00th=[40633], 60.00th=[43254],
     | 70.00th=[45351], 80.00th=[47449], 90.00th=[50070], 95.00th=[52691],
     | 99.00th=[57410], 99.50th=[60031], 99.90th=[63701], 99.95th=[66323],
     | 99.99th=[66323]
   bw (  KiB/s): min=  632, max=  832, per=24.81%, avg=724.08, stdev=70.35, samples=13
   iops        : min=  158, max=  208, avg=181.00, stdev=17.56, samples=13
  write: IOPS=188, BW=755KiB/s (773kB/s)(5192KiB/6876msec); 0 zone resets
    slat (usec): min=29, max=1923, avg=97.15, stdev=73.51
    clat (usec): min=15820, max=64908, avg=41276.66, stdev=6836.80
     lat (usec): min=15896, max=65002, avg=41374.92, stdev=6838.54
    clat percentiles (usec):
     |  1.00th=[28443],  5.00th=[31327], 10.00th=[32637], 20.00th=[34866],
     | 30.00th=[36439], 40.00th=[38536], 50.00th=[41157], 60.00th=[43254],
     | 70.00th=[45351], 80.00th=[47449], 90.00th=[50070], 95.00th=[52167],
     | 99.00th=[56886], 99.50th=[57934], 99.90th=[64750], 99.95th=[64750],
     | 99.99th=[64750]
   bw (  KiB/s): min=  616, max= 1032, per=24.61%, avg=747.62, stdev=119.33, samples=13
   iops        : min=  154, max=  258, avg=186.85, stdev=29.82, samples=13
  lat (msec)   : 10=0.08%, 20=0.23%, 50=89.14%, 100=10.55%
  cpu          : usr=0.44%, sys=3.84%, ctx=4809, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182236: Fri Sep 18 02:02:03 2026
  read: IOPS=181, BW=724KiB/s (742kB/s)(4976KiB/6869msec)
    slat (usec): min=1080, max=7268, avg=2966.52, stdev=1006.38
    clat (usec): min=8129, max=65184, avg=40899.30, stdev=7097.61
     lat (usec): min=10639, max=67437, avg=43867.44, stdev=7396.28
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[29492], 10.00th=[31851], 20.00th=[34341],
     | 30.00th=[36963], 40.00th=[39060], 50.00th=[40633], 60.00th=[42730],
     | 70.00th=[44827], 80.00th=[47449], 90.00th=[50070], 95.00th=[52691],
     | 99.00th=[57934], 99.50th=[60031], 99.90th=[63701], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  592, max=  880, per=24.64%, avg=718.92, stdev=87.90, samples=13
   iops        : min=  148, max=  220, avg=179.69, stdev=21.97, samples=13
  write: IOPS=191, BW=766KiB/s (785kB/s)(5264KiB/6869msec); 0 zone resets
    slat (usec): min=37, max=496, avg=95.26, stdev=55.39
    clat (usec): min=14744, max=60903, avg=41511.96, stdev=6763.47
     lat (usec): min=14883, max=60996, avg=41608.22, stdev=6766.13
    clat percentiles (usec):
     |  1.00th=[28181],  5.00th=[31589], 10.00th=[33162], 20.00th=[34866],
     | 30.00th=[36963], 40.00th=[39060], 50.00th=[41157], 60.00th=[43779],
     | 70.00th=[45876], 80.00th=[47973], 90.00th=[50070], 95.00th=[52167],
     | 99.00th=[55837], 99.50th=[57934], 99.90th=[60556], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min=  640, max=  944, per=25.05%, avg=761.00, stdev=101.24, samples=13
   iops        : min=  160, max=  236, avg=190.23, stdev=25.29, samples=13
  lat (msec)   : 10=0.04%, 20=0.08%, 50=89.34%, 100=10.55%
  cpu          : usr=0.41%, sys=3.87%, ctx=4780, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182237: Fri Sep 18 02:02:03 2026
  read: IOPS=181, BW=728KiB/s (745kB/s)(4984KiB/6850msec)
    slat (usec): min=23, max=7831, avg=2969.84, stdev=1024.40
    clat (usec): min=4887, max=71163, avg=41058.84, stdev=7664.99
     lat (usec): min=4924, max=75575, avg=44030.09, stdev=8014.13
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[29492], 10.00th=[32113], 20.00th=[34866],
     | 30.00th=[36963], 40.00th=[39060], 50.00th=[40633], 60.00th=[43254],
     | 70.00th=[44827], 80.00th=[47449], 90.00th=[50594], 95.00th=[52691],
     | 99.00th=[58459], 99.50th=[62129], 99.90th=[67634], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min=  600, max=  912, per=24.86%, avg=725.54, stdev=109.08, samples=13
   iops        : min=  150, max=  228, avg=181.38, stdev=27.27, samples=13
  write: IOPS=191, BW=767KiB/s (786kB/s)(5256KiB/6850msec); 0 zone resets
    slat (usec): min=25, max=3325, avg=99.31, stdev=106.80
    clat (usec): min=20876, max=67643, avg=41436.30, stdev=6657.58
     lat (usec): min=21100, max=67684, avg=41536.78, stdev=6657.48
    clat percentiles (usec):
     |  1.00th=[28181],  5.00th=[31065], 10.00th=[32900], 20.00th=[34866],
     | 30.00th=[37487], 40.00th=[39584], 50.00th=[41681], 60.00th=[43254],
     | 70.00th=[45351], 80.00th=[47449], 90.00th=[49546], 95.00th=[51643],
     | 99.00th=[55837], 99.50th=[58459], 99.90th=[67634], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  608, max=  904, per=25.01%, avg=759.77, stdev=87.30, samples=13
   iops        : min=  152, max=  226, avg=189.92, stdev=21.80, samples=13
  lat (msec)   : 10=0.12%, 20=0.27%, 50=89.34%, 100=10.27%
  cpu          : usr=0.35%, sys=3.90%, ctx=4747, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182238: Fri Sep 18 02:02:03 2026
  read: IOPS=184, BW=737KiB/s (754kB/s)(5060KiB/6869msec)
    slat (usec): min=1090, max=7736, avg=3018.44, stdev=997.24
    clat (usec): min=19345, max=67777, avg=40987.65, stdev=7126.42
     lat (usec): min=22482, max=71405, avg=44007.72, stdev=7445.51
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[30278], 10.00th=[32113], 20.00th=[34866],
     | 30.00th=[36439], 40.00th=[38536], 50.00th=[40633], 60.00th=[42730],
     | 70.00th=[44827], 80.00th=[47449], 90.00th=[50070], 95.00th=[52691],
     | 99.00th=[59507], 99.50th=[61604], 99.90th=[64750], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  632, max=  864, per=25.01%, avg=729.77, stdev=82.03, samples=13
   iops        : min=  158, max=  216, avg=182.38, stdev=20.44, samples=13
  write: IOPS=188, BW=754KiB/s (772kB/s)(5180KiB/6869msec); 0 zone resets
    slat (usec): min=38, max=696, avg=97.61, stdev=59.60
    clat (usec): min=24862, max=59528, avg=41377.24, stdev=6568.96
     lat (usec): min=24991, max=59638, avg=41476.05, stdev=6570.76
    clat percentiles (usec):
     |  1.00th=[28443],  5.00th=[31065], 10.00th=[32900], 20.00th=[34866],
     | 30.00th=[36963], 40.00th=[39060], 50.00th=[41157], 60.00th=[43254],
     | 70.00th=[45876], 80.00th=[47449], 90.00th=[50070], 95.00th=[51643],
     | 99.00th=[55313], 99.50th=[56361], 99.90th=[58983], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min=  584, max=  952, per=24.62%, avg=748.00, stdev=105.40, samples=13
   iops        : min=  146, max=  238, avg=187.00, stdev=26.35, samples=13
  lat (msec)   : 20=0.04%, 50=89.61%, 100=10.35%
  cpu          : usr=0.55%, sys=3.77%, ctx=4821, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2919KiB/s (2989kB/s), 724KiB/s-737KiB/s (742kB/s-754kB/s), io=19.6MiB (20.5MB), run=6850-6876msec
  WRITE: bw=3038KiB/s (3111kB/s), 754KiB/s-767KiB/s (772kB/s-786kB/s), io=20.4MiB (21.4MB), run=6850-6876msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=182242: Fri Sep 18 02:02:06 2026
  read: IOPS=9, BW=388MiB/s (407MB/s)(40.0MiB/103msec)
    slat (nsec): min=10721k, max=10721k, avg=10721390.00, stdev= 0.00
    clat (nsec): min=86932k, max=86932k, avg=86932153.00, stdev= 0.00
     lat (nsec): min=97658k, max=97658k, avg=97658059.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[86508],  5.00th=[86508], 10.00th=[86508], 20.00th=[86508],
     | 30.00th=[86508], 40.00th=[86508], 50.00th=[86508], 60.00th=[86508],
     | 70.00th=[86508], 80.00th=[86508], 90.00th=[86508], 95.00th=[86508],
     | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508],
     | 99.99th=[86508]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.82%, ctx=8, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=388MiB/s (407MB/s), 388MiB/s-388MiB/s (407MB/s-407MB/s), io=40.0MiB (41.9MB), run=103-103msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout