== sanity test 398c: run fio to test AIO ================= 21:36:28 (1788917788)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.133082 s, 315 MB/s
osc.lustre-OST0000-osc-ffff8b554564e000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=183960: Tue Sep  8 21:36:37 2026
  write: IOPS=472, BW=1889KiB/s (1934kB/s)(10.0MiB/5422msec); 0 zone resets
    slat (usec): min=27, max=25879, avg=110.85, stdev=513.01
    clat (usec): min=4531, max=63630, avg=33530.18, stdev=4632.52
     lat (usec): min=4618, max=63744, avg=33642.44, stdev=4605.77
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[27395], 10.00th=[28705], 20.00th=[30278],
     | 30.00th=[31327], 40.00th=[32113], 50.00th=[32900], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[39060], 95.00th=[41681],
     | 99.00th=[48497], 99.50th=[53216], 99.90th=[58983], 99.95th=[63701],
     | 99.99th=[63701]
   bw (  KiB/s): min= 1648, max= 2104, per=25.16%, avg=1894.60, stdev=117.89, samples=10
   iops        : min=  412, max=  526, avg=473.60, stdev=29.48, samples=10
  lat (msec)   : 10=0.20%, 20=0.08%, 50=98.91%, 100=0.82%
  cpu          : usr=0.46%, sys=4.45%, ctx=2735, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183961: Tue Sep  8 21:36:37 2026
  write: IOPS=474, BW=1897KiB/s (1942kB/s)(10.0MiB/5399msec); 0 zone resets
    slat (usec): min=21, max=682, avg=98.46, stdev=57.40
    clat (usec): min=4704, max=77188, avg=33591.05, stdev=4709.12
     lat (usec): min=4785, max=77257, avg=33690.49, stdev=4713.29
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[27657], 10.00th=[28967], 20.00th=[30278],
     | 30.00th=[31065], 40.00th=[32113], 50.00th=[33162], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[39060], 95.00th=[41157],
     | 99.00th=[48497], 99.50th=[53216], 99.90th=[69731], 99.95th=[73925],
     | 99.99th=[77071]
   bw (  KiB/s): min= 1640, max= 2112, per=25.26%, avg=1902.30, stdev=119.97, samples=10
   iops        : min=  410, max=  528, avg=475.50, stdev=29.96, samples=10
  lat (msec)   : 10=0.04%, 20=0.12%, 50=98.98%, 100=0.86%
  cpu          : usr=0.76%, sys=3.67%, ctx=2626, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183962: Tue Sep  8 21:36:37 2026
  write: IOPS=470, BW=1883KiB/s (1929kB/s)(10.0MiB/5437msec); 0 zone resets
    slat (usec): min=21, max=994, avg=96.73, stdev=57.87
    clat (usec): min=15015, max=58187, avg=33555.77, stdev=4318.24
     lat (usec): min=15101, max=58270, avg=33653.44, stdev=4324.43
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[27395], 10.00th=[28705], 20.00th=[30278],
     | 30.00th=[31327], 40.00th=[32113], 50.00th=[33162], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[39060], 95.00th=[41681],
     | 99.00th=[46924], 99.50th=[50070], 99.90th=[53740], 99.95th=[53740],
     | 99.99th=[57934]
   bw (  KiB/s): min= 1648, max= 2080, per=25.01%, avg=1883.60, stdev=116.24, samples=10
   iops        : min=  412, max=  520, avg=470.90, stdev=29.06, samples=10
  lat (msec)   : 20=0.04%, 50=99.49%, 100=0.47%
  cpu          : usr=0.48%, sys=3.81%, ctx=2656, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183963: Tue Sep  8 21:36:37 2026
  write: IOPS=470, BW=1883KiB/s (1928kB/s)(10.0MiB/5439msec); 0 zone resets
    slat (usec): min=21, max=3400, avg=99.30, stdev=88.24
    clat (usec): min=5346, max=61510, avg=33590.77, stdev=4501.95
     lat (usec): min=5419, max=61597, avg=33691.18, stdev=4506.22
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[27132], 10.00th=[28443], 20.00th=[30278],
     | 30.00th=[31327], 40.00th=[32375], 50.00th=[33162], 60.00th=[34341],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[39060], 95.00th=[41681],
     | 99.00th=[47449], 99.50th=[49546], 99.90th=[56886], 99.95th=[57934],
     | 99.99th=[61604]
   bw (  KiB/s): min= 1664, max= 2096, per=25.03%, avg=1884.70, stdev=112.86, samples=10
   iops        : min=  416, max=  524, avg=471.10, stdev=28.29, samples=10
  lat (msec)   : 10=0.08%, 20=0.04%, 50=99.41%, 100=0.47%
  cpu          : usr=0.57%, sys=4.23%, ctx=2735, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7531KiB/s (7712kB/s), 1883KiB/s-1897KiB/s (1928kB/s-1942kB/s), io=40.0MiB (41.9MB), run=5399-5439msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=183975: Tue Sep  8 21:36:46 2026
  read: IOPS=193, BW=772KiB/s (791kB/s)(5048KiB/6535msec)
    slat (usec): min=1032, max=7787, avg=2666.88, stdev=976.18
    clat (usec): min=11665, max=70718, avg=39023.92, stdev=7995.96
     lat (usec): min=13049, max=75374, avg=41693.19, stdev=8329.66
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[28443], 10.00th=[30278], 20.00th=[32900],
     | 30.00th=[34866], 40.00th=[36439], 50.00th=[38011], 60.00th=[39584],
     | 70.00th=[41681], 80.00th=[44303], 90.00th=[49546], 95.00th=[54789],
     | 99.00th=[64750], 99.50th=[67634], 99.90th=[69731], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min=  592, max=  856, per=24.84%, avg=762.67, stdev=91.58, samples=12
   iops        : min=  148, max=  214, avg=190.58, stdev=23.01, samples=12
  write: IOPS=198, BW=794KiB/s (814kB/s)(5192KiB/6535msec); 0 zone resets
    slat (usec): min=25, max=673, avg=94.72, stdev=55.57
    clat (usec): min=9685, max=72226, avg=39268.32, stdev=7667.89
     lat (usec): min=9757, max=72374, avg=39364.12, stdev=7670.87
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[30540], 10.00th=[31589], 20.00th=[33424],
     | 30.00th=[34866], 40.00th=[35914], 50.00th=[37487], 60.00th=[39584],
     | 70.00th=[41681], 80.00th=[43779], 90.00th=[48497], 95.00th=[55313],
     | 99.00th=[65274], 99.50th=[67634], 99.90th=[69731], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  638, max= 1064, per=24.84%, avg=793.83, stdev=115.38, samples=12
   iops        : min=  159, max=  266, avg=198.33, stdev=28.96, samples=12
  lat (msec)   : 10=0.04%, 20=0.31%, 50=91.05%, 100=8.59%
  cpu          : usr=0.52%, sys=4.15%, ctx=4766, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183976: Tue Sep  8 21:36:46 2026
  read: IOPS=191, BW=767KiB/s (785kB/s)(4976KiB/6490msec)
    slat (usec): min=28, max=9874, avg=2614.36, stdev=967.30
    clat (usec): min=22407, max=76503, avg=38935.53, stdev=7941.50
     lat (usec): min=24649, max=78610, avg=41552.22, stdev=8211.86
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[28181], 10.00th=[30016], 20.00th=[32637],
     | 30.00th=[34341], 40.00th=[36439], 50.00th=[38011], 60.00th=[39584],
     | 70.00th=[41681], 80.00th=[43779], 90.00th=[48497], 95.00th=[53740],
     | 99.00th=[65799], 99.50th=[69731], 99.90th=[73925], 99.95th=[76022],
     | 99.99th=[76022]
   bw (  KiB/s): min=  552, max=  904, per=24.81%, avg=761.58, stdev=95.12, samples=12
   iops        : min=  138, max=  226, avg=190.25, stdev=23.83, samples=12
  write: IOPS=202, BW=811KiB/s (831kB/s)(5264KiB/6490msec); 0 zone resets
    slat (usec): min=29, max=3248, avg=97.40, stdev=121.49
    clat (usec): min=11304, max=73755, avg=39383.74, stdev=8127.55
     lat (usec): min=11384, max=73829, avg=39482.33, stdev=8137.27
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[29754], 10.00th=[31327], 20.00th=[33424],
     | 30.00th=[34866], 40.00th=[36439], 50.00th=[37487], 60.00th=[39060],
     | 70.00th=[41157], 80.00th=[44303], 90.00th=[51119], 95.00th=[55837],
     | 99.00th=[65274], 99.50th=[69731], 99.90th=[72877], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min=  630, max=  969, per=25.46%, avg=813.75, stdev=110.88, samples=12
   iops        : min=  157, max=  242, avg=203.33, stdev=27.82, samples=12
  lat (msec)   : 20=0.23%, 50=90.35%, 100=9.41%
  cpu          : usr=0.32%, sys=4.28%, ctx=4569, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183977: Tue Sep  8 21:36:46 2026
  read: IOPS=190, BW=764KiB/s (782kB/s)(4984KiB/6524msec)
    slat (usec): min=1006, max=8967, avg=2646.33, stdev=996.60
    clat (usec): min=19822, max=71467, avg=38889.40, stdev=7844.40
     lat (usec): min=23656, max=75377, avg=41538.13, stdev=8194.92
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[28443], 10.00th=[30540], 20.00th=[32637],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[38011], 60.00th=[39584],
     | 70.00th=[41157], 80.00th=[43779], 90.00th=[49021], 95.00th=[54789],
     | 99.00th=[63701], 99.50th=[67634], 99.90th=[70779], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  624, max=  952, per=24.99%, avg=767.25, stdev=115.01, samples=12
   iops        : min=  156, max=  238, avg=191.67, stdev=28.80, samples=12
  write: IOPS=201, BW=806KiB/s (825kB/s)(5256KiB/6524msec); 0 zone resets
    slat (usec): min=30, max=1300, avg=94.42, stdev=68.75
    clat (usec): min=20927, max=71146, avg=39455.73, stdev=7455.10
     lat (usec): min=21020, max=71227, avg=39551.16, stdev=7461.05
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[30278], 10.00th=[31851], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[36439], 50.00th=[37487], 60.00th=[39584],
     | 70.00th=[41681], 80.00th=[44303], 90.00th=[49546], 95.00th=[53740],
     | 99.00th=[63177], 99.50th=[65799], 99.90th=[69731], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min=  640, max=  952, per=25.01%, avg=799.25, stdev=102.96, samples=12
   iops        : min=  160, max=  238, avg=199.67, stdev=25.83, samples=12
  lat (msec)   : 20=0.04%, 50=90.70%, 100=9.26%
  cpu          : usr=0.46%, sys=4.17%, ctx=4679, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183978: Tue Sep  8 21:36:46 2026
  read: IOPS=194, BW=778KiB/s (796kB/s)(5060KiB/6508msec)
    slat (usec): min=1030, max=9042, avg=2621.58, stdev=925.27
    clat (usec): min=6902, max=75254, avg=38995.67, stdev=7799.49
     lat (usec): min=8810, max=78405, avg=41619.31, stdev=8040.12
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[28443], 10.00th=[30540], 20.00th=[33162],
     | 30.00th=[34866], 40.00th=[36439], 50.00th=[38011], 60.00th=[39584],
     | 70.00th=[41157], 80.00th=[43779], 90.00th=[49021], 95.00th=[54264],
     | 99.00th=[62129], 99.50th=[68682], 99.90th=[74974], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min=  624, max=  872, per=25.23%, avg=774.67, stdev=80.64, samples=12
   iops        : min=  156, max=  218, avg=193.58, stdev=20.27, samples=12
  write: IOPS=198, BW=796KiB/s (815kB/s)(5180KiB/6508msec); 0 zone resets
    slat (usec): min=32, max=942, avg=96.60, stdev=67.92
    clat (usec): min=24751, max=70913, avg=39209.65, stdev=7546.90
     lat (usec): min=24826, max=70970, avg=39307.30, stdev=7548.72
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[30278], 10.00th=[31851], 20.00th=[33424],
     | 30.00th=[34866], 40.00th=[35914], 50.00th=[37487], 60.00th=[39060],
     | 70.00th=[40633], 80.00th=[43779], 90.00th=[49021], 95.00th=[55837],
     | 99.00th=[63701], 99.50th=[65274], 99.90th=[70779], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min=  624, max=  944, per=24.90%, avg=795.67, stdev=108.76, samples=12
   iops        : min=  156, max=  236, avg=198.83, stdev=27.31, samples=12
  lat (msec)   : 10=0.12%, 50=91.09%, 100=8.79%
  cpu          : usr=0.52%, sys=4.13%, ctx=4678, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3071KiB/s (3145kB/s), 764KiB/s-778KiB/s (782kB/s-796kB/s), io=19.6MiB (20.5MB), run=6490-6535msec
  WRITE: bw=3197KiB/s (3274kB/s), 794KiB/s-811KiB/s (814kB/s-831kB/s), io=20.4MiB (21.4MB), run=6490-6535msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=183982: Tue Sep  8 21:36:50 2026
  read: IOPS=10, BW=421MiB/s (442MB/s)(40.0MiB/95msec)
    slat (nsec): min=8051.4k, max=8051.4k, avg=8051417.00, stdev= 0.00
    clat (nsec): min=82375k, max=82375k, avg=82374999.00, stdev= 0.00
     lat (nsec): min=90430k, max=90430k, avg=90429904.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[82314],  5.00th=[82314], 10.00th=[82314], 20.00th=[82314],
     | 30.00th=[82314], 40.00th=[82314], 50.00th=[82314], 60.00th=[82314],
     | 70.00th=[82314], 80.00th=[82314], 90.00th=[82314], 95.00th=[82314],
     | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314],
     | 99.99th=[82314]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.51%, ctx=8, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=421MiB/s (442MB/s), 421MiB/s-421MiB/s (442MB/s-442MB/s), io=40.0MiB (41.9MB), run=95-95msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout