== sanity test 398c: run fio to test AIO ================= 07:21:37 (1788002497)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.130369 s, 322 MB/s
osc.lustre-OST0000-osc-ffff9ef50086a000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=182301: Sat Aug 29 07:21:46 2026
  write: IOPS=428, BW=1715KiB/s (1756kB/s)(10.0MiB/5972msec); 0 zone resets
    slat (usec): min=22, max=2224, avg=114.76, stdev=87.30
    clat (usec): min=6737, max=74842, avg=37150.49, stdev=6290.68
     lat (usec): min=6816, max=75119, avg=37266.41, stdev=6299.78
    clat percentiles (usec):
     |  1.00th=[21365],  5.00th=[27395], 10.00th=[29754], 20.00th=[32113],
     | 30.00th=[34341], 40.00th=[35390], 50.00th=[36963], 60.00th=[38536],
     | 70.00th=[40109], 80.00th=[42206], 90.00th=[45351], 95.00th=[46924],
     | 99.00th=[49546], 99.50th=[52167], 99.90th=[74974], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min= 1568, max= 1993, per=25.51%, avg=1729.55, stdev=144.35, samples=11
   iops        : min=  392, max=  498, avg=432.36, stdev=36.04, samples=11
  lat (msec)   : 10=0.12%, 20=0.66%, 50=98.40%, 100=0.82%
  cpu          : usr=0.55%, sys=4.14%, ctx=2502, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182302: Sat Aug 29 07:21:46 2026
  write: IOPS=423, BW=1695KiB/s (1736kB/s)(10.0MiB/6040msec); 0 zone resets
    slat (usec): min=20, max=893, avg=99.88, stdev=74.37
    clat (usec): min=8547, max=78445, avg=37277.83, stdev=6126.00
     lat (usec): min=8609, max=78618, avg=37378.92, stdev=6139.38
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[27657], 10.00th=[29754], 20.00th=[32113],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[37487], 60.00th=[38536],
     | 70.00th=[40109], 80.00th=[42206], 90.00th=[45351], 95.00th=[46924],
     | 99.00th=[49546], 99.50th=[51119], 99.90th=[73925], 99.95th=[73925],
     | 99.99th=[78119]
   bw (  KiB/s): min= 1552, max= 1952, per=25.25%, avg=1712.27, stdev=127.98, samples=11
   iops        : min=  388, max=  488, avg=428.00, stdev=31.94, samples=11
  lat (msec)   : 10=0.08%, 50=99.10%, 100=0.82%
  cpu          : usr=0.51%, sys=3.56%, ctx=2274, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182303: Sat Aug 29 07:21:46 2026
  write: IOPS=424, BW=1697KiB/s (1738kB/s)(10.0MiB/6034msec); 0 zone resets
    slat (usec): min=21, max=763, avg=108.75, stdev=73.84
    clat (usec): min=17096, max=83647, avg=37255.40, stdev=6283.53
     lat (usec): min=17258, max=83697, avg=37365.32, stdev=6293.75
    clat percentiles (usec):
     |  1.00th=[21103],  5.00th=[27395], 10.00th=[29754], 20.00th=[32375],
     | 30.00th=[34341], 40.00th=[35390], 50.00th=[36963], 60.00th=[38536],
     | 70.00th=[40109], 80.00th=[42206], 90.00th=[45351], 95.00th=[46924],
     | 99.00th=[49546], 99.50th=[52691], 99.90th=[76022], 99.95th=[76022],
     | 99.99th=[83362]
   bw (  KiB/s): min= 1584, max= 1944, per=25.30%, avg=1715.27, stdev=120.81, samples=11
   iops        : min=  396, max=  486, avg=428.82, stdev=30.20, samples=11
  lat (msec)   : 20=0.12%, 50=99.14%, 100=0.74%
  cpu          : usr=0.56%, sys=4.29%, ctx=2607, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182304: Sat Aug 29 07:21:46 2026
  write: IOPS=425, BW=1701KiB/s (1742kB/s)(10.0MiB/6020msec); 0 zone resets
    slat (usec): min=23, max=29643, avg=120.91, stdev=590.22
    clat (usec): min=3614, max=86125, avg=37200.80, stdev=6448.88
     lat (usec): min=3671, max=86175, avg=37322.87, stdev=6436.83
    clat percentiles (usec):
     |  1.00th=[21890],  5.00th=[27657], 10.00th=[29754], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[35914], 50.00th=[36963], 60.00th=[38536],
     | 70.00th=[40109], 80.00th=[42206], 90.00th=[45351], 95.00th=[46924],
     | 99.00th=[49546], 99.50th=[53216], 99.90th=[82314], 99.95th=[82314],
     | 99.99th=[86508]
   bw (  KiB/s): min= 1568, max= 1944, per=25.38%, avg=1721.00, stdev=128.05, samples=11
   iops        : min=  392, max=  486, avg=430.36, stdev=31.77, samples=11
  lat (msec)   : 4=0.04%, 20=0.74%, 50=98.36%, 100=0.86%
  cpu          : usr=0.60%, sys=4.25%, ctx=2543, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6781KiB/s (6944kB/s), 1695KiB/s-1715KiB/s (1736kB/s-1756kB/s), io=40.0MiB (41.9MB), run=5972-6040msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=182314: Sat Aug 29 07:21:56 2026
  read: IOPS=203, BW=814KiB/s (834kB/s)(5048KiB/6198msec)
    slat (usec): min=43, max=6906, avg=2514.98, stdev=873.67
    clat (usec): min=4921, max=63067, avg=37182.78, stdev=7141.04
     lat (usec): min=6675, max=66172, avg=39699.63, stdev=7450.93
    clat percentiles (usec):
     |  1.00th=[18482],  5.00th=[26084], 10.00th=[28967], 20.00th=[31589],
     | 30.00th=[33424], 40.00th=[35390], 50.00th=[36963], 60.00th=[39060],
     | 70.00th=[40633], 80.00th=[43254], 90.00th=[46400], 95.00th=[49021],
     | 99.00th=[53740], 99.50th=[55837], 99.90th=[60556], 99.95th=[63177],
     | 99.99th=[63177]
   bw (  KiB/s): min=  678, max=  920, per=25.08%, avg=811.83, stdev=84.02, samples=12
   iops        : min=  169, max=  230, avg=202.92, stdev=21.08, samples=12
  write: IOPS=209, BW=838KiB/s (858kB/s)(5192KiB/6198msec); 0 zone resets
    slat (usec): min=30, max=2331, avg=96.96, stdev=96.26
    clat (usec): min=16997, max=57997, avg=37377.01, stdev=6213.89
     lat (usec): min=17095, max=58064, avg=37474.95, stdev=6214.97
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[28181], 10.00th=[29754], 20.00th=[31589],
     | 30.00th=[33424], 40.00th=[35390], 50.00th=[36963], 60.00th=[38536],
     | 70.00th=[40633], 80.00th=[42730], 90.00th=[45876], 95.00th=[47973],
     | 99.00th=[53216], 99.50th=[54789], 99.90th=[56886], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min=  664, max= 1072, per=24.88%, avg=838.50, stdev=110.37, samples=12
   iops        : min=  166, max=  268, avg=209.58, stdev=27.65, samples=12
  lat (msec)   : 10=0.16%, 20=0.59%, 50=96.13%, 100=3.12%
  cpu          : usr=0.53%, sys=4.31%, ctx=4693, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182315: Sat Aug 29 07:21:56 2026
  read: IOPS=200, BW=803KiB/s (823kB/s)(4976KiB/6194msec)
    slat (usec): min=65, max=6909, avg=2528.13, stdev=810.97
    clat (usec): min=4852, max=60352, avg=37169.46, stdev=7103.99
     lat (usec): min=6924, max=63805, avg=39699.45, stdev=7385.66
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[26870], 10.00th=[28705], 20.00th=[31065],
     | 30.00th=[32900], 40.00th=[34866], 50.00th=[36439], 60.00th=[38536],
     | 70.00th=[40633], 80.00th=[43254], 90.00th=[46400], 95.00th=[49021],
     | 99.00th=[55313], 99.50th=[57934], 99.90th=[60031], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  652, max= 1024, per=24.75%, avg=801.00, stdev=110.33, samples=12
   iops        : min=  163, max=  256, avg=200.25, stdev=27.58, samples=12
  write: IOPS=212, BW=850KiB/s (870kB/s)(5264KiB/6194msec); 0 zone resets
    slat (usec): min=33, max=2569, avg=95.25, stdev=92.20
    clat (usec): min=20209, max=54238, avg=37361.55, stdev=6122.05
     lat (usec): min=20299, max=54304, avg=37457.77, stdev=6125.31
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[28181], 10.00th=[29754], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[36439], 60.00th=[38536],
     | 70.00th=[40633], 80.00th=[42730], 90.00th=[46400], 95.00th=[49021],
     | 99.00th=[51643], 99.50th=[53740], 99.90th=[53740], 99.95th=[54264],
     | 99.99th=[54264]
   bw (  KiB/s): min=  704, max= 1088, per=25.32%, avg=853.25, stdev=119.83, samples=12
   iops        : min=  176, max=  272, avg=213.25, stdev=30.02, samples=12
  lat (msec)   : 10=0.08%, 20=0.20%, 50=96.52%, 100=3.20%
  cpu          : usr=0.45%, sys=4.31%, ctx=4696, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182316: Sat Aug 29 07:21:56 2026
  read: IOPS=201, BW=806KiB/s (826kB/s)(4984KiB/6181msec)
    slat (usec): min=31, max=6701, avg=2520.14, stdev=828.11
    clat (usec): min=16593, max=57607, avg=37394.32, stdev=6636.31
     lat (usec): min=16631, max=60947, avg=39916.43, stdev=6901.67
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[27132], 10.00th=[29230], 20.00th=[31851],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[36963], 60.00th=[38536],
     | 70.00th=[40633], 80.00th=[43254], 90.00th=[45876], 95.00th=[49546],
     | 99.00th=[53216], 99.50th=[54789], 99.90th=[57410], 99.95th=[57410],
     | 99.99th=[57410]
   bw (  KiB/s): min=  711, max=  992, per=24.96%, avg=807.92, stdev=86.06, samples=12
   iops        : min=  177, max=  248, avg=201.92, stdev=21.59, samples=12
  write: IOPS=212, BW=850KiB/s (871kB/s)(5256KiB/6181msec); 0 zone resets
    slat (usec): min=30, max=1977, avg=95.37, stdev=80.17
    clat (usec): min=12172, max=54841, avg=37169.21, stdev=6475.62
     lat (usec): min=12387, max=54917, avg=37265.45, stdev=6481.54
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[28181], 10.00th=[29492], 20.00th=[31589],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[36439], 60.00th=[38536],
     | 70.00th=[40633], 80.00th=[42730], 90.00th=[45876], 95.00th=[47973],
     | 99.00th=[52167], 99.50th=[53740], 99.90th=[54264], 99.95th=[54789],
     | 99.99th=[54789]
   bw (  KiB/s): min=  680, max= 1048, per=25.18%, avg=848.50, stdev=132.80, samples=12
   iops        : min=  170, max=  262, avg=212.08, stdev=33.23, samples=12
  lat (msec)   : 20=0.66%, 50=95.62%, 100=3.71%
  cpu          : usr=0.52%, sys=4.29%, ctx=4662, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182317: Sat Aug 29 07:21:56 2026
  read: IOPS=204, BW=816KiB/s (836kB/s)(5060KiB/6198msec)
    slat (usec): min=943, max=6803, avg=2506.76, stdev=796.14
    clat (usec): min=6808, max=64341, avg=37168.87, stdev=6810.01
     lat (usec): min=8780, max=66357, avg=39677.48, stdev=7070.96
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[27132], 10.00th=[29230], 20.00th=[31589],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[36439], 60.00th=[38536],
     | 70.00th=[40633], 80.00th=[42730], 90.00th=[45876], 95.00th=[49021],
     | 99.00th=[54789], 99.50th=[56361], 99.90th=[63177], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  626, max=  952, per=25.15%, avg=814.17, stdev=104.65, samples=12
   iops        : min=  156, max=  238, avg=203.50, stdev=26.25, samples=12
  write: IOPS=208, BW=836KiB/s (856kB/s)(5180KiB/6198msec); 0 zone resets
    slat (usec): min=29, max=727, avg=92.64, stdev=59.15
    clat (usec): min=12758, max=57097, avg=37267.49, stdev=6105.84
     lat (usec): min=12819, max=57167, avg=37361.09, stdev=6109.67
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[28705], 10.00th=[30278], 20.00th=[31851],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[36963], 60.00th=[38536],
     | 70.00th=[40109], 80.00th=[42730], 90.00th=[45351], 95.00th=[48497],
     | 99.00th=[52691], 99.50th=[54264], 99.90th=[55837], 99.95th=[56886],
     | 99.99th=[56886]
   bw (  KiB/s): min=  624, max= 1064, per=24.72%, avg=833.00, stdev=122.19, samples=12
   iops        : min=  156, max=  266, avg=208.25, stdev=30.55, samples=12
  lat (msec)   : 10=0.04%, 20=0.35%, 50=96.48%, 100=3.12%
  cpu          : usr=0.55%, sys=4.29%, ctx=4699, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3238KiB/s (3316kB/s), 803KiB/s-816KiB/s (823kB/s-836kB/s), io=19.6MiB (20.5MB), run=6181-6198msec
  WRITE: bw=3371KiB/s (3452kB/s), 836KiB/s-850KiB/s (856kB/s-871kB/s), io=20.4MiB (21.4MB), run=6181-6198msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=182322: Sat Aug 29 07:21:59 2026
  read: IOPS=7, BW=315MiB/s (330MB/s)(40.0MiB/127msec)
    slat (nsec): min=8861.4k, max=8861.4k, avg=8861401.00, stdev= 0.00
    clat (nsec): min=112556k, max=112556k, avg=112556107.00, stdev= 0.00
     lat (nsec): min=121422k, max=121422k, avg=121421678.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  113],  5.00th=[  113], 10.00th=[  113], 20.00th=[  113],
     | 30.00th=[  113], 40.00th=[  113], 50.00th=[  113], 60.00th=[  113],
     | 70.00th=[  113], 80.00th=[  113], 90.00th=[  113], 95.00th=[  113],
     | 99.00th=[  113], 99.50th=[  113], 99.90th=[  113], 99.95th=[  113],
     | 99.99th=[  113]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=6.35%, ctx=12, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=315MiB/s (330MB/s), 315MiB/s-315MiB/s (330MB/s-330MB/s), io=40.0MiB (41.9MB), run=127-127msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout