== sanity test 398c: run fio to test AIO ================= 12:01:23 (1783353683)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.124758 s, 336 MB/s
osc.lustre-OST0000-osc-ffff922e87f65800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=182024: Mon Jul  6 12:01:31 2026
  write: IOPS=504, BW=2018KiB/s (2067kB/s)(10.0MiB/5074msec); 0 zone resets
    slat (usec): min=34, max=913, avg=166.22, stdev=90.54
    clat (usec): min=3878, max=49297, avg=31156.52, stdev=3240.69
     lat (usec): min=4129, max=49396, avg=31323.98, stdev=3243.91
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[26608], 10.00th=[27657], 20.00th=[28705],
     | 30.00th=[29492], 40.00th=[30278], 50.00th=[31065], 60.00th=[31589],
     | 70.00th=[32637], 80.00th=[33817], 90.00th=[35390], 95.00th=[36963],
     | 99.00th=[39060], 99.50th=[40109], 99.90th=[46400], 99.95th=[49021],
     | 99.99th=[49546]
   bw (  KiB/s): min= 1849, max= 2184, per=25.05%, avg=2022.33, stdev=116.97, samples=9
   iops        : min=  462, max=  546, avg=505.56, stdev=29.29, samples=9
  lat (msec)   : 4=0.04%, 20=0.20%, 50=99.77%
  cpu          : usr=0.47%, sys=6.51%, ctx=2498, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182025: Mon Jul  6 12:01:31 2026
  write: IOPS=506, BW=2026KiB/s (2075kB/s)(10.0MiB/5054msec); 0 zone resets
    slat (usec): min=36, max=24874, avg=187.74, stdev=496.56
    clat (usec): min=9325, max=51132, avg=31113.02, stdev=3299.78
     lat (usec): min=12242, max=51271, avg=31301.80, stdev=3276.40
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[26346], 10.00th=[27657], 20.00th=[28705],
     | 30.00th=[29492], 40.00th=[30278], 50.00th=[31065], 60.00th=[31589],
     | 70.00th=[32375], 80.00th=[33817], 90.00th=[35390], 95.00th=[36963],
     | 99.00th=[39060], 99.50th=[40109], 99.90th=[47449], 99.95th=[47449],
     | 99.99th=[51119]
   bw (  KiB/s): min= 1912, max= 2168, per=25.23%, avg=2036.56, stdev=91.75, samples=9
   iops        : min=  478, max=  542, avg=509.11, stdev=22.98, samples=9
  lat (msec)   : 10=0.04%, 20=0.59%, 50=99.34%, 100=0.04%
  cpu          : usr=0.46%, sys=7.50%, ctx=2734, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182026: Mon Jul  6 12:01:31 2026
  write: IOPS=509, BW=2038KiB/s (2087kB/s)(10.0MiB/5025msec); 0 zone resets
    slat (usec): min=38, max=931, avg=169.71, stdev=86.66
    clat (usec): min=4428, max=58072, avg=31168.56, stdev=3441.74
     lat (usec): min=4508, max=58247, avg=31339.26, stdev=3445.43
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[26084], 10.00th=[27657], 20.00th=[28705],
     | 30.00th=[29492], 40.00th=[30278], 50.00th=[31065], 60.00th=[31589],
     | 70.00th=[32375], 80.00th=[33817], 90.00th=[35390], 95.00th=[36963],
     | 99.00th=[40109], 99.50th=[40633], 99.90th=[51643], 99.95th=[55313],
     | 99.99th=[57934]
   bw (  KiB/s): min= 1920, max= 2176, per=25.33%, avg=2044.78, stdev=80.51, samples=9
   iops        : min=  480, max=  544, avg=511.11, stdev=20.18, samples=9
  lat (msec)   : 10=0.04%, 20=0.82%, 50=98.98%, 100=0.16%
  cpu          : usr=0.72%, sys=6.49%, ctx=2517, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182027: Mon Jul  6 12:01:31 2026
  write: IOPS=505, BW=2021KiB/s (2070kB/s)(10.0MiB/5066msec); 0 zone resets
    slat (usec): min=39, max=1035, avg=172.99, stdev=90.08
    clat (usec): min=5266, max=53556, avg=31146.68, stdev=3344.62
     lat (usec): min=5408, max=53703, avg=31320.65, stdev=3346.51
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[26346], 10.00th=[27657], 20.00th=[28705],
     | 30.00th=[29492], 40.00th=[30278], 50.00th=[31065], 60.00th=[31589],
     | 70.00th=[32375], 80.00th=[33817], 90.00th=[35390], 95.00th=[36963],
     | 99.00th=[39584], 99.50th=[41681], 99.90th=[53216], 99.95th=[53216],
     | 99.99th=[53740]
   bw (  KiB/s): min= 1874, max= 2176, per=25.18%, avg=2032.22, stdev=100.88, samples=9
   iops        : min=  468, max=  544, avg=508.00, stdev=25.32, samples=9
  lat (msec)   : 10=0.04%, 20=0.31%, 50=99.45%, 100=0.20%
  cpu          : usr=0.32%, sys=7.32%, ctx=2739, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8073KiB/s (8266kB/s), 2018KiB/s-2038KiB/s (2067kB/s-2087kB/s), io=40.0MiB (41.9MB), run=5025-5074msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=182037: Mon Jul  6 12:01:40 2026
  read: IOPS=203, BW=815KiB/s (835kB/s)(5048KiB/6191msec)
    slat (usec): min=192, max=7565, avg=2611.68, stdev=886.45
    clat (usec): min=22505, max=57710, avg=36946.30, stdev=6555.73
     lat (usec): min=24253, max=60319, avg=39559.60, stdev=6827.18
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[27132], 10.00th=[28967], 20.00th=[31327],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[36439], 60.00th=[38011],
     | 70.00th=[40109], 80.00th=[42730], 90.00th=[46924], 95.00th=[48497],
     | 99.00th=[52167], 99.50th=[53740], 99.90th=[56886], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min=  648, max=  904, per=25.08%, avg=812.33, stdev=86.15, samples=12
   iops        : min=  162, max=  226, avg=203.08, stdev=21.54, samples=12
  write: IOPS=209, BW=839KiB/s (859kB/s)(5192KiB/6191msec); 0 zone resets
    slat (usec): min=53, max=2369, avg=154.48, stdev=94.79
    clat (usec): min=8071, max=64610, avg=37258.70, stdev=5976.25
     lat (usec): min=8261, max=64762, avg=37414.47, stdev=5973.97
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[29230], 10.00th=[30278], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[36439], 60.00th=[38011],
     | 70.00th=[40109], 80.00th=[42206], 90.00th=[44827], 95.00th=[47973],
     | 99.00th=[53740], 99.50th=[57410], 99.90th=[63701], 99.95th=[64750],
     | 99.99th=[64750]
   bw (  KiB/s): min=  664, max= 1056, per=24.89%, avg=839.33, stdev=95.85, samples=12
   iops        : min=  166, max=  264, avg=209.83, stdev=23.96, samples=12
  lat (msec)   : 10=0.04%, 50=97.23%, 100=2.73%
  cpu          : usr=0.40%, sys=5.80%, ctx=4818, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182038: Mon Jul  6 12:01:40 2026
  read: IOPS=201, BW=805KiB/s (824kB/s)(4976KiB/6184msec)
    slat (usec): min=43, max=7687, avg=2606.83, stdev=917.98
    clat (usec): min=21708, max=69093, avg=37119.73, stdev=6978.39
     lat (usec): min=23879, max=72171, avg=39728.14, stdev=7253.43
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[27395], 10.00th=[28967], 20.00th=[31065],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[35914], 60.00th=[38011],
     | 70.00th=[40633], 80.00th=[42730], 90.00th=[46400], 95.00th=[49021],
     | 99.00th=[55837], 99.50th=[60556], 99.90th=[67634], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  712, max=  936, per=24.81%, avg=803.50, stdev=68.87, samples=12
   iops        : min=  178, max=  234, avg=200.83, stdev=17.21, samples=12
  write: IOPS=212, BW=851KiB/s (872kB/s)(5264KiB/6184msec); 0 zone resets
    slat (usec): min=34, max=2072, avg=152.16, stdev=97.28
    clat (usec): min=10838, max=67012, avg=37346.66, stdev=6342.72
     lat (usec): min=10954, max=67112, avg=37499.85, stdev=6348.62
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[28705], 10.00th=[30278], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[36439], 60.00th=[38011],
     | 70.00th=[40109], 80.00th=[42206], 90.00th=[45876], 95.00th=[48497],
     | 99.00th=[53740], 99.50th=[56886], 99.90th=[62129], 99.95th=[66847],
     | 99.99th=[66847]
   bw (  KiB/s): min=  632, max= 1064, per=25.23%, avg=850.83, stdev=112.43, samples=12
   iops        : min=  158, max=  266, avg=212.67, stdev=28.06, samples=12
  lat (msec)   : 20=0.23%, 50=96.02%, 100=3.75%
  cpu          : usr=0.40%, sys=5.76%, ctx=4690, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182039: Mon Jul  6 12:01:40 2026
  read: IOPS=201, BW=805KiB/s (824kB/s)(4984KiB/6194msec)
    slat (usec): min=996, max=7656, avg=2609.09, stdev=896.99
    clat (usec): min=9542, max=63089, avg=36908.98, stdev=6610.11
     lat (usec): min=12516, max=66569, avg=39519.51, stdev=6900.45
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[27132], 10.00th=[28967], 20.00th=[31327],
     | 30.00th=[33162], 40.00th=[34866], 50.00th=[36439], 60.00th=[38011],
     | 70.00th=[40109], 80.00th=[42206], 90.00th=[45351], 95.00th=[49021],
     | 99.00th=[54789], 99.50th=[55837], 99.90th=[61604], 99.95th=[63177],
     | 99.99th=[63177]
   bw (  KiB/s): min=  656, max=  976, per=24.81%, avg=803.67, stdev=91.56, samples=12
   iops        : min=  164, max=  244, avg=200.92, stdev=22.89, samples=12
  write: IOPS=212, BW=849KiB/s (869kB/s)(5256KiB/6194msec); 0 zone resets
    slat (usec): min=53, max=505, avg=152.02, stdev=67.94
    clat (usec): min=21238, max=60398, avg=37357.14, stdev=5894.25
     lat (usec): min=21332, max=60533, avg=37510.24, stdev=5896.50
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28967], 10.00th=[30278], 20.00th=[32375],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[36439], 60.00th=[38011],
     | 70.00th=[40109], 80.00th=[42206], 90.00th=[45876], 95.00th=[47973],
     | 99.00th=[52167], 99.50th=[53216], 99.90th=[59507], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  712, max= 1064, per=25.07%, avg=845.50, stdev=104.32, samples=12
   iops        : min=  178, max=  266, avg=211.33, stdev=26.08, samples=12
  lat (msec)   : 10=0.04%, 20=0.04%, 50=96.37%, 100=3.55%
  cpu          : usr=0.39%, sys=5.80%, ctx=4796, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182040: Mon Jul  6 12:01:40 2026
  read: IOPS=204, BW=817KiB/s (837kB/s)(5060KiB/6194msec)
    slat (usec): min=995, max=7674, avg=2633.32, stdev=896.69
    clat (usec): min=5678, max=61858, avg=36753.73, stdev=6785.80
     lat (usec): min=8706, max=65225, avg=39388.66, stdev=7091.19
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[26870], 10.00th=[28705], 20.00th=[30802],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[42206], 90.00th=[46400], 95.00th=[48497],
     | 99.00th=[55313], 99.50th=[57934], 99.90th=[61080], 99.95th=[61604],
     | 99.99th=[61604]
   bw (  KiB/s): min=  664, max=  984, per=25.16%, avg=814.83, stdev=91.95, samples=12
   iops        : min=  166, max=  246, avg=203.67, stdev=22.99, samples=12
  write: IOPS=209, BW=836KiB/s (856kB/s)(5180KiB/6194msec); 0 zone resets
    slat (usec): min=45, max=2148, avg=155.39, stdev=92.48
    clat (usec): min=18452, max=59412, avg=37375.94, stdev=6260.69
     lat (usec): min=18566, max=59570, avg=37532.40, stdev=6259.05
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[28967], 10.00th=[30540], 20.00th=[32113],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[40109], 80.00th=[42730], 90.00th=[46400], 95.00th=[49546],
     | 99.00th=[54264], 99.50th=[55313], 99.90th=[58983], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min=  664, max=  992, per=24.69%, avg=832.42, stdev=95.61, samples=12
   iops        : min=  166, max=  248, avg=208.08, stdev=23.90, samples=12
  lat (msec)   : 10=0.04%, 20=0.08%, 50=95.82%, 100=4.06%
  cpu          : usr=0.45%, sys=5.70%, ctx=4756, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3240KiB/s (3318kB/s), 805KiB/s-817KiB/s (824kB/s-837kB/s), io=19.6MiB (20.5MB), run=6184-6194msec
  WRITE: bw=3373KiB/s (3454kB/s), 836KiB/s-851KiB/s (856kB/s-872kB/s), io=20.4MiB (21.4MB), run=6184-6194msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=182045: Mon Jul  6 12:01:44 2026
  read: IOPS=10, BW=400MiB/s (419MB/s)(40.0MiB/100msec)
    slat (nsec): min=10157k, max=10157k, avg=10156717.00, stdev= 0.00
    clat (nsec): min=82118k, max=82118k, avg=82117516.00, stdev= 0.00
     lat (nsec): min=92279k, max=92279k, avg=92278612.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[82314],  5.00th=[82314], 10.00th=[82314], 20.00th=[82314],
     | 30.00th=[82314], 40.00th=[82314], 50.00th=[82314], 60.00th=[82314],
     | 70.00th=[82314], 80.00th=[82314], 90.00th=[82314], 95.00th=[82314],
     | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314],
     | 99.99th=[82314]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=10.10%, ctx=4, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=400MiB/s (419MB/s), 400MiB/s-400MiB/s (419MB/s-419MB/s), io=40.0MiB (41.9MB), run=100-100msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout