== sanity test 398c: run fio to test AIO ================= 18:58:50 (1784415530)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.120915 s, 347 MB/s
osc.lustre-OST0000-osc-ffff9db7c2e87000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=190822: Sat Jul 18 18:58:57 2026
  write: IOPS=569, BW=2278KiB/s (2332kB/s)(10.0MiB/4496msec); 0 zone resets
    slat (usec): min=45, max=19976, avg=180.18, stdev=403.70
    clat (usec): min=5878, max=45969, avg=27638.27, stdev=3035.32
     lat (usec): min=5962, max=46140, avg=27819.50, stdev=3021.94
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[23725], 10.00th=[24773], 20.00th=[26084],
     | 30.00th=[26608], 40.00th=[27132], 50.00th=[27657], 60.00th=[28443],
     | 70.00th=[28967], 80.00th=[29492], 90.00th=[30540], 95.00th=[31851],
     | 99.00th=[34341], 99.50th=[34866], 99.90th=[40633], 99.95th=[45876],
     | 99.99th=[45876]
   bw (  KiB/s): min= 2152, max= 2384, per=24.98%, avg=2275.62, stdev=88.72, samples=8
   iops        : min=  538, max=  596, avg=568.88, stdev=22.18, samples=8
  lat (msec)   : 10=0.62%, 20=0.59%, 50=98.79%
  cpu          : usr=0.53%, sys=7.76%, ctx=2770, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190823: Sat Jul 18 18:58:57 2026
  write: IOPS=572, BW=2290KiB/s (2345kB/s)(10.0MiB/4472msec); 0 zone resets
    slat (usec): min=44, max=1597, avg=184.99, stdev=106.64
    clat (usec): min=6424, max=47505, avg=27712.39, stdev=3163.28
     lat (usec): min=6571, max=47772, avg=27898.55, stdev=3169.20
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[23725], 10.00th=[24773], 20.00th=[25822],
     | 30.00th=[26608], 40.00th=[27132], 50.00th=[27657], 60.00th=[28181],
     | 70.00th=[28967], 80.00th=[29492], 90.00th=[30802], 95.00th=[32375],
     | 99.00th=[35390], 99.50th=[38536], 99.90th=[44303], 99.95th=[44303],
     | 99.99th=[47449]
   bw (  KiB/s): min= 2176, max= 2414, per=25.13%, avg=2289.75, stdev=87.93, samples=8
   iops        : min=  544, max=  603, avg=572.38, stdev=21.88, samples=8
  lat (msec)   : 10=0.55%, 20=0.59%, 50=98.87%
  cpu          : usr=0.72%, sys=7.63%, ctx=2802, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190824: Sat Jul 18 18:58:57 2026
  write: IOPS=569, BW=2278KiB/s (2332kB/s)(10.0MiB/4496msec); 0 zone resets
    slat (usec): min=35, max=1437, avg=171.81, stdev=104.34
    clat (usec): min=6385, max=48094, avg=27798.67, stdev=2735.17
     lat (usec): min=6532, max=48196, avg=27971.48, stdev=2739.69
    clat percentiles (usec):
     |  1.00th=[20841],  5.00th=[23725], 10.00th=[24773], 20.00th=[26084],
     | 30.00th=[26608], 40.00th=[27132], 50.00th=[27919], 60.00th=[28443],
     | 70.00th=[28967], 80.00th=[29754], 90.00th=[30802], 95.00th=[32113],
     | 99.00th=[34341], 99.50th=[35390], 99.90th=[45876], 99.95th=[45876],
     | 99.99th=[47973]
   bw (  KiB/s): min= 2184, max= 2400, per=24.96%, avg=2273.62, stdev=83.10, samples=8
   iops        : min=  546, max=  600, avg=568.38, stdev=20.76, samples=8
  lat (msec)   : 10=0.04%, 20=0.59%, 50=99.38%
  cpu          : usr=0.51%, sys=7.27%, ctx=2732, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190825: Sat Jul 18 18:58:57 2026
  write: IOPS=570, BW=2281KiB/s (2336kB/s)(10.0MiB/4489msec); 0 zone resets
    slat (usec): min=41, max=705, avg=177.82, stdev=97.81
    clat (usec): min=10195, max=44818, avg=27770.25, stdev=2822.50
     lat (usec): min=10353, max=44984, avg=27949.11, stdev=2827.38
    clat percentiles (usec):
     |  1.00th=[20841],  5.00th=[23725], 10.00th=[24773], 20.00th=[26084],
     | 30.00th=[26870], 40.00th=[27132], 50.00th=[27919], 60.00th=[28443],
     | 70.00th=[28967], 80.00th=[29492], 90.00th=[30540], 95.00th=[31851],
     | 99.00th=[34866], 99.50th=[36439], 99.90th=[41157], 99.95th=[41681],
     | 99.99th=[44827]
   bw (  KiB/s): min= 2136, max= 2408, per=25.03%, avg=2280.62, stdev=96.78, samples=8
   iops        : min=  534, max=  602, avg=570.12, stdev=24.17, samples=8
  lat (msec)   : 20=0.78%, 50=99.22%
  cpu          : usr=0.60%, sys=7.51%, ctx=2729, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=9110KiB/s (9329kB/s), 2278KiB/s-2290KiB/s (2332kB/s-2345kB/s), io=40.0MiB (41.9MB), run=4472-4496msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=190835: Sat Jul 18 18:59:07 2026
  read: IOPS=189, BW=757KiB/s (775kB/s)(5048KiB/6669msec)
    slat (usec): min=33, max=10991, avg=2962.94, stdev=1062.42
    clat (usec): min=10739, max=64995, avg=39881.29, stdev=6731.67
     lat (usec): min=11062, max=67584, avg=42845.57, stdev=6989.58
    clat percentiles (usec):
     |  1.00th=[22152],  5.00th=[29230], 10.00th=[31327], 20.00th=[34341],
     | 30.00th=[36963], 40.00th=[39060], 50.00th=[40109], 60.00th=[41681],
     | 70.00th=[43254], 80.00th=[44827], 90.00th=[47973], 95.00th=[50070],
     | 99.00th=[56886], 99.50th=[58459], 99.90th=[62129], 99.95th=[64750],
     | 99.99th=[64750]
   bw (  KiB/s): min=  651, max= 1064, per=25.22%, avg=757.15, stdev=105.40, samples=13
   iops        : min=  162, max=  266, avg=189.23, stdev=26.41, samples=13
  write: IOPS=194, BW=779KiB/s (797kB/s)(5192KiB/6669msec); 0 zone resets
    slat (usec): min=53, max=2352, avg=153.33, stdev=97.90
    clat (usec): min=23076, max=61192, avg=40262.20, stdev=5633.71
     lat (usec): min=23187, max=61321, avg=40416.56, stdev=5637.16
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[30278], 10.00th=[33162], 20.00th=[35914],
     | 30.00th=[37487], 40.00th=[39060], 50.00th=[40633], 60.00th=[41681],
     | 70.00th=[43254], 80.00th=[44303], 90.00th=[46924], 95.00th=[49021],
     | 99.00th=[54789], 99.50th=[55837], 99.90th=[61080], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min=  680, max=  936, per=24.93%, avg=779.23, stdev=69.72, samples=13
   iops        : min=  170, max=  234, avg=194.77, stdev=17.48, samples=13
  lat (msec)   : 20=0.43%, 50=95.04%, 100=4.53%
  cpu          : usr=0.30%, sys=5.34%, ctx=4689, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190836: Sat Jul 18 18:59:07 2026
  read: IOPS=186, BW=745KiB/s (762kB/s)(4976KiB/6683msec)
    slat (usec): min=990, max=7540, avg=2971.01, stdev=1040.94
    clat (usec): min=9587, max=65223, avg=39772.63, stdev=6832.22
     lat (usec): min=12120, max=68367, avg=42745.04, stdev=7076.52
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[29230], 10.00th=[31589], 20.00th=[34341],
     | 30.00th=[36439], 40.00th=[38536], 50.00th=[39584], 60.00th=[41681],
     | 70.00th=[43254], 80.00th=[45351], 90.00th=[47973], 95.00th=[50070],
     | 99.00th=[57410], 99.50th=[60556], 99.90th=[64226], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  568, max= 1000, per=24.68%, avg=740.92, stdev=101.22, samples=13
   iops        : min=  142, max=  250, avg=185.23, stdev=25.31, samples=13
  write: IOPS=196, BW=788KiB/s (807kB/s)(5264KiB/6683msec); 0 zone resets
    slat (usec): min=46, max=2906, avg=154.56, stdev=112.01
    clat (usec): min=9548, max=60105, avg=40135.17, stdev=5801.59
     lat (usec): min=9696, max=60171, avg=40290.76, stdev=5802.12
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[29492], 10.00th=[32637], 20.00th=[35914],
     | 30.00th=[38011], 40.00th=[39060], 50.00th=[40633], 60.00th=[41681],
     | 70.00th=[43254], 80.00th=[44827], 90.00th=[46924], 95.00th=[49021],
     | 99.00th=[55313], 99.50th=[56361], 99.90th=[58983], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min=  651, max= 1024, per=25.23%, avg=788.54, stdev=97.33, samples=13
   iops        : min=  162, max=  256, avg=197.08, stdev=24.42, samples=13
  lat (msec)   : 10=0.08%, 20=0.23%, 50=95.31%, 100=4.38%
  cpu          : usr=0.33%, sys=5.28%, ctx=4702, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190837: Sat Jul 18 18:59:07 2026
  read: IOPS=186, BW=746KiB/s (764kB/s)(4984KiB/6681msec)
    slat (usec): min=981, max=11413, avg=2976.76, stdev=1072.85
    clat (usec): min=20625, max=71325, avg=39984.34, stdev=6406.34
     lat (usec): min=22306, max=73764, avg=42962.35, stdev=6630.37
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[29754], 10.00th=[31851], 20.00th=[34866],
     | 30.00th=[36439], 40.00th=[38536], 50.00th=[40109], 60.00th=[41681],
     | 70.00th=[43254], 80.00th=[45351], 90.00th=[47973], 95.00th=[50594],
     | 99.00th=[55313], 99.50th=[57410], 99.90th=[66323], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  575, max= 1000, per=24.92%, avg=748.23, stdev=101.05, samples=13
   iops        : min=  143, max=  250, avg=187.00, stdev=25.37, samples=13
  write: IOPS=196, BW=787KiB/s (806kB/s)(5256KiB/6681msec); 0 zone resets
    slat (usec): min=56, max=680, avg=154.24, stdev=77.80
    clat (usec): min=20492, max=60559, avg=39889.46, stdev=5691.21
     lat (usec): min=20550, max=60676, avg=40044.72, stdev=5690.44
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[28967], 10.00th=[32375], 20.00th=[35914],
     | 30.00th=[37487], 40.00th=[39060], 50.00th=[40109], 60.00th=[41157],
     | 70.00th=[42730], 80.00th=[44303], 90.00th=[46400], 95.00th=[49021],
     | 99.00th=[52691], 99.50th=[55837], 99.90th=[58459], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  645, max= 1040, per=25.11%, avg=785.00, stdev=102.00, samples=13
   iops        : min=  161, max=  260, avg=196.23, stdev=25.53, samples=13
  lat (msec)   : 50=95.31%, 100=4.69%
  cpu          : usr=0.34%, sys=5.27%, ctx=4733, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190838: Sat Jul 18 18:59:07 2026
  read: IOPS=189, BW=758KiB/s (776kB/s)(5060KiB/6677msec)
    slat (usec): min=949, max=7733, avg=2915.54, stdev=977.93
    clat (usec): min=18018, max=67315, avg=39649.23, stdev=6370.05
     lat (usec): min=19738, max=71683, avg=42566.16, stdev=6583.20
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[29230], 10.00th=[31589], 20.00th=[34866],
     | 30.00th=[36439], 40.00th=[38011], 50.00th=[39584], 60.00th=[41157],
     | 70.00th=[42730], 80.00th=[44827], 90.00th=[47449], 95.00th=[49546],
     | 99.00th=[55313], 99.50th=[58983], 99.90th=[64750], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  587, max= 1000, per=25.30%, avg=759.62, stdev=101.31, samples=13
   iops        : min=  146, max=  250, avg=189.85, stdev=25.44, samples=13
  write: IOPS=193, BW=776KiB/s (794kB/s)(5180KiB/6677msec); 0 zone resets
    slat (usec): min=52, max=669, avg=152.09, stdev=76.22
    clat (usec): min=13252, max=57338, avg=40007.49, stdev=5835.65
     lat (usec): min=13388, max=57510, avg=40160.62, stdev=5836.48
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[28705], 10.00th=[32113], 20.00th=[35914],
     | 30.00th=[37487], 40.00th=[39060], 50.00th=[40109], 60.00th=[41681],
     | 70.00th=[43254], 80.00th=[44303], 90.00th=[46924], 95.00th=[48497],
     | 99.00th=[52691], 99.50th=[55837], 99.90th=[57410], 99.95th=[57410],
     | 99.99th=[57410]
   bw (  KiB/s): min=  613, max= 1072, per=24.80%, avg=775.15, stdev=117.89, samples=13
   iops        : min=  153, max=  268, avg=193.77, stdev=29.50, samples=13
  lat (msec)   : 20=0.23%, 50=95.94%, 100=3.83%
  cpu          : usr=0.39%, sys=5.24%, ctx=4714, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3003KiB/s (3075kB/s), 745KiB/s-758KiB/s (762kB/s-776kB/s), io=19.6MiB (20.5MB), run=6669-6683msec
  WRITE: bw=3126KiB/s (3201kB/s), 776KiB/s-788KiB/s (794kB/s-807kB/s), io=20.4MiB (21.4MB), run=6669-6683msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=190842: Sat Jul 18 18:59:11 2026
  read: IOPS=10, BW=412MiB/s (432MB/s)(40.0MiB/97msec)
    slat (nsec): min=9709.5k, max=9709.5k, avg=9709498.00, stdev= 0.00
    clat (nsec): min=82406k, max=82406k, avg=82406349.00, stdev= 0.00
     lat (nsec): min=92119k, max=92119k, avg=92119478.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[82314],  5.00th=[82314], 10.00th=[82314], 20.00th=[82314],
     | 30.00th=[82314], 40.00th=[82314], 50.00th=[82314], 60.00th=[82314],
     | 70.00th=[82314], 80.00th=[82314], 90.00th=[82314], 95.00th=[82314],
     | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314],
     | 99.99th=[82314]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.38%, ctx=10, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=412MiB/s (432MB/s), 412MiB/s-412MiB/s (432MB/s-432MB/s), io=40.0MiB (41.9MB), run=97-97msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout