== sanity test 398c: run fio to test AIO ================= 23:22:09 (1786936929)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.136797 s, 307 MB/s
osc.lustre-OST0000-osc-ffff8c813151e800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=195018: Sun Aug 16 23:22:20 2026
  write: IOPS=339, BW=1359KiB/s (1391kB/s)(10.0MiB/7537msec); 0 zone resets
    slat (usec): min=52, max=42036, avg=302.53, stdev=842.59
    clat (usec): min=9154, max=91455, avg=46323.28, stdev=11346.05
     lat (usec): min=9773, max=91867, avg=46627.42, stdev=11420.75
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[28705], 10.00th=[30802], 20.00th=[34341],
     | 30.00th=[39060], 40.00th=[44303], 50.00th=[46924], 60.00th=[50594],
     | 70.00th=[53740], 80.00th=[56886], 90.00th=[60031], 95.00th=[61604],
     | 99.00th=[65799], 99.50th=[80217], 99.90th=[91751], 99.95th=[91751],
     | 99.99th=[91751]
   bw (  KiB/s): min= 1080, max= 1768, per=24.80%, avg=1346.43, stdev=249.72, samples=14
   iops        : min=  270, max=  442, avg=336.57, stdev=62.37, samples=14
  lat (msec)   : 10=0.08%, 20=0.12%, 50=58.40%, 100=41.41%
  cpu          : usr=0.52%, sys=7.59%, ctx=2837, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=195019: Sun Aug 16 23:22:20 2026
  write: IOPS=339, BW=1358KiB/s (1390kB/s)(10.0MiB/7542msec); 0 zone resets
    slat (usec): min=49, max=1396, avg=276.26, stdev=167.27
    clat (usec): min=3036, max=90897, avg=46339.68, stdev=11189.47
     lat (usec): min=3315, max=91138, avg=46617.75, stdev=11256.50
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[28967], 10.00th=[30802], 20.00th=[33817],
     | 30.00th=[39584], 40.00th=[43779], 50.00th=[46924], 60.00th=[50594],
     | 70.00th=[53740], 80.00th=[56886], 90.00th=[60031], 95.00th=[61604],
     | 99.00th=[66323], 99.50th=[80217], 99.90th=[90702], 99.95th=[90702],
     | 99.99th=[90702]
   bw (  KiB/s): min= 1064, max= 1776, per=24.72%, avg=1342.21, stdev=247.15, samples=14
   iops        : min=  266, max=  444, avg=335.50, stdev=61.72, samples=14
  lat (msec)   : 4=0.04%, 50=58.24%, 100=41.72%
  cpu          : usr=0.54%, sys=7.29%, ctx=2758, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=195020: Sun Aug 16 23:22:20 2026
  write: IOPS=339, BW=1359KiB/s (1391kB/s)(10.0MiB/7537msec); 0 zone resets
    slat (usec): min=42, max=1390, avg=281.94, stdev=177.77
    clat (usec): min=22310, max=90651, avg=46657.37, stdev=11521.91
     lat (usec): min=22478, max=91190, avg=46941.09, stdev=11580.10
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[28705], 10.00th=[30802], 20.00th=[34866],
     | 30.00th=[39584], 40.00th=[44303], 50.00th=[47449], 60.00th=[51119],
     | 70.00th=[54264], 80.00th=[56886], 90.00th=[60031], 95.00th=[62129],
     | 99.00th=[82314], 99.50th=[85459], 99.90th=[89654], 99.95th=[89654],
     | 99.99th=[90702]
   bw (  KiB/s): min= 1080, max= 1776, per=24.77%, avg=1345.00, stdev=243.55, samples=14
   iops        : min=  270, max=  444, avg=336.21, stdev=60.83, samples=14
  lat (msec)   : 50=57.23%, 100=42.77%
  cpu          : usr=0.56%, sys=6.77%, ctx=2821, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=195021: Sun Aug 16 23:22:20 2026
  write: IOPS=340, BW=1363KiB/s (1396kB/s)(10.0MiB/7512msec); 0 zone resets
    slat (usec): min=61, max=1175, avg=276.55, stdev=164.75
    clat (usec): min=9625, max=96197, avg=46582.42, stdev=11627.76
     lat (usec): min=9847, max=96407, avg=46860.68, stdev=11683.13
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[28705], 10.00th=[30802], 20.00th=[34866],
     | 30.00th=[39584], 40.00th=[44303], 50.00th=[47449], 60.00th=[51119],
     | 70.00th=[54264], 80.00th=[56886], 90.00th=[60031], 95.00th=[62129],
     | 99.00th=[67634], 99.50th=[90702], 99.90th=[95945], 99.95th=[95945],
     | 99.99th=[95945]
   bw (  KiB/s): min= 1072, max= 1795, per=24.88%, avg=1351.07, stdev=254.84, samples=14
   iops        : min=  268, max=  448, avg=337.71, stdev=63.61, samples=14
  lat (msec)   : 10=0.04%, 20=0.08%, 50=57.23%, 100=42.66%
  cpu          : usr=0.48%, sys=7.02%, ctx=2759, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5431KiB/s (5561kB/s), 1358KiB/s-1363KiB/s (1390kB/s-1396kB/s), io=40.0MiB (41.9MB), run=7512-7542msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=195031: Sun Aug 16 23:22:29 2026
  read: IOPS=219, BW=879KiB/s (900kB/s)(5048KiB/5745msec)
    slat (usec): min=1053, max=6851, avg=2546.50, stdev=704.96
    clat (usec): min=10793, max=47128, avg=34249.84, stdev=4408.80
     lat (usec): min=12951, max=49413, avg=36797.76, stdev=4433.37
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[27132], 10.00th=[28443], 20.00th=[30802],
     | 30.00th=[32113], 40.00th=[33162], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[38011], 90.00th=[39584], 95.00th=[41157],
     | 99.00th=[44303], 99.50th=[45351], 99.90th=[46924], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min=  774, max=  960, per=25.08%, avg=876.18, stdev=62.13, samples=11
   iops        : min=  193, max=  240, avg=219.00, stdev=15.61, samples=11
  write: IOPS=225, BW=904KiB/s (925kB/s)(5192KiB/5745msec); 0 zone resets
    slat (usec): min=61, max=1244, avg=204.07, stdev=121.00
    clat (usec): min=10373, max=46284, avg=33953.45, stdev=3490.93
     lat (usec): min=10763, max=46346, avg=34158.62, stdev=3485.26
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[28705], 10.00th=[29754], 20.00th=[31327],
     | 30.00th=[32113], 40.00th=[32900], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[35390], 80.00th=[36963], 90.00th=[38011], 95.00th=[39584],
     | 99.00th=[43254], 99.50th=[44303], 99.90th=[45351], 99.95th=[46400],
     | 99.99th=[46400]
   bw (  KiB/s): min=  736, max= 1024, per=24.96%, avg=907.64, stdev=83.86, samples=11
   iops        : min=  184, max=  256, avg=226.91, stdev=20.96, samples=11
  lat (msec)   : 20=0.23%, 50=99.77%
  cpu          : usr=0.64%, sys=7.75%, ctx=4651, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=195032: Sun Aug 16 23:22:29 2026
  read: IOPS=216, BW=868KiB/s (889kB/s)(4976KiB/5733msec)
    slat (usec): min=1078, max=7776, avg=2569.52, stdev=729.59
    clat (usec): min=16599, max=51643, avg=34344.50, stdev=4490.60
     lat (usec): min=23765, max=53827, avg=36915.64, stdev=4487.73
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[26870], 10.00th=[28443], 20.00th=[30540],
     | 30.00th=[32113], 40.00th=[33424], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[38011], 90.00th=[39584], 95.00th=[41681],
     | 99.00th=[46400], 99.50th=[47449], 99.90th=[49546], 99.95th=[51643],
     | 99.99th=[51643]
   bw (  KiB/s): min=  743, max=  968, per=24.88%, avg=869.00, stdev=63.12, samples=11
   iops        : min=  185, max=  242, avg=217.18, stdev=15.93, samples=11
  write: IOPS=229, BW=918KiB/s (940kB/s)(5264KiB/5733msec); 0 zone resets
    slat (usec): min=62, max=855, avg=198.75, stdev=114.96
    clat (usec): min=12399, max=58588, avg=34134.62, stdev=3608.52
     lat (usec): min=12522, max=58771, avg=34334.48, stdev=3605.58
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28967], 10.00th=[29754], 20.00th=[31065],
     | 30.00th=[32375], 40.00th=[33162], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[36963], 90.00th=[38536], 95.00th=[39584],
     | 99.00th=[43254], 99.50th=[45351], 99.90th=[53740], 99.95th=[58459],
     | 99.99th=[58459]
   bw (  KiB/s): min=  856, max= 1000, per=25.31%, avg=920.09, stdev=49.07, samples=11
   iops        : min=  214, max=  250, avg=230.00, stdev=12.30, samples=11
  lat (msec)   : 20=0.08%, 50=99.80%, 100=0.12%
  cpu          : usr=0.61%, sys=7.78%, ctx=4598, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=195033: Sun Aug 16 23:22:29 2026
  read: IOPS=217, BW=870KiB/s (891kB/s)(4984KiB/5728msec)
    slat (usec): min=63, max=8510, avg=2580.36, stdev=746.05
    clat (usec): min=7583, max=56841, avg=34368.66, stdev=4918.83
     lat (usec): min=7664, max=59273, avg=36950.57, stdev=4963.41
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[26608], 10.00th=[28181], 20.00th=[30278],
     | 30.00th=[32113], 40.00th=[33424], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[38011], 90.00th=[40633], 95.00th=[42206],
     | 99.00th=[46400], 99.50th=[47973], 99.90th=[53740], 99.95th=[56886],
     | 99.99th=[56886]
   bw (  KiB/s): min=  816, max=  944, per=25.03%, avg=874.27, stdev=43.28, samples=11
   iops        : min=  204, max=  236, avg=218.55, stdev=10.85, samples=11
  write: IOPS=229, BW=918KiB/s (940kB/s)(5256KiB/5728msec); 0 zone resets
    slat (usec): min=56, max=894, avg=200.65, stdev=117.19
    clat (usec): min=24117, max=59623, avg=34343.77, stdev=3895.37
     lat (usec): min=24392, max=59691, avg=34545.48, stdev=3895.68
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[28705], 10.00th=[29754], 20.00th=[31327],
     | 30.00th=[32375], 40.00th=[33162], 50.00th=[34341], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[36963], 90.00th=[39060], 95.00th=[40633],
     | 99.00th=[44303], 99.50th=[49546], 99.90th=[59507], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min=  825, max= 1048, per=25.29%, avg=919.36, stdev=67.88, samples=11
   iops        : min=  206, max=  262, avg=229.82, stdev=17.00, samples=11
  lat (msec)   : 10=0.08%, 20=0.12%, 50=99.53%, 100=0.27%
  cpu          : usr=0.45%, sys=7.87%, ctx=4645, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=195034: Sun Aug 16 23:22:29 2026
  read: IOPS=220, BW=881KiB/s (902kB/s)(5060KiB/5743msec)
    slat (usec): min=1049, max=6547, avg=2545.18, stdev=714.35
    clat (usec): min=19600, max=49635, avg=34159.23, stdev=4279.76
     lat (usec): min=21796, max=51540, avg=36705.97, stdev=4305.89
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[27132], 10.00th=[28443], 20.00th=[30540],
     | 30.00th=[31851], 40.00th=[33162], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[37487], 90.00th=[39584], 95.00th=[41681],
     | 99.00th=[44827], 99.50th=[45351], 99.90th=[46924], 99.95th=[49546],
     | 99.99th=[49546]
   bw (  KiB/s): min=  755, max=  976, per=25.31%, avg=883.91, stdev=60.21, samples=11
   iops        : min=  188, max=  244, avg=220.91, stdev=15.21, samples=11
  write: IOPS=225, BW=902KiB/s (924kB/s)(5180KiB/5743msec); 0 zone resets
    slat (usec): min=50, max=909, avg=206.63, stdev=119.90
    clat (usec): min=24546, max=56061, avg=34197.04, stdev=3535.41
     lat (usec): min=24789, max=56472, avg=34404.79, stdev=3537.73
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28967], 10.00th=[29754], 20.00th=[31327],
     | 30.00th=[32113], 40.00th=[33162], 50.00th=[34341], 60.00th=[34866],
     | 70.00th=[35914], 80.00th=[36963], 90.00th=[38536], 95.00th=[40109],
     | 99.00th=[43254], 99.50th=[46400], 99.90th=[51119], 99.95th=[55837],
     | 99.99th=[55837]
   bw (  KiB/s): min=  780, max=  960, per=24.81%, avg=902.18, stdev=50.26, samples=11
   iops        : min=  195, max=  240, avg=225.55, stdev=12.56, samples=11
  lat (msec)   : 20=0.04%, 50=99.77%, 100=0.20%
  cpu          : usr=0.52%, sys=7.84%, ctx=4641, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3493KiB/s (3577kB/s), 868KiB/s-881KiB/s (889kB/s-902kB/s), io=19.6MiB (20.5MB), run=5728-5745msec
  WRITE: bw=3637KiB/s (3724kB/s), 902KiB/s-918KiB/s (924kB/s-940kB/s), io=20.4MiB (21.4MB), run=5728-5745msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=195039: Sun Aug 16 23:22:32 2026
  read: IOPS=9, BW=377MiB/s (396MB/s)(40.0MiB/106msec)
    slat (nsec): min=9652.5k, max=9652.5k, avg=9652528.00, stdev= 0.00
    clat (nsec): min=91111k, max=91111k, avg=91111059.00, stdev= 0.00
     lat (nsec): min=100766k, max=100766k, avg=100766402.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[90702],  5.00th=[90702], 10.00th=[90702], 20.00th=[90702],
     | 30.00th=[90702], 40.00th=[90702], 50.00th=[90702], 60.00th=[90702],
     | 70.00th=[90702], 80.00th=[90702], 90.00th=[90702], 95.00th=[90702],
     | 99.00th=[90702], 99.50th=[90702], 99.90th=[90702], 99.95th=[90702],
     | 99.99th=[90702]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.57%, ctx=6, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=377MiB/s (396MB/s), 377MiB/s-377MiB/s (396MB/s-396MB/s), io=40.0MiB (41.9MB), run=106-106msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout