== sanity test 398c: run fio to test AIO ================= 10:30:49 (1787063449)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.133387 s, 314 MB/s
osc.lustre-OST0000-osc-ffffa0af90609800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=196944: Tue Aug 18 10:31:00 2026
  write: IOPS=355, BW=1422KiB/s (1456kB/s)(10.0MiB/7203msec); 0 zone resets
    slat (usec): min=44, max=1438, avg=287.56, stdev=179.82
    clat (msec): min=24, max=111, avg=44.62, stdev=11.48
     lat (msec): min=24, max=111, avg=44.91, stdev=11.54
    clat percentiles (msec):
     |  1.00th=[   27],  5.00th=[   30], 10.00th=[   32], 20.00th=[   34],
     | 30.00th=[   35], 40.00th=[   39], 50.00th=[   46], 60.00th=[   50],
     | 70.00th=[   52], 80.00th=[   56], 90.00th=[   58], 95.00th=[   61],
     | 99.00th=[   77], 99.50th=[   93], 99.90th=[  100], 99.95th=[  106],
     | 99.99th=[  112]
   bw (  KiB/s): min= 1072, max= 1984, per=25.32%, avg=1430.14, stdev=283.81, samples=14
   iops        : min=  268, max=  496, avg=357.50, stdev=70.97, samples=14
  lat (msec)   : 50=60.90%, 100=39.02%, 250=0.08%
  cpu          : usr=0.53%, sys=7.50%, ctx=3124, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=196945: Tue Aug 18 10:31:00 2026
  write: IOPS=353, BW=1412KiB/s (1446kB/s)(10.0MiB/7250msec); 0 zone resets
    slat (usec): min=49, max=1320, avg=290.17, stdev=183.56
    clat (usec): min=24616, max=88284, avg=44347.02, stdev=10733.74
     lat (usec): min=24798, max=88754, avg=44638.84, stdev=10793.98
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[29492], 10.00th=[31589], 20.00th=[33162],
     | 30.00th=[34866], 40.00th=[37487], 50.00th=[45351], 60.00th=[50070],
     | 70.00th=[51643], 80.00th=[55313], 90.00th=[57934], 95.00th=[60031],
     | 99.00th=[64226], 99.50th=[64750], 99.90th=[83362], 99.95th=[87557],
     | 99.99th=[88605]
   bw (  KiB/s): min= 1080, max= 1936, per=25.17%, avg=1422.07, stdev=287.79, samples=14
   iops        : min=  270, max=  484, avg=355.50, stdev=71.97, samples=14
  lat (msec)   : 50=60.86%, 100=39.14%
  cpu          : usr=0.40%, sys=7.49%, ctx=3041, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=196946: Tue Aug 18 10:31:00 2026
  write: IOPS=353, BW=1412KiB/s (1446kB/s)(10.0MiB/7250msec); 0 zone resets
    slat (usec): min=61, max=1408, avg=291.70, stdev=187.08
    clat (usec): min=24570, max=82768, avg=44402.02, stdev=10679.54
     lat (usec): min=24710, max=82957, avg=44695.24, stdev=10747.41
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[29230], 10.00th=[31589], 20.00th=[33162],
     | 30.00th=[34866], 40.00th=[38536], 50.00th=[45351], 60.00th=[49546],
     | 70.00th=[52167], 80.00th=[55313], 90.00th=[58459], 95.00th=[60031],
     | 99.00th=[64226], 99.50th=[64750], 99.90th=[82314], 99.95th=[82314],
     | 99.99th=[82314]
   bw (  KiB/s): min= 1056, max= 1936, per=25.12%, avg=1419.21, stdev=284.70, samples=14
   iops        : min=  264, max=  484, avg=354.79, stdev=71.19, samples=14
  lat (msec)   : 50=60.98%, 100=39.02%
  cpu          : usr=0.40%, sys=8.08%, ctx=3154, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=196947: Tue Aug 18 10:31:00 2026
  write: IOPS=353, BW=1415KiB/s (1449kB/s)(10.0MiB/7237msec); 0 zone resets
    slat (usec): min=61, max=44156, avg=310.47, stdev=883.77
    clat (msec): min=7, max=109, avg=44.35, stdev=11.09
     lat (msec): min=7, max=109, avg=44.66, stdev=11.14
    clat percentiles (msec):
     |  1.00th=[   27],  5.00th=[   30], 10.00th=[   32], 20.00th=[   34],
     | 30.00th=[   35], 40.00th=[   39], 50.00th=[   46], 60.00th=[   50],
     | 70.00th=[   52], 80.00th=[   56], 90.00th=[   58], 95.00th=[   61],
     | 99.00th=[   65], 99.50th=[   65], 99.90th=[  105], 99.95th=[  106],
     | 99.99th=[  110]
   bw (  KiB/s): min= 1080, max= 1944, per=25.24%, avg=1425.71, stdev=285.13, samples=14
   iops        : min=  270, max=  486, avg=356.43, stdev=71.28, samples=14
  lat (msec)   : 10=0.04%, 20=0.04%, 50=61.21%, 100=38.59%, 250=0.12%
  cpu          : usr=0.43%, sys=8.46%, ctx=2993, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5650KiB/s (5785kB/s), 1412KiB/s-1422KiB/s (1446kB/s-1456kB/s), io=40.0MiB (41.9MB), run=7203-7250msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=196957: Tue Aug 18 10:31:08 2026
  read: IOPS=226, BW=906KiB/s (927kB/s)(5048KiB/5574msec)
    slat (usec): min=1045, max=6904, avg=2526.56, stdev=757.39
    clat (usec): min=20042, max=50325, avg=32718.31, stdev=4430.23
     lat (usec): min=23146, max=52542, avg=35246.42, stdev=4440.29
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[25560], 10.00th=[26870], 20.00th=[28967],
     | 30.00th=[30278], 40.00th=[31589], 50.00th=[32637], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[38011], 95.00th=[40109],
     | 99.00th=[43779], 99.50th=[45876], 99.90th=[49546], 99.95th=[50070],
     | 99.99th=[50070]
   bw (  KiB/s): min=  811, max=  976, per=25.03%, avg=899.50, stdev=51.43, samples=10
   iops        : min=  202, max=  244, avg=224.80, stdev=13.00, samples=10
  write: IOPS=232, BW=931KiB/s (954kB/s)(5192KiB/5574msec); 0 zone resets
    slat (usec): min=63, max=974, avg=196.20, stdev=111.48
    clat (usec): min=22757, max=48299, avg=33334.86, stdev=3225.51
     lat (usec): min=23130, max=48380, avg=33532.02, stdev=3225.13
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28443], 10.00th=[29230], 20.00th=[30540],
     | 30.00th=[31589], 40.00th=[32375], 50.00th=[33162], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[37487], 95.00th=[39060],
     | 99.00th=[41681], 99.50th=[42730], 99.90th=[45876], 99.95th=[48497],
     | 99.99th=[48497]
   bw (  KiB/s): min=  772, max= 1080, per=25.02%, avg=935.60, stdev=81.24, samples=10
   iops        : min=  193, max=  270, avg=233.90, stdev=20.31, samples=10
  lat (msec)   : 50=99.96%, 100=0.04%
  cpu          : usr=0.39%, sys=8.15%, ctx=4425, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=196958: Tue Aug 18 10:31:08 2026
  read: IOPS=222, BW=892KiB/s (913kB/s)(4976KiB/5580msec)
    slat (usec): min=1028, max=8849, avg=2512.56, stdev=759.35
    clat (usec): min=21134, max=49939, avg=32842.41, stdev=4395.60
     lat (usec): min=23032, max=52804, avg=35356.48, stdev=4413.81
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[26084], 10.00th=[27132], 20.00th=[28967],
     | 30.00th=[30540], 40.00th=[31589], 50.00th=[32900], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[38536], 95.00th=[40109],
     | 99.00th=[44303], 99.50th=[45351], 99.90th=[48497], 99.95th=[50070],
     | 99.99th=[50070]
   bw (  KiB/s): min=  721, max=  960, per=24.67%, avg=886.50, stdev=63.47, samples=10
   iops        : min=  180, max=  240, avg=221.60, stdev=15.94, samples=10
  write: IOPS=235, BW=943KiB/s (966kB/s)(5264KiB/5580msec); 0 zone resets
    slat (usec): min=61, max=1475, avg=191.62, stdev=115.72
    clat (usec): min=21487, max=47070, avg=33283.16, stdev=3339.17
     lat (usec): min=21564, max=47232, avg=33475.81, stdev=3340.79
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[28181], 10.00th=[29230], 20.00th=[30540],
     | 30.00th=[31589], 40.00th=[32375], 50.00th=[33162], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[37487], 95.00th=[39060],
     | 99.00th=[42206], 99.50th=[43779], 99.90th=[44303], 99.95th=[46924],
     | 99.99th=[46924]
   bw (  KiB/s): min=  850, max= 1024, per=25.37%, avg=949.00, stdev=63.23, samples=10
   iops        : min=  212, max=  256, avg=237.20, stdev=15.89, samples=10
  lat (msec)   : 50=100.00%
  cpu          : usr=0.50%, sys=8.03%, ctx=4424, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=196959: Tue Aug 18 10:31:08 2026
  read: IOPS=223, BW=892KiB/s (914kB/s)(4984KiB/5585msec)
    slat (usec): min=1060, max=5510, avg=2533.50, stdev=712.26
    clat (usec): min=12272, max=49099, avg=32859.28, stdev=4695.78
     lat (usec): min=13682, max=51392, avg=35394.13, stdev=4720.15
    clat percentiles (usec):
     |  1.00th=[22152],  5.00th=[25297], 10.00th=[27132], 20.00th=[28967],
     | 30.00th=[30540], 40.00th=[31851], 50.00th=[32900], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[38536], 95.00th=[40633],
     | 99.00th=[44827], 99.50th=[45876], 99.90th=[48497], 99.95th=[49021],
     | 99.99th=[49021]
   bw (  KiB/s): min=  729, max= 1008, per=24.72%, avg=888.10, stdev=75.85, samples=10
   iops        : min=  182, max=  252, avg=222.00, stdev=19.02, samples=10
  write: IOPS=235, BW=941KiB/s (964kB/s)(5256KiB/5585msec); 0 zone resets
    slat (usec): min=62, max=899, avg=196.54, stdev=115.08
    clat (usec): min=10373, max=48505, avg=33136.09, stdev=3798.84
     lat (usec): min=10552, max=48698, avg=33333.58, stdev=3802.66
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[28181], 10.00th=[28967], 20.00th=[30278],
     | 30.00th=[31327], 40.00th=[32113], 50.00th=[32900], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[37487], 95.00th=[39060],
     | 99.00th=[42730], 99.50th=[44303], 99.90th=[47449], 99.95th=[48497],
     | 99.99th=[48497]
   bw (  KiB/s): min=  787, max= 1048, per=25.03%, avg=936.30, stdev=88.12, samples=10
   iops        : min=  196, max=  262, avg=234.00, stdev=22.17, samples=10
  lat (msec)   : 20=0.55%, 50=99.45%
  cpu          : usr=0.36%, sys=8.24%, ctx=4408, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=196960: Tue Aug 18 10:31:08 2026
  read: IOPS=227, BW=911KiB/s (933kB/s)(5060KiB/5555msec)
    slat (usec): min=187, max=12603, avg=2537.10, stdev=860.19
    clat (usec): min=6875, max=55113, avg=32822.45, stdev=4647.83
     lat (usec): min=7238, max=55302, avg=35360.89, stdev=4720.81
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[26084], 10.00th=[27132], 20.00th=[28967],
     | 30.00th=[30278], 40.00th=[31327], 50.00th=[32637], 60.00th=[33817],
     | 70.00th=[34866], 80.00th=[36439], 90.00th=[38536], 95.00th=[40633],
     | 99.00th=[44303], 99.50th=[46924], 99.90th=[54789], 99.95th=[55313],
     | 99.99th=[55313]
   bw (  KiB/s): min=  869, max=  960, per=25.31%, avg=909.30, stdev=29.42, samples=10
   iops        : min=  217, max=  240, avg=227.30, stdev= 7.39, samples=10
  write: IOPS=233, BW=932KiB/s (955kB/s)(5180KiB/5555msec); 0 zone resets
    slat (usec): min=54, max=812, avg=195.67, stdev=111.30
    clat (usec): min=23309, max=63278, avg=33698.44, stdev=4222.22
     lat (usec): min=23600, max=63576, avg=33895.14, stdev=4228.87
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[28181], 10.00th=[29230], 20.00th=[30802],
     | 30.00th=[31851], 40.00th=[32637], 50.00th=[33424], 60.00th=[34341],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[38011], 95.00th=[40109],
     | 99.00th=[45876], 99.50th=[61080], 99.90th=[63177], 99.95th=[63177],
     | 99.99th=[63177]
   bw (  KiB/s): min=  850, max= 1008, per=24.95%, avg=933.00, stdev=56.77, samples=10
   iops        : min=  212, max=  252, avg=233.20, stdev=14.27, samples=10
  lat (msec)   : 10=0.04%, 20=0.08%, 50=99.38%, 100=0.51%
  cpu          : usr=0.50%, sys=8.10%, ctx=4414, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3593KiB/s (3679kB/s), 892KiB/s-911KiB/s (913kB/s-933kB/s), io=19.6MiB (20.5MB), run=5555-5585msec
  WRITE: bw=3741KiB/s (3831kB/s), 931KiB/s-943KiB/s (954kB/s-966kB/s), io=20.4MiB (21.4MB), run=5555-5585msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=196965: Tue Aug 18 10:31:11 2026
  read: IOPS=9, BW=360MiB/s (378MB/s)(40.0MiB/111msec)
    slat (nsec): min=7474.0k, max=7474.0k, avg=7474035.00, stdev= 0.00
    clat (nsec): min=96850k, max=96850k, avg=96849769.00, stdev= 0.00
     lat (nsec): min=104326k, max=104326k, avg=104326124.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[96994],  5.00th=[96994], 10.00th=[96994], 20.00th=[96994],
     | 30.00th=[96994], 40.00th=[96994], 50.00th=[96994], 60.00th=[96994],
     | 70.00th=[96994], 80.00th=[96994], 90.00th=[96994], 95.00th=[96994],
     | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994],
     | 99.99th=[96994]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.36%, ctx=3, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=360MiB/s (378MB/s), 360MiB/s-360MiB/s (378MB/s-378MB/s), io=40.0MiB (41.9MB), run=111-111msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout