== sanity test 398c: run fio to test AIO ================= 21:03:04 (1763085784)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.132627 s, 316 MB/s
osc.lustre-OST0000-osc-ffffa0e9914e9800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=184117: Thu Nov 13 21:03:13 2025
  write: IOPS=464, BW=1857KiB/s (1901kB/s)(10.0MiB/5515msec); 0 zone resets
    slat (usec): min=26, max=482, avg=99.94, stdev=47.34
    clat (usec): min=18053, max=57564, avg=33901.74, stdev=3995.13
     lat (usec): min=18156, max=57756, avg=34002.82, stdev=3997.61
    clat percentiles (usec):
     |  1.00th=[22152],  5.00th=[26870], 10.00th=[28705], 20.00th=[31065],
     | 30.00th=[32113], 40.00th=[33162], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[36963], 90.00th=[38536], 95.00th=[39584],
     | 99.00th=[42206], 99.50th=[42730], 99.90th=[45876], 99.95th=[54264],
     | 99.99th=[57410]
   bw (  KiB/s): min= 1704, max= 2025, per=24.86%, avg=1846.50, stdev=109.53, samples=10
   iops        : min=  426, max=  506, avg=461.60, stdev=27.34, samples=10
  lat (msec)   : 20=0.16%, 50=99.77%, 100=0.08%
  cpu          : usr=0.42%, sys=3.94%, ctx=2526, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=184118: Thu Nov 13 21:03:13 2025
  write: IOPS=467, BW=1870KiB/s (1915kB/s)(10.0MiB/5476msec); 0 zone resets
    slat (usec): min=24, max=517, avg=102.75, stdev=52.64
    clat (usec): min=3659, max=63095, avg=34054.67, stdev=4411.21
     lat (usec): min=3798, max=63175, avg=34158.45, stdev=4409.61
    clat percentiles (usec):
     |  1.00th=[21627],  5.00th=[26608], 10.00th=[28705], 20.00th=[31065],
     | 30.00th=[32375], 40.00th=[33162], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[36963], 90.00th=[38536], 95.00th=[40109],
     | 99.00th=[43779], 99.50th=[48497], 99.90th=[55837], 99.95th=[60556],
     | 99.99th=[63177]
   bw (  KiB/s): min= 1728, max= 2192, per=25.11%, avg=1864.80, stdev=143.33, samples=10
   iops        : min=  432, max=  548, avg=466.20, stdev=35.83, samples=10
  lat (msec)   : 4=0.04%, 10=0.08%, 20=0.27%, 50=99.38%, 100=0.23%
  cpu          : usr=0.46%, sys=3.96%, ctx=2656, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=184119: Thu Nov 13 21:03:13 2025
  write: IOPS=465, BW=1860KiB/s (1905kB/s)(10.0MiB/5504msec); 0 zone resets
    slat (usec): min=32, max=30090, avg=114.52, stdev=595.03
    clat (usec): min=5823, max=55468, avg=33925.63, stdev=4185.55
     lat (usec): min=5908, max=55629, avg=34041.29, stdev=4165.91
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[26870], 10.00th=[28705], 20.00th=[31065],
     | 30.00th=[32113], 40.00th=[33162], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[36963], 90.00th=[38536], 95.00th=[39584],
     | 99.00th=[42730], 99.50th=[43779], 99.90th=[55313], 99.95th=[55313],
     | 99.99th=[55313]
   bw (  KiB/s): min= 1680, max= 2096, per=24.98%, avg=1855.20, stdev=129.13, samples=10
   iops        : min=  420, max=  524, avg=463.80, stdev=32.28, samples=10
  lat (msec)   : 10=0.08%, 20=0.20%, 50=99.49%, 100=0.23%
  cpu          : usr=0.49%, sys=4.42%, ctx=2620, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=184120: Thu Nov 13 21:03:13 2025
  write: IOPS=464, BW=1857KiB/s (1901kB/s)(10.0MiB/5515msec); 0 zone resets
    slat (usec): min=28, max=989, avg=105.58, stdev=55.79
    clat (usec): min=4132, max=54483, avg=33949.95, stdev=4086.42
     lat (usec): min=4257, max=54544, avg=34056.61, stdev=4083.50
    clat percentiles (usec):
     |  1.00th=[22152],  5.00th=[27132], 10.00th=[28705], 20.00th=[31065],
     | 30.00th=[32113], 40.00th=[33162], 50.00th=[34341], 60.00th=[35390],
     | 70.00th=[36439], 80.00th=[36963], 90.00th=[38536], 95.00th=[39584],
     | 99.00th=[42206], 99.50th=[43254], 99.90th=[54264], 99.95th=[54264],
     | 99.99th=[54264]
   bw (  KiB/s): min= 1712, max= 2038, per=24.90%, avg=1849.40, stdev=121.61, samples=10
   iops        : min=  428, max=  509, avg=462.30, stdev=30.32, samples=10
  lat (msec)   : 10=0.04%, 20=0.35%, 50=99.41%, 100=0.20%
  cpu          : usr=0.56%, sys=4.43%, ctx=2691, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7427KiB/s (7605kB/s), 1857KiB/s-1870KiB/s (1901kB/s-1915kB/s), io=40.0MiB (41.9MB), run=5476-5515msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=184130: Thu Nov 13 21:03:23 2025
  read: IOPS=197, BW=792KiB/s (811kB/s)(5048KiB/6375msec)
    slat (usec): min=1079, max=8241, avg=2949.13, stdev=983.32
    clat (usec): min=18327, max=66260, avg=37868.39, stdev=7351.64
     lat (usec): min=20390, max=73101, avg=40819.18, stdev=7673.53
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[27657], 10.00th=[29754], 20.00th=[31851],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[36439], 60.00th=[38536],
     | 70.00th=[40633], 80.00th=[43254], 90.00th=[48497], 95.00th=[52167],
     | 99.00th=[57934], 99.50th=[61080], 99.90th=[64750], 99.95th=[66323],
     | 99.99th=[66323]
   bw (  KiB/s): min=  624, max=  936, per=24.85%, avg=782.17, stdev=105.46, samples=12
   iops        : min=  156, max=  234, avg=195.50, stdev=26.33, samples=12
  write: IOPS=203, BW=814KiB/s (834kB/s)(5192KiB/6375msec); 0 zone resets
    slat (usec): min=36, max=4221, avg=100.69, stdev=137.18
    clat (usec): min=8021, max=64400, avg=38299.64, stdev=6929.89
     lat (usec): min=8094, max=64454, avg=38401.30, stdev=6936.94
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[29230], 10.00th=[30802], 20.00th=[32637],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[36963], 60.00th=[38536],
     | 70.00th=[40633], 80.00th=[43779], 90.00th=[48497], 95.00th=[51643],
     | 99.00th=[58459], 99.50th=[60031], 99.90th=[63177], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  664, max=  984, per=24.64%, avg=807.33, stdev=94.81, samples=12
   iops        : min=  166, max=  246, avg=201.83, stdev=23.70, samples=12
  lat (msec)   : 10=0.04%, 20=0.04%, 50=92.85%, 100=7.07%
  cpu          : usr=0.63%, sys=4.31%, ctx=4468, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=184131: Thu Nov 13 21:03:23 2025
  read: IOPS=196, BW=785KiB/s (804kB/s)(4976KiB/6341msec)
    slat (usec): min=23, max=10051, avg=2962.25, stdev=1023.39
    clat (usec): min=2969, max=71423, avg=37584.50, stdev=7503.37
     lat (usec): min=5543, max=75445, avg=40548.44, stdev=7890.97
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[27919], 10.00th=[29754], 20.00th=[31589],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[36439], 60.00th=[38011],
     | 70.00th=[40109], 80.00th=[43254], 90.00th=[47973], 95.00th=[51119],
     | 99.00th=[57934], 99.50th=[58983], 99.90th=[68682], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  600, max=  912, per=24.71%, avg=777.75, stdev=110.48, samples=12
   iops        : min=  150, max=  228, avg=194.42, stdev=27.61, samples=12
  write: IOPS=207, BW=830KiB/s (850kB/s)(5264KiB/6341msec); 0 zone resets
    slat (usec): min=24, max=488, avg=94.96, stdev=49.60
    clat (usec): min=24130, max=67353, avg=38538.49, stdev=7002.21
     lat (usec): min=24225, max=67461, avg=38634.50, stdev=7006.95
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[29230], 10.00th=[30802], 20.00th=[32900],
     | 30.00th=[34341], 40.00th=[35390], 50.00th=[36963], 60.00th=[39060],
     | 70.00th=[41157], 80.00th=[44303], 90.00th=[48497], 95.00th=[52691],
     | 99.00th=[57934], 99.50th=[60556], 99.90th=[62129], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  664, max= 1020, per=25.09%, avg=822.33, stdev=97.47, samples=12
   iops        : min=  166, max=  255, avg=205.58, stdev=24.37, samples=12
  lat (msec)   : 4=0.04%, 10=0.27%, 20=0.04%, 50=92.19%, 100=7.46%
  cpu          : usr=0.55%, sys=4.34%, ctx=4495, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=184132: Thu Nov 13 21:03:23 2025
  read: IOPS=195, BW=782KiB/s (801kB/s)(4984KiB/6375msec)
    slat (usec): min=1110, max=14907, avg=2962.20, stdev=1039.44
    clat (usec): min=5005, max=73414, avg=38041.32, stdev=7524.02
     lat (usec): min=7562, max=76795, avg=41005.21, stdev=7801.50
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[28443], 10.00th=[30016], 20.00th=[31851],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[36439], 60.00th=[38536],
     | 70.00th=[40633], 80.00th=[43779], 90.00th=[48497], 95.00th=[52691],
     | 99.00th=[57410], 99.50th=[60556], 99.90th=[71828], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min=  616, max=  952, per=24.62%, avg=774.83, stdev=99.67, samples=12
   iops        : min=  154, max=  238, avg=193.67, stdev=24.90, samples=12
  write: IOPS=206, BW=824KiB/s (844kB/s)(5256KiB/6375msec); 0 zone resets
    slat (usec): min=35, max=2900, avg=99.99, stdev=93.31
    clat (usec): min=13578, max=64813, avg=37998.78, stdev=6995.01
     lat (usec): min=13622, max=64901, avg=38099.72, stdev=6995.94
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[29230], 10.00th=[30802], 20.00th=[32113],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[36439], 60.00th=[38011],
     | 70.00th=[40633], 80.00th=[43779], 90.00th=[47973], 95.00th=[52167],
     | 99.00th=[56361], 99.50th=[60556], 99.90th=[64750], 99.95th=[64750],
     | 99.99th=[64750]
   bw (  KiB/s): min=  648, max= 1016, per=24.73%, avg=810.42, stdev=107.86, samples=12
   iops        : min=  162, max=  254, avg=202.58, stdev=26.96, samples=12
  lat (msec)   : 10=0.08%, 20=0.16%, 50=92.11%, 100=7.66%
  cpu          : usr=0.53%, sys=4.41%, ctx=4521, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=184133: Thu Nov 13 21:03:23 2025
  read: IOPS=199, BW=797KiB/s (816kB/s)(5060KiB/6350msec)
    slat (usec): min=1169, max=14803, avg=2958.89, stdev=1030.62
    clat (usec): min=18645, max=67632, avg=37637.25, stdev=6988.40
     lat (usec): min=22783, max=73954, avg=40597.72, stdev=7327.26
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[28443], 10.00th=[30016], 20.00th=[31851],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[36439], 60.00th=[38011],
     | 70.00th=[40109], 80.00th=[42730], 90.00th=[47449], 95.00th=[50594],
     | 99.00th=[57934], 99.50th=[61604], 99.90th=[65799], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  616, max=  936, per=25.09%, avg=789.50, stdev=100.54, samples=12
   iops        : min=  154, max=  234, avg=197.33, stdev=25.12, samples=12
  write: IOPS=203, BW=816KiB/s (835kB/s)(5180KiB/6350msec); 0 zone resets
    slat (usec): min=38, max=3680, avg=97.85, stdev=111.33
    clat (usec): min=21308, max=64626, avg=38195.10, stdev=6843.03
     lat (usec): min=21383, max=64729, avg=38293.92, stdev=6845.91
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[29230], 10.00th=[30540], 20.00th=[32375],
     | 30.00th=[33817], 40.00th=[35390], 50.00th=[36439], 60.00th=[38536],
     | 70.00th=[40633], 80.00th=[44303], 90.00th=[47973], 95.00th=[50594],
     | 99.00th=[58983], 99.50th=[60556], 99.90th=[63701], 99.95th=[64750],
     | 99.99th=[64750]
   bw (  KiB/s): min=  656, max=  920, per=24.74%, avg=810.58, stdev=91.39, samples=12
   iops        : min=  164, max=  230, avg=202.58, stdev=22.82, samples=12
  lat (msec)   : 20=0.04%, 50=93.87%, 100=6.09%
  cpu          : usr=0.38%, sys=4.60%, ctx=4571, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3148KiB/s (3223kB/s), 782KiB/s-797KiB/s (801kB/s-816kB/s), io=19.6MiB (20.5MB), run=6341-6375msec
  WRITE: bw=3277KiB/s (3356kB/s), 814KiB/s-830KiB/s (834kB/s-850kB/s), io=20.4MiB (21.4MB), run=6341-6375msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=184137: Thu Nov 13 21:03:27 2025
  read: IOPS=8, BW=357MiB/s (374MB/s)(40.0MiB/112msec)
    slat (nsec): min=9941.8k, max=9941.8k, avg=9941803.00, stdev= 0.00
    clat (nsec): min=95944k, max=95944k, avg=95943737.00, stdev= 0.00
     lat (nsec): min=105889k, max=105889k, avg=105889493.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[95945],  5.00th=[95945], 10.00th=[95945], 20.00th=[95945],
     | 30.00th=[95945], 40.00th=[95945], 50.00th=[95945], 60.00th=[95945],
     | 70.00th=[95945], 80.00th=[95945], 90.00th=[95945], 95.00th=[95945],
     | 99.00th=[95945], 99.50th=[95945], 99.90th=[95945], 99.95th=[95945],
     | 99.99th=[95945]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.01%, ctx=7, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=357MiB/s (374MB/s), 357MiB/s-357MiB/s (374MB/s-374MB/s), io=40.0MiB (41.9MB), run=112-112msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout