== sanity test 398c: run fio to test AIO ================= 16:44:07 (1784407447)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.133458 s, 314 MB/s
osc.lustre-OST0000-osc-ffff9caa87330000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=183759: Sat Jul 18 16:44:16 2026
  write: IOPS=483, BW=1932KiB/s (1978kB/s)(10.0MiB/5300msec); 0 zone resets
    slat (usec): min=41, max=1635, avg=199.22, stdev=108.09
    clat (usec): min=5259, max=62466, avg=32791.59, stdev=4012.56
     lat (usec): min=5436, max=62650, avg=32992.32, stdev=4012.36
    clat percentiles (usec):
     |  1.00th=[19792],  5.00th=[26084], 10.00th=[27919], 20.00th=[30278],
     | 30.00th=[31589], 40.00th=[32113], 50.00th=[32900], 60.00th=[33424],
     | 70.00th=[34341], 80.00th=[35390], 90.00th=[36963], 95.00th=[39060],
     | 99.00th=[42730], 99.50th=[45876], 99.90th=[56361], 99.95th=[59507],
     | 99.99th=[62653]
   bw (  KiB/s): min= 1712, max= 2208, per=25.09%, avg=1924.80, stdev=138.68, samples=10
   iops        : min=  428, max=  552, avg=481.20, stdev=34.67, samples=10
  lat (msec)   : 10=0.08%, 20=0.98%, 50=98.75%, 100=0.20%
  cpu          : usr=0.49%, sys=7.21%, ctx=2671, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183760: Sat Jul 18 16:44:16 2026
  write: IOPS=483, BW=1932KiB/s (1978kB/s)(10.0MiB/5300msec); 0 zone resets
    slat (usec): min=52, max=796, avg=191.99, stdev=96.06
    clat (usec): min=4477, max=59147, avg=32872.67, stdev=3988.36
     lat (usec): min=4594, max=59284, avg=33065.94, stdev=3986.83
    clat percentiles (usec):
     |  1.00th=[21627],  5.00th=[26084], 10.00th=[28181], 20.00th=[30278],
     | 30.00th=[31589], 40.00th=[32375], 50.00th=[32900], 60.00th=[33817],
     | 70.00th=[34341], 80.00th=[35390], 90.00th=[36963], 95.00th=[39060],
     | 99.00th=[42730], 99.50th=[43779], 99.90th=[55837], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min= 1720, max= 2185, per=25.07%, avg=1923.30, stdev=130.65, samples=10
   iops        : min=  430, max=  546, avg=480.80, stdev=32.61, samples=10
  lat (msec)   : 10=0.04%, 20=0.66%, 50=99.10%, 100=0.20%
  cpu          : usr=0.57%, sys=7.02%, ctx=2673, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183761: Sat Jul 18 16:44:16 2026
  write: IOPS=480, BW=1920KiB/s (1966kB/s)(10.0MiB/5333msec); 0 zone resets
    slat (usec): min=45, max=732, avg=192.37, stdev=95.52
    clat (usec): min=11150, max=49591, avg=32778.35, stdev=3693.57
     lat (usec): min=11408, max=49682, avg=32972.09, stdev=3690.28
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[26084], 10.00th=[28181], 20.00th=[30278],
     | 30.00th=[31589], 40.00th=[32113], 50.00th=[32900], 60.00th=[33424],
     | 70.00th=[34341], 80.00th=[35390], 90.00th=[36963], 95.00th=[38536],
     | 99.00th=[42206], 99.50th=[43254], 99.90th=[49546], 99.95th=[49546],
     | 99.99th=[49546]
   bw (  KiB/s): min= 1728, max= 2060, per=24.90%, avg=1910.80, stdev=109.89, samples=10
   iops        : min=  432, max=  515, avg=477.70, stdev=27.47, samples=10
  lat (msec)   : 20=0.31%, 50=99.69%
  cpu          : usr=0.60%, sys=7.52%, ctx=2610, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183762: Sat Jul 18 16:44:16 2026
  write: IOPS=479, BW=1918KiB/s (1964kB/s)(10.0MiB/5338msec); 0 zone resets
    slat (usec): min=45, max=991, avg=184.77, stdev=95.34
    clat (usec): min=13628, max=46016, avg=32784.52, stdev=3625.29
     lat (usec): min=13748, max=46481, avg=32970.42, stdev=3632.43
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[26084], 10.00th=[27919], 20.00th=[30278],
     | 30.00th=[31589], 40.00th=[32113], 50.00th=[32900], 60.00th=[33817],
     | 70.00th=[34341], 80.00th=[35390], 90.00th=[36963], 95.00th=[38536],
     | 99.00th=[42730], 99.50th=[43254], 99.90th=[44827], 99.95th=[45351],
     | 99.99th=[45876]
   bw (  KiB/s): min= 1680, max= 2056, per=24.78%, avg=1901.40, stdev=105.00, samples=10
   iops        : min=  420, max=  514, avg=475.30, stdev=26.20, samples=10
  lat (msec)   : 20=0.31%, 50=99.69%
  cpu          : usr=0.39%, sys=6.78%, ctx=2508, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7673KiB/s (7857kB/s), 1918KiB/s-1932KiB/s (1964kB/s-1978kB/s), io=40.0MiB (41.9MB), run=5300-5338msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=183772: Sat Jul 18 16:44:26 2026
  read: IOPS=188, BW=753KiB/s (771kB/s)(5048KiB/6704msec)
    slat (usec): min=1060, max=13218, avg=2933.57, stdev=1018.61
    clat (usec): min=7871, max=65123, avg=39943.13, stdev=7410.56
     lat (usec): min=10559, max=68752, avg=42878.30, stdev=7687.17
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[28967], 10.00th=[31065], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[37487], 50.00th=[39584], 60.00th=[41157],
     | 70.00th=[43779], 80.00th=[46400], 90.00th=[49546], 95.00th=[52167],
     | 99.00th=[58459], 99.50th=[60556], 99.90th=[62129], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  568, max=  896, per=25.12%, avg=751.69, stdev=104.72, samples=13
   iops        : min=  142, max=  224, avg=187.92, stdev=26.18, samples=13
  write: IOPS=193, BW=774KiB/s (793kB/s)(5192KiB/6704msec); 0 zone resets
    slat (usec): min=50, max=880, avg=178.59, stdev=89.87
    clat (usec): min=7647, max=62830, avg=40046.91, stdev=6519.77
     lat (usec): min=7859, max=62940, avg=40226.86, stdev=6515.56
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[30278], 10.00th=[32113], 20.00th=[34341],
     | 30.00th=[35914], 40.00th=[38011], 50.00th=[39584], 60.00th=[41681],
     | 70.00th=[43779], 80.00th=[45876], 90.00th=[49021], 95.00th=[50594],
     | 99.00th=[53740], 99.50th=[55837], 99.90th=[60031], 99.95th=[62653],
     | 99.99th=[62653]
   bw (  KiB/s): min=  598, max=  984, per=24.92%, avg=776.46, stdev=112.57, samples=13
   iops        : min=  149, max=  246, avg=194.08, stdev=28.21, samples=13
  lat (msec)   : 10=0.16%, 20=0.12%, 50=91.41%, 100=8.32%
  cpu          : usr=0.48%, sys=6.06%, ctx=4717, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183773: Sat Jul 18 16:44:26 2026
  read: IOPS=186, BW=744KiB/s (762kB/s)(4976KiB/6688msec)
    slat (usec): min=1086, max=13461, avg=2931.13, stdev=1022.07
    clat (usec): min=14929, max=78294, avg=40118.83, stdev=7262.59
     lat (usec): min=18834, max=85018, avg=43051.51, stdev=7496.66
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[28967], 10.00th=[31065], 20.00th=[33817],
     | 30.00th=[35914], 40.00th=[38011], 50.00th=[39584], 60.00th=[41157],
     | 70.00th=[43779], 80.00th=[45876], 90.00th=[49546], 95.00th=[52167],
     | 99.00th=[57410], 99.50th=[63177], 99.90th=[73925], 99.95th=[78119],
     | 99.99th=[78119]
   bw (  KiB/s): min=  592, max=  920, per=24.91%, avg=745.69, stdev=96.43, samples=13
   iops        : min=  148, max=  230, avg=186.38, stdev=24.17, samples=13
  write: IOPS=196, BW=787KiB/s (806kB/s)(5264KiB/6688msec); 0 zone resets
    slat (usec): min=56, max=688, avg=176.74, stdev=89.83
    clat (usec): min=23543, max=78117, avg=40077.42, stdev=6819.48
     lat (usec): min=23696, max=78284, avg=40255.40, stdev=6823.25
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[30278], 10.00th=[31851], 20.00th=[33817],
     | 30.00th=[35914], 40.00th=[37487], 50.00th=[39584], 60.00th=[41157],
     | 70.00th=[43779], 80.00th=[45876], 90.00th=[49021], 95.00th=[51119],
     | 99.00th=[57410], 99.50th=[58983], 99.90th=[69731], 99.95th=[78119],
     | 99.99th=[78119]
   bw (  KiB/s): min=  624, max= 1048, per=25.45%, avg=793.00, stdev=118.69, samples=13
   iops        : min=  156, max=  262, avg=198.23, stdev=29.69, samples=13
  lat (msec)   : 20=0.08%, 50=91.45%, 100=8.48%
  cpu          : usr=0.46%, sys=5.98%, ctx=4742, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183774: Sat Jul 18 16:44:26 2026
  read: IOPS=187, BW=749KiB/s (767kB/s)(4984KiB/6652msec)
    slat (usec): min=41, max=8972, avg=2926.99, stdev=980.66
    clat (usec): min=5267, max=72241, avg=39988.03, stdev=7485.13
     lat (usec): min=7683, max=75972, avg=42916.59, stdev=7774.39
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[29230], 10.00th=[31065], 20.00th=[33424],
     | 30.00th=[35390], 40.00th=[37487], 50.00th=[39584], 60.00th=[41157],
     | 70.00th=[43254], 80.00th=[45876], 90.00th=[50070], 95.00th=[53216],
     | 99.00th=[59507], 99.50th=[62653], 99.90th=[65799], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  637, max=  952, per=25.16%, avg=753.00, stdev=85.80, samples=13
   iops        : min=  159, max=  238, avg=188.23, stdev=21.48, samples=13
  write: IOPS=197, BW=790KiB/s (809kB/s)(5256KiB/6652msec); 0 zone resets
    slat (usec): min=43, max=3385, avg=184.05, stdev=128.74
    clat (usec): min=8786, max=79287, avg=39969.24, stdev=7243.93
     lat (usec): min=9075, max=79571, avg=40154.46, stdev=7246.43
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[30016], 10.00th=[31851], 20.00th=[33817],
     | 30.00th=[35914], 40.00th=[37487], 50.00th=[39060], 60.00th=[41157],
     | 70.00th=[43254], 80.00th=[45876], 90.00th=[49021], 95.00th=[51643],
     | 99.00th=[59507], 99.50th=[65274], 99.90th=[79168], 99.95th=[79168],
     | 99.99th=[79168]
   bw (  KiB/s): min=  608, max= 1048, per=25.33%, avg=789.15, stdev=139.48, samples=13
   iops        : min=  152, max=  262, avg=197.23, stdev=34.90, samples=13
  lat (msec)   : 10=0.12%, 20=0.16%, 50=91.09%, 100=8.63%
  cpu          : usr=0.51%, sys=6.03%, ctx=4739, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183775: Sat Jul 18 16:44:26 2026
  read: IOPS=188, BW=755KiB/s (773kB/s)(5060KiB/6702msec)
    slat (usec): min=1104, max=13517, avg=2953.90, stdev=1000.63
    clat (usec): min=22417, max=67588, avg=39926.34, stdev=7220.52
     lat (usec): min=25899, max=70970, avg=42881.76, stdev=7489.76
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[29230], 10.00th=[30802], 20.00th=[33424],
     | 30.00th=[35914], 40.00th=[37487], 50.00th=[39584], 60.00th=[41157],
     | 70.00th=[43779], 80.00th=[45876], 90.00th=[49021], 95.00th=[52691],
     | 99.00th=[57934], 99.50th=[58983], 99.90th=[67634], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  598, max=  912, per=25.33%, avg=758.00, stdev=100.35, samples=13
   iops        : min=  149, max=  228, avg=189.46, stdev=25.15, samples=13
  write: IOPS=193, BW=773KiB/s (791kB/s)(5180KiB/6702msec); 0 zone resets
    slat (usec): min=54, max=886, avg=177.68, stdev=86.92
    clat (usec): min=25066, max=69117, avg=40211.92, stdev=6667.24
     lat (usec): min=25181, max=69241, avg=40390.75, stdev=6669.02
    clat percentiles (usec):
     |  1.00th=[28181],  5.00th=[30540], 10.00th=[32113], 20.00th=[34341],
     | 30.00th=[35914], 40.00th=[37487], 50.00th=[39584], 60.00th=[41157],
     | 70.00th=[43779], 80.00th=[45876], 90.00th=[49546], 95.00th=[51643],
     | 99.00th=[57410], 99.50th=[60031], 99.90th=[66323], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  632, max=  920, per=24.81%, avg=773.08, stdev=96.54, samples=13
   iops        : min=  158, max=  230, avg=193.23, stdev=24.19, samples=13
  lat (msec)   : 50=91.60%, 100=8.40%
  cpu          : usr=0.33%, sys=6.27%, ctx=4752, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2993KiB/s (3065kB/s), 744KiB/s-755KiB/s (762kB/s-773kB/s), io=19.6MiB (20.5MB), run=6652-6704msec
  WRITE: bw=3116KiB/s (3191kB/s), 773KiB/s-790KiB/s (791kB/s-809kB/s), io=20.4MiB (21.4MB), run=6652-6704msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=183780: Sat Jul 18 16:44:29 2026
  read: IOPS=9, BW=392MiB/s (411MB/s)(40.0MiB/102msec)
    slat (nsec): min=12794k, max=12794k, avg=12794179.00, stdev= 0.00
    clat (nsec): min=84266k, max=84266k, avg=84265999.00, stdev= 0.00
     lat (nsec): min=97064k, max=97064k, avg=97064134.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[84411],  5.00th=[84411], 10.00th=[84411], 20.00th=[84411],
     | 30.00th=[84411], 40.00th=[84411], 50.00th=[84411], 60.00th=[84411],
     | 70.00th=[84411], 80.00th=[84411], 90.00th=[84411], 95.00th=[84411],
     | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411],
     | 99.99th=[84411]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=11.88%, ctx=10, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=392MiB/s (411MB/s), 392MiB/s-392MiB/s (411MB/s-411MB/s), io=40.0MiB (41.9MB), run=102-102msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout