== sanity test 398c: run fio to test AIO ================= 12:15:30 (1788884130) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.140223 s, 299 MB/s osc.lustre-OST0000-osc-ffff987fd2c54800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=185296: Tue Sep 8 12:15:39 2026 write: IOPS=472, BW=1891KiB/s (1936kB/s)(10.0MiB/5415msec); 0 zone resets slat (usec): min=22, max=1145, avg=107.25, stdev=74.08 clat (usec): min=5745, max=44932, avg=33332.10, stdev=4695.81 lat (usec): min=5869, max=44991, avg=33440.48, stdev=4701.64 clat percentiles (usec): | 1.00th=[16712], 5.00th=[25035], 10.00th=[27919], 20.00th=[30802], | 30.00th=[31851], 40.00th=[32900], 50.00th=[33817], 60.00th=[34866], | 70.00th=[35914], 80.00th=[36439], 90.00th=[38536], 95.00th=[40109], | 99.00th=[42206], 99.50th=[44303], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 1736, max= 2061, per=24.55%, avg=1857.30, stdev=89.10, samples=10 iops : min= 434, max= 515, avg=464.30, stdev=22.21, samples=10 lat (msec) : 10=0.04%, 20=2.54%, 50=97.42% cpu : usr=0.59%, sys=3.93%, ctx=2721, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=185297: Tue Sep 8 12:15:39 2026 write: IOPS=475, BW=1902KiB/s (1947kB/s)(10.0MiB/5385msec); 0 zone resets slat (usec): min=27, max=663, avg=110.23, stdev=67.79 clat (usec): min=11634, max=47633, avg=33477.11, stdev=4891.05 lat (usec): min=11713, max=47699, avg=33588.52, stdev=4896.75 clat percentiles (usec): | 1.00th=[15008], 5.00th=[24249], 10.00th=[28443], 20.00th=[30802], | 30.00th=[31851], 40.00th=[32900], 50.00th=[33817], 60.00th=[34866], | 70.00th=[35914], 80.00th=[36963], 90.00th=[38536], 95.00th=[40109], | 99.00th=[43254], 99.50th=[46400], 99.90th=[46924], 99.95th=[46924], | 99.99th=[47449] bw ( KiB/s): min= 1784, max= 2247, per=24.83%, avg=1878.30, stdev=136.45, samples=10 iops : min= 446, max= 561, avg=469.50, stdev=33.89, samples=10 lat (msec) : 20=2.46%, 50=97.54% cpu : usr=0.61%, sys=4.05%, ctx=2733, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=185298: Tue Sep 8 12:15:39 2026 write: IOPS=475, BW=1903KiB/s (1949kB/s)(10.0MiB/5381msec); 0 zone resets slat (usec): min=24, max=2953, avg=108.89, stdev=90.29 clat (usec): min=11674, max=50105, avg=33394.53, stdev=4797.35 lat (usec): min=11762, max=50261, avg=33504.61, stdev=4804.02 clat percentiles (usec): | 1.00th=[15008], 5.00th=[25297], 10.00th=[28443], 20.00th=[30540], | 30.00th=[31851], 40.00th=[32900], 50.00th=[33817], 60.00th=[34866], | 70.00th=[35914], 80.00th=[36963], 90.00th=[38536], 95.00th=[39584], | 99.00th=[42206], 99.50th=[43254], 99.90th=[44827], 99.95th=[49546], | 99.99th=[50070] bw ( KiB/s): min= 1776, max= 2267, per=24.86%, avg=1880.30, stdev=143.81, samples=10 iops : min= 444, max= 566, avg=470.00, stdev=35.73, samples=10 lat (msec) : 20=2.70%, 50=97.27%, 100=0.04% cpu : usr=0.52%, sys=4.09%, ctx=2682, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=185299: Tue Sep 8 12:15:39 2026 write: IOPS=473, BW=1892KiB/s (1938kB/s)(10.0MiB/5411msec); 0 zone resets slat (usec): min=26, max=3125, avg=115.07, stdev=92.76 clat (usec): min=10727, max=44891, avg=33327.44, stdev=4624.10 lat (usec): min=10881, max=44953, avg=33443.66, stdev=4625.43 clat percentiles (usec): | 1.00th=[17957], 5.00th=[24773], 10.00th=[27919], 20.00th=[30540], | 30.00th=[31851], 40.00th=[32900], 50.00th=[33817], 60.00th=[34866], | 70.00th=[35914], 80.00th=[36963], 90.00th=[38536], 95.00th=[40109], | 99.00th=[41681], 99.50th=[43254], 99.90th=[44303], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 1776, max= 2067, per=24.62%, avg=1861.90, stdev=84.71, samples=10 iops : min= 444, max= 516, avg=465.40, stdev=20.98, samples=10 lat (msec) : 20=2.03%, 50=97.97% cpu : usr=0.54%, sys=4.88%, ctx=2919, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=7564KiB/s (7746kB/s), 1891KiB/s-1903KiB/s (1936kB/s-1949kB/s), io=40.0MiB (41.9MB), run=5381-5415msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=185310: Tue Sep 8 12:15:49 2026 read: IOPS=196, BW=785KiB/s (804kB/s)(5048KiB/6433msec) slat (usec): min=1071, max=14885, avg=2729.85, stdev=990.34 clat (usec): min=14874, max=63748, avg=38330.94, stdev=7110.10 lat (usec): min=15965, max=68974, avg=41062.32, stdev=7472.69 clat percentiles (usec): | 1.00th=[25560], 5.00th=[28705], 10.00th=[30802], 20.00th=[32637], | 30.00th=[34341], 40.00th=[35914], 50.00th=[36963], 60.00th=[38536], | 70.00th=[40633], 80.00th=[43254], 90.00th=[48497], 95.00th=[52167], | 99.00th=[58983], 99.50th=[61604], 99.90th=[62653], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 557, max= 904, per=25.04%, avg=781.08, stdev=117.42, samples=12 iops : min= 139, max= 226, avg=195.25, stdev=29.40, samples=12 write: IOPS=201, BW=807KiB/s (826kB/s)(5192KiB/6433msec); 0 zone resets slat (usec): min=36, max=583, avg=97.05, stdev=60.79 clat (usec): min=5154, max=61547, avg=38466.12, stdev=6605.06 lat (usec): min=5212, max=61645, avg=38564.36, stdev=6603.20 clat percentiles (usec): | 1.00th=[27919], 5.00th=[30540], 10.00th=[31589], 20.00th=[33424], | 30.00th=[34341], 40.00th=[35914], 50.00th=[36963], 60.00th=[38011], | 70.00th=[40109], 80.00th=[43779], 90.00th=[49021], 95.00th=[51643], | 99.00th=[56886], 99.50th=[58983], 99.90th=[61080], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 552, max= 952, per=24.78%, avg=804.67, stdev=135.44, samples=12 iops : min= 138, max= 238, avg=201.17, stdev=33.86, samples=12 lat (msec) : 10=0.04%, 20=0.27%, 50=91.91%, 100=7.77% cpu : usr=0.54%, sys=4.09%, ctx=4962, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=185311: Tue Sep 8 12:15:49 2026 read: IOPS=193, BW=774KiB/s (792kB/s)(4976KiB/6430msec) slat (usec): min=1015, max=7529, avg=2720.45, stdev=925.54 clat (usec): min=19660, max=66226, avg=38372.84, stdev=7016.55 lat (usec): min=22450, max=71022, avg=41094.72, stdev=7337.90 clat percentiles (usec): | 1.00th=[26084], 5.00th=[28443], 10.00th=[30540], 20.00th=[32637], | 30.00th=[34341], 40.00th=[35914], 50.00th=[36963], 60.00th=[39060], | 70.00th=[40633], 80.00th=[43254], 90.00th=[48497], 95.00th=[52167], | 99.00th=[57934], 99.50th=[61080], 99.90th=[64750], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 544, max= 896, per=24.64%, avg=768.42, stdev=129.05, samples=12 iops : min= 136, max= 224, avg=192.08, stdev=32.30, samples=12 write: IOPS=204, BW=819KiB/s (838kB/s)(5264KiB/6430msec); 0 zone resets slat (usec): min=24, max=2094, avg=99.10, stdev=81.53 clat (usec): min=19558, max=64531, avg=38641.74, stdev=6684.28 lat (usec): min=19650, max=64610, avg=38741.95, stdev=6684.35 clat percentiles (usec): | 1.00th=[28443], 5.00th=[30540], 10.00th=[31589], 20.00th=[33424], | 30.00th=[34866], 40.00th=[35914], 50.00th=[36963], 60.00th=[38536], | 70.00th=[40109], 80.00th=[43779], 90.00th=[48497], 95.00th=[52167], | 99.00th=[58459], 99.50th=[62129], 99.90th=[64226], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 602, max= 1016, per=25.36%, avg=823.50, stdev=127.52, samples=12 iops : min= 150, max= 254, avg=205.83, stdev=31.96, samples=12 lat (msec) : 20=0.08%, 50=92.46%, 100=7.46% cpu : usr=0.42%, sys=4.22%, ctx=4982, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=185312: Tue Sep 8 12:15:49 2026 read: IOPS=193, BW=776KiB/s (794kB/s)(4984KiB/6424msec) slat (usec): min=1087, max=13865, avg=2730.79, stdev=997.97 clat (usec): min=22154, max=65605, avg=38355.82, stdev=7135.33 lat (usec): min=23979, max=68401, avg=41088.09, stdev=7493.91 clat percentiles (usec): | 1.00th=[25560], 5.00th=[28705], 10.00th=[30278], 20.00th=[32637], | 30.00th=[34341], 40.00th=[35914], 50.00th=[36963], 60.00th=[38536], | 70.00th=[40633], 80.00th=[43779], 90.00th=[49021], 95.00th=[53216], | 99.00th=[58459], 99.50th=[60031], 99.90th=[62653], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 564, max= 928, per=24.98%, avg=779.00, stdev=110.79, samples=12 iops : min= 141, max= 232, avg=194.75, stdev=27.70, samples=12 write: IOPS=204, BW=818KiB/s (838kB/s)(5256KiB/6424msec); 0 zone resets slat (usec): min=33, max=863, avg=94.73, stdev=55.76 clat (usec): min=23901, max=64822, avg=38569.58, stdev=6730.83 lat (usec): min=24026, max=64898, avg=38665.49, stdev=6731.45 clat percentiles (usec): | 1.00th=[27657], 5.00th=[30278], 10.00th=[31851], 20.00th=[33162], | 30.00th=[34341], 40.00th=[35390], 50.00th=[36963], 60.00th=[38011], | 70.00th=[40109], 80.00th=[43779], 90.00th=[49021], 95.00th=[52167], | 99.00th=[57934], 99.50th=[61080], 99.90th=[63177], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 592, max= 1040, per=25.08%, avg=814.33, stdev=157.15, samples=12 iops : min= 148, max= 260, avg=203.58, stdev=39.29, samples=12 lat (msec) : 50=92.07%, 100=7.93% cpu : usr=0.48%, sys=4.17%, ctx=5002, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=185313: Tue Sep 8 12:15:49 2026 read: IOPS=197, BW=790KiB/s (809kB/s)(5060KiB/6403msec) slat (usec): min=55, max=14895, avg=2704.74, stdev=1002.22 clat (usec): min=3538, max=66556, avg=38478.46, stdev=7441.06 lat (usec): min=3703, max=70318, avg=41184.55, stdev=7795.53 clat percentiles (usec): | 1.00th=[25297], 5.00th=[28443], 10.00th=[30278], 20.00th=[32637], | 30.00th=[34866], 40.00th=[35914], 50.00th=[37487], 60.00th=[39060], | 70.00th=[40633], 80.00th=[43779], 90.00th=[49021], 95.00th=[52691], | 99.00th=[59507], 99.50th=[62129], 99.90th=[63701], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 568, max= 928, per=25.23%, avg=786.83, stdev=122.12, samples=12 iops : min= 142, max= 232, avg=196.67, stdev=30.59, samples=12 write: IOPS=202, BW=809KiB/s (828kB/s)(5180KiB/6403msec); 0 zone resets slat (usec): min=28, max=634, avg=97.76, stdev=58.85 clat (usec): min=13983, max=68698, avg=38623.48, stdev=6800.79 lat (usec): min=14160, max=68765, avg=38722.34, stdev=6801.55 clat percentiles (usec): | 1.00th=[27395], 5.00th=[30278], 10.00th=[31851], 20.00th=[33424], | 30.00th=[34866], 40.00th=[35914], 50.00th=[36963], 60.00th=[38536], | 70.00th=[40633], 80.00th=[43779], 90.00th=[48497], 95.00th=[52167], | 99.00th=[58983], 99.50th=[60556], 99.90th=[67634], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 592, max= 992, per=25.00%, avg=811.67, stdev=107.97, samples=12 iops : min= 148, max= 248, avg=202.92, stdev=26.99, samples=12 lat (msec) : 4=0.04%, 10=0.12%, 20=0.20%, 50=91.52%, 100=8.12% cpu : usr=0.42%, sys=4.28%, ctx=4855, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3120KiB/s (3194kB/s), 774KiB/s-790KiB/s (792kB/s-809kB/s), io=19.6MiB (20.5MB), run=6403-6433msec WRITE: bw=3248KiB/s (3326kB/s), 807KiB/s-819KiB/s (826kB/s-838kB/s), io=20.4MiB (21.4MB), run=6403-6433msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=185317: Tue Sep 8 12:15:52 2026 read: IOPS=8, BW=331MiB/s (347MB/s)(40.0MiB/121msec) slat (nsec): min=10911k, max=10911k, avg=10910807.00, stdev= 0.00 clat (nsec): min=104861k, max=104861k, avg=104861205.00, stdev= 0.00 lat (nsec): min=115775k, max=115775k, avg=115775473.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 106], 10.00th=[ 106], 20.00th=[ 106], | 30.00th=[ 106], 40.00th=[ 106], 50.00th=[ 106], 60.00th=[ 106], | 70.00th=[ 106], 80.00th=[ 106], 90.00th=[ 106], 95.00th=[ 106], | 99.00th=[ 106], 99.50th=[ 106], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=8.33%, ctx=10, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=331MiB/s (347MB/s), 331MiB/s-331MiB/s (347MB/s-347MB/s), io=40.0MiB (41.9MB), run=121-121msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout