== sanity test 398c: run fio to test AIO ================= 08:20:26 (1775046026) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.140791 s, 298 MB/s osc.lustre-OST0000-osc-ffff9499b30c9800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=182297: Wed Apr 1 08:20:35 2026 write: IOPS=488, BW=1953KiB/s (2000kB/s)(10.0MiB/5243msec); 0 zone resets slat (usec): min=34, max=714, avg=151.49, stdev=72.93 clat (usec): min=6947, max=57631, avg=32563.83, stdev=3758.82 lat (usec): min=7124, max=57826, avg=32716.31, stdev=3773.17 clat percentiles (usec): | 1.00th=[24511], 5.00th=[27395], 10.00th=[28705], 20.00th=[30016], | 30.00th=[30540], 40.00th=[31589], 50.00th=[32375], 60.00th=[33162], | 70.00th=[34341], 80.00th=[34866], 90.00th=[36439], 95.00th=[38536], | 99.00th=[45351], 99.50th=[45876], 99.90th=[47973], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 1776, max= 2143, per=25.05%, avg=1944.70, stdev=111.64, samples=10 iops : min= 444, max= 535, avg=486.10, stdev=27.76, samples=10 lat (msec) : 10=0.04%, 20=0.51%, 50=99.38%, 100=0.08% cpu : usr=0.50%, sys=5.95%, ctx=2393, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182298: Wed Apr 1 08:20:35 2026 write: IOPS=485, BW=1942KiB/s (1989kB/s)(10.0MiB/5273msec); 0 zone resets slat (usec): min=34, max=2095, avg=139.44, stdev=80.98 clat (usec): min=2632, max=51295, avg=32520.06, stdev=3710.78 lat (usec): min=3051, max=51440, avg=32660.58, stdev=3720.30 clat percentiles (usec): | 1.00th=[24511], 5.00th=[27657], 10.00th=[28705], 20.00th=[29754], | 30.00th=[30802], 40.00th=[31589], 50.00th=[32375], 60.00th=[33162], | 70.00th=[33817], 80.00th=[34866], 90.00th=[36963], 95.00th=[38011], | 99.00th=[44827], 99.50th=[46924], 99.90th=[47973], 99.95th=[47973], | 99.99th=[51119] bw ( KiB/s): min= 1792, max= 2176, per=24.93%, avg=1935.60, stdev=110.92, samples=10 iops : min= 448, max= 544, avg=483.90, stdev=27.73, samples=10 lat (msec) : 4=0.04%, 10=0.04%, 20=0.39%, 50=99.49%, 100=0.04% cpu : usr=0.63%, sys=5.48%, ctx=2507, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182299: Wed Apr 1 08:20:35 2026 write: IOPS=486, BW=1947KiB/s (1993kB/s)(10.0MiB/5260msec); 0 zone resets slat (usec): min=36, max=19522, avg=164.26, stdev=402.38 clat (usec): min=7428, max=55472, avg=32568.93, stdev=3622.51 lat (usec): min=13347, max=55623, avg=32734.13, stdev=3596.45 clat percentiles (usec): | 1.00th=[24773], 5.00th=[27132], 10.00th=[28443], 20.00th=[30016], | 30.00th=[30802], 40.00th=[31589], 50.00th=[32113], 60.00th=[33162], | 70.00th=[33817], 80.00th=[34866], 90.00th=[36963], 95.00th=[39060], | 99.00th=[44303], 99.50th=[44827], 99.90th=[45351], 99.95th=[47449], | 99.99th=[55313] bw ( KiB/s): min= 1752, max= 2072, per=25.05%, avg=1944.90, stdev=106.73, samples=10 iops : min= 438, max= 518, avg=486.20, stdev=26.66, samples=10 lat (msec) : 10=0.04%, 20=0.12%, 50=99.80%, 100=0.04% cpu : usr=0.72%, sys=6.16%, ctx=2553, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182300: Wed Apr 1 08:20:35 2026 write: IOPS=485, BW=1941KiB/s (1988kB/s)(10.0MiB/5275msec); 0 zone resets slat (usec): min=34, max=1203, avg=163.12, stdev=81.58 clat (usec): min=9817, max=46349, avg=32492.61, stdev=3701.03 lat (usec): min=9927, max=46611, avg=32656.83, stdev=3712.86 clat percentiles (usec): | 1.00th=[22938], 5.00th=[27395], 10.00th=[28705], 20.00th=[30016], | 30.00th=[30802], 40.00th=[31327], 50.00th=[32375], 60.00th=[33162], | 70.00th=[33817], 80.00th=[34866], 90.00th=[36439], 95.00th=[38536], | 99.00th=[43779], 99.50th=[44827], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 1736, max= 2088, per=24.85%, avg=1929.70, stdev=100.42, samples=10 iops : min= 434, max= 522, avg=482.40, stdev=25.10, samples=10 lat (msec) : 10=0.08%, 20=0.35%, 50=99.57% cpu : usr=0.44%, sys=6.33%, ctx=2517, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=7765KiB/s (7951kB/s), 1941KiB/s-1953KiB/s (1988kB/s-2000kB/s), io=40.0MiB (41.9MB), run=5243-5275msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=182310: Wed Apr 1 08:20:43 2026 read: IOPS=234, BW=940KiB/s (962kB/s)(5048KiB/5373msec) slat (usec): min=1017, max=4852, avg=2218.56, stdev=617.93 clat (usec): min=6857, max=46643, avg=31929.56, stdev=4417.95 lat (usec): min=8794, max=50717, avg=34149.75, stdev=4512.62 clat percentiles (usec): | 1.00th=[22152], 5.00th=[25297], 10.00th=[26608], 20.00th=[28181], | 30.00th=[29754], 40.00th=[30802], 50.00th=[32113], 60.00th=[32900], | 70.00th=[33817], 80.00th=[35390], 90.00th=[37487], 95.00th=[39060], | 99.00th=[43254], 99.50th=[44303], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 800, max= 1032, per=25.09%, avg=936.80, stdev=69.48, samples=10 iops : min= 200, max= 258, avg=234.20, stdev=17.37, samples=10 write: IOPS=241, BW=966KiB/s (990kB/s)(5192KiB/5373msec); 0 zone resets slat (usec): min=46, max=688, avg=155.17, stdev=76.21 clat (usec): min=10107, max=48890, avg=32284.03, stdev=3678.70 lat (usec): min=10257, max=49057, avg=32440.81, stdev=3682.68 clat percentiles (usec): | 1.00th=[24773], 5.00th=[27132], 10.00th=[27919], 20.00th=[29230], | 30.00th=[30278], 40.00th=[31327], 50.00th=[32113], 60.00th=[32900], | 70.00th=[33817], 80.00th=[34866], 90.00th=[36963], 95.00th=[38536], | 99.00th=[43254], 99.50th=[44827], 99.90th=[48497], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 774, max= 1112, per=24.89%, avg=967.80, stdev=98.70, samples=10 iops : min= 193, max= 278, avg=241.90, stdev=24.79, samples=10 lat (msec) : 10=0.04%, 20=0.27%, 50=99.69% cpu : usr=0.63%, sys=6.46%, ctx=4520, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182311: Wed Apr 1 08:20:43 2026 read: IOPS=231, BW=926KiB/s (949kB/s)(4976KiB/5371msec) slat (usec): min=1009, max=4520, avg=2225.21, stdev=612.93 clat (usec): min=18851, max=50213, avg=31882.07, stdev=4315.94 lat (usec): min=21413, max=53062, avg=34108.60, stdev=4407.89 clat percentiles (usec): | 1.00th=[22676], 5.00th=[25035], 10.00th=[26346], 20.00th=[28181], | 30.00th=[29754], 40.00th=[30802], 50.00th=[31851], 60.00th=[32900], | 70.00th=[33817], 80.00th=[35390], 90.00th=[36963], 95.00th=[39060], | 99.00th=[43779], 99.50th=[44827], 99.90th=[47449], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 742, max= 1024, per=24.76%, avg=924.60, stdev=75.32, samples=10 iops : min= 185, max= 256, avg=231.10, stdev=18.96, samples=10 write: IOPS=245, BW=980KiB/s (1004kB/s)(5264KiB/5371msec); 0 zone resets slat (usec): min=35, max=1485, avg=155.08, stdev=80.67 clat (usec): min=23162, max=50482, avg=32343.18, stdev=3707.53 lat (usec): min=23233, max=50592, avg=32499.28, stdev=3708.63 clat percentiles (usec): | 1.00th=[25560], 5.00th=[26870], 10.00th=[27919], 20.00th=[29230], | 30.00th=[30278], 40.00th=[31065], 50.00th=[32113], 60.00th=[32900], | 70.00th=[33817], 80.00th=[34866], 90.00th=[36963], 95.00th=[38536], | 99.00th=[44827], 99.50th=[45876], 99.90th=[49021], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 857, max= 1120, per=25.29%, avg=983.30, stdev=77.70, samples=10 iops : min= 214, max= 280, avg=245.80, stdev=19.47, samples=10 lat (msec) : 20=0.08%, 50=99.84%, 100=0.08% cpu : usr=0.37%, sys=6.72%, ctx=4488, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182312: Wed Apr 1 08:20:43 2026 read: IOPS=232, BW=930KiB/s (953kB/s)(4984KiB/5357msec) slat (usec): min=1028, max=4880, avg=2221.44, stdev=594.01 clat (usec): min=3951, max=48233, avg=31906.33, stdev=4286.05 lat (usec): min=6011, max=50461, avg=34129.48, stdev=4368.03 clat percentiles (usec): | 1.00th=[22938], 5.00th=[25035], 10.00th=[26870], 20.00th=[28443], | 30.00th=[29754], 40.00th=[30802], 50.00th=[31851], 60.00th=[32900], | 70.00th=[33817], 80.00th=[35390], 90.00th=[36963], 95.00th=[39060], | 99.00th=[43779], 99.50th=[45351], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 832, max= 1040, per=25.09%, avg=936.80, stdev=83.09, samples=10 iops : min= 208, max= 260, avg=234.20, stdev=20.77, samples=10 write: IOPS=245, BW=981KiB/s (1005kB/s)(5256KiB/5357msec); 0 zone resets slat (usec): min=53, max=827, avg=161.61, stdev=88.54 clat (usec): min=22103, max=57051, avg=32273.48, stdev=3776.55 lat (usec): min=22289, max=57155, avg=32436.15, stdev=3778.06 clat percentiles (usec): | 1.00th=[25297], 5.00th=[27395], 10.00th=[28181], 20.00th=[29230], | 30.00th=[30278], 40.00th=[31065], 50.00th=[31851], 60.00th=[32637], | 70.00th=[33817], 80.00th=[34866], 90.00th=[36963], 95.00th=[38536], | 99.00th=[42206], 99.50th=[47973], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 832, max= 1144, per=25.12%, avg=976.80, stdev=97.06, samples=10 iops : min= 208, max= 286, avg=244.20, stdev=24.27, samples=10 lat (msec) : 4=0.04%, 10=0.04%, 20=0.08%, 50=99.69%, 100=0.16% cpu : usr=0.37%, sys=6.76%, ctx=4599, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182313: Wed Apr 1 08:20:43 2026 read: IOPS=236, BW=947KiB/s (970kB/s)(5060KiB/5344msec) slat (usec): min=54, max=6864, avg=2207.00, stdev=657.94 clat (usec): min=6193, max=58080, avg=31948.62, stdev=4801.79 lat (usec): min=6357, max=60952, avg=34157.19, stdev=4914.42 clat percentiles (usec): | 1.00th=[21365], 5.00th=[25035], 10.00th=[26608], 20.00th=[28443], | 30.00th=[29754], 40.00th=[30802], 50.00th=[31851], 60.00th=[32900], | 70.00th=[33817], 80.00th=[35390], 90.00th=[37487], 95.00th=[39584], | 99.00th=[45351], 99.50th=[47449], 99.90th=[49021], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 880, max= 1016, per=25.35%, avg=946.70, stdev=43.74, samples=10 iops : min= 220, max= 254, avg=236.60, stdev=11.00, samples=10 write: IOPS=242, BW=969KiB/s (993kB/s)(5180KiB/5344msec); 0 zone resets slat (usec): min=50, max=1901, avg=156.11, stdev=94.19 clat (usec): min=13613, max=61354, avg=32372.10, stdev=4048.39 lat (usec): min=13780, max=61683, avg=32529.24, stdev=4058.96 clat percentiles (usec): | 1.00th=[24511], 5.00th=[26870], 10.00th=[28181], 20.00th=[29230], | 30.00th=[30278], 40.00th=[31327], 50.00th=[32113], 60.00th=[32900], | 70.00th=[33817], 80.00th=[34866], 90.00th=[36963], 95.00th=[39584], | 99.00th=[44303], 99.50th=[46400], 99.90th=[61080], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 880, max= 1056, per=25.00%, avg=972.00, stdev=56.73, samples=10 iops : min= 220, max= 264, avg=243.00, stdev=14.18, samples=10 lat (msec) : 10=0.23%, 20=0.27%, 50=99.34%, 100=0.16% cpu : usr=0.58%, sys=6.51%, ctx=4561, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3735KiB/s (3825kB/s), 926KiB/s-947KiB/s (949kB/s-970kB/s), io=19.6MiB (20.5MB), run=5344-5373msec WRITE: bw=3888KiB/s (3982kB/s), 966KiB/s-981KiB/s (990kB/s-1005kB/s), io=20.4MiB (21.4MB), run=5344-5373msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=182318: Wed Apr 1 08:20:47 2026 read: IOPS=8, BW=323MiB/s (338MB/s)(40.0MiB/124msec) slat (nsec): min=13493k, max=13493k, avg=13492993.00, stdev= 0.00 clat (nsec): min=105009k, max=105009k, avg=105009043.00, stdev= 0.00 lat (nsec): min=118506k, max=118506k, avg=118505757.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 106], 10.00th=[ 106], 20.00th=[ 106], | 30.00th=[ 106], 40.00th=[ 106], 50.00th=[ 106], 60.00th=[ 106], | 70.00th=[ 106], 80.00th=[ 106], 90.00th=[ 106], 95.00th=[ 106], | 99.00th=[ 106], 99.50th=[ 106], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=8.94%, ctx=11, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=323MiB/s (338MB/s), 323MiB/s-323MiB/s (338MB/s-338MB/s), io=40.0MiB (41.9MB), run=124-124msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout