== sanity test 398c: run fio to test AIO ================= 21:08:42 (1768010922) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.121081 s, 346 MB/s osc.lustre-OST0000-osc-ffff97deabc14000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=186848: Fri Jan 9 21:08:50 2026 write: IOPS=476, BW=1905KiB/s (1951kB/s)(10.0MiB/5374msec); 0 zone resets slat (usec): min=28, max=703, avg=87.94, stdev=47.21 clat (usec): min=23818, max=58337, avg=33104.26, stdev=5273.52 lat (usec): min=23867, max=58399, avg=33193.64, stdev=5274.85 clat percentiles (usec): | 1.00th=[25560], 5.00th=[26870], 10.00th=[27919], 20.00th=[28967], | 30.00th=[30016], 40.00th=[30540], 50.00th=[31851], 60.00th=[32900], | 70.00th=[34341], 80.00th=[36963], 90.00th=[41157], 95.00th=[43254], | 99.00th=[50070], 99.50th=[53216], 99.90th=[57410], 99.95th=[57934], | 99.99th=[58459] bw ( KiB/s): min= 1635, max= 2232, per=25.16%, avg=1916.30, stdev=193.18, samples=10 iops : min= 408, max= 558, avg=479.00, stdev=48.42, samples=10 lat (msec) : 50=99.10%, 100=0.90% cpu : usr=0.58%, sys=3.63%, ctx=2719, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=186849: Fri Jan 9 21:08:50 2026 write: IOPS=476, BW=1904KiB/s (1950kB/s)(10.0MiB/5378msec); 0 zone resets slat (usec): min=21, max=822, avg=86.60, stdev=48.57 clat (usec): min=22937, max=57859, avg=33117.68, stdev=5337.04 lat (usec): min=23032, max=57969, avg=33205.41, stdev=5338.74 clat percentiles (usec): | 1.00th=[25297], 5.00th=[26870], 10.00th=[27657], 20.00th=[28967], | 30.00th=[30016], 40.00th=[30802], 50.00th=[31589], 60.00th=[32900], | 70.00th=[34341], 80.00th=[36963], 90.00th=[40109], 95.00th=[43779], | 99.00th=[52167], 99.50th=[53216], 99.90th=[55837], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 1635, max= 2224, per=25.11%, avg=1912.30, stdev=190.06, samples=10 iops : min= 408, max= 556, avg=478.00, stdev=47.64, samples=10 lat (msec) : 50=98.36%, 100=1.64% cpu : usr=0.30%, sys=3.44%, ctx=2664, majf=1, minf=7 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=186850: Fri Jan 9 21:08:50 2026 write: IOPS=477, BW=1910KiB/s (1956kB/s)(10.0MiB/5361msec); 0 zone resets slat (usec): min=23, max=27888, avg=104.76, stdev=552.24 clat (usec): min=9795, max=58524, avg=33064.75, stdev=5517.08 lat (usec): min=13695, max=58571, avg=33170.60, stdev=5500.20 clat percentiles (usec): | 1.00th=[24249], 5.00th=[26870], 10.00th=[27657], 20.00th=[28967], | 30.00th=[30016], 40.00th=[30802], 50.00th=[31589], 60.00th=[32637], | 70.00th=[34341], 80.00th=[36963], 90.00th=[40633], 95.00th=[43779], | 99.00th=[51643], 99.50th=[54789], 99.90th=[57934], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 1696, max= 2248, per=25.22%, avg=1920.60, stdev=185.17, samples=10 iops : min= 424, max= 562, avg=480.10, stdev=46.35, samples=10 lat (msec) : 10=0.04%, 20=0.20%, 50=98.20%, 100=1.56% cpu : usr=0.30%, sys=4.27%, ctx=2803, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=186851: Fri Jan 9 21:08:50 2026 write: IOPS=480, BW=1922KiB/s (1968kB/s)(10.0MiB/5328msec); 0 zone resets slat (usec): min=26, max=2826, avg=92.49, stdev=81.89 clat (usec): min=10490, max=58224, avg=33134.59, stdev=5593.15 lat (usec): min=10553, max=58279, avg=33228.10, stdev=5594.36 clat percentiles (usec): | 1.00th=[25035], 5.00th=[26870], 10.00th=[27657], 20.00th=[28967], | 30.00th=[30016], 40.00th=[30540], 50.00th=[31589], 60.00th=[32900], | 70.00th=[34341], 80.00th=[36963], 90.00th=[41157], 95.00th=[44303], | 99.00th=[52691], 99.50th=[54789], 99.90th=[57934], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 1744, max= 2248, per=25.37%, avg=1932.50, stdev=168.86, samples=10 iops : min= 436, max= 562, avg=483.10, stdev=42.24, samples=10 lat (msec) : 20=0.12%, 50=98.20%, 100=1.68% cpu : usr=0.43%, sys=3.62%, ctx=2698, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=7616KiB/s (7799kB/s), 1904KiB/s-1922KiB/s (1950kB/s-1968kB/s), io=40.0MiB (41.9MB), run=5328-5378msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=186861: Fri Jan 9 21:08:59 2026 read: IOPS=220, BW=880KiB/s (901kB/s)(5048KiB/5734msec) slat (usec): min=968, max=8582, avg=2646.01, stdev=1007.95 clat (usec): min=3616, max=63804, avg=34099.35, stdev=7345.60 lat (usec): min=5036, max=66350, avg=36746.64, stdev=7739.76 clat percentiles (usec): | 1.00th=[21103], 5.00th=[24511], 10.00th=[26084], 20.00th=[28181], | 30.00th=[29754], 40.00th=[31065], 50.00th=[32637], 60.00th=[34341], | 70.00th=[36963], 80.00th=[40633], 90.00th=[44827], 95.00th=[47449], | 99.00th=[53740], 99.50th=[55837], 99.90th=[61604], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 704, max= 1072, per=25.00%, avg=874.91, stdev=131.61, samples=11 iops : min= 176, max= 268, avg=218.73, stdev=32.90, samples=11 write: IOPS=226, BW=905KiB/s (927kB/s)(5192KiB/5734msec); 0 zone resets slat (usec): min=29, max=605, avg=83.18, stdev=44.58 clat (usec): min=8408, max=61153, avg=34284.42, stdev=6710.25 lat (usec): min=8527, max=61323, avg=34368.46, stdev=6710.02 clat percentiles (usec): | 1.00th=[23987], 5.00th=[25822], 10.00th=[27132], 20.00th=[28967], | 30.00th=[30016], 40.00th=[31065], 50.00th=[32375], 60.00th=[34341], | 70.00th=[36963], 80.00th=[40633], 90.00th=[44303], 95.00th=[46924], | 99.00th=[51643], 99.50th=[54264], 99.90th=[57934], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 648, max= 1112, per=24.88%, avg=906.45, stdev=165.38, samples=11 iops : min= 162, max= 278, avg=226.55, stdev=41.41, samples=11 lat (msec) : 4=0.04%, 10=0.12%, 20=0.31%, 50=97.38%, 100=2.15% cpu : usr=0.38%, sys=4.13%, ctx=4706, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=186862: Fri Jan 9 21:08:59 2026 read: IOPS=217, BW=868KiB/s (889kB/s)(4976KiB/5732msec) slat (usec): min=891, max=8616, avg=2652.56, stdev=1006.08 clat (usec): min=10115, max=60035, avg=34160.43, stdev=7235.28 lat (usec): min=13120, max=63720, avg=36814.51, stdev=7592.81 clat percentiles (usec): | 1.00th=[21365], 5.00th=[24511], 10.00th=[26346], 20.00th=[28181], | 30.00th=[29754], 40.00th=[31065], 50.00th=[32375], 60.00th=[34341], | 70.00th=[36963], 80.00th=[41157], 90.00th=[44827], 95.00th=[47973], | 99.00th=[52167], 99.50th=[54264], 99.90th=[58983], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 688, max= 1088, per=24.82%, avg=868.55, stdev=133.84, samples=11 iops : min= 172, max= 272, avg=217.09, stdev=33.51, samples=11 write: IOPS=229, BW=918KiB/s (940kB/s)(5264KiB/5732msec); 0 zone resets slat (usec): min=21, max=839, avg=80.00, stdev=49.07 clat (usec): min=22584, max=56323, avg=34417.94, stdev=6543.32 lat (usec): min=22653, max=56380, avg=34498.73, stdev=6542.28 clat percentiles (usec): | 1.00th=[24511], 5.00th=[26084], 10.00th=[27395], 20.00th=[28705], | 30.00th=[30016], 40.00th=[31327], 50.00th=[32637], 60.00th=[34866], | 70.00th=[37487], 80.00th=[40109], 90.00th=[44303], 95.00th=[46924], | 99.00th=[52167], 99.50th=[53740], 99.90th=[55313], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 696, max= 1080, per=25.17%, avg=916.91, stdev=150.74, samples=11 iops : min= 174, max= 270, avg=229.18, stdev=37.69, samples=11 lat (msec) : 20=0.12%, 50=97.58%, 100=2.30% cpu : usr=0.40%, sys=4.07%, ctx=4716, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=186863: Fri Jan 9 21:08:59 2026 read: IOPS=219, BW=877KiB/s (898kB/s)(4984KiB/5684msec) slat (usec): min=34, max=8108, avg=2637.45, stdev=1029.11 clat (usec): min=3542, max=61283, avg=33826.19, stdev=7263.55 lat (usec): min=4750, max=63268, avg=36465.17, stdev=7684.93 clat percentiles (usec): | 1.00th=[20579], 5.00th=[24249], 10.00th=[26084], 20.00th=[28181], | 30.00th=[29492], 40.00th=[31065], 50.00th=[32375], 60.00th=[34341], | 70.00th=[36439], 80.00th=[40109], 90.00th=[44303], 95.00th=[47449], | 99.00th=[52691], 99.50th=[54789], 99.90th=[57410], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 640, max= 1064, per=25.10%, avg=878.09, stdev=147.60, samples=11 iops : min= 160, max= 266, avg=219.45, stdev=36.89, samples=11 write: IOPS=231, BW=925KiB/s (947kB/s)(5256KiB/5684msec); 0 zone resets slat (usec): min=26, max=1808, avg=83.00, stdev=66.84 clat (usec): min=14533, max=59737, avg=34412.72, stdev=6649.26 lat (usec): min=14691, max=59775, avg=34496.57, stdev=6652.13 clat percentiles (usec): | 1.00th=[23200], 5.00th=[25822], 10.00th=[27132], 20.00th=[28705], | 30.00th=[30016], 40.00th=[31327], 50.00th=[32900], 60.00th=[35390], | 70.00th=[37487], 80.00th=[40633], 90.00th=[44303], 95.00th=[46924], | 99.00th=[50070], 99.50th=[53740], 99.90th=[57934], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 712, max= 1120, per=25.37%, avg=924.18, stdev=127.12, samples=11 iops : min= 178, max= 280, avg=231.00, stdev=31.81, samples=11 lat (msec) : 4=0.04%, 10=0.16%, 20=0.39%, 50=97.54%, 100=1.88% cpu : usr=0.48%, sys=4.03%, ctx=4549, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=186864: Fri Jan 9 21:08:59 2026 read: IOPS=220, BW=883KiB/s (904kB/s)(5060KiB/5732msec) slat (usec): min=990, max=9391, avg=2652.89, stdev=997.03 clat (usec): min=19200, max=59073, avg=33996.85, stdev=6948.71 lat (usec): min=20852, max=62648, avg=36651.62, stdev=7370.66 clat percentiles (usec): | 1.00th=[21627], 5.00th=[24511], 10.00th=[26346], 20.00th=[28443], | 30.00th=[29754], 40.00th=[31065], 50.00th=[32375], 60.00th=[33817], | 70.00th=[36963], 80.00th=[40633], 90.00th=[44303], 95.00th=[46924], | 99.00th=[51643], 99.50th=[52691], 99.90th=[56361], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 656, max= 1040, per=25.23%, avg=882.82, stdev=130.12, samples=11 iops : min= 164, max= 260, avg=220.64, stdev=32.58, samples=11 write: IOPS=225, BW=904KiB/s (925kB/s)(5180KiB/5732msec); 0 zone resets slat (usec): min=25, max=1267, avg=81.78, stdev=55.88 clat (usec): min=20857, max=56766, avg=34299.35, stdev=6398.20 lat (usec): min=20909, max=56885, avg=34382.02, stdev=6398.68 clat percentiles (usec): | 1.00th=[24249], 5.00th=[25822], 10.00th=[27395], 20.00th=[28967], | 30.00th=[30278], 40.00th=[31327], 50.00th=[32900], 60.00th=[34341], | 70.00th=[36963], 80.00th=[40109], 90.00th=[43779], 95.00th=[46924], | 99.00th=[51643], 99.50th=[53216], 99.90th=[54789], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 616, max= 1144, per=24.72%, avg=900.64, stdev=147.67, samples=11 iops : min= 154, max= 286, avg=225.09, stdev=36.97, samples=11 lat (msec) : 20=0.20%, 50=97.93%, 100=1.88% cpu : usr=0.47%, sys=4.05%, ctx=4764, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3500KiB/s (3584kB/s), 868KiB/s-883KiB/s (889kB/s-904kB/s), io=19.6MiB (20.5MB), run=5684-5734msec WRITE: bw=3644KiB/s (3731kB/s), 904KiB/s-925KiB/s (925kB/s-947kB/s), io=20.4MiB (21.4MB), run=5684-5734msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=186868: Fri Jan 9 21:09:03 2026 read: IOPS=8, BW=333MiB/s (350MB/s)(40.0MiB/120msec) slat (nsec): min=9871.5k, max=9871.5k, avg=9871452.00, stdev= 0.00 clat (nsec): min=104546k, max=104546k, avg=104545942.00, stdev= 0.00 lat (nsec): min=114422k, max=114422k, avg=114422295.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 105], 10.00th=[ 105], 20.00th=[ 105], | 30.00th=[ 105], 40.00th=[ 105], 50.00th=[ 105], 60.00th=[ 105], | 70.00th=[ 105], 80.00th=[ 105], 90.00th=[ 105], 95.00th=[ 105], | 99.00th=[ 105], 99.50th=[ 105], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=8.40%, ctx=6, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=333MiB/s (350MB/s), 333MiB/s-333MiB/s (350MB/s-350MB/s), io=40.0MiB (41.9MB), run=120-120msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout