== sanity test 398c: run fio to test AIO ================= 16:58:13 (1768255093) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.44452 s, 94.4 MB/s osc.lustre-OST0000-osc-ffffa023091c1800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147747: Mon Jan 12 17:00:58 2026 write: IOPS=16, BW=65.4KiB/s (66.0kB/s)(10.0MiB/156612msec); 0 zone resets slat (usec): min=21, max=14369, avg=183.81, stdev=727.85 clat (msec): min=221, max=1990, avg=977.87, stdev=273.76 lat (msec): min=221, max=1990, avg=978.05, stdev=273.77 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 535], 10.00th=[ 667], 20.00th=[ 776], | 30.00th=[ 844], 40.00th=[ 919], 50.00th=[ 978], 60.00th=[ 1045], | 70.00th=[ 1116], 80.00th=[ 1183], 90.00th=[ 1334], 95.00th=[ 1435], | 99.00th=[ 1586], 99.50th=[ 1670], 99.90th=[ 1989], 99.95th=[ 1989], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 256, per=24.85%, avg=64.87, stdev=29.12, samples=311 iops : min= 1, max= 64, avg=15.81, stdev= 7.29, samples=311 lat (msec) : 250=1.88%, 500=2.58%, 750=13.44%, 1000=35.70%, 2000=46.41% cpu : usr=0.05%, sys=0.20%, ctx=2145, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147748: Mon Jan 12 17:00:58 2026 write: IOPS=16, BW=65.4KiB/s (66.9kB/s)(10.0MiB/156665msec); 0 zone resets slat (usec): min=21, max=16830, avg=177.87, stdev=758.38 clat (msec): min=217, max=2030, avg=978.61, stdev=277.35 lat (msec): min=217, max=2031, avg=978.79, stdev=277.37 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 542], 10.00th=[ 667], 20.00th=[ 768], | 30.00th=[ 835], 40.00th=[ 911], 50.00th=[ 986], 60.00th=[ 1045], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1334], 95.00th=[ 1435], | 99.00th=[ 1603], 99.50th=[ 1670], 99.90th=[ 1989], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 256, per=24.94%, avg=65.08, stdev=32.71, samples=310 iops : min= 1, max= 64, avg=15.87, stdev= 8.18, samples=310 lat (msec) : 250=1.29%, 500=2.89%, 750=13.67%, 1000=35.16%, 2000=46.91% lat (msec) : >=2000=0.08% cpu : usr=0.03%, sys=0.21%, ctx=1912, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147749: Mon Jan 12 17:00:58 2026 write: IOPS=16, BW=65.6KiB/s (67.1kB/s)(10.0MiB/156169msec); 0 zone resets slat (usec): min=22, max=9993, avg=169.78, stdev=582.81 clat (msec): min=74, max=2015, avg=975.48, stdev=276.40 lat (msec): min=75, max=2015, avg=975.66, stdev=276.39 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 542], 10.00th=[ 659], 20.00th=[ 768], | 30.00th=[ 844], 40.00th=[ 911], 50.00th=[ 978], 60.00th=[ 1045], | 70.00th=[ 1116], 80.00th=[ 1183], 90.00th=[ 1334], 95.00th=[ 1401], | 99.00th=[ 1569], 99.50th=[ 1653], 99.90th=[ 1989], 99.95th=[ 1989], | 99.99th=[ 2022] bw ( KiB/s): min= 8, max= 264, per=24.93%, avg=65.06, stdev=32.92, samples=310 iops : min= 2, max= 66, avg=15.86, stdev= 8.24, samples=310 lat (msec) : 100=0.04%, 250=2.03%, 500=2.23%, 750=13.40%, 1000=34.84% lat (msec) : 2000=47.42%, >=2000=0.04% cpu : usr=0.05%, sys=0.22%, ctx=2002, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147750: Mon Jan 12 17:00:58 2026 write: IOPS=16, BW=65.6KiB/s (67.1kB/s)(10.0MiB/156168msec); 0 zone resets slat (usec): min=22, max=12314, avg=152.46, stdev=553.09 clat (msec): min=193, max=2015, avg=975.05, stdev=274.46 lat (msec): min=193, max=2016, avg=975.21, stdev=274.46 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 567], 10.00th=[ 667], 20.00th=[ 768], | 30.00th=[ 844], 40.00th=[ 919], 50.00th=[ 978], 60.00th=[ 1045], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1334], 95.00th=[ 1418], | 99.00th=[ 1552], 99.50th=[ 1636], 99.90th=[ 1938], 99.95th=[ 1989], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 255, per=25.08%, avg=65.46, stdev=34.04, samples=308 iops : min= 1, max= 63, avg=15.96, stdev= 8.53, samples=308 lat (msec) : 250=2.50%, 500=1.52%, 750=14.73%, 1000=34.49%, 2000=46.72% lat (msec) : >=2000=0.04% cpu : usr=0.04%, sys=0.17%, ctx=1530, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=261KiB/s (268kB/s), 65.4KiB/s-65.6KiB/s (66.9kB/s-67.1kB/s), io=40.0MiB (41.9MB), run=156168-156665msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147793: Mon Jan 12 17:02:43 2026 read: IOPS=12, BW=51.6KiB/s (52.8kB/s)(5048KiB/97910msec) slat (usec): min=2536, max=49171, avg=12887.30, stdev=7192.40 clat (msec): min=19, max=786, avg=383.74, stdev=103.07 lat (msec): min=25, max=810, avg=396.63, stdev=104.36 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 228], 10.00th=[ 264], 20.00th=[ 300], | 30.00th=[ 326], 40.00th=[ 351], 50.00th=[ 376], 60.00th=[ 401], | 70.00th=[ 430], 80.00th=[ 468], 90.00th=[ 518], 95.00th=[ 550], | 99.00th=[ 651], 99.50th=[ 709], 99.90th=[ 768], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 7, max= 119, per=25.02%, avg=51.04, stdev=23.97, samples=194 iops : min= 1, max= 29, avg=12.32, stdev= 6.03, samples=194 write: IOPS=13, BW=53.0KiB/s (54.3kB/s)(5192KiB/97910msec); 0 zone resets slat (usec): min=30, max=14630, avg=242.86, stdev=783.69 clat (msec): min=210, max=1595, avg=818.15, stdev=233.15 lat (msec): min=210, max=1598, avg=818.39, stdev=233.13 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 456], 10.00th=[ 527], 20.00th=[ 609], | 30.00th=[ 684], 40.00th=[ 743], 50.00th=[ 810], 60.00th=[ 869], | 70.00th=[ 936], 80.00th=[ 1011], 90.00th=[ 1116], 95.00th=[ 1234], | 99.00th=[ 1401], 99.50th=[ 1552], 99.90th=[ 1586], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min= 7, max= 112, per=24.63%, avg=52.46, stdev=23.06, samples=194 iops : min= 1, max= 28, avg=12.67, stdev= 5.79, samples=194 lat (msec) : 20=0.04%, 50=0.20%, 100=0.08%, 250=3.63%, 500=42.81% lat (msec) : 750=23.01%, 1000=19.38%, 2000=10.86% cpu : usr=0.09%, sys=0.69%, ctx=4937, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147794: Mon Jan 12 17:02:43 2026 read: IOPS=12, BW=50.8KiB/s (52.0kB/s)(4976KiB/97912msec) slat (usec): min=53, max=55146, avg=12884.02, stdev=7309.59 clat (msec): min=14, max=762, avg=388.07, stdev=107.56 lat (msec): min=25, max=772, avg=400.95, stdev=108.75 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 232], 10.00th=[ 266], 20.00th=[ 300], | 30.00th=[ 334], 40.00th=[ 359], 50.00th=[ 384], 60.00th=[ 405], | 70.00th=[ 435], 80.00th=[ 472], 90.00th=[ 531], 95.00th=[ 567], | 99.00th=[ 651], 99.50th=[ 709], 99.90th=[ 735], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 7, max= 119, per=24.79%, avg=50.58, stdev=21.98, samples=193 iops : min= 1, max= 29, avg=12.20, stdev= 5.54, samples=193 write: IOPS=13, BW=53.8KiB/s (55.1kB/s)(5264KiB/97912msec); 0 zone resets slat (usec): min=26, max=14816, avg=249.50, stdev=837.07 clat (msec): min=221, max=1652, avg=810.41, stdev=242.97 lat (msec): min=221, max=1653, avg=810.67, stdev=242.97 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 435], 10.00th=[ 518], 20.00th=[ 600], | 30.00th=[ 667], 40.00th=[ 735], 50.00th=[ 793], 60.00th=[ 860], | 70.00th=[ 927], 80.00th=[ 1003], 90.00th=[ 1150], 95.00th=[ 1234], | 99.00th=[ 1469], 99.50th=[ 1519], 99.90th=[ 1569], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 7, max= 128, per=25.15%, avg=53.57, stdev=22.47, samples=193 iops : min= 1, max= 32, avg=12.95, stdev= 5.64, samples=193 lat (msec) : 20=0.04%, 50=0.27%, 100=0.23%, 250=3.40%, 500=41.80% lat (msec) : 750=25.39%, 1000=17.93%, 2000=10.94% cpu : usr=0.12%, sys=0.60%, ctx=4607, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147795: Mon Jan 12 17:02:43 2026 read: IOPS=12, BW=50.9KiB/s (52.1kB/s)(4984KiB/97911msec) slat (usec): min=2669, max=49097, avg=13173.05, stdev=7145.33 clat (msec): min=13, max=812, avg=387.49, stdev=102.90 lat (msec): min=34, max=818, avg=400.67, stdev=103.99 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 234], 10.00th=[ 262], 20.00th=[ 305], | 30.00th=[ 334], 40.00th=[ 359], 50.00th=[ 380], 60.00th=[ 405], | 70.00th=[ 439], 80.00th=[ 468], 90.00th=[ 523], 95.00th=[ 567], | 99.00th=[ 651], 99.50th=[ 701], 99.90th=[ 743], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 7, max= 120, per=24.92%, avg=50.83, stdev=23.67, samples=193 iops : min= 1, max= 30, avg=12.26, stdev= 5.96, samples=193 write: IOPS=13, BW=53.7KiB/s (54.0kB/s)(5256KiB/97911msec); 0 zone resets slat (usec): min=25, max=10909, avg=264.42, stdev=800.01 clat (msec): min=222, max=1646, avg=809.34, stdev=233.91 lat (msec): min=222, max=1646, avg=809.61, stdev=233.93 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 447], 10.00th=[ 514], 20.00th=[ 600], | 30.00th=[ 676], 40.00th=[ 743], 50.00th=[ 810], 60.00th=[ 860], | 70.00th=[ 919], 80.00th=[ 986], 90.00th=[ 1133], 95.00th=[ 1217], | 99.00th=[ 1418], 99.50th=[ 1502], 99.90th=[ 1569], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 7, max= 120, per=24.86%, avg=52.95, stdev=19.74, samples=194 iops : min= 1, max= 30, avg=12.79, stdev= 4.97, samples=194 lat (msec) : 20=0.04%, 50=0.08%, 100=0.23%, 250=3.44%, 500=42.85% lat (msec) : 750=22.70%, 1000=21.09%, 2000=9.57% cpu : usr=0.11%, sys=0.70%, ctx=4929, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147796: Mon Jan 12 17:02:43 2026 read: IOPS=12, BW=51.6KiB/s (52.9kB/s)(5060KiB/97970msec) slat (usec): min=2580, max=52098, avg=12960.32, stdev=7342.59 clat (msec): min=28, max=774, avg=392.46, stdev=104.40 lat (msec): min=35, max=785, avg=405.42, stdev=105.76 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 234], 10.00th=[ 271], 20.00th=[ 305], | 30.00th=[ 330], 40.00th=[ 359], 50.00th=[ 384], 60.00th=[ 414], | 70.00th=[ 443], 80.00th=[ 477], 90.00th=[ 527], 95.00th=[ 575], | 99.00th=[ 659], 99.50th=[ 718], 99.90th=[ 768], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 7, max= 144, per=25.21%, avg=51.43, stdev=22.61, samples=193 iops : min= 1, max= 36, avg=12.41, stdev= 5.71, samples=193 write: IOPS=13, BW=52.9KiB/s (54.1kB/s)(5180KiB/97970msec); 0 zone resets slat (usec): min=34, max=20196, avg=254.27, stdev=975.64 clat (msec): min=107, max=1578, avg=811.52, stdev=233.66 lat (msec): min=107, max=1578, avg=811.78, stdev=233.68 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 443], 10.00th=[ 523], 20.00th=[ 609], | 30.00th=[ 676], 40.00th=[ 743], 50.00th=[ 802], 60.00th=[ 869], | 70.00th=[ 936], 80.00th=[ 995], 90.00th=[ 1133], 95.00th=[ 1234], | 99.00th=[ 1368], 99.50th=[ 1418], 99.90th=[ 1569], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 15, max= 111, per=24.45%, avg=52.09, stdev=19.69, samples=194 iops : min= 3, max= 27, avg=12.58, stdev= 4.96, samples=194 lat (msec) : 50=0.08%, 100=0.08%, 250=3.28%, 500=42.66%, 750=24.34% lat (msec) : 1000=19.61%, 2000=9.96% cpu : usr=0.10%, sys=0.62%, ctx=4849, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=205KiB/s (210kB/s), 50.8KiB/s-51.6KiB/s (52.0kB/s-52.9kB/s), io=19.6MiB (20.5MB), run=97910-97970msec WRITE: bw=213KiB/s (218kB/s), 52.9KiB/s-53.8KiB/s (54.1kB/s-55.1kB/s), io=20.4MiB (21.4MB), run=97910-97970msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147814: Mon Jan 12 17:02:52 2026 read: IOPS=2, BW=91.3MiB/s (95.8MB/s)(40.0MiB/438msec) slat (nsec): min=17346k, max=17346k, avg=17346198.00, stdev= 0.00 clat (nsec): min=399867k, max=399867k, avg=399867481.00, stdev= 0.00 lat (nsec): min=417219k, max=417219k, avg=417218974.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 401], 10.00th=[ 401], 20.00th=[ 401], | 30.00th=[ 401], 40.00th=[ 401], 50.00th=[ 401], 60.00th=[ 401], | 70.00th=[ 401], 80.00th=[ 401], 90.00th=[ 401], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.29%, ctx=44, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=91.3MiB/s (95.8MB/s), 91.3MiB/s-91.3MiB/s (95.8MB/s-95.8MB/s), io=40.0MiB (41.9MB), run=438-438msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout