== sanity test 398c: run fio to test AIO ================= 14:33:53 (1776191633) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.37423 s, 112 MB/s osc.lustre-OST0000-osc-ffff9041c31f5800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=123212: Tue Apr 14 14:36:32 2026 write: IOPS=16, BW=67.6KiB/s (69.2kB/s)(10.0MiB/151498msec); 0 zone resets slat (usec): min=23, max=18424, avg=288.69, stdev=941.62 clat (msec): min=203, max=2420, avg=946.05, stdev=288.91 lat (msec): min=204, max=2420, avg=946.35, stdev=288.90 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 709], | 30.00th=[ 785], 40.00th=[ 852], 50.00th=[ 911], 60.00th=[ 978], | 70.00th=[ 1070], 80.00th=[ 1150], 90.00th=[ 1318], 95.00th=[ 1401], | 99.00th=[ 1921], 99.50th=[ 2165], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 152, per=24.95%, avg=67.35, stdev=32.91, samples=299 iops : min= 1, max= 38, avg=16.33, stdev= 8.29, samples=299 lat (msec) : 250=0.04%, 500=2.70%, 750=22.50%, 1000=36.64%, 2000=37.30% lat (msec) : >=2000=0.82% cpu : usr=0.04%, sys=0.29%, ctx=1687, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=123213: Tue Apr 14 14:36:32 2026 write: IOPS=16, BW=67.7KiB/s (69.3kB/s)(10.0MiB/151290msec); 0 zone resets slat (usec): min=21, max=15035, avg=304.10, stdev=1012.22 clat (msec): min=260, max=2651, avg=944.54, stdev=292.46 lat (msec): min=260, max=2651, avg=944.85, stdev=292.44 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 550], 10.00th=[ 625], 20.00th=[ 709], | 30.00th=[ 785], 40.00th=[ 852], 50.00th=[ 911], 60.00th=[ 978], | 70.00th=[ 1070], 80.00th=[ 1167], 90.00th=[ 1284], 95.00th=[ 1401], | 99.00th=[ 1955], 99.50th=[ 2165], 99.90th=[ 2400], 99.95th=[ 2433], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 232, per=25.64%, avg=69.23, stdev=33.44, samples=291 iops : min= 1, max= 58, avg=16.81, stdev= 8.36, samples=291 lat (msec) : 500=3.09%, 750=22.97%, 1000=36.41%, 2000=36.56%, >=2000=0.98% cpu : usr=0.06%, sys=0.27%, ctx=1613, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=123214: Tue Apr 14 14:36:32 2026 write: IOPS=16, BW=67.6KiB/s (69.3kB/s)(10.0MiB/151412msec); 0 zone resets slat (usec): min=20, max=15859, avg=314.73, stdev=923.94 clat (msec): min=106, max=2582, avg=945.61, stdev=293.15 lat (msec): min=107, max=2595, avg=945.93, stdev=293.14 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 550], 10.00th=[ 617], 20.00th=[ 701], | 30.00th=[ 785], 40.00th=[ 852], 50.00th=[ 902], 60.00th=[ 986], | 70.00th=[ 1070], 80.00th=[ 1150], 90.00th=[ 1318], 95.00th=[ 1435], | 99.00th=[ 1871], 99.50th=[ 2165], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 2567] bw ( KiB/s): min= 7, max= 162, per=25.19%, avg=68.02, stdev=32.04, samples=296 iops : min= 1, max= 40, avg=16.50, stdev= 8.03, samples=296 lat (msec) : 250=0.47%, 500=2.34%, 750=23.05%, 1000=35.78%, 2000=37.50% lat (msec) : >=2000=0.86% cpu : usr=0.06%, sys=0.29%, ctx=1731, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=123215: Tue Apr 14 14:36:32 2026 write: IOPS=16, BW=67.6KiB/s (69.2kB/s)(10.0MiB/151498msec); 0 zone resets slat (usec): min=26, max=18982, avg=298.91, stdev=893.69 clat (msec): min=261, max=2650, avg=945.85, stdev=288.44 lat (msec): min=263, max=2650, avg=946.16, stdev=288.44 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 550], 10.00th=[ 625], 20.00th=[ 709], | 30.00th=[ 776], 40.00th=[ 844], 50.00th=[ 911], 60.00th=[ 986], | 70.00th=[ 1070], 80.00th=[ 1150], 90.00th=[ 1301], 95.00th=[ 1401], | 99.00th=[ 1972], 99.50th=[ 2140], 99.90th=[ 2400], 99.95th=[ 2601], | 99.99th=[ 2635] bw ( KiB/s): min= 7, max= 136, per=24.95%, avg=67.35, stdev=30.23, samples=299 iops : min= 1, max= 34, avg=16.33, stdev= 7.60, samples=299 lat (msec) : 500=3.20%, 750=22.62%, 1000=35.20%, 2000=38.09%, >=2000=0.90% cpu : usr=0.05%, sys=0.30%, ctx=1691, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=270KiB/s (277kB/s), 67.6KiB/s-67.7KiB/s (69.2kB/s-69.3kB/s), io=40.0MiB (41.9MB), run=151290-151498msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=123247: Tue Apr 14 14:38:10 2026 read: IOPS=13, BW=55.7KiB/s (57.0kB/s)(5048KiB/90671msec) slat (usec): min=40, max=50569, avg=10843.89, stdev=5805.37 clat (msec): min=11, max=728, avg=375.57, stdev=91.83 lat (msec): min=13, max=746, avg=386.42, stdev=92.68 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 247], 10.00th=[ 275], 20.00th=[ 300], | 30.00th=[ 326], 40.00th=[ 351], 50.00th=[ 376], 60.00th=[ 397], | 70.00th=[ 422], 80.00th=[ 447], 90.00th=[ 489], 95.00th=[ 518], | 99.00th=[ 617], 99.50th=[ 659], 99.90th=[ 701], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 7, max= 127, per=25.04%, avg=55.08, stdev=23.10, samples=180 iops : min= 1, max= 31, avg=13.25, stdev= 5.79, samples=180 write: IOPS=14, BW=57.3KiB/s (58.6kB/s)(5192KiB/90671msec); 0 zone resets slat (usec): min=21, max=11906, avg=393.39, stdev=920.67 clat (msec): min=93, max=1435, avg=740.80, stdev=202.46 lat (msec): min=95, max=1435, avg=741.20, stdev=202.49 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 443], 10.00th=[ 502], 20.00th=[ 575], | 30.00th=[ 625], 40.00th=[ 667], 50.00th=[ 718], 60.00th=[ 768], | 70.00th=[ 835], 80.00th=[ 911], 90.00th=[ 1020], 95.00th=[ 1116], | 99.00th=[ 1267], 99.50th=[ 1318], 99.90th=[ 1385], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 7, max= 136, per=24.66%, avg=56.47, stdev=21.53, samples=180 iops : min= 1, max= 34, avg=13.59, stdev= 5.42, samples=180 lat (msec) : 20=0.12%, 50=0.23%, 100=0.27%, 250=2.27%, 500=47.23% lat (msec) : 750=28.01%, 1000=16.17%, 2000=5.70% cpu : usr=0.14%, sys=0.79%, ctx=3182, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=123248: Tue Apr 14 14:38:10 2026 read: IOPS=13, BW=54.7KiB/s (56.1kB/s)(4976KiB/90889msec) slat (usec): min=35, max=38174, avg=10784.34, stdev=5579.00 clat (msec): min=7, max=766, avg=377.47, stdev=90.12 lat (msec): min=15, max=780, avg=388.26, stdev=90.64 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 251], 10.00th=[ 275], 20.00th=[ 305], | 30.00th=[ 330], 40.00th=[ 355], 50.00th=[ 372], 60.00th=[ 393], | 70.00th=[ 418], 80.00th=[ 447], 90.00th=[ 489], 95.00th=[ 518], | 99.00th=[ 634], 99.50th=[ 659], 99.90th=[ 701], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 15, max= 135, per=24.68%, avg=54.30, stdev=23.49, samples=179 iops : min= 3, max= 33, avg=13.06, stdev= 5.89, samples=179 write: IOPS=14, BW=57.9KiB/s (59.3kB/s)(5264KiB/90889msec); 0 zone resets slat (usec): min=21, max=8818, avg=367.55, stdev=905.82 clat (msec): min=197, max=1459, avg=737.09, stdev=202.02 lat (msec): min=198, max=1459, avg=737.46, stdev=202.02 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 435], 10.00th=[ 502], 20.00th=[ 567], | 30.00th=[ 625], 40.00th=[ 667], 50.00th=[ 718], 60.00th=[ 768], | 70.00th=[ 835], 80.00th=[ 911], 90.00th=[ 995], 95.00th=[ 1099], | 99.00th=[ 1267], 99.50th=[ 1334], 99.90th=[ 1452], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 15, max= 119, per=25.01%, avg=57.28, stdev=22.13, samples=180 iops : min= 3, max= 29, avg=13.80, stdev= 5.57, samples=180 lat (msec) : 10=0.08%, 20=0.12%, 100=0.16%, 250=2.23%, 500=47.11% lat (msec) : 750=27.85%, 1000=17.66%, 2000=4.80% cpu : usr=0.13%, sys=0.80%, ctx=3239, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=123249: Tue Apr 14 14:38:10 2026 read: IOPS=13, BW=54.8KiB/s (56.2kB/s)(4984KiB/90888msec) slat (usec): min=41, max=36663, avg=10841.77, stdev=5705.89 clat (msec): min=16, max=747, avg=377.95, stdev=89.14 lat (msec): min=22, max=766, avg=388.80, stdev=89.66 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 247], 10.00th=[ 279], 20.00th=[ 309], | 30.00th=[ 334], 40.00th=[ 351], 50.00th=[ 372], 60.00th=[ 397], | 70.00th=[ 422], 80.00th=[ 447], 90.00th=[ 493], 95.00th=[ 523], | 99.00th=[ 600], 99.50th=[ 634], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 7, max= 144, per=24.86%, avg=54.70, stdev=24.30, samples=179 iops : min= 1, max= 36, avg=13.16, stdev= 6.12, samples=179 write: IOPS=14, BW=57.8KiB/s (59.2kB/s)(5256KiB/90888msec); 0 zone resets slat (usec): min=27, max=11666, avg=326.67, stdev=832.04 clat (msec): min=96, max=1397, avg=736.98, stdev=201.82 lat (msec): min=97, max=1397, avg=737.30, stdev=201.78 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 435], 10.00th=[ 502], 20.00th=[ 558], | 30.00th=[ 617], 40.00th=[ 667], 50.00th=[ 718], 60.00th=[ 776], | 70.00th=[ 835], 80.00th=[ 911], 90.00th=[ 995], 95.00th=[ 1116], | 99.00th=[ 1234], 99.50th=[ 1334], 99.90th=[ 1385], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 7, max= 111, per=24.91%, avg=57.05, stdev=21.23, samples=180 iops : min= 1, max= 27, avg=13.74, stdev= 5.33, samples=180 lat (msec) : 20=0.04%, 50=0.08%, 100=0.27%, 250=2.38%, 500=46.91% lat (msec) : 750=27.97%, 1000=17.27%, 2000=5.08% cpu : usr=0.11%, sys=0.83%, ctx=3210, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=123250: Tue Apr 14 14:38:10 2026 read: IOPS=13, BW=55.9KiB/s (57.3kB/s)(5060KiB/90466msec) slat (usec): min=2346, max=35898, avg=10821.84, stdev=5706.94 clat (msec): min=11, max=695, avg=375.64, stdev=87.55 lat (msec): min=17, max=703, avg=386.47, stdev=88.07 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 251], 10.00th=[ 279], 20.00th=[ 305], | 30.00th=[ 330], 40.00th=[ 351], 50.00th=[ 376], 60.00th=[ 397], | 70.00th=[ 418], 80.00th=[ 443], 90.00th=[ 485], 95.00th=[ 518], | 99.00th=[ 592], 99.50th=[ 651], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 15, max= 143, per=25.71%, avg=56.55, stdev=24.09, samples=176 iops : min= 3, max= 35, avg=13.62, stdev= 6.02, samples=176 write: IOPS=14, BW=57.3KiB/s (58.6kB/s)(5180KiB/90466msec); 0 zone resets slat (usec): min=27, max=15501, avg=347.99, stdev=920.83 clat (msec): min=221, max=1476, avg=738.67, stdev=199.57 lat (msec): min=222, max=1476, avg=739.02, stdev=199.58 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 439], 10.00th=[ 502], 20.00th=[ 567], | 30.00th=[ 617], 40.00th=[ 667], 50.00th=[ 718], 60.00th=[ 768], | 70.00th=[ 827], 80.00th=[ 902], 90.00th=[ 1003], 95.00th=[ 1116], | 99.00th=[ 1318], 99.50th=[ 1385], 99.90th=[ 1452], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 7, max= 110, per=24.72%, avg=56.60, stdev=20.00, samples=179 iops : min= 1, max= 27, avg=13.63, stdev= 5.03, samples=179 lat (msec) : 20=0.04%, 50=0.08%, 100=0.35%, 250=1.95%, 500=48.24% lat (msec) : 750=27.27%, 1000=16.76%, 2000=5.31% cpu : usr=0.12%, sys=0.81%, ctx=3325, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=221KiB/s (226kB/s), 54.7KiB/s-55.9KiB/s (56.1kB/s-57.3kB/s), io=19.6MiB (20.5MB), run=90466-90889msec WRITE: bw=230KiB/s (235kB/s), 57.3KiB/s-57.9KiB/s (58.6kB/s-59.3kB/s), io=20.4MiB (21.4MB), run=90466-90889msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=123268: Tue Apr 14 14:38:18 2026 read: IOPS=2, BW=95.5MiB/s (100MB/s)(40.0MiB/419msec) slat (nsec): min=45114k, max=45114k, avg=45114012.00, stdev= 0.00 clat (nsec): min=364165k, max=364165k, avg=364164723.00, stdev= 0.00 lat (nsec): min=409282k, max=409282k, avg=409282257.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 363], | 30.00th=[ 363], 40.00th=[ 363], 50.00th=[ 363], 60.00th=[ 363], | 70.00th=[ 363], 80.00th=[ 363], 90.00th=[ 363], 95.00th=[ 363], | 99.00th=[ 363], 99.50th=[ 363], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.15%, ctx=2, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=95.5MiB/s (100MB/s), 95.5MiB/s-95.5MiB/s (100MB/s-100MB/s), io=40.0MiB (41.9MB), run=419-419msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck