== sanity test 398c: run fio to test AIO ================= 23:58:52 (1787371132) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.473732 s, 88.5 MB/s osc.lustre-OST0000-osc-ffff93e090daa000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145709: Sat Aug 22 00:01:50 2026 write: IOPS=15, BW=61.6KiB/s (63.1kB/s)(10.0MiB/166194msec); 0 zone resets slat (usec): min=21, max=14353, avg=156.67, stdev=644.47 clat (msec): min=190, max=2426, avg=1037.78, stdev=292.51 lat (msec): min=190, max=2426, avg=1037.94, stdev=292.53 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 617], 10.00th=[ 709], 20.00th=[ 810], | 30.00th=[ 877], 40.00th=[ 953], 50.00th=[ 1020], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1401], 95.00th=[ 1519], | 99.00th=[ 1871], 99.50th=[ 1989], 99.90th=[ 2333], 99.95th=[ 2400], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 168, per=25.36%, avg=62.38, stdev=32.39, samples=323 iops : min= 1, max= 42, avg=15.14, stdev= 8.09, samples=323 lat (msec) : 250=0.62%, 500=3.32%, 750=9.73%, 1000=33.16%, 2000=52.70% lat (msec) : >=2000=0.47% cpu : usr=0.03%, sys=0.19%, ctx=1529, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145710: Sat Aug 22 00:01:50 2026 write: IOPS=15, BW=61.7KiB/s (63.2kB/s)(10.0MiB/166041msec); 0 zone resets slat (usec): min=21, max=17993, avg=149.23, stdev=559.91 clat (msec): min=187, max=2427, avg=1037.05, stdev=297.71 lat (msec): min=187, max=2427, avg=1037.20, stdev=297.75 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 609], 10.00th=[ 718], 20.00th=[ 802], | 30.00th=[ 877], 40.00th=[ 944], 50.00th=[ 1020], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1401], 95.00th=[ 1586], | 99.00th=[ 1754], 99.50th=[ 1938], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 255, per=25.35%, avg=62.37, stdev=31.12, samples=323 iops : min= 1, max= 63, avg=15.14, stdev= 7.80, samples=323 lat (msec) : 250=0.62%, 500=3.48%, 750=8.32%, 1000=35.08%, 2000=52.15% lat (msec) : >=2000=0.35% cpu : usr=0.04%, sys=0.19%, ctx=1524, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145711: Sat Aug 22 00:01:50 2026 write: IOPS=15, BW=61.6KiB/s (63.1kB/s)(10.0MiB/166150msec); 0 zone resets slat (usec): min=22, max=28746, avg=203.50, stdev=1023.09 clat (msec): min=87, max=2420, avg=1037.16, stdev=290.78 lat (msec): min=87, max=2420, avg=1037.36, stdev=290.79 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 617], 10.00th=[ 726], 20.00th=[ 810], | 30.00th=[ 877], 40.00th=[ 953], 50.00th=[ 1011], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1401], 95.00th=[ 1536], | 99.00th=[ 1854], 99.50th=[ 2056], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 136, per=24.89%, avg=61.23, stdev=26.87, samples=329 iops : min= 1, max= 34, avg=14.86, stdev= 6.73, samples=329 lat (msec) : 100=0.04%, 250=0.04%, 500=3.71%, 750=8.44%, 1000=35.66% lat (msec) : 2000=51.60%, >=2000=0.51% cpu : usr=0.04%, sys=0.20%, ctx=1638, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145712: Sat Aug 22 00:01:50 2026 write: IOPS=15, BW=61.6KiB/s (63.1kB/s)(10.0MiB/166193msec); 0 zone resets slat (usec): min=22, max=30319, avg=192.18, stdev=1049.04 clat (msec): min=100, max=2421, avg=1037.52, stdev=290.31 lat (msec): min=100, max=2422, avg=1037.72, stdev=290.27 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 609], 10.00th=[ 726], 20.00th=[ 810], | 30.00th=[ 894], 40.00th=[ 953], 50.00th=[ 1028], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1401], 95.00th=[ 1536], | 99.00th=[ 1804], 99.50th=[ 1989], 99.90th=[ 2232], 99.95th=[ 2400], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 159, per=25.04%, avg=61.61, stdev=32.86, samples=327 iops : min= 1, max= 39, avg=14.95, stdev= 8.25, samples=327 lat (msec) : 250=0.20%, 500=3.91%, 750=8.87%, 1000=33.40%, 2000=53.24% lat (msec) : >=2000=0.39% cpu : usr=0.05%, sys=0.17%, ctx=1570, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=246KiB/s (252kB/s), 61.6KiB/s-61.7KiB/s (63.1kB/s-63.2kB/s), io=40.0MiB (41.9MB), run=166041-166194msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145762: Sat Aug 22 00:03:40 2026 read: IOPS=12, BW=50.9KiB/s (52.1kB/s)(5048KiB/99215msec) slat (usec): min=33, max=50160, avg=13224.01, stdev=7426.28 clat (msec): min=10, max=1264, avg=387.36, stdev=117.03 lat (msec): min=16, max=1276, avg=400.59, stdev=118.49 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 243], 10.00th=[ 266], 20.00th=[ 305], | 30.00th=[ 330], 40.00th=[ 355], 50.00th=[ 372], 60.00th=[ 393], | 70.00th=[ 426], 80.00th=[ 464], 90.00th=[ 527], 95.00th=[ 575], | 99.00th=[ 785], 99.50th=[ 827], 99.90th=[ 1200], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 7, max= 120, per=25.10%, avg=50.46, stdev=23.24, samples=196 iops : min= 1, max= 30, avg=12.12, stdev= 5.88, samples=196 write: IOPS=13, BW=52.3KiB/s (53.6kB/s)(5192KiB/99215msec); 0 zone resets slat (usec): min=29, max=26625, avg=242.59, stdev=1222.07 clat (msec): min=141, max=1791, avg=831.96, stdev=225.22 lat (msec): min=141, max=1792, avg=832.20, stdev=225.21 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 535], 10.00th=[ 592], 20.00th=[ 659], | 30.00th=[ 709], 40.00th=[ 751], 50.00th=[ 802], 60.00th=[ 860], | 70.00th=[ 919], 80.00th=[ 995], 90.00th=[ 1116], 95.00th=[ 1267], | 99.00th=[ 1519], 99.50th=[ 1603], 99.90th=[ 1787], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 7, max= 112, per=24.90%, avg=52.03, stdev=20.12, samples=195 iops : min= 1, max= 28, avg=12.51, stdev= 5.06, samples=195 lat (msec) : 20=0.04%, 50=0.12%, 100=0.31%, 250=2.62%, 500=41.17% lat (msec) : 750=24.65%, 1000=21.17%, 2000=9.92% cpu : usr=0.10%, sys=0.65%, ctx=4242, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145763: Sat Aug 22 00:03:40 2026 read: IOPS=12, BW=49.8KiB/s (51.0kB/s)(4976KiB/99820msec) slat (msec): min=2, max=432, avg=14.10, stdev=14.12 clat (msec): min=21, max=955, avg=386.04, stdev=108.31 lat (msec): min=26, max=976, avg=400.16, stdev=109.08 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 243], 10.00th=[ 266], 20.00th=[ 305], | 30.00th=[ 334], 40.00th=[ 351], 50.00th=[ 372], 60.00th=[ 401], | 70.00th=[ 426], 80.00th=[ 464], 90.00th=[ 531], 95.00th=[ 567], | 99.00th=[ 751], 99.50th=[ 835], 99.90th=[ 927], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 7, max= 127, per=24.72%, avg=49.69, stdev=23.28, samples=196 iops : min= 1, max= 31, avg=11.91, stdev= 5.90, samples=196 write: IOPS=13, BW=52.7KiB/s (54.0kB/s)(5264KiB/99820msec); 0 zone resets slat (usec): min=30, max=13700, avg=214.79, stdev=745.25 clat (msec): min=130, max=1689, avg=832.30, stdev=217.05 lat (msec): min=130, max=1689, avg=832.51, stdev=217.10 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 531], 10.00th=[ 592], 20.00th=[ 659], | 30.00th=[ 718], 40.00th=[ 760], 50.00th=[ 802], 60.00th=[ 852], | 70.00th=[ 919], 80.00th=[ 986], 90.00th=[ 1116], 95.00th=[ 1200], | 99.00th=[ 1485], 99.50th=[ 1620], 99.90th=[ 1687], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 96, per=25.14%, avg=52.55, stdev=20.45, samples=197 iops : min= 1, max= 24, avg=12.63, stdev= 5.13, samples=197 lat (msec) : 50=0.27%, 100=0.12%, 250=2.89%, 500=40.47%, 750=23.55% lat (msec) : 1000=23.16%, 2000=9.53% cpu : usr=0.11%, sys=0.63%, ctx=4343, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145764: Sat Aug 22 00:03:40 2026 read: IOPS=12, BW=49.9KiB/s (51.1kB/s)(4984KiB/99819msec) slat (usec): min=2605, max=61876, avg=13909.44, stdev=7774.11 clat (msec): min=22, max=898, avg=394.98, stdev=110.34 lat (msec): min=27, max=908, avg=408.89, stdev=111.78 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 253], 10.00th=[ 279], 20.00th=[ 309], | 30.00th=[ 334], 40.00th=[ 355], 50.00th=[ 380], 60.00th=[ 401], | 70.00th=[ 435], 80.00th=[ 472], 90.00th=[ 535], 95.00th=[ 584], | 99.00th=[ 802], 99.50th=[ 827], 99.90th=[ 852], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 7, max= 111, per=24.75%, avg=49.75, stdev=20.94, samples=197 iops : min= 1, max= 27, avg=11.93, stdev= 5.28, samples=197 write: IOPS=13, BW=52.7KiB/s (53.9kB/s)(5256KiB/99819msec); 0 zone resets slat (usec): min=32, max=12424, avg=230.01, stdev=751.92 clat (msec): min=170, max=1693, avg=824.90, stdev=209.81 lat (msec): min=172, max=1693, avg=825.13, stdev=209.80 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 535], 10.00th=[ 592], 20.00th=[ 659], | 30.00th=[ 709], 40.00th=[ 751], 50.00th=[ 793], 60.00th=[ 852], | 70.00th=[ 911], 80.00th=[ 986], 90.00th=[ 1099], 95.00th=[ 1200], | 99.00th=[ 1469], 99.50th=[ 1569], 99.90th=[ 1670], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 127, per=24.85%, avg=51.93, stdev=21.91, samples=198 iops : min= 1, max= 31, avg=12.48, stdev= 5.49, samples=198 lat (msec) : 50=0.04%, 100=0.04%, 250=2.30%, 500=40.43%, 750=25.20% lat (msec) : 1000=22.89%, 2000=9.10% cpu : usr=0.08%, sys=0.62%, ctx=4330, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145765: Sat Aug 22 00:03:40 2026 read: IOPS=12, BW=50.7KiB/s (51.9kB/s)(5060KiB/99822msec) slat (usec): min=2603, max=62993, avg=13937.91, stdev=8127.87 clat (msec): min=10, max=855, avg=392.39, stdev=107.75 lat (msec): min=14, max=890, avg=406.34, stdev=108.92 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 247], 10.00th=[ 271], 20.00th=[ 309], | 30.00th=[ 338], 40.00th=[ 359], 50.00th=[ 384], 60.00th=[ 405], | 70.00th=[ 430], 80.00th=[ 468], 90.00th=[ 527], 95.00th=[ 575], | 99.00th=[ 743], 99.50th=[ 827], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 7, max= 104, per=25.32%, avg=50.89, stdev=20.33, samples=195 iops : min= 1, max= 26, avg=12.22, stdev= 5.12, samples=195 write: IOPS=12, BW=51.9KiB/s (53.1kB/s)(5180KiB/99822msec); 0 zone resets slat (usec): min=24, max=12499, avg=212.72, stdev=653.54 clat (msec): min=180, max=1730, avg=833.32, stdev=221.09 lat (msec): min=180, max=1730, avg=833.54, stdev=221.10 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 531], 10.00th=[ 592], 20.00th=[ 659], | 30.00th=[ 709], 40.00th=[ 760], 50.00th=[ 802], 60.00th=[ 852], | 70.00th=[ 919], 80.00th=[ 995], 90.00th=[ 1116], 95.00th=[ 1234], | 99.00th=[ 1586], 99.50th=[ 1636], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 7, max= 111, per=24.96%, avg=52.16, stdev=21.10, samples=195 iops : min= 1, max= 27, avg=12.54, stdev= 5.30, samples=195 lat (msec) : 20=0.04%, 50=0.04%, 100=0.23%, 250=2.50%, 500=41.64% lat (msec) : 750=23.48%, 1000=22.70%, 2000=9.38% cpu : usr=0.12%, sys=0.62%, ctx=4240, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=201KiB/s (206kB/s), 49.8KiB/s-50.9KiB/s (51.0kB/s-52.1kB/s), io=19.6MiB (20.5MB), run=99215-99822msec WRITE: bw=209KiB/s (214kB/s), 51.9KiB/s-52.7KiB/s (53.1kB/s-54.0kB/s), io=20.4MiB (21.4MB), run=99215-99822msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145783: Sat Aug 22 00:03:50 2026 read: IOPS=2, BW=90.7MiB/s (95.1MB/s)(40.0MiB/441msec) slat (nsec): min=36987k, max=36987k, avg=36987225.00, stdev= 0.00 clat (nsec): min=390923k, max=390923k, avg=390922999.00, stdev= 0.00 lat (nsec): min=427915k, max=427915k, avg=427915279.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 393], 10.00th=[ 393], 20.00th=[ 393], | 30.00th=[ 393], 40.00th=[ 393], 50.00th=[ 393], 60.00th=[ 393], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.27%, ctx=37, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=90.7MiB/s (95.1MB/s), 90.7MiB/s-90.7MiB/s (95.1MB/s-95.1MB/s), io=40.0MiB (41.9MB), run=441-441msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout