== sanity test 398c: run fio to test AIO ================= 14:15:54 (1790100954) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.551847 s, 76.0 MB/s osc.lustre-OST0000-osc-ffff961b51613000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145915: Tue Sep 22 14:19:40 2026 write: IOPS=11, BW=47.7KiB/s (48.8kB/s)(10.0MiB/214866msec); 0 zone resets slat (usec): min=23, max=23627, avg=291.31, stdev=1170.44 clat (msec): min=306, max=2989, avg=1340.92, stdev=427.42 lat (msec): min=306, max=2989, avg=1341.21, stdev=427.44 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 701], 10.00th=[ 869], 20.00th=[ 1003], | 30.00th=[ 1099], 40.00th=[ 1200], 50.00th=[ 1301], 60.00th=[ 1385], | 70.00th=[ 1519], 80.00th=[ 1687], 90.00th=[ 1905], 95.00th=[ 2072], | 99.00th=[ 2635], 99.50th=[ 2802], 99.90th=[ 2869], 99.95th=[ 2937], | 99.99th=[ 3004] bw ( KiB/s): min= 7, max= 151, per=25.91%, avg=49.23, stdev=26.17, samples=407 iops : min= 1, max= 37, avg=11.78, stdev= 6.56, samples=407 lat (msec) : 500=2.03%, 750=4.26%, 1000=13.09%, 2000=74.02%, >=2000=6.60% cpu : usr=0.04%, sys=0.22%, ctx=1706, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145916: Tue Sep 22 14:19:40 2026 write: IOPS=11, BW=47.8KiB/s (48.0kB/s)(10.0MiB/214144msec); 0 zone resets slat (usec): min=23, max=21833, avg=294.09, stdev=1166.03 clat (msec): min=147, max=2950, avg=1337.52, stdev=428.81 lat (msec): min=148, max=2950, avg=1337.82, stdev=428.82 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 676], 10.00th=[ 852], 20.00th=[ 1011], | 30.00th=[ 1099], 40.00th=[ 1200], 50.00th=[ 1301], 60.00th=[ 1401], | 70.00th=[ 1536], 80.00th=[ 1687], 90.00th=[ 1888], 95.00th=[ 2089], | 99.00th=[ 2534], 99.50th=[ 2702], 99.90th=[ 2937], 99.95th=[ 2937], | 99.99th=[ 2937] bw ( KiB/s): min= 7, max= 158, per=25.41%, avg=48.29, stdev=24.62, samples=415 iops : min= 1, max= 39, avg=11.55, stdev= 6.22, samples=415 lat (msec) : 250=0.16%, 500=2.50%, 750=4.38%, 1000=11.60%, 2000=74.96% lat (msec) : >=2000=6.41% cpu : usr=0.06%, sys=0.22%, ctx=1848, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145917: Tue Sep 22 14:19:40 2026 write: IOPS=11, BW=47.7KiB/s (48.9kB/s)(10.0MiB/214564msec); 0 zone resets slat (usec): min=22, max=43729, avg=276.23, stdev=1358.38 clat (msec): min=351, max=3028, avg=1339.30, stdev=423.26 lat (msec): min=351, max=3028, avg=1339.58, stdev=423.30 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 709], 10.00th=[ 869], 20.00th=[ 1011], | 30.00th=[ 1099], 40.00th=[ 1200], 50.00th=[ 1301], 60.00th=[ 1401], | 70.00th=[ 1502], 80.00th=[ 1670], 90.00th=[ 1888], 95.00th=[ 2072], | 99.00th=[ 2567], 99.50th=[ 2802], 99.90th=[ 3004], 99.95th=[ 3037], | 99.99th=[ 3037] bw ( KiB/s): min= 7, max= 143, per=25.48%, avg=48.41, stdev=26.02, samples=414 iops : min= 1, max= 35, avg=11.58, stdev= 6.56, samples=414 lat (msec) : 500=2.23%, 750=4.22%, 1000=13.32%, 2000=73.75%, >=2000=6.48% cpu : usr=0.03%, sys=0.21%, ctx=1827, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145918: Tue Sep 22 14:19:40 2026 write: IOPS=11, BW=47.7KiB/s (48.8kB/s)(10.0MiB/214859msec); 0 zone resets slat (usec): min=22, max=20210, avg=290.50, stdev=1103.48 clat (msec): min=332, max=2949, avg=1341.14, stdev=421.86 lat (msec): min=337, max=2949, avg=1341.44, stdev=421.89 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 718], 10.00th=[ 860], 20.00th=[ 1003], | 30.00th=[ 1116], 40.00th=[ 1200], 50.00th=[ 1301], 60.00th=[ 1401], | 70.00th=[ 1519], 80.00th=[ 1670], 90.00th=[ 1888], 95.00th=[ 2106], | 99.00th=[ 2534], 99.50th=[ 2735], 99.90th=[ 2869], 99.95th=[ 2937], | 99.99th=[ 2937] bw ( KiB/s): min= 7, max= 135, per=25.11%, avg=47.71, stdev=26.70, samples=420 iops : min= 1, max= 33, avg=11.40, stdev= 6.75, samples=420 lat (msec) : 500=1.64%, 750=4.69%, 1000=13.32%, 2000=73.01%, >=2000=7.34% cpu : usr=0.05%, sys=0.20%, ctx=1712, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=191KiB/s (195kB/s), 47.7KiB/s-47.8KiB/s (48.8kB/s-48.0kB/s), io=40.0MiB (41.9MB), run=214144-214866msec 4% of I/O are 1-page (PAGE_SIZE=4096), max 50% mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145959: Tue Sep 22 14:22:22 2026 read: IOPS=8, BW=33.1KiB/s (33.9kB/s)(5048KiB/152617msec) slat (usec): min=3696, max=80077, avg=18770.49, stdev=9786.41 clat (msec): min=35, max=1319, avg=584.29, stdev=162.18 lat (msec): min=50, max=1328, avg=603.06, stdev=163.67 clat percentiles (msec): | 1.00th=[ 259], 5.00th=[ 368], 10.00th=[ 405], 20.00th=[ 456], | 30.00th=[ 493], 40.00th=[ 531], 50.00th=[ 567], 60.00th=[ 609], | 70.00th=[ 651], 80.00th=[ 709], 90.00th=[ 802], 95.00th=[ 869], | 99.00th=[ 1045], 99.50th=[ 1167], 99.90th=[ 1284], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 7, max= 95, per=25.28%, avg=33.11, stdev=17.32, samples=297 iops : min= 1, max= 23, avg= 7.74, stdev= 4.37, samples=297 write: IOPS=8, BW=34.0KiB/s (34.8kB/s)(5192KiB/152617msec); 0 zone resets slat (usec): min=33, max=19502, avg=297.98, stdev=1128.81 clat (msec): min=431, max=2292, avg=1291.19, stdev=339.42 lat (msec): min=431, max=2292, avg=1291.50, stdev=339.40 clat percentiles (msec): | 1.00th=[ 592], 5.00th=[ 760], 10.00th=[ 852], 20.00th=[ 978], | 30.00th=[ 1083], 40.00th=[ 1200], 50.00th=[ 1284], 60.00th=[ 1368], | 70.00th=[ 1469], 80.00th=[ 1569], 90.00th=[ 1754], 95.00th=[ 1905], | 99.00th=[ 2106], 99.50th=[ 2140], 99.90th=[ 2265], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 80, per=25.07%, avg=34.10, stdev=17.17, samples=296 iops : min= 1, max= 20, avg= 7.99, stdev= 4.34, samples=296 lat (msec) : 50=0.08%, 100=0.12%, 250=0.27%, 500=15.35%, 750=28.48% lat (msec) : 1000=15.23%, 2000=38.87%, >=2000=1.60% cpu : usr=0.06%, sys=0.57%, ctx=4399, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145960: Tue Sep 22 14:22:22 2026 read: IOPS=8, BW=32.5KiB/s (33.3kB/s)(4976KiB/152875msec) slat (usec): min=3696, max=79302, avg=18898.74, stdev=10293.49 clat (msec): min=38, max=1223, avg=580.00, stdev=157.08 lat (msec): min=51, max=1234, avg=598.91, stdev=158.18 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 359], 10.00th=[ 401], 20.00th=[ 451], | 30.00th=[ 493], 40.00th=[ 527], 50.00th=[ 567], 60.00th=[ 609], | 70.00th=[ 651], 80.00th=[ 709], 90.00th=[ 793], 95.00th=[ 844], | 99.00th=[ 995], 99.50th=[ 1045], 99.90th=[ 1183], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 96, per=25.00%, avg=32.75, stdev=18.67, samples=295 iops : min= 1, max= 24, avg= 7.66, stdev= 4.69, samples=295 write: IOPS=8, BW=34.4KiB/s (35.3kB/s)(5264KiB/152875msec); 0 zone resets slat (usec): min=33, max=19403, avg=286.88, stdev=924.63 clat (msec): min=248, max=2292, avg=1289.24, stdev=340.81 lat (msec): min=248, max=2292, avg=1289.53, stdev=340.82 clat percentiles (msec): | 1.00th=[ 600], 5.00th=[ 751], 10.00th=[ 852], 20.00th=[ 986], | 30.00th=[ 1083], 40.00th=[ 1200], 50.00th=[ 1284], 60.00th=[ 1368], | 70.00th=[ 1469], 80.00th=[ 1586], 90.00th=[ 1754], 95.00th=[ 1888], | 99.00th=[ 2123], 99.50th=[ 2198], 99.90th=[ 2265], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 87, per=25.35%, avg=34.48, stdev=17.64, samples=298 iops : min= 1, max= 21, avg= 8.09, stdev= 4.43, samples=298 lat (msec) : 50=0.04%, 100=0.23%, 250=0.31%, 500=15.43%, 750=28.28% lat (msec) : 1000=14.77%, 2000=39.49%, >=2000=1.45% cpu : usr=0.09%, sys=0.52%, ctx=4459, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145961: Tue Sep 22 14:22:22 2026 read: IOPS=8, BW=32.6KiB/s (33.4kB/s)(4984KiB/152729msec) slat (usec): min=3308, max=80145, avg=18833.09, stdev=10539.48 clat (msec): min=32, max=1308, avg=582.24, stdev=157.25 lat (msec): min=46, max=1319, avg=601.08, stdev=158.81 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 359], 10.00th=[ 397], 20.00th=[ 451], | 30.00th=[ 493], 40.00th=[ 535], 50.00th=[ 575], 60.00th=[ 609], | 70.00th=[ 651], 80.00th=[ 701], 90.00th=[ 785], 95.00th=[ 852], | 99.00th=[ 995], 99.50th=[ 1150], 99.90th=[ 1284], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 7, max= 119, per=25.33%, avg=33.18, stdev=18.35, samples=293 iops : min= 1, max= 29, avg= 7.76, stdev= 4.59, samples=293 write: IOPS=8, BW=34.4KiB/s (35.2kB/s)(5256KiB/152729msec); 0 zone resets slat (usec): min=32, max=14569, avg=297.04, stdev=929.40 clat (msec): min=407, max=2313, avg=1285.83, stdev=348.03 lat (msec): min=407, max=2313, avg=1286.13, stdev=347.99 clat percentiles (msec): | 1.00th=[ 617], 5.00th=[ 743], 10.00th=[ 852], 20.00th=[ 969], | 30.00th=[ 1083], 40.00th=[ 1183], 50.00th=[ 1267], 60.00th=[ 1351], | 70.00th=[ 1469], 80.00th=[ 1586], 90.00th=[ 1770], 95.00th=[ 1921], | 99.00th=[ 2106], 99.50th=[ 2165], 99.90th=[ 2265], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 87, per=25.27%, avg=34.37, stdev=17.13, samples=297 iops : min= 1, max= 21, avg= 8.06, stdev= 4.28, samples=297 lat (msec) : 50=0.04%, 100=0.12%, 250=0.20%, 500=15.59%, 750=28.87% lat (msec) : 1000=15.08%, 2000=38.83%, >=2000=1.29% cpu : usr=0.09%, sys=0.52%, ctx=4750, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145962: Tue Sep 22 14:22:22 2026 read: IOPS=8, BW=33.3KiB/s (34.1kB/s)(5060KiB/151796msec) slat (usec): min=46, max=86463, avg=18640.67, stdev=10669.15 clat (msec): min=35, max=1762, avg=587.33, stdev=161.28 lat (msec): min=51, max=1792, avg=605.98, stdev=163.05 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 372], 10.00th=[ 414], 20.00th=[ 464], | 30.00th=[ 498], 40.00th=[ 527], 50.00th=[ 575], 60.00th=[ 609], | 70.00th=[ 651], 80.00th=[ 709], 90.00th=[ 785], 95.00th=[ 844], | 99.00th=[ 1116], 99.50th=[ 1167], 99.90th=[ 1519], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 7, max= 103, per=25.79%, avg=33.78, stdev=16.34, samples=292 iops : min= 1, max= 25, avg= 7.91, stdev= 4.11, samples=292 write: IOPS=8, BW=34.1KiB/s (34.9kB/s)(5180KiB/151796msec); 0 zone resets slat (usec): min=24, max=23026, avg=323.13, stdev=1183.41 clat (msec): min=125, max=2564, avg=1281.31, stdev=359.19 lat (msec): min=131, max=2564, avg=1281.63, stdev=359.12 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 743], 10.00th=[ 852], 20.00th=[ 969], | 30.00th=[ 1083], 40.00th=[ 1167], 50.00th=[ 1267], 60.00th=[ 1351], | 70.00th=[ 1452], 80.00th=[ 1569], 90.00th=[ 1770], 95.00th=[ 1905], | 99.00th=[ 2165], 99.50th=[ 2265], 99.90th=[ 2467], 99.95th=[ 2567], | 99.99th=[ 2567] bw ( KiB/s): min= 7, max= 72, per=25.08%, avg=34.11, stdev=16.38, samples=295 iops : min= 1, max= 18, avg= 7.99, stdev= 4.11, samples=295 lat (msec) : 50=0.12%, 250=0.51%, 500=15.35%, 750=29.38%, 1000=14.92% lat (msec) : 2000=38.12%, >=2000=1.60% cpu : usr=0.08%, sys=0.56%, ctx=4617, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=131KiB/s (134kB/s), 32.5KiB/s-33.3KiB/s (33.3kB/s-34.1kB/s), io=19.6MiB (20.5MB), run=151796-152875msec WRITE: bw=137KiB/s (140kB/s), 34.0KiB/s-34.4KiB/s (34.8kB/s-35.3kB/s), io=20.4MiB (21.4MB), run=151796-152875msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145987: Tue Sep 22 14:22:34 2026 read: IOPS=1, BW=71.8MiB/s (75.3MB/s)(40.0MiB/557msec) slat (nsec): min=29906k, max=29906k, avg=29905658.00, stdev= 0.00 clat (nsec): min=493052k, max=493052k, avg=493051928.00, stdev= 0.00 lat (nsec): min=522962k, max=522962k, avg=522961551.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 493], 10.00th=[ 493], 20.00th=[ 493], | 30.00th=[ 493], 40.00th=[ 493], 50.00th=[ 493], 60.00th=[ 493], | 70.00th=[ 493], 80.00th=[ 493], 90.00th=[ 493], 95.00th=[ 493], | 99.00th=[ 493], 99.50th=[ 493], 99.90th=[ 493], 99.95th=[ 493], | 99.99th=[ 493] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.16%, ctx=6, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=71.8MiB/s (75.3MB/s), 71.8MiB/s-71.8MiB/s (75.3MB/s-75.3MB/s), io=40.0MiB (41.9MB), run=557-557msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout