== sanity test 398c: run fio to test AIO ================= 20:48:20 (1786754900) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.825652 s, 50.8 MB/s osc.lustre-OST0000-osc-ffff9a25d311d800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146380: Fri Aug 14 20:51:21 2026 write: IOPS=15, BW=60.9KiB/s (62.3kB/s)(10.0MiB/168191msec); 0 zone resets slat (usec): min=51, max=25073, avg=290.28, stdev=1257.50 clat (msec): min=235, max=2668, avg=1050.13, stdev=353.90 lat (msec): min=235, max=2668, avg=1050.43, stdev=353.94 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 550], 10.00th=[ 667], 20.00th=[ 793], | 30.00th=[ 869], 40.00th=[ 927], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1150], 80.00th=[ 1250], 90.00th=[ 1502], 95.00th=[ 1720], | 99.00th=[ 2265], 99.50th=[ 2500], 99.90th=[ 2534], 99.95th=[ 2601], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 136, per=25.00%, avg=60.75, stdev=29.74, samples=331 iops : min= 1, max= 34, avg=14.71, stdev= 7.46, samples=331 lat (msec) : 250=0.62%, 500=2.85%, 750=11.60%, 1000=33.09%, 2000=49.49% lat (msec) : >=2000=2.34% cpu : usr=0.04%, sys=0.27%, ctx=1886, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146381: Fri Aug 14 20:51:21 2026 write: IOPS=15, BW=60.8KiB/s (62.3kB/s)(10.0MiB/168293msec); 0 zone resets slat (usec): min=51, max=18619, avg=294.48, stdev=963.77 clat (msec): min=237, max=2670, avg=1050.74, stdev=352.57 lat (msec): min=238, max=2670, avg=1051.03, stdev=352.60 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 567], 10.00th=[ 676], 20.00th=[ 785], | 30.00th=[ 869], 40.00th=[ 936], 50.00th=[ 1003], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1284], 90.00th=[ 1485], 95.00th=[ 1703], | 99.00th=[ 2299], 99.50th=[ 2534], 99.90th=[ 2601], 99.95th=[ 2668], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 159, per=25.18%, avg=61.19, stdev=28.79, samples=329 iops : min= 1, max= 39, avg=14.81, stdev= 7.22, samples=329 lat (msec) : 250=0.62%, 500=2.77%, 750=12.23%, 1000=33.95%, 2000=48.40% lat (msec) : >=2000=2.03% cpu : usr=0.04%, sys=0.27%, ctx=2310, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146382: Fri Aug 14 20:51:21 2026 write: IOPS=15, BW=61.0KiB/s (62.5kB/s)(10.0MiB/167774msec); 0 zone resets slat (usec): min=49, max=19255, avg=275.72, stdev=862.36 clat (msec): min=112, max=2581, avg=1048.08, stdev=354.15 lat (msec): min=118, max=2581, avg=1048.35, stdev=354.16 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 558], 10.00th=[ 684], 20.00th=[ 793], | 30.00th=[ 860], 40.00th=[ 936], 50.00th=[ 1011], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1267], 90.00th=[ 1485], 95.00th=[ 1703], | 99.00th=[ 2299], 99.50th=[ 2467], 99.90th=[ 2567], 99.95th=[ 2567], | 99.99th=[ 2567] bw ( KiB/s): min= 7, max= 184, per=25.17%, avg=61.16, stdev=29.10, samples=329 iops : min= 1, max= 46, avg=14.82, stdev= 7.31, samples=329 lat (msec) : 250=0.66%, 500=2.73%, 750=11.68%, 1000=33.67%, 2000=48.75% lat (msec) : >=2000=2.50% cpu : usr=0.04%, sys=0.26%, ctx=2117, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146383: Fri Aug 14 20:51:21 2026 write: IOPS=15, BW=60.9KiB/s (62.4kB/s)(10.0MiB/168081msec); 0 zone resets slat (usec): min=50, max=27631, avg=284.45, stdev=1054.24 clat (msec): min=216, max=2669, avg=1049.16, stdev=353.58 lat (msec): min=216, max=2669, avg=1049.44, stdev=353.65 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 542], 10.00th=[ 676], 20.00th=[ 793], | 30.00th=[ 869], 40.00th=[ 927], 50.00th=[ 1003], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1267], 90.00th=[ 1485], 95.00th=[ 1737], | 99.00th=[ 2299], 99.50th=[ 2500], 99.90th=[ 2567], 99.95th=[ 2601], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 207, per=24.95%, avg=60.64, stdev=29.43, samples=332 iops : min= 1, max= 51, avg=14.68, stdev= 7.40, samples=332 lat (msec) : 250=0.62%, 500=2.89%, 750=11.64%, 1000=34.84%, 2000=47.97% lat (msec) : >=2000=2.03% cpu : usr=0.04%, sys=0.27%, ctx=1972, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=243KiB/s (249kB/s), 60.8KiB/s-61.0KiB/s (62.3kB/s-62.5kB/s), io=40.0MiB (41.9MB), run=167774-168293msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146417: Fri Aug 14 20:53:29 2026 read: IOPS=10, BW=42.4KiB/s (43.4kB/s)(5048KiB/119084msec) slat (usec): min=2821, max=70378, avg=15891.50, stdev=8465.83 clat (msec): min=28, max=1156, avg=477.27, stdev=145.29 lat (msec): min=34, max=1170, avg=493.17, stdev=146.46 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 292], 10.00th=[ 321], 20.00th=[ 363], | 30.00th=[ 397], 40.00th=[ 426], 50.00th=[ 456], 60.00th=[ 489], | 70.00th=[ 527], 80.00th=[ 584], 90.00th=[ 667], 95.00th=[ 726], | 99.00th=[ 944], 99.50th=[ 969], 99.90th=[ 1116], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 119, per=25.27%, avg=42.45, stdev=20.21, samples=233 iops : min= 1, max= 29, avg=10.12, stdev= 5.11, samples=233 write: IOPS=10, BW=43.6KiB/s (44.6kB/s)(5192KiB/119084msec); 0 zone resets slat (usec): min=54, max=25958, avg=417.29, stdev=1368.30 clat (msec): min=391, max=1919, avg=984.75, stdev=255.00 lat (msec): min=391, max=1919, avg=985.17, stdev=255.02 clat percentiles (msec): | 1.00th=[ 510], 5.00th=[ 609], 10.00th=[ 667], 20.00th=[ 760], | 30.00th=[ 844], 40.00th=[ 894], 50.00th=[ 961], 60.00th=[ 1028], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1334], 95.00th=[ 1469], | 99.00th=[ 1687], 99.50th=[ 1821], 99.90th=[ 1905], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 103, per=24.90%, avg=43.58, stdev=19.49, samples=233 iops : min= 1, max= 25, avg=10.40, stdev= 4.91, samples=233 lat (msec) : 50=0.04%, 100=0.12%, 250=0.70%, 500=31.29%, 750=24.30% lat (msec) : 1000=20.94%, 2000=22.62% cpu : usr=0.10%, sys=0.69%, ctx=5040, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146418: Fri Aug 14 20:53:29 2026 read: IOPS=10, BW=41.8KiB/s (42.8kB/s)(4976KiB/119087msec) slat (usec): min=3467, max=76023, avg=15602.69, stdev=8525.30 clat (msec): min=19, max=1113, avg=473.33, stdev=144.58 lat (msec): min=29, max=1139, avg=488.94, stdev=145.64 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 279], 10.00th=[ 313], 20.00th=[ 355], | 30.00th=[ 393], 40.00th=[ 418], 50.00th=[ 451], 60.00th=[ 489], | 70.00th=[ 531], 80.00th=[ 584], 90.00th=[ 667], 95.00th=[ 726], | 99.00th=[ 894], 99.50th=[ 919], 99.90th=[ 986], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 120, per=24.87%, avg=41.79, stdev=20.29, samples=233 iops : min= 1, max= 30, avg= 9.96, stdev= 5.13, samples=233 write: IOPS=11, BW=44.2KiB/s (45.3kB/s)(5264KiB/119087msec); 0 zone resets slat (usec): min=54, max=15137, avg=328.08, stdev=897.17 clat (msec): min=313, max=2052, avg=983.36, stdev=261.47 lat (msec): min=313, max=2053, avg=983.69, stdev=261.50 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 600], 10.00th=[ 667], 20.00th=[ 760], | 30.00th=[ 835], 40.00th=[ 885], 50.00th=[ 953], 60.00th=[ 1028], | 70.00th=[ 1116], 80.00th=[ 1183], 90.00th=[ 1334], 95.00th=[ 1485], | 99.00th=[ 1653], 99.50th=[ 1737], 99.90th=[ 1989], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 104, per=25.05%, avg=43.83, stdev=19.42, samples=235 iops : min= 1, max= 26, avg=10.47, stdev= 4.87, samples=235 lat (msec) : 20=0.04%, 50=0.08%, 100=0.16%, 250=0.74%, 500=30.59% lat (msec) : 750=24.61%, 1000=20.82%, 2000=22.93%, >=2000=0.04% cpu : usr=0.10%, sys=0.67%, ctx=4642, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146419: Fri Aug 14 20:53:29 2026 read: IOPS=10, BW=41.9KiB/s (42.0kB/s)(4984KiB/118814msec) slat (usec): min=57, max=68044, avg=15334.74, stdev=8668.43 clat (msec): min=13, max=1419, avg=471.72, stdev=145.01 lat (msec): min=21, max=1430, avg=487.06, stdev=146.48 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 296], 10.00th=[ 317], 20.00th=[ 355], | 30.00th=[ 388], 40.00th=[ 418], 50.00th=[ 451], 60.00th=[ 485], | 70.00th=[ 523], 80.00th=[ 575], 90.00th=[ 667], 95.00th=[ 735], | 99.00th=[ 902], 99.50th=[ 944], 99.90th=[ 995], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 7, max= 110, per=24.90%, avg=41.83, stdev=20.54, samples=234 iops : min= 1, max= 27, avg= 9.96, stdev= 5.20, samples=234 write: IOPS=11, BW=44.2KiB/s (45.3kB/s)(5256KiB/118814msec); 0 zone resets slat (usec): min=56, max=16985, avg=400.07, stdev=1122.67 clat (msec): min=248, max=1997, avg=983.32, stdev=265.46 lat (msec): min=248, max=1997, avg=983.72, stdev=265.47 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 592], 10.00th=[ 667], 20.00th=[ 760], | 30.00th=[ 835], 40.00th=[ 885], 50.00th=[ 961], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1318], 95.00th=[ 1452], | 99.00th=[ 1720], 99.50th=[ 1804], 99.90th=[ 1972], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 119, per=25.01%, avg=43.76, stdev=20.27, samples=234 iops : min= 1, max= 29, avg=10.45, stdev= 5.11, samples=234 lat (msec) : 20=0.16%, 50=0.08%, 100=0.23%, 250=0.47%, 500=31.41% lat (msec) : 750=23.95%, 1000=20.62%, 2000=23.09% cpu : usr=0.08%, sys=0.65%, ctx=4506, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146420: Fri Aug 14 20:53:29 2026 read: IOPS=10, BW=42.5KiB/s (43.5kB/s)(5060KiB/119084msec) slat (usec): min=3248, max=76797, avg=15657.08, stdev=8826.13 clat (msec): min=61, max=954, avg=471.70, stdev=133.21 lat (msec): min=76, max=972, avg=487.37, stdev=134.78 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 296], 10.00th=[ 326], 20.00th=[ 363], | 30.00th=[ 393], 40.00th=[ 422], 50.00th=[ 456], 60.00th=[ 481], | 70.00th=[ 527], 80.00th=[ 575], 90.00th=[ 642], 95.00th=[ 718], | 99.00th=[ 869], 99.50th=[ 902], 99.90th=[ 953], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 7, max= 96, per=25.59%, avg=42.99, stdev=18.62, samples=231 iops : min= 1, max= 24, avg=10.26, stdev= 4.69, samples=231 write: IOPS=10, BW=43.5KiB/s (44.5kB/s)(5180KiB/119084msec); 0 zone resets slat (usec): min=56, max=27510, avg=403.85, stdev=1396.18 clat (msec): min=390, max=1918, avg=991.48, stdev=260.72 lat (msec): min=392, max=1918, avg=991.89, stdev=260.65 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 600], 10.00th=[ 676], 20.00th=[ 768], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 969], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1200], 90.00th=[ 1351], 95.00th=[ 1469], | 99.00th=[ 1737], 99.50th=[ 1821], 99.90th=[ 1905], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 103, per=24.70%, avg=43.23, stdev=20.55, samples=234 iops : min= 1, max= 25, avg=10.31, stdev= 5.19, samples=234 lat (msec) : 100=0.20%, 250=0.51%, 500=31.64%, 750=24.02%, 1000=20.98% lat (msec) : 2000=22.66% cpu : usr=0.08%, sys=0.64%, ctx=4735, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=169KiB/s (173kB/s), 41.8KiB/s-42.5KiB/s (42.8kB/s-43.5kB/s), io=19.6MiB (20.5MB), run=118814-119087msec WRITE: bw=175KiB/s (180kB/s), 43.5KiB/s-44.2KiB/s (44.5kB/s-45.3kB/s), io=20.4MiB (21.4MB), run=118814-119087msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146443: Fri Aug 14 20:53:37 2026 read: IOPS=2, BW=80.0MiB/s (83.9MB/s)(40.0MiB/500msec) slat (nsec): min=32678k, max=32678k, avg=32678397.00, stdev= 0.00 clat (nsec): min=446125k, max=446125k, avg=446125403.00, stdev= 0.00 lat (nsec): min=478808k, max=478808k, avg=478807510.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 447], 10.00th=[ 447], 20.00th=[ 447], | 30.00th=[ 447], 40.00th=[ 447], 50.00th=[ 447], 60.00th=[ 447], | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.00%, ctx=4, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=80.0MiB/s (83.9MB/s), 80.0MiB/s-80.0MiB/s (83.9MB/s-83.9MB/s), io=40.0MiB (41.9MB), run=500-500msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout