== sanity test 398c: run fio to test AIO ================= 02:46:40 (1790923600) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.602967 s, 69.6 MB/s osc.lustre-OST0000-osc-ffff9f20b41c1800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145649: Fri Oct 2 02:49:41 2026 write: IOPS=15, BW=60.2KiB/s (61.7kB/s)(10.0MiB/170074msec); 0 zone resets slat (usec): min=23, max=16519, avg=200.38, stdev=854.22 clat (msec): min=294, max=2572, avg=1061.78, stdev=328.85 lat (msec): min=294, max=2572, avg=1061.98, stdev=328.86 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 617], 10.00th=[ 709], 20.00th=[ 802], | 30.00th=[ 869], 40.00th=[ 953], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1183], 80.00th=[ 1301], 90.00th=[ 1452], 95.00th=[ 1670], | 99.00th=[ 2089], 99.50th=[ 2366], 99.90th=[ 2567], 99.95th=[ 2567], | 99.99th=[ 2567] bw ( KiB/s): min= 7, max= 128, per=25.81%, avg=61.94, stdev=33.13, samples=325 iops : min= 1, max= 32, avg=15.01, stdev= 8.34, samples=325 lat (msec) : 500=1.37%, 750=11.52%, 1000=35.39%, 2000=50.08%, >=2000=1.64% cpu : usr=0.04%, sys=0.20%, ctx=1623, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145650: Fri Oct 2 02:49:41 2026 write: IOPS=15, BW=60.2KiB/s (61.6kB/s)(10.0MiB/170206msec); 0 zone resets slat (usec): min=22, max=26999, avg=212.72, stdev=1083.83 clat (msec): min=279, max=2387, avg=1062.47, stdev=329.15 lat (msec): min=279, max=2387, avg=1062.69, stdev=329.18 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 642], 10.00th=[ 709], 20.00th=[ 793], | 30.00th=[ 877], 40.00th=[ 944], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1284], 90.00th=[ 1485], 95.00th=[ 1670], | 99.00th=[ 2140], 99.50th=[ 2232], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 128, per=25.10%, avg=60.24, stdev=30.30, samples=334 iops : min= 1, max= 32, avg=14.58, stdev= 7.60, samples=334 lat (msec) : 500=1.91%, 750=12.19%, 1000=33.71%, 2000=50.39%, >=2000=1.80% cpu : usr=0.04%, sys=0.20%, ctx=1718, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145651: Fri Oct 2 02:49:41 2026 write: IOPS=15, BW=60.2KiB/s (61.7kB/s)(10.0MiB/170044msec); 0 zone resets slat (usec): min=21, max=19386, avg=189.25, stdev=803.49 clat (msec): min=144, max=2568, avg=1062.13, stdev=331.35 lat (msec): min=145, max=2568, avg=1062.32, stdev=331.37 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 651], 10.00th=[ 718], 20.00th=[ 802], | 30.00th=[ 877], 40.00th=[ 944], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1284], 90.00th=[ 1485], 95.00th=[ 1670], | 99.00th=[ 2106], 99.50th=[ 2265], 99.90th=[ 2567], 99.95th=[ 2567], | 99.99th=[ 2567] bw ( KiB/s): min= 7, max= 155, per=24.95%, avg=59.87, stdev=25.75, samples=336 iops : min= 1, max= 38, avg=14.49, stdev= 6.46, samples=336 lat (msec) : 250=0.39%, 500=1.41%, 750=11.76%, 1000=35.00%, 2000=49.65% lat (msec) : >=2000=1.80% cpu : usr=0.04%, sys=0.20%, ctx=1839, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145652: Fri Oct 2 02:49:41 2026 write: IOPS=15, BW=60.2KiB/s (61.6kB/s)(10.0MiB/170203msec); 0 zone resets slat (usec): min=24, max=18946, avg=213.85, stdev=981.82 clat (msec): min=306, max=2570, avg=1062.61, stdev=331.66 lat (msec): min=306, max=2570, avg=1062.82, stdev=331.67 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 634], 10.00th=[ 701], 20.00th=[ 785], | 30.00th=[ 877], 40.00th=[ 944], 50.00th=[ 1020], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1284], 90.00th=[ 1452], 95.00th=[ 1670], | 99.00th=[ 2165], 99.50th=[ 2400], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 2567] bw ( KiB/s): min= 8, max= 128, per=25.26%, avg=60.61, stdev=26.38, samples=332 iops : min= 2, max= 32, avg=14.67, stdev= 6.62, samples=332 lat (msec) : 500=1.56%, 750=13.79%, 1000=31.72%, 2000=51.13%, >=2000=1.80% cpu : usr=0.05%, sys=0.21%, ctx=1931, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=241KiB/s (246kB/s), 60.2KiB/s-60.2KiB/s (61.6kB/s-61.7kB/s), io=40.0MiB (41.9MB), run=170044-170206msec 6% of I/O are 1-page (PAGE_SIZE=4096), max 50% mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145687: Fri Oct 2 02:51:51 2026 read: IOPS=10, BW=41.0KiB/s (42.0kB/s)(5048KiB/120214msec) slat (usec): min=3136, max=76147, avg=15224.82, stdev=9090.18 clat (msec): min=53, max=1391, avg=461.10, stdev=149.87 lat (msec): min=63, max=1445, avg=476.34, stdev=152.93 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 271], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 380], 40.00th=[ 409], 50.00th=[ 439], 60.00th=[ 468], | 70.00th=[ 502], 80.00th=[ 558], 90.00th=[ 651], 95.00th=[ 760], | 99.00th=[ 927], 99.50th=[ 1028], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 7, max= 112, per=25.31%, avg=42.01, stdev=21.68, samples=235 iops : min= 1, max= 28, avg=10.00, stdev= 5.53, samples=235 write: IOPS=10, BW=43.2KiB/s (44.2kB/s)(5192KiB/120214msec); 0 zone resets slat (usec): min=27, max=22156, avg=274.42, stdev=1121.72 clat (msec): min=367, max=2189, avg=1013.48, stdev=325.85 lat (msec): min=367, max=2189, avg=1013.76, stdev=325.86 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 575], 10.00th=[ 642], 20.00th=[ 751], | 30.00th=[ 818], 40.00th=[ 885], 50.00th=[ 953], 60.00th=[ 1053], | 70.00th=[ 1133], 80.00th=[ 1250], 90.00th=[ 1485], 95.00th=[ 1653], | 99.00th=[ 1989], 99.50th=[ 2106], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 120, per=25.09%, avg=43.40, stdev=21.00, samples=234 iops : min= 1, max= 30, avg=10.35, stdev= 5.32, samples=234 lat (msec) : 100=0.08%, 250=1.68%, 500=34.26%, 750=20.86%, 1000=19.77% lat (msec) : 2000=22.89%, >=2000=0.47% cpu : usr=0.09%, sys=0.50%, ctx=4635, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145688: Fri Oct 2 02:51:51 2026 read: IOPS=10, BW=41.5KiB/s (42.5kB/s)(4976KiB/119970msec) slat (usec): min=2631, max=61480, avg=15224.24, stdev=9206.74 clat (msec): min=18, max=1517, avg=455.13, stdev=159.94 lat (msec): min=26, max=1538, avg=470.36, stdev=162.30 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 255], 10.00th=[ 296], 20.00th=[ 342], | 30.00th=[ 372], 40.00th=[ 401], 50.00th=[ 430], 60.00th=[ 460], | 70.00th=[ 498], 80.00th=[ 558], 90.00th=[ 642], 95.00th=[ 726], | 99.00th=[ 969], 99.50th=[ 1150], 99.90th=[ 1485], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 6, max= 118, per=24.84%, avg=41.23, stdev=21.92, samples=236 iops : min= 1, max= 29, avg= 9.81, stdev= 5.53, samples=236 write: IOPS=10, BW=43.9KiB/s (44.9kB/s)(5264KiB/119970msec); 0 zone resets slat (usec): min=33, max=11149, avg=240.33, stdev=798.47 clat (msec): min=382, max=2248, avg=1010.76, stdev=330.69 lat (msec): min=382, max=2248, avg=1011.01, stdev=330.73 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 567], 10.00th=[ 642], 20.00th=[ 751], | 30.00th=[ 818], 40.00th=[ 885], 50.00th=[ 961], 60.00th=[ 1045], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1485], 95.00th=[ 1687], | 99.00th=[ 2022], 99.50th=[ 2072], 99.90th=[ 2232], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 120, per=25.33%, avg=43.82, stdev=21.37, samples=235 iops : min= 1, max= 30, avg=10.46, stdev= 5.44, samples=235 lat (msec) : 20=0.04%, 50=0.08%, 100=0.39%, 250=1.68%, 500=33.55% lat (msec) : 750=21.17%, 1000=19.57%, 2000=22.89%, >=2000=0.62% cpu : usr=0.10%, sys=0.56%, ctx=4312, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145689: Fri Oct 2 02:51:51 2026 read: IOPS=10, BW=41.6KiB/s (42.6kB/s)(4984KiB/119847msec) slat (usec): min=40, max=63595, avg=15343.82, stdev=9518.00 clat (msec): min=37, max=1420, avg=456.31, stdev=152.78 lat (msec): min=42, max=1440, avg=471.66, stdev=155.63 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 241], 10.00th=[ 288], 20.00th=[ 347], | 30.00th=[ 380], 40.00th=[ 409], 50.00th=[ 439], 60.00th=[ 472], | 70.00th=[ 510], 80.00th=[ 550], 90.00th=[ 642], 95.00th=[ 735], | 99.00th=[ 953], 99.50th=[ 1003], 99.90th=[ 1401], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 7, max= 112, per=25.09%, avg=41.65, stdev=20.24, samples=235 iops : min= 1, max= 28, avg= 9.91, stdev= 5.12, samples=235 write: IOPS=10, BW=43.9KiB/s (44.9kB/s)(5256KiB/119847msec); 0 zone resets slat (usec): min=23, max=21620, avg=246.71, stdev=999.53 clat (msec): min=272, max=2221, avg=1010.70, stdev=326.40 lat (msec): min=272, max=2221, avg=1010.95, stdev=326.42 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 567], 10.00th=[ 642], 20.00th=[ 751], | 30.00th=[ 810], 40.00th=[ 885], 50.00th=[ 961], 60.00th=[ 1045], | 70.00th=[ 1133], 80.00th=[ 1250], 90.00th=[ 1469], 95.00th=[ 1653], | 99.00th=[ 1972], 99.50th=[ 2039], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 110, per=25.25%, avg=43.68, stdev=20.97, samples=234 iops : min= 1, max= 27, avg=10.43, stdev= 5.33, samples=234 lat (msec) : 50=0.08%, 100=0.39%, 250=2.38%, 500=31.60%, 750=22.38% lat (msec) : 1000=20.04%, 2000=22.66%, >=2000=0.47% cpu : usr=0.09%, sys=0.52%, ctx=4359, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145690: Fri Oct 2 02:51:51 2026 read: IOPS=10, BW=42.2KiB/s (43.2kB/s)(5060KiB/119845msec) slat (msec): min=2, max=567, avg=15.78, stdev=18.09 clat (msec): min=47, max=1393, avg=465.96, stdev=162.63 lat (msec): min=61, max=1438, avg=481.74, stdev=164.86 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 264], 10.00th=[ 305], 20.00th=[ 351], | 30.00th=[ 384], 40.00th=[ 414], 50.00th=[ 443], 60.00th=[ 472], | 70.00th=[ 502], 80.00th=[ 567], 90.00th=[ 667], 95.00th=[ 751], | 99.00th=[ 969], 99.50th=[ 1167], 99.90th=[ 1385], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 7, max= 103, per=25.71%, avg=42.67, stdev=20.69, samples=232 iops : min= 1, max= 25, avg=10.17, stdev= 5.25, samples=232 write: IOPS=10, BW=43.2KiB/s (44.3kB/s)(5180KiB/119845msec); 0 zone resets slat (usec): min=22, max=15587, avg=218.38, stdev=773.59 clat (msec): min=385, max=2223, avg=1006.01, stdev=316.16 lat (msec): min=385, max=2223, avg=1006.23, stdev=316.16 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 567], 10.00th=[ 634], 20.00th=[ 751], | 30.00th=[ 818], 40.00th=[ 894], 50.00th=[ 969], 60.00th=[ 1045], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1435], 95.00th=[ 1603], | 99.00th=[ 1955], 99.50th=[ 2089], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 120, per=25.24%, avg=43.67, stdev=21.69, samples=232 iops : min= 1, max= 30, avg=10.42, stdev= 5.51, samples=232 lat (msec) : 50=0.04%, 100=0.23%, 250=1.84%, 500=33.55%, 750=21.25% lat (msec) : 1000=19.96%, 2000=22.73%, >=2000=0.39% cpu : usr=0.10%, sys=0.56%, ctx=4842, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=167KiB/s (171kB/s), 41.5KiB/s-42.2KiB/s (42.5kB/s-43.2kB/s), io=19.6MiB (20.5MB), run=119845-120214msec WRITE: bw=174KiB/s (178kB/s), 43.2KiB/s-43.9KiB/s (44.2kB/s-44.9kB/s), io=20.4MiB (21.4MB), run=119845-120214msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145712: Fri Oct 2 02:52:02 2026 read: IOPS=1, BW=55.9MiB/s (58.6MB/s)(40.0MiB/716msec) slat (nsec): min=47378k, max=47378k, avg=47377925.00, stdev= 0.00 clat (nsec): min=626678k, max=626678k, avg=626678371.00, stdev= 0.00 lat (nsec): min=674063k, max=674063k, avg=674062786.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 625], 5.00th=[ 625], 10.00th=[ 625], 20.00th=[ 625], | 30.00th=[ 625], 40.00th=[ 625], 50.00th=[ 625], 60.00th=[ 625], | 70.00th=[ 625], 80.00th=[ 625], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.96%, ctx=4, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=55.9MiB/s (58.6MB/s), 55.9MiB/s-55.9MiB/s (58.6MB/s-58.6MB/s), io=40.0MiB (41.9MB), run=716-716msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout