== sanity test 398c: run fio to test AIO ================= 02:31:02 (1788849062) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.459467 s, 91.3 MB/s osc.lustre-OST0000-osc-ffff9f5ed9115000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147244: Tue Sep 8 02:34:12 2026 write: IOPS=14, BW=57.1KiB/s (58.5kB/s)(10.0MiB/179190msec); 0 zone resets slat (usec): min=28, max=10655, avg=165.42, stdev=525.52 clat (msec): min=186, max=2460, avg=1118.78, stdev=288.35 lat (msec): min=187, max=2460, avg=1118.95, stdev=288.36 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 667], 10.00th=[ 785], 20.00th=[ 911], | 30.00th=[ 986], 40.00th=[ 1045], 50.00th=[ 1099], 60.00th=[ 1167], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1502], 95.00th=[ 1586], | 99.00th=[ 1838], 99.50th=[ 2106], 99.90th=[ 2366], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 135, per=25.59%, avg=58.34, stdev=27.78, samples=345 iops : min= 1, max= 33, avg=14.12, stdev= 7.00, samples=345 lat (msec) : 250=0.62%, 500=1.29%, 750=5.90%, 1000=24.57%, 2000=66.88% lat (msec) : >=2000=0.74% cpu : usr=0.03%, sys=0.18%, ctx=1528, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147245: Tue Sep 8 02:34:12 2026 write: IOPS=14, BW=57.1KiB/s (58.5kB/s)(10.0MiB/179316msec); 0 zone resets slat (usec): min=27, max=19010, avg=196.46, stdev=819.90 clat (msec): min=258, max=2423, avg=1119.51, stdev=273.48 lat (msec): min=258, max=2423, avg=1119.71, stdev=273.49 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 651], 10.00th=[ 810], 20.00th=[ 919], | 30.00th=[ 986], 40.00th=[ 1053], 50.00th=[ 1116], 60.00th=[ 1183], | 70.00th=[ 1250], 80.00th=[ 1334], 90.00th=[ 1469], 95.00th=[ 1552], | 99.00th=[ 1821], 99.50th=[ 1989], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 254, per=25.37%, avg=57.84, stdev=30.64, samples=348 iops : min= 1, max= 63, avg=13.99, stdev= 7.69, samples=348 lat (msec) : 500=1.99%, 750=5.51%, 1000=24.73%, 2000=67.30%, >=2000=0.47% cpu : usr=0.05%, sys=0.18%, ctx=1858, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147246: Tue Sep 8 02:34:12 2026 write: IOPS=14, BW=57.1KiB/s (58.5kB/s)(10.0MiB/179189msec); 0 zone resets slat (usec): min=26, max=18478, avg=173.02, stdev=777.90 clat (msec): min=358, max=2417, avg=1118.50, stdev=277.45 lat (msec): min=358, max=2417, avg=1118.68, stdev=277.45 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 667], 10.00th=[ 793], 20.00th=[ 911], | 30.00th=[ 978], 40.00th=[ 1045], 50.00th=[ 1116], 60.00th=[ 1167], | 70.00th=[ 1250], 80.00th=[ 1334], 90.00th=[ 1469], 95.00th=[ 1569], | 99.00th=[ 1787], 99.50th=[ 2106], 99.90th=[ 2299], 99.95th=[ 2299], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 254, per=25.52%, avg=58.18, stdev=28.08, samples=346 iops : min= 1, max= 63, avg=14.09, stdev= 7.06, samples=346 lat (msec) : 500=2.50%, 750=4.80%, 1000=25.35%, 2000=66.52%, >=2000=0.82% cpu : usr=0.03%, sys=0.20%, ctx=1710, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147247: Tue Sep 8 02:34:12 2026 write: IOPS=14, BW=57.2KiB/s (58.6kB/s)(10.0MiB/179057msec); 0 zone resets slat (usec): min=25, max=19669, avg=210.21, stdev=857.29 clat (msec): min=118, max=2636, avg=1118.24, stdev=292.27 lat (msec): min=118, max=2636, avg=1118.45, stdev=292.30 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 651], 10.00th=[ 793], 20.00th=[ 902], | 30.00th=[ 978], 40.00th=[ 1045], 50.00th=[ 1116], 60.00th=[ 1183], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1485], 95.00th=[ 1569], | 99.00th=[ 1888], 99.50th=[ 2165], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 2635] bw ( KiB/s): min= 7, max= 136, per=24.92%, avg=56.83, stdev=26.35, samples=354 iops : min= 1, max= 34, avg=13.74, stdev= 6.64, samples=354 lat (msec) : 250=0.39%, 500=1.80%, 750=5.94%, 1000=24.69%, 2000=66.33% lat (msec) : >=2000=0.86% cpu : usr=0.04%, sys=0.19%, ctx=1892, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=228KiB/s (234kB/s), 57.1KiB/s-57.2KiB/s (58.5kB/s-58.6kB/s), io=40.0MiB (41.9MB), run=179057-179316msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147292: Tue Sep 8 02:36:15 2026 read: IOPS=11, BW=44.7KiB/s (45.7kB/s)(5048KiB/112997msec) slat (usec): min=2537, max=56469, avg=14102.20, stdev=7762.92 clat (msec): min=14, max=809, avg=444.33, stdev=108.52 lat (msec): min=25, max=820, avg=458.44, stdev=109.53 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 284], 10.00th=[ 317], 20.00th=[ 351], | 30.00th=[ 384], 40.00th=[ 409], 50.00th=[ 439], 60.00th=[ 464], | 70.00th=[ 502], 80.00th=[ 535], 90.00th=[ 592], 95.00th=[ 634], | 99.00th=[ 701], 99.50th=[ 726], 99.90th=[ 793], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 7, max= 104, per=25.21%, avg=44.37, stdev=20.21, samples=223 iops : min= 1, max= 26, avg=10.61, stdev= 5.06, samples=223 write: IOPS=11, BW=45.9KiB/s (47.1kB/s)(5192KiB/112997msec); 0 zone resets slat (usec): min=31, max=13439, avg=267.25, stdev=960.55 clat (msec): min=281, max=1590, avg=943.21, stdev=237.58 lat (msec): min=281, max=1590, avg=943.48, stdev=237.57 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 575], 10.00th=[ 642], 20.00th=[ 735], | 30.00th=[ 802], 40.00th=[ 869], 50.00th=[ 936], 60.00th=[ 1003], | 70.00th=[ 1062], 80.00th=[ 1150], 90.00th=[ 1250], 95.00th=[ 1351], | 99.00th=[ 1552], 99.50th=[ 1569], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 6, max= 103, per=24.57%, avg=45.21, stdev=19.70, samples=224 iops : min= 1, max= 25, avg=10.82, stdev= 4.91, samples=224 lat (msec) : 20=0.08%, 50=0.04%, 100=0.12%, 250=0.82%, 500=34.53% lat (msec) : 750=24.49%, 1000=19.53%, 2000=20.39% cpu : usr=0.09%, sys=0.50%, ctx=3756, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147293: Tue Sep 8 02:36:15 2026 read: IOPS=10, BW=43.8KiB/s (44.9kB/s)(4976KiB/113535msec) slat (usec): min=2886, max=54186, avg=14048.84, stdev=7466.54 clat (msec): min=187, max=809, avg=444.23, stdev=104.20 lat (msec): min=190, max=829, avg=458.28, stdev=105.17 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 288], 10.00th=[ 317], 20.00th=[ 347], | 30.00th=[ 380], 40.00th=[ 409], 50.00th=[ 439], 60.00th=[ 468], | 70.00th=[ 493], 80.00th=[ 531], 90.00th=[ 584], 95.00th=[ 634], | 99.00th=[ 684], 99.50th=[ 718], 99.90th=[ 793], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 7, max= 119, per=24.55%, avg=43.21, stdev=22.28, samples=224 iops : min= 1, max= 29, avg=10.32, stdev= 5.57, samples=224 write: IOPS=11, BW=46.4KiB/s (47.5kB/s)(5264KiB/113535msec); 0 zone resets slat (usec): min=27, max=15475, avg=258.03, stdev=910.15 clat (msec): min=234, max=1594, avg=944.26, stdev=249.35 lat (msec): min=234, max=1594, avg=944.52, stdev=249.37 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 726], | 30.00th=[ 802], 40.00th=[ 877], 50.00th=[ 953], 60.00th=[ 1020], | 70.00th=[ 1083], 80.00th=[ 1150], 90.00th=[ 1250], 95.00th=[ 1334], | 99.00th=[ 1552], 99.50th=[ 1569], 99.90th=[ 1586], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min= 7, max= 111, per=25.03%, avg=46.05, stdev=19.67, samples=223 iops : min= 1, max= 27, avg=11.03, stdev= 4.97, samples=223 lat (msec) : 250=0.66%, 500=35.82%, 750=24.49%, 1000=16.80%, 2000=22.23% cpu : usr=0.09%, sys=0.50%, ctx=3633, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147294: Tue Sep 8 02:36:15 2026 read: IOPS=10, BW=43.9KiB/s (44.0kB/s)(4984KiB/113469msec) slat (msec): min=2, max=605, avg=15.04, stdev=22.91 clat (msec): min=14, max=1228, avg=442.51, stdev=111.95 lat (msec): min=23, max=1252, avg=457.55, stdev=114.24 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 288], 10.00th=[ 321], 20.00th=[ 355], | 30.00th=[ 384], 40.00th=[ 409], 50.00th=[ 435], 60.00th=[ 460], | 70.00th=[ 493], 80.00th=[ 527], 90.00th=[ 592], 95.00th=[ 634], | 99.00th=[ 726], 99.50th=[ 735], 99.90th=[ 827], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 6, max= 103, per=24.93%, avg=43.87, stdev=19.84, samples=222 iops : min= 1, max= 25, avg=10.49, stdev= 4.95, samples=222 write: IOPS=11, BW=46.3KiB/s (47.4kB/s)(5256KiB/113469msec); 0 zone resets slat (usec): min=29, max=14774, avg=244.83, stdev=917.78 clat (msec): min=171, max=1928, avg=944.11, stdev=256.27 lat (msec): min=171, max=1929, avg=944.36, stdev=256.26 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 550], 10.00th=[ 617], 20.00th=[ 735], | 30.00th=[ 810], 40.00th=[ 877], 50.00th=[ 944], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1150], 90.00th=[ 1267], 95.00th=[ 1351], | 99.00th=[ 1552], 99.50th=[ 1636], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 111, per=24.91%, avg=45.84, stdev=20.63, samples=223 iops : min= 1, max= 27, avg=10.98, stdev= 5.17, samples=223 lat (msec) : 20=0.12%, 50=0.08%, 100=0.12%, 250=1.13%, 500=35.82% lat (msec) : 750=23.05%, 1000=18.44%, 2000=21.25% cpu : usr=0.08%, sys=0.52%, ctx=3726, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147295: Tue Sep 8 02:36:15 2026 read: IOPS=11, BW=44.8KiB/s (45.9kB/s)(5060KiB/112998msec) slat (usec): min=52, max=55899, avg=14153.52, stdev=7956.36 clat (msec): min=41, max=826, avg=442.09, stdev=112.54 lat (msec): min=46, max=839, avg=456.25, stdev=113.81 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 279], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 384], 40.00th=[ 409], 50.00th=[ 439], 60.00th=[ 468], | 70.00th=[ 498], 80.00th=[ 531], 90.00th=[ 584], 95.00th=[ 642], | 99.00th=[ 726], 99.50th=[ 743], 99.90th=[ 810], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 7, max= 104, per=25.30%, avg=44.52, stdev=19.89, samples=223 iops : min= 1, max= 26, avg=10.65, stdev= 5.03, samples=223 write: IOPS=11, BW=45.8KiB/s (46.9kB/s)(5180KiB/112998msec); 0 zone resets slat (usec): min=24, max=17604, avg=262.58, stdev=1016.59 clat (msec): min=167, max=1646, avg=949.46, stdev=245.63 lat (msec): min=167, max=1646, avg=949.72, stdev=245.61 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 567], 10.00th=[ 642], 20.00th=[ 743], | 30.00th=[ 818], 40.00th=[ 877], 50.00th=[ 953], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1267], 95.00th=[ 1351], | 99.00th=[ 1536], 99.50th=[ 1569], 99.90th=[ 1636], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 7, max= 103, per=24.79%, avg=45.62, stdev=19.29, samples=221 iops : min= 1, max= 25, avg=10.91, stdev= 4.82, samples=221 lat (msec) : 50=0.20%, 100=0.31%, 250=0.98%, 500=35.20%, 750=22.93% lat (msec) : 1000=19.30%, 2000=21.09% cpu : usr=0.08%, sys=0.51%, ctx=4001, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=177KiB/s (181kB/s), 43.8KiB/s-44.8KiB/s (44.9kB/s-45.9kB/s), io=19.6MiB (20.5MB), run=112997-113535msec WRITE: bw=184KiB/s (188kB/s), 45.8KiB/s-46.4KiB/s (46.9kB/s-47.5kB/s), io=20.4MiB (21.4MB), run=112997-113535msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147315: Tue Sep 8 02:36:24 2026 read: IOPS=2, BW=88.1MiB/s (92.4MB/s)(40.0MiB/454msec) slat (nsec): min=35544k, max=35544k, avg=35543526.00, stdev= 0.00 clat (nsec): min=408008k, max=408008k, avg=408007966.00, stdev= 0.00 lat (nsec): min=443556k, max=443556k, avg=443556017.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 409], 10.00th=[ 409], 20.00th=[ 409], | 30.00th=[ 409], 40.00th=[ 409], 50.00th=[ 409], 60.00th=[ 409], | 70.00th=[ 409], 80.00th=[ 409], 90.00th=[ 409], 95.00th=[ 409], | 99.00th=[ 409], 99.50th=[ 409], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.64%, ctx=11, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=88.1MiB/s (92.4MB/s), 88.1MiB/s-88.1MiB/s (92.4MB/s-92.4MB/s), io=40.0MiB (41.9MB), run=454-454msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout