== sanity test 398c: run fio to test AIO ================= 19:22:24 (1788564144) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.469979 s, 89.2 MB/s osc.lustre-OST0000-osc-ffff8d6666c39000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144839: Fri Sep 4 19:25:13 2026 write: IOPS=15, BW=63.9KiB/s (65.4kB/s)(10.0MiB/160263msec); 0 zone resets slat (usec): min=25, max=15587, avg=222.22, stdev=788.89 clat (msec): min=369, max=2277, avg=999.93, stdev=272.12 lat (msec): min=369, max=2277, avg=1000.15, stdev=272.11 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 651], 10.00th=[ 709], 20.00th=[ 793], | 30.00th=[ 852], 40.00th=[ 902], 50.00th=[ 953], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1334], 95.00th=[ 1519], | 99.00th=[ 1989], 99.50th=[ 2140], 99.90th=[ 2232], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 128, per=26.11%, avg=66.59, stdev=29.78, samples=303 iops : min= 1, max= 32, avg=16.22, stdev= 7.47, samples=303 lat (msec) : 500=0.23%, 750=14.57%, 1000=43.05%, 2000=41.17%, >=2000=0.98% cpu : usr=0.04%, sys=0.21%, ctx=1767, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144840: Fri Sep 4 19:25:13 2026 write: IOPS=16, BW=64.1KiB/s (65.6kB/s)(10.0MiB/159832msec); 0 zone resets slat (usec): min=26, max=15811, avg=196.34, stdev=691.72 clat (msec): min=245, max=2414, avg=998.47, stdev=273.46 lat (msec): min=245, max=2415, avg=998.66, stdev=273.49 clat percentiles (msec): | 1.00th=[ 518], 5.00th=[ 667], 10.00th=[ 726], 20.00th=[ 793], | 30.00th=[ 844], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1334], 95.00th=[ 1485], | 99.00th=[ 2022], 99.50th=[ 2165], 99.90th=[ 2333], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 128, per=25.84%, avg=65.89, stdev=29.66, samples=306 iops : min= 1, max= 32, avg=16.03, stdev= 7.46, samples=306 lat (msec) : 250=0.62%, 500=0.31%, 750=12.85%, 1000=44.10%, 2000=41.02% lat (msec) : >=2000=1.09% cpu : usr=0.04%, sys=0.20%, ctx=1614, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144841: Fri Sep 4 19:25:13 2026 write: IOPS=15, BW=63.9KiB/s (65.5kB/s)(10.0MiB/160196msec); 0 zone resets slat (usec): min=28, max=17809, avg=243.91, stdev=952.10 clat (msec): min=225, max=2478, avg=999.78, stdev=275.45 lat (msec): min=226, max=2478, avg=1000.03, stdev=275.47 clat percentiles (msec): | 1.00th=[ 510], 5.00th=[ 659], 10.00th=[ 718], 20.00th=[ 802], | 30.00th=[ 852], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1011], | 70.00th=[ 1070], 80.00th=[ 1167], 90.00th=[ 1351], 95.00th=[ 1485], | 99.00th=[ 2106], 99.50th=[ 2299], 99.90th=[ 2400], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 128, per=25.42%, avg=64.82, stdev=29.16, samples=311 iops : min= 1, max= 32, avg=15.76, stdev= 7.33, samples=311 lat (msec) : 250=0.04%, 500=0.94%, 750=11.60%, 1000=45.31%, 2000=41.05% lat (msec) : >=2000=1.05% cpu : usr=0.04%, sys=0.22%, ctx=2191, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144842: Fri Sep 4 19:25:13 2026 write: IOPS=15, BW=63.0KiB/s (65.5kB/s)(10.0MiB/160038msec); 0 zone resets slat (usec): min=29, max=10793, avg=201.99, stdev=612.44 clat (msec): min=330, max=2332, avg=998.52, stdev=272.77 lat (msec): min=330, max=2333, avg=998.72, stdev=272.76 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 659], 10.00th=[ 718], 20.00th=[ 793], | 30.00th=[ 852], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1011], | 70.00th=[ 1070], 80.00th=[ 1167], 90.00th=[ 1351], 95.00th=[ 1519], | 99.00th=[ 2072], 99.50th=[ 2232], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 128, per=25.68%, avg=65.47, stdev=27.61, samples=308 iops : min= 1, max= 32, avg=15.93, stdev= 6.92, samples=308 lat (msec) : 500=0.62%, 750=12.50%, 1000=44.92%, 2000=40.82%, >=2000=1.13% cpu : usr=0.05%, sys=0.22%, ctx=1949, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=256KiB/s (262kB/s), 63.9KiB/s-64.1KiB/s (65.4kB/s-65.6kB/s), io=40.0MiB (41.9MB), run=159832-160263msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144874: Fri Sep 4 19:27:05 2026 read: IOPS=12, BW=48.9KiB/s (50.0kB/s)(5048KiB/103324msec) slat (usec): min=2293, max=58030, avg=12805.36, stdev=7046.69 clat (msec): min=15, max=1188, avg=405.87, stdev=126.17 lat (msec): min=21, max=1210, avg=418.69, stdev=127.40 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 259], 10.00th=[ 284], 20.00th=[ 317], | 30.00th=[ 347], 40.00th=[ 368], 50.00th=[ 388], 60.00th=[ 409], | 70.00th=[ 443], 80.00th=[ 485], 90.00th=[ 542], 95.00th=[ 617], | 99.00th=[ 944], 99.50th=[ 1045], 99.90th=[ 1116], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 8, max= 111, per=25.21%, avg=48.91, stdev=20.20, samples=203 iops : min= 2, max= 27, avg=11.82, stdev= 5.12, samples=203 write: IOPS=12, BW=50.2KiB/s (51.5kB/s)(5192KiB/103324msec); 0 zone resets slat (usec): min=34, max=20005, avg=347.21, stdev=1202.35 clat (msec): min=170, max=2165, avg=862.77, stdev=249.08 lat (msec): min=170, max=2165, avg=863.12, stdev=249.08 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 523], 10.00th=[ 592], 20.00th=[ 651], | 30.00th=[ 726], 40.00th=[ 785], 50.00th=[ 844], 60.00th=[ 902], | 70.00th=[ 961], 80.00th=[ 1045], 90.00th=[ 1167], 95.00th=[ 1318], | 99.00th=[ 1502], 99.50th=[ 2056], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 104, per=24.83%, avg=50.16, stdev=20.53, samples=203 iops : min= 1, max= 26, avg=12.13, stdev= 5.12, samples=203 lat (msec) : 20=0.08%, 50=0.16%, 100=0.12%, 250=1.68%, 500=41.13% lat (msec) : 750=22.89%, 1000=20.90%, 2000=12.77%, >=2000=0.27% cpu : usr=0.10%, sys=0.65%, ctx=4784, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144875: Fri Sep 4 19:27:05 2026 read: IOPS=12, BW=48.2KiB/s (49.3kB/s)(4976KiB/103322msec) slat (usec): min=42, max=61134, avg=13111.59, stdev=7649.03 clat (msec): min=11, max=1112, avg=406.09, stdev=121.79 lat (msec): min=12, max=1141, avg=419.20, stdev=123.51 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 255], 10.00th=[ 288], 20.00th=[ 317], | 30.00th=[ 347], 40.00th=[ 368], 50.00th=[ 388], 60.00th=[ 409], | 70.00th=[ 443], 80.00th=[ 485], 90.00th=[ 550], 95.00th=[ 625], | 99.00th=[ 793], 99.50th=[ 978], 99.90th=[ 1099], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 120, per=24.73%, avg=47.99, stdev=22.56, samples=204 iops : min= 1, max= 30, avg=11.59, stdev= 5.66, samples=204 write: IOPS=12, BW=50.9KiB/s (52.2kB/s)(5264KiB/103322msec); 0 zone resets slat (usec): min=27, max=24362, avg=312.31, stdev=1209.06 clat (msec): min=147, max=2225, avg=858.46, stdev=256.23 lat (msec): min=147, max=2225, avg=858.78, stdev=256.21 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 514], 10.00th=[ 584], 20.00th=[ 659], | 30.00th=[ 726], 40.00th=[ 776], 50.00th=[ 827], 60.00th=[ 894], | 70.00th=[ 969], 80.00th=[ 1045], 90.00th=[ 1150], 95.00th=[ 1284], | 99.00th=[ 1703], 99.50th=[ 2039], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 104, per=25.15%, avg=50.81, stdev=21.39, samples=203 iops : min= 1, max= 26, avg=12.30, stdev= 5.42, samples=203 lat (msec) : 20=0.04%, 50=0.23%, 100=0.27%, 250=1.88%, 500=39.49% lat (msec) : 750=24.57%, 1000=20.55%, 2000=12.66%, >=2000=0.31% cpu : usr=0.10%, sys=0.69%, ctx=4770, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144876: Fri Sep 4 19:27:05 2026 read: IOPS=12, BW=48.2KiB/s (49.3kB/s)(4984KiB/103417msec) slat (usec): min=2560, max=60581, avg=13383.17, stdev=7393.36 clat (msec): min=22, max=1096, avg=399.89, stdev=109.88 lat (msec): min=40, max=1123, avg=413.28, stdev=111.53 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 264], 10.00th=[ 288], 20.00th=[ 313], | 30.00th=[ 342], 40.00th=[ 368], 50.00th=[ 384], 60.00th=[ 409], | 70.00th=[ 439], 80.00th=[ 481], 90.00th=[ 535], 95.00th=[ 584], | 99.00th=[ 735], 99.50th=[ 802], 99.90th=[ 1020], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 7, max= 136, per=24.54%, avg=47.60, stdev=24.10, samples=205 iops : min= 1, max= 34, avg=11.49, stdev= 6.07, samples=205 write: IOPS=12, BW=50.8KiB/s (52.0kB/s)(5256KiB/103417msec); 0 zone resets slat (usec): min=34, max=11555, avg=242.15, stdev=616.62 clat (msec): min=98, max=2219, avg=864.83, stdev=256.92 lat (msec): min=98, max=2219, avg=865.07, stdev=256.91 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 502], 10.00th=[ 592], 20.00th=[ 667], | 30.00th=[ 726], 40.00th=[ 776], 50.00th=[ 844], 60.00th=[ 902], | 70.00th=[ 969], 80.00th=[ 1062], 90.00th=[ 1183], 95.00th=[ 1301], | 99.00th=[ 1636], 99.50th=[ 1821], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 15, max= 128, per=24.74%, avg=49.98, stdev=19.23, samples=204 iops : min= 3, max= 32, avg=12.09, stdev= 4.86, samples=204 lat (msec) : 50=0.16%, 100=0.20%, 250=2.03%, 500=41.02%, 750=23.01% lat (msec) : 1000=19.96%, 2000=13.44%, >=2000=0.20% cpu : usr=0.10%, sys=0.62%, ctx=4614, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144877: Fri Sep 4 19:27:05 2026 read: IOPS=12, BW=48.0KiB/s (50.1kB/s)(5060KiB/103338msec) slat (usec): min=2264, max=54622, avg=13290.27, stdev=7636.07 clat (msec): min=65, max=1110, avg=406.48, stdev=113.68 lat (msec): min=71, max=1130, avg=419.77, stdev=115.51 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 262], 10.00th=[ 288], 20.00th=[ 321], | 30.00th=[ 347], 40.00th=[ 368], 50.00th=[ 388], 60.00th=[ 414], | 70.00th=[ 439], 80.00th=[ 489], 90.00th=[ 550], 95.00th=[ 617], | 99.00th=[ 785], 99.50th=[ 827], 99.90th=[ 1028], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 128, per=25.30%, avg=49.09, stdev=20.91, samples=203 iops : min= 1, max= 32, avg=11.87, stdev= 5.28, samples=203 write: IOPS=12, BW=50.1KiB/s (51.3kB/s)(5180KiB/103338msec); 0 zone resets slat (usec): min=30, max=11694, avg=274.04, stdev=757.30 clat (msec): min=235, max=2189, avg=863.84, stdev=244.67 lat (msec): min=235, max=2190, avg=864.12, stdev=244.67 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 493], 10.00th=[ 592], 20.00th=[ 667], | 30.00th=[ 735], 40.00th=[ 785], 50.00th=[ 844], 60.00th=[ 894], | 70.00th=[ 969], 80.00th=[ 1045], 90.00th=[ 1167], 95.00th=[ 1284], | 99.00th=[ 1552], 99.50th=[ 1905], 99.90th=[ 2165], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 104, per=24.75%, avg=49.99, stdev=19.54, samples=203 iops : min= 1, max= 26, avg=12.09, stdev= 4.94, samples=203 lat (msec) : 100=0.23%, 250=1.72%, 500=41.84%, 750=21.56%, 1000=21.52% lat (msec) : 2000=12.93%, >=2000=0.20% cpu : usr=0.09%, sys=0.61%, ctx=4476, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=194KiB/s (199kB/s), 48.2KiB/s-48.0KiB/s (49.3kB/s-50.1kB/s), io=19.6MiB (20.5MB), run=103322-103417msec WRITE: bw=202KiB/s (207kB/s), 50.1KiB/s-50.9KiB/s (51.3kB/s-52.2kB/s), io=20.4MiB (21.4MB), run=103322-103417msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144898: Fri Sep 4 19:27:16 2026 read: IOPS=1, BW=57.0MiB/s (60.8MB/s)(40.0MiB/690msec) slat (nsec): min=39703k, max=39703k, avg=39703388.00, stdev= 0.00 clat (nsec): min=633628k, max=633628k, avg=633627724.00, stdev= 0.00 lat (nsec): min=673338k, max=673338k, avg=673337798.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 634], 5.00th=[ 634], 10.00th=[ 634], 20.00th=[ 634], | 30.00th=[ 634], 40.00th=[ 634], 50.00th=[ 634], 60.00th=[ 634], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] lat (msec) : 750=100.00% cpu : usr=0.14%, sys=1.88%, ctx=42, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=57.0MiB/s (60.8MB/s), 57.0MiB/s-57.0MiB/s (60.8MB/s-60.8MB/s), io=40.0MiB (41.9MB), run=690-690msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout