== sanity test 398c: run fio to test AIO ================= 19:31:53 (1763339513) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.379624 s, 110 MB/s osc.lustre-OST0000-osc-ffff904108020000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=148529: Sun Nov 16 19:34:39 2025 write: IOPS=16, BW=64.8KiB/s (66.3kB/s)(10.0MiB/158039msec); 0 zone resets slat (usec): min=27, max=7299, avg=203.18, stdev=505.58 clat (msec): min=185, max=2042, avg=986.63, stdev=237.90 lat (msec): min=185, max=2043, avg=986.84, stdev=237.92 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 642], 10.00th=[ 718], 20.00th=[ 793], | 30.00th=[ 860], 40.00th=[ 911], 50.00th=[ 969], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1301], 95.00th=[ 1385], | 99.00th=[ 1519], 99.50th=[ 1569], 99.90th=[ 1871], 99.95th=[ 1972], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 144, per=25.07%, avg=64.93, stdev=31.08, samples=310 iops : min= 1, max= 36, avg=15.74, stdev= 7.81, samples=310 lat (msec) : 250=0.78%, 500=1.25%, 750=12.15%, 1000=40.94%, 2000=44.84% lat (msec) : >=2000=0.04% cpu : usr=0.05%, sys=0.28%, ctx=1705, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148530: Sun Nov 16 19:34:39 2025 write: IOPS=16, BW=64.8KiB/s (66.3kB/s)(10.0MiB/158037msec); 0 zone resets slat (usec): min=27, max=7478, avg=186.75, stdev=488.38 clat (msec): min=173, max=1770, avg=986.75, stdev=229.18 lat (msec): min=174, max=1770, avg=986.94, stdev=229.20 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 651], 10.00th=[ 726], 20.00th=[ 802], | 30.00th=[ 860], 40.00th=[ 919], 50.00th=[ 969], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1301], 95.00th=[ 1368], | 99.00th=[ 1502], 99.50th=[ 1552], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1770] bw ( KiB/s): min= 7, max= 247, per=25.16%, avg=65.16, stdev=28.66, samples=309 iops : min= 1, max= 61, avg=15.79, stdev= 7.20, samples=309 lat (msec) : 250=0.62%, 500=1.33%, 750=11.25%, 1000=41.41%, 2000=45.39% cpu : usr=0.05%, sys=0.26%, ctx=1692, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148531: Sun Nov 16 19:34:39 2025 write: IOPS=16, BW=64.8KiB/s (66.3kB/s)(10.0MiB/158038msec); 0 zone resets slat (usec): min=28, max=7607, avg=186.33, stdev=460.17 clat (msec): min=163, max=2044, avg=986.51, stdev=231.71 lat (msec): min=163, max=2044, avg=986.70, stdev=231.73 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 659], 10.00th=[ 718], 20.00th=[ 793], | 30.00th=[ 860], 40.00th=[ 911], 50.00th=[ 969], 60.00th=[ 1028], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1301], 95.00th=[ 1385], | 99.00th=[ 1519], 99.50th=[ 1569], 99.90th=[ 1972], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 247, per=24.83%, avg=64.32, stdev=25.52, samples=313 iops : min= 1, max= 61, avg=15.58, stdev= 6.42, samples=313 lat (msec) : 250=0.62%, 500=1.33%, 750=11.60%, 1000=42.27%, 2000=44.10% lat (msec) : >=2000=0.08% cpu : usr=0.05%, sys=0.26%, ctx=1695, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148532: Sun Nov 16 19:34:39 2025 write: IOPS=16, BW=64.9KiB/s (66.4kB/s)(10.0MiB/157882msec); 0 zone resets slat (usec): min=28, max=7930, avg=199.86, stdev=535.38 clat (msec): min=136, max=2076, avg=985.96, stdev=236.34 lat (msec): min=136, max=2076, avg=986.16, stdev=236.34 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 651], 10.00th=[ 726], 20.00th=[ 793], | 30.00th=[ 860], 40.00th=[ 919], 50.00th=[ 969], 60.00th=[ 1045], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1284], 95.00th=[ 1385], | 99.00th=[ 1502], 99.50th=[ 1569], 99.90th=[ 1888], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 189, per=25.07%, avg=64.94, stdev=29.18, samples=310 iops : min= 1, max= 47, avg=15.74, stdev= 7.31, samples=310 lat (msec) : 250=0.98%, 500=1.48%, 750=11.17%, 1000=40.43%, 2000=45.86% lat (msec) : >=2000=0.08% cpu : usr=0.06%, sys=0.26%, ctx=1584, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=259KiB/s (265kB/s), 64.8KiB/s-64.9KiB/s (66.3kB/s-66.4kB/s), io=40.0MiB (41.9MB), run=157882-158039msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148564: Sun Nov 16 19:36:40 2025 read: IOPS=11, BW=44.9KiB/s (45.0kB/s)(5048KiB/112386msec) slat (usec): min=3079, max=62785, avg=13951.23, stdev=6376.29 clat (msec): min=11, max=987, avg=438.70, stdev=115.69 lat (msec): min=37, max=1000, avg=452.66, stdev=116.53 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 288], 10.00th=[ 317], 20.00th=[ 355], | 30.00th=[ 380], 40.00th=[ 405], 50.00th=[ 422], 60.00th=[ 447], | 70.00th=[ 481], 80.00th=[ 518], 90.00th=[ 584], 95.00th=[ 651], | 99.00th=[ 810], 99.50th=[ 852], 99.90th=[ 986], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 7, max= 119, per=25.27%, avg=44.73, stdev=21.00, samples=221 iops : min= 1, max= 29, avg=10.72, stdev= 5.28, samples=221 write: IOPS=11, BW=46.2KiB/s (47.3kB/s)(5192KiB/112386msec); 0 zone resets slat (usec): min=28, max=9160, avg=240.01, stdev=530.46 clat (msec): min=266, max=1905, avg=942.79, stdev=257.21 lat (msec): min=266, max=1905, avg=943.03, stdev=257.21 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 550], 10.00th=[ 651], 20.00th=[ 751], | 30.00th=[ 802], 40.00th=[ 860], 50.00th=[ 911], 60.00th=[ 969], | 70.00th=[ 1036], 80.00th=[ 1133], 90.00th=[ 1301], 95.00th=[ 1452], | 99.00th=[ 1687], 99.50th=[ 1737], 99.90th=[ 1854], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 103, per=24.97%, avg=46.20, stdev=18.39, samples=220 iops : min= 1, max= 25, avg=11.09, stdev= 4.60, samples=220 lat (msec) : 20=0.08%, 50=0.08%, 100=0.31%, 250=0.55%, 500=37.97% lat (msec) : 750=20.00%, 1000=22.81%, 2000=18.20% cpu : usr=0.10%, sys=0.70%, ctx=4550, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148565: Sun Nov 16 19:36:40 2025 read: IOPS=11, BW=44.1KiB/s (45.2kB/s)(4976KiB/112708msec) slat (usec): min=52, max=65279, avg=13930.26, stdev=6537.75 clat (msec): min=21, max=969, avg=441.69, stdev=115.91 lat (msec): min=21, max=984, avg=455.64, stdev=116.87 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 288], 10.00th=[ 321], 20.00th=[ 355], | 30.00th=[ 380], 40.00th=[ 409], 50.00th=[ 430], 60.00th=[ 451], | 70.00th=[ 481], 80.00th=[ 527], 90.00th=[ 584], 95.00th=[ 651], | 99.00th=[ 785], 99.50th=[ 835], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 7, max= 111, per=24.96%, avg=44.19, stdev=20.30, samples=220 iops : min= 1, max= 27, avg=10.59, stdev= 5.06, samples=220 write: IOPS=11, BW=46.7KiB/s (47.8kB/s)(5264KiB/112708msec); 0 zone resets slat (usec): min=30, max=12591, avg=261.10, stdev=678.16 clat (msec): min=164, max=1883, avg=938.69, stdev=263.02 lat (msec): min=164, max=1883, avg=938.95, stdev=263.01 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 550], 10.00th=[ 642], 20.00th=[ 735], | 30.00th=[ 802], 40.00th=[ 860], 50.00th=[ 919], 60.00th=[ 969], | 70.00th=[ 1036], 80.00th=[ 1133], 90.00th=[ 1267], 95.00th=[ 1435], | 99.00th=[ 1670], 99.50th=[ 1787], 99.90th=[ 1854], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 112, per=24.89%, avg=46.05, stdev=21.66, samples=224 iops : min= 1, max= 28, avg=11.05, stdev= 5.46, samples=224 lat (msec) : 50=0.08%, 100=0.31%, 250=1.09%, 500=36.88%, 750=20.70% lat (msec) : 1000=22.97%, 2000=17.97% cpu : usr=0.12%, sys=0.69%, ctx=4525, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148566: Sun Nov 16 19:36:40 2025 read: IOPS=11, BW=44.2KiB/s (45.2kB/s)(4984KiB/112796msec) slat (usec): min=2690, max=65740, avg=14059.54, stdev=6647.20 clat (msec): min=22, max=972, avg=441.28, stdev=113.99 lat (msec): min=38, max=981, avg=455.34, stdev=114.63 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 288], 10.00th=[ 326], 20.00th=[ 359], | 30.00th=[ 384], 40.00th=[ 405], 50.00th=[ 426], 60.00th=[ 451], | 70.00th=[ 477], 80.00th=[ 523], 90.00th=[ 592], 95.00th=[ 651], | 99.00th=[ 793], 99.50th=[ 835], 99.90th=[ 953], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 7, max= 104, per=25.23%, avg=44.65, stdev=21.51, samples=219 iops : min= 1, max= 26, avg=10.71, stdev= 5.38, samples=219 write: IOPS=11, BW=46.6KiB/s (47.7kB/s)(5256KiB/112796msec); 0 zone resets slat (usec): min=30, max=32386, avg=324.45, stdev=1230.65 clat (msec): min=147, max=1883, avg=939.18, stdev=274.93 lat (msec): min=147, max=1883, avg=939.51, stdev=274.96 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 523], 10.00th=[ 609], 20.00th=[ 735], | 30.00th=[ 802], 40.00th=[ 852], 50.00th=[ 911], 60.00th=[ 978], | 70.00th=[ 1053], 80.00th=[ 1133], 90.00th=[ 1301], 95.00th=[ 1452], | 99.00th=[ 1687], 99.50th=[ 1770], 99.90th=[ 1838], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 104, per=24.89%, avg=46.04, stdev=19.66, samples=223 iops : min= 1, max= 26, avg=11.05, stdev= 4.95, samples=223 lat (msec) : 50=0.12%, 100=0.27%, 250=0.74%, 500=38.20%, 750=20.70% lat (msec) : 1000=20.94%, 2000=19.02% cpu : usr=0.10%, sys=0.73%, ctx=4610, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148567: Sun Nov 16 19:36:40 2025 read: IOPS=11, BW=45.1KiB/s (46.2kB/s)(5060KiB/112073msec) slat (usec): min=3336, max=64259, avg=13880.07, stdev=6294.11 clat (msec): min=44, max=1180, avg=437.61, stdev=115.86 lat (msec): min=52, max=1189, avg=451.50, stdev=116.79 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 296], 10.00th=[ 317], 20.00th=[ 351], | 30.00th=[ 376], 40.00th=[ 401], 50.00th=[ 422], 60.00th=[ 447], | 70.00th=[ 477], 80.00th=[ 514], 90.00th=[ 575], 95.00th=[ 651], | 99.00th=[ 776], 99.50th=[ 852], 99.90th=[ 1167], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 7, max= 104, per=25.80%, avg=45.67, stdev=20.03, samples=217 iops : min= 1, max= 26, avg=10.96, stdev= 5.02, samples=217 write: IOPS=11, BW=46.2KiB/s (47.3kB/s)(5180KiB/112073msec); 0 zone resets slat (usec): min=29, max=13214, avg=238.53, stdev=553.25 clat (msec): min=277, max=1872, avg=940.98, stdev=268.08 lat (msec): min=277, max=1872, avg=941.22, stdev=268.09 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 550], 10.00th=[ 625], 20.00th=[ 735], | 30.00th=[ 793], 40.00th=[ 852], 50.00th=[ 911], 60.00th=[ 978], | 70.00th=[ 1036], 80.00th=[ 1133], 90.00th=[ 1301], 95.00th=[ 1452], | 99.00th=[ 1703], 99.50th=[ 1754], 99.90th=[ 1871], 99.95th=[ 1871], | 99.99th=[ 1871] bw ( KiB/s): min= 7, max= 112, per=24.68%, avg=45.66, stdev=18.77, samples=222 iops : min= 1, max= 28, avg=10.96, stdev= 4.71, samples=222 lat (msec) : 50=0.08%, 100=0.20%, 250=0.74%, 500=38.98%, 750=20.00% lat (msec) : 1000=21.60%, 2000=18.40% cpu : usr=0.10%, sys=0.69%, ctx=4407, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=178KiB/s (182kB/s), 44.1KiB/s-45.1KiB/s (45.2kB/s-46.2kB/s), io=19.6MiB (20.5MB), run=112073-112796msec WRITE: bw=185KiB/s (190kB/s), 46.2KiB/s-46.7KiB/s (47.3kB/s-47.8kB/s), io=20.4MiB (21.4MB), run=112073-112796msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=148587: Sun Nov 16 19:36:50 2025 read: IOPS=1, BW=56.5MiB/s (59.2MB/s)(40.0MiB/708msec) slat (nsec): min=72967k, max=72967k, avg=72966632.00, stdev= 0.00 clat (nsec): min=565357k, max=565357k, avg=565357112.00, stdev= 0.00 lat (nsec): min=638330k, max=638330k, avg=638329677.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 567], 10.00th=[ 567], 20.00th=[ 567], | 30.00th=[ 567], 40.00th=[ 567], 50.00th=[ 567], 60.00th=[ 567], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] lat (msec) : 750=100.00% cpu : usr=1.27%, sys=3.96%, ctx=45, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=56.5MiB/s (59.2MB/s), 56.5MiB/s-56.5MiB/s (59.2MB/s-59.2MB/s), io=40.0MiB (41.9MB), run=708-708msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout