== sanity test 398c: run fio to test AIO ================= 05:57:09 (1789120629) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.602286 s, 69.6 MB/s osc.lustre-OST0000-osc-ffff94c63af32800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145669: Fri Sep 11 06:00:04 2026 write: IOPS=15, BW=62.1KiB/s (63.6kB/s)(10.0MiB/164784msec); 0 zone resets slat (usec): min=21, max=25330, avg=238.19, stdev=965.78 clat (msec): min=165, max=2192, avg=1028.88, stdev=291.97 lat (msec): min=165, max=2192, avg=1029.12, stdev=291.97 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 617], 10.00th=[ 718], 20.00th=[ 793], | 30.00th=[ 869], 40.00th=[ 919], 50.00th=[ 978], 60.00th=[ 1062], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1401], 95.00th=[ 1586], | 99.00th=[ 1854], 99.50th=[ 1938], 99.90th=[ 2106], 99.95th=[ 2165], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 128, per=25.85%, avg=64.10, stdev=36.40, samples=314 iops : min= 1, max= 32, avg=15.52, stdev= 9.14, samples=314 lat (msec) : 250=0.62%, 500=0.70%, 750=11.72%, 1000=39.53%, 2000=46.95% lat (msec) : >=2000=0.47% cpu : usr=0.05%, sys=0.23%, ctx=1484, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145670: Fri Sep 11 06:00:04 2026 write: IOPS=15, BW=62.2KiB/s (63.7kB/s)(10.0MiB/164542msec); 0 zone resets slat (usec): min=21, max=16164, avg=249.32, stdev=941.44 clat (msec): min=159, max=2216, avg=1027.75, stdev=298.40 lat (msec): min=159, max=2217, avg=1028.01, stdev=298.41 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 609], 10.00th=[ 709], 20.00th=[ 793], | 30.00th=[ 860], 40.00th=[ 919], 50.00th=[ 986], 60.00th=[ 1062], | 70.00th=[ 1150], 80.00th=[ 1267], 90.00th=[ 1435], 95.00th=[ 1552], | 99.00th=[ 1854], 99.50th=[ 2022], 99.90th=[ 2198], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 128, per=26.00%, avg=64.49, stdev=29.28, samples=312 iops : min= 1, max= 32, avg=15.61, stdev= 7.37, samples=312 lat (msec) : 250=0.62%, 500=1.29%, 750=12.27%, 1000=37.50%, 2000=47.81% lat (msec) : >=2000=0.51% cpu : usr=0.06%, sys=0.22%, ctx=1440, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145671: Fri Sep 11 06:00:04 2026 write: IOPS=15, BW=62.1KiB/s (63.6kB/s)(10.0MiB/164780msec); 0 zone resets slat (usec): min=22, max=16625, avg=240.57, stdev=879.20 clat (msec): min=233, max=2314, avg=1028.69, stdev=298.35 lat (msec): min=233, max=2315, avg=1028.93, stdev=298.38 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 600], 10.00th=[ 701], 20.00th=[ 793], | 30.00th=[ 860], 40.00th=[ 919], 50.00th=[ 986], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1250], 90.00th=[ 1418], 95.00th=[ 1552], | 99.00th=[ 1854], 99.50th=[ 1989], 99.90th=[ 2265], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 128, per=25.27%, avg=62.68, stdev=28.43, samples=321 iops : min= 1, max= 32, avg=15.16, stdev= 7.13, samples=321 lat (msec) : 250=0.62%, 500=1.25%, 750=12.73%, 1000=36.72%, 2000=48.32% lat (msec) : >=2000=0.35% cpu : usr=0.05%, sys=0.24%, ctx=1661, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145672: Fri Sep 11 06:00:04 2026 write: IOPS=15, BW=62.1KiB/s (63.6kB/s)(10.0MiB/164847msec); 0 zone resets slat (usec): min=22, max=15924, avg=234.37, stdev=828.58 clat (msec): min=118, max=2199, avg=1029.01, stdev=294.02 lat (msec): min=119, max=2199, avg=1029.25, stdev=294.01 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 617], 10.00th=[ 709], 20.00th=[ 793], | 30.00th=[ 860], 40.00th=[ 919], 50.00th=[ 995], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1234], 90.00th=[ 1418], 95.00th=[ 1552], | 99.00th=[ 1955], 99.50th=[ 2106], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 143, per=25.52%, avg=63.28, stdev=33.09, samples=318 iops : min= 1, max= 35, avg=15.31, stdev= 8.30, samples=318 lat (msec) : 250=0.08%, 500=1.84%, 750=12.15%, 1000=37.23%, 2000=47.97% lat (msec) : >=2000=0.74% cpu : usr=0.04%, sys=0.26%, ctx=1774, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=248KiB/s (254kB/s), 62.1KiB/s-62.2KiB/s (63.6kB/s-63.7kB/s), io=40.0MiB (41.9MB), run=164542-164847msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145707: Fri Sep 11 06:01:52 2026 read: IOPS=12, BW=51.0KiB/s (52.2kB/s)(5048KiB/98961msec) slat (usec): min=3025, max=49896, avg=14452.97, stdev=7887.51 clat (msec): min=45, max=707, avg=378.76, stdev=94.25 lat (msec): min=56, max=719, avg=393.22, stdev=95.63 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 247], 10.00th=[ 271], 20.00th=[ 300], | 30.00th=[ 326], 40.00th=[ 351], 50.00th=[ 372], 60.00th=[ 397], | 70.00th=[ 422], 80.00th=[ 451], 90.00th=[ 498], 95.00th=[ 558], | 99.00th=[ 651], 99.50th=[ 676], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 7, max= 144, per=25.12%, avg=50.73, stdev=24.87, samples=195 iops : min= 1, max= 36, avg=12.12, stdev= 6.25, samples=195 write: IOPS=13, BW=52.5KiB/s (53.7kB/s)(5192KiB/98961msec); 0 zone resets slat (usec): min=28, max=14525, avg=324.17, stdev=840.08 clat (msec): min=282, max=1704, avg=833.12, stdev=217.47 lat (msec): min=282, max=1704, avg=833.44, stdev=217.46 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 514], 10.00th=[ 558], 20.00th=[ 642], | 30.00th=[ 701], 40.00th=[ 760], 50.00th=[ 810], 60.00th=[ 877], | 70.00th=[ 944], 80.00th=[ 1003], 90.00th=[ 1116], 95.00th=[ 1217], | 99.00th=[ 1418], 99.50th=[ 1469], 99.90th=[ 1653], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 110, per=24.76%, avg=51.99, stdev=20.55, samples=195 iops : min= 1, max= 27, avg=12.44, stdev= 5.15, samples=195 lat (msec) : 50=0.04%, 100=0.16%, 250=2.50%, 500=43.79%, 750=22.50% lat (msec) : 1000=20.59%, 2000=10.43% cpu : usr=0.12%, sys=0.80%, ctx=4485, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145708: Fri Sep 11 06:01:52 2026 read: IOPS=12, BW=50.2KiB/s (51.4kB/s)(4976KiB/99058msec) slat (usec): min=2685, max=61483, avg=14781.49, stdev=7947.47 clat (msec): min=22, max=731, avg=378.35, stdev=94.05 lat (msec): min=33, max=747, avg=393.13, stdev=95.13 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 243], 10.00th=[ 268], 20.00th=[ 305], | 30.00th=[ 330], 40.00th=[ 351], 50.00th=[ 372], 60.00th=[ 393], | 70.00th=[ 422], 80.00th=[ 447], 90.00th=[ 502], 95.00th=[ 550], | 99.00th=[ 609], 99.50th=[ 659], 99.90th=[ 701], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 7, max= 128, per=24.59%, avg=49.67, stdev=21.44, samples=195 iops : min= 1, max= 32, avg=11.85, stdev= 5.39, samples=195 write: IOPS=13, BW=53.1KiB/s (54.4kB/s)(5264KiB/99058msec); 0 zone resets slat (usec): min=30, max=15509, avg=393.36, stdev=1133.16 clat (msec): min=282, max=1657, avg=828.68, stdev=215.57 lat (msec): min=282, max=1657, avg=829.08, stdev=215.55 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 510], 10.00th=[ 550], 20.00th=[ 642], | 30.00th=[ 701], 40.00th=[ 751], 50.00th=[ 818], 60.00th=[ 869], | 70.00th=[ 927], 80.00th=[ 1003], 90.00th=[ 1116], 95.00th=[ 1217], | 99.00th=[ 1401], 99.50th=[ 1435], 99.90th=[ 1519], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 7, max= 104, per=25.05%, avg=52.59, stdev=21.17, samples=195 iops : min= 1, max= 26, avg=12.58, stdev= 5.27, samples=195 lat (msec) : 50=0.12%, 100=0.27%, 250=2.58%, 500=42.93%, 750=23.24% lat (msec) : 1000=20.47%, 2000=10.39% cpu : usr=0.17%, sys=0.83%, ctx=4724, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145709: Fri Sep 11 06:01:52 2026 read: IOPS=12, BW=50.4KiB/s (51.6kB/s)(4984KiB/98846msec) slat (usec): min=43, max=69796, avg=14800.90, stdev=8283.49 clat (msec): min=19, max=786, avg=377.33, stdev=98.74 lat (msec): min=25, max=793, avg=392.14, stdev=100.16 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 243], 10.00th=[ 271], 20.00th=[ 300], | 30.00th=[ 326], 40.00th=[ 347], 50.00th=[ 368], 60.00th=[ 388], | 70.00th=[ 418], 80.00th=[ 447], 90.00th=[ 502], 95.00th=[ 550], | 99.00th=[ 651], 99.50th=[ 701], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 7, max= 143, per=24.85%, avg=50.19, stdev=24.19, samples=195 iops : min= 1, max= 35, avg=11.98, stdev= 6.06, samples=195 write: IOPS=13, BW=53.2KiB/s (54.4kB/s)(5256KiB/98846msec); 0 zone resets slat (usec): min=26, max=11350, avg=384.32, stdev=983.13 clat (msec): min=208, max=1654, avg=830.05, stdev=214.41 lat (msec): min=210, max=1654, avg=830.44, stdev=214.37 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 514], 10.00th=[ 575], 20.00th=[ 651], | 30.00th=[ 701], 40.00th=[ 751], 50.00th=[ 810], 60.00th=[ 869], | 70.00th=[ 936], 80.00th=[ 1003], 90.00th=[ 1116], 95.00th=[ 1217], | 99.00th=[ 1401], 99.50th=[ 1485], 99.90th=[ 1620], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 7, max= 111, per=24.89%, avg=52.27, stdev=20.24, samples=196 iops : min= 1, max= 27, avg=12.49, stdev= 5.08, samples=196 lat (msec) : 20=0.04%, 50=0.16%, 100=0.27%, 250=2.50%, 500=42.66% lat (msec) : 750=23.52%, 1000=20.62%, 2000=10.23% cpu : usr=0.13%, sys=0.89%, ctx=4386, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145710: Fri Sep 11 06:01:52 2026 read: IOPS=12, BW=51.2KiB/s (52.4kB/s)(5060KiB/98885msec) slat (usec): min=2839, max=65760, avg=14440.91, stdev=7850.61 clat (msec): min=17, max=760, avg=377.72, stdev=97.38 lat (msec): min=30, max=778, avg=392.17, stdev=98.69 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 245], 10.00th=[ 271], 20.00th=[ 300], | 30.00th=[ 330], 40.00th=[ 351], 50.00th=[ 368], 60.00th=[ 393], | 70.00th=[ 418], 80.00th=[ 456], 90.00th=[ 506], 95.00th=[ 550], | 99.00th=[ 642], 99.50th=[ 667], 99.90th=[ 735], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 7, max= 108, per=25.60%, avg=51.72, stdev=22.06, samples=192 iops : min= 1, max= 27, avg=12.37, stdev= 5.56, samples=192 write: IOPS=13, BW=52.4KiB/s (53.6kB/s)(5180KiB/98885msec); 0 zone resets slat (usec): min=31, max=17127, avg=375.40, stdev=1118.97 clat (msec): min=355, max=1584, avg=834.53, stdev=220.86 lat (msec): min=355, max=1585, avg=834.91, stdev=220.82 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 506], 10.00th=[ 550], 20.00th=[ 642], | 30.00th=[ 701], 40.00th=[ 751], 50.00th=[ 810], 60.00th=[ 869], | 70.00th=[ 944], 80.00th=[ 1020], 90.00th=[ 1133], 95.00th=[ 1267], | 99.00th=[ 1401], 99.50th=[ 1469], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 7, max= 127, per=24.68%, avg=51.82, stdev=19.93, samples=195 iops : min= 1, max= 31, avg=12.38, stdev= 4.98, samples=195 lat (msec) : 20=0.08%, 50=0.04%, 100=0.47%, 250=2.27%, 500=43.55% lat (msec) : 750=23.01%, 1000=19.34%, 2000=11.25% cpu : usr=0.13%, sys=0.83%, ctx=4445, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=203KiB/s (207kB/s), 50.2KiB/s-51.2KiB/s (51.4kB/s-52.4kB/s), io=19.6MiB (20.5MB), run=98846-99058msec WRITE: bw=211KiB/s (216kB/s), 52.4KiB/s-53.2KiB/s (53.6kB/s-54.4kB/s), io=20.4MiB (21.4MB), run=98846-99058msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145739: Fri Sep 11 06:02:03 2026 read: IOPS=2, BW=82.8MiB/s (86.8MB/s)(40.0MiB/483msec) slat (nsec): min=33289k, max=33289k, avg=33289184.00, stdev= 0.00 clat (nsec): min=433647k, max=433647k, avg=433647381.00, stdev= 0.00 lat (nsec): min=466941k, max=466941k, avg=466940975.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 435], 10.00th=[ 435], 20.00th=[ 435], | 30.00th=[ 435], 40.00th=[ 435], 50.00th=[ 435], 60.00th=[ 435], | 70.00th=[ 435], 80.00th=[ 435], 90.00th=[ 435], 95.00th=[ 435], | 99.00th=[ 435], 99.50th=[ 435], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=0.21%, ctx=36, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=82.8MiB/s (86.8MB/s), 82.8MiB/s-82.8MiB/s (86.8MB/s-86.8MB/s), io=40.0MiB (41.9MB), run=483-483msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout