== sanity test 398c: run fio to test AIO ================= 20:37:41 (1786754261) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.63614 s, 65.9 MB/s osc.lustre-OST0000-osc-ffff9ac7c7ddc000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145344: Fri Aug 14 20:40:41 2026 write: IOPS=15, BW=60.7KiB/s (62.1kB/s)(10.0MiB/168716msec); 0 zone resets slat (usec): min=36, max=21045, avg=349.34, stdev=1127.34 clat (msec): min=221, max=2322, avg=1053.30, stdev=330.93 lat (msec): min=222, max=2323, avg=1053.65, stdev=330.92 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 506], 10.00th=[ 642], 20.00th=[ 785], | 30.00th=[ 877], 40.00th=[ 969], 50.00th=[ 1062], 60.00th=[ 1150], | 70.00th=[ 1217], 80.00th=[ 1301], 90.00th=[ 1435], 95.00th=[ 1586], | 99.00th=[ 1989], 99.50th=[ 2165], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 256, per=26.31%, avg=63.66, stdev=35.36, samples=316 iops : min= 1, max= 64, avg=15.43, stdev= 8.87, samples=316 lat (msec) : 250=0.62%, 500=4.18%, 750=11.91%, 1000=25.98%, 2000=56.41% lat (msec) : >=2000=0.90% cpu : usr=0.05%, sys=0.29%, ctx=1688, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145345: Fri Aug 14 20:40:41 2026 write: IOPS=15, BW=60.8KiB/s (62.2kB/s)(10.0MiB/168454msec); 0 zone resets slat (usec): min=36, max=23575, avg=381.57, stdev=1181.04 clat (msec): min=130, max=2293, avg=1051.98, stdev=330.99 lat (msec): min=130, max=2294, avg=1052.36, stdev=330.99 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 493], 10.00th=[ 634], 20.00th=[ 793], | 30.00th=[ 894], 40.00th=[ 978], 50.00th=[ 1062], 60.00th=[ 1133], | 70.00th=[ 1200], 80.00th=[ 1284], 90.00th=[ 1435], 95.00th=[ 1603], | 99.00th=[ 1972], 99.50th=[ 2089], 99.90th=[ 2299], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 199, per=25.26%, avg=61.12, stdev=31.03, samples=329 iops : min= 1, max= 49, avg=14.79, stdev= 7.78, samples=329 lat (msec) : 250=0.82%, 500=4.34%, 750=11.17%, 1000=26.21%, 2000=56.64% lat (msec) : >=2000=0.82% cpu : usr=0.06%, sys=0.32%, ctx=2377, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145346: Fri Aug 14 20:40:41 2026 write: IOPS=15, BW=60.8KiB/s (62.3kB/s)(10.0MiB/168380msec); 0 zone resets slat (usec): min=37, max=21299, avg=354.91, stdev=1119.67 clat (msec): min=197, max=2324, avg=1050.96, stdev=337.01 lat (msec): min=197, max=2324, avg=1051.32, stdev=337.00 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 514], 10.00th=[ 617], 20.00th=[ 785], | 30.00th=[ 877], 40.00th=[ 978], 50.00th=[ 1070], 60.00th=[ 1133], | 70.00th=[ 1200], 80.00th=[ 1301], 90.00th=[ 1452], 95.00th=[ 1620], | 99.00th=[ 1989], 99.50th=[ 2198], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 192, per=25.19%, avg=60.95, stdev=30.68, samples=330 iops : min= 1, max= 48, avg=14.75, stdev= 7.66, samples=330 lat (msec) : 250=0.94%, 500=3.63%, 750=11.72%, 1000=26.05%, 2000=56.68% lat (msec) : >=2000=0.98% cpu : usr=0.05%, sys=0.29%, ctx=2109, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145347: Fri Aug 14 20:40:41 2026 write: IOPS=15, BW=60.7KiB/s (62.2kB/s)(10.0MiB/168675msec); 0 zone resets slat (usec): min=38, max=20559, avg=387.75, stdev=1240.37 clat (msec): min=168, max=2310, avg=1052.67, stdev=333.49 lat (msec): min=168, max=2311, avg=1053.06, stdev=333.46 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 498], 10.00th=[ 634], 20.00th=[ 785], | 30.00th=[ 885], 40.00th=[ 978], 50.00th=[ 1062], 60.00th=[ 1150], | 70.00th=[ 1217], 80.00th=[ 1301], 90.00th=[ 1452], 95.00th=[ 1603], | 99.00th=[ 1955], 99.50th=[ 2165], 99.90th=[ 2299], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 256, per=24.96%, avg=60.40, stdev=29.60, samples=333 iops : min= 1, max= 64, avg=14.61, stdev= 7.43, samples=333 lat (msec) : 250=0.62%, 500=4.61%, 750=11.76%, 1000=25.78%, 2000=56.29% lat (msec) : >=2000=0.94% cpu : usr=0.05%, sys=0.31%, ctx=2101, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=243KiB/s (249kB/s), 60.7KiB/s-60.8KiB/s (62.1kB/s-62.3kB/s), io=40.0MiB (41.9MB), run=168380-168716msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145381: Fri Aug 14 20:42:48 2026 read: IOPS=10, BW=42.7KiB/s (43.7kB/s)(5048KiB/118182msec) slat (usec): min=3294, max=64018, avg=15978.38, stdev=8502.10 clat (msec): min=11, max=1594, avg=461.27, stdev=137.42 lat (msec): min=21, max=1607, avg=477.26, stdev=138.70 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 292], 10.00th=[ 326], 20.00th=[ 368], | 30.00th=[ 393], 40.00th=[ 414], 50.00th=[ 443], 60.00th=[ 472], | 70.00th=[ 502], 80.00th=[ 550], 90.00th=[ 609], 95.00th=[ 676], | 99.00th=[ 869], 99.50th=[ 1183], 99.90th=[ 1435], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min= 7, max= 103, per=25.41%, avg=42.94, stdev=20.49, samples=230 iops : min= 1, max= 25, avg=10.22, stdev= 5.17, samples=230 write: IOPS=10, BW=43.9KiB/s (44.0kB/s)(5192KiB/118182msec); 0 zone resets slat (usec): min=43, max=17988, avg=361.03, stdev=930.95 clat (msec): min=358, max=2025, avg=988.66, stdev=292.19 lat (msec): min=358, max=2027, avg=989.03, stdev=292.22 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 542], 10.00th=[ 609], 20.00th=[ 718], | 30.00th=[ 818], 40.00th=[ 894], 50.00th=[ 978], 60.00th=[ 1053], | 70.00th=[ 1133], 80.00th=[ 1250], 90.00th=[ 1368], 95.00th=[ 1469], | 99.00th=[ 1804], 99.50th=[ 1905], 99.90th=[ 2005], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 103, per=25.20%, avg=44.35, stdev=19.83, samples=229 iops : min= 1, max= 25, avg=10.57, stdev= 4.95, samples=229 lat (msec) : 20=0.08%, 50=0.04%, 100=0.12%, 250=0.59%, 500=34.10% lat (msec) : 750=24.53%, 1000=16.21%, 2000=24.26%, >=2000=0.08% cpu : usr=0.09%, sys=0.73%, ctx=4635, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145382: Fri Aug 14 20:42:48 2026 read: IOPS=10, BW=42.1KiB/s (43.1kB/s)(4976KiB/118183msec) slat (usec): min=2742, max=66218, avg=16024.27, stdev=8716.54 clat (msec): min=21, max=1140, avg=458.28, stdev=115.89 lat (msec): min=34, max=1147, avg=474.31, stdev=117.36 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 305], 10.00th=[ 338], 20.00th=[ 372], | 30.00th=[ 397], 40.00th=[ 418], 50.00th=[ 443], 60.00th=[ 472], | 70.00th=[ 506], 80.00th=[ 542], 90.00th=[ 600], 95.00th=[ 659], | 99.00th=[ 810], 99.50th=[ 869], 99.90th=[ 1011], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 7, max= 111, per=24.96%, avg=42.19, stdev=20.04, samples=231 iops : min= 1, max= 27, avg=10.03, stdev= 5.06, samples=231 write: IOPS=11, BW=44.5KiB/s (45.6kB/s)(5264KiB/118183msec); 0 zone resets slat (usec): min=45, max=17056, avg=387.00, stdev=907.83 clat (msec): min=267, max=1983, avg=985.57, stdev=300.63 lat (msec): min=267, max=1983, avg=985.96, stdev=300.63 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 531], 10.00th=[ 617], 20.00th=[ 718], | 30.00th=[ 802], 40.00th=[ 894], 50.00th=[ 969], 60.00th=[ 1045], | 70.00th=[ 1133], 80.00th=[ 1250], 90.00th=[ 1385], 95.00th=[ 1485], | 99.00th=[ 1854], 99.50th=[ 1921], 99.90th=[ 1972], 99.95th=[ 1989], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 95, per=25.05%, avg=44.09, stdev=19.95, samples=233 iops : min= 1, max= 23, avg=10.50, stdev= 5.00, samples=233 lat (msec) : 50=0.12%, 100=0.16%, 250=0.70%, 500=34.41%, 750=24.06% lat (msec) : 1000=16.99%, 2000=23.55% cpu : usr=0.09%, sys=0.71%, ctx=4490, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145383: Fri Aug 14 20:42:48 2026 read: IOPS=10, BW=42.2KiB/s (43.2kB/s)(4984KiB/118178msec) slat (usec): min=91, max=54054, avg=15492.63, stdev=8559.74 clat (msec): min=12, max=1512, avg=461.39, stdev=127.17 lat (msec): min=17, max=1534, avg=476.89, stdev=128.97 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 300], 10.00th=[ 334], 20.00th=[ 376], | 30.00th=[ 401], 40.00th=[ 422], 50.00th=[ 443], 60.00th=[ 472], | 70.00th=[ 506], 80.00th=[ 550], 90.00th=[ 600], 95.00th=[ 676], | 99.00th=[ 793], 99.50th=[ 869], 99.90th=[ 1502], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 7, max= 119, per=24.91%, avg=42.10, stdev=21.50, samples=232 iops : min= 1, max= 29, avg=10.01, stdev= 5.41, samples=232 write: IOPS=11, BW=44.5KiB/s (45.5kB/s)(5256KiB/118178msec); 0 zone resets slat (usec): min=42, max=18810, avg=370.29, stdev=958.66 clat (msec): min=117, max=2001, avg=985.10, stdev=305.23 lat (msec): min=121, max=2001, avg=985.47, stdev=305.20 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 542], 10.00th=[ 617], 20.00th=[ 726], | 30.00th=[ 810], 40.00th=[ 885], 50.00th=[ 961], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1267], 90.00th=[ 1385], 95.00th=[ 1502], | 99.00th=[ 1838], 99.50th=[ 1888], 99.90th=[ 1955], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 104, per=25.08%, avg=44.13, stdev=20.08, samples=232 iops : min= 1, max= 26, avg=10.52, stdev= 5.02, samples=232 lat (msec) : 20=0.04%, 50=0.20%, 100=0.20%, 250=0.86%, 500=33.12% lat (msec) : 750=24.41%, 1000=18.12%, 2000=23.01%, >=2000=0.04% cpu : usr=0.11%, sys=0.63%, ctx=4735, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145384: Fri Aug 14 20:42:48 2026 read: IOPS=10, BW=42.8KiB/s (43.8kB/s)(5060KiB/118175msec) slat (usec): min=3514, max=52843, avg=15596.95, stdev=8758.54 clat (msec): min=13, max=1513, avg=462.23, stdev=132.79 lat (msec): min=32, max=1534, avg=477.83, stdev=134.08 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 292], 10.00th=[ 330], 20.00th=[ 368], | 30.00th=[ 397], 40.00th=[ 422], 50.00th=[ 443], 60.00th=[ 472], | 70.00th=[ 510], 80.00th=[ 542], 90.00th=[ 609], 95.00th=[ 676], | 99.00th=[ 869], 99.50th=[ 1020], 99.90th=[ 1217], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 7, max= 112, per=25.17%, avg=42.54, stdev=19.17, samples=233 iops : min= 1, max= 28, avg=10.11, stdev= 4.81, samples=233 write: IOPS=10, BW=43.8KiB/s (44.9kB/s)(5180KiB/118175msec); 0 zone resets slat (usec): min=43, max=22327, avg=388.83, stdev=1214.56 clat (msec): min=289, max=1991, avg=990.25, stdev=302.60 lat (msec): min=290, max=1991, avg=990.64, stdev=302.58 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 542], 10.00th=[ 600], 20.00th=[ 709], | 30.00th=[ 818], 40.00th=[ 902], 50.00th=[ 978], 60.00th=[ 1045], | 70.00th=[ 1133], 80.00th=[ 1250], 90.00th=[ 1385], 95.00th=[ 1502], | 99.00th=[ 1854], 99.50th=[ 1888], 99.90th=[ 1972], 99.95th=[ 1989], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 112, per=24.76%, avg=43.57, stdev=19.12, samples=232 iops : min= 1, max= 28, avg=10.38, stdev= 4.82, samples=232 lat (msec) : 20=0.04%, 50=0.12%, 100=0.31%, 250=0.43%, 500=33.63% lat (msec) : 750=25.70%, 1000=15.51%, 2000=24.26% cpu : usr=0.08%, sys=0.71%, ctx=5016, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=170KiB/s (174kB/s), 42.1KiB/s-42.8KiB/s (43.1kB/s-43.8kB/s), io=19.6MiB (20.5MB), run=118175-118183msec WRITE: bw=177KiB/s (181kB/s), 43.8KiB/s-44.5KiB/s (44.9kB/s-45.6kB/s), io=20.4MiB (21.4MB), run=118175-118183msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145405: Fri Aug 14 20:42:58 2026 read: IOPS=1, BW=76.2MiB/s (79.9MB/s)(40.0MiB/525msec) slat (nsec): min=55691k, max=55691k, avg=55691469.00, stdev= 0.00 clat (nsec): min=449074k, max=449074k, avg=449074091.00, stdev= 0.00 lat (nsec): min=504770k, max=504770k, avg=504770248.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 451], 10.00th=[ 451], 20.00th=[ 451], | 30.00th=[ 451], 40.00th=[ 451], 50.00th=[ 451], 60.00th=[ 451], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.67%, ctx=44, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=76.2MiB/s (79.9MB/s), 76.2MiB/s-76.2MiB/s (79.9MB/s-79.9MB/s), io=40.0MiB (41.9MB), run=525-525msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout