== sanity test 398c: run fio to test AIO ================= 11:10:48 (1789139448) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.422956 s, 99.2 MB/s osc.lustre-OST0000-osc-ffff9b3ca0dea800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180421: Fri Sep 11 11:11:23 2026 write: IOPS=104, BW=418KiB/s (428kB/s)(10.0MiB/24475msec); 0 zone resets slat (usec): min=27, max=15131, avg=253.55, stdev=721.52 clat (msec): min=19, max=271, avg=152.45, stdev=34.11 lat (msec): min=20, max=272, avg=152.70, stdev=34.11 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 95], 10.00th=[ 108], 20.00th=[ 127], | 30.00th=[ 138], 40.00th=[ 146], 50.00th=[ 155], 60.00th=[ 161], | 70.00th=[ 169], 80.00th=[ 178], 90.00th=[ 194], 95.00th=[ 209], | 99.00th=[ 245], 99.50th=[ 255], 99.90th=[ 268], 99.95th=[ 268], | 99.99th=[ 271] bw ( KiB/s): min= 254, max= 596, per=25.01%, avg=417.10, stdev=76.72, samples=48 iops : min= 63, max= 149, avg=103.92, stdev=19.28, samples=48 lat (msec) : 20=0.04%, 50=0.23%, 100=6.45%, 250=92.54%, 500=0.74% cpu : usr=0.25%, sys=1.27%, ctx=3047, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180422: Fri Sep 11 11:11:23 2026 write: IOPS=104, BW=417KiB/s (427kB/s)(10.0MiB/24552msec); 0 zone resets slat (usec): min=28, max=15021, avg=176.53, stdev=606.18 clat (msec): min=19, max=269, avg=152.32, stdev=34.10 lat (msec): min=34, max=269, avg=152.50, stdev=34.09 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 96], 10.00th=[ 107], 20.00th=[ 127], | 30.00th=[ 138], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 161], | 70.00th=[ 169], 80.00th=[ 180], 90.00th=[ 194], 95.00th=[ 207], | 99.00th=[ 236], 99.50th=[ 259], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 271] bw ( KiB/s): min= 254, max= 613, per=24.79%, avg=413.54, stdev=72.53, samples=48 iops : min= 63, max= 153, avg=103.04, stdev=18.20, samples=48 lat (msec) : 20=0.04%, 50=0.12%, 100=7.73%, 250=91.56%, 500=0.55% cpu : usr=0.18%, sys=1.11%, ctx=2154, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180423: Fri Sep 11 11:11:23 2026 write: IOPS=104, BW=417KiB/s (427kB/s)(10.0MiB/24552msec); 0 zone resets slat (usec): min=27, max=13782, avg=214.23, stdev=659.21 clat (msec): min=22, max=271, avg=152.38, stdev=33.18 lat (msec): min=23, max=272, avg=152.59, stdev=33.19 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 94], 10.00th=[ 108], 20.00th=[ 126], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 155], 60.00th=[ 161], | 70.00th=[ 169], 80.00th=[ 182], 90.00th=[ 197], 95.00th=[ 205], | 99.00th=[ 230], 99.50th=[ 245], 99.90th=[ 249], 99.95th=[ 264], | 99.99th=[ 271] bw ( KiB/s): min= 256, max= 622, per=24.82%, avg=414.04, stdev=73.77, samples=48 iops : min= 64, max= 155, avg=103.15, stdev=18.50, samples=48 lat (msec) : 50=0.04%, 100=6.13%, 250=93.75%, 500=0.08% cpu : usr=0.20%, sys=1.14%, ctx=2444, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180424: Fri Sep 11 11:11:23 2026 write: IOPS=104, BW=417KiB/s (427kB/s)(10.0MiB/24556msec); 0 zone resets slat (usec): min=30, max=10502, avg=248.58, stdev=679.65 clat (msec): min=18, max=276, avg=152.41, stdev=33.24 lat (msec): min=19, max=276, avg=152.66, stdev=33.23 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 96], 10.00th=[ 108], 20.00th=[ 127], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 161], | 70.00th=[ 169], 80.00th=[ 180], 90.00th=[ 194], 95.00th=[ 207], | 99.00th=[ 239], 99.50th=[ 251], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 254, max= 574, per=24.88%, avg=415.04, stdev=67.05, samples=48 iops : min= 63, max= 143, avg=103.38, stdev=16.81, samples=48 lat (msec) : 20=0.04%, 50=0.08%, 100=5.82%, 250=93.28%, 500=0.78% cpu : usr=0.17%, sys=1.32%, ctx=2988, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1668KiB/s (1708kB/s), 417KiB/s-418KiB/s (427kB/s-428kB/s), io=40.0MiB (41.9MB), run=24475-24556msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180437: Fri Sep 11 11:11:59 2026 read: IOPS=42, BW=168KiB/s (172kB/s)(5048KiB/30043msec) slat (msec): min=2, max=131, avg=13.01, stdev= 8.26 clat (msec): min=26, max=385, avg=176.73, stdev=40.12 lat (msec): min=87, max=401, avg=189.75, stdev=41.41 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 118], 10.00th=[ 131], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 205], 90.00th=[ 230], 95.00th=[ 247], | 99.00th=[ 300], 99.50th=[ 309], 99.90th=[ 338], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 103, max= 287, per=25.15%, avg=167.76, stdev=36.82, samples=59 iops : min= 25, max= 71, avg=41.53, stdev= 9.20, samples=59 write: IOPS=43, BW=173KiB/s (177kB/s)(5192KiB/30043msec); 0 zone resets slat (usec): min=29, max=12627, avg=280.15, stdev=804.57 clat (msec): min=92, max=389, avg=183.09, stdev=38.88 lat (msec): min=93, max=389, avg=183.38, stdev=38.86 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 127], 10.00th=[ 136], 20.00th=[ 150], | 30.00th=[ 163], 40.00th=[ 171], 50.00th=[ 182], 60.00th=[ 190], | 70.00th=[ 201], 80.00th=[ 211], 90.00th=[ 230], 95.00th=[ 253], | 99.00th=[ 305], 99.50th=[ 309], 99.90th=[ 363], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 87, max= 248, per=24.86%, avg=172.78, stdev=35.54, samples=59 iops : min= 21, max= 62, avg=42.76, stdev= 8.92, samples=59 lat (msec) : 50=0.04%, 100=0.55%, 250=94.45%, 500=4.96% cpu : usr=0.24%, sys=1.83%, ctx=4022, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180438: Fri Sep 11 11:11:59 2026 read: IOPS=41, BW=167KiB/s (171kB/s)(4976KiB/29804msec) slat (usec): min=3040, max=51510, avg=12985.95, stdev=6931.65 clat (msec): min=16, max=370, avg=176.24, stdev=42.11 lat (msec): min=24, max=383, avg=189.24, stdev=43.47 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 118], 10.00th=[ 128], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 209], 90.00th=[ 232], 95.00th=[ 255], | 99.00th=[ 296], 99.50th=[ 305], 99.90th=[ 317], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 112, max= 263, per=24.86%, avg=165.80, stdev=30.61, samples=59 iops : min= 28, max= 65, avg=41.02, stdev= 7.65, samples=59 write: IOPS=44, BW=177KiB/s (181kB/s)(5264KiB/29804msec); 0 zone resets slat (usec): min=32, max=10118, avg=239.06, stdev=664.67 clat (msec): min=91, max=394, avg=181.06, stdev=39.47 lat (msec): min=91, max=394, avg=181.30, stdev=39.48 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 126], 10.00th=[ 136], 20.00th=[ 150], | 30.00th=[ 159], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 211], 90.00th=[ 232], 95.00th=[ 251], | 99.00th=[ 300], 99.50th=[ 313], 99.90th=[ 372], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 94, max= 247, per=25.33%, avg=176.07, stdev=39.18, samples=59 iops : min= 23, max= 61, avg=43.59, stdev= 9.82, samples=59 lat (msec) : 20=0.04%, 50=0.08%, 100=0.47%, 250=93.91%, 500=5.51% cpu : usr=0.31%, sys=1.89%, ctx=4242, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180439: Fri Sep 11 11:11:59 2026 read: IOPS=41, BW=167KiB/s (171kB/s)(4984KiB/29880msec) slat (usec): min=2781, max=54729, avg=12997.72, stdev=6983.93 clat (msec): min=80, max=318, avg=175.36, stdev=38.77 lat (msec): min=93, max=329, avg=188.37, stdev=40.24 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 117], 10.00th=[ 128], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 207], 90.00th=[ 230], 95.00th=[ 245], | 99.00th=[ 279], 99.50th=[ 292], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 79, max= 263, per=25.01%, avg=166.85, stdev=41.66, samples=59 iops : min= 19, max= 65, avg=41.29, stdev=10.40, samples=59 write: IOPS=43, BW=176KiB/s (180kB/s)(5256KiB/29880msec); 0 zone resets slat (usec): min=29, max=10500, avg=245.54, stdev=626.49 clat (msec): min=34, max=370, avg=182.74, stdev=38.46 lat (msec): min=34, max=370, avg=182.99, stdev=38.46 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 129], 10.00th=[ 138], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 169], 50.00th=[ 180], 60.00th=[ 190], | 70.00th=[ 201], 80.00th=[ 213], 90.00th=[ 234], 95.00th=[ 249], | 99.00th=[ 284], 99.50th=[ 300], 99.90th=[ 368], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 94, max= 247, per=25.18%, avg=175.03, stdev=35.29, samples=59 iops : min= 23, max= 61, avg=43.34, stdev= 8.87, samples=59 lat (msec) : 50=0.04%, 100=0.51%, 250=95.20%, 500=4.26% cpu : usr=0.26%, sys=1.85%, ctx=4521, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180440: Fri Sep 11 11:11:59 2026 read: IOPS=42, BW=168KiB/s (172kB/s)(5060KiB/30045msec) slat (usec): min=53, max=203976, avg=13189.22, stdev=11184.12 clat (msec): min=14, max=761, avg=176.12, stdev=51.66 lat (msec): min=14, max=769, avg=189.31, stdev=54.76 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 113], 10.00th=[ 126], 20.00th=[ 142], | 30.00th=[ 157], 40.00th=[ 165], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 205], 90.00th=[ 228], 95.00th=[ 247], | 99.00th=[ 284], 99.50th=[ 317], 99.90th=[ 726], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 23, max= 271, per=25.15%, avg=167.75, stdev=44.25, samples=59 iops : min= 5, max= 67, avg=41.51, stdev=11.08, samples=59 write: IOPS=43, BW=172KiB/s (177kB/s)(5180KiB/30045msec); 0 zone resets slat (usec): min=31, max=10104, avg=264.61, stdev=757.87 clat (msec): min=58, max=760, avg=185.61, stdev=60.89 lat (msec): min=58, max=761, avg=185.87, stdev=60.92 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 125], 10.00th=[ 138], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 169], 50.00th=[ 180], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 213], 90.00th=[ 234], 95.00th=[ 251], | 99.00th=[ 309], 99.50th=[ 743], 99.90th=[ 751], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 31, max= 264, per=24.58%, avg=170.83, stdev=41.68, samples=59 iops : min= 7, max= 66, avg=42.29, stdev=10.46, samples=59 lat (msec) : 20=0.04%, 50=0.16%, 100=1.33%, 250=93.87%, 500=4.06% lat (msec) : 750=0.43%, 1000=0.12% cpu : usr=0.27%, sys=1.93%, ctx=4398, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=668KiB/s (684kB/s), 167KiB/s-168KiB/s (171kB/s-172kB/s), io=19.6MiB (20.5MB), run=29804-30045msec WRITE: bw=695KiB/s (712kB/s), 172KiB/s-177KiB/s (177kB/s-181kB/s), io=20.4MiB (21.4MB), run=29804-30045msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180449: Fri Sep 11 11:12:08 2026 read: IOPS=3, BW=129MiB/s (135MB/s)(40.0MiB/310msec) slat (nsec): min=33094k, max=33094k, avg=33094184.00, stdev= 0.00 clat (nsec): min=264115k, max=264115k, avg=264115100.00, stdev= 0.00 lat (nsec): min=297213k, max=297213k, avg=297213464.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 264], 5.00th=[ 264], 10.00th=[ 264], 20.00th=[ 264], | 30.00th=[ 264], 40.00th=[ 264], 50.00th=[ 264], 60.00th=[ 264], | 70.00th=[ 264], 80.00th=[ 264], 90.00th=[ 264], 95.00th=[ 264], | 99.00th=[ 264], 99.50th=[ 264], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.88%, ctx=8, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=40.0MiB (41.9MB), run=310-310msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout