== sanity test 398c: run fio to test AIO ================= 09:20:43 (1790947243) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.45989 s, 91.2 MB/s osc.lustre-OST0000-osc-ffff922d2555e000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=181207: Fri Oct 2 09:21:20 2026 write: IOPS=96, BW=385KiB/s (395kB/s)(10.0MiB/26571msec); 0 zone resets slat (usec): min=22, max=33676, avg=180.75, stdev=936.75 clat (msec): min=38, max=412, avg=165.67, stdev=42.22 lat (msec): min=39, max=412, avg=165.86, stdev=42.22 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 106], 10.00th=[ 120], 20.00th=[ 131], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 161], 60.00th=[ 169], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 224], 95.00th=[ 247], | 99.00th=[ 271], 99.50th=[ 279], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 223, max= 512, per=25.10%, avg=385.33, stdev=83.72, samples=52 iops : min= 55, max= 128, avg=95.96, stdev=21.00, samples=52 lat (msec) : 50=0.04%, 100=2.85%, 250=92.50%, 500=4.61% cpu : usr=0.18%, sys=0.95%, ctx=1242, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181208: Fri Oct 2 09:21:20 2026 write: IOPS=95, BW=384KiB/s (393kB/s)(10.0MiB/26679msec); 0 zone resets slat (usec): min=22, max=13692, avg=180.04, stdev=639.55 clat (msec): min=24, max=363, avg=165.46, stdev=41.25 lat (msec): min=24, max=363, avg=165.64, stdev=41.26 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 112], 10.00th=[ 120], 20.00th=[ 130], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 174], | 70.00th=[ 182], 80.00th=[ 199], 90.00th=[ 226], 95.00th=[ 245], | 99.00th=[ 271], 99.50th=[ 275], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 363] bw ( KiB/s): min= 254, max= 518, per=25.07%, avg=384.79, stdev=80.61, samples=52 iops : min= 63, max= 129, avg=95.85, stdev=20.20, samples=52 lat (msec) : 50=0.23%, 100=0.47%, 250=95.66%, 500=3.63% cpu : usr=0.22%, sys=0.85%, ctx=2246, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181209: Fri Oct 2 09:21:20 2026 write: IOPS=96, BW=385KiB/s (394kB/s)(10.0MiB/26605msec); 0 zone resets slat (usec): min=22, max=18183, avg=236.21, stdev=912.11 clat (msec): min=12, max=389, avg=165.25, stdev=42.30 lat (msec): min=12, max=389, avg=165.49, stdev=42.32 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 108], 10.00th=[ 120], 20.00th=[ 131], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 161], 60.00th=[ 169], | 70.00th=[ 182], 80.00th=[ 201], 90.00th=[ 224], 95.00th=[ 243], | 99.00th=[ 275], 99.50th=[ 288], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 254, max= 526, per=25.12%, avg=385.52, stdev=78.22, samples=52 iops : min= 63, max= 131, avg=95.98, stdev=19.60, samples=52 lat (msec) : 20=0.08%, 50=0.23%, 100=1.95%, 250=93.91%, 500=3.83% cpu : usr=0.23%, sys=1.07%, ctx=2969, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181210: Fri Oct 2 09:21:20 2026 write: IOPS=95, BW=384KiB/s (393kB/s)(10.0MiB/26680msec); 0 zone resets slat (usec): min=23, max=20306, avg=189.54, stdev=813.08 clat (msec): min=19, max=276, avg=165.42, stdev=39.97 lat (msec): min=19, max=276, avg=165.61, stdev=39.95 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 111], 10.00th=[ 121], 20.00th=[ 132], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 157], 60.00th=[ 169], | 70.00th=[ 182], 80.00th=[ 203], 90.00th=[ 224], 95.00th=[ 241], | 99.00th=[ 262], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 191, max= 520, per=24.95%, avg=382.96, stdev=81.86, samples=52 iops : min= 47, max= 130, avg=95.37, stdev=20.48, samples=52 lat (msec) : 20=0.04%, 100=1.64%, 250=94.92%, 500=3.40% cpu : usr=0.16%, sys=0.99%, ctx=2111, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1535KiB/s (1572kB/s), 384KiB/s-385KiB/s (393kB/s-395kB/s), io=40.0MiB (41.9MB), run=26571-26680msec 0% of I/O are 1-page (PAGE_SIZE=4096), max 50% mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=181224: Fri Oct 2 09:21:58 2026 read: IOPS=45, BW=182KiB/s (186kB/s)(5048KiB/27790msec) slat (usec): min=45, max=46519, avg=12280.07, stdev=6783.83 clat (msec): min=17, max=377, avg=165.51, stdev=45.34 lat (msec): min=29, max=388, avg=177.79, stdev=47.03 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 103], 10.00th=[ 114], 20.00th=[ 130], | 30.00th=[ 140], 40.00th=[ 150], 50.00th=[ 161], 60.00th=[ 171], | 70.00th=[ 184], 80.00th=[ 201], 90.00th=[ 222], 95.00th=[ 243], | 99.00th=[ 305], 99.50th=[ 347], 99.90th=[ 368], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 71, max= 265, per=25.38%, avg=181.48, stdev=38.09, samples=54 iops : min= 17, max= 66, avg=44.94, stdev= 9.58, samples=54 write: IOPS=46, BW=187KiB/s (191kB/s)(5192KiB/27790msec); 0 zone resets slat (usec): min=26, max=15660, avg=248.33, stdev=927.47 clat (msec): min=20, max=357, avg=168.81, stdev=42.66 lat (msec): min=20, max=357, avg=169.06, stdev=42.65 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 110], 10.00th=[ 120], 20.00th=[ 133], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 163], 60.00th=[ 176], | 70.00th=[ 188], 80.00th=[ 201], 90.00th=[ 224], 95.00th=[ 247], | 99.00th=[ 296], 99.50th=[ 313], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 95, max= 271, per=25.09%, avg=186.67, stdev=42.11, samples=54 iops : min= 23, max= 67, avg=46.24, stdev=10.67, samples=54 lat (msec) : 20=0.04%, 50=0.23%, 100=3.01%, 250=92.50%, 500=4.22% cpu : usr=0.23%, sys=1.91%, ctx=4479, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181225: Fri Oct 2 09:21:58 2026 read: IOPS=44, BW=177KiB/s (182kB/s)(4976KiB/28058msec) slat (usec): min=2167, max=52126, avg=12302.74, stdev=6593.79 clat (msec): min=17, max=405, avg=166.51, stdev=46.03 lat (msec): min=27, max=423, avg=178.82, stdev=47.76 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 103], 10.00th=[ 113], 20.00th=[ 128], | 30.00th=[ 140], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 186], 80.00th=[ 201], 90.00th=[ 228], 95.00th=[ 251], | 99.00th=[ 313], 99.50th=[ 334], 99.90th=[ 376], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 95, max= 303, per=24.60%, avg=175.85, stdev=41.17, samples=55 iops : min= 23, max= 75, avg=43.53, stdev=10.33, samples=55 write: IOPS=46, BW=188KiB/s (192kB/s)(5264KiB/28058msec); 0 zone resets slat (usec): min=23, max=9809, avg=191.73, stdev=558.97 clat (msec): min=58, max=400, avg=169.28, stdev=41.92 lat (msec): min=58, max=400, avg=169.47, stdev=41.92 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 111], 10.00th=[ 122], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 222], 95.00th=[ 241], | 99.00th=[ 300], 99.50th=[ 321], 99.90th=[ 397], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 87, max= 271, per=25.23%, avg=187.69, stdev=44.86, samples=55 iops : min= 21, max= 67, avg=46.47, stdev=11.29, samples=55 lat (msec) : 20=0.04%, 50=0.04%, 100=3.16%, 250=92.62%, 500=4.14% cpu : usr=0.26%, sys=2.02%, ctx=4336, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181226: Fri Oct 2 09:21:58 2026 read: IOPS=44, BW=178KiB/s (182kB/s)(4984KiB/28059msec) slat (usec): min=2308, max=63196, avg=12456.64, stdev=7001.23 clat (msec): min=72, max=339, avg=163.84, stdev=42.37 lat (msec): min=77, max=362, avg=176.30, stdev=44.07 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 102], 10.00th=[ 115], 20.00th=[ 129], | 30.00th=[ 140], 40.00th=[ 150], 50.00th=[ 159], 60.00th=[ 169], | 70.00th=[ 182], 80.00th=[ 194], 90.00th=[ 220], 95.00th=[ 245], | 99.00th=[ 288], 99.50th=[ 313], 99.90th=[ 338], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 95, max= 287, per=24.79%, avg=177.22, stdev=47.42, samples=55 iops : min= 23, max= 71, avg=43.85, stdev=11.92, samples=55 write: IOPS=46, BW=187KiB/s (192kB/s)(5256KiB/28059msec); 0 zone resets slat (usec): min=25, max=14034, avg=206.04, stdev=648.03 clat (msec): min=78, max=399, avg=171.24, stdev=44.58 lat (msec): min=78, max=399, avg=171.45, stdev=44.59 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 108], 10.00th=[ 120], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 188], 80.00th=[ 201], 90.00th=[ 226], 95.00th=[ 247], | 99.00th=[ 330], 99.50th=[ 351], 99.90th=[ 397], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 95, max= 288, per=24.99%, avg=185.93, stdev=44.83, samples=55 iops : min= 23, max= 72, avg=46.04, stdev=11.28, samples=55 lat (msec) : 100=3.40%, 250=92.07%, 500=4.53% cpu : usr=0.29%, sys=1.80%, ctx=4074, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181227: Fri Oct 2 09:21:58 2026 read: IOPS=45, BW=181KiB/s (185kB/s)(5060KiB/28024msec) slat (usec): min=2473, max=82851, avg=12132.32, stdev=6894.71 clat (msec): min=23, max=375, avg=163.42, stdev=41.00 lat (msec): min=45, max=388, avg=175.56, stdev=42.52 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 104], 10.00th=[ 114], 20.00th=[ 129], | 30.00th=[ 142], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 171], | 70.00th=[ 180], 80.00th=[ 194], 90.00th=[ 218], 95.00th=[ 234], | 99.00th=[ 275], 99.50th=[ 300], 99.90th=[ 363], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 79, max= 272, per=25.17%, avg=179.96, stdev=41.28, samples=55 iops : min= 19, max= 68, avg=44.56, stdev=10.42, samples=55 write: IOPS=46, BW=185KiB/s (189kB/s)(5180KiB/28024msec); 0 zone resets slat (usec): min=31, max=9438, avg=244.81, stdev=772.14 clat (msec): min=22, max=366, avg=171.92, stdev=43.81 lat (msec): min=22, max=366, avg=172.17, stdev=43.84 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 113], 10.00th=[ 122], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 157], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 188], 80.00th=[ 203], 90.00th=[ 234], 95.00th=[ 253], | 99.00th=[ 300], 99.50th=[ 321], 99.90th=[ 363], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 103, max= 264, per=24.78%, avg=184.38, stdev=35.56, samples=55 iops : min= 25, max= 66, avg=45.67, stdev= 8.97, samples=55 lat (msec) : 50=0.12%, 100=2.62%, 250=93.09%, 500=4.18% cpu : usr=0.35%, sys=1.88%, ctx=4278, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=715KiB/s (732kB/s), 177KiB/s-182KiB/s (182kB/s-186kB/s), io=19.6MiB (20.5MB), run=27790-28059msec WRITE: bw=745KiB/s (762kB/s), 185KiB/s-188KiB/s (189kB/s-192kB/s), io=20.4MiB (21.4MB), run=27790-28059msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=181235: Fri Oct 2 09:22:07 2026 read: IOPS=1, BW=71.9MiB/s (75.4MB/s)(40.0MiB/556msec) slat (nsec): min=51664k, max=51664k, avg=51664437.00, stdev= 0.00 clat (nsec): min=485917k, max=485917k, avg=485917073.00, stdev= 0.00 lat (nsec): min=537586k, max=537586k, avg=537586137.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 485], 10.00th=[ 485], 20.00th=[ 485], | 30.00th=[ 485], 40.00th=[ 485], 50.00th=[ 485], 60.00th=[ 485], | 70.00th=[ 485], 80.00th=[ 485], 90.00th=[ 485], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] lat (msec) : 500=100.00% cpu : usr=0.18%, sys=2.70%, ctx=7, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=71.9MiB/s (75.4MB/s), 71.9MiB/s-71.9MiB/s (75.4MB/s-75.4MB/s), io=40.0MiB (41.9MB), run=556-556msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout