== sanity test 398c: run fio to test AIO ================= 20:49:46 (1773103786) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.15639 s, 268 MB/s osc.lustre-OST0000-osc-ffff96f9c6861000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=154452: Mon Mar 9 20:50:23 2026 write: IOPS=75, BW=302KiB/s (309kB/s)(10.0MiB/33927msec); 0 zone resets slat (usec): min=36, max=3885, avg=214.27, stdev=150.38 clat (msec): min=46, max=482, avg=211.60, stdev=48.96 lat (msec): min=46, max=482, avg=211.82, stdev=48.97 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 140], 10.00th=[ 159], 20.00th=[ 176], | 30.00th=[ 186], 40.00th=[ 197], 50.00th=[ 207], 60.00th=[ 220], | 70.00th=[ 232], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 359], 99.50th=[ 376], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 485] bw ( KiB/s): min= 159, max= 516, per=25.00%, avg=301.54, stdev=57.22, samples=67 iops : min= 39, max= 129, avg=75.31, stdev=14.35, samples=67 lat (msec) : 50=0.62%, 100=0.55%, 250=80.31%, 500=18.52% cpu : usr=0.15%, sys=1.29%, ctx=2617, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154453: Mon Mar 9 20:50:23 2026 write: IOPS=75, BW=303KiB/s (310kB/s)(10.0MiB/33791msec); 0 zone resets slat (usec): min=39, max=2462, avg=216.92, stdev=127.16 clat (msec): min=35, max=440, avg=210.84, stdev=48.31 lat (msec): min=35, max=440, avg=211.06, stdev=48.33 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 140], 10.00th=[ 159], 20.00th=[ 176], | 30.00th=[ 186], 40.00th=[ 197], 50.00th=[ 207], 60.00th=[ 218], | 70.00th=[ 230], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 355], 99.50th=[ 372], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 439] bw ( KiB/s): min= 191, max= 516, per=25.03%, avg=301.90, stdev=56.58, samples=67 iops : min= 47, max= 129, avg=75.40, stdev=14.18, samples=67 lat (msec) : 50=0.62%, 100=0.04%, 250=80.86%, 500=18.48% cpu : usr=0.14%, sys=1.30%, ctx=2655, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154454: Mon Mar 9 20:50:23 2026 write: IOPS=75, BW=302KiB/s (309kB/s)(10.0MiB/33939msec); 0 zone resets slat (usec): min=39, max=4020, avg=216.45, stdev=139.79 clat (msec): min=100, max=395, avg=211.62, stdev=45.90 lat (msec): min=100, max=396, avg=211.84, stdev=45.92 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 142], 10.00th=[ 159], 20.00th=[ 178], | 30.00th=[ 188], 40.00th=[ 199], 50.00th=[ 207], 60.00th=[ 218], | 70.00th=[ 232], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 338], 99.50th=[ 368], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 397] bw ( KiB/s): min= 200, max= 445, per=24.94%, avg=300.84, stdev=49.57, samples=67 iops : min= 50, max= 111, avg=75.13, stdev=12.41, samples=67 lat (msec) : 250=81.91%, 500=18.09% cpu : usr=0.16%, sys=1.31%, ctx=2633, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154455: Mon Mar 9 20:50:23 2026 write: IOPS=75, BW=302KiB/s (309kB/s)(10.0MiB/33926msec); 0 zone resets slat (usec): min=37, max=3011, avg=219.51, stdev=122.79 clat (msec): min=23, max=534, avg=211.54, stdev=48.86 lat (msec): min=24, max=534, avg=211.76, stdev=48.86 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 142], 10.00th=[ 159], 20.00th=[ 176], | 30.00th=[ 188], 40.00th=[ 197], 50.00th=[ 207], 60.00th=[ 218], | 70.00th=[ 230], 80.00th=[ 245], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 359], 99.50th=[ 380], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 535] bw ( KiB/s): min= 176, max= 497, per=25.04%, avg=301.97, stdev=53.73, samples=67 iops : min= 44, max= 124, avg=75.42, stdev=13.45, samples=67 lat (msec) : 50=0.08%, 100=0.55%, 250=81.91%, 500=17.34%, 750=0.12% cpu : usr=0.18%, sys=1.29%, ctx=2713, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1207KiB/s (1236kB/s), 302KiB/s-303KiB/s (309kB/s-310kB/s), io=40.0MiB (41.9MB), run=33791-33939msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=154469: Mon Mar 9 20:50:49 2026 read: IOPS=56, BW=225KiB/s (230kB/s)(5048KiB/22458msec) slat (usec): min=1113, max=9388, avg=2797.41, stdev=1041.56 clat (msec): min=6, max=158, avg=87.84, stdev=18.12 lat (msec): min=8, max=164, avg=90.65, stdev=18.28 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 64], 10.00th=[ 68], 20.00th=[ 73], | 30.00th=[ 79], 40.00th=[ 83], 50.00th=[ 88], 60.00th=[ 92], | 70.00th=[ 96], 80.00th=[ 102], 90.00th=[ 111], 95.00th=[ 120], | 99.00th=[ 138], 99.50th=[ 140], 99.90th=[ 155], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 136, max= 304, per=25.11%, avg=224.27, stdev=44.50, samples=44 iops : min= 34, max= 76, avg=55.95, stdev=11.12, samples=44 write: IOPS=57, BW=231KiB/s (237kB/s)(5192KiB/22458msec); 0 zone resets slat (usec): min=57, max=3124, avg=192.18, stdev=128.23 clat (msec): min=64, max=321, avg=187.86, stdev=41.91 lat (msec): min=64, max=321, avg=188.05, stdev=41.92 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 117], 10.00th=[ 131], 20.00th=[ 153], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 188], 60.00th=[ 197], | 70.00th=[ 207], 80.00th=[ 226], 90.00th=[ 243], 95.00th=[ 257], | 99.00th=[ 284], 99.50th=[ 292], 99.90th=[ 305], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 144, max= 280, per=24.73%, avg=229.95, stdev=30.68, samples=44 iops : min= 36, max= 70, avg=57.39, stdev= 7.63, samples=44 lat (msec) : 10=0.12%, 20=0.20%, 50=0.23%, 100=38.83%, 250=56.91% lat (msec) : 500=3.71% cpu : usr=0.28%, sys=2.04%, ctx=4922, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154470: Mon Mar 9 20:50:49 2026 read: IOPS=55, BW=222KiB/s (227kB/s)(4976KiB/22420msec) slat (usec): min=36, max=12195, avg=2836.38, stdev=1121.59 clat (msec): min=3, max=173, avg=87.80, stdev=18.85 lat (msec): min=5, max=175, avg=90.64, stdev=19.05 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 61], 10.00th=[ 67], 20.00th=[ 73], | 30.00th=[ 79], 40.00th=[ 84], 50.00th=[ 88], 60.00th=[ 92], | 70.00th=[ 96], 80.00th=[ 102], 90.00th=[ 112], 95.00th=[ 120], | 99.00th=[ 136], 99.50th=[ 140], 99.90th=[ 171], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 120, max= 358, per=24.80%, avg=221.48, stdev=50.62, samples=44 iops : min= 30, max= 89, avg=55.27, stdev=12.60, samples=44 write: IOPS=58, BW=235KiB/s (240kB/s)(5264KiB/22420msec); 0 zone resets slat (usec): min=37, max=2140, avg=192.79, stdev=116.75 clat (msec): min=31, max=325, avg=186.50, stdev=44.01 lat (msec): min=31, max=325, avg=186.69, stdev=44.01 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 112], 10.00th=[ 131], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 180], 50.00th=[ 188], 60.00th=[ 197], | 70.00th=[ 205], 80.00th=[ 220], 90.00th=[ 245], 95.00th=[ 262], | 99.00th=[ 288], 99.50th=[ 292], 99.90th=[ 313], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 152, max= 311, per=25.23%, avg=234.61, stdev=36.94, samples=44 iops : min= 38, max= 77, avg=58.55, stdev= 9.23, samples=44 lat (msec) : 4=0.04%, 10=0.08%, 20=0.23%, 50=0.98%, 100=37.85% lat (msec) : 250=56.60%, 500=4.22% cpu : usr=0.24%, sys=2.10%, ctx=4882, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154471: Mon Mar 9 20:50:49 2026 read: IOPS=55, BW=222KiB/s (227kB/s)(4984KiB/22448msec) slat (usec): min=1182, max=9637, avg=2874.03, stdev=1063.61 clat (msec): min=3, max=160, avg=87.38, stdev=17.82 lat (msec): min=6, max=164, avg=90.26, stdev=18.01 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 64], 10.00th=[ 69], 20.00th=[ 74], | 30.00th=[ 80], 40.00th=[ 84], 50.00th=[ 88], 60.00th=[ 92], | 70.00th=[ 95], 80.00th=[ 101], 90.00th=[ 109], 95.00th=[ 116], | 99.00th=[ 131], 99.50th=[ 138], 99.90th=[ 159], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 120, max= 304, per=24.85%, avg=221.91, stdev=46.26, samples=44 iops : min= 30, max= 76, avg=55.36, stdev=11.50, samples=44 write: IOPS=58, BW=234KiB/s (240kB/s)(5256KiB/22448msec); 0 zone resets slat (usec): min=53, max=1620, avg=194.99, stdev=111.45 clat (msec): min=43, max=302, avg=187.09, stdev=43.02 lat (msec): min=43, max=302, avg=187.29, stdev=43.02 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 113], 10.00th=[ 132], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 188], 60.00th=[ 197], | 70.00th=[ 207], 80.00th=[ 222], 90.00th=[ 245], 95.00th=[ 259], | 99.00th=[ 288], 99.50th=[ 296], 99.90th=[ 300], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 167, max= 288, per=24.93%, avg=231.86, stdev=36.04, samples=44 iops : min= 41, max= 72, avg=57.86, stdev= 9.01, samples=44 lat (msec) : 4=0.04%, 10=0.16%, 20=0.31%, 50=0.35%, 100=39.61% lat (msec) : 250=55.23%, 500=4.30% cpu : usr=0.17%, sys=2.18%, ctx=4873, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154472: Mon Mar 9 20:50:49 2026 read: IOPS=56, BW=226KiB/s (231kB/s)(5060KiB/22405msec) slat (usec): min=1115, max=9450, avg=2808.95, stdev=1045.96 clat (msec): min=3, max=159, avg=87.63, stdev=17.35 lat (msec): min=5, max=161, avg=90.44, stdev=17.51 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 73], | 30.00th=[ 79], 40.00th=[ 84], 50.00th=[ 88], 60.00th=[ 92], | 70.00th=[ 96], 80.00th=[ 101], 90.00th=[ 109], 95.00th=[ 118], | 99.00th=[ 130], 99.50th=[ 136], 99.90th=[ 155], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 128, max= 312, per=25.44%, avg=227.14, stdev=44.05, samples=44 iops : min= 32, max= 78, avg=56.68, stdev=11.01, samples=44 write: IOPS=57, BW=231KiB/s (237kB/s)(5180KiB/22405msec); 0 zone resets slat (usec): min=35, max=1706, avg=193.30, stdev=120.99 clat (msec): min=45, max=331, avg=187.71, stdev=43.47 lat (msec): min=46, max=332, avg=187.90, stdev=43.48 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 117], 10.00th=[ 131], 20.00th=[ 153], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 188], 60.00th=[ 197], | 70.00th=[ 207], 80.00th=[ 224], 90.00th=[ 243], 95.00th=[ 259], | 99.00th=[ 305], 99.50th=[ 313], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 151, max= 326, per=24.75%, avg=230.18, stdev=37.70, samples=44 iops : min= 37, max= 81, avg=57.45, stdev= 9.42, samples=44 lat (msec) : 4=0.04%, 20=0.12%, 50=0.31%, 100=39.65%, 250=55.94% lat (msec) : 500=3.95% cpu : usr=0.22%, sys=2.14%, ctx=4823, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=894KiB/s (915kB/s), 222KiB/s-226KiB/s (227kB/s-231kB/s), io=19.6MiB (20.5MB), run=22405-22458msec WRITE: bw=930KiB/s (953kB/s), 231KiB/s-235KiB/s (237kB/s-240kB/s), io=20.4MiB (21.4MB), run=22405-22458msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=154479: Mon Mar 9 20:50:53 2026 read: IOPS=8, BW=357MiB/s (374MB/s)(40.0MiB/112msec) slat (nsec): min=9065.4k, max=9065.4k, avg=9065368.00, stdev= 0.00 clat (nsec): min=98294k, max=98294k, avg=98294239.00, stdev= 0.00 lat (nsec): min=107363k, max=107363k, avg=107362807.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[98042], 5.00th=[98042], 10.00th=[98042], 20.00th=[98042], | 30.00th=[98042], 40.00th=[98042], 50.00th=[98042], 60.00th=[98042], | 70.00th=[98042], 80.00th=[98042], 90.00th=[98042], 95.00th=[98042], | 99.00th=[98042], 99.50th=[98042], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=7.21%, ctx=40, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=357MiB/s (374MB/s), 357MiB/s-357MiB/s (374MB/s-374MB/s), io=40.0MiB (41.9MB), run=112-112msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout