== sanity test 398c: run fio to test AIO ================= 06:13:17 (1788516797) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.723129 s, 58.0 MB/s osc.lustre-OST0000-osc-ffff93535257f800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180096: Fri Sep 4 06:13:55 2026 write: IOPS=90, BW=362KiB/s (371kB/s)(10.0MiB/28269msec); 0 zone resets slat (usec): min=22, max=16722, avg=167.79, stdev=670.41 clat (msec): min=42, max=409, avg=176.31, stdev=45.32 lat (msec): min=42, max=410, avg=176.48, stdev=45.37 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 110], 10.00th=[ 129], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 190], 80.00th=[ 209], 90.00th=[ 228], 95.00th=[ 255], | 99.00th=[ 334], 99.50th=[ 376], 99.90th=[ 401], 99.95th=[ 405], | 99.99th=[ 409] bw ( KiB/s): min= 182, max= 528, per=24.95%, avg=360.79, stdev=79.39, samples=56 iops : min= 45, max= 132, avg=89.82, stdev=19.91, samples=56 lat (msec) : 50=0.12%, 100=3.55%, 250=90.31%, 500=6.02% cpu : usr=0.16%, sys=0.72%, ctx=2116, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180097: Fri Sep 4 06:13:55 2026 write: IOPS=90, BW=362KiB/s (370kB/s)(10.0MiB/28312msec); 0 zone resets slat (usec): min=21, max=25196, avg=202.18, stdev=841.53 clat (msec): min=23, max=401, avg=176.21, stdev=46.66 lat (msec): min=23, max=401, avg=176.41, stdev=46.69 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 111], 10.00th=[ 127], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 213], 90.00th=[ 232], 95.00th=[ 255], | 99.00th=[ 309], 99.50th=[ 388], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 135, max= 535, per=24.95%, avg=360.77, stdev=78.63, samples=56 iops : min= 33, max= 133, avg=89.80, stdev=19.71, samples=56 lat (msec) : 50=0.08%, 100=3.63%, 250=90.55%, 500=5.74% cpu : usr=0.17%, sys=0.81%, ctx=2463, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180098: Fri Sep 4 06:13:55 2026 write: IOPS=90, BW=362KiB/s (370kB/s)(10.0MiB/28309msec); 0 zone resets slat (usec): min=21, max=40881, avg=199.66, stdev=1111.89 clat (msec): min=16, max=384, avg=175.80, stdev=46.67 lat (msec): min=16, max=384, avg=176.00, stdev=46.66 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 112], 10.00th=[ 124], 20.00th=[ 140], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 209], 90.00th=[ 236], 95.00th=[ 255], | 99.00th=[ 309], 99.50th=[ 363], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 128, max= 612, per=24.93%, avg=360.55, stdev=84.50, samples=56 iops : min= 32, max= 153, avg=89.75, stdev=21.18, samples=56 lat (msec) : 20=0.08%, 50=0.16%, 100=3.20%, 250=90.70%, 500=5.86% cpu : usr=0.11%, sys=0.85%, ctx=2066, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180099: Fri Sep 4 06:13:55 2026 write: IOPS=90, BW=362KiB/s (371kB/s)(10.0MiB/28294msec); 0 zone resets slat (usec): min=28, max=19861, avg=226.47, stdev=940.39 clat (msec): min=17, max=415, avg=176.04, stdev=46.74 lat (msec): min=17, max=415, avg=176.27, stdev=46.73 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 108], 10.00th=[ 126], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 180], | 70.00th=[ 192], 80.00th=[ 209], 90.00th=[ 232], 95.00th=[ 259], | 99.00th=[ 326], 99.50th=[ 363], 99.90th=[ 401], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 223, max= 512, per=24.96%, avg=360.89, stdev=67.91, samples=56 iops : min= 55, max= 128, avg=89.84, stdev=17.03, samples=56 lat (msec) : 20=0.04%, 50=0.27%, 100=3.24%, 250=90.08%, 500=6.37% cpu : usr=0.16%, sys=0.91%, ctx=2939, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1447KiB/s (1481kB/s), 362KiB/s-362KiB/s (370kB/s-371kB/s), io=40.0MiB (41.9MB), run=28269-28312msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180115: Fri Sep 4 06:14:40 2026 read: IOPS=35, BW=143KiB/s (147kB/s)(5048KiB/35183msec) slat (usec): min=2795, max=67864, avg=14930.86, stdev=8739.27 clat (msec): min=70, max=375, avg=207.33, stdev=48.77 lat (msec): min=85, max=400, avg=222.26, stdev=50.65 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 136], 10.00th=[ 148], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 205], 60.00th=[ 218], | 70.00th=[ 230], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 342], 99.50th=[ 359], 99.90th=[ 372], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 87, max= 232, per=24.99%, avg=142.46, stdev=31.86, samples=69 iops : min= 21, max= 58, avg=35.09, stdev= 8.01, samples=69 write: IOPS=36, BW=148KiB/s (151kB/s)(5192KiB/35183msec); 0 zone resets slat (usec): min=33, max=12700, avg=159.15, stdev=558.58 clat (msec): min=32, max=382, avg=214.63, stdev=47.96 lat (msec): min=32, max=382, avg=214.79, stdev=47.92 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 146], 10.00th=[ 157], 20.00th=[ 174], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 209], 60.00th=[ 222], | 70.00th=[ 239], 80.00th=[ 255], 90.00th=[ 284], 95.00th=[ 300], | 99.00th=[ 347], 99.50th=[ 368], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 63, max= 222, per=24.81%, avg=147.13, stdev=38.27, samples=69 iops : min= 15, max= 55, avg=36.28, stdev= 9.63, samples=69 lat (msec) : 50=0.04%, 100=0.39%, 250=79.41%, 500=20.16% cpu : usr=0.22%, sys=1.32%, ctx=4084, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180116: Fri Sep 4 06:14:40 2026 read: IOPS=35, BW=143KiB/s (146kB/s)(4976KiB/34900msec) slat (usec): min=3276, max=62156, avg=14839.14, stdev=8251.29 clat (msec): min=56, max=407, avg=207.30, stdev=48.91 lat (msec): min=73, max=426, avg=222.15, stdev=50.73 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 136], 10.00th=[ 146], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 192], 50.00th=[ 205], 60.00th=[ 218], | 70.00th=[ 232], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 338], 99.50th=[ 351], 99.90th=[ 405], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 78, max= 224, per=24.87%, avg=141.75, stdev=33.93, samples=69 iops : min= 19, max= 56, avg=34.93, stdev= 8.55, samples=69 write: IOPS=37, BW=151KiB/s (154kB/s)(5264KiB/34900msec); 0 zone resets slat (usec): min=30, max=19797, avg=183.52, stdev=743.10 clat (msec): min=72, max=438, avg=212.35, stdev=46.44 lat (msec): min=72, max=438, avg=212.53, stdev=46.45 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 144], 10.00th=[ 157], 20.00th=[ 174], | 30.00th=[ 186], 40.00th=[ 194], 50.00th=[ 209], 60.00th=[ 222], | 70.00th=[ 236], 80.00th=[ 253], 90.00th=[ 275], 95.00th=[ 292], | 99.00th=[ 326], 99.50th=[ 347], 99.90th=[ 359], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 71, max= 231, per=25.37%, avg=150.43, stdev=34.06, samples=69 iops : min= 17, max= 57, avg=37.10, stdev= 8.56, samples=69 lat (msec) : 100=0.43%, 250=79.41%, 500=20.16% cpu : usr=0.21%, sys=1.47%, ctx=4029, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180117: Fri Sep 4 06:14:40 2026 read: IOPS=35, BW=142KiB/s (146kB/s)(4984KiB/35011msec) slat (usec): min=52, max=66539, avg=15098.46, stdev=8776.68 clat (msec): min=42, max=422, avg=208.30, stdev=50.92 lat (msec): min=42, max=429, avg=223.41, stdev=52.59 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 138], 10.00th=[ 148], 20.00th=[ 165], | 30.00th=[ 180], 40.00th=[ 192], 50.00th=[ 203], 60.00th=[ 215], | 70.00th=[ 232], 80.00th=[ 249], 90.00th=[ 275], 95.00th=[ 300], | 99.00th=[ 338], 99.50th=[ 359], 99.90th=[ 393], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 55, max= 232, per=24.94%, avg=142.13, stdev=35.09, samples=69 iops : min= 13, max= 58, avg=35.01, stdev= 8.83, samples=69 write: IOPS=37, BW=150KiB/s (154kB/s)(5256KiB/35011msec); 0 zone resets slat (usec): min=30, max=10596, avg=189.45, stdev=622.86 clat (msec): min=82, max=377, avg=213.75, stdev=46.81 lat (msec): min=82, max=377, avg=213.94, stdev=46.83 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 146], 10.00th=[ 161], 20.00th=[ 176], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 211], 60.00th=[ 220], | 70.00th=[ 236], 80.00th=[ 253], 90.00th=[ 275], 95.00th=[ 296], | 99.00th=[ 347], 99.50th=[ 372], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 63, max= 224, per=25.19%, avg=149.36, stdev=38.33, samples=69 iops : min= 15, max= 56, avg=36.83, stdev= 9.62, samples=69 lat (msec) : 50=0.04%, 100=0.70%, 250=79.02%, 500=20.23% cpu : usr=0.25%, sys=1.44%, ctx=4004, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180118: Fri Sep 4 06:14:40 2026 read: IOPS=35, BW=144KiB/s (147kB/s)(5060KiB/35181msec) slat (usec): min=2556, max=72008, avg=15121.01, stdev=8529.77 clat (msec): min=94, max=449, avg=208.33, stdev=48.83 lat (msec): min=102, max=466, avg=223.45, stdev=50.41 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 136], 10.00th=[ 148], 20.00th=[ 165], | 30.00th=[ 180], 40.00th=[ 192], 50.00th=[ 205], 60.00th=[ 220], | 70.00th=[ 232], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 338], 99.50th=[ 372], 99.90th=[ 426], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 72, max= 207, per=25.03%, avg=142.70, stdev=30.94, samples=69 iops : min= 18, max= 51, avg=35.17, stdev= 7.78, samples=69 write: IOPS=36, BW=147KiB/s (151kB/s)(5180KiB/35181msec); 0 zone resets slat (usec): min=33, max=10784, avg=182.05, stdev=611.98 clat (msec): min=51, max=462, avg=213.73, stdev=47.50 lat (msec): min=51, max=462, avg=213.91, stdev=47.50 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 144], 10.00th=[ 159], 20.00th=[ 176], | 30.00th=[ 188], 40.00th=[ 199], 50.00th=[ 209], 60.00th=[ 222], | 70.00th=[ 236], 80.00th=[ 253], 90.00th=[ 275], 95.00th=[ 292], | 99.00th=[ 334], 99.50th=[ 363], 99.90th=[ 451], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 71, max= 240, per=24.64%, avg=146.10, stdev=36.41, samples=69 iops : min= 17, max= 60, avg=36.00, stdev= 9.14, samples=69 lat (msec) : 100=0.16%, 250=80.04%, 500=19.80% cpu : usr=0.25%, sys=1.38%, ctx=4045, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=570KiB/s (584kB/s), 142KiB/s-144KiB/s (146kB/s-147kB/s), io=19.6MiB (20.5MB), run=34900-35183msec WRITE: bw=594KiB/s (608kB/s), 147KiB/s-151KiB/s (151kB/s-154kB/s), io=20.4MiB (21.4MB), run=34900-35183msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180127: Fri Sep 4 06:14:50 2026 read: IOPS=2, BW=83.5MiB/s (87.6MB/s)(40.0MiB/479msec) slat (nsec): min=37752k, max=37752k, avg=37751617.00, stdev= 0.00 clat (nsec): min=411632k, max=411632k, avg=411632214.00, stdev= 0.00 lat (nsec): min=449388k, max=449388k, avg=449387784.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 414], 10.00th=[ 414], 20.00th=[ 414], | 30.00th=[ 414], 40.00th=[ 414], 50.00th=[ 414], 60.00th=[ 414], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.72%, ctx=11, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=83.5MiB/s (87.6MB/s), 83.5MiB/s-83.5MiB/s (87.6MB/s-87.6MB/s), io=40.0MiB (41.9MB), run=479-479msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout