== sanity test 398c: run fio to test AIO ================= 01:25:47 (1788672347) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.486636 s, 86.2 MB/s osc.lustre-OST0000-osc-ffff96fb852c3000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=181080: Sun Sep 6 01:26:24 2026 write: IOPS=86, BW=346KiB/s (354kB/s)(10.0MiB/29624msec); 0 zone resets slat (usec): min=30, max=20137, avg=281.79, stdev=881.23 clat (msec): min=52, max=340, avg=183.65, stdev=40.95 lat (msec): min=52, max=340, avg=183.93, stdev=40.92 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 118], 10.00th=[ 140], 20.00th=[ 155], | 30.00th=[ 163], 40.00th=[ 171], 50.00th=[ 182], 60.00th=[ 190], | 70.00th=[ 201], 80.00th=[ 215], 90.00th=[ 239], 95.00th=[ 255], | 99.00th=[ 284], 99.50th=[ 321], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 342] bw ( KiB/s): min= 254, max= 638, per=24.93%, avg=344.47, stdev=68.27, samples=58 iops : min= 63, max= 159, avg=85.69, stdev=17.06, samples=58 lat (msec) : 100=3.20%, 250=90.35%, 500=6.45% cpu : usr=0.17%, sys=0.96%, ctx=2390, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181081: Sun Sep 6 01:26:24 2026 write: IOPS=86, BW=348KiB/s (356kB/s)(10.0MiB/29452msec); 0 zone resets slat (usec): min=28, max=21034, avg=334.70, stdev=1149.66 clat (msec): min=26, max=342, avg=183.38, stdev=40.68 lat (msec): min=27, max=342, avg=183.72, stdev=40.68 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 125], 10.00th=[ 142], 20.00th=[ 155], | 30.00th=[ 163], 40.00th=[ 171], 50.00th=[ 180], 60.00th=[ 190], | 70.00th=[ 201], 80.00th=[ 215], 90.00th=[ 239], 95.00th=[ 257], | 99.00th=[ 279], 99.50th=[ 288], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 254, max= 630, per=25.07%, avg=346.53, stdev=73.21, samples=58 iops : min= 63, max= 157, avg=86.21, stdev=18.28, samples=58 lat (msec) : 50=0.08%, 100=3.28%, 250=90.35%, 500=6.29% cpu : usr=0.21%, sys=1.00%, ctx=2344, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181082: Sun Sep 6 01:26:24 2026 write: IOPS=86, BW=346KiB/s (354kB/s)(10.0MiB/29599msec); 0 zone resets slat (usec): min=30, max=20681, avg=308.17, stdev=1071.92 clat (msec): min=15, max=337, avg=183.40, stdev=42.17 lat (msec): min=15, max=337, avg=183.71, stdev=42.22 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 117], 10.00th=[ 140], 20.00th=[ 153], | 30.00th=[ 165], 40.00th=[ 171], 50.00th=[ 180], 60.00th=[ 188], | 70.00th=[ 199], 80.00th=[ 218], 90.00th=[ 241], 95.00th=[ 262], | 99.00th=[ 284], 99.50th=[ 296], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 246, max= 520, per=24.94%, avg=344.64, stdev=57.09, samples=58 iops : min= 61, max= 130, avg=85.74, stdev=14.29, samples=58 lat (msec) : 20=0.04%, 50=0.08%, 100=2.19%, 250=89.88%, 500=7.81% cpu : usr=0.27%, sys=1.08%, ctx=2631, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181083: Sun Sep 6 01:26:24 2026 write: IOPS=86, BW=346KiB/s (355kB/s)(10.0MiB/29557msec); 0 zone resets slat (usec): min=29, max=20879, avg=326.78, stdev=1181.42 clat (msec): min=19, max=337, avg=182.94, stdev=41.32 lat (msec): min=19, max=337, avg=183.27, stdev=41.32 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 117], 10.00th=[ 138], 20.00th=[ 153], | 30.00th=[ 161], 40.00th=[ 171], 50.00th=[ 180], 60.00th=[ 190], | 70.00th=[ 201], 80.00th=[ 218], 90.00th=[ 236], 95.00th=[ 257], | 99.00th=[ 284], 99.50th=[ 292], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 214, max= 552, per=25.03%, avg=345.95, stdev=63.89, samples=58 iops : min= 53, max= 138, avg=86.07, stdev=15.94, samples=58 lat (msec) : 20=0.04%, 50=0.08%, 100=2.42%, 250=90.82%, 500=6.64% cpu : usr=0.20%, sys=1.13%, ctx=2418, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1383KiB/s (1416kB/s), 346KiB/s-348KiB/s (354kB/s-356kB/s), io=40.0MiB (41.9MB), run=29452-29624msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=181097: Sun Sep 6 01:27:09 2026 read: IOPS=35, BW=141KiB/s (144kB/s)(5048KiB/35867msec) slat (usec): min=3062, max=71829, avg=15451.72, stdev=9287.62 clat (msec): min=29, max=452, avg=210.29, stdev=61.31 lat (msec): min=37, max=497, avg=225.74, stdev=64.20 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 121], 10.00th=[ 138], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 188], 50.00th=[ 205], 60.00th=[ 220], | 70.00th=[ 239], 80.00th=[ 266], 90.00th=[ 292], 95.00th=[ 317], | 99.00th=[ 372], 99.50th=[ 388], 99.90th=[ 422], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 71, max= 248, per=25.02%, avg=139.85, stdev=40.84, samples=71 iops : min= 17, max= 62, avg=34.51, stdev=10.23, samples=71 write: IOPS=36, BW=145KiB/s (148kB/s)(5192KiB/35867msec); 0 zone resets slat (usec): min=29, max=13974, avg=247.90, stdev=791.35 clat (msec): min=59, max=403, avg=218.76, stdev=61.78 lat (msec): min=59, max=403, avg=219.01, stdev=61.78 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 128], 10.00th=[ 144], 20.00th=[ 165], | 30.00th=[ 182], 40.00th=[ 197], 50.00th=[ 211], 60.00th=[ 230], | 70.00th=[ 249], 80.00th=[ 275], 90.00th=[ 305], 95.00th=[ 334], | 99.00th=[ 368], 99.50th=[ 388], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 40, max= 286, per=24.77%, avg=144.14, stdev=46.01, samples=71 iops : min= 10, max= 71, avg=35.58, stdev=11.58, samples=71 lat (msec) : 50=0.08%, 100=1.37%, 250=70.82%, 500=27.73% cpu : usr=0.25%, sys=1.46%, ctx=3907, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181098: Sun Sep 6 01:27:09 2026 read: IOPS=34, BW=139KiB/s (142kB/s)(4976KiB/35873msec) slat (usec): min=3194, max=75306, avg=16323.75, stdev=9841.18 clat (msec): min=25, max=455, avg=212.28, stdev=61.60 lat (msec): min=41, max=476, avg=228.62, stdev=64.75 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 126], 10.00th=[ 142], 20.00th=[ 159], | 30.00th=[ 176], 40.00th=[ 188], 50.00th=[ 207], 60.00th=[ 226], | 70.00th=[ 243], 80.00th=[ 266], 90.00th=[ 300], 95.00th=[ 317], | 99.00th=[ 363], 99.50th=[ 384], 99.90th=[ 414], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 55, max= 246, per=24.61%, avg=137.58, stdev=39.92, samples=71 iops : min= 13, max= 61, avg=33.92, stdev= 9.98, samples=71 write: IOPS=36, BW=147KiB/s (150kB/s)(5264KiB/35873msec); 0 zone resets slat (usec): min=31, max=25821, avg=293.38, stdev=1151.10 clat (msec): min=69, max=450, avg=217.12, stdev=63.91 lat (msec): min=69, max=451, avg=217.41, stdev=63.88 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 120], 10.00th=[ 138], 20.00th=[ 163], | 30.00th=[ 182], 40.00th=[ 197], 50.00th=[ 211], 60.00th=[ 226], | 70.00th=[ 247], 80.00th=[ 268], 90.00th=[ 305], 95.00th=[ 334], | 99.00th=[ 388], 99.50th=[ 397], 99.90th=[ 439], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 55, max= 286, per=25.15%, avg=146.35, stdev=44.00, samples=71 iops : min= 13, max= 71, avg=36.13, stdev=11.10, samples=71 lat (msec) : 50=0.12%, 100=1.52%, 250=70.90%, 500=27.46% cpu : usr=0.27%, sys=1.63%, ctx=4232, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181099: Sun Sep 6 01:27:09 2026 read: IOPS=34, BW=139KiB/s (143kB/s)(4984KiB/35776msec) slat (usec): min=2957, max=80601, avg=15988.96, stdev=9683.63 clat (msec): min=73, max=419, avg=211.90, stdev=61.00 lat (msec): min=80, max=448, avg=227.90, stdev=63.95 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 122], 10.00th=[ 138], 20.00th=[ 159], | 30.00th=[ 176], 40.00th=[ 192], 50.00th=[ 207], 60.00th=[ 224], | 70.00th=[ 241], 80.00th=[ 266], 90.00th=[ 292], 95.00th=[ 321], | 99.00th=[ 380], 99.50th=[ 393], 99.90th=[ 409], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 63, max= 238, per=24.88%, avg=139.09, stdev=39.42, samples=70 iops : min= 15, max= 59, avg=34.30, stdev= 9.93, samples=70 write: IOPS=36, BW=147KiB/s (150kB/s)(5256KiB/35776msec); 0 zone resets slat (usec): min=25, max=23189, avg=298.03, stdev=1028.15 clat (msec): min=72, max=445, avg=215.91, stdev=65.85 lat (msec): min=72, max=445, avg=216.21, stdev=65.89 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 120], 10.00th=[ 136], 20.00th=[ 159], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 209], 60.00th=[ 228], | 70.00th=[ 249], 80.00th=[ 271], 90.00th=[ 305], 95.00th=[ 330], | 99.00th=[ 397], 99.50th=[ 414], 99.90th=[ 435], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 55, max= 302, per=25.14%, avg=146.31, stdev=48.16, samples=70 iops : min= 13, max= 75, avg=36.13, stdev=12.09, samples=70 lat (msec) : 100=1.52%, 250=71.33%, 500=27.15% cpu : usr=0.30%, sys=1.55%, ctx=4071, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181100: Sun Sep 6 01:27:09 2026 read: IOPS=35, BW=142KiB/s (145kB/s)(5060KiB/35696msec) slat (usec): min=39, max=247783, avg=15899.52, stdev=12138.15 clat (msec): min=24, max=423, avg=212.85, stdev=62.67 lat (msec): min=33, max=449, avg=228.76, stdev=65.75 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 118], 10.00th=[ 136], 20.00th=[ 159], | 30.00th=[ 178], 40.00th=[ 192], 50.00th=[ 207], 60.00th=[ 224], | 70.00th=[ 243], 80.00th=[ 268], 90.00th=[ 300], 95.00th=[ 321], | 99.00th=[ 372], 99.50th=[ 384], 99.90th=[ 409], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 71, max= 270, per=25.31%, avg=141.49, stdev=38.27, samples=70 iops : min= 17, max= 67, avg=34.91, stdev= 9.65, samples=70 write: IOPS=36, BW=145KiB/s (149kB/s)(5180KiB/35696msec); 0 zone resets slat (usec): min=29, max=26277, avg=298.04, stdev=1205.89 clat (msec): min=76, max=443, avg=216.82, stdev=62.31 lat (msec): min=77, max=443, avg=217.12, stdev=62.32 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 126], 10.00th=[ 140], 20.00th=[ 163], | 30.00th=[ 180], 40.00th=[ 194], 50.00th=[ 209], 60.00th=[ 226], | 70.00th=[ 247], 80.00th=[ 268], 90.00th=[ 305], 95.00th=[ 334], | 99.00th=[ 376], 99.50th=[ 393], 99.90th=[ 414], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 71, max= 255, per=24.88%, avg=144.79, stdev=42.88, samples=70 iops : min= 17, max= 63, avg=35.73, stdev=10.74, samples=70 lat (msec) : 50=0.08%, 100=1.48%, 250=70.74%, 500=27.70% cpu : usr=0.31%, sys=1.76%, ctx=4450, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=559KiB/s (573kB/s), 139KiB/s-142KiB/s (142kB/s-145kB/s), io=19.6MiB (20.5MB), run=35696-35873msec WRITE: bw=582KiB/s (596kB/s), 145KiB/s-147KiB/s (148kB/s-150kB/s), io=20.4MiB (21.4MB), run=35696-35873msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=181109: Sun Sep 6 01:27:19 2026 read: IOPS=1, BW=72.1MiB/s (75.6MB/s)(40.0MiB/555msec) slat (nsec): min=40109k, max=40109k, avg=40109039.00, stdev= 0.00 clat (nsec): min=465300k, max=465300k, avg=465300104.00, stdev= 0.00 lat (nsec): min=505414k, max=505414k, avg=505413738.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 464], 10.00th=[ 464], 20.00th=[ 464], | 30.00th=[ 464], 40.00th=[ 464], 50.00th=[ 464], 60.00th=[ 464], | 70.00th=[ 464], 80.00th=[ 464], 90.00th=[ 464], 95.00th=[ 464], | 99.00th=[ 464], 99.50th=[ 464], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.17%, ctx=9, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=72.1MiB/s (75.6MB/s), 72.1MiB/s-72.1MiB/s (75.6MB/s-75.6MB/s), io=40.0MiB (41.9MB), run=555-555msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout