== sanity test 398c: run fio to test AIO ================= 14:03:39 (1771441419) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.151462 s, 277 MB/s osc.lustre-OST0000-osc-ffffa01f78b2c800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=154740: Wed Feb 18 14:04:12 2026 write: IOPS=85, BW=342KiB/s (350kB/s)(10.0MiB/29954msec); 0 zone resets slat (usec): min=34, max=1262, avg=216.24, stdev=96.20 clat (msec): min=59, max=319, avg=186.80, stdev=35.81 lat (msec): min=59, max=319, avg=187.01, stdev=35.82 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 132], 10.00th=[ 146], 20.00th=[ 163], | 30.00th=[ 171], 40.00th=[ 180], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 203], 80.00th=[ 213], 90.00th=[ 232], 95.00th=[ 247], | 99.00th=[ 271], 99.50th=[ 284], 99.90th=[ 313], 99.95th=[ 317], | 99.99th=[ 321] bw ( KiB/s): min= 256, max= 614, per=24.93%, avg=340.83, stdev=50.21, samples=59 iops : min= 64, max= 153, avg=85.19, stdev=12.51, samples=59 lat (msec) : 100=1.52%, 250=94.61%, 500=3.87% cpu : usr=0.18%, sys=1.49%, ctx=2926, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154741: Wed Feb 18 14:04:12 2026 write: IOPS=85, BW=342KiB/s (350kB/s)(10.0MiB/29947msec); 0 zone resets slat (usec): min=51, max=1099, avg=218.16, stdev=94.63 clat (msec): min=54, max=372, avg=186.71, stdev=36.56 lat (msec): min=55, max=373, avg=186.93, stdev=36.57 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 132], 10.00th=[ 146], 20.00th=[ 161], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 203], 80.00th=[ 215], 90.00th=[ 232], 95.00th=[ 247], | 99.00th=[ 279], 99.50th=[ 296], 99.90th=[ 321], 99.95th=[ 330], | 99.99th=[ 372] bw ( KiB/s): min= 256, max= 614, per=24.95%, avg=341.10, stdev=51.38, samples=59 iops : min= 64, max= 153, avg=85.25, stdev=12.80, samples=59 lat (msec) : 100=1.60%, 250=94.34%, 500=4.06% cpu : usr=0.17%, sys=1.53%, ctx=2884, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154742: Wed Feb 18 14:04:12 2026 write: IOPS=85, BW=343KiB/s (351kB/s)(10.0MiB/29845msec); 0 zone resets slat (usec): min=34, max=774, avg=202.36, stdev=98.47 clat (msec): min=32, max=317, avg=186.23, stdev=38.23 lat (msec): min=33, max=317, avg=186.44, stdev=38.24 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 127], 10.00th=[ 146], 20.00th=[ 159], | 30.00th=[ 169], 40.00th=[ 178], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 205], 80.00th=[ 215], 90.00th=[ 234], 95.00th=[ 249], | 99.00th=[ 271], 99.50th=[ 296], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 256, max= 614, per=25.00%, avg=341.78, stdev=60.44, samples=59 iops : min= 64, max= 153, avg=85.42, stdev=15.07, samples=59 lat (msec) : 50=0.62%, 100=1.25%, 250=93.32%, 500=4.80% cpu : usr=0.16%, sys=1.45%, ctx=2769, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154743: Wed Feb 18 14:04:12 2026 write: IOPS=85, BW=342KiB/s (350kB/s)(10.0MiB/29946msec); 0 zone resets slat (usec): min=43, max=1316, avg=218.18, stdev=104.74 clat (msec): min=34, max=350, avg=186.77, stdev=37.28 lat (msec): min=34, max=351, avg=186.99, stdev=37.29 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 128], 10.00th=[ 146], 20.00th=[ 161], | 30.00th=[ 171], 40.00th=[ 180], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 205], 80.00th=[ 215], 90.00th=[ 232], 95.00th=[ 247], | 99.00th=[ 279], 99.50th=[ 309], 99.90th=[ 321], 99.95th=[ 342], | 99.99th=[ 351] bw ( KiB/s): min= 256, max= 633, per=24.95%, avg=341.02, stdev=54.17, samples=59 iops : min= 64, max= 158, avg=85.24, stdev=13.52, samples=59 lat (msec) : 50=0.35%, 100=1.84%, 250=93.71%, 500=4.10% cpu : usr=0.13%, sys=1.54%, ctx=3015, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1367KiB/s (1400kB/s), 342KiB/s-343KiB/s (350kB/s-351kB/s), io=40.0MiB (41.9MB), run=29845-29954msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=154757: Wed Feb 18 14:04:36 2026 read: IOPS=61, BW=246KiB/s (252kB/s)(5048KiB/20490msec) slat (usec): min=34, max=6563, avg=2567.21, stdev=869.14 clat (msec): min=3, max=245, avg=81.72, stdev=18.54 lat (msec): min=5, max=249, avg=84.29, stdev=18.72 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 59], 10.00th=[ 64], 20.00th=[ 70], | 30.00th=[ 74], 40.00th=[ 79], 50.00th=[ 82], 60.00th=[ 86], | 70.00th=[ 89], 80.00th=[ 94], 90.00th=[ 102], 95.00th=[ 108], | 99.00th=[ 121], 99.50th=[ 138], 99.90th=[ 243], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min= 176, max= 328, per=25.37%, avg=246.30, stdev=43.49, samples=40 iops : min= 44, max= 82, avg=61.55, stdev=10.85, samples=40 write: IOPS=63, BW=253KiB/s (259kB/s)(5192KiB/20490msec); 0 zone resets slat (usec): min=37, max=2207, avg=189.37, stdev=114.85 clat (msec): min=27, max=301, avg=170.19, stdev=42.76 lat (msec): min=27, max=301, avg=170.39, stdev=42.77 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 103], 10.00th=[ 115], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 226], 95.00th=[ 241], | 99.00th=[ 262], 99.50th=[ 268], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 184, max= 325, per=25.13%, avg=254.10, stdev=38.59, samples=40 iops : min= 46, max= 81, avg=63.50, stdev= 9.66, samples=40 lat (msec) : 4=0.04%, 10=0.20%, 20=0.39%, 50=0.66%, 100=44.69% lat (msec) : 250=52.30%, 500=1.72% cpu : usr=0.20%, sys=2.35%, ctx=4841, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154758: Wed Feb 18 14:04:36 2026 read: IOPS=60, BW=242KiB/s (247kB/s)(4976KiB/20596msec) slat (usec): min=966, max=10247, avg=2527.17, stdev=865.79 clat (msec): min=4, max=138, avg=81.32, stdev=15.89 lat (msec): min=7, max=141, avg=83.85, stdev=15.98 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 61], 10.00th=[ 65], 20.00th=[ 70], | 30.00th=[ 74], 40.00th=[ 78], 50.00th=[ 81], 60.00th=[ 85], | 70.00th=[ 89], 80.00th=[ 94], 90.00th=[ 101], 95.00th=[ 108], | 99.00th=[ 120], 99.50th=[ 124], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 144, max= 352, per=24.81%, avg=240.87, stdev=44.67, samples=40 iops : min= 36, max= 88, avg=60.20, stdev=11.19, samples=40 write: IOPS=63, BW=256KiB/s (262kB/s)(5264KiB/20596msec); 0 zone resets slat (usec): min=46, max=2410, avg=195.24, stdev=117.60 clat (msec): min=68, max=273, avg=170.52, stdev=40.32 lat (msec): min=68, max=274, avg=170.71, stdev=40.32 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 104], 10.00th=[ 117], 20.00th=[ 134], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 171], 60.00th=[ 184], | 70.00th=[ 197], 80.00th=[ 207], 90.00th=[ 224], 95.00th=[ 236], | 99.00th=[ 257], 99.50th=[ 268], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 192, max= 336, per=25.16%, avg=254.40, stdev=37.90, samples=40 iops : min= 48, max= 84, avg=63.58, stdev= 9.48, samples=40 lat (msec) : 10=0.12%, 20=0.39%, 50=0.43%, 100=44.57%, 250=53.79% lat (msec) : 500=0.70% cpu : usr=0.20%, sys=2.35%, ctx=4896, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154759: Wed Feb 18 14:04:36 2026 read: IOPS=60, BW=241KiB/s (247kB/s)(4984KiB/20650msec) slat (usec): min=974, max=6381, avg=2524.10, stdev=853.67 clat (msec): min=4, max=137, avg=81.48, stdev=15.11 lat (msec): min=5, max=141, avg=84.00, stdev=15.20 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 61], 10.00th=[ 66], 20.00th=[ 70], | 30.00th=[ 74], 40.00th=[ 79], 50.00th=[ 82], 60.00th=[ 85], | 70.00th=[ 88], 80.00th=[ 93], 90.00th=[ 101], 95.00th=[ 108], | 99.00th=[ 117], 99.50th=[ 124], 99.90th=[ 127], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 128, max= 384, per=24.89%, avg=241.70, stdev=62.64, samples=40 iops : min= 32, max= 96, avg=60.40, stdev=15.68, samples=40 write: IOPS=63, BW=255KiB/s (261kB/s)(5256KiB/20650msec); 0 zone resets slat (usec): min=57, max=1062, avg=187.96, stdev=89.96 clat (msec): min=54, max=275, avg=171.22, stdev=41.42 lat (msec): min=54, max=275, avg=171.41, stdev=41.43 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 104], 10.00th=[ 117], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 161], 50.00th=[ 171], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 209], 90.00th=[ 226], 95.00th=[ 239], | 99.00th=[ 259], 99.50th=[ 268], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 192, max= 328, per=24.99%, avg=252.68, stdev=33.11, samples=40 iops : min= 48, max= 82, avg=63.15, stdev= 8.29, samples=40 lat (msec) : 10=0.16%, 20=0.20%, 50=0.31%, 100=45.43%, 250=52.70% lat (msec) : 500=1.21% cpu : usr=0.25%, sys=2.27%, ctx=4822, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154760: Wed Feb 18 14:04:36 2026 read: IOPS=61, BW=245KiB/s (251kB/s)(5060KiB/20617msec) slat (usec): min=969, max=8205, avg=2560.89, stdev=869.04 clat (msec): min=3, max=152, avg=81.76, stdev=15.81 lat (msec): min=5, max=155, avg=84.33, stdev=15.90 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 61], 10.00th=[ 64], 20.00th=[ 70], | 30.00th=[ 74], 40.00th=[ 79], 50.00th=[ 82], 60.00th=[ 86], | 70.00th=[ 90], 80.00th=[ 94], 90.00th=[ 102], 95.00th=[ 108], | 99.00th=[ 118], 99.50th=[ 124], 99.90th=[ 130], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 144, max= 328, per=25.38%, avg=246.45, stdev=43.99, samples=40 iops : min= 36, max= 82, avg=61.58, stdev=10.99, samples=40 write: IOPS=62, BW=251KiB/s (257kB/s)(5180KiB/20617msec); 0 zone resets slat (usec): min=57, max=3740, avg=193.18, stdev=138.48 clat (msec): min=68, max=274, avg=171.66, stdev=39.49 lat (msec): min=68, max=274, avg=171.85, stdev=39.50 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 108], 10.00th=[ 120], 20.00th=[ 138], | 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 171], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 224], 95.00th=[ 236], | 99.00th=[ 259], 99.50th=[ 266], 99.90th=[ 271], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 184, max= 312, per=24.77%, avg=250.48, stdev=35.72, samples=40 iops : min= 46, max= 78, avg=62.60, stdev= 8.96, samples=40 lat (msec) : 4=0.04%, 10=0.08%, 20=0.27%, 50=0.47%, 100=44.65% lat (msec) : 250=53.36%, 500=1.13% cpu : usr=0.18%, sys=2.39%, ctx=4855, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=972KiB/s (995kB/s), 241KiB/s-246KiB/s (247kB/s-252kB/s), io=19.6MiB (20.5MB), run=20490-20650msec WRITE: bw=1012KiB/s (1036kB/s), 251KiB/s-256KiB/s (257kB/s-262kB/s), io=20.4MiB (21.4MB), run=20490-20650msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=154766: Wed Feb 18 14:04:39 2026 read: IOPS=7, BW=292MiB/s (306MB/s)(40.0MiB/137msec) slat (nsec): min=10811k, max=10811k, avg=10811339.00, stdev= 0.00 clat (nsec): min=120398k, max=120398k, avg=120397711.00, stdev= 0.00 lat (nsec): min=131214k, max=131214k, avg=131213540.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 121], 10.00th=[ 121], 20.00th=[ 121], | 30.00th=[ 121], 40.00th=[ 121], 50.00th=[ 121], 60.00th=[ 121], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 121], 95.00th=[ 121], | 99.00th=[ 121], 99.50th=[ 121], 99.90th=[ 121], 99.95th=[ 121], | 99.99th=[ 121] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.62%, ctx=36, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=292MiB/s (306MB/s), 292MiB/s-292MiB/s (306MB/s-306MB/s), io=40.0MiB (41.9MB), run=137-137msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout