== sanity test 398c: run fio to test AIO ================= 09:29:35 (1786109375) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.53144 s, 78.9 MB/s osc.lustre-OST0000-osc-ffff96f8a07e3000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180116: Fri Aug 7 09:30:12 2026 write: IOPS=99, BW=396KiB/s (406kB/s)(10.0MiB/25838msec); 0 zone resets slat (usec): min=36, max=19565, avg=262.90, stdev=825.94 clat (msec): min=25, max=335, avg=160.80, stdev=47.48 lat (msec): min=25, max=335, avg=161.07, stdev=47.49 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 81], 10.00th=[ 95], 20.00th=[ 123], | 30.00th=[ 138], 40.00th=[ 148], 50.00th=[ 161], 60.00th=[ 171], | 70.00th=[ 182], 80.00th=[ 201], 90.00th=[ 222], 95.00th=[ 243], | 99.00th=[ 288], 99.50th=[ 326], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 334] bw ( KiB/s): min= 256, max= 776, per=25.02%, avg=395.55, stdev=109.23, samples=51 iops : min= 64, max= 194, avg=98.53, stdev=27.34, samples=51 lat (msec) : 50=0.12%, 100=11.05%, 250=85.98%, 500=2.85% cpu : usr=0.17%, sys=1.21%, ctx=2646, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180117: Fri Aug 7 09:30:12 2026 write: IOPS=98, BW=395KiB/s (405kB/s)(10.0MiB/25903msec); 0 zone resets slat (usec): min=36, max=14981, avg=246.60, stdev=707.00 clat (msec): min=24, max=335, avg=160.62, stdev=47.83 lat (msec): min=24, max=335, avg=160.87, stdev=47.83 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 81], 10.00th=[ 96], 20.00th=[ 123], | 30.00th=[ 138], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 171], | 70.00th=[ 182], 80.00th=[ 199], 90.00th=[ 228], 95.00th=[ 245], | 99.00th=[ 288], 99.50th=[ 296], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 236, max= 768, per=24.85%, avg=392.86, stdev=113.10, samples=51 iops : min= 59, max= 192, avg=97.86, stdev=28.32, samples=51 lat (msec) : 50=0.12%, 100=10.70%, 250=86.05%, 500=3.12% cpu : usr=0.17%, sys=1.12%, ctx=2307, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180118: Fri Aug 7 09:30:12 2026 write: IOPS=99, BW=397KiB/s (406kB/s)(10.0MiB/25806msec); 0 zone resets slat (usec): min=35, max=14763, avg=253.90, stdev=686.96 clat (msec): min=22, max=334, avg=160.81, stdev=47.17 lat (msec): min=23, max=334, avg=161.07, stdev=47.17 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 83], 10.00th=[ 95], 20.00th=[ 123], | 30.00th=[ 138], 40.00th=[ 150], 50.00th=[ 159], 60.00th=[ 171], | 70.00th=[ 184], 80.00th=[ 201], 90.00th=[ 226], 95.00th=[ 241], | 99.00th=[ 275], 99.50th=[ 300], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 334] bw ( KiB/s): min= 255, max= 768, per=25.04%, avg=395.88, stdev=106.83, samples=51 iops : min= 63, max= 192, avg=98.63, stdev=26.74, samples=51 lat (msec) : 50=0.12%, 100=11.33%, 250=86.52%, 500=2.03% cpu : usr=0.19%, sys=1.26%, ctx=2469, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180119: Fri Aug 7 09:30:12 2026 write: IOPS=99, BW=396KiB/s (406kB/s)(10.0MiB/25852msec); 0 zone resets slat (usec): min=38, max=27791, avg=321.06, stdev=1165.01 clat (msec): min=11, max=356, avg=160.38, stdev=48.50 lat (msec): min=11, max=357, avg=160.70, stdev=48.50 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 82], 10.00th=[ 94], 20.00th=[ 122], | 30.00th=[ 138], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 169], | 70.00th=[ 182], 80.00th=[ 201], 90.00th=[ 224], 95.00th=[ 243], | 99.00th=[ 284], 99.50th=[ 300], 99.90th=[ 355], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 237, max= 768, per=25.02%, avg=395.57, stdev=111.84, samples=51 iops : min= 59, max= 192, avg=98.53, stdev=27.99, samples=51 lat (msec) : 20=0.12%, 50=0.16%, 100=11.33%, 250=85.04%, 500=3.36% cpu : usr=0.24%, sys=1.47%, ctx=3036, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1581KiB/s (1619kB/s), 395KiB/s-397KiB/s (405kB/s-406kB/s), io=40.0MiB (41.9MB), run=25806-25903msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180134: Fri Aug 7 09:30:49 2026 read: IOPS=41, BW=167KiB/s (171kB/s)(5048KiB/30262msec) slat (usec): min=2728, max=55497, avg=13283.00, stdev=7639.85 clat (msec): min=74, max=420, avg=177.08, stdev=48.35 lat (msec): min=81, max=443, avg=190.37, stdev=50.63 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 108], 10.00th=[ 118], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 161], 50.00th=[ 174], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 215], 90.00th=[ 243], 95.00th=[ 262], | 99.00th=[ 309], 99.50th=[ 313], 99.90th=[ 393], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 73, max= 304, per=25.05%, avg=165.83, stdev=44.04, samples=60 iops : min= 18, max= 76, avg=40.93, stdev=11.03, samples=60 write: IOPS=42, BW=172KiB/s (176kB/s)(5192KiB/30262msec); 0 zone resets slat (usec): min=44, max=18598, avg=327.34, stdev=916.11 clat (msec): min=83, max=440, avg=184.62, stdev=48.16 lat (msec): min=83, max=440, avg=184.95, stdev=48.15 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 117], 10.00th=[ 127], 20.00th=[ 146], | 30.00th=[ 159], 40.00th=[ 169], 50.00th=[ 180], 60.00th=[ 192], | 70.00th=[ 205], 80.00th=[ 220], 90.00th=[ 243], 95.00th=[ 264], | 99.00th=[ 342], 99.50th=[ 397], 99.90th=[ 422], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 58, max= 271, per=24.82%, avg=171.03, stdev=43.06, samples=60 iops : min= 14, max= 67, avg=42.23, stdev=10.83, samples=60 lat (msec) : 100=1.84%, 250=90.16%, 500=8.01% cpu : usr=0.22%, sys=1.93%, ctx=4219, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180135: Fri Aug 7 09:30:49 2026 read: IOPS=41, BW=166KiB/s (170kB/s)(4976KiB/29931msec) slat (usec): min=61, max=58513, avg=13697.22, stdev=7718.62 clat (msec): min=19, max=415, avg=176.55, stdev=49.50 lat (msec): min=19, max=439, avg=190.25, stdev=51.77 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 109], 10.00th=[ 121], 20.00th=[ 138], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 171], 60.00th=[ 186], | 70.00th=[ 199], 80.00th=[ 213], 90.00th=[ 239], 95.00th=[ 266], | 99.00th=[ 313], 99.50th=[ 342], 99.90th=[ 363], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 87, max= 312, per=25.01%, avg=165.54, stdev=41.85, samples=59 iops : min= 21, max= 78, avg=40.86, stdev=10.53, samples=59 write: IOPS=43, BW=176KiB/s (180kB/s)(5264KiB/29931msec); 0 zone resets slat (usec): min=51, max=12529, avg=325.44, stdev=828.48 clat (msec): min=18, max=458, avg=183.02, stdev=49.53 lat (msec): min=21, max=458, avg=183.35, stdev=49.49 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 114], 10.00th=[ 127], 20.00th=[ 144], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 190], | 70.00th=[ 203], 80.00th=[ 218], 90.00th=[ 247], 95.00th=[ 268], | 99.00th=[ 326], 99.50th=[ 384], 99.90th=[ 451], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 79, max= 280, per=25.58%, avg=176.22, stdev=41.52, samples=59 iops : min= 19, max= 70, avg=43.53, stdev=10.42, samples=59 lat (msec) : 20=0.12%, 50=0.20%, 100=1.64%, 250=89.96%, 500=8.09% cpu : usr=0.20%, sys=1.84%, ctx=4542, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180136: Fri Aug 7 09:30:49 2026 read: IOPS=41, BW=165KiB/s (168kB/s)(4984KiB/30295msec) slat (usec): min=3036, max=75763, avg=13971.36, stdev=8065.27 clat (msec): min=48, max=447, avg=179.32, stdev=49.68 lat (msec): min=60, max=478, avg=193.30, stdev=51.91 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 110], 10.00th=[ 122], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 188], | 70.00th=[ 203], 80.00th=[ 220], 90.00th=[ 243], 95.00th=[ 266], | 99.00th=[ 330], 99.50th=[ 347], 99.90th=[ 405], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 66, max= 240, per=24.73%, avg=163.73, stdev=40.85, samples=60 iops : min= 16, max= 60, avg=40.40, stdev=10.25, samples=60 write: IOPS=43, BW=173KiB/s (178kB/s)(5256KiB/30295msec); 0 zone resets slat (usec): min=39, max=14867, avg=289.57, stdev=738.06 clat (msec): min=36, max=374, avg=182.73, stdev=50.11 lat (msec): min=36, max=374, avg=183.02, stdev=50.10 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 111], 10.00th=[ 126], 20.00th=[ 138], | 30.00th=[ 153], 40.00th=[ 167], 50.00th=[ 180], 60.00th=[ 192], | 70.00th=[ 205], 80.00th=[ 222], 90.00th=[ 249], 95.00th=[ 271], | 99.00th=[ 330], 99.50th=[ 347], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 79, max= 320, per=24.96%, avg=171.98, stdev=45.22, samples=60 iops : min= 19, max= 80, avg=42.47, stdev=11.35, samples=60 lat (msec) : 50=0.12%, 100=2.30%, 250=88.95%, 500=8.63% cpu : usr=0.24%, sys=1.97%, ctx=4151, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180137: Fri Aug 7 09:30:49 2026 read: IOPS=41, BW=167KiB/s (171kB/s)(5060KiB/30262msec) slat (usec): min=2487, max=65331, avg=13490.49, stdev=8016.51 clat (msec): min=72, max=429, avg=178.31, stdev=48.82 lat (msec): min=77, max=473, avg=191.81, stdev=50.83 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 109], 10.00th=[ 121], 20.00th=[ 138], | 30.00th=[ 150], 40.00th=[ 163], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 199], 80.00th=[ 215], 90.00th=[ 241], 95.00th=[ 271], | 99.00th=[ 313], 99.50th=[ 334], 99.90th=[ 418], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 73, max= 288, per=25.13%, avg=166.38, stdev=38.98, samples=60 iops : min= 18, max= 72, avg=41.07, stdev= 9.80, samples=60 write: IOPS=42, BW=171KiB/s (175kB/s)(5180KiB/30262msec); 0 zone resets slat (usec): min=43, max=8620, avg=304.02, stdev=659.68 clat (msec): min=49, max=422, avg=183.77, stdev=48.55 lat (msec): min=53, max=423, avg=184.08, stdev=48.56 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 114], 10.00th=[ 126], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 169], 50.00th=[ 182], 60.00th=[ 194], | 70.00th=[ 205], 80.00th=[ 220], 90.00th=[ 243], 95.00th=[ 266], | 99.00th=[ 330], 99.50th=[ 363], 99.90th=[ 422], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 88, max= 304, per=24.75%, avg=170.50, stdev=43.47, samples=60 iops : min= 22, max= 76, avg=42.08, stdev=10.86, samples=60 lat (msec) : 50=0.04%, 100=1.64%, 250=90.43%, 500=7.89% cpu : usr=0.20%, sys=1.81%, ctx=4406, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=662KiB/s (678kB/s), 165KiB/s-167KiB/s (168kB/s-171kB/s), io=19.6MiB (20.5MB), run=29931-30295msec WRITE: bw=690KiB/s (706kB/s), 171KiB/s-176KiB/s (175kB/s-180kB/s), io=20.4MiB (21.4MB), run=29931-30295msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180146: Fri Aug 7 09:31:01 2026 read: IOPS=1, BW=77.5MiB/s (81.3MB/s)(40.0MiB/516msec) slat (nsec): min=56538k, max=56538k, avg=56538281.00, stdev= 0.00 clat (nsec): min=446999k, max=446999k, avg=446999375.00, stdev= 0.00 lat (nsec): min=503542k, max=503542k, avg=503542411.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 447], 10.00th=[ 447], 20.00th=[ 447], | 30.00th=[ 447], 40.00th=[ 447], 50.00th=[ 447], 60.00th=[ 447], | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] lat (msec) : 500=100.00% cpu : usr=0.78%, sys=3.30%, ctx=13, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=77.5MiB/s (81.3MB/s), 77.5MiB/s-77.5MiB/s (81.3MB/s-81.3MB/s), io=40.0MiB (41.9MB), run=516-516msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout