== sanity test 398c: run fio to test AIO ================= 14:28:38 (1767036518) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.156129 s, 269 MB/s osc.lustre-OST0000-osc-ffff9a17c762d800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=152608: Mon Dec 29 14:29:16 2025 write: IOPS=75, BW=302KiB/s (310kB/s)(10.0MiB/33855msec); 0 zone resets slat (usec): min=23, max=499, avg=123.65, stdev=59.55 clat (msec): min=33, max=382, avg=211.34, stdev=42.04 lat (msec): min=33, max=382, avg=211.47, stdev=42.04 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 146], 10.00th=[ 163], 20.00th=[ 182], | 30.00th=[ 192], 40.00th=[ 203], 50.00th=[ 211], 60.00th=[ 222], | 70.00th=[ 230], 80.00th=[ 245], 90.00th=[ 262], 95.00th=[ 279], | 99.00th=[ 321], 99.50th=[ 355], 99.90th=[ 368], 99.95th=[ 372], | 99.99th=[ 384] bw ( KiB/s): min= 176, max= 521, per=25.04%, avg=301.96, stdev=47.77, samples=67 iops : min= 44, max= 130, avg=75.48, stdev=11.92, samples=67 lat (msec) : 50=0.62%, 100=0.66%, 250=82.58%, 500=16.13% cpu : usr=0.14%, sys=0.86%, ctx=2646, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=152609: Mon Dec 29 14:29:16 2025 write: IOPS=75, BW=302KiB/s (309kB/s)(10.0MiB/33947msec); 0 zone resets slat (usec): min=22, max=908, avg=128.06, stdev=65.11 clat (msec): min=86, max=369, avg=211.83, stdev=39.45 lat (msec): min=86, max=369, avg=211.96, stdev=39.46 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 148], 10.00th=[ 165], 20.00th=[ 182], | 30.00th=[ 194], 40.00th=[ 203], 50.00th=[ 211], 60.00th=[ 220], | 70.00th=[ 230], 80.00th=[ 243], 90.00th=[ 262], 95.00th=[ 279], | 99.00th=[ 313], 99.50th=[ 338], 99.90th=[ 359], 99.95th=[ 363], | 99.99th=[ 368] bw ( KiB/s): min= 200, max= 438, per=24.94%, avg=300.72, stdev=39.95, samples=67 iops : min= 50, max= 109, avg=75.16, stdev= 9.96, samples=67 lat (msec) : 100=0.66%, 250=84.02%, 500=15.31% cpu : usr=0.16%, sys=0.86%, ctx=2795, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=152610: Mon Dec 29 14:29:16 2025 write: IOPS=75, BW=302KiB/s (309kB/s)(10.0MiB/33921msec); 0 zone resets slat (usec): min=22, max=520, avg=121.70, stdev=60.86 clat (msec): min=46, max=372, avg=211.68, stdev=41.54 lat (msec): min=46, max=372, avg=211.81, stdev=41.55 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 148], 10.00th=[ 163], 20.00th=[ 182], | 30.00th=[ 194], 40.00th=[ 203], 50.00th=[ 211], 60.00th=[ 220], | 70.00th=[ 232], 80.00th=[ 245], 90.00th=[ 264], 95.00th=[ 279], | 99.00th=[ 317], 99.50th=[ 351], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 216, max= 515, per=25.02%, avg=301.75, stdev=45.79, samples=67 iops : min= 54, max= 128, avg=75.42, stdev=11.39, samples=67 lat (msec) : 50=0.62%, 100=0.62%, 250=82.42%, 500=16.33% cpu : usr=0.14%, sys=0.83%, ctx=2681, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=152611: Mon Dec 29 14:29:16 2025 write: IOPS=75, BW=302KiB/s (309kB/s)(10.0MiB/33946msec); 0 zone resets slat (usec): min=24, max=665, avg=126.19, stdev=62.04 clat (msec): min=23, max=373, avg=211.83, stdev=40.97 lat (msec): min=23, max=373, avg=211.96, stdev=40.97 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 150], 10.00th=[ 165], 20.00th=[ 182], | 30.00th=[ 192], 40.00th=[ 203], 50.00th=[ 211], 60.00th=[ 222], | 70.00th=[ 230], 80.00th=[ 245], 90.00th=[ 264], 95.00th=[ 279], | 99.00th=[ 313], 99.50th=[ 342], 99.90th=[ 359], 99.95th=[ 368], | 99.99th=[ 372] bw ( KiB/s): min= 224, max= 509, per=24.95%, avg=300.94, stdev=44.78, samples=67 iops : min= 56, max= 127, avg=75.22, stdev=11.17, samples=67 lat (msec) : 50=0.12%, 100=0.94%, 250=82.89%, 500=16.05% cpu : usr=0.14%, sys=0.86%, ctx=2790, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1207KiB/s (1236kB/s), 302KiB/s-302KiB/s (309kB/s-310kB/s), io=40.0MiB (41.9MB), run=33855-33947msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=152627: Mon Dec 29 14:29:44 2025 read: IOPS=51, BW=207KiB/s (212kB/s)(5048KiB/24389msec) slat (usec): min=1117, max=8113, avg=2828.30, stdev=920.69 clat (msec): min=3, max=193, avg=95.98, stdev=22.51 lat (msec): min=6, max=196, avg=98.81, stdev=22.66 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 67], 10.00th=[ 72], 20.00th=[ 79], | 30.00th=[ 84], 40.00th=[ 89], 50.00th=[ 94], 60.00th=[ 101], | 70.00th=[ 106], 80.00th=[ 114], 90.00th=[ 124], 95.00th=[ 133], | 99.00th=[ 159], 99.50th=[ 169], 99.90th=[ 192], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 112, max= 360, per=25.28%, avg=207.58, stdev=53.69, samples=48 iops : min= 28, max= 90, avg=51.90, stdev=13.42, samples=48 write: IOPS=53, BW=213KiB/s (218kB/s)(5192KiB/24389msec); 0 zone resets slat (usec): min=25, max=2034, avg=130.99, stdev=89.87 clat (msec): min=39, max=380, avg=203.87, stdev=53.85 lat (msec): min=39, max=380, avg=204.00, stdev=53.86 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 126], 10.00th=[ 140], 20.00th=[ 157], | 30.00th=[ 171], 40.00th=[ 186], 50.00th=[ 199], 60.00th=[ 213], | 70.00th=[ 228], 80.00th=[ 247], 90.00th=[ 279], 95.00th=[ 305], | 99.00th=[ 342], 99.50th=[ 351], 99.90th=[ 363], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 104, max= 288, per=24.89%, avg=212.77, stdev=38.17, samples=48 iops : min= 26, max= 72, avg=53.19, stdev= 9.54, samples=48 lat (msec) : 4=0.04%, 10=0.12%, 20=0.16%, 50=0.16%, 100=30.12% lat (msec) : 250=60.04%, 500=9.38% cpu : usr=0.26%, sys=1.69%, ctx=4758, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=152628: Mon Dec 29 14:29:44 2025 read: IOPS=50, BW=204KiB/s (209kB/s)(4976KiB/24425msec) slat (usec): min=34, max=7562, avg=2828.09, stdev=922.41 clat (msec): min=5, max=165, avg=95.81, stdev=20.56 lat (msec): min=7, max=169, avg=98.64, stdev=20.76 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 67], 10.00th=[ 73], 20.00th=[ 80], | 30.00th=[ 85], 40.00th=[ 90], 50.00th=[ 94], 60.00th=[ 100], | 70.00th=[ 105], 80.00th=[ 112], 90.00th=[ 123], 95.00th=[ 131], | 99.00th=[ 150], 99.50th=[ 155], 99.90th=[ 161], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 88, max= 312, per=24.70%, avg=202.75, stdev=56.85, samples=48 iops : min= 22, max= 78, avg=50.69, stdev=14.21, samples=48 write: IOPS=53, BW=216KiB/s (221kB/s)(5264KiB/24425msec); 0 zone resets slat (usec): min=28, max=940, avg=131.28, stdev=70.16 clat (msec): min=30, max=384, avg=203.43, stdev=55.21 lat (msec): min=30, max=384, avg=203.56, stdev=55.22 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 123], 10.00th=[ 140], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 226], 80.00th=[ 243], 90.00th=[ 279], 95.00th=[ 309], | 99.00th=[ 338], 99.50th=[ 347], 99.90th=[ 368], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 128, max= 296, per=25.14%, avg=214.92, stdev=38.16, samples=48 iops : min= 32, max= 74, avg=53.73, stdev= 9.54, samples=48 lat (msec) : 10=0.12%, 20=0.12%, 50=0.51%, 100=30.20%, 250=60.08% lat (msec) : 500=8.98% cpu : usr=0.24%, sys=1.68%, ctx=4701, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=152629: Mon Dec 29 14:29:44 2025 read: IOPS=51, BW=204KiB/s (209kB/s)(4984KiB/24425msec) slat (usec): min=1161, max=8090, avg=2885.01, stdev=940.24 clat (msec): min=10, max=165, avg=96.21, stdev=21.33 lat (msec): min=13, max=169, avg=99.10, stdev=21.52 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 66], 10.00th=[ 71], 20.00th=[ 80], | 30.00th=[ 85], 40.00th=[ 90], 50.00th=[ 95], 60.00th=[ 102], | 70.00th=[ 107], 80.00th=[ 114], 90.00th=[ 125], 95.00th=[ 132], | 99.00th=[ 153], 99.50th=[ 153], 99.90th=[ 163], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 104, max= 328, per=24.95%, avg=204.81, stdev=53.89, samples=48 iops : min= 26, max= 82, avg=51.19, stdev=13.48, samples=48 write: IOPS=53, BW=215KiB/s (220kB/s)(5256KiB/24425msec); 0 zone resets slat (usec): min=37, max=1241, avg=131.06, stdev=76.64 clat (msec): min=36, max=388, avg=202.73, stdev=55.34 lat (msec): min=36, max=388, avg=202.86, stdev=55.35 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 118], 10.00th=[ 138], 20.00th=[ 157], | 30.00th=[ 171], 40.00th=[ 186], 50.00th=[ 199], 60.00th=[ 215], | 70.00th=[ 226], 80.00th=[ 245], 90.00th=[ 275], 95.00th=[ 305], | 99.00th=[ 342], 99.50th=[ 359], 99.90th=[ 376], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 136, max= 320, per=24.96%, avg=213.42, stdev=43.26, samples=48 iops : min= 34, max= 80, avg=53.35, stdev=10.81, samples=48 lat (msec) : 20=0.27%, 50=0.23%, 100=28.91%, 250=61.37%, 500=9.22% cpu : usr=0.23%, sys=1.73%, ctx=4803, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=152630: Mon Dec 29 14:29:44 2025 read: IOPS=51, BW=207KiB/s (212kB/s)(5060KiB/24390msec) slat (usec): min=1213, max=8127, avg=2837.90, stdev=920.45 clat (msec): min=3, max=193, avg=96.54, stdev=21.55 lat (msec): min=5, max=196, avg=99.38, stdev=21.72 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 68], 10.00th=[ 73], 20.00th=[ 80], | 30.00th=[ 85], 40.00th=[ 90], 50.00th=[ 95], 60.00th=[ 101], | 70.00th=[ 106], 80.00th=[ 113], 90.00th=[ 124], 95.00th=[ 133], | 99.00th=[ 155], 99.50th=[ 161], 99.90th=[ 182], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 128, max= 328, per=25.39%, avg=208.48, stdev=43.52, samples=48 iops : min= 32, max= 82, avg=52.10, stdev=10.88, samples=48 write: IOPS=53, BW=212KiB/s (217kB/s)(5180KiB/24390msec); 0 zone resets slat (usec): min=27, max=1137, avg=129.14, stdev=73.48 clat (msec): min=45, max=386, avg=203.61, stdev=55.98 lat (msec): min=45, max=386, avg=203.74, stdev=55.99 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 122], 10.00th=[ 138], 20.00th=[ 157], | 30.00th=[ 171], 40.00th=[ 186], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 228], 80.00th=[ 249], 90.00th=[ 279], 95.00th=[ 309], | 99.00th=[ 342], 99.50th=[ 347], 99.90th=[ 384], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 136, max= 296, per=24.77%, avg=211.81, stdev=41.62, samples=48 iops : min= 34, max= 74, avg=52.94, stdev=10.41, samples=48 lat (msec) : 4=0.04%, 10=0.08%, 20=0.20%, 50=0.31%, 100=30.27% lat (msec) : 250=59.26%, 500=9.84% cpu : usr=0.28%, sys=1.64%, ctx=4794, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=822KiB/s (841kB/s), 204KiB/s-207KiB/s (209kB/s-212kB/s), io=19.6MiB (20.5MB), run=24389-24425msec WRITE: bw=855KiB/s (876kB/s), 212KiB/s-216KiB/s (217kB/s-221kB/s), io=20.4MiB (21.4MB), run=24389-24425msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=152637: Mon Dec 29 14:29:47 2025 read: IOPS=7, BW=315MiB/s (330MB/s)(40.0MiB/127msec) slat (nsec): min=11910k, max=11910k, avg=11909843.00, stdev= 0.00 clat (nsec): min=109879k, max=109879k, avg=109879445.00, stdev= 0.00 lat (nsec): min=121794k, max=121794k, avg=121793908.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 110], 10.00th=[ 110], 20.00th=[ 110], | 30.00th=[ 110], 40.00th=[ 110], 50.00th=[ 110], 60.00th=[ 110], | 70.00th=[ 110], 80.00th=[ 110], 90.00th=[ 110], 95.00th=[ 110], | 99.00th=[ 110], 99.50th=[ 110], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=7.94%, ctx=39, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=315MiB/s (330MB/s), 315MiB/s-315MiB/s (330MB/s-330MB/s), io=40.0MiB (41.9MB), run=127-127msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout