== sanity test 398c: run fio to test AIO ================= 14:59:24 (1773687564) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.190106 s, 221 MB/s osc.lustre-OST0000-osc-ffff9f78892a8000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=153003: Mon Mar 16 15:00:05 2026 write: IOPS=69, BW=278KiB/s (284kB/s)(10.0MiB/36874msec); 0 zone resets slat (usec): min=50, max=1825, avg=210.06, stdev=111.71 clat (msec): min=27, max=488, avg=230.00, stdev=51.64 lat (msec): min=27, max=488, avg=230.22, stdev=51.64 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 155], 10.00th=[ 169], 20.00th=[ 190], | 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 230], 60.00th=[ 241], | 70.00th=[ 253], 80.00th=[ 266], 90.00th=[ 292], 95.00th=[ 317], | 99.00th=[ 372], 99.50th=[ 426], 99.90th=[ 489], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 167, max= 466, per=25.00%, avg=277.26, stdev=53.24, samples=73 iops : min= 41, max= 116, avg=69.22, stdev=13.38, samples=73 lat (msec) : 50=0.04%, 100=0.82%, 250=67.34%, 500=31.80% cpu : usr=0.12%, sys=1.19%, ctx=2920, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=153004: Mon Mar 16 15:00:05 2026 write: IOPS=69, BW=277KiB/s (284kB/s)(10.0MiB/36907msec); 0 zone resets slat (usec): min=35, max=3886, avg=194.43, stdev=135.85 clat (msec): min=48, max=521, avg=230.31, stdev=52.28 lat (msec): min=49, max=521, avg=230.51, stdev=52.30 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 155], 10.00th=[ 171], 20.00th=[ 190], | 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 230], 60.00th=[ 241], | 70.00th=[ 253], 80.00th=[ 266], 90.00th=[ 296], 95.00th=[ 317], | 99.00th=[ 376], 99.50th=[ 430], 99.90th=[ 498], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 143, max= 518, per=24.99%, avg=277.10, stdev=59.70, samples=73 iops : min= 35, max= 129, avg=69.18, stdev=14.97, samples=73 lat (msec) : 50=0.62%, 250=67.42%, 500=31.88%, 750=0.08% cpu : usr=0.16%, sys=1.04%, ctx=2870, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=153005: Mon Mar 16 15:00:05 2026 write: IOPS=69, BW=277KiB/s (284kB/s)(10.0MiB/36908msec); 0 zone resets slat (usec): min=36, max=4118, avg=212.04, stdev=127.03 clat (msec): min=75, max=441, avg=230.17, stdev=49.25 lat (msec): min=76, max=441, avg=230.38, stdev=49.25 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 159], 10.00th=[ 171], 20.00th=[ 188], | 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 230], 60.00th=[ 241], | 70.00th=[ 253], 80.00th=[ 266], 90.00th=[ 292], 95.00th=[ 317], | 99.00th=[ 368], 99.50th=[ 388], 99.90th=[ 430], 99.95th=[ 439], | 99.99th=[ 443] bw ( KiB/s): min= 135, max= 424, per=24.97%, avg=276.88, stdev=52.11, samples=73 iops : min= 33, max= 106, avg=69.12, stdev=13.10, samples=73 lat (msec) : 100=0.08%, 250=68.44%, 500=31.48% cpu : usr=0.14%, sys=1.19%, ctx=2937, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=153006: Mon Mar 16 15:00:05 2026 write: IOPS=69, BW=278KiB/s (284kB/s)(10.0MiB/36874msec); 0 zone resets slat (usec): min=51, max=3879, avg=214.84, stdev=139.25 clat (msec): min=67, max=504, avg=229.85, stdev=50.93 lat (msec): min=67, max=506, avg=230.06, stdev=50.94 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 157], 10.00th=[ 169], 20.00th=[ 188], | 30.00th=[ 203], 40.00th=[ 218], 50.00th=[ 230], 60.00th=[ 241], | 70.00th=[ 253], 80.00th=[ 266], 90.00th=[ 292], 95.00th=[ 317], | 99.00th=[ 376], 99.50th=[ 426], 99.90th=[ 464], 99.95th=[ 489], | 99.99th=[ 506] bw ( KiB/s): min= 151, max= 432, per=25.02%, avg=277.42, stdev=51.71, samples=73 iops : min= 37, max= 108, avg=69.26, stdev=13.00, samples=73 lat (msec) : 100=0.12%, 250=68.44%, 500=31.41%, 750=0.04% cpu : usr=0.13%, sys=1.18%, ctx=2936, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1110KiB/s (1136kB/s), 277KiB/s-278KiB/s (284kB/s-284kB/s), io=40.0MiB (41.9MB), run=36874-36908msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=153021: Mon Mar 16 15:00:30 2026 read: IOPS=57, BW=228KiB/s (234kB/s)(5048KiB/22137msec) slat (usec): min=1106, max=7567, avg=2714.14, stdev=970.16 clat (msec): min=2, max=159, avg=86.78, stdev=17.33 lat (msec): min=5, max=163, avg=89.49, stdev=17.45 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 73], | 30.00th=[ 79], 40.00th=[ 83], 50.00th=[ 87], 60.00th=[ 91], | 70.00th=[ 94], 80.00th=[ 99], 90.00th=[ 109], 95.00th=[ 115], | 99.00th=[ 133], 99.50th=[ 138], 99.90th=[ 159], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 144, max= 344, per=25.09%, avg=227.35, stdev=54.31, samples=43 iops : min= 36, max= 86, avg=56.81, stdev=13.59, samples=43 write: IOPS=58, BW=235KiB/s (240kB/s)(5192KiB/22137msec); 0 zone resets slat (usec): min=53, max=2478, avg=189.16, stdev=114.30 clat (msec): min=77, max=310, avg=184.91, stdev=39.69 lat (msec): min=77, max=311, avg=185.10, stdev=39.69 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 125], 10.00th=[ 136], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 174], 50.00th=[ 182], 60.00th=[ 192], | 70.00th=[ 205], 80.00th=[ 220], 90.00th=[ 239], 95.00th=[ 251], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 300], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 160, max= 328, per=24.81%, avg=233.98, stdev=32.75, samples=43 iops : min= 40, max= 82, avg=58.47, stdev= 8.20, samples=43 lat (msec) : 4=0.04%, 10=0.04%, 20=0.20%, 50=0.23%, 100=40.27% lat (msec) : 250=56.37%, 500=2.85% cpu : usr=0.23%, sys=2.08%, ctx=4861, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=153022: Mon Mar 16 15:00:30 2026 read: IOPS=56, BW=225KiB/s (230kB/s)(4976KiB/22147msec) slat (usec): min=1033, max=10242, avg=2793.36, stdev=1060.85 clat (msec): min=2, max=169, avg=86.30, stdev=17.85 lat (msec): min=5, max=172, avg=89.10, stdev=17.99 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 73], | 30.00th=[ 78], 40.00th=[ 82], 50.00th=[ 86], 60.00th=[ 90], | 70.00th=[ 94], 80.00th=[ 100], 90.00th=[ 110], 95.00th=[ 116], | 99.00th=[ 134], 99.50th=[ 138], 99.90th=[ 150], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 120, max= 376, per=24.76%, avg=224.30, stdev=56.73, samples=43 iops : min= 30, max= 94, avg=56.05, stdev=14.21, samples=43 write: IOPS=59, BW=238KiB/s (243kB/s)(5264KiB/22147msec); 0 zone resets slat (usec): min=57, max=4498, avg=187.62, stdev=153.03 clat (msec): min=27, max=312, avg=184.14, stdev=42.13 lat (msec): min=29, max=312, avg=184.33, stdev=42.13 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 120], 10.00th=[ 133], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 171], 50.00th=[ 184], 60.00th=[ 194], | 70.00th=[ 205], 80.00th=[ 218], 90.00th=[ 239], 95.00th=[ 257], | 99.00th=[ 292], 99.50th=[ 296], 99.90th=[ 305], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 176, max= 336, per=25.07%, avg=236.40, stdev=33.18, samples=43 iops : min= 44, max= 84, avg=59.07, stdev= 8.29, samples=43 lat (msec) : 4=0.08%, 20=0.27%, 50=0.43%, 100=39.88%, 250=55.70% lat (msec) : 500=3.63% cpu : usr=0.24%, sys=2.09%, ctx=4866, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=153023: Mon Mar 16 15:00:30 2026 read: IOPS=56, BW=225KiB/s (231kB/s)(4984KiB/22104msec) slat (usec): min=36, max=10646, avg=2790.01, stdev=1059.25 clat (msec): min=3, max=208, avg=86.30, stdev=19.26 lat (msec): min=6, max=211, avg=89.09, stdev=19.44 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 61], 10.00th=[ 66], 20.00th=[ 72], | 30.00th=[ 78], 40.00th=[ 82], 50.00th=[ 87], 60.00th=[ 91], | 70.00th=[ 95], 80.00th=[ 101], 90.00th=[ 110], 95.00th=[ 116], | 99.00th=[ 136], 99.50th=[ 142], 99.90th=[ 165], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 136, max= 336, per=25.01%, avg=226.60, stdev=45.79, samples=43 iops : min= 34, max= 84, avg=56.63, stdev=11.46, samples=43 write: IOPS=59, BW=238KiB/s (243kB/s)(5256KiB/22104msec); 0 zone resets slat (usec): min=37, max=2788, avg=188.24, stdev=133.80 clat (msec): min=44, max=310, avg=184.23, stdev=42.09 lat (msec): min=44, max=311, avg=184.41, stdev=42.11 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 122], 10.00th=[ 132], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 174], 50.00th=[ 184], 60.00th=[ 192], | 70.00th=[ 205], 80.00th=[ 218], 90.00th=[ 239], 95.00th=[ 257], | 99.00th=[ 288], 99.50th=[ 296], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 144, max= 312, per=25.11%, avg=236.74, stdev=36.06, samples=43 iops : min= 36, max= 78, avg=59.16, stdev= 8.99, samples=43 lat (msec) : 4=0.04%, 10=0.12%, 20=0.35%, 50=0.59%, 100=38.95% lat (msec) : 250=56.68%, 500=3.28% cpu : usr=0.22%, sys=2.12%, ctx=4785, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=153024: Mon Mar 16 15:00:30 2026 read: IOPS=57, BW=229KiB/s (234kB/s)(5060KiB/22105msec) slat (usec): min=1058, max=7690, avg=2735.16, stdev=1011.48 clat (msec): min=4, max=177, avg=86.86, stdev=18.77 lat (msec): min=7, max=180, avg=89.60, stdev=18.90 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 63], 10.00th=[ 67], 20.00th=[ 72], | 30.00th=[ 78], 40.00th=[ 83], 50.00th=[ 87], 60.00th=[ 90], | 70.00th=[ 94], 80.00th=[ 100], 90.00th=[ 110], 95.00th=[ 120], | 99.00th=[ 136], 99.50th=[ 146], 99.90th=[ 176], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 104, max= 360, per=25.39%, avg=230.07, stdev=50.94, samples=43 iops : min= 26, max= 90, avg=57.49, stdev=12.74, samples=43 write: IOPS=58, BW=234KiB/s (240kB/s)(5180KiB/22105msec); 0 zone resets slat (usec): min=55, max=1813, avg=186.88, stdev=110.31 clat (msec): min=58, max=312, avg=184.79, stdev=41.64 lat (msec): min=58, max=312, avg=184.98, stdev=41.64 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 124], 10.00th=[ 132], 20.00th=[ 148], | 30.00th=[ 161], 40.00th=[ 174], 50.00th=[ 184], 60.00th=[ 192], | 70.00th=[ 205], 80.00th=[ 220], 90.00th=[ 241], 95.00th=[ 259], | 99.00th=[ 292], 99.50th=[ 305], 99.90th=[ 309], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 168, max= 296, per=24.80%, avg=233.91, stdev=30.74, samples=43 iops : min= 42, max= 74, avg=58.44, stdev= 7.70, samples=43 lat (msec) : 10=0.08%, 20=0.20%, 50=0.66%, 100=39.61%, 250=55.74% lat (msec) : 500=3.71% cpu : usr=0.18%, sys=2.13%, ctx=4856, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=906KiB/s (928kB/s), 225KiB/s-229KiB/s (230kB/s-234kB/s), io=19.6MiB (20.5MB), run=22104-22147msec WRITE: bw=943KiB/s (966kB/s), 234KiB/s-238KiB/s (240kB/s-243kB/s), io=20.4MiB (21.4MB), run=22104-22147msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=153031: Mon Mar 16 15:00:34 2026 read: IOPS=7, BW=299MiB/s (313MB/s)(40.0MiB/134msec) slat (nsec): min=13073k, max=13073k, avg=13073482.00, stdev= 0.00 clat (nsec): min=113564k, max=113564k, avg=113563525.00, stdev= 0.00 lat (nsec): min=126679k, max=126679k, avg=126679478.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 114], 10.00th=[ 114], 20.00th=[ 114], | 30.00th=[ 114], 40.00th=[ 114], 50.00th=[ 114], 60.00th=[ 114], | 70.00th=[ 114], 80.00th=[ 114], 90.00th=[ 114], 95.00th=[ 114], | 99.00th=[ 114], 99.50th=[ 114], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=9.02%, ctx=32, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=299MiB/s (313MB/s), 299MiB/s-299MiB/s (313MB/s-313MB/s), io=40.0MiB (41.9MB), run=134-134msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout