== sanity test 398c: run fio to test AIO ================= 22:12:00 (1781662320) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.118181 s, 355 MB/s osc.lustre-OST0000-osc-ffff94ae85324000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=159847: Tue Jun 16 22:12:30 2026 write: IOPS=95, BW=381KiB/s (390kB/s)(10.0MiB/26877msec); 0 zone resets slat (usec): min=43, max=756, avg=199.03, stdev=91.31 clat (msec): min=58, max=312, avg=167.61, stdev=32.97 lat (msec): min=58, max=312, avg=167.81, stdev=32.98 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 114], 10.00th=[ 130], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 169], 60.00th=[ 176], | 70.00th=[ 184], 80.00th=[ 192], 90.00th=[ 205], 95.00th=[ 215], | 99.00th=[ 253], 99.50th=[ 288], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 313] bw ( KiB/s): min= 224, max= 646, per=24.89%, avg=379.13, stdev=55.78, samples=53 iops : min= 56, max= 161, avg=94.77, stdev=13.90, samples=53 lat (msec) : 100=3.36%, 250=95.39%, 500=1.25% cpu : usr=0.18%, sys=1.51%, ctx=2761, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=159848: Tue Jun 16 22:12:30 2026 write: IOPS=95, BW=382KiB/s (391kB/s)(10.0MiB/26826msec); 0 zone resets slat (usec): min=35, max=2105, avg=194.97, stdev=99.42 clat (msec): min=33, max=325, avg=167.37, stdev=33.84 lat (msec): min=33, max=325, avg=167.57, stdev=33.86 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 113], 10.00th=[ 130], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 176], | 70.00th=[ 182], 80.00th=[ 190], 90.00th=[ 207], 95.00th=[ 218], | 99.00th=[ 249], 99.50th=[ 292], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 326] bw ( KiB/s): min= 248, max= 709, per=24.95%, avg=380.02, stdev=59.15, samples=53 iops : min= 62, max= 177, avg=95.00, stdev=14.76, samples=53 lat (msec) : 50=0.62%, 100=3.24%, 250=95.16%, 500=0.98% cpu : usr=0.13%, sys=1.51%, ctx=2633, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=159849: Tue Jun 16 22:12:30 2026 write: IOPS=95, BW=381KiB/s (390kB/s)(10.0MiB/26877msec); 0 zone resets slat (usec): min=42, max=757, avg=206.12, stdev=90.40 clat (msec): min=54, max=312, avg=167.56, stdev=32.91 lat (msec): min=54, max=313, avg=167.77, stdev=32.91 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 114], 10.00th=[ 130], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 169], 60.00th=[ 176], | 70.00th=[ 182], 80.00th=[ 192], 90.00th=[ 205], 95.00th=[ 215], | 99.00th=[ 251], 99.50th=[ 284], 99.90th=[ 296], 99.95th=[ 300], | 99.99th=[ 313] bw ( KiB/s): min= 232, max= 646, per=24.91%, avg=379.43, stdev=53.60, samples=53 iops : min= 58, max= 161, avg=94.85, stdev=13.35, samples=53 lat (msec) : 100=3.24%, 250=95.70%, 500=1.05% cpu : usr=0.17%, sys=1.56%, ctx=2796, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=159850: Tue Jun 16 22:12:30 2026 write: IOPS=95, BW=382KiB/s (391kB/s)(10.0MiB/26805msec); 0 zone resets slat (usec): min=41, max=656, avg=206.21, stdev=86.26 clat (msec): min=30, max=303, avg=167.17, stdev=33.64 lat (msec): min=30, max=303, avg=167.38, stdev=33.64 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 115], 10.00th=[ 129], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 169], 60.00th=[ 176], | 70.00th=[ 182], 80.00th=[ 192], 90.00th=[ 205], 95.00th=[ 215], | 99.00th=[ 253], 99.50th=[ 284], 99.90th=[ 300], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 256, max= 697, per=24.98%, avg=380.40, stdev=57.76, samples=53 iops : min= 64, max= 174, avg=95.09, stdev=14.41, samples=53 lat (msec) : 50=0.47%, 100=3.05%, 250=95.39%, 500=1.09% cpu : usr=0.16%, sys=1.60%, ctx=2820, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1524KiB/s (1561kB/s), 381KiB/s-382KiB/s (390kB/s-391kB/s), io=40.0MiB (41.9MB), run=26805-26877msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=159863: Tue Jun 16 22:12:51 2026 read: IOPS=68, BW=273KiB/s (279kB/s)(5048KiB/18506msec) slat (usec): min=1047, max=6312, avg=2304.57, stdev=729.24 clat (msec): min=2, max=159, avg=72.95, stdev=14.44 lat (msec): min=4, max=161, avg=75.26, stdev=14.51 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 53], 10.00th=[ 58], 20.00th=[ 64], | 30.00th=[ 66], 40.00th=[ 70], 50.00th=[ 73], 60.00th=[ 75], | 70.00th=[ 80], 80.00th=[ 83], 90.00th=[ 90], 95.00th=[ 96], | 99.00th=[ 110], 99.50th=[ 120], 99.90th=[ 138], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 160, max= 352, per=25.05%, avg=271.28, stdev=45.00, samples=36 iops : min= 40, max= 88, avg=67.81, stdev=11.25, samples=36 write: IOPS=70, BW=281KiB/s (287kB/s)(5192KiB/18506msec); 0 zone resets slat (usec): min=43, max=1428, avg=179.09, stdev=97.17 clat (msec): min=44, max=269, avg=154.27, stdev=33.20 lat (msec): min=44, max=270, avg=154.45, stdev=33.20 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 102], 10.00th=[ 114], 20.00th=[ 126], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 163], | 70.00th=[ 171], 80.00th=[ 182], 90.00th=[ 199], 95.00th=[ 211], | 99.00th=[ 241], 99.50th=[ 249], 99.90th=[ 259], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 184, max= 336, per=24.83%, avg=280.11, stdev=36.76, samples=36 iops : min= 46, max= 84, avg=70.00, stdev= 9.18, samples=36 lat (msec) : 4=0.16%, 20=0.23%, 50=1.33%, 100=48.09%, 250=50.00% lat (msec) : 500=0.20% cpu : usr=0.17%, sys=2.31%, ctx=4881, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=159864: Tue Jun 16 22:12:51 2026 read: IOPS=67, BW=269KiB/s (275kB/s)(4976KiB/18513msec) slat (usec): min=986, max=6875, avg=2275.84, stdev=754.43 clat (msec): min=4, max=129, avg=72.85, stdev=14.06 lat (msec): min=6, max=132, avg=75.13, stdev=14.16 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 54], 10.00th=[ 58], 20.00th=[ 63], | 30.00th=[ 67], 40.00th=[ 70], 50.00th=[ 72], 60.00th=[ 75], | 70.00th=[ 80], 80.00th=[ 84], 90.00th=[ 90], 95.00th=[ 99], | 99.00th=[ 110], 99.50th=[ 113], 99.90th=[ 123], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 152, max= 376, per=24.73%, avg=267.83, stdev=55.37, samples=36 iops : min= 38, max= 94, avg=66.94, stdev=13.83, samples=36 write: IOPS=71, BW=284KiB/s (291kB/s)(5264KiB/18513msec); 0 zone resets slat (usec): min=36, max=611, avg=175.82, stdev=84.82 clat (msec): min=30, max=266, avg=153.55, stdev=34.21 lat (msec): min=30, max=266, avg=153.73, stdev=34.22 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 100], 10.00th=[ 115], 20.00th=[ 126], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 163], | 70.00th=[ 171], 80.00th=[ 182], 90.00th=[ 199], 95.00th=[ 213], | 99.00th=[ 236], 99.50th=[ 255], 99.90th=[ 262], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 200, max= 360, per=25.06%, avg=282.67, stdev=40.81, samples=36 iops : min= 50, max= 90, avg=70.64, stdev=10.19, samples=36 lat (msec) : 10=0.08%, 20=0.23%, 50=1.41%, 100=47.34%, 250=50.62% lat (msec) : 500=0.31% cpu : usr=0.21%, sys=2.29%, ctx=4851, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=159865: Tue Jun 16 22:12:51 2026 read: IOPS=67, BW=269KiB/s (276kB/s)(4984KiB/18497msec) slat (usec): min=39, max=7106, avg=2283.39, stdev=770.68 clat (msec): min=2, max=128, avg=73.03, stdev=14.20 lat (msec): min=5, max=131, avg=75.32, stdev=14.32 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 53], 10.00th=[ 58], 20.00th=[ 64], | 30.00th=[ 67], 40.00th=[ 70], 50.00th=[ 73], 60.00th=[ 77], | 70.00th=[ 80], 80.00th=[ 84], 90.00th=[ 90], 95.00th=[ 95], | 99.00th=[ 110], 99.50th=[ 114], 99.90th=[ 127], 99.95th=[ 129], | 99.99th=[ 129] bw ( KiB/s): min= 176, max= 384, per=24.86%, avg=269.25, stdev=53.82, samples=36 iops : min= 44, max= 96, avg=67.28, stdev=13.44, samples=36 write: IOPS=71, BW=284KiB/s (291kB/s)(5256KiB/18497msec); 0 zone resets slat (usec): min=48, max=638, avg=174.85, stdev=80.56 clat (msec): min=25, max=256, avg=153.48, stdev=33.81 lat (msec): min=25, max=256, avg=153.65, stdev=33.81 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 100], 10.00th=[ 114], 20.00th=[ 127], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 155], 60.00th=[ 163], | 70.00th=[ 171], 80.00th=[ 182], 90.00th=[ 197], 95.00th=[ 209], | 99.00th=[ 228], 99.50th=[ 243], 99.90th=[ 257], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 184, max= 329, per=24.96%, avg=281.53, stdev=30.23, samples=36 iops : min= 46, max= 82, avg=70.36, stdev= 7.55, samples=36 lat (msec) : 4=0.12%, 10=0.12%, 20=0.20%, 50=1.76%, 100=47.42% lat (msec) : 250=50.23%, 500=0.16% cpu : usr=0.16%, sys=2.30%, ctx=4940, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=159866: Tue Jun 16 22:12:51 2026 read: IOPS=68, BW=274KiB/s (280kB/s)(5060KiB/18487msec) slat (usec): min=999, max=6953, avg=2310.71, stdev=751.63 clat (msec): min=2, max=174, avg=73.10, stdev=14.28 lat (msec): min=4, max=177, avg=75.41, stdev=14.34 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 54], 10.00th=[ 59], 20.00th=[ 64], | 30.00th=[ 67], 40.00th=[ 70], 50.00th=[ 73], 60.00th=[ 77], | 70.00th=[ 80], 80.00th=[ 83], 90.00th=[ 89], 95.00th=[ 94], | 99.00th=[ 107], 99.50th=[ 124], 99.90th=[ 174], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 184, max= 368, per=25.29%, avg=273.86, stdev=47.12, samples=36 iops : min= 46, max= 92, avg=68.44, stdev=11.77, samples=36 write: IOPS=70, BW=280KiB/s (287kB/s)(5180KiB/18487msec); 0 zone resets slat (usec): min=55, max=852, avg=175.89, stdev=84.98 clat (msec): min=44, max=260, avg=154.03, stdev=33.23 lat (msec): min=44, max=260, avg=154.20, stdev=33.24 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 101], 10.00th=[ 113], 20.00th=[ 127], | 30.00th=[ 136], 40.00th=[ 146], 50.00th=[ 155], 60.00th=[ 161], | 70.00th=[ 171], 80.00th=[ 182], 90.00th=[ 197], 95.00th=[ 209], | 99.00th=[ 243], 99.50th=[ 253], 99.90th=[ 259], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 232, max= 352, per=24.71%, avg=278.78, stdev=28.89, samples=36 iops : min= 58, max= 88, avg=69.67, stdev= 7.22, samples=36 lat (msec) : 4=0.12%, 20=0.16%, 50=1.33%, 100=48.98%, 250=49.14% lat (msec) : 500=0.27% cpu : usr=0.22%, sys=2.26%, ctx=4975, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1084KiB/s (1110kB/s), 269KiB/s-274KiB/s (275kB/s-280kB/s), io=19.6MiB (20.5MB), run=18487-18513msec WRITE: bw=1129KiB/s (1156kB/s), 280KiB/s-284KiB/s (287kB/s-291kB/s), io=20.4MiB (21.4MB), run=18487-18513msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=159873: Tue Jun 16 22:12:54 2026 read: IOPS=9, BW=364MiB/s (381MB/s)(40.0MiB/110msec) slat (nsec): min=9687.9k, max=9687.9k, avg=9687861.00, stdev= 0.00 clat (nsec): min=96152k, max=96152k, avg=96151840.00, stdev= 0.00 lat (nsec): min=105844k, max=105844k, avg=105844031.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[95945], 5.00th=[95945], 10.00th=[95945], 20.00th=[95945], | 30.00th=[95945], 40.00th=[95945], 50.00th=[95945], 60.00th=[95945], | 70.00th=[95945], 80.00th=[95945], 90.00th=[95945], 95.00th=[95945], | 99.00th=[95945], 99.50th=[95945], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=7.34%, ctx=42, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=364MiB/s (381MB/s), 364MiB/s-364MiB/s (381MB/s-381MB/s), io=40.0MiB (41.9MB), run=110-110msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout