== sanity test 398c: run fio to test AIO ================= 11:21:59 (1786116119) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.131645 s, 319 MB/s osc.lustre-OST0000-osc-ffff9f7d486ad800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=151735: Fri Aug 7 11:22:38 2026 write: IOPS=70, BW=282KiB/s (289kB/s)(10.0MiB/36297msec); 0 zone resets slat (usec): min=35, max=2267, avg=224.63, stdev=132.22 clat (msec): min=58, max=445, avg=226.40, stdev=60.84 lat (msec): min=59, max=446, avg=226.63, stdev=60.85 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 136], 10.00th=[ 163], 20.00th=[ 180], | 30.00th=[ 194], 40.00th=[ 209], 50.00th=[ 222], 60.00th=[ 236], | 70.00th=[ 253], 80.00th=[ 271], 90.00th=[ 300], 95.00th=[ 330], | 99.00th=[ 405], 99.50th=[ 414], 99.90th=[ 443], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 160, max= 619, per=24.88%, avg=280.65, stdev=70.36, samples=72 iops : min= 40, max= 154, avg=70.11, stdev=17.56, samples=72 lat (msec) : 100=2.34%, 250=65.12%, 500=32.54% cpu : usr=0.14%, sys=1.26%, ctx=2815, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151736: Fri Aug 7 11:22:38 2026 write: IOPS=70, BW=283KiB/s (289kB/s)(10.0MiB/36234msec); 0 zone resets slat (usec): min=36, max=2641, avg=215.39, stdev=137.46 clat (msec): min=20, max=445, avg=226.11, stdev=62.09 lat (msec): min=20, max=445, avg=226.33, stdev=62.11 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 136], 10.00th=[ 163], 20.00th=[ 182], | 30.00th=[ 194], 40.00th=[ 209], 50.00th=[ 222], 60.00th=[ 236], | 70.00th=[ 253], 80.00th=[ 275], 90.00th=[ 305], 95.00th=[ 334], | 99.00th=[ 401], 99.50th=[ 414], 99.90th=[ 439], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 184, max= 657, per=24.87%, avg=280.51, stdev=70.90, samples=72 iops : min= 46, max= 164, avg=70.08, stdev=17.73, samples=72 lat (msec) : 50=0.66%, 100=2.54%, 250=64.84%, 500=31.95% cpu : usr=0.13%, sys=1.22%, ctx=2758, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151737: Fri Aug 7 11:22:38 2026 write: IOPS=70, BW=282KiB/s (289kB/s)(10.0MiB/36299msec); 0 zone resets slat (usec): min=35, max=1647, avg=223.71, stdev=129.10 clat (msec): min=64, max=446, avg=226.46, stdev=60.05 lat (msec): min=64, max=446, avg=226.68, stdev=60.07 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 136], 10.00th=[ 163], 20.00th=[ 180], | 30.00th=[ 194], 40.00th=[ 209], 50.00th=[ 222], 60.00th=[ 236], | 70.00th=[ 253], 80.00th=[ 271], 90.00th=[ 300], 95.00th=[ 330], | 99.00th=[ 401], 99.50th=[ 418], 99.90th=[ 443], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 160, max= 619, per=24.86%, avg=280.43, stdev=66.41, samples=72 iops : min= 40, max= 154, avg=70.06, stdev=16.57, samples=72 lat (msec) : 100=2.23%, 250=65.66%, 500=32.11% cpu : usr=0.15%, sys=1.23%, ctx=2882, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151738: Fri Aug 7 11:22:38 2026 write: IOPS=70, BW=282KiB/s (289kB/s)(10.0MiB/36273msec); 0 zone resets slat (usec): min=38, max=1864, avg=217.58, stdev=123.96 clat (msec): min=33, max=430, avg=226.30, stdev=61.24 lat (msec): min=33, max=430, avg=226.52, stdev=61.25 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 133], 10.00th=[ 161], 20.00th=[ 180], | 30.00th=[ 194], 40.00th=[ 211], 50.00th=[ 224], 60.00th=[ 239], | 70.00th=[ 255], 80.00th=[ 275], 90.00th=[ 305], 95.00th=[ 326], | 99.00th=[ 405], 99.50th=[ 414], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 160, max= 650, per=24.87%, avg=280.51, stdev=73.61, samples=72 iops : min= 40, max= 162, avg=70.08, stdev=18.39, samples=72 lat (msec) : 50=0.39%, 100=2.58%, 250=64.34%, 500=32.70% cpu : usr=0.15%, sys=1.22%, ctx=2906, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1128KiB/s (1155kB/s), 282KiB/s-283KiB/s (289kB/s-289kB/s), io=40.0MiB (41.9MB), run=36234-36299msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=151753: Fri Aug 7 11:23:05 2026 read: IOPS=53, BW=216KiB/s (221kB/s)(5048KiB/23376msec) slat (usec): min=1227, max=7854, avg=2966.98, stdev=930.35 clat (msec): min=2, max=168, avg=91.97, stdev=21.03 lat (msec): min=5, max=171, avg=94.94, stdev=21.20 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 62], 10.00th=[ 69], 20.00th=[ 77], | 30.00th=[ 83], 40.00th=[ 87], 50.00th=[ 92], 60.00th=[ 96], | 70.00th=[ 102], 80.00th=[ 108], 90.00th=[ 117], 95.00th=[ 127], | 99.00th=[ 146], 99.50th=[ 157], 99.90th=[ 167], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 120, max= 328, per=25.17%, avg=215.93, stdev=45.52, samples=46 iops : min= 30, max= 82, avg=53.96, stdev=11.37, samples=46 write: IOPS=55, BW=222KiB/s (227kB/s)(5192KiB/23376msec); 0 zone resets slat (usec): min=47, max=1091, avg=206.57, stdev=111.03 clat (msec): min=73, max=336, avg=195.06, stdev=43.53 lat (msec): min=73, max=336, avg=195.27, stdev=43.52 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 131], 10.00th=[ 144], 20.00th=[ 159], | 30.00th=[ 171], 40.00th=[ 182], 50.00th=[ 190], 60.00th=[ 203], | 70.00th=[ 215], 80.00th=[ 230], 90.00th=[ 251], 95.00th=[ 279], | 99.00th=[ 317], 99.50th=[ 326], 99.90th=[ 330], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 144, max= 296, per=24.85%, avg=221.93, stdev=37.14, samples=46 iops : min= 36, max= 74, avg=55.46, stdev= 9.29, samples=46 lat (msec) : 4=0.04%, 10=0.20%, 20=0.12%, 50=0.39%, 100=32.89% lat (msec) : 250=61.25%, 500=5.12% cpu : usr=0.21%, sys=2.16%, ctx=5031, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151754: Fri Aug 7 11:23:05 2026 read: IOPS=53, BW=213KiB/s (218kB/s)(4976KiB/23376msec) slat (usec): min=1042, max=7507, avg=2943.89, stdev=974.66 clat (msec): min=5, max=165, avg=92.53, stdev=19.51 lat (msec): min=8, max=168, avg=95.47, stdev=19.71 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 64], 10.00th=[ 69], 20.00th=[ 77], | 30.00th=[ 83], 40.00th=[ 87], 50.00th=[ 92], 60.00th=[ 97], | 70.00th=[ 102], 80.00th=[ 108], 90.00th=[ 118], 95.00th=[ 126], | 99.00th=[ 144], 99.50th=[ 146], 99.90th=[ 150], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 80, max= 312, per=24.66%, avg=211.59, stdev=48.99, samples=46 iops : min= 20, max= 78, avg=52.87, stdev=12.24, samples=46 write: IOPS=56, BW=225KiB/s (231kB/s)(5264KiB/23376msec); 0 zone resets slat (usec): min=52, max=1558, avg=206.70, stdev=116.92 clat (msec): min=65, max=342, avg=193.27, stdev=44.39 lat (msec): min=65, max=342, avg=193.48, stdev=44.39 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 129], 10.00th=[ 142], 20.00th=[ 155], | 30.00th=[ 169], 40.00th=[ 182], 50.00th=[ 190], 60.00th=[ 201], | 70.00th=[ 213], 80.00th=[ 230], 90.00th=[ 251], 95.00th=[ 271], | 99.00th=[ 326], 99.50th=[ 330], 99.90th=[ 338], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 168, max= 288, per=25.17%, avg=224.76, stdev=30.21, samples=46 iops : min= 42, max= 72, avg=56.17, stdev= 7.56, samples=46 lat (msec) : 10=0.08%, 20=0.08%, 50=0.20%, 100=32.62%, 250=61.84% lat (msec) : 500=5.20% cpu : usr=0.27%, sys=2.07%, ctx=4911, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151755: Fri Aug 7 11:23:05 2026 read: IOPS=53, BW=213KiB/s (218kB/s)(4984KiB/23363msec) slat (usec): min=94, max=7464, avg=2946.04, stdev=984.58 clat (msec): min=3, max=203, avg=91.21, stdev=21.12 lat (msec): min=5, max=204, avg=94.16, stdev=21.28 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 62], 10.00th=[ 69], 20.00th=[ 77], | 30.00th=[ 82], 40.00th=[ 86], 50.00th=[ 90], 60.00th=[ 95], | 70.00th=[ 101], 80.00th=[ 109], 90.00th=[ 117], 95.00th=[ 126], | 99.00th=[ 144], 99.50th=[ 148], 99.90th=[ 167], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 120, max= 312, per=24.99%, avg=214.41, stdev=41.35, samples=46 iops : min= 30, max= 78, avg=53.59, stdev=10.35, samples=46 write: IOPS=56, BW=225KiB/s (230kB/s)(5256KiB/23363msec); 0 zone resets slat (usec): min=52, max=1153, avg=199.62, stdev=106.14 clat (msec): min=33, max=342, avg=194.76, stdev=44.58 lat (msec): min=33, max=343, avg=194.96, stdev=44.58 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 133], 10.00th=[ 144], 20.00th=[ 159], | 30.00th=[ 171], 40.00th=[ 182], 50.00th=[ 190], 60.00th=[ 201], | 70.00th=[ 213], 80.00th=[ 230], 90.00th=[ 251], 95.00th=[ 275], | 99.00th=[ 317], 99.50th=[ 330], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 144, max= 280, per=25.11%, avg=224.22, stdev=33.28, samples=46 iops : min= 36, max= 70, avg=56.02, stdev= 8.30, samples=46 lat (msec) : 4=0.04%, 10=0.23%, 20=0.20%, 50=0.59%, 100=33.12% lat (msec) : 250=60.51%, 500=5.31% cpu : usr=0.28%, sys=2.09%, ctx=4890, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=151756: Fri Aug 7 11:23:05 2026 read: IOPS=54, BW=217KiB/s (222kB/s)(5060KiB/23326msec) slat (usec): min=1219, max=8786, avg=2969.42, stdev=974.45 clat (msec): min=3, max=167, avg=92.04, stdev=20.23 lat (msec): min=5, max=171, avg=95.01, stdev=20.37 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 64], 10.00th=[ 69], 20.00th=[ 78], | 30.00th=[ 82], 40.00th=[ 87], 50.00th=[ 91], 60.00th=[ 96], | 70.00th=[ 102], 80.00th=[ 108], 90.00th=[ 118], 95.00th=[ 126], | 99.00th=[ 140], 99.50th=[ 144], 99.90th=[ 167], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 128, max= 296, per=25.36%, avg=217.63, stdev=41.30, samples=46 iops : min= 32, max= 74, avg=54.39, stdev=10.34, samples=46 write: IOPS=55, BW=222KiB/s (227kB/s)(5180KiB/23326msec); 0 zone resets slat (usec): min=53, max=3027, avg=203.63, stdev=131.47 clat (msec): min=72, max=348, avg=194.52, stdev=44.03 lat (msec): min=73, max=348, avg=194.72, stdev=44.03 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 131], 10.00th=[ 142], 20.00th=[ 155], | 30.00th=[ 169], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 215], 80.00th=[ 228], 90.00th=[ 251], 95.00th=[ 275], | 99.00th=[ 321], 99.50th=[ 326], 99.90th=[ 334], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 128, max= 296, per=24.81%, avg=221.59, stdev=39.11, samples=46 iops : min= 32, max= 74, avg=55.37, stdev= 9.77, samples=46 lat (msec) : 4=0.04%, 10=0.12%, 20=0.20%, 50=0.43%, 100=32.97% lat (msec) : 250=61.05%, 500=5.20% cpu : usr=0.21%, sys=2.16%, ctx=4879, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=858KiB/s (879kB/s), 213KiB/s-217KiB/s (218kB/s-222kB/s), io=19.6MiB (20.5MB), run=23326-23376msec WRITE: bw=894KiB/s (915kB/s), 222KiB/s-225KiB/s (227kB/s-231kB/s), io=20.4MiB (21.4MB), run=23326-23376msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=151772: Fri Aug 7 11:23:09 2026 read: IOPS=6, BW=260MiB/s (272MB/s)(40.0MiB/154msec) slat (nsec): min=12936k, max=12936k, avg=12936009.00, stdev= 0.00 clat (nsec): min=132704k, max=132704k, avg=132703716.00, stdev= 0.00 lat (nsec): min=145644k, max=145644k, avg=145643754.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 133], | 30.00th=[ 133], 40.00th=[ 133], 50.00th=[ 133], 60.00th=[ 133], | 70.00th=[ 133], 80.00th=[ 133], 90.00th=[ 133], 95.00th=[ 133], | 99.00th=[ 133], 99.50th=[ 133], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=7.84%, ctx=23, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=260MiB/s (272MB/s), 260MiB/s-260MiB/s (272MB/s-272MB/s), io=40.0MiB (41.9MB), run=154-154msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout