== sanity test 398c: run fio to test AIO ================= 22:19:30 (1775873970) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.575133 s, 72.9 MB/s osc.lustre-OST0000-osc-ffff9d67d8fd3800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=184357: Fri Apr 10 22:20:03 2026 write: IOPS=106, BW=426KiB/s (436kB/s)(10.0MiB/24060msec); 0 zone resets slat (usec): min=37, max=12381, avg=245.65, stdev=609.90 clat (msec): min=17, max=310, avg=149.18, stdev=34.15 lat (msec): min=17, max=310, avg=149.43, stdev=34.17 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 102], 10.00th=[ 111], 20.00th=[ 123], | 30.00th=[ 130], 40.00th=[ 136], 50.00th=[ 144], 60.00th=[ 155], | 70.00th=[ 165], 80.00th=[ 174], 90.00th=[ 192], 95.00th=[ 209], | 99.00th=[ 255], 99.50th=[ 266], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 262, max= 598, per=24.96%, avg=424.85, stdev=77.05, samples=47 iops : min= 65, max= 149, avg=105.83, stdev=19.31, samples=47 lat (msec) : 20=0.04%, 50=0.08%, 100=4.18%, 250=94.45%, 500=1.25% cpu : usr=0.21%, sys=1.32%, ctx=2317, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184358: Fri Apr 10 22:20:03 2026 write: IOPS=106, BW=426KiB/s (436kB/s)(10.0MiB/24055msec); 0 zone resets slat (usec): min=37, max=12520, avg=253.78, stdev=662.00 clat (msec): min=13, max=284, avg=149.21, stdev=34.65 lat (msec): min=14, max=285, avg=149.46, stdev=34.63 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 105], 10.00th=[ 113], 20.00th=[ 122], | 30.00th=[ 130], 40.00th=[ 138], 50.00th=[ 146], 60.00th=[ 155], | 70.00th=[ 163], 80.00th=[ 174], 90.00th=[ 194], 95.00th=[ 207], | 99.00th=[ 259], 99.50th=[ 279], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 250, max= 600, per=24.94%, avg=424.55, stdev=78.88, samples=47 iops : min= 62, max= 150, avg=105.74, stdev=19.77, samples=47 lat (msec) : 20=0.04%, 50=0.16%, 100=3.24%, 250=94.26%, 500=2.30% cpu : usr=0.18%, sys=1.38%, ctx=2717, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184359: Fri Apr 10 22:20:03 2026 write: IOPS=106, BW=426KiB/s (437kB/s)(10.0MiB/24013msec); 0 zone resets slat (usec): min=35, max=16248, avg=236.78, stdev=664.08 clat (msec): min=20, max=279, avg=149.04, stdev=35.15 lat (msec): min=20, max=279, avg=149.27, stdev=35.11 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 102], 10.00th=[ 110], 20.00th=[ 120], | 30.00th=[ 129], 40.00th=[ 138], 50.00th=[ 144], 60.00th=[ 153], | 70.00th=[ 163], 80.00th=[ 176], 90.00th=[ 197], 95.00th=[ 218], | 99.00th=[ 259], 99.50th=[ 266], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 279] bw ( KiB/s): min= 256, max= 628, per=25.03%, avg=426.04, stdev=76.72, samples=47 iops : min= 64, max= 157, avg=106.15, stdev=19.19, samples=47 lat (msec) : 50=0.12%, 100=3.63%, 250=94.88%, 500=1.37% cpu : usr=0.16%, sys=1.31%, ctx=2445, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184360: Fri Apr 10 22:20:03 2026 write: IOPS=106, BW=428KiB/s (438kB/s)(10.0MiB/23952msec); 0 zone resets slat (usec): min=37, max=13653, avg=276.48, stdev=700.12 clat (msec): min=11, max=309, avg=149.26, stdev=35.92 lat (msec): min=11, max=309, avg=149.54, stdev=35.89 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 99], 10.00th=[ 112], 20.00th=[ 123], | 30.00th=[ 129], 40.00th=[ 138], 50.00th=[ 146], 60.00th=[ 153], | 70.00th=[ 163], 80.00th=[ 178], 90.00th=[ 197], 95.00th=[ 213], | 99.00th=[ 268], 99.50th=[ 271], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 309] bw ( KiB/s): min= 272, max= 622, per=25.14%, avg=427.85, stdev=68.88, samples=47 iops : min= 68, max= 155, avg=106.55, stdev=17.21, samples=47 lat (msec) : 20=0.08%, 50=0.20%, 100=5.59%, 250=92.66%, 500=1.48% cpu : usr=0.26%, sys=1.63%, ctx=2792, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1702KiB/s (1743kB/s), 426KiB/s-428KiB/s (436kB/s-438kB/s), io=40.0MiB (41.9MB), run=23952-24060msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=184373: Fri Apr 10 22:20:32 2026 read: IOPS=55, BW=221KiB/s (227kB/s)(5048KiB/22805msec) slat (usec): min=2384, max=44679, avg=10075.25, stdev=5512.56 clat (msec): min=27, max=306, avg=134.22, stdev=38.24 lat (msec): min=37, max=319, avg=144.30, stdev=40.58 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 88], 10.00th=[ 94], 20.00th=[ 103], | 30.00th=[ 110], 40.00th=[ 118], 50.00th=[ 126], 60.00th=[ 136], | 70.00th=[ 148], 80.00th=[ 165], 90.00th=[ 192], 95.00th=[ 209], | 99.00th=[ 247], 99.50th=[ 259], 99.90th=[ 300], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 117, max= 356, per=25.17%, avg=221.22, stdev=58.91, samples=45 iops : min= 29, max= 89, avg=54.89, stdev=14.74, samples=45 write: IOPS=56, BW=228KiB/s (233kB/s)(5192KiB/22805msec); 0 zone resets slat (usec): min=40, max=9684, avg=276.38, stdev=634.85 clat (msec): min=16, max=272, avg=137.97, stdev=36.34 lat (msec): min=18, max=272, avg=138.25, stdev=36.35 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 92], 10.00th=[ 97], 20.00th=[ 108], | 30.00th=[ 115], 40.00th=[ 122], 50.00th=[ 130], 60.00th=[ 140], | 70.00th=[ 155], 80.00th=[ 169], 90.00th=[ 190], 95.00th=[ 205], | 99.00th=[ 245], 99.50th=[ 253], 99.90th=[ 264], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 133, max= 343, per=24.92%, avg=228.02, stdev=51.25, samples=45 iops : min= 33, max= 85, avg=56.58, stdev=12.87, samples=45 lat (msec) : 20=0.04%, 50=0.04%, 100=14.53%, 250=84.61%, 500=0.78% cpu : usr=0.23%, sys=2.42%, ctx=4934, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184374: Fri Apr 10 22:20:32 2026 read: IOPS=54, BW=218KiB/s (223kB/s)(4976KiB/22826msec) slat (usec): min=2429, max=44188, avg=9932.83, stdev=5675.93 clat (msec): min=16, max=297, avg=133.39, stdev=37.43 lat (msec): min=25, max=309, avg=143.33, stdev=39.62 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 84], 10.00th=[ 92], 20.00th=[ 102], | 30.00th=[ 110], 40.00th=[ 117], 50.00th=[ 126], 60.00th=[ 136], | 70.00th=[ 150], 80.00th=[ 165], 90.00th=[ 184], 95.00th=[ 205], | 99.00th=[ 236], 99.50th=[ 253], 99.90th=[ 279], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 93, max= 351, per=24.66%, avg=216.80, stdev=62.70, samples=45 iops : min= 23, max= 87, avg=53.78, stdev=15.74, samples=45 write: IOPS=57, BW=231KiB/s (236kB/s)(5264KiB/22826msec); 0 zone resets slat (usec): min=39, max=6448, avg=233.95, stdev=458.90 clat (msec): min=42, max=286, avg=139.29, stdev=37.89 lat (msec): min=42, max=286, avg=139.53, stdev=37.87 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 92], 10.00th=[ 99], 20.00th=[ 106], | 30.00th=[ 114], 40.00th=[ 124], 50.00th=[ 132], 60.00th=[ 142], | 70.00th=[ 155], 80.00th=[ 169], 90.00th=[ 192], 95.00th=[ 213], | 99.00th=[ 249], 99.50th=[ 262], 99.90th=[ 275], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 120, max= 392, per=25.17%, avg=230.29, stdev=61.25, samples=45 iops : min= 30, max= 98, avg=57.13, stdev=15.35, samples=45 lat (msec) : 20=0.04%, 50=0.08%, 100=15.47%, 250=83.75%, 500=0.66% cpu : usr=0.18%, sys=2.29%, ctx=4725, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184375: Fri Apr 10 22:20:32 2026 read: IOPS=54, BW=219KiB/s (224kB/s)(4984KiB/22810msec) slat (usec): min=91, max=42120, avg=9924.77, stdev=5564.92 clat (msec): min=14, max=278, avg=134.30, stdev=39.10 lat (msec): min=18, max=309, avg=144.24, stdev=41.36 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 84], 10.00th=[ 91], 20.00th=[ 103], | 30.00th=[ 111], 40.00th=[ 120], 50.00th=[ 129], 60.00th=[ 138], | 70.00th=[ 150], 80.00th=[ 165], 90.00th=[ 188], 95.00th=[ 207], | 99.00th=[ 255], 99.50th=[ 259], 99.90th=[ 275], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 127, max= 328, per=24.92%, avg=219.07, stdev=54.40, samples=45 iops : min= 31, max= 82, avg=54.33, stdev=13.67, samples=45 write: IOPS=57, BW=230KiB/s (236kB/s)(5256KiB/22810msec); 0 zone resets slat (usec): min=38, max=18310, avg=271.35, stdev=769.88 clat (msec): min=74, max=289, avg=140.27, stdev=39.38 lat (msec): min=74, max=289, avg=140.55, stdev=39.41 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 92], 10.00th=[ 99], 20.00th=[ 107], | 30.00th=[ 114], 40.00th=[ 123], 50.00th=[ 132], 60.00th=[ 142], | 70.00th=[ 157], 80.00th=[ 174], 90.00th=[ 197], 95.00th=[ 218], | 99.00th=[ 262], 99.50th=[ 275], 99.90th=[ 288], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 112, max= 349, per=25.12%, avg=229.84, stdev=53.77, samples=45 iops : min= 28, max= 87, avg=57.04, stdev=13.44, samples=45 lat (msec) : 20=0.04%, 50=0.23%, 100=14.57%, 250=83.67%, 500=1.48% cpu : usr=0.32%, sys=2.49%, ctx=4809, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184376: Fri Apr 10 22:20:32 2026 read: IOPS=55, BW=222KiB/s (227kB/s)(5060KiB/22790msec) slat (usec): min=2598, max=41202, avg=9766.27, stdev=5362.75 clat (msec): min=18, max=284, avg=133.50, stdev=37.68 lat (msec): min=31, max=299, avg=143.27, stdev=39.96 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 85], 10.00th=[ 91], 20.00th=[ 101], | 30.00th=[ 109], 40.00th=[ 120], 50.00th=[ 128], 60.00th=[ 138], | 70.00th=[ 150], 80.00th=[ 165], 90.00th=[ 184], 95.00th=[ 203], | 99.00th=[ 247], 99.50th=[ 255], 99.90th=[ 275], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 108, max= 357, per=25.26%, avg=222.07, stdev=59.48, samples=45 iops : min= 27, max= 89, avg=55.11, stdev=14.88, samples=45 write: IOPS=56, BW=227KiB/s (233kB/s)(5180KiB/22790msec); 0 zone resets slat (usec): min=38, max=10426, avg=284.42, stdev=682.49 clat (msec): min=20, max=291, avg=138.68, stdev=38.03 lat (msec): min=20, max=291, avg=138.96, stdev=38.07 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 91], 10.00th=[ 99], 20.00th=[ 107], | 30.00th=[ 114], 40.00th=[ 123], 50.00th=[ 132], 60.00th=[ 142], | 70.00th=[ 153], 80.00th=[ 169], 90.00th=[ 192], 95.00th=[ 211], | 99.00th=[ 251], 99.50th=[ 262], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 116, max= 328, per=24.72%, avg=226.20, stdev=56.79, samples=45 iops : min= 29, max= 82, avg=56.13, stdev=14.26, samples=45 lat (msec) : 20=0.04%, 50=0.04%, 100=15.39%, 250=83.63%, 500=0.90% cpu : usr=0.29%, sys=2.40%, ctx=4733, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=879KiB/s (900kB/s), 218KiB/s-222KiB/s (223kB/s-227kB/s), io=19.6MiB (20.5MB), run=22790-22826msec WRITE: bw=915KiB/s (937kB/s), 227KiB/s-231KiB/s (233kB/s-236kB/s), io=20.4MiB (21.4MB), run=22790-22826msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=184384: Fri Apr 10 22:20:40 2026 read: IOPS=2, BW=105MiB/s (110MB/s)(40.0MiB/381msec) slat (nsec): min=17935k, max=17935k, avg=17934508.00, stdev= 0.00 clat (nsec): min=346271k, max=346271k, avg=346271414.00, stdev= 0.00 lat (nsec): min=364244k, max=364244k, avg=364243584.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 347], 10.00th=[ 347], 20.00th=[ 347], | 30.00th=[ 347], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 347], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.42%, ctx=5, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=105MiB/s (110MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=40.0MiB (41.9MB), run=381-381msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout