== sanity test 398c: run fio to test AIO ================= 14:14:12 (1761243252) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.2866 s, 146 MB/s osc.lustre-OST0000-osc-ffff957e0547f800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=184155: Thu Oct 23 14:14:40 2025 write: IOPS=119, BW=479KiB/s (491kB/s)(10.0MiB/21375msec); 0 zone resets slat (usec): min=27, max=5930, avg=134.29, stdev=290.91 clat (msec): min=12, max=318, avg=133.21, stdev=27.93 lat (msec): min=12, max=318, avg=133.34, stdev=27.93 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 94], 10.00th=[ 101], 20.00th=[ 111], | 30.00th=[ 117], 40.00th=[ 126], 50.00th=[ 131], 60.00th=[ 138], | 70.00th=[ 144], 80.00th=[ 155], 90.00th=[ 171], 95.00th=[ 180], | 99.00th=[ 209], 99.50th=[ 213], 99.90th=[ 296], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 384, max= 617, per=25.17%, avg=479.93, stdev=56.93, samples=42 iops : min= 96, max= 154, avg=119.62, stdev=14.24, samples=42 lat (msec) : 20=0.08%, 50=0.35%, 100=9.26%, 250=90.08%, 500=0.23% cpu : usr=0.17%, sys=1.18%, ctx=3034, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184156: Thu Oct 23 14:14:40 2025 write: IOPS=119, BW=478KiB/s (490kB/s)(10.0MiB/21408msec); 0 zone resets slat (usec): min=26, max=5829, avg=156.29, stdev=370.43 clat (msec): min=17, max=286, avg=133.03, stdev=27.70 lat (msec): min=18, max=286, avg=133.19, stdev=27.70 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 95], 10.00th=[ 102], 20.00th=[ 110], | 30.00th=[ 116], 40.00th=[ 125], 50.00th=[ 132], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 155], 90.00th=[ 171], 95.00th=[ 182], | 99.00th=[ 207], 99.50th=[ 211], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 288] bw ( KiB/s): min= 335, max= 598, per=25.12%, avg=479.12, stdev=59.40, samples=42 iops : min= 83, max= 149, avg=119.40, stdev=14.88, samples=42 lat (msec) : 20=0.04%, 50=0.16%, 100=9.41%, 250=90.12%, 500=0.27% cpu : usr=0.19%, sys=1.25%, ctx=3021, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184157: Thu Oct 23 14:14:40 2025 write: IOPS=119, BW=477KiB/s (489kB/s)(10.0MiB/21449msec); 0 zone resets slat (usec): min=26, max=4595, avg=116.43, stdev=279.54 clat (msec): min=32, max=228, avg=132.83, stdev=26.89 lat (msec): min=33, max=228, avg=132.95, stdev=26.89 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 94], 10.00th=[ 101], 20.00th=[ 109], | 30.00th=[ 117], 40.00th=[ 125], 50.00th=[ 131], 60.00th=[ 138], | 70.00th=[ 144], 80.00th=[ 157], 90.00th=[ 171], 95.00th=[ 184], | 99.00th=[ 199], 99.50th=[ 205], 99.90th=[ 224], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 373, max= 638, per=24.94%, avg=475.57, stdev=71.83, samples=42 iops : min= 93, max= 159, avg=118.55, stdev=17.99, samples=42 lat (msec) : 50=0.04%, 100=10.12%, 250=89.84% cpu : usr=0.15%, sys=0.93%, ctx=1822, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184158: Thu Oct 23 14:14:40 2025 write: IOPS=119, BW=477KiB/s (488kB/s)(10.0MiB/21470msec); 0 zone resets slat (usec): min=26, max=8148, avg=153.91, stdev=387.63 clat (msec): min=22, max=226, avg=132.96, stdev=27.15 lat (msec): min=22, max=226, avg=133.12, stdev=27.15 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 92], 10.00th=[ 100], 20.00th=[ 110], | 30.00th=[ 117], 40.00th=[ 126], 50.00th=[ 132], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 155], 90.00th=[ 171], 95.00th=[ 182], | 99.00th=[ 199], 99.50th=[ 209], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 383, max= 614, per=24.88%, avg=474.40, stdev=59.70, samples=42 iops : min= 95, max= 153, avg=118.24, stdev=14.95, samples=42 lat (msec) : 50=0.12%, 100=10.23%, 250=89.65% cpu : usr=0.20%, sys=1.31%, ctx=2964, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1908KiB/s (1954kB/s), 477KiB/s-479KiB/s (488kB/s-491kB/s), io=40.0MiB (41.9MB), run=21375-21470msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=184171: Thu Oct 23 14:15:09 2025 read: IOPS=57, BW=232KiB/s (237kB/s)(5048KiB/21796msec) slat (msec): min=2, max=140, avg= 9.20, stdev= 5.35 clat (msec): min=9, max=265, avg=129.10, stdev=27.89 lat (msec): min=13, max=274, avg=138.31, stdev=28.59 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 89], 10.00th=[ 97], 20.00th=[ 107], | 30.00th=[ 114], 40.00th=[ 121], 50.00th=[ 127], 60.00th=[ 133], | 70.00th=[ 140], 80.00th=[ 150], 90.00th=[ 167], 95.00th=[ 180], | 99.00th=[ 205], 99.50th=[ 218], 99.90th=[ 253], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 138, max= 333, per=25.02%, avg=230.16, stdev=42.54, samples=43 iops : min= 34, max= 83, avg=57.12, stdev=10.61, samples=43 write: IOPS=59, BW=238KiB/s (244kB/s)(5192KiB/21796msec); 0 zone resets slat (usec): min=30, max=3969, avg=150.92, stdev=302.79 clat (msec): min=55, max=274, avg=132.49, stdev=25.56 lat (msec): min=56, max=274, avg=132.64, stdev=25.56 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 94], 10.00th=[ 104], 20.00th=[ 113], | 30.00th=[ 120], 40.00th=[ 125], 50.00th=[ 129], 60.00th=[ 136], | 70.00th=[ 144], 80.00th=[ 153], 90.00th=[ 165], 95.00th=[ 178], | 99.00th=[ 203], 99.50th=[ 213], 99.90th=[ 266], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 166, max= 343, per=24.74%, avg=237.05, stdev=42.49, samples=43 iops : min= 41, max= 85, avg=58.86, stdev=10.69, samples=43 lat (msec) : 10=0.04%, 100=9.80%, 250=90.00%, 500=0.16% cpu : usr=0.25%, sys=1.88%, ctx=4005, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184172: Thu Oct 23 14:15:09 2025 read: IOPS=57, BW=229KiB/s (234kB/s)(4976KiB/21766msec) slat (usec): min=2616, max=32548, avg=9014.04, stdev=3775.97 clat (msec): min=20, max=221, avg=128.95, stdev=26.85 lat (msec): min=24, max=229, avg=137.97, stdev=27.38 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 89], 10.00th=[ 97], 20.00th=[ 107], | 30.00th=[ 113], 40.00th=[ 121], 50.00th=[ 127], 60.00th=[ 136], | 70.00th=[ 142], 80.00th=[ 150], 90.00th=[ 165], 95.00th=[ 178], | 99.00th=[ 201], 99.50th=[ 207], 99.90th=[ 215], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 153, max= 335, per=24.75%, avg=227.74, stdev=37.57, samples=43 iops : min= 38, max= 83, avg=56.51, stdev= 9.41, samples=43 write: IOPS=60, BW=242KiB/s (248kB/s)(5264KiB/21766msec); 0 zone resets slat (usec): min=30, max=4849, avg=145.07, stdev=267.41 clat (msec): min=27, max=217, avg=133.00, stdev=25.50 lat (msec): min=27, max=217, avg=133.15, stdev=25.50 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 95], 10.00th=[ 103], 20.00th=[ 111], | 30.00th=[ 120], 40.00th=[ 126], 50.00th=[ 132], 60.00th=[ 140], | 70.00th=[ 144], 80.00th=[ 155], 90.00th=[ 169], 95.00th=[ 178], | 99.00th=[ 201], 99.50th=[ 209], 99.90th=[ 213], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 168, max= 327, per=25.23%, avg=241.74, stdev=40.65, samples=43 iops : min= 42, max= 81, avg=60.05, stdev=10.22, samples=43 lat (msec) : 50=0.12%, 100=10.70%, 250=89.18% cpu : usr=0.21%, sys=1.93%, ctx=4354, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184173: Thu Oct 23 14:15:09 2025 read: IOPS=57, BW=229KiB/s (234kB/s)(4984KiB/21782msec) slat (usec): min=2360, max=32242, avg=9064.91, stdev=3726.82 clat (msec): min=33, max=211, avg=128.75, stdev=26.48 lat (msec): min=37, max=221, avg=137.83, stdev=27.12 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 89], 10.00th=[ 96], 20.00th=[ 107], | 30.00th=[ 114], 40.00th=[ 122], 50.00th=[ 128], 60.00th=[ 134], | 70.00th=[ 142], 80.00th=[ 148], 90.00th=[ 163], 95.00th=[ 178], | 99.00th=[ 201], 99.50th=[ 205], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 135, max= 327, per=24.88%, avg=228.86, stdev=42.25, samples=43 iops : min= 33, max= 81, avg=56.81, stdev=10.57, samples=43 write: IOPS=60, BW=241KiB/s (247kB/s)(5256KiB/21782msec); 0 zone resets slat (usec): min=29, max=9686, avg=170.95, stdev=436.13 clat (msec): min=60, max=209, avg=132.91, stdev=24.74 lat (msec): min=60, max=209, avg=133.08, stdev=24.76 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 94], 10.00th=[ 104], 20.00th=[ 112], | 30.00th=[ 120], 40.00th=[ 126], 50.00th=[ 132], 60.00th=[ 140], | 70.00th=[ 146], 80.00th=[ 155], 90.00th=[ 167], 95.00th=[ 178], | 99.00th=[ 190], 99.50th=[ 197], 99.90th=[ 201], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 142, max= 351, per=25.07%, avg=240.19, stdev=48.55, samples=43 iops : min= 35, max= 87, avg=59.63, stdev=12.20, samples=43 lat (msec) : 50=0.16%, 100=10.70%, 250=89.14% cpu : usr=0.34%, sys=1.91%, ctx=4539, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184174: Thu Oct 23 14:15:09 2025 read: IOPS=58, BW=233KiB/s (239kB/s)(5060KiB/21694msec) slat (usec): min=50, max=32102, avg=9013.48, stdev=3849.18 clat (msec): min=9, max=219, avg=128.74, stdev=27.74 lat (msec): min=9, max=233, avg=137.76, stdev=28.55 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 90], 10.00th=[ 97], 20.00th=[ 107], | 30.00th=[ 114], 40.00th=[ 122], 50.00th=[ 128], 60.00th=[ 134], | 70.00th=[ 140], 80.00th=[ 150], 90.00th=[ 165], 95.00th=[ 178], | 99.00th=[ 199], 99.50th=[ 209], 99.90th=[ 218], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 135, max= 320, per=25.19%, avg=231.72, stdev=38.44, samples=43 iops : min= 33, max= 80, avg=57.51, stdev= 9.66, samples=43 write: IOPS=59, BW=239KiB/s (245kB/s)(5180KiB/21694msec); 0 zone resets slat (usec): min=29, max=4592, avg=146.50, stdev=274.22 clat (msec): min=48, max=214, avg=132.97, stdev=25.23 lat (msec): min=48, max=214, avg=133.12, stdev=25.22 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 94], 10.00th=[ 103], 20.00th=[ 113], | 30.00th=[ 121], 40.00th=[ 127], 50.00th=[ 132], 60.00th=[ 140], | 70.00th=[ 146], 80.00th=[ 153], 90.00th=[ 165], 95.00th=[ 178], | 99.00th=[ 199], 99.50th=[ 201], 99.90th=[ 209], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 157, max= 367, per=24.77%, avg=237.26, stdev=48.15, samples=43 iops : min= 39, max= 91, avg=58.91, stdev=12.00, samples=43 lat (msec) : 10=0.08%, 20=0.16%, 50=0.20%, 100=10.27%, 250=89.30% cpu : usr=0.31%, sys=1.79%, ctx=4417, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=921KiB/s (943kB/s), 229KiB/s-233KiB/s (234kB/s-239kB/s), io=19.6MiB (20.5MB), run=21694-21796msec WRITE: bw=959KiB/s (982kB/s), 238KiB/s-242KiB/s (244kB/s-248kB/s), io=20.4MiB (21.4MB), run=21694-21796msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=184181: Thu Oct 23 14:15:15 2025 read: IOPS=3, BW=126MiB/s (132MB/s)(40.0MiB/317msec) slat (nsec): min=13343k, max=13343k, avg=13342887.00, stdev= 0.00 clat (nsec): min=286323k, max=286323k, avg=286322520.00, stdev= 0.00 lat (nsec): min=299671k, max=299671k, avg=299670903.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 288], 10.00th=[ 288], 20.00th=[ 288], | 30.00th=[ 288], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 288], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.80%, ctx=4, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=40.0MiB (41.9MB), run=317-317msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout