== sanity test 398c: run fio to test AIO ================= 02:46:12 (1788849972) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.375039 s, 112 MB/s osc.lustre-OST0000-osc-ffff8a2e46408000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180387: Tue Sep 8 02:46:42 2026 write: IOPS=118, BW=473KiB/s (484kB/s)(10.0MiB/21655msec); 0 zone resets slat (usec): min=21, max=8630, avg=145.90, stdev=423.62 clat (msec): min=50, max=241, avg=134.69, stdev=33.12 lat (msec): min=50, max=242, avg=134.85, stdev=33.11 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 85], 10.00th=[ 94], 20.00th=[ 109], | 30.00th=[ 116], 40.00th=[ 125], 50.00th=[ 130], 60.00th=[ 140], | 70.00th=[ 150], 80.00th=[ 163], 90.00th=[ 180], 95.00th=[ 192], | 99.00th=[ 234], 99.50th=[ 243], 99.90th=[ 243], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min= 312, max= 675, per=24.87%, avg=470.33, stdev=80.90, samples=42 iops : min= 78, max= 168, avg=117.29, stdev=20.18, samples=42 lat (msec) : 100=12.23%, 250=87.77% cpu : usr=0.19%, sys=1.06%, ctx=2863, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180388: Tue Sep 8 02:46:42 2026 write: IOPS=118, BW=473KiB/s (485kB/s)(10.0MiB/21631msec); 0 zone resets slat (usec): min=21, max=15022, avg=141.27, stdev=556.59 clat (msec): min=28, max=245, avg=134.81, stdev=32.61 lat (msec): min=28, max=245, avg=134.95, stdev=32.61 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 83], 10.00th=[ 94], 20.00th=[ 109], | 30.00th=[ 117], 40.00th=[ 126], 50.00th=[ 133], 60.00th=[ 142], | 70.00th=[ 148], 80.00th=[ 161], 90.00th=[ 178], 95.00th=[ 199], | 99.00th=[ 213], 99.50th=[ 230], 99.90th=[ 232], 99.95th=[ 234], | 99.99th=[ 247] bw ( KiB/s): min= 272, max= 761, per=24.91%, avg=471.02, stdev=97.57, samples=42 iops : min= 68, max= 190, avg=117.50, stdev=24.37, samples=42 lat (msec) : 50=0.08%, 100=14.26%, 250=85.66% cpu : usr=0.18%, sys=0.93%, ctx=2190, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180389: Tue Sep 8 02:46:42 2026 write: IOPS=118, BW=473KiB/s (485kB/s)(10.0MiB/21634msec); 0 zone resets slat (usec): min=24, max=15312, avg=182.08, stdev=672.45 clat (msec): min=11, max=241, avg=134.92, stdev=33.37 lat (msec): min=11, max=241, avg=135.11, stdev=33.37 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 85], 10.00th=[ 96], 20.00th=[ 109], | 30.00th=[ 116], 40.00th=[ 124], 50.00th=[ 131], 60.00th=[ 138], | 70.00th=[ 150], 80.00th=[ 165], 90.00th=[ 182], 95.00th=[ 197], | 99.00th=[ 224], 99.50th=[ 228], 99.90th=[ 239], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min= 368, max= 783, per=24.94%, avg=471.62, stdev=90.07, samples=42 iops : min= 92, max= 195, avg=117.60, stdev=22.47, samples=42 lat (msec) : 20=0.04%, 50=0.08%, 100=12.07%, 250=87.81% cpu : usr=0.24%, sys=1.09%, ctx=2975, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180390: Tue Sep 8 02:46:42 2026 write: IOPS=118, BW=475KiB/s (487kB/s)(10.0MiB/21548msec); 0 zone resets slat (usec): min=21, max=14548, avg=182.43, stdev=681.89 clat (msec): min=14, max=280, avg=134.32, stdev=33.82 lat (msec): min=14, max=280, avg=134.50, stdev=33.80 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 81], 10.00th=[ 94], 20.00th=[ 109], | 30.00th=[ 117], 40.00th=[ 126], 50.00th=[ 132], 60.00th=[ 140], | 70.00th=[ 150], 80.00th=[ 163], 90.00th=[ 178], 95.00th=[ 190], | 99.00th=[ 224], 99.50th=[ 230], 99.90th=[ 243], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 328, max= 835, per=25.03%, avg=473.26, stdev=93.96, samples=42 iops : min= 82, max= 208, avg=118.02, stdev=23.44, samples=42 lat (msec) : 20=0.12%, 50=0.31%, 100=13.91%, 250=85.59%, 500=0.08% cpu : usr=0.21%, sys=1.10%, ctx=2831, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1891KiB/s (1937kB/s), 473KiB/s-475KiB/s (484kB/s-487kB/s), io=40.0MiB (41.9MB), run=21548-21655msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180403: Tue Sep 8 02:47:16 2026 read: IOPS=49, BW=199KiB/s (204kB/s)(5048KiB/25323msec) slat (usec): min=2380, max=48410, avg=10966.26, stdev=6112.49 clat (msec): min=18, max=329, avg=148.31, stdev=40.74 lat (msec): min=26, max=352, avg=159.28, stdev=42.74 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 90], 10.00th=[ 102], 20.00th=[ 114], | 30.00th=[ 126], 40.00th=[ 134], 50.00th=[ 144], 60.00th=[ 155], | 70.00th=[ 165], 80.00th=[ 182], 90.00th=[ 201], 95.00th=[ 220], | 99.00th=[ 262], 99.50th=[ 284], 99.90th=[ 313], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 78, max= 302, per=25.16%, avg=198.98, stdev=52.32, samples=50 iops : min= 19, max= 75, avg=49.36, stdev=13.16, samples=50 write: IOPS=51, BW=205KiB/s (210kB/s)(5192KiB/25323msec); 0 zone resets slat (usec): min=26, max=14985, avg=227.30, stdev=804.65 clat (msec): min=64, max=313, avg=154.19, stdev=40.00 lat (msec): min=64, max=313, avg=154.42, stdev=40.02 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 96], 10.00th=[ 108], 20.00th=[ 118], | 30.00th=[ 129], 40.00th=[ 140], 50.00th=[ 150], 60.00th=[ 163], | 70.00th=[ 178], 80.00th=[ 188], 90.00th=[ 205], 95.00th=[ 222], | 99.00th=[ 271], 99.50th=[ 288], 99.90th=[ 305], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 56, max= 336, per=24.87%, avg=204.94, stdev=52.63, samples=50 iops : min= 14, max= 84, avg=50.86, stdev=13.17, samples=50 lat (msec) : 20=0.04%, 50=0.08%, 100=7.66%, 250=90.55%, 500=1.68% cpu : usr=0.30%, sys=2.04%, ctx=4713, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180404: Tue Sep 8 02:47:16 2026 read: IOPS=49, BW=197KiB/s (202kB/s)(4976KiB/25220msec) slat (usec): min=55, max=53981, avg=11002.81, stdev=6531.82 clat (msec): min=23, max=319, avg=149.05, stdev=42.74 lat (msec): min=23, max=340, avg=160.07, stdev=44.90 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 90], 10.00th=[ 100], 20.00th=[ 110], | 30.00th=[ 124], 40.00th=[ 134], 50.00th=[ 144], 60.00th=[ 157], | 70.00th=[ 169], 80.00th=[ 184], 90.00th=[ 211], 95.00th=[ 226], | 99.00th=[ 268], 99.50th=[ 284], 99.90th=[ 317], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 110, max= 320, per=24.79%, avg=196.08, stdev=51.83, samples=50 iops : min= 27, max= 80, avg=48.62, stdev=13.03, samples=50 write: IOPS=52, BW=209KiB/s (214kB/s)(5264KiB/25220msec); 0 zone resets slat (usec): min=34, max=13012, avg=195.55, stdev=594.68 clat (msec): min=61, max=336, avg=154.57, stdev=41.24 lat (msec): min=61, max=337, avg=154.77, stdev=41.23 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 97], 10.00th=[ 107], 20.00th=[ 118], | 30.00th=[ 128], 40.00th=[ 140], 50.00th=[ 153], 60.00th=[ 163], | 70.00th=[ 176], 80.00th=[ 188], 90.00th=[ 207], 95.00th=[ 222], | 99.00th=[ 288], 99.50th=[ 317], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 120, max= 336, per=25.30%, avg=208.44, stdev=50.24, samples=50 iops : min= 30, max= 84, avg=51.74, stdev=12.66, samples=50 lat (msec) : 50=0.12%, 100=8.63%, 250=89.41%, 500=1.84% cpu : usr=0.29%, sys=1.92%, ctx=4675, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180405: Tue Sep 8 02:47:16 2026 read: IOPS=49, BW=197KiB/s (201kB/s)(4984KiB/25350msec) slat (usec): min=2191, max=80711, avg=11212.23, stdev=6647.72 clat (msec): min=17, max=309, avg=150.28, stdev=41.79 lat (msec): min=25, max=337, avg=161.50, stdev=43.87 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 90], 10.00th=[ 100], 20.00th=[ 114], | 30.00th=[ 126], 40.00th=[ 136], 50.00th=[ 146], 60.00th=[ 159], | 70.00th=[ 171], 80.00th=[ 186], 90.00th=[ 205], 95.00th=[ 224], | 99.00th=[ 268], 99.50th=[ 275], 99.90th=[ 305], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 84, max= 326, per=24.87%, avg=196.70, stdev=47.84, samples=50 iops : min= 21, max= 81, avg=48.78, stdev=12.05, samples=50 write: IOPS=51, BW=207KiB/s (212kB/s)(5256KiB/25350msec); 0 zone resets slat (usec): min=29, max=12430, avg=206.24, stdev=593.86 clat (msec): min=65, max=303, avg=153.20, stdev=40.56 lat (msec): min=65, max=303, avg=153.41, stdev=40.58 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 94], 10.00th=[ 106], 20.00th=[ 117], | 30.00th=[ 130], 40.00th=[ 140], 50.00th=[ 148], 60.00th=[ 161], | 70.00th=[ 171], 80.00th=[ 186], 90.00th=[ 207], 95.00th=[ 224], | 99.00th=[ 279], 99.50th=[ 288], 99.90th=[ 300], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 99, max= 343, per=25.07%, avg=206.58, stdev=58.91, samples=50 iops : min= 24, max= 85, avg=51.26, stdev=14.76, samples=50 lat (msec) : 20=0.04%, 100=8.98%, 250=88.91%, 500=2.07% cpu : usr=0.29%, sys=1.98%, ctx=4652, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180406: Tue Sep 8 02:47:16 2026 read: IOPS=49, BW=200KiB/s (204kB/s)(5060KiB/25352msec) slat (usec): min=2233, max=83963, avg=11082.46, stdev=6463.60 clat (msec): min=26, max=291, avg=147.81, stdev=42.33 lat (msec): min=32, max=309, avg=158.89, stdev=44.40 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 91], 10.00th=[ 100], 20.00th=[ 111], | 30.00th=[ 123], 40.00th=[ 132], 50.00th=[ 144], 60.00th=[ 157], | 70.00th=[ 167], 80.00th=[ 180], 90.00th=[ 209], 95.00th=[ 226], | 99.00th=[ 266], 99.50th=[ 284], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 71, max= 344, per=25.06%, avg=198.22, stdev=51.63, samples=50 iops : min= 17, max= 86, avg=49.16, stdev=13.01, samples=50 write: IOPS=51, BW=204KiB/s (209kB/s)(5180KiB/25352msec); 0 zone resets slat (usec): min=24, max=19498, avg=189.62, stdev=675.32 clat (msec): min=32, max=306, avg=155.58, stdev=39.63 lat (msec): min=32, max=306, avg=155.77, stdev=39.61 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 100], 10.00th=[ 109], 20.00th=[ 121], | 30.00th=[ 131], 40.00th=[ 142], 50.00th=[ 153], 60.00th=[ 165], | 70.00th=[ 176], 80.00th=[ 188], 90.00th=[ 205], 95.00th=[ 228], | 99.00th=[ 271], 99.50th=[ 284], 99.90th=[ 300], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 85, max= 296, per=24.57%, avg=202.46, stdev=50.31, samples=50 iops : min= 21, max= 74, avg=50.24, stdev=12.67, samples=50 lat (msec) : 50=0.39%, 100=7.73%, 250=89.96%, 500=1.91% cpu : usr=0.34%, sys=2.02%, ctx=4301, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=792KiB/s (811kB/s), 197KiB/s-200KiB/s (201kB/s-204kB/s), io=19.6MiB (20.5MB), run=25220-25352msec WRITE: bw=824KiB/s (844kB/s), 204KiB/s-209KiB/s (209kB/s-214kB/s), io=20.4MiB (21.4MB), run=25220-25352msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180415: Tue Sep 8 02:47:25 2026 read: IOPS=2, BW=92.4MiB/s (96.9MB/s)(40.0MiB/433msec) slat (nsec): min=32128k, max=32128k, avg=32128121.00, stdev= 0.00 clat (nsec): min=381828k, max=381828k, avg=381828319.00, stdev= 0.00 lat (nsec): min=413961k, max=413961k, avg=413960698.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 384], 10.00th=[ 384], 20.00th=[ 384], | 30.00th=[ 384], 40.00th=[ 384], 50.00th=[ 384], 60.00th=[ 384], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.01%, ctx=4, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=92.4MiB/s (96.9MB/s), 92.4MiB/s-92.4MiB/s (96.9MB/s-96.9MB/s), io=40.0MiB (41.9MB), run=433-433msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout