== sanity test 398c: run fio to test AIO ================= 02:25:24 (1781245524) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.855933 s, 49.0 MB/s osc.lustre-OST0000-osc-ffff8f451f77b000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180248: Fri Jun 12 02:26:03 2026 write: IOPS=91, BW=366KiB/s (375kB/s)(10.0MiB/27953msec); 0 zone resets slat (usec): min=37, max=20951, avg=273.51, stdev=785.85 clat (msec): min=74, max=388, avg=173.08, stdev=48.39 lat (msec): min=74, max=388, avg=173.35, stdev=48.41 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 110], 10.00th=[ 118], 20.00th=[ 131], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 169], 60.00th=[ 178], | 70.00th=[ 192], 80.00th=[ 209], 90.00th=[ 234], 95.00th=[ 253], | 99.00th=[ 334], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 200, max= 598, per=24.98%, avg=365.49, stdev=89.22, samples=55 iops : min= 50, max= 149, avg=91.02, stdev=22.34, samples=55 lat (msec) : 100=2.30%, 250=92.30%, 500=5.39% cpu : usr=0.21%, sys=1.14%, ctx=2270, majf=0, minf=7 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180249: Fri Jun 12 02:26:03 2026 write: IOPS=92, BW=368KiB/s (377kB/s)(10.0MiB/27823msec); 0 zone resets slat (usec): min=39, max=22669, avg=327.29, stdev=1008.66 clat (msec): min=26, max=444, avg=173.28, stdev=49.89 lat (msec): min=26, max=444, avg=173.61, stdev=49.92 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 108], 10.00th=[ 117], 20.00th=[ 132], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 167], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 211], 90.00th=[ 234], 95.00th=[ 255], | 99.00th=[ 326], 99.50th=[ 372], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 184, max= 534, per=25.12%, avg=367.44, stdev=80.77, samples=55 iops : min= 46, max= 133, avg=91.49, stdev=20.23, samples=55 lat (msec) : 50=0.08%, 100=2.50%, 250=90.55%, 500=6.88% cpu : usr=0.18%, sys=1.12%, ctx=2738, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180250: Fri Jun 12 02:26:03 2026 write: IOPS=91, BW=366KiB/s (375kB/s)(10.0MiB/27973msec); 0 zone resets slat (usec): min=38, max=16806, avg=309.62, stdev=835.45 clat (msec): min=19, max=358, avg=173.38, stdev=47.62 lat (msec): min=19, max=358, avg=173.69, stdev=47.67 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 109], 10.00th=[ 118], 20.00th=[ 133], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 167], 60.00th=[ 180], | 70.00th=[ 194], 80.00th=[ 213], 90.00th=[ 234], 95.00th=[ 259], | 99.00th=[ 309], 99.50th=[ 321], 99.90th=[ 355], 99.95th=[ 355], | 99.99th=[ 359] bw ( KiB/s): min= 236, max= 574, per=25.04%, avg=366.36, stdev=78.98, samples=55 iops : min= 59, max= 143, avg=91.24, stdev=19.74, samples=55 lat (msec) : 20=0.04%, 50=0.04%, 100=2.46%, 250=91.17%, 500=6.29% cpu : usr=0.17%, sys=1.29%, ctx=2892, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180251: Fri Jun 12 02:26:03 2026 write: IOPS=91, BW=366KiB/s (375kB/s)(10.0MiB/27993msec); 0 zone resets slat (usec): min=38, max=17440, avg=265.47, stdev=797.50 clat (msec): min=16, max=357, avg=173.13, stdev=47.40 lat (msec): min=16, max=357, avg=173.40, stdev=47.42 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 111], 10.00th=[ 120], 20.00th=[ 131], | 30.00th=[ 146], 40.00th=[ 155], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 194], 80.00th=[ 211], 90.00th=[ 236], 95.00th=[ 257], | 99.00th=[ 321], 99.50th=[ 351], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 208, max= 539, per=25.01%, avg=365.89, stdev=76.91, samples=55 iops : min= 52, max= 134, avg=91.11, stdev=19.28, samples=55 lat (msec) : 20=0.04%, 50=0.04%, 100=1.99%, 250=91.29%, 500=6.64% cpu : usr=0.15%, sys=1.09%, ctx=2182, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1463KiB/s (1498kB/s), 366KiB/s-368KiB/s (375kB/s-377kB/s), io=40.0MiB (41.9MB), run=27823-27993msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180266: Fri Jun 12 02:26:40 2026 read: IOPS=50, BW=202KiB/s (207kB/s)(5048KiB/25006msec) slat (usec): min=2426, max=58543, avg=11334.71, stdev=6674.15 clat (msec): min=19, max=364, avg=145.25, stdev=48.17 lat (msec): min=31, max=380, avg=156.59, stdev=50.63 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 84], 10.00th=[ 93], 20.00th=[ 106], | 30.00th=[ 116], 40.00th=[ 126], 50.00th=[ 138], 60.00th=[ 148], | 70.00th=[ 163], 80.00th=[ 182], 90.00th=[ 211], 95.00th=[ 241], | 99.00th=[ 292], 99.50th=[ 317], 99.90th=[ 359], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 43, max= 359, per=24.84%, avg=199.18, stdev=61.30, samples=49 iops : min= 10, max= 89, avg=49.39, stdev=15.35, samples=49 write: IOPS=51, BW=208KiB/s (213kB/s)(5192KiB/25006msec); 0 zone resets slat (usec): min=39, max=5395, avg=239.26, stdev=389.96 clat (msec): min=26, max=362, avg=152.04, stdev=45.78 lat (msec): min=26, max=362, avg=152.28, stdev=45.80 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 95], 10.00th=[ 102], 20.00th=[ 113], | 30.00th=[ 124], 40.00th=[ 133], 50.00th=[ 144], 60.00th=[ 155], | 70.00th=[ 167], 80.00th=[ 186], 90.00th=[ 220], 95.00th=[ 241], | 99.00th=[ 292], 99.50th=[ 300], 99.90th=[ 338], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 36, max= 320, per=24.70%, avg=206.24, stdev=60.48, samples=49 iops : min= 9, max= 80, avg=51.16, stdev=15.13, samples=49 lat (msec) : 20=0.04%, 50=0.16%, 100=11.76%, 250=84.69%, 500=3.36% cpu : usr=0.30%, sys=2.20%, ctx=4520, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180267: Fri Jun 12 02:26:40 2026 read: IOPS=49, BW=200KiB/s (205kB/s)(4976KiB/24898msec) slat (usec): min=2209, max=51570, avg=10999.16, stdev=6521.71 clat (msec): min=9, max=365, avg=146.82, stdev=47.37 lat (msec): min=16, max=391, avg=157.83, stdev=49.82 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 87], 10.00th=[ 95], 20.00th=[ 107], | 30.00th=[ 118], 40.00th=[ 129], 50.00th=[ 138], 60.00th=[ 150], | 70.00th=[ 165], 80.00th=[ 182], 90.00th=[ 211], 95.00th=[ 243], | 99.00th=[ 284], 99.50th=[ 330], 99.90th=[ 363], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 58, max= 311, per=24.68%, avg=197.94, stdev=59.66, samples=49 iops : min= 14, max= 77, avg=49.08, stdev=14.94, samples=49 write: IOPS=52, BW=211KiB/s (216kB/s)(5264KiB/24898msec); 0 zone resets slat (usec): min=41, max=14140, avg=252.34, stdev=555.79 clat (msec): min=64, max=337, avg=151.05, stdev=46.32 lat (msec): min=64, max=337, avg=151.30, stdev=46.34 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 94], 10.00th=[ 102], 20.00th=[ 113], | 30.00th=[ 121], 40.00th=[ 130], 50.00th=[ 140], 60.00th=[ 153], | 70.00th=[ 169], 80.00th=[ 190], 90.00th=[ 222], 95.00th=[ 241], | 99.00th=[ 292], 99.50th=[ 305], 99.90th=[ 330], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 95, max= 335, per=25.27%, avg=211.04, stdev=60.32, samples=49 iops : min= 23, max= 83, avg=52.35, stdev=15.06, samples=49 lat (msec) : 10=0.04%, 100=11.29%, 250=85.39%, 500=3.28% cpu : usr=0.24%, sys=2.17%, ctx=4671, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180268: Fri Jun 12 02:26:40 2026 read: IOPS=49, BW=199KiB/s (204kB/s)(4984KiB/25000msec) slat (usec): min=2567, max=63981, avg=10998.81, stdev=6401.77 clat (msec): min=21, max=378, avg=145.83, stdev=46.69 lat (msec): min=31, max=401, avg=156.83, stdev=49.14 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 86], 10.00th=[ 94], 20.00th=[ 108], | 30.00th=[ 117], 40.00th=[ 128], 50.00th=[ 138], 60.00th=[ 150], | 70.00th=[ 165], 80.00th=[ 182], 90.00th=[ 213], 95.00th=[ 232], | 99.00th=[ 284], 99.50th=[ 305], 99.90th=[ 376], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 36, max= 327, per=24.61%, avg=197.39, stdev=61.30, samples=49 iops : min= 9, max= 81, avg=48.92, stdev=15.33, samples=49 write: IOPS=52, BW=210KiB/s (215kB/s)(5256KiB/25000msec); 0 zone resets slat (usec): min=45, max=17966, avg=294.04, stdev=755.32 clat (msec): min=26, max=367, avg=152.10, stdev=47.05 lat (msec): min=26, max=367, avg=152.40, stdev=47.06 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 94], 10.00th=[ 102], 20.00th=[ 112], | 30.00th=[ 122], 40.00th=[ 133], 50.00th=[ 144], 60.00th=[ 157], | 70.00th=[ 171], 80.00th=[ 190], 90.00th=[ 213], 95.00th=[ 236], | 99.00th=[ 305], 99.50th=[ 317], 99.90th=[ 359], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 44, max= 343, per=24.78%, avg=206.94, stdev=59.93, samples=49 iops : min= 11, max= 85, avg=51.35, stdev=14.98, samples=49 lat (msec) : 50=0.16%, 100=11.25%, 250=85.43%, 500=3.16% cpu : usr=0.31%, sys=2.17%, ctx=4545, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180269: Fri Jun 12 02:26:40 2026 read: IOPS=50, BW=204KiB/s (209kB/s)(5060KiB/24815msec) slat (usec): min=93, max=63978, avg=11001.23, stdev=6745.53 clat (msec): min=17, max=369, avg=146.08, stdev=46.67 lat (msec): min=23, max=396, avg=157.09, stdev=49.04 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 88], 10.00th=[ 96], 20.00th=[ 108], | 30.00th=[ 118], 40.00th=[ 129], 50.00th=[ 138], 60.00th=[ 148], | 70.00th=[ 163], 80.00th=[ 180], 90.00th=[ 209], 95.00th=[ 234], | 99.00th=[ 296], 99.50th=[ 317], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 95, max= 383, per=25.36%, avg=203.41, stdev=58.76, samples=49 iops : min= 23, max= 95, avg=50.45, stdev=14.67, samples=49 write: IOPS=52, BW=209KiB/s (214kB/s)(5180KiB/24815msec); 0 zone resets slat (usec): min=41, max=16482, avg=320.92, stdev=915.76 clat (msec): min=35, max=365, avg=152.10, stdev=46.25 lat (msec): min=35, max=365, avg=152.42, stdev=46.27 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 96], 10.00th=[ 103], 20.00th=[ 113], | 30.00th=[ 122], 40.00th=[ 133], 50.00th=[ 142], 60.00th=[ 153], | 70.00th=[ 167], 80.00th=[ 192], 90.00th=[ 218], 95.00th=[ 236], | 99.00th=[ 296], 99.50th=[ 317], 99.90th=[ 363], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 103, max= 375, per=24.82%, avg=207.24, stdev=56.32, samples=49 iops : min= 25, max= 93, avg=51.41, stdev=14.13, samples=49 lat (msec) : 20=0.04%, 50=0.23%, 100=10.12%, 250=86.52%, 500=3.09% cpu : usr=0.28%, sys=2.32%, ctx=4667, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=803KiB/s (822kB/s), 199KiB/s-204KiB/s (204kB/s-209kB/s), io=19.6MiB (20.5MB), run=24815-25006msec WRITE: bw=835KiB/s (856kB/s), 208KiB/s-211KiB/s (213kB/s-216kB/s), io=20.4MiB (21.4MB), run=24815-25006msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180277: Fri Jun 12 02:26:46 2026 read: IOPS=2, BW=117MiB/s (123MB/s)(40.0MiB/341msec) slat (nsec): min=16242k, max=16242k, avg=16242217.00, stdev= 0.00 clat (nsec): min=303886k, max=303886k, avg=303885549.00, stdev= 0.00 lat (nsec): min=320133k, max=320133k, avg=320132606.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 305], 10.00th=[ 305], 20.00th=[ 305], | 30.00th=[ 305], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 305], | 70.00th=[ 305], 80.00th=[ 305], 90.00th=[ 305], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.94%, ctx=10, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=117MiB/s (123MB/s), 117MiB/s-117MiB/s (123MB/s-123MB/s), io=40.0MiB (41.9MB), run=341-341msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout