== sanity test 398c: run fio to test AIO ================= 10:35:57 (1753367757) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.821289 s, 51.1 MB/s osc.lustre-OST0000-osc-ffff9d0918574000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=191094: Thu Jul 24 10:36:35 2025 write: IOPS=103, BW=413KiB/s (423kB/s)(10.0MiB/24786msec); 0 zone resets slat (usec): min=26, max=9587, avg=147.92, stdev=422.58 clat (msec): min=18, max=343, avg=154.65, stdev=38.88 lat (msec): min=19, max=343, avg=154.80, stdev=38.87 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 106], 10.00th=[ 113], 20.00th=[ 126], | 30.00th=[ 136], 40.00th=[ 142], 50.00th=[ 153], 60.00th=[ 161], | 70.00th=[ 171], 80.00th=[ 182], 90.00th=[ 194], 95.00th=[ 211], | 99.00th=[ 326], 99.50th=[ 334], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 253, max= 601, per=25.27%, avg=412.20, stdev=89.11, samples=49 iops : min= 63, max= 150, avg=102.69, stdev=22.32, samples=49 lat (msec) : 20=0.04%, 50=0.66%, 100=2.19%, 250=95.51%, 500=1.60% cpu : usr=0.16%, sys=1.09%, ctx=1110, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=191095: Thu Jul 24 10:36:35 2025 write: IOPS=102, BW=409KiB/s (418kB/s)(10.0MiB/25064msec); 0 zone resets slat (usec): min=27, max=13187, avg=160.05, stdev=481.29 clat (msec): min=25, max=330, avg=155.64, stdev=36.77 lat (msec): min=25, max=331, avg=155.80, stdev=36.76 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 106], 10.00th=[ 115], 20.00th=[ 128], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 163], | 70.00th=[ 171], 80.00th=[ 184], 90.00th=[ 194], 95.00th=[ 209], | 99.00th=[ 309], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 253, max= 552, per=24.95%, avg=407.00, stdev=75.94, samples=49 iops : min= 63, max= 138, avg=101.37, stdev=18.99, samples=49 lat (msec) : 50=0.66%, 100=2.03%, 250=95.78%, 500=1.52% cpu : usr=0.20%, sys=1.14%, ctx=1208, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=191096: Thu Jul 24 10:36:35 2025 write: IOPS=102, BW=408KiB/s (418kB/s)(10.0MiB/25086msec); 0 zone resets slat (usec): min=22, max=7194, avg=181.78, stdev=469.75 clat (msec): min=13, max=341, avg=155.62, stdev=35.55 lat (msec): min=14, max=341, avg=155.81, stdev=35.56 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 106], 10.00th=[ 116], 20.00th=[ 128], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 155], 60.00th=[ 161], | 70.00th=[ 171], 80.00th=[ 184], 90.00th=[ 199], 95.00th=[ 220], | 99.00th=[ 275], 99.50th=[ 317], 99.90th=[ 326], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 216, max= 566, per=24.90%, avg=406.10, stdev=77.66, samples=49 iops : min= 54, max= 141, avg=101.10, stdev=19.43, samples=49 lat (msec) : 20=0.04%, 50=0.20%, 100=3.01%, 250=95.20%, 500=1.56% cpu : usr=0.18%, sys=1.25%, ctx=1478, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=191097: Thu Jul 24 10:36:35 2025 write: IOPS=101, BW=408KiB/s (418kB/s)(10.0MiB/25106msec); 0 zone resets slat (usec): min=24, max=10525, avg=156.61, stdev=463.11 clat (msec): min=32, max=340, avg=155.85, stdev=36.30 lat (msec): min=33, max=340, avg=156.01, stdev=36.29 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 105], 10.00th=[ 116], 20.00th=[ 129], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 159], | 70.00th=[ 169], 80.00th=[ 180], 90.00th=[ 197], 95.00th=[ 218], | 99.00th=[ 313], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 246, max= 622, per=24.88%, avg=405.84, stdev=85.15, samples=49 iops : min= 61, max= 155, avg=101.04, stdev=21.28, samples=49 lat (msec) : 50=0.04%, 100=2.58%, 250=95.90%, 500=1.48% cpu : usr=0.15%, sys=1.06%, ctx=1015, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1631KiB/s (1671kB/s), 408KiB/s-413KiB/s (418kB/s-423kB/s), io=40.0MiB (41.9MB), run=24786-25106msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=191110: Thu Jul 24 10:37:03 2025 read: IOPS=57, BW=228KiB/s (234kB/s)(5048KiB/22110msec) slat (usec): min=2537, max=28491, avg=9599.66, stdev=3862.52 clat (msec): min=41, max=297, avg=129.91, stdev=27.50 lat (msec): min=52, max=308, avg=139.52, stdev=28.26 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 91], 10.00th=[ 97], 20.00th=[ 108], | 30.00th=[ 115], 40.00th=[ 122], 50.00th=[ 129], 60.00th=[ 134], | 70.00th=[ 142], 80.00th=[ 153], 90.00th=[ 165], 95.00th=[ 178], | 99.00th=[ 197], 99.50th=[ 230], 99.90th=[ 292], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 151, max= 312, per=25.17%, avg=228.07, stdev=38.37, samples=43 iops : min= 37, max= 78, avg=56.51, stdev= 9.62, samples=43 write: IOPS=58, BW=235KiB/s (240kB/s)(5192KiB/22110msec); 0 zone resets slat (usec): min=29, max=4947, avg=205.04, stdev=401.51 clat (msec): min=30, max=268, avg=135.12, stdev=24.80 lat (msec): min=30, max=268, avg=135.33, stdev=24.79 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 101], 10.00th=[ 106], 20.00th=[ 115], | 30.00th=[ 122], 40.00th=[ 128], 50.00th=[ 134], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 155], 90.00th=[ 169], 95.00th=[ 180], | 99.00th=[ 197], 99.50th=[ 203], 99.90th=[ 253], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 160, max= 303, per=24.94%, avg=235.47, stdev=33.19, samples=43 iops : min= 40, max= 75, avg=58.37, stdev= 8.31, samples=43 lat (msec) : 50=0.08%, 100=8.59%, 250=91.13%, 500=0.20% cpu : usr=0.44%, sys=2.60%, ctx=2699, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=191111: Thu Jul 24 10:37:03 2025 read: IOPS=56, BW=225KiB/s (230kB/s)(4976KiB/22126msec) slat (usec): min=2943, max=33597, avg=9368.50, stdev=3876.85 clat (msec): min=43, max=223, avg=130.21, stdev=26.36 lat (msec): min=49, max=233, avg=139.59, stdev=26.94 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 92], 10.00th=[ 100], 20.00th=[ 109], | 30.00th=[ 114], 40.00th=[ 122], 50.00th=[ 129], 60.00th=[ 136], | 70.00th=[ 144], 80.00th=[ 153], 90.00th=[ 165], 95.00th=[ 176], | 99.00th=[ 194], 99.50th=[ 209], 99.90th=[ 222], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 143, max= 302, per=24.77%, avg=224.42, stdev=43.12, samples=43 iops : min= 35, max= 75, avg=55.60, stdev=10.88, samples=43 write: IOPS=59, BW=238KiB/s (244kB/s)(5264KiB/22126msec); 0 zone resets slat (usec): min=28, max=6079, avg=210.95, stdev=455.88 clat (msec): min=42, max=219, avg=135.23, stdev=25.99 lat (msec): min=43, max=219, avg=135.44, stdev=25.99 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 99], 10.00th=[ 105], 20.00th=[ 113], | 30.00th=[ 121], 40.00th=[ 127], 50.00th=[ 133], 60.00th=[ 142], | 70.00th=[ 148], 80.00th=[ 157], 90.00th=[ 167], 95.00th=[ 182], | 99.00th=[ 209], 99.50th=[ 211], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 167, max= 327, per=25.12%, avg=237.12, stdev=40.91, samples=43 iops : min= 41, max= 81, avg=58.79, stdev=10.21, samples=43 lat (msec) : 50=0.12%, 100=8.48%, 250=91.41% cpu : usr=0.35%, sys=2.52%, ctx=2695, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=191112: Thu Jul 24 10:37:03 2025 read: IOPS=56, BW=226KiB/s (232kB/s)(4984KiB/22044msec) slat (usec): min=41, max=32612, avg=9553.43, stdev=3846.82 clat (msec): min=24, max=322, avg=130.86, stdev=27.89 lat (msec): min=33, max=334, avg=140.42, stdev=28.59 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 92], 10.00th=[ 100], 20.00th=[ 108], | 30.00th=[ 115], 40.00th=[ 123], 50.00th=[ 129], 60.00th=[ 136], | 70.00th=[ 142], 80.00th=[ 153], 90.00th=[ 165], 95.00th=[ 176], | 99.00th=[ 205], 99.50th=[ 247], 99.90th=[ 317], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 151, max= 295, per=25.01%, avg=226.56, stdev=37.31, samples=43 iops : min= 37, max= 73, avg=56.14, stdev= 9.43, samples=43 write: IOPS=59, BW=238KiB/s (244kB/s)(5256KiB/22044msec); 0 zone resets slat (usec): min=29, max=6134, avg=218.45, stdev=447.84 clat (msec): min=25, max=313, avg=134.53, stdev=27.16 lat (msec): min=25, max=313, avg=134.75, stdev=27.15 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 97], 10.00th=[ 104], 20.00th=[ 113], | 30.00th=[ 120], 40.00th=[ 126], 50.00th=[ 131], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 155], 90.00th=[ 171], 95.00th=[ 182], | 99.00th=[ 207], 99.50th=[ 222], 99.90th=[ 296], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 136, max= 350, per=25.22%, avg=238.05, stdev=46.90, samples=43 iops : min= 34, max= 87, avg=59.02, stdev=11.68, samples=43 lat (msec) : 50=0.20%, 100=8.67%, 250=90.82%, 500=0.31% cpu : usr=0.43%, sys=2.54%, ctx=2641, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=191113: Thu Jul 24 10:37:03 2025 read: IOPS=57, BW=229KiB/s (234kB/s)(5060KiB/22117msec) slat (usec): min=2903, max=32363, avg=9584.95, stdev=3794.40 clat (msec): min=54, max=231, avg=130.68, stdev=26.23 lat (msec): min=64, max=244, avg=140.27, stdev=27.01 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 93], 10.00th=[ 100], 20.00th=[ 109], | 30.00th=[ 115], 40.00th=[ 122], 50.00th=[ 129], 60.00th=[ 136], | 70.00th=[ 144], 80.00th=[ 153], 90.00th=[ 165], 95.00th=[ 176], | 99.00th=[ 207], 99.50th=[ 211], 99.90th=[ 226], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 128, max= 327, per=25.26%, avg=228.81, stdev=40.81, samples=43 iops : min= 32, max= 81, avg=56.72, stdev=10.19, samples=43 write: IOPS=58, BW=234KiB/s (240kB/s)(5180KiB/22117msec); 0 zone resets slat (usec): min=30, max=11113, avg=236.77, stdev=517.74 clat (msec): min=16, max=252, avg=134.39, stdev=24.90 lat (msec): min=17, max=252, avg=134.63, stdev=24.89 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 99], 10.00th=[ 106], 20.00th=[ 114], | 30.00th=[ 122], 40.00th=[ 127], 50.00th=[ 132], 60.00th=[ 138], | 70.00th=[ 144], 80.00th=[ 153], 90.00th=[ 167], 95.00th=[ 180], | 99.00th=[ 209], 99.50th=[ 215], 99.90th=[ 228], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 167, max= 303, per=24.80%, avg=234.12, stdev=37.90, samples=43 iops : min= 41, max= 75, avg=58.05, stdev= 9.50, samples=43 lat (msec) : 20=0.04%, 100=8.44%, 250=91.48%, 500=0.04% cpu : usr=0.37%, sys=2.75%, ctx=2865, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=907KiB/s (929kB/s), 225KiB/s-229KiB/s (230kB/s-234kB/s), io=19.6MiB (20.5MB), run=22044-22126msec WRITE: bw=944KiB/s (967kB/s), 234KiB/s-238KiB/s (240kB/s-244kB/s), io=20.4MiB (21.4MB), run=22044-22126msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=191120: Thu Jul 24 10:37:10 2025 read: IOPS=2, BW=91.0MiB/s (96.4MB/s)(40.0MiB/435msec) slat (nsec): min=36119k, max=36119k, avg=36118678.00, stdev= 0.00 clat (nsec): min=358949k, max=358949k, avg=358948997.00, stdev= 0.00 lat (nsec): min=395072k, max=395072k, avg=395072106.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 359], | 30.00th=[ 359], 40.00th=[ 359], 50.00th=[ 359], 60.00th=[ 359], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] lat (msec) : 500=100.00% cpu : usr=0.46%, sys=3.92%, ctx=3, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=91.0MiB/s (96.4MB/s), 91.0MiB/s-91.0MiB/s (96.4MB/s-96.4MB/s), io=40.0MiB (41.9MB), run=435-435msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout