== sanity test 398c: run fio to test AIO ================= 16:07:04 (1738962424) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0939078 s, 447 MB/s osc.lustre-OST0000-osc-ffff8800a8b22800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=7561: Fri Feb 7 16:07:41 2025 write: IOPS=70, BW=284KiB/s (290kB/s)(10.0MiB/36104msec) slat (usec): min=14, max=365, avg=74.54, stdev=45.47 clat (msec): min=27, max=366, avg=225.22, stdev=40.74 lat (msec): min=27, max=366, avg=225.29, stdev=40.73 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 159], 10.00th=[ 176], 20.00th=[ 190], | 30.00th=[ 203], 40.00th=[ 215], 50.00th=[ 224], 60.00th=[ 234], | 70.00th=[ 247], 80.00th=[ 262], 90.00th=[ 279], 95.00th=[ 292], | 99.00th=[ 321], 99.50th=[ 338], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 184, max= 512, per=24.91%, avg=282.53, stdev=49.90, samples=72 iops : min= 46, max= 128, avg=70.44, stdev=12.43, samples=72 lat (msec) : 50=0.04%, 250=73.95%, 500=26.02% cpu : usr=0.15%, sys=0.66%, ctx=1845, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=7562: Fri Feb 7 16:07:41 2025 write: IOPS=71, BW=285KiB/s (291kB/s)(10.0MiB/35986msec) slat (usec): min=13, max=589, avg=75.69, stdev=46.58 clat (msec): min=50, max=416, avg=224.61, stdev=41.93 lat (msec): min=50, max=416, avg=224.69, stdev=41.93 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 155], 10.00th=[ 171], 20.00th=[ 190], | 30.00th=[ 203], 40.00th=[ 215], 50.00th=[ 224], 60.00th=[ 234], | 70.00th=[ 247], 80.00th=[ 262], 90.00th=[ 279], 95.00th=[ 292], | 99.00th=[ 321], 99.50th=[ 338], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 418] bw ( KiB/s): min= 207, max= 408, per=25.04%, avg=283.97, stdev=43.58, samples=71 iops : min= 51, max= 102, avg=70.85, stdev=10.90, samples=71 lat (msec) : 100=0.16%, 250=73.01%, 500=26.84% cpu : usr=0.12%, sys=0.73%, ctx=1925, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=7563: Fri Feb 7 16:07:41 2025 write: IOPS=70, BW=284KiB/s (291kB/s)(10.0MiB/36090msec) slat (usec): min=14, max=383, avg=75.11, stdev=47.07 clat (msec): min=94, max=383, avg=225.12, stdev=40.51 lat (msec): min=95, max=383, avg=225.20, stdev=40.51 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 159], 10.00th=[ 174], 20.00th=[ 192], | 30.00th=[ 203], 40.00th=[ 215], 50.00th=[ 224], 60.00th=[ 234], | 70.00th=[ 247], 80.00th=[ 262], 90.00th=[ 279], 95.00th=[ 292], | 99.00th=[ 326], 99.50th=[ 338], 99.90th=[ 355], 99.95th=[ 355], | 99.99th=[ 384] bw ( KiB/s): min= 200, max= 512, per=24.92%, avg=282.54, stdev=47.43, samples=72 iops : min= 50, max= 128, avg=70.46, stdev=11.83, samples=72 lat (msec) : 100=0.04%, 250=72.81%, 500=27.15% cpu : usr=0.13%, sys=0.68%, ctx=1813, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=7564: Fri Feb 7 16:07:41 2025 write: IOPS=70, BW=284KiB/s (291kB/s)(10.0MiB/36075msec) slat (usec): min=14, max=515, avg=78.21, stdev=48.37 clat (msec): min=103, max=414, avg=225.13, stdev=41.34 lat (msec): min=103, max=414, avg=225.21, stdev=41.34 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 155], 10.00th=[ 171], 20.00th=[ 192], | 30.00th=[ 205], 40.00th=[ 215], 50.00th=[ 226], 60.00th=[ 234], | 70.00th=[ 247], 80.00th=[ 262], 90.00th=[ 279], 95.00th=[ 292], | 99.00th=[ 326], 99.50th=[ 338], 99.90th=[ 401], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 208, max= 512, per=24.92%, avg=282.56, stdev=46.95, samples=72 iops : min= 52, max= 128, avg=70.47, stdev=11.74, samples=72 lat (msec) : 250=72.97%, 500=27.03% cpu : usr=0.14%, sys=0.72%, ctx=1942, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1135KiB/s (1162kB/s), 284KiB/s-285KiB/s (290kB/s-291kB/s), io=40.0MiB (41.9MB), run=35986-36104msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7603: Fri Feb 7 16:08:04 2025 read: IOPS=59, BW=239KiB/s (244kB/s)(5048KiB/21153msec) slat (usec): min=27, max=10380, avg=1518.08, stdev=511.37 clat (msec): min=2, max=177, avg=86.47, stdev=20.76 lat (msec): min=3, max=178, avg=87.99, stdev=20.84 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 58], 10.00th=[ 65], 20.00th=[ 73], | 30.00th=[ 79], 40.00th=[ 82], 50.00th=[ 87], 60.00th=[ 92], | 70.00th=[ 96], 80.00th=[ 102], 90.00th=[ 111], 95.00th=[ 117], | 99.00th=[ 140], 99.50th=[ 148], 99.90th=[ 163], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 144, max= 328, per=25.32%, avg=238.02, stdev=47.63, samples=42 iops : min= 36, max= 82, avg=59.45, stdev=11.96, samples=42 write: IOPS=61, BW=245KiB/s (251kB/s)(5192KiB/21153msec) slat (usec): min=14, max=377, avg=67.26, stdev=42.14 clat (msec): min=29, max=321, avg=174.93, stdev=43.53 lat (msec): min=30, max=321, avg=174.99, stdev=43.53 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 103], 10.00th=[ 120], 20.00th=[ 136], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 178], 60.00th=[ 190], | 70.00th=[ 201], 80.00th=[ 213], 90.00th=[ 228], 95.00th=[ 245], | 99.00th=[ 268], 99.50th=[ 288], 99.90th=[ 313], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 176, max= 352, per=24.99%, avg=244.69, stdev=40.94, samples=42 iops : min= 44, max= 88, avg=61.12, stdev=10.26, samples=42 lat (msec) : 4=0.16%, 10=0.20%, 20=0.59%, 50=0.51%, 100=39.02% lat (msec) : 250=57.93%, 500=1.60% cpu : usr=0.20%, sys=1.46%, ctx=3265, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7604: Fri Feb 7 16:08:04 2025 read: IOPS=58, BW=233KiB/s (239kB/s)(4976KiB/21339msec) slat (usec): min=763, max=7164, avg=1516.50, stdev=438.89 clat (usec): min=1958, max=144148, avg=86893.45, stdev=18356.77 lat (msec): min=3, max=145, avg=88.41, stdev=18.45 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 61], 10.00th=[ 65], 20.00th=[ 72], | 30.00th=[ 79], 40.00th=[ 83], 50.00th=[ 88], 60.00th=[ 92], | 70.00th=[ 96], 80.00th=[ 102], 90.00th=[ 109], 95.00th=[ 117], | 99.00th=[ 133], 99.50th=[ 136], 99.90th=[ 140], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 144, max= 400, per=24.53%, avg=230.62, stdev=53.21, samples=42 iops : min= 36, max= 100, avg=57.62, stdev=13.33, samples=42 write: IOPS=61, BW=247KiB/s (253kB/s)(5264KiB/21339msec) slat (usec): min=13, max=431, avg=66.40, stdev=41.21 clat (msec): min=29, max=321, avg=175.43, stdev=43.85 lat (msec): min=29, max=321, avg=175.50, stdev=43.85 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 104], 10.00th=[ 118], 20.00th=[ 134], | 30.00th=[ 148], 40.00th=[ 163], 50.00th=[ 178], 60.00th=[ 192], | 70.00th=[ 203], 80.00th=[ 213], 90.00th=[ 230], 95.00th=[ 243], | 99.00th=[ 271], 99.50th=[ 288], 99.90th=[ 309], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 168, max= 368, per=24.86%, avg=243.38, stdev=44.22, samples=42 iops : min= 42, max= 92, avg=60.81, stdev=11.06, samples=42 lat (msec) : 2=0.04%, 4=0.08%, 10=0.04%, 20=0.20%, 50=0.51% lat (msec) : 100=39.18%, 250=58.16%, 500=1.80% cpu : usr=0.26%, sys=1.39%, ctx=3127, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7605: Fri Feb 7 16:08:04 2025 read: IOPS=58, BW=234KiB/s (240kB/s)(4984KiB/21309msec) slat (usec): min=702, max=6639, avg=1540.31, stdev=446.63 clat (usec): min=1736, max=143349, avg=87382.95, stdev=18811.57 lat (msec): min=3, max=144, avg=88.93, stdev=18.89 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 60], 10.00th=[ 65], 20.00th=[ 72], | 30.00th=[ 79], 40.00th=[ 83], 50.00th=[ 87], 60.00th=[ 93], | 70.00th=[ 96], 80.00th=[ 104], 90.00th=[ 112], 95.00th=[ 118], | 99.00th=[ 134], 99.50th=[ 136], 99.90th=[ 140], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 144, max= 456, per=24.78%, avg=232.90, stdev=62.05, samples=42 iops : min= 36, max= 114, avg=58.19, stdev=15.54, samples=42 write: IOPS=61, BW=247KiB/s (253kB/s)(5256KiB/21309msec) slat (usec): min=15, max=320, avg=65.48, stdev=39.24 clat (msec): min=65, max=298, avg=174.72, stdev=42.67 lat (msec): min=65, max=298, avg=174.79, stdev=42.67 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 105], 10.00th=[ 118], 20.00th=[ 134], | 30.00th=[ 148], 40.00th=[ 161], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 199], 80.00th=[ 213], 90.00th=[ 230], 95.00th=[ 245], | 99.00th=[ 268], 99.50th=[ 288], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 152, max= 328, per=24.94%, avg=244.14, stdev=38.75, samples=42 iops : min= 38, max= 82, avg=61.00, stdev= 9.71, samples=42 lat (msec) : 2=0.04%, 4=0.04%, 20=0.23%, 50=0.51%, 100=38.12% lat (msec) : 250=59.30%, 500=1.76% cpu : usr=0.16%, sys=1.51%, ctx=3187, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7606: Fri Feb 7 16:08:04 2025 read: IOPS=59, BW=237KiB/s (243kB/s)(5060KiB/21309msec) slat (usec): min=729, max=10099, avg=1523.88, stdev=454.95 clat (usec): min=1384, max=138221, avg=86889.73, stdev=18497.97 lat (msec): min=2, max=140, avg=88.42, stdev=18.57 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 61], 10.00th=[ 65], 20.00th=[ 72], | 30.00th=[ 80], 40.00th=[ 84], 50.00th=[ 88], 60.00th=[ 91], | 70.00th=[ 95], 80.00th=[ 102], 90.00th=[ 111], 95.00th=[ 115], | 99.00th=[ 133], 99.50th=[ 136], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 120, max= 384, per=25.11%, avg=236.07, stdev=58.41, samples=42 iops : min= 30, max= 96, avg=58.93, stdev=14.62, samples=42 write: IOPS=60, BW=243KiB/s (249kB/s)(5180KiB/21309msec) slat (usec): min=14, max=553, avg=68.29, stdev=45.16 clat (msec): min=63, max=311, avg=176.33, stdev=41.55 lat (msec): min=63, max=311, avg=176.40, stdev=41.55 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 110], 10.00th=[ 122], 20.00th=[ 140], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 178], 60.00th=[ 192], | 70.00th=[ 201], 80.00th=[ 213], 90.00th=[ 230], 95.00th=[ 243], | 99.00th=[ 264], 99.50th=[ 271], 99.90th=[ 300], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 152, max= 312, per=24.56%, avg=240.45, stdev=33.48, samples=42 iops : min= 38, max= 78, avg=60.02, stdev= 8.36, samples=42 lat (msec) : 2=0.04%, 4=0.12%, 20=0.31%, 50=0.39%, 100=39.06% lat (msec) : 250=58.48%, 500=1.60% cpu : usr=0.24%, sys=1.44%, ctx=3492, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=940KiB/s (963kB/s), 233KiB/s-239KiB/s (239kB/s-244kB/s), io=19.6MiB (20.5MB), run=21153-21339msec WRITE: bw=979KiB/s (1003kB/s), 243KiB/s-247KiB/s (249kB/s-253kB/s), io=20.4MiB (21.4MB), run=21153-21339msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=7627: Fri Feb 7 16:08:05 2025 read: IOPS=5, BW=231MiB/s (242MB/s)(40.0MiB/173msec) slat (nsec): min=5764.9k, max=5764.9k, avg=5764917.00, stdev= 0.00 clat (nsec): min=159859k, max=159859k, avg=159858926.00, stdev= 0.00 lat (nsec): min=165633k, max=165633k, avg=165633220.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 161], | 70.00th=[ 161], 80.00th=[ 161], 90.00th=[ 161], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=2.91%, ctx=16, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=231MiB/s (242MB/s), 231MiB/s-231MiB/s (242MB/s-242MB/s), io=40.0MiB (41.9MB), run=173-173msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout