== sanity test 398c: run fio to test AIO ================= 01:44:01 (1746078241) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.165169 s, 254 MB/s osc.lustre-OST0000-osc-ffff8801369cc800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=2051: Thu May 1 01:45:29 2025 write: IOPS=29, BW=120KiB/s (123kB/s)(10.0MiB/85462msec) slat (usec): min=12, max=3107, avg=129.81, stdev=237.13 clat (msec): min=191, max=1032, avg=533.45, stdev=129.09 lat (msec): min=191, max=1032, avg=533.58, stdev=129.09 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 342], 10.00th=[ 376], 20.00th=[ 422], | 30.00th=[ 460], 40.00th=[ 493], 50.00th=[ 527], 60.00th=[ 558], | 70.00th=[ 592], 80.00th=[ 634], 90.00th=[ 701], 95.00th=[ 768], | 99.00th=[ 894], 99.50th=[ 944], 99.90th=[ 1011], 99.95th=[ 1020], | 99.99th=[ 1036] bw ( KiB/s): min= 40, max= 256, per=24.85%, avg=119.05, stdev=32.99, samples=170 iops : min= 10, max= 64, avg=29.56, stdev= 8.29, samples=170 lat (msec) : 250=0.35%, 500=41.64%, 750=52.11%, 1000=5.70% cpu : usr=0.14%, sys=0.58%, ctx=1635, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=2052: Thu May 1 01:45:29 2025 write: IOPS=30, BW=120KiB/s (123kB/s)(10.0MiB/85333msec) slat (usec): min=11, max=5113, avg=132.38, stdev=309.86 clat (msec): min=151, max=1021, avg=532.74, stdev=130.27 lat (msec): min=151, max=1021, avg=532.88, stdev=130.28 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 334], 10.00th=[ 376], 20.00th=[ 430], | 30.00th=[ 460], 40.00th=[ 493], 50.00th=[ 527], 60.00th=[ 558], | 70.00th=[ 584], 80.00th=[ 634], 90.00th=[ 693], 95.00th=[ 751], | 99.00th=[ 911], 99.50th=[ 944], 99.90th=[ 1003], 99.95th=[ 1011], | 99.99th=[ 1020] bw ( KiB/s): min= 8, max= 256, per=24.94%, avg=119.44, stdev=35.02, samples=170 iops : min= 2, max= 64, avg=29.70, stdev= 8.80, samples=170 lat (msec) : 250=0.66%, 500=41.72%, 750=52.30%, 1000=5.08% cpu : usr=0.06%, sys=0.59%, ctx=1340, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=2053: Thu May 1 01:45:29 2025 write: IOPS=29, BW=120KiB/s (123kB/s)(10.0MiB/85435msec) slat (usec): min=13, max=4337, avg=131.04, stdev=282.30 clat (msec): min=177, max=1021, avg=533.22, stdev=130.73 lat (msec): min=177, max=1021, avg=533.35, stdev=130.74 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 342], 10.00th=[ 376], 20.00th=[ 426], | 30.00th=[ 460], 40.00th=[ 493], 50.00th=[ 523], 60.00th=[ 550], | 70.00th=[ 592], 80.00th=[ 634], 90.00th=[ 701], 95.00th=[ 776], | 99.00th=[ 894], 99.50th=[ 919], 99.90th=[ 1003], 99.95th=[ 1011], | 99.99th=[ 1020] bw ( KiB/s): min= 15, max= 256, per=24.93%, avg=119.43, stdev=38.73, samples=170 iops : min= 3, max= 64, avg=29.69, stdev= 9.71, samples=170 lat (msec) : 250=0.78%, 500=40.78%, 750=52.23%, 1000=6.02% cpu : usr=0.12%, sys=0.54%, ctx=1477, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=2054: Thu May 1 01:45:29 2025 write: IOPS=29, BW=120KiB/s (123kB/s)(10.0MiB/85361msec) slat (usec): min=12, max=4393, avg=136.57, stdev=270.96 clat (msec): min=70, max=1057, avg=532.88, stdev=134.29 lat (msec): min=70, max=1057, avg=533.02, stdev=134.30 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 330], 10.00th=[ 376], 20.00th=[ 422], | 30.00th=[ 460], 40.00th=[ 493], 50.00th=[ 527], 60.00th=[ 558], | 70.00th=[ 592], 80.00th=[ 634], 90.00th=[ 701], 95.00th=[ 760], | 99.00th=[ 944], 99.50th=[ 995], 99.90th=[ 1036], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 16, max= 256, per=24.94%, avg=119.45, stdev=35.98, samples=170 iops : min= 4, max= 64, avg=29.64, stdev= 9.02, samples=170 lat (msec) : 100=0.04%, 250=0.74%, 500=40.66%, 750=53.16%, 1000=4.92% cpu : usr=0.14%, sys=0.56%, ctx=1507, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=479KiB/s (491kB/s), 120KiB/s-120KiB/s (123kB/s-123kB/s), io=40.0MiB (41.9MB), run=85333-85462msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2132: Thu May 1 01:46:19 2025 read: IOPS=25, BW=103KiB/s (106kB/s)(5048KiB/48949msec) slat (usec): min=20, max=12891, avg=4063.76, stdev=1744.44 clat (msec): min=5, max=557, avg=200.92, stdev=51.60 lat (msec): min=5, max=564, avg=204.99, stdev=52.05 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 132], 10.00th=[ 148], 20.00th=[ 167], | 30.00th=[ 178], 40.00th=[ 186], 50.00th=[ 201], 60.00th=[ 211], | 70.00th=[ 224], 80.00th=[ 241], 90.00th=[ 259], 95.00th=[ 275], | 99.00th=[ 321], 99.50th=[ 359], 99.90th=[ 550], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 24, max= 216, per=25.34%, avg=103.40, stdev=39.42, samples=97 iops : min= 6, max= 54, avg=25.71, stdev= 9.91, samples=97 write: IOPS=26, BW=106KiB/s (109kB/s)(5192KiB/48949msec) slat (usec): min=15, max=1998, avg=121.61, stdev=193.74 clat (msec): min=63, max=861, avg=403.46, stdev=110.01 lat (msec): min=63, max=861, avg=403.58, stdev=110.03 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 243], 10.00th=[ 275], 20.00th=[ 313], | 30.00th=[ 338], 40.00th=[ 372], 50.00th=[ 401], 60.00th=[ 426], | 70.00th=[ 456], 80.00th=[ 493], 90.00th=[ 550], 95.00th=[ 600], | 99.00th=[ 676], 99.50th=[ 684], 99.90th=[ 835], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 40, max= 176, per=24.95%, avg=105.79, stdev=28.98, samples=97 iops : min= 10, max= 44, avg=26.31, stdev= 7.27, samples=97 lat (msec) : 10=0.20%, 20=0.31%, 50=0.23%, 100=0.47%, 250=44.02% lat (msec) : 500=45.23%, 750=9.41%, 1000=0.12% cpu : usr=0.15%, sys=1.67%, ctx=2704, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2133: Thu May 1 01:46:19 2025 read: IOPS=25, BW=101KiB/s (104kB/s)(4976KiB/49115msec) slat (usec): min=1348, max=13157, avg=4099.95, stdev=1767.42 clat (msec): min=6, max=428, avg=200.84, stdev=47.13 lat (msec): min=7, max=434, avg=204.94, stdev=47.41 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 136], 10.00th=[ 150], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 186], 50.00th=[ 199], 60.00th=[ 211], | 70.00th=[ 222], 80.00th=[ 239], 90.00th=[ 262], 95.00th=[ 275], | 99.00th=[ 309], 99.50th=[ 342], 99.90th=[ 422], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 23, max= 184, per=24.67%, avg=100.65, stdev=32.46, samples=98 iops : min= 5, max= 46, avg=24.98, stdev= 8.17, samples=98 write: IOPS=26, BW=107KiB/s (110kB/s)(5264KiB/49115msec) slat (usec): min=13, max=2112, avg=128.35, stdev=207.15 clat (msec): min=135, max=790, avg=402.50, stdev=104.45 lat (msec): min=136, max=790, avg=402.63, stdev=104.47 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 241], 10.00th=[ 275], 20.00th=[ 309], | 30.00th=[ 342], 40.00th=[ 368], 50.00th=[ 401], 60.00th=[ 422], | 70.00th=[ 456], 80.00th=[ 493], 90.00th=[ 542], 95.00th=[ 584], | 99.00th=[ 651], 99.50th=[ 676], 99.90th=[ 701], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 56, max= 168, per=25.14%, avg=106.61, stdev=27.62, samples=98 iops : min= 14, max= 42, avg=26.47, stdev= 6.91, samples=98 lat (msec) : 10=0.04%, 20=0.04%, 50=0.47%, 100=0.23%, 250=43.55% lat (msec) : 500=46.52%, 750=9.10%, 1000=0.04% cpu : usr=0.00%, sys=1.81%, ctx=2697, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2134: Thu May 1 01:46:19 2025 read: IOPS=25, BW=101KiB/s (104kB/s)(4984KiB/49160msec) slat (usec): min=1234, max=13508, avg=4110.15, stdev=1701.33 clat (msec): min=5, max=348, avg=202.90, stdev=45.45 lat (msec): min=9, max=360, avg=207.02, stdev=45.76 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 138], 10.00th=[ 150], 20.00th=[ 167], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 213], | 70.00th=[ 226], 80.00th=[ 243], 90.00th=[ 262], 95.00th=[ 275], | 99.00th=[ 309], 99.50th=[ 321], 99.90th=[ 347], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 40, max= 200, per=24.83%, avg=101.32, stdev=32.24, samples=98 iops : min= 10, max= 50, avg=25.21, stdev= 8.05, samples=98 write: IOPS=26, BW=107KiB/s (109kB/s)(5256KiB/49160msec) slat (usec): min=13, max=2960, avg=131.19, stdev=226.06 clat (msec): min=137, max=798, avg=401.44, stdev=105.63 lat (msec): min=137, max=798, avg=401.57, stdev=105.63 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 241], 10.00th=[ 271], 20.00th=[ 309], | 30.00th=[ 338], 40.00th=[ 368], 50.00th=[ 397], 60.00th=[ 422], | 70.00th=[ 451], 80.00th=[ 493], 90.00th=[ 542], 95.00th=[ 592], | 99.00th=[ 651], 99.50th=[ 676], 99.90th=[ 793], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 48, max= 176, per=25.03%, avg=106.13, stdev=27.77, samples=98 iops : min= 12, max= 44, avg=26.42, stdev= 6.92, samples=98 lat (msec) : 10=0.08%, 20=0.04%, 50=0.23%, 100=0.39%, 250=43.75% lat (msec) : 500=45.90%, 750=9.53%, 1000=0.08% cpu : usr=0.13%, sys=1.72%, ctx=2732, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2135: Thu May 1 01:46:19 2025 read: IOPS=25, BW=103KiB/s (105kB/s)(5060KiB/49139msec) slat (usec): min=1185, max=12365, avg=4064.38, stdev=1662.23 clat (msec): min=5, max=359, avg=200.44, stdev=44.32 lat (msec): min=9, max=361, avg=204.51, stdev=44.69 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 138], 10.00th=[ 148], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 188], 50.00th=[ 199], 60.00th=[ 209], | 70.00th=[ 222], 80.00th=[ 241], 90.00th=[ 259], 95.00th=[ 271], | 99.00th=[ 305], 99.50th=[ 313], 99.90th=[ 355], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 16, max= 192, per=25.16%, avg=102.66, stdev=35.46, samples=98 iops : min= 4, max= 48, avg=25.52, stdev= 8.88, samples=98 write: IOPS=26, BW=105KiB/s (108kB/s)(5180KiB/49139msec) slat (usec): min=18, max=2581, avg=138.71, stdev=244.09 clat (msec): min=161, max=718, avg=406.27, stdev=104.20 lat (msec): min=161, max=718, avg=406.41, stdev=104.20 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 249], 10.00th=[ 284], 20.00th=[ 313], | 30.00th=[ 342], 40.00th=[ 372], 50.00th=[ 401], 60.00th=[ 430], | 70.00th=[ 460], 80.00th=[ 493], 90.00th=[ 550], 95.00th=[ 592], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 709], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 56, max= 167, per=24.69%, avg=104.70, stdev=24.82, samples=98 iops : min= 14, max= 41, avg=26.03, stdev= 6.18, samples=98 lat (msec) : 10=0.04%, 50=0.23%, 100=0.27%, 250=44.38%, 500=45.66% lat (msec) : 750=9.41% cpu : usr=0.28%, sys=1.57%, ctx=2818, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=408KiB/s (418kB/s), 101KiB/s-103KiB/s (104kB/s-106kB/s), io=19.6MiB (20.5MB), run=48949-49160msec WRITE: bw=425KiB/s (435kB/s), 105KiB/s-107KiB/s (108kB/s-110kB/s), io=20.4MiB (21.4MB), run=48949-49160msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=2178: Thu May 1 01:46:21 2025 read: IOPS=4, BW=172MiB/s (180MB/s)(40.0MiB/233msec) slat (nsec): min=10490k, max=10490k, avg=10490145.00, stdev= 0.00 clat (nsec): min=211031k, max=211031k, avg=211030743.00, stdev= 0.00 lat (nsec): min=221544k, max=221544k, avg=221544265.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 211], 10.00th=[ 211], 20.00th=[ 211], | 30.00th=[ 211], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 211], | 70.00th=[ 211], 80.00th=[ 211], 90.00th=[ 211], 95.00th=[ 211], | 99.00th=[ 211], 99.50th=[ 211], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] lat (msec) : 250=100.00% cpu : usr=0.43%, sys=3.88%, ctx=5, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=172MiB/s (180MB/s), 172MiB/s-172MiB/s (180MB/s-180MB/s), io=40.0MiB (41.9MB), run=233-233msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout