== sanity test 398c: run fio to test AIO ================= 09:39:33 (1741700373) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.15533 s, 270 MB/s osc.lustre-OST0000-osc-ffff8800b5ce7800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=8509: Tue Mar 11 09:40:28 2025 write: IOPS=47, BW=191KiB/s (195kB/s)(10.0MiB/53708msec) slat (usec): min=15, max=981, avg=103.92, stdev=68.77 clat (msec): min=39, max=566, avg=335.38, stdev=93.05 lat (msec): min=39, max=566, avg=335.48, stdev=93.06 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 188], 10.00th=[ 209], 20.00th=[ 241], | 30.00th=[ 275], 40.00th=[ 321], 50.00th=[ 351], 60.00th=[ 376], | 70.00th=[ 393], 80.00th=[ 418], 90.00th=[ 451], 95.00th=[ 481], | 99.00th=[ 518], 99.50th=[ 535], 99.90th=[ 550], 99.95th=[ 558], | 99.99th=[ 567] bw ( KiB/s): min= 104, max= 344, per=24.93%, avg=189.49, stdev=53.89, samples=107 iops : min= 26, max= 86, avg=47.22, stdev=13.48, samples=107 lat (msec) : 50=0.08%, 100=0.39%, 250=22.85%, 500=74.49%, 750=2.19% cpu : usr=0.12%, sys=0.60%, ctx=1972, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=8510: Tue Mar 11 09:40:28 2025 write: IOPS=47, BW=190KiB/s (195kB/s)(10.0MiB/53825msec) slat (usec): min=15, max=713, avg=104.75, stdev=64.62 clat (msec): min=87, max=566, avg=335.77, stdev=92.50 lat (msec): min=87, max=567, avg=335.88, stdev=92.52 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 184], 10.00th=[ 207], 20.00th=[ 243], | 30.00th=[ 275], 40.00th=[ 317], 50.00th=[ 351], 60.00th=[ 376], | 70.00th=[ 393], 80.00th=[ 418], 90.00th=[ 451], 95.00th=[ 477], | 99.00th=[ 518], 99.50th=[ 527], 99.90th=[ 558], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 120, max= 352, per=24.78%, avg=188.30, stdev=52.77, samples=107 iops : min= 30, max= 88, avg=46.99, stdev=13.20, samples=107 lat (msec) : 100=0.08%, 250=23.01%, 500=74.69%, 750=2.23% cpu : usr=0.14%, sys=0.57%, ctx=2178, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=8511: Tue Mar 11 09:40:28 2025 write: IOPS=47, BW=190KiB/s (195kB/s)(10.0MiB/53834msec) slat (usec): min=14, max=750, avg=102.67, stdev=65.12 clat (msec): min=85, max=579, avg=335.80, stdev=91.82 lat (msec): min=85, max=579, avg=335.91, stdev=91.83 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 190], 10.00th=[ 209], 20.00th=[ 239], | 30.00th=[ 275], 40.00th=[ 321], 50.00th=[ 351], 60.00th=[ 376], | 70.00th=[ 393], 80.00th=[ 418], 90.00th=[ 447], 95.00th=[ 481], | 99.00th=[ 518], 99.50th=[ 542], 99.90th=[ 558], 99.95th=[ 567], | 99.99th=[ 584] bw ( KiB/s): min= 112, max= 360, per=24.84%, avg=188.75, stdev=52.58, samples=107 iops : min= 28, max= 90, avg=47.10, stdev=13.15, samples=107 lat (msec) : 100=0.04%, 250=23.32%, 500=74.61%, 750=2.03% cpu : usr=0.14%, sys=0.58%, ctx=2004, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=8512: Tue Mar 11 09:40:28 2025 write: IOPS=47, BW=190KiB/s (195kB/s)(10.0MiB/53780msec) slat (usec): min=13, max=918, avg=100.55, stdev=64.62 clat (msec): min=125, max=579, avg=335.69, stdev=92.22 lat (msec): min=125, max=579, avg=335.79, stdev=92.23 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 186], 10.00th=[ 209], 20.00th=[ 241], | 30.00th=[ 275], 40.00th=[ 317], 50.00th=[ 351], 60.00th=[ 376], | 70.00th=[ 397], 80.00th=[ 418], 90.00th=[ 451], 95.00th=[ 477], | 99.00th=[ 518], 99.50th=[ 531], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 584] bw ( KiB/s): min= 120, max= 336, per=24.87%, avg=189.03, stdev=56.08, samples=107 iops : min= 30, max= 84, avg=47.21, stdev=14.03, samples=107 lat (msec) : 250=24.14%, 500=73.75%, 750=2.11% cpu : usr=0.11%, sys=0.59%, ctx=1873, majf=0, minf=27 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=761KiB/s (779kB/s), 190KiB/s-191KiB/s (195kB/s-195kB/s), io=40.0MiB (41.9MB), run=53708-53834msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8564: Tue Mar 11 09:40:53 2025 read: IOPS=52, BW=212KiB/s (217kB/s)(5048KiB/23815msec) slat (usec): min=16, max=3990, avg=1448.14, stdev=571.07 clat (msec): min=2, max=241, avg=100.81, stdev=38.63 lat (msec): min=2, max=242, avg=102.26, stdev=39.03 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 60], 10.00th=[ 67], 20.00th=[ 73], | 30.00th=[ 79], 40.00th=[ 85], 50.00th=[ 90], 60.00th=[ 96], | 70.00th=[ 106], 80.00th=[ 129], 90.00th=[ 163], 95.00th=[ 182], | 99.00th=[ 215], 99.50th=[ 220], 99.90th=[ 230], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min= 64, max= 352, per=25.29%, avg=212.15, stdev=70.49, samples=47 iops : min= 16, max= 88, avg=52.96, stdev=17.68, samples=47 write: IOPS=54, BW=218KiB/s (223kB/s)(5192KiB/23815msec) slat (usec): min=15, max=526, avg=68.00, stdev=41.85 clat (msec): min=24, max=475, avg=193.83, stdev=73.47 lat (msec): min=24, max=475, avg=193.90, stdev=73.49 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 115], 10.00th=[ 125], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 205], 80.00th=[ 236], 90.00th=[ 300], 95.00th=[ 363], | 99.00th=[ 435], 99.50th=[ 456], 99.90th=[ 477], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 80, max= 352, per=24.93%, avg=217.60, stdev=67.60, samples=47 iops : min= 20, max= 88, avg=54.32, stdev=16.92, samples=47 lat (msec) : 4=0.16%, 10=0.12%, 20=0.23%, 50=0.70%, 100=31.21% lat (msec) : 250=58.63%, 500=8.95% cpu : usr=0.19%, sys=1.31%, ctx=3121, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8565: Tue Mar 11 09:40:53 2025 read: IOPS=52, BW=208KiB/s (213kB/s)(4976KiB/23917msec) slat (usec): min=733, max=5286, avg=1455.61, stdev=578.52 clat (msec): min=13, max=218, avg=101.06, stdev=37.67 lat (msec): min=14, max=221, avg=102.52, stdev=38.07 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 61], 10.00th=[ 67], 20.00th=[ 74], | 30.00th=[ 80], 40.00th=[ 85], 50.00th=[ 89], 60.00th=[ 95], | 70.00th=[ 106], 80.00th=[ 131], 90.00th=[ 163], 95.00th=[ 182], | 99.00th=[ 209], 99.50th=[ 211], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 56, max= 392, per=24.68%, avg=207.04, stdev=75.00, samples=47 iops : min= 14, max= 98, avg=51.68, stdev=18.82, samples=47 write: IOPS=55, BW=220KiB/s (225kB/s)(5264KiB/23917msec) slat (usec): min=15, max=673, avg=69.86, stdev=46.76 clat (msec): min=20, max=495, avg=193.51, stdev=72.21 lat (msec): min=20, max=495, avg=193.58, stdev=72.22 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 115], 10.00th=[ 127], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 190], | 70.00th=[ 205], 80.00th=[ 232], 90.00th=[ 292], 95.00th=[ 351], | 99.00th=[ 447], 99.50th=[ 464], 99.90th=[ 493], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 88, max= 360, per=24.91%, avg=217.43, stdev=65.53, samples=47 iops : min= 22, max= 90, avg=54.28, stdev=16.38, samples=47 lat (msec) : 20=0.20%, 50=0.51%, 100=31.95%, 250=59.45%, 500=7.89% cpu : usr=0.20%, sys=1.26%, ctx=3188, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8566: Tue Mar 11 09:40:53 2025 read: IOPS=52, BW=208KiB/s (213kB/s)(4984KiB/23916msec) slat (usec): min=729, max=5322, avg=1467.78, stdev=570.48 clat (usec): min=1988, max=223204, avg=101510.78, stdev=37400.64 lat (msec): min=3, max=225, avg=102.98, stdev=37.80 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 61], 10.00th=[ 66], 20.00th=[ 74], | 30.00th=[ 81], 40.00th=[ 85], 50.00th=[ 90], 60.00th=[ 99], | 70.00th=[ 108], 80.00th=[ 130], 90.00th=[ 161], 95.00th=[ 178], | 99.00th=[ 215], 99.50th=[ 222], 99.90th=[ 222], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 88, max= 400, per=24.84%, avg=208.40, stdev=72.08, samples=47 iops : min= 22, max= 100, avg=52.02, stdev=18.01, samples=47 write: IOPS=54, BW=220KiB/s (225kB/s)(5256KiB/23916msec) slat (usec): min=15, max=829, avg=70.16, stdev=47.52 clat (msec): min=64, max=516, avg=193.13, stdev=72.87 lat (msec): min=64, max=516, avg=193.20, stdev=72.89 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 115], 10.00th=[ 125], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 205], 80.00th=[ 232], 90.00th=[ 292], 95.00th=[ 351], | 99.00th=[ 451], 99.50th=[ 477], 99.90th=[ 502], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 88, max= 344, per=24.87%, avg=217.09, stdev=67.49, samples=47 iops : min= 22, max= 86, avg=54.19, stdev=16.94, samples=47 lat (msec) : 2=0.08%, 4=0.04%, 20=0.12%, 50=0.55%, 100=30.70% lat (msec) : 250=60.23%, 500=8.20%, 750=0.08% cpu : usr=0.18%, sys=1.31%, ctx=3040, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8567: Tue Mar 11 09:40:53 2025 read: IOPS=53, BW=212KiB/s (218kB/s)(5060KiB/23815msec) slat (usec): min=687, max=4596, avg=1460.00, stdev=573.20 clat (msec): min=9, max=223, avg=100.61, stdev=36.70 lat (msec): min=11, max=225, avg=102.07, stdev=37.10 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 62], 10.00th=[ 68], 20.00th=[ 74], | 30.00th=[ 80], 40.00th=[ 85], 50.00th=[ 90], 60.00th=[ 95], | 70.00th=[ 106], 80.00th=[ 127], 90.00th=[ 161], 95.00th=[ 178], | 99.00th=[ 211], 99.50th=[ 218], 99.90th=[ 220], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 87, max= 368, per=25.45%, avg=213.51, stdev=72.10, samples=47 iops : min= 21, max= 92, avg=53.30, stdev=18.06, samples=47 write: IOPS=54, BW=218KiB/s (223kB/s)(5180KiB/23815msec) slat (usec): min=14, max=936, avg=71.06, stdev=48.81 clat (msec): min=75, max=483, avg=194.15, stdev=71.79 lat (msec): min=75, max=483, avg=194.22, stdev=71.81 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 116], 10.00th=[ 126], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 205], 80.00th=[ 234], 90.00th=[ 296], 95.00th=[ 359], | 99.00th=[ 435], 99.50th=[ 451], 99.90th=[ 481], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 88, max= 360, per=24.77%, avg=216.23, stdev=65.20, samples=47 iops : min= 22, max= 90, avg=53.98, stdev=16.33, samples=47 lat (msec) : 10=0.04%, 20=0.39%, 50=0.27%, 100=32.46%, 250=58.40% lat (msec) : 500=8.44% cpu : usr=0.21%, sys=1.29%, ctx=3126, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=839KiB/s (859kB/s), 208KiB/s-212KiB/s (213kB/s-218kB/s), io=19.6MiB (20.5MB), run=23815-23917msec WRITE: bw=874KiB/s (894kB/s), 218KiB/s-220KiB/s (223kB/s-225kB/s), io=20.4MiB (21.4MB), run=23815-23917msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=8590: Tue Mar 11 09:40:55 2025 read: IOPS=12, BW=500MiB/s (524MB/s)(40.0MiB/80msec) slat (nsec): min=6306.5k, max=6306.5k, avg=6306496.00, stdev= 0.00 clat (nsec): min=68007k, max=68007k, avg=68006950.00, stdev= 0.00 lat (nsec): min=74324k, max=74324k, avg=74323697.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[67634], 5.00th=[67634], 10.00th=[67634], 20.00th=[67634], | 30.00th=[67634], 40.00th=[67634], 50.00th=[67634], 60.00th=[67634], | 70.00th=[67634], 80.00th=[67634], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=7.59%, ctx=17, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=500MiB/s (524MB/s), 500MiB/s-500MiB/s (524MB/s-524MB/s), io=40.0MiB (41.9MB), run=80-80msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout