== sanity test 398c: run fio to test AIO ================= 14:14:17 (1744136057) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.128217 s, 327 MB/s osc.lustre-OST0000-osc-ffff8800a8510800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=11934: Tue Apr 8 14:14:52 2025 write: IOPS=76, BW=307KiB/s (314kB/s)(10.0MiB/33359msec) slat (usec): min=14, max=1149, avg=70.51, stdev=51.15 clat (msec): min=38, max=424, avg=208.21, stdev=52.93 lat (msec): min=38, max=424, avg=208.28, stdev=52.94 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 136], 10.00th=[ 146], 20.00th=[ 163], | 30.00th=[ 176], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 218], | 70.00th=[ 230], 80.00th=[ 249], 90.00th=[ 279], 95.00th=[ 309], | 99.00th=[ 351], 99.50th=[ 376], 99.90th=[ 409], 99.95th=[ 422], | 99.99th=[ 426] bw ( KiB/s): min= 200, max= 456, per=25.05%, avg=306.62, stdev=64.67, samples=66 iops : min= 50, max= 114, avg=76.62, stdev=16.15, samples=66 lat (msec) : 50=0.23%, 100=0.23%, 250=80.23%, 500=19.30% cpu : usr=0.12%, sys=0.64%, ctx=2021, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=11935: Tue Apr 8 14:14:52 2025 write: IOPS=76, BW=307KiB/s (314kB/s)(10.0MiB/33384msec) slat (usec): min=13, max=642, avg=71.71, stdev=52.30 clat (msec): min=58, max=405, avg=208.33, stdev=51.72 lat (msec): min=58, max=405, avg=208.40, stdev=51.74 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 136], 10.00th=[ 144], 20.00th=[ 161], | 30.00th=[ 178], 40.00th=[ 192], 50.00th=[ 205], 60.00th=[ 218], | 70.00th=[ 232], 80.00th=[ 247], 90.00th=[ 275], 95.00th=[ 309], | 99.00th=[ 355], 99.50th=[ 363], 99.90th=[ 393], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 176, max= 472, per=24.94%, avg=305.29, stdev=64.93, samples=66 iops : min= 44, max= 118, avg=76.29, stdev=16.22, samples=66 lat (msec) : 100=0.08%, 250=81.84%, 500=18.09% cpu : usr=0.10%, sys=0.67%, ctx=1980, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=11936: Tue Apr 8 14:14:52 2025 write: IOPS=76, BW=306KiB/s (314kB/s)(10.0MiB/33441msec) slat (usec): min=12, max=416, avg=71.31, stdev=47.18 clat (msec): min=39, max=424, avg=208.53, stdev=51.37 lat (msec): min=39, max=424, avg=208.60, stdev=51.38 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 138], 10.00th=[ 148], 20.00th=[ 163], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 215], | 70.00th=[ 230], 80.00th=[ 249], 90.00th=[ 275], 95.00th=[ 309], | 99.00th=[ 347], 99.50th=[ 363], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 426] bw ( KiB/s): min= 184, max= 424, per=24.92%, avg=305.06, stdev=62.87, samples=66 iops : min= 46, max= 106, avg=76.24, stdev=15.71, samples=66 lat (msec) : 50=0.04%, 100=0.12%, 250=80.59%, 500=19.26% cpu : usr=0.10%, sys=0.69%, ctx=2010, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=11937: Tue Apr 8 14:14:52 2025 write: IOPS=76, BW=306KiB/s (314kB/s)(10.0MiB/33415msec) slat (usec): min=13, max=386, avg=73.38, stdev=48.09 clat (msec): min=63, max=422, avg=208.55, stdev=51.30 lat (msec): min=63, max=422, avg=208.63, stdev=51.31 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 136], 10.00th=[ 146], 20.00th=[ 161], | 30.00th=[ 178], 40.00th=[ 192], 50.00th=[ 205], 60.00th=[ 218], | 70.00th=[ 230], 80.00th=[ 249], 90.00th=[ 279], 95.00th=[ 309], | 99.00th=[ 351], 99.50th=[ 363], 99.90th=[ 401], 99.95th=[ 409], | 99.99th=[ 422] bw ( KiB/s): min= 183, max= 512, per=24.99%, avg=305.85, stdev=65.72, samples=66 iops : min= 45, max= 128, avg=76.39, stdev=16.50, samples=66 lat (msec) : 100=0.04%, 250=81.48%, 500=18.48% cpu : usr=0.09%, sys=0.69%, ctx=2183, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1225KiB/s (1254kB/s), 306KiB/s-307KiB/s (314kB/s-314kB/s), io=40.0MiB (41.9MB), run=33359-33441msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11973: Tue Apr 8 14:15:16 2025 read: IOPS=56, BW=226KiB/s (231kB/s)(5048KiB/22359msec) slat (usec): min=23, max=5225, avg=1702.72, stdev=586.75 clat (msec): min=3, max=175, avg=93.14, stdev=27.53 lat (msec): min=3, max=177, avg=94.85, stdev=27.88 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 52], 10.00th=[ 57], 20.00th=[ 67], | 30.00th=[ 77], 40.00th=[ 87], 50.00th=[ 94], 60.00th=[ 103], | 70.00th=[ 111], 80.00th=[ 118], 90.00th=[ 128], 95.00th=[ 136], | 99.00th=[ 153], 99.50th=[ 157], 99.90th=[ 163], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 120, max= 400, per=25.00%, avg=223.75, stdev=70.91, samples=44 iops : min= 30, max= 100, avg=55.89, stdev=17.70, samples=44 write: IOPS=58, BW=232KiB/s (238kB/s)(5192KiB/22359msec) slat (usec): min=13, max=1016, avg=87.94, stdev=64.46 clat (msec): min=28, max=380, avg=183.09, stdev=55.43 lat (msec): min=28, max=380, avg=183.18, stdev=55.44 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 102], 10.00th=[ 114], 20.00th=[ 134], | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 178], 60.00th=[ 194], | 70.00th=[ 211], 80.00th=[ 232], 90.00th=[ 257], 95.00th=[ 279], | 99.00th=[ 326], 99.50th=[ 355], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 152, max= 408, per=24.69%, avg=230.11, stdev=57.57, samples=44 iops : min= 38, max= 102, avg=57.48, stdev=14.41, samples=44 lat (msec) : 4=0.08%, 10=0.04%, 20=0.27%, 50=1.84%, 100=28.12% lat (msec) : 250=63.44%, 500=6.21% cpu : usr=0.22%, sys=1.74%, ctx=3261, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11974: Tue Apr 8 14:15:16 2025 read: IOPS=55, BW=222KiB/s (227kB/s)(4976KiB/22414msec) slat (usec): min=638, max=4708, avg=1715.77, stdev=566.66 clat (msec): min=9, max=170, avg=92.55, stdev=26.87 lat (msec): min=10, max=173, avg=94.27, stdev=27.20 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 52], 10.00th=[ 56], 20.00th=[ 67], | 30.00th=[ 75], 40.00th=[ 86], 50.00th=[ 93], 60.00th=[ 101], | 70.00th=[ 110], 80.00th=[ 118], 90.00th=[ 127], 95.00th=[ 136], | 99.00th=[ 148], 99.50th=[ 155], 99.90th=[ 169], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 88, max= 456, per=24.35%, avg=217.93, stdev=73.83, samples=44 iops : min= 22, max= 114, avg=54.43, stdev=18.47, samples=44 write: IOPS=58, BW=235KiB/s (240kB/s)(5264KiB/22414msec) slat (usec): min=15, max=538, avg=85.68, stdev=50.42 clat (msec): min=45, max=375, avg=182.92, stdev=55.58 lat (msec): min=45, max=375, avg=183.01, stdev=55.59 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 102], 10.00th=[ 113], 20.00th=[ 134], | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 178], 60.00th=[ 190], | 70.00th=[ 211], 80.00th=[ 230], 90.00th=[ 262], 95.00th=[ 279], | 99.00th=[ 326], 99.50th=[ 359], 99.90th=[ 372], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 136, max= 392, per=24.79%, avg=231.02, stdev=57.55, samples=44 iops : min= 34, max= 98, avg=57.70, stdev=14.36, samples=44 lat (msec) : 10=0.04%, 20=0.16%, 50=1.88%, 100=29.45%, 250=61.80% lat (msec) : 500=6.68% cpu : usr=0.21%, sys=1.78%, ctx=3299, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11975: Tue Apr 8 14:15:16 2025 read: IOPS=55, BW=222KiB/s (228kB/s)(4984KiB/22414msec) slat (usec): min=714, max=4961, avg=1732.41, stdev=547.20 clat (msec): min=3, max=167, avg=94.01, stdev=26.89 lat (msec): min=4, max=170, avg=95.74, stdev=27.21 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 53], 10.00th=[ 59], 20.00th=[ 70], | 30.00th=[ 78], 40.00th=[ 88], 50.00th=[ 95], 60.00th=[ 102], | 70.00th=[ 111], 80.00th=[ 118], 90.00th=[ 129], 95.00th=[ 136], | 99.00th=[ 153], 99.50th=[ 157], 99.90th=[ 167], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 104, max= 456, per=24.65%, avg=220.66, stdev=65.96, samples=44 iops : min= 26, max= 114, avg=55.11, stdev=16.48, samples=44 write: IOPS=58, BW=234KiB/s (240kB/s)(5256KiB/22414msec) slat (usec): min=13, max=507, avg=89.13, stdev=54.82 clat (msec): min=55, max=368, avg=181.68, stdev=54.20 lat (msec): min=55, max=368, avg=181.77, stdev=54.22 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 104], 10.00th=[ 115], 20.00th=[ 134], | 30.00th=[ 150], 40.00th=[ 163], 50.00th=[ 176], 60.00th=[ 192], | 70.00th=[ 209], 80.00th=[ 226], 90.00th=[ 255], 95.00th=[ 275], | 99.00th=[ 326], 99.50th=[ 334], 99.90th=[ 359], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 144, max= 376, per=24.67%, avg=229.93, stdev=58.89, samples=44 iops : min= 36, max= 94, avg=57.43, stdev=14.73, samples=44 lat (msec) : 4=0.04%, 10=0.04%, 20=0.16%, 50=1.52%, 100=28.36% lat (msec) : 250=63.87%, 500=6.02% cpu : usr=0.20%, sys=1.77%, ctx=3365, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11976: Tue Apr 8 14:15:16 2025 read: IOPS=56, BW=226KiB/s (231kB/s)(5060KiB/22413msec) slat (usec): min=669, max=4726, avg=1721.13, stdev=557.52 clat (msec): min=2, max=169, avg=93.36, stdev=27.32 lat (msec): min=5, max=172, avg=95.08, stdev=27.64 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 53], 10.00th=[ 57], 20.00th=[ 68], | 30.00th=[ 75], 40.00th=[ 88], 50.00th=[ 95], 60.00th=[ 102], | 70.00th=[ 109], 80.00th=[ 118], 90.00th=[ 128], 95.00th=[ 140], | 99.00th=[ 153], 99.50th=[ 155], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 88, max= 440, per=25.02%, avg=223.95, stdev=69.80, samples=44 iops : min= 22, max= 110, avg=55.95, stdev=17.43, samples=44 write: IOPS=57, BW=231KiB/s (237kB/s)(5180KiB/22413msec) slat (usec): min=15, max=470, avg=87.89, stdev=50.18 clat (msec): min=55, max=378, avg=183.43, stdev=55.01 lat (msec): min=55, max=378, avg=183.52, stdev=55.02 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 102], 10.00th=[ 114], 20.00th=[ 134], | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 178], 60.00th=[ 194], | 70.00th=[ 209], 80.00th=[ 230], 90.00th=[ 259], 95.00th=[ 284], | 99.00th=[ 321], 99.50th=[ 334], 99.90th=[ 363], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 144, max= 392, per=24.46%, avg=227.95, stdev=59.24, samples=44 iops : min= 36, max= 98, avg=56.95, stdev=14.80, samples=44 lat (msec) : 4=0.08%, 20=0.16%, 50=1.68%, 100=29.18%, 250=62.62% lat (msec) : 500=6.29% cpu : usr=0.27%, sys=1.74%, ctx=3335, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=895KiB/s (917kB/s), 222KiB/s-226KiB/s (227kB/s-231kB/s), io=19.6MiB (20.5MB), run=22359-22414msec WRITE: bw=932KiB/s (954kB/s), 231KiB/s-235KiB/s (237kB/s-240kB/s), io=20.4MiB (21.4MB), run=22359-22414msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=11998: Tue Apr 8 14:15:17 2025 read: IOPS=7, BW=303MiB/s (318MB/s)(40.0MiB/132msec) slat (nsec): min=6538.9k, max=6538.9k, avg=6538867.00, stdev= 0.00 clat (nsec): min=119894k, max=119894k, avg=119894382.00, stdev= 0.00 lat (nsec): min=126448k, max=126448k, avg=126447536.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 121], 10.00th=[ 121], 20.00th=[ 121], | 30.00th=[ 121], 40.00th=[ 121], 50.00th=[ 121], 60.00th=[ 121], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 121], 95.00th=[ 121], | 99.00th=[ 121], 99.50th=[ 121], 99.90th=[ 121], 99.95th=[ 121], | 99.99th=[ 121] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.58%, ctx=12, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=303MiB/s (318MB/s), 303MiB/s-303MiB/s (318MB/s-318MB/s), io=40.0MiB (41.9MB), run=132-132msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout