== sanity test 398c: run fio to test AIO ================= 06:25:57 (1712485557) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.131766 s, 318 MB/s osc.lustre-OST0000-osc-ffff88009191f800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=31835: Sun Apr 7 06:26:29 2024 write: IOPS=82, BW=331KiB/s (339kB/s)(10.0MiB/30948msec) slat (usec): min=13, max=774, avg=53.31, stdev=33.39 clat (msec): min=84, max=344, avg=193.20, stdev=34.77 lat (msec): min=84, max=344, avg=193.25, stdev=34.78 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 138], 10.00th=[ 150], 20.00th=[ 165], | 30.00th=[ 176], 40.00th=[ 184], 50.00th=[ 192], 60.00th=[ 201], | 70.00th=[ 209], 80.00th=[ 222], 90.00th=[ 236], 95.00th=[ 253], | 99.00th=[ 284], 99.50th=[ 300], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 232, max= 512, per=25.03%, avg=330.93, stdev=43.31, samples=61 iops : min= 58, max= 128, avg=82.67, stdev=10.81, samples=61 lat (msec) : 100=0.23%, 250=94.14%, 500=5.62% cpu : usr=0.10%, sys=0.59%, ctx=2395, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=31836: Sun Apr 7 06:26:29 2024 write: IOPS=82, BW=332KiB/s (340kB/s)(10.0MiB/30855msec) slat (usec): min=12, max=402, avg=53.27, stdev=32.68 clat (msec): min=27, max=379, avg=192.61, stdev=36.01 lat (msec): min=27, max=379, avg=192.67, stdev=36.01 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 136], 10.00th=[ 150], 20.00th=[ 165], | 30.00th=[ 176], 40.00th=[ 184], 50.00th=[ 192], 60.00th=[ 201], | 70.00th=[ 211], 80.00th=[ 222], 90.00th=[ 236], 95.00th=[ 253], | 99.00th=[ 288], 99.50th=[ 296], 99.90th=[ 347], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 248, max= 512, per=25.10%, avg=331.85, stdev=46.09, samples=61 iops : min= 62, max= 128, avg=82.90, stdev=11.49, samples=61 lat (msec) : 50=0.08%, 100=0.35%, 250=93.71%, 500=5.86% cpu : usr=0.12%, sys=0.57%, ctx=2199, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=31837: Sun Apr 7 06:26:29 2024 write: IOPS=82, BW=331KiB/s (338kB/s)(10.0MiB/30980msec) slat (usec): min=12, max=739, avg=55.15, stdev=36.70 clat (msec): min=78, max=317, avg=193.32, stdev=34.60 lat (msec): min=78, max=317, avg=193.38, stdev=34.61 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 138], 10.00th=[ 150], 20.00th=[ 165], | 30.00th=[ 176], 40.00th=[ 184], 50.00th=[ 192], 60.00th=[ 201], | 70.00th=[ 211], 80.00th=[ 222], 90.00th=[ 236], 95.00th=[ 255], | 99.00th=[ 279], 99.50th=[ 288], 99.90th=[ 305], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 216, max= 512, per=24.95%, avg=329.89, stdev=46.19, samples=61 iops : min= 54, max= 128, avg=82.41, stdev=11.54, samples=61 lat (msec) : 100=0.16%, 250=93.71%, 500=6.13% cpu : usr=0.11%, sys=0.57%, ctx=2565, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=31838: Sun Apr 7 06:26:29 2024 write: IOPS=82, BW=331KiB/s (338kB/s)(10.0MiB/30980msec) slat (usec): min=12, max=827, avg=53.77, stdev=36.42 clat (msec): min=59, max=317, avg=193.42, stdev=34.14 lat (msec): min=59, max=317, avg=193.48, stdev=34.14 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 138], 10.00th=[ 153], 20.00th=[ 165], | 30.00th=[ 176], 40.00th=[ 184], 50.00th=[ 194], 60.00th=[ 201], | 70.00th=[ 209], 80.00th=[ 222], 90.00th=[ 236], 95.00th=[ 251], | 99.00th=[ 284], 99.50th=[ 292], 99.90th=[ 300], 99.95th=[ 313], | 99.99th=[ 317] bw ( KiB/s): min= 248, max= 520, per=24.96%, avg=329.92, stdev=44.60, samples=61 iops : min= 62, max= 130, avg=82.44, stdev=11.14, samples=61 lat (msec) : 100=0.12%, 250=94.41%, 500=5.47% cpu : usr=0.09%, sys=0.62%, ctx=2402, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1322KiB/s (1354kB/s), 331KiB/s-332KiB/s (338kB/s-340kB/s), io=40.0MiB (41.9MB), run=30855-30980msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31873: Sun Apr 7 06:26:48 2024 read: IOPS=70, BW=283KiB/s (290kB/s)(5048KiB/17812msec) slat (usec): min=21, max=3929, avg=1194.37, stdev=322.06 clat (usec): min=1641, max=182226, avg=74450.39, stdev=14209.06 lat (msec): min=5, max=183, avg=75.65, stdev=14.23 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 55], 10.00th=[ 61], 20.00th=[ 65], | 30.00th=[ 67], 40.00th=[ 71], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 81], 80.00th=[ 85], 90.00th=[ 91], 95.00th=[ 95], | 99.00th=[ 110], 99.50th=[ 114], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 168, max= 367, per=25.21%, avg=282.14, stdev=49.30, samples=35 iops : min= 42, max= 91, avg=70.43, stdev=12.30, samples=35 write: IOPS=72, BW=291KiB/s (298kB/s)(5192KiB/17812msec) slat (usec): min=13, max=257, avg=45.49, stdev=20.85 clat (msec): min=43, max=268, avg=145.89, stdev=35.66 lat (msec): min=43, max=269, avg=145.93, stdev=35.66 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 91], 10.00th=[ 103], 20.00th=[ 115], | 30.00th=[ 125], 40.00th=[ 136], 50.00th=[ 144], 60.00th=[ 155], | 70.00th=[ 165], 80.00th=[ 176], 90.00th=[ 194], 95.00th=[ 209], | 99.00th=[ 228], 99.50th=[ 247], 99.90th=[ 259], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 215, max= 367, per=24.90%, avg=290.14, stdev=40.68, samples=35 iops : min= 53, max= 91, avg=72.43, stdev=10.14, samples=35 lat (msec) : 2=0.04%, 20=0.35%, 50=1.02%, 100=51.05%, 250=47.38% lat (msec) : 500=0.16% cpu : usr=0.13%, sys=1.34%, ctx=3160, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31874: Sun Apr 7 06:26:48 2024 read: IOPS=69, BW=278KiB/s (284kB/s)(4976KiB/17919msec) slat (usec): min=23, max=3525, avg=1198.49, stdev=317.05 clat (usec): min=1851, max=117706, avg=74683.07, stdev=13395.85 lat (msec): min=2, max=118, avg=75.88, stdev=13.41 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 54], 10.00th=[ 60], 20.00th=[ 65], | 30.00th=[ 68], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 81], 80.00th=[ 85], 90.00th=[ 91], 95.00th=[ 95], | 99.00th=[ 110], 99.50th=[ 113], 99.90th=[ 117], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 160, max= 376, per=24.63%, avg=275.63, stdev=54.00, samples=35 iops : min= 40, max= 94, avg=68.89, stdev=13.51, samples=35 write: IOPS=73, BW=294KiB/s (301kB/s)(5264KiB/17919msec) slat (usec): min=12, max=1417, avg=44.40, stdev=42.79 clat (msec): min=43, max=260, avg=145.99, stdev=35.93 lat (msec): min=43, max=260, avg=146.04, stdev=35.93 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 92], 10.00th=[ 103], 20.00th=[ 115], | 30.00th=[ 126], 40.00th=[ 136], 50.00th=[ 146], 60.00th=[ 155], | 70.00th=[ 165], 80.00th=[ 176], 90.00th=[ 194], 95.00th=[ 209], | 99.00th=[ 236], 99.50th=[ 249], 99.90th=[ 257], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 216, max= 383, per=25.03%, avg=291.63, stdev=39.33, samples=35 iops : min= 54, max= 95, avg=72.89, stdev= 9.78, samples=35 lat (msec) : 2=0.04%, 10=0.04%, 20=0.20%, 50=1.17%, 100=50.51% lat (msec) : 250=47.93%, 500=0.12% cpu : usr=0.18%, sys=1.28%, ctx=2768, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31875: Sun Apr 7 06:26:48 2024 read: IOPS=69, BW=278KiB/s (285kB/s)(4984KiB/17930msec) slat (usec): min=22, max=3220, avg=1200.01, stdev=302.07 clat (usec): min=1165, max=115402, avg=73862.13, stdev=12679.29 lat (msec): min=2, max=116, avg=75.06, stdev=12.70 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 54], 10.00th=[ 61], 20.00th=[ 65], | 30.00th=[ 68], 40.00th=[ 71], 50.00th=[ 74], 60.00th=[ 78], | 70.00th=[ 80], 80.00th=[ 84], 90.00th=[ 90], 95.00th=[ 95], | 99.00th=[ 106], 99.50th=[ 111], 99.90th=[ 114], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 168, max= 440, per=24.80%, avg=277.46, stdev=58.73, samples=35 iops : min= 42, max= 110, avg=69.34, stdev=14.67, samples=35 write: IOPS=73, BW=293KiB/s (300kB/s)(5256KiB/17930msec) slat (usec): min=12, max=1269, avg=46.14, stdev=40.02 clat (msec): min=43, max=260, avg=147.03, stdev=34.71 lat (msec): min=43, max=260, avg=147.08, stdev=34.70 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 93], 10.00th=[ 105], 20.00th=[ 117], | 30.00th=[ 128], 40.00th=[ 138], 50.00th=[ 146], 60.00th=[ 155], | 70.00th=[ 165], 80.00th=[ 178], 90.00th=[ 197], 95.00th=[ 205], | 99.00th=[ 228], 99.50th=[ 239], 99.90th=[ 259], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 216, max= 352, per=24.80%, avg=288.89, stdev=33.71, samples=35 iops : min= 54, max= 88, avg=72.20, stdev= 8.39, samples=35 lat (msec) : 2=0.04%, 4=0.04%, 20=0.16%, 50=1.05%, 100=50.04% lat (msec) : 250=48.55%, 500=0.12% cpu : usr=0.27%, sys=1.18%, ctx=3309, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31876: Sun Apr 7 06:26:48 2024 read: IOPS=71, BW=284KiB/s (291kB/s)(5060KiB/17800msec) slat (usec): min=21, max=3953, avg=1190.51, stdev=307.23 clat (msec): min=3, max=171, avg=74.63, stdev=14.02 lat (msec): min=4, max=172, avg=75.82, stdev=14.05 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 55], 10.00th=[ 61], 20.00th=[ 65], | 30.00th=[ 68], 40.00th=[ 71], 50.00th=[ 75], 60.00th=[ 78], | 70.00th=[ 81], 80.00th=[ 85], 90.00th=[ 91], 95.00th=[ 95], | 99.00th=[ 112], 99.50th=[ 114], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 176, max= 376, per=25.43%, avg=284.54, stdev=54.34, samples=35 iops : min= 44, max= 94, avg=71.11, stdev=13.56, samples=35 write: IOPS=72, BW=291KiB/s (298kB/s)(5180KiB/17800msec) slat (usec): min=13, max=403, avg=46.16, stdev=23.07 clat (msec): min=43, max=269, avg=145.73, stdev=35.14 lat (msec): min=43, max=269, avg=145.78, stdev=35.14 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 92], 10.00th=[ 102], 20.00th=[ 115], | 30.00th=[ 126], 40.00th=[ 136], 50.00th=[ 144], 60.00th=[ 153], | 70.00th=[ 163], 80.00th=[ 176], 90.00th=[ 194], 95.00th=[ 207], | 99.00th=[ 226], 99.50th=[ 230], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 200, max= 360, per=24.82%, avg=289.11, stdev=33.25, samples=35 iops : min= 50, max= 90, avg=72.26, stdev= 8.29, samples=35 lat (msec) : 4=0.04%, 20=0.27%, 50=1.05%, 100=51.13%, 250=47.38% lat (msec) : 500=0.12% cpu : usr=0.20%, sys=1.27%, ctx=3509, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1119KiB/s (1146kB/s), 278KiB/s-284KiB/s (284kB/s-291kB/s), io=19.6MiB (20.5MB), run=17800-17930msec WRITE: bw=1165KiB/s (1193kB/s), 291KiB/s-294KiB/s (298kB/s-301kB/s), io=20.4MiB (21.4MB), run=17800-17930msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=31894: Sun Apr 7 06:26:50 2024 read: IOPS=6, BW=250MiB/s (262MB/s)(40.0MiB/160msec) slat (nsec): min=6642.9k, max=6642.9k, avg=6642876.00, stdev= 0.00 clat (nsec): min=152419k, max=152419k, avg=152419341.00, stdev= 0.00 lat (nsec): min=159072k, max=159072k, avg=159072027.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 153], 10.00th=[ 153], 20.00th=[ 153], | 30.00th=[ 153], 40.00th=[ 153], 50.00th=[ 153], 60.00th=[ 153], | 70.00th=[ 153], 80.00th=[ 153], 90.00th=[ 153], 95.00th=[ 153], | 99.00th=[ 153], 99.50th=[ 153], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.40%, ctx=16, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=250MiB/s (262MB/s), 250MiB/s-250MiB/s (262MB/s-262MB/s), io=40.0MiB (41.9MB), run=160-160msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout