== sanity test 398c: run fio to test AIO ================= 23:07:23 (1712027243) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.127013 s, 330 MB/s osc.lustre-OST0000-osc-ffff8800a7c1f800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=9942: Mon Apr 1 23:07:35 2024 write: IOPS=237, BW=949KiB/s (971kB/s)(10.0MiB/10795msec) slat (usec): min=13, max=2035, avg=93.43, stdev=168.71 clat (msec): min=5, max=126, avg=67.14, stdev=13.93 lat (msec): min=5, max=126, avg=67.23, stdev=13.93 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 46], 10.00th=[ 51], 20.00th=[ 55], | 30.00th=[ 59], 40.00th=[ 64], 50.00th=[ 68], 60.00th=[ 71], | 70.00th=[ 74], 80.00th=[ 80], 90.00th=[ 85], 95.00th=[ 91], | 99.00th=[ 101], 99.50th=[ 101], 99.90th=[ 112], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 784, max= 1256, per=24.92%, avg=945.48, stdev=119.85, samples=21 iops : min= 196, max= 314, avg=236.29, stdev=30.00, samples=21 lat (msec) : 10=0.08%, 50=9.73%, 100=89.10%, 250=1.09% cpu : usr=0.00%, sys=3.58%, ctx=1129, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=9943: Mon Apr 1 23:07:35 2024 write: IOPS=237, BW=949KiB/s (972kB/s)(10.0MiB/10788msec) slat (usec): min=13, max=2062, avg=93.35, stdev=167.77 clat (msec): min=8, max=152, avg=67.11, stdev=14.61 lat (msec): min=8, max=152, avg=67.20, stdev=14.61 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 45], 10.00th=[ 50], 20.00th=[ 54], | 30.00th=[ 59], 40.00th=[ 63], 50.00th=[ 68], 60.00th=[ 71], | 70.00th=[ 77], 80.00th=[ 80], 90.00th=[ 85], 95.00th=[ 90], | 99.00th=[ 99], 99.50th=[ 102], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 764, max= 1176, per=24.98%, avg=947.71, stdev=117.44, samples=21 iops : min= 191, max= 294, avg=236.81, stdev=29.36, samples=21 lat (msec) : 10=0.04%, 20=0.08%, 50=11.05%, 100=88.32%, 250=0.51% cpu : usr=0.52%, sys=3.16%, ctx=1124, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=9944: Mon Apr 1 23:07:35 2024 write: IOPS=238, BW=952KiB/s (975kB/s)(10.0MiB/10756msec) slat (usec): min=13, max=2741, avg=99.42, stdev=188.04 clat (msec): min=7, max=146, avg=66.91, stdev=14.64 lat (msec): min=7, max=146, avg=67.01, stdev=14.64 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 44], 10.00th=[ 50], 20.00th=[ 55], | 30.00th=[ 59], 40.00th=[ 64], 50.00th=[ 67], 60.00th=[ 71], | 70.00th=[ 75], 80.00th=[ 80], 90.00th=[ 85], 95.00th=[ 91], | 99.00th=[ 102], 99.50th=[ 103], 99.90th=[ 134], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 768, max= 1232, per=25.06%, avg=950.71, stdev=121.69, samples=21 iops : min= 192, max= 308, avg=237.57, stdev=30.43, samples=21 lat (msec) : 10=0.08%, 50=11.29%, 100=87.50%, 250=1.13% cpu : usr=1.45%, sys=2.19%, ctx=1200, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=9945: Mon Apr 1 23:07:35 2024 write: IOPS=237, BW=949KiB/s (972kB/s)(10.0MiB/10785msec) slat (usec): min=12, max=2866, avg=93.11, stdev=184.33 clat (msec): min=6, max=152, avg=67.09, stdev=14.45 lat (msec): min=6, max=152, avg=67.18, stdev=14.46 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 45], 10.00th=[ 50], 20.00th=[ 55], | 30.00th=[ 59], 40.00th=[ 63], 50.00th=[ 67], 60.00th=[ 71], | 70.00th=[ 75], 80.00th=[ 80], 90.00th=[ 86], 95.00th=[ 91], | 99.00th=[ 100], 99.50th=[ 105], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 153] bw ( KiB/s): min= 792, max= 1232, per=24.96%, avg=946.81, stdev=119.78, samples=21 iops : min= 198, max= 308, avg=236.52, stdev=30.03, samples=21 lat (msec) : 10=0.04%, 20=0.04%, 50=10.62%, 100=88.48%, 250=0.82% cpu : usr=0.50%, sys=3.29%, ctx=1225, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3794KiB/s (3885kB/s), 949KiB/s-952KiB/s (971kB/s-975kB/s), io=40.0MiB (41.9MB), run=10756-10795msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9964: Mon Apr 1 23:07:47 2024 read: IOPS=125, BW=502KiB/s (514kB/s)(5048KiB/10054msec) slat (usec): min=19, max=16066, avg=4062.12, stdev=1621.15 clat (msec): min=7, max=109, avg=59.61, stdev=12.86 lat (msec): min=7, max=117, avg=63.68, stdev=13.32 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 41], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 54], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 70], 90.00th=[ 75], 95.00th=[ 81], | 99.00th=[ 93], 99.50th=[ 96], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 416, max= 600, per=25.09%, avg=500.80, stdev=47.46, samples=20 iops : min= 104, max= 150, avg=125.05, stdev=11.94, samples=20 write: IOPS=129, BW=516KiB/s (529kB/s)(5192KiB/10054msec) slat (usec): min=13, max=3511, avg=91.31, stdev=164.29 clat (msec): min=17, max=104, avg=61.83, stdev=11.87 lat (msec): min=17, max=104, avg=61.92, stdev=11.87 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 45], 10.00th=[ 48], 20.00th=[ 52], | 30.00th=[ 55], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 68], 80.00th=[ 71], 90.00th=[ 78], 95.00th=[ 83], | 99.00th=[ 94], 99.50th=[ 99], 99.90th=[ 102], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 384, max= 622, per=24.86%, avg=516.25, stdev=71.47, samples=20 iops : min= 96, max= 155, avg=128.95, stdev=17.84, samples=20 lat (msec) : 10=0.12%, 20=0.39%, 50=18.32%, 100=80.94%, 250=0.23% cpu : usr=0.00%, sys=6.95%, ctx=2227, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9965: Mon Apr 1 23:07:47 2024 read: IOPS=123, BW=495KiB/s (507kB/s)(4976KiB/10046msec) slat (usec): min=18, max=16204, avg=4181.66, stdev=1754.80 clat (msec): min=15, max=109, avg=59.46, stdev=12.46 lat (msec): min=15, max=116, avg=63.65, stdev=12.88 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 41], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 66], 80.00th=[ 69], 90.00th=[ 75], 95.00th=[ 82], | 99.00th=[ 93], 99.50th=[ 101], 99.90th=[ 108], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 408, max= 616, per=24.73%, avg=493.55, stdev=47.97, samples=20 iops : min= 102, max= 154, avg=123.20, stdev=11.99, samples=20 write: IOPS=130, BW=524KiB/s (537kB/s)(5264KiB/10046msec) slat (usec): min=16, max=2784, avg=101.38, stdev=179.75 clat (msec): min=23, max=121, avg=61.76, stdev=11.77 lat (msec): min=23, max=121, avg=61.86, stdev=11.77 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 44], 10.00th=[ 48], 20.00th=[ 53], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 68], 80.00th=[ 71], 90.00th=[ 78], 95.00th=[ 83], | 99.00th=[ 91], 99.50th=[ 96], 99.90th=[ 112], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 415, max= 630, per=25.21%, avg=523.55, stdev=62.11, samples=20 iops : min= 103, max= 157, avg=130.70, stdev=15.57, samples=20 lat (msec) : 20=0.20%, 50=18.28%, 100=81.17%, 250=0.35% cpu : usr=0.48%, sys=6.56%, ctx=2268, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9966: Mon Apr 1 23:07:47 2024 read: IOPS=124, BW=497KiB/s (509kB/s)(4984KiB/10033msec) slat (usec): min=20, max=12559, avg=4065.95, stdev=1622.70 clat (msec): min=3, max=120, avg=59.27, stdev=12.28 lat (msec): min=4, max=130, avg=63.34, stdev=12.73 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 41], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 59], 60.00th=[ 63], | 70.00th=[ 65], 80.00th=[ 69], 90.00th=[ 74], 95.00th=[ 81], | 99.00th=[ 92], 99.50th=[ 97], 99.90th=[ 109], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 416, max= 592, per=24.87%, avg=496.50, stdev=55.34, samples=20 iops : min= 104, max= 148, avg=124.00, stdev=13.92, samples=20 write: IOPS=130, BW=524KiB/s (536kB/s)(5256KiB/10033msec) slat (usec): min=15, max=4508, avg=107.12, stdev=242.06 clat (msec): min=6, max=107, avg=61.87, stdev=11.41 lat (msec): min=6, max=108, avg=61.98, stdev=11.41 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 45], 10.00th=[ 48], 20.00th=[ 53], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 68], 80.00th=[ 71], 90.00th=[ 77], 95.00th=[ 82], | 99.00th=[ 91], 99.50th=[ 96], 99.90th=[ 104], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 408, max= 646, per=25.07%, avg=520.80, stdev=64.53, samples=20 iops : min= 102, max= 161, avg=130.10, stdev=16.08, samples=20 lat (msec) : 4=0.04%, 10=0.08%, 20=0.08%, 50=17.58%, 100=81.91% lat (msec) : 250=0.31% cpu : usr=0.63%, sys=6.38%, ctx=2263, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9967: Mon Apr 1 23:07:47 2024 read: IOPS=125, BW=503KiB/s (515kB/s)(5060KiB/10054msec) slat (usec): min=29, max=16091, avg=4073.04, stdev=1684.34 clat (msec): min=13, max=108, avg=59.39, stdev=12.46 lat (msec): min=17, max=110, avg=63.47, stdev=12.84 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 40], 10.00th=[ 44], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 67], 80.00th=[ 70], 90.00th=[ 75], 95.00th=[ 81], | 99.00th=[ 92], 99.50th=[ 95], 99.90th=[ 102], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 391, max= 600, per=25.17%, avg=502.45, stdev=60.32, samples=20 iops : min= 97, max= 150, avg=125.40, stdev=15.24, samples=20 write: IOPS=128, BW=515KiB/s (528kB/s)(5180KiB/10054msec) slat (usec): min=16, max=5424, avg=102.58, stdev=216.85 clat (msec): min=28, max=103, avg=62.00, stdev=11.68 lat (msec): min=28, max=104, avg=62.10, stdev=11.68 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 44], 10.00th=[ 48], 20.00th=[ 52], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 68], 80.00th=[ 72], 90.00th=[ 78], 95.00th=[ 83], | 99.00th=[ 92], 99.50th=[ 95], 99.90th=[ 100], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 408, max= 632, per=24.78%, avg=514.70, stdev=64.65, samples=20 iops : min= 102, max= 158, avg=128.50, stdev=16.13, samples=20 lat (msec) : 20=0.04%, 50=18.75%, 100=81.09%, 250=0.12% cpu : usr=1.48%, sys=5.60%, ctx=2273, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1996KiB/s (2044kB/s), 495KiB/s-503KiB/s (507kB/s-515kB/s), io=19.6MiB (20.5MB), run=10033-10054msec WRITE: bw=2078KiB/s (2128kB/s), 515KiB/s-524KiB/s (528kB/s-537kB/s), io=20.4MiB (21.4MB), run=10033-10054msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=9979: Mon Apr 1 23:07:49 2024 read: IOPS=4, BW=163MiB/s (171MB/s)(40.0MiB/246msec) slat (nsec): min=17804k, max=17804k, avg=17803587.00, stdev= 0.00 clat (nsec): min=224716k, max=224716k, avg=224716130.00, stdev= 0.00 lat (nsec): min=242545k, max=242545k, avg=242545035.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 226], 10.00th=[ 226], 20.00th=[ 226], | 30.00th=[ 226], 40.00th=[ 226], 50.00th=[ 226], 60.00th=[ 226], | 70.00th=[ 226], 80.00th=[ 226], 90.00th=[ 226], 95.00th=[ 226], | 99.00th=[ 226], 99.50th=[ 226], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 226] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=3.67%, ctx=12, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=163MiB/s (171MB/s), 163MiB/s-163MiB/s (171MB/s-171MB/s), io=40.0MiB (41.9MB), run=246-246msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout