== sanity test 398c: run fio to test AIO ================= 08:24:27 (1733405067) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0871375 s, 481 MB/s osc.lustre-OST0000-osc-ffff8800abc18000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=21715: Thu Dec 5 08:24:34 2024 write: IOPS=435, BW=1741KiB/s (1782kB/s)(10.0MiB/5883msec) slat (usec): min=12, max=1557, avg=69.57, stdev=92.11 clat (usec): min=3187, max=73068, avg=36583.29, stdev=12532.91 lat (usec): min=3219, max=73199, avg=36654.34, stdev=12545.05 clat percentiles (usec): | 1.00th=[11994], 5.00th=[17433], 10.00th=[19792], 20.00th=[24249], | 30.00th=[28181], 40.00th=[32637], 50.00th=[36439], 60.00th=[40633], | 70.00th=[44827], 80.00th=[48497], 90.00th=[52691], 95.00th=[56361], | 99.00th=[64226], 99.50th=[66323], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 1280, max= 3208, per=25.35%, avg=1756.00, stdev=581.44, samples=11 iops : min= 320, max= 802, avg=439.00, stdev=145.36, samples=11 lat (msec) : 4=0.04%, 10=0.39%, 20=9.96%, 50=73.44%, 100=16.17% cpu : usr=0.58%, sys=3.84%, ctx=1338, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=21716: Thu Dec 5 08:24:34 2024 write: IOPS=434, BW=1737KiB/s (1779kB/s)(10.0MiB/5894msec) slat (usec): min=12, max=1956, avg=70.96, stdev=94.80 clat (usec): min=3060, max=75814, avg=36639.37, stdev=12633.77 lat (usec): min=3114, max=75905, avg=36712.09, stdev=12646.73 clat percentiles (usec): | 1.00th=[12125], 5.00th=[17433], 10.00th=[19792], 20.00th=[23987], | 30.00th=[28181], 40.00th=[32375], 50.00th=[36439], 60.00th=[40633], | 70.00th=[44303], 80.00th=[47973], 90.00th=[52691], 95.00th=[56361], | 99.00th=[65799], 99.50th=[68682], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 1264, max= 3136, per=25.30%, avg=1753.09, stdev=573.60, samples=11 iops : min= 316, max= 784, avg=438.27, stdev=143.40, samples=11 lat (msec) : 4=0.08%, 10=0.16%, 20=10.59%, 50=73.40%, 100=15.78% cpu : usr=0.95%, sys=3.63%, ctx=1364, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=21717: Thu Dec 5 08:24:34 2024 write: IOPS=433, BW=1733KiB/s (1774kB/s)(10.0MiB/5910msec) slat (usec): min=12, max=1909, avg=77.01, stdev=114.82 clat (usec): min=2854, max=74989, avg=36711.35, stdev=12471.53 lat (usec): min=2907, max=75031, avg=36790.16, stdev=12490.64 clat percentiles (usec): | 1.00th=[12518], 5.00th=[17433], 10.00th=[19792], 20.00th=[24249], | 30.00th=[28967], 40.00th=[32637], 50.00th=[36439], 60.00th=[40633], | 70.00th=[44303], 80.00th=[47973], 90.00th=[52691], 95.00th=[56361], | 99.00th=[64226], 99.50th=[65799], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 1272, max= 3104, per=25.23%, avg=1748.00, stdev=563.30, samples=11 iops : min= 318, max= 776, avg=437.00, stdev=140.82, samples=11 lat (msec) : 4=0.04%, 10=0.04%, 20=10.31%, 50=74.06%, 100=15.55% cpu : usr=0.81%, sys=3.86%, ctx=1323, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=21718: Thu Dec 5 08:24:34 2024 write: IOPS=433, BW=1732KiB/s (1774kB/s)(10.0MiB/5912msec) slat (usec): min=12, max=1522, avg=74.01, stdev=93.09 clat (usec): min=3860, max=71867, avg=36713.00, stdev=12383.43 lat (usec): min=3936, max=71897, avg=36788.52, stdev=12399.85 clat percentiles (usec): | 1.00th=[11994], 5.00th=[17957], 10.00th=[20317], 20.00th=[24249], | 30.00th=[28967], 40.00th=[32900], 50.00th=[36439], 60.00th=[40109], | 70.00th=[44303], 80.00th=[48497], 90.00th=[52167], 95.00th=[56361], | 99.00th=[64750], 99.50th=[66323], 99.90th=[68682], 99.95th=[68682], | 99.99th=[71828] bw ( KiB/s): min= 1232, max= 3072, per=25.18%, avg=1744.36, stdev=558.41, samples=11 iops : min= 308, max= 768, avg=436.09, stdev=139.60, samples=11 lat (msec) : 4=0.04%, 10=0.62%, 20=8.71%, 50=75.27%, 100=15.35% cpu : usr=0.59%, sys=4.20%, ctx=1373, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=6928KiB/s (7095kB/s), 1732KiB/s-1741KiB/s (1774kB/s-1782kB/s), io=40.0MiB (41.9MB), run=5883-5912msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21733: Thu Dec 5 08:24:39 2024 read: IOPS=321, BW=1287KiB/s (1318kB/s)(5048KiB/3923msec) slat (usec): min=723, max=3620, avg=1486.30, stdev=414.91 clat (usec): min=4182, max=42421, avg=23209.88, stdev=4707.20 lat (usec): min=5704, max=45310, avg=24698.36, stdev=4828.49 clat percentiles (usec): | 1.00th=[14746], 5.00th=[16909], 10.00th=[17957], 20.00th=[19268], | 30.00th=[20579], 40.00th=[21365], 50.00th=[22676], 60.00th=[23725], | 70.00th=[25035], 80.00th=[26608], 90.00th=[29230], 95.00th=[32375], | 99.00th=[38011], 99.50th=[40109], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 1120, max= 1432, per=25.07%, avg=1282.29, stdev=124.25, samples=7 iops : min= 280, max= 358, avg=320.57, stdev=31.06, samples=7 write: IOPS=330, BW=1323KiB/s (1355kB/s)(5192KiB/3923msec) slat (usec): min=13, max=695, avg=55.16, stdev=45.83 clat (usec): min=13449, max=40786, avg=23821.16, stdev=4374.58 lat (usec): min=13484, max=40827, avg=23877.46, stdev=4377.76 clat percentiles (usec): | 1.00th=[15795], 5.00th=[17957], 10.00th=[19006], 20.00th=[20317], | 30.00th=[21365], 40.00th=[22152], 50.00th=[22938], 60.00th=[23987], | 70.00th=[25297], 80.00th=[27395], 90.00th=[30278], 95.00th=[32900], | 99.00th=[35390], 99.50th=[36439], 99.90th=[40109], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 1048, max= 1480, per=25.20%, avg=1341.71, stdev=169.12, samples=7 iops : min= 262, max= 370, avg=335.43, stdev=42.28, samples=7 lat (msec) : 10=0.08%, 20=21.68%, 50=78.24% cpu : usr=1.05%, sys=6.43%, ctx=2525, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21734: Thu Dec 5 08:24:39 2024 read: IOPS=317, BW=1270KiB/s (1301kB/s)(4976KiB/3917msec) slat (usec): min=793, max=4742, avg=1464.29, stdev=402.01 clat (usec): min=2781, max=43341, avg=23235.16, stdev=4667.08 lat (usec): min=4933, max=44889, avg=24702.10, stdev=4756.24 clat percentiles (usec): | 1.00th=[15008], 5.00th=[16909], 10.00th=[17695], 20.00th=[19530], | 30.00th=[20579], 40.00th=[21627], 50.00th=[22676], 60.00th=[23725], | 70.00th=[25035], 80.00th=[26608], 90.00th=[29230], 95.00th=[32113], | 99.00th=[36439], 99.50th=[38536], 99.90th=[40633], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 1016, max= 1424, per=24.88%, avg=1272.71, stdev=139.24, samples=7 iops : min= 254, max= 356, avg=318.14, stdev=34.77, samples=7 write: IOPS=335, BW=1344KiB/s (1376kB/s)(5264KiB/3917msec) slat (usec): min=15, max=1276, avg=54.50, stdev=52.86 clat (usec): min=13254, max=42220, avg=23965.25, stdev=4512.97 lat (usec): min=13298, max=42239, avg=24021.22, stdev=4518.00 clat percentiles (usec): | 1.00th=[16450], 5.00th=[17957], 10.00th=[19268], 20.00th=[20317], | 30.00th=[21365], 40.00th=[22152], 50.00th=[23200], 60.00th=[24249], | 70.00th=[25297], 80.00th=[27395], 90.00th=[30540], 95.00th=[33162], | 99.00th=[36963], 99.50th=[39584], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 1200, max= 1528, per=25.64%, avg=1365.29, stdev=138.11, samples=7 iops : min= 300, max= 382, avg=341.29, stdev=34.50, samples=7 lat (msec) : 4=0.04%, 10=0.08%, 20=20.59%, 50=79.30% cpu : usr=1.10%, sys=6.28%, ctx=2489, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21735: Thu Dec 5 08:24:39 2024 read: IOPS=317, BW=1271KiB/s (1302kB/s)(4984KiB/3921msec) slat (usec): min=710, max=3451, avg=1487.76, stdev=410.33 clat (usec): min=1229, max=44591, avg=23130.46, stdev=4698.15 lat (usec): min=2762, max=46947, avg=24620.72, stdev=4820.97 clat percentiles (usec): | 1.00th=[15008], 5.00th=[16909], 10.00th=[17957], 20.00th=[19268], | 30.00th=[20579], 40.00th=[21365], 50.00th=[22414], 60.00th=[23725], | 70.00th=[25035], 80.00th=[26608], 90.00th=[28967], 95.00th=[31851], | 99.00th=[38011], 99.50th=[39584], 99.90th=[42730], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 1136, max= 1496, per=25.20%, avg=1289.14, stdev=150.90, samples=7 iops : min= 284, max= 374, avg=322.29, stdev=37.73, samples=7 write: IOPS=335, BW=1340KiB/s (1373kB/s)(5256KiB/3921msec) slat (usec): min=13, max=599, avg=54.21, stdev=42.32 clat (usec): min=13602, max=41491, avg=23951.54, stdev=4690.17 lat (usec): min=13668, max=41593, avg=24006.94, stdev=4692.06 clat percentiles (usec): | 1.00th=[16188], 5.00th=[17957], 10.00th=[18744], 20.00th=[20317], | 30.00th=[21365], 40.00th=[22152], 50.00th=[22938], 60.00th=[24249], | 70.00th=[25560], 80.00th=[27132], 90.00th=[30540], 95.00th=[33817], | 99.00th=[38011], 99.50th=[38536], 99.90th=[40109], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 1064, max= 1512, per=25.33%, avg=1348.57, stdev=161.78, samples=7 iops : min= 266, max= 378, avg=337.14, stdev=40.45, samples=7 lat (msec) : 2=0.04%, 4=0.04%, 20=21.56%, 50=78.36% cpu : usr=0.69%, sys=6.84%, ctx=2483, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21736: Thu Dec 5 08:24:39 2024 read: IOPS=323, BW=1293KiB/s (1324kB/s)(5060KiB/3913msec) slat (usec): min=27, max=4973, avg=1490.38, stdev=449.53 clat (usec): min=2898, max=44482, avg=23192.99, stdev=4693.16 lat (usec): min=3304, max=46624, avg=24685.46, stdev=4840.71 clat percentiles (usec): | 1.00th=[14353], 5.00th=[16909], 10.00th=[17957], 20.00th=[19530], | 30.00th=[20579], 40.00th=[21627], 50.00th=[22676], 60.00th=[23987], | 70.00th=[25035], 80.00th=[26608], 90.00th=[28967], 95.00th=[31589], | 99.00th=[36439], 99.50th=[38536], 99.90th=[42206], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 1160, max= 1488, per=25.57%, avg=1307.71, stdev=123.97, samples=7 iops : min= 290, max= 372, avg=326.86, stdev=31.00, samples=7 write: IOPS=330, BW=1324KiB/s (1356kB/s)(5180KiB/3913msec) slat (usec): min=12, max=2014, avg=55.72, stdev=77.09 clat (usec): min=3452, max=40817, avg=24053.14, stdev=4715.55 lat (usec): min=3507, max=40851, avg=24110.19, stdev=4714.98 clat percentiles (usec): | 1.00th=[15795], 5.00th=[17957], 10.00th=[19006], 20.00th=[20317], | 30.00th=[21365], 40.00th=[22152], 50.00th=[23200], 60.00th=[23987], | 70.00th=[25297], 80.00th=[27657], 90.00th=[31065], 95.00th=[33817], | 99.00th=[37487], 99.50th=[39060], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 1128, max= 1464, per=24.94%, avg=1328.29, stdev=120.73, samples=7 iops : min= 282, max= 366, avg=332.00, stdev=30.20, samples=7 lat (msec) : 4=0.20%, 10=0.12%, 20=20.00%, 50=79.69% cpu : usr=0.66%, sys=6.75%, ctx=2515, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5115KiB/s (5238kB/s), 1270KiB/s-1293KiB/s (1301kB/s-1324kB/s), io=19.6MiB (20.5MB), run=3913-3923msec WRITE: bw=5326KiB/s (5453kB/s), 1323KiB/s-1344KiB/s (1355kB/s-1376kB/s), io=20.4MiB (21.4MB), run=3913-3923msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=21743: Thu Dec 5 08:24:40 2024 read: IOPS=7, BW=290MiB/s (304MB/s)(40.0MiB/138msec) slat (nsec): min=7253.1k, max=7253.1k, avg=7253078.00, stdev= 0.00 clat (nsec): min=124756k, max=124756k, avg=124755529.00, stdev= 0.00 lat (nsec): min=132032k, max=132032k, avg=132031968.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 125], 10.00th=[ 125], 20.00th=[ 125], | 30.00th=[ 125], 40.00th=[ 125], 50.00th=[ 125], 60.00th=[ 125], | 70.00th=[ 125], 80.00th=[ 125], 90.00th=[ 125], 95.00th=[ 125], | 99.00th=[ 125], 99.50th=[ 125], 99.90th=[ 125], 99.95th=[ 125], | 99.99th=[ 125] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=5.11%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=290MiB/s (304MB/s), 290MiB/s-290MiB/s (304MB/s-304MB/s), io=40.0MiB (41.9MB), run=138-138msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout