== sanity test 398c: run fio to test AIO ================= 15:46:51 (1744055211) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.142284 s, 295 MB/s osc.lustre-OST0000-osc-ffff88012b47d000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=29180: Mon Apr 7 15:46:59 2025 write: IOPS=365, BW=1461KiB/s (1496kB/s)(10.0MiB/7009msec) slat (usec): min=15, max=877, avg=86.04, stdev=47.06 clat (usec): min=6136, max=81938, avg=43603.39, stdev=6600.45 lat (usec): min=6164, max=81985, avg=43691.39, stdev=6603.97 clat percentiles (usec): | 1.00th=[21627], 5.00th=[32900], 10.00th=[36439], 20.00th=[40109], | 30.00th=[41157], 40.00th=[42730], 50.00th=[43779], 60.00th=[45351], | 70.00th=[46400], 80.00th=[47973], 90.00th=[50594], 95.00th=[53740], | 99.00th=[56886], 99.50th=[57410], 99.90th=[72877], 99.95th=[77071], | 99.99th=[82314] bw ( KiB/s): min= 1280, max= 1776, per=24.96%, avg=1453.57, stdev=147.93, samples=14 iops : min= 320, max= 444, avg=363.36, stdev=37.01, samples=14 lat (msec) : 10=0.16%, 20=0.82%, 50=88.01%, 100=11.02% cpu : usr=0.70%, sys=3.51%, ctx=1781, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=29181: Mon Apr 7 15:46:59 2025 write: IOPS=364, BW=1457KiB/s (1492kB/s)(10.0MiB/7029msec) slat (usec): min=17, max=1220, avg=88.33, stdev=50.66 clat (usec): min=20241, max=62259, avg=43726.96, stdev=5829.72 lat (usec): min=20357, max=62353, avg=43817.37, stdev=5836.26 clat percentiles (usec): | 1.00th=[28181], 5.00th=[33817], 10.00th=[36963], 20.00th=[40109], | 30.00th=[41157], 40.00th=[42730], 50.00th=[43779], 60.00th=[45351], | 70.00th=[46400], 80.00th=[47973], 90.00th=[50594], 95.00th=[53740], | 99.00th=[56361], 99.50th=[56886], 99.90th=[57410], 99.95th=[57410], | 99.99th=[62129] bw ( KiB/s): min= 1280, max= 1672, per=24.92%, avg=1450.93, stdev=126.75, samples=14 iops : min= 320, max= 418, avg=362.71, stdev=31.70, samples=14 lat (msec) : 50=89.34%, 100=10.66% cpu : usr=0.60%, sys=3.84%, ctx=2005, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=29182: Mon Apr 7 15:46:59 2025 write: IOPS=363, BW=1456KiB/s (1491kB/s)(10.0MiB/7034msec) slat (usec): min=16, max=343, avg=61.56, stdev=37.28 clat (usec): min=4893, max=70030, avg=43691.76, stdev=5941.88 lat (usec): min=4993, max=70080, avg=43754.87, stdev=5945.29 clat percentiles (usec): | 1.00th=[26084], 5.00th=[33817], 10.00th=[36963], 20.00th=[40109], | 30.00th=[41157], 40.00th=[42206], 50.00th=[43779], 60.00th=[45351], | 70.00th=[46400], 80.00th=[47449], 90.00th=[51119], 95.00th=[53216], | 99.00th=[56361], 99.50th=[56886], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 1280, max= 1784, per=24.85%, avg=1447.21, stdev=150.14, samples=14 iops : min= 320, max= 446, avg=361.79, stdev=37.54, samples=14 lat (msec) : 10=0.04%, 50=88.01%, 100=11.95% cpu : usr=0.54%, sys=2.64%, ctx=650, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=29183: Mon Apr 7 15:46:59 2025 write: IOPS=364, BW=1458KiB/s (1493kB/s)(10.0MiB/7025msec) slat (usec): min=14, max=362, avg=89.49, stdev=46.49 clat (usec): min=4381, max=71035, avg=43651.63, stdev=6288.96 lat (usec): min=4486, max=71120, avg=43743.21, stdev=6294.66 clat percentiles (usec): | 1.00th=[28181], 5.00th=[33424], 10.00th=[36963], 20.00th=[40109], | 30.00th=[41157], 40.00th=[42730], 50.00th=[43779], 60.00th=[45351], | 70.00th=[46400], 80.00th=[47973], 90.00th=[50594], 95.00th=[53740], | 99.00th=[56886], 99.50th=[57410], 99.90th=[65799], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 1280, max= 1672, per=24.97%, avg=1454.29, stdev=127.54, samples=14 iops : min= 320, max= 418, avg=363.57, stdev=31.89, samples=14 lat (msec) : 10=0.08%, 20=0.74%, 50=88.36%, 100=10.82% cpu : usr=0.50%, sys=3.87%, ctx=2056, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=5823KiB/s (5963kB/s), 1456KiB/s-1461KiB/s (1491kB/s-1496kB/s), io=40.0MiB (41.9MB), run=7009-7034msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29198: Mon Apr 7 15:47:04 2025 read: IOPS=340, BW=1361KiB/s (1394kB/s)(5048KiB/3708msec) slat (usec): min=734, max=3592, avg=1547.13, stdev=384.84 clat (usec): min=6425, max=40813, avg=21771.45, stdev=4020.37 lat (usec): min=8241, max=43259, avg=23321.66, stdev=4171.14 clat percentiles (usec): | 1.00th=[14484], 5.00th=[16319], 10.00th=[17171], 20.00th=[18482], | 30.00th=[19530], 40.00th=[20579], 50.00th=[21365], 60.00th=[22414], | 70.00th=[23200], 80.00th=[24511], 90.00th=[26346], 95.00th=[29230], | 99.00th=[35390], 99.50th=[37487], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 944, max= 1480, per=25.02%, avg=1354.29, stdev=185.31, samples=7 iops : min= 236, max= 370, avg=338.57, stdev=46.33, samples=7 write: IOPS=350, BW=1400KiB/s (1434kB/s)(5192KiB/3708msec) slat (usec): min=14, max=1500, avg=66.80, stdev=58.70 clat (usec): min=12294, max=40348, avg=22424.83, stdev=3789.68 lat (usec): min=12349, max=40409, avg=22492.92, stdev=3794.03 clat percentiles (usec): | 1.00th=[16319], 5.00th=[17957], 10.00th=[18744], 20.00th=[19530], | 30.00th=[20317], 40.00th=[21103], 50.00th=[21890], 60.00th=[22414], | 70.00th=[23462], 80.00th=[24249], 90.00th=[26346], 95.00th=[30016], | 99.00th=[36963], 99.50th=[38536], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 920, max= 1664, per=24.77%, avg=1395.43, stdev=245.53, samples=7 iops : min= 230, max= 416, avg=348.86, stdev=61.38, samples=7 lat (msec) : 10=0.04%, 20=29.34%, 50=70.62% cpu : usr=1.32%, sys=8.77%, ctx=2227, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29199: Mon Apr 7 15:47:04 2025 read: IOPS=337, BW=1350KiB/s (1383kB/s)(4976KiB/3685msec) slat (usec): min=767, max=3402, avg=1536.44, stdev=373.04 clat (usec): min=5214, max=45062, avg=21715.87, stdev=4369.98 lat (usec): min=6742, max=46932, avg=23255.11, stdev=4498.53 clat percentiles (usec): | 1.00th=[14353], 5.00th=[16057], 10.00th=[17171], 20.00th=[18482], | 30.00th=[19530], 40.00th=[20317], 50.00th=[21103], 60.00th=[22152], | 70.00th=[23200], 80.00th=[24511], 90.00th=[26346], 95.00th=[29754], | 99.00th=[36963], 99.50th=[38536], 99.90th=[43779], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 912, max= 1520, per=24.79%, avg=1341.71, stdev=204.78, samples=7 iops : min= 228, max= 380, avg=335.43, stdev=51.20, samples=7 write: IOPS=357, BW=1428KiB/s (1463kB/s)(5264KiB/3685msec) slat (usec): min=13, max=1525, avg=66.99, stdev=68.98 clat (usec): min=13954, max=40842, avg=22464.78, stdev=3859.54 lat (usec): min=14012, max=40897, avg=22533.11, stdev=3861.09 clat percentiles (usec): | 1.00th=[16057], 5.00th=[17957], 10.00th=[18482], 20.00th=[19792], | 30.00th=[20579], 40.00th=[21103], 50.00th=[21627], 60.00th=[22414], | 70.00th=[23200], 80.00th=[24249], 90.00th=[26608], 95.00th=[29754], | 99.00th=[37487], 99.50th=[38536], 99.90th=[40109], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 1056, max= 1592, per=25.44%, avg=1433.14, stdev=184.89, samples=7 iops : min= 264, max= 398, avg=358.29, stdev=46.22, samples=7 lat (msec) : 10=0.12%, 20=30.08%, 50=69.80% cpu : usr=0.71%, sys=9.20%, ctx=2201, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29200: Mon Apr 7 15:47:04 2025 read: IOPS=336, BW=1344KiB/s (1377kB/s)(4984KiB/3707msec) slat (usec): min=782, max=3040, avg=1533.65, stdev=353.75 clat (usec): min=4218, max=38681, avg=21684.86, stdev=4041.49 lat (usec): min=6544, max=41579, avg=23221.34, stdev=4178.91 clat percentiles (usec): | 1.00th=[14746], 5.00th=[16319], 10.00th=[17171], 20.00th=[18482], | 30.00th=[19530], 40.00th=[20317], 50.00th=[21103], 60.00th=[22152], | 70.00th=[22938], 80.00th=[24511], 90.00th=[26346], 95.00th=[29230], | 99.00th=[34341], 99.50th=[35914], 99.90th=[38011], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 912, max= 1552, per=24.94%, avg=1349.71, stdev=223.75, samples=7 iops : min= 228, max= 388, avg=337.43, stdev=55.94, samples=7 write: IOPS=354, BW=1418KiB/s (1452kB/s)(5256KiB/3707msec) slat (usec): min=13, max=316, avg=67.19, stdev=34.47 clat (usec): min=14134, max=39745, avg=22648.44, stdev=4122.89 lat (usec): min=14267, max=39830, avg=22717.16, stdev=4123.94 clat percentiles (usec): | 1.00th=[16188], 5.00th=[17957], 10.00th=[18744], 20.00th=[19792], | 30.00th=[20317], 40.00th=[21365], 50.00th=[21890], 60.00th=[22676], | 70.00th=[23462], 80.00th=[24773], 90.00th=[27132], 95.00th=[32113], | 99.00th=[38011], 99.50th=[39060], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 960, max= 1584, per=24.95%, avg=1405.71, stdev=213.75, samples=7 iops : min= 240, max= 396, avg=351.43, stdev=53.44, samples=7 lat (msec) : 10=0.04%, 20=29.96%, 50=70.00% cpu : usr=1.38%, sys=8.88%, ctx=2173, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29201: Mon Apr 7 15:47:04 2025 read: IOPS=342, BW=1371KiB/s (1403kB/s)(5060KiB/3692msec) slat (usec): min=61, max=3196, avg=1536.38, stdev=380.66 clat (usec): min=2428, max=43931, avg=21835.16, stdev=4192.54 lat (usec): min=2569, max=45891, avg=23374.29, stdev=4348.57 clat percentiles (usec): | 1.00th=[14353], 5.00th=[16319], 10.00th=[17171], 20.00th=[18482], | 30.00th=[19792], 40.00th=[20579], 50.00th=[21365], 60.00th=[22414], | 70.00th=[23200], 80.00th=[24511], 90.00th=[26346], 95.00th=[30016], | 99.00th=[35390], 99.50th=[38536], 99.90th=[42206], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 1032, max= 1504, per=25.21%, avg=1364.57, stdev=158.58, samples=7 iops : min= 258, max= 376, avg=341.14, stdev=39.65, samples=7 write: IOPS=350, BW=1403KiB/s (1437kB/s)(5180KiB/3692msec) slat (usec): min=13, max=1520, avg=66.40, stdev=56.29 clat (usec): min=8669, max=43523, avg=22595.24, stdev=4087.47 lat (usec): min=8706, max=43572, avg=22663.04, stdev=4086.50 clat percentiles (usec): | 1.00th=[16057], 5.00th=[17695], 10.00th=[18482], 20.00th=[19792], | 30.00th=[20579], 40.00th=[21103], 50.00th=[21890], 60.00th=[22676], | 70.00th=[23462], 80.00th=[24511], 90.00th=[26608], 95.00th=[31589], | 99.00th=[37487], 99.50th=[38536], 99.90th=[42730], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 1024, max= 1608, per=24.95%, avg=1405.71, stdev=197.62, samples=7 iops : min= 256, max= 402, avg=351.43, stdev=49.41, samples=7 lat (msec) : 4=0.12%, 10=0.12%, 20=27.38%, 50=72.38% cpu : usr=1.46%, sys=8.40%, ctx=2234, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5412KiB/s (5542kB/s), 1344KiB/s-1371KiB/s (1377kB/s-1403kB/s), io=19.6MiB (20.5MB), run=3685-3708msec WRITE: bw=5634KiB/s (5770kB/s), 1400KiB/s-1428KiB/s (1434kB/s-1463kB/s), io=20.4MiB (21.4MB), run=3685-3708msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=29208: Mon Apr 7 15:47:05 2025 read: IOPS=5, BW=230MiB/s (241MB/s)(40.0MiB/174msec) slat (nsec): min=5267.6k, max=5267.6k, avg=5267585.00, stdev= 0.00 clat (nsec): min=161344k, max=161344k, avg=161343707.00, stdev= 0.00 lat (nsec): min=166620k, max=166620k, avg=166620329.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 161], | 70.00th=[ 161], 80.00th=[ 161], 90.00th=[ 161], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=3.47%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=230MiB/s (241MB/s), 230MiB/s-230MiB/s (241MB/s-241MB/s), io=40.0MiB (41.9MB), run=174-174msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout