== sanity test 398c: run fio to test AIO ================= 10:24:25 (1743517465) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0764511 s, 549 MB/s osc.lustre-OST0000-osc-ffff8800b5c76000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=30564: Tue Apr 1 10:24:33 2025 write: IOPS=425, BW=1703KiB/s (1744kB/s)(10.0MiB/6014msec) slat (usec): min=12, max=814, avg=71.16, stdev=45.68 clat (usec): min=9676, max=72977, avg=37402.93, stdev=7916.30 lat (usec): min=9763, max=73038, avg=37476.02, stdev=7927.39 clat percentiles (usec): | 1.00th=[20055], 5.00th=[22414], 10.00th=[24511], 20.00th=[30278], | 30.00th=[34341], 40.00th=[37487], 50.00th=[39584], 60.00th=[41157], | 70.00th=[42206], 80.00th=[43779], 90.00th=[45876], 95.00th=[46924], | 99.00th=[50594], 99.50th=[52167], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 1437, max= 2288, per=24.93%, avg=1696.42, stdev=274.08, samples=12 iops : min= 359, max= 572, avg=424.08, stdev=68.54, samples=12 lat (msec) : 10=0.04%, 20=0.94%, 50=97.89%, 100=1.13% cpu : usr=0.65%, sys=3.56%, ctx=1371, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=30565: Tue Apr 1 10:24:33 2025 write: IOPS=425, BW=1703KiB/s (1744kB/s)(10.0MiB/6014msec) slat (usec): min=12, max=527, avg=64.25, stdev=40.46 clat (usec): min=15409, max=77859, avg=37388.30, stdev=7955.46 lat (usec): min=15469, max=77915, avg=37454.19, stdev=7963.21 clat percentiles (usec): | 1.00th=[20317], 5.00th=[22414], 10.00th=[24511], 20.00th=[29754], | 30.00th=[34341], 40.00th=[37487], 50.00th=[39584], 60.00th=[41157], | 70.00th=[42206], 80.00th=[43779], 90.00th=[45876], 95.00th=[46924], | 99.00th=[50594], 99.50th=[52167], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 1413, max= 2224, per=24.93%, avg=1696.42, stdev=262.63, samples=12 iops : min= 353, max= 556, avg=424.08, stdev=65.68, samples=12 lat (msec) : 20=0.90%, 50=97.77%, 100=1.33% cpu : usr=0.63%, sys=3.09%, ctx=1250, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=30566: Tue Apr 1 10:24:33 2025 write: IOPS=427, BW=1711KiB/s (1752kB/s)(10.0MiB/5986msec) slat (usec): min=12, max=682, avg=70.00, stdev=43.70 clat (usec): min=3772, max=86203, avg=37229.65, stdev=8462.28 lat (usec): min=3854, max=86331, avg=37301.39, stdev=8471.48 clat percentiles (usec): | 1.00th=[16450], 5.00th=[22152], 10.00th=[24511], 20.00th=[29230], | 30.00th=[33817], 40.00th=[37487], 50.00th=[39060], 60.00th=[41157], | 70.00th=[42206], 80.00th=[43779], 90.00th=[45876], 95.00th=[46924], | 99.00th=[50594], 99.50th=[53740], 99.90th=[82314], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 1445, max= 2296, per=25.26%, avg=1719.00, stdev=279.09, samples=11 iops : min= 361, max= 574, avg=429.73, stdev=69.80, samples=11 lat (msec) : 4=0.04%, 10=0.39%, 20=0.86%, 50=96.99%, 100=1.72% cpu : usr=0.58%, sys=3.64%, ctx=1473, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=30567: Tue Apr 1 10:24:33 2025 write: IOPS=425, BW=1701KiB/s (1742kB/s)(10.0MiB/6019msec) slat (usec): min=13, max=494, avg=59.24, stdev=40.03 clat (usec): min=15855, max=67390, avg=37410.72, stdev=7729.40 lat (usec): min=15914, max=67469, avg=37471.50, stdev=7736.85 clat percentiles (usec): | 1.00th=[20317], 5.00th=[22152], 10.00th=[24511], 20.00th=[30016], | 30.00th=[34341], 40.00th=[37487], 50.00th=[39584], 60.00th=[41681], | 70.00th=[42730], 80.00th=[43779], 90.00th=[45876], 95.00th=[46400], | 99.00th=[49546], 99.50th=[51119], 99.90th=[53740], 99.95th=[53740], | 99.99th=[67634] bw ( KiB/s): min= 1424, max= 2296, per=24.93%, avg=1696.33, stdev=266.06, samples=12 iops : min= 356, max= 574, avg=424.08, stdev=66.51, samples=12 lat (msec) : 20=0.51%, 50=98.52%, 100=0.98% cpu : usr=0.50%, sys=2.92%, ctx=981, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=6805KiB/s (6968kB/s), 1701KiB/s-1711KiB/s (1742kB/s-1752kB/s), io=40.0MiB (41.9MB), run=5986-6019msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30581: Tue Apr 1 10:24:37 2025 read: IOPS=460, BW=1842KiB/s (1886kB/s)(5048KiB/2741msec) slat (usec): min=651, max=2238, avg=1088.26, stdev=209.59 clat (usec): min=10115, max=25983, avg=16180.61, stdev=2163.08 lat (usec): min=10973, max=26872, avg=17270.71, stdev=2160.01 clat percentiles (usec): | 1.00th=[11338], 5.00th=[12780], 10.00th=[13304], 20.00th=[14353], | 30.00th=[15008], 40.00th=[15533], 50.00th=[16188], 60.00th=[16909], | 70.00th=[17433], 80.00th=[17957], 90.00th=[18744], 95.00th=[19530], | 99.00th=[21365], 99.50th=[21627], 99.90th=[25297], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 1656, max= 1920, per=24.98%, avg=1828.80, stdev=108.01, samples=5 iops : min= 414, max= 480, avg=457.20, stdev=27.00, samples=5 write: IOPS=473, BW=1894KiB/s (1940kB/s)(5192KiB/2741msec) slat (usec): min=12, max=1023, avg=44.46, stdev=35.46 clat (usec): min=11972, max=28484, avg=16672.91, stdev=1674.23 lat (usec): min=12008, max=28515, avg=16718.50, stdev=1673.16 clat percentiles (usec): | 1.00th=[13173], 5.00th=[14222], 10.00th=[14484], 20.00th=[15270], | 30.00th=[15795], 40.00th=[16188], 50.00th=[16712], 60.00th=[17171], | 70.00th=[17433], 80.00th=[17957], 90.00th=[18744], 95.00th=[19530], | 99.00th=[21103], 99.50th=[21627], 99.90th=[25822], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 1784, max= 1952, per=24.96%, avg=1902.40, stdev=70.97, samples=5 iops : min= 446, max= 488, avg=475.60, stdev=17.74, samples=5 lat (msec) : 20=97.19%, 50=2.81% cpu : usr=0.99%, sys=7.74%, ctx=2324, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30582: Tue Apr 1 10:24:37 2025 read: IOPS=453, BW=1815KiB/s (1859kB/s)(4976KiB/2741msec) slat (usec): min=632, max=1939, avg=1089.03, stdev=206.98 clat (usec): min=2382, max=23601, avg=16166.38, stdev=2190.93 lat (usec): min=3382, max=25085, avg=17257.20, stdev=2196.84 clat percentiles (usec): | 1.00th=[11076], 5.00th=[12780], 10.00th=[13566], 20.00th=[14353], | 30.00th=[15008], 40.00th=[15664], 50.00th=[16188], 60.00th=[16712], | 70.00th=[17171], 80.00th=[17957], 90.00th=[18744], 95.00th=[19792], | 99.00th=[21103], 99.50th=[21890], 99.90th=[22676], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 1592, max= 1912, per=24.54%, avg=1796.80, stdev=132.98, samples=5 iops : min= 398, max= 478, avg=449.20, stdev=33.24, samples=5 write: IOPS=480, BW=1920KiB/s (1967kB/s)(5264KiB/2741msec) slat (usec): min=13, max=1066, avg=43.72, stdev=34.20 clat (usec): min=5123, max=22988, avg=16720.60, stdev=1775.58 lat (usec): min=5155, max=23006, avg=16765.32, stdev=1773.80 clat percentiles (usec): | 1.00th=[12911], 5.00th=[13960], 10.00th=[14484], 20.00th=[15270], | 30.00th=[15795], 40.00th=[16319], 50.00th=[16712], 60.00th=[17171], | 70.00th=[17695], 80.00th=[18220], 90.00th=[19006], 95.00th=[19792], | 99.00th=[21627], 99.50th=[21890], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 1800, max= 2024, per=25.19%, avg=1920.00, stdev=82.37, samples=5 iops : min= 450, max= 506, avg=480.00, stdev=20.59, samples=5 lat (msec) : 4=0.04%, 10=0.20%, 20=96.29%, 50=3.48% cpu : usr=1.20%, sys=7.52%, ctx=2239, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30583: Tue Apr 1 10:24:37 2025 read: IOPS=459, BW=1837KiB/s (1881kB/s)(4984KiB/2713msec) slat (usec): min=13, max=1995, avg=1094.61, stdev=227.42 clat (usec): min=1262, max=24899, avg=16081.99, stdev=2351.73 lat (usec): min=1282, max=25986, avg=17178.34, stdev=2379.67 clat percentiles (usec): | 1.00th=[10683], 5.00th=[12518], 10.00th=[13173], 20.00th=[14222], | 30.00th=[14877], 40.00th=[15664], 50.00th=[16188], 60.00th=[16712], | 70.00th=[17433], 80.00th=[17957], 90.00th=[18744], 95.00th=[19792], | 99.00th=[21103], 99.50th=[21890], 99.90th=[23987], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 1808, max= 1900, per=25.23%, avg=1847.20, stdev=37.35, samples=5 iops : min= 452, max= 475, avg=461.80, stdev= 9.34, samples=5 write: IOPS=484, BW=1937KiB/s (1984kB/s)(5256KiB/2713msec) slat (usec): min=12, max=1159, avg=44.15, stdev=39.62 clat (usec): min=6024, max=28367, avg=16610.89, stdev=1863.24 lat (usec): min=6063, max=28401, avg=16656.13, stdev=1863.52 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13829], 10.00th=[14484], 20.00th=[15270], | 30.00th=[15795], 40.00th=[16188], 50.00th=[16581], 60.00th=[16909], | 70.00th=[17433], 80.00th=[17957], 90.00th=[18744], 95.00th=[19530], | 99.00th=[20841], 99.50th=[21890], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 1800, max= 2024, per=25.33%, avg=1930.40, stdev=110.76, samples=5 iops : min= 450, max= 506, avg=482.60, stdev=27.69, samples=5 lat (msec) : 2=0.12%, 4=0.04%, 10=0.43%, 20=96.13%, 50=3.28% cpu : usr=0.92%, sys=7.78%, ctx=2257, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30584: Tue Apr 1 10:24:37 2025 read: IOPS=462, BW=1851KiB/s (1895kB/s)(5060KiB/2734msec) slat (usec): min=641, max=2206, avg=1095.61, stdev=209.75 clat (usec): min=9729, max=27527, avg=16188.11, stdev=2165.06 lat (usec): min=10586, max=28584, avg=17285.41, stdev=2169.97 clat percentiles (usec): | 1.00th=[11338], 5.00th=[12780], 10.00th=[13435], 20.00th=[14222], | 30.00th=[15008], 40.00th=[15664], 50.00th=[16188], 60.00th=[16712], | 70.00th=[17433], 80.00th=[17957], 90.00th=[18744], 95.00th=[19530], | 99.00th=[21365], 99.50th=[21627], 99.90th=[23725], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 1736, max= 1968, per=25.33%, avg=1854.40, stdev=97.06, samples=5 iops : min= 434, max= 492, avg=463.60, stdev=24.27, samples=5 write: IOPS=473, BW=1895KiB/s (1940kB/s)(5180KiB/2734msec) slat (usec): min=13, max=1012, avg=44.72, stdev=40.61 clat (usec): min=11451, max=28740, avg=16662.06, stdev=1782.24 lat (usec): min=11560, max=28808, avg=16708.01, stdev=1781.81 clat percentiles (usec): | 1.00th=[12911], 5.00th=[13829], 10.00th=[14484], 20.00th=[15139], | 30.00th=[15795], 40.00th=[16188], 50.00th=[16712], 60.00th=[17171], | 70.00th=[17433], 80.00th=[17957], 90.00th=[18744], 95.00th=[19530], | 99.00th=[20841], 99.50th=[21103], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 1776, max= 1936, per=24.79%, avg=1889.60, stdev=66.07, samples=5 iops : min= 444, max= 484, avg=472.40, stdev=16.52, samples=5 lat (msec) : 10=0.08%, 20=96.68%, 50=3.24% cpu : usr=1.06%, sys=7.68%, ctx=2385, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7321KiB/s (7497kB/s), 1815KiB/s-1851KiB/s (1859kB/s-1895kB/s), io=19.6MiB (20.5MB), run=2713-2741msec WRITE: bw=7622KiB/s (7805kB/s), 1894KiB/s-1937KiB/s (1940kB/s-1984kB/s), io=20.4MiB (21.4MB), run=2713-2741msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=30590: Tue Apr 1 10:24:38 2025 read: IOPS=6, BW=278MiB/s (291MB/s)(40.0MiB/144msec) slat (nsec): min=5094.0k, max=5094.0k, avg=5094014.00, stdev= 0.00 clat (nsec): min=133465k, max=133465k, avg=133465392.00, stdev= 0.00 lat (nsec): min=138568k, max=138568k, avg=138568132.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 134], 10.00th=[ 134], 20.00th=[ 134], | 30.00th=[ 134], 40.00th=[ 134], 50.00th=[ 134], 60.00th=[ 134], | 70.00th=[ 134], 80.00th=[ 134], 90.00th=[ 134], 95.00th=[ 134], | 99.00th=[ 134], 99.50th=[ 134], 99.90th=[ 134], 99.95th=[ 134], | 99.99th=[ 134] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=3.50%, ctx=3, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=278MiB/s (291MB/s), 278MiB/s-278MiB/s (291MB/s-291MB/s), io=40.0MiB (41.9MB), run=144-144msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout