== sanity test 398c: run fio to test AIO ================= 14:36:22 (1744137382) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0935169 s, 449 MB/s osc.lustre-OST0000-osc-ffff88012a45c000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=30410: Tue Apr 8 14:36:27 2025 write: IOPS=652, BW=2608KiB/s (2671kB/s)(10.0MiB/3926msec) slat (usec): min=13, max=329, avg=53.45, stdev=30.90 clat (usec): min=4646, max=39588, avg=24377.17, stdev=4578.59 lat (usec): min=4692, max=39636, avg=24432.04, stdev=4583.02 clat percentiles (usec): | 1.00th=[13304], 5.00th=[16712], 10.00th=[19006], 20.00th=[21103], | 30.00th=[22414], 40.00th=[23200], 50.00th=[23987], 60.00th=[25035], | 70.00th=[26346], 80.00th=[27919], 90.00th=[30802], 95.00th=[32375], | 99.00th=[34866], 99.50th=[35914], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 2304, max= 2832, per=24.50%, avg=2554.57, stdev=191.08, samples=7 iops : min= 576, max= 708, avg=638.57, stdev=47.72, samples=7 lat (msec) : 10=0.08%, 20=12.97%, 50=86.95% cpu : usr=0.66%, sys=4.10%, ctx=1597, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=30411: Tue Apr 8 14:36:27 2025 write: IOPS=653, BW=2616KiB/s (2678kB/s)(10.0MiB/3915msec) slat (usec): min=13, max=686, avg=58.55, stdev=39.23 clat (usec): min=2704, max=47015, avg=24308.75, stdev=4813.86 lat (usec): min=2757, max=47045, avg=24368.59, stdev=4816.54 clat percentiles (usec): | 1.00th=[11469], 5.00th=[16712], 10.00th=[19006], 20.00th=[20841], | 30.00th=[22152], 40.00th=[23200], 50.00th=[23987], 60.00th=[25035], | 70.00th=[26084], 80.00th=[28181], 90.00th=[30802], 95.00th=[32113], | 99.00th=[34866], 99.50th=[35390], 99.90th=[42730], 99.95th=[44827], | 99.99th=[46924] bw ( KiB/s): min= 2312, max= 2936, per=24.66%, avg=2571.43, stdev=210.69, samples=7 iops : min= 578, max= 734, avg=642.86, stdev=52.67, samples=7 lat (msec) : 4=0.08%, 10=0.55%, 20=12.66%, 50=86.72% cpu : usr=0.56%, sys=4.39%, ctx=1731, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=30412: Tue Apr 8 14:36:27 2025 write: IOPS=652, BW=2610KiB/s (2672kB/s)(10.0MiB/3924msec) slat (usec): min=12, max=345, avg=56.52, stdev=32.70 clat (usec): min=8921, max=40091, avg=24337.16, stdev=4622.64 lat (usec): min=8974, max=40144, avg=24395.06, stdev=4628.33 clat percentiles (usec): | 1.00th=[12387], 5.00th=[16581], 10.00th=[19006], 20.00th=[21103], | 30.00th=[22414], 40.00th=[23200], 50.00th=[23987], 60.00th=[25035], | 70.00th=[26084], 80.00th=[28181], 90.00th=[30802], 95.00th=[32113], | 99.00th=[34866], 99.50th=[34866], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 2312, max= 2864, per=24.53%, avg=2557.71, stdev=191.37, samples=7 iops : min= 578, max= 716, avg=639.43, stdev=47.84, samples=7 lat (msec) : 10=0.51%, 20=12.46%, 50=87.03% cpu : usr=0.76%, sys=4.00%, ctx=1676, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=30413: Tue Apr 8 14:36:27 2025 write: IOPS=651, BW=2607KiB/s (2669kB/s)(10.0MiB/3928msec) slat (usec): min=12, max=433, avg=51.83, stdev=32.70 clat (usec): min=3455, max=35891, avg=24349.61, stdev=4528.85 lat (usec): min=3513, max=36126, avg=24402.71, stdev=4532.38 clat percentiles (usec): | 1.00th=[13042], 5.00th=[16909], 10.00th=[19268], 20.00th=[21103], | 30.00th=[22152], 40.00th=[23200], 50.00th=[24249], 60.00th=[24773], | 70.00th=[26084], 80.00th=[27919], 90.00th=[30802], 95.00th=[32113], | 99.00th=[34341], 99.50th=[34866], 99.90th=[35390], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 2320, max= 2816, per=24.45%, avg=2549.71, stdev=171.19, samples=7 iops : min= 580, max= 704, avg=637.43, stdev=42.80, samples=7 lat (msec) : 4=0.04%, 10=0.04%, 20=12.54%, 50=87.38% cpu : usr=0.79%, sys=3.64%, ctx=1263, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=10.2MiB/s (10.7MB/s), 2607KiB/s-2616KiB/s (2669kB/s-2678kB/s), io=40.0MiB (41.9MB), run=3915-3928msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30426: Tue Apr 8 14:36:31 2025 read: IOPS=417, BW=1672KiB/s (1712kB/s)(5048KiB/3020msec) slat (usec): min=682, max=2572, avg=1203.97, stdev=245.84 clat (usec): min=8775, max=29301, avg=17858.93, stdev=2429.80 lat (usec): min=10172, max=30334, avg=19065.00, stdev=2460.01 clat percentiles (usec): | 1.00th=[12649], 5.00th=[14091], 10.00th=[14877], 20.00th=[15795], | 30.00th=[16581], 40.00th=[17171], 50.00th=[17957], 60.00th=[18482], | 70.00th=[19006], 80.00th=[19792], 90.00th=[20841], 95.00th=[21627], | 99.00th=[23987], 99.50th=[25035], 99.90th=[28181], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 1616, max= 1768, per=25.13%, avg=1668.00, stdev=60.03, samples=6 iops : min= 404, max= 442, avg=417.00, stdev=15.01, samples=6 write: IOPS=429, BW=1719KiB/s (1760kB/s)(5192KiB/3020msec) slat (usec): min=13, max=1033, avg=50.47, stdev=37.90 clat (usec): min=6494, max=28069, avg=18403.27, stdev=2109.22 lat (usec): min=6555, max=28123, avg=18455.04, stdev=2108.61 clat percentiles (usec): | 1.00th=[13960], 5.00th=[15270], 10.00th=[15926], 20.00th=[16712], | 30.00th=[17171], 40.00th=[17957], 50.00th=[18220], 60.00th=[18744], | 70.00th=[19268], 80.00th=[20055], 90.00th=[21103], 95.00th=[21890], | 99.00th=[23987], 99.50th=[24249], 99.90th=[26870], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 1656, max= 1776, per=24.89%, avg=1720.00, stdev=52.82, samples=6 iops : min= 414, max= 444, avg=430.00, stdev=13.21, samples=6 lat (msec) : 10=0.08%, 20=81.45%, 50=18.48% cpu : usr=0.76%, sys=7.78%, ctx=2365, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30427: Tue Apr 8 14:36:31 2025 read: IOPS=411, BW=1646KiB/s (1686kB/s)(4976KiB/3023msec) slat (usec): min=683, max=2453, avg=1217.16, stdev=247.27 clat (usec): min=3332, max=31036, avg=17898.56, stdev=2575.76 lat (usec): min=4499, max=32589, avg=19117.69, stdev=2601.15 clat percentiles (usec): | 1.00th=[12256], 5.00th=[13829], 10.00th=[14877], 20.00th=[15926], | 30.00th=[16450], 40.00th=[17171], 50.00th=[17695], 60.00th=[18482], | 70.00th=[19006], 80.00th=[20055], 90.00th=[21365], 95.00th=[22152], | 99.00th=[24249], 99.50th=[25560], 99.90th=[27395], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 1552, max= 1696, per=24.69%, avg=1638.67, stdev=54.83, samples=6 iops : min= 388, max= 424, avg=409.67, stdev=13.71, samples=6 write: IOPS=435, BW=1741KiB/s (1783kB/s)(5264KiB/3023msec) slat (usec): min=13, max=292, avg=49.64, stdev=26.86 clat (usec): min=3899, max=27286, avg=18385.46, stdev=2193.11 lat (usec): min=3946, max=27322, avg=18436.38, stdev=2193.26 clat percentiles (usec): | 1.00th=[13698], 5.00th=[14877], 10.00th=[15664], 20.00th=[16581], | 30.00th=[17171], 40.00th=[17957], 50.00th=[18482], 60.00th=[19006], | 70.00th=[19530], 80.00th=[20055], 90.00th=[21103], 95.00th=[22152], | 99.00th=[23725], 99.50th=[24249], 99.90th=[25297], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 1600, max= 1848, per=25.22%, avg=1742.67, stdev=81.18, samples=6 iops : min= 400, max= 462, avg=435.67, stdev=20.29, samples=6 lat (msec) : 4=0.08%, 20=79.49%, 50=20.43% cpu : usr=0.96%, sys=7.51%, ctx=2390, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30428: Tue Apr 8 14:36:31 2025 read: IOPS=412, BW=1649KiB/s (1688kB/s)(4984KiB/3023msec) slat (usec): min=717, max=2592, avg=1215.43, stdev=254.06 clat (usec): min=3762, max=32274, avg=17919.36, stdev=2549.61 lat (usec): min=5120, max=33206, avg=19136.89, stdev=2566.34 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13960], 10.00th=[14746], 20.00th=[15795], | 30.00th=[16581], 40.00th=[17171], 50.00th=[17957], 60.00th=[18744], | 70.00th=[19268], 80.00th=[19792], 90.00th=[21103], 95.00th=[21890], | 99.00th=[23987], 99.50th=[24773], 99.90th=[30802], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 1440, max= 1856, per=24.85%, avg=1649.33, stdev=146.68, samples=6 iops : min= 360, max= 464, avg=412.33, stdev=36.67, samples=6 write: IOPS=434, BW=1739KiB/s (1780kB/s)(5256KiB/3023msec) slat (usec): min=13, max=364, avg=50.71, stdev=27.67 clat (usec): min=12047, max=29849, avg=18307.97, stdev=2047.43 lat (usec): min=12084, max=29889, avg=18360.19, stdev=2047.17 clat percentiles (usec): | 1.00th=[13960], 5.00th=[15008], 10.00th=[15795], 20.00th=[16450], | 30.00th=[17171], 40.00th=[17695], 50.00th=[18220], 60.00th=[18744], | 70.00th=[19268], 80.00th=[20055], 90.00th=[20841], 95.00th=[21627], | 99.00th=[23200], 99.50th=[24249], 99.90th=[28705], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 1640, max= 1872, per=25.08%, avg=1733.33, stdev=91.87, samples=6 iops : min= 410, max= 468, avg=433.33, stdev=22.97, samples=6 lat (msec) : 4=0.04%, 10=0.04%, 20=80.47%, 50=19.45% cpu : usr=1.13%, sys=7.31%, ctx=2378, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30430: Tue Apr 8 14:36:31 2025 read: IOPS=421, BW=1685KiB/s (1725kB/s)(5060KiB/3003msec) slat (usec): min=17, max=2437, avg=1198.81, stdev=257.37 clat (usec): min=1267, max=30534, avg=17894.16, stdev=2772.53 lat (usec): min=1294, max=31843, avg=19094.90, stdev=2816.12 clat percentiles (usec): | 1.00th=[11076], 5.00th=[13960], 10.00th=[14746], 20.00th=[15795], | 30.00th=[16581], 40.00th=[17171], 50.00th=[17957], 60.00th=[18482], | 70.00th=[19268], 80.00th=[19792], 90.00th=[21103], 95.00th=[22152], | 99.00th=[25035], 99.50th=[26346], 99.90th=[29230], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 1584, max= 1736, per=25.25%, avg=1676.00, stdev=54.67, samples=6 iops : min= 396, max= 434, avg=419.00, stdev=13.67, samples=6 write: IOPS=431, BW=1725KiB/s (1766kB/s)(5180KiB/3003msec) slat (usec): min=13, max=376, avg=50.32, stdev=29.74 clat (usec): min=2810, max=35101, avg=18296.62, stdev=2415.30 lat (usec): min=2856, max=35147, avg=18348.13, stdev=2412.75 clat percentiles (usec): | 1.00th=[12256], 5.00th=[15008], 10.00th=[15664], 20.00th=[16581], | 30.00th=[17171], 40.00th=[17695], 50.00th=[18220], 60.00th=[18744], | 70.00th=[19268], 80.00th=[20055], 90.00th=[21103], 95.00th=[22152], | 99.00th=[23987], 99.50th=[24511], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 1640, max= 1776, per=24.85%, avg=1717.33, stdev=52.74, samples=6 iops : min= 410, max= 444, avg=429.33, stdev=13.19, samples=6 lat (msec) : 2=0.12%, 4=0.16%, 10=0.23%, 20=79.49%, 50=20.00% cpu : usr=0.87%, sys=7.56%, ctx=2386, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6638KiB/s (6798kB/s), 1646KiB/s-1685KiB/s (1686kB/s-1725kB/s), io=19.6MiB (20.5MB), run=3003-3023msec WRITE: bw=6911KiB/s (7077kB/s), 1719KiB/s-1741KiB/s (1760kB/s-1783kB/s), io=20.4MiB (21.4MB), run=3003-3023msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=30436: Tue Apr 8 14:36:33 2025 read: IOPS=9, BW=360MiB/s (378MB/s)(40.0MiB/111msec) slat (nsec): min=6673.9k, max=6673.9k, avg=6673935.00, stdev= 0.00 clat (nsec): min=99887k, max=99887k, avg=99886636.00, stdev= 0.00 lat (nsec): min=106581k, max=106581k, avg=106581478.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 101], 10.00th=[ 101], 20.00th=[ 101], | 30.00th=[ 101], 40.00th=[ 101], 50.00th=[ 101], 60.00th=[ 101], | 70.00th=[ 101], 80.00th=[ 101], 90.00th=[ 101], 95.00th=[ 101], | 99.00th=[ 101], 99.50th=[ 101], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=6.36%, ctx=4, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=360MiB/s (378MB/s), 360MiB/s-360MiB/s (378MB/s-378MB/s), io=40.0MiB (41.9MB), run=111-111msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout