== sanity test 398c: run fio to test AIO ================= 10:47:35 (1749480455) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0996038 s, 421 MB/s osc.lustre-OST0000-osc-ffff8800b3634000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=24425: Mon Jun 9 10:47:41 2025 write: IOPS=631, BW=2524KiB/s (2585kB/s)(10.0MiB/4057msec) slat (usec): min=13, max=506, avg=53.96, stdev=34.22 clat (usec): min=12808, max=48056, avg=25176.03, stdev=6003.66 lat (usec): min=12943, max=48140, avg=25231.34, stdev=6012.40 clat percentiles (usec): | 1.00th=[15401], 5.00th=[17957], 10.00th=[19268], 20.00th=[20841], | 30.00th=[21627], 40.00th=[22676], 50.00th=[23462], 60.00th=[24511], | 70.00th=[26346], 80.00th=[29492], 90.00th=[33817], 95.00th=[38536], | 99.00th=[43779], 99.50th=[44303], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 1744, max= 3088, per=25.02%, avg=2525.00, stdev=445.13, samples=8 iops : min= 436, max= 772, avg=631.25, stdev=111.28, samples=8 lat (msec) : 20=12.62%, 50=87.38% cpu : usr=0.59%, sys=4.12%, ctx=1486, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24426: Mon Jun 9 10:47:41 2025 write: IOPS=633, BW=2534KiB/s (2595kB/s)(10.0MiB/4041msec) slat (usec): min=13, max=421, avg=53.06, stdev=32.57 clat (usec): min=5016, max=50735, avg=25092.47, stdev=6087.75 lat (usec): min=5104, max=50790, avg=25146.85, stdev=6095.18 clat percentiles (usec): | 1.00th=[15008], 5.00th=[17957], 10.00th=[19268], 20.00th=[20841], | 30.00th=[21627], 40.00th=[22414], 50.00th=[23462], 60.00th=[24249], | 70.00th=[26608], 80.00th=[29754], 90.00th=[33162], 95.00th=[38536], | 99.00th=[44827], 99.50th=[45351], 99.90th=[46400], 99.95th=[48497], | 99.99th=[50594] bw ( KiB/s): min= 1704, max= 3096, per=25.09%, avg=2532.00, stdev=465.42, samples=8 iops : min= 426, max= 774, avg=633.00, stdev=116.35, samples=8 lat (msec) : 10=0.08%, 20=13.28%, 50=86.60%, 100=0.04% cpu : usr=0.72%, sys=3.84%, ctx=1464, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24427: Mon Jun 9 10:47:41 2025 write: IOPS=631, BW=2524KiB/s (2585kB/s)(10.0MiB/4057msec) slat (usec): min=13, max=385, avg=53.28, stdev=32.08 clat (usec): min=2835, max=47888, avg=25154.55, stdev=6084.74 lat (usec): min=2882, max=47952, avg=25209.14, stdev=6093.12 clat percentiles (usec): | 1.00th=[15270], 5.00th=[17957], 10.00th=[19268], 20.00th=[20841], | 30.00th=[21627], 40.00th=[22414], 50.00th=[23462], 60.00th=[24511], | 70.00th=[26608], 80.00th=[29754], 90.00th=[33162], 95.00th=[38536], | 99.00th=[44303], 99.50th=[44827], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 1712, max= 3112, per=25.01%, avg=2524.00, stdev=454.83, samples=8 iops : min= 428, max= 778, avg=631.00, stdev=113.71, samples=8 lat (msec) : 4=0.04%, 20=13.75%, 50=86.21% cpu : usr=0.52%, sys=4.19%, ctx=1500, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24428: Mon Jun 9 10:47:41 2025 write: IOPS=630, BW=2523KiB/s (2583kB/s)(10.0MiB/4059msec) slat (usec): min=13, max=598, avg=53.41, stdev=36.52 clat (usec): min=12437, max=46432, avg=25153.50, stdev=5997.67 lat (usec): min=12490, max=46515, avg=25208.21, stdev=6007.89 clat percentiles (usec): | 1.00th=[15270], 5.00th=[17957], 10.00th=[19268], 20.00th=[20841], | 30.00th=[21627], 40.00th=[22676], 50.00th=[23725], 60.00th=[24511], | 70.00th=[26346], 80.00th=[29492], 90.00th=[33817], 95.00th=[38011], | 99.00th=[44303], 99.50th=[44827], 99.90th=[45351], 99.95th=[45351], | 99.99th=[46400] bw ( KiB/s): min= 1736, max= 3080, per=24.96%, avg=2519.00, stdev=450.51, samples=8 iops : min= 434, max= 770, avg=629.75, stdev=112.63, samples=8 lat (msec) : 20=12.70%, 50=87.30% cpu : usr=0.76%, sys=3.67%, ctx=1573, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=9.85MiB/s (10.3MB/s), 2523KiB/s-2534KiB/s (2583kB/s-2595kB/s), io=40.0MiB (41.9MB), run=4041-4059msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24441: Mon Jun 9 10:47:45 2025 read: IOPS=436, BW=1747KiB/s (1789kB/s)(5048KiB/2889msec) slat (usec): min=698, max=2527, avg=1176.43, stdev=243.42 clat (usec): min=6847, max=25256, avg=16970.48, stdev=2431.76 lat (usec): min=7914, max=26510, avg=18148.46, stdev=2456.44 clat percentiles (usec): | 1.00th=[11994], 5.00th=[13173], 10.00th=[13960], 20.00th=[14877], | 30.00th=[15664], 40.00th=[16319], 50.00th=[16909], 60.00th=[17433], | 70.00th=[18220], 80.00th=[19006], 90.00th=[20055], 95.00th=[21103], | 99.00th=[23200], 99.50th=[23462], 99.90th=[25035], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 1632, max= 1816, per=25.04%, avg=1739.20, stdev=68.96, samples=5 iops : min= 408, max= 454, avg=434.80, stdev=17.24, samples=5 write: IOPS=449, BW=1797KiB/s (1840kB/s)(5192KiB/2889msec) slat (usec): min=13, max=1346, avg=43.63, stdev=41.51 clat (usec): min=8184, max=25506, avg=17678.20, stdev=1923.08 lat (usec): min=8220, max=25578, avg=17722.92, stdev=1923.86 clat percentiles (usec): | 1.00th=[12780], 5.00th=[14746], 10.00th=[15401], 20.00th=[16188], | 30.00th=[16712], 40.00th=[17171], 50.00th=[17695], 60.00th=[18220], | 70.00th=[18482], 80.00th=[19006], 90.00th=[20055], 95.00th=[20579], | 99.00th=[22938], 99.50th=[23987], 99.90th=[25297], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 1752, max= 1936, per=25.05%, avg=1811.20, stdev=76.44, samples=5 iops : min= 438, max= 484, avg=452.80, stdev=19.11, samples=5 lat (msec) : 10=0.31%, 20=89.38%, 50=10.31% cpu : usr=0.80%, sys=6.96%, ctx=2376, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24442: Mon Jun 9 10:47:45 2025 read: IOPS=430, BW=1724KiB/s (1765kB/s)(4976KiB/2887msec) slat (usec): min=649, max=3447, avg=1169.94, stdev=244.28 clat (usec): min=7932, max=25850, avg=17020.89, stdev=2356.46 lat (usec): min=9295, max=26991, avg=18192.46, stdev=2379.06 clat percentiles (usec): | 1.00th=[11994], 5.00th=[13435], 10.00th=[14091], 20.00th=[15008], | 30.00th=[15664], 40.00th=[16319], 50.00th=[16909], 60.00th=[17433], | 70.00th=[18220], 80.00th=[19006], 90.00th=[20055], 95.00th=[21103], | 99.00th=[22676], 99.50th=[23462], 99.90th=[24511], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 1576, max= 1776, per=24.58%, avg=1707.20, stdev=84.02, samples=5 iops : min= 394, max= 444, avg=426.80, stdev=21.00, samples=5 write: IOPS=455, BW=1823KiB/s (1867kB/s)(5264KiB/2887msec) slat (usec): min=14, max=337, avg=43.41, stdev=22.46 clat (usec): min=10560, max=28095, avg=17616.73, stdev=1894.67 lat (usec): min=10594, max=28130, avg=17661.14, stdev=1895.56 clat percentiles (usec): | 1.00th=[13435], 5.00th=[14746], 10.00th=[15270], 20.00th=[16188], | 30.00th=[16581], 40.00th=[16909], 50.00th=[17433], 60.00th=[17957], | 70.00th=[18482], 80.00th=[19006], 90.00th=[20055], 95.00th=[20841], | 99.00th=[22414], 99.50th=[23725], 99.90th=[26870], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 1736, max= 1960, per=25.47%, avg=1841.60, stdev=100.14, samples=5 iops : min= 434, max= 490, avg=460.40, stdev=25.04, samples=5 lat (msec) : 10=0.08%, 20=89.57%, 50=10.35% cpu : usr=0.55%, sys=7.07%, ctx=2395, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24443: Mon Jun 9 10:47:45 2025 read: IOPS=433, BW=1734KiB/s (1776kB/s)(4984KiB/2874msec) slat (usec): min=644, max=2005, avg=1177.92, stdev=232.43 clat (usec): min=4609, max=39781, avg=16943.79, stdev=2664.89 lat (usec): min=6102, max=40707, avg=18123.40, stdev=2677.55 clat percentiles (usec): | 1.00th=[11731], 5.00th=[13173], 10.00th=[13960], 20.00th=[14877], | 30.00th=[15664], 40.00th=[16319], 50.00th=[16909], 60.00th=[17695], | 70.00th=[18220], 80.00th=[18744], 90.00th=[19792], 95.00th=[20579], | 99.00th=[23462], 99.50th=[26084], 99.90th=[38536], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 1584, max= 1960, per=24.92%, avg=1731.20, stdev=142.17, samples=5 iops : min= 396, max= 490, avg=432.80, stdev=35.54, samples=5 write: IOPS=457, BW=1829KiB/s (1873kB/s)(5256KiB/2874msec) slat (usec): min=14, max=257, avg=42.12, stdev=20.16 clat (usec): min=9933, max=38577, avg=17575.26, stdev=2158.27 lat (usec): min=9973, max=38594, avg=17618.35, stdev=2157.86 clat percentiles (usec): | 1.00th=[13042], 5.00th=[14484], 10.00th=[15270], 20.00th=[16057], | 30.00th=[16581], 40.00th=[16909], 50.00th=[17433], 60.00th=[17957], | 70.00th=[18482], 80.00th=[19006], 90.00th=[19792], 95.00th=[20841], | 99.00th=[23725], 99.50th=[25822], 99.90th=[37487], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 1744, max= 1936, per=25.51%, avg=1844.80, stdev=76.65, samples=5 iops : min= 436, max= 484, avg=461.20, stdev=19.16, samples=5 lat (msec) : 10=0.31%, 20=91.37%, 50=8.32% cpu : usr=0.70%, sys=7.03%, ctx=2338, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24444: Mon Jun 9 10:47:45 2025 read: IOPS=440, BW=1760KiB/s (1802kB/s)(5060KiB/2875msec) slat (usec): min=22, max=2294, avg=1163.91, stdev=243.73 clat (usec): min=1073, max=29443, avg=17027.49, stdev=2683.31 lat (usec): min=1186, max=30545, avg=18193.08, stdev=2719.99 clat percentiles (usec): | 1.00th=[10814], 5.00th=[13173], 10.00th=[14091], 20.00th=[15008], | 30.00th=[15664], 40.00th=[16319], 50.00th=[16909], 60.00th=[17695], | 70.00th=[18482], 80.00th=[19268], 90.00th=[20317], 95.00th=[21103], | 99.00th=[23200], 99.50th=[23987], 99.90th=[28443], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 1736, max= 1784, per=25.41%, avg=1764.80, stdev=18.42, samples=5 iops : min= 434, max= 446, avg=441.20, stdev= 4.60, samples=5 write: IOPS=450, BW=1802KiB/s (1845kB/s)(5180KiB/2875msec) slat (usec): min=14, max=1105, avg=43.96, stdev=36.62 clat (usec): min=6083, max=34827, avg=17620.48, stdev=2320.99 lat (usec): min=6116, max=34873, avg=17665.50, stdev=2319.16 clat percentiles (usec): | 1.00th=[ 9896], 5.00th=[14484], 10.00th=[15270], 20.00th=[16057], | 30.00th=[16581], 40.00th=[17171], 50.00th=[17695], 60.00th=[17957], | 70.00th=[18482], 80.00th=[19006], 90.00th=[20055], 95.00th=[21103], | 99.00th=[23725], 99.50th=[26084], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 1752, max= 1944, per=25.07%, avg=1812.80, stdev=77.69, samples=5 iops : min= 438, max= 486, avg=453.20, stdev=19.42, samples=5 lat (msec) : 2=0.16%, 4=0.04%, 10=0.70%, 20=88.79%, 50=10.31% cpu : usr=1.04%, sys=6.75%, ctx=2419, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6946KiB/s (7113kB/s), 1724KiB/s-1760KiB/s (1765kB/s-1802kB/s), io=19.6MiB (20.5MB), run=2874-2889msec WRITE: bw=7232KiB/s (7405kB/s), 1797KiB/s-1829KiB/s (1840kB/s-1873kB/s), io=20.4MiB (21.4MB), run=2874-2889msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=24450: Mon Jun 9 10:47:46 2025 read: IOPS=7, BW=301MiB/s (315MB/s)(40.0MiB/133msec) slat (nsec): min=7455.8k, max=7455.8k, avg=7455768.00, stdev= 0.00 clat (nsec): min=119998k, max=119998k, avg=119997917.00, stdev= 0.00 lat (nsec): min=127463k, max=127463k, avg=127463130.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 121], 10.00th=[ 121], 20.00th=[ 121], | 30.00th=[ 121], 40.00th=[ 121], 50.00th=[ 121], 60.00th=[ 121], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 121], 95.00th=[ 121], | 99.00th=[ 121], 99.50th=[ 121], 99.90th=[ 121], 99.95th=[ 121], | 99.99th=[ 121] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.06%, ctx=5, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=301MiB/s (315MB/s), 301MiB/s-301MiB/s (315MB/s-315MB/s), io=40.0MiB (41.9MB), run=133-133msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout