== sanity test 398c: run fio to test AIO ================= 02:28:10 (1744093690) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.140698 s, 298 MB/s osc.lustre-OST0000-osc-ffff8800b59ad000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=31280: Tue Apr 8 02:28:19 2025 write: IOPS=393, BW=1575KiB/s (1612kB/s)(10.0MiB/6503msec) slat (usec): min=15, max=442, avg=91.22, stdev=50.27 clat (usec): min=6541, max=75107, avg=40402.72, stdev=7169.95 lat (usec): min=6670, max=75136, avg=40496.44, stdev=7184.51 clat percentiles (usec): | 1.00th=[19792], 5.00th=[23462], 10.00th=[29230], 20.00th=[35914], | 30.00th=[40109], 40.00th=[41157], 50.00th=[42206], 60.00th=[43254], | 70.00th=[44303], 80.00th=[45876], 90.00th=[46924], 95.00th=[47449], | 99.00th=[49546], 99.50th=[49546], 99.90th=[70779], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 1416, max= 1936, per=24.89%, avg=1565.85, stdev=173.38, samples=13 iops : min= 354, max= 484, avg=391.46, stdev=43.34, samples=13 lat (msec) : 10=0.08%, 20=1.21%, 50=98.36%, 100=0.35% cpu : usr=0.66%, sys=3.84%, ctx=2042, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=31281: Tue Apr 8 02:28:19 2025 write: IOPS=394, BW=1579KiB/s (1617kB/s)(10.0MiB/6486msec) slat (usec): min=15, max=465, avg=90.15, stdev=49.61 clat (usec): min=6280, max=67213, avg=40314.58, stdev=7283.95 lat (usec): min=6338, max=67307, avg=40407.05, stdev=7291.27 clat percentiles (usec): | 1.00th=[18482], 5.00th=[22938], 10.00th=[28967], 20.00th=[35914], | 30.00th=[40109], 40.00th=[41157], 50.00th=[42206], 60.00th=[43254], | 70.00th=[44303], 80.00th=[45351], 90.00th=[46924], 95.00th=[47449], | 99.00th=[49546], 99.50th=[49546], 99.90th=[58983], 99.95th=[63177], | 99.99th=[67634] bw ( KiB/s): min= 1408, max= 1936, per=24.68%, avg=1552.42, stdev=184.40, samples=12 iops : min= 352, max= 484, avg=388.08, stdev=46.11, samples=12 lat (msec) : 10=0.04%, 20=1.64%, 50=97.93%, 100=0.39% cpu : usr=0.69%, sys=3.78%, ctx=1947, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=31282: Tue Apr 8 02:28:19 2025 write: IOPS=393, BW=1574KiB/s (1611kB/s)(10.0MiB/6507msec) slat (usec): min=14, max=1132, avg=88.40, stdev=53.56 clat (usec): min=5753, max=66778, avg=40403.47, stdev=7200.81 lat (usec): min=5851, max=66838, avg=40494.20, stdev=7214.70 clat percentiles (usec): | 1.00th=[19268], 5.00th=[23462], 10.00th=[28967], 20.00th=[35914], | 30.00th=[40109], 40.00th=[41157], 50.00th=[42206], 60.00th=[43254], | 70.00th=[44303], 80.00th=[45876], 90.00th=[46924], 95.00th=[47449], | 99.00th=[49546], 99.50th=[49546], 99.90th=[62129], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 1408, max= 1936, per=24.89%, avg=1565.85, stdev=180.58, samples=13 iops : min= 352, max= 484, avg=391.46, stdev=45.15, samples=13 lat (msec) : 10=0.04%, 20=1.41%, 50=98.24%, 100=0.31% cpu : usr=0.60%, sys=3.69%, ctx=1923, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=31283: Tue Apr 8 02:28:19 2025 write: IOPS=393, BW=1573KiB/s (1610kB/s)(10.0MiB/6511msec) slat (usec): min=12, max=533, avg=73.63, stdev=45.35 clat (usec): min=3745, max=61431, avg=40405.21, stdev=7082.12 lat (usec): min=3854, max=61516, avg=40480.76, stdev=7090.27 clat percentiles (usec): | 1.00th=[19268], 5.00th=[23462], 10.00th=[29492], 20.00th=[36439], | 30.00th=[40109], 40.00th=[41681], 50.00th=[42730], 60.00th=[43779], | 70.00th=[44827], 80.00th=[45876], 90.00th=[46924], 95.00th=[47449], | 99.00th=[49021], 99.50th=[50070], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61604] bw ( KiB/s): min= 1408, max= 1880, per=24.90%, avg=1566.15, stdev=163.30, samples=13 iops : min= 352, max= 470, avg=391.54, stdev=40.83, samples=13 lat (msec) : 4=0.04%, 20=1.25%, 50=98.28%, 100=0.43% cpu : usr=0.72%, sys=3.13%, ctx=1112, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=6291KiB/s (6442kB/s), 1573KiB/s-1579KiB/s (1610kB/s-1617kB/s), io=40.0MiB (41.9MB), run=6486-6511msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31298: Tue Apr 8 02:28:22 2025 read: IOPS=447, BW=1791KiB/s (1834kB/s)(5048KiB/2819msec) slat (usec): min=627, max=3387, avg=1124.04, stdev=304.15 clat (usec): min=9234, max=39120, avg=16426.76, stdev=4034.48 lat (usec): min=10322, max=42005, avg=17552.64, stdev=4209.42 clat percentiles (usec): | 1.00th=[11338], 5.00th=[12256], 10.00th=[12911], 20.00th=[13698], | 30.00th=[14484], 40.00th=[15139], 50.00th=[15795], 60.00th=[16319], | 70.00th=[17171], 80.00th=[17957], 90.00th=[19530], 95.00th=[22152], | 99.00th=[35914], 99.50th=[36963], 99.90th=[38536], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 1224, max= 1952, per=24.53%, avg=1746.40, stdev=299.91, samples=5 iops : min= 306, max= 488, avg=436.60, stdev=74.98, samples=5 write: IOPS=460, BW=1842KiB/s (1886kB/s)(5192KiB/2819msec) slat (usec): min=12, max=1421, avg=41.84, stdev=43.81 clat (usec): min=10466, max=42693, avg=17065.54, stdev=3781.52 lat (usec): min=10515, max=42742, avg=17108.43, stdev=3786.16 clat percentiles (usec): | 1.00th=[12649], 5.00th=[13566], 10.00th=[14091], 20.00th=[14877], | 30.00th=[15401], 40.00th=[15926], 50.00th=[16450], 60.00th=[16909], | 70.00th=[17433], 80.00th=[18220], 90.00th=[19530], 95.00th=[23200], | 99.00th=[35390], 99.50th=[40633], 99.90th=[41157], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 1200, max= 2104, per=24.66%, avg=1827.80, stdev=360.63, samples=5 iops : min= 300, max= 526, avg=456.80, stdev=90.06, samples=5 lat (msec) : 10=0.04%, 20=91.76%, 50=8.20% cpu : usr=0.89%, sys=6.85%, ctx=2290, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31299: Tue Apr 8 02:28:22 2025 read: IOPS=441, BW=1766KiB/s (1808kB/s)(4976KiB/2818msec) slat (usec): min=603, max=3145, avg=1124.73, stdev=321.89 clat (usec): min=9693, max=41393, avg=16463.44, stdev=4013.47 lat (usec): min=10892, max=44213, avg=17590.06, stdev=4203.45 clat percentiles (usec): | 1.00th=[11338], 5.00th=[12256], 10.00th=[13042], 20.00th=[13829], | 30.00th=[14615], 40.00th=[15139], 50.00th=[15795], 60.00th=[16450], | 70.00th=[17171], 80.00th=[17957], 90.00th=[19530], 95.00th=[22676], | 99.00th=[34866], 99.50th=[36963], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 1120, max= 1940, per=24.29%, avg=1728.80, stdev=348.03, samples=5 iops : min= 280, max= 485, avg=432.20, stdev=87.01, samples=5 write: IOPS=466, BW=1868KiB/s (1913kB/s)(5264KiB/2818msec) slat (usec): min=12, max=1516, avg=43.65, stdev=64.78 clat (usec): min=11128, max=41288, avg=17187.62, stdev=3686.23 lat (usec): min=11172, max=41336, avg=17232.38, stdev=3692.16 clat percentiles (usec): | 1.00th=[12780], 5.00th=[13698], 10.00th=[14222], 20.00th=[14877], | 30.00th=[15533], 40.00th=[16057], 50.00th=[16450], 60.00th=[17171], | 70.00th=[17695], 80.00th=[18220], 90.00th=[19530], 95.00th=[24511], | 99.00th=[34341], 99.50th=[38011], 99.90th=[39584], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 1384, max= 2080, per=25.14%, avg=1863.20, stdev=274.35, samples=5 iops : min= 346, max= 520, avg=465.80, stdev=68.59, samples=5 lat (msec) : 10=0.08%, 20=91.60%, 50=8.32% cpu : usr=1.21%, sys=6.60%, ctx=2255, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31300: Tue Apr 8 02:28:22 2025 read: IOPS=445, BW=1781KiB/s (1824kB/s)(4984KiB/2798msec) slat (usec): min=27, max=3334, avg=1124.68, stdev=332.50 clat (usec): min=1504, max=37127, avg=16324.04, stdev=4017.79 lat (usec): min=2305, max=40466, avg=17450.49, stdev=4204.98 clat percentiles (usec): | 1.00th=[10683], 5.00th=[12125], 10.00th=[12911], 20.00th=[13829], | 30.00th=[14484], 40.00th=[15008], 50.00th=[15664], 60.00th=[16319], | 70.00th=[17171], 80.00th=[17957], 90.00th=[19530], 95.00th=[22676], | 99.00th=[34866], 99.50th=[36439], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 1344, max= 2056, per=24.77%, avg=1763.20, stdev=258.53, samples=5 iops : min= 336, max= 514, avg=440.80, stdev=64.63, samples=5 write: IOPS=469, BW=1878KiB/s (1924kB/s)(5256KiB/2798msec) slat (usec): min=13, max=286, avg=41.45, stdev=23.78 clat (usec): min=10872, max=40314, avg=17346.11, stdev=3662.74 lat (usec): min=10911, max=40348, avg=17388.58, stdev=3670.50 clat percentiles (usec): | 1.00th=[12911], 5.00th=[13698], 10.00th=[14484], 20.00th=[15139], | 30.00th=[15795], 40.00th=[16188], 50.00th=[16712], 60.00th=[17171], | 70.00th=[17433], 80.00th=[18220], 90.00th=[19530], 95.00th=[24773], | 99.00th=[34866], 99.50th=[36963], 99.90th=[38536], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 1336, max= 2096, per=25.04%, avg=1856.00, stdev=308.23, samples=5 iops : min= 334, max= 524, avg=464.00, stdev=77.06, samples=5 lat (msec) : 2=0.04%, 4=0.20%, 10=0.12%, 20=90.47%, 50=9.18% cpu : usr=1.00%, sys=6.83%, ctx=2249, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31301: Tue Apr 8 02:28:22 2025 read: IOPS=449, BW=1797KiB/s (1840kB/s)(5060KiB/2816msec) slat (usec): min=672, max=3826, avg=1133.79, stdev=311.00 clat (usec): min=9092, max=43281, avg=16458.87, stdev=4244.92 lat (usec): min=10380, max=44994, avg=17594.55, stdev=4415.37 clat percentiles (usec): | 1.00th=[10945], 5.00th=[12125], 10.00th=[12780], 20.00th=[13829], | 30.00th=[14484], 40.00th=[15008], 50.00th=[15664], 60.00th=[16450], | 70.00th=[17171], 80.00th=[17957], 90.00th=[19530], 95.00th=[22414], | 99.00th=[36439], 99.50th=[39060], 99.90th=[42730], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 1248, max= 1952, per=24.82%, avg=1766.40, stdev=292.49, samples=5 iops : min= 312, max= 488, avg=441.60, stdev=73.12, samples=5 write: IOPS=459, BW=1839KiB/s (1884kB/s)(5180KiB/2816msec) slat (usec): min=12, max=283, avg=39.72, stdev=22.04 clat (usec): min=7101, max=37344, avg=17187.39, stdev=3691.43 lat (usec): min=7148, max=37408, avg=17228.12, stdev=3696.37 clat percentiles (usec): | 1.00th=[12780], 5.00th=[13566], 10.00th=[14222], 20.00th=[14877], | 30.00th=[15664], 40.00th=[16057], 50.00th=[16450], 60.00th=[16909], | 70.00th=[17433], 80.00th=[18220], 90.00th=[19530], 95.00th=[24773], | 99.00th=[35914], 99.50th=[36439], 99.90th=[36963], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 1264, max= 2024, per=24.44%, avg=1811.20, stdev=318.23, samples=5 iops : min= 316, max= 506, avg=452.80, stdev=79.56, samples=5 lat (msec) : 10=0.16%, 20=91.13%, 50=8.71% cpu : usr=0.67%, sys=7.07%, ctx=2212, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7119KiB/s (7290kB/s), 1766KiB/s-1797KiB/s (1808kB/s-1840kB/s), io=19.6MiB (20.5MB), run=2798-2819msec WRITE: bw=7411KiB/s (7589kB/s), 1839KiB/s-1878KiB/s (1884kB/s-1924kB/s), io=20.4MiB (21.4MB), run=2798-2819msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=31307: Tue Apr 8 02:28:24 2025 read: IOPS=8, BW=348MiB/s (365MB/s)(40.0MiB/115msec) slat (nsec): min=5112.3k, max=5112.3k, avg=5112295.00, stdev= 0.00 clat (nsec): min=103786k, max=103786k, avg=103786259.00, stdev= 0.00 lat (nsec): min=108908k, max=108908k, avg=108908052.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 104], | 30.00th=[ 104], 40.00th=[ 104], 50.00th=[ 104], 60.00th=[ 104], | 70.00th=[ 104], 80.00th=[ 104], 90.00th=[ 104], 95.00th=[ 104], | 99.00th=[ 104], 99.50th=[ 104], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.39%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=348MiB/s (365MB/s), 348MiB/s-348MiB/s (365MB/s-365MB/s), io=40.0MiB (41.9MB), run=115-115msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout