== sanity test 398c: run fio to test AIO ================= 22:45:56 (1733802356) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.13792 s, 304 MB/s osc.lustre-OST0000-osc-ffff88012eba7000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=20443: Mon Dec 9 22:46:05 2024 write: IOPS=368, BW=1474KiB/s (1509kB/s)(10.0MiB/6947msec) slat (usec): min=13, max=623, avg=79.01, stdev=48.32 clat (usec): min=5901, max=86443, avg=43074.20, stdev=4548.78 lat (usec): min=6027, max=86524, avg=43155.14, stdev=4550.98 clat percentiles (usec): | 1.00th=[32637], 5.00th=[36439], 10.00th=[38011], 20.00th=[40109], | 30.00th=[41157], 40.00th=[42206], 50.00th=[43254], 60.00th=[44303], | 70.00th=[45351], 80.00th=[46400], 90.00th=[47973], 95.00th=[48497], | 99.00th=[51643], 99.50th=[52691], 99.90th=[81265], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 1392, max= 1624, per=25.14%, avg=1482.15, stdev=75.45, samples=13 iops : min= 348, max= 406, avg=370.54, stdev=18.86, samples=13 lat (msec) : 10=0.12%, 20=0.04%, 50=97.62%, 100=2.23% cpu : usr=0.53%, sys=3.43%, ctx=1257, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=20444: Mon Dec 9 22:46:05 2024 write: IOPS=368, BW=1475KiB/s (1510kB/s)(10.0MiB/6942msec) slat (usec): min=17, max=579, avg=88.56, stdev=49.29 clat (usec): min=19139, max=86271, avg=43068.90, stdev=4661.89 lat (usec): min=19283, max=86407, avg=43159.41, stdev=4665.27 clat percentiles (usec): | 1.00th=[28181], 5.00th=[36439], 10.00th=[38011], 20.00th=[40109], | 30.00th=[41157], 40.00th=[42206], 50.00th=[43254], 60.00th=[44303], | 70.00th=[45351], 80.00th=[46400], 90.00th=[47973], 95.00th=[49021], | 99.00th=[52691], 99.50th=[55313], 99.90th=[70779], 99.95th=[76022], | 99.99th=[86508] bw ( KiB/s): min= 1368, max= 1640, per=25.17%, avg=1484.08, stdev=73.94, samples=13 iops : min= 342, max= 410, avg=371.00, stdev=18.48, samples=13 lat (msec) : 20=0.62%, 50=96.45%, 100=2.93% cpu : usr=0.49%, sys=3.75%, ctx=1789, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=20445: Mon Dec 9 22:46:05 2024 write: IOPS=369, BW=1479KiB/s (1515kB/s)(10.0MiB/6922msec) slat (usec): min=13, max=386, avg=87.85, stdev=47.69 clat (usec): min=4287, max=96216, avg=42998.67, stdev=5085.38 lat (usec): min=4483, max=96354, avg=43088.49, stdev=5083.40 clat percentiles (usec): | 1.00th=[27395], 5.00th=[35914], 10.00th=[38011], 20.00th=[40109], | 30.00th=[41157], 40.00th=[42206], 50.00th=[43254], 60.00th=[44303], | 70.00th=[45351], 80.00th=[46400], 90.00th=[47973], 95.00th=[49021], | 99.00th=[52691], 99.50th=[55313], 99.90th=[81265], 99.95th=[81265], | 99.99th=[95945] bw ( KiB/s): min= 1408, max= 1616, per=25.28%, avg=1490.23, stdev=77.17, samples=13 iops : min= 352, max= 404, avg=372.54, stdev=19.30, samples=13 lat (msec) : 10=0.08%, 20=0.39%, 50=97.23%, 100=2.30% cpu : usr=0.61%, sys=3.66%, ctx=1763, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=20446: Mon Dec 9 22:46:05 2024 write: IOPS=368, BW=1475KiB/s (1510kB/s)(10.0MiB/6942msec) slat (usec): min=17, max=1096, avg=91.05, stdev=52.06 clat (msec): min=9, max=100, avg=43.10, stdev= 5.05 lat (msec): min=9, max=100, avg=43.20, stdev= 5.05 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 36], 10.00th=[ 38], 20.00th=[ 41], | 30.00th=[ 42], 40.00th=[ 43], 50.00th=[ 44], 60.00th=[ 45], | 70.00th=[ 46], 80.00th=[ 47], 90.00th=[ 48], 95.00th=[ 50], | 99.00th=[ 53], 99.50th=[ 56], 99.90th=[ 96], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 1360, max= 1624, per=25.20%, avg=1485.92, stdev=74.78, samples=13 iops : min= 340, max= 406, avg=371.46, stdev=18.70, samples=13 lat (msec) : 10=0.08%, 50=96.99%, 100=2.85%, 250=0.08% cpu : usr=0.61%, sys=3.86%, ctx=2028, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=5896KiB/s (6038kB/s), 1474KiB/s-1479KiB/s (1509kB/s-1515kB/s), io=40.0MiB (41.9MB), run=6922-6947msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20461: Mon Dec 9 22:46:09 2024 read: IOPS=427, BW=1708KiB/s (1749kB/s)(5048KiB/2955msec) slat (usec): min=24, max=3467, avg=1217.57, stdev=381.88 clat (usec): min=2163, max=48783, avg=17480.31, stdev=4723.21 lat (usec): min=3413, max=51795, avg=18700.19, stdev=4946.73 clat percentiles (usec): | 1.00th=[10290], 5.00th=[12649], 10.00th=[13566], 20.00th=[14615], | 30.00th=[15270], 40.00th=[15926], 50.00th=[16581], 60.00th=[17171], | 70.00th=[18220], 80.00th=[19268], 90.00th=[21365], 95.00th=[28181], | 99.00th=[36963], 99.50th=[39060], 99.90th=[43254], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 1184, max= 1800, per=24.58%, avg=1652.80, stdev=263.07, samples=5 iops : min= 296, max= 450, avg=413.20, stdev=65.77, samples=5 write: IOPS=439, BW=1757KiB/s (1799kB/s)(5192KiB/2955msec) slat (usec): min=12, max=1120, avg=54.00, stdev=47.83 clat (usec): min=8435, max=43671, avg=18063.83, stdev=4320.41 lat (usec): min=8605, max=43812, avg=18119.16, stdev=4332.32 clat percentiles (usec): | 1.00th=[12780], 5.00th=[13960], 10.00th=[14746], 20.00th=[15664], | 30.00th=[16188], 40.00th=[16712], 50.00th=[17171], 60.00th=[17695], | 70.00th=[18220], 80.00th=[19268], 90.00th=[20579], 95.00th=[27919], | 99.00th=[36439], 99.50th=[40109], 99.90th=[41681], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 1144, max= 2016, per=25.02%, avg=1752.00, stdev=348.99, samples=5 iops : min= 286, max= 504, avg=438.00, stdev=87.25, samples=5 lat (msec) : 4=0.12%, 10=0.31%, 20=85.08%, 50=14.49% cpu : usr=1.05%, sys=7.62%, ctx=2827, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20462: Mon Dec 9 22:46:09 2024 read: IOPS=417, BW=1671KiB/s (1712kB/s)(4976KiB/2977msec) slat (usec): min=652, max=4054, avg=1223.03, stdev=373.52 clat (usec): min=3815, max=45798, avg=17543.60, stdev=4767.92 lat (usec): min=4755, max=48017, avg=18768.76, stdev=5015.54 clat percentiles (usec): | 1.00th=[11731], 5.00th=[12780], 10.00th=[13566], 20.00th=[14484], | 30.00th=[15270], 40.00th=[15926], 50.00th=[16581], 60.00th=[17433], | 70.00th=[18220], 80.00th=[19268], 90.00th=[21365], 95.00th=[27395], | 99.00th=[39060], 99.50th=[41157], 99.90th=[45351], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 968, max= 1856, per=24.20%, avg=1627.20, stdev=373.55, samples=5 iops : min= 242, max= 464, avg=406.80, stdev=93.39, samples=5 write: IOPS=442, BW=1768KiB/s (1811kB/s)(5264KiB/2977msec) slat (usec): min=13, max=1142, avg=53.00, stdev=42.97 clat (usec): min=11330, max=45851, avg=18063.11, stdev=4185.82 lat (usec): min=11399, max=45975, avg=18117.29, stdev=4193.43 clat percentiles (usec): | 1.00th=[12780], 5.00th=[13960], 10.00th=[14615], 20.00th=[15533], | 30.00th=[16188], 40.00th=[16712], 50.00th=[17171], 60.00th=[17695], | 70.00th=[18220], 80.00th=[19006], 90.00th=[21103], 95.00th=[27657], | 99.00th=[36439], 99.50th=[38536], 99.90th=[42730], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 1192, max= 1944, per=24.96%, avg=1747.20, stdev=313.00, samples=5 iops : min= 298, max= 486, avg=436.80, stdev=78.25, samples=5 lat (msec) : 4=0.04%, 10=0.12%, 20=86.02%, 50=13.83% cpu : usr=1.01%, sys=7.56%, ctx=2827, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20463: Mon Dec 9 22:46:09 2024 read: IOPS=417, BW=1670KiB/s (1710kB/s)(4984KiB/2984msec) slat (usec): min=654, max=4231, avg=1220.66, stdev=367.99 clat (usec): min=2595, max=42062, avg=17387.61, stdev=4384.27 lat (usec): min=3879, max=44872, avg=18610.47, stdev=4604.46 clat percentiles (usec): | 1.00th=[11207], 5.00th=[12780], 10.00th=[13566], 20.00th=[14484], | 30.00th=[15270], 40.00th=[16057], 50.00th=[16581], 60.00th=[17171], | 70.00th=[17957], 80.00th=[19006], 90.00th=[21103], 95.00th=[27395], | 99.00th=[35390], 99.50th=[38536], 99.90th=[41681], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 1024, max= 1976, per=24.24%, avg=1630.40, stdev=359.22, samples=5 iops : min= 256, max= 494, avg=407.60, stdev=89.80, samples=5 write: IOPS=440, BW=1761KiB/s (1804kB/s)(5256KiB/2984msec) slat (usec): min=13, max=968, avg=52.69, stdev=42.74 clat (usec): min=9670, max=42937, avg=18143.06, stdev=4464.72 lat (usec): min=9728, max=43104, avg=18196.92, stdev=4476.40 clat percentiles (usec): | 1.00th=[12780], 5.00th=[14091], 10.00th=[14615], 20.00th=[15533], | 30.00th=[16188], 40.00th=[16712], 50.00th=[17171], 60.00th=[17695], | 70.00th=[18220], 80.00th=[19268], 90.00th=[20841], 95.00th=[28181], | 99.00th=[38536], 99.50th=[40633], 99.90th=[41681], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 1024, max= 2016, per=24.50%, avg=1715.20, stdev=395.44, samples=5 iops : min= 256, max= 504, avg=428.80, stdev=98.86, samples=5 lat (msec) : 4=0.04%, 10=0.08%, 20=86.33%, 50=13.55% cpu : usr=1.01%, sys=7.41%, ctx=2928, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20464: Mon Dec 9 22:46:09 2024 read: IOPS=423, BW=1696KiB/s (1736kB/s)(5060KiB/2984msec) slat (usec): min=609, max=3117, avg=1212.10, stdev=348.06 clat (usec): min=3347, max=43164, avg=17362.81, stdev=4486.04 lat (usec): min=4273, max=45606, avg=18576.94, stdev=4701.25 clat percentiles (usec): | 1.00th=[11207], 5.00th=[12649], 10.00th=[13566], 20.00th=[14484], | 30.00th=[15270], 40.00th=[15795], 50.00th=[16450], 60.00th=[17171], | 70.00th=[17957], 80.00th=[19006], 90.00th=[21365], 95.00th=[26084], | 99.00th=[37487], 99.50th=[41157], 99.90th=[42206], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 1016, max= 1832, per=24.42%, avg=1642.40, stdev=351.86, samples=5 iops : min= 254, max= 458, avg=410.60, stdev=87.96, samples=5 write: IOPS=433, BW=1736KiB/s (1778kB/s)(5180KiB/2984msec) slat (usec): min=13, max=772, avg=50.63, stdev=36.46 clat (usec): min=3756, max=42833, avg=18117.06, stdev=4416.87 lat (usec): min=3817, max=42937, avg=18168.79, stdev=4425.07 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13960], 10.00th=[14746], 20.00th=[15533], | 30.00th=[16188], 40.00th=[16712], 50.00th=[17171], 60.00th=[17957], | 70.00th=[18482], 80.00th=[19268], 90.00th=[20841], 95.00th=[27132], | 99.00th=[38536], 99.50th=[40633], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 1040, max= 1928, per=24.33%, avg=1703.20, stdev=375.22, samples=5 iops : min= 260, max= 482, avg=425.80, stdev=93.80, samples=5 lat (msec) : 4=0.08%, 10=0.16%, 20=86.05%, 50=13.71% cpu : usr=0.84%, sys=7.41%, ctx=2848, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6725KiB/s (6887kB/s), 1670KiB/s-1708KiB/s (1710kB/s-1749kB/s), io=19.6MiB (20.5MB), run=2955-2984msec WRITE: bw=7001KiB/s (7169kB/s), 1736KiB/s-1768KiB/s (1778kB/s-1811kB/s), io=20.4MiB (21.4MB), run=2955-2984msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=20470: Mon Dec 9 22:46:10 2024 read: IOPS=10, BW=408MiB/s (428MB/s)(40.0MiB/98msec) slat (nsec): min=10333k, max=10333k, avg=10332929.00, stdev= 0.00 clat (nsec): min=81341k, max=81341k, avg=81341152.00, stdev= 0.00 lat (nsec): min=91686k, max=91686k, avg=91686493.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[81265], 5.00th=[81265], 10.00th=[81265], 20.00th=[81265], | 30.00th=[81265], 40.00th=[81265], 50.00th=[81265], 60.00th=[81265], | 70.00th=[81265], 80.00th=[81265], 90.00th=[81265], 95.00th=[81265], | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=10.31%, ctx=5, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=408MiB/s (428MB/s), 408MiB/s-408MiB/s (428MB/s-428MB/s), io=40.0MiB (41.9MB), run=98-98msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout