== sanity test 398c: run fio to test AIO ================= 02:17:13 (1744179433) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.136561 s, 307 MB/s osc.lustre-OST0000-osc-ffff8800b62b0000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=32621: Wed Apr 9 02:17:21 2025 write: IOPS=397, BW=1590KiB/s (1628kB/s)(10.0MiB/6441msec) slat (usec): min=13, max=1233, avg=90.70, stdev=58.32 clat (usec): min=5548, max=83874, avg=40021.11, stdev=5562.43 lat (usec): min=5660, max=83969, avg=40113.93, stdev=5566.08 clat percentiles (usec): | 1.00th=[21890], 5.00th=[31327], 10.00th=[33817], 20.00th=[36439], | 30.00th=[38011], 40.00th=[39584], 50.00th=[41157], 60.00th=[42206], | 70.00th=[43254], 80.00th=[43779], 90.00th=[44827], 95.00th=[46924], | 99.00th=[49021], 99.50th=[54789], 99.90th=[70779], 99.95th=[80217], | 99.99th=[83362] bw ( KiB/s): min= 1368, max= 1840, per=25.19%, avg=1597.33, stdev=134.36, samples=12 iops : min= 342, max= 460, avg=399.33, stdev=33.59, samples=12 lat (msec) : 10=0.12%, 20=0.43%, 50=98.67%, 100=0.78% cpu : usr=0.51%, sys=4.29%, ctx=1464, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=32622: Wed Apr 9 02:17:21 2025 write: IOPS=396, BW=1585KiB/s (1623kB/s)(10.0MiB/6459msec) slat (usec): min=13, max=696, avg=84.27, stdev=53.52 clat (usec): min=9005, max=59332, avg=40063.43, stdev=4854.99 lat (usec): min=9070, max=59366, avg=40149.54, stdev=4859.61 clat percentiles (usec): | 1.00th=[23462], 5.00th=[31065], 10.00th=[33424], 20.00th=[36439], | 30.00th=[38011], 40.00th=[39584], 50.00th=[41157], 60.00th=[42206], | 70.00th=[43254], 80.00th=[43779], 90.00th=[44827], 95.00th=[46400], | 99.00th=[48497], 99.50th=[49021], 99.90th=[50594], 99.95th=[54789], | 99.99th=[59507] bw ( KiB/s): min= 1312, max= 1896, per=24.95%, avg=1582.00, stdev=149.05, samples=12 iops : min= 328, max= 474, avg=395.50, stdev=37.26, samples=12 lat (msec) : 10=0.04%, 20=0.12%, 50=99.69%, 100=0.16% cpu : usr=0.54%, sys=3.93%, ctx=1315, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=32623: Wed Apr 9 02:17:21 2025 write: IOPS=396, BW=1586KiB/s (1624kB/s)(10.0MiB/6457msec) slat (usec): min=13, max=442, avg=77.06, stdev=46.85 clat (usec): min=21749, max=61400, avg=40130.63, stdev=4832.45 lat (usec): min=21807, max=61541, avg=40209.36, stdev=4835.55 clat percentiles (usec): | 1.00th=[23200], 5.00th=[31851], 10.00th=[33817], 20.00th=[36439], | 30.00th=[38536], 40.00th=[39584], 50.00th=[41157], 60.00th=[42206], | 70.00th=[43254], 80.00th=[43779], 90.00th=[44827], 95.00th=[46924], | 99.00th=[48497], 99.50th=[49021], 99.90th=[56361], 99.95th=[56361], | 99.99th=[61604] bw ( KiB/s): min= 1408, max= 1872, per=25.06%, avg=1588.67, stdev=142.60, samples=12 iops : min= 352, max= 468, avg=397.17, stdev=35.65, samples=12 lat (msec) : 50=99.84%, 100=0.16% cpu : usr=0.59%, sys=3.59%, ctx=1036, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=32624: Wed Apr 9 02:17:21 2025 write: IOPS=396, BW=1585KiB/s (1623kB/s)(10.0MiB/6460msec) slat (usec): min=17, max=1045, avg=91.10, stdev=58.96 clat (usec): min=4649, max=77922, avg=40091.13, stdev=5189.38 lat (usec): min=4979, max=77992, avg=40184.35, stdev=5193.86 clat percentiles (usec): | 1.00th=[21890], 5.00th=[31327], 10.00th=[33817], 20.00th=[36439], | 30.00th=[38011], 40.00th=[39584], 50.00th=[41157], 60.00th=[42206], | 70.00th=[43254], 80.00th=[43779], 90.00th=[44827], 95.00th=[46400], | 99.00th=[49021], 99.50th=[50070], 99.90th=[73925], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 1384, max= 1856, per=25.10%, avg=1591.33, stdev=129.38, samples=12 iops : min= 346, max= 464, avg=397.83, stdev=32.34, samples=12 lat (msec) : 10=0.04%, 50=99.41%, 100=0.55% cpu : usr=0.50%, sys=4.23%, ctx=1584, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=6341KiB/s (6493kB/s), 1585KiB/s-1590KiB/s (1623kB/s-1628kB/s), io=40.0MiB (41.9MB), run=6441-6460msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32639: Wed Apr 9 02:17:25 2025 read: IOPS=440, BW=1760KiB/s (1802kB/s)(5048KiB/2868msec) slat (usec): min=41, max=3184, avg=1178.39, stdev=428.05 clat (usec): min=2250, max=41945, avg=17039.33, stdev=5647.36 lat (usec): min=3330, max=44131, avg=18220.45, stdev=5970.37 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11863], 10.00th=[12649], 20.00th=[13435], | 30.00th=[14353], 40.00th=[14877], 50.00th=[15533], 60.00th=[16188], | 70.00th=[17171], 80.00th=[18482], 90.00th=[24511], 95.00th=[31065], | 99.00th=[36439], 99.50th=[38536], 99.90th=[41157], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 992, max= 2128, per=24.82%, avg=1721.60, stdev=450.26, samples=5 iops : min= 248, max= 532, avg=430.40, stdev=112.56, samples=5 write: IOPS=452, BW=1810KiB/s (1854kB/s)(5192KiB/2868msec) slat (usec): min=13, max=1304, avg=66.66, stdev=52.20 clat (usec): min=3521, max=41699, avg=17454.82, stdev=5593.06 lat (usec): min=3576, max=41746, avg=17523.05, stdev=5602.83 clat percentiles (usec): | 1.00th=[11600], 5.00th=[12780], 10.00th=[13566], 20.00th=[14222], | 30.00th=[14746], 40.00th=[15270], 50.00th=[15795], 60.00th=[16319], | 70.00th=[17171], 80.00th=[18482], 90.00th=[23462], 95.00th=[33424], | 99.00th=[36963], 99.50th=[39060], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 920, max= 2112, per=24.91%, avg=1798.40, stdev=500.51, samples=5 iops : min= 230, max= 528, avg=449.60, stdev=125.13, samples=5 lat (msec) : 4=0.27%, 10=0.16%, 20=84.45%, 50=15.12% cpu : usr=1.15%, sys=11.02%, ctx=2277, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32640: Wed Apr 9 02:17:25 2025 read: IOPS=430, BW=1723KiB/s (1764kB/s)(4976KiB/2888msec) slat (usec): min=601, max=4342, avg=1177.77, stdev=414.32 clat (usec): min=8977, max=42721, avg=16921.54, stdev=5352.40 lat (usec): min=10312, max=45522, avg=18102.19, stdev=5660.04 clat percentiles (usec): | 1.00th=[11076], 5.00th=[12125], 10.00th=[12649], 20.00th=[13435], | 30.00th=[14091], 40.00th=[14746], 50.00th=[15533], 60.00th=[16188], | 70.00th=[17171], 80.00th=[18220], 90.00th=[23462], 95.00th=[30278], | 99.00th=[36439], 99.50th=[37487], 99.90th=[40633], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 824, max= 2088, per=24.18%, avg=1676.80, stdev=507.78, samples=5 iops : min= 206, max= 522, avg=419.20, stdev=126.95, samples=5 write: IOPS=455, BW=1823KiB/s (1866kB/s)(5264KiB/2888msec) slat (usec): min=13, max=486, avg=64.51, stdev=37.97 clat (usec): min=2459, max=40573, avg=17548.90, stdev=5578.59 lat (usec): min=2514, max=40604, avg=17614.78, stdev=5586.06 clat percentiles (usec): | 1.00th=[11731], 5.00th=[12911], 10.00th=[13566], 20.00th=[14222], | 30.00th=[14746], 40.00th=[15401], 50.00th=[15795], 60.00th=[16319], | 70.00th=[16909], 80.00th=[18482], 90.00th=[26346], 95.00th=[33424], | 99.00th=[36963], 99.50th=[37487], 99.90th=[38011], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 920, max= 2208, per=24.99%, avg=1804.80, stdev=511.99, samples=5 iops : min= 230, max= 552, avg=451.20, stdev=128.00, samples=5 lat (msec) : 4=0.04%, 10=0.08%, 20=85.35%, 50=14.53% cpu : usr=1.42%, sys=10.70%, ctx=2224, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32641: Wed Apr 9 02:17:25 2025 read: IOPS=430, BW=1723KiB/s (1764kB/s)(4984KiB/2893msec) slat (usec): min=591, max=4319, avg=1190.79, stdev=414.08 clat (usec): min=2082, max=45863, avg=17066.27, stdev=5643.96 lat (usec): min=3371, max=48366, avg=18259.84, stdev=5941.97 clat percentiles (usec): | 1.00th=[10552], 5.00th=[11731], 10.00th=[12387], 20.00th=[13435], | 30.00th=[14353], 40.00th=[15008], 50.00th=[15533], 60.00th=[16188], | 70.00th=[17171], 80.00th=[18744], 90.00th=[23462], 95.00th=[31589], | 99.00th=[37487], 99.50th=[39060], 99.90th=[42206], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 824, max= 1976, per=24.27%, avg=1683.20, stdev=484.14, samples=5 iops : min= 206, max= 494, avg=420.80, stdev=121.03, samples=5 write: IOPS=454, BW=1817KiB/s (1860kB/s)(5256KiB/2893msec) slat (usec): min=15, max=776, avg=68.09, stdev=46.33 clat (usec): min=8429, max=40900, avg=17489.28, stdev=5655.94 lat (usec): min=8521, max=40963, avg=17558.87, stdev=5665.25 clat percentiles (usec): | 1.00th=[11731], 5.00th=[12780], 10.00th=[13435], 20.00th=[14091], | 30.00th=[14746], 40.00th=[15270], 50.00th=[15795], 60.00th=[16450], | 70.00th=[17171], 80.00th=[18482], 90.00th=[23200], 95.00th=[33817], | 99.00th=[37487], 99.50th=[39584], 99.90th=[40633], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 896, max= 2328, per=24.75%, avg=1787.20, stdev=562.67, samples=5 iops : min= 224, max= 582, avg=446.80, stdev=140.67, samples=5 lat (msec) : 4=0.04%, 10=0.31%, 20=84.57%, 50=15.08% cpu : usr=1.38%, sys=11.07%, ctx=2270, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32642: Wed Apr 9 02:17:25 2025 read: IOPS=437, BW=1749KiB/s (1791kB/s)(5060KiB/2893msec) slat (usec): min=599, max=3316, avg=1168.42, stdev=401.47 clat (usec): min=7428, max=40223, avg=16953.99, stdev=5410.17 lat (usec): min=8554, max=42346, avg=18124.84, stdev=5712.65 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11863], 10.00th=[12649], 20.00th=[13435], | 30.00th=[14222], 40.00th=[15008], 50.00th=[15533], 60.00th=[16188], | 70.00th=[16909], 80.00th=[18482], 90.00th=[22938], 95.00th=[31589], | 99.00th=[36963], 99.50th=[38536], 99.90th=[39584], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 808, max= 2104, per=24.66%, avg=1710.40, stdev=525.00, samples=5 iops : min= 202, max= 526, avg=427.60, stdev=131.25, samples=5 write: IOPS=447, BW=1791KiB/s (1834kB/s)(5180KiB/2893msec) slat (usec): min=13, max=2049, avg=68.44, stdev=69.03 clat (usec): min=2322, max=41508, avg=17484.60, stdev=5694.97 lat (usec): min=2426, max=41609, avg=17554.66, stdev=5709.88 clat percentiles (usec): | 1.00th=[11600], 5.00th=[12911], 10.00th=[13566], 20.00th=[14222], | 30.00th=[14877], 40.00th=[15401], 50.00th=[15795], 60.00th=[16319], | 70.00th=[16909], 80.00th=[18220], 90.00th=[22938], 95.00th=[33162], | 99.00th=[38011], 99.50th=[39060], 99.90th=[39584], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 864, max= 2096, per=24.31%, avg=1755.20, stdev=511.08, samples=5 iops : min= 216, max= 524, avg=438.80, stdev=127.77, samples=5 lat (msec) : 4=0.08%, 10=0.27%, 20=85.27%, 50=14.38% cpu : usr=1.24%, sys=10.93%, ctx=2256, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6937KiB/s (7103kB/s), 1723KiB/s-1760KiB/s (1764kB/s-1802kB/s), io=19.6MiB (20.5MB), run=2868-2893msec WRITE: bw=7222KiB/s (7395kB/s), 1791KiB/s-1823KiB/s (1834kB/s-1866kB/s), io=20.4MiB (21.4MB), run=2868-2893msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=32648: Wed Apr 9 02:17:26 2025 read: IOPS=6, BW=268MiB/s (281MB/s)(40.0MiB/149msec) slat (nsec): min=5714.6k, max=5714.6k, avg=5714578.00, stdev= 0.00 clat (nsec): min=135713k, max=135713k, avg=135712825.00, stdev= 0.00 lat (nsec): min=141437k, max=141437k, avg=141437353.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 136], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.05%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=268MiB/s (281MB/s), 268MiB/s-268MiB/s (281MB/s-281MB/s), io=40.0MiB (41.9MB), run=149-149msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout