== sanity test 398c: run fio to test AIO ================= 13:32:38 (1746120758) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.136438 s, 307 MB/s osc.lustre-OST0000-osc-ffff880095a71000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=1371: Thu May 1 13:32:47 2025 write: IOPS=378, BW=1514KiB/s (1550kB/s)(10.0MiB/6765msec) slat (usec): min=15, max=578, avg=96.02, stdev=50.35 clat (usec): min=5566, max=70586, avg=42086.94, stdev=4227.58 lat (usec): min=5678, max=70750, avg=42185.46, stdev=4230.51 clat percentiles (usec): | 1.00th=[27132], 5.00th=[34866], 10.00th=[38011], 20.00th=[40109], | 30.00th=[41157], 40.00th=[42206], 50.00th=[43254], 60.00th=[43779], | 70.00th=[44303], 80.00th=[44827], 90.00th=[45351], 95.00th=[45876], | 99.00th=[48497], 99.50th=[49546], 99.90th=[61080], 99.95th=[66323], | 99.99th=[70779] bw ( KiB/s): min= 1416, max= 1696, per=25.13%, avg=1512.00, stdev=80.12, samples=13 iops : min= 354, max= 424, avg=378.00, stdev=20.03, samples=13 lat (msec) : 10=0.16%, 20=0.35%, 50=99.10%, 100=0.39% cpu : usr=0.69%, sys=4.12%, ctx=2079, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=1372: Thu May 1 13:32:47 2025 write: IOPS=376, BW=1505KiB/s (1541kB/s)(10.0MiB/6803msec) slat (usec): min=14, max=439, avg=87.64, stdev=49.99 clat (usec): min=23823, max=71899, avg=42312.35, stdev=3680.42 lat (usec): min=23881, max=72020, avg=42402.11, stdev=3683.93 clat percentiles (usec): | 1.00th=[26608], 5.00th=[35390], 10.00th=[38536], 20.00th=[40109], | 30.00th=[41157], 40.00th=[42206], 50.00th=[43254], 60.00th=[43779], | 70.00th=[44303], 80.00th=[44827], 90.00th=[45351], 95.00th=[46400], | 99.00th=[47973], 99.50th=[50070], 99.90th=[62653], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 1416, max= 1664, per=25.01%, avg=1504.62, stdev=64.14, samples=13 iops : min= 354, max= 416, avg=376.15, stdev=16.04, samples=13 lat (msec) : 50=99.53%, 100=0.47% cpu : usr=0.54%, sys=3.78%, ctx=1681, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=1373: Thu May 1 13:32:47 2025 write: IOPS=376, BW=1505KiB/s (1541kB/s)(10.0MiB/6803msec) slat (usec): min=17, max=386, avg=88.48, stdev=48.05 clat (usec): min=10080, max=80465, avg=42271.84, stdev=3875.65 lat (usec): min=10219, max=80513, avg=42362.49, stdev=3880.75 clat percentiles (usec): | 1.00th=[28443], 5.00th=[35390], 10.00th=[38011], 20.00th=[40109], | 30.00th=[41157], 40.00th=[42206], 50.00th=[43254], 60.00th=[43779], | 70.00th=[44303], 80.00th=[44827], 90.00th=[45351], 95.00th=[45876], | 99.00th=[47973], 99.50th=[49021], 99.90th=[76022], 99.95th=[76022], | 99.99th=[80217] bw ( KiB/s): min= 1424, max= 1664, per=25.01%, avg=1504.62, stdev=70.69, samples=13 iops : min= 356, max= 416, avg=376.15, stdev=17.67, samples=13 lat (msec) : 20=0.04%, 50=99.57%, 100=0.39% cpu : usr=0.51%, sys=3.85%, ctx=1596, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=1374: Thu May 1 13:32:47 2025 write: IOPS=376, BW=1504KiB/s (1540kB/s)(10.0MiB/6807msec) slat (usec): min=13, max=464, avg=84.08, stdev=47.15 clat (usec): min=8656, max=66451, avg=42264.02, stdev=3494.25 lat (usec): min=8769, max=66501, avg=42350.16, stdev=3499.14 clat percentiles (usec): | 1.00th=[30802], 5.00th=[35914], 10.00th=[38011], 20.00th=[39584], | 30.00th=[41157], 40.00th=[42206], 50.00th=[43254], 60.00th=[43779], | 70.00th=[44303], 80.00th=[44827], 90.00th=[45351], 95.00th=[45876], | 99.00th=[48497], 99.50th=[49546], 99.90th=[53216], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 1408, max= 1608, per=24.94%, avg=1500.92, stdev=67.29, samples=13 iops : min= 352, max= 402, avg=375.23, stdev=16.82, samples=13 lat (msec) : 10=0.04%, 50=99.53%, 100=0.43% cpu : usr=0.53%, sys=3.69%, ctx=1433, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=6017KiB/s (6162kB/s), 1504KiB/s-1514KiB/s (1540kB/s-1550kB/s), io=40.0MiB (41.9MB), run=6765-6807msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1390: Thu May 1 13:32:53 2025 read: IOPS=256, BW=1027KiB/s (1051kB/s)(5048KiB/4917msec) slat (usec): min=689, max=6336, avg=1999.19, stdev=465.42 clat (usec): min=10707, max=53175, avg=29016.94, stdev=5981.96 lat (usec): min=12523, max=55496, avg=31019.69, stdev=6172.06 clat percentiles (usec): | 1.00th=[13960], 5.00th=[17695], 10.00th=[20317], 20.00th=[24511], | 30.00th=[26608], 40.00th=[28181], 50.00th=[29492], 60.00th=[30802], | 70.00th=[32113], 80.00th=[33817], 90.00th=[35914], 95.00th=[37487], | 99.00th=[42730], 99.50th=[47449], 99.90th=[51643], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 832, max= 1184, per=24.85%, avg=1014.00, stdev=112.85, samples=9 iops : min= 208, max= 296, avg=253.44, stdev=28.27, samples=9 write: IOPS=263, BW=1056KiB/s (1081kB/s)(5192KiB/4917msec) slat (usec): min=14, max=2120, avg=86.36, stdev=74.27 clat (usec): min=12453, max=53258, avg=29936.61, stdev=5732.99 lat (usec): min=12521, max=53327, avg=30024.71, stdev=5740.02 clat percentiles (usec): | 1.00th=[15664], 5.00th=[17957], 10.00th=[21627], 20.00th=[26346], | 30.00th=[27919], 40.00th=[29492], 50.00th=[30540], 60.00th=[31589], | 70.00th=[32900], 80.00th=[34341], 90.00th=[36439], 95.00th=[38011], | 99.00th=[42730], 99.50th=[44827], 99.90th=[52691], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 904, max= 1416, per=24.77%, avg=1052.11, stdev=158.39, samples=9 iops : min= 226, max= 354, avg=263.00, stdev=39.60, samples=9 lat (msec) : 20=9.10%, 50=90.59%, 100=0.31% cpu : usr=0.96%, sys=7.89%, ctx=2601, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1391: Thu May 1 13:32:53 2025 read: IOPS=252, BW=1012KiB/s (1036kB/s)(4976KiB/4917msec) slat (usec): min=711, max=11676, avg=2006.73, stdev=530.95 clat (usec): min=5084, max=58735, avg=29079.39, stdev=6013.36 lat (usec): min=7879, max=61382, avg=31089.44, stdev=6221.66 clat percentiles (usec): | 1.00th=[14746], 5.00th=[17695], 10.00th=[21103], 20.00th=[24511], | 30.00th=[26608], 40.00th=[28181], 50.00th=[29754], 60.00th=[31065], | 70.00th=[32375], 80.00th=[33817], 90.00th=[35914], 95.00th=[37487], | 99.00th=[41681], 99.50th=[46924], 99.90th=[56361], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 880, max= 1320, per=24.46%, avg=998.22, stdev=134.15, samples=9 iops : min= 220, max= 330, avg=249.56, stdev=33.54, samples=9 write: IOPS=267, BW=1071KiB/s (1096kB/s)(5264KiB/4917msec) slat (usec): min=13, max=360, avg=80.90, stdev=44.00 clat (usec): min=6316, max=54264, avg=29817.90, stdev=5827.02 lat (usec): min=6378, max=54327, avg=29900.52, stdev=5830.86 clat percentiles (usec): | 1.00th=[15401], 5.00th=[17171], 10.00th=[21103], 20.00th=[26608], | 30.00th=[27919], 40.00th=[29492], 50.00th=[30540], 60.00th=[31327], | 70.00th=[32637], 80.00th=[33817], 90.00th=[35914], 95.00th=[38536], | 99.00th=[42730], 99.50th=[45876], 99.90th=[51119], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 840, max= 1264, per=25.17%, avg=1069.33, stdev=136.29, samples=9 iops : min= 210, max= 316, avg=267.33, stdev=34.07, samples=9 lat (msec) : 10=0.08%, 20=8.32%, 50=91.37%, 100=0.23% cpu : usr=1.00%, sys=7.81%, ctx=2549, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1392: Thu May 1 13:32:53 2025 read: IOPS=253, BW=1014KiB/s (1038kB/s)(4984KiB/4915msec) slat (usec): min=732, max=12192, avg=2004.76, stdev=562.29 clat (usec): min=6068, max=54107, avg=28805.12, stdev=5900.56 lat (usec): min=8442, max=56917, avg=30813.26, stdev=6126.04 clat percentiles (usec): | 1.00th=[14484], 5.00th=[17695], 10.00th=[20055], 20.00th=[24773], | 30.00th=[26346], 40.00th=[28181], 50.00th=[29230], 60.00th=[30540], | 70.00th=[31851], 80.00th=[33424], 90.00th=[35390], 95.00th=[36963], | 99.00th=[42206], 99.50th=[44827], 99.90th=[50594], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 800, max= 1288, per=24.66%, avg=1006.22, stdev=150.13, samples=9 iops : min= 200, max= 322, avg=251.56, stdev=37.53, samples=9 write: IOPS=267, BW=1069KiB/s (1095kB/s)(5256KiB/4915msec) slat (usec): min=13, max=510, avg=83.57, stdev=46.98 clat (usec): min=13393, max=55918, avg=30079.66, stdev=5717.13 lat (usec): min=13489, max=55959, avg=30164.93, stdev=5723.23 clat percentiles (usec): | 1.00th=[14746], 5.00th=[17957], 10.00th=[22676], 20.00th=[26608], | 30.00th=[28181], 40.00th=[29492], 50.00th=[30540], 60.00th=[31589], | 70.00th=[32900], 80.00th=[34341], 90.00th=[36439], 95.00th=[38011], | 99.00th=[43779], 99.50th=[47449], 99.90th=[53740], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 936, max= 1320, per=25.03%, avg=1063.11, stdev=114.15, samples=9 iops : min= 234, max= 330, avg=265.78, stdev=28.54, samples=9 lat (msec) : 10=0.04%, 20=8.71%, 50=91.05%, 100=0.20% cpu : usr=0.90%, sys=7.83%, ctx=2605, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1393: Thu May 1 13:32:53 2025 read: IOPS=259, BW=1038KiB/s (1063kB/s)(5060KiB/4874msec) slat (usec): min=33, max=12418, avg=2003.82, stdev=572.57 clat (usec): min=2681, max=56096, avg=28914.88, stdev=6275.24 lat (usec): min=3046, max=58965, avg=30921.93, stdev=6519.40 clat percentiles (usec): | 1.00th=[13435], 5.00th=[17171], 10.00th=[19792], 20.00th=[24249], | 30.00th=[26608], 40.00th=[28443], 50.00th=[29492], 60.00th=[30802], | 70.00th=[32113], 80.00th=[33817], 90.00th=[35914], 95.00th=[38011], | 99.00th=[43254], 99.50th=[47973], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 840, max= 1240, per=25.28%, avg=1031.67, stdev=127.55, samples=9 iops : min= 210, max= 310, avg=257.89, stdev=31.89, samples=9 write: IOPS=265, BW=1063KiB/s (1088kB/s)(5180KiB/4874msec) slat (usec): min=14, max=1373, avg=83.35, stdev=65.15 clat (usec): min=4672, max=49452, avg=29797.78, stdev=5733.55 lat (usec): min=4774, max=49560, avg=29882.85, stdev=5736.84 clat percentiles (usec): | 1.00th=[14746], 5.00th=[17695], 10.00th=[21890], 20.00th=[26346], | 30.00th=[27919], 40.00th=[29230], 50.00th=[30540], 60.00th=[31589], | 70.00th=[32637], 80.00th=[34341], 90.00th=[35914], 95.00th=[37487], | 99.00th=[42206], 99.50th=[46924], 99.90th=[49021], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 896, max= 1384, per=24.90%, avg=1057.56, stdev=155.38, samples=9 iops : min= 224, max= 346, avg=264.33, stdev=38.88, samples=9 lat (msec) : 4=0.16%, 10=0.08%, 20=9.06%, 50=90.62%, 100=0.08% cpu : usr=0.88%, sys=7.92%, ctx=2713, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4081KiB/s (4179kB/s), 1012KiB/s-1038KiB/s (1036kB/s-1063kB/s), io=19.6MiB (20.5MB), run=4874-4917msec WRITE: bw=4249KiB/s (4351kB/s), 1056KiB/s-1071KiB/s (1081kB/s-1096kB/s), io=20.4MiB (21.4MB), run=4874-4917msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=1400: Thu May 1 13:32:54 2025 read: IOPS=5, BW=237MiB/s (248MB/s)(40.0MiB/169msec) slat (nsec): min=5819.7k, max=5819.7k, avg=5819731.00, stdev= 0.00 clat (nsec): min=155844k, max=155844k, avg=155843992.00, stdev= 0.00 lat (nsec): min=161674k, max=161674k, avg=161673806.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 157], 50.00th=[ 157], 60.00th=[ 157], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=3.57%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=237MiB/s (248MB/s), 237MiB/s-237MiB/s (248MB/s-248MB/s), io=40.0MiB (41.9MB), run=169-169msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout