== sanity test 398c: run fio to test AIO ================= 14:33:28 (1744137208) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.112729 s, 372 MB/s osc.lustre-OST0000-osc-ffff8800b623c000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=31119: Tue Apr 8 14:33:37 2025 write: IOPS=391, BW=1565KiB/s (1603kB/s)(10.0MiB/6542msec) slat (usec): min=15, max=1043, avg=74.55, stdev=46.99 clat (usec): min=18510, max=66918, avg=40602.33, stdev=4941.35 lat (usec): min=18682, max=66991, avg=40678.60, stdev=4946.67 clat percentiles (usec): | 1.00th=[21365], 5.00th=[29754], 10.00th=[33424], 20.00th=[37487], | 30.00th=[39584], 40.00th=[40633], 50.00th=[41681], 60.00th=[42730], | 70.00th=[43779], 80.00th=[44303], 90.00th=[45351], 95.00th=[45876], | 99.00th=[46924], 99.50th=[49021], 99.90th=[50070], 99.95th=[53740], | 99.99th=[66847] bw ( KiB/s): min= 1408, max= 1792, per=24.92%, avg=1558.77, stdev=107.72, samples=13 iops : min= 352, max= 448, avg=389.69, stdev=26.93, samples=13 lat (msec) : 20=0.62%, 50=99.22%, 100=0.16% cpu : usr=0.50%, sys=3.67%, ctx=873, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=31120: Tue Apr 8 14:33:37 2025 write: IOPS=391, BW=1567KiB/s (1605kB/s)(10.0MiB/6533msec) slat (usec): min=17, max=394, avg=76.16, stdev=42.60 clat (usec): min=5334, max=65911, avg=40576.51, stdev=5188.71 lat (usec): min=5433, max=66015, avg=40654.43, stdev=5192.75 clat percentiles (usec): | 1.00th=[22414], 5.00th=[30540], 10.00th=[33817], 20.00th=[38011], | 30.00th=[39584], 40.00th=[40633], 50.00th=[41681], 60.00th=[42730], | 70.00th=[43779], 80.00th=[44303], 90.00th=[45351], 95.00th=[46400], | 99.00th=[47449], 99.50th=[49546], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 1381, max= 1792, per=25.01%, avg=1564.46, stdev=107.10, samples=13 iops : min= 345, max= 448, avg=391.08, stdev=26.83, samples=13 lat (msec) : 10=0.12%, 20=0.62%, 50=98.87%, 100=0.39% cpu : usr=0.61%, sys=3.72%, ctx=938, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=31121: Tue Apr 8 14:33:37 2025 write: IOPS=393, BW=1573KiB/s (1610kB/s)(10.0MiB/6511msec) slat (usec): min=18, max=455, avg=83.83, stdev=45.01 clat (usec): min=5780, max=78633, avg=40475.66, stdev=5622.74 lat (usec): min=5872, max=78692, avg=40561.33, stdev=5623.40 clat percentiles (usec): | 1.00th=[21365], 5.00th=[29230], 10.00th=[33817], 20.00th=[38011], | 30.00th=[39584], 40.00th=[40633], 50.00th=[41681], 60.00th=[42730], | 70.00th=[43779], 80.00th=[44303], 90.00th=[45351], 95.00th=[46400], | 99.00th=[49021], 99.50th=[50070], 99.90th=[69731], 99.95th=[73925], | 99.99th=[78119] bw ( KiB/s): min= 1317, max= 1912, per=25.02%, avg=1565.00, stdev=142.24, samples=13 iops : min= 329, max= 478, avg=391.23, stdev=35.60, samples=13 lat (msec) : 10=0.35%, 20=0.43%, 50=98.59%, 100=0.62% cpu : usr=0.58%, sys=4.16%, ctx=1287, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=31122: Tue Apr 8 14:33:37 2025 write: IOPS=391, BW=1564KiB/s (1602kB/s)(10.0MiB/6547msec) slat (usec): min=12, max=332, avg=65.59, stdev=40.37 clat (usec): min=18450, max=53135, avg=40603.09, stdev=4717.09 lat (usec): min=18495, max=53177, avg=40670.22, stdev=4718.95 clat percentiles (usec): | 1.00th=[25822], 5.00th=[30540], 10.00th=[33817], 20.00th=[38011], | 30.00th=[39060], 40.00th=[40109], 50.00th=[41157], 60.00th=[42730], | 70.00th=[43779], 80.00th=[44303], 90.00th=[45351], 95.00th=[46400], | 99.00th=[48497], 99.50th=[49546], 99.90th=[50070], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 1408, max= 1672, per=24.89%, avg=1557.23, stdev=101.53, samples=13 iops : min= 352, max= 418, avg=389.31, stdev=25.38, samples=13 lat (msec) : 20=0.16%, 50=99.45%, 100=0.39% cpu : usr=0.46%, sys=3.13%, ctx=678, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=6256KiB/s (6406kB/s), 1564KiB/s-1573KiB/s (1602kB/s-1610kB/s), io=40.0MiB (41.9MB), run=6511-6547msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31137: Tue Apr 8 14:33:40 2025 read: IOPS=472, BW=1891KiB/s (1936kB/s)(5048KiB/2670msec) slat (usec): min=665, max=3347, avg=1200.27, stdev=348.13 clat (usec): min=2083, max=40716, avg=15599.39, stdev=3996.51 lat (usec): min=3276, max=43499, avg=16802.59, stdev=4192.33 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11731], 10.00th=[12256], 20.00th=[13042], | 30.00th=[13566], 40.00th=[14222], 50.00th=[14877], 60.00th=[15401], | 70.00th=[16057], 80.00th=[17171], 90.00th=[19006], 95.00th=[23725], | 99.00th=[32113], 99.50th=[33817], 99.90th=[38536], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 1224, max= 2136, per=24.72%, avg=1858.20, stdev=367.20, samples=5 iops : min= 306, max= 534, avg=464.40, stdev=91.69, samples=5 write: IOPS=486, BW=1945KiB/s (1991kB/s)(5192KiB/2670msec) slat (usec): min=16, max=1191, avg=71.22, stdev=50.32 clat (usec): min=2038, max=36578, avg=15992.73, stdev=3892.82 lat (usec): min=2074, max=36615, avg=16065.52, stdev=3894.52 clat percentiles (usec): | 1.00th=[11600], 5.00th=[12649], 10.00th=[13173], 20.00th=[13829], | 30.00th=[14353], 40.00th=[14746], 50.00th=[15139], 60.00th=[15533], | 70.00th=[16188], 80.00th=[16909], 90.00th=[18220], 95.00th=[26084], | 99.00th=[32375], 99.50th=[34341], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 1200, max= 2155, per=24.71%, avg=1933.40, stdev=411.39, samples=5 iops : min= 300, max= 538, avg=483.20, stdev=102.75, samples=5 lat (msec) : 4=0.23%, 10=0.20%, 20=91.72%, 50=7.85% cpu : usr=1.31%, sys=12.14%, ctx=2488, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31138: Tue Apr 8 14:33:40 2025 read: IOPS=466, BW=1865KiB/s (1910kB/s)(4976KiB/2668msec) slat (usec): min=643, max=3241, avg=1200.77, stdev=344.88 clat (usec): min=2590, max=40474, avg=15753.77, stdev=4059.86 lat (usec): min=4297, max=42908, avg=16957.41, stdev=4258.85 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11731], 10.00th=[12387], 20.00th=[13173], | 30.00th=[13829], 40.00th=[14484], 50.00th=[15008], 60.00th=[15533], | 70.00th=[16188], 80.00th=[16909], 90.00th=[19268], 95.00th=[23725], | 99.00th=[32375], 99.50th=[33817], 99.90th=[39060], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 1125, max= 2080, per=24.37%, avg=1831.40, stdev=401.56, samples=5 iops : min= 281, max= 520, avg=457.80, stdev=100.50, samples=5 write: IOPS=493, BW=1973KiB/s (2020kB/s)(5264KiB/2668msec) slat (usec): min=17, max=1242, avg=70.89, stdev=59.74 clat (usec): min=8601, max=40360, avg=16044.91, stdev=4007.58 lat (usec): min=8679, max=40421, avg=16117.45, stdev=4010.16 clat percentiles (usec): | 1.00th=[11600], 5.00th=[12518], 10.00th=[13042], 20.00th=[13829], | 30.00th=[14222], 40.00th=[14746], 50.00th=[15139], 60.00th=[15533], | 70.00th=[16057], 80.00th=[16909], 90.00th=[19006], 95.00th=[25035], | 99.00th=[33424], 99.50th=[35390], 99.90th=[38536], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 1405, max= 2208, per=25.25%, avg=1975.40, stdev=334.22, samples=5 iops : min= 351, max= 552, avg=493.80, stdev=83.66, samples=5 lat (msec) : 4=0.08%, 10=0.23%, 20=90.86%, 50=8.83% cpu : usr=1.80%, sys=11.62%, ctx=2416, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31139: Tue Apr 8 14:33:40 2025 read: IOPS=468, BW=1873KiB/s (1918kB/s)(4984KiB/2661msec) slat (usec): min=634, max=3966, avg=1197.44, stdev=356.42 clat (usec): min=7917, max=36698, avg=15569.68, stdev=3797.22 lat (usec): min=8943, max=39150, avg=16769.91, stdev=4011.38 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11600], 10.00th=[12256], 20.00th=[13042], | 30.00th=[13698], 40.00th=[14353], 50.00th=[14877], 60.00th=[15401], | 70.00th=[16057], 80.00th=[16909], 90.00th=[19006], 95.00th=[23200], | 99.00th=[32113], 99.50th=[33817], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 1256, max= 2120, per=24.91%, avg=1872.60, stdev=360.40, samples=5 iops : min= 314, max= 530, avg=468.00, stdev=90.00, samples=5 write: IOPS=493, BW=1975KiB/s (2023kB/s)(5256KiB/2661msec) slat (usec): min=13, max=461, avg=70.36, stdev=38.86 clat (usec): min=9847, max=36285, avg=16022.34, stdev=3947.31 lat (usec): min=9906, max=36332, avg=16094.30, stdev=3951.20 clat percentiles (usec): | 1.00th=[11469], 5.00th=[12518], 10.00th=[13042], 20.00th=[13698], | 30.00th=[14222], 40.00th=[14746], 50.00th=[15139], 60.00th=[15533], | 70.00th=[16057], 80.00th=[16909], 90.00th=[18744], 95.00th=[26870], | 99.00th=[31851], 99.50th=[34341], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 1232, max= 2232, per=24.96%, avg=1952.60, stdev=412.58, samples=5 iops : min= 308, max= 558, avg=488.00, stdev=103.04, samples=5 lat (msec) : 10=0.27%, 20=91.37%, 50=8.36% cpu : usr=1.35%, sys=11.99%, ctx=2493, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31140: Tue Apr 8 14:33:40 2025 read: IOPS=476, BW=1907KiB/s (1953kB/s)(5060KiB/2653msec) slat (usec): min=26, max=3358, avg=1196.89, stdev=375.33 clat (usec): min=2997, max=43706, avg=15673.64, stdev=4124.69 lat (usec): min=3033, max=47068, avg=16873.20, stdev=4346.53 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11600], 10.00th=[12256], 20.00th=[13042], | 30.00th=[13698], 40.00th=[14353], 50.00th=[15008], 60.00th=[15533], | 70.00th=[16057], 80.00th=[16909], 90.00th=[19006], 95.00th=[23987], | 99.00th=[33817], 99.50th=[34866], 99.90th=[41681], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 1352, max= 2147, per=25.19%, avg=1893.40, stdev=318.42, samples=5 iops : min= 338, max= 536, avg=473.20, stdev=79.46, samples=5 write: IOPS=488, BW=1953KiB/s (1999kB/s)(5180KiB/2653msec) slat (usec): min=14, max=1254, avg=70.88, stdev=55.12 clat (usec): min=8288, max=36920, avg=16104.16, stdev=3952.38 lat (usec): min=8323, max=36975, avg=16176.65, stdev=3956.29 clat percentiles (usec): | 1.00th=[11338], 5.00th=[12649], 10.00th=[13173], 20.00th=[13829], | 30.00th=[14222], 40.00th=[14746], 50.00th=[15270], 60.00th=[15664], | 70.00th=[16188], 80.00th=[17171], 90.00th=[19006], 95.00th=[25822], | 99.00th=[33817], 99.50th=[34866], 99.90th=[36439], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 1328, max= 2120, per=24.85%, avg=1944.60, stdev=345.68, samples=5 iops : min= 332, max= 530, avg=486.00, stdev=86.34, samples=5 lat (msec) : 4=0.12%, 10=0.27%, 20=91.48%, 50=8.12% cpu : usr=1.66%, sys=11.58%, ctx=2516, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7516KiB/s (7696kB/s), 1865KiB/s-1907KiB/s (1910kB/s-1953kB/s), io=19.6MiB (20.5MB), run=2653-2670msec WRITE: bw=7825KiB/s (8013kB/s), 1945KiB/s-1975KiB/s (1991kB/s-2023kB/s), io=20.4MiB (21.4MB), run=2653-2670msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=31145: Tue Apr 8 14:33:42 2025 read: IOPS=6, BW=248MiB/s (261MB/s)(40.0MiB/161msec) slat (nsec): min=6398.8k, max=6398.8k, avg=6398811.00, stdev= 0.00 clat (nsec): min=148401k, max=148401k, avg=148400652.00, stdev= 0.00 lat (nsec): min=154812k, max=154812k, avg=154811629.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 148], 10.00th=[ 148], 20.00th=[ 148], | 30.00th=[ 148], 40.00th=[ 148], 50.00th=[ 148], 60.00th=[ 148], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.38%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=248MiB/s (261MB/s), 248MiB/s-248MiB/s (261MB/s-261MB/s), io=40.0MiB (41.9MB), run=161-161msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout