== sanity test 398c: run fio to test AIO ================= 04:48:58 (1738489738) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.114581 s, 366 MB/s osc.lustre-OST0000-osc-ffff880080496000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=28455: Sun Feb 2 04:49:05 2025 write: IOPS=502, BW=2012KiB/s (2060kB/s)(10.0MiB/5090msec) slat (usec): min=11, max=763, avg=64.88, stdev=57.66 clat (usec): min=4801, max=49398, avg=31608.38, stdev=6757.59 lat (usec): min=4919, max=49435, avg=31674.82, stdev=6764.66 clat percentiles (usec): | 1.00th=[15008], 5.00th=[22152], 10.00th=[23462], 20.00th=[25297], | 30.00th=[27657], 40.00th=[29754], 50.00th=[31065], 60.00th=[32637], | 70.00th=[34866], 80.00th=[37487], 90.00th=[40633], 95.00th=[44303], | 99.00th=[47973], 99.50th=[48497], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 1536, max= 2592, per=24.80%, avg=1995.60, stdev=311.02, samples=10 iops : min= 384, max= 648, avg=498.90, stdev=77.76, samples=10 lat (msec) : 10=0.08%, 20=1.84%, 50=98.09% cpu : usr=1.08%, sys=3.18%, ctx=1672, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=28456: Sun Feb 2 04:49:05 2025 write: IOPS=503, BW=2013KiB/s (2062kB/s)(10.0MiB/5086msec) slat (usec): min=11, max=756, avg=61.96, stdev=55.63 clat (usec): min=4636, max=49578, avg=31610.10, stdev=6714.43 lat (usec): min=4671, max=49620, avg=31673.54, stdev=6719.27 clat percentiles (usec): | 1.00th=[15008], 5.00th=[21890], 10.00th=[23462], 20.00th=[25297], | 30.00th=[27919], 40.00th=[29754], 50.00th=[31065], 60.00th=[32375], | 70.00th=[34866], 80.00th=[36963], 90.00th=[41157], 95.00th=[44303], | 99.00th=[46924], 99.50th=[47973], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 1544, max= 2592, per=24.84%, avg=1999.20, stdev=305.69, samples=10 iops : min= 386, max= 648, avg=499.80, stdev=76.42, samples=10 lat (msec) : 10=0.16%, 20=2.03%, 50=97.81% cpu : usr=0.31%, sys=3.87%, ctx=1539, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=28457: Sun Feb 2 04:49:05 2025 write: IOPS=502, BW=2012KiB/s (2060kB/s)(10.0MiB/5090msec) slat (usec): min=10, max=1022, avg=61.54, stdev=57.90 clat (usec): min=4386, max=49528, avg=31584.40, stdev=6633.75 lat (usec): min=4474, max=49560, avg=31647.23, stdev=6638.58 clat percentiles (usec): | 1.00th=[18744], 5.00th=[22414], 10.00th=[23462], 20.00th=[25297], | 30.00th=[27657], 40.00th=[29492], 50.00th=[31065], 60.00th=[32113], | 70.00th=[34866], 80.00th=[36963], 90.00th=[41157], 95.00th=[44303], | 99.00th=[47973], 99.50th=[48497], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 1536, max= 2568, per=24.78%, avg=1994.00, stdev=301.61, samples=10 iops : min= 384, max= 642, avg=498.50, stdev=75.40, samples=10 lat (msec) : 10=0.12%, 20=1.41%, 50=98.48% cpu : usr=0.63%, sys=3.64%, ctx=1453, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=28458: Sun Feb 2 04:49:05 2025 write: IOPS=505, BW=2021KiB/s (2070kB/s)(10.0MiB/5066msec) slat (usec): min=11, max=1122, avg=67.64, stdev=66.98 clat (usec): min=3042, max=49929, avg=31488.62, stdev=6884.12 lat (usec): min=3254, max=49997, avg=31557.65, stdev=6889.93 clat percentiles (usec): | 1.00th=[14877], 5.00th=[21890], 10.00th=[23462], 20.00th=[25297], | 30.00th=[27657], 40.00th=[29754], 50.00th=[31327], 60.00th=[32637], | 70.00th=[34866], 80.00th=[36963], 90.00th=[41157], 95.00th=[43254], | 99.00th=[48497], 99.50th=[48497], 99.90th=[49546], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 1536, max= 2592, per=25.00%, avg=2011.60, stdev=313.55, samples=10 iops : min= 384, max= 648, avg=502.90, stdev=78.39, samples=10 lat (msec) : 4=0.04%, 10=0.51%, 20=1.48%, 50=97.97% cpu : usr=0.57%, sys=3.85%, ctx=1662, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=8047KiB/s (8240kB/s), 2012KiB/s-2021KiB/s (2060kB/s-2070kB/s), io=40.0MiB (41.9MB), run=5066-5090msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28472: Sun Feb 2 04:49:10 2025 read: IOPS=317, BW=1271KiB/s (1301kB/s)(5048KiB/3972msec) slat (usec): min=23, max=5086, avg=1489.65, stdev=396.67 clat (usec): min=1747, max=57414, avg=23720.93, stdev=4987.00 lat (usec): min=2568, max=58810, avg=25212.98, stdev=5133.06 clat percentiles (usec): | 1.00th=[ 9372], 5.00th=[17171], 10.00th=[18220], 20.00th=[20055], | 30.00th=[21103], 40.00th=[22152], 50.00th=[23462], 60.00th=[24511], | 70.00th=[25822], 80.00th=[27395], 90.00th=[29492], 95.00th=[31589], | 99.00th=[36963], 99.50th=[40109], 99.90th=[50070], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 1208, max= 1424, per=25.71%, avg=1286.43, stdev=74.75, samples=7 iops : min= 302, max= 356, avg=321.57, stdev=18.67, samples=7 write: IOPS=326, BW=1307KiB/s (1339kB/s)(5192KiB/3972msec) slat (usec): min=12, max=1663, avg=55.66, stdev=57.72 clat (usec): min=6207, max=50215, avg=24286.59, stdev=4036.66 lat (usec): min=6237, max=50280, avg=24343.54, stdev=4038.44 clat percentiles (usec): | 1.00th=[13435], 5.00th=[19006], 10.00th=[19792], 20.00th=[21365], | 30.00th=[22414], 40.00th=[23200], 50.00th=[23987], 60.00th=[25035], | 70.00th=[25822], 80.00th=[27132], 90.00th=[28967], 95.00th=[31065], | 99.00th=[34341], 99.50th=[35390], 99.90th=[47973], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 1272, max= 1488, per=25.84%, avg=1345.86, stdev=89.46, samples=7 iops : min= 318, max= 372, avg=336.43, stdev=22.39, samples=7 lat (msec) : 2=0.04%, 4=0.16%, 10=0.59%, 20=14.10%, 50=85.00% lat (msec) : 100=0.12% cpu : usr=1.51%, sys=5.97%, ctx=2399, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28473: Sun Feb 2 04:49:10 2025 read: IOPS=310, BW=1242KiB/s (1272kB/s)(4976KiB/4007msec) slat (usec): min=823, max=5010, avg=1483.31, stdev=358.85 clat (usec): min=2072, max=46751, avg=23862.57, stdev=4279.72 lat (usec): min=3003, max=48625, avg=25348.02, stdev=4371.06 clat percentiles (usec): | 1.00th=[15533], 5.00th=[17695], 10.00th=[18744], 20.00th=[20317], | 30.00th=[21627], 40.00th=[22676], 50.00th=[23725], 60.00th=[24511], | 70.00th=[25822], 80.00th=[27132], 90.00th=[28967], 95.00th=[30540], | 99.00th=[35390], 99.50th=[37487], 99.90th=[42206], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 1088, max= 1312, per=24.68%, avg=1235.00, stdev=68.94, samples=8 iops : min= 272, max= 328, avg=308.75, stdev=17.24, samples=8 write: IOPS=328, BW=1314KiB/s (1345kB/s)(5264KiB/4007msec) slat (usec): min=12, max=1688, avg=53.21, stdev=54.04 clat (usec): min=15794, max=48631, avg=24436.91, stdev=3655.09 lat (usec): min=15830, max=48681, avg=24491.58, stdev=3657.15 clat percentiles (usec): | 1.00th=[17433], 5.00th=[19006], 10.00th=[20055], 20.00th=[21365], | 30.00th=[22414], 40.00th=[23200], 50.00th=[24249], 60.00th=[25035], | 70.00th=[26084], 80.00th=[27132], 90.00th=[28967], 95.00th=[31065], | 99.00th=[34341], 99.50th=[35914], 99.90th=[46924], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 1064, max= 1488, per=25.15%, avg=1310.00, stdev=130.95, samples=8 iops : min= 266, max= 372, avg=327.50, stdev=32.74, samples=8 lat (msec) : 4=0.08%, 10=0.04%, 20=13.09%, 50=86.80% cpu : usr=1.35%, sys=6.14%, ctx=2404, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28474: Sun Feb 2 04:49:10 2025 read: IOPS=310, BW=1243KiB/s (1273kB/s)(4984KiB/4010msec) slat (usec): min=809, max=3352, avg=1483.16, stdev=363.22 clat (usec): min=1912, max=46338, avg=23827.51, stdev=4274.81 lat (usec): min=4079, max=48136, avg=25313.11, stdev=4388.88 clat percentiles (usec): | 1.00th=[15270], 5.00th=[17695], 10.00th=[19006], 20.00th=[20579], | 30.00th=[21365], 40.00th=[22414], 50.00th=[23462], 60.00th=[24511], | 70.00th=[25560], 80.00th=[27132], 90.00th=[29230], 95.00th=[30802], | 99.00th=[35914], 99.50th=[38011], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 1072, max= 1336, per=24.80%, avg=1241.00, stdev=87.79, samples=8 iops : min= 268, max= 334, avg=310.25, stdev=21.95, samples=8 write: IOPS=327, BW=1311KiB/s (1342kB/s)(5256KiB/4010msec) slat (usec): min=14, max=1443, avg=56.37, stdev=59.02 clat (usec): min=14229, max=37937, avg=24420.95, stdev=3672.62 lat (usec): min=14290, max=38008, avg=24478.53, stdev=3675.86 clat percentiles (usec): | 1.00th=[17171], 5.00th=[19006], 10.00th=[19792], 20.00th=[21365], | 30.00th=[22414], 40.00th=[23200], 50.00th=[23987], 60.00th=[25035], | 70.00th=[26084], 80.00th=[27395], 90.00th=[29754], 95.00th=[31065], | 99.00th=[33817], 99.50th=[33817], 99.90th=[36963], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 1112, max= 1480, per=25.03%, avg=1304.00, stdev=102.45, samples=8 iops : min= 278, max= 370, avg=326.00, stdev=25.61, samples=8 lat (msec) : 2=0.04%, 10=0.04%, 20=13.48%, 50=86.45% cpu : usr=1.05%, sys=6.51%, ctx=2336, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28475: Sun Feb 2 04:49:10 2025 read: IOPS=317, BW=1269KiB/s (1299kB/s)(5060KiB/3988msec) slat (usec): min=821, max=4908, avg=1481.86, stdev=359.03 clat (usec): min=8238, max=47640, avg=23793.20, stdev=4293.64 lat (usec): min=9432, max=49877, avg=25277.42, stdev=4377.27 clat percentiles (usec): | 1.00th=[15139], 5.00th=[17433], 10.00th=[18744], 20.00th=[20317], | 30.00th=[21365], 40.00th=[22414], 50.00th=[23462], 60.00th=[24511], | 70.00th=[25560], 80.00th=[26870], 90.00th=[29230], 95.00th=[30802], | 99.00th=[36439], 99.50th=[39060], 99.90th=[44303], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 1224, max= 1384, per=25.94%, avg=1298.29, stdev=63.45, samples=7 iops : min= 306, max= 346, avg=324.57, stdev=15.86, samples=7 write: IOPS=324, BW=1299KiB/s (1330kB/s)(5180KiB/3988msec) slat (usec): min=12, max=566, avg=55.03, stdev=42.27 clat (usec): min=11233, max=50658, avg=24284.69, stdev=3936.64 lat (usec): min=11261, max=50741, avg=24340.91, stdev=3938.84 clat percentiles (usec): | 1.00th=[16057], 5.00th=[18744], 10.00th=[19792], 20.00th=[21365], | 30.00th=[22152], 40.00th=[22938], 50.00th=[23987], 60.00th=[24773], | 70.00th=[26084], 80.00th=[27395], 90.00th=[29230], 95.00th=[31065], | 99.00th=[34341], 99.50th=[35914], 99.90th=[50594], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 1224, max= 1408, per=25.30%, avg=1317.71, stdev=74.15, samples=7 iops : min= 306, max= 352, avg=329.43, stdev=18.54, samples=7 lat (msec) : 10=0.12%, 20=13.87%, 50=85.94%, 100=0.08% cpu : usr=1.00%, sys=6.52%, ctx=2404, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5004KiB/s (5125kB/s), 1242KiB/s-1271KiB/s (1272kB/s-1301kB/s), io=19.6MiB (20.5MB), run=3972-4010msec WRITE: bw=5210KiB/s (5335kB/s), 1299KiB/s-1314KiB/s (1330kB/s-1345kB/s), io=20.4MiB (21.4MB), run=3972-4010msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=28482: Sun Feb 2 04:49:11 2025 read: IOPS=8, BW=333MiB/s (350MB/s)(40.0MiB/120msec) slat (nsec): min=8002.3k, max=8002.3k, avg=8002298.00, stdev= 0.00 clat (nsec): min=103774k, max=103774k, avg=103773838.00, stdev= 0.00 lat (nsec): min=111798k, max=111798k, avg=111798216.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 104], | 30.00th=[ 104], 40.00th=[ 104], 50.00th=[ 104], 60.00th=[ 104], | 70.00th=[ 104], 80.00th=[ 104], 90.00th=[ 104], 95.00th=[ 104], | 99.00th=[ 104], 99.50th=[ 104], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] lat (msec) : 250=100.00% cpu : usr=0.84%, sys=6.72%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=333MiB/s (350MB/s), 333MiB/s-333MiB/s (350MB/s-350MB/s), io=40.0MiB (41.9MB), run=120-120msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout