== sanity test 398c: run fio to test AIO ================= 03:38:45 (1746085125) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.111437 s, 376 MB/s osc.lustre-OST0000-osc-ffff880091eef000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=26447: Thu May 1 03:38:52 2025 write: IOPS=467, BW=1872KiB/s (1917kB/s)(10.0MiB/5471msec) slat (usec): min=11, max=2463, avg=75.15, stdev=97.89 clat (usec): min=3355, max=76275, avg=34010.30, stdev=11170.04 lat (usec): min=3380, max=76340, avg=34087.25, stdev=11179.51 clat percentiles (usec): | 1.00th=[16188], 5.00th=[20841], 10.00th=[22676], 20.00th=[25560], | 30.00th=[27395], 40.00th=[29230], 50.00th=[31589], 60.00th=[33817], | 70.00th=[37487], 80.00th=[41157], 90.00th=[49021], 95.00th=[55837], | 99.00th=[71828], 99.50th=[74974], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 1157, max= 2464, per=24.66%, avg=1839.20, stdev=465.48, samples=10 iops : min= 289, max= 616, avg=459.70, stdev=116.31, samples=10 lat (msec) : 4=0.04%, 10=0.23%, 20=3.52%, 50=87.46%, 100=8.75% cpu : usr=1.04%, sys=3.53%, ctx=1778, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=26448: Thu May 1 03:38:52 2025 write: IOPS=466, BW=1865KiB/s (1910kB/s)(10.0MiB/5490msec) slat (usec): min=11, max=1263, avg=67.26, stdev=85.94 clat (usec): min=10178, max=78444, avg=34058.56, stdev=11040.33 lat (usec): min=10238, max=78481, avg=34127.70, stdev=11051.70 clat percentiles (usec): | 1.00th=[18482], 5.00th=[21103], 10.00th=[22676], 20.00th=[25822], | 30.00th=[27919], 40.00th=[29230], 50.00th=[30802], 60.00th=[33817], | 70.00th=[37487], 80.00th=[40633], 90.00th=[47973], 95.00th=[57934], | 99.00th=[71828], 99.50th=[73925], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 1117, max= 2432, per=24.44%, avg=1822.90, stdev=453.83, samples=10 iops : min= 279, max= 608, avg=455.70, stdev=113.50, samples=10 lat (msec) : 20=2.19%, 50=90.08%, 100=7.73% cpu : usr=0.69%, sys=3.37%, ctx=1346, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=26449: Thu May 1 03:38:52 2025 write: IOPS=466, BW=1865KiB/s (1910kB/s)(10.0MiB/5491msec) slat (usec): min=12, max=1259, avg=70.32, stdev=78.77 clat (usec): min=3181, max=76764, avg=34084.02, stdev=11071.82 lat (usec): min=3263, max=76816, avg=34155.99, stdev=11081.10 clat percentiles (usec): | 1.00th=[16057], 5.00th=[20841], 10.00th=[23200], 20.00th=[25822], | 30.00th=[27657], 40.00th=[28967], 50.00th=[31327], 60.00th=[34341], | 70.00th=[37487], 80.00th=[41681], 90.00th=[47973], 95.00th=[56361], | 99.00th=[71828], 99.50th=[73925], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 1152, max= 2400, per=24.48%, avg=1826.30, stdev=448.96, samples=10 iops : min= 288, max= 600, avg=456.50, stdev=112.19, samples=10 lat (msec) : 4=0.04%, 20=3.24%, 50=88.12%, 100=8.59% cpu : usr=0.78%, sys=3.61%, ctx=1729, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=26450: Thu May 1 03:38:52 2025 write: IOPS=466, BW=1865KiB/s (1910kB/s)(10.0MiB/5490msec) slat (usec): min=13, max=1136, avg=73.18, stdev=84.37 clat (usec): min=15274, max=76474, avg=34111.38, stdev=10934.61 lat (usec): min=15321, max=76573, avg=34186.77, stdev=10946.72 clat percentiles (usec): | 1.00th=[17957], 5.00th=[20841], 10.00th=[23200], 20.00th=[26084], | 30.00th=[27657], 40.00th=[29230], 50.00th=[31327], 60.00th=[33817], | 70.00th=[37487], 80.00th=[41157], 90.00th=[47449], 95.00th=[56886], | 99.00th=[71828], 99.50th=[72877], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 1149, max= 2403, per=24.48%, avg=1825.60, stdev=451.30, samples=10 iops : min= 287, max= 600, avg=456.30, stdev=112.76, samples=10 lat (msec) : 20=2.77%, 50=88.91%, 100=8.32% cpu : usr=0.62%, sys=3.68%, ctx=1786, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=7459KiB/s (7639kB/s), 1865KiB/s-1872KiB/s (1910kB/s-1917kB/s), io=40.0MiB (41.9MB), run=5471-5491msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=26464: Thu May 1 03:38:58 2025 read: IOPS=290, BW=1162KiB/s (1190kB/s)(5048KiB/4345msec) slat (usec): min=708, max=6499, avg=1700.11, stdev=652.61 clat (usec): min=3224, max=55273, avg=25758.77, stdev=5966.73 lat (usec): min=4873, max=57201, avg=27462.25, stdev=6107.54 clat percentiles (usec): | 1.00th=[15270], 5.00th=[17695], 10.00th=[19006], 20.00th=[20841], | 30.00th=[22152], 40.00th=[23725], 50.00th=[25035], 60.00th=[26608], | 70.00th=[27919], 80.00th=[30278], 90.00th=[33424], 95.00th=[36439], | 99.00th=[43779], 99.50th=[44827], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 1048, max= 1392, per=25.04%, avg=1153.63, stdev=115.98, samples=8 iops : min= 262, max= 348, avg=288.38, stdev=29.02, samples=8 write: IOPS=298, BW=1195KiB/s (1224kB/s)(5192KiB/4345msec) slat (usec): min=14, max=1372, avg=60.55, stdev=75.07 clat (usec): min=15077, max=50598, avg=26419.46, stdev=5132.50 lat (usec): min=15113, max=50638, avg=26481.33, stdev=5137.41 clat percentiles (usec): | 1.00th=[17433], 5.00th=[19268], 10.00th=[20841], 20.00th=[22152], | 30.00th=[23200], 40.00th=[24511], 50.00th=[25822], 60.00th=[26870], | 70.00th=[28443], 80.00th=[30278], 90.00th=[33817], 95.00th=[35914], | 99.00th=[42206], 99.50th=[43779], 99.90th=[46924], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 1024, max= 1424, per=24.95%, avg=1196.62, stdev=132.71, samples=8 iops : min= 256, max= 356, avg=299.12, stdev=33.16, samples=8 lat (msec) : 4=0.04%, 10=0.12%, 20=11.37%, 50=88.36%, 100=0.12% cpu : usr=1.04%, sys=6.19%, ctx=2293, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26465: Thu May 1 03:38:58 2025 read: IOPS=289, BW=1157KiB/s (1185kB/s)(4976KiB/4299msec) slat (usec): min=27, max=6436, avg=1716.16, stdev=693.45 clat (usec): min=2929, max=44740, avg=25559.28, stdev=5581.09 lat (usec): min=2959, max=47249, avg=27278.92, stdev=5768.39 clat percentiles (usec): | 1.00th=[15270], 5.00th=[17957], 10.00th=[19268], 20.00th=[20841], | 30.00th=[22414], 40.00th=[23725], 50.00th=[24773], 60.00th=[26346], | 70.00th=[27919], 80.00th=[30016], 90.00th=[32900], 95.00th=[35914], | 99.00th=[40633], 99.50th=[41681], 99.90th=[43779], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 912, max= 1304, per=24.88%, avg=1146.62, stdev=122.61, samples=8 iops : min= 228, max= 326, avg=286.62, stdev=30.64, samples=8 write: IOPS=306, BW=1224KiB/s (1254kB/s)(5264KiB/4299msec) slat (usec): min=12, max=1030, avg=60.02, stdev=63.28 clat (usec): min=10270, max=48723, avg=26317.44, stdev=5537.23 lat (usec): min=10386, max=48757, avg=26378.73, stdev=5540.09 clat percentiles (usec): | 1.00th=[16712], 5.00th=[18744], 10.00th=[20055], 20.00th=[21627], | 30.00th=[23200], 40.00th=[24249], 50.00th=[25560], 60.00th=[26870], | 70.00th=[28443], 80.00th=[30540], 90.00th=[33817], 95.00th=[36439], | 99.00th=[42730], 99.50th=[44827], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 1133, max= 1528, per=25.74%, avg=1234.62, stdev=137.21, samples=8 iops : min= 283, max= 382, avg=308.62, stdev=34.33, samples=8 lat (msec) : 4=0.08%, 10=0.16%, 20=11.60%, 50=88.16% cpu : usr=1.00%, sys=6.28%, ctx=2360, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26466: Thu May 1 03:38:58 2025 read: IOPS=286, BW=1144KiB/s (1172kB/s)(4984KiB/4355msec) slat (usec): min=776, max=6145, avg=1695.70, stdev=613.93 clat (usec): min=2646, max=49826, avg=25658.70, stdev=5653.29 lat (usec): min=4884, max=51912, avg=27356.89, stdev=5791.41 clat percentiles (usec): | 1.00th=[15401], 5.00th=[17695], 10.00th=[19530], 20.00th=[21103], | 30.00th=[22414], 40.00th=[23725], 50.00th=[24773], 60.00th=[26346], | 70.00th=[27919], 80.00th=[30016], 90.00th=[33162], 95.00th=[35914], | 99.00th=[42730], 99.50th=[44827], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 1024, max= 1344, per=24.71%, avg=1138.62, stdev=98.16, samples=8 iops : min= 256, max= 336, avg=284.62, stdev=24.52, samples=8 write: IOPS=301, BW=1207KiB/s (1236kB/s)(5256KiB/4355msec) slat (usec): min=13, max=2759, avg=59.46, stdev=100.73 clat (usec): min=15096, max=47815, avg=26530.46, stdev=5271.36 lat (usec): min=15140, max=47875, avg=26591.79, stdev=5276.31 clat percentiles (usec): | 1.00th=[17433], 5.00th=[19530], 10.00th=[20579], 20.00th=[22152], | 30.00th=[23200], 40.00th=[24249], 50.00th=[25560], 60.00th=[27132], | 70.00th=[28705], 80.00th=[30540], 90.00th=[34341], 95.00th=[35914], | 99.00th=[41157], 99.50th=[44303], 99.90th=[46400], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 968, max= 1432, per=25.03%, avg=1200.62, stdev=144.57, samples=8 iops : min= 242, max= 358, avg=300.12, stdev=36.14, samples=8 lat (msec) : 4=0.04%, 10=0.08%, 20=10.16%, 50=89.73% cpu : usr=0.85%, sys=6.22%, ctx=2338, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26467: Thu May 1 03:38:58 2025 read: IOPS=290, BW=1162KiB/s (1190kB/s)(5060KiB/4355msec) slat (usec): min=727, max=5464, avg=1702.16, stdev=647.56 clat (usec): min=2602, max=55352, avg=25702.11, stdev=5742.33 lat (usec): min=4796, max=57191, avg=27406.23, stdev=5900.38 clat percentiles (usec): | 1.00th=[15795], 5.00th=[17957], 10.00th=[19006], 20.00th=[21103], | 30.00th=[22414], 40.00th=[23725], 50.00th=[24773], 60.00th=[26084], | 70.00th=[27919], 80.00th=[30016], 90.00th=[33424], 95.00th=[36439], | 99.00th=[41681], 99.50th=[43779], 99.90th=[48497], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 1029, max= 1392, per=25.14%, avg=1158.62, stdev=104.43, samples=8 iops : min= 257, max= 348, avg=289.62, stdev=26.15, samples=8 write: IOPS=297, BW=1189KiB/s (1218kB/s)(5180KiB/4355msec) slat (usec): min=13, max=2844, avg=64.77, stdev=126.42 clat (usec): min=7275, max=51395, avg=26536.19, stdev=5296.22 lat (usec): min=7355, max=51560, avg=26602.16, stdev=5296.05 clat percentiles (usec): | 1.00th=[17433], 5.00th=[19268], 10.00th=[20317], 20.00th=[22152], | 30.00th=[23462], 40.00th=[24773], 50.00th=[26084], 60.00th=[27132], | 70.00th=[28443], 80.00th=[30278], 90.00th=[33424], 95.00th=[35914], | 99.00th=[43779], 99.50th=[48497], 99.90th=[51119], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 968, max= 1432, per=24.70%, avg=1184.63, stdev=149.75, samples=8 iops : min= 242, max= 358, avg=296.12, stdev=37.41, samples=8 lat (msec) : 4=0.04%, 10=0.04%, 20=11.05%, 50=88.71%, 100=0.16% cpu : usr=1.01%, sys=6.16%, ctx=2447, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4608KiB/s (4719kB/s), 1144KiB/s-1162KiB/s (1172kB/s-1190kB/s), io=19.6MiB (20.5MB), run=4299-4355msec WRITE: bw=4797KiB/s (4912kB/s), 1189KiB/s-1224KiB/s (1218kB/s-1254kB/s), io=20.4MiB (21.4MB), run=4299-4355msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=26474: Thu May 1 03:39:00 2025 read: IOPS=7, BW=305MiB/s (320MB/s)(40.0MiB/131msec) slat (nsec): min=10032k, max=10032k, avg=10032005.00, stdev= 0.00 clat (nsec): min=106044k, max=106044k, avg=106044130.00, stdev= 0.00 lat (nsec): min=116100k, max=116100k, avg=116100215.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 107], 50.00th=[ 107], 60.00th=[ 107], | 70.00th=[ 107], 80.00th=[ 107], 90.00th=[ 107], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.92%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=305MiB/s (320MB/s), 305MiB/s-305MiB/s (320MB/s-320MB/s), io=40.0MiB (41.9MB), run=131-131msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout