== sanity test 398c: run fio to test AIO ================= 14:38:33 (1780857513) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.129672 s, 323 MB/s osc.lustre-OST0000-osc-ffff95f110d80800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=181116: Sun Jun 7 14:38:41 2026 write: IOPS=541, BW=2164KiB/s (2216kB/s)(10.0MiB/4731msec); 0 zone resets slat (usec): min=37, max=817, avg=164.09, stdev=81.60 clat (usec): min=7946, max=51221, avg=29353.60, stdev=4209.97 lat (usec): min=8105, max=51402, avg=29518.86, stdev=4220.34 clat percentiles (usec): | 1.00th=[18482], 5.00th=[23462], 10.00th=[24511], 20.00th=[26084], | 30.00th=[27395], 40.00th=[28443], 50.00th=[29230], 60.00th=[30278], | 70.00th=[31327], 80.00th=[32637], 90.00th=[34866], 95.00th=[35914], | 99.00th=[39584], 99.50th=[40633], 99.90th=[47973], 99.95th=[48497], | 99.99th=[51119] bw ( KiB/s): min= 1936, max= 2489, per=25.12%, avg=2156.56, stdev=195.32, samples=9 iops : min= 484, max= 622, avg=539.11, stdev=48.78, samples=9 lat (msec) : 10=0.08%, 20=1.91%, 50=97.97%, 100=0.04% cpu : usr=0.49%, sys=6.85%, ctx=2593, majf=0, minf=7 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181117: Sun Jun 7 14:38:41 2026 write: IOPS=536, BW=2148KiB/s (2199kB/s)(10.0MiB/4768msec); 0 zone resets slat (usec): min=38, max=686, avg=160.92, stdev=84.16 clat (usec): min=3377, max=56131, avg=29344.28, stdev=4238.23 lat (usec): min=3494, max=56336, avg=29506.35, stdev=4253.39 clat percentiles (usec): | 1.00th=[18744], 5.00th=[22938], 10.00th=[24511], 20.00th=[26084], | 30.00th=[27395], 40.00th=[28443], 50.00th=[29230], 60.00th=[30016], | 70.00th=[31327], 80.00th=[32375], 90.00th=[34341], 95.00th=[35914], | 99.00th=[39584], 99.50th=[40633], 99.90th=[55837], 99.95th=[55837], | 99.99th=[56361] bw ( KiB/s): min= 1944, max= 2393, per=24.98%, avg=2144.11, stdev=166.80, samples=9 iops : min= 486, max= 598, avg=536.00, stdev=41.65, samples=9 lat (msec) : 4=0.04%, 10=0.08%, 20=1.91%, 50=97.85%, 100=0.12% cpu : usr=0.52%, sys=7.13%, ctx=2628, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181118: Sun Jun 7 14:38:41 2026 write: IOPS=538, BW=2153KiB/s (2205kB/s)(10.0MiB/4756msec); 0 zone resets slat (usec): min=47, max=23336, avg=180.18, stdev=465.27 clat (usec): min=7793, max=57431, avg=29284.96, stdev=4289.18 lat (usec): min=7969, max=57563, avg=29466.13, stdev=4275.69 clat percentiles (usec): | 1.00th=[17957], 5.00th=[22938], 10.00th=[24511], 20.00th=[26084], | 30.00th=[27395], 40.00th=[28443], 50.00th=[29230], 60.00th=[30278], | 70.00th=[31327], 80.00th=[32375], 90.00th=[34341], 95.00th=[35914], | 99.00th=[39060], 99.50th=[41157], 99.90th=[54264], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 1928, max= 2425, per=25.01%, avg=2146.78, stdev=180.10, samples=9 iops : min= 482, max= 606, avg=536.67, stdev=44.98, samples=9 lat (msec) : 10=0.12%, 20=1.99%, 50=97.73%, 100=0.16% cpu : usr=0.59%, sys=7.40%, ctx=2533, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181119: Sun Jun 7 14:38:41 2026 write: IOPS=536, BW=2146KiB/s (2198kB/s)(10.0MiB/4771msec); 0 zone resets slat (usec): min=36, max=1193, avg=160.70, stdev=91.74 clat (usec): min=2997, max=41529, avg=29341.74, stdev=3992.75 lat (usec): min=3122, max=41647, avg=29503.63, stdev=4014.82 clat percentiles (usec): | 1.00th=[19792], 5.00th=[23200], 10.00th=[24773], 20.00th=[26346], | 30.00th=[27395], 40.00th=[28443], 50.00th=[29230], 60.00th=[30016], | 70.00th=[31327], 80.00th=[32375], 90.00th=[34341], 95.00th=[35914], | 99.00th=[39060], 99.50th=[39584], 99.90th=[40633], 99.95th=[41157], | 99.99th=[41681] bw ( KiB/s): min= 1928, max= 2360, per=24.83%, avg=2131.33, stdev=164.05, samples=9 iops : min= 482, max= 590, avg=532.78, stdev=40.93, samples=9 lat (msec) : 4=0.04%, 10=0.16%, 20=0.90%, 50=98.91% cpu : usr=0.67%, sys=6.29%, ctx=2595, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=8585KiB/s (8791kB/s), 2146KiB/s-2164KiB/s (2198kB/s-2216kB/s), io=40.0MiB (41.9MB), run=4731-4771msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=181129: Sun Jun 7 14:38:51 2026 read: IOPS=191, BW=766KiB/s (784kB/s)(5048KiB/6590msec) slat (usec): min=80, max=9356, avg=2703.91, stdev=1022.84 clat (usec): min=5781, max=82067, avg=39578.89, stdev=8938.54 lat (usec): min=7616, max=85499, avg=42284.80, stdev=9259.01 clat percentiles (usec): | 1.00th=[23462], 5.00th=[28443], 10.00th=[30540], 20.00th=[33162], | 30.00th=[35390], 40.00th=[36963], 50.00th=[38536], 60.00th=[40109], | 70.00th=[42206], 80.00th=[44827], 90.00th=[50070], 95.00th=[56361], | 99.00th=[71828], 99.50th=[74974], 99.90th=[81265], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 560, max= 876, per=24.92%, avg=757.33, stdev=75.55, samples=12 iops : min= 140, max= 219, avg=189.25, stdev=18.93, samples=12 write: IOPS=196, BW=788KiB/s (807kB/s)(5192KiB/6590msec); 0 zone resets slat (usec): min=57, max=3645, avg=171.29, stdev=137.15 clat (usec): min=6663, max=75670, avg=39602.20, stdev=7494.25 lat (usec): min=6770, max=76023, avg=39774.63, stdev=7498.29 clat percentiles (usec): | 1.00th=[25035], 5.00th=[30540], 10.00th=[32375], 20.00th=[34341], | 30.00th=[35914], 40.00th=[36963], 50.00th=[38536], 60.00th=[40109], | 70.00th=[41681], 80.00th=[43779], 90.00th=[47973], 95.00th=[54264], | 99.00th=[66847], 99.50th=[69731], 99.90th=[74974], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 632, max= 1016, per=24.97%, avg=790.00, stdev=97.13, samples=12 iops : min= 158, max= 254, avg=197.42, stdev=24.31, samples=12 lat (msec) : 10=0.16%, 20=0.55%, 50=90.55%, 100=8.75% cpu : usr=0.52%, sys=5.71%, ctx=4572, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181130: Sun Jun 7 14:38:51 2026 read: IOPS=188, BW=754KiB/s (772kB/s)(4976KiB/6598msec) slat (usec): min=55, max=8978, avg=2705.64, stdev=983.61 clat (usec): min=6966, max=79672, avg=39604.21, stdev=8201.47 lat (usec): min=7036, max=85137, avg=42312.50, stdev=8545.84 clat percentiles (usec): | 1.00th=[24511], 5.00th=[29230], 10.00th=[31065], 20.00th=[33424], | 30.00th=[35390], 40.00th=[36963], 50.00th=[38536], 60.00th=[40109], | 70.00th=[42730], 80.00th=[45351], 90.00th=[49021], 95.00th=[53216], | 99.00th=[68682], 99.50th=[70779], 99.90th=[78119], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 568, max= 864, per=24.51%, avg=744.83, stdev=75.61, samples=12 iops : min= 142, max= 216, avg=186.08, stdev=18.93, samples=12 write: IOPS=199, BW=798KiB/s (817kB/s)(5264KiB/6598msec); 0 zone resets slat (usec): min=51, max=1775, avg=169.63, stdev=99.62 clat (usec): min=15331, max=80948, avg=39705.31, stdev=7380.42 lat (usec): min=15398, max=81128, avg=39876.19, stdev=7386.72 clat percentiles (usec): | 1.00th=[26870], 5.00th=[30540], 10.00th=[32375], 20.00th=[34341], | 30.00th=[35914], 40.00th=[36963], 50.00th=[38536], 60.00th=[39584], | 70.00th=[41681], 80.00th=[44303], 90.00th=[48497], 95.00th=[53740], | 99.00th=[65274], 99.50th=[68682], 99.90th=[77071], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 632, max= 904, per=25.27%, avg=799.67, stdev=81.41, samples=12 iops : min= 158, max= 226, avg=199.83, stdev=20.40, samples=12 lat (msec) : 10=0.20%, 20=0.16%, 50=91.45%, 100=8.20% cpu : usr=0.50%, sys=5.82%, ctx=4575, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181131: Sun Jun 7 14:38:51 2026 read: IOPS=189, BW=758KiB/s (776kB/s)(4984KiB/6577msec) slat (usec): min=36, max=9836, avg=2692.51, stdev=1028.60 clat (usec): min=6179, max=80941, avg=39450.72, stdev=8476.85 lat (usec): min=7688, max=83900, avg=42145.55, stdev=8798.31 clat percentiles (usec): | 1.00th=[20317], 5.00th=[28443], 10.00th=[30802], 20.00th=[33424], | 30.00th=[35390], 40.00th=[36439], 50.00th=[38536], 60.00th=[40109], | 70.00th=[41681], 80.00th=[44303], 90.00th=[49546], 95.00th=[54789], | 99.00th=[68682], 99.50th=[73925], 99.90th=[80217], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 520, max= 904, per=24.93%, avg=757.75, stdev=106.19, samples=12 iops : min= 130, max= 226, avg=189.33, stdev=26.53, samples=12 write: IOPS=199, BW=799KiB/s (818kB/s)(5256KiB/6577msec); 0 zone resets slat (usec): min=37, max=1222, avg=169.21, stdev=88.57 clat (usec): min=9006, max=79357, avg=39811.65, stdev=7873.33 lat (usec): min=9295, max=79449, avg=39982.06, stdev=7872.66 clat percentiles (usec): | 1.00th=[26084], 5.00th=[30802], 10.00th=[32637], 20.00th=[34341], | 30.00th=[35914], 40.00th=[37487], 50.00th=[38536], 60.00th=[40109], | 70.00th=[41681], 80.00th=[43779], 90.00th=[48497], 95.00th=[53740], | 99.00th=[69731], 99.50th=[76022], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 654, max= 912, per=25.06%, avg=792.83, stdev=81.23, samples=12 iops : min= 163, max= 228, avg=198.08, stdev=20.37, samples=12 lat (msec) : 10=0.20%, 20=0.47%, 50=90.62%, 100=8.71% cpu : usr=0.49%, sys=5.87%, ctx=4555, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181132: Sun Jun 7 14:38:51 2026 read: IOPS=191, BW=766KiB/s (785kB/s)(5060KiB/6603msec) slat (usec): min=61, max=17132, avg=2706.91, stdev=1063.62 clat (usec): min=1737, max=81816, avg=39755.92, stdev=8370.89 lat (usec): min=4804, max=85524, avg=42465.05, stdev=8714.66 clat percentiles (usec): | 1.00th=[25297], 5.00th=[29230], 10.00th=[31065], 20.00th=[33424], | 30.00th=[35390], 40.00th=[36963], 50.00th=[39060], 60.00th=[40633], | 70.00th=[42730], 80.00th=[44827], 90.00th=[49546], 95.00th=[54264], | 99.00th=[69731], 99.50th=[74974], 99.90th=[81265], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 576, max= 840, per=24.99%, avg=759.50, stdev=69.20, samples=12 iops : min= 144, max= 210, avg=189.75, stdev=17.29, samples=12 write: IOPS=196, BW=784KiB/s (803kB/s)(5180KiB/6603msec); 0 zone resets slat (usec): min=46, max=1270, avg=171.75, stdev=93.63 clat (usec): min=4812, max=81894, avg=39547.25, stdev=7417.74 lat (usec): min=5018, max=82010, avg=39720.20, stdev=7418.55 clat percentiles (usec): | 1.00th=[27395], 5.00th=[30802], 10.00th=[32637], 20.00th=[34341], | 30.00th=[35914], 40.00th=[36963], 50.00th=[38536], 60.00th=[39584], | 70.00th=[41157], 80.00th=[43779], 90.00th=[48497], 95.00th=[53216], | 99.00th=[65274], 99.50th=[68682], 99.90th=[81265], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 600, max= 896, per=24.62%, avg=778.83, stdev=80.71, samples=12 iops : min= 150, max= 224, avg=194.58, stdev=20.22, samples=12 lat (msec) : 2=0.04%, 10=0.20%, 20=0.23%, 50=91.13%, 100=8.40% cpu : usr=0.39%, sys=5.89%, ctx=4585, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3039KiB/s (3112kB/s), 754KiB/s-766KiB/s (772kB/s-785kB/s), io=19.6MiB (20.5MB), run=6577-6603msec WRITE: bw=3164KiB/s (3240kB/s), 784KiB/s-799KiB/s (803kB/s-818kB/s), io=20.4MiB (21.4MB), run=6577-6603msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=181137: Sun Jun 7 14:38:54 2026 read: IOPS=7, BW=308MiB/s (323MB/s)(40.0MiB/130msec) slat (nsec): min=11545k, max=11545k, avg=11544670.00, stdev= 0.00 clat (nsec): min=110885k, max=110885k, avg=110884969.00, stdev= 0.00 lat (nsec): min=122434k, max=122434k, avg=122433644.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 111], 50.00th=[ 111], 60.00th=[ 111], | 70.00th=[ 111], 80.00th=[ 111], 90.00th=[ 111], 95.00th=[ 111], | 99.00th=[ 111], 99.50th=[ 111], 99.90th=[ 111], 99.95th=[ 111], | 99.99th=[ 111] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=9.30%, ctx=8, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=308MiB/s (323MB/s), 308MiB/s-308MiB/s (323MB/s-323MB/s), io=40.0MiB (41.9MB), run=130-130msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout