== sanity test 398c: run fio to test AIO ================= 17:31:48 (1788730308) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.141175 s, 297 MB/s osc.lustre-OST0000-osc-ffff8ee5c2e00000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=181051: Sun Sep 6 17:31:57 2026 write: IOPS=465, BW=1861KiB/s (1905kB/s)(10.0MiB/5503msec); 0 zone resets slat (usec): min=21, max=1684, avg=102.74, stdev=67.38 clat (usec): min=7689, max=62731, avg=34241.12, stdev=6767.51 lat (usec): min=7767, max=62872, avg=34345.09, stdev=6773.92 clat percentiles (usec): | 1.00th=[18482], 5.00th=[26346], 10.00th=[28443], 20.00th=[29492], | 30.00th=[30540], 40.00th=[31851], 50.00th=[32900], 60.00th=[33817], | 70.00th=[35390], 80.00th=[38011], 90.00th=[43254], 95.00th=[47973], | 99.00th=[57410], 99.50th=[60031], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 1544, max= 2064, per=25.12%, avg=1855.00, stdev=166.31, samples=10 iops : min= 386, max= 516, avg=463.70, stdev=41.59, samples=10 lat (msec) : 10=0.08%, 20=1.37%, 50=94.73%, 100=3.83% cpu : usr=0.44%, sys=3.89%, ctx=2691, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181052: Sun Sep 6 17:31:57 2026 write: IOPS=462, BW=1849KiB/s (1893kB/s)(10.0MiB/5538msec); 0 zone resets slat (usec): min=22, max=22904, avg=109.19, stdev=454.94 clat (usec): min=7911, max=62607, avg=34243.24, stdev=6804.30 lat (usec): min=8811, max=62659, avg=34353.57, stdev=6788.07 clat percentiles (usec): | 1.00th=[18744], 5.00th=[26346], 10.00th=[28443], 20.00th=[29492], | 30.00th=[30540], 40.00th=[31589], 50.00th=[32637], 60.00th=[33817], | 70.00th=[35390], 80.00th=[38011], 90.00th=[43779], 95.00th=[48497], | 99.00th=[56886], 99.50th=[60031], 99.90th=[62129], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 1536, max= 2048, per=24.95%, avg=1842.90, stdev=168.91, samples=10 iops : min= 384, max= 512, avg=460.70, stdev=42.25, samples=10 lat (msec) : 10=0.08%, 20=1.21%, 50=95.04%, 100=3.67% cpu : usr=0.54%, sys=4.01%, ctx=2634, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181053: Sun Sep 6 17:31:57 2026 write: IOPS=461, BW=1848KiB/s (1892kB/s)(10.0MiB/5542msec); 0 zone resets slat (usec): min=24, max=1941, avg=98.82, stdev=69.36 clat (usec): min=7766, max=62637, avg=34245.98, stdev=6716.49 lat (usec): min=7952, max=62735, avg=34345.97, stdev=6722.58 clat percentiles (usec): | 1.00th=[21365], 5.00th=[26346], 10.00th=[28443], 20.00th=[29492], | 30.00th=[30540], 40.00th=[31589], 50.00th=[32637], 60.00th=[33817], | 70.00th=[35390], 80.00th=[38011], 90.00th=[43254], 95.00th=[47973], | 99.00th=[57934], 99.50th=[62129], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 1536, max= 2072, per=24.89%, avg=1837.80, stdev=175.09, samples=10 iops : min= 384, max= 518, avg=459.40, stdev=43.83, samples=10 lat (msec) : 10=0.08%, 20=0.43%, 50=96.05%, 100=3.44% cpu : usr=0.61%, sys=3.88%, ctx=2716, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181054: Sun Sep 6 17:31:57 2026 write: IOPS=461, BW=1846KiB/s (1891kB/s)(10.0MiB/5546msec); 0 zone resets slat (usec): min=21, max=3910, avg=96.67, stdev=105.74 clat (usec): min=14493, max=63824, avg=34239.30, stdev=6696.97 lat (usec): min=14690, max=63894, avg=34337.13, stdev=6705.98 clat percentiles (usec): | 1.00th=[21365], 5.00th=[26608], 10.00th=[28181], 20.00th=[29492], | 30.00th=[30540], 40.00th=[31589], 50.00th=[32637], 60.00th=[33817], | 70.00th=[35390], 80.00th=[38011], 90.00th=[43779], 95.00th=[47973], | 99.00th=[56886], 99.50th=[60556], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 1536, max= 2064, per=24.85%, avg=1835.50, stdev=182.08, samples=10 iops : min= 384, max= 516, avg=458.80, stdev=45.61, samples=10 lat (msec) : 20=0.31%, 50=96.37%, 100=3.32% cpu : usr=0.38%, sys=3.62%, ctx=2592, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=7386KiB/s (7563kB/s), 1846KiB/s-1861KiB/s (1891kB/s-1905kB/s), io=40.0MiB (41.9MB), run=5503-5546msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=181064: Sun Sep 6 17:32:07 2026 read: IOPS=200, BW=803KiB/s (823kB/s)(5048KiB/6283msec) slat (usec): min=1002, max=7459, avg=2514.98, stdev=817.37 clat (usec): min=7749, max=64325, avg=37316.94, stdev=6204.80 lat (usec): min=10763, max=67726, avg=39834.23, stdev=6376.02 clat percentiles (usec): | 1.00th=[25035], 5.00th=[28443], 10.00th=[30016], 20.00th=[32113], | 30.00th=[33817], 40.00th=[35390], 50.00th=[36963], 60.00th=[38011], | 70.00th=[39584], 80.00th=[42206], 90.00th=[45351], 95.00th=[48497], | 99.00th=[55837], 99.50th=[58459], 99.90th=[63177], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 656, max= 960, per=25.12%, avg=802.33, stdev=81.81, samples=12 iops : min= 164, max= 240, avg=200.58, stdev=20.45, samples=12 write: IOPS=206, BW=826KiB/s (846kB/s)(5192KiB/6283msec); 0 zone resets slat (usec): min=33, max=1773, avg=94.71, stdev=82.00 clat (usec): min=19816, max=65317, avg=37978.00, stdev=5672.18 lat (usec): min=19908, max=65473, avg=38073.72, stdev=5678.16 clat percentiles (usec): | 1.00th=[27395], 5.00th=[30016], 10.00th=[31327], 20.00th=[33162], | 30.00th=[34341], 40.00th=[35914], 50.00th=[37487], 60.00th=[39060], | 70.00th=[40633], 80.00th=[42730], 90.00th=[45351], 95.00th=[48497], | 99.00th=[53740], 99.50th=[56361], 99.90th=[60556], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 727, max= 928, per=24.84%, avg=825.92, stdev=64.06, samples=12 iops : min= 181, max= 232, avg=206.42, stdev=16.12, samples=12 lat (msec) : 10=0.04%, 20=0.08%, 50=96.68%, 100=3.20% cpu : usr=0.45%, sys=4.30%, ctx=4642, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181065: Sun Sep 6 17:32:07 2026 read: IOPS=198, BW=793KiB/s (812kB/s)(4976KiB/6277msec) slat (usec): min=1032, max=9468, avg=2501.99, stdev=825.94 clat (usec): min=9515, max=65051, avg=37264.52, stdev=6321.23 lat (usec): min=11688, max=66894, avg=39768.80, stdev=6489.52 clat percentiles (usec): | 1.00th=[25297], 5.00th=[28443], 10.00th=[30016], 20.00th=[32113], | 30.00th=[33817], 40.00th=[34866], 50.00th=[36439], 60.00th=[38011], | 70.00th=[39584], 80.00th=[42206], 90.00th=[45351], 95.00th=[49021], | 99.00th=[56361], 99.50th=[58459], 99.90th=[61604], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 676, max= 912, per=24.85%, avg=793.67, stdev=76.08, samples=12 iops : min= 169, max= 228, avg=198.42, stdev=19.02, samples=12 write: IOPS=209, BW=839KiB/s (859kB/s)(5264KiB/6277msec); 0 zone resets slat (usec): min=33, max=882, avg=90.78, stdev=58.98 clat (usec): min=6803, max=67898, avg=38152.53, stdev=5875.06 lat (usec): min=6896, max=67998, avg=38244.36, stdev=5873.92 clat percentiles (usec): | 1.00th=[27919], 5.00th=[30278], 10.00th=[31589], 20.00th=[33424], | 30.00th=[34866], 40.00th=[36439], 50.00th=[37487], 60.00th=[38536], | 70.00th=[40109], 80.00th=[42206], 90.00th=[45876], 95.00th=[49021], | 99.00th=[53740], 99.50th=[62129], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 760, max= 896, per=25.20%, avg=838.00, stdev=51.75, samples=12 iops : min= 190, max= 224, avg=209.50, stdev=12.94, samples=12 lat (msec) : 10=0.08%, 20=0.04%, 50=95.90%, 100=3.98% cpu : usr=0.62%, sys=4.11%, ctx=4637, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181066: Sun Sep 6 17:32:07 2026 read: IOPS=198, BW=793KiB/s (812kB/s)(4984KiB/6283msec) slat (usec): min=968, max=10073, avg=2525.24, stdev=891.65 clat (usec): min=19256, max=69531, avg=37664.78, stdev=6465.59 lat (usec): min=21389, max=72952, avg=40192.13, stdev=6627.88 clat percentiles (usec): | 1.00th=[25560], 5.00th=[28443], 10.00th=[30278], 20.00th=[32375], | 30.00th=[33817], 40.00th=[35390], 50.00th=[36963], 60.00th=[38536], | 70.00th=[40109], 80.00th=[42206], 90.00th=[46400], 95.00th=[49546], | 99.00th=[57410], 99.50th=[61080], 99.90th=[66323], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 664, max= 880, per=24.85%, avg=793.67, stdev=65.83, samples=12 iops : min= 166, max= 220, avg=198.42, stdev=16.46, samples=12 write: IOPS=209, BW=837KiB/s (857kB/s)(5256KiB/6283msec); 0 zone resets slat (usec): min=30, max=1127, avg=93.46, stdev=63.55 clat (usec): min=19364, max=65292, avg=37817.77, stdev=5634.95 lat (usec): min=19576, max=65369, avg=37912.26, stdev=5635.73 clat percentiles (usec): | 1.00th=[27657], 5.00th=[30016], 10.00th=[31327], 20.00th=[33424], | 30.00th=[34866], 40.00th=[35914], 50.00th=[36963], 60.00th=[38536], | 70.00th=[39584], 80.00th=[42206], 90.00th=[45351], 95.00th=[49021], | 99.00th=[54264], 99.50th=[55837], 99.90th=[58983], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 744, max= 960, per=25.03%, avg=832.08, stdev=74.01, samples=12 iops : min= 186, max= 240, avg=208.00, stdev=18.53, samples=12 lat (msec) : 20=0.16%, 50=95.90%, 100=3.95% cpu : usr=0.48%, sys=4.22%, ctx=4683, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181067: Sun Sep 6 17:32:07 2026 read: IOPS=202, BW=809KiB/s (829kB/s)(5060KiB/6251msec) slat (usec): min=32, max=8188, avg=2518.09, stdev=875.72 clat (usec): min=3108, max=64280, avg=37433.88, stdev=6416.07 lat (usec): min=5496, max=68027, avg=39954.05, stdev=6660.74 clat percentiles (usec): | 1.00th=[24511], 5.00th=[28705], 10.00th=[30540], 20.00th=[32637], | 30.00th=[34341], 40.00th=[35390], 50.00th=[36963], 60.00th=[38011], | 70.00th=[39584], 80.00th=[42206], 90.00th=[46400], 95.00th=[49021], | 99.00th=[54789], 99.50th=[56886], 99.90th=[62653], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 728, max= 912, per=25.36%, avg=810.08, stdev=47.72, samples=12 iops : min= 182, max= 228, avg=202.50, stdev=11.94, samples=12 write: IOPS=207, BW=829KiB/s (849kB/s)(5180KiB/6251msec); 0 zone resets slat (usec): min=30, max=3089, avg=96.83, stdev=136.74 clat (usec): min=10854, max=69059, avg=37972.64, stdev=5825.22 lat (usec): min=10951, max=69144, avg=38070.46, stdev=5837.68 clat percentiles (usec): | 1.00th=[27132], 5.00th=[30540], 10.00th=[31589], 20.00th=[33424], | 30.00th=[34866], 40.00th=[35914], 50.00th=[37487], 60.00th=[38536], | 70.00th=[40109], 80.00th=[41681], 90.00th=[45351], 95.00th=[48497], | 99.00th=[57410], 99.50th=[62129], 99.90th=[65274], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 720, max= 928, per=24.88%, avg=827.17, stdev=66.53, samples=12 iops : min= 180, max= 232, avg=206.75, stdev=16.65, samples=12 lat (msec) : 4=0.04%, 10=0.12%, 20=0.16%, 50=95.74%, 100=3.95% cpu : usr=0.58%, sys=4.13%, ctx=4594, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3194KiB/s (3271kB/s), 793KiB/s-809KiB/s (812kB/s-829kB/s), io=19.6MiB (20.5MB), run=6251-6283msec WRITE: bw=3325KiB/s (3405kB/s), 826KiB/s-839KiB/s (846kB/s-859kB/s), io=20.4MiB (21.4MB), run=6251-6283msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=181074: Sun Sep 6 17:32:10 2026 read: IOPS=8, BW=345MiB/s (362MB/s)(40.0MiB/116msec) slat (nsec): min=9968.2k, max=9968.2k, avg=9968223.00, stdev= 0.00 clat (nsec): min=100908k, max=100908k, avg=100908039.00, stdev= 0.00 lat (nsec): min=110880k, max=110880k, avg=110880033.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 102], 10.00th=[ 102], 20.00th=[ 102], | 30.00th=[ 102], 40.00th=[ 102], 50.00th=[ 102], 60.00th=[ 102], | 70.00th=[ 102], 80.00th=[ 102], 90.00th=[ 102], 95.00th=[ 102], | 99.00th=[ 102], 99.50th=[ 102], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=7.83%, ctx=10, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=345MiB/s (362MB/s), 345MiB/s-345MiB/s (362MB/s-362MB/s), io=40.0MiB (41.9MB), run=116-116msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout