== sanity test 398c: run fio to test AIO ================= 18:39:27 (1775169567) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.135729 s, 309 MB/s osc.lustre-OST0000-osc-ffffa002f2aef800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=185249: Thu Apr 2 18:39:35 2026 write: IOPS=495, BW=1982KiB/s (2029kB/s)(10.0MiB/5167msec); 0 zone resets slat (usec): min=37, max=721, avg=192.37, stdev=96.87 clat (usec): min=12382, max=55309, avg=32051.05, stdev=3451.39 lat (usec): min=12642, max=55558, avg=32244.75, stdev=3458.60 clat percentiles (usec): | 1.00th=[23200], 5.00th=[27132], 10.00th=[28443], 20.00th=[29492], | 30.00th=[30540], 40.00th=[31327], 50.00th=[31851], 60.00th=[32637], | 70.00th=[33424], 80.00th=[34341], 90.00th=[35914], 95.00th=[37487], | 99.00th=[41681], 99.50th=[46924], 99.90th=[48497], 99.95th=[51119], | 99.99th=[55313] bw ( KiB/s): min= 1872, max= 2089, per=25.09%, avg=1979.30, stdev=64.32, samples=10 iops : min= 468, max= 522, avg=494.80, stdev=16.03, samples=10 lat (msec) : 20=0.35%, 50=99.57%, 100=0.08% cpu : usr=0.68%, sys=7.34%, ctx=2873, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=185250: Thu Apr 2 18:39:35 2026 write: IOPS=493, BW=1972KiB/s (2020kB/s)(10.0MiB/5192msec); 0 zone resets slat (usec): min=42, max=22019, avg=194.75, stdev=442.39 clat (usec): min=6569, max=48039, avg=31993.38, stdev=3388.57 lat (usec): min=6757, max=48212, avg=32189.18, stdev=3388.88 clat percentiles (usec): | 1.00th=[23462], 5.00th=[26870], 10.00th=[28181], 20.00th=[29492], | 30.00th=[30540], 40.00th=[31327], 50.00th=[31851], 60.00th=[32637], | 70.00th=[33424], 80.00th=[34341], 90.00th=[35914], 95.00th=[37487], | 99.00th=[41681], 99.50th=[44827], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 1864, max= 2040, per=24.95%, avg=1968.30, stdev=55.79, samples=10 iops : min= 466, max= 510, avg=492.00, stdev=13.92, samples=10 lat (msec) : 10=0.04%, 20=0.23%, 50=99.73% cpu : usr=0.62%, sys=7.49%, ctx=2856, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=185251: Thu Apr 2 18:39:35 2026 write: IOPS=493, BW=1972KiB/s (2020kB/s)(10.0MiB/5192msec); 0 zone resets slat (usec): min=36, max=856, avg=176.28, stdev=95.05 clat (usec): min=14533, max=48161, avg=31977.10, stdev=3401.08 lat (usec): min=14695, max=48311, avg=32154.47, stdev=3410.33 clat percentiles (usec): | 1.00th=[22938], 5.00th=[27132], 10.00th=[28181], 20.00th=[29492], | 30.00th=[30540], 40.00th=[31065], 50.00th=[31851], 60.00th=[32637], | 70.00th=[33424], 80.00th=[34341], 90.00th=[35914], 95.00th=[37487], | 99.00th=[39584], 99.50th=[43779], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 1880, max= 2040, per=24.88%, avg=1962.70, stdev=47.35, samples=10 iops : min= 470, max= 510, avg=490.60, stdev=11.85, samples=10 lat (msec) : 20=0.43%, 50=99.57% cpu : usr=0.94%, sys=6.30%, ctx=2763, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=185252: Thu Apr 2 18:39:35 2026 write: IOPS=494, BW=1977KiB/s (2024kB/s)(10.0MiB/5180msec); 0 zone resets slat (usec): min=52, max=23117, avg=186.27, stdev=463.30 clat (usec): min=6584, max=57633, avg=31958.50, stdev=3818.74 lat (usec): min=6703, max=57738, avg=32145.78, stdev=3795.86 clat percentiles (usec): | 1.00th=[20579], 5.00th=[26870], 10.00th=[28181], 20.00th=[29492], | 30.00th=[30540], 40.00th=[31065], 50.00th=[31851], 60.00th=[32637], | 70.00th=[33424], 80.00th=[34341], 90.00th=[35914], 95.00th=[37487], | 99.00th=[41157], 99.50th=[45351], 99.90th=[49021], 99.95th=[49546], | 99.99th=[57410] bw ( KiB/s): min= 1856, max= 2082, per=25.03%, avg=1974.60, stdev=69.45, samples=10 iops : min= 464, max= 520, avg=493.60, stdev=17.28, samples=10 lat (msec) : 10=0.35%, 20=0.62%, 50=98.98%, 100=0.04% cpu : usr=0.46%, sys=7.16%, ctx=2899, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=7889KiB/s (8078kB/s), 1972KiB/s-1982KiB/s (2020kB/s-2029kB/s), io=40.0MiB (41.9MB), run=5167-5192msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=185262: Thu Apr 2 18:39:44 2026 read: IOPS=200, BW=803KiB/s (822kB/s)(5048KiB/6285msec) slat (usec): min=1096, max=9207, avg=2652.20, stdev=916.67 clat (usec): min=23031, max=62153, avg=37420.91, stdev=6346.74 lat (usec): min=24894, max=65044, avg=40075.83, stdev=6538.48 clat percentiles (usec): | 1.00th=[25822], 5.00th=[28181], 10.00th=[30016], 20.00th=[32375], | 30.00th=[33817], 40.00th=[35390], 50.00th=[36439], 60.00th=[38011], | 70.00th=[39584], 80.00th=[42206], 90.00th=[45876], 95.00th=[50070], | 99.00th=[54264], 99.50th=[56361], 99.90th=[60556], 99.95th=[62129], | 99.99th=[62129] bw ( KiB/s): min= 669, max= 896, per=25.18%, avg=803.75, stdev=78.37, samples=12 iops : min= 167, max= 224, avg=200.92, stdev=19.63, samples=12 write: IOPS=206, BW=826KiB/s (846kB/s)(5192KiB/6285msec); 0 zone resets slat (usec): min=50, max=1436, avg=170.08, stdev=87.04 clat (usec): min=26196, max=60838, avg=37681.11, stdev=5518.05 lat (usec): min=26444, max=60946, avg=37852.27, stdev=5518.35 clat percentiles (usec): | 1.00th=[28181], 5.00th=[30540], 10.00th=[31589], 20.00th=[33162], | 30.00th=[34341], 40.00th=[35390], 50.00th=[36439], 60.00th=[38011], | 70.00th=[39584], 80.00th=[41681], 90.00th=[45351], 95.00th=[48497], | 99.00th=[54264], 99.50th=[55837], 99.90th=[59507], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 695, max= 960, per=24.93%, avg=828.58, stdev=81.55, samples=12 iops : min= 173, max= 240, avg=207.08, stdev=20.50, samples=12 lat (msec) : 50=95.82%, 100=4.18% cpu : usr=0.53%, sys=6.76%, ctx=4515, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=185263: Thu Apr 2 18:39:44 2026 read: IOPS=199, BW=797KiB/s (816kB/s)(4976KiB/6243msec) slat (usec): min=36, max=10847, avg=2595.51, stdev=899.96 clat (usec): min=21504, max=73306, avg=37620.02, stdev=6535.76 lat (usec): min=24237, max=75306, avg=40218.24, stdev=6726.55 clat percentiles (usec): | 1.00th=[25035], 5.00th=[28443], 10.00th=[30278], 20.00th=[32637], | 30.00th=[33817], 40.00th=[35390], 50.00th=[36439], 60.00th=[38011], | 70.00th=[39584], 80.00th=[42206], 90.00th=[46400], 95.00th=[50594], | 99.00th=[56361], 99.50th=[56886], 99.90th=[69731], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 702, max= 912, per=24.99%, avg=797.83, stdev=63.18, samples=12 iops : min= 175, max= 228, avg=199.42, stdev=15.87, samples=12 write: IOPS=210, BW=843KiB/s (863kB/s)(5264KiB/6243msec); 0 zone resets slat (usec): min=38, max=2877, avg=176.24, stdev=116.93 clat (usec): min=7375, max=60207, avg=37547.95, stdev=5710.55 lat (usec): min=7494, max=60414, avg=37725.26, stdev=5716.02 clat percentiles (usec): | 1.00th=[27132], 5.00th=[30540], 10.00th=[31589], 20.00th=[33162], | 30.00th=[34341], 40.00th=[35390], 50.00th=[36963], 60.00th=[38011], | 70.00th=[39584], 80.00th=[41681], 90.00th=[45351], 95.00th=[48497], | 99.00th=[54264], 99.50th=[56886], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 680, max= 976, per=25.43%, avg=845.17, stdev=81.92, samples=12 iops : min= 170, max= 244, avg=211.25, stdev=20.51, samples=12 lat (msec) : 10=0.04%, 20=0.20%, 50=95.08%, 100=4.69% cpu : usr=0.53%, sys=6.74%, ctx=4488, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=185264: Thu Apr 2 18:39:44 2026 read: IOPS=198, BW=793KiB/s (812kB/s)(4984KiB/6285msec) slat (usec): min=1102, max=7278, avg=2620.76, stdev=841.02 clat (usec): min=21590, max=64493, avg=37477.87, stdev=6309.72 lat (usec): min=24210, max=66157, avg=40102.34, stdev=6484.73 clat percentiles (usec): | 1.00th=[25297], 5.00th=[28443], 10.00th=[30016], 20.00th=[32375], | 30.00th=[33817], 40.00th=[35390], 50.00th=[36439], 60.00th=[38011], | 70.00th=[40109], 80.00th=[42206], 90.00th=[46400], 95.00th=[49546], | 99.00th=[54789], 99.50th=[57934], 99.90th=[61080], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 631, max= 920, per=25.04%, avg=799.25, stdev=80.22, samples=12 iops : min= 157, max= 230, avg=199.75, stdev=20.20, samples=12 write: IOPS=209, BW=836KiB/s (856kB/s)(5256KiB/6285msec); 0 zone resets slat (usec): min=51, max=1955, avg=179.96, stdev=113.92 clat (usec): min=25077, max=64869, avg=37488.05, stdev=5588.51 lat (usec): min=25251, max=65076, avg=37669.18, stdev=5588.78 clat percentiles (usec): | 1.00th=[28181], 5.00th=[30016], 10.00th=[31065], 20.00th=[32900], | 30.00th=[34341], 40.00th=[35390], 50.00th=[36963], 60.00th=[38011], | 70.00th=[39584], 80.00th=[41681], 90.00th=[45351], 95.00th=[47973], | 99.00th=[53740], 99.50th=[54789], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 688, max= 984, per=25.10%, avg=834.33, stdev=102.68, samples=12 iops : min= 172, max= 246, avg=208.58, stdev=25.67, samples=12 lat (msec) : 50=95.98%, 100=4.02% cpu : usr=0.65%, sys=6.72%, ctx=4472, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=185265: Thu Apr 2 18:39:44 2026 read: IOPS=201, BW=806KiB/s (825kB/s)(5060KiB/6281msec) slat (usec): min=1097, max=8948, avg=2618.66, stdev=910.00 clat (usec): min=6975, max=65458, avg=37236.57, stdev=6320.72 lat (usec): min=9988, max=69490, avg=39858.07, stdev=6545.71 clat percentiles (usec): | 1.00th=[25297], 5.00th=[28181], 10.00th=[30016], 20.00th=[32113], | 30.00th=[33817], 40.00th=[34866], 50.00th=[36439], 60.00th=[38011], | 70.00th=[39584], 80.00th=[41681], 90.00th=[45351], 95.00th=[49546], | 99.00th=[56361], 99.50th=[57410], 99.90th=[62129], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 648, max= 936, per=25.36%, avg=809.42, stdev=90.51, samples=12 iops : min= 162, max= 234, avg=202.33, stdev=22.66, samples=12 write: IOPS=206, BW=825KiB/s (845kB/s)(5180KiB/6281msec); 0 zone resets slat (usec): min=57, max=829, avg=177.21, stdev=90.50 clat (usec): min=24387, max=72074, avg=37854.73, stdev=5770.74 lat (usec): min=24618, max=72207, avg=38033.10, stdev=5767.97 clat percentiles (usec): | 1.00th=[28181], 5.00th=[30278], 10.00th=[31327], 20.00th=[33162], | 30.00th=[34866], 40.00th=[35914], 50.00th=[36963], 60.00th=[38011], | 70.00th=[39584], 80.00th=[41681], 90.00th=[45351], 95.00th=[49021], | 99.00th=[54789], 99.50th=[57410], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 708, max= 928, per=24.82%, avg=825.00, stdev=71.07, samples=12 iops : min= 177, max= 232, avg=206.25, stdev=17.77, samples=12 lat (msec) : 10=0.04%, 50=95.94%, 100=4.02% cpu : usr=0.64%, sys=6.70%, ctx=4515, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3193KiB/s (3270kB/s), 793KiB/s-806KiB/s (812kB/s-825kB/s), io=19.6MiB (20.5MB), run=6243-6285msec WRITE: bw=3324KiB/s (3404kB/s), 825KiB/s-843KiB/s (845kB/s-863kB/s), io=20.4MiB (21.4MB), run=6243-6285msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=185269: Thu Apr 2 18:39:48 2026 read: IOPS=8, BW=325MiB/s (341MB/s)(40.0MiB/123msec) slat (nsec): min=12274k, max=12274k, avg=12273983.00, stdev= 0.00 clat (nsec): min=105952k, max=105952k, avg=105951503.00, stdev= 0.00 lat (nsec): min=118230k, max=118230k, avg=118229698.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 107], 50.00th=[ 107], 60.00th=[ 107], | 70.00th=[ 107], 80.00th=[ 107], 90.00th=[ 107], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=8.20%, ctx=12, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=325MiB/s (341MB/s), 325MiB/s-325MiB/s (341MB/s-341MB/s), io=40.0MiB (41.9MB), run=123-123msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout