== sanity test 398c: run fio to test AIO ================= 11:24:19 (1761319459) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.149521 s, 281 MB/s osc.lustre-OST0000-osc-ffff9dd259dac000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=148791: Fri Oct 24 11:24:33 2025 write: IOPS=265, BW=1062KiB/s (1088kB/s)(10.0MiB/9641msec); 0 zone resets slat (usec): min=21, max=29429, avg=137.91, stdev=590.79 clat (msec): min=12, max=109, avg=59.76, stdev=12.80 lat (msec): min=12, max=109, avg=59.90, stdev=12.80 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 41], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 54], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 63], | 70.00th=[ 65], 80.00th=[ 69], 90.00th=[ 75], 95.00th=[ 84], | 99.00th=[ 97], 99.50th=[ 101], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 872, max= 1376, per=24.92%, avg=1058.79, stdev=135.75, samples=19 iops : min= 218, max= 344, avg=264.63, stdev=33.96, samples=19 lat (msec) : 20=0.04%, 50=21.60%, 100=77.77%, 250=0.59% cpu : usr=0.38%, sys=3.29%, ctx=1491, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148792: Fri Oct 24 11:24:33 2025 write: IOPS=266, BW=1064KiB/s (1090kB/s)(10.0MiB/9620msec); 0 zone resets slat (usec): min=24, max=1788, avg=126.61, stdev=115.45 clat (msec): min=6, max=106, avg=59.91, stdev=12.75 lat (msec): min=6, max=106, avg=60.04, stdev=12.75 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 42], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 63], | 70.00th=[ 66], 80.00th=[ 70], 90.00th=[ 77], 95.00th=[ 83], | 99.00th=[ 96], 99.50th=[ 101], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 848, max= 1360, per=24.85%, avg=1055.72, stdev=135.56, samples=18 iops : min= 212, max= 340, avg=263.83, stdev=33.91, samples=18 lat (msec) : 10=0.04%, 20=0.08%, 50=22.77%, 100=76.56%, 250=0.55% cpu : usr=0.47%, sys=3.09%, ctx=1470, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148793: Fri Oct 24 11:24:33 2025 write: IOPS=265, BW=1063KiB/s (1088kB/s)(10.0MiB/9635msec); 0 zone resets slat (usec): min=24, max=31942, avg=138.61, stdev=640.43 clat (msec): min=6, max=109, avg=59.72, stdev=12.86 lat (msec): min=6, max=109, avg=59.86, stdev=12.87 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 41], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 57], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 65], 80.00th=[ 69], 90.00th=[ 77], 95.00th=[ 83], | 99.00th=[ 97], 99.50th=[ 102], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 864, max= 1368, per=24.93%, avg=1059.16, stdev=127.27, samples=19 iops : min= 216, max= 342, avg=264.68, stdev=31.86, samples=19 lat (msec) : 10=0.16%, 20=0.08%, 50=22.11%, 100=76.95%, 250=0.70% cpu : usr=0.54%, sys=3.17%, ctx=1431, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148794: Fri Oct 24 11:24:33 2025 write: IOPS=265, BW=1062KiB/s (1088kB/s)(10.0MiB/9642msec); 0 zone resets slat (usec): min=21, max=30374, avg=137.10, stdev=616.16 clat (msec): min=6, max=106, avg=59.76, stdev=12.86 lat (msec): min=6, max=107, avg=59.90, stdev=12.90 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 41], 10.00th=[ 45], 20.00th=[ 49], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 65], 80.00th=[ 69], 90.00th=[ 77], 95.00th=[ 84], | 99.00th=[ 95], 99.50th=[ 99], 99.90th=[ 103], 99.95th=[ 104], | 99.99th=[ 107] bw ( KiB/s): min= 886, max= 1384, per=24.94%, avg=1059.53, stdev=140.88, samples=19 iops : min= 221, max= 346, avg=264.79, stdev=35.27, samples=19 lat (msec) : 10=0.04%, 20=0.04%, 50=22.54%, 100=77.19%, 250=0.20% cpu : usr=0.36%, sys=3.34%, ctx=1453, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=4248KiB/s (4350kB/s), 1062KiB/s-1064KiB/s (1088kB/s-1090kB/s), io=40.0MiB (41.9MB), run=9620-9642msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148805: Fri Oct 24 11:24:42 2025 read: IOPS=226, BW=906KiB/s (928kB/s)(5048KiB/5573msec) slat (usec): min=903, max=6425, avg=2033.66, stdev=707.07 clat (usec): min=9914, max=62249, avg=33130.79, stdev=8479.09 lat (usec): min=11351, max=66671, avg=35167.97, stdev=8816.94 clat percentiles (usec): | 1.00th=[19006], 5.00th=[22152], 10.00th=[23462], 20.00th=[25297], | 30.00th=[27395], 40.00th=[29754], 50.00th=[31851], 60.00th=[34341], | 70.00th=[36963], 80.00th=[40109], 90.00th=[44827], 95.00th=[50070], | 99.00th=[55313], 99.50th=[57934], 99.90th=[62129], 99.95th=[62129], | 99.99th=[62129] bw ( KiB/s): min= 663, max= 1104, per=25.26%, avg=908.50, stdev=162.18, samples=10 iops : min= 165, max= 276, avg=227.00, stdev=40.71, samples=10 write: IOPS=232, BW=932KiB/s (954kB/s)(5192KiB/5573msec); 0 zone resets slat (usec): min=29, max=856, avg=98.40, stdev=69.43 clat (usec): min=4954, max=63314, avg=34068.41, stdev=7942.22 lat (usec): min=5027, max=63384, avg=34167.93, stdev=7946.15 clat percentiles (usec): | 1.00th=[20841], 5.00th=[22938], 10.00th=[24511], 20.00th=[27132], | 30.00th=[29230], 40.00th=[31065], 50.00th=[32900], 60.00th=[34866], | 70.00th=[37487], 80.00th=[41157], 90.00th=[45876], 95.00th=[49021], | 99.00th=[54264], 99.50th=[56361], 99.90th=[58983], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 695, max= 1136, per=25.20%, avg=943.70, stdev=168.24, samples=10 iops : min= 173, max= 284, avg=235.80, stdev=42.22, samples=10 lat (msec) : 10=0.08%, 20=0.86%, 50=94.61%, 100=4.45% cpu : usr=0.79%, sys=6.41%, ctx=2314, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148806: Fri Oct 24 11:24:42 2025 read: IOPS=223, BW=892KiB/s (913kB/s)(4976KiB/5578msec) slat (usec): min=904, max=7188, avg=1999.40, stdev=701.86 clat (usec): min=17321, max=62660, avg=32903.27, stdev=8178.51 lat (usec): min=18547, max=66212, avg=34905.44, stdev=8508.83 clat percentiles (usec): | 1.00th=[19530], 5.00th=[22152], 10.00th=[23462], 20.00th=[25560], | 30.00th=[27657], 40.00th=[29492], 50.00th=[31589], 60.00th=[33817], | 70.00th=[36439], 80.00th=[39584], 90.00th=[44827], 95.00th=[49021], | 99.00th=[54264], 99.50th=[57410], 99.90th=[61604], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 643, max= 1096, per=24.78%, avg=891.30, stdev=183.88, samples=10 iops : min= 160, max= 274, avg=222.70, stdev=46.15, samples=10 write: IOPS=235, BW=944KiB/s (966kB/s)(5264KiB/5578msec); 0 zone resets slat (usec): min=26, max=821, avg=95.15, stdev=64.34 clat (usec): min=16745, max=59948, avg=34458.89, stdev=8246.65 lat (usec): min=16826, max=60019, avg=34555.23, stdev=8249.52 clat percentiles (usec): | 1.00th=[20841], 5.00th=[23200], 10.00th=[25035], 20.00th=[27132], | 30.00th=[28967], 40.00th=[30802], 50.00th=[33162], 60.00th=[35390], | 70.00th=[38011], 80.00th=[41681], 90.00th=[46924], 95.00th=[50070], | 99.00th=[54264], 99.50th=[55313], 99.90th=[58983], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 721, max= 1160, per=25.52%, avg=955.90, stdev=157.57, samples=10 iops : min= 180, max= 290, avg=238.90, stdev=39.42, samples=10 lat (msec) : 20=0.98%, 50=94.49%, 100=4.53% cpu : usr=0.66%, sys=6.26%, ctx=2266, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148807: Fri Oct 24 11:24:42 2025 read: IOPS=224, BW=899KiB/s (921kB/s)(4984KiB/5541msec) slat (usec): min=40, max=7487, avg=2034.72, stdev=771.56 clat (usec): min=3922, max=84514, avg=33409.22, stdev=8870.35 lat (usec): min=3983, max=88094, avg=35447.08, stdev=9239.65 clat percentiles (usec): | 1.00th=[18744], 5.00th=[22152], 10.00th=[23725], 20.00th=[25822], | 30.00th=[27919], 40.00th=[29754], 50.00th=[31851], 60.00th=[34341], | 70.00th=[36963], 80.00th=[40633], 90.00th=[45876], 95.00th=[50070], | 99.00th=[56361], 99.50th=[58983], 99.90th=[76022], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 696, max= 1104, per=25.28%, avg=909.20, stdev=135.22, samples=10 iops : min= 174, max= 276, avg=227.20, stdev=33.91, samples=10 write: IOPS=237, BW=949KiB/s (971kB/s)(5256KiB/5541msec); 0 zone resets slat (usec): min=29, max=636, avg=95.99, stdev=57.48 clat (usec): min=3820, max=84288, avg=33610.89, stdev=8297.27 lat (usec): min=3886, max=84333, avg=33707.83, stdev=8300.57 clat percentiles (usec): | 1.00th=[20055], 5.00th=[22676], 10.00th=[24511], 20.00th=[26870], | 30.00th=[28443], 40.00th=[30278], 50.00th=[32113], 60.00th=[34341], | 70.00th=[36963], 80.00th=[40109], 90.00th=[45351], 95.00th=[49021], | 99.00th=[54789], 99.50th=[56886], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 704, max= 1240, per=25.36%, avg=949.90, stdev=204.67, samples=10 iops : min= 176, max= 310, avg=237.40, stdev=51.24, samples=10 lat (msec) : 4=0.08%, 10=0.16%, 20=1.09%, 50=94.30%, 100=4.38% cpu : usr=0.87%, sys=6.37%, ctx=2268, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148808: Fri Oct 24 11:24:42 2025 read: IOPS=227, BW=910KiB/s (932kB/s)(5060KiB/5558msec) slat (usec): min=803, max=7235, avg=2017.62, stdev=725.32 clat (usec): min=16330, max=63103, avg=33197.58, stdev=8406.01 lat (usec): min=17757, max=66594, avg=35218.06, stdev=8761.22 clat percentiles (usec): | 1.00th=[19530], 5.00th=[22152], 10.00th=[23725], 20.00th=[26084], | 30.00th=[27919], 40.00th=[29754], 50.00th=[31589], 60.00th=[33817], | 70.00th=[36439], 80.00th=[39584], 90.00th=[45351], 95.00th=[50070], | 99.00th=[55837], 99.50th=[56361], 99.90th=[60556], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 657, max= 1120, per=25.46%, avg=915.90, stdev=164.31, samples=10 iops : min= 164, max= 280, avg=228.90, stdev=41.15, samples=10 write: IOPS=232, BW=932KiB/s (954kB/s)(5180KiB/5558msec); 0 zone resets slat (usec): min=29, max=1144, avg=98.66, stdev=76.71 clat (usec): min=10456, max=74889, avg=33883.74, stdev=8163.42 lat (usec): min=10531, max=74959, avg=33983.59, stdev=8170.44 clat percentiles (usec): | 1.00th=[20317], 5.00th=[23200], 10.00th=[24773], 20.00th=[26870], | 30.00th=[28705], 40.00th=[30540], 50.00th=[32375], 60.00th=[34341], | 70.00th=[36963], 80.00th=[40633], 90.00th=[45876], 95.00th=[50070], | 99.00th=[53740], 99.50th=[55313], 99.90th=[66323], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 688, max= 1120, per=25.07%, avg=938.90, stdev=179.01, samples=10 iops : min= 172, max= 280, avg=234.60, stdev=44.92, samples=10 lat (msec) : 20=0.90%, 50=93.95%, 100=5.16% cpu : usr=0.88%, sys=6.30%, ctx=2357, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3598KiB/s (3684kB/s), 892KiB/s-910KiB/s (913kB/s-932kB/s), io=19.6MiB (20.5MB), run=5541-5578msec WRITE: bw=3745KiB/s (3835kB/s), 932KiB/s-949KiB/s (954kB/s-971kB/s), io=20.4MiB (21.4MB), run=5541-5578msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=148812: Fri Oct 24 11:24:46 2025 read: IOPS=6, BW=256MiB/s (269MB/s)(40.0MiB/156msec) slat (nsec): min=14511k, max=14511k, avg=14510694.00, stdev= 0.00 clat (nsec): min=134305k, max=134305k, avg=134305200.00, stdev= 0.00 lat (nsec): min=148821k, max=148821k, avg=148820802.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 136], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=9.03%, ctx=3, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=256MiB/s (269MB/s), 256MiB/s-256MiB/s (269MB/s-269MB/s), io=40.0MiB (41.9MB), run=156-156msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck