== sanity test 398c: run fio to test AIO ================= 12:08:24 (1761322104) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.178055 s, 236 MB/s osc.lustre-OST0000-osc-ffff8af710c5d000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=148371: Fri Oct 24 12:08:41 2025 write: IOPS=220, BW=881KiB/s (902kB/s)(10.0MiB/11625msec); 0 zone resets slat (usec): min=22, max=4102, avg=116.96, stdev=172.60 clat (msec): min=8, max=129, avg=72.43, stdev=14.82 lat (msec): min=8, max=129, avg=72.54, stdev=14.81 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 52], 10.00th=[ 55], 20.00th=[ 61], | 30.00th=[ 65], 40.00th=[ 68], 50.00th=[ 71], 60.00th=[ 75], | 70.00th=[ 79], 80.00th=[ 84], 90.00th=[ 93], 95.00th=[ 102], | 99.00th=[ 113], 99.50th=[ 116], 99.90th=[ 122], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 648, max= 1065, per=24.80%, avg=873.77, stdev=102.82, samples=22 iops : min= 162, max= 266, avg=218.18, stdev=25.72, samples=22 lat (msec) : 10=0.04%, 20=0.12%, 50=3.91%, 100=90.47%, 250=5.47% cpu : usr=0.38%, sys=2.16%, ctx=1299, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148372: Fri Oct 24 12:08:41 2025 write: IOPS=220, BW=882KiB/s (903kB/s)(10.0MiB/11610msec); 0 zone resets slat (usec): min=21, max=3431, avg=111.85, stdev=176.00 clat (msec): min=12, max=126, avg=72.38, stdev=15.05 lat (msec): min=12, max=126, avg=72.49, stdev=15.06 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 49], 10.00th=[ 56], 20.00th=[ 61], | 30.00th=[ 64], 40.00th=[ 68], 50.00th=[ 72], 60.00th=[ 75], | 70.00th=[ 80], 80.00th=[ 84], 90.00th=[ 93], 95.00th=[ 101], | 99.00th=[ 113], 99.50th=[ 116], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 704, max= 1169, per=24.95%, avg=878.91, stdev=111.27, samples=22 iops : min= 176, max= 292, avg=219.45, stdev=27.86, samples=22 lat (msec) : 20=0.08%, 50=5.35%, 100=89.73%, 250=4.84% cpu : usr=0.40%, sys=1.92%, ctx=1159, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148373: Fri Oct 24 12:08:41 2025 write: IOPS=220, BW=881KiB/s (902kB/s)(10.0MiB/11620msec); 0 zone resets slat (usec): min=24, max=2474, avg=113.17, stdev=156.35 clat (msec): min=7, max=126, avg=72.43, stdev=14.97 lat (msec): min=7, max=126, avg=72.54, stdev=14.97 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 50], 10.00th=[ 56], 20.00th=[ 60], | 30.00th=[ 64], 40.00th=[ 68], 50.00th=[ 72], 60.00th=[ 75], | 70.00th=[ 80], 80.00th=[ 84], 90.00th=[ 93], 95.00th=[ 101], | 99.00th=[ 112], 99.50th=[ 118], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 648, max= 1156, per=24.84%, avg=875.00, stdev=110.07, samples=22 iops : min= 162, max= 289, avg=218.50, stdev=27.58, samples=22 lat (msec) : 10=0.04%, 20=0.08%, 50=5.66%, 100=89.53%, 250=4.69% cpu : usr=0.38%, sys=1.98%, ctx=1236, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148374: Fri Oct 24 12:08:41 2025 write: IOPS=220, BW=881KiB/s (902kB/s)(10.0MiB/11626msec); 0 zone resets slat (usec): min=19, max=3518, avg=117.11, stdev=184.75 clat (msec): min=7, max=125, avg=72.44, stdev=14.70 lat (msec): min=7, max=125, avg=72.55, stdev=14.71 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 52], 10.00th=[ 56], 20.00th=[ 61], | 30.00th=[ 64], 40.00th=[ 68], 50.00th=[ 71], 60.00th=[ 75], | 70.00th=[ 79], 80.00th=[ 84], 90.00th=[ 94], 95.00th=[ 101], | 99.00th=[ 111], 99.50th=[ 114], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 126] bw ( KiB/s): min= 648, max= 1086, per=24.86%, avg=875.82, stdev=99.90, samples=22 iops : min= 162, max= 271, avg=218.68, stdev=24.98, samples=22 lat (msec) : 10=0.04%, 20=0.12%, 50=3.91%, 100=90.98%, 250=4.96% cpu : usr=0.36%, sys=2.00%, ctx=1283, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3523KiB/s (3608kB/s), 881KiB/s-882KiB/s (902kB/s-903kB/s), io=40.0MiB (41.9MB), run=11610-11626msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148385: Fri Oct 24 12:08:54 2025 read: IOPS=135, BW=544KiB/s (557kB/s)(5048KiB/9284msec) slat (usec): min=1408, max=10342, avg=3358.84, stdev=1322.67 clat (usec): min=9802, max=96656, avg=55029.21, stdev=12343.01 lat (msec): min=14, max=101, avg=58.39, stdev=12.66 clat percentiles (usec): | 1.00th=[29230], 5.00th=[36963], 10.00th=[40109], 20.00th=[44303], | 30.00th=[47973], 40.00th=[51643], 50.00th=[54264], 60.00th=[56886], | 70.00th=[60556], 80.00th=[65274], 90.00th=[70779], 95.00th=[77071], | 99.00th=[85459], 99.50th=[89654], 99.90th=[94897], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 391, max= 640, per=24.99%, avg=540.00, stdev=72.20, samples=18 iops : min= 97, max= 160, avg=134.78, stdev=18.11, samples=18 write: IOPS=139, BW=559KiB/s (573kB/s)(5192KiB/9284msec); 0 zone resets slat (usec): min=25, max=2837, avg=123.42, stdev=173.79 clat (usec): min=9679, max=97427, avg=57058.65, stdev=11552.15 lat (usec): min=9778, max=97467, avg=57183.56, stdev=11553.29 clat percentiles (usec): | 1.00th=[35390], 5.00th=[40109], 10.00th=[43254], 20.00th=[47449], | 30.00th=[50070], 40.00th=[53216], 50.00th=[55313], 60.00th=[58983], | 70.00th=[62653], 80.00th=[66323], 90.00th=[72877], 95.00th=[78119], | 99.00th=[86508], 99.50th=[89654], 99.90th=[94897], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 456, max= 688, per=24.82%, avg=558.39, stdev=80.22, samples=18 iops : min= 114, max= 172, avg=139.39, stdev=20.09, samples=18 lat (msec) : 10=0.08%, 20=0.16%, 50=32.11%, 100=67.66% cpu : usr=0.74%, sys=4.13%, ctx=2707, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148386: Fri Oct 24 12:08:54 2025 read: IOPS=134, BW=538KiB/s (551kB/s)(4976KiB/9252msec) slat (usec): min=24, max=10194, avg=3359.11, stdev=1336.34 clat (usec): min=5740, max=97384, avg=55005.17, stdev=12875.66 lat (usec): min=6032, max=99483, avg=58368.26, stdev=13219.29 clat percentiles (usec): | 1.00th=[27919], 5.00th=[36439], 10.00th=[39584], 20.00th=[44303], | 30.00th=[47973], 40.00th=[51119], 50.00th=[54264], 60.00th=[57410], | 70.00th=[61080], 80.00th=[64750], 90.00th=[71828], 95.00th=[78119], | 99.00th=[87557], 99.50th=[91751], 99.90th=[95945], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 376, max= 680, per=24.77%, avg=535.22, stdev=88.48, samples=18 iops : min= 94, max= 170, avg=133.61, stdev=22.19, samples=18 write: IOPS=142, BW=569KiB/s (583kB/s)(5264KiB/9252msec); 0 zone resets slat (usec): min=25, max=2074, avg=118.34, stdev=147.65 clat (msec): min=14, max=100, avg=57.01, stdev=11.79 lat (msec): min=14, max=100, avg=57.13, stdev=11.80 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 41], 10.00th=[ 44], 20.00th=[ 47], | 30.00th=[ 51], 40.00th=[ 54], 50.00th=[ 56], 60.00th=[ 60], | 70.00th=[ 63], 80.00th=[ 68], 90.00th=[ 73], 95.00th=[ 79], | 99.00th=[ 87], 99.50th=[ 88], 99.90th=[ 97], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 480, max= 710, per=25.34%, avg=570.06, stdev=69.42, samples=18 iops : min= 120, max= 177, avg=142.33, stdev=17.31, samples=18 lat (msec) : 10=0.12%, 20=0.31%, 50=31.68%, 100=67.85%, 250=0.04% cpu : usr=0.53%, sys=4.30%, ctx=2595, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148387: Fri Oct 24 12:08:54 2025 read: IOPS=134, BW=537KiB/s (550kB/s)(4984KiB/9275msec) slat (usec): min=1205, max=11409, avg=3381.49, stdev=1265.52 clat (msec): min=18, max=102, avg=55.07, stdev=12.01 lat (msec): min=22, max=105, avg=58.46, stdev=12.32 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 37], 10.00th=[ 41], 20.00th=[ 45], | 30.00th=[ 48], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 58], | 70.00th=[ 61], 80.00th=[ 65], 90.00th=[ 71], 95.00th=[ 77], | 99.00th=[ 87], 99.50th=[ 91], 99.90th=[ 101], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 424, max= 686, per=24.96%, avg=539.33, stdev=79.59, samples=18 iops : min= 106, max= 171, avg=134.61, stdev=19.94, samples=18 write: IOPS=141, BW=567KiB/s (580kB/s)(5256KiB/9275msec); 0 zone resets slat (usec): min=25, max=1329, avg=112.37, stdev=115.83 clat (msec): min=19, max=105, avg=57.04, stdev=11.77 lat (msec): min=19, max=105, avg=57.15, stdev=11.77 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 41], 10.00th=[ 44], 20.00th=[ 47], | 30.00th=[ 51], 40.00th=[ 54], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 63], 80.00th=[ 67], 90.00th=[ 73], 95.00th=[ 80], | 99.00th=[ 87], 99.50th=[ 93], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 424, max= 709, per=25.15%, avg=565.89, stdev=86.26, samples=18 iops : min= 106, max= 177, avg=141.28, stdev=21.63, samples=18 lat (msec) : 20=0.08%, 50=31.84%, 100=67.89%, 250=0.20% cpu : usr=0.59%, sys=4.20%, ctx=2476, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148388: Fri Oct 24 12:08:54 2025 read: IOPS=136, BW=545KiB/s (558kB/s)(5060KiB/9280msec) slat (usec): min=1179, max=9513, avg=3371.38, stdev=1285.88 clat (usec): min=4410, max=90243, avg=55236.79, stdev=12208.79 lat (usec): min=7668, max=96219, avg=58612.00, stdev=12521.80 clat percentiles (usec): | 1.00th=[30540], 5.00th=[35914], 10.00th=[40633], 20.00th=[44827], | 30.00th=[48497], 40.00th=[51119], 50.00th=[54264], 60.00th=[57410], | 70.00th=[61080], 80.00th=[65274], 90.00th=[71828], 95.00th=[77071], | 99.00th=[85459], 99.50th=[87557], 99.90th=[89654], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 408, max= 677, per=25.23%, avg=545.17, stdev=69.78, samples=18 iops : min= 102, max= 169, avg=136.11, stdev=17.42, samples=18 write: IOPS=139, BW=558KiB/s (572kB/s)(5180KiB/9280msec); 0 zone resets slat (usec): min=28, max=2410, avg=121.96, stdev=173.20 clat (usec): min=26247, max=95303, avg=56885.64, stdev=11644.38 lat (usec): min=26332, max=95351, avg=57008.98, stdev=11641.11 clat percentiles (usec): | 1.00th=[33162], 5.00th=[39584], 10.00th=[43254], 20.00th=[46924], | 30.00th=[50070], 40.00th=[52691], 50.00th=[55837], 60.00th=[58459], | 70.00th=[61604], 80.00th=[66323], 90.00th=[72877], 95.00th=[78119], | 99.00th=[87557], 99.50th=[91751], 99.90th=[93848], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 399, max= 728, per=24.75%, avg=556.94, stdev=80.09, samples=18 iops : min= 99, max= 182, avg=139.06, stdev=20.13, samples=18 lat (msec) : 10=0.04%, 50=32.38%, 100=67.58% cpu : usr=0.59%, sys=4.14%, ctx=2598, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2162KiB/s (2213kB/s), 537KiB/s-545KiB/s (550kB/s-558kB/s), io=19.6MiB (20.5MB), run=9252-9284msec WRITE: bw=2250KiB/s (2304kB/s), 558KiB/s-569KiB/s (572kB/s-583kB/s), io=20.4MiB (21.4MB), run=9252-9284msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=148393: Fri Oct 24 12:08:58 2025 read: IOPS=6, BW=241MiB/s (253MB/s)(40.0MiB/166msec) slat (nsec): min=14605k, max=14605k, avg=14604924.00, stdev= 0.00 clat (nsec): min=135079k, max=135079k, avg=135078710.00, stdev= 0.00 lat (nsec): min=149687k, max=149687k, avg=149686967.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 136], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=8.43%, ctx=2, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=241MiB/s (253MB/s), 241MiB/s-241MiB/s (253MB/s-253MB/s), io=40.0MiB (41.9MB), run=166-166msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck