== sanity test 398c: run fio to test AIO ================= 10:51:39 (1761317499) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.293564 s, 143 MB/s osc.lustre-OST0000-osc-ffff925e906c2000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=122899: Fri Oct 24 10:53:55 2025 write: IOPS=19, BW=79.5KiB/s (81.4kB/s)(10.0MiB/128847msec); 0 zone resets slat (usec): min=21, max=8023, avg=221.69, stdev=513.85 clat (msec): min=206, max=1603, avg=804.59, stdev=181.41 lat (msec): min=207, max=1603, avg=804.82, stdev=181.43 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 518], 10.00th=[ 592], 20.00th=[ 659], | 30.00th=[ 709], 40.00th=[ 751], 50.00th=[ 802], 60.00th=[ 844], | 70.00th=[ 902], 80.00th=[ 953], 90.00th=[ 1020], 95.00th=[ 1083], | 99.00th=[ 1318], 99.50th=[ 1435], 99.90th=[ 1536], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min= 15, max= 247, per=24.84%, avg=78.76, stdev=27.87, samples=256 iops : min= 3, max= 61, avg=19.20, stdev= 7.02, samples=256 lat (msec) : 250=0.08%, 500=3.20%, 750=36.17%, 1000=47.58%, 2000=12.97% cpu : usr=0.05%, sys=0.33%, ctx=1742, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=122900: Fri Oct 24 10:53:55 2025 write: IOPS=19, BW=79.6KiB/s (81.5kB/s)(10.0MiB/128613msec); 0 zone resets slat (usec): min=25, max=7162, avg=204.99, stdev=490.79 clat (msec): min=183, max=1659, avg=803.15, stdev=188.76 lat (msec): min=183, max=1659, avg=803.35, stdev=188.77 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 514], 10.00th=[ 600], 20.00th=[ 659], | 30.00th=[ 701], 40.00th=[ 760], 50.00th=[ 802], 60.00th=[ 852], | 70.00th=[ 902], 80.00th=[ 953], 90.00th=[ 1028], 95.00th=[ 1083], | 99.00th=[ 1301], 99.50th=[ 1385], 99.90th=[ 1586], 99.95th=[ 1636], | 99.99th=[ 1653] bw ( KiB/s): min= 15, max= 255, per=24.95%, avg=79.08, stdev=31.34, samples=255 iops : min= 3, max= 63, avg=19.28, stdev= 7.86, samples=255 lat (msec) : 250=1.25%, 500=3.48%, 750=34.26%, 1000=48.12%, 2000=12.89% cpu : usr=0.05%, sys=0.32%, ctx=1391, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=122901: Fri Oct 24 10:53:55 2025 write: IOPS=19, BW=79.9KiB/s (81.8kB/s)(10.0MiB/128219msec); 0 zone resets slat (usec): min=23, max=7497, avg=208.42, stdev=491.15 clat (msec): min=83, max=1679, avg=800.77, stdev=190.54 lat (msec): min=84, max=1679, avg=800.98, stdev=190.53 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 493], 10.00th=[ 592], 20.00th=[ 659], | 30.00th=[ 709], 40.00th=[ 751], 50.00th=[ 802], 60.00th=[ 844], | 70.00th=[ 894], 80.00th=[ 944], 90.00th=[ 1028], 95.00th=[ 1099], | 99.00th=[ 1301], 99.50th=[ 1401], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 224, per=25.03%, avg=79.33, stdev=29.40, samples=254 iops : min= 1, max= 56, avg=19.34, stdev= 7.36, samples=254 lat (msec) : 100=0.08%, 250=1.17%, 500=3.95%, 750=34.49%, 1000=47.50% lat (msec) : 2000=12.81% cpu : usr=0.05%, sys=0.32%, ctx=1551, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=122902: Fri Oct 24 10:53:55 2025 write: IOPS=19, BW=79.7KiB/s (81.6kB/s)(10.0MiB/128525msec); 0 zone resets slat (usec): min=25, max=13378, avg=221.30, stdev=586.21 clat (msec): min=173, max=1679, avg=802.50, stdev=184.60 lat (msec): min=173, max=1679, avg=802.73, stdev=184.60 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 514], 10.00th=[ 592], 20.00th=[ 659], | 30.00th=[ 709], 40.00th=[ 751], 50.00th=[ 793], 60.00th=[ 844], | 70.00th=[ 894], 80.00th=[ 953], 90.00th=[ 1028], 95.00th=[ 1083], | 99.00th=[ 1318], 99.50th=[ 1368], 99.90th=[ 1620], 99.95th=[ 1636], | 99.99th=[ 1687] bw ( KiB/s): min= 15, max= 223, per=24.94%, avg=79.07, stdev=27.11, samples=255 iops : min= 3, max= 55, avg=19.28, stdev= 6.77, samples=255 lat (msec) : 250=0.35%, 500=3.87%, 750=35.16%, 1000=46.64%, 2000=13.98% cpu : usr=0.07%, sys=0.33%, ctx=1669, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=318KiB/s (326kB/s), 79.5KiB/s-79.9KiB/s (81.4kB/s-81.8kB/s), io=40.0MiB (41.9MB), run=128219-128847msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=122930: Fri Oct 24 10:55:17 2025 read: IOPS=17, BW=68.1KiB/s (69.8kB/s)(5048KiB/74097msec) slat (usec): min=34, max=26403, avg=8249.37, stdev=3785.92 clat (msec): min=22, max=883, avg=309.50, stdev=77.47 lat (msec): min=33, max=895, avg=317.76, stdev=78.43 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 192], 10.00th=[ 222], 20.00th=[ 249], | 30.00th=[ 271], 40.00th=[ 292], 50.00th=[ 309], 60.00th=[ 326], | 70.00th=[ 342], 80.00th=[ 363], 90.00th=[ 393], 95.00th=[ 430], | 99.00th=[ 506], 99.50th=[ 617], 99.90th=[ 827], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 7, max= 136, per=25.07%, avg=67.44, stdev=26.98, samples=147 iops : min= 1, max= 34, avg=16.33, stdev= 6.79, samples=147 write: IOPS=17, BW=70.1KiB/s (71.8kB/s)(5192KiB/74097msec); 0 zone resets slat (usec): min=30, max=15488, avg=276.82, stdev=664.75 clat (msec): min=91, max=1255, avg=603.32, stdev=163.12 lat (msec): min=91, max=1256, avg=603.60, stdev=163.18 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 376], 10.00th=[ 422], 20.00th=[ 468], | 30.00th=[ 510], 40.00th=[ 550], 50.00th=[ 584], 60.00th=[ 625], | 70.00th=[ 667], 80.00th=[ 726], 90.00th=[ 818], 95.00th=[ 911], | 99.00th=[ 1062], 99.50th=[ 1116], 99.90th=[ 1183], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 23, max= 114, per=24.76%, avg=69.34, stdev=20.76, samples=147 iops : min= 5, max= 28, avg=16.80, stdev= 5.21, samples=147 lat (msec) : 50=0.20%, 100=0.20%, 250=10.00%, 500=52.11%, 750=28.52% lat (msec) : 1000=8.09%, 2000=0.90% cpu : usr=0.14%, sys=0.91%, ctx=3102, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=122931: Fri Oct 24 10:55:17 2025 read: IOPS=16, BW=66.9KiB/s (68.6kB/s)(4976KiB/74331msec) slat (usec): min=1986, max=31159, avg=8411.92, stdev=3872.79 clat (msec): min=21, max=576, avg=309.36, stdev=73.16 lat (msec): min=32, max=590, avg=317.79, stdev=73.89 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 201], 10.00th=[ 226], 20.00th=[ 253], | 30.00th=[ 271], 40.00th=[ 288], 50.00th=[ 309], 60.00th=[ 326], | 70.00th=[ 342], 80.00th=[ 368], 90.00th=[ 401], 95.00th=[ 430], | 99.00th=[ 510], 99.50th=[ 542], 99.90th=[ 567], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 15, max= 151, per=24.63%, avg=66.26, stdev=28.70, samples=147 iops : min= 3, max= 37, avg=16.04, stdev= 7.18, samples=147 write: IOPS=17, BW=70.8KiB/s (72.5kB/s)(5264KiB/74331msec); 0 zone resets slat (usec): min=27, max=17928, avg=246.04, stdev=659.54 clat (msec): min=142, max=1192, avg=601.72, stdev=160.64 lat (msec): min=142, max=1192, avg=601.97, stdev=160.68 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 384], 10.00th=[ 422], 20.00th=[ 464], | 30.00th=[ 506], 40.00th=[ 550], 50.00th=[ 592], 60.00th=[ 625], | 70.00th=[ 667], 80.00th=[ 726], 90.00th=[ 810], 95.00th=[ 894], | 99.00th=[ 1083], 99.50th=[ 1116], 99.90th=[ 1167], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 16, max= 128, per=25.12%, avg=70.33, stdev=21.28, samples=147 iops : min= 4, max= 32, avg=17.05, stdev= 5.34, samples=147 lat (msec) : 50=0.20%, 100=0.31%, 250=9.18%, 500=52.42%, 750=28.87% lat (msec) : 1000=8.36%, 2000=0.66% cpu : usr=0.12%, sys=0.89%, ctx=3019, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=122932: Fri Oct 24 10:55:17 2025 read: IOPS=16, BW=66.0KiB/s (68.6kB/s)(4984KiB/74410msec) slat (usec): min=2126, max=35325, avg=8583.16, stdev=4127.92 clat (msec): min=16, max=568, avg=313.54, stdev=72.25 lat (msec): min=32, max=576, avg=322.13, stdev=73.05 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 201], 10.00th=[ 226], 20.00th=[ 255], | 30.00th=[ 279], 40.00th=[ 296], 50.00th=[ 313], 60.00th=[ 330], | 70.00th=[ 351], 80.00th=[ 372], 90.00th=[ 405], 95.00th=[ 430], | 99.00th=[ 481], 99.50th=[ 493], 99.90th=[ 542], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 7, max= 160, per=24.65%, avg=66.30, stdev=26.04, samples=148 iops : min= 1, max= 40, avg=16.05, stdev= 6.54, samples=148 write: IOPS=17, BW=70.6KiB/s (72.3kB/s)(5256KiB/74410msec); 0 zone resets slat (usec): min=28, max=7698, avg=267.15, stdev=602.28 clat (msec): min=87, max=1159, avg=598.86, stdev=159.29 lat (msec): min=92, max=1159, avg=599.13, stdev=159.26 clat percentiles (msec): | 1.00th=[ 259], 5.00th=[ 368], 10.00th=[ 414], 20.00th=[ 464], | 30.00th=[ 510], 40.00th=[ 558], 50.00th=[ 592], 60.00th=[ 625], | 70.00th=[ 667], 80.00th=[ 718], 90.00th=[ 810], 95.00th=[ 877], | 99.00th=[ 1028], 99.50th=[ 1116], 99.90th=[ 1150], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 23, max= 128, per=24.86%, avg=69.60, stdev=22.10, samples=148 iops : min= 5, max= 32, avg=16.87, stdev= 5.57, samples=148 lat (msec) : 20=0.04%, 50=0.12%, 100=0.39%, 250=8.98%, 500=52.62% lat (msec) : 750=29.84%, 1000=7.30%, 2000=0.70% cpu : usr=0.11%, sys=0.90%, ctx=2939, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=122933: Fri Oct 24 10:55:17 2025 read: IOPS=17, BW=68.5KiB/s (70.1kB/s)(5060KiB/73880msec) slat (usec): min=2085, max=27049, avg=8483.03, stdev=3925.85 clat (msec): min=10, max=944, avg=310.78, stdev=83.10 lat (msec): min=18, max=953, avg=319.27, stdev=83.78 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 199], 10.00th=[ 224], 20.00th=[ 249], | 30.00th=[ 271], 40.00th=[ 292], 50.00th=[ 309], 60.00th=[ 326], | 70.00th=[ 347], 80.00th=[ 372], 90.00th=[ 401], 95.00th=[ 439], | 99.00th=[ 527], 99.50th=[ 617], 99.90th=[ 793], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 16, max= 166, per=25.31%, avg=68.10, stdev=25.50, samples=146 iops : min= 4, max= 41, avg=16.50, stdev= 6.41, samples=146 write: IOPS=17, BW=70.1KiB/s (71.8kB/s)(5180KiB/73880msec); 0 zone resets slat (usec): min=27, max=6119, avg=261.73, stdev=526.18 clat (msec): min=111, max=1240, avg=599.11, stdev=161.85 lat (msec): min=111, max=1241, avg=599.38, stdev=161.84 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 372], 10.00th=[ 409], 20.00th=[ 460], | 30.00th=[ 510], 40.00th=[ 550], 50.00th=[ 592], 60.00th=[ 625], | 70.00th=[ 659], 80.00th=[ 718], 90.00th=[ 810], 95.00th=[ 894], | 99.00th=[ 1028], 99.50th=[ 1070], 99.90th=[ 1200], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 23, max= 128, per=24.88%, avg=69.67, stdev=21.43, samples=146 iops : min= 5, max= 32, avg=16.89, stdev= 5.41, samples=146 lat (msec) : 20=0.12%, 50=0.39%, 100=0.16%, 250=9.80%, 500=51.84% lat (msec) : 750=29.53%, 1000=7.38%, 2000=0.78% cpu : usr=0.13%, sys=0.98%, ctx=3288, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=270KiB/s (276kB/s), 66.9KiB/s-68.5KiB/s (68.6kB/s-70.1kB/s), io=19.6MiB (20.5MB), run=73880-74410msec WRITE: bw=281KiB/s (288kB/s), 70.1KiB/s-70.8KiB/s (71.8kB/s-72.5kB/s), io=20.4MiB (21.4MB), run=73880-74410msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=122947: Fri Oct 24 10:55:24 2025 read: IOPS=3, BW=158MiB/s (166MB/s)(40.0MiB/253msec) slat (nsec): min=21022k, max=21022k, avg=21022445.00, stdev= 0.00 clat (nsec): min=214602k, max=214602k, avg=214601536.00, stdev= 0.00 lat (nsec): min=235629k, max=235629k, avg=235629016.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 215], 10.00th=[ 215], 20.00th=[ 215], | 30.00th=[ 215], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 215], | 70.00th=[ 215], 80.00th=[ 215], 90.00th=[ 215], 95.00th=[ 215], | 99.00th=[ 215], 99.50th=[ 215], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] lat (msec) : 250=100.00% cpu : usr=0.40%, sys=6.75%, ctx=7, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=158MiB/s (166MB/s), 158MiB/s-158MiB/s (166MB/s-166MB/s), io=40.0MiB (41.9MB), run=253-253msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck