== sanity test 398c: run fio to test AIO ================= 11:55:53 (1776268553) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.739801 s, 56.7 MB/s osc.lustre-OST0000-osc-ffff90a80f26e800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=123537: Wed Apr 15 11:59:31 2026 write: IOPS=12, BW=49.3KiB/s (50.5kB/s)(10.0MiB/207713msec); 0 zone resets slat (usec): min=20, max=40614, avg=320.83, stdev=1611.29 clat (msec): min=373, max=3215, avg=1297.29, stdev=439.44 lat (msec): min=373, max=3215, avg=1297.61, stdev=439.44 clat percentiles (msec): | 1.00th=[ 535], 5.00th=[ 642], 10.00th=[ 835], 20.00th=[ 953], | 30.00th=[ 1070], 40.00th=[ 1150], 50.00th=[ 1217], 60.00th=[ 1318], | 70.00th=[ 1435], 80.00th=[ 1603], 90.00th=[ 1905], 95.00th=[ 2198], | 99.00th=[ 2567], 99.50th=[ 2903], 99.90th=[ 2937], 99.95th=[ 2937], | 99.99th=[ 3205] bw ( KiB/s): min= 7, max= 128, per=25.53%, avg=50.30, stdev=28.32, samples=399 iops : min= 1, max= 32, avg=12.11, stdev= 7.12, samples=399 lat (msec) : 500=0.82%, 750=6.91%, 1000=17.03%, 2000=66.91%, >=2000=8.32% cpu : usr=0.06%, sys=0.23%, ctx=1732, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=123538: Wed Apr 15 11:59:31 2026 write: IOPS=12, BW=49.4KiB/s (50.5kB/s)(10.0MiB/207463msec); 0 zone resets slat (usec): min=19, max=16443, avg=255.17, stdev=1129.01 clat (msec): min=125, max=3093, avg=1296.00, stdev=439.35 lat (msec): min=125, max=3093, avg=1296.25, stdev=439.36 clat percentiles (msec): | 1.00th=[ 514], 5.00th=[ 659], 10.00th=[ 835], 20.00th=[ 953], | 30.00th=[ 1053], 40.00th=[ 1133], 50.00th=[ 1217], 60.00th=[ 1318], | 70.00th=[ 1452], 80.00th=[ 1603], 90.00th=[ 1888], 95.00th=[ 2165], | 99.00th=[ 2567], 99.50th=[ 2735], 99.90th=[ 3071], 99.95th=[ 3104], | 99.99th=[ 3104] bw ( KiB/s): min= 7, max= 135, per=26.13%, avg=51.48, stdev=28.84, samples=390 iops : min= 1, max= 33, avg=12.40, stdev= 7.27, samples=390 lat (msec) : 250=0.08%, 500=0.90%, 750=6.25%, 1000=18.48%, 2000=66.84% lat (msec) : >=2000=7.46% cpu : usr=0.05%, sys=0.19%, ctx=1390, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=123539: Wed Apr 15 11:59:31 2026 write: IOPS=12, BW=49.3KiB/s (50.5kB/s)(10.0MiB/207716msec); 0 zone resets slat (usec): min=21, max=34184, avg=332.08, stdev=1737.38 clat (msec): min=167, max=3079, avg=1297.25, stdev=441.76 lat (msec): min=167, max=3079, avg=1297.58, stdev=441.82 clat percentiles (msec): | 1.00th=[ 523], 5.00th=[ 676], 10.00th=[ 835], 20.00th=[ 953], | 30.00th=[ 1045], 40.00th=[ 1133], 50.00th=[ 1217], 60.00th=[ 1318], | 70.00th=[ 1435], 80.00th=[ 1620], 90.00th=[ 1921], 95.00th=[ 2198], | 99.00th=[ 2500], 99.50th=[ 2903], 99.90th=[ 3071], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min= 7, max= 127, per=25.94%, avg=51.11, stdev=28.20, samples=393 iops : min= 1, max= 31, avg=12.30, stdev= 7.09, samples=393 lat (msec) : 250=0.04%, 500=0.55%, 750=6.05%, 1000=18.52%, 2000=66.48% lat (msec) : >=2000=8.36% cpu : usr=0.04%, sys=0.22%, ctx=1578, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=123540: Wed Apr 15 11:59:31 2026 write: IOPS=12, BW=49.3KiB/s (50.5kB/s)(10.0MiB/207809msec); 0 zone resets slat (usec): min=23, max=23650, avg=324.07, stdev=1483.15 clat (msec): min=348, max=3059, avg=1297.81, stdev=435.01 lat (msec): min=348, max=3060, avg=1298.13, stdev=435.09 clat percentiles (msec): | 1.00th=[ 531], 5.00th=[ 726], 10.00th=[ 844], 20.00th=[ 944], | 30.00th=[ 1045], 40.00th=[ 1133], 50.00th=[ 1234], 60.00th=[ 1318], | 70.00th=[ 1435], 80.00th=[ 1603], 90.00th=[ 1871], 95.00th=[ 2198], | 99.00th=[ 2668], 99.50th=[ 2903], 99.90th=[ 3071], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min= 7, max= 128, per=26.56%, avg=52.33, stdev=30.47, samples=384 iops : min= 1, max= 32, avg=12.61, stdev= 7.67, samples=384 lat (msec) : 500=0.27%, 750=5.51%, 1000=19.96%, 2000=66.91%, >=2000=7.34% cpu : usr=0.06%, sys=0.21%, ctx=1691, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=197KiB/s (202kB/s), 49.3KiB/s-49.4KiB/s (50.5kB/s-50.5kB/s), io=40.0MiB (41.9MB), run=207463-207809msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=123582: Wed Apr 15 12:01:42 2026 read: IOPS=10, BW=42.0KiB/s (43.0kB/s)(5048KiB/120096msec) slat (usec): min=60, max=72137, avg=14018.72, stdev=9336.66 clat (msec): min=13, max=1108, avg=491.50, stdev=169.69 lat (msec): min=18, max=1131, avg=505.52, stdev=171.26 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 284], 10.00th=[ 326], 20.00th=[ 359], | 30.00th=[ 393], 40.00th=[ 422], 50.00th=[ 456], 60.00th=[ 502], | 70.00th=[ 550], 80.00th=[ 625], 90.00th=[ 735], 95.00th=[ 827], | 99.00th=[ 953], 99.50th=[ 1011], 99.90th=[ 1083], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 104, per=24.99%, avg=41.48, stdev=22.08, samples=237 iops : min= 1, max= 26, avg= 9.81, stdev= 5.58, samples=237 write: IOPS=10, BW=43.2KiB/s (44.3kB/s)(5192KiB/120096msec); 0 zone resets slat (usec): min=20, max=19607, avg=344.97, stdev=1375.23 clat (msec): min=147, max=2069, avg=987.43, stdev=318.49 lat (msec): min=147, max=2069, avg=987.77, stdev=318.41 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 531], 10.00th=[ 625], 20.00th=[ 735], | 30.00th=[ 802], 40.00th=[ 869], 50.00th=[ 936], 60.00th=[ 1011], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1452], 95.00th=[ 1603], | 99.00th=[ 1838], 99.50th=[ 1989], 99.90th=[ 2022], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 111, per=24.80%, avg=42.91, stdev=19.23, samples=235 iops : min= 1, max= 27, avg=10.16, stdev= 4.84, samples=235 lat (msec) : 20=0.08%, 50=0.27%, 100=0.27%, 250=0.78%, 500=30.16% lat (msec) : 750=24.18%, 1000=22.54%, 2000=21.52%, >=2000=0.20% cpu : usr=0.11%, sys=0.66%, ctx=3218, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=123583: Wed Apr 15 12:01:42 2026 read: IOPS=10, BW=41.3KiB/s (42.2kB/s)(4976KiB/120607msec) slat (usec): min=1998, max=74255, avg=13885.92, stdev=9283.97 clat (msec): min=6, max=1044, avg=485.60, stdev=156.54 lat (msec): min=22, max=1064, avg=499.49, stdev=158.23 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 288], 10.00th=[ 321], 20.00th=[ 363], | 30.00th=[ 397], 40.00th=[ 422], 50.00th=[ 451], 60.00th=[ 493], | 70.00th=[ 542], 80.00th=[ 600], 90.00th=[ 709], 95.00th=[ 785], | 99.00th=[ 961], 99.50th=[ 995], 99.90th=[ 1045], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 7, max= 119, per=24.80%, avg=41.17, stdev=21.73, samples=235 iops : min= 1, max= 29, avg= 9.74, stdev= 5.49, samples=235 write: IOPS=10, BW=43.6KiB/s (44.7kB/s)(5264KiB/120607msec); 0 zone resets slat (usec): min=26, max=25430, avg=387.42, stdev=1666.96 clat (msec): min=341, max=2038, avg=991.41, stdev=309.40 lat (msec): min=342, max=2039, avg=991.82, stdev=309.41 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 518], 10.00th=[ 625], 20.00th=[ 735], | 30.00th=[ 802], 40.00th=[ 877], 50.00th=[ 953], 60.00th=[ 1036], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1452], 95.00th=[ 1586], | 99.00th=[ 1804], 99.50th=[ 1888], 99.90th=[ 2022], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 119, per=24.92%, avg=43.11, stdev=21.28, samples=238 iops : min= 1, max= 29, avg=10.22, stdev= 5.36, samples=238 lat (msec) : 10=0.04%, 20=0.04%, 50=0.08%, 100=0.12%, 250=0.78% lat (msec) : 500=31.45%, 750=23.91%, 1000=20.55%, 2000=22.97%, >=2000=0.08% cpu : usr=0.11%, sys=0.69%, ctx=3128, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=123584: Wed Apr 15 12:01:42 2026 read: IOPS=10, BW=41.4KiB/s (42.4kB/s)(4984KiB/120389msec) slat (usec): min=2253, max=72100, avg=14460.26, stdev=9484.23 clat (msec): min=19, max=1060, avg=482.60, stdev=154.81 lat (msec): min=25, max=1070, avg=497.07, stdev=156.67 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 284], 10.00th=[ 317], 20.00th=[ 359], | 30.00th=[ 397], 40.00th=[ 422], 50.00th=[ 447], 60.00th=[ 489], | 70.00th=[ 535], 80.00th=[ 600], 90.00th=[ 701], 95.00th=[ 793], | 99.00th=[ 919], 99.50th=[ 978], 99.90th=[ 1053], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 7, max= 141, per=25.08%, avg=41.64, stdev=22.28, samples=234 iops : min= 1, max= 35, avg= 9.86, stdev= 5.60, samples=234 write: IOPS=10, BW=43.7KiB/s (44.7kB/s)(5256KiB/120389msec); 0 zone resets slat (usec): min=22, max=33540, avg=374.10, stdev=1578.19 clat (msec): min=291, max=2039, avg=991.34, stdev=305.72 lat (msec): min=291, max=2040, avg=991.72, stdev=305.70 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 567], 10.00th=[ 642], 20.00th=[ 743], | 30.00th=[ 810], 40.00th=[ 885], 50.00th=[ 953], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1217], 90.00th=[ 1435], 95.00th=[ 1569], | 99.00th=[ 1838], 99.50th=[ 1905], 99.90th=[ 2022], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 96, per=24.77%, avg=42.84, stdev=19.92, samples=238 iops : min= 1, max= 24, avg=10.15, stdev= 5.04, samples=238 lat (msec) : 20=0.04%, 50=0.08%, 100=0.08%, 250=0.90%, 500=31.68% lat (msec) : 750=23.12%, 1000=21.48%, 2000=22.54%, >=2000=0.08% cpu : usr=0.12%, sys=0.71%, ctx=3229, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=123585: Wed Apr 15 12:01:42 2026 read: IOPS=10, BW=42.0KiB/s (43.0kB/s)(5060KiB/120376msec) slat (usec): min=2089, max=73803, avg=13829.83, stdev=9094.46 clat (msec): min=12, max=1108, avg=489.01, stdev=158.12 lat (msec): min=29, max=1130, avg=502.84, stdev=159.54 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 288], 10.00th=[ 317], 20.00th=[ 359], | 30.00th=[ 397], 40.00th=[ 426], 50.00th=[ 460], 60.00th=[ 493], | 70.00th=[ 542], 80.00th=[ 617], 90.00th=[ 718], 95.00th=[ 793], | 99.00th=[ 944], 99.50th=[ 995], 99.90th=[ 1020], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 119, per=25.79%, avg=42.81, stdev=21.12, samples=231 iops : min= 1, max= 29, avg=10.14, stdev= 5.30, samples=231 write: IOPS=10, BW=43.0KiB/s (44.1kB/s)(5180KiB/120376msec); 0 zone resets slat (usec): min=26, max=25872, avg=436.98, stdev=1799.58 clat (msec): min=216, max=2038, avg=993.22, stdev=307.15 lat (msec): min=217, max=2038, avg=993.66, stdev=307.19 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 550], 10.00th=[ 625], 20.00th=[ 735], | 30.00th=[ 810], 40.00th=[ 885], 50.00th=[ 953], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1435], 95.00th=[ 1569], | 99.00th=[ 1804], 99.50th=[ 1871], 99.90th=[ 1921], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 96, per=24.64%, avg=42.62, stdev=19.46, samples=236 iops : min= 1, max= 24, avg=10.10, stdev= 4.93, samples=236 lat (msec) : 20=0.04%, 100=0.16%, 250=0.78%, 500=31.48%, 750=24.45% lat (msec) : 1000=20.47%, 2000=22.58%, >=2000=0.04% cpu : usr=0.11%, sys=0.67%, ctx=3202, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=166KiB/s (170kB/s), 41.3KiB/s-42.0KiB/s (42.2kB/s-43.0kB/s), io=19.6MiB (20.5MB), run=120096-120607msec WRITE: bw=173KiB/s (177kB/s), 43.0KiB/s-43.7KiB/s (44.1kB/s-44.7kB/s), io=20.4MiB (21.4MB), run=120096-120607msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=123622: Wed Apr 15 12:01:54 2026 read: IOPS=1, BW=48.1MiB/s (50.5MB/s)(40.0MiB/831msec) slat (nsec): min=76622k, max=76622k, avg=76621690.00, stdev= 0.00 clat (nsec): min=721670k, max=721670k, avg=721669678.00, stdev= 0.00 lat (nsec): min=798297k, max=798297k, avg=798296960.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 726], 5.00th=[ 726], 10.00th=[ 726], 20.00th=[ 726], | 30.00th=[ 726], 40.00th=[ 726], 50.00th=[ 726], 60.00th=[ 726], | 70.00th=[ 726], 80.00th=[ 726], 90.00th=[ 726], 95.00th=[ 726], | 99.00th=[ 726], 99.50th=[ 726], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 726] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.20%, ctx=7, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=48.1MiB/s (50.5MB/s), 48.1MiB/s-48.1MiB/s (50.5MB/s-50.5MB/s), io=40.0MiB (41.9MB), run=831-831msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck