== sanity test 398c: run fio to test AIO ================= 20:25:59 (1788567959) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.505239 s, 83.0 MB/s osc.lustre-OST0000-osc-ffff895349cd4000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144446: Fri Sep 4 20:29:58 2026 write: IOPS=11, BW=45.6KiB/s (46.7kB/s)(10.0MiB/224736msec); 0 zone resets slat (usec): min=23, max=37498, avg=237.10, stdev=1319.04 clat (msec): min=50, max=3321, avg=1402.87, stdev=495.93 lat (msec): min=51, max=3322, avg=1403.11, stdev=495.94 clat percentiles (msec): | 1.00th=[ 510], 5.00th=[ 667], 10.00th=[ 827], 20.00th=[ 1020], | 30.00th=[ 1116], 40.00th=[ 1217], 50.00th=[ 1334], 60.00th=[ 1469], | 70.00th=[ 1620], 80.00th=[ 1804], 90.00th=[ 2072], 95.00th=[ 2265], | 99.00th=[ 2869], 99.50th=[ 2970], 99.90th=[ 3239], 99.95th=[ 3306], | 99.99th=[ 3306] bw ( KiB/s): min= 7, max= 129, per=25.93%, avg=47.19, stdev=29.22, samples=425 iops : min= 1, max= 32, avg=11.29, stdev= 7.35, samples=425 lat (msec) : 100=0.04%, 250=0.20%, 500=0.70%, 750=6.52%, 1000=11.05% lat (msec) : 2000=69.41%, >=2000=12.07% cpu : usr=0.04%, sys=0.17%, ctx=1763, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144447: Fri Sep 4 20:29:58 2026 write: IOPS=11, BW=45.6KiB/s (46.7kB/s)(10.0MiB/224430msec); 0 zone resets slat (usec): min=22, max=31845, avg=260.08, stdev=1403.64 clat (msec): min=136, max=3065, avg=1401.75, stdev=499.58 lat (msec): min=136, max=3065, avg=1402.01, stdev=499.58 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 676], 10.00th=[ 818], 20.00th=[ 1003], | 30.00th=[ 1116], 40.00th=[ 1217], 50.00th=[ 1351], 60.00th=[ 1469], | 70.00th=[ 1620], 80.00th=[ 1787], 90.00th=[ 2056], 95.00th=[ 2333], | 99.00th=[ 2869], 99.50th=[ 2937], 99.90th=[ 3071], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min= 7, max= 135, per=26.83%, avg=48.82, stdev=29.99, samples=411 iops : min= 1, max= 33, avg=11.70, stdev= 7.56, samples=411 lat (msec) : 250=0.12%, 500=1.37%, 750=5.31%, 1000=13.01%, 2000=68.24% lat (msec) : >=2000=11.95% cpu : usr=0.04%, sys=0.17%, ctx=1597, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144448: Fri Sep 4 20:29:58 2026 write: IOPS=11, BW=45.5KiB/s (46.6kB/s)(10.0MiB/224813msec); 0 zone resets slat (usec): min=22, max=20064, avg=202.25, stdev=976.53 clat (msec): min=221, max=3195, avg=1403.10, stdev=496.18 lat (msec): min=221, max=3195, avg=1403.30, stdev=496.17 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 693], 10.00th=[ 844], 20.00th=[ 1003], | 30.00th=[ 1116], 40.00th=[ 1217], 50.00th=[ 1334], 60.00th=[ 1469], | 70.00th=[ 1620], 80.00th=[ 1804], 90.00th=[ 2039], 95.00th=[ 2333], | 99.00th=[ 2937], 99.50th=[ 3004], 99.90th=[ 3138], 99.95th=[ 3138], | 99.99th=[ 3205] bw ( KiB/s): min= 7, max= 135, per=26.12%, avg=47.53, stdev=28.89, samples=422 iops : min= 1, max= 33, avg=11.39, stdev= 7.28, samples=422 lat (msec) : 250=0.62%, 500=0.70%, 750=5.59%, 1000=13.05%, 2000=68.67% lat (msec) : >=2000=11.37% cpu : usr=0.03%, sys=0.16%, ctx=1696, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144449: Fri Sep 4 20:29:58 2026 write: IOPS=11, BW=45.6KiB/s (46.6kB/s)(10.0MiB/224802msec); 0 zone resets slat (usec): min=21, max=40633, avg=243.31, stdev=1307.99 clat (msec): min=252, max=3224, avg=1403.18, stdev=494.91 lat (msec): min=252, max=3224, avg=1403.42, stdev=494.91 clat percentiles (msec): | 1.00th=[ 489], 5.00th=[ 684], 10.00th=[ 827], 20.00th=[ 995], | 30.00th=[ 1116], 40.00th=[ 1217], 50.00th=[ 1334], 60.00th=[ 1469], | 70.00th=[ 1620], 80.00th=[ 1804], 90.00th=[ 2056], 95.00th=[ 2333], | 99.00th=[ 2903], 99.50th=[ 2970], 99.90th=[ 3171], 99.95th=[ 3171], | 99.99th=[ 3239] bw ( KiB/s): min= 7, max= 128, per=26.00%, avg=47.31, stdev=26.98, samples=424 iops : min= 1, max= 32, avg=11.33, stdev= 6.81, samples=424 lat (msec) : 500=1.09%, 750=6.37%, 1000=13.24%, 2000=67.62%, >=2000=11.68% cpu : usr=0.04%, sys=0.19%, ctx=2037, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=182KiB/s (187kB/s), 45.5KiB/s-45.6KiB/s (46.6kB/s-46.7kB/s), io=40.0MiB (41.9MB), run=224430-224813msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144500: Fri Sep 4 20:32:07 2026 read: IOPS=10, BW=42.3KiB/s (43.3kB/s)(5048KiB/119402msec) slat (usec): min=2091, max=69996, avg=16212.40, stdev=9775.62 clat (msec): min=18, max=1111, avg=467.29, stdev=160.89 lat (msec): min=36, max=1141, avg=483.51, stdev=163.24 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 264], 10.00th=[ 300], 20.00th=[ 347], | 30.00th=[ 384], 40.00th=[ 409], 50.00th=[ 430], 60.00th=[ 464], | 70.00th=[ 510], 80.00th=[ 584], 90.00th=[ 684], 95.00th=[ 793], | 99.00th=[ 961], 99.50th=[ 1020], 99.90th=[ 1099], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 151, per=24.99%, avg=41.98, stdev=21.97, samples=235 iops : min= 1, max= 37, avg= 9.98, stdev= 5.53, samples=235 write: IOPS=10, BW=43.5KiB/s (44.5kB/s)(5192KiB/119402msec); 0 zone resets slat (usec): min=32, max=15946, avg=245.66, stdev=906.11 clat (msec): min=255, max=2068, avg=998.84, stdev=322.75 lat (msec): min=255, max=2068, avg=999.09, stdev=322.77 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 542], 10.00th=[ 617], 20.00th=[ 726], | 30.00th=[ 802], 40.00th=[ 869], 50.00th=[ 944], 60.00th=[ 1036], | 70.00th=[ 1150], 80.00th=[ 1284], 90.00th=[ 1435], 95.00th=[ 1569], | 99.00th=[ 1921], 99.50th=[ 1989], 99.90th=[ 2039], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 104, per=25.07%, avg=43.61, stdev=21.57, samples=233 iops : min= 1, max= 26, avg=10.38, stdev= 5.47, samples=233 lat (msec) : 20=0.04%, 100=0.16%, 250=1.72%, 500=33.52%, 750=22.15% lat (msec) : 1000=20.16%, 2000=22.07%, >=2000=0.20% cpu : usr=0.09%, sys=0.51%, ctx=4413, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144501: Fri Sep 4 20:32:07 2026 read: IOPS=10, BW=41.7KiB/s (42.7kB/s)(4976KiB/119347msec) slat (usec): min=2264, max=99486, avg=15720.45, stdev=9629.12 clat (msec): min=45, max=1138, avg=471.38, stdev=160.05 lat (msec): min=52, max=1159, avg=487.11, stdev=162.19 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 266], 10.00th=[ 305], 20.00th=[ 347], | 30.00th=[ 380], 40.00th=[ 409], 50.00th=[ 435], 60.00th=[ 472], | 70.00th=[ 518], 80.00th=[ 609], 90.00th=[ 701], 95.00th=[ 785], | 99.00th=[ 936], 99.50th=[ 978], 99.90th=[ 1028], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 7, max= 120, per=24.72%, avg=41.53, stdev=21.06, samples=234 iops : min= 1, max= 30, avg= 9.86, stdev= 5.29, samples=234 write: IOPS=11, BW=44.1KiB/s (45.2kB/s)(5264KiB/119347msec); 0 zone resets slat (usec): min=24, max=7828, avg=198.59, stdev=513.17 clat (msec): min=253, max=1984, avg=987.84, stdev=322.21 lat (msec): min=253, max=1984, avg=988.04, stdev=322.21 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 498], 10.00th=[ 600], 20.00th=[ 709], | 30.00th=[ 793], 40.00th=[ 877], 50.00th=[ 953], 60.00th=[ 1045], | 70.00th=[ 1133], 80.00th=[ 1267], 90.00th=[ 1435], 95.00th=[ 1552], | 99.00th=[ 1854], 99.50th=[ 1905], 99.90th=[ 1972], 99.95th=[ 1989], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 112, per=25.11%, avg=43.69, stdev=21.33, samples=236 iops : min= 1, max= 28, avg=10.40, stdev= 5.37, samples=236 lat (msec) : 50=0.08%, 100=0.20%, 250=1.48%, 500=33.01%, 750=23.40% lat (msec) : 1000=19.02%, 2000=22.81% cpu : usr=0.08%, sys=0.54%, ctx=4155, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144502: Fri Sep 4 20:32:07 2026 read: IOPS=10, BW=41.8KiB/s (42.8kB/s)(4984KiB/119227msec) slat (usec): min=2778, max=85489, avg=16020.18, stdev=9787.05 clat (msec): min=11, max=1042, avg=472.22, stdev=167.72 lat (msec): min=21, max=1065, avg=488.25, stdev=170.27 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 266], 10.00th=[ 300], 20.00th=[ 342], | 30.00th=[ 376], 40.00th=[ 405], 50.00th=[ 439], 60.00th=[ 472], | 70.00th=[ 518], 80.00th=[ 600], 90.00th=[ 726], 95.00th=[ 802], | 99.00th=[ 953], 99.50th=[ 1003], 99.90th=[ 1028], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 7, max= 111, per=25.08%, avg=42.13, stdev=21.54, samples=232 iops : min= 1, max= 27, avg=10.01, stdev= 5.42, samples=232 write: IOPS=11, BW=44.1KiB/s (45.1kB/s)(5256KiB/119227msec); 0 zone resets slat (usec): min=24, max=29012, avg=266.15, stdev=1280.30 clat (msec): min=202, max=2128, avg=985.98, stdev=319.32 lat (msec): min=202, max=2128, avg=986.25, stdev=319.27 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 535], 10.00th=[ 609], 20.00th=[ 709], | 30.00th=[ 793], 40.00th=[ 860], 50.00th=[ 927], 60.00th=[ 1028], | 70.00th=[ 1133], 80.00th=[ 1267], 90.00th=[ 1435], 95.00th=[ 1569], | 99.00th=[ 1838], 99.50th=[ 1938], 99.90th=[ 2039], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 126, per=25.19%, avg=43.83, stdev=20.97, samples=234 iops : min= 1, max= 31, avg=10.44, stdev= 5.28, samples=234 lat (msec) : 20=0.04%, 50=0.16%, 100=0.27%, 250=1.56%, 500=31.84% lat (msec) : 750=23.40%, 1000=20.62%, 2000=22.03%, >=2000=0.08% cpu : usr=0.09%, sys=0.53%, ctx=4126, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144503: Fri Sep 4 20:32:07 2026 read: IOPS=10, BW=42.7KiB/s (43.7kB/s)(5060KiB/118478msec) slat (usec): min=54, max=109284, avg=15574.13, stdev=10305.28 clat (msec): min=13, max=1182, avg=475.75, stdev=164.56 lat (msec): min=19, max=1211, avg=491.33, stdev=167.20 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 275], 10.00th=[ 313], 20.00th=[ 355], | 30.00th=[ 380], 40.00th=[ 409], 50.00th=[ 443], 60.00th=[ 472], | 70.00th=[ 518], 80.00th=[ 600], 90.00th=[ 718], 95.00th=[ 810], | 99.00th=[ 953], 99.50th=[ 978], 99.90th=[ 1053], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 7, max= 136, per=25.91%, avg=43.53, stdev=21.25, samples=228 iops : min= 1, max= 34, avg=10.36, stdev= 5.36, samples=228 write: IOPS=10, BW=43.7KiB/s (44.8kB/s)(5180KiB/118478msec); 0 zone resets slat (usec): min=25, max=66985, avg=323.20, stdev=2343.62 clat (msec): min=138, max=2118, avg=982.82, stdev=329.32 lat (msec): min=138, max=2118, avg=983.15, stdev=329.36 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 493], 10.00th=[ 592], 20.00th=[ 709], | 30.00th=[ 793], 40.00th=[ 869], 50.00th=[ 944], 60.00th=[ 1011], | 70.00th=[ 1116], 80.00th=[ 1267], 90.00th=[ 1418], 95.00th=[ 1569], | 99.00th=[ 1905], 99.50th=[ 1989], 99.90th=[ 2089], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 104, per=25.00%, avg=43.49, stdev=21.31, samples=232 iops : min= 1, max= 26, avg=10.36, stdev= 5.37, samples=232 lat (msec) : 20=0.04%, 50=0.16%, 100=0.20%, 250=1.56%, 500=33.05% lat (msec) : 750=22.85%, 1000=20.66%, 2000=21.29%, >=2000=0.20% cpu : usr=0.09%, sys=0.52%, ctx=4525, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=168KiB/s (172kB/s), 41.7KiB/s-42.7KiB/s (42.7kB/s-43.7kB/s), io=19.6MiB (20.5MB), run=118478-119402msec WRITE: bw=175KiB/s (179kB/s), 43.5KiB/s-44.1KiB/s (44.5kB/s-45.2kB/s), io=20.4MiB (21.4MB), run=118478-119402msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144525: Fri Sep 4 20:32:20 2026 read: IOPS=1, BW=63.6MiB/s (66.7MB/s)(40.0MiB/629msec) slat (nsec): min=42376k, max=42376k, avg=42375996.00, stdev= 0.00 clat (nsec): min=562113k, max=562113k, avg=562113214.00, stdev= 0.00 lat (nsec): min=604494k, max=604494k, avg=604493945.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 567], 10.00th=[ 567], 20.00th=[ 567], | 30.00th=[ 567], 40.00th=[ 567], 50.00th=[ 567], 60.00th=[ 567], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.27%, ctx=5, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=63.6MiB/s (66.7MB/s), 63.6MiB/s-63.6MiB/s (66.7MB/s-66.7MB/s), io=40.0MiB (41.9MB), run=629-629msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout