== sanity test 398c: run fio to test AIO ================= 17:15:20 (1787865320) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.448022 s, 93.6 MB/s osc.lustre-OST0000-osc-ffff9c594397c800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145469: Thu Aug 27 17:18:20 2026 write: IOPS=15, BW=60.9KiB/s (62.4kB/s)(10.0MiB/168060msec); 0 zone resets slat (usec): min=22, max=20486, avg=183.64, stdev=805.83 clat (msec): min=220, max=2127, avg=1049.75, stdev=273.22 lat (msec): min=220, max=2128, avg=1049.93, stdev=273.23 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 642], 10.00th=[ 726], 20.00th=[ 844], | 30.00th=[ 927], 40.00th=[ 978], 50.00th=[ 1045], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1401], 95.00th=[ 1519], | 99.00th=[ 1737], 99.50th=[ 1854], 99.90th=[ 2056], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 160, per=25.12%, avg=61.04, stdev=26.66, samples=330 iops : min= 1, max= 40, avg=14.82, stdev= 6.69, samples=330 lat (msec) : 250=0.62%, 500=1.99%, 750=9.06%, 1000=31.52%, 2000=56.60% lat (msec) : >=2000=0.20% cpu : usr=0.04%, sys=0.20%, ctx=1501, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145470: Thu Aug 27 17:18:20 2026 write: IOPS=15, BW=60.9KiB/s (62.3kB/s)(10.0MiB/168179msec); 0 zone resets slat (usec): min=25, max=17678, avg=177.67, stdev=731.90 clat (msec): min=201, max=2114, avg=1049.69, stdev=268.07 lat (msec): min=201, max=2114, avg=1049.87, stdev=268.08 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 651], 10.00th=[ 751], 20.00th=[ 844], | 30.00th=[ 919], 40.00th=[ 969], 50.00th=[ 1036], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1401], 95.00th=[ 1536], | 99.00th=[ 1754], 99.50th=[ 1921], 99.90th=[ 1938], 99.95th=[ 2039], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 192, per=25.03%, avg=60.82, stdev=26.31, samples=331 iops : min= 1, max= 48, avg=14.77, stdev= 6.61, samples=331 lat (msec) : 250=0.04%, 500=2.23%, 750=7.97%, 1000=34.10%, 2000=55.59% lat (msec) : >=2000=0.08% cpu : usr=0.04%, sys=0.23%, ctx=2039, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145471: Thu Aug 27 17:18:20 2026 write: IOPS=15, BW=60.9KiB/s (62.4kB/s)(10.0MiB/168126msec); 0 zone resets slat (usec): min=22, max=14152, avg=176.58, stdev=651.84 clat (msec): min=341, max=2036, avg=1049.43, stdev=267.78 lat (msec): min=341, max=2036, avg=1049.61, stdev=267.79 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 634], 10.00th=[ 735], 20.00th=[ 844], | 30.00th=[ 919], 40.00th=[ 978], 50.00th=[ 1036], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1385], 95.00th=[ 1536], | 99.00th=[ 1754], 99.50th=[ 1854], 99.90th=[ 2022], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 256, per=25.40%, avg=61.72, stdev=29.95, samples=326 iops : min= 1, max= 64, avg=14.99, stdev= 7.50, samples=326 lat (msec) : 500=2.15%, 750=8.59%, 1000=33.40%, 2000=55.74%, >=2000=0.12% cpu : usr=0.04%, sys=0.18%, ctx=1551, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145472: Thu Aug 27 17:18:20 2026 write: IOPS=15, BW=60.9KiB/s (62.4kB/s)(10.0MiB/168058msec); 0 zone resets slat (usec): min=22, max=25486, avg=209.56, stdev=924.07 clat (msec): min=86, max=2134, avg=1049.03, stdev=273.65 lat (msec): min=86, max=2134, avg=1049.24, stdev=273.65 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 634], 10.00th=[ 735], 20.00th=[ 844], | 30.00th=[ 927], 40.00th=[ 986], 50.00th=[ 1036], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1401], 95.00th=[ 1519], | 99.00th=[ 1770], 99.50th=[ 1871], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2140] bw ( KiB/s): min= 7, max= 200, per=25.12%, avg=61.03, stdev=29.76, samples=330 iops : min= 1, max= 50, avg=14.82, stdev= 7.44, samples=330 lat (msec) : 100=0.04%, 250=0.20%, 500=2.19%, 750=8.95%, 1000=32.54% lat (msec) : 2000=55.86%, >=2000=0.23% cpu : usr=0.04%, sys=0.20%, ctx=1942, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=244KiB/s (249kB/s), 60.9KiB/s-60.9KiB/s (62.3kB/s-62.4kB/s), io=40.0MiB (41.9MB), run=168058-168179msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145506: Thu Aug 27 17:20:27 2026 read: IOPS=10, BW=43.0KiB/s (45.0kB/s)(5048KiB/114801msec) slat (msec): min=3, max=113, avg=15.85, stdev= 9.42 clat (msec): min=35, max=969, avg=435.39, stdev=122.19 lat (msec): min=48, max=992, avg=451.25, stdev=123.81 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 271], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 372], 40.00th=[ 401], 50.00th=[ 422], 60.00th=[ 447], | 70.00th=[ 477], 80.00th=[ 510], 90.00th=[ 584], 95.00th=[ 659], | 99.00th=[ 860], 99.50th=[ 894], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 7, max= 103, per=25.02%, avg=43.53, stdev=20.25, samples=227 iops : min= 1, max= 25, avg=10.37, stdev= 5.08, samples=227 write: IOPS=11, BW=45.2KiB/s (46.3kB/s)(5192KiB/114801msec); 0 zone resets slat (usec): min=25, max=24680, avg=281.95, stdev=1321.35 clat (msec): min=400, max=1808, avg=972.45, stdev=254.95 lat (msec): min=401, max=1808, avg=972.74, stdev=254.95 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 592], 10.00th=[ 651], 20.00th=[ 743], | 30.00th=[ 818], 40.00th=[ 885], 50.00th=[ 961], 60.00th=[ 1028], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1586], 99.50th=[ 1703], 99.90th=[ 1787], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 7, max= 104, per=24.69%, avg=44.69, stdev=18.48, samples=227 iops : min= 1, max= 26, avg=10.66, stdev= 4.66, samples=227 lat (msec) : 50=0.04%, 100=0.43%, 250=1.09%, 500=37.15%, 750=20.16% lat (msec) : 1000=18.83%, 2000=22.30% cpu : usr=0.08%, sys=0.56%, ctx=4288, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145507: Thu Aug 27 17:20:27 2026 read: IOPS=10, BW=43.3KiB/s (44.3kB/s)(4976KiB/114923msec) slat (usec): min=3225, max=85502, avg=15538.45, stdev=9301.80 clat (msec): min=27, max=936, avg=437.39, stdev=124.37 lat (msec): min=34, max=970, avg=452.94, stdev=126.00 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 271], 10.00th=[ 305], 20.00th=[ 347], | 30.00th=[ 376], 40.00th=[ 397], 50.00th=[ 422], 60.00th=[ 447], | 70.00th=[ 481], 80.00th=[ 518], 90.00th=[ 600], 95.00th=[ 667], | 99.00th=[ 827], 99.50th=[ 885], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 7, max= 111, per=24.91%, avg=43.35, stdev=20.05, samples=224 iops : min= 1, max= 27, avg=10.33, stdev= 4.99, samples=224 write: IOPS=11, BW=45.8KiB/s (46.9kB/s)(5264KiB/114923msec); 0 zone resets slat (usec): min=27, max=23646, avg=255.93, stdev=999.39 clat (msec): min=338, max=1804, avg=965.46, stdev=253.94 lat (msec): min=338, max=1804, avg=965.71, stdev=253.98 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 567], 10.00th=[ 651], 20.00th=[ 735], | 30.00th=[ 818], 40.00th=[ 894], 50.00th=[ 961], 60.00th=[ 1028], | 70.00th=[ 1099], 80.00th=[ 1167], 90.00th=[ 1301], 95.00th=[ 1401], | 99.00th=[ 1620], 99.50th=[ 1687], 99.90th=[ 1703], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 7, max= 112, per=25.22%, avg=45.64, stdev=20.41, samples=226 iops : min= 1, max= 28, avg=10.90, stdev= 5.13, samples=226 lat (msec) : 50=0.20%, 100=0.23%, 250=1.09%, 500=36.09%, 750=21.17% lat (msec) : 1000=18.67%, 2000=22.54% cpu : usr=0.09%, sys=0.61%, ctx=4436, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145508: Thu Aug 27 17:20:27 2026 read: IOPS=10, BW=43.4KiB/s (44.4kB/s)(4984KiB/114823msec) slat (msec): min=2, max=111, avg=15.35, stdev= 9.14 clat (msec): min=9, max=969, avg=435.16, stdev=122.19 lat (msec): min=21, max=992, avg=450.52, stdev=123.86 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 275], 10.00th=[ 309], 20.00th=[ 342], | 30.00th=[ 368], 40.00th=[ 397], 50.00th=[ 418], 60.00th=[ 447], | 70.00th=[ 477], 80.00th=[ 518], 90.00th=[ 592], 95.00th=[ 659], | 99.00th=[ 827], 99.50th=[ 860], 99.90th=[ 953], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 7, max= 119, per=24.63%, avg=42.86, stdev=22.36, samples=228 iops : min= 1, max= 29, avg=10.20, stdev= 5.58, samples=228 write: IOPS=11, BW=45.8KiB/s (46.9kB/s)(5256KiB/114823msec); 0 zone resets slat (usec): min=27, max=28384, avg=246.42, stdev=1091.21 clat (msec): min=296, max=1807, avg=967.16, stdev=256.61 lat (msec): min=296, max=1807, avg=967.41, stdev=256.63 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 575], 10.00th=[ 642], 20.00th=[ 743], | 30.00th=[ 810], 40.00th=[ 877], 50.00th=[ 953], 60.00th=[ 1028], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1318], 95.00th=[ 1401], | 99.00th=[ 1620], 99.50th=[ 1720], 99.90th=[ 1787], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 7, max= 95, per=25.06%, avg=45.37, stdev=19.02, samples=226 iops : min= 1, max= 23, avg=10.83, stdev= 4.80, samples=226 lat (msec) : 10=0.04%, 50=0.04%, 100=0.35%, 250=1.05%, 500=36.60% lat (msec) : 750=20.31%, 1000=19.30%, 2000=22.30% cpu : usr=0.09%, sys=0.55%, ctx=4225, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145509: Thu Aug 27 17:20:27 2026 read: IOPS=11, BW=44.2KiB/s (45.3kB/s)(5060KiB/114391msec) slat (usec): min=51, max=93826, avg=15497.80, stdev=9276.42 clat (msec): min=19, max=1181, avg=439.93, stdev=131.57 lat (msec): min=30, max=1194, avg=455.44, stdev=133.24 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 266], 10.00th=[ 305], 20.00th=[ 351], | 30.00th=[ 380], 40.00th=[ 401], 50.00th=[ 426], 60.00th=[ 451], | 70.00th=[ 481], 80.00th=[ 514], 90.00th=[ 592], 95.00th=[ 684], | 99.00th=[ 844], 99.50th=[ 919], 99.90th=[ 1167], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 7, max= 103, per=25.45%, avg=44.29, stdev=18.59, samples=224 iops : min= 1, max= 25, avg=10.56, stdev= 4.66, samples=224 write: IOPS=11, BW=45.3KiB/s (46.4kB/s)(5180KiB/114391msec); 0 zone resets slat (usec): min=31, max=16736, avg=264.26, stdev=820.74 clat (msec): min=154, max=2049, avg=966.97, stdev=268.14 lat (msec): min=154, max=2049, avg=967.24, stdev=268.14 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 584], 10.00th=[ 642], 20.00th=[ 743], | 30.00th=[ 802], 40.00th=[ 869], 50.00th=[ 953], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1167], 90.00th=[ 1318], 95.00th=[ 1435], | 99.00th=[ 1670], 99.50th=[ 1770], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 103, per=24.68%, avg=44.67, stdev=18.33, samples=226 iops : min= 1, max= 25, avg=10.66, stdev= 4.62, samples=226 lat (msec) : 20=0.04%, 50=0.23%, 100=0.27%, 250=1.52%, 500=36.56% lat (msec) : 750=19.84%, 1000=19.10%, 2000=22.34%, >=2000=0.08% cpu : usr=0.10%, sys=0.59%, ctx=4827, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=175KiB/s (179kB/s), 43.3KiB/s-44.2KiB/s (44.3kB/s-45.3kB/s), io=19.6MiB (20.5MB), run=114391-114923msec WRITE: bw=182KiB/s (186kB/s), 45.2KiB/s-45.8KiB/s (46.3kB/s-46.9kB/s), io=20.4MiB (21.4MB), run=114391-114923msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145530: Thu Aug 27 17:20:40 2026 read: IOPS=2, BW=107MiB/s (112MB/s)(40.0MiB/373msec) slat (nsec): min=21787k, max=21787k, avg=21787205.00, stdev= 0.00 clat (nsec): min=319946k, max=319946k, avg=319945724.00, stdev= 0.00 lat (nsec): min=341765k, max=341765k, avg=341764582.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 321], 10.00th=[ 321], 20.00th=[ 321], | 30.00th=[ 321], 40.00th=[ 321], 50.00th=[ 321], 60.00th=[ 321], | 70.00th=[ 321], 80.00th=[ 321], 90.00th=[ 321], 95.00th=[ 321], | 99.00th=[ 321], 99.50th=[ 321], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.96%, ctx=11, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=107MiB/s (112MB/s), 107MiB/s-107MiB/s (112MB/s-112MB/s), io=40.0MiB (41.9MB), run=373-373msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout