== sanity test 398c: run fio to test AIO ================= 18:38:08 (1773614288) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.965372 s, 43.4 MB/s osc.lustre-OST0000-osc-ffff9315c03d1800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147265: Sun Mar 15 18:43:23 2026 write: IOPS=8, BW=34.5KiB/s (35.4kB/s)(10.0MiB/296552msec); 0 zone resets slat (usec): min=41, max=31476, avg=607.74, stdev=1861.88 clat (msec): min=86, max=4220, avg=1851.07, stdev=587.83 lat (msec): min=86, max=4220, avg=1851.68, stdev=587.88 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 953], 10.00th=[ 1234], 20.00th=[ 1435], | 30.00th=[ 1536], 40.00th=[ 1653], 50.00th=[ 1804], 60.00th=[ 1938], | 70.00th=[ 2089], 80.00th=[ 2299], 90.00th=[ 2567], 95.00th=[ 2802], | 99.00th=[ 3775], 99.50th=[ 4010], 99.90th=[ 4178], 99.95th=[ 4212], | 99.99th=[ 4212] bw ( KiB/s): min= 7, max= 199, per=27.52%, avg=37.98, stdev=25.43, samples=524 iops : min= 1, max= 49, avg= 8.88, stdev= 6.39, samples=524 lat (msec) : 100=0.04%, 250=0.08%, 500=1.56%, 750=1.64%, 1000=2.07% lat (msec) : 2000=57.66%, >=2000=36.95% cpu : usr=0.05%, sys=0.25%, ctx=1892, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147266: Sun Mar 15 18:43:23 2026 write: IOPS=8, BW=34.6KiB/s (35.4kB/s)(10.0MiB/296311msec); 0 zone resets slat (usec): min=40, max=30077, avg=580.09, stdev=1825.66 clat (msec): min=310, max=4208, avg=1850.43, stdev=592.25 lat (msec): min=310, max=4211, avg=1851.02, stdev=592.29 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 978], 10.00th=[ 1234], 20.00th=[ 1401], | 30.00th=[ 1536], 40.00th=[ 1670], 50.00th=[ 1804], 60.00th=[ 1955], | 70.00th=[ 2140], 80.00th=[ 2265], 90.00th=[ 2601], 95.00th=[ 2802], | 99.00th=[ 3742], 99.50th=[ 4044], 99.90th=[ 4178], 99.95th=[ 4212], | 99.99th=[ 4212] bw ( KiB/s): min= 7, max= 128, per=27.28%, avg=37.65, stdev=24.17, samples=528 iops : min= 1, max= 32, avg= 8.80, stdev= 6.07, samples=528 lat (msec) : 500=1.88%, 750=1.29%, 1000=2.19%, 2000=57.03%, >=2000=37.62% cpu : usr=0.04%, sys=0.28%, ctx=1791, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147267: Sun Mar 15 18:43:23 2026 write: IOPS=8, BW=34.5KiB/s (35.3kB/s)(10.0MiB/296643msec); 0 zone resets slat (usec): min=38, max=23238, avg=498.32, stdev=1445.36 clat (msec): min=341, max=4231, avg=1851.71, stdev=577.03 lat (msec): min=341, max=4231, avg=1852.21, stdev=577.09 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 978], 10.00th=[ 1200], 20.00th=[ 1401], | 30.00th=[ 1536], 40.00th=[ 1687], 50.00th=[ 1838], 60.00th=[ 1989], | 70.00th=[ 2106], 80.00th=[ 2299], 90.00th=[ 2567], 95.00th=[ 2769], | 99.00th=[ 3574], 99.50th=[ 3910], 99.90th=[ 4212], 99.95th=[ 4245], | 99.99th=[ 4245] bw ( KiB/s): min= 7, max= 255, per=27.39%, avg=37.79, stdev=24.36, samples=527 iops : min= 1, max= 63, avg= 8.84, stdev= 6.09, samples=527 lat (msec) : 500=1.25%, 750=2.07%, 1000=1.99%, 2000=56.13%, >=2000=38.55% cpu : usr=0.04%, sys=0.25%, ctx=1517, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147268: Sun Mar 15 18:43:23 2026 write: IOPS=8, BW=34.5KiB/s (35.3kB/s)(10.0MiB/296753msec); 0 zone resets slat (usec): min=41, max=30367, avg=524.22, stdev=1599.26 clat (msec): min=336, max=4317, avg=1852.46, stdev=579.33 lat (msec): min=337, max=4317, avg=1852.99, stdev=579.33 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 953], 10.00th=[ 1217], 20.00th=[ 1435], | 30.00th=[ 1536], 40.00th=[ 1670], 50.00th=[ 1787], 60.00th=[ 1989], | 70.00th=[ 2123], 80.00th=[ 2265], 90.00th=[ 2534], 95.00th=[ 2836], | 99.00th=[ 3608], 99.50th=[ 4010], 99.90th=[ 4245], 99.95th=[ 4329], | 99.99th=[ 4329] bw ( KiB/s): min= 7, max= 255, per=28.20%, avg=38.91, stdev=26.85, samples=512 iops : min= 1, max= 63, avg= 9.12, stdev= 6.73, samples=512 lat (msec) : 500=1.25%, 750=1.91%, 1000=2.15%, 2000=55.70%, >=2000=38.98% cpu : usr=0.04%, sys=0.25%, ctx=1578, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=138KiB/s (141kB/s), 34.5KiB/s-34.6KiB/s (35.3kB/s-35.4kB/s), io=40.0MiB (41.9MB), run=296311-296753msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147321: Sun Mar 15 18:46:43 2026 read: IOPS=6, BW=27.2KiB/s (27.9kB/s)(5048KiB/185358msec) slat (usec): min=4251, max=85325, avg=23415.50, stdev=11917.49 clat (msec): min=42, max=1624, avg=739.47, stdev=219.70 lat (msec): min=52, max=1656, avg=762.90, stdev=222.38 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 435], 10.00th=[ 493], 20.00th=[ 558], | 30.00th=[ 617], 40.00th=[ 659], 50.00th=[ 709], 60.00th=[ 768], | 70.00th=[ 835], 80.00th=[ 919], 90.00th=[ 1028], 95.00th=[ 1116], | 99.00th=[ 1385], 99.50th=[ 1485], 99.90th=[ 1620], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 7, max= 79, per=26.56%, avg=28.69, stdev=16.16, samples=340 iops : min= 1, max= 19, avg= 6.53, stdev= 4.07, samples=340 write: IOPS=7, BW=28.0KiB/s (28.7kB/s)(5192KiB/185358msec); 0 zone resets slat (usec): min=53, max=35632, avg=755.56, stdev=2178.78 clat (msec): min=685, max=3105, avg=1536.67, stdev=422.25 lat (msec): min=685, max=3105, avg=1537.43, stdev=422.29 clat percentiles (msec): | 1.00th=[ 852], 5.00th=[ 944], 10.00th=[ 1020], 20.00th=[ 1150], | 30.00th=[ 1267], 40.00th=[ 1401], 50.00th=[ 1502], 60.00th=[ 1620], | 70.00th=[ 1737], 80.00th=[ 1838], 90.00th=[ 2072], 95.00th=[ 2299], | 99.00th=[ 2802], 99.50th=[ 2970], 99.90th=[ 3104], 99.95th=[ 3104], | 99.99th=[ 3104] bw ( KiB/s): min= 7, max= 71, per=25.64%, avg=28.72, stdev=15.19, samples=348 iops : min= 1, max= 17, avg= 6.53, stdev= 3.80, samples=348 lat (msec) : 50=0.08%, 100=0.08%, 500=5.23%, 750=22.85%, 1000=19.30% lat (msec) : 2000=46.37%, >=2000=6.09% cpu : usr=0.10%, sys=0.68%, ctx=4388, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147322: Sun Mar 15 18:46:43 2026 read: IOPS=6, BW=26.8KiB/s (27.5kB/s)(4976KiB/185359msec) slat (usec): min=5024, max=89864, avg=24052.59, stdev=12423.51 clat (msec): min=22, max=1623, avg=742.32, stdev=225.49 lat (msec): min=55, max=1673, avg=766.38, stdev=227.67 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 443], 10.00th=[ 489], 20.00th=[ 567], | 30.00th=[ 617], 40.00th=[ 667], 50.00th=[ 709], 60.00th=[ 768], | 70.00th=[ 844], 80.00th=[ 919], 90.00th=[ 1045], 95.00th=[ 1133], | 99.00th=[ 1368], 99.50th=[ 1469], 99.90th=[ 1586], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 7, max= 79, per=25.97%, avg=28.05, stdev=15.52, samples=342 iops : min= 1, max= 19, avg= 6.37, stdev= 3.89, samples=342 write: IOPS=7, BW=28.4KiB/s (29.1kB/s)(5264KiB/185359msec); 0 zone resets slat (usec): min=44, max=22310, avg=719.83, stdev=1706.01 clat (msec): min=251, max=3021, avg=1524.17, stdev=425.69 lat (msec): min=259, max=3026, avg=1524.90, stdev=425.65 clat percentiles (msec): | 1.00th=[ 785], 5.00th=[ 927], 10.00th=[ 1011], 20.00th=[ 1133], | 30.00th=[ 1267], 40.00th=[ 1368], 50.00th=[ 1502], 60.00th=[ 1603], | 70.00th=[ 1720], 80.00th=[ 1854], 90.00th=[ 2056], 95.00th=[ 2333], | 99.00th=[ 2735], 99.50th=[ 2802], 99.90th=[ 3037], 99.95th=[ 3037], | 99.99th=[ 3037] bw ( KiB/s): min= 7, max= 79, per=25.81%, avg=28.91, stdev=15.95, samples=351 iops : min= 1, max= 19, avg= 6.58, stdev= 3.99, samples=351 lat (msec) : 50=0.12%, 100=0.16%, 250=0.43%, 500=4.69%, 750=22.07% lat (msec) : 1000=19.45%, 2000=46.72%, >=2000=6.37% cpu : usr=0.09%, sys=0.67%, ctx=4397, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147323: Sun Mar 15 18:46:43 2026 read: IOPS=6, BW=26.9KiB/s (27.5kB/s)(4984KiB/185364msec) slat (usec): min=58, max=84145, avg=23756.58, stdev=12116.87 clat (msec): min=54, max=1966, avg=741.91, stdev=223.80 lat (msec): min=82, max=2012, avg=765.68, stdev=226.45 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 439], 10.00th=[ 493], 20.00th=[ 575], | 30.00th=[ 617], 40.00th=[ 659], 50.00th=[ 718], 60.00th=[ 768], | 70.00th=[ 827], 80.00th=[ 911], 90.00th=[ 1036], 95.00th=[ 1150], | 99.00th=[ 1368], 99.50th=[ 1502], 99.90th=[ 1620], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 87, per=25.81%, avg=27.88, stdev=14.99, samples=345 iops : min= 1, max= 21, avg= 6.33, stdev= 3.77, samples=345 write: IOPS=7, BW=28.4KiB/s (29.0kB/s)(5256KiB/185364msec); 0 zone resets slat (usec): min=49, max=33215, avg=705.68, stdev=1870.07 clat (msec): min=306, max=3211, avg=1528.91, stdev=424.91 lat (msec): min=306, max=3212, avg=1529.62, stdev=424.93 clat percentiles (msec): | 1.00th=[ 743], 5.00th=[ 944], 10.00th=[ 1003], 20.00th=[ 1150], | 30.00th=[ 1284], 40.00th=[ 1401], 50.00th=[ 1502], 60.00th=[ 1620], | 70.00th=[ 1737], 80.00th=[ 1871], 90.00th=[ 2072], 95.00th=[ 2299], | 99.00th=[ 2635], 99.50th=[ 2802], 99.90th=[ 3071], 99.95th=[ 3205], | 99.99th=[ 3205] bw ( KiB/s): min= 7, max= 87, per=25.40%, avg=28.45, stdev=16.08, samples=354 iops : min= 1, max= 21, avg= 6.47, stdev= 4.02, samples=354 lat (msec) : 100=0.16%, 250=0.39%, 500=5.04%, 750=21.91%, 1000=20.23% lat (msec) : 2000=45.98%, >=2000=6.29% cpu : usr=0.11%, sys=0.66%, ctx=4458, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147324: Sun Mar 15 18:46:43 2026 read: IOPS=6, BW=27.3KiB/s (27.0kB/s)(5060KiB/185062msec) slat (usec): min=4211, max=97920, avg=23614.50, stdev=12278.41 clat (msec): min=33, max=1927, avg=734.19, stdev=222.32 lat (msec): min=55, max=1948, avg=757.81, stdev=224.97 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 422], 10.00th=[ 485], 20.00th=[ 558], | 30.00th=[ 609], 40.00th=[ 659], 50.00th=[ 718], 60.00th=[ 760], | 70.00th=[ 818], 80.00th=[ 894], 90.00th=[ 1028], 95.00th=[ 1133], | 99.00th=[ 1351], 99.50th=[ 1469], 99.90th=[ 1905], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 5, max= 79, per=26.73%, avg=28.87, stdev=16.09, samples=339 iops : min= 1, max= 19, avg= 6.58, stdev= 4.03, samples=339 write: IOPS=6, BW=27.0KiB/s (28.7kB/s)(5180KiB/185062msec); 0 zone resets slat (usec): min=48, max=15076, avg=567.97, stdev=1337.74 clat (msec): min=458, max=3122, avg=1540.02, stdev=428.66 lat (msec): min=459, max=3122, avg=1540.60, stdev=428.57 clat percentiles (msec): | 1.00th=[ 785], 5.00th=[ 927], 10.00th=[ 1011], 20.00th=[ 1167], | 30.00th=[ 1284], 40.00th=[ 1401], 50.00th=[ 1519], 60.00th=[ 1620], | 70.00th=[ 1737], 80.00th=[ 1871], 90.00th=[ 2072], 95.00th=[ 2366], | 99.00th=[ 2735], 99.50th=[ 2802], 99.90th=[ 3104], 99.95th=[ 3138], | 99.99th=[ 3138] bw ( KiB/s): min= 7, max= 87, per=25.12%, avg=28.13, stdev=15.47, samples=354 iops : min= 1, max= 21, avg= 6.39, stdev= 3.88, samples=354 lat (msec) : 50=0.08%, 100=0.08%, 250=0.23%, 500=5.51%, 750=22.54% lat (msec) : 1000=19.65%, 2000=45.35%, >=2000=6.56% cpu : usr=0.09%, sys=0.62%, ctx=4314, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=108KiB/s (111kB/s), 26.8KiB/s-27.3KiB/s (27.5kB/s-27.0kB/s), io=19.6MiB (20.5MB), run=185062-185364msec WRITE: bw=113KiB/s (115kB/s), 27.0KiB/s-28.4KiB/s (28.7kB/s-29.1kB/s), io=20.4MiB (21.4MB), run=185062-185364msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147356: Sun Mar 15 18:47:01 2026 read: IOPS=1, BW=42.3MiB/s (44.3MB/s)(40.0MiB/946msec) slat (nsec): min=48132k, max=48132k, avg=48132186.00, stdev= 0.00 clat (nsec): min=859756k, max=859756k, avg=859755635.00, stdev= 0.00 lat (nsec): min=907895k, max=907895k, avg=907894907.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 860], 5.00th=[ 860], 10.00th=[ 860], 20.00th=[ 860], | 30.00th=[ 860], 40.00th=[ 860], 50.00th=[ 860], 60.00th=[ 860], | 70.00th=[ 860], 80.00th=[ 860], 90.00th=[ 860], 95.00th=[ 860], | 99.00th=[ 860], 99.50th=[ 860], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] lat (msec) : 1000=100.00% cpu : usr=0.00%, sys=1.69%, ctx=25, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=42.3MiB/s (44.3MB/s), 42.3MiB/s-42.3MiB/s (44.3MB/s-44.3MB/s), io=40.0MiB (41.9MB), run=946-946msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout