== sanity test 398c: run fio to test AIO ================= 08:54:56 (1785416096) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.655838 s, 64.0 MB/s osc.lustre-OST0000-osc-ffff8938a09e5000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144437: Thu Jul 30 08:59:40 2026 write: IOPS=9, BW=38.0KiB/s (38.9kB/s)(10.0MiB/269368msec); 0 zone resets slat (usec): min=49, max=30605, avg=408.97, stdev=1875.68 clat (msec): min=227, max=4320, avg=1682.15, stdev=626.89 lat (msec): min=227, max=4320, avg=1682.57, stdev=626.88 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 642], 10.00th=[ 827], 20.00th=[ 1167], | 30.00th=[ 1385], 40.00th=[ 1536], 50.00th=[ 1687], 60.00th=[ 1804], | 70.00th=[ 1955], 80.00th=[ 2165], 90.00th=[ 2467], 95.00th=[ 2769], | 99.00th=[ 3406], 99.50th=[ 3842], 99.90th=[ 4111], 99.95th=[ 4329], | 99.99th=[ 4329] bw ( KiB/s): min= 7, max= 150, per=27.08%, avg=40.89, stdev=25.99, samples=490 iops : min= 1, max= 37, avg= 9.76, stdev= 6.51, samples=490 lat (msec) : 250=0.04%, 500=1.80%, 750=5.86%, 1000=6.76%, 2000=58.36% lat (msec) : >=2000=27.19% cpu : usr=0.03%, sys=0.20%, ctx=1939, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144438: Thu Jul 30 08:59:40 2026 write: IOPS=9, BW=37.9KiB/s (38.8kB/s)(10.0MiB/270093msec); 0 zone resets slat (usec): min=49, max=42661, avg=377.91, stdev=1617.89 clat (msec): min=377, max=3945, avg=1686.26, stdev=606.85 lat (msec): min=378, max=3945, avg=1686.64, stdev=606.86 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 676], 10.00th=[ 835], 20.00th=[ 1167], | 30.00th=[ 1385], 40.00th=[ 1536], 50.00th=[ 1670], 60.00th=[ 1804], | 70.00th=[ 1972], 80.00th=[ 2198], 90.00th=[ 2467], 95.00th=[ 2702], | 99.00th=[ 3138], 99.50th=[ 3507], 99.90th=[ 3943], 99.95th=[ 3943], | 99.99th=[ 3943] bw ( KiB/s): min= 7, max= 142, per=26.98%, avg=40.74, stdev=26.05, samples=492 iops : min= 1, max= 35, avg= 9.72, stdev= 6.53, samples=492 lat (msec) : 500=1.05%, 750=6.33%, 1000=6.95%, 2000=57.11%, >=2000=28.55% cpu : usr=0.03%, sys=0.19%, ctx=2093, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144439: Thu Jul 30 08:59:40 2026 write: IOPS=9, BW=37.9KiB/s (38.9kB/s)(10.0MiB/269834msec); 0 zone resets slat (usec): min=39, max=26722, avg=288.02, stdev=1192.00 clat (msec): min=378, max=4067, avg=1684.58, stdev=607.21 lat (msec): min=379, max=4067, avg=1684.87, stdev=607.22 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 634], 10.00th=[ 835], 20.00th=[ 1183], | 30.00th=[ 1385], 40.00th=[ 1536], 50.00th=[ 1687], 60.00th=[ 1804], | 70.00th=[ 1972], 80.00th=[ 2165], 90.00th=[ 2467], 95.00th=[ 2769], | 99.00th=[ 3138], 99.50th=[ 3440], 99.90th=[ 3876], 99.95th=[ 3876], | 99.99th=[ 4077] bw ( KiB/s): min= 7, max= 152, per=27.05%, avg=40.84, stdev=26.22, samples=491 iops : min= 1, max= 38, avg= 9.75, stdev= 6.59, samples=491 lat (msec) : 500=1.02%, 750=7.54%, 1000=4.61%, 2000=58.83%, >=2000=28.01% cpu : usr=0.03%, sys=0.18%, ctx=1787, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144440: Thu Jul 30 08:59:40 2026 write: IOPS=9, BW=37.9KiB/s (38.8kB/s)(10.0MiB/270040msec); 0 zone resets slat (usec): min=49, max=18309, avg=331.32, stdev=1155.37 clat (msec): min=341, max=4770, avg=1685.59, stdev=615.28 lat (msec): min=346, max=4770, avg=1685.92, stdev=615.31 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 642], 10.00th=[ 835], 20.00th=[ 1167], | 30.00th=[ 1385], 40.00th=[ 1519], 50.00th=[ 1687], 60.00th=[ 1821], | 70.00th=[ 1972], 80.00th=[ 2165], 90.00th=[ 2467], 95.00th=[ 2702], | 99.00th=[ 3205], 99.50th=[ 3876], 99.90th=[ 4329], 99.95th=[ 4463], | 99.99th=[ 4799] bw ( KiB/s): min= 7, max= 152, per=26.98%, avg=40.74, stdev=28.74, samples=492 iops : min= 1, max= 38, avg= 9.72, stdev= 7.23, samples=492 lat (msec) : 500=0.94%, 750=6.68%, 1000=5.35%, 2000=58.44%, >=2000=28.59% cpu : usr=0.03%, sys=0.17%, ctx=1714, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=152KiB/s (155kB/s), 37.9KiB/s-38.0KiB/s (38.8kB/s-38.9kB/s), io=40.0MiB (41.9MB), run=269368-270093msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144489: Thu Jul 30 09:03:10 2026 read: IOPS=6, BW=25.8KiB/s (26.5kB/s)(5048KiB/195425msec) slat (msec): min=2, max=129, avg=21.50, stdev=15.26 clat (msec): min=28, max=1620, avg=754.90, stdev=236.54 lat (msec): min=45, max=1639, avg=776.41, stdev=239.79 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 376], 10.00th=[ 451], 20.00th=[ 558], | 30.00th=[ 651], 40.00th=[ 709], 50.00th=[ 760], 60.00th=[ 810], | 70.00th=[ 860], 80.00th=[ 927], 90.00th=[ 1045], 95.00th=[ 1150], | 99.00th=[ 1385], 99.50th=[ 1452], 99.90th=[ 1620], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 7, max= 79, per=27.25%, avg=27.79, stdev=15.23, samples=354 iops : min= 1, max= 19, avg= 6.50, stdev= 3.82, samples=354 write: IOPS=6, BW=26.6KiB/s (27.2kB/s)(5192KiB/195425msec); 0 zone resets slat (usec): min=50, max=14431, avg=294.13, stdev=916.82 clat (msec): min=218, max=3066, avg=1649.39, stdev=457.60 lat (msec): min=219, max=3066, avg=1649.69, stdev=457.61 clat percentiles (msec): | 1.00th=[ 718], 5.00th=[ 953], 10.00th=[ 1083], 20.00th=[ 1250], | 30.00th=[ 1385], 40.00th=[ 1502], 50.00th=[ 1603], 60.00th=[ 1737], | 70.00th=[ 1871], 80.00th=[ 2056], 90.00th=[ 2299], 95.00th=[ 2433], | 99.00th=[ 2735], 99.50th=[ 2802], 99.90th=[ 3037], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min= 7, max= 95, per=25.97%, avg=27.52, stdev=14.94, samples=366 iops : min= 1, max= 23, avg= 6.43, stdev= 3.77, samples=366 lat (msec) : 50=0.16%, 100=0.31%, 250=0.31%, 500=5.86%, 750=18.09% lat (msec) : 1000=21.45%, 2000=42.27%, >=2000=11.56% cpu : usr=0.05%, sys=0.39%, ctx=4237, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144490: Thu Jul 30 09:03:10 2026 read: IOPS=6, BW=25.4KiB/s (26.0kB/s)(4976KiB/195656msec) slat (msec): min=2, max=133, avg=20.73, stdev=14.43 clat (msec): min=13, max=1633, avg=758.27, stdev=220.56 lat (msec): min=19, max=1672, avg=779.01, stdev=223.23 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 393], 10.00th=[ 472], 20.00th=[ 592], | 30.00th=[ 651], 40.00th=[ 709], 50.00th=[ 768], 60.00th=[ 810], | 70.00th=[ 860], 80.00th=[ 927], 90.00th=[ 1020], 95.00th=[ 1116], | 99.00th=[ 1334], 99.50th=[ 1469], 99.90th=[ 1620], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 7, max= 96, per=26.40%, avg=26.93, stdev=15.43, samples=359 iops : min= 1, max= 24, avg= 6.28, stdev= 3.92, samples=359 write: IOPS=6, BW=26.9KiB/s (27.6kB/s)(5264KiB/195656msec); 0 zone resets slat (usec): min=51, max=37568, avg=345.24, stdev=1479.44 clat (msec): min=496, max=3184, avg=1638.96, stdev=491.83 lat (msec): min=497, max=3184, avg=1639.31, stdev=491.88 clat percentiles (msec): | 1.00th=[ 743], 5.00th=[ 885], 10.00th=[ 1020], 20.00th=[ 1200], | 30.00th=[ 1351], 40.00th=[ 1469], 50.00th=[ 1586], 60.00th=[ 1720], | 70.00th=[ 1871], 80.00th=[ 2072], 90.00th=[ 2333], 95.00th=[ 2534], | 99.00th=[ 2869], 99.50th=[ 2937], 99.90th=[ 3071], 99.95th=[ 3171], | 99.99th=[ 3171] bw ( KiB/s): min= 7, max= 87, per=26.31%, avg=27.89, stdev=15.31, samples=367 iops : min= 1, max= 21, avg= 6.53, stdev= 3.87, samples=367 lat (msec) : 20=0.04%, 50=0.04%, 100=0.20%, 250=0.31%, 500=5.20% lat (msec) : 750=17.93%, 1000=24.53%, 2000=39.73%, >=2000=12.03% cpu : usr=0.07%, sys=0.37%, ctx=3993, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144491: Thu Jul 30 09:03:10 2026 read: IOPS=6, BW=25.5KiB/s (26.1kB/s)(4984KiB/195250msec) slat (usec): min=58, max=117489, avg=21096.25, stdev=14492.27 clat (msec): min=49, max=1653, avg=763.32, stdev=237.80 lat (msec): min=50, max=1706, avg=784.42, stdev=240.52 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 380], 10.00th=[ 472], 20.00th=[ 592], | 30.00th=[ 659], 40.00th=[ 718], 50.00th=[ 760], 60.00th=[ 818], | 70.00th=[ 860], 80.00th=[ 944], 90.00th=[ 1053], 95.00th=[ 1167], | 99.00th=[ 1368], 99.50th=[ 1452], 99.90th=[ 1653], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 7, max= 80, per=26.33%, avg=26.86, stdev=15.71, samples=361 iops : min= 1, max= 20, avg= 6.25, stdev= 3.97, samples=361 write: IOPS=6, BW=26.9KiB/s (27.6kB/s)(5256KiB/195250msec); 0 zone resets slat (usec): min=51, max=21760, avg=320.59, stdev=1243.52 clat (msec): min=211, max=3070, avg=1632.19, stdev=475.23 lat (msec): min=212, max=3070, avg=1632.51, stdev=475.20 clat percentiles (msec): | 1.00th=[ 651], 5.00th=[ 902], 10.00th=[ 1045], 20.00th=[ 1250], | 30.00th=[ 1368], 40.00th=[ 1485], 50.00th=[ 1603], 60.00th=[ 1703], | 70.00th=[ 1854], 80.00th=[ 2056], 90.00th=[ 2232], 95.00th=[ 2433], | 99.00th=[ 2802], 99.50th=[ 2903], 99.90th=[ 3071], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min= 7, max= 87, per=25.87%, avg=27.43, stdev=15.62, samples=370 iops : min= 1, max= 21, avg= 6.41, stdev= 3.94, samples=370 lat (msec) : 50=0.04%, 100=0.20%, 250=1.09%, 500=4.88%, 750=18.67% lat (msec) : 1000=21.13%, 2000=42.23%, >=2000=11.76% cpu : usr=0.06%, sys=0.39%, ctx=3893, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144492: Thu Jul 30 09:03:10 2026 read: IOPS=6, BW=25.9KiB/s (26.5kB/s)(5060KiB/195409msec) slat (msec): min=2, max=309, avg=20.95, stdev=16.62 clat (msec): min=19, max=1836, avg=773.01, stdev=225.57 lat (msec): min=22, max=1857, avg=793.97, stdev=228.34 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 409], 10.00th=[ 481], 20.00th=[ 600], | 30.00th=[ 667], 40.00th=[ 718], 50.00th=[ 768], 60.00th=[ 818], | 70.00th=[ 869], 80.00th=[ 936], 90.00th=[ 1053], 95.00th=[ 1150], | 99.00th=[ 1452], 99.50th=[ 1452], 99.90th=[ 1787], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 5, max= 80, per=26.53%, avg=27.06, stdev=16.67, samples=363 iops : min= 1, max= 20, avg= 6.30, stdev= 4.22, samples=363 write: IOPS=6, BW=26.5KiB/s (27.1kB/s)(5180KiB/195409msec); 0 zone resets slat (usec): min=52, max=28458, avg=342.17, stdev=1541.32 clat (msec): min=232, max=3034, avg=1634.78, stdev=467.92 lat (msec): min=233, max=3034, avg=1635.12, stdev=467.95 clat percentiles (msec): | 1.00th=[ 693], 5.00th=[ 885], 10.00th=[ 1070], 20.00th=[ 1234], | 30.00th=[ 1368], 40.00th=[ 1469], 50.00th=[ 1586], 60.00th=[ 1720], | 70.00th=[ 1854], 80.00th=[ 2072], 90.00th=[ 2299], 95.00th=[ 2433], | 99.00th=[ 2735], 99.50th=[ 2802], 99.90th=[ 2937], 99.95th=[ 3037], | 99.99th=[ 3037] bw ( KiB/s): min= 5, max= 87, per=26.53%, avg=28.12, stdev=15.56, samples=358 iops : min= 1, max= 21, avg= 6.57, stdev= 3.90, samples=358 lat (msec) : 20=0.04%, 100=0.08%, 250=0.16%, 500=5.39%, 750=18.40% lat (msec) : 1000=22.66%, 2000=41.64%, >=2000=11.64% cpu : usr=0.05%, sys=0.39%, ctx=3997, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=103KiB/s (105kB/s), 25.4KiB/s-25.9KiB/s (26.0kB/s-26.5kB/s), io=19.6MiB (20.5MB), run=195250-195656msec WRITE: bw=107KiB/s (109kB/s), 26.5KiB/s-26.9KiB/s (27.1kB/s-27.6kB/s), io=20.4MiB (21.4MB), run=195250-195656msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144539: Thu Jul 30 09:03:25 2026 read: IOPS=1, BW=53.6MiB/s (56.2MB/s)(40.0MiB/746msec) slat (nsec): min=27945k, max=27945k, avg=27944882.00, stdev= 0.00 clat (nsec): min=684963k, max=684963k, avg=684962766.00, stdev= 0.00 lat (nsec): min=712912k, max=712912k, avg=712911707.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 684], 5.00th=[ 684], 10.00th=[ 684], 20.00th=[ 684], | 30.00th=[ 684], 40.00th=[ 684], 50.00th=[ 684], 60.00th=[ 684], | 70.00th=[ 684], 80.00th=[ 684], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.21%, ctx=38, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=53.6MiB/s (56.2MB/s), 53.6MiB/s-53.6MiB/s (56.2MB/s-56.2MB/s), io=40.0MiB (41.9MB), run=746-746msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout