== sanity test 398c: run fio to test AIO ================= 17:29:17 (1787606957) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.513551 s, 81.7 MB/s osc.lustre-OST0000-osc-ffff96af4577f000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=176404: Mon Aug 24 17:32:34 2026 write: IOPS=13, BW=54.6KiB/s (55.9kB/s)(10.0MiB/187412msec); 0 zone resets slat (usec): min=29, max=13078, avg=196.26, stdev=741.99 clat (msec): min=168, max=2186, avg=1170.01, stdev=291.62 lat (msec): min=168, max=2187, avg=1170.21, stdev=291.62 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 718], 10.00th=[ 835], 20.00th=[ 927], | 30.00th=[ 1011], 40.00th=[ 1083], 50.00th=[ 1150], 60.00th=[ 1234], | 70.00th=[ 1301], 80.00th=[ 1401], 90.00th=[ 1552], 95.00th=[ 1687], | 99.00th=[ 1989], 99.50th=[ 2039], 99.90th=[ 2123], 99.95th=[ 2140], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 136, per=25.53%, avg=55.65, stdev=27.56, samples=362 iops : min= 1, max= 34, avg=13.49, stdev= 6.91, samples=362 lat (msec) : 250=0.04%, 500=1.21%, 750=4.30%, 1000=22.27%, 2000=71.21% lat (msec) : >=2000=0.98% cpu : usr=0.04%, sys=0.19%, ctx=1765, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=176405: Mon Aug 24 17:32:34 2026 write: IOPS=13, BW=54.8KiB/s (56.1kB/s)(10.0MiB/186921msec); 0 zone resets slat (usec): min=26, max=17995, avg=236.48, stdev=1017.99 clat (msec): min=211, max=2235, avg=1167.45, stdev=296.93 lat (msec): min=212, max=2235, avg=1167.69, stdev=296.91 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 718], 10.00th=[ 844], 20.00th=[ 936], | 30.00th=[ 1011], 40.00th=[ 1083], 50.00th=[ 1150], 60.00th=[ 1217], | 70.00th=[ 1301], 80.00th=[ 1401], 90.00th=[ 1552], 95.00th=[ 1670], | 99.00th=[ 1989], 99.50th=[ 2056], 99.90th=[ 2198], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 203, per=25.32%, avg=55.19, stdev=24.83, samples=365 iops : min= 1, max= 50, avg=13.38, stdev= 6.28, samples=365 lat (msec) : 250=0.39%, 500=0.94%, 750=4.26%, 1000=22.30%, 2000=71.13% lat (msec) : >=2000=0.98% cpu : usr=0.04%, sys=0.20%, ctx=1770, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=176406: Mon Aug 24 17:32:34 2026 write: IOPS=13, BW=54.6KiB/s (55.9kB/s)(10.0MiB/187414msec); 0 zone resets slat (usec): min=22, max=18414, avg=205.78, stdev=927.91 clat (msec): min=85, max=2218, avg=1170.08, stdev=292.16 lat (msec): min=86, max=2218, avg=1170.28, stdev=292.18 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 726], 10.00th=[ 852], 20.00th=[ 936], | 30.00th=[ 1020], 40.00th=[ 1083], 50.00th=[ 1150], 60.00th=[ 1217], | 70.00th=[ 1284], 80.00th=[ 1385], 90.00th=[ 1552], 95.00th=[ 1687], | 99.00th=[ 2005], 99.50th=[ 2072], 99.90th=[ 2165], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 128, per=25.46%, avg=55.50, stdev=24.49, samples=363 iops : min= 1, max= 32, avg=13.45, stdev= 6.15, samples=363 lat (msec) : 100=0.04%, 500=1.37%, 750=4.34%, 1000=21.64%, 2000=71.64% lat (msec) : >=2000=0.98% cpu : usr=0.04%, sys=0.17%, ctx=1819, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=176407: Mon Aug 24 17:32:34 2026 write: IOPS=13, BW=54.6KiB/s (55.9kB/s)(10.0MiB/187444msec); 0 zone resets slat (usec): min=27, max=15387, avg=205.49, stdev=777.43 clat (msec): min=418, max=2288, avg=1170.43, stdev=284.60 lat (msec): min=419, max=2288, avg=1170.64, stdev=284.62 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 735], 10.00th=[ 852], 20.00th=[ 944], | 30.00th=[ 1020], 40.00th=[ 1083], 50.00th=[ 1150], 60.00th=[ 1217], | 70.00th=[ 1301], 80.00th=[ 1385], 90.00th=[ 1536], 95.00th=[ 1653], | 99.00th=[ 2022], 99.50th=[ 2072], 99.90th=[ 2140], 99.95th=[ 2198], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 128, per=24.97%, avg=54.44, stdev=24.46, samples=370 iops : min= 1, max= 32, avg=13.19, stdev= 6.18, samples=370 lat (msec) : 500=0.59%, 750=4.49%, 1000=21.76%, 2000=72.07%, >=2000=1.09% cpu : usr=0.04%, sys=0.17%, ctx=1724, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=219KiB/s (224kB/s), 54.6KiB/s-54.8KiB/s (55.9kB/s-56.1kB/s), io=40.0MiB (41.9MB), run=186921-187444msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=176443: Mon Aug 24 17:34:45 2026 read: IOPS=10, BW=40.2KiB/s (41.2kB/s)(5048KiB/125516msec) slat (usec): min=2526, max=59786, avg=14182.64, stdev=8019.01 clat (msec): min=35, max=1288, avg=497.56, stdev=144.07 lat (msec): min=42, max=1303, avg=511.74, stdev=145.43 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 300], 10.00th=[ 338], 20.00th=[ 384], | 30.00th=[ 422], 40.00th=[ 451], 50.00th=[ 481], 60.00th=[ 514], | 70.00th=[ 550], 80.00th=[ 600], 90.00th=[ 667], 95.00th=[ 760], | 99.00th=[ 953], 99.50th=[ 1020], 99.90th=[ 1116], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 7, max= 111, per=25.22%, avg=40.11, stdev=18.63, samples=247 iops : min= 1, max= 27, avg= 9.59, stdev= 4.71, samples=247 write: IOPS=10, BW=41.4KiB/s (42.4kB/s)(5192KiB/125516msec); 0 zone resets slat (usec): min=31, max=12884, avg=259.76, stdev=802.47 clat (msec): min=230, max=2105, avg=1046.58, stdev=304.25 lat (msec): min=230, max=2105, avg=1046.84, stdev=304.26 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 609], 10.00th=[ 693], 20.00th=[ 785], | 30.00th=[ 877], 40.00th=[ 953], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1284], 90.00th=[ 1469], 95.00th=[ 1603], | 99.00th=[ 1905], 99.50th=[ 1938], 99.90th=[ 2072], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 7, max= 120, per=24.77%, avg=41.12, stdev=19.14, samples=247 iops : min= 1, max= 30, avg= 9.85, stdev= 4.82, samples=247 lat (msec) : 50=0.08%, 100=0.08%, 250=0.82%, 500=26.91%, 750=26.76% lat (msec) : 1000=18.95%, 2000=26.25%, >=2000=0.16% cpu : usr=0.05%, sys=0.46%, ctx=4123, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=176444: Mon Aug 24 17:34:45 2026 read: IOPS=9, BW=39.6KiB/s (40.6kB/s)(4976KiB/125606msec) slat (usec): min=2338, max=59115, avg=14043.05, stdev=8182.96 clat (msec): min=55, max=1023, avg=493.27, stdev=132.59 lat (msec): min=65, max=1035, avg=507.32, stdev=134.32 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 296], 10.00th=[ 338], 20.00th=[ 384], | 30.00th=[ 418], 40.00th=[ 451], 50.00th=[ 481], 60.00th=[ 518], | 70.00th=[ 550], 80.00th=[ 600], 90.00th=[ 659], 95.00th=[ 726], | 99.00th=[ 869], 99.50th=[ 902], 99.90th=[ 961], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 7, max= 104, per=25.10%, avg=39.90, stdev=21.09, samples=244 iops : min= 1, max= 26, avg= 9.55, stdev= 5.31, samples=244 write: IOPS=10, BW=41.9KiB/s (42.9kB/s)(5264KiB/125606msec); 0 zone resets slat (usec): min=30, max=9258, avg=208.59, stdev=606.18 clat (msec): min=83, max=2094, avg=1045.59, stdev=312.29 lat (msec): min=83, max=2094, avg=1045.80, stdev=312.28 clat percentiles (msec): | 1.00th=[ 264], 5.00th=[ 592], 10.00th=[ 676], 20.00th=[ 785], | 30.00th=[ 877], 40.00th=[ 961], 50.00th=[ 1011], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1284], 90.00th=[ 1469], 95.00th=[ 1586], | 99.00th=[ 1905], 99.50th=[ 2005], 99.90th=[ 2089], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 88, per=24.95%, avg=41.41, stdev=18.49, samples=249 iops : min= 1, max= 22, avg= 9.92, stdev= 4.63, samples=249 lat (msec) : 100=0.12%, 250=1.13%, 500=26.45%, 750=27.62%, 1000=17.19% lat (msec) : 2000=27.23%, >=2000=0.27% cpu : usr=0.07%, sys=0.46%, ctx=4286, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=176445: Mon Aug 24 17:34:45 2026 read: IOPS=9, BW=39.7KiB/s (40.7kB/s)(4984KiB/125395msec) slat (usec): min=27, max=53397, avg=14075.79, stdev=8284.54 clat (msec): min=30, max=1464, avg=486.30, stdev=140.22 lat (msec): min=37, max=1477, avg=500.38, stdev=141.62 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 288], 10.00th=[ 330], 20.00th=[ 384], | 30.00th=[ 414], 40.00th=[ 447], 50.00th=[ 477], 60.00th=[ 510], | 70.00th=[ 550], 80.00th=[ 592], 90.00th=[ 651], 95.00th=[ 726], | 99.00th=[ 885], 99.50th=[ 978], 99.90th=[ 1028], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 7, max= 112, per=25.28%, avg=40.20, stdev=20.78, samples=244 iops : min= 1, max= 28, avg= 9.61, stdev= 5.23, samples=244 write: IOPS=10, BW=41.9KiB/s (42.9kB/s)(5256KiB/125395msec); 0 zone resets slat (usec): min=28, max=13560, avg=256.45, stdev=865.62 clat (msec): min=135, max=2422, avg=1050.81, stdev=328.75 lat (msec): min=135, max=2423, avg=1051.06, stdev=328.80 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 592], 10.00th=[ 659], 20.00th=[ 785], | 30.00th=[ 885], 40.00th=[ 953], 50.00th=[ 1011], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1301], 90.00th=[ 1502], 95.00th=[ 1653], | 99.00th=[ 1938], 99.50th=[ 2056], 99.90th=[ 2333], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 96, per=25.09%, avg=41.65, stdev=18.74, samples=246 iops : min= 1, max= 24, avg= 9.98, stdev= 4.71, samples=246 lat (msec) : 50=0.20%, 100=0.31%, 250=1.29%, 500=27.42%, 750=26.60% lat (msec) : 1000=17.07%, 2000=26.72%, >=2000=0.39% cpu : usr=0.08%, sys=0.47%, ctx=4133, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=176446: Mon Aug 24 17:34:45 2026 read: IOPS=10, BW=40.3KiB/s (41.3kB/s)(5060KiB/125598msec) slat (usec): min=2742, max=72801, avg=14056.22, stdev=8401.26 clat (msec): min=46, max=1033, avg=498.88, stdev=140.11 lat (msec): min=61, max=1055, avg=512.94, stdev=142.02 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 300], 10.00th=[ 342], 20.00th=[ 384], | 30.00th=[ 426], 40.00th=[ 456], 50.00th=[ 489], 60.00th=[ 523], | 70.00th=[ 558], 80.00th=[ 609], 90.00th=[ 676], 95.00th=[ 751], | 99.00th=[ 885], 99.50th=[ 927], 99.90th=[ 986], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 7, max= 104, per=25.52%, avg=40.58, stdev=19.23, samples=245 iops : min= 1, max= 26, avg= 9.72, stdev= 4.83, samples=245 write: IOPS=10, BW=41.2KiB/s (42.2kB/s)(5180KiB/125598msec); 0 zone resets slat (usec): min=32, max=13027, avg=268.02, stdev=889.14 clat (msec): min=137, max=2060, avg=1048.46, stdev=310.16 lat (msec): min=137, max=2060, avg=1048.73, stdev=310.19 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 575], 10.00th=[ 676], 20.00th=[ 793], | 30.00th=[ 894], 40.00th=[ 953], 50.00th=[ 1028], 60.00th=[ 1099], | 70.00th=[ 1200], 80.00th=[ 1301], 90.00th=[ 1469], 95.00th=[ 1586], | 99.00th=[ 1854], 99.50th=[ 1905], 99.90th=[ 2039], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 88, per=24.49%, avg=40.65, stdev=19.05, samples=249 iops : min= 1, max= 22, avg= 9.73, stdev= 4.78, samples=249 lat (msec) : 50=0.04%, 100=0.31%, 250=1.05%, 500=26.33%, 750=27.46% lat (msec) : 1000=17.38%, 2000=27.30%, >=2000=0.12% cpu : usr=0.07%, sys=0.50%, ctx=4164, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=160KiB/s (164kB/s), 39.6KiB/s-40.3KiB/s (40.6kB/s-41.3kB/s), io=19.6MiB (20.5MB), run=125395-125606msec WRITE: bw=166KiB/s (170kB/s), 41.2KiB/s-41.9KiB/s (42.2kB/s-42.9kB/s), io=20.4MiB (21.4MB), run=125395-125606msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=176470: Mon Aug 24 17:34:54 2026 read: IOPS=1, BW=56.3MiB/s (58.0MB/s)(40.0MiB/711msec) slat (nsec): min=40350k, max=40350k, avg=40350410.00, stdev= 0.00 clat (nsec): min=637052k, max=637052k, avg=637051973.00, stdev= 0.00 lat (nsec): min=677407k, max=677407k, avg=677407108.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 634], 5.00th=[ 634], 10.00th=[ 634], 20.00th=[ 634], | 30.00th=[ 634], 40.00th=[ 634], 50.00th=[ 634], 60.00th=[ 634], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.13%, ctx=12, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=56.3MiB/s (58.0MB/s), 56.3MiB/s-56.3MiB/s (58.0MB/s-58.0MB/s), io=40.0MiB (41.9MB), run=711-711msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout