== sanity test 398c: run fio to test AIO ================= 03:17:40 (1790925460) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.56611 s, 74.1 MB/s osc.lustre-OST0000-osc-ffff8fe3c81b1000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=176011: Fri Oct 2 03:20:11 2026 write: IOPS=18, BW=72.9KiB/s (74.7kB/s)(10.0MiB/140462msec); 0 zone resets slat (usec): min=24, max=25184, avg=170.38, stdev=811.95 clat (msec): min=284, max=1996, avg=876.41, stdev=218.09 lat (msec): min=284, max=1996, avg=876.58, stdev=218.07 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 558], 10.00th=[ 642], 20.00th=[ 709], | 30.00th=[ 768], 40.00th=[ 818], 50.00th=[ 869], 60.00th=[ 919], | 70.00th=[ 969], 80.00th=[ 1028], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1519], 99.50th=[ 1703], 99.90th=[ 1938], 99.95th=[ 1938], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 254, per=24.94%, avg=72.58, stdev=30.91, samples=278 iops : min= 1, max= 63, avg=17.73, stdev= 7.76, samples=278 lat (msec) : 500=3.59%, 750=22.97%, 1000=49.06%, 2000=24.38% cpu : usr=0.05%, sys=0.21%, ctx=1960, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=176012: Fri Oct 2 03:20:11 2026 write: IOPS=18, BW=72.9KiB/s (74.7kB/s)(10.0MiB/140374msec); 0 zone resets slat (usec): min=21, max=18504, avg=171.43, stdev=674.49 clat (msec): min=202, max=2014, avg=876.17, stdev=222.85 lat (msec): min=202, max=2014, avg=876.34, stdev=222.86 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 567], 10.00th=[ 634], 20.00th=[ 709], | 30.00th=[ 768], 40.00th=[ 810], 50.00th=[ 860], 60.00th=[ 919], | 70.00th=[ 969], 80.00th=[ 1036], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1485], 99.50th=[ 1670], 99.90th=[ 1972], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 8, max= 254, per=25.12%, avg=73.09, stdev=31.98, samples=276 iops : min= 2, max= 63, avg=17.86, stdev= 8.03, samples=276 lat (msec) : 250=0.62%, 500=3.20%, 750=23.98%, 1000=46.60%, 2000=25.51% lat (msec) : >=2000=0.08% cpu : usr=0.04%, sys=0.24%, ctx=1929, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=176013: Fri Oct 2 03:20:11 2026 write: IOPS=18, BW=72.9KiB/s (74.7kB/s)(10.0MiB/140374msec); 0 zone resets slat (usec): min=24, max=12382, avg=158.22, stdev=588.28 clat (msec): min=182, max=1969, avg=876.06, stdev=220.17 lat (msec): min=182, max=1969, avg=876.22, stdev=220.16 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 550], 10.00th=[ 642], 20.00th=[ 709], | 30.00th=[ 776], 40.00th=[ 818], 50.00th=[ 860], 60.00th=[ 919], | 70.00th=[ 969], 80.00th=[ 1036], 90.00th=[ 1133], 95.00th=[ 1250], | 99.00th=[ 1469], 99.50th=[ 1653], 99.90th=[ 1921], 99.95th=[ 1955], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 254, per=25.12%, avg=73.09, stdev=34.84, samples=276 iops : min= 1, max= 63, avg=17.86, stdev= 8.76, samples=276 lat (msec) : 250=0.62%, 500=3.44%, 750=21.91%, 1000=48.67%, 2000=25.35% cpu : usr=0.05%, sys=0.20%, ctx=1646, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=176014: Fri Oct 2 03:20:11 2026 write: IOPS=18, BW=73.2KiB/s (74.9kB/s)(10.0MiB/139944msec); 0 zone resets slat (usec): min=23, max=26402, avg=207.79, stdev=945.72 clat (msec): min=79, max=2073, avg=873.80, stdev=231.02 lat (msec): min=80, max=2073, avg=874.01, stdev=230.96 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 567], 10.00th=[ 634], 20.00th=[ 701], | 30.00th=[ 760], 40.00th=[ 810], 50.00th=[ 860], 60.00th=[ 919], | 70.00th=[ 969], 80.00th=[ 1028], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1653], 99.50th=[ 1921], 99.90th=[ 1989], 99.95th=[ 2022], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 199, per=25.20%, avg=73.33, stdev=31.14, samples=275 iops : min= 1, max= 49, avg=17.92, stdev= 7.81, samples=275 lat (msec) : 100=0.04%, 250=0.43%, 500=3.52%, 750=24.57%, 1000=47.15% lat (msec) : 2000=24.22%, >=2000=0.08% cpu : usr=0.06%, sys=0.23%, ctx=2045, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=292KiB/s (299kB/s), 72.9KiB/s-73.2KiB/s (74.7kB/s-74.9kB/s), io=40.0MiB (41.9MB), run=139944-140462msec 4% of I/O are 1-page (PAGE_SIZE=4096), max 50% mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=176045: Fri Oct 2 03:21:58 2026 read: IOPS=13, BW=52.3KiB/s (53.6kB/s)(5048KiB/96465msec) slat (usec): min=36, max=358871, avg=13227.51, stdev=12171.66 clat (msec): min=10, max=932, avg=368.44, stdev=104.08 lat (msec): min=18, max=939, avg=381.68, stdev=105.29 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 234], 10.00th=[ 262], 20.00th=[ 292], | 30.00th=[ 317], 40.00th=[ 334], 50.00th=[ 359], 60.00th=[ 380], | 70.00th=[ 405], 80.00th=[ 443], 90.00th=[ 506], 95.00th=[ 550], | 99.00th=[ 634], 99.50th=[ 651], 99.90th=[ 827], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 7, max= 104, per=25.23%, avg=52.23, stdev=21.60, samples=190 iops : min= 1, max= 26, avg=12.57, stdev= 5.48, samples=190 write: IOPS=13, BW=53.8KiB/s (55.1kB/s)(5192KiB/96465msec); 0 zone resets slat (usec): min=29, max=16284, avg=209.79, stdev=801.36 clat (msec): min=360, max=1715, avg=817.00, stdev=213.27 lat (msec): min=360, max=1715, avg=817.21, stdev=213.27 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 506], 10.00th=[ 558], 20.00th=[ 642], | 30.00th=[ 701], 40.00th=[ 751], 50.00th=[ 802], 60.00th=[ 852], | 70.00th=[ 902], 80.00th=[ 969], 90.00th=[ 1070], 95.00th=[ 1183], | 99.00th=[ 1452], 99.50th=[ 1569], 99.90th=[ 1703], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min= 8, max= 135, per=24.94%, avg=53.61, stdev=21.83, samples=190 iops : min= 2, max= 33, avg=12.92, stdev= 5.47, samples=190 lat (msec) : 20=0.04%, 50=0.12%, 100=0.78%, 250=2.81%, 500=42.58% lat (msec) : 750=23.36%, 1000=22.11%, 2000=8.20% cpu : usr=0.11%, sys=0.71%, ctx=4643, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=176046: Fri Oct 2 03:21:58 2026 read: IOPS=12, BW=51.3KiB/s (52.6kB/s)(4976KiB/96905msec) slat (usec): min=2873, max=69412, avg=12772.44, stdev=7188.28 clat (msec): min=14, max=841, avg=367.07, stdev=97.47 lat (msec): min=18, max=855, avg=379.85, stdev=98.87 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 243], 10.00th=[ 262], 20.00th=[ 292], | 30.00th=[ 313], 40.00th=[ 334], 50.00th=[ 355], 60.00th=[ 380], | 70.00th=[ 405], 80.00th=[ 435], 90.00th=[ 498], 95.00th=[ 535], | 99.00th=[ 651], 99.50th=[ 693], 99.90th=[ 802], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 7, max= 112, per=24.53%, avg=50.79, stdev=23.25, samples=192 iops : min= 1, max= 28, avg=12.24, stdev= 5.85, samples=192 write: IOPS=13, BW=54.3KiB/s (55.6kB/s)(5264KiB/96905msec); 0 zone resets slat (usec): min=26, max=10418, avg=211.73, stdev=612.12 clat (msec): min=317, max=1783, avg=816.07, stdev=213.43 lat (msec): min=317, max=1783, avg=816.28, stdev=213.42 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 506], 10.00th=[ 567], 20.00th=[ 642], | 30.00th=[ 693], 40.00th=[ 743], 50.00th=[ 802], 60.00th=[ 852], | 70.00th=[ 902], 80.00th=[ 969], 90.00th=[ 1083], 95.00th=[ 1183], | 99.00th=[ 1452], 99.50th=[ 1670], 99.90th=[ 1720], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 7, max= 128, per=25.00%, avg=53.76, stdev=20.84, samples=192 iops : min= 1, max= 32, avg=12.98, stdev= 5.26, samples=192 lat (msec) : 20=0.08%, 50=0.16%, 100=0.12%, 250=2.77%, 500=43.52% lat (msec) : 750=22.81%, 1000=21.80%, 2000=8.75% cpu : usr=0.11%, sys=0.64%, ctx=4713, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=176047: Fri Oct 2 03:21:58 2026 read: IOPS=12, BW=51.4KiB/s (52.7kB/s)(4984KiB/96909msec) slat (usec): min=2447, max=66589, avg=12803.88, stdev=7292.76 clat (msec): min=18, max=845, avg=368.14, stdev=98.85 lat (msec): min=33, max=855, avg=380.95, stdev=99.87 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 239], 10.00th=[ 259], 20.00th=[ 292], | 30.00th=[ 317], 40.00th=[ 338], 50.00th=[ 359], 60.00th=[ 380], | 70.00th=[ 405], 80.00th=[ 439], 90.00th=[ 489], 95.00th=[ 542], | 99.00th=[ 659], 99.50th=[ 726], 99.90th=[ 810], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 7, max= 119, per=24.97%, avg=51.68, stdev=23.24, samples=190 iops : min= 1, max= 29, avg=12.44, stdev= 5.87, samples=190 write: IOPS=13, BW=54.2KiB/s (55.5kB/s)(5256KiB/96909msec); 0 zone resets slat (usec): min=29, max=28028, avg=263.41, stdev=1110.92 clat (msec): min=263, max=1762, avg=815.49, stdev=218.22 lat (msec): min=263, max=1762, avg=815.76, stdev=218.19 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 498], 10.00th=[ 558], 20.00th=[ 634], | 30.00th=[ 693], 40.00th=[ 743], 50.00th=[ 793], 60.00th=[ 844], | 70.00th=[ 902], 80.00th=[ 986], 90.00th=[ 1083], 95.00th=[ 1200], | 99.00th=[ 1452], 99.50th=[ 1569], 99.90th=[ 1687], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 15, max= 120, per=25.04%, avg=53.83, stdev=18.50, samples=191 iops : min= 3, max= 30, avg=12.97, stdev= 4.70, samples=191 lat (msec) : 20=0.04%, 50=0.20%, 100=0.27%, 250=2.97%, 500=43.75% lat (msec) : 750=22.27%, 1000=21.68%, 2000=8.83% cpu : usr=0.11%, sys=0.66%, ctx=4612, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=176048: Fri Oct 2 03:21:58 2026 read: IOPS=13, BW=52.4KiB/s (53.6kB/s)(5060KiB/96629msec) slat (usec): min=2540, max=62201, avg=12974.27, stdev=7487.62 clat (msec): min=11, max=817, avg=369.31, stdev=99.38 lat (msec): min=31, max=849, avg=382.29, stdev=100.66 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 236], 10.00th=[ 259], 20.00th=[ 292], | 30.00th=[ 317], 40.00th=[ 338], 50.00th=[ 355], 60.00th=[ 380], | 70.00th=[ 409], 80.00th=[ 443], 90.00th=[ 493], 95.00th=[ 542], | 99.00th=[ 684], 99.50th=[ 760], 99.90th=[ 802], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 7, max= 128, per=25.33%, avg=52.44, stdev=23.80, samples=190 iops : min= 1, max= 32, avg=12.63, stdev= 6.01, samples=190 write: IOPS=13, BW=53.6KiB/s (54.9kB/s)(5180KiB/96629msec); 0 zone resets slat (usec): min=23, max=20129, avg=241.70, stdev=898.61 clat (msec): min=293, max=1729, avg=817.18, stdev=218.23 lat (msec): min=293, max=1729, avg=817.42, stdev=218.23 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 489], 10.00th=[ 567], 20.00th=[ 642], | 30.00th=[ 701], 40.00th=[ 743], 50.00th=[ 793], 60.00th=[ 844], | 70.00th=[ 902], 80.00th=[ 978], 90.00th=[ 1099], 95.00th=[ 1200], | 99.00th=[ 1469], 99.50th=[ 1519], 99.90th=[ 1687], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 7, max= 112, per=24.58%, avg=52.85, stdev=20.67, samples=192 iops : min= 1, max= 28, avg=12.73, stdev= 5.21, samples=192 lat (msec) : 20=0.08%, 50=0.12%, 100=0.16%, 250=3.24%, 500=43.83% lat (msec) : 750=22.46%, 1000=21.37%, 2000=8.75% cpu : usr=0.11%, sys=0.71%, ctx=4629, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=207KiB/s (212kB/s), 51.3KiB/s-52.4KiB/s (52.6kB/s-53.6kB/s), io=19.6MiB (20.5MB), run=96465-96909msec WRITE: bw=216KiB/s (221kB/s), 53.6KiB/s-54.3KiB/s (54.9kB/s-55.6kB/s), io=20.4MiB (21.4MB), run=96465-96909msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=176066: Fri Oct 2 03:22:07 2026 read: IOPS=1, BW=77.0MiB/s (81.8MB/s)(40.0MiB/513msec) slat (nsec): min=44763k, max=44763k, avg=44763070.00, stdev= 0.00 clat (nsec): min=437355k, max=437355k, avg=437355270.00, stdev= 0.00 lat (nsec): min=482122k, max=482122k, avg=482122348.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 439], 10.00th=[ 439], 20.00th=[ 439], | 30.00th=[ 439], 40.00th=[ 439], 50.00th=[ 439], 60.00th=[ 439], | 70.00th=[ 439], 80.00th=[ 439], 90.00th=[ 439], 95.00th=[ 439], | 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] lat (msec) : 500=100.00% cpu : usr=0.20%, sys=2.93%, ctx=45, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=77.0MiB/s (81.8MB/s), 77.0MiB/s-77.0MiB/s (81.8MB/s-81.8MB/s), io=40.0MiB (41.9MB), run=513-513msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout