== sanity test 398c: run fio to test AIO ================= 19:07:15 (1786835235) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.492626 s, 85.1 MB/s osc.lustre-OST0000-osc-ffff9fc970a1a000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145643: Sat Aug 15 19:09:29 2026 write: IOPS=20, BW=81.7KiB/s (83.7kB/s)(10.0MiB/125313msec); 0 zone resets slat (usec): min=47, max=23221, avg=325.80, stdev=831.87 clat (msec): min=135, max=1563, avg=782.59, stdev=187.02 lat (msec): min=135, max=1563, avg=782.92, stdev=187.03 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 502], 10.00th=[ 575], 20.00th=[ 659], | 30.00th=[ 693], 40.00th=[ 743], 50.00th=[ 785], 60.00th=[ 818], | 70.00th=[ 869], 80.00th=[ 919], 90.00th=[ 995], 95.00th=[ 1083], | 99.00th=[ 1284], 99.50th=[ 1318], 99.90th=[ 1469], 99.95th=[ 1469], | 99.99th=[ 1569] bw ( KiB/s): min= 8, max= 224, per=24.84%, avg=80.97, stdev=33.15, samples=249 iops : min= 2, max= 56, avg=19.80, stdev= 8.31, samples=249 lat (msec) : 250=1.25%, 500=3.71%, 750=36.52%, 1000=48.71%, 2000=9.80% cpu : usr=0.05%, sys=0.37%, ctx=1913, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145644: Sat Aug 15 19:09:29 2026 write: IOPS=20, BW=81.6KiB/s (83.5kB/s)(10.0MiB/125504msec); 0 zone resets slat (usec): min=45, max=23831, avg=386.69, stdev=881.19 clat (msec): min=116, max=1533, avg=783.03, stdev=182.90 lat (msec): min=134, max=1534, avg=783.42, stdev=182.88 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 477], 10.00th=[ 584], 20.00th=[ 651], | 30.00th=[ 701], 40.00th=[ 735], 50.00th=[ 785], 60.00th=[ 827], | 70.00th=[ 877], 80.00th=[ 919], 90.00th=[ 995], 95.00th=[ 1070], | 99.00th=[ 1284], 99.50th=[ 1351], 99.90th=[ 1485], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 23, max= 206, per=24.79%, avg=80.82, stdev=27.22, samples=249 iops : min= 5, max= 51, avg=19.76, stdev= 6.83, samples=249 lat (msec) : 250=0.62%, 500=4.84%, 750=37.70%, 1000=47.42%, 2000=9.41% cpu : usr=0.06%, sys=0.38%, ctx=2492, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145645: Sat Aug 15 19:09:29 2026 write: IOPS=20, BW=81.6KiB/s (83.6kB/s)(10.0MiB/125448msec); 0 zone resets slat (usec): min=50, max=13286, avg=367.85, stdev=827.13 clat (msec): min=146, max=1563, avg=782.96, stdev=183.99 lat (msec): min=146, max=1563, avg=783.33, stdev=183.98 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 510], 10.00th=[ 575], 20.00th=[ 651], | 30.00th=[ 693], 40.00th=[ 735], 50.00th=[ 785], 60.00th=[ 827], | 70.00th=[ 869], 80.00th=[ 919], 90.00th=[ 1003], 95.00th=[ 1083], | 99.00th=[ 1250], 99.50th=[ 1351], 99.90th=[ 1536], 99.95th=[ 1536], | 99.99th=[ 1569] bw ( KiB/s): min= 15, max= 243, per=24.79%, avg=80.82, stdev=29.99, samples=249 iops : min= 3, max= 60, avg=19.76, stdev= 7.53, samples=249 lat (msec) : 250=0.66%, 500=4.18%, 750=38.16%, 1000=46.91%, 2000=10.08% cpu : usr=0.05%, sys=0.38%, ctx=2377, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145646: Sat Aug 15 19:09:29 2026 write: IOPS=20, BW=81.6KiB/s (83.6kB/s)(10.0MiB/125455msec); 0 zone resets slat (usec): min=48, max=18256, avg=358.03, stdev=808.01 clat (msec): min=65, max=1506, avg=783.06, stdev=187.90 lat (msec): min=67, max=1507, avg=783.42, stdev=187.89 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 489], 10.00th=[ 575], 20.00th=[ 651], | 30.00th=[ 693], 40.00th=[ 743], 50.00th=[ 785], 60.00th=[ 827], | 70.00th=[ 869], 80.00th=[ 919], 90.00th=[ 995], 95.00th=[ 1083], | 99.00th=[ 1284], 99.50th=[ 1351], 99.90th=[ 1485], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 15, max= 231, per=24.83%, avg=80.94, stdev=28.96, samples=249 iops : min= 3, max= 57, avg=19.79, stdev= 7.26, samples=249 lat (msec) : 100=0.04%, 250=0.86%, 500=4.30%, 750=37.07%, 1000=47.97% lat (msec) : 2000=9.77% cpu : usr=0.05%, sys=0.40%, ctx=2379, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=326KiB/s (334kB/s), 81.6KiB/s-81.7KiB/s (83.5kB/s-83.7kB/s), io=40.0MiB (41.9MB), run=125313-125504msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145673: Sat Aug 15 19:10:53 2026 read: IOPS=16, BW=64.7KiB/s (66.2kB/s)(5048KiB/78067msec) slat (usec): min=2770, max=42684, avg=10149.16, stdev=4589.03 clat (msec): min=8, max=725, avg=302.28, stdev=68.00 lat (msec): min=20, max=736, avg=312.44, stdev=68.73 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 205], 10.00th=[ 224], 20.00th=[ 247], | 30.00th=[ 268], 40.00th=[ 284], 50.00th=[ 300], 60.00th=[ 313], | 70.00th=[ 334], 80.00th=[ 351], 90.00th=[ 384], 95.00th=[ 409], | 99.00th=[ 489], 99.50th=[ 514], 99.90th=[ 718], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 15, max= 176, per=25.07%, avg=64.19, stdev=25.21, samples=155 iops : min= 3, max= 44, avg=15.68, stdev= 6.33, samples=155 write: IOPS=16, BW=66.5KiB/s (68.1kB/s)(5192KiB/78067msec); 0 zone resets slat (usec): min=56, max=10075, avg=389.70, stdev=652.87 clat (msec): min=263, max=1146, avg=654.82, stdev=145.82 lat (msec): min=263, max=1146, avg=655.21, stdev=145.82 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 418], 10.00th=[ 460], 20.00th=[ 531], | 30.00th=[ 575], 40.00th=[ 617], 50.00th=[ 651], 60.00th=[ 693], | 70.00th=[ 735], 80.00th=[ 785], 90.00th=[ 852], 95.00th=[ 902], | 99.00th=[ 969], 99.50th=[ 995], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 23, max= 128, per=24.93%, avg=66.57, stdev=21.23, samples=154 iops : min= 5, max= 32, avg=16.28, stdev= 5.36, samples=154 lat (msec) : 10=0.04%, 100=0.27%, 250=10.16%, 500=45.59%, 750=30.70% lat (msec) : 1000=13.01%, 2000=0.23% cpu : usr=0.09%, sys=0.94%, ctx=5020, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145674: Sat Aug 15 19:10:53 2026 read: IOPS=15, BW=63.7KiB/s (65.2kB/s)(4976KiB/78123msec) slat (usec): min=2714, max=42158, avg=10260.20, stdev=4667.35 clat (msec): min=12, max=678, avg=301.84, stdev=70.86 lat (msec): min=22, max=685, avg=312.11, stdev=71.14 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 203], 10.00th=[ 222], 20.00th=[ 247], | 30.00th=[ 266], 40.00th=[ 284], 50.00th=[ 300], 60.00th=[ 317], | 70.00th=[ 334], 80.00th=[ 355], 90.00th=[ 388], 95.00th=[ 409], | 99.00th=[ 493], 99.50th=[ 550], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 15, max= 143, per=24.72%, avg=63.27, stdev=26.51, samples=155 iops : min= 3, max= 35, avg=15.45, stdev= 6.66, samples=155 write: IOPS=16, BW=67.4KiB/s (68.0kB/s)(5264KiB/78123msec); 0 zone resets slat (usec): min=56, max=9007, avg=389.90, stdev=702.41 clat (msec): min=271, max=1134, avg=651.78, stdev=149.70 lat (msec): min=272, max=1134, avg=652.17, stdev=149.70 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 393], 10.00th=[ 447], 20.00th=[ 527], | 30.00th=[ 575], 40.00th=[ 617], 50.00th=[ 651], 60.00th=[ 693], | 70.00th=[ 735], 80.00th=[ 793], 90.00th=[ 844], 95.00th=[ 902], | 99.00th=[ 961], 99.50th=[ 995], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 15, max= 136, per=25.09%, avg=66.99, stdev=25.14, samples=155 iops : min= 3, max= 34, avg=16.39, stdev= 6.28, samples=155 lat (msec) : 20=0.04%, 50=0.08%, 100=0.31%, 250=10.08%, 500=46.02% lat (msec) : 750=29.77%, 1000=13.55%, 2000=0.16% cpu : usr=0.12%, sys=0.91%, ctx=4922, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145675: Sat Aug 15 19:10:53 2026 read: IOPS=15, BW=63.8KiB/s (65.3kB/s)(4984KiB/78151msec) slat (usec): min=2955, max=42223, avg=10320.76, stdev=4617.36 clat (msec): min=18, max=520, avg=301.73, stdev=62.89 lat (msec): min=28, max=534, avg=312.06, stdev=63.61 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 207], 10.00th=[ 228], 20.00th=[ 247], | 30.00th=[ 268], 40.00th=[ 284], 50.00th=[ 300], 60.00th=[ 317], | 70.00th=[ 330], 80.00th=[ 351], 90.00th=[ 384], 95.00th=[ 409], | 99.00th=[ 460], 99.50th=[ 485], 99.90th=[ 510], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 7, max= 128, per=24.84%, avg=63.58, stdev=25.26, samples=155 iops : min= 1, max= 32, avg=15.54, stdev= 6.32, samples=155 write: IOPS=16, BW=67.3KiB/s (68.9kB/s)(5256KiB/78151msec); 0 zone resets slat (usec): min=62, max=11061, avg=438.03, stdev=794.73 clat (msec): min=233, max=1143, avg=652.14, stdev=148.56 lat (msec): min=233, max=1144, avg=652.58, stdev=148.56 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 409], 10.00th=[ 451], 20.00th=[ 527], | 30.00th=[ 575], 40.00th=[ 609], 50.00th=[ 651], 60.00th=[ 684], | 70.00th=[ 735], 80.00th=[ 785], 90.00th=[ 852], 95.00th=[ 894], | 99.00th=[ 969], 99.50th=[ 995], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 15, max= 135, per=24.93%, avg=66.57, stdev=20.75, samples=155 iops : min= 3, max= 33, avg=16.28, stdev= 5.24, samples=155 lat (msec) : 20=0.04%, 50=0.08%, 100=0.12%, 250=10.27%, 500=46.09% lat (msec) : 750=29.69%, 1000=13.52%, 2000=0.20% cpu : usr=0.13%, sys=0.95%, ctx=5409, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145676: Sat Aug 15 19:10:53 2026 read: IOPS=16, BW=65.1KiB/s (66.7kB/s)(5060KiB/77704msec) slat (usec): min=102, max=35740, avg=9919.47, stdev=4422.99 clat (msec): min=14, max=796, avg=298.81, stdev=71.23 lat (msec): min=20, max=807, avg=308.74, stdev=71.83 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 199], 10.00th=[ 222], 20.00th=[ 247], | 30.00th=[ 268], 40.00th=[ 284], 50.00th=[ 300], 60.00th=[ 317], | 70.00th=[ 330], 80.00th=[ 351], 90.00th=[ 380], 95.00th=[ 409], | 99.00th=[ 451], 99.50th=[ 489], 99.90th=[ 743], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 8, max= 128, per=25.37%, avg=64.94, stdev=24.27, samples=154 iops : min= 2, max= 32, avg=15.88, stdev= 6.11, samples=154 write: IOPS=16, BW=66.7KiB/s (68.3kB/s)(5180KiB/77704msec); 0 zone resets slat (usec): min=56, max=12362, avg=381.75, stdev=713.35 clat (msec): min=137, max=1230, avg=657.22, stdev=156.17 lat (msec): min=137, max=1230, avg=657.60, stdev=156.14 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 405], 10.00th=[ 460], 20.00th=[ 527], | 30.00th=[ 575], 40.00th=[ 617], 50.00th=[ 659], 60.00th=[ 693], | 70.00th=[ 743], 80.00th=[ 793], 90.00th=[ 852], 95.00th=[ 902], | 99.00th=[ 1003], 99.50th=[ 1150], 99.90th=[ 1217], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 16, max= 126, per=24.77%, avg=66.13, stdev=21.93, samples=154 iops : min= 4, max= 31, avg=16.18, stdev= 5.48, samples=154 lat (msec) : 20=0.12%, 50=0.23%, 100=0.43%, 250=10.00%, 500=45.86% lat (msec) : 750=29.06%, 1000=13.79%, 2000=0.51% cpu : usr=0.10%, sys=0.92%, ctx=4956, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=257KiB/s (263kB/s), 63.7KiB/s-65.1KiB/s (65.2kB/s-66.7kB/s), io=19.6MiB (20.5MB), run=77704-78151msec WRITE: bw=267KiB/s (274kB/s), 66.5KiB/s-67.4KiB/s (68.1kB/s-68.0kB/s), io=20.4MiB (21.4MB), run=77704-78151msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145691: Sat Aug 15 19:11:00 2026 read: IOPS=2, BW=110MiB/s (116MB/s)(40.0MiB/362msec) slat (nsec): min=35317k, max=35317k, avg=35316632.00, stdev= 0.00 clat (nsec): min=307506k, max=307506k, avg=307505856.00, stdev= 0.00 lat (nsec): min=342828k, max=342828k, avg=342828418.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 309], 10.00th=[ 309], 20.00th=[ 309], | 30.00th=[ 309], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=4.16%, ctx=38, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=110MiB/s (116MB/s), 110MiB/s-110MiB/s (116MB/s-116MB/s), io=40.0MiB (41.9MB), run=362-362msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout