== sanity test 398c: run fio to test AIO ================= 05:33:24 (1788514404) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.494185 s, 84.9 MB/s osc.lustre-OST0000-osc-ffff9b8bc4f44000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145767: Fri Sep 4 05:36:06 2026 write: IOPS=16, BW=67.5KiB/s (69.1kB/s)(10.0MiB/151678msec); 0 zone resets slat (usec): min=50, max=19665, avg=327.69, stdev=949.82 clat (msec): min=310, max=1695, avg=946.64, stdev=245.21 lat (msec): min=310, max=1695, avg=946.97, stdev=245.26 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 584], 10.00th=[ 676], 20.00th=[ 760], | 30.00th=[ 810], 40.00th=[ 877], 50.00th=[ 927], 60.00th=[ 995], | 70.00th=[ 1053], 80.00th=[ 1150], 90.00th=[ 1284], 95.00th=[ 1385], | 99.00th=[ 1586], 99.50th=[ 1636], 99.90th=[ 1687], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 251, per=24.80%, avg=66.97, stdev=28.01, samples=301 iops : min= 1, max= 62, avg=16.28, stdev= 7.02, samples=301 lat (msec) : 500=2.62%, 750=16.48%, 1000=42.46%, 2000=38.44% cpu : usr=0.05%, sys=0.31%, ctx=2265, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145768: Fri Sep 4 05:36:06 2026 write: IOPS=16, BW=67.8KiB/s (69.4kB/s)(10.0MiB/151125msec); 0 zone resets slat (usec): min=50, max=15012, avg=314.20, stdev=844.03 clat (msec): min=154, max=1745, avg=943.32, stdev=249.84 lat (msec): min=155, max=1745, avg=943.64, stdev=249.87 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 584], 10.00th=[ 659], 20.00th=[ 751], | 30.00th=[ 810], 40.00th=[ 877], 50.00th=[ 927], 60.00th=[ 995], | 70.00th=[ 1062], 80.00th=[ 1133], 90.00th=[ 1267], 95.00th=[ 1368], | 99.00th=[ 1569], 99.50th=[ 1636], 99.90th=[ 1720], 99.95th=[ 1720], | 99.99th=[ 1754] bw ( KiB/s): min= 7, max= 256, per=25.05%, avg=67.62, stdev=30.20, samples=298 iops : min= 1, max= 64, avg=16.45, stdev= 7.60, samples=298 lat (msec) : 250=0.62%, 500=2.85%, 750=16.95%, 1000=40.27%, 2000=39.30% cpu : usr=0.04%, sys=0.31%, ctx=2291, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145769: Fri Sep 4 05:36:06 2026 write: IOPS=16, BW=67.6KiB/s (69.2kB/s)(10.0MiB/151567msec); 0 zone resets slat (usec): min=46, max=18065, avg=321.96, stdev=942.71 clat (msec): min=142, max=1758, avg=946.55, stdev=251.08 lat (msec): min=143, max=1759, avg=946.87, stdev=251.13 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 600], 10.00th=[ 667], 20.00th=[ 743], | 30.00th=[ 818], 40.00th=[ 877], 50.00th=[ 936], 60.00th=[ 995], | 70.00th=[ 1062], 80.00th=[ 1150], 90.00th=[ 1267], 95.00th=[ 1368], | 99.00th=[ 1603], 99.50th=[ 1620], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 7, max= 234, per=25.12%, avg=67.82, stdev=31.78, samples=297 iops : min= 1, max= 58, avg=16.51, stdev= 7.98, samples=297 lat (msec) : 250=0.62%, 500=3.20%, 750=17.66%, 1000=39.10%, 2000=39.41% cpu : usr=0.05%, sys=0.30%, ctx=1886, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145770: Fri Sep 4 05:36:06 2026 write: IOPS=16, BW=67.6KiB/s (69.3kB/s)(10.0MiB/151382msec); 0 zone resets slat (usec): min=50, max=14291, avg=311.30, stdev=903.96 clat (msec): min=77, max=1759, avg=944.81, stdev=249.47 lat (msec): min=77, max=1759, avg=945.12, stdev=249.45 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 575], 10.00th=[ 667], 20.00th=[ 743], | 30.00th=[ 810], 40.00th=[ 877], 50.00th=[ 936], 60.00th=[ 986], | 70.00th=[ 1053], 80.00th=[ 1150], 90.00th=[ 1250], 95.00th=[ 1368], | 99.00th=[ 1586], 99.50th=[ 1687], 99.90th=[ 1720], 99.95th=[ 1737], | 99.99th=[ 1754] bw ( KiB/s): min= 7, max= 220, per=24.88%, avg=67.19, stdev=28.94, samples=300 iops : min= 1, max= 55, avg=16.34, stdev= 7.27, samples=300 lat (msec) : 100=0.04%, 250=0.20%, 500=2.70%, 750=17.81%, 1000=41.52% lat (msec) : 2000=37.73% cpu : usr=0.05%, sys=0.29%, ctx=2146, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=270KiB/s (277kB/s), 67.5KiB/s-67.8KiB/s (69.1kB/s-69.4kB/s), io=40.0MiB (41.9MB), run=151125-151678msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145802: Fri Sep 4 05:38:00 2026 read: IOPS=12, BW=48.1KiB/s (49.2kB/s)(5048KiB/104982msec) slat (usec): min=2577, max=49786, avg=14289.71, stdev=7487.67 clat (msec): min=13, max=753, avg=402.99, stdev=100.34 lat (msec): min=20, max=764, avg=417.29, stdev=101.66 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 266], 10.00th=[ 288], 20.00th=[ 321], | 30.00th=[ 347], 40.00th=[ 372], 50.00th=[ 397], 60.00th=[ 426], | 70.00th=[ 447], 80.00th=[ 485], 90.00th=[ 527], 95.00th=[ 575], | 99.00th=[ 676], 99.50th=[ 709], 99.90th=[ 743], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 7, max= 127, per=25.03%, avg=47.80, stdev=20.41, samples=207 iops : min= 1, max= 31, avg=11.44, stdev= 5.15, samples=207 write: IOPS=12, BW=49.5KiB/s (50.6kB/s)(5192KiB/104982msec); 0 zone resets slat (usec): min=52, max=14059, avg=358.92, stdev=926.44 clat (msec): min=256, max=1522, avg=885.41, stdev=224.98 lat (msec): min=256, max=1523, avg=885.77, stdev=225.06 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 523], 10.00th=[ 592], 20.00th=[ 684], | 30.00th=[ 768], 40.00th=[ 818], 50.00th=[ 869], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1083], 90.00th=[ 1200], 95.00th=[ 1284], | 99.00th=[ 1418], 99.50th=[ 1435], 99.90th=[ 1485], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 6, max= 111, per=25.00%, avg=49.51, stdev=21.80, samples=205 iops : min= 1, max= 27, avg=11.88, stdev= 5.50, samples=205 lat (msec) : 20=0.04%, 50=0.08%, 100=0.31%, 250=1.37%, 500=41.68% lat (msec) : 750=19.53%, 1000=21.91%, 2000=15.08% cpu : usr=0.09%, sys=0.73%, ctx=5015, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145803: Fri Sep 4 05:38:01 2026 read: IOPS=11, BW=47.5KiB/s (48.7kB/s)(4976KiB/104734msec) slat (usec): min=59, max=51153, avg=14095.82, stdev=7595.76 clat (msec): min=26, max=1141, avg=402.07, stdev=108.29 lat (msec): min=30, max=1145, avg=416.18, stdev=109.35 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 255], 10.00th=[ 288], 20.00th=[ 321], | 30.00th=[ 342], 40.00th=[ 372], 50.00th=[ 397], 60.00th=[ 418], | 70.00th=[ 447], 80.00th=[ 481], 90.00th=[ 527], 95.00th=[ 575], | 99.00th=[ 684], 99.50th=[ 726], 99.90th=[ 1133], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 119, per=24.79%, avg=47.34, stdev=22.72, samples=206 iops : min= 1, max= 29, avg=11.33, stdev= 5.72, samples=206 write: IOPS=12, BW=50.3KiB/s (51.5kB/s)(5264KiB/104734msec); 0 zone resets slat (usec): min=56, max=16824, avg=400.96, stdev=1158.67 clat (msec): min=160, max=1756, avg=878.47, stdev=234.91 lat (msec): min=160, max=1756, avg=878.87, stdev=234.95 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 518], 10.00th=[ 575], 20.00th=[ 676], | 30.00th=[ 751], 40.00th=[ 810], 50.00th=[ 869], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1083], 90.00th=[ 1200], 95.00th=[ 1267], | 99.00th=[ 1418], 99.50th=[ 1452], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 7, max= 111, per=25.33%, avg=50.16, stdev=21.53, samples=205 iops : min= 1, max= 27, avg=12.04, stdev= 5.42, samples=205 lat (msec) : 50=0.20%, 100=0.39%, 250=1.88%, 500=40.62%, 750=20.59% lat (msec) : 1000=21.29%, 2000=15.04% cpu : usr=0.10%, sys=0.65%, ctx=4799, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145804: Fri Sep 4 05:38:01 2026 read: IOPS=11, BW=47.4KiB/s (48.6kB/s)(4984KiB/105051msec) slat (usec): min=3056, max=50510, avg=14193.45, stdev=7428.04 clat (msec): min=6, max=752, avg=403.59, stdev=98.03 lat (msec): min=11, max=778, avg=417.79, stdev=99.37 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 268], 10.00th=[ 296], 20.00th=[ 321], | 30.00th=[ 347], 40.00th=[ 376], 50.00th=[ 397], 60.00th=[ 422], | 70.00th=[ 447], 80.00th=[ 481], 90.00th=[ 531], 95.00th=[ 575], | 99.00th=[ 667], 99.50th=[ 676], 99.90th=[ 735], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 7, max= 127, per=24.60%, avg=46.99, stdev=20.80, samples=207 iops : min= 1, max= 31, avg=11.24, stdev= 5.27, samples=207 write: IOPS=12, BW=50.0KiB/s (51.2kB/s)(5256KiB/105051msec); 0 zone resets slat (usec): min=55, max=33192, avg=376.65, stdev=1237.93 clat (msec): min=119, max=1533, avg=880.01, stdev=229.23 lat (msec): min=119, max=1533, avg=880.39, stdev=229.23 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 514], 10.00th=[ 592], 20.00th=[ 693], | 30.00th=[ 760], 40.00th=[ 818], 50.00th=[ 877], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1083], 90.00th=[ 1183], 95.00th=[ 1267], | 99.00th=[ 1401], 99.50th=[ 1435], 99.90th=[ 1485], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 7, max= 119, per=24.73%, avg=48.97, stdev=19.79, samples=208 iops : min= 1, max= 29, avg=11.74, stdev= 5.02, samples=208 lat (msec) : 10=0.04%, 50=0.08%, 100=0.12%, 250=1.91%, 500=41.02% lat (msec) : 750=19.53%, 1000=22.30%, 2000=15.00% cpu : usr=0.12%, sys=0.67%, ctx=5057, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145805: Fri Sep 4 05:38:01 2026 read: IOPS=12, BW=48.3KiB/s (49.5kB/s)(5060KiB/104730msec) slat (usec): min=3068, max=51563, avg=13967.67, stdev=7343.95 clat (msec): min=54, max=1191, avg=404.67, stdev=100.82 lat (msec): min=60, max=1201, avg=418.64, stdev=101.84 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 271], 10.00th=[ 300], 20.00th=[ 326], | 30.00th=[ 351], 40.00th=[ 376], 50.00th=[ 397], 60.00th=[ 422], | 70.00th=[ 443], 80.00th=[ 477], 90.00th=[ 523], 95.00th=[ 575], | 99.00th=[ 676], 99.50th=[ 718], 99.90th=[ 1133], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 7, max= 104, per=25.24%, avg=48.20, stdev=20.92, samples=206 iops : min= 1, max= 26, avg=11.54, stdev= 5.26, samples=206 write: IOPS=12, BW=49.5KiB/s (50.6kB/s)(5180KiB/104730msec); 0 zone resets slat (usec): min=58, max=11583, avg=305.86, stdev=626.10 clat (msec): min=120, max=1755, avg=882.00, stdev=226.00 lat (msec): min=120, max=1755, avg=882.30, stdev=226.02 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 527], 10.00th=[ 592], 20.00th=[ 693], | 30.00th=[ 768], 40.00th=[ 818], 50.00th=[ 869], 60.00th=[ 927], | 70.00th=[ 995], 80.00th=[ 1070], 90.00th=[ 1167], 95.00th=[ 1267], | 99.00th=[ 1418], 99.50th=[ 1569], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 7, max= 103, per=24.68%, avg=48.86, stdev=19.47, samples=207 iops : min= 1, max= 25, avg=11.71, stdev= 4.91, samples=207 lat (msec) : 100=0.20%, 250=1.60%, 500=42.19%, 750=18.87%, 1000=22.34% lat (msec) : 2000=14.80% cpu : usr=0.09%, sys=0.68%, ctx=5091, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=191KiB/s (196kB/s), 47.4KiB/s-48.3KiB/s (48.6kB/s-49.5kB/s), io=19.6MiB (20.5MB), run=104730-105051msec WRITE: bw=199KiB/s (204kB/s), 49.5KiB/s-50.3KiB/s (50.6kB/s-51.5kB/s), io=20.4MiB (21.4MB), run=104730-105051msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145824: Fri Sep 4 05:38:12 2026 read: IOPS=1, BW=66.8MiB/s (70.0MB/s)(40.0MiB/599msec) slat (nsec): min=47086k, max=47086k, avg=47085981.00, stdev= 0.00 clat (nsec): min=524433k, max=524433k, avg=524432933.00, stdev= 0.00 lat (nsec): min=571524k, max=571524k, avg=571523602.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 527], 5.00th=[ 527], 10.00th=[ 527], 20.00th=[ 527], | 30.00th=[ 527], 40.00th=[ 527], 50.00th=[ 527], 60.00th=[ 527], | 70.00th=[ 527], 80.00th=[ 527], 90.00th=[ 527], 95.00th=[ 527], | 99.00th=[ 527], 99.50th=[ 527], 99.90th=[ 527], 99.95th=[ 527], | 99.99th=[ 527] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.67%, ctx=43, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=66.8MiB/s (70.0MB/s), 66.8MiB/s-66.8MiB/s (70.0MB/s-70.0MB/s), io=40.0MiB (41.9MB), run=599-599msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout