== sanity test 398c: run fio to test AIO ================= 05:56:36 (1779098196) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.513421 s, 81.7 MB/s osc.lustre-OST0000-osc-ffff90921e02b000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145525: Mon May 18 05:59:50 2026 write: IOPS=13, BW=55.2KiB/s (56.5kB/s)(10.0MiB/185558msec); 0 zone resets slat (usec): min=36, max=21015, avg=293.76, stdev=1042.53 clat (msec): min=57, max=3176, avg=1158.52, stdev=401.66 lat (msec): min=57, max=3176, avg=1158.81, stdev=401.66 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 584], 10.00th=[ 743], 20.00th=[ 894], | 30.00th=[ 953], 40.00th=[ 1028], 50.00th=[ 1099], 60.00th=[ 1183], | 70.00th=[ 1301], 80.00th=[ 1435], 90.00th=[ 1636], 95.00th=[ 1821], | 99.00th=[ 2534], 99.50th=[ 2836], 99.90th=[ 3071], 99.95th=[ 3171], | 99.99th=[ 3171] bw ( KiB/s): min= 7, max= 247, per=25.53%, avg=56.18, stdev=29.44, samples=358 iops : min= 1, max= 61, avg=13.58, stdev= 7.38, samples=358 lat (msec) : 100=0.04%, 500=4.38%, 750=6.13%, 1000=25.82%, 2000=60.47% lat (msec) : >=2000=3.16% cpu : usr=0.03%, sys=0.21%, ctx=1738, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145526: Mon May 18 05:59:50 2026 write: IOPS=13, BW=55.2KiB/s (56.5kB/s)(10.0MiB/185631msec); 0 zone resets slat (usec): min=37, max=24533, avg=309.95, stdev=981.10 clat (msec): min=269, max=3066, avg=1158.81, stdev=397.08 lat (msec): min=269, max=3066, avg=1159.12, stdev=397.11 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 625], 10.00th=[ 751], 20.00th=[ 885], | 30.00th=[ 953], 40.00th=[ 1020], 50.00th=[ 1116], 60.00th=[ 1200], | 70.00th=[ 1284], 80.00th=[ 1435], 90.00th=[ 1653], 95.00th=[ 1804], | 99.00th=[ 2635], 99.50th=[ 2769], 99.90th=[ 3071], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min= 7, max= 254, per=25.39%, avg=55.86, stdev=31.13, samples=360 iops : min= 1, max= 63, avg=13.50, stdev= 7.81, samples=360 lat (msec) : 500=4.41%, 750=5.35%, 1000=26.84%, 2000=60.90%, >=2000=2.50% cpu : usr=0.04%, sys=0.22%, ctx=2039, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145527: Mon May 18 05:59:50 2026 write: IOPS=13, BW=55.3KiB/s (56.6kB/s)(10.0MiB/185306msec); 0 zone resets slat (usec): min=36, max=16811, avg=320.60, stdev=1065.11 clat (msec): min=150, max=3070, avg=1157.39, stdev=405.26 lat (msec): min=150, max=3071, avg=1157.71, stdev=405.28 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 542], 10.00th=[ 735], 20.00th=[ 869], | 30.00th=[ 953], 40.00th=[ 1028], 50.00th=[ 1116], 60.00th=[ 1200], | 70.00th=[ 1318], 80.00th=[ 1435], 90.00th=[ 1620], 95.00th=[ 1838], | 99.00th=[ 2601], 99.50th=[ 2869], 99.90th=[ 3004], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min= 7, max= 255, per=25.15%, avg=55.33, stdev=33.68, samples=363 iops : min= 1, max= 63, avg=13.37, stdev= 8.47, samples=363 lat (msec) : 250=0.62%, 500=3.87%, 750=6.02%, 1000=24.77%, 2000=61.99% lat (msec) : >=2000=2.73% cpu : usr=0.03%, sys=0.27%, ctx=2182, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145528: Mon May 18 05:59:50 2026 write: IOPS=13, BW=55.3KiB/s (56.6kB/s)(10.0MiB/185119msec); 0 zone resets slat (usec): min=48, max=25699, avg=337.11, stdev=1121.98 clat (msec): min=64, max=3061, avg=1155.78, stdev=404.72 lat (msec): min=65, max=3061, avg=1156.12, stdev=404.68 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 531], 10.00th=[ 735], 20.00th=[ 877], | 30.00th=[ 961], 40.00th=[ 1028], 50.00th=[ 1116], 60.00th=[ 1183], | 70.00th=[ 1301], 80.00th=[ 1452], 90.00th=[ 1636], 95.00th=[ 1821], | 99.00th=[ 2534], 99.50th=[ 2836], 99.90th=[ 2937], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min= 7, max= 207, per=25.90%, avg=56.99, stdev=30.53, samples=353 iops : min= 1, max= 51, avg=13.78, stdev= 7.68, samples=353 lat (msec) : 100=0.08%, 250=0.39%, 500=4.22%, 750=6.48%, 1000=24.69% lat (msec) : 2000=61.09%, >=2000=3.05% cpu : usr=0.04%, sys=0.23%, ctx=2066, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=221KiB/s (226kB/s), 55.2KiB/s-55.3KiB/s (56.5kB/s-56.6kB/s), io=40.0MiB (41.9MB), run=185119-185631msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145565: Mon May 18 06:02:05 2026 read: IOPS=9, BW=39.3KiB/s (40.2kB/s)(5048KiB/128606msec) slat (usec): min=59, max=94946, avg=15725.09, stdev=9845.49 clat (msec): min=48, max=1306, avg=499.53, stdev=168.55 lat (msec): min=52, max=1314, avg=515.26, stdev=171.29 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 292], 10.00th=[ 338], 20.00th=[ 380], | 30.00th=[ 409], 40.00th=[ 439], 50.00th=[ 468], 60.00th=[ 498], | 70.00th=[ 542], 80.00th=[ 600], 90.00th=[ 718], 95.00th=[ 844], | 99.00th=[ 1062], 99.50th=[ 1133], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 7, max= 112, per=25.63%, avg=39.72, stdev=20.69, samples=248 iops : min= 1, max= 28, avg= 9.46, stdev= 5.24, samples=248 write: IOPS=10, BW=40.4KiB/s (41.3kB/s)(5192KiB/128606msec); 0 zone resets slat (usec): min=46, max=44434, avg=446.92, stdev=1836.78 clat (msec): min=96, max=3020, avg=1082.97, stdev=353.66 lat (msec): min=96, max=3020, avg=1083.42, stdev=353.73 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 600], 10.00th=[ 701], 20.00th=[ 827], | 30.00th=[ 902], 40.00th=[ 961], 50.00th=[ 1028], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1318], 90.00th=[ 1519], 95.00th=[ 1720], | 99.00th=[ 2265], 99.50th=[ 2702], 99.90th=[ 3004], 99.95th=[ 3037], | 99.99th=[ 3037] bw ( KiB/s): min= 7, max= 111, per=25.00%, avg=40.51, stdev=20.27, samples=251 iops : min= 1, max= 27, avg= 9.66, stdev= 5.11, samples=251 lat (msec) : 50=0.08%, 100=0.27%, 250=0.82%, 500=29.69%, 750=21.29% lat (msec) : 1000=19.45%, 2000=27.58%, >=2000=0.82% cpu : usr=0.08%, sys=0.59%, ctx=4698, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145566: Mon May 18 06:02:05 2026 read: IOPS=9, BW=38.7KiB/s (39.6kB/s)(4976KiB/128710msec) slat (usec): min=2602, max=87276, avg=15676.62, stdev=9345.69 clat (msec): min=21, max=1390, avg=500.70, stdev=163.68 lat (msec): min=43, max=1411, avg=516.38, stdev=165.76 clat percentiles (msec): | 1.00th=[ 253], 5.00th=[ 313], 10.00th=[ 338], 20.00th=[ 384], | 30.00th=[ 409], 40.00th=[ 439], 50.00th=[ 468], 60.00th=[ 498], | 70.00th=[ 535], 80.00th=[ 600], 90.00th=[ 735], 95.00th=[ 835], | 99.00th=[ 1020], 99.50th=[ 1099], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 7, max= 88, per=24.87%, avg=38.55, stdev=18.70, samples=252 iops : min= 1, max= 22, avg= 9.17, stdev= 4.69, samples=252 write: IOPS=10, BW=40.9KiB/s (41.9kB/s)(5264KiB/128710msec); 0 zone resets slat (usec): min=45, max=13384, avg=375.35, stdev=870.78 clat (msec): min=211, max=3015, avg=1074.77, stdev=341.31 lat (msec): min=211, max=3015, avg=1075.15, stdev=341.32 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 609], 10.00th=[ 701], 20.00th=[ 827], | 30.00th=[ 911], 40.00th=[ 969], 50.00th=[ 1028], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1301], 90.00th=[ 1519], 95.00th=[ 1687], | 99.00th=[ 2232], 99.50th=[ 2467], 99.90th=[ 2836], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 7, max= 96, per=24.85%, avg=40.25, stdev=19.19, samples=256 iops : min= 1, max= 24, avg= 9.59, stdev= 4.86, samples=256 lat (msec) : 50=0.08%, 100=0.12%, 250=0.43%, 500=30.12%, 750=20.51% lat (msec) : 1000=20.35%, 2000=27.58%, >=2000=0.82% cpu : usr=0.08%, sys=0.58%, ctx=4741, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145567: Mon May 18 06:02:05 2026 read: IOPS=9, BW=38.7KiB/s (39.6kB/s)(4984KiB/128878msec) slat (usec): min=2844, max=82047, avg=15586.45, stdev=9057.36 clat (msec): min=16, max=1392, avg=496.57, stdev=160.41 lat (msec): min=33, max=1418, avg=512.16, stdev=162.21 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 309], 10.00th=[ 342], 20.00th=[ 384], | 30.00th=[ 414], 40.00th=[ 439], 50.00th=[ 464], 60.00th=[ 493], | 70.00th=[ 535], 80.00th=[ 592], 90.00th=[ 701], 95.00th=[ 810], | 99.00th=[ 1083], 99.50th=[ 1167], 99.90th=[ 1267], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 7, max= 128, per=25.09%, avg=38.88, stdev=19.96, samples=250 iops : min= 1, max= 32, avg= 9.25, stdev= 5.04, samples=250 write: IOPS=10, BW=40.8KiB/s (41.8kB/s)(5256KiB/128878msec); 0 zone resets slat (usec): min=57, max=30254, avg=434.65, stdev=1629.70 clat (msec): min=91, max=3017, avg=1081.10, stdev=353.73 lat (msec): min=95, max=3017, avg=1081.54, stdev=353.78 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 609], 10.00th=[ 701], 20.00th=[ 827], | 30.00th=[ 902], 40.00th=[ 969], 50.00th=[ 1036], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1334], 90.00th=[ 1552], 95.00th=[ 1670], | 99.00th=[ 2333], 99.50th=[ 2567], 99.90th=[ 3004], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 7, max= 88, per=24.83%, avg=40.22, stdev=18.46, samples=254 iops : min= 1, max= 22, avg= 9.59, stdev= 4.64, samples=254 lat (msec) : 20=0.04%, 50=0.08%, 100=0.20%, 250=0.55%, 500=30.39% lat (msec) : 750=21.33%, 1000=18.44%, 2000=28.05%, >=2000=0.94% cpu : usr=0.07%, sys=0.61%, ctx=4521, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145568: Mon May 18 06:02:05 2026 read: IOPS=9, BW=39.3KiB/s (40.2kB/s)(5060KiB/128814msec) slat (usec): min=2812, max=94095, avg=15919.25, stdev=9506.25 clat (msec): min=25, max=1419, avg=506.30, stdev=165.90 lat (msec): min=38, max=1427, avg=522.23, stdev=168.29 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 313], 10.00th=[ 347], 20.00th=[ 393], | 30.00th=[ 418], 40.00th=[ 451], 50.00th=[ 472], 60.00th=[ 506], | 70.00th=[ 542], 80.00th=[ 592], 90.00th=[ 726], 95.00th=[ 835], | 99.00th=[ 1099], 99.50th=[ 1167], 99.90th=[ 1401], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 7, max= 96, per=25.84%, avg=40.04, stdev=19.01, samples=247 iops : min= 1, max= 24, avg= 9.54, stdev= 4.80, samples=247 write: IOPS=10, BW=40.2KiB/s (41.2kB/s)(5180KiB/128814msec); 0 zone resets slat (usec): min=56, max=24317, avg=438.02, stdev=1490.89 clat (msec): min=143, max=3008, avg=1079.00, stdev=339.45 lat (msec): min=143, max=3008, avg=1079.44, stdev=339.51 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 609], 10.00th=[ 701], 20.00th=[ 818], | 30.00th=[ 902], 40.00th=[ 961], 50.00th=[ 1028], 60.00th=[ 1083], | 70.00th=[ 1183], 80.00th=[ 1318], 90.00th=[ 1569], 95.00th=[ 1703], | 99.00th=[ 2140], 99.50th=[ 2400], 99.90th=[ 2567], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 7, max= 95, per=24.63%, avg=39.90, stdev=18.51, samples=254 iops : min= 1, max= 23, avg= 9.50, stdev= 4.65, samples=254 lat (msec) : 50=0.04%, 100=0.20%, 250=0.43%, 500=28.95%, 750=22.85% lat (msec) : 1000=19.49%, 2000=27.27%, >=2000=0.78% cpu : usr=0.08%, sys=0.59%, ctx=4557, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=156KiB/s (159kB/s), 38.7KiB/s-39.3KiB/s (39.6kB/s-40.2kB/s), io=19.6MiB (20.5MB), run=128606-128878msec WRITE: bw=162KiB/s (166kB/s), 40.2KiB/s-40.9KiB/s (41.2kB/s-41.9kB/s), io=20.4MiB (21.4MB), run=128606-128878msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145603: Mon May 18 06:02:17 2026 read: IOPS=1, BW=66.0MiB/s (69.2MB/s)(40.0MiB/606msec) slat (nsec): min=63699k, max=63699k, avg=63698947.00, stdev= 0.00 clat (nsec): min=498652k, max=498652k, avg=498651545.00, stdev= 0.00 lat (nsec): min=562356k, max=562356k, avg=562355827.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 498], 5.00th=[ 498], 10.00th=[ 498], 20.00th=[ 498], | 30.00th=[ 498], 40.00th=[ 498], 50.00th=[ 498], 60.00th=[ 498], | 70.00th=[ 498], 80.00th=[ 498], 90.00th=[ 498], 95.00th=[ 498], | 99.00th=[ 498], 99.50th=[ 498], 99.90th=[ 498], 99.95th=[ 498], | 99.99th=[ 498] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=0.99%, ctx=6, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=66.0MiB/s (69.2MB/s), 66.0MiB/s-66.0MiB/s (69.2MB/s-69.2MB/s), io=40.0MiB (41.9MB), run=606-606msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout