== sanity test 398c: run fio to test AIO ================= 19:54:33 (1763340873) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.33865 s, 124 MB/s osc.lustre-OST0000-osc-ffff960a08376800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=149692: Sun Nov 16 19:57:00 2025 write: IOPS=18, BW=73.3KiB/s (75.1kB/s)(10.0MiB/139648msec); 0 zone resets slat (usec): min=24, max=8962, avg=151.66, stdev=472.21 clat (msec): min=112, max=1461, avg=872.30, stdev=204.54 lat (msec): min=112, max=1461, avg=872.45, stdev=204.56 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 592], 10.00th=[ 651], 20.00th=[ 735], | 30.00th=[ 785], 40.00th=[ 827], 50.00th=[ 877], 60.00th=[ 911], | 70.00th=[ 953], 80.00th=[ 1011], 90.00th=[ 1133], 95.00th=[ 1217], | 99.00th=[ 1351], 99.50th=[ 1401], 99.90th=[ 1452], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 7, max= 319, per=24.85%, avg=72.81, stdev=31.78, samples=276 iops : min= 1, max= 79, avg=17.71, stdev= 7.97, samples=276 lat (msec) : 250=1.88%, 500=1.37%, 750=20.39%, 1000=55.23%, 2000=21.13% cpu : usr=0.04%, sys=0.27%, ctx=1543, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149693: Sun Nov 16 19:57:00 2025 write: IOPS=18, BW=73.3KiB/s (75.0kB/s)(10.0MiB/139766msec); 0 zone resets slat (usec): min=24, max=7764, avg=163.17, stdev=435.73 clat (msec): min=173, max=1428, avg=872.74, stdev=195.75 lat (msec): min=173, max=1428, avg=872.91, stdev=195.76 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 600], 10.00th=[ 659], 20.00th=[ 726], | 30.00th=[ 785], 40.00th=[ 827], 50.00th=[ 869], 60.00th=[ 902], | 70.00th=[ 961], 80.00th=[ 1020], 90.00th=[ 1116], 95.00th=[ 1217], | 99.00th=[ 1351], 99.50th=[ 1368], 99.90th=[ 1401], 99.95th=[ 1401], | 99.99th=[ 1435] bw ( KiB/s): min= 23, max= 214, per=24.88%, avg=72.89, stdev=25.37, samples=276 iops : min= 5, max= 53, avg=17.74, stdev= 6.36, samples=276 lat (msec) : 250=0.78%, 500=2.30%, 750=20.43%, 1000=53.67%, 2000=22.81% cpu : usr=0.05%, sys=0.28%, ctx=1874, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149694: Sun Nov 16 19:57:00 2025 write: IOPS=18, BW=73.3KiB/s (75.0kB/s)(10.0MiB/139766msec); 0 zone resets slat (usec): min=26, max=6139, avg=142.28, stdev=331.58 clat (msec): min=170, max=1469, avg=872.63, stdev=199.86 lat (msec): min=170, max=1469, avg=872.77, stdev=199.87 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 592], 10.00th=[ 659], 20.00th=[ 735], | 30.00th=[ 785], 40.00th=[ 827], 50.00th=[ 869], 60.00th=[ 911], | 70.00th=[ 953], 80.00th=[ 1020], 90.00th=[ 1116], 95.00th=[ 1217], | 99.00th=[ 1368], 99.50th=[ 1385], 99.90th=[ 1452], 99.95th=[ 1452], | 99.99th=[ 1469] bw ( KiB/s): min= 15, max= 239, per=24.98%, avg=73.19, stdev=31.15, samples=275 iops : min= 3, max= 59, avg=17.81, stdev= 7.81, samples=275 lat (msec) : 250=1.25%, 500=2.19%, 750=19.41%, 1000=54.61%, 2000=22.54% cpu : usr=0.05%, sys=0.25%, ctx=1503, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149695: Sun Nov 16 19:57:00 2025 write: IOPS=18, BW=73.3KiB/s (75.0kB/s)(10.0MiB/139767msec); 0 zone resets slat (usec): min=26, max=12221, avg=175.82, stdev=545.72 clat (msec): min=94, max=1424, avg=872.77, stdev=200.43 lat (msec): min=94, max=1424, avg=872.95, stdev=200.42 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 600], 10.00th=[ 667], 20.00th=[ 726], | 30.00th=[ 785], 40.00th=[ 827], 50.00th=[ 860], 60.00th=[ 902], | 70.00th=[ 961], 80.00th=[ 1020], 90.00th=[ 1133], 95.00th=[ 1234], | 99.00th=[ 1351], 99.50th=[ 1385], 99.90th=[ 1418], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 7, max= 214, per=24.70%, avg=72.38, stdev=26.97, samples=278 iops : min= 1, max= 53, avg=17.60, stdev= 6.78, samples=278 lat (msec) : 100=0.04%, 250=1.25%, 500=1.84%, 750=20.31%, 1000=53.48% lat (msec) : 2000=23.09% cpu : usr=0.06%, sys=0.28%, ctx=1928, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=293KiB/s (300kB/s), 73.3KiB/s-73.3KiB/s (75.0kB/s-75.1kB/s), io=40.0MiB (41.9MB), run=139648-139767msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=149724: Sun Nov 16 19:58:44 2025 read: IOPS=12, BW=51.5KiB/s (52.7kB/s)(5048KiB/98019msec) slat (usec): min=2786, max=38119, avg=11132.59, stdev=4760.16 clat (msec): min=10, max=958, avg=384.44, stdev=88.76 lat (msec): min=21, max=970, avg=395.58, stdev=89.31 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 262], 10.00th=[ 292], 20.00th=[ 317], | 30.00th=[ 342], 40.00th=[ 359], 50.00th=[ 380], 60.00th=[ 397], | 70.00th=[ 418], 80.00th=[ 447], 90.00th=[ 493], 95.00th=[ 542], | 99.00th=[ 625], 99.50th=[ 659], 99.90th=[ 877], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 7, max= 119, per=25.06%, avg=51.11, stdev=21.83, samples=194 iops : min= 1, max= 29, avg=12.28, stdev= 5.53, samples=194 write: IOPS=13, BW=52.0KiB/s (54.2kB/s)(5192KiB/98019msec); 0 zone resets slat (usec): min=30, max=3352, avg=188.74, stdev=340.10 clat (msec): min=183, max=1437, avg=821.02, stdev=211.00 lat (msec): min=183, max=1437, avg=821.21, stdev=211.00 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 481], 10.00th=[ 542], 20.00th=[ 634], | 30.00th=[ 693], 40.00th=[ 760], 50.00th=[ 818], 60.00th=[ 885], | 70.00th=[ 953], 80.00th=[ 1020], 90.00th=[ 1099], 95.00th=[ 1150], | 99.00th=[ 1318], 99.50th=[ 1368], 99.90th=[ 1418], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 7, max= 111, per=24.73%, avg=52.42, stdev=20.71, samples=194 iops : min= 1, max= 27, avg=12.61, stdev= 5.19, samples=194 lat (msec) : 20=0.08%, 50=0.04%, 100=0.16%, 250=1.91%, 500=46.13% lat (msec) : 750=20.78%, 1000=19.61%, 2000=11.29% cpu : usr=0.09%, sys=0.69%, ctx=3982, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149725: Sun Nov 16 19:58:44 2025 read: IOPS=12, BW=50.7KiB/s (51.9kB/s)(4976KiB/98229msec) slat (msec): min=2, max=466, avg=12.14, stdev=20.41 clat (msec): min=14, max=1424, avg=379.22, stdev=96.74 lat (msec): min=25, max=1432, avg=391.36, stdev=101.01 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 262], 10.00th=[ 288], 20.00th=[ 313], | 30.00th=[ 330], 40.00th=[ 355], 50.00th=[ 372], 60.00th=[ 388], | 70.00th=[ 414], 80.00th=[ 443], 90.00th=[ 485], 95.00th=[ 531], | 99.00th=[ 609], 99.50th=[ 667], 99.90th=[ 1200], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 7, max= 144, per=24.69%, avg=50.36, stdev=23.39, samples=194 iops : min= 1, max= 36, avg=12.10, stdev= 5.89, samples=194 write: IOPS=13, BW=53.6KiB/s (54.9kB/s)(5264KiB/98229msec); 0 zone resets slat (usec): min=27, max=10012, avg=210.80, stdev=540.86 clat (msec): min=189, max=2099, avg=822.20, stdev=222.03 lat (msec): min=190, max=2099, avg=822.41, stdev=222.02 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 481], 10.00th=[ 535], 20.00th=[ 634], | 30.00th=[ 701], 40.00th=[ 760], 50.00th=[ 818], 60.00th=[ 877], | 70.00th=[ 936], 80.00th=[ 1003], 90.00th=[ 1099], 95.00th=[ 1167], | 99.00th=[ 1368], 99.50th=[ 1603], 99.90th=[ 1972], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 15, max= 126, per=25.08%, avg=53.16, stdev=19.89, samples=194 iops : min= 3, max= 31, avg=12.80, stdev= 4.99, samples=194 lat (msec) : 20=0.04%, 50=0.12%, 100=0.23%, 250=1.76%, 500=46.13% lat (msec) : 750=19.96%, 1000=21.29%, 2000=10.43%, >=2000=0.04% cpu : usr=0.10%, sys=0.64%, ctx=3695, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149726: Sun Nov 16 19:58:44 2025 read: IOPS=12, BW=50.8KiB/s (51.0kB/s)(4984KiB/98150msec) slat (usec): min=3094, max=39684, avg=11214.81, stdev=4782.52 clat (msec): min=16, max=700, avg=380.25, stdev=86.45 lat (msec): min=20, max=711, avg=391.48, stdev=86.91 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 264], 10.00th=[ 288], 20.00th=[ 317], | 30.00th=[ 338], 40.00th=[ 355], 50.00th=[ 376], 60.00th=[ 393], | 70.00th=[ 418], 80.00th=[ 447], 90.00th=[ 498], 95.00th=[ 527], | 99.00th=[ 584], 99.50th=[ 651], 99.90th=[ 693], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 7, max= 119, per=24.70%, avg=50.38, stdev=22.88, samples=195 iops : min= 1, max= 29, avg=12.10, stdev= 5.77, samples=195 write: IOPS=13, BW=53.5KiB/s (54.8kB/s)(5256KiB/98150msec); 0 zone resets slat (usec): min=28, max=6804, avg=204.31, stdev=454.29 clat (msec): min=220, max=1397, avg=821.62, stdev=209.55 lat (msec): min=220, max=1397, avg=821.82, stdev=209.57 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 498], 10.00th=[ 542], 20.00th=[ 625], | 30.00th=[ 693], 40.00th=[ 768], 50.00th=[ 835], 60.00th=[ 885], | 70.00th=[ 953], 80.00th=[ 1011], 90.00th=[ 1083], 95.00th=[ 1167], | 99.00th=[ 1267], 99.50th=[ 1334], 99.90th=[ 1401], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 15, max= 111, per=24.82%, avg=52.61, stdev=19.77, samples=195 iops : min= 3, max= 27, avg=12.66, stdev= 4.94, samples=195 lat (msec) : 20=0.04%, 50=0.16%, 100=0.27%, 250=1.29%, 500=45.43% lat (msec) : 750=21.13%, 1000=20.31%, 2000=11.37% cpu : usr=0.10%, sys=0.65%, ctx=3828, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149727: Sun Nov 16 19:58:44 2025 read: IOPS=12, BW=51.6KiB/s (52.8kB/s)(5060KiB/98088msec) slat (usec): min=56, max=477709, avg=11589.29, stdev=16249.09 clat (msec): min=14, max=981, avg=383.44, stdev=92.31 lat (msec): min=22, max=990, avg=395.03, stdev=93.41 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 259], 10.00th=[ 288], 20.00th=[ 321], | 30.00th=[ 338], 40.00th=[ 359], 50.00th=[ 380], 60.00th=[ 393], | 70.00th=[ 418], 80.00th=[ 443], 90.00th=[ 502], 95.00th=[ 542], | 99.00th=[ 634], 99.50th=[ 676], 99.90th=[ 894], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 7, max= 119, per=25.53%, avg=52.09, stdev=22.34, samples=191 iops : min= 1, max= 29, avg=12.52, stdev= 5.60, samples=191 write: IOPS=13, BW=52.8KiB/s (54.1kB/s)(5180KiB/98088msec); 0 zone resets slat (usec): min=28, max=6267, avg=215.64, stdev=478.76 clat (msec): min=89, max=1625, avg=824.96, stdev=221.52 lat (msec): min=90, max=1625, avg=825.18, stdev=221.54 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 485], 10.00th=[ 542], 20.00th=[ 634], | 30.00th=[ 701], 40.00th=[ 768], 50.00th=[ 835], 60.00th=[ 894], | 70.00th=[ 953], 80.00th=[ 1011], 90.00th=[ 1099], 95.00th=[ 1183], | 99.00th=[ 1351], 99.50th=[ 1435], 99.90th=[ 1552], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 7, max= 134, per=24.49%, avg=51.92, stdev=21.69, samples=195 iops : min= 1, max= 33, avg=12.49, stdev= 5.44, samples=195 lat (msec) : 20=0.04%, 50=0.20%, 100=0.27%, 250=1.84%, 500=45.62% lat (msec) : 750=20.47%, 1000=20.43%, 2000=11.13% cpu : usr=0.11%, sys=0.64%, ctx=3960, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=204KiB/s (209kB/s), 50.7KiB/s-51.6KiB/s (51.9kB/s-52.8kB/s), io=19.6MiB (20.5MB), run=98019-98229msec WRITE: bw=213KiB/s (218kB/s), 52.8KiB/s-53.6KiB/s (54.1kB/s-54.9kB/s), io=20.4MiB (21.4MB), run=98019-98229msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=149745: Sun Nov 16 19:58:51 2025 read: IOPS=2, BW=104MiB/s (109MB/s)(40.0MiB/386msec) slat (nsec): min=28370k, max=28370k, avg=28370113.00, stdev= 0.00 clat (nsec): min=333164k, max=333164k, avg=333164047.00, stdev= 0.00 lat (nsec): min=361540k, max=361540k, avg=361540463.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 334], 10.00th=[ 334], 20.00th=[ 334], | 30.00th=[ 334], 40.00th=[ 334], 50.00th=[ 334], 60.00th=[ 334], | 70.00th=[ 334], 80.00th=[ 334], 90.00th=[ 334], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] lat (msec) : 500=100.00% cpu : usr=0.26%, sys=2.60%, ctx=5, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=40.0MiB (41.9MB), run=386-386msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout