== sanity test 398c: run fio to test AIO ================= 19:33:33 (1788564813) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.35301 s, 119 MB/s osc.lustre-OST0000-osc-ffff99d5c8bd2800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145562: Fri Sep 4 19:35:51 2026 write: IOPS=19, BW=78.4KiB/s (80.3kB/s)(10.0MiB/130595msec); 0 zone resets slat (usec): min=27, max=11690, avg=195.98, stdev=657.31 clat (msec): min=134, max=1938, avg=815.26, stdev=230.58 lat (msec): min=134, max=1938, avg=815.46, stdev=230.58 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 464], 10.00th=[ 567], 20.00th=[ 642], | 30.00th=[ 709], 40.00th=[ 760], 50.00th=[ 810], 60.00th=[ 860], | 70.00th=[ 919], 80.00th=[ 969], 90.00th=[ 1083], 95.00th=[ 1183], | 99.00th=[ 1502], 99.50th=[ 1586], 99.90th=[ 1821], 99.95th=[ 1821], | 99.99th=[ 1938] bw ( KiB/s): min= 7, max= 256, per=24.80%, avg=77.63, stdev=33.14, samples=260 iops : min= 1, max= 64, avg=19.00, stdev= 8.29, samples=260 lat (msec) : 250=1.25%, 500=4.69%, 750=31.68%, 1000=45.39%, 2000=16.99% cpu : usr=0.04%, sys=0.25%, ctx=2171, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145563: Fri Sep 4 19:35:51 2026 write: IOPS=19, BW=78.3KiB/s (80.2kB/s)(10.0MiB/130695msec); 0 zone resets slat (usec): min=26, max=14316, avg=182.46, stdev=680.98 clat (msec): min=134, max=2047, avg=815.89, stdev=231.20 lat (msec): min=134, max=2047, avg=816.07, stdev=231.20 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 472], 10.00th=[ 558], 20.00th=[ 642], | 30.00th=[ 709], 40.00th=[ 760], 50.00th=[ 818], 60.00th=[ 860], | 70.00th=[ 911], 80.00th=[ 978], 90.00th=[ 1062], 95.00th=[ 1183], | 99.00th=[ 1485], 99.50th=[ 1552], 99.90th=[ 1938], 99.95th=[ 2022], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 256, per=24.81%, avg=77.65, stdev=31.16, samples=260 iops : min= 1, max= 64, avg=19.00, stdev= 7.78, samples=260 lat (msec) : 250=1.25%, 500=4.49%, 750=31.95%, 1000=44.80%, 2000=17.42% lat (msec) : >=2000=0.08% cpu : usr=0.04%, sys=0.24%, ctx=1722, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145564: Fri Sep 4 19:35:51 2026 write: IOPS=19, BW=78.5KiB/s (80.3kB/s)(10.0MiB/130518msec); 0 zone resets slat (usec): min=27, max=14994, avg=187.65, stdev=656.10 clat (msec): min=140, max=2045, avg=815.13, stdev=237.18 lat (msec): min=140, max=2045, avg=815.32, stdev=237.20 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 468], 10.00th=[ 567], 20.00th=[ 642], | 30.00th=[ 709], 40.00th=[ 760], 50.00th=[ 810], 60.00th=[ 860], | 70.00th=[ 911], 80.00th=[ 978], 90.00th=[ 1083], 95.00th=[ 1183], | 99.00th=[ 1485], 99.50th=[ 1720], 99.90th=[ 2022], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 15, max= 255, per=24.90%, avg=77.94, stdev=33.27, samples=259 iops : min= 3, max= 63, avg=19.08, stdev= 8.30, samples=259 lat (msec) : 250=1.25%, 500=5.04%, 750=32.11%, 1000=44.34%, 2000=17.15% lat (msec) : >=2000=0.12% cpu : usr=0.05%, sys=0.23%, ctx=1781, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145565: Fri Sep 4 19:35:51 2026 write: IOPS=19, BW=78.4KiB/s (80.2kB/s)(10.0MiB/130694msec); 0 zone resets slat (usec): min=23, max=19412, avg=211.40, stdev=790.51 clat (msec): min=51, max=2047, avg=815.84, stdev=234.02 lat (msec): min=51, max=2047, avg=816.06, stdev=234.02 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 460], 10.00th=[ 550], 20.00th=[ 642], | 30.00th=[ 701], 40.00th=[ 760], 50.00th=[ 818], 60.00th=[ 869], | 70.00th=[ 919], 80.00th=[ 978], 90.00th=[ 1070], 95.00th=[ 1167], | 99.00th=[ 1502], 99.50th=[ 1620], 99.90th=[ 1905], 99.95th=[ 1905], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 248, per=24.81%, avg=77.67, stdev=34.37, samples=260 iops : min= 1, max= 62, avg=19.01, stdev= 8.57, samples=260 lat (msec) : 100=0.12%, 250=1.37%, 500=5.04%, 750=31.09%, 1000=45.35% lat (msec) : 2000=16.99%, >=2000=0.04% cpu : usr=0.04%, sys=0.23%, ctx=2086, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=313KiB/s (321kB/s), 78.3KiB/s-78.5KiB/s (80.2kB/s-80.3kB/s), io=40.0MiB (41.9MB), run=130518-130695msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145594: Fri Sep 4 19:37:39 2026 read: IOPS=12, BW=49.9KiB/s (51.1kB/s)(5048KiB/101185msec) slat (usec): min=29, max=82643, avg=12308.58, stdev=6756.72 clat (msec): min=29, max=1176, avg=392.67, stdev=116.14 lat (msec): min=34, max=1201, avg=404.99, stdev=117.57 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 255], 10.00th=[ 284], 20.00th=[ 309], | 30.00th=[ 330], 40.00th=[ 351], 50.00th=[ 372], 60.00th=[ 397], | 70.00th=[ 426], 80.00th=[ 468], 90.00th=[ 531], 95.00th=[ 592], | 99.00th=[ 768], 99.50th=[ 953], 99.90th=[ 1167], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 7, max= 151, per=25.16%, avg=49.58, stdev=22.21, samples=200 iops : min= 1, max= 37, avg=11.94, stdev= 5.54, samples=200 write: IOPS=12, BW=51.3KiB/s (52.5kB/s)(5192KiB/101185msec); 0 zone resets slat (usec): min=31, max=14084, avg=192.92, stdev=603.79 clat (msec): min=87, max=1891, avg=852.55, stdev=233.38 lat (msec): min=87, max=1891, avg=852.75, stdev=233.39 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 518], 10.00th=[ 592], 20.00th=[ 667], | 30.00th=[ 726], 40.00th=[ 776], 50.00th=[ 835], 60.00th=[ 885], | 70.00th=[ 953], 80.00th=[ 1020], 90.00th=[ 1167], 95.00th=[ 1284], | 99.00th=[ 1435], 99.50th=[ 1552], 99.90th=[ 1888], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 119, per=24.66%, avg=50.56, stdev=19.51, samples=201 iops : min= 1, max= 29, avg=12.18, stdev= 4.93, samples=201 lat (msec) : 50=0.04%, 100=0.23%, 250=2.23%, 500=41.76%, 750=21.72% lat (msec) : 1000=21.95%, 2000=12.07% cpu : usr=0.08%, sys=0.56%, ctx=4544, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145595: Fri Sep 4 19:37:39 2026 read: IOPS=12, BW=49.1KiB/s (50.3kB/s)(4976KiB/101388msec) slat (usec): min=2130, max=54536, avg=12401.68, stdev=6966.10 clat (msec): min=18, max=1035, avg=387.02, stdev=108.73 lat (msec): min=28, max=1047, avg=399.43, stdev=110.08 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 251], 10.00th=[ 279], 20.00th=[ 309], | 30.00th=[ 334], 40.00th=[ 355], 50.00th=[ 372], 60.00th=[ 397], | 70.00th=[ 418], 80.00th=[ 456], 90.00th=[ 510], 95.00th=[ 600], | 99.00th=[ 743], 99.50th=[ 768], 99.90th=[ 969], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 7, max= 119, per=25.02%, avg=49.29, stdev=22.81, samples=198 iops : min= 1, max= 29, avg=11.87, stdev= 5.71, samples=198 write: IOPS=12, BW=51.9KiB/s (53.2kB/s)(5264KiB/101388msec); 0 zone resets slat (usec): min=32, max=26103, avg=243.99, stdev=1003.09 clat (msec): min=118, max=1611, avg=853.56, stdev=234.28 lat (msec): min=118, max=1612, avg=853.81, stdev=234.29 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 518], 10.00th=[ 584], 20.00th=[ 667], | 30.00th=[ 726], 40.00th=[ 785], 50.00th=[ 835], 60.00th=[ 894], | 70.00th=[ 953], 80.00th=[ 1028], 90.00th=[ 1167], 95.00th=[ 1284], | 99.00th=[ 1418], 99.50th=[ 1469], 99.90th=[ 1552], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 8, max= 151, per=25.27%, avg=51.79, stdev=21.28, samples=199 iops : min= 2, max= 37, avg=12.50, stdev= 5.32, samples=199 lat (msec) : 20=0.04%, 50=0.04%, 100=0.27%, 250=2.54%, 500=42.15% lat (msec) : 750=20.98%, 1000=22.11%, 2000=11.88% cpu : usr=0.08%, sys=0.55%, ctx=4749, majf=0, minf=18 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145596: Fri Sep 4 19:37:39 2026 read: IOPS=12, BW=49.1KiB/s (50.3kB/s)(4984KiB/101491msec) slat (usec): min=2068, max=59361, avg=12525.25, stdev=7112.61 clat (msec): min=25, max=968, avg=387.21, stdev=108.98 lat (msec): min=37, max=975, avg=399.74, stdev=110.30 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 251], 10.00th=[ 279], 20.00th=[ 305], | 30.00th=[ 330], 40.00th=[ 351], 50.00th=[ 372], 60.00th=[ 397], | 70.00th=[ 422], 80.00th=[ 460], 90.00th=[ 523], 95.00th=[ 592], | 99.00th=[ 735], 99.50th=[ 768], 99.90th=[ 818], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 7, max= 120, per=24.75%, avg=48.76, stdev=22.76, samples=200 iops : min= 1, max= 30, avg=11.74, stdev= 5.69, samples=200 write: IOPS=12, BW=51.8KiB/s (53.0kB/s)(5256KiB/101491msec); 0 zone resets slat (usec): min=32, max=13821, avg=234.10, stdev=792.12 clat (msec): min=72, max=1650, avg=855.22, stdev=235.44 lat (msec): min=72, max=1650, avg=855.46, stdev=235.43 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 514], 10.00th=[ 592], 20.00th=[ 676], | 30.00th=[ 735], 40.00th=[ 776], 50.00th=[ 827], 60.00th=[ 885], | 70.00th=[ 961], 80.00th=[ 1045], 90.00th=[ 1167], 95.00th=[ 1301], | 99.00th=[ 1401], 99.50th=[ 1452], 99.90th=[ 1586], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 8, max= 103, per=24.84%, avg=50.92, stdev=19.70, samples=200 iops : min= 2, max= 25, avg=12.28, stdev= 5.00, samples=200 lat (msec) : 50=0.16%, 100=0.39%, 250=2.30%, 500=41.76%, 750=21.13% lat (msec) : 1000=21.60%, 2000=12.66% cpu : usr=0.07%, sys=0.60%, ctx=5032, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145597: Fri Sep 4 19:37:39 2026 read: IOPS=12, BW=49.9KiB/s (51.1kB/s)(5060KiB/101330msec) slat (usec): min=2232, max=65411, avg=12418.61, stdev=6893.83 clat (msec): min=29, max=1037, avg=389.35, stdev=111.95 lat (msec): min=42, max=1050, avg=401.77, stdev=113.11 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 251], 10.00th=[ 279], 20.00th=[ 305], | 30.00th=[ 326], 40.00th=[ 351], 50.00th=[ 372], 60.00th=[ 397], | 70.00th=[ 430], 80.00th=[ 464], 90.00th=[ 527], 95.00th=[ 600], | 99.00th=[ 760], 99.50th=[ 802], 99.90th=[ 1020], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 7, max= 120, per=25.52%, avg=50.28, stdev=22.86, samples=198 iops : min= 1, max= 30, avg=12.12, stdev= 5.73, samples=198 write: IOPS=12, BW=51.1KiB/s (52.3kB/s)(5180KiB/101330msec); 0 zone resets slat (usec): min=33, max=11271, avg=227.73, stdev=656.14 clat (msec): min=99, max=2045, avg=857.75, stdev=238.84 lat (msec): min=100, max=2045, avg=857.99, stdev=238.86 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 542], 10.00th=[ 609], 20.00th=[ 667], | 30.00th=[ 735], 40.00th=[ 768], 50.00th=[ 827], 60.00th=[ 894], | 70.00th=[ 953], 80.00th=[ 1028], 90.00th=[ 1167], 95.00th=[ 1301], | 99.00th=[ 1502], 99.50th=[ 1620], 99.90th=[ 1972], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 112, per=24.58%, avg=50.39, stdev=21.36, samples=201 iops : min= 1, max= 28, avg=12.15, stdev= 5.36, samples=201 lat (msec) : 50=0.16%, 100=0.12%, 250=2.42%, 500=42.03%, 750=22.11% lat (msec) : 1000=21.41%, 2000=11.72%, >=2000=0.04% cpu : usr=0.08%, sys=0.59%, ctx=4970, majf=0, minf=18 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=198KiB/s (202kB/s), 49.1KiB/s-49.9KiB/s (50.3kB/s-51.1kB/s), io=19.6MiB (20.5MB), run=101185-101491msec WRITE: bw=206KiB/s (211kB/s), 51.1KiB/s-51.9KiB/s (52.3kB/s-53.2kB/s), io=20.4MiB (21.4MB), run=101185-101491msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145617: Fri Sep 4 19:37:49 2026 read: IOPS=1, BW=69.8MiB/s (73.2MB/s)(40.0MiB/573msec) slat (nsec): min=58177k, max=58177k, avg=58177130.00, stdev= 0.00 clat (nsec): min=500188k, max=500188k, avg=500187891.00, stdev= 0.00 lat (nsec): min=558369k, max=558369k, avg=558368718.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 502], 10.00th=[ 502], 20.00th=[ 502], | 30.00th=[ 502], 40.00th=[ 502], 50.00th=[ 502], 60.00th=[ 502], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=0.70%, ctx=43, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=69.8MiB/s (73.2MB/s), 69.8MiB/s-69.8MiB/s (73.2MB/s-73.2MB/s), io=40.0MiB (41.9MB), run=573-573msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout