== sanity test 398c: run fio to test AIO ================= 13:25:41 (1788974741) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.591247 s, 70.9 MB/s osc.lustre-OST0000-osc-ffff8ba6d1554800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147830: Wed Sep 9 13:28:29 2026 write: IOPS=16, BW=64.4KiB/s (65.9kB/s)(10.0MiB/158999msec); 0 zone resets slat (usec): min=28, max=18050, avg=219.59, stdev=806.67 clat (msec): min=209, max=2598, avg=992.39, stdev=281.40 lat (msec): min=210, max=2598, avg=992.61, stdev=281.40 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 617], 10.00th=[ 718], 20.00th=[ 802], | 30.00th=[ 852], 40.00th=[ 911], 50.00th=[ 969], 60.00th=[ 1028], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1318], 95.00th=[ 1452], | 99.00th=[ 1921], 99.50th=[ 2232], 99.90th=[ 2567], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 256, per=25.30%, avg=65.01, stdev=30.31, samples=310 iops : min= 1, max= 64, avg=15.79, stdev= 7.59, samples=310 lat (msec) : 250=0.62%, 500=3.20%, 750=9.77%, 1000=41.45%, 2000=44.34% lat (msec) : >=2000=0.62% cpu : usr=0.04%, sys=0.22%, ctx=1740, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147831: Wed Sep 9 13:28:29 2026 write: IOPS=16, BW=64.6KiB/s (66.2kB/s)(10.0MiB/158404msec); 0 zone resets slat (usec): min=26, max=17270, avg=246.18, stdev=949.64 clat (msec): min=68, max=2563, avg=989.32, stdev=281.04 lat (msec): min=68, max=2563, avg=989.57, stdev=281.01 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 584], 10.00th=[ 701], 20.00th=[ 802], | 30.00th=[ 860], 40.00th=[ 919], 50.00th=[ 969], 60.00th=[ 1028], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1318], 95.00th=[ 1452], | 99.00th=[ 1804], 99.50th=[ 2265], 99.90th=[ 2500], 99.95th=[ 2567], | 99.99th=[ 2567] bw ( KiB/s): min= 8, max= 184, per=24.96%, avg=64.15, stdev=28.26, samples=314 iops : min= 2, max= 46, avg=15.57, stdev= 7.08, samples=314 lat (msec) : 100=0.04%, 250=0.47%, 500=3.55%, 750=10.08%, 1000=41.13% lat (msec) : 2000=44.06%, >=2000=0.66% cpu : usr=0.05%, sys=0.20%, ctx=2012, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147832: Wed Sep 9 13:28:29 2026 write: IOPS=16, BW=64.4KiB/s (65.9kB/s)(10.0MiB/159048msec); 0 zone resets slat (usec): min=29, max=14638, avg=201.33, stdev=737.74 clat (msec): min=190, max=2602, avg=992.74, stdev=275.88 lat (msec): min=194, max=2602, avg=992.94, stdev=275.87 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 600], 10.00th=[ 718], 20.00th=[ 802], | 30.00th=[ 860], 40.00th=[ 919], 50.00th=[ 978], 60.00th=[ 1028], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1787], 99.50th=[ 2366], 99.90th=[ 2534], 99.95th=[ 2534], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 256, per=25.94%, avg=66.67, stdev=35.83, samples=302 iops : min= 1, max= 64, avg=16.21, stdev= 9.00, samples=302 lat (msec) : 250=0.04%, 500=4.06%, 750=9.26%, 1000=41.21%, 2000=44.80% lat (msec) : >=2000=0.62% cpu : usr=0.04%, sys=0.19%, ctx=1686, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147833: Wed Sep 9 13:28:29 2026 write: IOPS=16, BW=64.4KiB/s (65.9kB/s)(10.0MiB/159050msec); 0 zone resets slat (usec): min=23, max=15518, avg=218.29, stdev=818.13 clat (msec): min=199, max=2683, avg=992.65, stdev=280.98 lat (msec): min=199, max=2683, avg=992.87, stdev=280.99 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 609], 10.00th=[ 718], 20.00th=[ 802], | 30.00th=[ 860], 40.00th=[ 919], 50.00th=[ 969], 60.00th=[ 1020], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1318], 95.00th=[ 1435], | 99.00th=[ 1921], 99.50th=[ 2165], 99.90th=[ 2635], 99.95th=[ 2668], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 200, per=24.97%, avg=64.17, stdev=27.14, samples=314 iops : min= 1, max= 50, avg=15.58, stdev= 6.80, samples=314 lat (msec) : 250=0.62%, 500=3.12%, 750=9.26%, 1000=42.50%, 2000=43.75% lat (msec) : >=2000=0.74% cpu : usr=0.04%, sys=0.21%, ctx=2053, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=258KiB/s (264kB/s), 64.4KiB/s-64.6KiB/s (65.9kB/s-66.2kB/s), io=40.0MiB (41.9MB), run=158404-159050msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147866: Wed Sep 9 13:30:22 2026 read: IOPS=12, BW=48.8KiB/s (49.0kB/s)(5048KiB/103417msec) slat (usec): min=2240, max=51759, avg=13161.43, stdev=7473.47 clat (msec): min=16, max=799, avg=395.03, stdev=111.60 lat (msec): min=28, max=842, avg=408.20, stdev=112.70 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 239], 10.00th=[ 271], 20.00th=[ 309], | 30.00th=[ 338], 40.00th=[ 363], 50.00th=[ 388], 60.00th=[ 409], | 70.00th=[ 439], 80.00th=[ 477], 90.00th=[ 542], 95.00th=[ 592], | 99.00th=[ 701], 99.50th=[ 751], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 7, max= 111, per=25.20%, avg=48.63, stdev=21.33, samples=204 iops : min= 1, max= 27, avg=11.67, stdev= 5.38, samples=204 write: IOPS=12, BW=50.2KiB/s (51.4kB/s)(5192KiB/103417msec); 0 zone resets slat (usec): min=31, max=15043, avg=260.54, stdev=849.16 clat (msec): min=349, max=1680, avg=874.77, stdev=242.86 lat (msec): min=349, max=1680, avg=875.03, stdev=242.84 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 527], 10.00th=[ 592], 20.00th=[ 659], | 30.00th=[ 718], 40.00th=[ 785], 50.00th=[ 852], 60.00th=[ 919], | 70.00th=[ 995], 80.00th=[ 1083], 90.00th=[ 1217], 95.00th=[ 1318], | 99.00th=[ 1519], 99.50th=[ 1536], 99.90th=[ 1586], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 119, per=24.78%, avg=49.81, stdev=21.18, samples=204 iops : min= 1, max= 29, avg=11.97, stdev= 5.30, samples=204 lat (msec) : 20=0.04%, 50=0.23%, 100=0.43%, 250=2.66%, 500=40.35% lat (msec) : 750=23.05%, 1000=18.40%, 2000=14.84% cpu : usr=0.07%, sys=0.58%, ctx=4390, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147867: Wed Sep 9 13:30:22 2026 read: IOPS=12, BW=48.1KiB/s (49.2kB/s)(4976KiB/103496msec) slat (usec): min=2644, max=94734, avg=13314.10, stdev=7508.80 clat (msec): min=21, max=829, avg=393.46, stdev=111.74 lat (msec): min=25, max=841, avg=406.79, stdev=113.13 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 236], 10.00th=[ 271], 20.00th=[ 313], | 30.00th=[ 338], 40.00th=[ 363], 50.00th=[ 384], 60.00th=[ 405], | 70.00th=[ 435], 80.00th=[ 472], 90.00th=[ 527], 95.00th=[ 592], | 99.00th=[ 709], 99.50th=[ 776], 99.90th=[ 810], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 7, max= 111, per=24.92%, avg=48.09, stdev=21.83, samples=203 iops : min= 1, max= 27, avg=11.54, stdev= 5.48, samples=203 write: IOPS=12, BW=50.9KiB/s (52.1kB/s)(5264KiB/103496msec); 0 zone resets slat (usec): min=32, max=26072, avg=267.87, stdev=1127.86 clat (msec): min=339, max=1676, avg=870.87, stdev=239.53 lat (msec): min=339, max=1685, avg=871.14, stdev=239.57 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 510], 10.00th=[ 584], 20.00th=[ 659], | 30.00th=[ 726], 40.00th=[ 785], 50.00th=[ 852], 60.00th=[ 919], | 70.00th=[ 1003], 80.00th=[ 1070], 90.00th=[ 1183], 95.00th=[ 1301], | 99.00th=[ 1469], 99.50th=[ 1519], 99.90th=[ 1603], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 7, max= 112, per=25.07%, avg=50.39, stdev=20.37, samples=205 iops : min= 1, max= 28, avg=12.11, stdev= 5.12, samples=205 lat (msec) : 50=0.27%, 100=0.35%, 250=2.77%, 500=40.74%, 750=22.50% lat (msec) : 1000=18.05%, 2000=15.31% cpu : usr=0.11%, sys=0.60%, ctx=4112, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147868: Wed Sep 9 13:30:22 2026 read: IOPS=12, BW=48.2KiB/s (49.3kB/s)(4984KiB/103497msec) slat (usec): min=2198, max=93245, avg=13323.12, stdev=7823.79 clat (msec): min=15, max=829, avg=401.25, stdev=113.77 lat (msec): min=23, max=841, avg=414.58, stdev=114.98 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 236], 10.00th=[ 275], 20.00th=[ 317], | 30.00th=[ 347], 40.00th=[ 372], 50.00th=[ 393], 60.00th=[ 418], | 70.00th=[ 443], 80.00th=[ 485], 90.00th=[ 550], 95.00th=[ 592], | 99.00th=[ 743], 99.50th=[ 776], 99.90th=[ 810], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 7, max= 119, per=24.67%, avg=47.61, stdev=21.92, samples=206 iops : min= 1, max= 29, avg=11.41, stdev= 5.51, samples=206 write: IOPS=12, BW=50.8KiB/s (52.0kB/s)(5256KiB/103497msec); 0 zone resets slat (usec): min=30, max=23804, avg=288.73, stdev=1235.34 clat (msec): min=238, max=1709, avg=864.64, stdev=249.00 lat (msec): min=239, max=1709, avg=864.93, stdev=249.00 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 485], 10.00th=[ 575], 20.00th=[ 642], | 30.00th=[ 709], 40.00th=[ 768], 50.00th=[ 852], 60.00th=[ 919], | 70.00th=[ 986], 80.00th=[ 1070], 90.00th=[ 1200], 95.00th=[ 1318], | 99.00th=[ 1452], 99.50th=[ 1519], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 119, per=24.82%, avg=49.89, stdev=21.15, samples=206 iops : min= 1, max= 29, avg=11.99, stdev= 5.30, samples=206 lat (msec) : 20=0.04%, 50=0.12%, 100=0.31%, 250=3.05%, 500=39.84% lat (msec) : 750=23.55%, 1000=18.48%, 2000=14.61% cpu : usr=0.08%, sys=0.55%, ctx=3857, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147869: Wed Sep 9 13:30:22 2026 read: IOPS=12, BW=49.2KiB/s (50.4kB/s)(5060KiB/102883msec) slat (usec): min=51, max=66880, avg=13092.81, stdev=7431.78 clat (msec): min=17, max=808, avg=399.78, stdev=111.62 lat (msec): min=24, max=831, avg=412.88, stdev=113.06 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 239], 10.00th=[ 284], 20.00th=[ 317], | 30.00th=[ 342], 40.00th=[ 368], 50.00th=[ 393], 60.00th=[ 418], | 70.00th=[ 447], 80.00th=[ 477], 90.00th=[ 542], 95.00th=[ 600], | 99.00th=[ 718], 99.50th=[ 743], 99.90th=[ 810], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 7, max= 112, per=25.39%, avg=49.00, stdev=21.63, samples=203 iops : min= 1, max= 28, avg=11.76, stdev= 5.47, samples=203 write: IOPS=12, BW=50.3KiB/s (51.6kB/s)(5180KiB/102883msec); 0 zone resets slat (usec): min=30, max=10025, avg=236.82, stdev=739.64 clat (msec): min=133, max=1687, avg=866.72, stdev=250.24 lat (msec): min=133, max=1687, avg=866.96, stdev=250.27 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 498], 10.00th=[ 567], 20.00th=[ 651], | 30.00th=[ 709], 40.00th=[ 776], 50.00th=[ 852], 60.00th=[ 919], | 70.00th=[ 995], 80.00th=[ 1083], 90.00th=[ 1200], 95.00th=[ 1318], | 99.00th=[ 1469], 99.50th=[ 1536], 99.90th=[ 1653], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 126, per=24.82%, avg=49.90, stdev=22.49, samples=203 iops : min= 1, max= 31, avg=11.99, stdev= 5.63, samples=203 lat (msec) : 20=0.08%, 50=0.16%, 100=0.39%, 250=2.62%, 500=41.05% lat (msec) : 750=22.81%, 1000=18.09%, 2000=14.80% cpu : usr=0.11%, sys=0.57%, ctx=4038, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=194KiB/s (199kB/s), 48.1KiB/s-49.2KiB/s (49.2kB/s-50.4kB/s), io=19.6MiB (20.5MB), run=102883-103497msec WRITE: bw=202KiB/s (207kB/s), 50.2KiB/s-50.9KiB/s (51.4kB/s-52.1kB/s), io=20.4MiB (21.4MB), run=102883-103497msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147888: Wed Sep 9 13:30:31 2026 read: IOPS=2, BW=106MiB/s (112MB/s)(40.0MiB/376msec) slat (nsec): min=29203k, max=29203k, avg=29202640.00, stdev= 0.00 clat (nsec): min=326870k, max=326870k, avg=326870155.00, stdev= 0.00 lat (nsec): min=356077k, max=356077k, avg=356076915.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 326], 10.00th=[ 326], 20.00th=[ 326], | 30.00th=[ 326], 40.00th=[ 326], 50.00th=[ 326], 60.00th=[ 326], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.93%, ctx=42, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=106MiB/s (112MB/s), 106MiB/s-106MiB/s (112MB/s-112MB/s), io=40.0MiB (41.9MB), run=376-376msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout