== sanity test 398c: run fio to test AIO ================= 20:42:24 (1787618544) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.437792 s, 95.8 MB/s osc.lustre-OST0000-osc-ffff913bd262c800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144935: Mon Aug 24 20:45:23 2026 write: IOPS=15, BW=60.1KiB/s (61.6kB/s)(10.0MiB/170347msec); 0 zone resets slat (usec): min=21, max=18538, avg=216.12, stdev=838.82 clat (msec): min=272, max=2452, avg=1063.46, stdev=299.41 lat (msec): min=272, max=2452, avg=1063.68, stdev=299.40 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 676], 10.00th=[ 735], 20.00th=[ 827], | 30.00th=[ 894], 40.00th=[ 961], 50.00th=[ 1020], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1267], 90.00th=[ 1418], 95.00th=[ 1586], | 99.00th=[ 2022], 99.50th=[ 2198], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 128, per=26.27%, avg=63.04, stdev=34.08, samples=319 iops : min= 1, max= 32, avg=15.27, stdev= 8.57, samples=319 lat (msec) : 500=0.62%, 750=10.90%, 1000=34.73%, 2000=52.70%, >=2000=1.05% cpu : usr=0.04%, sys=0.21%, ctx=1531, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144936: Mon Aug 24 20:45:23 2026 write: IOPS=15, BW=60.3KiB/s (61.7kB/s)(10.0MiB/169939msec); 0 zone resets slat (usec): min=21, max=22655, avg=272.32, stdev=1219.82 clat (msec): min=176, max=2446, avg=1060.96, stdev=301.90 lat (msec): min=176, max=2446, avg=1061.23, stdev=301.94 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 676], 10.00th=[ 726], 20.00th=[ 818], | 30.00th=[ 902], 40.00th=[ 961], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1469], 95.00th=[ 1653], | 99.00th=[ 1989], 99.50th=[ 2198], 99.90th=[ 2366], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 128, per=26.44%, avg=63.46, stdev=30.05, samples=317 iops : min= 1, max= 32, avg=15.38, stdev= 7.55, samples=317 lat (msec) : 250=0.62%, 500=0.08%, 750=11.25%, 1000=36.05%, 2000=51.29% lat (msec) : >=2000=0.70% cpu : usr=0.04%, sys=0.21%, ctx=1588, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144937: Mon Aug 24 20:45:23 2026 write: IOPS=15, BW=60.3KiB/s (61.8kB/s)(10.0MiB/169771msec); 0 zone resets slat (usec): min=20, max=21396, avg=222.65, stdev=992.95 clat (msec): min=181, max=2449, avg=1060.39, stdev=296.99 lat (msec): min=181, max=2451, avg=1060.62, stdev=297.00 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 676], 10.00th=[ 726], 20.00th=[ 835], | 30.00th=[ 902], 40.00th=[ 953], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1452], 95.00th=[ 1670], | 99.00th=[ 1938], 99.50th=[ 2072], 99.90th=[ 2333], 99.95th=[ 2366], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 128, per=25.46%, avg=61.10, stdev=30.40, samples=329 iops : min= 1, max= 32, avg=14.78, stdev= 7.65, samples=329 lat (msec) : 250=0.62%, 500=0.66%, 750=10.27%, 1000=36.60%, 2000=51.25% lat (msec) : >=2000=0.59% cpu : usr=0.04%, sys=0.21%, ctx=1559, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144938: Mon Aug 24 20:45:23 2026 write: IOPS=15, BW=60.1KiB/s (61.6kB/s)(10.0MiB/170345msec); 0 zone resets slat (usec): min=20, max=26635, avg=266.06, stdev=1301.69 clat (msec): min=209, max=2445, avg=1063.29, stdev=295.48 lat (msec): min=209, max=2445, avg=1063.56, stdev=295.46 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 684], 10.00th=[ 735], 20.00th=[ 827], | 30.00th=[ 902], 40.00th=[ 961], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1435], 95.00th=[ 1636], | 99.00th=[ 1955], 99.50th=[ 2165], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 160, per=25.24%, avg=60.58, stdev=31.64, samples=332 iops : min= 1, max= 40, avg=14.65, stdev= 7.94, samples=332 lat (msec) : 250=0.04%, 500=1.05%, 750=9.92%, 1000=36.25%, 2000=51.88% lat (msec) : >=2000=0.86% cpu : usr=0.05%, sys=0.21%, ctx=1959, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=240KiB/s (246kB/s), 60.1KiB/s-60.3KiB/s (61.6kB/s-61.8kB/s), io=40.0MiB (41.9MB), run=169771-170347msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144971: Mon Aug 24 20:47:32 2026 read: IOPS=10, BW=41.3KiB/s (42.3kB/s)(5048KiB/122262msec) slat (usec): min=2553, max=88432, avg=15497.49, stdev=9576.09 clat (msec): min=17, max=1149, avg=459.65, stdev=160.98 lat (msec): min=30, max=1173, avg=475.15, stdev=163.35 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 271], 10.00th=[ 292], 20.00th=[ 334], | 30.00th=[ 363], 40.00th=[ 393], 50.00th=[ 422], 60.00th=[ 464], | 70.00th=[ 514], 80.00th=[ 575], 90.00th=[ 676], 95.00th=[ 760], | 99.00th=[ 1011], 99.50th=[ 1099], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 128, per=25.25%, avg=41.41, stdev=22.08, samples=239 iops : min= 1, max= 32, avg= 9.88, stdev= 5.58, samples=239 write: IOPS=10, BW=42.5KiB/s (43.5kB/s)(5192KiB/122262msec); 0 zone resets slat (usec): min=24, max=13367, avg=228.76, stdev=802.37 clat (msec): min=350, max=2292, avg=1042.07, stdev=361.61 lat (msec): min=350, max=2292, avg=1042.31, stdev=361.64 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 567], 10.00th=[ 651], 20.00th=[ 751], | 30.00th=[ 827], 40.00th=[ 885], 50.00th=[ 953], 60.00th=[ 1045], | 70.00th=[ 1150], 80.00th=[ 1318], 90.00th=[ 1569], 95.00th=[ 1787], | 99.00th=[ 2072], 99.50th=[ 2140], 99.90th=[ 2198], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 103, per=25.32%, avg=43.05, stdev=19.58, samples=236 iops : min= 1, max= 25, avg=10.29, stdev= 4.92, samples=236 lat (msec) : 20=0.04%, 50=0.08%, 100=0.04%, 250=1.21%, 500=33.09% lat (msec) : 750=21.99%, 1000=20.39%, 2000=22.19%, >=2000=0.98% cpu : usr=0.09%, sys=0.51%, ctx=4015, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144972: Mon Aug 24 20:47:32 2026 read: IOPS=10, BW=40.7KiB/s (41.7kB/s)(4976KiB/122264msec) slat (usec): min=2697, max=85311, avg=15459.12, stdev=9986.23 clat (msec): min=37, max=1152, avg=463.53, stdev=164.47 lat (msec): min=49, max=1187, avg=478.99, stdev=166.66 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 264], 10.00th=[ 292], 20.00th=[ 330], | 30.00th=[ 363], 40.00th=[ 397], 50.00th=[ 435], 60.00th=[ 477], | 70.00th=[ 527], 80.00th=[ 584], 90.00th=[ 659], 95.00th=[ 768], | 99.00th=[ 1045], 99.50th=[ 1099], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 104, per=24.85%, avg=40.76, stdev=20.77, samples=239 iops : min= 1, max= 26, avg= 9.72, stdev= 5.26, samples=239 write: IOPS=10, BW=43.1KiB/s (44.1kB/s)(5264KiB/122264msec); 0 zone resets slat (usec): min=25, max=17600, avg=218.97, stdev=843.65 clat (msec): min=329, max=2343, avg=1031.09, stdev=362.02 lat (msec): min=329, max=2343, avg=1031.32, stdev=362.04 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 575], 10.00th=[ 642], 20.00th=[ 743], | 30.00th=[ 810], 40.00th=[ 877], 50.00th=[ 944], 60.00th=[ 1036], | 70.00th=[ 1150], 80.00th=[ 1284], 90.00th=[ 1552], 95.00th=[ 1787], | 99.00th=[ 2072], 99.50th=[ 2123], 99.90th=[ 2165], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 136, per=25.15%, avg=42.76, stdev=22.49, samples=241 iops : min= 1, max= 34, avg=10.22, stdev= 5.69, samples=241 lat (msec) : 50=0.08%, 100=0.20%, 250=1.21%, 500=31.72%, 750=23.48% lat (msec) : 1000=19.73%, 2000=22.77%, >=2000=0.82% cpu : usr=0.09%, sys=0.51%, ctx=3993, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144973: Mon Aug 24 20:47:32 2026 read: IOPS=10, BW=40.8KiB/s (41.8kB/s)(4984KiB/122208msec) slat (usec): min=34, max=88720, avg=15174.81, stdev=9757.89 clat (msec): min=6, max=1157, avg=458.36, stdev=162.31 lat (msec): min=38, max=1187, avg=473.55, stdev=165.09 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 262], 10.00th=[ 292], 20.00th=[ 330], | 30.00th=[ 363], 40.00th=[ 393], 50.00th=[ 426], 60.00th=[ 468], | 70.00th=[ 514], 80.00th=[ 575], 90.00th=[ 684], 95.00th=[ 751], | 99.00th=[ 986], 99.50th=[ 1070], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 120, per=24.89%, avg=40.82, stdev=23.16, samples=240 iops : min= 1, max= 30, avg= 9.73, stdev= 5.86, samples=240 write: IOPS=10, BW=43.0KiB/s (44.0kB/s)(5256KiB/122208msec); 0 zone resets slat (usec): min=23, max=18852, avg=245.41, stdev=1030.44 clat (msec): min=167, max=2181, avg=1038.00, stdev=365.65 lat (msec): min=168, max=2181, avg=1038.25, stdev=365.65 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 575], 10.00th=[ 642], 20.00th=[ 751], | 30.00th=[ 818], 40.00th=[ 885], 50.00th=[ 953], 60.00th=[ 1053], | 70.00th=[ 1167], 80.00th=[ 1301], 90.00th=[ 1586], 95.00th=[ 1821], | 99.00th=[ 2056], 99.50th=[ 2140], 99.90th=[ 2165], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 135, per=25.02%, avg=42.53, stdev=20.88, samples=241 iops : min= 1, max= 33, avg=10.16, stdev= 5.29, samples=241 lat (msec) : 10=0.04%, 50=0.08%, 100=0.27%, 250=1.72%, 500=31.99% lat (msec) : 750=22.34%, 1000=19.61%, 2000=23.20%, >=2000=0.74% cpu : usr=0.08%, sys=0.50%, ctx=4055, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144974: Mon Aug 24 20:47:32 2026 read: IOPS=10, BW=41.4KiB/s (42.4kB/s)(5060KiB/122256msec) slat (usec): min=2643, max=74092, avg=15312.93, stdev=9177.34 clat (msec): min=7, max=1148, avg=462.19, stdev=166.36 lat (msec): min=26, max=1188, avg=477.51, stdev=168.74 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 266], 10.00th=[ 292], 20.00th=[ 334], | 30.00th=[ 368], 40.00th=[ 393], 50.00th=[ 426], 60.00th=[ 464], | 70.00th=[ 510], 80.00th=[ 567], 90.00th=[ 684], 95.00th=[ 793], | 99.00th=[ 1053], 99.50th=[ 1116], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 111, per=25.54%, avg=41.88, stdev=21.43, samples=237 iops : min= 1, max= 27, avg=10.00, stdev= 5.43, samples=237 write: IOPS=10, BW=42.4KiB/s (43.4kB/s)(5180KiB/122256msec); 0 zone resets slat (usec): min=26, max=21889, avg=233.37, stdev=947.17 clat (msec): min=346, max=2355, avg=1040.73, stdev=372.16 lat (msec): min=347, max=2355, avg=1040.96, stdev=372.19 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 575], 10.00th=[ 634], 20.00th=[ 743], | 30.00th=[ 818], 40.00th=[ 877], 50.00th=[ 953], 60.00th=[ 1062], | 70.00th=[ 1167], 80.00th=[ 1301], 90.00th=[ 1569], 95.00th=[ 1838], | 99.00th=[ 2106], 99.50th=[ 2198], 99.90th=[ 2299], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 119, per=25.46%, avg=43.29, stdev=20.32, samples=234 iops : min= 1, max= 29, avg=10.35, stdev= 5.16, samples=234 lat (msec) : 10=0.04%, 50=0.04%, 100=0.08%, 250=1.72%, 500=32.34% lat (msec) : 750=22.93%, 1000=18.71%, 2000=22.93%, >=2000=1.21% cpu : usr=0.07%, sys=0.50%, ctx=4044, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=164KiB/s (168kB/s), 40.7KiB/s-41.4KiB/s (41.7kB/s-42.4kB/s), io=19.6MiB (20.5MB), run=122208-122264msec WRITE: bw=171KiB/s (175kB/s), 42.4KiB/s-43.1KiB/s (43.4kB/s-44.1kB/s), io=20.4MiB (21.4MB), run=122208-122264msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144997: Mon Aug 24 20:47:40 2026 read: IOPS=2, BW=100MiB/s (105MB/s)(40.0MiB/399msec) slat (nsec): min=23231k, max=23231k, avg=23230964.00, stdev= 0.00 clat (nsec): min=349204k, max=349204k, avg=349203973.00, stdev= 0.00 lat (nsec): min=372439k, max=372439k, avg=372439353.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 351], 10.00th=[ 351], 20.00th=[ 351], | 30.00th=[ 351], 40.00th=[ 351], 50.00th=[ 351], 60.00th=[ 351], | 70.00th=[ 351], 80.00th=[ 351], 90.00th=[ 351], 95.00th=[ 351], | 99.00th=[ 351], 99.50th=[ 351], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.02%, ctx=42, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=100MiB/s (105MB/s), 100MiB/s-100MiB/s (105MB/s-105MB/s), io=40.0MiB (41.9MB), run=399-399msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout