== sanity test 398c: run fio to test AIO ================= 15:00:00 (1777575600) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.542251 s, 77.3 MB/s osc.lustre-OST0000-osc-ffff9e4c065d1000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=159228: Thu Apr 30 15:03:20 2026 write: IOPS=13, BW=54.4KiB/s (55.7kB/s)(10.0MiB/188399msec); 0 zone resets slat (usec): min=44, max=19303, avg=328.95, stdev=1006.63 clat (msec): min=78, max=2075, avg=1176.31, stdev=310.84 lat (msec): min=78, max=2075, avg=1176.64, stdev=310.87 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 709], 10.00th=[ 860], 20.00th=[ 961], | 30.00th=[ 1028], 40.00th=[ 1083], 50.00th=[ 1150], 60.00th=[ 1217], | 70.00th=[ 1301], 80.00th=[ 1418], 90.00th=[ 1569], 95.00th=[ 1754], | 99.00th=[ 1989], 99.50th=[ 2039], 99.90th=[ 2072], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 192, per=25.52%, avg=55.37, stdev=29.20, samples=363 iops : min= 1, max= 48, avg=13.39, stdev= 7.33, samples=363 lat (msec) : 100=0.04%, 250=0.39%, 500=2.58%, 750=2.97%, 1000=19.30% lat (msec) : 2000=73.87%, >=2000=0.86% cpu : usr=0.04%, sys=0.25%, ctx=1936, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=159229: Thu Apr 30 15:03:20 2026 write: IOPS=13, BW=54.4KiB/s (55.7kB/s)(10.0MiB/188392msec); 0 zone resets slat (usec): min=39, max=21867, avg=327.10, stdev=1181.49 clat (msec): min=234, max=2138, avg=1176.28, stdev=306.70 lat (msec): min=234, max=2138, avg=1176.61, stdev=306.76 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 718], 10.00th=[ 869], 20.00th=[ 961], | 30.00th=[ 1028], 40.00th=[ 1083], 50.00th=[ 1150], 60.00th=[ 1234], | 70.00th=[ 1318], 80.00th=[ 1401], 90.00th=[ 1569], 95.00th=[ 1720], | 99.00th=[ 1955], 99.50th=[ 2039], 99.90th=[ 2106], 99.95th=[ 2123], | 99.99th=[ 2140] bw ( KiB/s): min= 7, max= 200, per=24.96%, avg=54.16, stdev=28.85, samples=371 iops : min= 1, max= 50, avg=13.08, stdev= 7.28, samples=371 lat (msec) : 250=0.04%, 500=3.40%, 750=2.34%, 1000=19.45%, 2000=74.22% lat (msec) : >=2000=0.55% cpu : usr=0.03%, sys=0.27%, ctx=1979, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=159230: Thu Apr 30 15:03:20 2026 write: IOPS=13, BW=54.5KiB/s (55.8kB/s)(10.0MiB/187836msec); 0 zone resets slat (usec): min=50, max=21766, avg=333.47, stdev=1136.47 clat (msec): min=173, max=2281, avg=1173.06, stdev=318.98 lat (msec): min=173, max=2281, avg=1173.40, stdev=319.02 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 701], 10.00th=[ 869], 20.00th=[ 953], | 30.00th=[ 1020], 40.00th=[ 1083], 50.00th=[ 1150], 60.00th=[ 1234], | 70.00th=[ 1318], 80.00th=[ 1418], 90.00th=[ 1569], 95.00th=[ 1737], | 99.00th=[ 2022], 99.50th=[ 2039], 99.90th=[ 2165], 99.95th=[ 2198], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 220, per=24.94%, avg=54.12, stdev=27.08, samples=371 iops : min= 1, max= 55, avg=13.07, stdev= 6.83, samples=371 lat (msec) : 250=1.25%, 500=2.11%, 750=2.81%, 1000=20.12%, 2000=72.62% lat (msec) : >=2000=1.09% cpu : usr=0.03%, sys=0.26%, ctx=2129, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=159231: Thu Apr 30 15:03:20 2026 write: IOPS=13, BW=54.4KiB/s (55.7kB/s)(10.0MiB/188394msec); 0 zone resets slat (usec): min=50, max=27748, avg=314.53, stdev=1128.27 clat (msec): min=259, max=2152, avg=1176.38, stdev=304.94 lat (msec): min=259, max=2152, avg=1176.70, stdev=304.97 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 709], 10.00th=[ 852], 20.00th=[ 961], | 30.00th=[ 1028], 40.00th=[ 1099], 50.00th=[ 1167], 60.00th=[ 1234], | 70.00th=[ 1301], 80.00th=[ 1401], 90.00th=[ 1552], 95.00th=[ 1703], | 99.00th=[ 1955], 99.50th=[ 2039], 99.90th=[ 2140], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 192, per=24.87%, avg=53.96, stdev=24.72, samples=372 iops : min= 1, max= 48, avg=13.03, stdev= 6.25, samples=372 lat (msec) : 500=3.36%, 750=2.46%, 1000=19.34%, 2000=74.22%, >=2000=0.62% cpu : usr=0.04%, sys=0.26%, ctx=2221, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=217KiB/s (223kB/s), 54.4KiB/s-54.5KiB/s (55.7kB/s-55.8kB/s), io=40.0MiB (41.9MB), run=187836-188399msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=159282: Thu Apr 30 15:05:34 2026 read: IOPS=10, BW=41.6KiB/s (42.6kB/s)(5048KiB/121473msec) slat (usec): min=3325, max=68700, avg=15545.42, stdev=8427.37 clat (msec): min=16, max=1243, avg=471.39, stdev=132.35 lat (msec): min=38, max=1254, avg=486.95, stdev=133.97 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 309], 10.00th=[ 334], 20.00th=[ 372], | 30.00th=[ 405], 40.00th=[ 426], 50.00th=[ 456], 60.00th=[ 481], | 70.00th=[ 514], 80.00th=[ 550], 90.00th=[ 625], 95.00th=[ 701], | 99.00th=[ 927], 99.50th=[ 986], 99.90th=[ 1116], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 7, max= 111, per=25.31%, avg=41.50, stdev=21.24, samples=238 iops : min= 1, max= 27, avg= 9.89, stdev= 5.31, samples=238 write: IOPS=10, BW=42.7KiB/s (43.8kB/s)(5192KiB/121473msec); 0 zone resets slat (usec): min=54, max=27291, avg=399.90, stdev=1375.93 clat (msec): min=291, max=2292, avg=1020.04, stdev=329.58 lat (msec): min=291, max=2292, avg=1020.44, stdev=329.68 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 575], 10.00th=[ 651], 20.00th=[ 751], | 30.00th=[ 827], 40.00th=[ 894], 50.00th=[ 961], 60.00th=[ 1036], | 70.00th=[ 1150], 80.00th=[ 1267], 90.00th=[ 1469], 95.00th=[ 1636], | 99.00th=[ 2123], 99.50th=[ 2232], 99.90th=[ 2299], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 96, per=24.62%, avg=42.10, stdev=18.57, samples=241 iops : min= 1, max= 24, avg=10.03, stdev= 4.68, samples=241 lat (msec) : 20=0.04%, 50=0.04%, 100=0.12%, 250=0.35%, 500=32.97% lat (msec) : 750=23.83%, 1000=19.49%, 2000=22.38%, >=2000=0.78% cpu : usr=0.09%, sys=0.57%, ctx=4815, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=159283: Thu Apr 30 15:05:34 2026 read: IOPS=10, BW=40.9KiB/s (41.9kB/s)(4976KiB/121633msec) slat (usec): min=3052, max=50744, avg=15168.51, stdev=7635.61 clat (msec): min=27, max=1219, avg=475.92, stdev=138.77 lat (msec): min=46, max=1229, avg=491.09, stdev=139.92 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 309], 10.00th=[ 334], 20.00th=[ 372], | 30.00th=[ 401], 40.00th=[ 426], 50.00th=[ 456], 60.00th=[ 485], | 70.00th=[ 523], 80.00th=[ 558], 90.00th=[ 634], 95.00th=[ 760], | 99.00th=[ 936], 99.50th=[ 995], 99.90th=[ 1200], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 103, per=24.85%, avg=40.76, stdev=20.72, samples=238 iops : min= 1, max= 25, avg= 9.70, stdev= 5.21, samples=238 write: IOPS=10, BW=43.3KiB/s (44.3kB/s)(5264KiB/121633msec); 0 zone resets slat (usec): min=44, max=15877, avg=368.20, stdev=976.71 clat (msec): min=297, max=2338, avg=1010.09, stdev=336.78 lat (msec): min=297, max=2338, avg=1010.46, stdev=336.80 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 743], | 30.00th=[ 818], 40.00th=[ 877], 50.00th=[ 944], 60.00th=[ 1028], | 70.00th=[ 1150], 80.00th=[ 1267], 90.00th=[ 1469], 95.00th=[ 1620], | 99.00th=[ 2123], 99.50th=[ 2265], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 103, per=25.39%, avg=43.41, stdev=20.02, samples=237 iops : min= 1, max= 25, avg=10.36, stdev= 5.02, samples=237 lat (msec) : 50=0.08%, 100=0.16%, 250=0.51%, 500=31.41%, 750=24.96% lat (msec) : 1000=20.86%, 2000=21.17%, >=2000=0.86% cpu : usr=0.08%, sys=0.64%, ctx=4713, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=159284: Thu Apr 30 15:05:34 2026 read: IOPS=10, BW=40.0KiB/s (41.9kB/s)(4984KiB/121700msec) slat (usec): min=3234, max=58365, avg=15580.08, stdev=8275.87 clat (msec): min=45, max=1279, avg=480.63, stdev=137.58 lat (msec): min=60, max=1292, avg=496.23, stdev=139.04 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 300], 10.00th=[ 330], 20.00th=[ 372], | 30.00th=[ 409], 40.00th=[ 439], 50.00th=[ 464], 60.00th=[ 493], | 70.00th=[ 523], 80.00th=[ 567], 90.00th=[ 651], 95.00th=[ 751], | 99.00th=[ 927], 99.50th=[ 961], 99.90th=[ 995], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 7, max= 103, per=25.25%, avg=41.41, stdev=20.47, samples=236 iops : min= 1, max= 25, avg= 9.86, stdev= 5.13, samples=236 write: IOPS=10, BW=43.2KiB/s (44.2kB/s)(5256KiB/121700msec); 0 zone resets slat (usec): min=43, max=11887, avg=342.06, stdev=814.44 clat (msec): min=94, max=2358, avg=1007.25, stdev=324.61 lat (msec): min=94, max=2363, avg=1007.60, stdev=324.63 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 558], 10.00th=[ 642], 20.00th=[ 751], | 30.00th=[ 827], 40.00th=[ 877], 50.00th=[ 944], 60.00th=[ 1036], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1435], 95.00th=[ 1620], | 99.00th=[ 2106], 99.50th=[ 2198], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 95, per=24.92%, avg=42.62, stdev=20.16, samples=240 iops : min= 1, max= 23, avg=10.17, stdev= 5.06, samples=240 lat (msec) : 50=0.08%, 100=0.16%, 250=0.51%, 500=31.02%, 750=24.96% lat (msec) : 1000=20.66%, 2000=21.84%, >=2000=0.78% cpu : usr=0.08%, sys=0.58%, ctx=4727, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=159285: Thu Apr 30 15:05:34 2026 read: IOPS=10, BW=41.9KiB/s (42.9kB/s)(5060KiB/120880msec) slat (usec): min=68, max=56288, avg=15252.95, stdev=8371.31 clat (msec): min=9, max=1402, avg=474.45, stdev=141.39 lat (msec): min=20, max=1410, avg=489.71, stdev=142.64 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 305], 10.00th=[ 334], 20.00th=[ 372], | 30.00th=[ 401], 40.00th=[ 426], 50.00th=[ 456], 60.00th=[ 485], | 70.00th=[ 518], 80.00th=[ 558], 90.00th=[ 625], 95.00th=[ 726], | 99.00th=[ 944], 99.50th=[ 1062], 99.90th=[ 1351], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 7, max= 104, per=25.55%, avg=41.90, stdev=21.15, samples=236 iops : min= 1, max= 26, avg= 9.99, stdev= 5.32, samples=236 write: IOPS=10, BW=42.9KiB/s (43.9kB/s)(5180KiB/120880msec); 0 zone resets slat (usec): min=58, max=11608, avg=324.68, stdev=779.52 clat (msec): min=229, max=2452, avg=1013.45, stdev=341.13 lat (msec): min=229, max=2452, avg=1013.78, stdev=341.13 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 535], 10.00th=[ 625], 20.00th=[ 743], | 30.00th=[ 827], 40.00th=[ 877], 50.00th=[ 953], 60.00th=[ 1045], | 70.00th=[ 1150], 80.00th=[ 1267], 90.00th=[ 1502], 95.00th=[ 1653], | 99.00th=[ 2140], 99.50th=[ 2232], 99.90th=[ 2400], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 96, per=24.83%, avg=42.46, stdev=20.22, samples=238 iops : min= 1, max= 24, avg=10.13, stdev= 5.04, samples=238 lat (msec) : 10=0.04%, 50=0.04%, 100=0.20%, 250=0.59%, 500=32.50% lat (msec) : 750=24.34%, 1000=19.96%, 2000=21.56%, >=2000=0.78% cpu : usr=0.09%, sys=0.58%, ctx=4634, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=165KiB/s (169kB/s), 40.9KiB/s-41.9KiB/s (41.9kB/s-42.9kB/s), io=19.6MiB (20.5MB), run=120880-121700msec WRITE: bw=172KiB/s (176kB/s), 42.7KiB/s-43.3KiB/s (43.8kB/s-44.3kB/s), io=20.4MiB (21.4MB), run=120880-121700msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=159306: Thu Apr 30 15:05:44 2026 read: IOPS=1, BW=74.8MiB/s (78.4MB/s)(40.0MiB/535msec) slat (nsec): min=28042k, max=28042k, avg=28041735.00, stdev= 0.00 clat (nsec): min=490089k, max=490089k, avg=490089404.00, stdev= 0.00 lat (nsec): min=518136k, max=518136k, avg=518136099.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 489], 5.00th=[ 489], 10.00th=[ 489], 20.00th=[ 489], | 30.00th=[ 489], 40.00th=[ 489], 50.00th=[ 489], 60.00th=[ 489], | 70.00th=[ 489], 80.00th=[ 489], 90.00th=[ 489], 95.00th=[ 489], | 99.00th=[ 489], 99.50th=[ 489], 99.90th=[ 489], 99.95th=[ 489], | 99.99th=[ 489] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.62%, ctx=5, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=74.8MiB/s (78.4MB/s), 74.8MiB/s-74.8MiB/s (78.4MB/s-78.4MB/s), io=40.0MiB (41.9MB), run=535-535msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout