== sanity test 398c: run fio to test AIO ================= 09:17:41 (1787577461) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.204358 s, 205 MB/s osc.lustre-OST0000-osc-ffff903059970000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144498: Mon Aug 24 09:20:31 2026 write: IOPS=15, BW=63.8KiB/s (65.4kB/s)(10.0MiB/160437msec); 0 zone resets slat (usec): min=20, max=16916, avg=163.57, stdev=659.89 clat (msec): min=260, max=2298, avg=1001.53, stdev=306.23 lat (msec): min=260, max=2298, avg=1001.69, stdev=306.25 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 584], 10.00th=[ 667], 20.00th=[ 768], | 30.00th=[ 844], 40.00th=[ 902], 50.00th=[ 953], 60.00th=[ 1020], | 70.00th=[ 1099], 80.00th=[ 1217], 90.00th=[ 1435], 95.00th=[ 1586], | 99.00th=[ 1821], 99.50th=[ 2089], 99.90th=[ 2299], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 256, per=25.72%, avg=65.59, stdev=32.13, samples=307 iops : min= 1, max= 64, avg=15.93, stdev= 8.08, samples=307 lat (msec) : 500=3.79%, 750=13.32%, 1000=39.45%, 2000=42.77%, >=2000=0.66% cpu : usr=0.03%, sys=0.22%, ctx=1594, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144499: Mon Aug 24 09:20:31 2026 write: IOPS=15, BW=63.9KiB/s (65.5kB/s)(10.0MiB/160201msec); 0 zone resets slat (usec): min=20, max=17721, avg=170.64, stdev=696.68 clat (msec): min=164, max=2296, avg=1000.68, stdev=311.22 lat (msec): min=164, max=2296, avg=1000.85, stdev=311.20 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 567], 10.00th=[ 684], 20.00th=[ 768], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1020], | 70.00th=[ 1099], 80.00th=[ 1250], 90.00th=[ 1435], 95.00th=[ 1569], | 99.00th=[ 1821], 99.50th=[ 1955], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 207, per=25.43%, avg=64.85, stdev=29.90, samples=310 iops : min= 1, max= 51, avg=15.75, stdev= 7.50, samples=310 lat (msec) : 250=0.62%, 500=3.71%, 750=13.87%, 1000=38.01%, 2000=43.44% lat (msec) : >=2000=0.35% cpu : usr=0.04%, sys=0.21%, ctx=1660, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144500: Mon Aug 24 09:20:31 2026 write: IOPS=15, BW=63.9KiB/s (65.5kB/s)(10.0MiB/160206msec); 0 zone resets slat (usec): min=22, max=18214, avg=185.09, stdev=818.01 clat (msec): min=217, max=2295, avg=1000.15, stdev=309.46 lat (msec): min=217, max=2295, avg=1000.34, stdev=309.45 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 575], 10.00th=[ 676], 20.00th=[ 768], | 30.00th=[ 844], 40.00th=[ 894], 50.00th=[ 953], 60.00th=[ 1028], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1418], 95.00th=[ 1569], | 99.00th=[ 1838], 99.50th=[ 1955], 99.90th=[ 2299], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 256, per=25.22%, avg=64.32, stdev=31.05, samples=313 iops : min= 1, max= 64, avg=15.62, stdev= 7.81, samples=313 lat (msec) : 250=0.62%, 500=3.75%, 750=12.73%, 1000=38.98%, 2000=43.48% lat (msec) : >=2000=0.43% cpu : usr=0.05%, sys=0.21%, ctx=1720, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144501: Mon Aug 24 09:20:31 2026 write: IOPS=15, BW=63.8KiB/s (65.4kB/s)(10.0MiB/160438msec); 0 zone resets slat (usec): min=22, max=17100, avg=194.78, stdev=777.37 clat (msec): min=257, max=2295, avg=1001.60, stdev=305.90 lat (msec): min=257, max=2295, avg=1001.80, stdev=305.87 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 584], 10.00th=[ 667], 20.00th=[ 760], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1028], | 70.00th=[ 1099], 80.00th=[ 1234], 90.00th=[ 1452], 95.00th=[ 1569], | 99.00th=[ 1787], 99.50th=[ 1871], 99.90th=[ 2165], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 240, per=24.82%, avg=63.30, stdev=28.24, samples=318 iops : min= 1, max= 60, avg=15.36, stdev= 7.12, samples=318 lat (msec) : 500=3.71%, 750=14.77%, 1000=36.99%, 2000=44.18%, >=2000=0.35% cpu : usr=0.04%, sys=0.23%, ctx=1772, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=255KiB/s (261kB/s), 63.8KiB/s-63.9KiB/s (65.4kB/s-65.5kB/s), io=40.0MiB (41.9MB), run=160201-160438msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144534: Mon Aug 24 09:23:06 2026 read: IOPS=9, BW=36.0KiB/s (36.9kB/s)(5048KiB/140216msec) slat (usec): min=37, max=85730, avg=17924.15, stdev=11594.33 clat (msec): min=54, max=1680, avg=556.71, stdev=205.63 lat (msec): min=54, max=1693, avg=574.65, stdev=208.58 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 279], 10.00th=[ 338], 20.00th=[ 384], | 30.00th=[ 426], 40.00th=[ 472], 50.00th=[ 523], 60.00th=[ 575], | 70.00th=[ 651], 80.00th=[ 718], 90.00th=[ 827], 95.00th=[ 953], | 99.00th=[ 1150], 99.50th=[ 1167], 99.90th=[ 1267], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 98, per=26.05%, avg=36.99, stdev=19.85, samples=266 iops : min= 1, max= 24, avg= 8.70, stdev= 4.99, samples=266 write: IOPS=9, BW=37.0KiB/s (37.9kB/s)(5192KiB/140216msec); 0 zone resets slat (usec): min=26, max=14358, avg=217.66, stdev=694.58 clat (msec): min=277, max=2920, avg=1168.43, stdev=407.28 lat (msec): min=277, max=2920, avg=1168.65, stdev=407.27 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 600], 10.00th=[ 709], 20.00th=[ 818], | 30.00th=[ 902], 40.00th=[ 1003], 50.00th=[ 1099], 60.00th=[ 1234], | 70.00th=[ 1351], 80.00th=[ 1519], 90.00th=[ 1737], 95.00th=[ 1871], | 99.00th=[ 2400], 99.50th=[ 2635], 99.90th=[ 2769], 99.95th=[ 2937], | 99.99th=[ 2937] bw ( KiB/s): min= 7, max= 102, per=25.17%, avg=37.25, stdev=19.36, samples=271 iops : min= 1, max= 25, avg= 8.77, stdev= 4.86, samples=271 lat (msec) : 100=0.47%, 250=0.70%, 500=22.07%, 750=24.45%, 1000=19.88% lat (msec) : 2000=30.86%, >=2000=1.56% cpu : usr=0.07%, sys=0.45%, ctx=4112, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144535: Mon Aug 24 09:23:06 2026 read: IOPS=8, BW=35.3KiB/s (36.2kB/s)(4976KiB/140847msec) slat (usec): min=3077, max=92509, avg=17536.93, stdev=11231.35 clat (msec): min=13, max=1671, avg=543.42, stdev=200.04 lat (msec): min=27, max=1685, avg=560.96, stdev=203.01 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 292], 10.00th=[ 334], 20.00th=[ 380], | 30.00th=[ 422], 40.00th=[ 468], 50.00th=[ 510], 60.00th=[ 558], | 70.00th=[ 625], 80.00th=[ 701], 90.00th=[ 802], 95.00th=[ 927], | 99.00th=[ 1116], 99.50th=[ 1183], 99.90th=[ 1368], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 7, max= 120, per=25.95%, avg=36.84, stdev=20.85, samples=263 iops : min= 1, max= 30, avg= 8.66, stdev= 5.24, samples=263 write: IOPS=9, BW=37.4KiB/s (38.3kB/s)(5264KiB/140847msec); 0 zone resets slat (usec): min=31, max=24411, avg=276.58, stdev=1375.35 clat (msec): min=63, max=3137, avg=1178.60, stdev=414.71 lat (msec): min=64, max=3137, avg=1178.88, stdev=414.83 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 609], 10.00th=[ 726], 20.00th=[ 818], | 30.00th=[ 919], 40.00th=[ 1020], 50.00th=[ 1099], 60.00th=[ 1217], | 70.00th=[ 1368], 80.00th=[ 1519], 90.00th=[ 1754], 95.00th=[ 1905], | 99.00th=[ 2299], 99.50th=[ 2668], 99.90th=[ 3071], 99.95th=[ 3138], | 99.99th=[ 3138] bw ( KiB/s): min= 7, max= 104, per=25.41%, avg=37.61, stdev=20.31, samples=273 iops : min= 1, max= 26, avg= 8.86, stdev= 5.11, samples=273 lat (msec) : 20=0.08%, 50=0.04%, 100=0.31%, 250=1.21%, 500=22.62% lat (msec) : 750=23.59%, 1000=18.98%, 2000=31.33%, >=2000=1.84% cpu : usr=0.07%, sys=0.46%, ctx=4130, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144536: Mon Aug 24 09:23:06 2026 read: IOPS=8, BW=35.4KiB/s (36.3kB/s)(4984KiB/140652msec) slat (usec): min=3119, max=80965, avg=17155.53, stdev=10587.09 clat (msec): min=12, max=1676, avg=540.48, stdev=194.68 lat (msec): min=16, max=1696, avg=557.64, stdev=197.25 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 296], 10.00th=[ 334], 20.00th=[ 376], | 30.00th=[ 422], 40.00th=[ 468], 50.00th=[ 510], 60.00th=[ 558], | 70.00th=[ 609], 80.00th=[ 693], 90.00th=[ 785], 95.00th=[ 894], | 99.00th=[ 1150], 99.50th=[ 1167], 99.90th=[ 1284], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 7, max= 126, per=26.24%, avg=37.26, stdev=21.32, samples=262 iops : min= 1, max= 31, avg= 8.77, stdev= 5.37, samples=262 write: IOPS=9, BW=37.4KiB/s (38.3kB/s)(5256KiB/140652msec); 0 zone resets slat (usec): min=25, max=15416, avg=198.20, stdev=662.71 clat (msec): min=405, max=3139, avg=1179.87, stdev=421.86 lat (msec): min=405, max=3140, avg=1180.07, stdev=421.86 clat percentiles (msec): | 1.00th=[ 498], 5.00th=[ 634], 10.00th=[ 718], 20.00th=[ 810], | 30.00th=[ 902], 40.00th=[ 1011], 50.00th=[ 1099], 60.00th=[ 1250], | 70.00th=[ 1368], 80.00th=[ 1502], 90.00th=[ 1754], 95.00th=[ 1871], | 99.00th=[ 2500], 99.50th=[ 2836], 99.90th=[ 3138], 99.95th=[ 3138], | 99.99th=[ 3138] bw ( KiB/s): min= 7, max= 102, per=25.62%, avg=37.91, stdev=19.96, samples=269 iops : min= 1, max= 25, avg= 8.94, stdev= 5.03, samples=269 lat (msec) : 20=0.12%, 50=0.04%, 100=0.27%, 250=0.82%, 500=22.46% lat (msec) : 750=24.61%, 1000=19.06%, 2000=30.98%, >=2000=1.64% cpu : usr=0.10%, sys=0.47%, ctx=4072, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144537: Mon Aug 24 09:23:06 2026 read: IOPS=8, BW=35.0KiB/s (36.8kB/s)(5060KiB/140653msec) slat (msec): min=2, max=102, avg=17.89, stdev=11.51 clat (msec): min=22, max=1673, avg=555.86, stdev=198.57 lat (msec): min=32, max=1685, avg=573.75, stdev=201.42 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 305], 10.00th=[ 347], 20.00th=[ 393], | 30.00th=[ 439], 40.00th=[ 477], 50.00th=[ 514], 60.00th=[ 575], | 70.00th=[ 634], 80.00th=[ 709], 90.00th=[ 810], 95.00th=[ 927], | 99.00th=[ 1150], 99.50th=[ 1200], 99.90th=[ 1653], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 7, max= 88, per=26.30%, avg=37.35, stdev=19.12, samples=265 iops : min= 1, max= 22, avg= 8.80, stdev= 4.82, samples=265 write: IOPS=9, BW=36.8KiB/s (37.7kB/s)(5180KiB/140653msec); 0 zone resets slat (usec): min=24, max=20232, avg=236.67, stdev=930.24 clat (msec): min=392, max=3086, avg=1172.78, stdev=404.48 lat (msec): min=392, max=3086, avg=1173.02, stdev=404.58 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 642], 10.00th=[ 735], 20.00th=[ 827], | 30.00th=[ 894], 40.00th=[ 1003], 50.00th=[ 1116], 60.00th=[ 1234], | 70.00th=[ 1351], 80.00th=[ 1485], 90.00th=[ 1737], 95.00th=[ 1888], | 99.00th=[ 2333], 99.50th=[ 2735], 99.90th=[ 3071], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min= 7, max= 103, per=24.90%, avg=36.85, stdev=19.39, samples=273 iops : min= 1, max= 25, avg= 8.67, stdev= 4.86, samples=273 lat (msec) : 50=0.08%, 100=0.23%, 250=0.35%, 500=22.77%, 750=23.91% lat (msec) : 1000=20.43%, 2000=30.70%, >=2000=1.52% cpu : usr=0.09%, sys=0.49%, ctx=4297, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=142KiB/s (146kB/s), 35.3KiB/s-36.0KiB/s (36.2kB/s-36.9kB/s), io=19.6MiB (20.5MB), run=140216-140847msec WRITE: bw=148KiB/s (152kB/s), 36.8KiB/s-37.4KiB/s (37.7kB/s-38.3kB/s), io=20.4MiB (21.4MB), run=140216-140847msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144562: Mon Aug 24 09:23:17 2026 read: IOPS=1, BW=65.9MiB/s (69.1MB/s)(40.0MiB/607msec) slat (nsec): min=35856k, max=35856k, avg=35856418.00, stdev= 0.00 clat (nsec): min=527753k, max=527753k, avg=527752627.00, stdev= 0.00 lat (nsec): min=563615k, max=563615k, avg=563615302.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 527], 5.00th=[ 527], 10.00th=[ 527], 20.00th=[ 527], | 30.00th=[ 527], 40.00th=[ 527], 50.00th=[ 527], 60.00th=[ 527], | 70.00th=[ 527], 80.00th=[ 527], 90.00th=[ 527], 95.00th=[ 527], | 99.00th=[ 527], 99.50th=[ 527], 99.90th=[ 527], 99.95th=[ 527], | 99.99th=[ 527] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.31%, ctx=5, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=65.9MiB/s (69.1MB/s), 65.9MiB/s-65.9MiB/s (69.1MB/s-69.1MB/s), io=40.0MiB (41.9MB), run=607-607msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout