== sanity test 398c: run fio to test AIO ================= 14:49:05 (1788202145) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.621698 s, 67.5 MB/s osc.lustre-OST0000-osc-ffff967b4388f800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145696: Mon Aug 31 14:51:53 2026 write: IOPS=16, BW=65.3KiB/s (66.9kB/s)(10.0MiB/156801msec); 0 zone resets slat (usec): min=28, max=14860, avg=285.63, stdev=935.54 clat (msec): min=197, max=2338, avg=978.71, stdev=291.58 lat (msec): min=202, max=2338, avg=978.99, stdev=291.58 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 592], 10.00th=[ 684], 20.00th=[ 776], | 30.00th=[ 835], 40.00th=[ 885], 50.00th=[ 953], 60.00th=[ 1011], | 70.00th=[ 1070], 80.00th=[ 1183], 90.00th=[ 1301], 95.00th=[ 1502], | 99.00th=[ 1972], 99.50th=[ 2123], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 255, per=24.88%, avg=64.95, stdev=27.87, samples=310 iops : min= 1, max= 63, avg=15.73, stdev= 7.01, samples=310 lat (msec) : 250=0.59%, 500=2.97%, 750=13.05%, 1000=41.68%, 2000=40.78% lat (msec) : >=2000=0.94% cpu : usr=0.05%, sys=0.25%, ctx=1647, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145697: Mon Aug 31 14:51:53 2026 write: IOPS=16, BW=65.3KiB/s (66.9kB/s)(10.0MiB/156852msec); 0 zone resets slat (usec): min=27, max=15937, avg=278.58, stdev=992.48 clat (msec): min=198, max=2312, avg=979.08, stdev=291.36 lat (msec): min=198, max=2313, avg=979.36, stdev=291.37 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 584], 10.00th=[ 684], 20.00th=[ 768], | 30.00th=[ 827], 40.00th=[ 894], 50.00th=[ 953], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1334], 95.00th=[ 1502], | 99.00th=[ 1888], 99.50th=[ 2072], 99.90th=[ 2299], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 255, per=24.80%, avg=64.73, stdev=29.65, samples=311 iops : min= 1, max= 63, avg=15.67, stdev= 7.45, samples=311 lat (msec) : 250=0.20%, 500=3.28%, 750=14.61%, 1000=40.51%, 2000=40.74% lat (msec) : >=2000=0.66% cpu : usr=0.06%, sys=0.28%, ctx=2003, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145698: Mon Aug 31 14:51:53 2026 write: IOPS=16, BW=65.4KiB/s (66.9kB/s)(10.0MiB/156650msec); 0 zone resets slat (usec): min=27, max=12502, avg=233.08, stdev=711.05 clat (msec): min=214, max=2367, avg=978.39, stdev=299.50 lat (msec): min=214, max=2367, avg=978.63, stdev=299.53 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 567], 10.00th=[ 667], 20.00th=[ 768], | 30.00th=[ 835], 40.00th=[ 894], 50.00th=[ 953], 60.00th=[ 1020], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1334], 95.00th=[ 1485], | 99.00th=[ 2005], 99.50th=[ 2123], 99.90th=[ 2299], 99.95th=[ 2333], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 255, per=25.45%, avg=66.43, stdev=34.52, samples=303 iops : min= 1, max= 63, avg=16.10, stdev= 8.67, samples=303 lat (msec) : 250=1.25%, 500=2.73%, 750=13.63%, 1000=39.26%, 2000=41.91% lat (msec) : >=2000=1.21% cpu : usr=0.04%, sys=0.26%, ctx=1825, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145699: Mon Aug 31 14:51:53 2026 write: IOPS=16, BW=65.5KiB/s (67.1kB/s)(10.0MiB/156344msec); 0 zone resets slat (usec): min=27, max=14774, avg=251.26, stdev=805.40 clat (msec): min=140, max=2334, avg=975.87, stdev=296.19 lat (msec): min=140, max=2334, avg=976.13, stdev=296.18 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 592], 10.00th=[ 676], 20.00th=[ 768], | 30.00th=[ 827], 40.00th=[ 877], 50.00th=[ 944], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1318], 95.00th=[ 1485], | 99.00th=[ 1955], 99.50th=[ 2123], 99.90th=[ 2299], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 255, per=24.88%, avg=64.94, stdev=29.19, samples=310 iops : min= 1, max= 63, avg=15.73, stdev= 7.32, samples=310 lat (msec) : 250=1.21%, 500=2.19%, 750=14.49%, 1000=40.59%, 2000=40.70% lat (msec) : >=2000=0.82% cpu : usr=0.05%, sys=0.25%, ctx=1855, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=261KiB/s (267kB/s), 65.3KiB/s-65.5KiB/s (66.9kB/s-67.1kB/s), io=40.0MiB (41.9MB), run=156344-156852msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145733: Mon Aug 31 14:53:50 2026 read: IOPS=11, BW=46.7KiB/s (47.8kB/s)(5048KiB/108186msec) slat (usec): min=2974, max=62584, avg=14083.57, stdev=8040.56 clat (msec): min=14, max=926, avg=414.90, stdev=109.25 lat (msec): min=19, max=941, avg=428.99, stdev=110.37 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 264], 10.00th=[ 296], 20.00th=[ 330], | 30.00th=[ 355], 40.00th=[ 380], 50.00th=[ 401], 60.00th=[ 435], | 70.00th=[ 460], 80.00th=[ 502], 90.00th=[ 567], 95.00th=[ 609], | 99.00th=[ 701], 99.50th=[ 726], 99.90th=[ 827], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 7, max= 104, per=24.99%, avg=46.23, stdev=20.11, samples=214 iops : min= 1, max= 26, avg=11.07, stdev= 5.08, samples=214 write: IOPS=11, BW=47.0KiB/s (49.1kB/s)(5192KiB/108186msec); 0 zone resets slat (usec): min=32, max=22119, avg=319.48, stdev=1046.03 clat (msec): min=363, max=1969, avg=913.06, stdev=243.00 lat (msec): min=364, max=1970, avg=913.38, stdev=242.97 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 592], 10.00th=[ 642], 20.00th=[ 718], | 30.00th=[ 776], 40.00th=[ 835], 50.00th=[ 885], 60.00th=[ 927], | 70.00th=[ 986], 80.00th=[ 1099], 90.00th=[ 1217], 95.00th=[ 1351], | 99.00th=[ 1720], 99.50th=[ 1787], 99.90th=[ 1921], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 103, per=24.71%, avg=47.45, stdev=21.58, samples=214 iops : min= 1, max= 25, avg=11.37, stdev= 5.43, samples=214 lat (msec) : 20=0.08%, 50=0.04%, 100=0.31%, 250=1.45%, 500=38.24% lat (msec) : 750=21.80%, 1000=23.98%, 2000=14.10% cpu : usr=0.10%, sys=0.66%, ctx=4503, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145734: Mon Aug 31 14:53:50 2026 read: IOPS=11, BW=45.9KiB/s (47.0kB/s)(4976KiB/108380msec) slat (usec): min=2545, max=60538, avg=13732.25, stdev=7771.55 clat (msec): min=36, max=930, avg=418.17, stdev=108.18 lat (msec): min=44, max=952, avg=431.91, stdev=109.16 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 268], 10.00th=[ 292], 20.00th=[ 330], | 30.00th=[ 355], 40.00th=[ 380], 50.00th=[ 405], 60.00th=[ 430], | 70.00th=[ 464], 80.00th=[ 502], 90.00th=[ 567], 95.00th=[ 617], | 99.00th=[ 709], 99.50th=[ 760], 99.90th=[ 919], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 7, max= 104, per=24.55%, avg=45.41, stdev=21.06, samples=214 iops : min= 1, max= 26, avg=10.86, stdev= 5.32, samples=214 write: IOPS=12, BW=48.6KiB/s (49.7kB/s)(5264KiB/108380msec); 0 zone resets slat (usec): min=30, max=15021, avg=339.89, stdev=974.75 clat (msec): min=329, max=2039, avg=907.17, stdev=251.60 lat (msec): min=330, max=2039, avg=907.51, stdev=251.63 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 709], | 30.00th=[ 785], 40.00th=[ 835], 50.00th=[ 869], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1099], 90.00th=[ 1217], 95.00th=[ 1334], | 99.00th=[ 1770], 99.50th=[ 1888], 99.90th=[ 2039], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 95, per=25.01%, avg=48.01, stdev=18.31, samples=215 iops : min= 1, max= 23, avg=11.51, stdev= 4.59, samples=215 lat (msec) : 50=0.08%, 100=0.12%, 250=1.21%, 500=38.75%, 750=21.02% lat (msec) : 1000=23.79%, 2000=14.96%, >=2000=0.08% cpu : usr=0.11%, sys=0.63%, ctx=4362, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145735: Mon Aug 31 14:53:50 2026 read: IOPS=11, BW=46.0KiB/s (47.1kB/s)(4984KiB/108328msec) slat (usec): min=2960, max=60652, avg=14039.00, stdev=7748.52 clat (msec): min=23, max=935, avg=415.12, stdev=110.72 lat (msec): min=36, max=956, avg=429.17, stdev=111.77 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 271], 10.00th=[ 292], 20.00th=[ 330], | 30.00th=[ 351], 40.00th=[ 376], 50.00th=[ 401], 60.00th=[ 430], | 70.00th=[ 460], 80.00th=[ 493], 90.00th=[ 567], 95.00th=[ 617], | 99.00th=[ 726], 99.50th=[ 785], 99.90th=[ 927], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 7, max= 111, per=25.18%, avg=46.59, stdev=20.15, samples=210 iops : min= 1, max= 27, avg=11.15, stdev= 5.08, samples=210 write: IOPS=12, BW=48.5KiB/s (49.7kB/s)(5256KiB/108328msec); 0 zone resets slat (usec): min=30, max=22196, avg=317.26, stdev=1023.35 clat (msec): min=356, max=2051, avg=909.46, stdev=239.29 lat (msec): min=356, max=2051, avg=909.78, stdev=239.33 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 567], 10.00th=[ 625], 20.00th=[ 718], | 30.00th=[ 785], 40.00th=[ 835], 50.00th=[ 885], 60.00th=[ 936], | 70.00th=[ 986], 80.00th=[ 1083], 90.00th=[ 1217], 95.00th=[ 1318], | 99.00th=[ 1653], 99.50th=[ 1888], 99.90th=[ 1972], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 96, per=24.87%, avg=47.74, stdev=19.12, samples=215 iops : min= 1, max= 24, avg=11.45, stdev= 4.81, samples=215 lat (msec) : 50=0.12%, 100=0.20%, 250=0.98%, 500=39.10%, 750=20.16% lat (msec) : 1000=24.96%, 2000=14.45%, >=2000=0.04% cpu : usr=0.11%, sys=0.67%, ctx=4678, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145736: Mon Aug 31 14:53:50 2026 read: IOPS=11, BW=46.0KiB/s (48.1kB/s)(5060KiB/107662msec) slat (usec): min=48, max=73883, avg=13809.43, stdev=8087.53 clat (msec): min=6, max=1029, avg=413.02, stdev=117.59 lat (msec): min=6, max=1035, avg=426.84, stdev=118.70 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 257], 10.00th=[ 296], 20.00th=[ 321], | 30.00th=[ 347], 40.00th=[ 376], 50.00th=[ 401], 60.00th=[ 430], | 70.00th=[ 460], 80.00th=[ 502], 90.00th=[ 567], 95.00th=[ 609], | 99.00th=[ 709], 99.50th=[ 776], 99.90th=[ 1011], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 7, max= 96, per=25.55%, avg=47.26, stdev=19.54, samples=210 iops : min= 1, max= 24, avg=11.31, stdev= 4.94, samples=210 write: IOPS=12, BW=48.1KiB/s (49.3kB/s)(5180KiB/107662msec); 0 zone resets slat (usec): min=29, max=17653, avg=308.42, stdev=887.81 clat (msec): min=132, max=1965, avg=911.70, stdev=246.08 lat (msec): min=132, max=1965, avg=912.02, stdev=246.13 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 567], 10.00th=[ 634], 20.00th=[ 718], | 30.00th=[ 785], 40.00th=[ 835], 50.00th=[ 885], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1099], 90.00th=[ 1217], 95.00th=[ 1334], | 99.00th=[ 1737], 99.50th=[ 1838], 99.90th=[ 1955], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 103, per=24.95%, avg=47.91, stdev=19.94, samples=211 iops : min= 1, max= 25, avg=11.48, stdev= 5.00, samples=211 lat (msec) : 10=0.08%, 20=0.04%, 50=0.16%, 100=0.43%, 250=1.56% lat (msec) : 500=37.97%, 750=20.62%, 1000=23.98%, 2000=15.16% cpu : usr=0.11%, sys=0.71%, ctx=4469, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=185KiB/s (190kB/s), 45.9KiB/s-46.0KiB/s (47.0kB/s-48.1kB/s), io=19.6MiB (20.5MB), run=107662-108380msec WRITE: bw=193KiB/s (197kB/s), 47.0KiB/s-48.6KiB/s (49.1kB/s-49.7kB/s), io=20.4MiB (21.4MB), run=107662-108380msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145755: Mon Aug 31 14:53:58 2026 read: IOPS=1, BW=74.2MiB/s (77.8MB/s)(40.0MiB/539msec) slat (nsec): min=20654k, max=20654k, avg=20654345.00, stdev= 0.00 clat (nsec): min=504871k, max=504871k, avg=504870830.00, stdev= 0.00 lat (nsec): min=525529k, max=525529k, avg=525529145.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 506], 10.00th=[ 506], 20.00th=[ 506], | 30.00th=[ 506], 40.00th=[ 506], 50.00th=[ 506], 60.00th=[ 506], | 70.00th=[ 506], 80.00th=[ 506], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.23%, ctx=4, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=74.2MiB/s (77.8MB/s), 74.2MiB/s-74.2MiB/s (77.8MB/s-77.8MB/s), io=40.0MiB (41.9MB), run=539-539msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout