== sanity test 398c: run fio to test AIO ================= 18:18:05 (1787869085) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.595482 s, 70.4 MB/s osc.lustre-OST0000-osc-ffff8c954a42d000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145784: Thu Aug 27 18:21:06 2026 write: IOPS=14, BW=59.8KiB/s (61.2kB/s)(10.0MiB/171209msec); 0 zone resets slat (usec): min=30, max=20423, avg=307.25, stdev=1144.48 clat (msec): min=297, max=2083, avg=1068.66, stdev=277.41 lat (msec): min=298, max=2083, avg=1068.97, stdev=277.42 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 634], 10.00th=[ 735], 20.00th=[ 844], | 30.00th=[ 944], 40.00th=[ 1011], 50.00th=[ 1070], 60.00th=[ 1133], | 70.00th=[ 1200], 80.00th=[ 1284], 90.00th=[ 1401], 95.00th=[ 1519], | 99.00th=[ 1821], 99.50th=[ 1938], 99.90th=[ 2056], 99.95th=[ 2072], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 128, per=24.95%, avg=59.62, stdev=26.50, samples=337 iops : min= 1, max= 32, avg=14.39, stdev= 6.66, samples=337 lat (msec) : 500=2.50%, 750=8.48%, 1000=27.23%, 2000=61.60%, >=2000=0.20% cpu : usr=0.06%, sys=0.26%, ctx=1550, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145785: Thu Aug 27 18:21:06 2026 write: IOPS=14, BW=59.0KiB/s (61.4kB/s)(10.0MiB/170773msec); 0 zone resets slat (usec): min=30, max=14073, avg=290.31, stdev=912.15 clat (msec): min=113, max=2096, avg=1066.52, stdev=276.92 lat (msec): min=113, max=2096, avg=1066.82, stdev=276.92 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 617], 10.00th=[ 743], 20.00th=[ 860], | 30.00th=[ 936], 40.00th=[ 1003], 50.00th=[ 1070], 60.00th=[ 1133], | 70.00th=[ 1200], 80.00th=[ 1284], 90.00th=[ 1401], 95.00th=[ 1502], | 99.00th=[ 1871], 99.50th=[ 1989], 99.90th=[ 2072], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 167, per=25.17%, avg=60.16, stdev=27.69, samples=334 iops : min= 1, max= 41, avg=14.52, stdev= 6.97, samples=334 lat (msec) : 250=0.43%, 500=2.27%, 750=8.01%, 1000=29.61%, 2000=59.30% lat (msec) : >=2000=0.39% cpu : usr=0.05%, sys=0.24%, ctx=1800, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145786: Thu Aug 27 18:21:06 2026 write: IOPS=14, BW=59.8KiB/s (61.2kB/s)(10.0MiB/171253msec); 0 zone resets slat (usec): min=29, max=21127, avg=296.07, stdev=1069.87 clat (msec): min=314, max=2082, avg=1068.99, stdev=273.73 lat (msec): min=314, max=2082, avg=1069.29, stdev=273.76 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 592], 10.00th=[ 743], 20.00th=[ 860], | 30.00th=[ 936], 40.00th=[ 995], 50.00th=[ 1062], 60.00th=[ 1133], | 70.00th=[ 1200], 80.00th=[ 1284], 90.00th=[ 1385], 95.00th=[ 1519], | 99.00th=[ 1821], 99.50th=[ 1989], 99.90th=[ 2039], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 128, per=24.95%, avg=59.63, stdev=27.02, samples=337 iops : min= 1, max= 32, avg=14.39, stdev= 6.79, samples=337 lat (msec) : 500=2.46%, 750=7.97%, 1000=29.88%, 2000=59.34%, >=2000=0.35% cpu : usr=0.05%, sys=0.25%, ctx=1640, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145787: Thu Aug 27 18:21:06 2026 write: IOPS=14, BW=59.8KiB/s (61.2kB/s)(10.0MiB/171378msec); 0 zone resets slat (usec): min=28, max=12437, avg=287.95, stdev=879.20 clat (msec): min=290, max=2084, avg=1069.78, stdev=273.55 lat (msec): min=290, max=2084, avg=1070.07, stdev=273.58 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 642], 10.00th=[ 751], 20.00th=[ 844], | 30.00th=[ 936], 40.00th=[ 1003], 50.00th=[ 1062], 60.00th=[ 1133], | 70.00th=[ 1200], 80.00th=[ 1284], 90.00th=[ 1401], 95.00th=[ 1519], | 99.00th=[ 1838], 99.50th=[ 1871], 99.90th=[ 2056], 99.95th=[ 2072], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 128, per=24.71%, avg=59.05, stdev=24.36, samples=340 iops : min= 1, max= 32, avg=14.25, stdev= 6.13, samples=340 lat (msec) : 500=1.72%, 750=8.40%, 1000=29.22%, 2000=60.55%, >=2000=0.12% cpu : usr=0.04%, sys=0.25%, ctx=1845, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=239KiB/s (245kB/s), 59.8KiB/s-59.0KiB/s (61.2kB/s-61.4kB/s), io=40.0MiB (41.9MB), run=170773-171378msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145821: Thu Aug 27 18:23:26 2026 read: IOPS=9, BW=39.0KiB/s (39.0kB/s)(5048KiB/129361msec) slat (usec): min=3332, max=60586, avg=16768.94, stdev=8940.60 clat (msec): min=33, max=1667, avg=505.91, stdev=141.57 lat (msec): min=47, max=1715, avg=522.69, stdev=143.43 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 313], 10.00th=[ 347], 20.00th=[ 401], | 30.00th=[ 435], 40.00th=[ 468], 50.00th=[ 498], 60.00th=[ 531], | 70.00th=[ 567], 80.00th=[ 609], 90.00th=[ 667], 95.00th=[ 726], | 99.00th=[ 877], 99.50th=[ 978], 99.90th=[ 1452], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 7, max= 103, per=25.27%, avg=38.92, stdev=18.93, samples=253 iops : min= 1, max= 25, avg= 9.15, stdev= 4.76, samples=253 write: IOPS=10, BW=40.1KiB/s (41.1kB/s)(5192KiB/129361msec); 0 zone resets slat (usec): min=33, max=22070, avg=349.27, stdev=1048.28 clat (msec): min=439, max=2241, avg=1081.41, stdev=296.81 lat (msec): min=439, max=2241, avg=1081.77, stdev=296.81 clat percentiles (msec): | 1.00th=[ 514], 5.00th=[ 659], 10.00th=[ 718], 20.00th=[ 827], | 30.00th=[ 911], 40.00th=[ 995], 50.00th=[ 1062], 60.00th=[ 1133], | 70.00th=[ 1200], 80.00th=[ 1301], 90.00th=[ 1469], 95.00th=[ 1603], | 99.00th=[ 1938], 99.50th=[ 2022], 99.90th=[ 2165], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 95, per=25.12%, avg=40.44, stdev=19.91, samples=250 iops : min= 1, max= 23, avg= 9.54, stdev= 4.99, samples=250 lat (msec) : 50=0.08%, 100=0.23%, 250=0.70%, 500=24.45%, 750=28.40% lat (msec) : 1000=15.94%, 2000=29.84%, >=2000=0.35% cpu : usr=0.11%, sys=0.65%, ctx=4145, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145822: Thu Aug 27 18:23:26 2026 read: IOPS=9, BW=38.4KiB/s (39.3kB/s)(4976KiB/129653msec) slat (usec): min=3324, max=78684, avg=16960.10, stdev=9144.76 clat (msec): min=10, max=1102, avg=508.55, stdev=131.88 lat (msec): min=17, max=1117, avg=525.52, stdev=133.38 clat percentiles (msec): | 1.00th=[ 251], 5.00th=[ 321], 10.00th=[ 355], 20.00th=[ 401], | 30.00th=[ 435], 40.00th=[ 464], 50.00th=[ 498], 60.00th=[ 531], | 70.00th=[ 567], 80.00th=[ 617], 90.00th=[ 676], 95.00th=[ 743], | 99.00th=[ 869], 99.50th=[ 961], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 7, max= 80, per=24.89%, avg=38.32, stdev=18.37, samples=253 iops : min= 1, max= 20, avg= 9.00, stdev= 4.65, samples=253 write: IOPS=10, BW=40.6KiB/s (41.6kB/s)(5264KiB/129653msec); 0 zone resets slat (usec): min=35, max=26118, avg=339.48, stdev=1105.21 clat (msec): min=333, max=2079, avg=1076.21, stdev=296.96 lat (msec): min=333, max=2079, avg=1076.55, stdev=296.92 clat percentiles (msec): | 1.00th=[ 514], 5.00th=[ 625], 10.00th=[ 718], 20.00th=[ 810], | 30.00th=[ 919], 40.00th=[ 995], 50.00th=[ 1053], 60.00th=[ 1116], | 70.00th=[ 1200], 80.00th=[ 1318], 90.00th=[ 1469], 95.00th=[ 1603], | 99.00th=[ 1905], 99.50th=[ 1972], 99.90th=[ 2056], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 119, per=25.39%, avg=40.88, stdev=21.23, samples=251 iops : min= 1, max= 29, avg= 9.65, stdev= 5.31, samples=251 lat (msec) : 20=0.04%, 50=0.04%, 100=0.04%, 250=0.43%, 500=24.53% lat (msec) : 750=28.16%, 1000=16.21%, 2000=30.39%, >=2000=0.16% cpu : usr=0.10%, sys=0.63%, ctx=4208, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145823: Thu Aug 27 18:23:26 2026 read: IOPS=9, BW=38.4KiB/s (39.4kB/s)(4984KiB/129654msec) slat (usec): min=3851, max=63154, avg=16784.88, stdev=8836.71 clat (msec): min=34, max=1299, avg=500.89, stdev=129.84 lat (msec): min=42, max=1313, avg=517.68, stdev=131.68 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 317], 10.00th=[ 351], 20.00th=[ 397], | 30.00th=[ 426], 40.00th=[ 464], 50.00th=[ 493], 60.00th=[ 523], | 70.00th=[ 558], 80.00th=[ 609], 90.00th=[ 659], 95.00th=[ 726], | 99.00th=[ 844], 99.50th=[ 927], 99.90th=[ 1116], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 7, max= 95, per=25.13%, avg=38.69, stdev=17.97, samples=252 iops : min= 1, max= 23, avg= 9.10, stdev= 4.50, samples=252 write: IOPS=10, BW=40.5KiB/s (41.5kB/s)(5256KiB/129654msec); 0 zone resets slat (usec): min=34, max=7052, avg=288.81, stdev=582.90 clat (msec): min=439, max=2132, avg=1083.43, stdev=293.06 lat (msec): min=440, max=2133, avg=1083.72, stdev=293.06 clat percentiles (msec): | 1.00th=[ 531], 5.00th=[ 642], 10.00th=[ 718], 20.00th=[ 827], | 30.00th=[ 919], 40.00th=[ 995], 50.00th=[ 1062], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1318], 90.00th=[ 1452], 95.00th=[ 1603], | 99.00th=[ 1921], 99.50th=[ 2022], 99.90th=[ 2106], 99.95th=[ 2140], | 99.99th=[ 2140] bw ( KiB/s): min= 7, max= 103, per=24.85%, avg=40.02, stdev=19.53, samples=255 iops : min= 1, max= 25, avg= 9.43, stdev= 4.92, samples=255 lat (msec) : 50=0.08%, 100=0.08%, 250=0.74%, 500=24.69%, 750=27.46% lat (msec) : 1000=16.29%, 2000=30.39%, >=2000=0.27% cpu : usr=0.09%, sys=0.62%, ctx=4183, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145824: Thu Aug 27 18:23:26 2026 read: IOPS=9, BW=39.3KiB/s (40.3kB/s)(5060KiB/128694msec) slat (usec): min=50, max=57479, avg=16586.83, stdev=8936.24 clat (msec): min=19, max=1460, avg=505.05, stdev=138.32 lat (msec): min=34, max=1479, avg=521.64, stdev=139.91 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 313], 10.00th=[ 351], 20.00th=[ 401], | 30.00th=[ 435], 40.00th=[ 464], 50.00th=[ 493], 60.00th=[ 527], | 70.00th=[ 567], 80.00th=[ 609], 90.00th=[ 667], 95.00th=[ 726], | 99.00th=[ 927], 99.50th=[ 969], 99.90th=[ 1368], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 7, max= 104, per=25.63%, avg=39.48, stdev=17.63, samples=250 iops : min= 1, max= 26, avg= 9.30, stdev= 4.44, samples=250 write: IOPS=10, BW=40.2KiB/s (41.2kB/s)(5180KiB/128694msec); 0 zone resets slat (usec): min=34, max=18424, avg=362.37, stdev=1064.27 clat (msec): min=205, max=2228, avg=1078.73, stdev=302.82 lat (msec): min=205, max=2228, avg=1079.10, stdev=302.84 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 634], 10.00th=[ 726], 20.00th=[ 818], | 30.00th=[ 919], 40.00th=[ 995], 50.00th=[ 1062], 60.00th=[ 1116], | 70.00th=[ 1217], 80.00th=[ 1301], 90.00th=[ 1452], 95.00th=[ 1603], | 99.00th=[ 1972], 99.50th=[ 2072], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 103, per=25.12%, avg=40.44, stdev=17.99, samples=249 iops : min= 1, max= 25, avg= 9.53, stdev= 4.53, samples=249 lat (msec) : 20=0.04%, 50=0.12%, 100=0.16%, 250=0.55%, 500=25.12% lat (msec) : 750=28.05%, 1000=15.94%, 2000=29.65%, >=2000=0.39% cpu : usr=0.10%, sys=0.63%, ctx=4354, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=155KiB/s (158kB/s), 38.4KiB/s-39.3KiB/s (39.3kB/s-40.3kB/s), io=19.6MiB (20.5MB), run=128694-129654msec WRITE: bw=161KiB/s (165kB/s), 40.1KiB/s-40.6KiB/s (41.1kB/s-41.6kB/s), io=20.4MiB (21.4MB), run=128694-129654msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145848: Thu Aug 27 18:23:37 2026 read: IOPS=1, BW=62.9MiB/s (65.9MB/s)(40.0MiB/636msec) slat (nsec): min=31653k, max=31653k, avg=31653058.00, stdev= 0.00 clat (nsec): min=579391k, max=579391k, avg=579390744.00, stdev= 0.00 lat (nsec): min=611048k, max=611048k, avg=611047819.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 584], 5.00th=[ 584], 10.00th=[ 584], 20.00th=[ 584], | 30.00th=[ 584], 40.00th=[ 584], 50.00th=[ 584], 60.00th=[ 584], | 70.00th=[ 584], 80.00th=[ 584], 90.00th=[ 584], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.20%, ctx=11, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=62.9MiB/s (65.9MB/s), 62.9MiB/s-62.9MiB/s (65.9MB/s-65.9MB/s), io=40.0MiB (41.9MB), run=636-636msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout