== sanity test 398c: run fio to test AIO ================= 13:22:46 (1785950566) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.428074 s, 98.0 MB/s osc.lustre-OST0000-osc-ffff9aa690f9b800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144749: Wed Aug 5 13:27:53 2026 write: IOPS=8, BW=35.7KiB/s (36.6kB/s)(10.0MiB/286479msec); 0 zone resets slat (usec): min=44, max=47762, avg=258.65, stdev=1471.12 clat (msec): min=344, max=3771, avg=1788.74, stdev=645.79 lat (msec): min=344, max=3771, avg=1789.00, stdev=645.87 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 718], 10.00th=[ 869], 20.00th=[ 1183], | 30.00th=[ 1401], 40.00th=[ 1653], 50.00th=[ 1854], 60.00th=[ 2005], | 70.00th=[ 2165], 80.00th=[ 2366], 90.00th=[ 2601], 95.00th=[ 2802], | 99.00th=[ 3171], 99.50th=[ 3406], 99.90th=[ 3742], 99.95th=[ 3775], | 99.99th=[ 3775] bw ( KiB/s): min= 7, max= 128, per=28.34%, avg=40.24, stdev=29.53, samples=498 iops : min= 1, max= 32, avg= 9.61, stdev= 7.42, samples=498 lat (msec) : 500=2.42%, 750=3.55%, 1000=7.11%, 2000=46.64%, >=2000=40.27% cpu : usr=0.02%, sys=0.14%, ctx=1436, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144750: Wed Aug 5 13:27:53 2026 write: IOPS=8, BW=35.8KiB/s (36.6kB/s)(10.0MiB/286326msec); 0 zone resets slat (usec): min=41, max=12295, avg=204.34, stdev=680.17 clat (msec): min=340, max=3803, avg=1787.83, stdev=659.14 lat (msec): min=341, max=3803, avg=1788.03, stdev=659.16 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 684], 10.00th=[ 844], 20.00th=[ 1167], | 30.00th=[ 1435], 40.00th=[ 1670], 50.00th=[ 1838], 60.00th=[ 1989], | 70.00th=[ 2165], 80.00th=[ 2366], 90.00th=[ 2635], 95.00th=[ 2802], | 99.00th=[ 3306], 99.50th=[ 3574], 99.90th=[ 3775], 99.95th=[ 3809], | 99.99th=[ 3809] bw ( KiB/s): min= 7, max= 128, per=27.61%, avg=39.20, stdev=27.03, samples=511 iops : min= 1, max= 32, avg= 9.35, stdev= 6.79, samples=511 lat (msec) : 500=2.30%, 750=4.18%, 1000=7.30%, 2000=47.62%, >=2000=38.59% cpu : usr=0.02%, sys=0.14%, ctx=1683, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144751: Wed Aug 5 13:27:53 2026 write: IOPS=8, BW=35.9KiB/s (36.7kB/s)(10.0MiB/285426msec); 0 zone resets slat (usec): min=43, max=31637, avg=243.58, stdev=1180.61 clat (msec): min=109, max=4595, avg=1782.53, stdev=676.15 lat (msec): min=109, max=4595, avg=1782.78, stdev=676.13 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 642], 10.00th=[ 844], 20.00th=[ 1183], | 30.00th=[ 1435], 40.00th=[ 1636], 50.00th=[ 1821], 60.00th=[ 1989], | 70.00th=[ 2198], 80.00th=[ 2366], 90.00th=[ 2601], 95.00th=[ 2836], | 99.00th=[ 3339], 99.50th=[ 3641], 99.90th=[ 4597], 99.95th=[ 4597], | 99.99th=[ 4597] bw ( KiB/s): min= 7, max= 175, per=31.05%, avg=44.09, stdev=29.25, samples=455 iops : min= 1, max= 43, avg=10.57, stdev= 7.37, samples=455 lat (msec) : 250=0.08%, 500=2.19%, 750=5.59%, 1000=7.81%, 2000=44.88% lat (msec) : >=2000=39.45% cpu : usr=0.03%, sys=0.13%, ctx=1588, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144752: Wed Aug 5 13:27:53 2026 write: IOPS=8, BW=35.8KiB/s (36.6kB/s)(10.0MiB/286148msec); 0 zone resets slat (usec): min=50, max=15190, avg=235.12, stdev=934.38 clat (msec): min=322, max=4203, avg=1786.85, stdev=667.54 lat (msec): min=322, max=4203, avg=1787.09, stdev=667.56 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 676], 10.00th=[ 852], 20.00th=[ 1183], | 30.00th=[ 1418], 40.00th=[ 1636], 50.00th=[ 1854], 60.00th=[ 2022], | 70.00th=[ 2165], 80.00th=[ 2333], 90.00th=[ 2601], 95.00th=[ 2769], | 99.00th=[ 3306], 99.50th=[ 3540], 99.90th=[ 3842], 99.95th=[ 3943], | 99.99th=[ 4212] bw ( KiB/s): min= 7, max= 128, per=28.57%, avg=40.57, stdev=28.06, samples=494 iops : min= 1, max= 32, avg= 9.70, stdev= 7.05, samples=494 lat (msec) : 500=2.42%, 750=5.23%, 1000=7.66%, 2000=43.79%, >=2000=40.90% cpu : usr=0.02%, sys=0.14%, ctx=1669, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=143KiB/s (146kB/s), 35.7KiB/s-35.9KiB/s (36.6kB/s-36.7kB/s), io=40.0MiB (41.9MB), run=285426-286479msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144804: Wed Aug 5 13:30:57 2026 read: IOPS=7, BW=30.4KiB/s (31.2kB/s)(5048KiB/165872msec) slat (usec): min=2352, max=86197, avg=18377.38, stdev=11494.76 clat (msec): min=27, max=1915, avg=646.20, stdev=235.67 lat (msec): min=43, max=1939, avg=664.58, stdev=238.06 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 309], 10.00th=[ 355], 20.00th=[ 422], | 30.00th=[ 502], 40.00th=[ 575], 50.00th=[ 642], 60.00th=[ 709], | 70.00th=[ 768], 80.00th=[ 844], 90.00th=[ 927], 95.00th=[ 1011], | 99.00th=[ 1217], 99.50th=[ 1284], 99.90th=[ 1888], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 124, per=26.27%, avg=31.53, stdev=17.75, samples=312 iops : min= 1, max= 31, avg= 7.44, stdev= 4.45, samples=312 write: IOPS=7, BW=31.3KiB/s (32.1kB/s)(5192KiB/165872msec); 0 zone resets slat (usec): min=49, max=11782, avg=257.63, stdev=792.83 clat (msec): min=355, max=2910, avg=1392.92, stdev=496.80 lat (msec): min=359, max=2910, avg=1393.18, stdev=496.77 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 667], 10.00th=[ 785], 20.00th=[ 936], | 30.00th=[ 1083], 40.00th=[ 1217], 50.00th=[ 1351], 60.00th=[ 1485], | 70.00th=[ 1620], 80.00th=[ 1838], 90.00th=[ 2106], 95.00th=[ 2299], | 99.00th=[ 2567], 99.50th=[ 2635], 99.90th=[ 2802], 99.95th=[ 2903], | 99.99th=[ 2903] bw ( KiB/s): min= 7, max= 95, per=25.37%, avg=31.72, stdev=19.21, samples=319 iops : min= 1, max= 23, avg= 7.48, stdev= 4.84, samples=319 lat (msec) : 50=0.12%, 100=0.08%, 250=0.43%, 500=14.49%, 750=22.03% lat (msec) : 1000=22.07%, 2000=34.41%, >=2000=6.37% cpu : usr=0.06%, sys=0.41%, ctx=4026, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144805: Wed Aug 5 13:30:57 2026 read: IOPS=7, BW=29.0KiB/s (30.7kB/s)(4976KiB/166011msec) slat (usec): min=2878, max=84473, avg=18790.60, stdev=12051.19 clat (msec): min=24, max=2222, avg=640.79, stdev=236.24 lat (msec): min=47, max=2240, avg=659.59, stdev=238.03 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 300], 10.00th=[ 347], 20.00th=[ 414], | 30.00th=[ 498], 40.00th=[ 575], 50.00th=[ 651], 60.00th=[ 709], | 70.00th=[ 768], 80.00th=[ 827], 90.00th=[ 911], 95.00th=[ 978], | 99.00th=[ 1217], 99.50th=[ 1620], 99.90th=[ 2039], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 95, per=26.32%, avg=31.58, stdev=18.24, samples=307 iops : min= 1, max= 23, avg= 7.45, stdev= 4.57, samples=307 write: IOPS=7, BW=31.7KiB/s (32.5kB/s)(5264KiB/166011msec); 0 zone resets slat (usec): min=51, max=23593, avg=263.24, stdev=979.85 clat (msec): min=396, max=2921, avg=1390.14, stdev=506.47 lat (msec): min=397, max=2921, avg=1390.40, stdev=506.48 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 667], 10.00th=[ 768], 20.00th=[ 911], | 30.00th=[ 1070], 40.00th=[ 1200], 50.00th=[ 1351], 60.00th=[ 1502], | 70.00th=[ 1653], 80.00th=[ 1821], 90.00th=[ 2089], 95.00th=[ 2333], | 99.00th=[ 2635], 99.50th=[ 2668], 99.90th=[ 2903], 99.95th=[ 2937], | 99.99th=[ 2937] bw ( KiB/s): min= 7, max= 101, per=26.33%, avg=32.91, stdev=19.07, samples=312 iops : min= 1, max= 25, avg= 7.79, stdev= 4.81, samples=312 lat (msec) : 50=0.12%, 100=0.16%, 250=0.55%, 500=14.73%, 750=21.95% lat (msec) : 1000=21.84%, 2000=34.06%, >=2000=6.60% cpu : usr=0.06%, sys=0.41%, ctx=4363, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144806: Wed Aug 5 13:30:57 2026 read: IOPS=7, BW=30.0KiB/s (30.7kB/s)(4984KiB/166013msec) slat (usec): min=49, max=98070, avg=18538.36, stdev=12142.70 clat (msec): min=41, max=2031, avg=628.93, stdev=229.15 lat (msec): min=56, max=2076, avg=647.47, stdev=232.01 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 300], 10.00th=[ 351], 20.00th=[ 405], | 30.00th=[ 468], 40.00th=[ 567], 50.00th=[ 617], 60.00th=[ 701], | 70.00th=[ 751], 80.00th=[ 827], 90.00th=[ 911], 95.00th=[ 995], | 99.00th=[ 1167], 99.50th=[ 1200], 99.90th=[ 1821], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 111, per=25.49%, avg=30.59, stdev=19.93, samples=318 iops : min= 1, max= 27, avg= 7.21, stdev= 4.98, samples=318 write: IOPS=7, BW=31.7KiB/s (32.4kB/s)(5256KiB/166013msec); 0 zone resets slat (usec): min=50, max=14298, avg=242.10, stdev=730.43 clat (msec): min=335, max=2804, avg=1405.81, stdev=510.90 lat (msec): min=336, max=2804, avg=1406.06, stdev=510.89 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 659], 10.00th=[ 760], 20.00th=[ 936], | 30.00th=[ 1083], 40.00th=[ 1234], 50.00th=[ 1351], 60.00th=[ 1536], | 70.00th=[ 1670], 80.00th=[ 1854], 90.00th=[ 2123], 95.00th=[ 2333], | 99.00th=[ 2635], 99.50th=[ 2668], 99.90th=[ 2735], 99.95th=[ 2802], | 99.99th=[ 2802] bw ( KiB/s): min= 7, max= 96, per=26.30%, avg=32.87, stdev=17.73, samples=310 iops : min= 1, max= 24, avg= 7.77, stdev= 4.44, samples=310 lat (msec) : 50=0.04%, 100=0.20%, 250=0.86%, 500=15.90%, 750=21.60% lat (msec) : 1000=20.08%, 2000=34.10%, >=2000=7.23% cpu : usr=0.07%, sys=0.39%, ctx=3948, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144807: Wed Aug 5 13:30:57 2026 read: IOPS=7, BW=30.5KiB/s (31.2kB/s)(5060KiB/166013msec) slat (usec): min=2485, max=76683, avg=18350.88, stdev=11583.32 clat (msec): min=9, max=1653, avg=634.65, stdev=223.92 lat (msec): min=19, max=1669, avg=653.00, stdev=226.11 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 309], 10.00th=[ 355], 20.00th=[ 414], | 30.00th=[ 485], 40.00th=[ 558], 50.00th=[ 625], 60.00th=[ 709], | 70.00th=[ 760], 80.00th=[ 827], 90.00th=[ 919], 95.00th=[ 995], | 99.00th=[ 1183], 99.50th=[ 1217], 99.90th=[ 1620], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 7, max= 88, per=27.00%, avg=32.40, stdev=19.03, samples=305 iops : min= 1, max= 22, avg= 7.66, stdev= 4.76, samples=305 write: IOPS=7, BW=31.2KiB/s (31.0kB/s)(5180KiB/166013msec); 0 zone resets slat (usec): min=53, max=27434, avg=323.31, stdev=1348.28 clat (msec): min=330, max=2895, avg=1409.00, stdev=508.26 lat (msec): min=336, max=2895, avg=1409.33, stdev=508.31 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 651], 10.00th=[ 751], 20.00th=[ 936], | 30.00th=[ 1099], 40.00th=[ 1250], 50.00th=[ 1368], 60.00th=[ 1536], | 70.00th=[ 1687], 80.00th=[ 1854], 90.00th=[ 2123], 95.00th=[ 2299], | 99.00th=[ 2567], 99.50th=[ 2635], 99.90th=[ 2802], 99.95th=[ 2903], | 99.99th=[ 2903] bw ( KiB/s): min= 7, max= 88, per=25.49%, avg=31.86, stdev=17.86, samples=316 iops : min= 1, max= 22, avg= 7.52, stdev= 4.51, samples=316 lat (msec) : 10=0.04%, 20=0.04%, 50=0.04%, 100=0.16%, 250=0.39% lat (msec) : 500=15.62%, 750=22.27%, 1000=20.31%, 2000=34.30%, >=2000=6.84% cpu : usr=0.07%, sys=0.39%, ctx=3881, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=121KiB/s (124kB/s), 29.0KiB/s-30.5KiB/s (30.7kB/s-31.2kB/s), io=19.6MiB (20.5MB), run=165872-166013msec WRITE: bw=126KiB/s (129kB/s), 31.2KiB/s-31.7KiB/s (31.0kB/s-32.5kB/s), io=20.4MiB (21.4MB), run=165872-166013msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144836: Wed Aug 5 13:31:11 2026 read: IOPS=2, BW=81.8MiB/s (85.8MB/s)(40.0MiB/489msec) slat (nsec): min=46950k, max=46950k, avg=46950048.00, stdev= 0.00 clat (nsec): min=419632k, max=419632k, avg=419631588.00, stdev= 0.00 lat (nsec): min=466585k, max=466585k, avg=466585075.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 422], 10.00th=[ 422], 20.00th=[ 422], | 30.00th=[ 422], 40.00th=[ 422], 50.00th=[ 422], 60.00th=[ 422], | 70.00th=[ 422], 80.00th=[ 422], 90.00th=[ 422], 95.00th=[ 422], | 99.00th=[ 422], 99.50th=[ 422], 99.90th=[ 422], 99.95th=[ 422], | 99.99th=[ 422] lat (msec) : 500=100.00% cpu : usr=0.82%, sys=4.71%, ctx=44, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=81.8MiB/s (85.8MB/s), 81.8MiB/s-81.8MiB/s (85.8MB/s-85.8MB/s), io=40.0MiB (41.9MB), run=489-489msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout