== sanity test 398c: run fio to test AIO ================= 10:42:23 (1785940943) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.77376 s, 54.2 MB/s osc.lustre-OST0000-osc-ffff9f088eb11800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144490: Wed Aug 5 10:47:21 2026 write: IOPS=9, BW=36.4KiB/s (37.2kB/s)(10.0MiB/281652msec); 0 zone resets slat (usec): min=27, max=40662, avg=211.71, stdev=1474.06 clat (msec): min=379, max=3695, avg=1758.83, stdev=672.27 lat (msec): min=379, max=3695, avg=1759.04, stdev=672.30 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 718], 10.00th=[ 818], 20.00th=[ 1099], | 30.00th=[ 1334], 40.00th=[ 1620], 50.00th=[ 1787], 60.00th=[ 1955], | 70.00th=[ 2140], 80.00th=[ 2366], 90.00th=[ 2668], 95.00th=[ 2836], | 99.00th=[ 3239], 99.50th=[ 3373], 99.90th=[ 3641], 99.95th=[ 3641], | 99.99th=[ 3708] bw ( KiB/s): min= 7, max= 159, per=28.30%, avg=41.03, stdev=28.79, samples=488 iops : min= 1, max= 39, avg= 9.76, stdev= 7.23, samples=488 lat (msec) : 500=1.41%, 750=5.20%, 1000=9.34%, 2000=46.95%, >=2000=37.11% cpu : usr=0.02%, sys=0.11%, ctx=1513, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144491: Wed Aug 5 10:47:21 2026 write: IOPS=9, BW=36.4KiB/s (37.3kB/s)(10.0MiB/281323msec); 0 zone resets slat (usec): min=27, max=27792, avg=197.61, stdev=1195.22 clat (msec): min=271, max=3829, avg=1756.58, stdev=681.12 lat (msec): min=271, max=3829, avg=1756.78, stdev=681.14 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 676], 10.00th=[ 852], 20.00th=[ 1083], | 30.00th=[ 1351], 40.00th=[ 1603], 50.00th=[ 1804], 60.00th=[ 1921], | 70.00th=[ 2165], 80.00th=[ 2333], 90.00th=[ 2668], 95.00th=[ 2903], | 99.00th=[ 3239], 99.50th=[ 3373], 99.90th=[ 3708], 99.95th=[ 3842], | 99.99th=[ 3842] bw ( KiB/s): min= 7, max= 191, per=27.73%, avg=40.21, stdev=27.94, samples=498 iops : min= 1, max= 47, avg= 9.55, stdev= 7.01, samples=498 lat (msec) : 500=1.84%, 750=4.84%, 1000=9.77%, 2000=46.91%, >=2000=36.64% cpu : usr=0.02%, sys=0.10%, ctx=1446, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144492: Wed Aug 5 10:47:21 2026 write: IOPS=9, BW=36.4KiB/s (37.3kB/s)(10.0MiB/281326msec); 0 zone resets slat (usec): min=27, max=41664, avg=205.10, stdev=1335.10 clat (msec): min=226, max=3702, avg=1757.50, stdev=674.45 lat (msec): min=226, max=3702, avg=1757.71, stdev=674.51 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 709], 10.00th=[ 869], 20.00th=[ 1083], | 30.00th=[ 1334], 40.00th=[ 1569], 50.00th=[ 1787], 60.00th=[ 1955], | 70.00th=[ 2106], 80.00th=[ 2366], 90.00th=[ 2668], 95.00th=[ 2903], | 99.00th=[ 3239], 99.50th=[ 3306], 99.90th=[ 3708], 99.95th=[ 3708], | 99.99th=[ 3708] bw ( KiB/s): min= 7, max= 135, per=32.86%, avg=47.65, stdev=33.92, samples=421 iops : min= 1, max= 33, avg=11.42, stdev= 8.51, samples=421 lat (msec) : 250=0.62%, 500=0.86%, 750=4.41%, 1000=9.22%, 2000=49.14% lat (msec) : >=2000=35.74% cpu : usr=0.02%, sys=0.11%, ctx=1324, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144493: Wed Aug 5 10:47:21 2026 write: IOPS=9, BW=36.4KiB/s (37.2kB/s)(10.0MiB/281651msec); 0 zone resets slat (usec): min=27, max=41931, avg=221.41, stdev=1531.03 clat (msec): min=377, max=3813, avg=1758.56, stdev=671.99 lat (msec): min=377, max=3814, avg=1758.79, stdev=671.99 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 709], 10.00th=[ 844], 20.00th=[ 1099], | 30.00th=[ 1368], 40.00th=[ 1586], 50.00th=[ 1770], 60.00th=[ 1938], | 70.00th=[ 2140], 80.00th=[ 2366], 90.00th=[ 2635], 95.00th=[ 2869], | 99.00th=[ 3272], 99.50th=[ 3473], 99.90th=[ 3608], 99.95th=[ 3809], | 99.99th=[ 3809] bw ( KiB/s): min= 7, max= 159, per=27.28%, avg=39.56, stdev=27.69, samples=506 iops : min= 1, max= 39, avg= 9.40, stdev= 6.94, samples=506 lat (msec) : 500=0.90%, 750=6.05%, 1000=9.38%, 2000=47.54%, >=2000=36.13% cpu : usr=0.02%, sys=0.12%, ctx=1526, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=145KiB/s (149kB/s), 36.4KiB/s-36.4KiB/s (37.2kB/s-37.3kB/s), io=40.0MiB (41.9MB), run=281323-281652msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144546: Wed Aug 5 10:50:24 2026 read: IOPS=7, BW=30.1KiB/s (30.9kB/s)(5048KiB/167502msec) slat (usec): min=2529, max=91717, avg=17289.02, stdev=11794.65 clat (msec): min=72, max=1422, avg=643.51, stdev=222.55 lat (msec): min=80, max=1435, avg=660.81, stdev=225.09 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 313], 10.00th=[ 351], 20.00th=[ 435], | 30.00th=[ 510], 40.00th=[ 575], 50.00th=[ 634], 60.00th=[ 701], | 70.00th=[ 760], 80.00th=[ 835], 90.00th=[ 944], 95.00th=[ 1028], | 99.00th=[ 1167], 99.50th=[ 1200], 99.90th=[ 1284], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 7, max= 127, per=26.28%, avg=31.28, stdev=17.65, samples=316 iops : min= 1, max= 31, avg= 7.40, stdev= 4.45, samples=316 write: IOPS=7, BW=30.0KiB/s (31.7kB/s)(5192KiB/167502msec); 0 zone resets slat (usec): min=31, max=29873, avg=249.75, stdev=1298.21 clat (msec): min=365, max=2892, avg=1417.36, stdev=472.41 lat (msec): min=365, max=2892, avg=1417.61, stdev=472.42 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 659], 10.00th=[ 827], 20.00th=[ 1003], | 30.00th=[ 1116], 40.00th=[ 1267], 50.00th=[ 1385], 60.00th=[ 1519], | 70.00th=[ 1687], 80.00th=[ 1838], 90.00th=[ 2089], 95.00th=[ 2232], | 99.00th=[ 2534], 99.50th=[ 2635], 99.90th=[ 2769], 99.95th=[ 2903], | 99.99th=[ 2903] bw ( KiB/s): min= 7, max= 112, per=25.82%, avg=32.02, stdev=19.61, samples=316 iops : min= 1, max= 28, avg= 7.59, stdev= 4.95, samples=316 lat (msec) : 100=0.16%, 250=0.66%, 500=13.83%, 750=22.97%, 1000=18.55% lat (msec) : 2000=37.15%, >=2000=6.68% cpu : usr=0.07%, sys=0.41%, ctx=4143, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144547: Wed Aug 5 10:50:24 2026 read: IOPS=7, BW=29.7KiB/s (30.4kB/s)(4976KiB/167820msec) slat (usec): min=2539, max=86743, avg=17575.84, stdev=11809.15 clat (msec): min=35, max=1414, avg=638.66, stdev=234.79 lat (msec): min=60, max=1429, avg=656.24, stdev=237.58 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 296], 10.00th=[ 338], 20.00th=[ 418], | 30.00th=[ 498], 40.00th=[ 567], 50.00th=[ 625], 60.00th=[ 693], | 70.00th=[ 751], 80.00th=[ 835], 90.00th=[ 961], 95.00th=[ 1062], | 99.00th=[ 1200], 99.50th=[ 1250], 99.90th=[ 1401], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 7, max= 96, per=25.97%, avg=30.91, stdev=17.83, samples=314 iops : min= 1, max= 24, avg= 7.31, stdev= 4.49, samples=314 write: IOPS=7, BW=31.4KiB/s (32.1kB/s)(5264KiB/167820msec); 0 zone resets slat (usec): min=28, max=14260, avg=191.02, stdev=682.75 clat (msec): min=352, max=2811, avg=1415.65, stdev=480.26 lat (msec): min=352, max=2811, avg=1415.85, stdev=480.29 clat percentiles (msec): | 1.00th=[ 489], 5.00th=[ 651], 10.00th=[ 818], 20.00th=[ 995], | 30.00th=[ 1116], 40.00th=[ 1267], 50.00th=[ 1368], 60.00th=[ 1519], | 70.00th=[ 1670], 80.00th=[ 1854], 90.00th=[ 2072], 95.00th=[ 2232], | 99.00th=[ 2534], 99.50th=[ 2635], 99.90th=[ 2802], 99.95th=[ 2802], | 99.99th=[ 2802] bw ( KiB/s): min= 7, max= 112, per=26.21%, avg=32.50, stdev=18.56, samples=317 iops : min= 1, max= 28, avg= 7.70, stdev= 4.66, samples=317 lat (msec) : 50=0.04%, 100=0.12%, 250=1.02%, 500=14.30%, 750=22.54% lat (msec) : 1000=17.23%, 2000=38.44%, >=2000=6.33% cpu : usr=0.06%, sys=0.34%, ctx=3826, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144548: Wed Aug 5 10:50:24 2026 read: IOPS=7, BW=29.7KiB/s (30.4kB/s)(4984KiB/167775msec) slat (usec): min=2188, max=81256, avg=17195.15, stdev=11399.99 clat (msec): min=16, max=1541, avg=635.53, stdev=228.39 lat (msec): min=28, max=1558, avg=652.73, stdev=230.71 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 300], 10.00th=[ 351], 20.00th=[ 430], | 30.00th=[ 498], 40.00th=[ 558], 50.00th=[ 625], 60.00th=[ 684], | 70.00th=[ 751], 80.00th=[ 818], 90.00th=[ 944], 95.00th=[ 1036], | 99.00th=[ 1200], 99.50th=[ 1217], 99.90th=[ 1536], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 7, max= 112, per=25.35%, avg=30.17, stdev=18.53, samples=324 iops : min= 1, max= 28, avg= 7.12, stdev= 4.63, samples=324 write: IOPS=7, BW=31.3KiB/s (32.1kB/s)(5256KiB/167775msec); 0 zone resets slat (usec): min=32, max=15970, avg=197.09, stdev=877.19 clat (msec): min=354, max=2806, avg=1419.18, stdev=474.88 lat (msec): min=354, max=2807, avg=1419.38, stdev=474.90 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 651], 10.00th=[ 802], 20.00th=[ 1003], | 30.00th=[ 1150], 40.00th=[ 1267], 50.00th=[ 1385], 60.00th=[ 1519], | 70.00th=[ 1670], 80.00th=[ 1838], 90.00th=[ 2072], 95.00th=[ 2232], | 99.00th=[ 2567], 99.50th=[ 2668], 99.90th=[ 2802], 99.95th=[ 2802], | 99.99th=[ 2802] bw ( KiB/s): min= 7, max= 86, per=25.27%, avg=31.33, stdev=16.92, samples=326 iops : min= 1, max= 21, avg= 7.41, stdev= 4.25, samples=326 lat (msec) : 20=0.04%, 50=0.04%, 100=0.16%, 250=0.82%, 500=14.26% lat (msec) : 750=22.66%, 1000=17.50%, 2000=38.20%, >=2000=6.33% cpu : usr=0.07%, sys=0.39%, ctx=4437, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144549: Wed Aug 5 10:50:24 2026 read: IOPS=7, BW=30.3KiB/s (31.0kB/s)(5060KiB/167001msec) slat (usec): min=44, max=90471, avg=17377.95, stdev=11739.61 clat (msec): min=24, max=1518, avg=636.98, stdev=232.13 lat (msec): min=34, max=1533, avg=654.36, stdev=235.02 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 296], 10.00th=[ 347], 20.00th=[ 422], | 30.00th=[ 493], 40.00th=[ 567], 50.00th=[ 634], 60.00th=[ 693], | 70.00th=[ 751], 80.00th=[ 827], 90.00th=[ 936], 95.00th=[ 1053], | 99.00th=[ 1234], 99.50th=[ 1401], 99.90th=[ 1469], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 7, max= 104, per=26.18%, avg=31.15, stdev=17.28, samples=318 iops : min= 1, max= 26, avg= 7.37, stdev= 4.33, samples=318 write: IOPS=7, BW=31.0KiB/s (31.8kB/s)(5180KiB/167001msec); 0 zone resets slat (usec): min=31, max=10920, avg=205.06, stdev=740.01 clat (msec): min=243, max=2781, avg=1421.71, stdev=482.75 lat (msec): min=243, max=2781, avg=1421.91, stdev=482.78 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 642], 10.00th=[ 810], 20.00th=[ 1011], | 30.00th=[ 1150], 40.00th=[ 1267], 50.00th=[ 1385], 60.00th=[ 1536], | 70.00th=[ 1687], 80.00th=[ 1838], 90.00th=[ 2089], 95.00th=[ 2265], | 99.00th=[ 2567], 99.50th=[ 2601], 99.90th=[ 2769], 99.95th=[ 2769], | 99.99th=[ 2769] bw ( KiB/s): min= 7, max= 104, per=25.51%, avg=31.63, stdev=19.42, samples=318 iops : min= 1, max= 26, avg= 7.49, stdev= 4.90, samples=318 lat (msec) : 50=0.16%, 100=0.08%, 250=0.86%, 500=14.92%, 750=22.77% lat (msec) : 1000=17.23%, 2000=37.15%, >=2000=6.84% cpu : usr=0.07%, sys=0.38%, ctx=4190, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=120KiB/s (122kB/s), 29.7KiB/s-30.3KiB/s (30.4kB/s-31.0kB/s), io=19.6MiB (20.5MB), run=167001-167820msec WRITE: bw=124KiB/s (127kB/s), 30.0KiB/s-31.4KiB/s (31.7kB/s-32.1kB/s), io=20.4MiB (21.4MB), run=167001-167820msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144579: Wed Aug 5 10:50:36 2026 read: IOPS=1, BW=55.6MiB/s (58.3MB/s)(40.0MiB/719msec) slat (nsec): min=42997k, max=42997k, avg=42997228.00, stdev= 0.00 clat (nsec): min=649388k, max=649388k, avg=649388193.00, stdev= 0.00 lat (nsec): min=692390k, max=692390k, avg=692389765.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 651], 5.00th=[ 651], 10.00th=[ 651], 20.00th=[ 651], | 30.00th=[ 651], 40.00th=[ 651], 50.00th=[ 651], 60.00th=[ 651], | 70.00th=[ 651], 80.00th=[ 651], 90.00th=[ 651], 95.00th=[ 651], | 99.00th=[ 651], 99.50th=[ 651], 99.90th=[ 651], 99.95th=[ 651], | 99.99th=[ 651] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.39%, ctx=6, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=55.6MiB/s (58.3MB/s), 55.6MiB/s-55.6MiB/s (58.3MB/s-58.3MB/s), io=40.0MiB (41.9MB), run=719-719msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout