== sanity test 398c: run fio to test AIO ================= 03:07:57 (1778656077) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.586531 s, 71.5 MB/s osc.lustre-OST0000-osc-ffff9ad350cb7000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=149724: Wed May 13 03:10:56 2026 write: IOPS=15, BW=60.3KiB/s (61.7kB/s)(10.0MiB/169866msec); 0 zone resets slat (usec): min=40, max=23099, avg=299.70, stdev=1060.31 clat (msec): min=291, max=2254, avg=1060.07, stdev=281.52 lat (msec): min=291, max=2254, avg=1060.38, stdev=281.56 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 676], 10.00th=[ 760], 20.00th=[ 844], | 30.00th=[ 911], 40.00th=[ 969], 50.00th=[ 1020], 60.00th=[ 1099], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1418], 95.00th=[ 1603], | 99.00th=[ 1938], 99.50th=[ 2056], 99.90th=[ 2106], 99.95th=[ 2106], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 223, per=26.30%, avg=63.38, stdev=30.15, samples=318 iops : min= 1, max= 55, avg=15.40, stdev= 7.55, samples=318 lat (msec) : 500=1.25%, 750=8.09%, 1000=37.11%, 2000=52.85%, >=2000=0.70% cpu : usr=0.04%, sys=0.25%, ctx=2189, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149725: Wed May 13 03:10:56 2026 write: IOPS=15, BW=60.5KiB/s (61.9kB/s)(10.0MiB/169392msec); 0 zone resets slat (usec): min=37, max=16318, avg=256.45, stdev=816.63 clat (msec): min=220, max=2503, avg=1057.08, stdev=286.85 lat (msec): min=220, max=2503, avg=1057.34, stdev=286.87 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 642], 10.00th=[ 751], 20.00th=[ 835], | 30.00th=[ 902], 40.00th=[ 969], 50.00th=[ 1020], 60.00th=[ 1083], | 70.00th=[ 1183], 80.00th=[ 1267], 90.00th=[ 1401], 95.00th=[ 1569], | 99.00th=[ 2022], 99.50th=[ 2072], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 128, per=25.65%, avg=61.81, stdev=29.14, samples=326 iops : min= 1, max= 32, avg=15.01, stdev= 7.32, samples=326 lat (msec) : 250=0.66%, 500=0.08%, 750=9.34%, 1000=35.98%, 2000=52.77% lat (msec) : >=2000=1.17% cpu : usr=0.03%, sys=0.25%, ctx=1873, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149726: Wed May 13 03:10:56 2026 write: IOPS=15, BW=60.5KiB/s (61.9kB/s)(10.0MiB/169394msec); 0 zone resets slat (usec): min=49, max=16239, avg=292.96, stdev=934.81 clat (msec): min=146, max=2733, avg=1057.72, stdev=293.26 lat (msec): min=146, max=2733, avg=1058.01, stdev=293.26 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 634], 10.00th=[ 743], 20.00th=[ 844], | 30.00th=[ 919], 40.00th=[ 969], 50.00th=[ 1020], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1401], 95.00th=[ 1586], | 99.00th=[ 2039], 99.50th=[ 2123], 99.90th=[ 2500], 99.95th=[ 2668], | 99.99th=[ 2735] bw ( KiB/s): min= 7, max= 135, per=25.18%, avg=60.68, stdev=26.39, samples=332 iops : min= 1, max= 33, avg=14.73, stdev= 6.61, samples=332 lat (msec) : 250=0.43%, 500=1.21%, 750=8.83%, 1000=35.70%, 2000=52.70% lat (msec) : >=2000=1.13% cpu : usr=0.03%, sys=0.27%, ctx=2141, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149727: Wed May 13 03:10:56 2026 write: IOPS=15, BW=60.3KiB/s (61.7kB/s)(10.0MiB/169868msec); 0 zone resets slat (usec): min=50, max=19646, avg=285.09, stdev=847.78 clat (msec): min=268, max=2228, avg=1059.98, stdev=280.72 lat (msec): min=268, max=2228, avg=1060.27, stdev=280.72 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 659], 10.00th=[ 760], 20.00th=[ 844], | 30.00th=[ 911], 40.00th=[ 978], 50.00th=[ 1028], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1284], 90.00th=[ 1418], 95.00th=[ 1552], | 99.00th=[ 1938], 99.50th=[ 2072], 99.90th=[ 2165], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 128, per=25.18%, avg=60.69, stdev=28.25, samples=332 iops : min= 1, max= 32, avg=14.72, stdev= 7.05, samples=332 lat (msec) : 500=0.86%, 750=8.75%, 1000=35.16%, 2000=54.30%, >=2000=0.94% cpu : usr=0.04%, sys=0.27%, ctx=2170, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=241KiB/s (247kB/s), 60.3KiB/s-60.5KiB/s (61.7kB/s-61.9kB/s), io=40.0MiB (41.9MB), run=169392-169868msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=149761: Wed May 13 03:13:17 2026 read: IOPS=9, BW=38.4KiB/s (39.4kB/s)(5048KiB/131316msec) slat (usec): min=89, max=396729, avg=16817.40, stdev=15241.61 clat (msec): min=13, max=1407, avg=519.41, stdev=199.67 lat (msec): min=21, max=1451, avg=536.24, stdev=202.98 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 288], 10.00th=[ 326], 20.00th=[ 380], | 30.00th=[ 409], 40.00th=[ 443], 50.00th=[ 481], 60.00th=[ 523], | 70.00th=[ 575], 80.00th=[ 642], 90.00th=[ 776], 95.00th=[ 911], | 99.00th=[ 1234], 99.50th=[ 1284], 99.90th=[ 1401], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 7, max= 103, per=26.10%, avg=39.42, stdev=21.25, samples=250 iops : min= 1, max= 25, avg= 9.34, stdev= 5.36, samples=250 write: IOPS=9, BW=39.5KiB/s (40.5kB/s)(5192KiB/131316msec); 0 zone resets slat (usec): min=43, max=10547, avg=302.10, stdev=685.51 clat (msec): min=174, max=3268, avg=1095.85, stdev=422.08 lat (msec): min=177, max=3268, avg=1096.15, stdev=422.11 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 592], 10.00th=[ 667], 20.00th=[ 793], | 30.00th=[ 860], 40.00th=[ 927], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1183], 80.00th=[ 1351], 90.00th=[ 1620], 95.00th=[ 1905], | 99.00th=[ 2635], 99.50th=[ 2869], 99.90th=[ 3239], 99.95th=[ 3272], | 99.99th=[ 3272] bw ( KiB/s): min= 7, max= 103, per=25.20%, avg=39.57, stdev=20.46, samples=256 iops : min= 1, max= 25, avg= 9.38, stdev= 5.14, samples=256 lat (msec) : 20=0.12%, 50=0.23%, 100=0.20%, 250=1.09%, 500=26.21% lat (msec) : 750=23.83%, 1000=20.82%, 2000=25.51%, >=2000=1.99% cpu : usr=0.07%, sys=0.55%, ctx=4220, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149762: Wed May 13 03:13:17 2026 read: IOPS=9, BW=37.6KiB/s (38.5kB/s)(4976KiB/132381msec) slat (usec): min=2829, max=99745, avg=16487.06, stdev=10478.70 clat (msec): min=59, max=1550, avg=521.82, stdev=195.36 lat (msec): min=64, max=1585, avg=538.31, stdev=198.28 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 292], 10.00th=[ 330], 20.00th=[ 380], | 30.00th=[ 414], 40.00th=[ 443], 50.00th=[ 481], 60.00th=[ 514], | 70.00th=[ 567], 80.00th=[ 651], 90.00th=[ 785], 95.00th=[ 902], | 99.00th=[ 1217], 99.50th=[ 1334], 99.90th=[ 1536], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 7, max= 119, per=25.51%, avg=38.52, stdev=20.92, samples=252 iops : min= 1, max= 29, avg= 9.12, stdev= 5.25, samples=252 write: IOPS=9, BW=39.8KiB/s (40.7kB/s)(5264KiB/132381msec); 0 zone resets slat (usec): min=47, max=18265, avg=326.41, stdev=948.28 clat (msec): min=320, max=3293, avg=1098.02, stdev=410.90 lat (msec): min=320, max=3293, avg=1098.35, stdev=410.92 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 600], 10.00th=[ 667], 20.00th=[ 802], | 30.00th=[ 877], 40.00th=[ 944], 50.00th=[ 1011], 60.00th=[ 1099], | 70.00th=[ 1200], 80.00th=[ 1334], 90.00th=[ 1620], 95.00th=[ 1888], | 99.00th=[ 2433], 99.50th=[ 2903], 99.90th=[ 3205], 99.95th=[ 3306], | 99.99th=[ 3306] bw ( KiB/s): min= 7, max= 96, per=25.38%, avg=39.84, stdev=19.98, samples=258 iops : min= 1, max= 24, avg= 9.45, stdev= 5.04, samples=258 lat (msec) : 100=0.12%, 250=0.74%, 500=26.88%, 750=22.70%, 1000=21.21% lat (msec) : 2000=26.13%, >=2000=2.23% cpu : usr=0.08%, sys=0.54%, ctx=4362, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149763: Wed May 13 03:13:17 2026 read: IOPS=9, BW=37.6KiB/s (38.6kB/s)(4984KiB/132380msec) slat (msec): min=2, max=521, avg=16.73, stdev=17.61 clat (msec): min=14, max=1436, avg=521.10, stdev=188.93 lat (msec): min=20, max=1453, avg=537.84, stdev=192.06 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 292], 10.00th=[ 330], 20.00th=[ 384], | 30.00th=[ 418], 40.00th=[ 451], 50.00th=[ 485], 60.00th=[ 523], | 70.00th=[ 567], 80.00th=[ 634], 90.00th=[ 768], 95.00th=[ 902], | 99.00th=[ 1183], 99.50th=[ 1250], 99.90th=[ 1301], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 7, max= 104, per=25.44%, avg=38.41, stdev=20.74, samples=254 iops : min= 1, max= 26, avg= 9.09, stdev= 5.25, samples=254 write: IOPS=9, BW=39.7KiB/s (40.7kB/s)(5256KiB/132380msec); 0 zone resets slat (usec): min=51, max=23279, avg=305.84, stdev=933.04 clat (msec): min=329, max=3292, avg=1098.79, stdev=410.42 lat (msec): min=329, max=3293, avg=1099.10, stdev=410.41 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 575], 10.00th=[ 676], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 961], 50.00th=[ 1020], 60.00th=[ 1099], | 70.00th=[ 1217], 80.00th=[ 1334], 90.00th=[ 1586], 95.00th=[ 1921], | 99.00th=[ 2567], 99.50th=[ 3004], 99.90th=[ 3205], 99.95th=[ 3306], | 99.99th=[ 3306] bw ( KiB/s): min= 7, max= 112, per=25.25%, avg=39.64, stdev=20.50, samples=258 iops : min= 1, max= 28, avg= 9.40, stdev= 5.17, samples=258 lat (msec) : 20=0.08%, 50=0.12%, 100=0.04%, 250=0.74%, 500=26.33% lat (msec) : 750=24.10%, 1000=20.59%, 2000=25.94%, >=2000=2.07% cpu : usr=0.06%, sys=0.53%, ctx=4122, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149764: Wed May 13 03:13:17 2026 read: IOPS=9, BW=38.3KiB/s (39.3kB/s)(5060KiB/131987msec) slat (usec): min=2502, max=99496, avg=16599.89, stdev=10354.46 clat (msec): min=28, max=1575, avg=524.48, stdev=196.39 lat (msec): min=33, max=1626, avg=541.08, stdev=199.46 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 296], 10.00th=[ 330], 20.00th=[ 384], | 30.00th=[ 418], 40.00th=[ 451], 50.00th=[ 485], 60.00th=[ 518], | 70.00th=[ 567], 80.00th=[ 634], 90.00th=[ 785], 95.00th=[ 911], | 99.00th=[ 1234], 99.50th=[ 1318], 99.90th=[ 1552], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 7, max= 95, per=25.98%, avg=39.23, stdev=19.78, samples=252 iops : min= 1, max= 23, avg= 9.30, stdev= 4.94, samples=252 write: IOPS=9, BW=39.2KiB/s (40.2kB/s)(5180KiB/131987msec); 0 zone resets slat (usec): min=55, max=16214, avg=297.10, stdev=709.79 clat (msec): min=335, max=3414, avg=1099.29, stdev=413.13 lat (msec): min=335, max=3414, avg=1099.59, stdev=413.14 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 600], 10.00th=[ 676], 20.00th=[ 793], | 30.00th=[ 869], 40.00th=[ 936], 50.00th=[ 1011], 60.00th=[ 1099], | 70.00th=[ 1200], 80.00th=[ 1334], 90.00th=[ 1620], 95.00th=[ 1854], | 99.00th=[ 2500], 99.50th=[ 2903], 99.90th=[ 3373], 99.95th=[ 3406], | 99.99th=[ 3406] bw ( KiB/s): min= 7, max= 96, per=25.25%, avg=39.65, stdev=20.03, samples=255 iops : min= 1, max= 24, avg= 9.40, stdev= 5.07, samples=255 lat (msec) : 50=0.08%, 100=0.04%, 250=0.82%, 500=26.88%, 750=23.83% lat (msec) : 1000=20.66%, 2000=25.74%, >=2000=1.95% cpu : usr=0.05%, sys=0.53%, ctx=4473, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=152KiB/s (155kB/s), 37.6KiB/s-38.4KiB/s (38.5kB/s-39.4kB/s), io=19.6MiB (20.5MB), run=131316-132381msec WRITE: bw=158KiB/s (162kB/s), 39.2KiB/s-39.8KiB/s (40.2kB/s-40.7kB/s), io=20.4MiB (21.4MB), run=131316-132381msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=149788: Wed May 13 03:13:29 2026 read: IOPS=1, BW=72.2MiB/s (75.7MB/s)(40.0MiB/554msec) slat (nsec): min=81118k, max=81118k, avg=81118151.00, stdev= 0.00 clat (nsec): min=461358k, max=461358k, avg=461358153.00, stdev= 0.00 lat (nsec): min=542481k, max=542481k, avg=542480611.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 460], 10.00th=[ 460], 20.00th=[ 460], | 30.00th=[ 460], 40.00th=[ 460], 50.00th=[ 460], 60.00th=[ 460], | 70.00th=[ 460], 80.00th=[ 460], 90.00th=[ 460], 95.00th=[ 460], | 99.00th=[ 460], 99.50th=[ 460], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=0.36%, ctx=25, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=72.2MiB/s (75.7MB/s), 72.2MiB/s-72.2MiB/s (75.7MB/s-75.7MB/s), io=40.0MiB (41.9MB), run=554-554msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout