== sanity test 398c: run fio to test AIO ================= 12:10:32 (1787847032) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.435225 s, 96.4 MB/s osc.lustre-OST0000-osc-ffff9e5fc7853800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145554: Thu Aug 27 12:13:32 2026 write: IOPS=15, BW=60.2KiB/s (61.7kB/s)(10.0MiB/169989msec); 0 zone resets slat (usec): min=22, max=15027, avg=257.55, stdev=927.56 clat (msec): min=208, max=2608, avg=1061.12, stdev=385.33 lat (msec): min=209, max=2608, avg=1061.38, stdev=385.38 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 531], 10.00th=[ 634], 20.00th=[ 751], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1062], | 70.00th=[ 1234], 80.00th=[ 1401], 90.00th=[ 1620], 95.00th=[ 1754], | 99.00th=[ 2123], 99.50th=[ 2265], 99.90th=[ 2601], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 143, per=26.12%, avg=62.69, stdev=34.02, samples=321 iops : min= 1, max= 35, avg=15.16, stdev= 8.51, samples=321 lat (msec) : 250=0.62%, 500=3.24%, 750=16.17%, 1000=34.49%, 2000=43.83% lat (msec) : >=2000=1.64% cpu : usr=0.06%, sys=0.24%, ctx=1644, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145555: Thu Aug 27 12:13:32 2026 write: IOPS=15, BW=60.2KiB/s (61.7kB/s)(10.0MiB/170053msec); 0 zone resets slat (usec): min=22, max=19665, avg=278.23, stdev=1013.21 clat (msec): min=219, max=2619, avg=1061.30, stdev=385.76 lat (msec): min=219, max=2631, avg=1061.58, stdev=385.79 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 523], 10.00th=[ 617], 20.00th=[ 743], | 30.00th=[ 844], 40.00th=[ 911], 50.00th=[ 978], 60.00th=[ 1083], | 70.00th=[ 1234], 80.00th=[ 1385], 90.00th=[ 1620], 95.00th=[ 1754], | 99.00th=[ 2140], 99.50th=[ 2265], 99.90th=[ 2601], 99.95th=[ 2601], | 99.99th=[ 2635] bw ( KiB/s): min= 7, max= 253, per=25.02%, avg=60.05, stdev=34.86, samples=335 iops : min= 1, max= 63, avg=14.51, stdev= 8.74, samples=335 lat (msec) : 250=0.04%, 500=3.87%, 750=16.64%, 1000=31.72%, 2000=46.09% lat (msec) : >=2000=1.64% cpu : usr=0.06%, sys=0.23%, ctx=1895, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145556: Thu Aug 27 12:13:32 2026 write: IOPS=15, BW=60.4KiB/s (61.8kB/s)(10.0MiB/169563msec); 0 zone resets slat (usec): min=23, max=19233, avg=282.70, stdev=1025.42 clat (msec): min=112, max=2646, avg=1059.23, stdev=388.13 lat (msec): min=112, max=2646, avg=1059.52, stdev=388.14 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 535], 10.00th=[ 625], 20.00th=[ 751], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 969], 60.00th=[ 1070], | 70.00th=[ 1234], 80.00th=[ 1401], 90.00th=[ 1620], 95.00th=[ 1770], | 99.00th=[ 2106], 99.50th=[ 2299], 99.90th=[ 2601], 99.95th=[ 2635], | 99.99th=[ 2635] bw ( KiB/s): min= 7, max= 174, per=25.86%, avg=62.07, stdev=33.40, samples=324 iops : min= 1, max= 43, avg=15.01, stdev= 8.35, samples=324 lat (msec) : 250=0.47%, 500=3.28%, 750=16.52%, 1000=33.36%, 2000=44.96% lat (msec) : >=2000=1.41% cpu : usr=0.06%, sys=0.26%, ctx=1965, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145557: Thu Aug 27 12:13:32 2026 write: IOPS=15, BW=60.2KiB/s (61.7kB/s)(10.0MiB/170049msec); 0 zone resets slat (usec): min=22, max=16948, avg=240.93, stdev=866.48 clat (msec): min=98, max=2629, avg=1061.38, stdev=381.75 lat (msec): min=98, max=2633, avg=1061.63, stdev=381.76 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 542], 10.00th=[ 625], 20.00th=[ 751], | 30.00th=[ 844], 40.00th=[ 902], 50.00th=[ 969], 60.00th=[ 1083], | 70.00th=[ 1234], 80.00th=[ 1385], 90.00th=[ 1636], 95.00th=[ 1754], | 99.00th=[ 2056], 99.50th=[ 2232], 99.90th=[ 2601], 99.95th=[ 2601], | 99.99th=[ 2635] bw ( KiB/s): min= 7, max= 253, per=25.48%, avg=61.16, stdev=37.84, samples=329 iops : min= 1, max= 63, avg=14.78, stdev= 9.47, samples=329 lat (msec) : 100=0.04%, 250=0.04%, 500=3.36%, 750=16.56%, 1000=32.42% lat (msec) : 2000=46.09%, >=2000=1.48% cpu : usr=0.05%, sys=0.23%, ctx=1773, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=241KiB/s (247kB/s), 60.2KiB/s-60.4KiB/s (61.7kB/s-61.8kB/s), io=40.0MiB (41.9MB), run=169563-170053msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145591: Thu Aug 27 12:15:37 2026 read: IOPS=10, BW=43.6KiB/s (44.6kB/s)(5048KiB/115783msec) slat (usec): min=3242, max=58793, avg=15864.66, stdev=9407.05 clat (msec): min=10, max=1219, avg=440.14, stdev=154.10 lat (msec): min=23, max=1240, avg=456.01, stdev=156.63 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 239], 10.00th=[ 271], 20.00th=[ 321], | 30.00th=[ 359], 40.00th=[ 384], 50.00th=[ 418], 60.00th=[ 451], | 70.00th=[ 493], 80.00th=[ 550], 90.00th=[ 634], 95.00th=[ 709], | 99.00th=[ 885], 99.50th=[ 1045], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 167, per=24.75%, avg=42.82, stdev=23.71, samples=230 iops : min= 1, max= 41, avg=10.15, stdev= 5.96, samples=230 write: IOPS=11, BW=44.8KiB/s (45.9kB/s)(5192KiB/115783msec); 0 zone resets slat (usec): min=24, max=29307, avg=381.02, stdev=1598.15 clat (msec): min=289, max=2231, avg=980.53, stdev=313.70 lat (msec): min=289, max=2231, avg=980.91, stdev=313.71 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 542], 10.00th=[ 609], 20.00th=[ 693], | 30.00th=[ 785], 40.00th=[ 894], 50.00th=[ 969], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1368], 95.00th=[ 1536], | 99.00th=[ 1938], 99.50th=[ 2005], 99.90th=[ 2123], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 127, per=24.72%, avg=44.50, stdev=20.93, samples=228 iops : min= 1, max= 31, avg=10.57, stdev= 5.24, samples=228 lat (msec) : 20=0.08%, 50=0.12%, 100=0.16%, 250=3.32%, 500=32.97% lat (msec) : 750=24.06%, 1000=15.94%, 2000=23.12%, >=2000=0.23% cpu : usr=0.12%, sys=0.67%, ctx=4283, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145592: Thu Aug 27 12:15:37 2026 read: IOPS=10, BW=43.0KiB/s (44.1kB/s)(4976KiB/115642msec) slat (usec): min=2313, max=66646, avg=15800.33, stdev=9524.75 clat (msec): min=37, max=1207, avg=442.85, stdev=153.01 lat (msec): min=51, max=1235, avg=458.66, stdev=155.80 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 236], 10.00th=[ 271], 20.00th=[ 321], | 30.00th=[ 355], 40.00th=[ 393], 50.00th=[ 422], 60.00th=[ 456], | 70.00th=[ 502], 80.00th=[ 550], 90.00th=[ 634], 95.00th=[ 718], | 99.00th=[ 969], 99.50th=[ 1028], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 7, max= 103, per=24.68%, avg=42.69, stdev=21.42, samples=227 iops : min= 1, max= 25, avg=10.11, stdev= 5.37, samples=227 write: IOPS=11, BW=45.5KiB/s (46.6kB/s)(5264KiB/115642msec); 0 zone resets slat (usec): min=27, max=19355, avg=343.49, stdev=1193.04 clat (msec): min=296, max=2294, avg=970.06, stdev=321.27 lat (msec): min=296, max=2294, avg=970.41, stdev=321.28 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 535], 10.00th=[ 600], 20.00th=[ 676], | 30.00th=[ 776], 40.00th=[ 869], 50.00th=[ 953], 60.00th=[ 1020], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1368], 95.00th=[ 1552], | 99.00th=[ 1972], 99.50th=[ 2022], 99.90th=[ 2165], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 111, per=24.99%, avg=44.99, stdev=22.06, samples=229 iops : min= 1, max= 27, avg=10.69, stdev= 5.52, samples=229 lat (msec) : 50=0.08%, 100=0.08%, 250=3.16%, 500=32.54%, 750=24.84% lat (msec) : 1000=17.19%, 2000=21.72%, >=2000=0.39% cpu : usr=0.12%, sys=0.69%, ctx=4268, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145593: Thu Aug 27 12:15:37 2026 read: IOPS=10, BW=43.1KiB/s (44.1kB/s)(4984KiB/115658msec) slat (usec): min=3168, max=91129, avg=15673.74, stdev=9638.94 clat (msec): min=30, max=1185, avg=444.76, stdev=148.31 lat (msec): min=45, max=1221, avg=460.44, stdev=150.04 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 249], 10.00th=[ 279], 20.00th=[ 330], | 30.00th=[ 363], 40.00th=[ 393], 50.00th=[ 422], 60.00th=[ 460], | 70.00th=[ 498], 80.00th=[ 542], 90.00th=[ 625], 95.00th=[ 718], | 99.00th=[ 927], 99.50th=[ 1099], 99.90th=[ 1150], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 7, max= 119, per=24.84%, avg=42.96, stdev=21.34, samples=227 iops : min= 1, max= 29, avg=10.19, stdev= 5.36, samples=227 write: IOPS=11, BW=45.4KiB/s (46.5kB/s)(5256KiB/115658msec); 0 zone resets slat (usec): min=33, max=39453, avg=447.75, stdev=2242.52 clat (msec): min=211, max=2351, avg=968.27, stdev=320.20 lat (msec): min=211, max=2351, avg=968.72, stdev=320.23 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 527], 10.00th=[ 600], 20.00th=[ 676], | 30.00th=[ 785], 40.00th=[ 877], 50.00th=[ 953], 60.00th=[ 1028], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1334], 95.00th=[ 1502], | 99.00th=[ 1989], 99.50th=[ 2140], 99.90th=[ 2232], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 119, per=25.05%, avg=45.09, stdev=21.95, samples=227 iops : min= 1, max= 29, avg=10.72, stdev= 5.50, samples=227 lat (msec) : 50=0.08%, 100=0.12%, 250=2.46%, 500=33.79%, 750=24.26% lat (msec) : 1000=17.30%, 2000=21.52%, >=2000=0.47% cpu : usr=0.12%, sys=0.73%, ctx=4543, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145594: Thu Aug 27 12:15:37 2026 read: IOPS=11, BW=44.1KiB/s (45.1kB/s)(5060KiB/114845msec) slat (usec): min=40, max=96801, avg=15447.70, stdev=9662.69 clat (msec): min=17, max=1207, avg=437.96, stdev=164.88 lat (msec): min=17, max=1246, avg=453.41, stdev=167.31 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 230], 10.00th=[ 262], 20.00th=[ 313], | 30.00th=[ 347], 40.00th=[ 376], 50.00th=[ 414], 60.00th=[ 451], | 70.00th=[ 493], 80.00th=[ 550], 90.00th=[ 642], 95.00th=[ 735], | 99.00th=[ 1020], 99.50th=[ 1167], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 7, max= 135, per=25.54%, avg=44.19, stdev=23.06, samples=224 iops : min= 1, max= 33, avg=10.49, stdev= 5.80, samples=224 write: IOPS=11, BW=45.1KiB/s (46.2kB/s)(5180KiB/114845msec); 0 zone resets slat (usec): min=26, max=14084, avg=285.03, stdev=783.93 clat (msec): min=135, max=2293, avg=974.75, stdev=319.00 lat (msec): min=136, max=2293, avg=975.04, stdev=319.01 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 550], 10.00th=[ 600], 20.00th=[ 684], | 30.00th=[ 802], 40.00th=[ 869], 50.00th=[ 953], 60.00th=[ 1028], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1351], 95.00th=[ 1552], | 99.00th=[ 1989], 99.50th=[ 2123], 99.90th=[ 2232], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 135, per=25.00%, avg=45.00, stdev=22.37, samples=224 iops : min= 1, max= 33, avg=10.70, stdev= 5.60, samples=224 lat (msec) : 20=0.16%, 50=0.12%, 100=0.12%, 250=3.52%, 500=33.16% lat (msec) : 750=23.16%, 1000=17.07%, 2000=22.30%, >=2000=0.39% cpu : usr=0.10%, sys=0.65%, ctx=4203, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=173KiB/s (177kB/s), 43.0KiB/s-44.1KiB/s (44.1kB/s-45.1kB/s), io=19.6MiB (20.5MB), run=114845-115783msec WRITE: bw=180KiB/s (185kB/s), 44.8KiB/s-45.5KiB/s (45.9kB/s-46.6kB/s), io=20.4MiB (21.4MB), run=114845-115783msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145616: Thu Aug 27 12:15:47 2026 read: IOPS=2, BW=85.7MiB/s (89.8MB/s)(40.0MiB/467msec) slat (nsec): min=47564k, max=47564k, avg=47563610.00, stdev= 0.00 clat (nsec): min=395689k, max=395689k, avg=395689364.00, stdev= 0.00 lat (nsec): min=443257k, max=443257k, avg=443257252.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 397], 10.00th=[ 397], 20.00th=[ 397], | 30.00th=[ 397], 40.00th=[ 397], 50.00th=[ 397], 60.00th=[ 397], | 70.00th=[ 397], 80.00th=[ 397], 90.00th=[ 397], 95.00th=[ 397], | 99.00th=[ 397], 99.50th=[ 397], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.00%, ctx=44, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=85.7MiB/s (89.8MB/s), 85.7MiB/s-85.7MiB/s (89.8MB/s-89.8MB/s), io=40.0MiB (41.9MB), run=467-467msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout