== sanity test 398c: run fio to test AIO ================= 06:27:43 (1789122463) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.794083 s, 52.8 MB/s osc.lustre-OST0000-osc-ffff8eef0339b800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145361: Fri Sep 11 06:30:32 2026 write: IOPS=16, BW=65.7KiB/s (67.3kB/s)(10.0MiB/155760msec); 0 zone resets slat (usec): min=25, max=21817, avg=320.06, stdev=1196.00 clat (msec): min=69, max=2472, avg=972.81, stdev=263.96 lat (msec): min=69, max=2472, avg=973.14, stdev=263.94 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 567], 10.00th=[ 693], 20.00th=[ 785], | 30.00th=[ 827], 40.00th=[ 885], 50.00th=[ 944], 60.00th=[ 1003], | 70.00th=[ 1070], 80.00th=[ 1167], 90.00th=[ 1284], 95.00th=[ 1418], | 99.00th=[ 1720], 99.50th=[ 1972], 99.90th=[ 2366], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 128, per=25.10%, avg=65.76, stdev=29.68, samples=306 iops : min= 1, max= 32, avg=15.92, stdev= 7.46, samples=306 lat (msec) : 100=0.04%, 500=2.70%, 750=11.88%, 1000=44.14%, 2000=40.82% lat (msec) : >=2000=0.43% cpu : usr=0.06%, sys=0.29%, ctx=2295, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145362: Fri Sep 11 06:30:32 2026 write: IOPS=16, BW=65.8KiB/s (67.4kB/s)(10.0MiB/155628msec); 0 zone resets slat (usec): min=23, max=20243, avg=311.11, stdev=1189.92 clat (msec): min=182, max=2444, avg=971.88, stdev=266.80 lat (msec): min=182, max=2444, avg=972.20, stdev=266.83 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 542], 10.00th=[ 693], 20.00th=[ 785], | 30.00th=[ 844], 40.00th=[ 894], 50.00th=[ 944], 60.00th=[ 995], | 70.00th=[ 1070], 80.00th=[ 1167], 90.00th=[ 1301], 95.00th=[ 1435], | 99.00th=[ 1770], 99.50th=[ 1972], 99.90th=[ 2140], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 128, per=25.78%, avg=67.55, stdev=31.78, samples=298 iops : min= 1, max= 32, avg=16.37, stdev= 7.97, samples=298 lat (msec) : 250=0.62%, 500=2.27%, 750=12.19%, 1000=46.17%, 2000=38.44% lat (msec) : >=2000=0.31% cpu : usr=0.06%, sys=0.33%, ctx=1949, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145363: Fri Sep 11 06:30:32 2026 write: IOPS=16, BW=65.7KiB/s (67.3kB/s)(10.0MiB/155765msec); 0 zone resets slat (usec): min=28, max=15875, avg=243.01, stdev=924.84 clat (msec): min=356, max=2466, avg=972.32, stdev=260.98 lat (msec): min=356, max=2466, avg=972.57, stdev=261.00 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 609], 10.00th=[ 701], 20.00th=[ 776], | 30.00th=[ 835], 40.00th=[ 885], 50.00th=[ 944], 60.00th=[ 995], | 70.00th=[ 1070], 80.00th=[ 1150], 90.00th=[ 1301], 95.00th=[ 1435], | 99.00th=[ 1854], 99.50th=[ 1989], 99.90th=[ 2299], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 256, per=25.62%, avg=67.12, stdev=31.94, samples=300 iops : min= 1, max= 64, avg=16.27, stdev= 8.05, samples=300 lat (msec) : 500=3.16%, 750=12.93%, 1000=45.00%, 2000=38.48%, >=2000=0.43% cpu : usr=0.06%, sys=0.28%, ctx=1805, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145364: Fri Sep 11 06:30:32 2026 write: IOPS=16, BW=65.8KiB/s (67.4kB/s)(10.0MiB/155623msec); 0 zone resets slat (usec): min=23, max=26841, avg=261.53, stdev=1002.26 clat (msec): min=160, max=2464, avg=971.91, stdev=257.39 lat (msec): min=160, max=2464, avg=972.17, stdev=257.41 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 600], 10.00th=[ 701], 20.00th=[ 776], | 30.00th=[ 835], 40.00th=[ 885], 50.00th=[ 944], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1150], 90.00th=[ 1267], 95.00th=[ 1418], | 99.00th=[ 1787], 99.50th=[ 1955], 99.90th=[ 2299], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 136, per=24.85%, avg=65.12, stdev=30.09, samples=309 iops : min= 1, max= 34, avg=15.76, stdev= 7.53, samples=309 lat (msec) : 250=0.08%, 500=2.15%, 750=13.67%, 1000=42.85%, 2000=41.02% lat (msec) : >=2000=0.23% cpu : usr=0.07%, sys=0.27%, ctx=1897, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=263KiB/s (269kB/s), 65.7KiB/s-65.8KiB/s (67.3kB/s-67.4kB/s), io=40.0MiB (41.9MB), run=155623-155765msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145396: Fri Sep 11 06:32:44 2026 read: IOPS=10, BW=41.0KiB/s (42.0kB/s)(5048KiB/123013msec) slat (usec): min=2866, max=89332, avg=16519.87, stdev=10048.99 clat (msec): min=16, max=1304, avg=486.78, stdev=188.59 lat (msec): min=25, max=1334, avg=503.32, stdev=191.45 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 275], 10.00th=[ 305], 20.00th=[ 347], | 30.00th=[ 384], 40.00th=[ 414], 50.00th=[ 443], 60.00th=[ 485], | 70.00th=[ 527], 80.00th=[ 600], 90.00th=[ 735], 95.00th=[ 902], | 99.00th=[ 1083], 99.50th=[ 1150], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 7, max= 111, per=25.42%, avg=41.44, stdev=21.42, samples=238 iops : min= 1, max= 27, avg= 9.78, stdev= 5.37, samples=238 write: IOPS=10, BW=42.2KiB/s (43.2kB/s)(5192KiB/123013msec); 0 zone resets slat (usec): min=26, max=30935, avg=435.19, stdev=1565.08 clat (msec): min=338, max=2859, avg=1022.89, stdev=362.47 lat (msec): min=338, max=2859, avg=1023.32, stdev=362.44 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 609], 10.00th=[ 667], 20.00th=[ 743], | 30.00th=[ 818], 40.00th=[ 885], 50.00th=[ 944], 60.00th=[ 1011], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1519], 95.00th=[ 1787], | 99.00th=[ 2232], 99.50th=[ 2601], 99.90th=[ 2869], 99.95th=[ 2869], | 99.99th=[ 2869] bw ( KiB/s): min= 7, max= 111, per=24.81%, avg=41.93, stdev=20.48, samples=241 iops : min= 1, max= 27, avg= 9.90, stdev= 5.19, samples=241 lat (msec) : 20=0.04%, 50=0.08%, 100=0.27%, 250=1.25%, 500=30.51% lat (msec) : 750=23.05%, 1000=22.42%, 2000=21.13%, >=2000=1.25% cpu : usr=0.12%, sys=0.82%, ctx=4728, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145397: Fri Sep 11 06:32:44 2026 read: IOPS=10, BW=40.5KiB/s (41.5kB/s)(4976KiB/122889msec) slat (usec): min=2412, max=76326, avg=16316.14, stdev=9462.84 clat (msec): min=15, max=1320, avg=482.33, stdev=197.68 lat (msec): min=22, max=1343, avg=498.66, stdev=200.17 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 271], 10.00th=[ 305], 20.00th=[ 347], | 30.00th=[ 376], 40.00th=[ 405], 50.00th=[ 430], 60.00th=[ 464], | 70.00th=[ 518], 80.00th=[ 600], 90.00th=[ 743], 95.00th=[ 885], | 99.00th=[ 1200], 99.50th=[ 1267], 99.90th=[ 1301], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 7, max= 103, per=25.22%, avg=41.11, stdev=21.36, samples=236 iops : min= 1, max= 25, avg= 9.70, stdev= 5.39, samples=236 write: IOPS=10, BW=42.8KiB/s (43.9kB/s)(5264KiB/122889msec); 0 zone resets slat (usec): min=24, max=24142, avg=478.91, stdev=1517.43 clat (msec): min=335, max=2817, avg=1018.17, stdev=357.25 lat (msec): min=335, max=2817, avg=1018.65, stdev=357.32 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 600], 10.00th=[ 676], 20.00th=[ 743], | 30.00th=[ 810], 40.00th=[ 877], 50.00th=[ 927], 60.00th=[ 1011], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1469], 95.00th=[ 1754], | 99.00th=[ 2198], 99.50th=[ 2467], 99.90th=[ 2769], 99.95th=[ 2802], | 99.99th=[ 2802] bw ( KiB/s): min= 7, max= 104, per=25.53%, avg=43.14, stdev=20.21, samples=238 iops : min= 1, max= 26, avg=10.21, stdev= 5.08, samples=238 lat (msec) : 20=0.08%, 50=0.20%, 100=0.35%, 250=0.98%, 500=31.17% lat (msec) : 750=22.15%, 1000=22.46%, 2000=21.33%, >=2000=1.29% cpu : usr=0.12%, sys=0.81%, ctx=4616, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145398: Fri Sep 11 06:32:44 2026 read: IOPS=10, BW=40.6KiB/s (41.5kB/s)(4984KiB/122893msec) slat (usec): min=2996, max=97829, avg=16498.00, stdev=9678.95 clat (msec): min=39, max=1297, avg=488.91, stdev=184.06 lat (msec): min=47, max=1320, avg=505.42, stdev=186.99 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 275], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 384], 40.00th=[ 414], 50.00th=[ 439], 60.00th=[ 485], | 70.00th=[ 535], 80.00th=[ 609], 90.00th=[ 751], 95.00th=[ 860], | 99.00th=[ 1070], 99.50th=[ 1116], 99.90th=[ 1284], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 7, max= 103, per=25.15%, avg=40.99, stdev=20.29, samples=238 iops : min= 1, max= 25, avg= 9.66, stdev= 5.10, samples=238 write: IOPS=10, BW=42.8KiB/s (43.8kB/s)(5256KiB/122893msec); 0 zone resets slat (usec): min=27, max=21225, avg=442.30, stdev=1484.88 clat (msec): min=346, max=2805, avg=1012.92, stdev=356.90 lat (msec): min=347, max=2805, avg=1013.37, stdev=356.88 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 609], 10.00th=[ 667], 20.00th=[ 743], | 30.00th=[ 810], 40.00th=[ 869], 50.00th=[ 936], 60.00th=[ 1003], | 70.00th=[ 1099], 80.00th=[ 1217], 90.00th=[ 1469], 95.00th=[ 1737], | 99.00th=[ 2232], 99.50th=[ 2400], 99.90th=[ 2769], 99.95th=[ 2802], | 99.99th=[ 2802] bw ( KiB/s): min= 7, max= 95, per=25.60%, avg=43.26, stdev=20.43, samples=236 iops : min= 1, max= 23, avg=10.23, stdev= 5.17, samples=236 lat (msec) : 50=0.12%, 100=0.20%, 250=1.25%, 500=29.53%, 750=23.91% lat (msec) : 1000=23.24%, 2000=20.59%, >=2000=1.17% cpu : usr=0.11%, sys=0.77%, ctx=4703, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145399: Fri Sep 11 06:32:44 2026 read: IOPS=10, BW=41.2KiB/s (42.2kB/s)(5060KiB/122683msec) slat (usec): min=50, max=73973, avg=16477.32, stdev=9808.77 clat (msec): min=17, max=1824, avg=489.09, stdev=197.59 lat (msec): min=27, max=1863, avg=505.58, stdev=200.93 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 284], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 380], 40.00th=[ 414], 50.00th=[ 443], 60.00th=[ 485], | 70.00th=[ 531], 80.00th=[ 609], 90.00th=[ 743], 95.00th=[ 885], | 99.00th=[ 1150], 99.50th=[ 1334], 99.90th=[ 1787], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 7, max= 128, per=25.45%, avg=41.49, stdev=21.03, samples=238 iops : min= 1, max= 32, avg= 9.79, stdev= 5.28, samples=238 write: IOPS=10, BW=42.2KiB/s (43.2kB/s)(5180KiB/122683msec); 0 zone resets slat (usec): min=31, max=20918, avg=427.12, stdev=1367.06 clat (msec): min=117, max=2828, avg=1020.12, stdev=370.32 lat (msec): min=117, max=2828, avg=1020.56, stdev=370.38 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 575], 10.00th=[ 659], 20.00th=[ 743], | 30.00th=[ 818], 40.00th=[ 869], 50.00th=[ 936], 60.00th=[ 1020], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1469], 95.00th=[ 1804], | 99.00th=[ 2265], 99.50th=[ 2467], 99.90th=[ 2802], 99.95th=[ 2836], | 99.99th=[ 2836] bw ( KiB/s): min= 7, max= 103, per=24.86%, avg=42.02, stdev=19.74, samples=240 iops : min= 1, max= 25, avg= 9.93, stdev= 5.00, samples=240 lat (msec) : 20=0.08%, 50=0.12%, 100=0.27%, 250=1.17%, 500=30.98% lat (msec) : 750=22.89%, 1000=22.11%, 2000=20.70%, >=2000=1.68% cpu : usr=0.12%, sys=0.79%, ctx=4893, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=163KiB/s (167kB/s), 40.5KiB/s-41.2KiB/s (41.5kB/s-42.2kB/s), io=19.6MiB (20.5MB), run=122683-123013msec WRITE: bw=170KiB/s (174kB/s), 42.2KiB/s-42.8KiB/s (43.2kB/s-43.9kB/s), io=20.4MiB (21.4MB), run=122683-123013msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145420: Fri Sep 11 06:32:51 2026 read: IOPS=1, BW=62.0MiB/s (65.0MB/s)(40.0MiB/645msec) slat (nsec): min=56051k, max=56051k, avg=56050766.00, stdev= 0.00 clat (nsec): min=553062k, max=553062k, avg=553061756.00, stdev= 0.00 lat (nsec): min=609118k, max=609118k, avg=609118237.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 550], 5.00th=[ 550], 10.00th=[ 550], 20.00th=[ 550], | 30.00th=[ 550], 40.00th=[ 550], 50.00th=[ 550], 60.00th=[ 550], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.33%, ctx=6, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=62.0MiB/s (65.0MB/s), 62.0MiB/s-62.0MiB/s (65.0MB/s-65.0MB/s), io=40.0MiB (41.9MB), run=645-645msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout