== sanity test 398c: run fio to test AIO ================= 11:41:36 (1790005296) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.73963 s, 56.7 MB/s osc.lustre-OST0000-osc-ffff91e0d0b6a800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145003: Mon Sep 21 11:45:09 2026 write: IOPS=12, BW=50.9KiB/s (52.1kB/s)(10.0MiB/201213msec); 0 zone resets slat (usec): min=22, max=38735, avg=298.27, stdev=1493.73 clat (msec): min=66, max=3640, avg=1255.64, stdev=475.46 lat (msec): min=66, max=3641, avg=1255.94, stdev=475.48 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 609], 10.00th=[ 735], 20.00th=[ 902], | 30.00th=[ 1003], 40.00th=[ 1083], 50.00th=[ 1183], 60.00th=[ 1301], | 70.00th=[ 1435], 80.00th=[ 1620], 90.00th=[ 1821], 95.00th=[ 2089], | 99.00th=[ 2769], 99.50th=[ 3104], 99.90th=[ 3641], 99.95th=[ 3641], | 99.99th=[ 3641] bw ( KiB/s): min= 6, max= 207, per=25.72%, avg=52.20, stdev=29.55, samples=384 iops : min= 1, max= 51, avg=12.51, stdev= 7.44, samples=384 lat (msec) : 100=0.04%, 250=0.08%, 500=3.55%, 750=7.85%, 1000=18.28% lat (msec) : 2000=63.79%, >=2000=6.41% cpu : usr=0.04%, sys=0.22%, ctx=1766, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145004: Mon Sep 21 11:45:09 2026 write: IOPS=12, BW=50.0KiB/s (52.2kB/s)(10.0MiB/200855msec); 0 zone resets slat (usec): min=23, max=48043, avg=294.12, stdev=1440.52 clat (msec): min=247, max=3676, avg=1254.39, stdev=478.95 lat (msec): min=247, max=3676, avg=1254.69, stdev=478.99 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 584], 10.00th=[ 709], 20.00th=[ 877], | 30.00th=[ 1003], 40.00th=[ 1083], 50.00th=[ 1183], 60.00th=[ 1301], | 70.00th=[ 1452], 80.00th=[ 1603], 90.00th=[ 1838], 95.00th=[ 2089], | 99.00th=[ 2836], 99.50th=[ 2970], 99.90th=[ 3473], 99.95th=[ 3473], | 99.99th=[ 3675] bw ( KiB/s): min= 7, max= 254, per=26.45%, avg=53.70, stdev=31.26, samples=373 iops : min= 1, max= 63, avg=12.89, stdev= 7.85, samples=373 lat (msec) : 250=0.08%, 500=3.52%, 750=8.32%, 1000=17.85%, 2000=64.26% lat (msec) : >=2000=5.98% cpu : usr=0.05%, sys=0.21%, ctx=1854, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145005: Mon Sep 21 11:45:09 2026 write: IOPS=12, BW=50.9KiB/s (52.1kB/s)(10.0MiB/201207msec); 0 zone resets slat (usec): min=23, max=30401, avg=268.58, stdev=1181.35 clat (msec): min=277, max=3627, avg=1255.66, stdev=471.94 lat (msec): min=277, max=3631, avg=1255.93, stdev=471.93 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 609], 10.00th=[ 709], 20.00th=[ 911], | 30.00th=[ 1011], 40.00th=[ 1099], 50.00th=[ 1183], 60.00th=[ 1284], | 70.00th=[ 1418], 80.00th=[ 1603], 90.00th=[ 1871], 95.00th=[ 2106], | 99.00th=[ 2836], 99.50th=[ 2970], 99.90th=[ 3171], 99.95th=[ 3272], | 99.99th=[ 3641] bw ( KiB/s): min= 6, max= 254, per=26.50%, avg=53.79, stdev=32.54, samples=373 iops : min= 1, max= 63, avg=12.91, stdev= 8.20, samples=373 lat (msec) : 500=3.24%, 750=8.40%, 1000=17.58%, 2000=63.67%, >=2000=7.11% cpu : usr=0.05%, sys=0.20%, ctx=1811, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145006: Mon Sep 21 11:45:09 2026 write: IOPS=12, BW=50.0KiB/s (52.2kB/s)(10.0MiB/200960msec); 0 zone resets slat (usec): min=23, max=22265, avg=286.89, stdev=1205.61 clat (msec): min=216, max=3520, avg=1253.81, stdev=475.50 lat (msec): min=216, max=3520, avg=1254.11, stdev=475.56 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 600], 10.00th=[ 701], 20.00th=[ 902], | 30.00th=[ 1003], 40.00th=[ 1099], 50.00th=[ 1183], 60.00th=[ 1301], | 70.00th=[ 1418], 80.00th=[ 1603], 90.00th=[ 1838], 95.00th=[ 2089], | 99.00th=[ 2769], 99.50th=[ 2903], 99.90th=[ 3473], 99.95th=[ 3473], | 99.99th=[ 3507] bw ( KiB/s): min= 7, max= 254, per=25.59%, avg=51.95, stdev=31.56, samples=386 iops : min= 1, max= 63, avg=12.45, stdev= 7.92, samples=386 lat (msec) : 250=0.62%, 500=2.73%, 750=8.63%, 1000=17.30%, 2000=63.98% lat (msec) : >=2000=6.72% cpu : usr=0.05%, sys=0.23%, ctx=1727, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=204KiB/s (208kB/s), 50.9KiB/s-50.0KiB/s (52.1kB/s-52.2kB/s), io=40.0MiB (41.9MB), run=200855-201213msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145054: Mon Sep 21 11:47:19 2026 read: IOPS=10, BW=42.5KiB/s (43.5kB/s)(5048KiB/118771msec) slat (msec): min=2, max=310, avg=16.79, stdev=13.76 clat (msec): min=43, max=1389, avg=456.77, stdev=174.36 lat (msec): min=50, max=1393, avg=473.57, stdev=178.13 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 271], 10.00th=[ 296], 20.00th=[ 334], | 30.00th=[ 363], 40.00th=[ 388], 50.00th=[ 418], 60.00th=[ 451], | 70.00th=[ 493], 80.00th=[ 550], 90.00th=[ 659], 95.00th=[ 835], | 99.00th=[ 1070], 99.50th=[ 1150], 99.90th=[ 1318], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 7, max= 119, per=25.37%, avg=42.63, stdev=25.30, samples=231 iops : min= 1, max= 29, avg=10.10, stdev= 6.37, samples=231 write: IOPS=10, BW=43.7KiB/s (44.8kB/s)(5192KiB/118771msec); 0 zone resets slat (usec): min=24, max=20318, avg=342.27, stdev=1187.37 clat (msec): min=380, max=3011, avg=1000.17, stdev=412.28 lat (msec): min=380, max=3011, avg=1000.51, stdev=412.30 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 535], 10.00th=[ 592], 20.00th=[ 684], | 30.00th=[ 760], 40.00th=[ 835], 50.00th=[ 902], 60.00th=[ 986], | 70.00th=[ 1099], 80.00th=[ 1234], 90.00th=[ 1552], 95.00th=[ 1804], | 99.00th=[ 2467], 99.50th=[ 2903], 99.90th=[ 3004], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 7, max= 112, per=25.20%, avg=44.10, stdev=22.72, samples=230 iops : min= 1, max= 28, avg=10.47, stdev= 5.70, samples=230 lat (msec) : 50=0.12%, 100=0.12%, 250=1.25%, 500=35.20%, 750=23.59% lat (msec) : 1000=19.10%, 2000=18.87%, >=2000=1.76% cpu : usr=0.10%, sys=0.65%, ctx=4545, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145055: Mon Sep 21 11:47:19 2026 read: IOPS=10, BW=41.9KiB/s (42.9kB/s)(4976KiB/118832msec) slat (usec): min=3102, max=83846, avg=16354.13, stdev=9850.48 clat (msec): min=43, max=1301, avg=464.88, stdev=173.05 lat (msec): min=60, max=1333, avg=481.24, stdev=175.91 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 275], 10.00th=[ 300], 20.00th=[ 342], | 30.00th=[ 372], 40.00th=[ 397], 50.00th=[ 422], 60.00th=[ 460], | 70.00th=[ 502], 80.00th=[ 567], 90.00th=[ 676], 95.00th=[ 827], | 99.00th=[ 1116], 99.50th=[ 1217], 99.90th=[ 1284], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 7, max= 112, per=24.98%, avg=41.97, stdev=24.00, samples=231 iops : min= 1, max= 28, avg= 9.93, stdev= 6.07, samples=231 write: IOPS=11, BW=44.3KiB/s (45.4kB/s)(5264KiB/118832msec); 0 zone resets slat (usec): min=25, max=14640, avg=343.69, stdev=1078.08 clat (msec): min=270, max=3026, avg=987.23, stdev=409.59 lat (msec): min=270, max=3026, avg=987.58, stdev=409.54 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 531], 10.00th=[ 584], 20.00th=[ 667], | 30.00th=[ 751], 40.00th=[ 827], 50.00th=[ 894], 60.00th=[ 986], | 70.00th=[ 1083], 80.00th=[ 1200], 90.00th=[ 1469], 95.00th=[ 1821], | 99.00th=[ 2534], 99.50th=[ 2869], 99.90th=[ 3004], 99.95th=[ 3037], | 99.99th=[ 3037] bw ( KiB/s): min= 7, max= 95, per=25.34%, avg=44.35, stdev=20.42, samples=232 iops : min= 1, max= 23, avg=10.52, stdev= 5.13, samples=232 lat (msec) : 50=0.08%, 100=0.08%, 250=0.90%, 500=34.38%, 750=24.80% lat (msec) : 1000=19.45%, 2000=18.67%, >=2000=1.64% cpu : usr=0.12%, sys=0.65%, ctx=4186, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145056: Mon Sep 21 11:47:19 2026 read: IOPS=10, BW=41.0KiB/s (43.0kB/s)(4984KiB/118687msec) slat (msec): min=3, max=212, avg=17.01, stdev=12.28 clat (msec): min=14, max=1337, avg=460.34, stdev=171.55 lat (msec): min=25, max=1369, avg=477.35, stdev=175.15 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 271], 10.00th=[ 300], 20.00th=[ 334], | 30.00th=[ 368], 40.00th=[ 397], 50.00th=[ 430], 60.00th=[ 460], | 70.00th=[ 502], 80.00th=[ 550], 90.00th=[ 676], 95.00th=[ 818], | 99.00th=[ 1083], 99.50th=[ 1150], 99.90th=[ 1234], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 7, max= 111, per=25.57%, avg=42.96, stdev=23.47, samples=227 iops : min= 1, max= 27, avg=10.19, stdev= 5.93, samples=227 write: IOPS=11, BW=44.3KiB/s (45.3kB/s)(5256KiB/118687msec); 0 zone resets slat (usec): min=28, max=33177, avg=312.05, stdev=1323.55 clat (msec): min=305, max=3148, avg=988.47, stdev=417.32 lat (msec): min=305, max=3149, avg=988.78, stdev=417.43 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 531], 10.00th=[ 575], 20.00th=[ 659], | 30.00th=[ 743], 40.00th=[ 827], 50.00th=[ 894], 60.00th=[ 978], | 70.00th=[ 1099], 80.00th=[ 1217], 90.00th=[ 1502], 95.00th=[ 1838], | 99.00th=[ 2500], 99.50th=[ 2903], 99.90th=[ 3138], 99.95th=[ 3138], | 99.99th=[ 3138] bw ( KiB/s): min= 7, max= 111, per=25.21%, avg=44.13, stdev=21.67, samples=232 iops : min= 1, max= 27, avg=10.47, stdev= 5.44, samples=232 lat (msec) : 20=0.04%, 50=0.16%, 100=0.35%, 250=1.13%, 500=33.98% lat (msec) : 750=25.23%, 1000=18.87%, 2000=18.87%, >=2000=1.37% cpu : usr=0.10%, sys=0.62%, ctx=4442, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145057: Mon Sep 21 11:47:19 2026 read: IOPS=10, BW=42.0KiB/s (44.0kB/s)(5060KiB/117737msec) slat (usec): min=52, max=307603, avg=16615.58, stdev=13029.76 clat (msec): min=15, max=1273, avg=455.71, stdev=171.65 lat (msec): min=16, max=1301, avg=472.34, stdev=175.05 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 268], 10.00th=[ 292], 20.00th=[ 330], | 30.00th=[ 363], 40.00th=[ 388], 50.00th=[ 422], 60.00th=[ 456], | 70.00th=[ 498], 80.00th=[ 550], 90.00th=[ 659], 95.00th=[ 802], | 99.00th=[ 1062], 99.50th=[ 1150], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 7, max= 118, per=26.32%, avg=44.21, stdev=22.13, samples=224 iops : min= 1, max= 29, avg=10.50, stdev= 5.60, samples=224 write: IOPS=10, BW=43.0KiB/s (45.1kB/s)(5180KiB/117737msec); 0 zone resets slat (usec): min=23, max=51386, avg=301.66, stdev=1578.10 clat (msec): min=189, max=3024, avg=992.13, stdev=428.63 lat (msec): min=189, max=3028, avg=992.43, stdev=428.78 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 523], 10.00th=[ 575], 20.00th=[ 676], | 30.00th=[ 751], 40.00th=[ 818], 50.00th=[ 894], 60.00th=[ 978], | 70.00th=[ 1099], 80.00th=[ 1234], 90.00th=[ 1519], 95.00th=[ 1821], | 99.00th=[ 2802], 99.50th=[ 2903], 99.90th=[ 3004], 99.95th=[ 3037], | 99.99th=[ 3037] bw ( KiB/s): min= 7, max= 103, per=25.27%, avg=44.23, stdev=21.13, samples=228 iops : min= 1, max= 25, avg=10.49, stdev= 5.32, samples=228 lat (msec) : 20=0.16%, 50=0.31%, 100=0.16%, 250=1.45%, 500=34.65% lat (msec) : 750=24.65%, 1000=18.48%, 2000=18.32%, >=2000=1.84% cpu : usr=0.10%, sys=0.67%, ctx=4466, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=169KiB/s (173kB/s), 41.9KiB/s-42.0KiB/s (42.9kB/s-44.0kB/s), io=19.6MiB (20.5MB), run=117737-118832msec WRITE: bw=176KiB/s (180kB/s), 43.7KiB/s-44.3KiB/s (44.8kB/s-45.4kB/s), io=20.4MiB (21.4MB), run=117737-118832msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145080: Mon Sep 21 11:47:32 2026 read: IOPS=1, BW=51.9MiB/s (54.5MB/s)(40.0MiB/770msec) slat (nsec): min=60056k, max=60056k, avg=60056075.00, stdev= 0.00 clat (nsec): min=688940k, max=688940k, avg=688939978.00, stdev= 0.00 lat (nsec): min=749001k, max=749001k, avg=749001491.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 693], 5.00th=[ 693], 10.00th=[ 693], 20.00th=[ 693], | 30.00th=[ 693], 40.00th=[ 693], 50.00th=[ 693], 60.00th=[ 693], | 70.00th=[ 693], 80.00th=[ 693], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.08%, ctx=21, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=51.9MiB/s (54.5MB/s), 51.9MiB/s-51.9MiB/s (54.5MB/s-54.5MB/s), io=40.0MiB (41.9MB), run=770-770msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout