== sanity test 398c: run fio to test AIO ================= 15:28:06 (1790105286) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.554088 s, 75.7 MB/s osc.lustre-OST0000-osc-ffff984638bb2000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144583: Tue Sep 22 15:31:38 2026 write: IOPS=12, BW=51.0KiB/s (52.3kB/s)(10.0MiB/200629msec); 0 zone resets slat (usec): min=24, max=17097, avg=223.58, stdev=867.49 clat (msec): min=341, max=2848, avg=1252.62, stdev=394.09 lat (msec): min=341, max=2849, avg=1252.85, stdev=394.11 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 718], 10.00th=[ 802], 20.00th=[ 911], | 30.00th=[ 1020], 40.00th=[ 1116], 50.00th=[ 1200], 60.00th=[ 1301], | 70.00th=[ 1435], 80.00th=[ 1586], 90.00th=[ 1770], 95.00th=[ 1955], | 99.00th=[ 2333], 99.50th=[ 2467], 99.90th=[ 2769], 99.95th=[ 2769], | 99.99th=[ 2836] bw ( KiB/s): min= 7, max= 128, per=25.69%, avg=52.14, stdev=25.64, samples=385 iops : min= 1, max= 32, avg=12.55, stdev= 6.45, samples=385 lat (msec) : 500=1.33%, 750=5.70%, 1000=20.98%, 2000=67.58%, >=2000=4.41% cpu : usr=0.04%, sys=0.20%, ctx=1772, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144584: Tue Sep 22 15:31:38 2026 write: IOPS=12, BW=50.0KiB/s (52.2kB/s)(10.0MiB/200833msec); 0 zone resets slat (usec): min=24, max=38562, avg=219.62, stdev=1089.65 clat (msec): min=311, max=2756, avg=1253.67, stdev=393.57 lat (msec): min=311, max=2756, avg=1253.89, stdev=393.59 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 718], 10.00th=[ 810], 20.00th=[ 919], | 30.00th=[ 1020], 40.00th=[ 1116], 50.00th=[ 1200], 60.00th=[ 1301], | 70.00th=[ 1418], 80.00th=[ 1586], 90.00th=[ 1787], 95.00th=[ 1938], | 99.00th=[ 2366], 99.50th=[ 2500], 99.90th=[ 2635], 99.95th=[ 2769], | 99.99th=[ 2769] bw ( KiB/s): min= 7, max= 128, per=25.29%, avg=51.35, stdev=28.87, samples=391 iops : min= 1, max= 32, avg=12.34, stdev= 7.24, samples=391 lat (msec) : 500=1.25%, 750=5.23%, 1000=20.82%, 2000=68.52%, >=2000=4.18% cpu : usr=0.04%, sys=0.19%, ctx=1801, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144585: Tue Sep 22 15:31:38 2026 write: IOPS=12, BW=51.1KiB/s (52.3kB/s)(10.0MiB/200451msec); 0 zone resets slat (usec): min=22, max=19019, avg=204.48, stdev=811.37 clat (msec): min=174, max=3283, avg=1251.66, stdev=413.48 lat (msec): min=174, max=3284, avg=1251.87, stdev=413.49 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 701], 10.00th=[ 785], 20.00th=[ 902], | 30.00th=[ 1028], 40.00th=[ 1116], 50.00th=[ 1200], 60.00th=[ 1284], | 70.00th=[ 1435], 80.00th=[ 1586], 90.00th=[ 1787], 95.00th=[ 1972], | 99.00th=[ 2467], 99.50th=[ 2769], 99.90th=[ 3205], 99.95th=[ 3272], | 99.99th=[ 3272] bw ( KiB/s): min= 7, max= 220, per=25.52%, avg=51.81, stdev=29.71, samples=387 iops : min= 1, max= 55, avg=12.46, stdev= 7.44, samples=387 lat (msec) : 250=0.62%, 500=1.25%, 750=5.00%, 1000=21.02%, 2000=67.58% lat (msec) : >=2000=4.53% cpu : usr=0.04%, sys=0.20%, ctx=1789, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144586: Tue Sep 22 15:31:38 2026 write: IOPS=12, BW=51.0KiB/s (52.3kB/s)(10.0MiB/200628msec); 0 zone resets slat (usec): min=24, max=18380, avg=236.40, stdev=1010.10 clat (msec): min=202, max=2833, avg=1252.61, stdev=401.38 lat (msec): min=202, max=2833, avg=1252.85, stdev=401.40 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 701], 10.00th=[ 793], 20.00th=[ 911], | 30.00th=[ 1028], 40.00th=[ 1116], 50.00th=[ 1200], 60.00th=[ 1301], | 70.00th=[ 1418], 80.00th=[ 1569], 90.00th=[ 1804], 95.00th=[ 1955], | 99.00th=[ 2400], 99.50th=[ 2467], 99.90th=[ 2735], 99.95th=[ 2769], | 99.99th=[ 2836] bw ( KiB/s): min= 7, max= 128, per=25.35%, avg=51.46, stdev=26.86, samples=390 iops : min= 1, max= 32, avg=12.37, stdev= 6.75, samples=390 lat (msec) : 250=0.43%, 500=0.86%, 750=6.09%, 1000=20.66%, 2000=67.85% lat (msec) : >=2000=4.10% cpu : usr=0.04%, sys=0.19%, ctx=1672, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=204KiB/s (209kB/s), 50.0KiB/s-51.1KiB/s (52.2kB/s-52.3kB/s), io=40.0MiB (41.9MB), run=200451-200833msec 5% of I/O are 1-page (PAGE_SIZE=4096), max 50% mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144625: Tue Sep 22 15:34:04 2026 read: IOPS=9, BW=37.7KiB/s (38.6kB/s)(5048KiB/133804msec) slat (usec): min=2916, max=89410, avg=17171.87, stdev=10153.76 clat (msec): min=40, max=1075, avg=519.10, stdev=146.65 lat (msec): min=53, max=1111, avg=536.28, stdev=148.18 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 326], 10.00th=[ 359], 20.00th=[ 401], | 30.00th=[ 439], 40.00th=[ 464], 50.00th=[ 493], 60.00th=[ 535], | 70.00th=[ 575], 80.00th=[ 625], 90.00th=[ 718], 95.00th=[ 802], | 99.00th=[ 936], 99.50th=[ 978], 99.90th=[ 1045], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 87, per=25.05%, avg=37.32, stdev=17.93, samples=263 iops : min= 1, max= 21, avg= 8.82, stdev= 4.51, samples=263 write: IOPS=9, BW=38.8KiB/s (39.7kB/s)(5192KiB/133804msec); 0 zone resets slat (usec): min=32, max=18063, avg=264.55, stdev=1068.21 clat (msec): min=347, max=2170, avg=1124.08, stdev=322.38 lat (msec): min=347, max=2170, avg=1124.35, stdev=322.39 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 600], 10.00th=[ 726], 20.00th=[ 852], | 30.00th=[ 936], 40.00th=[ 1028], 50.00th=[ 1116], 60.00th=[ 1183], | 70.00th=[ 1284], 80.00th=[ 1385], 90.00th=[ 1536], 95.00th=[ 1703], | 99.00th=[ 1989], 99.50th=[ 2039], 99.90th=[ 2140], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 104, per=25.00%, avg=38.74, stdev=18.90, samples=262 iops : min= 1, max= 26, avg= 9.18, stdev= 4.73, samples=262 lat (msec) : 50=0.04%, 100=0.20%, 250=0.23%, 500=25.55%, 750=25.20% lat (msec) : 1000=17.03%, 2000=31.33%, >=2000=0.43% cpu : usr=0.09%, sys=0.54%, ctx=4135, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144626: Tue Sep 22 15:34:04 2026 read: IOPS=9, BW=37.1KiB/s (38.0kB/s)(4976KiB/133958msec) slat (usec): min=3199, max=81728, avg=17252.42, stdev=10044.45 clat (msec): min=78, max=1224, avg=519.00, stdev=146.60 lat (msec): min=86, max=1254, avg=536.26, stdev=148.29 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 326], 10.00th=[ 355], 20.00th=[ 393], | 30.00th=[ 430], 40.00th=[ 460], 50.00th=[ 493], 60.00th=[ 531], | 70.00th=[ 575], 80.00th=[ 634], 90.00th=[ 718], 95.00th=[ 802], | 99.00th=[ 927], 99.50th=[ 1003], 99.90th=[ 1200], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 104, per=25.09%, avg=37.38, stdev=18.38, samples=259 iops : min= 1, max= 26, avg= 8.84, stdev= 4.62, samples=259 write: IOPS=9, BW=39.3KiB/s (40.2kB/s)(5264KiB/133958msec); 0 zone resets slat (usec): min=30, max=17020, avg=321.94, stdev=1244.23 clat (msec): min=374, max=2145, avg=1118.83, stdev=324.60 lat (msec): min=374, max=2145, avg=1119.15, stdev=324.63 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 625], 10.00th=[ 718], 20.00th=[ 844], | 30.00th=[ 936], 40.00th=[ 1020], 50.00th=[ 1083], 60.00th=[ 1183], | 70.00th=[ 1267], 80.00th=[ 1385], 90.00th=[ 1536], 95.00th=[ 1720], | 99.00th=[ 1989], 99.50th=[ 2039], 99.90th=[ 2106], 99.95th=[ 2140], | 99.99th=[ 2140] bw ( KiB/s): min= 7, max= 119, per=25.26%, avg=39.15, stdev=18.66, samples=263 iops : min= 1, max= 29, avg= 9.28, stdev= 4.68, samples=263 lat (msec) : 100=0.04%, 250=0.08%, 500=25.70%, 750=25.16%, 1000=17.07% lat (msec) : 2000=31.48%, >=2000=0.47% cpu : usr=0.10%, sys=0.57%, ctx=4261, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144627: Tue Sep 22 15:34:04 2026 read: IOPS=9, BW=37.3KiB/s (38.1kB/s)(4984KiB/133794msec) slat (usec): min=2939, max=73901, avg=16977.49, stdev=9441.92 clat (msec): min=26, max=1058, avg=509.05, stdev=146.50 lat (msec): min=42, max=1085, avg=526.03, stdev=147.77 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 305], 10.00th=[ 347], 20.00th=[ 393], | 30.00th=[ 426], 40.00th=[ 460], 50.00th=[ 489], 60.00th=[ 531], | 70.00th=[ 567], 80.00th=[ 617], 90.00th=[ 701], 95.00th=[ 793], | 99.00th=[ 877], 99.50th=[ 936], 99.90th=[ 1045], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 5, max= 88, per=25.17%, avg=37.50, stdev=19.60, samples=260 iops : min= 1, max= 22, avg= 8.87, stdev= 4.88, samples=260 write: IOPS=9, BW=39.3KiB/s (40.2kB/s)(5256KiB/133794msec); 0 zone resets slat (usec): min=25, max=32030, avg=292.08, stdev=1319.59 clat (msec): min=345, max=2151, avg=1127.23, stdev=322.64 lat (msec): min=345, max=2151, avg=1127.53, stdev=322.62 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 617], 10.00th=[ 743], 20.00th=[ 852], | 30.00th=[ 936], 40.00th=[ 1028], 50.00th=[ 1116], 60.00th=[ 1183], | 70.00th=[ 1284], 80.00th=[ 1385], 90.00th=[ 1552], 95.00th=[ 1720], | 99.00th=[ 1972], 99.50th=[ 2039], 99.90th=[ 2140], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 95, per=25.29%, avg=39.20, stdev=19.27, samples=261 iops : min= 1, max= 23, avg= 9.30, stdev= 4.82, samples=261 lat (msec) : 50=0.23%, 100=0.16%, 250=0.23%, 500=25.82%, 750=24.02% lat (msec) : 1000=17.42%, 2000=31.68%, >=2000=0.43% cpu : usr=0.07%, sys=0.50%, ctx=3896, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144628: Tue Sep 22 15:34:04 2026 read: IOPS=9, BW=38.1KiB/s (39.0kB/s)(5060KiB/132689msec) slat (usec): min=50, max=81476, avg=16710.89, stdev=9522.92 clat (msec): min=15, max=1020, avg=510.36, stdev=143.94 lat (msec): min=37, max=1044, avg=527.08, stdev=145.52 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 321], 10.00th=[ 351], 20.00th=[ 397], | 30.00th=[ 430], 40.00th=[ 464], 50.00th=[ 493], 60.00th=[ 531], | 70.00th=[ 567], 80.00th=[ 625], 90.00th=[ 701], 95.00th=[ 776], | 99.00th=[ 902], 99.50th=[ 927], 99.90th=[ 1003], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 7, max= 111, per=26.01%, avg=38.75, stdev=18.95, samples=255 iops : min= 1, max= 27, avg= 9.18, stdev= 4.75, samples=255 write: IOPS=9, BW=39.0KiB/s (39.0kB/s)(5180KiB/132689msec); 0 zone resets slat (usec): min=34, max=19224, avg=275.87, stdev=1001.63 clat (msec): min=192, max=2179, avg=1123.15, stdev=330.09 lat (msec): min=192, max=2179, avg=1123.42, stdev=330.12 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 625], 10.00th=[ 735], 20.00th=[ 852], | 30.00th=[ 936], 40.00th=[ 1020], 50.00th=[ 1099], 60.00th=[ 1183], | 70.00th=[ 1267], 80.00th=[ 1401], 90.00th=[ 1569], 95.00th=[ 1720], | 99.00th=[ 1989], 99.50th=[ 2056], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 111, per=24.87%, avg=38.56, stdev=19.17, samples=261 iops : min= 1, max= 27, avg= 9.13, stdev= 4.81, samples=261 lat (msec) : 20=0.04%, 50=0.08%, 100=0.39%, 250=0.35%, 500=25.66% lat (msec) : 750=25.27%, 1000=16.76%, 2000=31.13%, >=2000=0.31% cpu : usr=0.09%, sys=0.50%, ctx=4280, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=150KiB/s (153kB/s), 37.1KiB/s-38.1KiB/s (38.0kB/s-39.0kB/s), io=19.6MiB (20.5MB), run=132689-133958msec WRITE: bw=156KiB/s (160kB/s), 38.8KiB/s-39.3KiB/s (39.7kB/s-40.2kB/s), io=20.4MiB (21.4MB), run=132689-133958msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144652: Tue Sep 22 15:34:15 2026 read: IOPS=1, BW=42.1MiB/s (44.2MB/s)(40.0MiB/949msec) slat (nsec): min=72379k, max=72379k, avg=72379203.00, stdev= 0.00 clat (nsec): min=856762k, max=856762k, avg=856761537.00, stdev= 0.00 lat (nsec): min=929158k, max=929158k, avg=929157965.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 860], 5.00th=[ 860], 10.00th=[ 860], 20.00th=[ 860], | 30.00th=[ 860], 40.00th=[ 860], 50.00th=[ 860], 60.00th=[ 860], | 70.00th=[ 860], 80.00th=[ 860], 90.00th=[ 860], 95.00th=[ 860], | 99.00th=[ 860], 99.50th=[ 860], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] lat (msec) : 1000=100.00% cpu : usr=0.00%, sys=1.16%, ctx=32, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=42.1MiB/s (44.2MB/s), 42.1MiB/s-42.1MiB/s (44.2MB/s-44.2MB/s), io=40.0MiB (41.9MB), run=949-949msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout