== sanity test 398c: run fio to test AIO ================= 20:22:13 (1788740533) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.513779 s, 81.6 MB/s osc.lustre-OST0000-osc-ffff89c9a0be2000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=154983: Sun Sep 6 20:24:26 2026 write: IOPS=20, BW=82.4KiB/s (84.3kB/s)(10.0MiB/124329msec); 0 zone resets slat (usec): min=20, max=13519, avg=131.96, stdev=420.17 clat (msec): min=151, max=1539, avg=775.78, stdev=177.81 lat (msec): min=151, max=1539, avg=775.91, stdev=177.82 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 477], 10.00th=[ 584], 20.00th=[ 642], | 30.00th=[ 693], 40.00th=[ 726], 50.00th=[ 768], 60.00th=[ 810], | 70.00th=[ 852], 80.00th=[ 911], 90.00th=[ 978], 95.00th=[ 1070], | 99.00th=[ 1284], 99.50th=[ 1385], 99.90th=[ 1469], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 16, max= 255, per=25.29%, avg=83.20, stdev=28.41, samples=243 iops : min= 4, max= 63, avg=20.40, stdev= 7.08, samples=243 lat (msec) : 250=0.66%, 500=4.80%, 750=40.16%, 1000=45.55%, 2000=8.83% cpu : usr=0.06%, sys=0.24%, ctx=1592, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154984: Sun Sep 6 20:24:26 2026 write: IOPS=20, BW=82.5KiB/s (84.5kB/s)(10.0MiB/124082msec); 0 zone resets slat (usec): min=20, max=14527, avg=154.54, stdev=570.08 clat (msec): min=165, max=1499, avg=775.05, stdev=182.17 lat (msec): min=165, max=1499, avg=775.20, stdev=182.16 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 485], 10.00th=[ 575], 20.00th=[ 642], | 30.00th=[ 684], 40.00th=[ 726], 50.00th=[ 768], 60.00th=[ 810], | 70.00th=[ 852], 80.00th=[ 902], 90.00th=[ 995], 95.00th=[ 1099], | 99.00th=[ 1301], 99.50th=[ 1334], 99.90th=[ 1469], 99.95th=[ 1485], | 99.99th=[ 1502] bw ( KiB/s): min= 15, max= 254, per=25.50%, avg=83.90, stdev=32.21, samples=241 iops : min= 3, max= 63, avg=20.58, stdev= 8.07, samples=241 lat (msec) : 250=0.62%, 500=4.69%, 750=40.94%, 1000=44.26%, 2000=9.49% cpu : usr=0.05%, sys=0.22%, ctx=1870, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154985: Sun Sep 6 20:24:26 2026 write: IOPS=20, BW=82.4KiB/s (84.4kB/s)(10.0MiB/124221msec); 0 zone resets slat (usec): min=20, max=14377, avg=166.83, stdev=704.45 clat (msec): min=117, max=1409, avg=775.17, stdev=178.77 lat (msec): min=122, max=1409, avg=775.34, stdev=178.76 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 477], 10.00th=[ 575], 20.00th=[ 642], | 30.00th=[ 693], 40.00th=[ 726], 50.00th=[ 768], 60.00th=[ 810], | 70.00th=[ 860], 80.00th=[ 919], 90.00th=[ 986], 95.00th=[ 1062], | 99.00th=[ 1267], 99.50th=[ 1351], 99.90th=[ 1401], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 7, max= 239, per=24.98%, avg=82.17, stdev=30.62, samples=246 iops : min= 1, max= 59, avg=20.14, stdev= 7.65, samples=246 lat (msec) : 250=0.62%, 500=5.59%, 750=39.14%, 1000=46.02%, 2000=8.63% cpu : usr=0.05%, sys=0.24%, ctx=1951, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154986: Sun Sep 6 20:24:26 2026 write: IOPS=20, BW=82.4KiB/s (84.3kB/s)(10.0MiB/124321msec); 0 zone resets slat (usec): min=21, max=13903, avg=145.59, stdev=544.74 clat (msec): min=67, max=1511, avg=775.69, stdev=181.77 lat (msec): min=67, max=1511, avg=775.84, stdev=181.75 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 489], 10.00th=[ 575], 20.00th=[ 642], | 30.00th=[ 684], 40.00th=[ 726], 50.00th=[ 768], 60.00th=[ 810], | 70.00th=[ 852], 80.00th=[ 902], 90.00th=[ 986], 95.00th=[ 1083], | 99.00th=[ 1318], 99.50th=[ 1401], 99.90th=[ 1502], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 7, max= 198, per=25.37%, avg=83.46, stdev=33.30, samples=242 iops : min= 1, max= 49, avg=20.46, stdev= 8.32, samples=242 lat (msec) : 100=0.08%, 250=0.35%, 500=5.27%, 750=40.66%, 1000=44.61% lat (msec) : 2000=9.02% cpu : usr=0.04%, sys=0.24%, ctx=1687, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=329KiB/s (337kB/s), 82.4KiB/s-82.5KiB/s (84.3kB/s-84.5kB/s), io=40.0MiB (41.9MB), run=124082-124329msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=155014: Sun Sep 6 20:26:11 2026 read: IOPS=12, BW=51.5KiB/s (52.8kB/s)(5048KiB/97954msec) slat (usec): min=2417, max=68032, avg=12850.60, stdev=7878.26 clat (msec): min=9, max=1325, avg=378.37, stdev=133.44 lat (msec): min=15, max=1339, avg=391.23, stdev=135.56 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 218], 10.00th=[ 251], 20.00th=[ 284], | 30.00th=[ 309], 40.00th=[ 330], 50.00th=[ 355], 60.00th=[ 380], | 70.00th=[ 418], 80.00th=[ 468], 90.00th=[ 531], 95.00th=[ 600], | 99.00th=[ 802], 99.50th=[ 1083], 99.90th=[ 1318], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 7, max= 127, per=25.16%, avg=51.33, stdev=24.15, samples=193 iops : min= 1, max= 31, avg=12.37, stdev= 6.10, samples=193 write: IOPS=13, BW=53.0KiB/s (54.3kB/s)(5192KiB/97954msec); 0 zone resets slat (usec): min=28, max=10153, avg=194.35, stdev=502.99 clat (msec): min=159, max=2111, avg=824.51, stdev=291.30 lat (msec): min=159, max=2111, avg=824.70, stdev=291.28 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 401], 10.00th=[ 506], 20.00th=[ 575], | 30.00th=[ 659], 40.00th=[ 726], 50.00th=[ 785], 60.00th=[ 860], | 70.00th=[ 936], 80.00th=[ 1045], 90.00th=[ 1200], 95.00th=[ 1401], | 99.00th=[ 1687], 99.50th=[ 1720], 99.90th=[ 2106], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 7, max= 120, per=24.88%, avg=52.99, stdev=23.62, samples=192 iops : min= 1, max= 30, avg=12.79, stdev= 5.94, samples=192 lat (msec) : 10=0.04%, 20=0.04%, 50=0.04%, 100=0.16%, 250=4.88% lat (msec) : 500=42.03%, 750=23.79%, 1000=17.03%, 2000=11.88%, >=2000=0.12% cpu : usr=0.12%, sys=0.69%, ctx=4885, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=155015: Sun Sep 6 20:26:11 2026 read: IOPS=12, BW=50.8KiB/s (51.0kB/s)(4976KiB/98021msec) slat (usec): min=2123, max=70469, avg=12620.71, stdev=7497.58 clat (msec): min=10, max=1152, avg=374.86, stdev=124.45 lat (msec): min=20, max=1170, avg=387.48, stdev=126.18 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 220], 10.00th=[ 249], 20.00th=[ 279], | 30.00th=[ 309], 40.00th=[ 330], 50.00th=[ 355], 60.00th=[ 380], | 70.00th=[ 414], 80.00th=[ 456], 90.00th=[ 531], 95.00th=[ 609], | 99.00th=[ 776], 99.50th=[ 911], 99.90th=[ 1028], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 142, per=24.84%, avg=50.68, stdev=25.52, samples=192 iops : min= 1, max= 35, avg=12.21, stdev= 6.43, samples=192 write: IOPS=13, BW=53.7KiB/s (54.0kB/s)(5264KiB/98021msec); 0 zone resets slat (usec): min=23, max=11586, avg=186.39, stdev=588.21 clat (msec): min=114, max=2112, avg=823.62, stdev=296.13 lat (msec): min=114, max=2112, avg=823.81, stdev=296.15 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 405], 10.00th=[ 502], 20.00th=[ 592], | 30.00th=[ 659], 40.00th=[ 718], 50.00th=[ 785], 60.00th=[ 852], | 70.00th=[ 927], 80.00th=[ 1045], 90.00th=[ 1217], 95.00th=[ 1385], | 99.00th=[ 1653], 99.50th=[ 1804], 99.90th=[ 2072], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 7, max= 112, per=25.02%, avg=53.30, stdev=23.18, samples=193 iops : min= 1, max= 28, avg=12.86, stdev= 5.79, samples=193 lat (msec) : 20=0.12%, 50=0.16%, 100=0.16%, 250=5.23%, 500=41.25% lat (msec) : 750=24.22%, 1000=17.15%, 2000=11.64%, >=2000=0.08% cpu : usr=0.10%, sys=0.60%, ctx=4926, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=155016: Sun Sep 6 20:26:11 2026 read: IOPS=12, BW=50.9KiB/s (52.1kB/s)(4984KiB/97952msec) slat (usec): min=37, max=85163, avg=13192.44, stdev=8246.35 clat (msec): min=19, max=1326, avg=379.04, stdev=132.77 lat (msec): min=25, max=1346, avg=392.24, stdev=135.00 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 211], 10.00th=[ 247], 20.00th=[ 288], | 30.00th=[ 309], 40.00th=[ 334], 50.00th=[ 355], 60.00th=[ 380], | 70.00th=[ 418], 80.00th=[ 468], 90.00th=[ 542], 95.00th=[ 609], | 99.00th=[ 835], 99.50th=[ 894], 99.90th=[ 1167], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 7, max= 143, per=25.04%, avg=51.08, stdev=23.84, samples=192 iops : min= 1, max= 35, avg=12.31, stdev= 6.00, samples=192 write: IOPS=13, BW=53.7KiB/s (54.9kB/s)(5256KiB/97952msec); 0 zone resets slat (usec): min=24, max=12669, avg=197.14, stdev=672.78 clat (msec): min=116, max=2165, avg=819.62, stdev=310.28 lat (msec): min=116, max=2165, avg=819.82, stdev=310.27 clat percentiles (msec): | 1.00th=[ 251], 5.00th=[ 363], 10.00th=[ 468], 20.00th=[ 575], | 30.00th=[ 651], 40.00th=[ 718], 50.00th=[ 785], 60.00th=[ 852], | 70.00th=[ 927], 80.00th=[ 1053], 90.00th=[ 1183], 95.00th=[ 1435], | 99.00th=[ 1720], 99.50th=[ 1787], 99.90th=[ 2106], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 127, per=25.07%, avg=53.40, stdev=25.09, samples=192 iops : min= 1, max= 31, avg=12.89, stdev= 6.28, samples=192 lat (msec) : 20=0.04%, 50=0.16%, 100=0.20%, 250=5.55%, 500=41.88% lat (msec) : 750=22.30%, 1000=17.62%, 2000=12.11%, >=2000=0.16% cpu : usr=0.09%, sys=0.57%, ctx=4934, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=155017: Sun Sep 6 20:26:11 2026 read: IOPS=12, BW=51.6KiB/s (52.9kB/s)(5060KiB/98022msec) slat (usec): min=2345, max=74988, avg=12856.55, stdev=7751.81 clat (msec): min=12, max=1153, avg=377.06, stdev=126.81 lat (msec): min=18, max=1170, avg=389.92, stdev=129.08 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 220], 10.00th=[ 249], 20.00th=[ 279], | 30.00th=[ 309], 40.00th=[ 330], 50.00th=[ 359], 60.00th=[ 384], | 70.00th=[ 418], 80.00th=[ 464], 90.00th=[ 535], 95.00th=[ 609], | 99.00th=[ 810], 99.50th=[ 902], 99.90th=[ 1133], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 120, per=25.67%, avg=52.37, stdev=25.50, samples=190 iops : min= 1, max= 30, avg=12.63, stdev= 6.39, samples=190 write: IOPS=13, BW=52.8KiB/s (54.1kB/s)(5180KiB/98022msec); 0 zone resets slat (usec): min=29, max=14848, avg=221.82, stdev=859.41 clat (msec): min=112, max=2066, avg=828.11, stdev=302.34 lat (msec): min=112, max=2066, avg=828.33, stdev=302.31 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 393], 10.00th=[ 493], 20.00th=[ 592], | 30.00th=[ 659], 40.00th=[ 718], 50.00th=[ 785], 60.00th=[ 860], | 70.00th=[ 927], 80.00th=[ 1045], 90.00th=[ 1234], 95.00th=[ 1435], | 99.00th=[ 1670], 99.50th=[ 1687], 99.90th=[ 2056], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 120, per=24.61%, avg=52.42, stdev=23.19, samples=193 iops : min= 1, max= 30, avg=12.65, stdev= 5.83, samples=193 lat (msec) : 20=0.04%, 50=0.12%, 100=0.20%, 250=4.96%, 500=42.62% lat (msec) : 750=23.01%, 1000=17.30%, 2000=11.68%, >=2000=0.08% cpu : usr=0.11%, sys=0.56%, ctx=4851, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=205KiB/s (210kB/s), 50.8KiB/s-51.6KiB/s (51.0kB/s-52.9kB/s), io=19.6MiB (20.5MB), run=97952-98022msec WRITE: bw=213KiB/s (218kB/s), 52.8KiB/s-53.7KiB/s (54.1kB/s-54.0kB/s), io=20.4MiB (21.4MB), run=97952-98022msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=155035: Sun Sep 6 20:26:20 2026 read: IOPS=2, BW=83.7MiB/s (87.7MB/s)(40.0MiB/478msec) slat (nsec): min=40645k, max=40645k, avg=40644887.00, stdev= 0.00 clat (nsec): min=399075k, max=399075k, avg=399074950.00, stdev= 0.00 lat (nsec): min=439724k, max=439724k, avg=439724245.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 401], 10.00th=[ 401], 20.00th=[ 401], | 30.00th=[ 401], 40.00th=[ 401], 50.00th=[ 401], 60.00th=[ 401], | 70.00th=[ 401], 80.00th=[ 401], 90.00th=[ 401], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] lat (msec) : 500=100.00% cpu : usr=0.42%, sys=3.56%, ctx=7, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=83.7MiB/s (87.7MB/s), 83.7MiB/s-83.7MiB/s (87.7MB/s-87.7MB/s), io=40.0MiB (41.9MB), run=478-478msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout