== sanity test 398c: run fio to test AIO ================= 00:13:51 (1788840831) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.474593 s, 88.4 MB/s osc.lustre-OST0000-osc-ffff9392b1472800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145797: Tue Sep 8 00:16:43 2026 write: IOPS=15, BW=63.6KiB/s (65.1kB/s)(10.0MiB/161043msec); 0 zone resets slat (usec): min=29, max=22493, avg=195.34, stdev=870.14 clat (msec): min=276, max=2010, avg=1005.12, stdev=245.46 lat (msec): min=276, max=2010, avg=1005.32, stdev=245.45 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 634], 10.00th=[ 701], 20.00th=[ 810], | 30.00th=[ 877], 40.00th=[ 944], 50.00th=[ 995], 60.00th=[ 1062], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1334], 95.00th=[ 1418], | 99.00th=[ 1586], 99.50th=[ 1653], 99.90th=[ 1838], 99.95th=[ 1838], | 99.99th=[ 2005] bw ( KiB/s): min= 15, max= 208, per=25.19%, avg=63.99, stdev=27.04, samples=315 iops : min= 3, max= 52, avg=15.57, stdev= 6.78, samples=315 lat (msec) : 500=2.50%, 750=11.56%, 1000=36.45%, 2000=49.45%, >=2000=0.04% cpu : usr=0.04%, sys=0.20%, ctx=1634, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145798: Tue Sep 8 00:16:43 2026 write: IOPS=15, BW=63.6KiB/s (65.1kB/s)(10.0MiB/161006msec); 0 zone resets slat (usec): min=28, max=10981, avg=183.23, stdev=660.90 clat (msec): min=314, max=1897, avg=1005.14, stdev=243.03 lat (msec): min=314, max=1897, avg=1005.32, stdev=243.04 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 609], 10.00th=[ 701], 20.00th=[ 818], | 30.00th=[ 885], 40.00th=[ 944], 50.00th=[ 1003], 60.00th=[ 1062], | 70.00th=[ 1133], 80.00th=[ 1200], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1603], 99.50th=[ 1653], 99.90th=[ 1854], 99.95th=[ 1871], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 160, per=24.88%, avg=63.18, stdev=30.21, samples=319 iops : min= 1, max= 40, avg=15.38, stdev= 7.56, samples=319 lat (msec) : 500=2.58%, 750=10.94%, 1000=36.45%, 2000=50.04% cpu : usr=0.05%, sys=0.20%, ctx=1672, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145799: Tue Sep 8 00:16:43 2026 write: IOPS=15, BW=63.7KiB/s (65.2kB/s)(10.0MiB/160809msec); 0 zone resets slat (usec): min=28, max=21328, avg=199.01, stdev=813.15 clat (msec): min=115, max=1897, avg=1004.41, stdev=248.11 lat (msec): min=115, max=1897, avg=1004.61, stdev=248.11 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 625], 10.00th=[ 701], 20.00th=[ 810], | 30.00th=[ 877], 40.00th=[ 944], 50.00th=[ 995], 60.00th=[ 1062], | 70.00th=[ 1133], 80.00th=[ 1200], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1569], 99.50th=[ 1804], 99.90th=[ 1905], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 151, per=25.19%, avg=63.99, stdev=28.02, samples=315 iops : min= 1, max= 37, avg=15.58, stdev= 7.04, samples=315 lat (msec) : 250=0.31%, 500=2.27%, 750=10.59%, 1000=37.11%, 2000=49.73% cpu : usr=0.05%, sys=0.20%, ctx=1776, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145800: Tue Sep 8 00:16:43 2026 write: IOPS=15, BW=63.6KiB/s (65.1kB/s)(10.0MiB/160951msec); 0 zone resets slat (usec): min=28, max=17703, avg=200.18, stdev=856.63 clat (msec): min=286, max=2011, avg=1004.60, stdev=246.48 lat (msec): min=286, max=2011, avg=1004.81, stdev=246.50 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 617], 10.00th=[ 709], 20.00th=[ 810], | 30.00th=[ 877], 40.00th=[ 936], 50.00th=[ 986], 60.00th=[ 1053], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1620], 99.50th=[ 1754], 99.90th=[ 1854], 99.95th=[ 1871], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 128, per=25.53%, avg=64.85, stdev=30.07, samples=311 iops : min= 1, max= 32, avg=15.79, stdev= 7.53, samples=311 lat (msec) : 500=2.50%, 750=10.59%, 1000=39.14%, 2000=47.73%, >=2000=0.04% cpu : usr=0.04%, sys=0.19%, ctx=1632, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=254KiB/s (260kB/s), 63.6KiB/s-63.7KiB/s (65.1kB/s-65.2kB/s), io=40.0MiB (41.9MB), run=160809-161043msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145833: Tue Sep 8 00:18:49 2026 read: IOPS=10, BW=43.5KiB/s (44.6kB/s)(5048KiB/115972msec) slat (usec): min=2727, max=68722, avg=14454.18, stdev=8134.33 clat (msec): min=12, max=1189, avg=441.09, stdev=132.84 lat (msec): min=24, max=1203, avg=455.55, stdev=133.95 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 275], 10.00th=[ 305], 20.00th=[ 338], | 30.00th=[ 368], 40.00th=[ 388], 50.00th=[ 418], 60.00th=[ 451], | 70.00th=[ 485], 80.00th=[ 542], 90.00th=[ 617], 95.00th=[ 684], | 99.00th=[ 810], 99.50th=[ 869], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 7, max= 119, per=25.29%, avg=43.50, stdev=21.24, samples=227 iops : min= 1, max= 29, avg=10.41, stdev= 5.36, samples=227 write: IOPS=11, BW=44.8KiB/s (45.8kB/s)(5192KiB/115972msec); 0 zone resets slat (usec): min=33, max=32678, avg=337.47, stdev=1391.78 clat (msec): min=325, max=2155, avg=982.78, stdev=276.87 lat (msec): min=325, max=2155, avg=983.12, stdev=276.89 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 558], 10.00th=[ 634], 20.00th=[ 751], | 30.00th=[ 827], 40.00th=[ 894], 50.00th=[ 969], 60.00th=[ 1053], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1334], 95.00th=[ 1452], | 99.00th=[ 1787], 99.50th=[ 1838], 99.90th=[ 2140], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 112, per=24.97%, avg=44.70, stdev=20.29, samples=228 iops : min= 1, max= 28, avg=10.71, stdev= 5.12, samples=228 lat (msec) : 20=0.04%, 50=0.08%, 100=0.20%, 250=1.09%, 500=35.90% lat (msec) : 750=21.02%, 1000=17.81%, 2000=23.71%, >=2000=0.16% cpu : usr=0.10%, sys=0.58%, ctx=4620, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145834: Tue Sep 8 00:18:49 2026 read: IOPS=10, BW=42.9KiB/s (43.9kB/s)(4976KiB/116092msec) slat (usec): min=3078, max=63849, avg=14922.39, stdev=8439.24 clat (msec): min=13, max=1113, avg=443.04, stdev=131.10 lat (msec): min=21, max=1134, avg=457.97, stdev=132.47 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 271], 10.00th=[ 305], 20.00th=[ 347], | 30.00th=[ 372], 40.00th=[ 397], 50.00th=[ 418], 60.00th=[ 447], | 70.00th=[ 485], 80.00th=[ 542], 90.00th=[ 617], 95.00th=[ 684], | 99.00th=[ 835], 99.50th=[ 877], 99.90th=[ 1099], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 119, per=24.93%, avg=42.87, stdev=19.94, samples=227 iops : min= 1, max= 29, avg=10.26, stdev= 5.05, samples=227 write: IOPS=11, BW=45.3KiB/s (46.4kB/s)(5264KiB/116092msec); 0 zone resets slat (usec): min=31, max=14515, avg=296.38, stdev=1033.93 clat (msec): min=244, max=2157, avg=975.57, stdev=281.72 lat (msec): min=244, max=2157, avg=975.87, stdev=281.72 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 542], 10.00th=[ 617], 20.00th=[ 743], | 30.00th=[ 810], 40.00th=[ 894], 50.00th=[ 961], 60.00th=[ 1053], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1334], 95.00th=[ 1435], | 99.00th=[ 1787], 99.50th=[ 1921], 99.90th=[ 2072], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 96, per=25.20%, avg=45.11, stdev=20.65, samples=229 iops : min= 1, max= 24, avg=10.82, stdev= 5.18, samples=229 lat (msec) : 20=0.08%, 50=0.08%, 100=0.23%, 250=1.17%, 500=35.31% lat (msec) : 750=21.45%, 1000=18.24%, 2000=23.32%, >=2000=0.12% cpu : usr=0.09%, sys=0.57%, ctx=4276, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145835: Tue Sep 8 00:18:49 2026 read: IOPS=10, BW=42.9KiB/s (43.0kB/s)(4984KiB/116095msec) slat (msec): min=2, max=176, avg=15.15, stdev= 9.27 clat (msec): min=15, max=1037, avg=435.86, stdev=127.24 lat (msec): min=26, max=1049, avg=451.01, stdev=128.48 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 268], 10.00th=[ 300], 20.00th=[ 338], | 30.00th=[ 368], 40.00th=[ 388], 50.00th=[ 414], 60.00th=[ 447], | 70.00th=[ 481], 80.00th=[ 527], 90.00th=[ 609], 95.00th=[ 676], | 99.00th=[ 776], 99.50th=[ 810], 99.90th=[ 1028], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 7, max= 128, per=24.96%, avg=42.93, stdev=20.81, samples=228 iops : min= 1, max= 32, avg=10.27, stdev= 5.27, samples=228 write: IOPS=11, BW=45.3KiB/s (46.4kB/s)(5256KiB/116095msec); 0 zone resets slat (usec): min=31, max=13711, avg=250.26, stdev=744.32 clat (msec): min=223, max=2060, avg=982.77, stdev=282.93 lat (msec): min=223, max=2061, avg=983.02, stdev=282.96 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 550], 10.00th=[ 642], 20.00th=[ 743], | 30.00th=[ 810], 40.00th=[ 902], 50.00th=[ 978], 60.00th=[ 1053], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1334], 95.00th=[ 1435], | 99.00th=[ 1804], 99.50th=[ 1888], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 111, per=24.96%, avg=44.67, stdev=19.82, samples=230 iops : min= 1, max= 27, avg=10.71, stdev= 4.97, samples=230 lat (msec) : 20=0.04%, 50=0.16%, 100=0.12%, 250=1.21%, 500=36.29% lat (msec) : 750=20.86%, 1000=16.88%, 2000=24.34%, >=2000=0.12% cpu : usr=0.09%, sys=0.54%, ctx=4158, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145836: Tue Sep 8 00:18:49 2026 read: IOPS=10, BW=43.0KiB/s (45.0kB/s)(5060KiB/115117msec) slat (usec): min=50, max=62149, avg=14875.37, stdev=8078.11 clat (msec): min=14, max=1081, avg=439.26, stdev=133.43 lat (msec): min=14, max=1086, avg=454.14, stdev=134.41 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 271], 10.00th=[ 305], 20.00th=[ 342], | 30.00th=[ 372], 40.00th=[ 388], 50.00th=[ 414], 60.00th=[ 447], | 70.00th=[ 485], 80.00th=[ 542], 90.00th=[ 625], 95.00th=[ 693], | 99.00th=[ 802], 99.50th=[ 885], 99.90th=[ 1070], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 107, per=25.68%, avg=44.18, stdev=20.65, samples=225 iops : min= 1, max= 26, avg=10.58, stdev= 5.19, samples=225 write: IOPS=11, BW=44.0KiB/s (46.1kB/s)(5180KiB/115117msec); 0 zone resets slat (usec): min=29, max=13240, avg=231.75, stdev=608.58 clat (msec): min=119, max=2155, avg=977.32, stdev=290.64 lat (msec): min=120, max=2156, avg=977.56, stdev=290.65 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 735], | 30.00th=[ 818], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1334], 95.00th=[ 1469], | 99.00th=[ 1804], 99.50th=[ 1838], 99.90th=[ 2022], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 103, per=24.88%, avg=44.54, stdev=20.25, samples=227 iops : min= 1, max= 25, avg=10.67, stdev= 5.09, samples=227 lat (msec) : 20=0.04%, 50=0.12%, 100=0.47%, 250=1.52%, 500=35.82% lat (msec) : 750=21.68%, 1000=17.85%, 2000=22.38%, >=2000=0.12% cpu : usr=0.10%, sys=0.55%, ctx=4299, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=173KiB/s (177kB/s), 42.9KiB/s-43.0KiB/s (43.9kB/s-45.0kB/s), io=19.6MiB (20.5MB), run=115117-116095msec WRITE: bw=180KiB/s (184kB/s), 44.8KiB/s-45.3KiB/s (45.8kB/s-46.4kB/s), io=20.4MiB (21.4MB), run=115117-116095msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145857: Tue Sep 8 00:19:00 2026 read: IOPS=1, BW=74.3MiB/s (77.0MB/s)(40.0MiB/538msec) slat (nsec): min=59705k, max=59705k, avg=59705365.00, stdev= 0.00 clat (nsec): min=430713k, max=430713k, avg=430713249.00, stdev= 0.00 lat (nsec): min=490423k, max=490423k, avg=490422919.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 430], 10.00th=[ 430], 20.00th=[ 430], | 30.00th=[ 430], 40.00th=[ 430], 50.00th=[ 430], 60.00th=[ 430], | 70.00th=[ 430], 80.00th=[ 430], 90.00th=[ 430], 95.00th=[ 430], | 99.00th=[ 430], 99.50th=[ 430], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.98%, ctx=45, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=74.3MiB/s (77.0MB/s), 74.3MiB/s-74.3MiB/s (77.0MB/s-77.0MB/s), io=40.0MiB (41.9MB), run=538-538msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout