== sanity test 398c: run fio to test AIO ================= 19:00:47 (1784761247) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.457703 s, 91.6 MB/s osc.lustre-OST0000-osc-ffff8969c4d89800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145822: Wed Jul 22 19:04:08 2026 write: IOPS=13, BW=53.0KiB/s (55.3kB/s)(10.0MiB/189772msec); 0 zone resets slat (usec): min=48, max=19790, avg=310.92, stdev=902.64 clat (msec): min=420, max=2807, avg=1184.34, stdev=348.00 lat (msec): min=421, max=2807, avg=1184.65, stdev=347.96 clat percentiles (msec): | 1.00th=[ 600], 5.00th=[ 726], 10.00th=[ 818], 20.00th=[ 911], | 30.00th=[ 995], 40.00th=[ 1062], 50.00th=[ 1133], 60.00th=[ 1217], | 70.00th=[ 1301], 80.00th=[ 1418], 90.00th=[ 1603], 95.00th=[ 1787], | 99.00th=[ 2500], 99.50th=[ 2601], 99.90th=[ 2702], 99.95th=[ 2702], | 99.99th=[ 2802] bw ( KiB/s): min= 7, max= 128, per=27.07%, avg=58.21, stdev=25.53, samples=345 iops : min= 1, max= 32, avg=14.08, stdev= 6.41, samples=345 lat (msec) : 500=0.62%, 750=5.70%, 1000=24.45%, 2000=66.21%, >=2000=3.01% cpu : usr=0.03%, sys=0.27%, ctx=2102, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145823: Wed Jul 22 19:04:08 2026 write: IOPS=13, BW=54.1KiB/s (55.4kB/s)(10.0MiB/189119msec); 0 zone resets slat (usec): min=42, max=19797, avg=332.44, stdev=991.59 clat (msec): min=195, max=2651, avg=1181.26, stdev=338.34 lat (msec): min=197, max=2657, avg=1181.60, stdev=338.34 clat percentiles (msec): | 1.00th=[ 625], 5.00th=[ 726], 10.00th=[ 818], 20.00th=[ 911], | 30.00th=[ 986], 40.00th=[ 1062], 50.00th=[ 1133], 60.00th=[ 1217], | 70.00th=[ 1301], 80.00th=[ 1401], 90.00th=[ 1603], 95.00th=[ 1821], | 99.00th=[ 2299], 99.50th=[ 2400], 99.90th=[ 2534], 99.95th=[ 2567], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 128, per=26.44%, avg=56.86, stdev=27.65, samples=353 iops : min= 1, max= 32, avg=13.74, stdev= 6.92, samples=353 lat (msec) : 250=0.08%, 500=0.55%, 750=5.35%, 1000=25.00%, 2000=65.86% lat (msec) : >=2000=3.16% cpu : usr=0.04%, sys=0.26%, ctx=2155, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145824: Wed Jul 22 19:04:08 2026 write: IOPS=13, BW=54.0KiB/s (55.3kB/s)(10.0MiB/189538msec); 0 zone resets slat (usec): min=35, max=19111, avg=338.65, stdev=985.24 clat (msec): min=225, max=2680, avg=1182.87, stdev=339.06 lat (msec): min=232, max=2680, avg=1183.21, stdev=339.05 clat percentiles (msec): | 1.00th=[ 600], 5.00th=[ 735], 10.00th=[ 818], 20.00th=[ 911], | 30.00th=[ 1003], 40.00th=[ 1070], 50.00th=[ 1133], 60.00th=[ 1200], | 70.00th=[ 1301], 80.00th=[ 1418], 90.00th=[ 1586], 95.00th=[ 1854], | 99.00th=[ 2265], 99.50th=[ 2333], 99.90th=[ 2534], 99.95th=[ 2668], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 128, per=26.30%, avg=56.55, stdev=28.20, samples=355 iops : min= 1, max= 32, avg=13.66, stdev= 7.07, samples=355 lat (msec) : 250=0.04%, 500=0.59%, 750=5.66%, 1000=23.83%, 2000=66.48% lat (msec) : >=2000=3.40% cpu : usr=0.04%, sys=0.26%, ctx=2145, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145825: Wed Jul 22 19:04:08 2026 write: IOPS=13, BW=53.0KiB/s (55.3kB/s)(10.0MiB/189774msec); 0 zone resets slat (usec): min=44, max=29580, avg=331.27, stdev=1100.13 clat (msec): min=348, max=2607, avg=1184.37, stdev=335.21 lat (msec): min=348, max=2607, avg=1184.71, stdev=335.19 clat percentiles (msec): | 1.00th=[ 600], 5.00th=[ 751], 10.00th=[ 835], 20.00th=[ 919], | 30.00th=[ 1003], 40.00th=[ 1070], 50.00th=[ 1133], 60.00th=[ 1200], | 70.00th=[ 1284], 80.00th=[ 1401], 90.00th=[ 1586], 95.00th=[ 1804], | 99.00th=[ 2400], 99.50th=[ 2534], 99.90th=[ 2601], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 128, per=26.94%, avg=57.93, stdev=31.02, samples=347 iops : min= 1, max= 32, avg=14.01, stdev= 7.76, samples=347 lat (msec) : 500=0.12%, 750=4.73%, 1000=24.84%, 2000=67.30%, >=2000=3.01% cpu : usr=0.04%, sys=0.24%, ctx=2008, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=216KiB/s (221kB/s), 53.0KiB/s-54.1KiB/s (55.3kB/s-55.4kB/s), io=40.0MiB (41.9MB), run=189119-189774msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145872: Wed Jul 22 19:06:12 2026 read: IOPS=10, BW=43.3KiB/s (44.3kB/s)(5048KiB/116702msec) slat (usec): min=3160, max=56116, avg=14689.57, stdev=8259.67 clat (msec): min=17, max=1096, avg=459.39, stdev=136.76 lat (msec): min=36, max=1129, avg=474.09, stdev=138.77 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 271], 10.00th=[ 309], 20.00th=[ 355], | 30.00th=[ 384], 40.00th=[ 414], 50.00th=[ 447], 60.00th=[ 477], | 70.00th=[ 514], 80.00th=[ 558], 90.00th=[ 642], 95.00th=[ 718], | 99.00th=[ 818], 99.50th=[ 877], 99.90th=[ 953], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 7, max= 118, per=25.14%, avg=42.98, stdev=21.83, samples=230 iops : min= 1, max= 29, avg=10.25, stdev= 5.48, samples=230 write: IOPS=11, BW=44.5KiB/s (45.6kB/s)(5192KiB/116702msec); 0 zone resets slat (usec): min=46, max=31821, avg=336.06, stdev=1086.85 clat (msec): min=302, max=2155, avg=974.14, stdev=291.56 lat (msec): min=302, max=2155, avg=974.48, stdev=291.53 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 558], 10.00th=[ 642], 20.00th=[ 743], | 30.00th=[ 810], 40.00th=[ 877], 50.00th=[ 927], 60.00th=[ 995], | 70.00th=[ 1099], 80.00th=[ 1200], 90.00th=[ 1368], 95.00th=[ 1469], | 99.00th=[ 1921], 99.50th=[ 2039], 99.90th=[ 2089], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 6, max= 119, per=24.97%, avg=44.70, stdev=20.76, samples=227 iops : min= 1, max= 29, avg=10.67, stdev= 5.23, samples=227 lat (msec) : 20=0.04%, 50=0.04%, 100=0.39%, 250=0.98%, 500=32.97% lat (msec) : 750=23.40%, 1000=22.23%, 2000=19.65%, >=2000=0.31% cpu : usr=0.08%, sys=0.61%, ctx=4459, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145873: Wed Jul 22 19:06:12 2026 read: IOPS=10, BW=42.6KiB/s (43.7kB/s)(4976KiB/116700msec) slat (usec): min=56, max=72467, avg=15429.36, stdev=9050.87 clat (msec): min=10, max=1102, avg=462.53, stdev=137.58 lat (msec): min=17, max=1114, avg=477.96, stdev=139.88 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 275], 10.00th=[ 300], 20.00th=[ 355], | 30.00th=[ 388], 40.00th=[ 418], 50.00th=[ 447], 60.00th=[ 477], | 70.00th=[ 506], 80.00th=[ 550], 90.00th=[ 642], 95.00th=[ 735], | 99.00th=[ 877], 99.50th=[ 919], 99.90th=[ 1036], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 7, max= 103, per=25.08%, avg=42.89, stdev=20.23, samples=227 iops : min= 1, max= 25, avg=10.22, stdev= 5.09, samples=227 write: IOPS=11, BW=45.1KiB/s (46.2kB/s)(5264KiB/116700msec); 0 zone resets slat (usec): min=47, max=18646, avg=385.97, stdev=912.56 clat (msec): min=158, max=2134, avg=965.61, stdev=290.82 lat (msec): min=158, max=2134, avg=966.00, stdev=290.83 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 558], 10.00th=[ 642], 20.00th=[ 743], | 30.00th=[ 818], 40.00th=[ 869], 50.00th=[ 927], 60.00th=[ 1003], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1334], 95.00th=[ 1452], | 99.00th=[ 1888], 99.50th=[ 1989], 99.90th=[ 2089], 99.95th=[ 2140], | 99.99th=[ 2140] bw ( KiB/s): min= 7, max= 119, per=24.98%, avg=44.72, stdev=20.83, samples=230 iops : min= 1, max= 29, avg=10.68, stdev= 5.25, samples=230 lat (msec) : 20=0.08%, 50=0.08%, 100=0.04%, 250=1.68%, 500=33.01% lat (msec) : 750=22.70%, 1000=21.99%, 2000=20.20%, >=2000=0.23% cpu : usr=0.10%, sys=0.67%, ctx=4981, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145874: Wed Jul 22 19:06:12 2026 read: IOPS=10, BW=42.7KiB/s (43.8kB/s)(4984KiB/116649msec) slat (usec): min=3012, max=54188, avg=15283.23, stdev=8558.65 clat (msec): min=24, max=1127, avg=459.22, stdev=146.30 lat (msec): min=37, max=1136, avg=474.51, stdev=148.10 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 264], 10.00th=[ 300], 20.00th=[ 342], | 30.00th=[ 376], 40.00th=[ 409], 50.00th=[ 443], 60.00th=[ 472], | 70.00th=[ 514], 80.00th=[ 575], 90.00th=[ 667], 95.00th=[ 726], | 99.00th=[ 860], 99.50th=[ 927], 99.90th=[ 1045], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 7, max= 119, per=25.58%, avg=43.74, stdev=20.48, samples=224 iops : min= 1, max= 29, avg=10.44, stdev= 5.17, samples=224 write: IOPS=11, BW=45.1KiB/s (46.1kB/s)(5256KiB/116649msec); 0 zone resets slat (usec): min=56, max=24064, avg=375.90, stdev=1133.84 clat (msec): min=198, max=2128, avg=967.51, stdev=281.06 lat (msec): min=199, max=2128, avg=967.89, stdev=281.06 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 550], 10.00th=[ 634], 20.00th=[ 743], | 30.00th=[ 818], 40.00th=[ 877], 50.00th=[ 944], 60.00th=[ 1020], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1318], 95.00th=[ 1418], | 99.00th=[ 1854], 99.50th=[ 1972], 99.90th=[ 2123], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 111, per=24.85%, avg=44.48, stdev=19.40, samples=230 iops : min= 1, max= 27, avg=10.62, stdev= 4.89, samples=230 lat (msec) : 50=0.20%, 100=0.31%, 250=1.37%, 500=32.50%, 750=23.32% lat (msec) : 1000=20.43%, 2000=21.68%, >=2000=0.20% cpu : usr=0.09%, sys=0.65%, ctx=4819, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145875: Wed Jul 22 19:06:12 2026 read: IOPS=10, BW=43.4KiB/s (44.4kB/s)(5060KiB/116700msec) slat (usec): min=3056, max=65320, avg=14906.98, stdev=8637.37 clat (msec): min=17, max=1016, avg=463.96, stdev=143.23 lat (msec): min=29, max=1038, avg=478.88, stdev=145.02 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 279], 10.00th=[ 305], 20.00th=[ 351], | 30.00th=[ 388], 40.00th=[ 418], 50.00th=[ 443], 60.00th=[ 472], | 70.00th=[ 514], 80.00th=[ 567], 90.00th=[ 676], 95.00th=[ 735], | 99.00th=[ 885], 99.50th=[ 944], 99.90th=[ 1011], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 7, max= 127, per=25.59%, avg=43.76, stdev=21.06, samples=227 iops : min= 1, max= 31, avg=10.44, stdev= 5.26, samples=227 write: IOPS=11, BW=44.4KiB/s (45.5kB/s)(5180KiB/116700msec); 0 zone resets slat (usec): min=53, max=14551, avg=412.18, stdev=1028.59 clat (msec): min=213, max=2160, avg=971.11, stdev=279.99 lat (msec): min=213, max=2160, avg=971.53, stdev=279.96 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 567], 10.00th=[ 634], 20.00th=[ 743], | 30.00th=[ 818], 40.00th=[ 885], 50.00th=[ 936], 60.00th=[ 1003], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1334], 95.00th=[ 1452], | 99.00th=[ 1838], 99.50th=[ 2039], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 111, per=24.32%, avg=43.53, stdev=20.26, samples=232 iops : min= 1, max= 27, avg=10.38, stdev= 5.11, samples=232 lat (msec) : 20=0.04%, 50=0.23%, 100=0.12%, 250=0.86%, 500=33.40% lat (msec) : 750=23.16%, 1000=21.60%, 2000=20.31%, >=2000=0.27% cpu : usr=0.10%, sys=0.67%, ctx=4922, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=172KiB/s (176kB/s), 42.6KiB/s-43.4KiB/s (43.7kB/s-44.4kB/s), io=19.6MiB (20.5MB), run=116649-116702msec WRITE: bw=179KiB/s (183kB/s), 44.4KiB/s-45.1KiB/s (45.5kB/s-46.2kB/s), io=20.4MiB (21.4MB), run=116649-116702msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145896: Wed Jul 22 19:06:24 2026 read: IOPS=1, BW=68.0MiB/s (72.3MB/s)(40.0MiB/580msec) slat (nsec): min=45545k, max=45545k, avg=45545036.00, stdev= 0.00 clat (nsec): min=509553k, max=509553k, avg=509553429.00, stdev= 0.00 lat (nsec): min=555102k, max=555102k, avg=555102368.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 510], 5.00th=[ 510], 10.00th=[ 510], 20.00th=[ 510], | 30.00th=[ 510], 40.00th=[ 510], 50.00th=[ 510], 60.00th=[ 510], | 70.00th=[ 510], 80.00th=[ 510], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.25%, ctx=43, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=68.0MiB/s (72.3MB/s), 68.0MiB/s-68.0MiB/s (72.3MB/s-72.3MB/s), io=40.0MiB (41.9MB), run=580-580msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout