== sanity test 398c: run fio to test AIO ================= 15:55:30 (1788465330) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.512634 s, 81.8 MB/s osc.lustre-OST0000-osc-ffff94fa48ff7800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145256: Thu Sep 3 15:58:50 2026 write: IOPS=13, BW=53.7KiB/s (55.0kB/s)(10.0MiB/190523msec); 0 zone resets slat (usec): min=21, max=13522, avg=170.39, stdev=680.36 clat (msec): min=390, max=2590, avg=1188.82, stdev=393.65 lat (msec): min=391, max=2590, avg=1188.99, stdev=393.66 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 625], 10.00th=[ 726], 20.00th=[ 852], | 30.00th=[ 961], 40.00th=[ 1062], 50.00th=[ 1133], 60.00th=[ 1250], | 70.00th=[ 1351], 80.00th=[ 1485], 90.00th=[ 1720], 95.00th=[ 1888], | 99.00th=[ 2299], 99.50th=[ 2400], 99.90th=[ 2567], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 251, per=27.35%, avg=58.53, stdev=32.47, samples=344 iops : min= 1, max= 62, avg=14.17, stdev= 8.16, samples=344 lat (msec) : 500=2.27%, 750=10.59%, 1000=21.33%, 2000=62.42%, >=2000=3.40% cpu : usr=0.04%, sys=0.17%, ctx=1533, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145257: Thu Sep 3 15:58:50 2026 write: IOPS=13, BW=53.9KiB/s (55.2kB/s)(10.0MiB/190004msec); 0 zone resets slat (usec): min=22, max=16550, avg=175.18, stdev=741.55 clat (msec): min=343, max=2865, avg=1186.63, stdev=402.06 lat (msec): min=343, max=2865, avg=1186.81, stdev=402.07 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 584], 10.00th=[ 709], 20.00th=[ 852], | 30.00th=[ 953], 40.00th=[ 1053], 50.00th=[ 1150], 60.00th=[ 1250], | 70.00th=[ 1351], 80.00th=[ 1502], 90.00th=[ 1737], 95.00th=[ 1905], | 99.00th=[ 2366], 99.50th=[ 2500], 99.90th=[ 2601], 99.95th=[ 2601], | 99.99th=[ 2869] bw ( KiB/s): min= 7, max= 175, per=26.28%, avg=56.25, stdev=31.54, samples=357 iops : min= 1, max= 43, avg=13.59, stdev= 7.93, samples=357 lat (msec) : 500=2.70%, 750=9.45%, 1000=22.77%, 2000=61.72%, >=2000=3.36% cpu : usr=0.04%, sys=0.19%, ctx=1672, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145258: Thu Sep 3 15:58:50 2026 write: IOPS=13, BW=53.9KiB/s (55.2kB/s)(10.0MiB/190007msec); 0 zone resets slat (usec): min=24, max=24607, avg=181.93, stdev=840.69 clat (msec): min=334, max=2584, avg=1185.99, stdev=399.76 lat (msec): min=337, max=2584, avg=1186.18, stdev=399.76 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 609], 10.00th=[ 718], 20.00th=[ 852], | 30.00th=[ 944], 40.00th=[ 1045], 50.00th=[ 1150], 60.00th=[ 1267], | 70.00th=[ 1368], 80.00th=[ 1502], 90.00th=[ 1687], 95.00th=[ 1905], | 99.00th=[ 2333], 99.50th=[ 2467], 99.90th=[ 2567], 99.95th=[ 2567], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 151, per=26.43%, avg=56.56, stdev=31.19, samples=355 iops : min= 1, max= 37, avg=13.66, stdev= 7.84, samples=355 lat (msec) : 500=2.23%, 750=10.23%, 1000=22.58%, 2000=61.17%, >=2000=3.79% cpu : usr=0.04%, sys=0.17%, ctx=1496, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145259: Thu Sep 3 15:58:50 2026 write: IOPS=13, BW=53.8KiB/s (55.1kB/s)(10.0MiB/190324msec); 0 zone resets slat (usec): min=21, max=33867, avg=220.83, stdev=1207.47 clat (msec): min=332, max=2606, avg=1187.55, stdev=402.73 lat (msec): min=332, max=2606, avg=1187.77, stdev=402.73 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 609], 10.00th=[ 693], 20.00th=[ 835], | 30.00th=[ 953], 40.00th=[ 1062], 50.00th=[ 1167], 60.00th=[ 1250], | 70.00th=[ 1368], 80.00th=[ 1502], 90.00th=[ 1720], 95.00th=[ 1888], | 99.00th=[ 2400], 99.50th=[ 2534], 99.90th=[ 2601], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 151, per=25.57%, avg=54.72, stdev=28.78, samples=367 iops : min= 1, max= 37, avg=13.22, stdev= 7.23, samples=367 lat (msec) : 500=2.27%, 750=11.41%, 1000=22.03%, 2000=60.98%, >=2000=3.32% cpu : usr=0.04%, sys=0.16%, ctx=1740, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=215KiB/s (220kB/s), 53.7KiB/s-53.9KiB/s (55.0kB/s-55.2kB/s), io=40.0MiB (41.9MB), run=190004-190523msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145297: Thu Sep 3 16:00:38 2026 read: IOPS=12, BW=51.7KiB/s (52.9kB/s)(5048KiB/97673msec) slat (usec): min=2609, max=72630, avg=13321.99, stdev=7560.52 clat (msec): min=17, max=861, avg=379.88, stdev=99.13 lat (msec): min=45, max=885, avg=393.21, stdev=99.99 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 245], 10.00th=[ 279], 20.00th=[ 309], | 30.00th=[ 334], 40.00th=[ 351], 50.00th=[ 372], 60.00th=[ 393], | 70.00th=[ 414], 80.00th=[ 443], 90.00th=[ 493], 95.00th=[ 535], | 99.00th=[ 701], 99.50th=[ 785], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 7, max= 120, per=24.97%, avg=51.18, stdev=22.27, samples=194 iops : min= 1, max= 30, avg=12.35, stdev= 5.58, samples=194 write: IOPS=13, BW=53.2KiB/s (54.4kB/s)(5192KiB/97673msec); 0 zone resets slat (usec): min=29, max=10821, avg=191.59, stdev=643.45 clat (msec): min=261, max=1880, avg=818.46, stdev=226.66 lat (msec): min=261, max=1880, avg=818.66, stdev=226.65 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 493], 10.00th=[ 550], 20.00th=[ 634], | 30.00th=[ 701], 40.00th=[ 751], 50.00th=[ 810], 60.00th=[ 852], | 70.00th=[ 911], 80.00th=[ 978], 90.00th=[ 1070], 95.00th=[ 1183], | 99.00th=[ 1620], 99.50th=[ 1670], 99.90th=[ 1871], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 103, per=24.78%, avg=52.78, stdev=20.71, samples=193 iops : min= 1, max= 25, avg=12.75, stdev= 5.15, samples=193 lat (msec) : 20=0.04%, 50=0.12%, 100=0.23%, 250=2.50%, 500=44.92% lat (msec) : 750=20.78%, 1000=22.50%, 2000=8.91% cpu : usr=0.10%, sys=0.61%, ctx=4084, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145298: Thu Sep 3 16:00:38 2026 read: IOPS=12, BW=50.9KiB/s (52.1kB/s)(4976KiB/97779msec) slat (usec): min=35, max=473379, avg=14527.92, stdev=22470.72 clat (msec): min=51, max=1561, avg=374.78, stdev=110.39 lat (msec): min=56, max=1573, avg=389.32, stdev=115.97 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 241], 10.00th=[ 271], 20.00th=[ 305], | 30.00th=[ 326], 40.00th=[ 347], 50.00th=[ 368], 60.00th=[ 384], | 70.00th=[ 405], 80.00th=[ 435], 90.00th=[ 493], 95.00th=[ 550], | 99.00th=[ 701], 99.50th=[ 852], 99.90th=[ 1250], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 8, max= 143, per=24.83%, avg=50.91, stdev=22.97, samples=191 iops : min= 2, max= 35, avg=12.28, stdev= 5.75, samples=191 write: IOPS=13, BW=53.8KiB/s (55.1kB/s)(5264KiB/97779msec); 0 zone resets slat (usec): min=27, max=9670, avg=199.61, stdev=542.84 clat (msec): min=54, max=2180, avg=820.34, stdev=259.10 lat (msec): min=54, max=2180, avg=820.54, stdev=259.10 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 456], 10.00th=[ 550], 20.00th=[ 634], | 30.00th=[ 693], 40.00th=[ 751], 50.00th=[ 802], 60.00th=[ 852], | 70.00th=[ 911], 80.00th=[ 978], 90.00th=[ 1083], 95.00th=[ 1267], | 99.00th=[ 1703], 99.50th=[ 1854], 99.90th=[ 2106], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 111, per=25.13%, avg=53.53, stdev=20.60, samples=192 iops : min= 1, max= 27, avg=12.94, stdev= 5.14, samples=192 lat (msec) : 100=0.82%, 250=2.85%, 500=44.18%, 750=21.29%, 1000=21.76% lat (msec) : 2000=9.02%, >=2000=0.08% cpu : usr=0.11%, sys=0.54%, ctx=4318, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145299: Thu Sep 3 16:00:38 2026 read: IOPS=12, BW=51.1KiB/s (52.3kB/s)(4984KiB/97538msec) slat (usec): min=3121, max=50151, avg=13511.16, stdev=7317.68 clat (msec): min=31, max=854, avg=372.11, stdev=93.50 lat (msec): min=45, max=882, avg=385.62, stdev=94.09 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 247], 10.00th=[ 275], 20.00th=[ 305], | 30.00th=[ 326], 40.00th=[ 342], 50.00th=[ 363], 60.00th=[ 384], | 70.00th=[ 409], 80.00th=[ 439], 90.00th=[ 485], 95.00th=[ 523], | 99.00th=[ 667], 99.50th=[ 743], 99.90th=[ 760], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 7, max= 127, per=24.85%, avg=50.93, stdev=24.17, samples=193 iops : min= 1, max= 31, avg=12.30, stdev= 6.04, samples=193 write: IOPS=13, BW=53.9KiB/s (55.2kB/s)(5256KiB/97538msec); 0 zone resets slat (usec): min=23, max=7803, avg=195.23, stdev=508.66 clat (msec): min=144, max=1850, avg=819.57, stdev=225.50 lat (msec): min=144, max=1850, avg=819.76, stdev=225.49 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 477], 10.00th=[ 558], 20.00th=[ 642], | 30.00th=[ 709], 40.00th=[ 760], 50.00th=[ 810], 60.00th=[ 860], | 70.00th=[ 902], 80.00th=[ 969], 90.00th=[ 1070], 95.00th=[ 1183], | 99.00th=[ 1569], 99.50th=[ 1620], 99.90th=[ 1703], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 8, max= 111, per=24.99%, avg=53.23, stdev=19.04, samples=193 iops : min= 2, max= 27, avg=12.87, stdev= 4.75, samples=193 lat (msec) : 50=0.16%, 100=0.27%, 250=2.46%, 500=45.51%, 750=19.41% lat (msec) : 1000=23.59%, 2000=8.59% cpu : usr=0.09%, sys=0.61%, ctx=4152, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145300: Thu Sep 3 16:00:38 2026 read: IOPS=12, BW=51.8KiB/s (53.1kB/s)(5060KiB/97667msec) slat (usec): min=2620, max=93107, avg=13482.70, stdev=7616.25 clat (msec): min=26, max=864, avg=377.76, stdev=95.58 lat (msec): min=30, max=884, avg=391.24, stdev=96.50 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 247], 10.00th=[ 279], 20.00th=[ 305], | 30.00th=[ 330], 40.00th=[ 347], 50.00th=[ 363], 60.00th=[ 388], | 70.00th=[ 414], 80.00th=[ 447], 90.00th=[ 498], 95.00th=[ 542], | 99.00th=[ 693], 99.50th=[ 726], 99.90th=[ 844], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 7, max= 127, per=25.18%, avg=51.63, stdev=23.51, samples=193 iops : min= 1, max= 31, avg=12.47, stdev= 5.89, samples=193 write: IOPS=13, BW=53.0KiB/s (54.3kB/s)(5180KiB/97667msec); 0 zone resets slat (usec): min=24, max=14990, avg=201.25, stdev=669.83 clat (msec): min=164, max=1949, avg=821.50, stdev=229.66 lat (msec): min=164, max=1950, avg=821.70, stdev=229.66 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 481], 10.00th=[ 558], 20.00th=[ 651], | 30.00th=[ 701], 40.00th=[ 760], 50.00th=[ 810], 60.00th=[ 852], | 70.00th=[ 902], 80.00th=[ 986], 90.00th=[ 1070], 95.00th=[ 1167], | 99.00th=[ 1620], 99.50th=[ 1854], 99.90th=[ 1905], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 104, per=24.55%, avg=52.30, stdev=19.37, samples=194 iops : min= 1, max= 26, avg=12.63, stdev= 4.84, samples=194 lat (msec) : 50=0.16%, 100=0.12%, 250=2.73%, 500=45.20%, 750=20.35% lat (msec) : 1000=22.42%, 2000=9.02% cpu : usr=0.10%, sys=0.58%, ctx=4303, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=205KiB/s (210kB/s), 50.9KiB/s-51.8KiB/s (52.1kB/s-53.1kB/s), io=19.6MiB (20.5MB), run=97538-97779msec WRITE: bw=214KiB/s (219kB/s), 53.0KiB/s-53.9KiB/s (54.3kB/s-55.2kB/s), io=20.4MiB (21.4MB), run=97538-97779msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145318: Thu Sep 3 16:00:47 2026 read: IOPS=2, BW=86.2MiB/s (90.4MB/s)(40.0MiB/464msec) slat (nsec): min=27681k, max=27681k, avg=27681237.00, stdev= 0.00 clat (nsec): min=414571k, max=414571k, avg=414571433.00, stdev= 0.00 lat (nsec): min=442258k, max=442258k, avg=442257878.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 414], 10.00th=[ 414], 20.00th=[ 414], | 30.00th=[ 414], 40.00th=[ 414], 50.00th=[ 414], 60.00th=[ 414], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.59%, ctx=42, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=86.2MiB/s (90.4MB/s), 86.2MiB/s-86.2MiB/s (90.4MB/s-90.4MB/s), io=40.0MiB (41.9MB), run=464-464msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout