== sanity test 398c: run fio to test AIO ================= 13:46:08 (1761241568) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.325824 s, 129 MB/s osc.lustre-OST0000-osc-ffff8fb8413f2800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147771: Thu Oct 23 13:48:26 2025 write: IOPS=19, BW=78.0KiB/s (79.9kB/s)(10.0MiB/131213msec); 0 zone resets slat (usec): min=28, max=4886, avg=143.98, stdev=300.15 clat (msec): min=122, max=1563, avg=819.19, stdev=201.50 lat (msec): min=122, max=1563, avg=819.33, stdev=201.50 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 527], 10.00th=[ 617], 20.00th=[ 684], | 30.00th=[ 735], 40.00th=[ 768], 50.00th=[ 810], 60.00th=[ 852], | 70.00th=[ 911], 80.00th=[ 969], 90.00th=[ 1062], 95.00th=[ 1133], | 99.00th=[ 1368], 99.50th=[ 1452], 99.90th=[ 1536], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 7, max= 255, per=25.00%, avg=77.75, stdev=30.55, samples=259 iops : min= 1, max= 63, avg=18.92, stdev= 7.64, samples=259 lat (msec) : 250=1.88%, 500=2.34%, 750=30.20%, 1000=49.77%, 2000=15.82% cpu : usr=0.04%, sys=0.27%, ctx=2001, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147772: Thu Oct 23 13:48:26 2025 write: IOPS=19, BW=77.0KiB/s (79.8kB/s)(10.0MiB/131353msec); 0 zone resets slat (usec): min=28, max=4506, avg=139.22, stdev=290.64 clat (msec): min=149, max=1544, avg=820.30, stdev=197.40 lat (msec): min=149, max=1545, avg=820.44, stdev=197.41 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 550], 10.00th=[ 609], 20.00th=[ 676], | 30.00th=[ 726], 40.00th=[ 776], 50.00th=[ 818], 60.00th=[ 860], | 70.00th=[ 919], 80.00th=[ 969], 90.00th=[ 1062], 95.00th=[ 1133], | 99.00th=[ 1351], 99.50th=[ 1469], 99.90th=[ 1502], 99.95th=[ 1536], | 99.99th=[ 1552] bw ( KiB/s): min= 7, max= 255, per=24.90%, avg=77.44, stdev=33.34, samples=260 iops : min= 1, max= 63, avg=18.84, stdev= 8.34, samples=260 lat (msec) : 250=1.25%, 500=2.73%, 750=30.12%, 1000=50.90%, 2000=15.00% cpu : usr=0.05%, sys=0.26%, ctx=1590, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147773: Thu Oct 23 13:48:26 2025 write: IOPS=19, BW=78.0KiB/s (79.9kB/s)(10.0MiB/131262msec); 0 zone resets slat (usec): min=27, max=4297, avg=156.91, stdev=313.98 clat (msec): min=70, max=1573, avg=819.54, stdev=202.01 lat (msec): min=70, max=1574, avg=819.70, stdev=202.01 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 535], 10.00th=[ 617], 20.00th=[ 684], | 30.00th=[ 735], 40.00th=[ 776], 50.00th=[ 818], 60.00th=[ 852], | 70.00th=[ 902], 80.00th=[ 969], 90.00th=[ 1062], 95.00th=[ 1150], | 99.00th=[ 1385], 99.50th=[ 1452], 99.90th=[ 1552], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 8, max= 255, per=24.81%, avg=77.15, stdev=29.73, samples=261 iops : min= 2, max= 63, avg=18.77, stdev= 7.45, samples=261 lat (msec) : 100=0.04%, 250=1.64%, 500=2.73%, 750=29.34%, 1000=51.13% lat (msec) : 2000=15.12% cpu : usr=0.05%, sys=0.28%, ctx=1884, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147774: Thu Oct 23 13:48:26 2025 write: IOPS=19, BW=78.0KiB/s (79.9kB/s)(10.0MiB/131216msec); 0 zone resets slat (usec): min=28, max=12400, avg=172.20, stdev=473.34 clat (msec): min=80, max=1545, avg=819.60, stdev=201.03 lat (msec): min=81, max=1546, avg=819.77, stdev=201.04 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 523], 10.00th=[ 609], 20.00th=[ 684], | 30.00th=[ 735], 40.00th=[ 768], 50.00th=[ 810], 60.00th=[ 860], | 70.00th=[ 919], 80.00th=[ 978], 90.00th=[ 1053], 95.00th=[ 1133], | 99.00th=[ 1368], 99.50th=[ 1418], 99.90th=[ 1536], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 15, max= 243, per=24.90%, avg=77.44, stdev=27.89, samples=260 iops : min= 3, max= 60, avg=18.84, stdev= 7.00, samples=260 lat (msec) : 100=0.04%, 250=1.72%, 500=2.58%, 750=30.04%, 1000=48.98% lat (msec) : 2000=16.64% cpu : usr=0.04%, sys=0.28%, ctx=2063, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=312KiB/s (319kB/s), 77.0KiB/s-78.0KiB/s (79.8kB/s-79.9kB/s), io=40.0MiB (41.9MB), run=131213-131353msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147802: Thu Oct 23 13:50:01 2025 read: IOPS=14, BW=56.6KiB/s (57.0kB/s)(5048KiB/89169msec) slat (usec): min=2850, max=49393, avg=9949.85, stdev=4326.02 clat (msec): min=17, max=1133, avg=349.75, stdev=92.52 lat (msec): min=24, max=1146, avg=359.71, stdev=93.67 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 234], 10.00th=[ 262], 20.00th=[ 284], | 30.00th=[ 305], 40.00th=[ 326], 50.00th=[ 342], 60.00th=[ 363], | 70.00th=[ 380], 80.00th=[ 409], 90.00th=[ 447], 95.00th=[ 498], | 99.00th=[ 600], 99.50th=[ 701], 99.90th=[ 1003], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 8, max= 136, per=25.17%, avg=56.39, stdev=23.88, samples=176 iops : min= 2, max= 34, avg=13.63, stdev= 5.99, samples=176 write: IOPS=14, BW=58.2KiB/s (59.6kB/s)(5192KiB/89169msec); 0 zone resets slat (usec): min=30, max=6464, avg=200.29, stdev=400.81 clat (msec): min=247, max=1677, avg=747.03, stdev=198.61 lat (msec): min=247, max=1677, avg=747.24, stdev=198.60 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 447], 10.00th=[ 502], 20.00th=[ 575], | 30.00th=[ 642], 40.00th=[ 693], 50.00th=[ 735], 60.00th=[ 785], | 70.00th=[ 835], 80.00th=[ 902], 90.00th=[ 1011], 95.00th=[ 1099], | 99.00th=[ 1284], 99.50th=[ 1368], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 15, max= 117, per=24.85%, avg=57.90, stdev=19.49, samples=176 iops : min= 3, max= 29, avg=14.01, stdev= 4.90, samples=176 lat (msec) : 20=0.04%, 50=0.08%, 100=0.35%, 250=3.09%, 500=48.44% lat (msec) : 750=24.10%, 1000=18.28%, 2000=5.62% cpu : usr=0.07%, sys=0.70%, ctx=4391, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147803: Thu Oct 23 13:50:01 2025 read: IOPS=13, BW=55.7KiB/s (57.0kB/s)(4976KiB/89317msec) slat (usec): min=2772, max=38075, avg=10059.73, stdev=4377.75 clat (msec): min=10, max=706, avg=353.00, stdev=80.96 lat (msec): min=17, max=720, avg=363.07, stdev=82.26 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 241], 10.00th=[ 262], 20.00th=[ 292], | 30.00th=[ 309], 40.00th=[ 326], 50.00th=[ 342], 60.00th=[ 363], | 70.00th=[ 384], 80.00th=[ 414], 90.00th=[ 456], 95.00th=[ 502], | 99.00th=[ 575], 99.50th=[ 625], 99.90th=[ 701], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 7, max= 127, per=24.64%, avg=55.19, stdev=23.37, samples=177 iops : min= 1, max= 31, avg=13.34, stdev= 5.85, samples=177 write: IOPS=14, BW=58.9KiB/s (60.3kB/s)(5264KiB/89317msec); 0 zone resets slat (usec): min=30, max=4943, avg=195.99, stdev=363.17 clat (msec): min=247, max=1400, avg=741.03, stdev=199.45 lat (msec): min=247, max=1400, avg=741.23, stdev=199.45 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 447], 10.00th=[ 489], 20.00th=[ 558], | 30.00th=[ 625], 40.00th=[ 676], 50.00th=[ 735], 60.00th=[ 785], | 70.00th=[ 835], 80.00th=[ 911], 90.00th=[ 1011], 95.00th=[ 1099], | 99.00th=[ 1250], 99.50th=[ 1284], 99.90th=[ 1368], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 7, max= 143, per=25.01%, avg=58.27, stdev=21.97, samples=177 iops : min= 1, max= 35, avg=14.11, stdev= 5.50, samples=177 lat (msec) : 20=0.04%, 100=0.08%, 250=3.36%, 500=48.40%, 750=23.91% lat (msec) : 1000=18.71%, 2000=5.51% cpu : usr=0.11%, sys=0.70%, ctx=4765, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147804: Thu Oct 23 13:50:01 2025 read: IOPS=13, BW=55.8KiB/s (57.2kB/s)(4984KiB/89258msec) slat (usec): min=52, max=394073, avg=10606.13, stdev=14738.86 clat (msec): min=11, max=1001, avg=349.79, stdev=92.26 lat (msec): min=13, max=1009, avg=360.41, stdev=93.39 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 228], 10.00th=[ 255], 20.00th=[ 284], | 30.00th=[ 305], 40.00th=[ 330], 50.00th=[ 347], 60.00th=[ 363], | 70.00th=[ 384], 80.00th=[ 418], 90.00th=[ 456], 95.00th=[ 502], | 99.00th=[ 567], 99.50th=[ 625], 99.90th=[ 995], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 7, max= 119, per=24.79%, avg=55.53, stdev=24.03, samples=177 iops : min= 1, max= 29, avg=13.42, stdev= 6.00, samples=177 write: IOPS=14, BW=58.9KiB/s (60.3kB/s)(5256KiB/89258msec); 0 zone resets slat (usec): min=29, max=4574, avg=199.00, stdev=346.12 clat (msec): min=78, max=1584, avg=744.40, stdev=209.81 lat (msec): min=78, max=1585, avg=744.61, stdev=209.82 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 447], 10.00th=[ 502], 20.00th=[ 567], | 30.00th=[ 625], 40.00th=[ 684], 50.00th=[ 735], 60.00th=[ 785], | 70.00th=[ 827], 80.00th=[ 911], 90.00th=[ 1036], 95.00th=[ 1116], | 99.00th=[ 1318], 99.50th=[ 1385], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 8, max= 120, per=24.89%, avg=57.99, stdev=20.27, samples=177 iops : min= 2, max= 30, avg=14.04, stdev= 5.11, samples=177 lat (msec) : 20=0.23%, 50=0.16%, 100=0.31%, 250=3.91%, 500=46.88% lat (msec) : 750=24.49%, 1000=18.01%, 2000=6.02% cpu : usr=0.08%, sys=0.75%, ctx=4855, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147805: Thu Oct 23 13:50:01 2025 read: IOPS=14, BW=56.7KiB/s (58.0kB/s)(5060KiB/89260msec) slat (usec): min=2357, max=42503, avg=10126.56, stdev=4588.45 clat (msec): min=16, max=897, avg=353.71, stdev=92.57 lat (msec): min=26, max=907, avg=363.84, stdev=93.72 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 230], 10.00th=[ 257], 20.00th=[ 284], | 30.00th=[ 305], 40.00th=[ 330], 50.00th=[ 347], 60.00th=[ 368], | 70.00th=[ 388], 80.00th=[ 414], 90.00th=[ 468], 95.00th=[ 514], | 99.00th=[ 617], 99.50th=[ 693], 99.90th=[ 885], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 8, max= 134, per=25.28%, avg=56.64, stdev=23.43, samples=176 iops : min= 2, max= 33, avg=13.70, stdev= 5.86, samples=176 write: IOPS=14, BW=58.0KiB/s (59.4kB/s)(5180KiB/89260msec); 0 zone resets slat (usec): min=30, max=13493, avg=220.12, stdev=608.93 clat (msec): min=264, max=1502, avg=745.64, stdev=207.55 lat (msec): min=264, max=1503, avg=745.86, stdev=207.54 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 439], 10.00th=[ 485], 20.00th=[ 558], | 30.00th=[ 625], 40.00th=[ 684], 50.00th=[ 726], 60.00th=[ 776], | 70.00th=[ 844], 80.00th=[ 911], 90.00th=[ 1003], 95.00th=[ 1116], | 99.00th=[ 1385], 99.50th=[ 1435], 99.90th=[ 1502], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 15, max= 119, per=24.61%, avg=57.34, stdev=21.05, samples=177 iops : min= 3, max= 29, avg=13.88, stdev= 5.28, samples=177 lat (msec) : 20=0.04%, 50=0.08%, 100=0.51%, 250=3.67%, 500=48.12% lat (msec) : 750=24.06%, 1000=18.16%, 2000=5.35% cpu : usr=0.10%, sys=0.68%, ctx=4721, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=225KiB/s (230kB/s), 55.7KiB/s-56.7KiB/s (57.0kB/s-58.0kB/s), io=19.6MiB (20.5MB), run=89169-89317msec WRITE: bw=234KiB/s (240kB/s), 58.0KiB/s-58.9KiB/s (59.4kB/s-60.3kB/s), io=20.4MiB (21.4MB), run=89169-89317msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147822: Thu Oct 23 13:50:06 2025 read: IOPS=3, BW=137MiB/s (143MB/s)(40.0MiB/293msec) slat (nsec): min=22244k, max=22244k, avg=22243964.00, stdev= 0.00 clat (nsec): min=253753k, max=253753k, avg=253753319.00, stdev= 0.00 lat (nsec): min=276002k, max=276002k, avg=276002193.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 253], 5.00th=[ 253], 10.00th=[ 253], 20.00th=[ 253], | 30.00th=[ 253], 40.00th=[ 253], 50.00th=[ 253], 60.00th=[ 253], | 70.00th=[ 253], 80.00th=[ 253], 90.00th=[ 253], 95.00th=[ 253], | 99.00th=[ 253], 99.50th=[ 253], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 253] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.08%, ctx=42, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=137MiB/s (143MB/s), 137MiB/s-137MiB/s (143MB/s-143MB/s), io=40.0MiB (41.9MB), run=293-293msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout