== sanity test 398c: run fio to test AIO ================= 07:22:23 (1756812143) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.155488 s, 270 MB/s osc.lustre-OST0000-osc-ffff9ced201c8800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=152767: Tue Sep 2 07:23:25 2025 write: IOPS=44, BW=177KiB/s (181kB/s)(10.0MiB/57905msec); 0 zone resets slat (usec): min=24, max=3784, avg=152.67, stdev=167.36 clat (msec): min=89, max=777, avg=361.51, stdev=112.03 lat (msec): min=89, max=778, avg=361.66, stdev=112.04 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 197], 10.00th=[ 241], 20.00th=[ 279], | 30.00th=[ 300], 40.00th=[ 321], 50.00th=[ 347], 60.00th=[ 372], | 70.00th=[ 405], 80.00th=[ 443], 90.00th=[ 498], 95.00th=[ 558], | 99.00th=[ 718], 99.50th=[ 743], 99.90th=[ 768], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 72, max= 375, per=24.89%, avg=175.94, stdev=51.57, samples=115 iops : min= 18, max= 93, avg=43.72, stdev=12.95, samples=115 lat (msec) : 100=0.12%, 250=12.30%, 500=78.01%, 750=9.34%, 1000=0.23% cpu : usr=0.09%, sys=0.64%, ctx=1928, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=152768: Tue Sep 2 07:23:25 2025 write: IOPS=44, BW=177KiB/s (181kB/s)(10.0MiB/57905msec); 0 zone resets slat (usec): min=21, max=2906, avg=161.01, stdev=182.55 clat (msec): min=68, max=777, avg=361.47, stdev=112.49 lat (msec): min=68, max=777, avg=361.63, stdev=112.50 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 197], 10.00th=[ 236], 20.00th=[ 279], | 30.00th=[ 305], 40.00th=[ 321], 50.00th=[ 342], 60.00th=[ 372], | 70.00th=[ 405], 80.00th=[ 447], 90.00th=[ 502], 95.00th=[ 558], | 99.00th=[ 718], 99.50th=[ 743], 99.90th=[ 768], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 79, max= 369, per=24.86%, avg=175.73, stdev=52.01, samples=115 iops : min= 19, max= 92, avg=43.68, stdev=13.08, samples=115 lat (msec) : 100=0.04%, 250=12.89%, 500=76.88%, 750=9.77%, 1000=0.43% cpu : usr=0.09%, sys=0.65%, ctx=2059, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=152769: Tue Sep 2 07:23:25 2025 write: IOPS=44, BW=177KiB/s (181kB/s)(10.0MiB/57863msec); 0 zone resets slat (usec): min=26, max=2374, avg=149.10, stdev=151.44 clat (msec): min=25, max=801, avg=361.34, stdev=114.66 lat (msec): min=25, max=801, avg=361.50, stdev=114.67 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 190], 10.00th=[ 236], 20.00th=[ 279], | 30.00th=[ 305], 40.00th=[ 321], 50.00th=[ 342], 60.00th=[ 368], | 70.00th=[ 405], 80.00th=[ 447], 90.00th=[ 498], 95.00th=[ 558], | 99.00th=[ 726], 99.50th=[ 768], 99.90th=[ 776], 99.95th=[ 776], | 99.99th=[ 802] bw ( KiB/s): min= 63, max= 427, per=24.86%, avg=175.76, stdev=55.57, samples=115 iops : min= 15, max= 106, avg=43.68, stdev=13.95, samples=115 lat (msec) : 50=0.12%, 100=0.59%, 250=11.99%, 500=77.50%, 750=9.22% lat (msec) : 1000=0.59% cpu : usr=0.11%, sys=0.62%, ctx=1832, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=152770: Tue Sep 2 07:23:25 2025 write: IOPS=44, BW=177KiB/s (181kB/s)(10.0MiB/57906msec); 0 zone resets slat (usec): min=24, max=2012, avg=152.14, stdev=157.00 clat (msec): min=88, max=800, avg=361.51, stdev=112.49 lat (msec): min=88, max=800, avg=361.66, stdev=112.49 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 197], 10.00th=[ 232], 20.00th=[ 284], | 30.00th=[ 305], 40.00th=[ 321], 50.00th=[ 347], 60.00th=[ 372], | 70.00th=[ 405], 80.00th=[ 443], 90.00th=[ 502], 95.00th=[ 558], | 99.00th=[ 726], 99.50th=[ 743], 99.90th=[ 760], 99.95th=[ 768], | 99.99th=[ 802] bw ( KiB/s): min= 71, max= 388, per=24.86%, avg=175.77, stdev=52.29, samples=115 iops : min= 17, max= 97, avg=43.69, stdev=13.17, samples=115 lat (msec) : 100=0.04%, 250=12.70%, 500=77.07%, 750=10.00%, 1000=0.20% cpu : usr=0.10%, sys=0.63%, ctx=1994, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=707KiB/s (724kB/s), 177KiB/s-177KiB/s (181kB/s-181kB/s), io=40.0MiB (41.9MB), run=57863-57906msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=152787: Tue Sep 2 07:24:05 2025 read: IOPS=35, BW=141KiB/s (144kB/s)(5048KiB/35904msec) slat (usec): min=1250, max=13738, avg=4017.58, stdev=1770.46 clat (msec): min=6, max=378, avg=145.88, stdev=42.73 lat (msec): min=8, max=382, avg=149.90, stdev=43.18 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 93], 10.00th=[ 103], 20.00th=[ 113], | 30.00th=[ 123], 40.00th=[ 131], 50.00th=[ 142], 60.00th=[ 148], | 70.00th=[ 159], 80.00th=[ 178], 90.00th=[ 201], 95.00th=[ 224], | 99.00th=[ 279], 99.50th=[ 309], 99.90th=[ 376], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 55, max= 240, per=25.19%, avg=140.56, stdev=42.06, samples=71 iops : min= 13, max= 60, avg=34.90, stdev=10.57, samples=71 write: IOPS=36, BW=145KiB/s (148kB/s)(5192KiB/35904msec); 0 zone resets slat (usec): min=23, max=2976, avg=147.47, stdev=178.95 clat (msec): min=94, max=523, avg=295.61, stdev=76.11 lat (msec): min=94, max=524, avg=295.76, stdev=76.11 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 182], 10.00th=[ 199], 20.00th=[ 226], | 30.00th=[ 247], 40.00th=[ 268], 50.00th=[ 292], 60.00th=[ 317], | 70.00th=[ 342], 80.00th=[ 368], 90.00th=[ 397], 95.00th=[ 422], | 99.00th=[ 472], 99.50th=[ 489], 99.90th=[ 510], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 79, max= 216, per=24.87%, avg=144.51, stdev=33.41, samples=71 iops : min= 19, max= 54, avg=35.90, stdev= 8.46, samples=71 lat (msec) : 10=0.08%, 20=0.16%, 50=0.20%, 100=4.14%, 250=59.61% lat (msec) : 500=35.70%, 750=0.12% cpu : usr=0.19%, sys=1.39%, ctx=2936, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=152788: Tue Sep 2 07:24:05 2025 read: IOPS=34, BW=139KiB/s (142kB/s)(4976KiB/35908msec) slat (usec): min=1312, max=12597, avg=4019.85, stdev=1662.25 clat (msec): min=8, max=376, avg=146.20, stdev=41.29 lat (msec): min=13, max=382, avg=150.22, stdev=41.75 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 93], 10.00th=[ 104], 20.00th=[ 113], | 30.00th=[ 124], 40.00th=[ 131], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 176], 90.00th=[ 197], 95.00th=[ 220], | 99.00th=[ 284], 99.50th=[ 321], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 31, max= 256, per=24.76%, avg=138.17, stdev=48.07, samples=71 iops : min= 7, max= 64, avg=34.31, stdev=12.09, samples=71 write: IOPS=36, BW=147KiB/s (150kB/s)(5264KiB/35908msec); 0 zone resets slat (usec): min=26, max=3349, avg=153.03, stdev=200.39 clat (msec): min=113, max=521, avg=293.68, stdev=77.82 lat (msec): min=114, max=521, avg=293.83, stdev=77.83 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 178], 10.00th=[ 197], 20.00th=[ 222], | 30.00th=[ 245], 40.00th=[ 264], 50.00th=[ 288], 60.00th=[ 317], | 70.00th=[ 342], 80.00th=[ 368], 90.00th=[ 397], 95.00th=[ 430], | 99.00th=[ 477], 99.50th=[ 485], 99.90th=[ 514], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 72, max= 232, per=25.21%, avg=146.49, stdev=33.68, samples=71 iops : min= 18, max= 58, avg=36.38, stdev= 8.49, samples=71 lat (msec) : 10=0.08%, 20=0.04%, 50=0.12%, 100=3.67%, 250=61.05% lat (msec) : 500=34.96%, 750=0.08% cpu : usr=0.23%, sys=1.35%, ctx=2891, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=152789: Tue Sep 2 07:24:05 2025 read: IOPS=34, BW=139KiB/s (142kB/s)(4984KiB/35885msec) slat (usec): min=1240, max=15033, avg=4030.32, stdev=1773.54 clat (msec): min=5, max=355, avg=145.24, stdev=40.48 lat (msec): min=7, max=359, avg=149.27, stdev=41.01 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 96], 10.00th=[ 104], 20.00th=[ 112], | 30.00th=[ 122], 40.00th=[ 130], 50.00th=[ 140], 60.00th=[ 148], | 70.00th=[ 159], 80.00th=[ 176], 90.00th=[ 199], 95.00th=[ 220], | 99.00th=[ 288], 99.50th=[ 300], 99.90th=[ 330], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 48, max= 312, per=24.94%, avg=139.17, stdev=49.92, samples=71 iops : min= 12, max= 78, avg=34.56, stdev=12.59, samples=71 write: IOPS=36, BW=146KiB/s (150kB/s)(5256KiB/35885msec); 0 zone resets slat (usec): min=29, max=3648, avg=152.19, stdev=226.31 clat (msec): min=113, max=522, avg=294.21, stdev=77.21 lat (msec): min=113, max=522, avg=294.36, stdev=77.23 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 178], 10.00th=[ 199], 20.00th=[ 224], | 30.00th=[ 243], 40.00th=[ 264], 50.00th=[ 292], 60.00th=[ 313], | 70.00th=[ 342], 80.00th=[ 368], 90.00th=[ 401], 95.00th=[ 426], | 99.00th=[ 472], 99.50th=[ 489], 99.90th=[ 510], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 63, max= 240, per=25.11%, avg=145.89, stdev=37.60, samples=71 iops : min= 15, max= 60, avg=36.24, stdev= 9.43, samples=71 lat (msec) : 10=0.04%, 20=0.08%, 50=0.08%, 100=3.48%, 250=61.29% lat (msec) : 500=34.92%, 750=0.12% cpu : usr=0.19%, sys=1.36%, ctx=2934, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=152790: Tue Sep 2 07:24:05 2025 read: IOPS=35, BW=142KiB/s (146kB/s)(5060KiB/35580msec) slat (usec): min=47, max=12891, avg=4047.39, stdev=1822.37 clat (msec): min=11, max=487, avg=147.06, stdev=43.62 lat (msec): min=11, max=491, avg=151.11, stdev=44.02 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 96], 10.00th=[ 105], 20.00th=[ 114], | 30.00th=[ 123], 40.00th=[ 131], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 163], 80.00th=[ 176], 90.00th=[ 199], 95.00th=[ 222], | 99.00th=[ 284], 99.50th=[ 317], 99.90th=[ 464], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 64, max= 224, per=25.66%, avg=143.17, stdev=38.73, samples=70 iops : min= 16, max= 56, avg=35.57, stdev= 9.71, samples=70 write: IOPS=36, BW=146KiB/s (149kB/s)(5180KiB/35580msec); 0 zone resets slat (usec): min=22, max=1806, avg=150.87, stdev=160.88 clat (msec): min=51, max=659, avg=291.37, stdev=82.95 lat (msec): min=51, max=659, avg=291.52, stdev=82.95 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 174], 10.00th=[ 192], 20.00th=[ 220], | 30.00th=[ 241], 40.00th=[ 259], 50.00th=[ 284], 60.00th=[ 309], | 70.00th=[ 338], 80.00th=[ 363], 90.00th=[ 393], 95.00th=[ 430], | 99.00th=[ 514], 99.50th=[ 642], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 72, max= 264, per=25.04%, avg=145.47, stdev=40.56, samples=70 iops : min= 18, max= 66, avg=36.14, stdev=10.23, samples=70 lat (msec) : 20=0.12%, 50=0.27%, 100=3.05%, 250=62.70%, 500=33.28% lat (msec) : 750=0.59% cpu : usr=0.21%, sys=1.39%, ctx=2929, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=559KiB/s (572kB/s), 139KiB/s-142KiB/s (142kB/s-146kB/s), io=19.6MiB (20.5MB), run=35580-35908msec WRITE: bw=582KiB/s (596kB/s), 145KiB/s-147KiB/s (148kB/s-150kB/s), io=20.4MiB (21.4MB), run=35580-35908msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=152799: Tue Sep 2 07:24:10 2025 read: IOPS=5, BW=221MiB/s (232MB/s)(40.0MiB/181msec) slat (nsec): min=13108k, max=13108k, avg=13107699.00, stdev= 0.00 clat (nsec): min=160843k, max=160843k, avg=160842988.00, stdev= 0.00 lat (nsec): min=173955k, max=173955k, avg=173955143.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 161], | 70.00th=[ 161], 80.00th=[ 161], 90.00th=[ 161], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=7.78%, ctx=2, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=221MiB/s (232MB/s), 221MiB/s-221MiB/s (232MB/s-232MB/s), io=40.0MiB (41.9MB), run=181-181msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout