== sanity test 398c: run fio to test AIO ================= 15:37:59 (1764794279) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.243118 s, 173 MB/s osc.lustre-OST0000-osc-ffff9804d875c000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=151995: Wed Dec 3 15:39:03 2025 write: IOPS=43, BW=174KiB/s (178kB/s)(10.0MiB/58867msec); 0 zone resets slat (usec): min=28, max=1803, avg=132.23, stdev=125.35 clat (msec): min=84, max=765, avg=367.28, stdev=109.75 lat (msec): min=84, max=765, avg=367.41, stdev=109.74 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 232], 10.00th=[ 257], 20.00th=[ 284], | 30.00th=[ 305], 40.00th=[ 326], 50.00th=[ 347], 60.00th=[ 368], | 70.00th=[ 397], 80.00th=[ 443], 90.00th=[ 510], 95.00th=[ 600], | 99.00th=[ 735], 99.50th=[ 760], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 23, max= 280, per=25.15%, avg=174.79, stdev=47.75, samples=116 iops : min= 5, max= 70, avg=43.42, stdev=12.07, samples=116 lat (msec) : 100=0.04%, 250=8.59%, 500=80.39%, 750=10.27%, 1000=0.70% cpu : usr=0.09%, sys=0.46%, ctx=2766, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151996: Wed Dec 3 15:39:03 2025 write: IOPS=43, BW=174KiB/s (178kB/s)(10.0MiB/58931msec); 0 zone resets slat (usec): min=28, max=4100, avg=138.08, stdev=161.22 clat (msec): min=129, max=810, avg=367.57, stdev=108.75 lat (msec): min=129, max=810, avg=367.72, stdev=108.75 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 232], 10.00th=[ 253], 20.00th=[ 284], | 30.00th=[ 300], 40.00th=[ 321], 50.00th=[ 342], 60.00th=[ 368], | 70.00th=[ 401], 80.00th=[ 447], 90.00th=[ 518], 95.00th=[ 575], | 99.00th=[ 743], 99.50th=[ 776], 99.90th=[ 810], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 32, max= 272, per=24.95%, avg=173.38, stdev=47.61, samples=117 iops : min= 8, max= 68, avg=43.07, stdev=11.99, samples=117 lat (msec) : 250=9.18%, 500=78.63%, 750=11.52%, 1000=0.66% cpu : usr=0.09%, sys=0.51%, ctx=3100, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151997: Wed Dec 3 15:39:03 2025 write: IOPS=43, BW=174KiB/s (178kB/s)(10.0MiB/58847msec); 0 zone resets slat (usec): min=28, max=6146, avg=144.76, stdev=237.90 clat (msec): min=78, max=806, avg=367.43, stdev=110.57 lat (msec): min=78, max=806, avg=367.58, stdev=110.58 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 232], 10.00th=[ 253], 20.00th=[ 279], | 30.00th=[ 300], 40.00th=[ 326], 50.00th=[ 342], 60.00th=[ 368], | 70.00th=[ 401], 80.00th=[ 447], 90.00th=[ 518], 95.00th=[ 592], | 99.00th=[ 718], 99.50th=[ 735], 99.90th=[ 802], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 55, max= 304, per=24.93%, avg=173.27, stdev=48.05, samples=117 iops : min= 13, max= 76, avg=43.04, stdev=12.14, samples=117 lat (msec) : 100=0.12%, 250=9.26%, 500=78.40%, 750=11.91%, 1000=0.31% cpu : usr=0.08%, sys=0.50%, ctx=3041, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=151998: Wed Dec 3 15:39:03 2025 write: IOPS=43, BW=174KiB/s (178kB/s)(10.0MiB/58901msec); 0 zone resets slat (usec): min=27, max=3748, avg=140.36, stdev=166.55 clat (msec): min=149, max=841, avg=367.40, stdev=108.68 lat (msec): min=149, max=841, avg=367.55, stdev=108.68 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 234], 10.00th=[ 255], 20.00th=[ 279], | 30.00th=[ 300], 40.00th=[ 321], 50.00th=[ 342], 60.00th=[ 368], | 70.00th=[ 401], 80.00th=[ 447], 90.00th=[ 510], 95.00th=[ 584], | 99.00th=[ 735], 99.50th=[ 751], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 15, max= 272, per=24.95%, avg=173.44, stdev=49.28, samples=117 iops : min= 3, max= 68, avg=43.09, stdev=12.41, samples=117 lat (msec) : 250=8.75%, 500=79.73%, 750=10.94%, 1000=0.59% cpu : usr=0.09%, sys=0.50%, ctx=3028, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=695KiB/s (712kB/s), 174KiB/s-174KiB/s (178kB/s-178kB/s), io=40.0MiB (41.9MB), run=58847-58931msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=152016: Wed Dec 3 15:39:36 2025 read: IOPS=43, BW=173KiB/s (177kB/s)(5048KiB/29253msec) slat (usec): min=1593, max=11232, avg=3795.98, stdev=1198.61 clat (msec): min=5, max=228, avg=116.30, stdev=26.99 lat (msec): min=8, max=235, avg=120.10, stdev=27.35 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 81], 10.00th=[ 86], 20.00th=[ 95], | 30.00th=[ 103], 40.00th=[ 108], 50.00th=[ 114], 60.00th=[ 121], | 70.00th=[ 127], 80.00th=[ 138], 90.00th=[ 150], 95.00th=[ 167], | 99.00th=[ 190], 99.50th=[ 205], 99.90th=[ 224], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 87, max= 264, per=25.13%, avg=172.41, stdev=42.54, samples=58 iops : min= 21, max= 66, avg=43.00, stdev=10.73, samples=58 write: IOPS=44, BW=177KiB/s (182kB/s)(5192KiB/29253msec); 0 zone resets slat (usec): min=36, max=3924, avg=164.22, stdev=170.86 clat (msec): min=95, max=495, avg=242.61, stdev=58.82 lat (msec): min=95, max=495, avg=242.78, stdev=58.82 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 150], 10.00th=[ 171], 20.00th=[ 197], | 30.00th=[ 211], 40.00th=[ 226], 50.00th=[ 241], 60.00th=[ 255], | 70.00th=[ 268], 80.00th=[ 284], 90.00th=[ 313], 95.00th=[ 342], | 99.00th=[ 430], 99.50th=[ 456], 99.90th=[ 493], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 80, max= 264, per=24.78%, avg=176.93, stdev=35.87, samples=58 iops : min= 20, max= 66, avg=44.12, stdev= 9.06, samples=58 lat (msec) : 10=0.04%, 20=0.12%, 50=0.20%, 100=12.81%, 250=65.00% lat (msec) : 500=21.84% cpu : usr=0.24%, sys=1.71%, ctx=5007, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=152017: Wed Dec 3 15:39:36 2025 read: IOPS=42, BW=171KiB/s (175kB/s)(4976KiB/29081msec) slat (usec): min=34, max=10541, avg=3737.46, stdev=1249.07 clat (msec): min=10, max=340, avg=116.58, stdev=30.83 lat (msec): min=12, max=344, avg=120.32, stdev=31.15 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 79], 10.00th=[ 86], 20.00th=[ 93], | 30.00th=[ 102], 40.00th=[ 108], 50.00th=[ 114], 60.00th=[ 121], | 70.00th=[ 127], 80.00th=[ 136], 90.00th=[ 150], 95.00th=[ 169], | 99.00th=[ 194], 99.50th=[ 271], 99.90th=[ 338], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 55, max= 241, per=24.90%, avg=170.82, stdev=43.86, samples=57 iops : min= 13, max= 60, avg=42.60, stdev=11.05, samples=57 write: IOPS=45, BW=181KiB/s (185kB/s)(5264KiB/29081msec); 0 zone resets slat (usec): min=32, max=1416, avg=156.87, stdev=97.00 clat (msec): min=50, max=478, avg=239.27, stdev=61.55 lat (msec): min=51, max=478, avg=239.43, stdev=61.55 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 150], 10.00th=[ 169], 20.00th=[ 190], | 30.00th=[ 211], 40.00th=[ 224], 50.00th=[ 236], 60.00th=[ 251], | 70.00th=[ 266], 80.00th=[ 284], 90.00th=[ 309], 95.00th=[ 334], | 99.00th=[ 447], 99.50th=[ 460], 99.90th=[ 477], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 104, max= 296, per=25.40%, avg=181.39, stdev=34.95, samples=57 iops : min= 26, max= 74, avg=45.23, stdev= 8.82, samples=57 lat (msec) : 20=0.35%, 50=0.08%, 100=14.14%, 250=64.34%, 500=21.09% cpu : usr=0.27%, sys=1.67%, ctx=4873, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=152018: Wed Dec 3 15:39:36 2025 read: IOPS=42, BW=170KiB/s (175kB/s)(4984KiB/29236msec) slat (usec): min=1462, max=11592, avg=3806.53, stdev=1275.39 clat (msec): min=4, max=279, avg=115.97, stdev=27.31 lat (msec): min=7, max=282, avg=119.79, stdev=27.64 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 79], 10.00th=[ 87], 20.00th=[ 94], | 30.00th=[ 103], 40.00th=[ 109], 50.00th=[ 115], 60.00th=[ 121], | 70.00th=[ 127], 80.00th=[ 134], 90.00th=[ 150], 95.00th=[ 163], | 99.00th=[ 194], 99.50th=[ 209], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 55, max= 288, per=24.90%, avg=170.81, stdev=46.88, samples=58 iops : min= 13, max= 72, avg=42.59, stdev=11.78, samples=58 write: IOPS=44, BW=180KiB/s (184kB/s)(5256KiB/29236msec); 0 zone resets slat (usec): min=34, max=1078, avg=156.20, stdev=89.53 clat (msec): min=51, max=516, avg=241.28, stdev=59.38 lat (msec): min=51, max=516, avg=241.43, stdev=59.38 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 148], 10.00th=[ 171], 20.00th=[ 194], | 30.00th=[ 213], 40.00th=[ 226], 50.00th=[ 241], 60.00th=[ 251], | 70.00th=[ 266], 80.00th=[ 284], 90.00th=[ 309], 95.00th=[ 338], | 99.00th=[ 439], 99.50th=[ 481], 99.90th=[ 506], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 87, max= 328, per=25.02%, avg=178.64, stdev=39.78, samples=58 iops : min= 21, max= 82, avg=44.55, stdev=10.04, samples=58 lat (msec) : 10=0.08%, 20=0.08%, 50=0.23%, 100=13.09%, 250=65.35% lat (msec) : 500=21.05%, 750=0.12% cpu : usr=0.24%, sys=1.71%, ctx=4892, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=152019: Wed Dec 3 15:39:36 2025 read: IOPS=43, BW=173KiB/s (177kB/s)(5060KiB/29236msec) slat (usec): min=1355, max=11564, avg=3743.79, stdev=1220.29 clat (msec): min=28, max=246, avg=117.40, stdev=26.39 lat (msec): min=33, max=250, avg=121.15, stdev=26.73 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 81], 10.00th=[ 87], 20.00th=[ 96], | 30.00th=[ 103], 40.00th=[ 109], 50.00th=[ 115], 60.00th=[ 121], | 70.00th=[ 128], 80.00th=[ 136], 90.00th=[ 155], 95.00th=[ 169], | 99.00th=[ 192], 99.50th=[ 194], 99.90th=[ 211], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min= 87, max= 256, per=25.23%, avg=173.09, stdev=40.21, samples=58 iops : min= 21, max= 64, avg=43.16, stdev=10.15, samples=58 write: IOPS=44, BW=177KiB/s (181kB/s)(5180KiB/29236msec); 0 zone resets slat (usec): min=28, max=1934, avg=160.38, stdev=115.06 clat (msec): min=81, max=515, avg=241.62, stdev=57.67 lat (msec): min=82, max=516, avg=241.78, stdev=57.68 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 153], 10.00th=[ 178], 20.00th=[ 197], | 30.00th=[ 211], 40.00th=[ 226], 50.00th=[ 236], 60.00th=[ 251], | 70.00th=[ 264], 80.00th=[ 284], 90.00th=[ 309], 95.00th=[ 334], | 99.00th=[ 439], 99.50th=[ 481], 99.90th=[ 514], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 103, max= 264, per=24.70%, avg=176.38, stdev=33.59, samples=58 iops : min= 25, max= 66, avg=43.98, stdev= 8.48, samples=58 lat (msec) : 50=0.04%, 100=13.48%, 250=66.21%, 500=20.20%, 750=0.08% cpu : usr=0.24%, sys=1.67%, ctx=4882, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=686KiB/s (702kB/s), 170KiB/s-173KiB/s (175kB/s-177kB/s), io=19.6MiB (20.5MB), run=29081-29253msec WRITE: bw=714KiB/s (731kB/s), 177KiB/s-181KiB/s (181kB/s-185kB/s), io=20.4MiB (21.4MB), run=29081-29253msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=152026: Wed Dec 3 15:39:41 2025 read: IOPS=6, BW=265MiB/s (278MB/s)(40.0MiB/151msec) slat (nsec): min=15982k, max=15982k, avg=15981668.00, stdev= 0.00 clat (nsec): min=127416k, max=127416k, avg=127415932.00, stdev= 0.00 lat (nsec): min=143403k, max=143403k, avg=143402758.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 128], 10.00th=[ 128], 20.00th=[ 128], | 30.00th=[ 128], 40.00th=[ 128], 50.00th=[ 128], 60.00th=[ 128], | 70.00th=[ 128], 80.00th=[ 128], 90.00th=[ 128], 95.00th=[ 128], | 99.00th=[ 128], 99.50th=[ 128], 99.90th=[ 128], 99.95th=[ 128], | 99.99th=[ 128] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=8.67%, ctx=42, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=265MiB/s (278MB/s), 265MiB/s-265MiB/s (278MB/s-278MB/s), io=40.0MiB (41.9MB), run=151-151msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout