== sanity test 398c: run fio to test AIO ================= 01:42:18 (1777009338) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.703707 s, 59.6 MB/s osc.lustre-OST0000-osc-ffff94d305649800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=181255: Fri Apr 24 01:43:35 2026 write: IOPS=42, BW=172KiB/s (176kB/s)(10.0MiB/59632msec); 0 zone resets slat (usec): min=55, max=24158, avg=786.21, stdev=1778.72 clat (msec): min=57, max=597, avg=370.25, stdev=65.22 lat (msec): min=73, max=597, avg=371.05, stdev=65.22 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 262], 10.00th=[ 296], 20.00th=[ 317], | 30.00th=[ 338], 40.00th=[ 355], 50.00th=[ 363], 60.00th=[ 380], | 70.00th=[ 397], 80.00th=[ 422], 90.00th=[ 468], 95.00th=[ 489], | 99.00th=[ 527], 99.50th=[ 527], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 600] bw ( KiB/s): min= 115, max= 255, per=24.92%, avg=170.95, stdev=43.24, samples=118 iops : min= 28, max= 63, avg=42.19, stdev=10.84, samples=118 lat (msec) : 100=0.04%, 250=3.32%, 500=93.01%, 750=3.63% cpu : usr=0.15%, sys=1.16%, ctx=2768, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181256: Fri Apr 24 01:43:35 2026 write: IOPS=43, BW=173KiB/s (177kB/s)(10.0MiB/59249msec); 0 zone resets slat (usec): min=59, max=19941, avg=760.26, stdev=1580.65 clat (msec): min=38, max=602, avg=369.10, stdev=67.27 lat (msec): min=42, max=603, avg=369.88, stdev=67.28 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 259], 10.00th=[ 292], 20.00th=[ 317], | 30.00th=[ 338], 40.00th=[ 355], 50.00th=[ 368], 60.00th=[ 380], | 70.00th=[ 401], 80.00th=[ 422], 90.00th=[ 456], 95.00th=[ 481], | 99.00th=[ 531], 99.50th=[ 558], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 600] bw ( KiB/s): min= 126, max= 294, per=25.11%, avg=172.22, stdev=36.37, samples=117 iops : min= 31, max= 73, avg=42.51, stdev= 9.15, samples=117 lat (msec) : 50=0.04%, 250=3.63%, 500=93.75%, 750=2.58% cpu : usr=0.18%, sys=1.24%, ctx=2746, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181257: Fri Apr 24 01:43:35 2026 write: IOPS=43, BW=172KiB/s (176kB/s)(10.0MiB/59492msec); 0 zone resets slat (usec): min=62, max=21419, avg=708.24, stdev=1510.06 clat (msec): min=89, max=631, avg=369.99, stdev=67.87 lat (msec): min=90, max=631, avg=370.71, stdev=67.91 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 271], 10.00th=[ 292], 20.00th=[ 317], | 30.00th=[ 334], 40.00th=[ 351], 50.00th=[ 363], 60.00th=[ 380], | 70.00th=[ 401], 80.00th=[ 430], 90.00th=[ 464], 95.00th=[ 485], | 99.00th=[ 550], 99.50th=[ 567], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 31, max= 255, per=24.91%, avg=170.86, stdev=45.65, samples=118 iops : min= 7, max= 63, avg=42.16, stdev=11.45, samples=118 lat (msec) : 100=0.04%, 250=3.09%, 500=94.77%, 750=2.11% cpu : usr=0.16%, sys=1.26%, ctx=2707, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181258: Fri Apr 24 01:43:35 2026 write: IOPS=42, BW=172KiB/s (176kB/s)(10.0MiB/59556msec); 0 zone resets slat (usec): min=81, max=29389, avg=759.74, stdev=1628.87 clat (msec): min=41, max=609, avg=369.98, stdev=66.67 lat (msec): min=41, max=610, avg=370.75, stdev=66.62 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 275], 10.00th=[ 296], 20.00th=[ 317], | 30.00th=[ 338], 40.00th=[ 351], 50.00th=[ 363], 60.00th=[ 380], | 70.00th=[ 397], 80.00th=[ 422], 90.00th=[ 460], 95.00th=[ 481], | 99.00th=[ 542], 99.50th=[ 584], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 111, max= 263, per=24.92%, avg=170.92, stdev=38.25, samples=118 iops : min= 27, max= 65, avg=42.19, stdev= 9.61, samples=118 lat (msec) : 50=0.08%, 100=0.04%, 250=3.40%, 500=93.52%, 750=2.97% cpu : usr=0.17%, sys=1.16%, ctx=2626, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=687KiB/s (703kB/s), 172KiB/s-173KiB/s (176kB/s-177kB/s), io=40.0MiB (41.9MB), run=59249-59632msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=181279: Fri Apr 24 01:44:48 2026 read: IOPS=23, BW=92.6KiB/s (94.8kB/s)(5048KiB/54534msec) slat (usec): min=5900, max=84724, avg=21861.38, stdev=9629.99 clat (msec): min=45, max=555, avg=323.44, stdev=69.40 lat (msec): min=72, max=589, avg=345.32, stdev=71.04 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 220], 10.00th=[ 243], 20.00th=[ 264], | 30.00th=[ 284], 40.00th=[ 300], 50.00th=[ 317], 60.00th=[ 338], | 70.00th=[ 359], 80.00th=[ 380], 90.00th=[ 418], 95.00th=[ 451], | 99.00th=[ 493], 99.50th=[ 518], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 15, max= 176, per=24.95%, avg=91.58, stdev=25.60, samples=108 iops : min= 3, max= 44, avg=22.26, stdev= 6.46, samples=108 write: IOPS=23, BW=95.2KiB/s (97.5kB/s)(5192KiB/54534msec); 0 zone resets slat (usec): min=97, max=22674, avg=749.39, stdev=1407.27 clat (msec): min=111, max=523, avg=330.42, stdev=64.47 lat (msec): min=112, max=524, avg=331.19, stdev=64.57 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 228], 10.00th=[ 251], 20.00th=[ 275], | 30.00th=[ 296], 40.00th=[ 313], 50.00th=[ 326], 60.00th=[ 342], | 70.00th=[ 359], 80.00th=[ 380], 90.00th=[ 418], 95.00th=[ 451], | 99.00th=[ 498], 99.50th=[ 510], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 23, max= 182, per=24.90%, avg=95.38, stdev=26.25, samples=107 iops : min= 5, max= 45, avg=23.21, stdev= 6.56, samples=107 lat (msec) : 50=0.04%, 250=11.25%, 500=87.93%, 750=0.78% cpu : usr=0.25%, sys=1.89%, ctx=4901, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181280: Fri Apr 24 01:44:48 2026 read: IOPS=22, BW=91.8KiB/s (94.0kB/s)(4976KiB/54183msec) slat (usec): min=138, max=94813, avg=21764.51, stdev=9970.26 clat (msec): min=49, max=572, avg=322.16, stdev=70.67 lat (msec): min=50, max=585, avg=343.94, stdev=72.41 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 222], 10.00th=[ 241], 20.00th=[ 264], | 30.00th=[ 279], 40.00th=[ 296], 50.00th=[ 313], 60.00th=[ 334], | 70.00th=[ 355], 80.00th=[ 380], 90.00th=[ 414], 95.00th=[ 447], | 99.00th=[ 527], 99.50th=[ 542], 99.90th=[ 567], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 39, max= 159, per=24.81%, avg=91.05, stdev=25.20, samples=107 iops : min= 9, max= 39, avg=22.12, stdev= 6.35, samples=107 write: IOPS=24, BW=97.2KiB/s (99.5kB/s)(5264KiB/54183msec); 0 zone resets slat (usec): min=85, max=19775, avg=667.27, stdev=1169.34 clat (msec): min=114, max=566, avg=332.10, stdev=66.33 lat (msec): min=115, max=566, avg=332.78, stdev=66.33 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 236], 10.00th=[ 255], 20.00th=[ 279], | 30.00th=[ 296], 40.00th=[ 313], 50.00th=[ 326], 60.00th=[ 342], | 70.00th=[ 359], 80.00th=[ 384], 90.00th=[ 422], 95.00th=[ 456], | 99.00th=[ 502], 99.50th=[ 510], 99.90th=[ 550], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 40, max= 174, per=25.25%, avg=96.72, stdev=26.95, samples=107 iops : min= 10, max= 43, avg=23.54, stdev= 6.77, samples=107 lat (msec) : 50=0.04%, 100=0.08%, 250=10.90%, 500=87.58%, 750=1.41% cpu : usr=0.25%, sys=1.88%, ctx=4868, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181281: Fri Apr 24 01:44:48 2026 read: IOPS=22, BW=91.5KiB/s (93.7kB/s)(4984KiB/54465msec) slat (usec): min=5467, max=83717, avg=21272.67, stdev=9644.74 clat (msec): min=40, max=598, avg=319.45, stdev=68.45 lat (msec): min=53, max=632, avg=340.75, stdev=69.93 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 218], 10.00th=[ 234], 20.00th=[ 259], | 30.00th=[ 279], 40.00th=[ 296], 50.00th=[ 317], 60.00th=[ 334], | 70.00th=[ 355], 80.00th=[ 376], 90.00th=[ 409], 95.00th=[ 439], | 99.00th=[ 477], 99.50th=[ 510], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 7, max= 166, per=24.73%, avg=90.76, stdev=28.30, samples=108 iops : min= 1, max= 41, avg=22.06, stdev= 7.11, samples=108 write: IOPS=24, BW=96.5KiB/s (98.8kB/s)(5256KiB/54465msec); 0 zone resets slat (usec): min=96, max=22263, avg=685.50, stdev=1276.76 clat (msec): min=166, max=575, avg=334.92, stdev=65.24 lat (msec): min=167, max=576, avg=335.62, stdev=65.31 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 239], 10.00th=[ 257], 20.00th=[ 284], | 30.00th=[ 300], 40.00th=[ 313], 50.00th=[ 330], 60.00th=[ 342], | 70.00th=[ 359], 80.00th=[ 384], 90.00th=[ 426], 95.00th=[ 460], | 99.00th=[ 518], 99.50th=[ 523], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 15, max= 158, per=24.89%, avg=95.32, stdev=26.28, samples=108 iops : min= 3, max= 39, avg=23.19, stdev= 6.61, samples=108 lat (msec) : 50=0.04%, 250=11.91%, 500=87.07%, 750=0.98% cpu : usr=0.27%, sys=1.85%, ctx=4923, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181282: Fri Apr 24 01:44:48 2026 read: IOPS=23, BW=92.8KiB/s (95.0kB/s)(5060KiB/54540msec) slat (usec): min=6423, max=81147, avg=21660.89, stdev=9571.34 clat (msec): min=95, max=557, avg=321.79, stdev=69.23 lat (msec): min=134, max=586, avg=343.47, stdev=71.10 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 218], 10.00th=[ 236], 20.00th=[ 262], | 30.00th=[ 284], 40.00th=[ 300], 50.00th=[ 317], 60.00th=[ 334], | 70.00th=[ 355], 80.00th=[ 380], 90.00th=[ 418], 95.00th=[ 447], | 99.00th=[ 498], 99.50th=[ 527], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 30, max= 143, per=25.05%, avg=91.92, stdev=23.76, samples=108 iops : min= 7, max= 35, avg=22.34, stdev= 5.98, samples=108 write: IOPS=23, BW=94.0KiB/s (97.3kB/s)(5180KiB/54540msec); 0 zone resets slat (usec): min=77, max=9993, avg=664.78, stdev=1033.05 clat (msec): min=162, max=534, avg=333.66, stdev=62.99 lat (msec): min=162, max=534, avg=334.33, stdev=62.96 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 236], 10.00th=[ 255], 20.00th=[ 279], | 30.00th=[ 300], 40.00th=[ 317], 50.00th=[ 330], 60.00th=[ 347], | 70.00th=[ 359], 80.00th=[ 380], 90.00th=[ 418], 95.00th=[ 451], | 99.00th=[ 498], 99.50th=[ 514], 99.90th=[ 523], 99.95th=[ 535], | 99.99th=[ 535] bw ( KiB/s): min= 38, max= 206, per=24.58%, avg=94.14, stdev=26.84, samples=108 iops : min= 9, max= 51, avg=22.90, stdev= 6.73, samples=108 lat (msec) : 100=0.04%, 250=11.25%, 500=87.89%, 750=0.82% cpu : usr=0.24%, sys=1.97%, ctx=4851, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=368KiB/s (377kB/s), 91.5KiB/s-92.8KiB/s (93.7kB/s-95.0kB/s), io=19.6MiB (20.5MB), run=54183-54540msec WRITE: bw=383KiB/s (392kB/s), 94.0KiB/s-97.2KiB/s (97.3kB/s-99.5kB/s), io=20.4MiB (21.4MB), run=54183-54540msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=181294: Fri Apr 24 01:45:05 2026 read: IOPS=1, BW=51.5MiB/s (54.1MB/s)(40.0MiB/776msec) slat (nsec): min=65648k, max=65648k, avg=65647630.00, stdev= 0.00 clat (nsec): min=659017k, max=659017k, avg=659016881.00, stdev= 0.00 lat (nsec): min=724672k, max=724672k, avg=724672166.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 659], 5.00th=[ 659], 10.00th=[ 659], 20.00th=[ 659], | 30.00th=[ 659], 40.00th=[ 659], 50.00th=[ 659], 60.00th=[ 659], | 70.00th=[ 659], 80.00th=[ 659], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.32%, ctx=5, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=51.5MiB/s (54.1MB/s), 51.5MiB/s-51.5MiB/s (54.1MB/s-54.1MB/s), io=40.0MiB (41.9MB), run=776-776msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout