== sanity test 398c: run fio to test AIO ================= 07:51:11 (1764939071) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.435078 s, 96.4 MB/s osc.lustre-OST0000-osc-ffff9b5a50fd9000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=124020: Fri Dec 5 07:53:43 2025 write: IOPS=17, BW=70.9KiB/s (72.6kB/s)(10.0MiB/144504msec); 0 zone resets slat (usec): min=24, max=8582, avg=228.28, stdev=515.31 clat (msec): min=309, max=1630, avg=902.14, stdev=199.62 lat (msec): min=309, max=1630, avg=902.37, stdev=199.64 clat percentiles (msec): | 1.00th=[ 498], 5.00th=[ 592], 10.00th=[ 651], 20.00th=[ 735], | 30.00th=[ 793], 40.00th=[ 852], 50.00th=[ 894], 60.00th=[ 944], | 70.00th=[ 1003], 80.00th=[ 1062], 90.00th=[ 1150], 95.00th=[ 1217], | 99.00th=[ 1452], 99.50th=[ 1536], 99.90th=[ 1552], 99.95th=[ 1552], | 99.99th=[ 1636] bw ( KiB/s): min= 15, max= 128, per=25.12%, avg=71.09, stdev=27.68, samples=283 iops : min= 3, max= 32, avg=17.22, stdev= 6.94, samples=283 lat (msec) : 500=1.02%, 750=20.86%, 1000=48.24%, 2000=29.88% cpu : usr=0.06%, sys=0.30%, ctx=1472, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=124021: Fri Dec 5 07:53:43 2025 write: IOPS=17, BW=71.0KiB/s (72.7kB/s)(10.0MiB/144186msec); 0 zone resets slat (usec): min=24, max=6666, avg=220.01, stdev=497.85 clat (msec): min=221, max=1630, avg=900.36, stdev=199.48 lat (msec): min=222, max=1630, avg=900.59, stdev=199.50 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 592], 10.00th=[ 651], 20.00th=[ 735], | 30.00th=[ 793], 40.00th=[ 852], 50.00th=[ 902], 60.00th=[ 944], | 70.00th=[ 995], 80.00th=[ 1062], 90.00th=[ 1150], 95.00th=[ 1234], | 99.00th=[ 1435], 99.50th=[ 1469], 99.90th=[ 1552], 99.95th=[ 1552], | 99.99th=[ 1636] bw ( KiB/s): min= 7, max= 136, per=25.39%, avg=71.85, stdev=30.65, samples=280 iops : min= 1, max= 34, avg=17.41, stdev= 7.69, samples=280 lat (msec) : 250=0.62%, 500=0.74%, 750=21.25%, 1000=48.71%, 2000=28.67% cpu : usr=0.05%, sys=0.30%, ctx=1425, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=124022: Fri Dec 5 07:53:43 2025 write: IOPS=17, BW=70.9KiB/s (72.6kB/s)(10.0MiB/144351msec); 0 zone resets slat (usec): min=23, max=9244, avg=228.37, stdev=568.24 clat (msec): min=167, max=1701, avg=901.54, stdev=206.33 lat (msec): min=167, max=1701, avg=901.77, stdev=206.35 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 584], 10.00th=[ 651], 20.00th=[ 743], | 30.00th=[ 793], 40.00th=[ 844], 50.00th=[ 894], 60.00th=[ 944], | 70.00th=[ 995], 80.00th=[ 1070], 90.00th=[ 1167], 95.00th=[ 1234], | 99.00th=[ 1452], 99.50th=[ 1502], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 128, per=24.85%, avg=70.34, stdev=27.28, samples=286 iops : min= 1, max= 32, avg=17.03, stdev= 6.88, samples=286 lat (msec) : 250=0.62%, 500=0.90%, 750=20.43%, 1000=48.95%, 2000=29.10% cpu : usr=0.06%, sys=0.28%, ctx=1382, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=124023: Fri Dec 5 07:53:43 2025 write: IOPS=17, BW=70.9KiB/s (72.6kB/s)(10.0MiB/144505msec); 0 zone resets slat (usec): min=21, max=8410, avg=233.77, stdev=525.27 clat (msec): min=192, max=1700, avg=902.13, stdev=198.54 lat (msec): min=193, max=1700, avg=902.36, stdev=198.54 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 592], 10.00th=[ 659], 20.00th=[ 735], | 30.00th=[ 793], 40.00th=[ 852], 50.00th=[ 902], 60.00th=[ 953], | 70.00th=[ 1003], 80.00th=[ 1062], 90.00th=[ 1133], 95.00th=[ 1217], | 99.00th=[ 1452], 99.50th=[ 1519], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 144, per=25.04%, avg=70.85, stdev=26.80, samples=284 iops : min= 1, max= 36, avg=17.16, stdev= 6.72, samples=284 lat (msec) : 250=0.04%, 500=1.45%, 750=20.47%, 1000=47.97%, 2000=30.08% cpu : usr=0.04%, sys=0.30%, ctx=1536, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=283KiB/s (290kB/s), 70.9KiB/s-71.0KiB/s (72.6kB/s-72.7kB/s), io=40.0MiB (41.9MB), run=144186-144505msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=124058: Fri Dec 5 07:55:17 2025 read: IOPS=14, BW=57.6KiB/s (59.0kB/s)(5048KiB/87608msec) slat (usec): min=25, max=36111, avg=9490.18, stdev=4472.84 clat (msec): min=14, max=713, avg=361.58, stdev=88.56 lat (msec): min=20, max=724, avg=371.07, stdev=89.47 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 230], 10.00th=[ 259], 20.00th=[ 288], | 30.00th=[ 313], 40.00th=[ 338], 50.00th=[ 359], 60.00th=[ 380], | 70.00th=[ 405], 80.00th=[ 435], 90.00th=[ 477], 95.00th=[ 506], | 99.00th=[ 567], 99.50th=[ 609], 99.90th=[ 701], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 7, max= 128, per=24.86%, avg=56.92, stdev=24.08, samples=174 iops : min= 1, max= 32, avg=13.69, stdev= 6.07, samples=174 write: IOPS=14, BW=59.3KiB/s (60.7kB/s)(5192KiB/87608msec); 0 zone resets slat (usec): min=24, max=12457, avg=271.56, stdev=650.55 clat (msec): min=182, max=1296, avg=718.03, stdev=187.34 lat (msec): min=182, max=1297, avg=718.30, stdev=187.39 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 439], 10.00th=[ 489], 20.00th=[ 550], | 30.00th=[ 617], 40.00th=[ 651], 50.00th=[ 701], 60.00th=[ 751], | 70.00th=[ 810], 80.00th=[ 885], 90.00th=[ 969], 95.00th=[ 1036], | 99.00th=[ 1183], 99.50th=[ 1250], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 15, max= 143, per=24.55%, avg=58.42, stdev=20.41, samples=174 iops : min= 3, max= 35, avg=14.06, stdev= 5.09, samples=174 lat (msec) : 20=0.08%, 50=0.12%, 100=0.08%, 250=3.91%, 500=48.12% lat (msec) : 750=27.11%, 1000=16.84%, 2000=3.75% cpu : usr=0.14%, sys=0.86%, ctx=3325, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=124059: Fri Dec 5 07:55:17 2025 read: IOPS=14, BW=56.8KiB/s (58.2kB/s)(4976KiB/87608msec) slat (usec): min=2434, max=31678, avg=9645.40, stdev=4257.59 clat (msec): min=8, max=638, avg=357.17, stdev=83.64 lat (msec): min=16, max=655, avg=366.82, stdev=84.52 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 230], 10.00th=[ 257], 20.00th=[ 288], | 30.00th=[ 313], 40.00th=[ 334], 50.00th=[ 351], 60.00th=[ 376], | 70.00th=[ 401], 80.00th=[ 426], 90.00th=[ 468], 95.00th=[ 498], | 99.00th=[ 558], 99.50th=[ 567], 99.90th=[ 617], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 7, max= 111, per=24.48%, avg=56.07, stdev=25.02, samples=174 iops : min= 1, max= 27, avg=13.48, stdev= 6.30, samples=174 write: IOPS=15, BW=60.1KiB/s (61.5kB/s)(5264KiB/87608msec); 0 zone resets slat (usec): min=27, max=13352, avg=297.65, stdev=804.41 clat (msec): min=233, max=1309, avg=717.10, stdev=182.80 lat (msec): min=234, max=1309, avg=717.40, stdev=182.81 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 435], 10.00th=[ 493], 20.00th=[ 558], | 30.00th=[ 609], 40.00th=[ 651], 50.00th=[ 701], 60.00th=[ 751], | 70.00th=[ 810], 80.00th=[ 885], 90.00th=[ 969], 95.00th=[ 1020], | 99.00th=[ 1150], 99.50th=[ 1183], 99.90th=[ 1301], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 8, max= 127, per=24.92%, avg=59.30, stdev=23.20, samples=174 iops : min= 2, max= 31, avg=14.29, stdev= 5.75, samples=174 lat (msec) : 10=0.04%, 100=0.08%, 250=3.98%, 500=47.54%, 750=27.81% lat (msec) : 1000=16.99%, 2000=3.55% cpu : usr=0.11%, sys=0.81%, ctx=3104, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=124060: Fri Dec 5 07:55:17 2025 read: IOPS=14, BW=56.9KiB/s (58.3kB/s)(4984KiB/87608msec) slat (usec): min=2360, max=47456, avg=9679.64, stdev=4386.93 clat (msec): min=7, max=639, avg=358.16, stdev=84.89 lat (msec): min=15, max=657, avg=367.85, stdev=85.86 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 230], 10.00th=[ 257], 20.00th=[ 292], | 30.00th=[ 313], 40.00th=[ 334], 50.00th=[ 355], 60.00th=[ 376], | 70.00th=[ 397], 80.00th=[ 426], 90.00th=[ 468], 95.00th=[ 502], | 99.00th=[ 558], 99.50th=[ 584], 99.90th=[ 609], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 8, max= 127, per=24.58%, avg=56.30, stdev=25.02, samples=174 iops : min= 2, max= 31, avg=13.53, stdev= 6.26, samples=174 write: IOPS=14, BW=59.0KiB/s (61.4kB/s)(5256KiB/87608msec); 0 zone resets slat (usec): min=26, max=8254, avg=275.59, stdev=564.60 clat (msec): min=85, max=1304, avg=716.38, stdev=184.01 lat (msec): min=86, max=1304, avg=716.66, stdev=184.03 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 447], 10.00th=[ 493], 20.00th=[ 567], | 30.00th=[ 617], 40.00th=[ 659], 50.00th=[ 701], 60.00th=[ 743], | 70.00th=[ 802], 80.00th=[ 885], 90.00th=[ 969], 95.00th=[ 1036], | 99.00th=[ 1183], 99.50th=[ 1200], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 15, max= 112, per=24.81%, avg=59.04, stdev=20.62, samples=174 iops : min= 3, max= 28, avg=14.22, stdev= 5.18, samples=174 lat (msec) : 10=0.04%, 20=0.04%, 50=0.04%, 100=0.23%, 250=4.18% lat (msec) : 500=47.27%, 750=28.12%, 1000=16.25%, 2000=3.83% cpu : usr=0.13%, sys=0.80%, ctx=3108, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=124061: Fri Dec 5 07:55:17 2025 read: IOPS=14, BW=57.9KiB/s (59.3kB/s)(5060KiB/87341msec) slat (usec): min=32, max=35626, avg=9539.17, stdev=4395.65 clat (msec): min=14, max=892, avg=360.55, stdev=86.71 lat (msec): min=19, max=900, avg=370.10, stdev=87.69 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 228], 10.00th=[ 259], 20.00th=[ 292], | 30.00th=[ 313], 40.00th=[ 334], 50.00th=[ 359], 60.00th=[ 380], | 70.00th=[ 405], 80.00th=[ 430], 90.00th=[ 468], 95.00th=[ 502], | 99.00th=[ 567], 99.50th=[ 592], 99.90th=[ 885], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 7, max= 119, per=25.20%, avg=57.70, stdev=23.77, samples=172 iops : min= 1, max= 29, avg=13.88, stdev= 5.97, samples=172 write: IOPS=14, BW=59.3KiB/s (60.7kB/s)(5180KiB/87341msec); 0 zone resets slat (usec): min=26, max=15159, avg=266.90, stdev=655.61 clat (msec): min=101, max=1437, avg=716.65, stdev=193.11 lat (msec): min=101, max=1437, avg=716.92, stdev=193.13 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 405], 10.00th=[ 477], 20.00th=[ 558], | 30.00th=[ 609], 40.00th=[ 659], 50.00th=[ 709], 60.00th=[ 768], | 70.00th=[ 827], 80.00th=[ 885], 90.00th=[ 969], 95.00th=[ 1028], | 99.00th=[ 1167], 99.50th=[ 1183], 99.90th=[ 1284], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 7, max= 119, per=24.62%, avg=58.58, stdev=21.24, samples=173 iops : min= 1, max= 29, avg=14.10, stdev= 5.31, samples=173 lat (msec) : 20=0.04%, 100=0.04%, 250=4.45%, 500=48.32%, 750=25.62% lat (msec) : 1000=18.05%, 2000=3.48% cpu : usr=0.12%, sys=0.80%, ctx=3075, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=229KiB/s (235kB/s), 56.8KiB/s-57.9KiB/s (58.2kB/s-59.3kB/s), io=19.6MiB (20.5MB), run=87341-87608msec WRITE: bw=238KiB/s (244kB/s), 59.3KiB/s-60.1KiB/s (60.7kB/s-61.5kB/s), io=20.4MiB (21.4MB), run=87341-87608msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=124079: Fri Dec 5 07:55:23 2025 read: IOPS=2, BW=105MiB/s (110MB/s)(40.0MiB/382msec) slat (nsec): min=53161k, max=53161k, avg=53160932.00, stdev= 0.00 clat (nsec): min=292174k, max=292174k, avg=292174325.00, stdev= 0.00 lat (nsec): min=345342k, max=345342k, avg=345341681.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 292], 10.00th=[ 292], 20.00th=[ 292], | 30.00th=[ 292], 40.00th=[ 292], 50.00th=[ 292], 60.00th=[ 292], | 70.00th=[ 292], 80.00th=[ 292], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=4.46%, ctx=2, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=105MiB/s (110MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=40.0MiB (41.9MB), run=382-382msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck