== sanity test 398c: run fio to test AIO ================= 06:24:44 (1773224684) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.400314 s, 105 MB/s osc.lustre-OST0000-osc-ffff9ff60680c800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147310: Wed Mar 11 06:26:38 2026 write: IOPS=24, BW=96.4KiB/s (98.7kB/s)(10.0MiB/106240msec); 0 zone resets slat (usec): min=37, max=14589, avg=327.85, stdev=731.15 clat (msec): min=103, max=1257, avg=662.94, stdev=141.19 lat (msec): min=103, max=1257, avg=663.27, stdev=141.20 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 439], 10.00th=[ 498], 20.00th=[ 558], | 30.00th=[ 600], 40.00th=[ 634], 50.00th=[ 659], 60.00th=[ 693], | 70.00th=[ 726], 80.00th=[ 768], 90.00th=[ 835], 95.00th=[ 902], | 99.00th=[ 1036], 99.50th=[ 1116], 99.90th=[ 1200], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 16, max= 200, per=24.94%, avg=95.78, stdev=27.60, samples=211 iops : min= 4, max= 50, avg=23.53, stdev= 6.96, samples=211 lat (msec) : 250=0.35%, 500=10.39%, 750=64.53%, 1000=23.55%, 2000=1.17% cpu : usr=0.07%, sys=0.47%, ctx=2612, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147311: Wed Mar 11 06:26:38 2026 write: IOPS=24, BW=96.2KiB/s (98.5kB/s)(10.0MiB/106418msec); 0 zone resets slat (usec): min=37, max=15322, avg=322.24, stdev=663.71 clat (msec): min=255, max=1216, avg=664.07, stdev=140.07 lat (msec): min=257, max=1218, avg=664.39, stdev=140.11 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 414], 10.00th=[ 498], 20.00th=[ 558], | 30.00th=[ 600], 40.00th=[ 634], 50.00th=[ 659], 60.00th=[ 693], | 70.00th=[ 726], 80.00th=[ 768], 90.00th=[ 844], 95.00th=[ 894], | 99.00th=[ 1028], 99.50th=[ 1116], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 15, max= 192, per=24.87%, avg=95.51, stdev=29.32, samples=211 iops : min= 3, max= 48, avg=23.46, stdev= 7.38, samples=211 lat (msec) : 500=10.47%, 750=65.31%, 1000=23.12%, 2000=1.09% cpu : usr=0.06%, sys=0.46%, ctx=2645, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147312: Wed Mar 11 06:26:38 2026 write: IOPS=24, BW=96.5KiB/s (98.8kB/s)(10.0MiB/106079msec); 0 zone resets slat (usec): min=38, max=16879, avg=347.27, stdev=869.25 clat (msec): min=116, max=1261, avg=662.39, stdev=143.33 lat (msec): min=116, max=1261, avg=662.74, stdev=143.36 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 430], 10.00th=[ 502], 20.00th=[ 558], | 30.00th=[ 600], 40.00th=[ 634], 50.00th=[ 659], 60.00th=[ 693], | 70.00th=[ 726], 80.00th=[ 768], 90.00th=[ 844], 95.00th=[ 894], | 99.00th=[ 1045], 99.50th=[ 1150], 99.90th=[ 1217], 99.95th=[ 1250], | 99.99th=[ 1267] bw ( KiB/s): min= 15, max= 255, per=25.05%, avg=96.19, stdev=32.83, samples=210 iops : min= 3, max= 63, avg=23.63, stdev= 8.25, samples=210 lat (msec) : 250=0.62%, 500=9.34%, 750=67.27%, 1000=21.64%, 2000=1.13% cpu : usr=0.07%, sys=0.46%, ctx=2612, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147313: Wed Mar 11 06:26:38 2026 write: IOPS=24, BW=96.3KiB/s (98.6kB/s)(10.0MiB/106317msec); 0 zone resets slat (usec): min=36, max=9744, avg=319.12, stdev=655.20 clat (msec): min=241, max=1237, avg=663.39, stdev=138.61 lat (msec): min=241, max=1237, avg=663.71, stdev=138.63 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 430], 10.00th=[ 493], 20.00th=[ 558], | 30.00th=[ 600], 40.00th=[ 634], 50.00th=[ 659], 60.00th=[ 693], | 70.00th=[ 726], 80.00th=[ 768], 90.00th=[ 835], 95.00th=[ 894], | 99.00th=[ 1020], 99.50th=[ 1116], 99.90th=[ 1217], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 15, max= 199, per=24.93%, avg=95.74, stdev=29.55, samples=211 iops : min= 3, max= 49, avg=23.52, stdev= 7.45, samples=211 lat (msec) : 250=0.04%, 500=10.27%, 750=65.20%, 1000=23.44%, 2000=1.05% cpu : usr=0.05%, sys=0.48%, ctx=2383, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=385KiB/s (394kB/s), 96.2KiB/s-96.5KiB/s (98.5kB/s-98.8kB/s), io=40.0MiB (41.9MB), run=106079-106418msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147338: Wed Mar 11 06:27:52 2026 read: IOPS=18, BW=74.0KiB/s (75.8kB/s)(5048KiB/68199msec) slat (usec): min=2362, max=39091, avg=9273.83, stdev=4477.96 clat (msec): min=26, max=619, avg=260.40, stdev=59.61 lat (msec): min=32, max=625, avg=269.68, stdev=60.14 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 176], 10.00th=[ 192], 20.00th=[ 215], | 30.00th=[ 230], 40.00th=[ 243], 50.00th=[ 257], 60.00th=[ 271], | 70.00th=[ 288], 80.00th=[ 309], 90.00th=[ 334], 95.00th=[ 363], | 99.00th=[ 405], 99.50th=[ 443], 99.90th=[ 550], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 23, max= 152, per=25.16%, avg=73.71, stdev=27.02, samples=135 iops : min= 5, max= 38, avg=18.04, stdev= 6.81, samples=135 write: IOPS=19, BW=76.1KiB/s (77.0kB/s)(5192KiB/68199msec); 0 zone resets slat (usec): min=48, max=12692, avg=365.01, stdev=706.95 clat (msec): min=116, max=1048, avg=576.50, stdev=139.61 lat (msec): min=118, max=1049, avg=576.87, stdev=139.66 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 355], 10.00th=[ 409], 20.00th=[ 460], | 30.00th=[ 502], 40.00th=[ 535], 50.00th=[ 575], 60.00th=[ 609], | 70.00th=[ 651], 80.00th=[ 693], 90.00th=[ 760], 95.00th=[ 818], | 99.00th=[ 936], 99.50th=[ 953], 99.90th=[ 1036], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 23, max= 136, per=24.82%, avg=75.69, stdev=21.26, samples=135 iops : min= 5, max= 34, avg=18.53, stdev= 5.33, samples=135 lat (msec) : 50=0.16%, 100=0.31%, 250=22.42%, 500=41.45%, 750=30.16% lat (msec) : 1000=5.35%, 2000=0.16% cpu : usr=0.12%, sys=1.11%, ctx=4900, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147339: Wed Mar 11 06:27:52 2026 read: IOPS=18, BW=72.0KiB/s (74.7kB/s)(4976KiB/68198msec) slat (usec): min=67, max=37139, avg=9202.99, stdev=4305.87 clat (msec): min=22, max=618, avg=262.60, stdev=60.42 lat (msec): min=22, max=631, avg=271.81, stdev=61.11 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 178], 10.00th=[ 192], 20.00th=[ 215], | 30.00th=[ 232], 40.00th=[ 245], 50.00th=[ 259], 60.00th=[ 275], | 70.00th=[ 288], 80.00th=[ 313], 90.00th=[ 338], 95.00th=[ 363], | 99.00th=[ 414], 99.50th=[ 422], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 15, max= 143, per=24.77%, avg=72.58, stdev=27.87, samples=135 iops : min= 3, max= 35, avg=17.75, stdev= 7.01, samples=135 write: IOPS=19, BW=77.2KiB/s (79.0kB/s)(5264KiB/68198msec); 0 zone resets slat (usec): min=37, max=9557, avg=339.47, stdev=612.26 clat (msec): min=95, max=1015, avg=571.24, stdev=144.26 lat (msec): min=95, max=1015, avg=571.58, stdev=144.26 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 342], 10.00th=[ 401], 20.00th=[ 456], | 30.00th=[ 493], 40.00th=[ 531], 50.00th=[ 567], 60.00th=[ 609], | 70.00th=[ 651], 80.00th=[ 701], 90.00th=[ 751], 95.00th=[ 802], | 99.00th=[ 894], 99.50th=[ 927], 99.90th=[ 1003], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 15, max= 160, per=25.15%, avg=76.71, stdev=25.51, samples=135 iops : min= 3, max= 40, avg=18.79, stdev= 6.41, samples=135 lat (msec) : 50=0.23%, 100=0.16%, 250=22.07%, 500=42.34%, 750=29.88% lat (msec) : 1000=5.23%, 2000=0.08% cpu : usr=0.15%, sys=1.05%, ctx=4794, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147340: Wed Mar 11 06:27:52 2026 read: IOPS=18, BW=72.9KiB/s (74.7kB/s)(4984KiB/68350msec) slat (usec): min=2359, max=31286, avg=9295.96, stdev=4412.42 clat (msec): min=11, max=451, avg=262.33, stdev=59.33 lat (msec): min=19, max=460, avg=271.63, stdev=60.03 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 174], 10.00th=[ 190], 20.00th=[ 215], | 30.00th=[ 232], 40.00th=[ 245], 50.00th=[ 262], 60.00th=[ 275], | 70.00th=[ 292], 80.00th=[ 309], 90.00th=[ 338], 95.00th=[ 359], | 99.00th=[ 405], 99.50th=[ 409], 99.90th=[ 439], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 15, max= 144, per=24.61%, avg=72.10, stdev=27.05, samples=135 iops : min= 3, max= 36, avg=17.63, stdev= 6.82, samples=135 write: IOPS=19, BW=76.9KiB/s (78.7kB/s)(5256KiB/68350msec); 0 zone resets slat (usec): min=39, max=20401, avg=359.86, stdev=774.64 clat (msec): min=59, max=1012, avg=572.86, stdev=147.14 lat (msec): min=59, max=1012, avg=573.22, stdev=147.14 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 342], 10.00th=[ 401], 20.00th=[ 460], | 30.00th=[ 502], 40.00th=[ 535], 50.00th=[ 567], 60.00th=[ 609], | 70.00th=[ 651], 80.00th=[ 701], 90.00th=[ 760], 95.00th=[ 802], | 99.00th=[ 927], 99.50th=[ 978], 99.90th=[ 1011], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 23, max= 158, per=24.72%, avg=75.41, stdev=23.61, samples=135 iops : min= 5, max= 39, avg=18.47, stdev= 5.91, samples=135 lat (msec) : 20=0.04%, 50=0.20%, 100=0.62%, 250=20.66%, 500=42.30% lat (msec) : 750=30.47%, 1000=5.47%, 2000=0.23% cpu : usr=0.13%, sys=1.10%, ctx=5073, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147341: Wed Mar 11 06:27:52 2026 read: IOPS=18, BW=74.3KiB/s (76.1kB/s)(5060KiB/68110msec) slat (usec): min=1886, max=40282, avg=9178.20, stdev=4408.00 clat (msec): min=12, max=717, avg=259.36, stdev=61.73 lat (msec): min=18, max=730, avg=268.55, stdev=62.37 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 176], 10.00th=[ 190], 20.00th=[ 213], | 30.00th=[ 230], 40.00th=[ 245], 50.00th=[ 257], 60.00th=[ 271], | 70.00th=[ 288], 80.00th=[ 305], 90.00th=[ 330], 95.00th=[ 355], | 99.00th=[ 409], 99.50th=[ 447], 99.90th=[ 718], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 8, max= 144, per=25.46%, avg=74.59, stdev=27.22, samples=134 iops : min= 2, max= 36, avg=18.25, stdev= 6.82, samples=134 write: IOPS=19, BW=76.1KiB/s (77.9kB/s)(5180KiB/68110msec); 0 zone resets slat (usec): min=41, max=8133, avg=342.25, stdev=556.61 clat (msec): min=117, max=1048, avg=576.94, stdev=142.45 lat (msec): min=117, max=1048, avg=577.28, stdev=142.45 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 351], 10.00th=[ 405], 20.00th=[ 460], | 30.00th=[ 502], 40.00th=[ 535], 50.00th=[ 575], 60.00th=[ 609], | 70.00th=[ 651], 80.00th=[ 693], 90.00th=[ 760], 95.00th=[ 827], | 99.00th=[ 944], 99.50th=[ 1011], 99.90th=[ 1045], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 24, max= 135, per=24.72%, avg=75.41, stdev=22.31, samples=135 iops : min= 6, max= 33, avg=18.46, stdev= 5.60, samples=135 lat (msec) : 20=0.12%, 50=0.23%, 100=0.12%, 250=22.38%, 500=41.64% lat (msec) : 750=30.20%, 1000=5.00%, 2000=0.31% cpu : usr=0.14%, sys=1.05%, ctx=5141, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=294KiB/s (301kB/s), 72.9KiB/s-74.3KiB/s (74.7kB/s-76.1kB/s), io=19.6MiB (20.5MB), run=68110-68350msec WRITE: bw=306KiB/s (313kB/s), 76.1KiB/s-77.2KiB/s (77.9kB/s-79.0kB/s), io=20.4MiB (21.4MB), run=68110-68350msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147357: Wed Mar 11 06:27:59 2026 read: IOPS=3, BW=143MiB/s (150MB/s)(40.0MiB/280msec) slat (nsec): min=22381k, max=22381k, avg=22380637.00, stdev= 0.00 clat (nsec): min=243413k, max=243413k, avg=243412561.00, stdev= 0.00 lat (nsec): min=265797k, max=265797k, avg=265797081.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 245], 10.00th=[ 245], 20.00th=[ 245], | 30.00th=[ 245], 40.00th=[ 245], 50.00th=[ 245], 60.00th=[ 245], | 70.00th=[ 245], 80.00th=[ 245], 90.00th=[ 245], 95.00th=[ 245], | 99.00th=[ 245], 99.50th=[ 245], 99.90th=[ 245], 99.95th=[ 245], | 99.99th=[ 245] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=5.38%, ctx=43, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=143MiB/s (150MB/s), 143MiB/s-143MiB/s (150MB/s-150MB/s), io=40.0MiB (41.9MB), run=280-280msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout