== sanity test 398c: run fio to test AIO ================= 03:31:01 (1789025461) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.657399 s, 63.8 MB/s osc.lustre-OST0000-osc-ffff991b513f7800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146249: Thu Sep 10 03:34:11 2026 write: IOPS=14, BW=57.6KiB/s (58.9kB/s)(10.0MiB/177905msec); 0 zone resets slat (usec): min=24, max=18681, avg=224.28, stdev=1021.80 clat (msec): min=210, max=2505, avg=1110.48, stdev=358.51 lat (msec): min=212, max=2505, avg=1110.71, stdev=358.50 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 659], 10.00th=[ 751], 20.00th=[ 852], | 30.00th=[ 911], 40.00th=[ 978], 50.00th=[ 1045], 60.00th=[ 1116], | 70.00th=[ 1200], 80.00th=[ 1351], 90.00th=[ 1586], 95.00th=[ 1821], | 99.00th=[ 2265], 99.50th=[ 2400], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 172, per=25.50%, avg=58.65, stdev=33.51, samples=342 iops : min= 1, max= 43, avg=14.21, stdev= 8.43, samples=342 lat (msec) : 250=0.62%, 500=1.25%, 750=7.81%, 1000=34.18%, 2000=52.85% lat (msec) : >=2000=3.28% cpu : usr=0.05%, sys=0.20%, ctx=1822, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146250: Thu Sep 10 03:34:11 2026 write: IOPS=14, BW=57.7KiB/s (59.0kB/s)(10.0MiB/177621msec); 0 zone resets slat (usec): min=23, max=21279, avg=258.16, stdev=1231.03 clat (msec): min=144, max=2531, avg=1109.24, stdev=358.01 lat (msec): min=144, max=2531, avg=1109.50, stdev=357.98 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 651], 10.00th=[ 743], 20.00th=[ 844], | 30.00th=[ 919], 40.00th=[ 978], 50.00th=[ 1062], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1351], 90.00th=[ 1569], 95.00th=[ 1804], | 99.00th=[ 2299], 99.50th=[ 2366], 99.90th=[ 2534], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 7, max= 198, per=25.26%, avg=58.10, stdev=29.88, samples=345 iops : min= 1, max= 49, avg=14.07, stdev= 7.49, samples=345 lat (msec) : 250=0.86%, 500=1.13%, 750=8.20%, 1000=32.27%, 2000=54.30% lat (msec) : >=2000=3.24% cpu : usr=0.05%, sys=0.21%, ctx=1826, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146251: Thu Sep 10 03:34:11 2026 write: IOPS=14, BW=57.6KiB/s (58.0kB/s)(10.0MiB/177742msec); 0 zone resets slat (usec): min=22, max=19264, avg=212.18, stdev=981.49 clat (msec): min=165, max=2616, avg=1109.34, stdev=358.02 lat (msec): min=165, max=2616, avg=1109.57, stdev=358.01 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 676], 10.00th=[ 751], 20.00th=[ 852], | 30.00th=[ 911], 40.00th=[ 995], 50.00th=[ 1062], 60.00th=[ 1116], | 70.00th=[ 1217], 80.00th=[ 1334], 90.00th=[ 1586], 95.00th=[ 1804], | 99.00th=[ 2333], 99.50th=[ 2400], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 172, per=25.96%, avg=59.71, stdev=30.96, samples=336 iops : min= 1, max= 43, avg=14.48, stdev= 7.76, samples=336 lat (msec) : 250=1.29%, 500=1.25%, 750=7.54%, 1000=30.70%, 2000=56.25% lat (msec) : >=2000=2.97% cpu : usr=0.04%, sys=0.19%, ctx=1609, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146252: Thu Sep 10 03:34:11 2026 write: IOPS=14, BW=57.6KiB/s (59.0kB/s)(10.0MiB/177687msec); 0 zone resets slat (usec): min=22, max=23297, avg=242.41, stdev=1045.70 clat (msec): min=169, max=2617, avg=1108.84, stdev=357.94 lat (msec): min=169, max=2618, avg=1109.08, stdev=357.94 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 659], 10.00th=[ 751], 20.00th=[ 844], | 30.00th=[ 919], 40.00th=[ 986], 50.00th=[ 1062], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1334], 90.00th=[ 1569], 95.00th=[ 1821], | 99.00th=[ 2299], 99.50th=[ 2400], 99.90th=[ 2467], 99.95th=[ 2500], | 99.99th=[ 2635] bw ( KiB/s): min= 7, max= 172, per=25.35%, avg=58.31, stdev=27.85, samples=344 iops : min= 1, max= 43, avg=14.12, stdev= 6.99, samples=344 lat (msec) : 250=0.98%, 500=1.33%, 750=7.66%, 1000=32.23%, 2000=54.61% lat (msec) : >=2000=3.20% cpu : usr=0.04%, sys=0.22%, ctx=2109, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=230KiB/s (236kB/s), 57.6KiB/s-57.7KiB/s (58.9kB/s-59.0kB/s), io=40.0MiB (41.9MB), run=177621-177905msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146288: Thu Sep 10 03:36:02 2026 read: IOPS=12, BW=49.4KiB/s (50.6kB/s)(5048KiB/102094msec) slat (usec): min=3222, max=52033, avg=13852.33, stdev=7792.42 clat (msec): min=22, max=929, avg=395.54, stdev=92.15 lat (msec): min=35, max=961, avg=409.40, stdev=93.06 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 275], 10.00th=[ 296], 20.00th=[ 326], | 30.00th=[ 347], 40.00th=[ 368], 50.00th=[ 384], 60.00th=[ 405], | 70.00th=[ 430], 80.00th=[ 460], 90.00th=[ 510], 95.00th=[ 558], | 99.00th=[ 676], 99.50th=[ 718], 99.90th=[ 927], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 7, max= 111, per=25.09%, avg=49.17, stdev=22.46, samples=202 iops : min= 1, max= 27, avg=11.87, stdev= 5.64, samples=202 write: IOPS=12, BW=50.9KiB/s (52.1kB/s)(5192KiB/102094msec); 0 zone resets slat (usec): min=29, max=9633, avg=209.77, stdev=606.63 clat (msec): min=209, max=1932, avg=858.06, stdev=231.52 lat (msec): min=209, max=1932, avg=858.27, stdev=231.54 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 531], 10.00th=[ 600], 20.00th=[ 676], | 30.00th=[ 718], 40.00th=[ 776], 50.00th=[ 827], 60.00th=[ 902], | 70.00th=[ 961], 80.00th=[ 1045], 90.00th=[ 1150], 95.00th=[ 1234], | 99.00th=[ 1603], 99.50th=[ 1821], 99.90th=[ 1921], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 7, max= 103, per=24.59%, avg=50.16, stdev=20.18, samples=203 iops : min= 1, max= 25, avg=12.11, stdev= 5.08, samples=203 lat (msec) : 50=0.08%, 100=0.04%, 250=1.37%, 500=43.67%, 750=21.80% lat (msec) : 1000=20.70%, 2000=12.34% cpu : usr=0.10%, sys=0.62%, ctx=4809, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146289: Thu Sep 10 03:36:02 2026 read: IOPS=12, BW=48.6KiB/s (49.8kB/s)(4976KiB/102310msec) slat (msec): min=2, max=659, avg=14.81, stdev=20.00 clat (msec): min=49, max=1167, avg=392.31, stdev=94.45 lat (msec): min=55, max=1185, avg=407.12, stdev=95.69 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 271], 10.00th=[ 292], 20.00th=[ 326], | 30.00th=[ 347], 40.00th=[ 363], 50.00th=[ 384], 60.00th=[ 405], | 70.00th=[ 426], 80.00th=[ 451], 90.00th=[ 498], 95.00th=[ 542], | 99.00th=[ 667], 99.50th=[ 827], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 7, max= 119, per=24.99%, avg=48.97, stdev=21.76, samples=199 iops : min= 1, max= 29, avg=11.81, stdev= 5.47, samples=199 write: IOPS=12, BW=51.5KiB/s (52.7kB/s)(5264KiB/102310msec); 0 zone resets slat (usec): min=28, max=19071, avg=242.83, stdev=911.52 clat (msec): min=212, max=2042, avg=856.97, stdev=233.78 lat (msec): min=212, max=2042, avg=857.22, stdev=233.76 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 542], 10.00th=[ 609], 20.00th=[ 667], | 30.00th=[ 726], 40.00th=[ 776], 50.00th=[ 835], 60.00th=[ 894], | 70.00th=[ 961], 80.00th=[ 1036], 90.00th=[ 1116], 95.00th=[ 1234], | 99.00th=[ 1636], 99.50th=[ 1787], 99.90th=[ 1938], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 112, per=24.89%, avg=50.77, stdev=20.95, samples=203 iops : min= 1, max= 28, avg=12.27, stdev= 5.28, samples=203 lat (msec) : 50=0.04%, 100=0.20%, 250=1.09%, 500=44.61%, 750=20.00% lat (msec) : 1000=21.21%, 2000=12.81%, >=2000=0.04% cpu : usr=0.11%, sys=0.65%, ctx=4502, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146290: Thu Sep 10 03:36:02 2026 read: IOPS=12, BW=48.8KiB/s (49.9kB/s)(4984KiB/102211msec) slat (usec): min=38, max=52893, avg=14035.10, stdev=7912.43 clat (msec): min=27, max=855, avg=393.60, stdev=90.59 lat (msec): min=33, max=864, avg=407.64, stdev=91.93 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 264], 10.00th=[ 296], 20.00th=[ 326], | 30.00th=[ 351], 40.00th=[ 368], 50.00th=[ 384], 60.00th=[ 405], | 70.00th=[ 426], 80.00th=[ 460], 90.00th=[ 506], 95.00th=[ 550], | 99.00th=[ 651], 99.50th=[ 743], 99.90th=[ 835], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 7, max= 120, per=24.93%, avg=48.87, stdev=22.87, samples=201 iops : min= 1, max= 30, avg=11.80, stdev= 5.74, samples=201 write: IOPS=12, BW=51.4KiB/s (52.7kB/s)(5256KiB/102211msec); 0 zone resets slat (usec): min=32, max=8077, avg=203.72, stdev=500.21 clat (msec): min=164, max=2015, avg=857.18, stdev=234.87 lat (msec): min=164, max=2015, avg=857.38, stdev=234.88 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 542], 10.00th=[ 609], 20.00th=[ 676], | 30.00th=[ 718], 40.00th=[ 776], 50.00th=[ 835], 60.00th=[ 894], | 70.00th=[ 961], 80.00th=[ 1036], 90.00th=[ 1133], 95.00th=[ 1234], | 99.00th=[ 1603], 99.50th=[ 1687], 99.90th=[ 1955], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 104, per=24.85%, avg=50.70, stdev=20.39, samples=203 iops : min= 1, max= 26, avg=12.25, stdev= 5.15, samples=203 lat (msec) : 50=0.20%, 100=0.12%, 250=1.52%, 500=43.44%, 750=21.45% lat (msec) : 1000=20.62%, 2000=12.62%, >=2000=0.04% cpu : usr=0.11%, sys=0.61%, ctx=4619, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146291: Thu Sep 10 03:36:02 2026 read: IOPS=12, BW=49.6KiB/s (50.8kB/s)(5060KiB/102095msec) slat (usec): min=3142, max=59166, avg=14332.75, stdev=7853.61 clat (msec): min=15, max=917, avg=395.61, stdev=91.90 lat (msec): min=28, max=933, avg=409.95, stdev=92.95 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 266], 10.00th=[ 292], 20.00th=[ 330], | 30.00th=[ 347], 40.00th=[ 368], 50.00th=[ 388], 60.00th=[ 409], | 70.00th=[ 430], 80.00th=[ 460], 90.00th=[ 506], 95.00th=[ 550], | 99.00th=[ 676], 99.50th=[ 768], 99.90th=[ 844], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 7, max= 104, per=25.30%, avg=49.58, stdev=19.79, samples=201 iops : min= 1, max= 26, avg=11.97, stdev= 5.00, samples=201 write: IOPS=12, BW=50.7KiB/s (51.0kB/s)(5180KiB/102095msec); 0 zone resets slat (usec): min=33, max=18316, avg=255.43, stdev=926.72 clat (msec): min=191, max=2036, avg=858.75, stdev=238.22 lat (msec): min=195, max=2036, avg=859.01, stdev=238.23 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 531], 10.00th=[ 584], 20.00th=[ 667], | 30.00th=[ 726], 40.00th=[ 776], 50.00th=[ 844], 60.00th=[ 894], | 70.00th=[ 961], 80.00th=[ 1036], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1586], 99.50th=[ 1687], 99.90th=[ 2022], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 111, per=24.63%, avg=50.24, stdev=18.55, samples=202 iops : min= 1, max= 27, avg=12.14, stdev= 4.68, samples=202 lat (msec) : 20=0.04%, 50=0.08%, 100=0.04%, 250=1.45%, 500=43.63% lat (msec) : 750=21.64%, 1000=20.66%, 2000=12.38%, >=2000=0.08% cpu : usr=0.12%, sys=0.66%, ctx=5257, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=196KiB/s (201kB/s), 48.6KiB/s-49.6KiB/s (49.8kB/s-50.8kB/s), io=19.6MiB (20.5MB), run=102094-102310msec WRITE: bw=204KiB/s (209kB/s), 50.7KiB/s-51.5KiB/s (51.0kB/s-52.7kB/s), io=20.4MiB (21.4MB), run=102094-102310msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146310: Thu Sep 10 03:36:14 2026 read: IOPS=1, BW=65.8MiB/s (68.0MB/s)(40.0MiB/608msec) slat (nsec): min=30560k, max=30560k, avg=30560101.00, stdev= 0.00 clat (nsec): min=548479k, max=548479k, avg=548479025.00, stdev= 0.00 lat (nsec): min=579043k, max=579043k, avg=579043100.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 550], 5.00th=[ 550], 10.00th=[ 550], 20.00th=[ 550], | 30.00th=[ 550], 40.00th=[ 550], 50.00th=[ 550], 60.00th=[ 550], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] lat (msec) : 750=100.00% cpu : usr=0.16%, sys=1.98%, ctx=7, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=65.8MiB/s (68.0MB/s), 65.8MiB/s-65.8MiB/s (68.0MB/s-68.0MB/s), io=40.0MiB (41.9MB), run=608-608msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout