== sanity test 398c: run fio to test AIO ================= 11:42:15 (1736268135) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0915365 s, 458 MB/s osc.lustre-OST0000-osc-ffff8800b3ea6800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=17409: Tue Jan 7 11:42:19 2025 write: IOPS=1019, BW=4076KiB/s (4174kB/s)(10.0MiB/2512msec) slat (usec): min=12, max=301, avg=44.43, stdev=24.46 clat (usec): min=1663, max=34230, avg=15616.25, stdev=2861.76 lat (usec): min=1683, max=34251, avg=15662.08, stdev=2863.36 clat percentiles (usec): | 1.00th=[ 7111], 5.00th=[10814], 10.00th=[12518], 20.00th=[14091], | 30.00th=[14877], 40.00th=[15533], 50.00th=[15926], 60.00th=[16319], | 70.00th=[16712], 80.00th=[17171], 90.00th=[17957], 95.00th=[18744], | 99.00th=[25297], 99.50th=[30802], 99.90th=[32637], 99.95th=[32637], | 99.99th=[34341] bw ( KiB/s): min= 3552, max= 4688, per=25.12%, avg=4070.20, stdev=404.26, samples=5 iops : min= 888, max= 1172, avg=1017.40, stdev=101.08, samples=5 lat (msec) : 2=0.16%, 4=0.31%, 10=2.46%, 20=94.14%, 50=2.93% cpu : usr=0.88%, sys=4.86%, ctx=1661, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=17410: Tue Jan 7 11:42:19 2025 write: IOPS=1013, BW=4055KiB/s (4153kB/s)(10.0MiB/2525msec) slat (usec): min=13, max=209, avg=43.26, stdev=22.86 clat (usec): min=3838, max=28975, avg=15686.66, stdev=2596.62 lat (usec): min=3897, max=29015, avg=15731.14, stdev=2598.36 clat percentiles (usec): | 1.00th=[ 8979], 5.00th=[10814], 10.00th=[12780], 20.00th=[14353], | 30.00th=[15008], 40.00th=[15533], 50.00th=[15926], 60.00th=[16319], | 70.00th=[16712], 80.00th=[17171], 90.00th=[17957], 95.00th=[18744], | 99.00th=[22938], 99.50th=[28705], 99.90th=[28705], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 3592, max= 4560, per=25.01%, avg=4052.60, stdev=343.15, samples=5 iops : min= 898, max= 1140, avg=1013.00, stdev=85.79, samples=5 lat (msec) : 4=0.16%, 10=1.80%, 20=95.39%, 50=2.66% cpu : usr=0.75%, sys=5.03%, ctx=1505, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=17411: Tue Jan 7 11:42:19 2025 write: IOPS=1013, BW=4052KiB/s (4149kB/s)(10.0MiB/2527msec) slat (usec): min=13, max=353, avg=41.66, stdev=22.77 clat (usec): min=1741, max=33980, avg=15697.86, stdev=2718.89 lat (usec): min=1787, max=33999, avg=15740.73, stdev=2719.31 clat percentiles (usec): | 1.00th=[ 9110], 5.00th=[10945], 10.00th=[12780], 20.00th=[14353], | 30.00th=[14877], 40.00th=[15533], 50.00th=[15926], 60.00th=[16319], | 70.00th=[16712], 80.00th=[17171], 90.00th=[17957], 95.00th=[18744], | 99.00th=[23987], 99.50th=[28705], 99.90th=[32637], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 3600, max= 4552, per=25.06%, avg=4060.80, stdev=337.31, samples=5 iops : min= 900, max= 1138, avg=1015.20, stdev=84.33, samples=5 lat (msec) : 2=0.04%, 4=0.20%, 10=1.68%, 20=95.16%, 50=2.93% cpu : usr=0.79%, sys=4.75%, ctx=1350, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=17412: Tue Jan 7 11:42:19 2025 write: IOPS=1012, BW=4051KiB/s (4148kB/s)(10.0MiB/2528msec) slat (usec): min=12, max=334, avg=43.91, stdev=23.93 clat (usec): min=1621, max=29042, avg=15706.27, stdev=2524.69 lat (usec): min=1677, max=29070, avg=15751.50, stdev=2524.62 clat percentiles (usec): | 1.00th=[ 8979], 5.00th=[10945], 10.00th=[12780], 20.00th=[14484], | 30.00th=[15008], 40.00th=[15533], 50.00th=[15926], 60.00th=[16319], | 70.00th=[16712], 80.00th=[17171], 90.00th=[17957], 95.00th=[19006], | 99.00th=[23200], 99.50th=[28967], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 3608, max= 4528, per=24.95%, avg=4043.00, stdev=326.55, samples=5 iops : min= 902, max= 1132, avg=1010.60, stdev=81.63, samples=5 lat (msec) : 2=0.04%, 10=1.88%, 20=95.43%, 50=2.66% cpu : usr=1.07%, sys=4.67%, ctx=1657, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=15.8MiB/s (16.6MB/s), 4051KiB/s-4076KiB/s (4148kB/s-4174kB/s), io=40.0MiB (41.9MB), run=2512-2528msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17424: Tue Jan 7 11:42:22 2025 read: IOPS=633, BW=2534KiB/s (2595kB/s)(5048KiB/1992msec) slat (usec): min=485, max=1605, avg=761.86, stdev=131.59 clat (usec): min=7328, max=19271, avg=11815.93, stdev=1621.00 lat (usec): min=8095, max=20038, avg=12579.47, stdev=1631.59 clat percentiles (usec): | 1.00th=[ 8356], 5.00th=[ 9241], 10.00th=[ 9765], 20.00th=[10290], | 30.00th=[10814], 40.00th=[11338], 50.00th=[11863], 60.00th=[12256], | 70.00th=[12649], 80.00th=[13173], 90.00th=[13829], 95.00th=[14484], | 99.00th=[15664], 99.50th=[16319], 99.90th=[18744], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 2384, max= 2552, per=24.71%, avg=2488.00, stdev=90.86, samples=3 iops : min= 596, max= 638, avg=622.00, stdev=22.72, samples=3 write: IOPS=651, BW=2606KiB/s (2669kB/s)(5192KiB/1992msec) slat (usec): min=12, max=850, avg=42.03, stdev=29.86 clat (usec): min=7706, max=18650, avg=12139.15, stdev=1261.42 lat (usec): min=7750, max=18696, avg=12182.36, stdev=1261.30 clat percentiles (usec): | 1.00th=[ 8979], 5.00th=[10028], 10.00th=[10552], 20.00th=[11076], | 30.00th=[11469], 40.00th=[11863], 50.00th=[12125], 60.00th=[12518], | 70.00th=[12911], 80.00th=[13173], 90.00th=[13698], 95.00th=[14091], | 99.00th=[15008], 99.50th=[15664], 99.90th=[17695], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 2584, max= 2680, per=25.06%, avg=2626.67, stdev=48.88, samples=3 iops : min= 646, max= 670, avg=656.67, stdev=12.22, samples=3 lat (msec) : 10=9.22%, 20=90.78% cpu : usr=1.16%, sys=9.54%, ctx=2249, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17425: Tue Jan 7 11:42:22 2025 read: IOPS=629, BW=2519KiB/s (2580kB/s)(4976KiB/1975msec) slat (usec): min=12, max=1404, avg=758.32, stdev=132.57 clat (usec): min=1082, max=20714, avg=11691.37, stdev=1746.07 lat (usec): min=1100, max=21362, avg=12451.39, stdev=1762.86 clat percentiles (usec): | 1.00th=[ 7898], 5.00th=[ 9110], 10.00th=[ 9634], 20.00th=[10290], | 30.00th=[10814], 40.00th=[11338], 50.00th=[11731], 60.00th=[12125], | 70.00th=[12649], 80.00th=[13042], 90.00th=[13698], 95.00th=[14353], | 99.00th=[15664], 99.50th=[16188], 99.90th=[18220], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 2400, max= 2520, per=24.63%, avg=2480.00, stdev=69.28, samples=3 iops : min= 600, max= 630, avg=620.00, stdev=17.32, samples=3 write: IOPS=666, BW=2665KiB/s (2729kB/s)(5264KiB/1975msec) slat (usec): min=12, max=713, avg=41.62, stdev=29.61 clat (usec): min=3130, max=18999, avg=12132.97, stdev=1378.61 lat (usec): min=3144, max=19040, avg=12175.72, stdev=1378.27 clat percentiles (usec): | 1.00th=[ 8717], 5.00th=[10028], 10.00th=[10552], 20.00th=[11076], | 30.00th=[11600], 40.00th=[11863], 50.00th=[12256], 60.00th=[12518], | 70.00th=[12911], 80.00th=[13173], 90.00th=[13698], 95.00th=[14222], | 99.00th=[15139], 99.50th=[15401], 99.90th=[16319], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 2664, max= 2720, per=25.69%, avg=2693.33, stdev=28.10, samples=3 iops : min= 666, max= 680, avg=673.33, stdev= 7.02, samples=3 lat (msec) : 2=0.12%, 4=0.20%, 10=9.77%, 20=89.88%, 50=0.04% cpu : usr=1.06%, sys=9.52%, ctx=2296, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17426: Tue Jan 7 11:42:22 2025 read: IOPS=625, BW=2501KiB/s (2561kB/s)(4984KiB/1993msec) slat (usec): min=416, max=1559, avg=760.61, stdev=135.37 clat (usec): min=1687, max=20830, avg=11792.18, stdev=1711.39 lat (usec): min=2399, max=21607, avg=12554.41, stdev=1728.14 clat percentiles (usec): | 1.00th=[ 8225], 5.00th=[ 9110], 10.00th=[ 9765], 20.00th=[10290], | 30.00th=[10814], 40.00th=[11338], 50.00th=[11731], 60.00th=[12256], | 70.00th=[12649], 80.00th=[13173], 90.00th=[13960], 95.00th=[14615], | 99.00th=[15795], 99.50th=[16319], 99.90th=[20317], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 2488, max= 2568, per=25.05%, avg=2522.67, stdev=41.05, samples=3 iops : min= 622, max= 642, avg=630.67, stdev=10.26, samples=3 write: IOPS=659, BW=2637KiB/s (2701kB/s)(5256KiB/1993msec) slat (usec): min=12, max=603, avg=41.22, stdev=30.66 clat (usec): min=1725, max=18512, avg=12205.90, stdev=1314.63 lat (usec): min=1808, max=18539, avg=12248.29, stdev=1313.59 clat percentiles (usec): | 1.00th=[ 9110], 5.00th=[10159], 10.00th=[10552], 20.00th=[11207], | 30.00th=[11600], 40.00th=[11994], 50.00th=[12256], 60.00th=[12518], | 70.00th=[12780], 80.00th=[13173], 90.00th=[13829], 95.00th=[14353], | 99.00th=[15139], 99.50th=[15795], 99.90th=[18482], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 2480, max= 2648, per=24.70%, avg=2589.33, stdev=94.77, samples=3 iops : min= 620, max= 662, avg=647.33, stdev=23.69, samples=3 lat (msec) : 2=0.08%, 10=8.63%, 20=91.21%, 50=0.08% cpu : usr=1.36%, sys=9.24%, ctx=2251, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17427: Tue Jan 7 11:42:22 2025 read: IOPS=634, BW=2539KiB/s (2600kB/s)(5060KiB/1993msec) slat (usec): min=440, max=1279, avg=758.98, stdev=130.47 clat (usec): min=6070, max=17840, avg=11780.90, stdev=1650.83 lat (usec): min=7042, max=18752, avg=12541.56, stdev=1660.42 clat percentiles (usec): | 1.00th=[ 7963], 5.00th=[ 9241], 10.00th=[ 9634], 20.00th=[10421], | 30.00th=[10814], 40.00th=[11338], 50.00th=[11863], 60.00th=[12256], | 70.00th=[12649], 80.00th=[13173], 90.00th=[13960], 95.00th=[14484], | 99.00th=[15533], 99.50th=[15926], 99.90th=[16319], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 2392, max= 2584, per=24.97%, avg=2514.67, stdev=106.53, samples=3 iops : min= 598, max= 646, avg=628.67, stdev=26.63, samples=3 write: IOPS=649, BW=2599KiB/s (2661kB/s)(5180KiB/1993msec) slat (usec): min=13, max=433, avg=40.85, stdev=22.74 clat (usec): min=8247, max=17286, avg=12218.47, stdev=1297.35 lat (usec): min=8283, max=17323, avg=12260.49, stdev=1296.25 clat percentiles (usec): | 1.00th=[ 9110], 5.00th=[10028], 10.00th=[10421], 20.00th=[11076], | 30.00th=[11600], 40.00th=[11994], 50.00th=[12256], 60.00th=[12649], | 70.00th=[12911], 80.00th=[13304], 90.00th=[13829], 95.00th=[14222], | 99.00th=[15139], 99.50th=[15270], 99.90th=[17171], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 2576, max= 2616, per=24.80%, avg=2600.00, stdev=21.17, samples=3 iops : min= 644, max= 654, avg=650.00, stdev= 5.29, samples=3 lat (msec) : 10=9.88%, 20=90.12% cpu : usr=0.80%, sys=9.99%, ctx=2288, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9.83MiB/s (10.3MB/s), 2501KiB/s-2539KiB/s (2561kB/s-2600kB/s), io=19.6MiB (20.5MB), run=1975-1993msec WRITE: bw=10.2MiB/s (10.7MB/s), 2599KiB/s-2665KiB/s (2661kB/s-2729kB/s), io=20.4MiB (21.4MB), run=1975-1993msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=17432: Tue Jan 7 11:42:24 2025 read: IOPS=8, BW=333MiB/s (350MB/s)(40.0MiB/120msec) slat (nsec): min=8959.9k, max=8959.9k, avg=8959923.00, stdev= 0.00 clat (nsec): min=105798k, max=105798k, avg=105798353.00, stdev= 0.00 lat (nsec): min=114770k, max=114770k, avg=114769936.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 106], 10.00th=[ 106], 20.00th=[ 106], | 30.00th=[ 106], 40.00th=[ 106], 50.00th=[ 106], 60.00th=[ 106], | 70.00th=[ 106], 80.00th=[ 106], 90.00th=[ 106], 95.00th=[ 106], | 99.00th=[ 106], 99.50th=[ 106], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=7.56%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=333MiB/s (350MB/s), 333MiB/s-333MiB/s (350MB/s-350MB/s), io=40.0MiB (41.9MB), run=120-120msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck