== sanity test 398c: run fio to test AIO ================= 19:31:10 (1736987470) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.069842 s, 601 MB/s osc.lustre-OST0000-osc-ffff88009e48e800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=24121: Wed Jan 15 19:31:36 2025 write: IOPS=104, BW=417KiB/s (427kB/s)(10.0MiB/24533msec) slat (usec): min=11, max=326, avg=67.52, stdev=39.66 clat (msec): min=41, max=237, avg=153.18, stdev=30.21 lat (msec): min=41, max=237, avg=153.25, stdev=30.22 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 102], 10.00th=[ 118], 20.00th=[ 130], | 30.00th=[ 140], 40.00th=[ 146], 50.00th=[ 155], 60.00th=[ 161], | 70.00th=[ 169], 80.00th=[ 180], 90.00th=[ 190], 95.00th=[ 201], | 99.00th=[ 220], 99.50th=[ 226], 99.90th=[ 232], 99.95th=[ 232], | 99.99th=[ 239] bw ( KiB/s): min= 328, max= 768, per=24.89%, avg=415.37, stdev=68.90, samples=49 iops : min= 82, max= 192, avg=103.76, stdev=17.21, samples=49 lat (msec) : 50=0.08%, 100=4.38%, 250=95.55% cpu : usr=0.20%, sys=0.73%, ctx=2130, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24122: Wed Jan 15 19:31:36 2025 write: IOPS=104, BW=419KiB/s (429kB/s)(10.0MiB/24440msec) slat (usec): min=11, max=306, avg=70.13, stdev=41.03 clat (msec): min=25, max=268, avg=152.60, stdev=32.08 lat (msec): min=25, max=268, avg=152.67, stdev=32.08 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 101], 10.00th=[ 116], 20.00th=[ 129], | 30.00th=[ 138], 40.00th=[ 146], 50.00th=[ 155], 60.00th=[ 161], | 70.00th=[ 171], 80.00th=[ 180], 90.00th=[ 192], 95.00th=[ 203], | 99.00th=[ 226], 99.50th=[ 230], 99.90th=[ 259], 99.95th=[ 259], | 99.99th=[ 271] bw ( KiB/s): min= 320, max= 816, per=25.05%, avg=418.12, stdev=75.54, samples=48 iops : min= 80, max= 204, avg=104.50, stdev=18.87, samples=48 lat (msec) : 50=0.47%, 100=4.34%, 250=95.08%, 500=0.12% cpu : usr=0.19%, sys=0.77%, ctx=2108, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24123: Wed Jan 15 19:31:36 2025 write: IOPS=104, BW=418KiB/s (428kB/s)(10.0MiB/24487msec) slat (usec): min=12, max=418, avg=70.33, stdev=40.16 clat (msec): min=41, max=244, avg=152.88, stdev=30.16 lat (msec): min=42, max=244, avg=152.96, stdev=30.16 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 103], 10.00th=[ 117], 20.00th=[ 129], | 30.00th=[ 138], 40.00th=[ 146], 50.00th=[ 155], 60.00th=[ 161], | 70.00th=[ 169], 80.00th=[ 180], 90.00th=[ 190], 95.00th=[ 201], | 99.00th=[ 220], 99.50th=[ 228], 99.90th=[ 239], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 328, max= 768, per=24.97%, avg=416.69, stdev=68.84, samples=48 iops : min= 82, max= 192, avg=104.08, stdev=17.18, samples=48 lat (msec) : 50=0.08%, 100=4.26%, 250=95.66% cpu : usr=0.20%, sys=0.79%, ctx=2153, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=24124: Wed Jan 15 19:31:36 2025 write: IOPS=104, BW=417KiB/s (427kB/s)(10.0MiB/24534msec) slat (usec): min=12, max=284, avg=66.33, stdev=39.16 clat (msec): min=41, max=231, avg=153.17, stdev=29.95 lat (msec): min=42, max=231, avg=153.24, stdev=29.96 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 105], 10.00th=[ 118], 20.00th=[ 129], | 30.00th=[ 138], 40.00th=[ 146], 50.00th=[ 155], 60.00th=[ 163], | 70.00th=[ 171], 80.00th=[ 180], 90.00th=[ 190], 95.00th=[ 201], | 99.00th=[ 218], 99.50th=[ 222], 99.90th=[ 230], 99.95th=[ 230], | 99.99th=[ 232] bw ( KiB/s): min= 336, max= 720, per=24.89%, avg=415.35, stdev=64.10, samples=49 iops : min= 84, max= 180, avg=103.76, stdev=16.00, samples=49 lat (msec) : 50=0.04%, 100=3.98%, 250=95.98% cpu : usr=0.16%, sys=0.79%, ctx=1891, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1670KiB/s (1710kB/s), 417KiB/s-419KiB/s (427kB/s-429kB/s), io=40.0MiB (41.9MB), run=24440-24534msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24153: Wed Jan 15 19:31:52 2025 read: IOPS=86, BW=347KiB/s (356kB/s)(5048KiB/14537msec) slat (usec): min=12, max=2848, avg=847.00, stdev=201.89 clat (usec): min=895, max=141269, avg=61857.94, stdev=11490.09 lat (usec): min=1339, max=142061, avg=62706.75, stdev=11516.67 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 46], 10.00th=[ 50], 20.00th=[ 54], | 30.00th=[ 57], 40.00th=[ 60], 50.00th=[ 63], 60.00th=[ 65], | 70.00th=[ 68], 80.00th=[ 71], 90.00th=[ 75], 95.00th=[ 79], | 99.00th=[ 87], 99.50th=[ 92], 99.90th=[ 140], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 240, max= 424, per=25.27%, avg=345.76, stdev=52.71, samples=29 iops : min= 60, max= 106, avg=86.31, stdev=13.18, samples=29 write: IOPS=89, BW=357KiB/s (366kB/s)(5192KiB/14537msec) slat (usec): min=12, max=685, avg=58.75, stdev=40.72 clat (msec): min=11, max=214, avg=118.05, stdev=27.48 lat (msec): min=11, max=215, avg=118.11, stdev=27.48 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 78], 10.00th=[ 86], 20.00th=[ 97], | 30.00th=[ 104], 40.00th=[ 109], 50.00th=[ 116], 60.00th=[ 123], | 70.00th=[ 131], 80.00th=[ 140], 90.00th=[ 155], 95.00th=[ 165], | 99.00th=[ 188], 99.50th=[ 197], 99.90th=[ 207], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 264, max= 440, per=25.00%, avg=355.97, stdev=43.52, samples=29 iops : min= 66, max= 110, avg=88.86, stdev=10.95, samples=29 lat (usec) : 1000=0.04% lat (msec) : 2=0.12%, 4=0.08%, 20=0.27%, 50=5.20%, 100=56.17% lat (msec) : 250=38.12% cpu : usr=0.34%, sys=1.66%, ctx=2852, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24154: Wed Jan 15 19:31:52 2025 read: IOPS=84, BW=340KiB/s (348kB/s)(4976KiB/14653msec) slat (usec): min=536, max=4116, avg=863.14, stdev=228.26 clat (usec): min=868, max=112766, avg=62195.48, stdev=11016.05 lat (usec): min=1786, max=113599, avg=63060.68, stdev=11024.05 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 47], 10.00th=[ 51], 20.00th=[ 54], | 30.00th=[ 57], 40.00th=[ 61], 50.00th=[ 63], 60.00th=[ 66], | 70.00th=[ 68], 80.00th=[ 71], 90.00th=[ 75], 95.00th=[ 79], | 99.00th=[ 87], 99.50th=[ 92], 99.90th=[ 112], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 216, max= 456, per=24.62%, avg=336.83, stdev=60.41, samples=29 iops : min= 54, max= 114, avg=84.21, stdev=15.10, samples=29 write: IOPS=89, BW=359KiB/s (368kB/s)(5264KiB/14653msec) slat (usec): min=13, max=847, avg=58.08, stdev=37.36 clat (msec): min=52, max=207, avg=118.34, stdev=27.33 lat (msec): min=52, max=207, avg=118.40, stdev=27.33 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 77], 10.00th=[ 85], 20.00th=[ 95], | 30.00th=[ 104], 40.00th=[ 111], 50.00th=[ 117], 60.00th=[ 124], | 70.00th=[ 131], 80.00th=[ 140], 90.00th=[ 153], 95.00th=[ 165], | 99.00th=[ 192], 99.50th=[ 199], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 264, max= 448, per=25.18%, avg=358.62, stdev=49.13, samples=29 iops : min= 66, max= 112, avg=89.66, stdev=12.28, samples=29 lat (usec) : 1000=0.04% lat (msec) : 2=0.04%, 4=0.08%, 20=0.31%, 50=3.95%, 100=57.23% lat (msec) : 250=38.36% cpu : usr=0.21%, sys=1.78%, ctx=2862, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24155: Wed Jan 15 19:31:52 2025 read: IOPS=84, BW=340KiB/s (348kB/s)(4984KiB/14669msec) slat (usec): min=477, max=2451, avg=857.84, stdev=187.47 clat (msec): min=9, max=109, avg=61.85, stdev=10.30 lat (msec): min=10, max=110, avg=62.71, stdev=10.30 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 46], 10.00th=[ 51], 20.00th=[ 54], | 30.00th=[ 57], 40.00th=[ 60], 50.00th=[ 63], 60.00th=[ 65], | 70.00th=[ 67], 80.00th=[ 70], 90.00th=[ 75], 95.00th=[ 79], | 99.00th=[ 84], 99.50th=[ 88], 99.90th=[ 109], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 192, max= 496, per=24.79%, avg=339.14, stdev=73.58, samples=29 iops : min= 48, max= 124, avg=84.66, stdev=18.42, samples=29 write: IOPS=89, BW=358KiB/s (367kB/s)(5256KiB/14669msec) slat (usec): min=12, max=230, avg=58.20, stdev=32.91 clat (msec): min=45, max=206, avg=118.93, stdev=26.97 lat (msec): min=45, max=206, avg=118.99, stdev=26.98 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 79], 10.00th=[ 86], 20.00th=[ 96], | 30.00th=[ 104], 40.00th=[ 110], 50.00th=[ 116], 60.00th=[ 124], | 70.00th=[ 132], 80.00th=[ 142], 90.00th=[ 155], 95.00th=[ 169], | 99.00th=[ 188], 99.50th=[ 194], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 256, max= 424, per=24.90%, avg=354.59, stdev=33.71, samples=29 iops : min= 64, max= 106, avg=88.52, stdev= 8.45, samples=29 lat (msec) : 10=0.04%, 20=0.23%, 50=4.69%, 100=56.88%, 250=38.16% cpu : usr=0.27%, sys=1.75%, ctx=2819, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24156: Wed Jan 15 19:31:52 2025 read: IOPS=86, BW=345KiB/s (354kB/s)(5060KiB/14652msec) slat (usec): min=512, max=2795, avg=855.41, stdev=199.67 clat (msec): min=10, max=113, avg=62.70, stdev= 9.61 lat (msec): min=11, max=114, avg=63.56, stdev= 9.62 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 48], 10.00th=[ 52], 20.00th=[ 55], | 30.00th=[ 58], 40.00th=[ 61], 50.00th=[ 64], 60.00th=[ 66], | 70.00th=[ 68], 80.00th=[ 71], 90.00th=[ 75], 95.00th=[ 79], | 99.00th=[ 82], 99.50th=[ 87], 99.90th=[ 93], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 192, max= 448, per=25.20%, avg=344.79, stdev=63.34, samples=29 iops : min= 48, max= 112, avg=86.17, stdev=15.81, samples=29 write: IOPS=88, BW=354KiB/s (362kB/s)(5180KiB/14652msec) slat (usec): min=13, max=757, avg=58.35, stdev=36.45 clat (msec): min=12, max=215, avg=118.70, stdev=27.76 lat (msec): min=12, max=215, avg=118.76, stdev=27.76 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 78], 10.00th=[ 86], 20.00th=[ 95], | 30.00th=[ 104], 40.00th=[ 110], 50.00th=[ 115], 60.00th=[ 124], | 70.00th=[ 132], 80.00th=[ 142], 90.00th=[ 155], 95.00th=[ 167], | 99.00th=[ 192], 99.50th=[ 199], 99.90th=[ 211], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 288, max= 424, per=24.66%, avg=351.14, stdev=34.28, samples=29 iops : min= 72, max= 106, avg=87.76, stdev= 8.61, samples=29 lat (msec) : 20=0.23%, 50=4.06%, 100=58.16%, 250=37.54% cpu : usr=0.24%, sys=1.77%, ctx=2888, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1368KiB/s (1401kB/s), 340KiB/s-347KiB/s (348kB/s-356kB/s), io=19.6MiB (20.5MB), run=14537-14669msec WRITE: bw=1424KiB/s (1458kB/s), 354KiB/s-359KiB/s (362kB/s-368kB/s), io=20.4MiB (21.4MB), run=14537-14669msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=24172: Wed Jan 15 19:31:53 2025 read: IOPS=13, BW=533MiB/s (559MB/s)(40.0MiB/75msec) slat (nsec): min=7693.9k, max=7693.9k, avg=7693893.00, stdev= 0.00 clat (nsec): min=63751k, max=63751k, avg=63750881.00, stdev= 0.00 lat (nsec): min=71466k, max=71466k, avg=71465599.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[63701], 5.00th=[63701], 10.00th=[63701], 20.00th=[63701], | 30.00th=[63701], 40.00th=[63701], 50.00th=[63701], 60.00th=[63701], | 70.00th=[63701], 80.00th=[63701], 90.00th=[63701], 95.00th=[63701], | 99.00th=[63701], 99.50th=[63701], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=9.46%, ctx=17, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=533MiB/s (559MB/s), 533MiB/s-533MiB/s (559MB/s-559MB/s), io=40.0MiB (41.9MB), run=75-75msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck