== sanity test 398c: run fio to test AIO ================= 04:17:57 (1724314677) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0893316 s, 470 MB/s osc.lustre-OST0000-osc-ffff88008765e800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=19285: Thu Aug 22 04:18:22 2024 write: IOPS=107, BW=429KiB/s (439kB/s)(10.0MiB/23879msec) slat (usec): min=11, max=461, avg=75.78, stdev=46.34 clat (msec): min=15, max=308, avg=149.08, stdev=31.95 lat (msec): min=15, max=308, avg=149.16, stdev=31.95 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 97], 10.00th=[ 114], 20.00th=[ 128], | 30.00th=[ 138], 40.00th=[ 144], 50.00th=[ 150], 60.00th=[ 157], | 70.00th=[ 165], 80.00th=[ 174], 90.00th=[ 186], 95.00th=[ 194], | 99.00th=[ 213], 99.50th=[ 224], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 336, max= 888, per=25.08%, avg=429.70, stdev=82.54, samples=47 iops : min= 84, max= 222, avg=107.36, stdev=20.64, samples=47 lat (msec) : 20=0.12%, 50=0.70%, 100=5.27%, 250=93.55%, 500=0.35% cpu : usr=0.21%, sys=0.83%, ctx=2613, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=19286: Thu Aug 22 04:18:22 2024 write: IOPS=107, BW=428KiB/s (439kB/s)(10.0MiB/23907msec) slat (usec): min=11, max=765, avg=74.41, stdev=48.16 clat (msec): min=47, max=260, avg=149.22, stdev=29.25 lat (msec): min=47, max=260, avg=149.30, stdev=29.26 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 101], 10.00th=[ 115], 20.00th=[ 128], | 30.00th=[ 138], 40.00th=[ 144], 50.00th=[ 150], 60.00th=[ 157], | 70.00th=[ 165], 80.00th=[ 174], 90.00th=[ 186], 95.00th=[ 194], | 99.00th=[ 209], 99.50th=[ 215], 99.90th=[ 232], 99.95th=[ 232], | 99.99th=[ 262] bw ( KiB/s): min= 328, max= 768, per=24.98%, avg=427.87, stdev=71.53, samples=47 iops : min= 82, max= 192, avg=106.94, stdev=17.90, samples=47 lat (msec) : 50=0.62%, 100=4.26%, 250=95.08%, 500=0.04% cpu : usr=0.23%, sys=0.77%, ctx=2715, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=19287: Thu Aug 22 04:18:22 2024 write: IOPS=107, BW=428KiB/s (439kB/s)(10.0MiB/23907msec) slat (usec): min=12, max=436, avg=75.65, stdev=45.95 clat (msec): min=30, max=291, avg=149.23, stdev=30.37 lat (msec): min=30, max=291, avg=149.30, stdev=30.37 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 97], 10.00th=[ 115], 20.00th=[ 128], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 150], 60.00th=[ 157], | 70.00th=[ 165], 80.00th=[ 174], 90.00th=[ 186], 95.00th=[ 197], | 99.00th=[ 213], 99.50th=[ 218], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 292] bw ( KiB/s): min= 344, max= 792, per=25.02%, avg=428.53, stdev=72.02, samples=47 iops : min= 86, max= 198, avg=107.09, stdev=18.01, samples=47 lat (msec) : 50=0.62%, 100=5.04%, 250=94.18%, 500=0.16% cpu : usr=0.18%, sys=0.83%, ctx=2708, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=19288: Thu Aug 22 04:18:22 2024 write: IOPS=107, BW=428KiB/s (439kB/s)(10.0MiB/23898msec) slat (usec): min=11, max=361, avg=74.54, stdev=44.52 clat (msec): min=47, max=263, avg=149.14, stdev=29.48 lat (msec): min=47, max=263, avg=149.21, stdev=29.49 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 103], 10.00th=[ 115], 20.00th=[ 129], | 30.00th=[ 138], 40.00th=[ 144], 50.00th=[ 150], 60.00th=[ 157], | 70.00th=[ 165], 80.00th=[ 174], 90.00th=[ 186], 95.00th=[ 194], | 99.00th=[ 211], 99.50th=[ 215], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 264] bw ( KiB/s): min= 328, max= 768, per=25.00%, avg=428.26, stdev=72.82, samples=47 iops : min= 82, max= 192, avg=107.06, stdev=18.21, samples=47 lat (msec) : 50=0.31%, 100=4.45%, 250=95.12%, 500=0.12% cpu : usr=0.22%, sys=0.77%, ctx=2680, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1713KiB/s (1754kB/s), 428KiB/s-429KiB/s (439kB/s-439kB/s), io=40.0MiB (41.9MB), run=23879-23907msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=19316: Thu Aug 22 04:18:40 2024 read: IOPS=77, BW=310KiB/s (318kB/s)(5048KiB/16272msec) slat (usec): min=20, max=3198, avg=1210.50, stdev=353.96 clat (usec): min=1217, max=241444, avg=66506.83, stdev=17816.07 lat (usec): min=1261, max=242565, avg=67719.36, stdev=17889.33 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 47], 10.00th=[ 51], 20.00th=[ 56], | 30.00th=[ 60], 40.00th=[ 63], 50.00th=[ 66], 60.00th=[ 69], | 70.00th=[ 73], 80.00th=[ 78], 90.00th=[ 86], 95.00th=[ 90], | 99.00th=[ 103], 99.50th=[ 112], 99.90th=[ 241], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min= 208, max= 480, per=25.46%, avg=311.84, stdev=61.96, samples=32 iops : min= 52, max= 120, avg=77.84, stdev=15.39, samples=32 write: IOPS=79, BW=319KiB/s (327kB/s)(5192KiB/16272msec) slat (usec): min=12, max=2501, avg=66.94, stdev=78.48 clat (msec): min=15, max=288, avg=134.48, stdev=34.70 lat (msec): min=15, max=288, avg=134.54, stdev=34.71 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 85], 10.00th=[ 95], 20.00th=[ 109], | 30.00th=[ 117], 40.00th=[ 124], 50.00th=[ 131], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 161], 90.00th=[ 182], 95.00th=[ 201], | 99.00th=[ 215], 99.50th=[ 228], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 232, max= 448, per=25.11%, avg=320.34, stdev=50.32, samples=32 iops : min= 58, max= 112, avg=79.97, stdev=12.59, samples=32 lat (msec) : 2=0.31%, 10=0.12%, 20=0.35%, 50=4.41%, 100=50.66% lat (msec) : 250=44.02%, 500=0.12% cpu : usr=0.22%, sys=1.67%, ctx=3337, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19317: Thu Aug 22 04:18:40 2024 read: IOPS=76, BW=304KiB/s (312kB/s)(4976KiB/16354msec) slat (usec): min=694, max=3601, avg=1223.13, stdev=338.31 clat (msec): min=9, max=111, avg=66.36, stdev=13.32 lat (msec): min=10, max=112, avg=67.59, stdev=13.39 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 49], 10.00th=[ 52], 20.00th=[ 56], | 30.00th=[ 59], 40.00th=[ 62], 50.00th=[ 66], 60.00th=[ 69], | 70.00th=[ 73], 80.00th=[ 78], 90.00th=[ 84], 95.00th=[ 90], | 99.00th=[ 101], 99.50th=[ 103], 99.90th=[ 110], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 152, max= 464, per=24.84%, avg=304.34, stdev=66.98, samples=32 iops : min= 38, max= 116, avg=75.97, stdev=16.74, samples=32 write: IOPS=80, BW=322KiB/s (330kB/s)(5264KiB/16354msec) slat (usec): min=14, max=776, avg=67.33, stdev=44.68 clat (msec): min=19, max=234, avg=134.64, stdev=32.83 lat (msec): min=19, max=234, avg=134.70, stdev=32.84 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 85], 10.00th=[ 96], 20.00th=[ 108], | 30.00th=[ 117], 40.00th=[ 126], 50.00th=[ 133], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 161], 90.00th=[ 180], 95.00th=[ 194], | 99.00th=[ 213], 99.50th=[ 224], 99.90th=[ 230], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 224, max= 415, per=25.07%, avg=319.84, stdev=43.80, samples=32 iops : min= 56, max= 103, avg=79.84, stdev=10.91, samples=32 lat (msec) : 10=0.12%, 20=0.20%, 50=3.32%, 100=51.05%, 250=45.31% cpu : usr=0.34%, sys=1.58%, ctx=3546, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19318: Thu Aug 22 04:18:40 2024 read: IOPS=76, BW=304KiB/s (312kB/s)(4984KiB/16373msec) slat (usec): min=658, max=4331, avg=1221.73, stdev=358.26 clat (msec): min=10, max=111, avg=66.54, stdev=13.60 lat (msec): min=11, max=113, avg=67.76, stdev=13.66 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 48], 10.00th=[ 52], 20.00th=[ 56], | 30.00th=[ 59], 40.00th=[ 62], 50.00th=[ 66], 60.00th=[ 69], | 70.00th=[ 73], 80.00th=[ 79], 90.00th=[ 86], 95.00th=[ 90], | 99.00th=[ 101], 99.50th=[ 104], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 184, max= 463, per=24.84%, avg=304.34, stdev=69.43, samples=32 iops : min= 46, max= 115, avg=75.97, stdev=17.29, samples=32 write: IOPS=80, BW=321KiB/s (329kB/s)(5256KiB/16373msec) slat (usec): min=13, max=536, avg=65.17, stdev=40.34 clat (msec): min=40, max=234, avg=134.86, stdev=33.80 lat (msec): min=40, max=234, avg=134.93, stdev=33.80 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 85], 10.00th=[ 95], 20.00th=[ 108], | 30.00th=[ 117], 40.00th=[ 125], 50.00th=[ 132], 60.00th=[ 140], | 70.00th=[ 150], 80.00th=[ 161], 90.00th=[ 182], 95.00th=[ 199], | 99.00th=[ 215], 99.50th=[ 226], 99.90th=[ 230], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 208, max= 400, per=24.83%, avg=316.84, stdev=46.09, samples=32 iops : min= 52, max= 100, avg=79.09, stdev=11.55, samples=32 lat (msec) : 20=0.27%, 50=3.67%, 100=51.05%, 250=45.00% cpu : usr=0.36%, sys=1.53%, ctx=3451, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=19319: Thu Aug 22 04:18:40 2024 read: IOPS=77, BW=310KiB/s (318kB/s)(5060KiB/16317msec) slat (usec): min=668, max=3962, avg=1230.01, stdev=374.76 clat (msec): min=11, max=139, avg=67.08, stdev=12.72 lat (msec): min=12, max=141, avg=68.31, stdev=12.80 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 50], 10.00th=[ 53], 20.00th=[ 57], | 30.00th=[ 61], 40.00th=[ 63], 50.00th=[ 67], 60.00th=[ 70], | 70.00th=[ 73], 80.00th=[ 79], 90.00th=[ 84], 95.00th=[ 89], | 99.00th=[ 99], 99.50th=[ 104], 99.90th=[ 128], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 216, max= 408, per=25.47%, avg=312.00, stdev=58.26, samples=32 iops : min= 54, max= 102, avg=78.00, stdev=14.56, samples=32 write: IOPS=79, BW=317KiB/s (325kB/s)(5180KiB/16317msec) slat (usec): min=13, max=491, avg=67.48, stdev=42.18 clat (msec): min=47, max=251, avg=134.54, stdev=34.01 lat (msec): min=47, max=251, avg=134.61, stdev=34.01 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 85], 10.00th=[ 95], 20.00th=[ 107], | 30.00th=[ 116], 40.00th=[ 124], 50.00th=[ 130], 60.00th=[ 140], | 70.00th=[ 150], 80.00th=[ 161], 90.00th=[ 184], 95.00th=[ 197], | 99.00th=[ 218], 99.50th=[ 241], 99.90th=[ 251], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 232, max= 400, per=24.88%, avg=317.50, stdev=46.53, samples=32 iops : min= 58, max= 100, avg=79.38, stdev=11.63, samples=32 lat (msec) : 20=0.12%, 50=2.89%, 100=52.81%, 250=44.06%, 500=0.12% cpu : usr=0.29%, sys=1.64%, ctx=3391, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1226KiB/s (1255kB/s), 304KiB/s-310KiB/s (312kB/s-318kB/s), io=19.6MiB (20.5MB), run=16272-16373msec WRITE: bw=1276KiB/s (1307kB/s), 317KiB/s-322KiB/s (325kB/s-330kB/s), io=20.4MiB (21.4MB), run=16272-16373msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=19336: Thu Aug 22 04:18:41 2024 read: IOPS=13, BW=548MiB/s (575MB/s)(40.0MiB/73msec) slat (nsec): min=6786.7k, max=6786.7k, avg=6786718.00, stdev= 0.00 clat (nsec): min=62821k, max=62821k, avg=62820960.00, stdev= 0.00 lat (nsec): min=69626k, max=69626k, avg=69626270.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[62653], 5.00th=[62653], 10.00th=[62653], 20.00th=[62653], | 30.00th=[62653], 40.00th=[62653], 50.00th=[62653], 60.00th=[62653], | 70.00th=[62653], 80.00th=[62653], 90.00th=[62653], 95.00th=[62653], | 99.00th=[62653], 99.50th=[62653], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=9.59%, ctx=13, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=548MiB/s (575MB/s), 548MiB/s-548MiB/s (575MB/s-575MB/s), io=40.0MiB (41.9MB), run=73-73msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout