== sanity test 398c: run fio to test AIO ================= 09:28:05 (1721222885) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.137392 s, 305 MB/s osc.lustre-OST0000-osc-ffff8800b5d54800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=25431: Wed Jul 17 09:28:38 2024 write: IOPS=80, BW=323KiB/s (331kB/s)(10.0MiB/31697msec) slat (usec): min=12, max=597, avg=70.76, stdev=45.51 clat (msec): min=66, max=415, avg=197.91, stdev=49.04 lat (msec): min=66, max=415, avg=197.98, stdev=49.06 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 128], 10.00th=[ 142], 20.00th=[ 159], | 30.00th=[ 171], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 218], 80.00th=[ 239], 90.00th=[ 266], 95.00th=[ 284], | 99.00th=[ 334], 99.50th=[ 347], 99.90th=[ 393], 99.95th=[ 405], | 99.99th=[ 418] bw ( KiB/s): min= 184, max= 536, per=25.01%, avg=322.68, stdev=66.95, samples=63 iops : min= 46, max= 134, avg=80.59, stdev=16.74, samples=63 lat (msec) : 100=1.21%, 250=83.24%, 500=15.55% cpu : usr=0.17%, sys=0.67%, ctx=1926, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=25432: Wed Jul 17 09:28:38 2024 write: IOPS=80, BW=323KiB/s (330kB/s)(10.0MiB/31728msec) slat (usec): min=12, max=454, avg=72.81, stdev=44.63 clat (msec): min=20, max=364, avg=198.13, stdev=48.00 lat (msec): min=20, max=364, avg=198.21, stdev=48.02 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 129], 10.00th=[ 144], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 192], 60.00th=[ 205], | 70.00th=[ 218], 80.00th=[ 241], 90.00th=[ 266], 95.00th=[ 284], | 99.00th=[ 330], 99.50th=[ 342], 99.90th=[ 355], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 216, max= 520, per=24.89%, avg=321.05, stdev=63.86, samples=63 iops : min= 54, max= 130, avg=80.19, stdev=15.98, samples=63 lat (msec) : 50=0.04%, 100=1.05%, 250=83.83%, 500=15.08% cpu : usr=0.13%, sys=0.70%, ctx=2095, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=25433: Wed Jul 17 09:28:38 2024 write: IOPS=80, BW=323KiB/s (330kB/s)(10.0MiB/31728msec) slat (usec): min=12, max=445, avg=72.23, stdev=45.55 clat (msec): min=66, max=358, avg=198.13, stdev=47.67 lat (msec): min=66, max=358, avg=198.20, stdev=47.68 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 131], 10.00th=[ 144], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 220], 80.00th=[ 239], 90.00th=[ 266], 95.00th=[ 284], | 99.00th=[ 330], 99.50th=[ 334], 99.90th=[ 347], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 199, max= 536, per=24.93%, avg=321.62, stdev=64.49, samples=63 iops : min= 49, max= 134, avg=80.29, stdev=16.21, samples=63 lat (msec) : 100=1.21%, 250=83.75%, 500=15.04% cpu : usr=0.14%, sys=0.70%, ctx=1983, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=25434: Wed Jul 17 09:28:38 2024 write: IOPS=80, BW=323KiB/s (331kB/s)(10.0MiB/31687msec) slat (usec): min=12, max=35818, avg=88.92, stdev=708.25 clat (msec): min=43, max=393, avg=197.87, stdev=49.32 lat (msec): min=43, max=393, avg=197.96, stdev=49.29 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 129], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 171], 40.00th=[ 184], 50.00th=[ 194], 60.00th=[ 205], | 70.00th=[ 218], 80.00th=[ 239], 90.00th=[ 266], 95.00th=[ 284], | 99.00th=[ 330], 99.50th=[ 342], 99.90th=[ 359], 99.95th=[ 372], | 99.99th=[ 393] bw ( KiB/s): min= 224, max= 616, per=25.01%, avg=322.63, stdev=67.50, samples=63 iops : min= 56, max= 154, avg=80.63, stdev=16.88, samples=63 lat (msec) : 50=0.31%, 100=1.56%, 250=83.20%, 500=14.92% cpu : usr=0.15%, sys=0.75%, ctx=2021, majf=0, minf=27 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1291KiB/s (1322kB/s), 323KiB/s-323KiB/s (330kB/s-331kB/s), io=40.0MiB (41.9MB), run=31687-31728msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25469: Wed Jul 17 09:28:56 2024 read: IOPS=77, BW=308KiB/s (316kB/s)(5048KiB/16376msec) slat (usec): min=25, max=2120, avg=819.87, stdev=216.73 clat (msec): min=2, max=293, avg=69.56, stdev=17.33 lat (msec): min=2, max=294, avg=70.39, stdev=17.35 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 53], 10.00th=[ 56], 20.00th=[ 59], | 30.00th=[ 63], 40.00th=[ 66], 50.00th=[ 70], 60.00th=[ 72], | 70.00th=[ 74], 80.00th=[ 80], 90.00th=[ 85], 95.00th=[ 89], | 99.00th=[ 106], 99.50th=[ 136], 99.90th=[ 271], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 216, max= 415, per=25.48%, avg=310.91, stdev=53.01, samples=32 iops : min= 54, max= 103, avg=77.66, stdev=13.19, samples=32 write: IOPS=79, BW=317KiB/s (325kB/s)(5192KiB/16376msec) slat (usec): min=13, max=844, avg=55.76, stdev=39.52 clat (msec): min=27, max=376, avg=133.12, stdev=35.19 lat (msec): min=27, max=376, avg=133.18, stdev=35.19 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 83], 10.00th=[ 92], 20.00th=[ 105], | 30.00th=[ 113], 40.00th=[ 124], 50.00th=[ 131], 60.00th=[ 140], | 70.00th=[ 150], 80.00th=[ 161], 90.00th=[ 176], 95.00th=[ 188], | 99.00th=[ 222], 99.50th=[ 292], 99.90th=[ 351], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 256, max= 384, per=25.13%, avg=319.16, stdev=36.23, samples=32 iops : min= 64, max= 96, avg=79.72, stdev= 9.05, samples=32 lat (msec) : 4=0.12%, 10=0.08%, 20=0.31%, 50=1.56%, 100=54.41% lat (msec) : 250=43.05%, 500=0.47% cpu : usr=0.27%, sys=1.48%, ctx=2982, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25470: Wed Jul 17 09:28:56 2024 read: IOPS=75, BW=303KiB/s (310kB/s)(4976KiB/16442msec) slat (usec): min=489, max=3465, avg=844.46, stdev=230.70 clat (usec): min=1425, max=136654, avg=69521.87, stdev=12485.80 lat (msec): min=2, max=138, avg=70.37, stdev=12.52 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 51], 10.00th=[ 56], 20.00th=[ 60], | 30.00th=[ 64], 40.00th=[ 68], 50.00th=[ 70], 60.00th=[ 72], | 70.00th=[ 74], 80.00th=[ 80], 90.00th=[ 85], 95.00th=[ 89], | 99.00th=[ 102], 99.50th=[ 112], 99.90th=[ 136], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 192, max= 472, per=24.82%, avg=302.75, stdev=70.86, samples=32 iops : min= 48, max= 118, avg=75.69, stdev=17.72, samples=32 write: IOPS=80, BW=320KiB/s (328kB/s)(5264KiB/16442msec) slat (usec): min=12, max=629, avg=55.35, stdev=36.32 clat (msec): min=43, max=223, avg=133.13, stdev=31.10 lat (msec): min=43, max=223, avg=133.19, stdev=31.10 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 86], 10.00th=[ 94], 20.00th=[ 106], | 30.00th=[ 115], 40.00th=[ 124], 50.00th=[ 132], 60.00th=[ 142], | 70.00th=[ 150], 80.00th=[ 161], 90.00th=[ 176], 95.00th=[ 188], | 99.00th=[ 209], 99.50th=[ 213], 99.90th=[ 222], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 240, max= 400, per=25.14%, avg=319.25, stdev=42.35, samples=32 iops : min= 60, max= 100, avg=79.81, stdev=10.59, samples=32 lat (msec) : 2=0.08%, 10=0.04%, 20=0.08%, 50=2.03%, 100=53.48% lat (msec) : 250=44.30% cpu : usr=0.23%, sys=1.51%, ctx=2982, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25471: Wed Jul 17 09:28:56 2024 read: IOPS=75, BW=304KiB/s (311kB/s)(4984KiB/16421msec) slat (usec): min=458, max=2341, avg=828.25, stdev=216.33 clat (usec): min=1418, max=136671, avg=69514.33, stdev=12869.67 lat (msec): min=2, max=138, avg=70.34, stdev=12.89 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 51], 10.00th=[ 56], 20.00th=[ 59], | 30.00th=[ 63], 40.00th=[ 67], 50.00th=[ 70], 60.00th=[ 72], | 70.00th=[ 75], 80.00th=[ 80], 90.00th=[ 85], 95.00th=[ 89], | 99.00th=[ 101], 99.50th=[ 113], 99.90th=[ 136], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 208, max= 456, per=25.02%, avg=305.22, stdev=65.09, samples=32 iops : min= 52, max= 114, avg=76.28, stdev=16.22, samples=32 write: IOPS=80, BW=320KiB/s (328kB/s)(5256KiB/16421msec) slat (usec): min=14, max=920, avg=56.56, stdev=40.90 clat (msec): min=44, max=226, avg=132.99, stdev=32.04 lat (msec): min=44, max=226, avg=133.05, stdev=32.05 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 84], 10.00th=[ 93], 20.00th=[ 105], | 30.00th=[ 114], 40.00th=[ 124], 50.00th=[ 132], 60.00th=[ 140], | 70.00th=[ 150], 80.00th=[ 161], 90.00th=[ 176], 95.00th=[ 188], | 99.00th=[ 209], 99.50th=[ 215], 99.90th=[ 228], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 248, max= 392, per=25.04%, avg=317.97, stdev=35.44, samples=32 iops : min= 62, max= 98, avg=79.47, stdev= 8.86, samples=32 lat (msec) : 2=0.04%, 10=0.04%, 20=0.16%, 50=2.27%, 100=53.95% lat (msec) : 250=43.55% cpu : usr=0.18%, sys=1.55%, ctx=3080, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25472: Wed Jul 17 09:28:56 2024 read: IOPS=77, BW=309KiB/s (316kB/s)(5060KiB/16388msec) slat (usec): min=468, max=3178, avg=825.85, stdev=223.02 clat (msec): min=12, max=136, avg=69.73, stdev=12.76 lat (msec): min=12, max=138, avg=70.56, stdev=12.79 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 54], 10.00th=[ 56], 20.00th=[ 60], | 30.00th=[ 63], 40.00th=[ 68], 50.00th=[ 70], 60.00th=[ 72], | 70.00th=[ 74], 80.00th=[ 80], 90.00th=[ 85], 95.00th=[ 90], | 99.00th=[ 105], 99.50th=[ 124], 99.90th=[ 136], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 152, max= 456, per=25.51%, avg=311.22, stdev=68.34, samples=32 iops : min= 38, max= 114, avg=77.78, stdev=17.06, samples=32 write: IOPS=79, BW=316KiB/s (324kB/s)(5180KiB/16388msec) slat (usec): min=13, max=354, avg=55.79, stdev=32.61 clat (msec): min=54, max=284, avg=133.24, stdev=32.27 lat (msec): min=54, max=285, avg=133.30, stdev=32.27 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 86], 10.00th=[ 94], 20.00th=[ 106], | 30.00th=[ 114], 40.00th=[ 124], 50.00th=[ 132], 60.00th=[ 142], | 70.00th=[ 148], 80.00th=[ 161], 90.00th=[ 176], 95.00th=[ 188], | 99.00th=[ 213], 99.50th=[ 259], 99.90th=[ 275], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 232, max= 376, per=24.90%, avg=316.22, stdev=31.71, samples=32 iops : min= 58, max= 94, avg=79.03, stdev= 7.91, samples=32 lat (msec) : 20=0.23%, 50=1.72%, 100=54.80%, 250=42.97%, 500=0.27% cpu : usr=0.24%, sys=1.50%, ctx=3019, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1221KiB/s (1250kB/s), 303KiB/s-309KiB/s (310kB/s-316kB/s), io=19.6MiB (20.5MB), run=16376-16442msec WRITE: bw=1271KiB/s (1301kB/s), 316KiB/s-320KiB/s (324kB/s-328kB/s), io=20.4MiB (21.4MB), run=16376-16442msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=25489: Wed Jul 17 09:28:57 2024 read: IOPS=8, BW=339MiB/s (355MB/s)(40.0MiB/118msec) slat (nsec): min=10659k, max=10659k, avg=10658809.00, stdev= 0.00 clat (nsec): min=102409k, max=102409k, avg=102408880.00, stdev= 0.00 lat (nsec): min=113077k, max=113077k, avg=113077489.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 103], 10.00th=[ 103], 20.00th=[ 103], | 30.00th=[ 103], 40.00th=[ 103], 50.00th=[ 103], 60.00th=[ 103], | 70.00th=[ 103], 80.00th=[ 103], 90.00th=[ 103], 95.00th=[ 103], | 99.00th=[ 103], 99.50th=[ 103], 99.90th=[ 103], 99.95th=[ 103], | 99.99th=[ 103] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=9.40%, ctx=6, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=339MiB/s (355MB/s), 339MiB/s-339MiB/s (355MB/s-355MB/s), io=40.0MiB (41.9MB), run=118-118msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck