== sanity test 398c: run fio to test AIO ================= 00:55:19 (1718859319) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.082854 s, 506 MB/s osc.lustre-OST0000-osc-ffff88012a9e6000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=21272: Thu Jun 20 00:55:53 2024 write: IOPS=79, BW=317KiB/s (325kB/s)(10.0MiB/32259msec) slat (usec): min=12, max=944, avg=87.77, stdev=63.45 clat (msec): min=14, max=373, avg=201.32, stdev=42.91 lat (msec): min=14, max=373, avg=201.41, stdev=42.91 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 132], 10.00th=[ 153], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 211], | 70.00th=[ 222], 80.00th=[ 236], 90.00th=[ 255], 95.00th=[ 275], | 99.00th=[ 309], 99.50th=[ 326], 99.90th=[ 355], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 200, max= 520, per=25.00%, avg=317.25, stdev=52.60, samples=64 iops : min= 50, max= 130, avg=79.22, stdev=13.17, samples=64 lat (msec) : 20=0.04%, 100=1.21%, 250=86.68%, 500=12.07% cpu : usr=0.20%, sys=0.72%, ctx=2442, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=21273: Thu Jun 20 00:55:53 2024 write: IOPS=79, BW=318KiB/s (325kB/s)(10.0MiB/32246msec) slat (usec): min=12, max=785, avg=87.08, stdev=59.51 clat (msec): min=69, max=373, avg=201.19, stdev=42.86 lat (msec): min=69, max=373, avg=201.28, stdev=42.86 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 138], 10.00th=[ 150], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 211], | 70.00th=[ 222], 80.00th=[ 234], 90.00th=[ 257], 95.00th=[ 275], | 99.00th=[ 313], 99.50th=[ 326], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 372] bw ( KiB/s): min= 224, max= 512, per=25.04%, avg=317.70, stdev=51.56, samples=64 iops : min= 56, max= 128, avg=79.39, stdev=12.92, samples=64 lat (msec) : 100=1.37%, 250=86.84%, 500=11.80% cpu : usr=0.19%, sys=0.72%, ctx=2387, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=21274: Thu Jun 20 00:55:53 2024 write: IOPS=79, BW=318KiB/s (326kB/s)(10.0MiB/32208msec) slat (usec): min=12, max=1284, avg=86.72, stdev=65.27 clat (msec): min=14, max=397, avg=200.94, stdev=45.09 lat (msec): min=14, max=397, avg=201.02, stdev=45.09 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 128], 10.00th=[ 150], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 211], | 70.00th=[ 224], 80.00th=[ 236], 90.00th=[ 257], 95.00th=[ 279], | 99.00th=[ 313], 99.50th=[ 351], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 397] bw ( KiB/s): min= 176, max= 600, per=25.06%, avg=317.98, stdev=58.87, samples=64 iops : min= 44, max= 150, avg=79.39, stdev=14.71, samples=64 lat (msec) : 20=0.08%, 50=0.27%, 100=1.52%, 250=86.52%, 500=11.60% cpu : usr=0.18%, sys=0.71%, ctx=2472, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=21275: Thu Jun 20 00:55:53 2024 write: IOPS=79, BW=317KiB/s (325kB/s)(10.0MiB/32275msec) slat (usec): min=11, max=2894, avg=90.80, stdev=83.16 clat (msec): min=14, max=378, avg=201.40, stdev=43.19 lat (msec): min=14, max=378, avg=201.49, stdev=43.19 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 129], 10.00th=[ 150], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 224], 80.00th=[ 236], 90.00th=[ 255], 95.00th=[ 275], | 99.00th=[ 309], 99.50th=[ 326], 99.90th=[ 347], 99.95th=[ 355], | 99.99th=[ 380] bw ( KiB/s): min= 216, max= 528, per=24.95%, avg=316.66, stdev=54.38, samples=64 iops : min= 54, max= 132, avg=79.09, stdev=13.62, samples=64 lat (msec) : 20=0.04%, 50=0.04%, 100=1.25%, 250=86.80%, 500=11.88% cpu : usr=0.17%, sys=0.73%, ctx=2594, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1269KiB/s (1300kB/s), 317KiB/s-318KiB/s (325kB/s-326kB/s), io=40.0MiB (41.9MB), run=32208-32275msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21311: Thu Jun 20 00:56:15 2024 read: IOPS=59, BW=237KiB/s (243kB/s)(5048KiB/21269msec) slat (usec): min=17, max=5183, avg=1413.92, stdev=447.82 clat (usec): min=1513, max=201152, avg=87797.25, stdev=18696.81 lat (msec): min=2, max=202, avg=89.21, stdev=18.78 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 63], 10.00th=[ 69], 20.00th=[ 74], | 30.00th=[ 80], 40.00th=[ 84], 50.00th=[ 87], 60.00th=[ 91], | 70.00th=[ 95], 80.00th=[ 103], 90.00th=[ 109], 95.00th=[ 115], | 99.00th=[ 140], 99.50th=[ 146], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 111, max= 360, per=25.20%, avg=237.14, stdev=48.08, samples=42 iops : min= 27, max= 90, avg=59.14, stdev=12.05, samples=42 write: IOPS=61, BW=244KiB/s (250kB/s)(5192KiB/21269msec) slat (usec): min=14, max=1534, avg=75.48, stdev=72.57 clat (msec): min=29, max=310, avg=175.15, stdev=38.60 lat (msec): min=29, max=310, avg=175.23, stdev=38.60 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 115], 10.00th=[ 128], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 184], | 70.00th=[ 197], 80.00th=[ 207], 90.00th=[ 226], 95.00th=[ 239], | 99.00th=[ 271], 99.50th=[ 275], 99.90th=[ 309], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 183, max= 320, per=24.86%, avg=243.62, stdev=31.75, samples=42 iops : min= 45, max= 80, avg=60.76, stdev= 7.97, samples=42 lat (msec) : 2=0.04%, 4=0.12%, 10=0.08%, 20=0.20%, 50=0.47% lat (msec) : 100=38.20%, 250=59.41%, 500=1.48% cpu : usr=0.29%, sys=1.46%, ctx=2911, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21312: Thu Jun 20 00:56:15 2024 read: IOPS=58, BW=234KiB/s (239kB/s)(4976KiB/21307msec) slat (usec): min=765, max=3718, avg=1441.51, stdev=439.59 clat (usec): min=1906, max=154362, avg=87222.46, stdev=17301.55 lat (msec): min=3, max=156, avg=88.67, stdev=17.37 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 75], | 30.00th=[ 80], 40.00th=[ 84], 50.00th=[ 87], 60.00th=[ 91], | 70.00th=[ 95], 80.00th=[ 102], 90.00th=[ 108], 95.00th=[ 114], | 99.00th=[ 132], 99.50th=[ 138], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 135, max= 408, per=24.64%, avg=231.86, stdev=57.53, samples=42 iops : min= 33, max= 102, avg=57.86, stdev=14.43, samples=42 write: IOPS=61, BW=247KiB/s (253kB/s)(5264KiB/21307msec) slat (usec): min=13, max=527, avg=72.19, stdev=47.78 clat (msec): min=28, max=307, avg=174.90, stdev=39.09 lat (msec): min=28, max=307, avg=174.97, stdev=39.09 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 114], 10.00th=[ 127], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 226], 95.00th=[ 247], | 99.00th=[ 264], 99.50th=[ 275], 99.90th=[ 288], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 192, max= 320, per=24.92%, avg=244.24, stdev=32.04, samples=42 iops : min= 48, max= 80, avg=60.95, stdev= 8.00, samples=42 lat (msec) : 2=0.04%, 10=0.16%, 20=0.12%, 50=0.47%, 100=38.16% lat (msec) : 250=58.95%, 500=2.11% cpu : usr=0.21%, sys=1.54%, ctx=3005, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21313: Thu Jun 20 00:56:15 2024 read: IOPS=58, BW=235KiB/s (240kB/s)(4984KiB/21242msec) slat (usec): min=20, max=3797, avg=1424.03, stdev=431.69 clat (msec): min=13, max=188, avg=87.24, stdev=17.50 lat (msec): min=14, max=189, avg=88.66, stdev=17.57 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 63], 10.00th=[ 67], 20.00th=[ 73], | 30.00th=[ 79], 40.00th=[ 83], 50.00th=[ 87], 60.00th=[ 91], | 70.00th=[ 95], 80.00th=[ 101], 90.00th=[ 109], 95.00th=[ 116], | 99.00th=[ 138], 99.50th=[ 142], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 152, max= 351, per=24.98%, avg=235.07, stdev=42.81, samples=42 iops : min= 38, max= 87, avg=58.64, stdev=10.68, samples=42 write: IOPS=61, BW=247KiB/s (253kB/s)(5256KiB/21242msec) slat (usec): min=14, max=450, avg=73.16, stdev=47.54 clat (msec): min=66, max=309, avg=174.29, stdev=38.06 lat (msec): min=66, max=309, avg=174.37, stdev=38.06 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 115], 10.00th=[ 126], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 207], 90.00th=[ 224], 95.00th=[ 239], | 99.00th=[ 262], 99.50th=[ 271], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 168, max= 352, per=25.11%, avg=246.12, stdev=36.95, samples=42 iops : min= 42, max= 88, avg=61.40, stdev= 9.30, samples=42 lat (msec) : 20=0.16%, 50=0.16%, 100=39.69%, 250=58.95%, 500=1.05% cpu : usr=0.20%, sys=1.54%, ctx=2942, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21314: Thu Jun 20 00:56:15 2024 read: IOPS=59, BW=237KiB/s (243kB/s)(5060KiB/21306msec) slat (usec): min=756, max=4040, avg=1405.39, stdev=402.30 clat (msec): min=15, max=156, avg=88.04, stdev=16.03 lat (msec): min=17, max=158, avg=89.45, stdev=16.07 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 64], 10.00th=[ 68], 20.00th=[ 75], | 30.00th=[ 81], 40.00th=[ 85], 50.00th=[ 88], 60.00th=[ 92], | 70.00th=[ 95], 80.00th=[ 102], 90.00th=[ 109], 95.00th=[ 113], | 99.00th=[ 130], 99.50th=[ 138], 99.90th=[ 140], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 144, max= 384, per=25.18%, avg=236.98, stdev=53.25, samples=42 iops : min= 36, max= 96, avg=59.12, stdev=13.31, samples=42 write: IOPS=60, BW=243KiB/s (249kB/s)(5180KiB/21306msec) slat (usec): min=12, max=1765, avg=74.41, stdev=69.65 clat (msec): min=77, max=311, avg=175.47, stdev=37.59 lat (msec): min=77, max=311, avg=175.54, stdev=37.60 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 115], 10.00th=[ 127], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 184], | 70.00th=[ 197], 80.00th=[ 209], 90.00th=[ 224], 95.00th=[ 239], | 99.00th=[ 264], 99.50th=[ 275], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 176, max= 336, per=24.65%, avg=241.55, stdev=30.91, samples=42 iops : min= 44, max= 84, avg=60.26, stdev= 7.79, samples=42 lat (msec) : 20=0.12%, 50=0.16%, 100=38.71%, 250=59.77%, 500=1.25% cpu : usr=0.25%, sys=1.49%, ctx=3048, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=942KiB/s (964kB/s), 234KiB/s-237KiB/s (239kB/s-243kB/s), io=19.6MiB (20.5MB), run=21242-21307msec WRITE: bw=981KiB/s (1004kB/s), 243KiB/s-247KiB/s (249kB/s-253kB/s), io=20.4MiB (21.4MB), run=21242-21307msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=21335: Thu Jun 20 00:56:16 2024 read: IOPS=10, BW=421MiB/s (442MB/s)(40.0MiB/95msec) slat (nsec): min=6248.7k, max=6248.7k, avg=6248665.00, stdev= 0.00 clat (nsec): min=84579k, max=84579k, avg=84579386.00, stdev= 0.00 lat (nsec): min=90855k, max=90855k, avg=90855060.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[84411], 5.00th=[84411], 10.00th=[84411], 20.00th=[84411], | 30.00th=[84411], 40.00th=[84411], 50.00th=[84411], 60.00th=[84411], | 70.00th=[84411], 80.00th=[84411], 90.00th=[84411], 95.00th=[84411], | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=7.45%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=421MiB/s (442MB/s), 421MiB/s-421MiB/s (442MB/s-442MB/s), io=40.0MiB (41.9MB), run=95-95msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout