== sanity test 398c: run fio to test AIO ================= 09:59:11 (1746799151) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.137108 s, 306 MB/s osc.lustre-OST0000-osc-ffff880136990000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=8148: Fri May 9 09:59:48 2025 write: IOPS=72, BW=291KiB/s (298kB/s)(10.0MiB/35193msec) slat (usec): min=13, max=347, avg=80.88, stdev=45.25 clat (msec): min=36, max=360, avg=219.48, stdev=38.10 lat (msec): min=36, max=360, avg=219.56, stdev=38.10 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 161], 10.00th=[ 171], 20.00th=[ 186], | 30.00th=[ 197], 40.00th=[ 207], 50.00th=[ 218], 60.00th=[ 230], | 70.00th=[ 241], 80.00th=[ 253], 90.00th=[ 271], 95.00th=[ 284], | 99.00th=[ 305], 99.50th=[ 317], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 184, max= 392, per=24.93%, avg=289.93, stdev=44.98, samples=70 iops : min= 46, max= 98, avg=72.47, stdev=11.24, samples=70 lat (msec) : 50=0.04%, 250=77.97%, 500=21.99% cpu : usr=0.11%, sys=0.77%, ctx=2123, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=8149: Fri May 9 09:59:48 2025 write: IOPS=72, BW=291KiB/s (298kB/s)(10.0MiB/35131msec) slat (usec): min=13, max=1085, avg=80.77, stdev=49.11 clat (msec): min=114, max=363, avg=219.27, stdev=38.51 lat (msec): min=115, max=363, avg=219.35, stdev=38.51 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 161], 10.00th=[ 171], 20.00th=[ 186], | 30.00th=[ 197], 40.00th=[ 207], 50.00th=[ 218], 60.00th=[ 228], | 70.00th=[ 241], 80.00th=[ 253], 90.00th=[ 271], 95.00th=[ 284], | 99.00th=[ 317], 99.50th=[ 330], 99.90th=[ 359], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 216, max= 384, per=25.01%, avg=290.83, stdev=43.01, samples=70 iops : min= 54, max= 96, avg=72.69, stdev=10.76, samples=70 lat (msec) : 250=78.55%, 500=21.45% cpu : usr=0.13%, sys=0.77%, ctx=2054, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=8150: Fri May 9 09:59:48 2025 write: IOPS=72, BW=291KiB/s (298kB/s)(10.0MiB/35165msec) slat (usec): min=13, max=420, avg=80.37, stdev=45.81 clat (msec): min=122, max=366, avg=219.39, stdev=38.17 lat (msec): min=122, max=366, avg=219.47, stdev=38.17 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 163], 10.00th=[ 174], 20.00th=[ 188], | 30.00th=[ 197], 40.00th=[ 207], 50.00th=[ 218], 60.00th=[ 228], | 70.00th=[ 239], 80.00th=[ 251], 90.00th=[ 268], 95.00th=[ 288], | 99.00th=[ 313], 99.50th=[ 334], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 368] bw ( KiB/s): min= 216, max= 384, per=25.00%, avg=290.79, stdev=44.01, samples=70 iops : min= 54, max= 96, avg=72.64, stdev=11.03, samples=70 lat (msec) : 250=79.06%, 500=20.94% cpu : usr=0.12%, sys=0.75%, ctx=2182, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=8151: Fri May 9 09:59:48 2025 write: IOPS=72, BW=292KiB/s (299kB/s)(10.0MiB/35114msec) slat (usec): min=14, max=345, avg=82.23, stdev=44.87 clat (msec): min=39, max=433, avg=219.18, stdev=40.87 lat (msec): min=39, max=433, avg=219.26, stdev=40.88 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 161], 10.00th=[ 169], 20.00th=[ 186], | 30.00th=[ 197], 40.00th=[ 207], 50.00th=[ 218], 60.00th=[ 228], | 70.00th=[ 241], 80.00th=[ 251], 90.00th=[ 271], 95.00th=[ 284], | 99.00th=[ 321], 99.50th=[ 342], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 435] bw ( KiB/s): min= 208, max= 384, per=25.01%, avg=290.83, stdev=42.94, samples=70 iops : min= 52, max= 96, avg=72.69, stdev=10.77, samples=70 lat (msec) : 50=0.23%, 100=0.23%, 250=77.93%, 500=21.60% cpu : usr=0.13%, sys=0.80%, ctx=2090, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1164KiB/s (1192kB/s), 291KiB/s-292KiB/s (298kB/s-299kB/s), io=40.0MiB (41.9MB), run=35114-35193msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8189: Fri May 9 10:00:11 2025 read: IOPS=59, BW=237KiB/s (243kB/s)(5048KiB/21305msec) slat (usec): min=29, max=10667, avg=1514.19, stdev=506.60 clat (msec): min=2, max=172, avg=86.29, stdev=19.54 lat (msec): min=4, max=174, avg=87.80, stdev=19.67 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 61], 10.00th=[ 66], 20.00th=[ 72], | 30.00th=[ 78], 40.00th=[ 82], 50.00th=[ 86], 60.00th=[ 90], | 70.00th=[ 95], 80.00th=[ 101], 90.00th=[ 110], 95.00th=[ 117], | 99.00th=[ 142], 99.50th=[ 157], 99.90th=[ 161], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 128, max= 336, per=25.27%, avg=236.55, stdev=45.50, samples=42 iops : min= 32, max= 84, avg=59.12, stdev=11.40, samples=42 write: IOPS=60, BW=244KiB/s (250kB/s)(5192KiB/21305msec) slat (usec): min=15, max=279, avg=70.26, stdev=38.55 clat (msec): min=30, max=289, avg=176.99, stdev=41.68 lat (msec): min=30, max=289, avg=177.06, stdev=41.68 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 112], 10.00th=[ 125], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 199], 80.00th=[ 213], 90.00th=[ 234], 95.00th=[ 249], | 99.00th=[ 275], 99.50th=[ 284], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 152, max= 328, per=24.96%, avg=243.40, stdev=42.39, samples=42 iops : min= 38, max= 82, avg=60.83, stdev=10.61, samples=42 lat (msec) : 4=0.08%, 10=0.23%, 20=0.16%, 50=0.62%, 100=39.38% lat (msec) : 250=57.38%, 500=2.15% cpu : usr=0.31%, sys=1.47%, ctx=2955, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8190: Fri May 9 10:00:11 2025 read: IOPS=58, BW=232KiB/s (238kB/s)(4976KiB/21403msec) slat (usec): min=782, max=5815, avg=1518.46, stdev=427.57 clat (msec): min=2, max=172, avg=85.57, stdev=18.88 lat (msec): min=3, max=175, avg=87.09, stdev=18.99 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 61], 10.00th=[ 65], 20.00th=[ 71], | 30.00th=[ 77], 40.00th=[ 81], 50.00th=[ 85], 60.00th=[ 89], | 70.00th=[ 94], 80.00th=[ 101], 90.00th=[ 108], 95.00th=[ 116], | 99.00th=[ 140], 99.50th=[ 144], 99.90th=[ 171], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 120, max= 352, per=24.79%, avg=232.00, stdev=56.79, samples=42 iops : min= 30, max= 88, avg=58.00, stdev=14.20, samples=42 write: IOPS=61, BW=246KiB/s (252kB/s)(5264KiB/21403msec) slat (usec): min=13, max=1043, avg=71.08, stdev=47.93 clat (msec): min=70, max=289, avg=177.49, stdev=41.43 lat (msec): min=70, max=289, avg=177.57, stdev=41.43 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 111], 10.00th=[ 125], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 199], 80.00th=[ 213], 90.00th=[ 236], 95.00th=[ 251], | 99.00th=[ 275], 99.50th=[ 279], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 160, max= 336, per=25.08%, avg=244.57, stdev=45.64, samples=42 iops : min= 40, max= 84, avg=61.14, stdev=11.41, samples=42 lat (msec) : 4=0.04%, 20=0.31%, 50=0.47%, 100=39.18%, 250=57.54% lat (msec) : 500=2.46% cpu : usr=0.17%, sys=1.62%, ctx=2946, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8191: Fri May 9 10:00:11 2025 read: IOPS=58, BW=233KiB/s (238kB/s)(4984KiB/21424msec) slat (usec): min=814, max=10411, avg=1507.36, stdev=460.77 clat (usec): min=1964, max=170942, avg=85607.85, stdev=19291.97 lat (msec): min=3, max=173, avg=87.12, stdev=19.40 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 61], 10.00th=[ 66], 20.00th=[ 71], | 30.00th=[ 77], 40.00th=[ 81], 50.00th=[ 85], 60.00th=[ 88], | 70.00th=[ 94], 80.00th=[ 100], 90.00th=[ 109], 95.00th=[ 117], | 99.00th=[ 136], 99.50th=[ 157], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 104, max= 360, per=24.83%, avg=232.43, stdev=61.81, samples=42 iops : min= 26, max= 90, avg=58.00, stdev=15.46, samples=42 write: IOPS=61, BW=245KiB/s (251kB/s)(5256KiB/21424msec) slat (usec): min=13, max=931, avg=69.21, stdev=44.53 clat (msec): min=81, max=282, avg=177.70, stdev=39.84 lat (msec): min=81, max=282, avg=177.77, stdev=39.84 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 113], 10.00th=[ 127], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 199], 80.00th=[ 211], 90.00th=[ 232], 95.00th=[ 249], | 99.00th=[ 271], 99.50th=[ 275], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 144, max= 328, per=24.87%, avg=242.52, stdev=44.60, samples=42 iops : min= 36, max= 82, avg=60.52, stdev=11.19, samples=42 lat (msec) : 2=0.04%, 4=0.16%, 20=0.20%, 50=0.51%, 100=39.26% lat (msec) : 250=57.66%, 500=2.19% cpu : usr=0.22%, sys=1.55%, ctx=2989, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8192: Fri May 9 10:00:11 2025 read: IOPS=59, BW=237KiB/s (242kB/s)(5060KiB/21377msec) slat (usec): min=826, max=5584, avg=1513.50, stdev=399.29 clat (msec): min=3, max=174, avg=86.01, stdev=18.43 lat (msec): min=5, max=176, avg=87.53, stdev=18.54 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 63], 10.00th=[ 66], 20.00th=[ 71], | 30.00th=[ 77], 40.00th=[ 81], 50.00th=[ 85], 60.00th=[ 89], | 70.00th=[ 94], 80.00th=[ 100], 90.00th=[ 109], 95.00th=[ 116], | 99.00th=[ 138], 99.50th=[ 146], 99.90th=[ 171], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 168, max= 344, per=25.31%, avg=236.86, stdev=47.76, samples=42 iops : min= 42, max= 86, avg=59.14, stdev=11.93, samples=42 write: IOPS=60, BW=242KiB/s (248kB/s)(5180KiB/21377msec) slat (usec): min=14, max=310, avg=70.69, stdev=40.38 clat (msec): min=74, max=299, avg=178.06, stdev=40.13 lat (msec): min=74, max=299, avg=178.14, stdev=40.13 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 115], 10.00th=[ 127], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 199], 80.00th=[ 213], 90.00th=[ 234], 95.00th=[ 249], | 99.00th=[ 271], 99.50th=[ 284], 99.90th=[ 288], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 144, max= 384, per=24.74%, avg=241.24, stdev=45.88, samples=42 iops : min= 36, max= 96, avg=60.24, stdev=11.48, samples=42 lat (msec) : 4=0.04%, 20=0.20%, 50=0.16%, 100=40.12%, 250=57.27% lat (msec) : 500=2.23% cpu : usr=0.20%, sys=1.59%, ctx=3041, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=937KiB/s (959kB/s), 232KiB/s-237KiB/s (238kB/s-243kB/s), io=19.6MiB (20.5MB), run=21305-21424msec WRITE: bw=975KiB/s (999kB/s), 242KiB/s-246KiB/s (248kB/s-252kB/s), io=20.4MiB (21.4MB), run=21305-21424msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=8214: Fri May 9 10:00:12 2025 read: IOPS=5, BW=237MiB/s (248MB/s)(40.0MiB/169msec) slat (nsec): min=8293.4k, max=8293.4k, avg=8293353.00, stdev= 0.00 clat (nsec): min=153834k, max=153834k, avg=153834244.00, stdev= 0.00 lat (nsec): min=162139k, max=162139k, avg=162138951.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 155], | 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.76%, ctx=5, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=237MiB/s (248MB/s), 237MiB/s-237MiB/s (248MB/s-248MB/s), io=40.0MiB (41.9MB), run=169-169msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout