== sanity test 398c: run fio to test AIO ================= 06:42:55 (1731757375) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.104504 s, 401 MB/s osc.lustre-OST0000-osc-ffff8800b610b000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=29010: Sat Nov 16 06:43:31 2024 write: IOPS=74, BW=299KiB/s (306kB/s)(10.0MiB/34295msec) slat (usec): min=13, max=605, avg=80.71, stdev=49.70 clat (msec): min=100, max=558, avg=213.97, stdev=69.97 lat (msec): min=100, max=558, avg=214.06, stdev=69.96 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 136], 10.00th=[ 144], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 176], 50.00th=[ 188], 60.00th=[ 205], | 70.00th=[ 247], 80.00th=[ 288], 90.00th=[ 321], 95.00th=[ 342], | 99.00th=[ 376], 99.50th=[ 388], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 168, max= 456, per=25.02%, avg=298.69, stdev=84.95, samples=68 iops : min= 42, max= 114, avg=74.57, stdev=21.20, samples=68 lat (msec) : 250=70.51%, 500=29.26%, 750=0.23% cpu : usr=0.12%, sys=0.73%, ctx=2209, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=29011: Sat Nov 16 06:43:31 2024 write: IOPS=74, BW=299KiB/s (306kB/s)(10.0MiB/34295msec) slat (usec): min=12, max=490, avg=83.06, stdev=49.19 clat (msec): min=45, max=445, avg=213.95, stdev=68.61 lat (msec): min=45, max=445, avg=214.03, stdev=68.60 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 136], 10.00th=[ 146], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 176], 50.00th=[ 188], 60.00th=[ 207], | 70.00th=[ 243], 80.00th=[ 288], 90.00th=[ 321], 95.00th=[ 342], | 99.00th=[ 376], 99.50th=[ 384], 99.90th=[ 409], 99.95th=[ 426], | 99.99th=[ 447] bw ( KiB/s): min= 176, max= 456, per=25.02%, avg=298.79, stdev=84.36, samples=68 iops : min= 44, max= 114, avg=74.59, stdev=21.13, samples=68 lat (msec) : 50=0.04%, 100=0.08%, 250=71.05%, 500=28.83% cpu : usr=0.11%, sys=0.75%, ctx=2379, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=29012: Sat Nov 16 06:43:31 2024 write: IOPS=74, BW=299KiB/s (306kB/s)(10.0MiB/34295msec) slat (usec): min=14, max=445, avg=82.81, stdev=47.73 clat (msec): min=39, max=520, avg=213.94, stdev=69.54 lat (msec): min=39, max=520, avg=214.02, stdev=69.53 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 136], 10.00th=[ 144], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 188], 60.00th=[ 207], | 70.00th=[ 243], 80.00th=[ 288], 90.00th=[ 321], 95.00th=[ 342], | 99.00th=[ 380], 99.50th=[ 393], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 168, max= 472, per=25.02%, avg=298.79, stdev=87.24, samples=68 iops : min= 42, max= 118, avg=74.59, stdev=21.85, samples=68 lat (msec) : 50=0.04%, 100=0.08%, 250=71.37%, 500=28.40%, 750=0.12% cpu : usr=0.11%, sys=0.76%, ctx=2288, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=29013: Sat Nov 16 06:43:31 2024 write: IOPS=74, BW=299KiB/s (306kB/s)(10.0MiB/34247msec) slat (usec): min=13, max=511, avg=84.05, stdev=48.88 clat (msec): min=45, max=584, avg=213.67, stdev=71.00 lat (msec): min=45, max=584, avg=213.75, stdev=70.99 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 134], 10.00th=[ 144], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 176], 50.00th=[ 188], 60.00th=[ 205], | 70.00th=[ 241], 80.00th=[ 288], 90.00th=[ 326], 95.00th=[ 347], | 99.00th=[ 384], 99.50th=[ 397], 99.90th=[ 550], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 160, max= 472, per=25.07%, avg=299.28, stdev=85.29, samples=68 iops : min= 40, max= 118, avg=74.72, stdev=21.34, samples=68 lat (msec) : 50=0.08%, 100=0.31%, 250=71.33%, 500=28.09%, 750=0.20% cpu : usr=0.13%, sys=0.74%, ctx=2595, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1194KiB/s (1223kB/s), 299KiB/s-299KiB/s (306kB/s-306kB/s), io=40.0MiB (41.9MB), run=34247-34295msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29051: Sat Nov 16 06:43:58 2024 read: IOPS=49, BW=196KiB/s (201kB/s)(5048KiB/25712msec) slat (usec): min=25, max=8017, avg=1715.65, stdev=515.22 clat (msec): min=2, max=300, avg=103.85, stdev=22.68 lat (msec): min=3, max=302, avg=105.56, stdev=22.78 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 72], 10.00th=[ 80], 20.00th=[ 88], | 30.00th=[ 94], 40.00th=[ 100], 50.00th=[ 105], 60.00th=[ 110], | 70.00th=[ 114], 80.00th=[ 120], 90.00th=[ 129], 95.00th=[ 136], | 99.00th=[ 150], 99.50th=[ 163], 99.90th=[ 253], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 96, max= 312, per=25.31%, avg=196.65, stdev=45.47, samples=51 iops : min= 24, max= 78, avg=49.12, stdev=11.40, samples=51 write: IOPS=50, BW=202KiB/s (207kB/s)(5192KiB/25712msec) slat (usec): min=13, max=296, avg=53.81, stdev=30.59 clat (msec): min=31, max=392, avg=214.00, stdev=52.98 lat (msec): min=31, max=392, avg=214.05, stdev=52.98 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 132], 10.00th=[ 146], 20.00th=[ 167], | 30.00th=[ 186], 40.00th=[ 201], 50.00th=[ 213], 60.00th=[ 226], | 70.00th=[ 239], 80.00th=[ 257], 90.00th=[ 288], 95.00th=[ 305], | 99.00th=[ 342], 99.50th=[ 363], 99.90th=[ 388], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 120, max= 280, per=24.97%, avg=201.98, stdev=37.74, samples=51 iops : min= 30, max= 70, avg=50.45, stdev= 9.42, samples=51 lat (msec) : 4=0.12%, 10=0.12%, 20=0.12%, 50=0.31%, 100=20.12% lat (msec) : 250=67.07%, 500=12.15% cpu : usr=0.17%, sys=0.95%, ctx=3043, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29052: Sat Nov 16 06:43:58 2024 read: IOPS=48, BW=193KiB/s (198kB/s)(4976KiB/25798msec) slat (usec): min=837, max=4349, avg=1685.45, stdev=416.51 clat (usec): min=1933, max=165883, avg=103809.45, stdev=20654.49 lat (msec): min=3, max=169, avg=105.50, stdev=20.72 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 73], 10.00th=[ 80], 20.00th=[ 89], | 30.00th=[ 95], 40.00th=[ 101], 50.00th=[ 106], 60.00th=[ 110], | 70.00th=[ 114], 80.00th=[ 121], 90.00th=[ 129], 95.00th=[ 134], | 99.00th=[ 148], 99.50th=[ 150], 99.90th=[ 165], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 48, max= 296, per=24.62%, avg=191.33, stdev=44.36, samples=51 iops : min= 12, max= 74, avg=47.80, stdev=11.11, samples=51 write: IOPS=51, BW=204KiB/s (209kB/s)(5264KiB/25798msec) slat (usec): min=13, max=2333, avg=53.42, stdev=68.63 clat (msec): min=82, max=392, avg=213.55, stdev=51.06 lat (msec): min=82, max=392, avg=213.60, stdev=51.06 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 136], 10.00th=[ 148], 20.00th=[ 167], | 30.00th=[ 186], 40.00th=[ 201], 50.00th=[ 211], 60.00th=[ 226], | 70.00th=[ 239], 80.00th=[ 255], 90.00th=[ 284], 95.00th=[ 300], | 99.00th=[ 342], 99.50th=[ 376], 99.90th=[ 384], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 128, max= 280, per=25.10%, avg=203.10, stdev=37.95, samples=51 iops : min= 32, max= 70, avg=50.75, stdev= 9.48, samples=51 lat (msec) : 2=0.04%, 4=0.12%, 10=0.04%, 20=0.12%, 50=0.20% lat (msec) : 100=19.30%, 250=68.40%, 500=11.80% cpu : usr=0.11%, sys=1.00%, ctx=2781, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29053: Sat Nov 16 06:43:58 2024 read: IOPS=48, BW=193KiB/s (198kB/s)(4984KiB/25820msec) slat (usec): min=672, max=4395, avg=1688.26, stdev=406.92 clat (msec): min=5, max=171, avg=103.49, stdev=19.20 lat (msec): min=7, max=173, avg=105.18, stdev=19.26 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 73], 10.00th=[ 81], 20.00th=[ 88], | 30.00th=[ 94], 40.00th=[ 100], 50.00th=[ 104], 60.00th=[ 109], | 70.00th=[ 113], 80.00th=[ 121], 90.00th=[ 129], 95.00th=[ 134], | 99.00th=[ 144], 99.50th=[ 148], 99.90th=[ 167], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 96, max= 360, per=24.80%, avg=192.69, stdev=58.21, samples=51 iops : min= 24, max= 90, avg=48.10, stdev=14.52, samples=51 write: IOPS=50, BW=204KiB/s (208kB/s)(5256KiB/25820msec) slat (usec): min=15, max=1842, avg=54.07, stdev=57.37 clat (msec): min=83, max=374, avg=214.16, stdev=50.01 lat (msec): min=83, max=375, avg=214.22, stdev=50.01 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 136], 10.00th=[ 148], 20.00th=[ 169], | 30.00th=[ 188], 40.00th=[ 203], 50.00th=[ 213], 60.00th=[ 226], | 70.00th=[ 239], 80.00th=[ 253], 90.00th=[ 284], 95.00th=[ 300], | 99.00th=[ 338], 99.50th=[ 359], 99.90th=[ 368], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 159, max= 272, per=24.92%, avg=201.63, stdev=27.36, samples=51 iops : min= 39, max= 68, avg=50.33, stdev= 6.88, samples=51 lat (msec) : 10=0.04%, 20=0.12%, 50=0.08%, 100=20.59%, 250=68.16% lat (msec) : 500=11.02% cpu : usr=0.12%, sys=0.96%, ctx=3088, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29054: Sat Nov 16 06:43:58 2024 read: IOPS=49, BW=196KiB/s (201kB/s)(5060KiB/25783msec) slat (usec): min=927, max=4948, avg=1715.20, stdev=470.72 clat (msec): min=17, max=185, avg=104.94, stdev=20.08 lat (msec): min=18, max=186, avg=106.65, stdev=20.15 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 73], 10.00th=[ 82], 20.00th=[ 90], | 30.00th=[ 95], 40.00th=[ 101], 50.00th=[ 106], 60.00th=[ 110], | 70.00th=[ 115], 80.00th=[ 121], 90.00th=[ 131], 95.00th=[ 138], | 99.00th=[ 155], 99.50th=[ 161], 99.90th=[ 167], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 96, max= 312, per=25.29%, avg=196.53, stdev=47.64, samples=51 iops : min= 24, max= 78, avg=49.12, stdev=11.90, samples=51 write: IOPS=50, BW=201KiB/s (206kB/s)(5180KiB/25783msec) slat (usec): min=13, max=363, avg=52.44, stdev=29.79 clat (msec): min=92, max=381, avg=213.75, stdev=50.18 lat (msec): min=93, max=381, avg=213.81, stdev=50.18 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 136], 10.00th=[ 148], 20.00th=[ 167], | 30.00th=[ 186], 40.00th=[ 201], 50.00th=[ 211], 60.00th=[ 224], | 70.00th=[ 241], 80.00th=[ 255], 90.00th=[ 284], 95.00th=[ 300], | 99.00th=[ 334], 99.50th=[ 355], 99.90th=[ 376], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 144, max= 304, per=24.68%, avg=199.67, stdev=33.68, samples=51 iops : min= 36, max= 76, avg=49.90, stdev= 8.42, samples=51 lat (msec) : 20=0.08%, 50=0.23%, 100=19.41%, 250=68.24%, 500=12.03% cpu : usr=0.14%, sys=0.99%, ctx=3065, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=777KiB/s (796kB/s), 193KiB/s-196KiB/s (198kB/s-201kB/s), io=19.6MiB (20.5MB), run=25712-25820msec WRITE: bw=809KiB/s (829kB/s), 201KiB/s-204KiB/s (206kB/s-209kB/s), io=20.4MiB (21.4MB), run=25712-25820msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=29078: Sat Nov 16 06:43:59 2024 read: IOPS=8, BW=320MiB/s (336MB/s)(40.0MiB/125msec) slat (nsec): min=5503.8k, max=5503.8k, avg=5503800.00, stdev= 0.00 clat (nsec): min=111938k, max=111938k, avg=111937815.00, stdev= 0.00 lat (nsec): min=117451k, max=117451k, avg=117451407.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 112], 50.00th=[ 112], 60.00th=[ 112], | 70.00th=[ 112], 80.00th=[ 112], 90.00th=[ 112], 95.00th=[ 112], | 99.00th=[ 112], 99.50th=[ 112], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.84%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=320MiB/s (336MB/s), 320MiB/s-320MiB/s (336MB/s-336MB/s), io=40.0MiB (41.9MB), run=125-125msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout