== sanity test 398b: DIO and buffer IO race ============== 18:54:09 (1717023249) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.5283 s, 32.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8082: Wed May 29 18:54:36 2024 read: IOPS=63, BW=252KiB/s (259kB/s)(5992KiB/23735msec) clat (usec): min=41, max=225710, avg=6698.04, stdev=8685.91 lat (usec): min=41, max=225711, avg=6699.83, stdev=8685.95 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 80], 10.00th=[ 88], 20.00th=[ 113], | 30.00th=[ 3130], 40.00th=[ 4621], 50.00th=[ 5735], 60.00th=[ 6849], | 70.00th=[ 8455], 80.00th=[ 10683], 90.00th=[ 14746], 95.00th=[ 17695], | 99.00th=[ 23987], 99.50th=[ 34866], 99.90th=[103285], 99.95th=[225444], | 99.99th=[225444] bw ( KiB/s): min= 55, max= 400, per=25.14%, avg=251.68, stdev=73.58, samples=47 iops : min= 13, max= 100, avg=62.79, stdev=18.45, samples=47 write: IOPS=66, BW=265KiB/s (272kB/s)(6296KiB/23735msec) clat (usec): min=467, max=53601, avg=8628.76, stdev=8328.40 lat (usec): min=468, max=53602, avg=8630.61, stdev=8328.37 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 709], 10.00th=[ 766], 20.00th=[ 947], | 30.00th=[ 1762], 40.00th=[ 4424], 50.00th=[ 6980], 60.00th=[ 8848], | 70.00th=[11338], 80.00th=[14615], 90.00th=[19792], 95.00th=[25297], | 99.00th=[35914], 99.50th=[39584], 99.90th=[49021], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 47, max= 496, per=25.11%, avg=263.11, stdev=73.31, samples=47 iops : min= 11, max= 124, avg=65.64, stdev=18.40, samples=47 lat (usec) : 50=0.10%, 100=7.45%, 250=5.44%, 500=0.39%, 750=4.69% lat (usec) : 1000=6.61% lat (msec) : 2=5.27%, 4=7.10%, 10=33.98%, 20=22.82%, 50=5.96% lat (msec) : 100=0.13%, 250=0.07% cpu : usr=0.18%, sys=17.49%, ctx=5261, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8084: Wed May 29 18:54:36 2024 read: IOPS=61, BW=248KiB/s (254kB/s)(5940KiB/23979msec) clat (usec): min=43, max=112097, avg=6649.93, stdev=6729.87 lat (usec): min=43, max=112099, avg=6651.34, stdev=6729.70 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 80], 10.00th=[ 88], 20.00th=[ 115], | 30.00th=[ 3392], 40.00th=[ 4621], 50.00th=[ 5604], 60.00th=[ 6980], | 70.00th=[ 8717], 80.00th=[ 10683], 90.00th=[ 14222], 95.00th=[ 17171], | 99.00th=[ 24511], 99.50th=[ 30016], 99.90th=[ 69731], 99.95th=[111674], | 99.99th=[111674] bw ( KiB/s): min= 48, max= 416, per=24.65%, avg=246.72, stdev=74.93, samples=47 iops : min= 12, max= 104, avg=61.53, stdev=18.75, samples=47 write: IOPS=66, BW=265KiB/s (271kB/s)(6348KiB/23979msec) clat (usec): min=410, max=219219, avg=8814.58, stdev=10595.17 lat (usec): min=411, max=219220, avg=8816.12, stdev=10594.98 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 709], 10.00th=[ 758], 20.00th=[ 947], | 30.00th=[ 1614], 40.00th=[ 3916], 50.00th=[ 6456], 60.00th=[ 8717], | 70.00th=[ 11338], 80.00th=[ 14222], 90.00th=[ 20579], 95.00th=[ 26084], | 99.00th=[ 39584], 99.50th=[ 45876], 99.90th=[108528], 99.95th=[219153], | 99.99th=[219153] bw ( KiB/s): min= 112, max= 440, per=24.83%, avg=260.19, stdev=76.07, samples=47 iops : min= 28, max= 110, avg=64.89, stdev=19.06, samples=47 lat (usec) : 50=0.23%, 100=6.97%, 250=5.11%, 500=0.78%, 750=4.75% lat (usec) : 1000=6.71% lat (msec) : 2=5.92%, 4=6.64%, 10=33.59%, 20=22.27%, 50=6.74% lat (msec) : 100=0.20%, 250=0.10% cpu : usr=0.20%, sys=17.34%, ctx=5205, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8087: Wed May 29 18:54:36 2024 read: IOPS=62, BW=252KiB/s (258kB/s)(6028KiB/23952msec) clat (usec): min=42, max=91288, avg=6705.97, stdev=6542.90 lat (usec): min=42, max=91289, avg=6707.46, stdev=6542.56 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 79], 10.00th=[ 87], 20.00th=[ 115], | 30.00th=[ 3589], 40.00th=[ 4752], 50.00th=[ 5866], 60.00th=[ 6980], | 70.00th=[ 8356], 80.00th=[10683], 90.00th=[14746], 95.00th=[17695], | 99.00th=[25297], 99.50th=[33162], 99.90th=[63177], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 88, max= 424, per=24.97%, avg=249.94, stdev=66.10, samples=47 iops : min= 22, max= 106, avg=62.32, stdev=16.56, samples=47 write: IOPS=65, BW=261KiB/s (268kB/s)(6260KiB/23952msec) clat (usec): min=426, max=245153, avg=8769.29, stdev=10811.40 lat (usec): min=427, max=245154, avg=8771.34, stdev=10811.07 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 701], 10.00th=[ 758], 20.00th=[ 938], | 30.00th=[ 1696], 40.00th=[ 4080], 50.00th=[ 6783], 60.00th=[ 8717], | 70.00th=[ 11207], 80.00th=[ 14615], 90.00th=[ 20317], 95.00th=[ 25297], | 99.00th=[ 32375], 99.50th=[ 43779], 99.90th=[105382], 99.95th=[244319], | 99.99th=[244319] bw ( KiB/s): min= 88, max= 408, per=24.71%, avg=258.96, stdev=63.74, samples=47 iops : min= 22, max= 102, avg=64.57, stdev=15.96, samples=47 lat (usec) : 50=0.16%, 100=7.36%, 250=4.72%, 500=0.39%, 750=4.69% lat (usec) : 1000=6.41% lat (msec) : 2=5.53%, 4=7.26%, 10=34.73%, 20=22.07%, 50=6.38% lat (msec) : 100=0.23%, 250=0.07% cpu : usr=0.15%, sys=17.40%, ctx=5211, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8089: Wed May 29 18:54:36 2024 read: IOPS=63, BW=254KiB/s (260kB/s)(6052KiB/23829msec) clat (usec): min=44, max=93864, avg=6589.45, stdev=6420.33 lat (usec): min=44, max=93864, avg=6590.64, stdev=6420.36 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 78], 10.00th=[ 87], 20.00th=[ 108], | 30.00th=[ 3097], 40.00th=[ 4686], 50.00th=[ 5735], 60.00th=[ 6915], | 70.00th=[ 8455], 80.00th=[10814], 90.00th=[14746], 95.00th=[18220], | 99.00th=[24249], 99.50th=[26870], 99.90th=[43779], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 135, max= 415, per=25.59%, avg=256.17, stdev=67.25, samples=46 iops : min= 33, max= 103, avg=63.80, stdev=16.82, samples=46 write: IOPS=65, BW=262KiB/s (268kB/s)(6236KiB/23829msec) clat (usec): min=449, max=256660, avg=8812.34, stdev=11244.26 lat (usec): min=450, max=256662, avg=8813.96, stdev=11244.19 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 701], 10.00th=[ 750], 20.00th=[ 914], | 30.00th=[ 1582], 40.00th=[ 3687], 50.00th=[ 6456], 60.00th=[ 8717], | 70.00th=[ 10945], 80.00th=[ 14091], 90.00th=[ 20841], 95.00th=[ 26608], | 99.00th=[ 39060], 99.50th=[ 53740], 99.90th=[ 92799], 99.95th=[256902], | 99.99th=[256902] bw ( KiB/s): min= 24, max= 400, per=24.64%, avg=258.19, stdev=65.19, samples=47 iops : min= 6, max= 100, avg=64.32, stdev=16.26, samples=47 lat (usec) : 50=0.07%, 100=8.40%, 250=4.98%, 500=0.62%, 750=4.95% lat (usec) : 1000=6.64% lat (msec) : 2=5.63%, 4=6.12%, 10=34.41%, 20=20.90%, 50=6.93% lat (msec) : 100=0.33%, 500=0.03% cpu : usr=0.17%, sys=17.29%, ctx=5077, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1001KiB/s (1025kB/s), 248KiB/s-254KiB/s (254kB/s-260kB/s), io=23.4MiB (24.6MB), run=23735-23979msec WRITE: bw=1048KiB/s (1074kB/s), 261KiB/s-265KiB/s (268kB/s-272kB/s), io=24.6MiB (25.7MB), run=23735-23979msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8083: Wed May 29 18:55:41 2024 read: IOPS=16, BW=67.7KiB/s (69.3kB/s)(5992KiB/88567msec) clat (usec): min=1986, max=253161, avg=7729.90, stdev=9086.05 lat (usec): min=1987, max=253163, avg=7731.63, stdev=9085.99 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 12], 90.00th=[ 16], 95.00th=[ 21], | 99.00th=[ 31], 99.50th=[ 36], 99.90th=[ 89], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 8, max= 168, per=25.08%, avg=67.46, stdev=36.50, samples=177 iops : min= 2, max= 42, avg=16.68, stdev= 9.15, samples=177 write: IOPS=17, BW=71.1KiB/s (72.8kB/s)(6296KiB/88567msec) clat (msec): min=14, max=167, avg=48.83, stdev=21.74 lat (msec): min=14, max=167, avg=48.83, stdev=21.74 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 23], 10.00th=[ 26], 20.00th=[ 31], | 30.00th=[ 35], 40.00th=[ 40], 50.00th=[ 45], 60.00th=[ 50], | 70.00th=[ 56], 80.00th=[ 64], 90.00th=[ 77], 95.00th=[ 92], | 99.00th=[ 121], 99.50th=[ 132], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 15, max= 136, per=25.09%, avg=70.76, stdev=24.80, samples=177 iops : min= 3, max= 34, avg=17.50, stdev= 6.25, samples=177 lat (msec) : 2=0.03%, 4=16.28%, 10=21.00%, 20=9.90%, 50=32.94% lat (msec) : 100=18.10%, 250=1.73%, 500=0.03% cpu : usr=0.12%, sys=3.68%, ctx=3792, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8085: Wed May 29 18:55:41 2024 read: IOPS=16, BW=66.7KiB/s (68.3kB/s)(5940KiB/89058msec) clat (usec): min=1766, max=223444, avg=7774.37, stdev=8836.82 lat (usec): min=1767, max=223445, avg=7776.25, stdev=8836.74 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 12], 90.00th=[ 16], 95.00th=[ 19], | 99.00th=[ 33], 99.50th=[ 46], 99.90th=[ 109], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 7, max= 199, per=24.85%, avg=66.85, stdev=36.42, samples=177 iops : min= 1, max= 49, avg=16.55, stdev= 9.13, samples=177 write: IOPS=17, BW=71.3KiB/s (72.0kB/s)(6348KiB/89058msec) clat (msec): min=14, max=152, avg=48.76, stdev=21.06 lat (msec): min=14, max=152, avg=48.76, stdev=21.06 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 23], 10.00th=[ 27], 20.00th=[ 32], | 30.00th=[ 36], 40.00th=[ 40], 50.00th=[ 45], 60.00th=[ 50], | 70.00th=[ 55], 80.00th=[ 65], 90.00th=[ 79], 95.00th=[ 92], | 99.00th=[ 115], 99.50th=[ 130], 99.90th=[ 142], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 15, max= 151, per=25.18%, avg=71.02, stdev=25.36, samples=178 iops : min= 3, max= 37, avg=17.59, stdev= 6.39, samples=178 lat (msec) : 2=0.10%, 4=14.91%, 10=21.71%, 20=10.87%, 50=32.68% lat (msec) : 100=18.26%, 250=1.46% cpu : usr=0.08%, sys=3.79%, ctx=3843, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8086: Wed May 29 18:55:41 2024 read: IOPS=17, BW=68.3KiB/s (69.9kB/s)(6028KiB/88275msec) clat (usec): min=2178, max=88519, avg=7898.16, stdev=6878.65 lat (usec): min=2179, max=88520, avg=7900.03, stdev=6878.44 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2835], 10.00th=[ 3097], 20.00th=[ 3556], | 30.00th=[ 3982], 40.00th=[ 4490], 50.00th=[ 5145], 60.00th=[ 6259], | 70.00th=[ 8455], 80.00th=[11863], 90.00th=[16319], 95.00th=[19792], | 99.00th=[33424], 99.50th=[40633], 99.90th=[86508], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 7, max= 200, per=25.45%, avg=68.45, stdev=35.45, samples=175 iops : min= 1, max= 50, avg=16.95, stdev= 8.90, samples=175 write: IOPS=17, BW=70.9KiB/s (72.6kB/s)(6260KiB/88275msec) clat (msec): min=15, max=280, avg=48.71, stdev=21.41 lat (msec): min=15, max=280, avg=48.72, stdev=21.41 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 24], 10.00th=[ 27], 20.00th=[ 32], | 30.00th=[ 36], 40.00th=[ 40], 50.00th=[ 45], 60.00th=[ 50], | 70.00th=[ 56], 80.00th=[ 64], 90.00th=[ 75], 95.00th=[ 88], | 99.00th=[ 115], 99.50th=[ 132], 99.90th=[ 155], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 15, max= 136, per=25.05%, avg=70.65, stdev=25.02, samples=176 iops : min= 3, max= 34, avg=17.50, stdev= 6.29, samples=176 lat (msec) : 4=14.81%, 10=21.71%, 20=11.13%, 50=32.36%, 100=18.55% lat (msec) : 250=1.40%, 500=0.03% cpu : usr=0.06%, sys=3.78%, ctx=3825, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8088: Wed May 29 18:55:41 2024 read: IOPS=17, BW=68.1KiB/s (69.8kB/s)(6052KiB/88842msec) clat (usec): min=1791, max=114677, avg=7894.95, stdev=7588.21 lat (usec): min=1792, max=114679, avg=7896.53, stdev=7588.16 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 12], 90.00th=[ 16], 95.00th=[ 21], | 99.00th=[ 36], 99.50th=[ 46], 99.90th=[ 89], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 7, max= 208, per=25.39%, avg=68.30, stdev=35.64, samples=176 iops : min= 1, max= 52, avg=16.89, stdev= 8.92, samples=176 write: IOPS=17, BW=70.2KiB/s (71.9kB/s)(6236KiB/88842msec) clat (msec): min=13, max=255, avg=49.24, stdev=21.81 lat (msec): min=13, max=255, avg=49.24, stdev=21.81 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 24], 10.00th=[ 27], 20.00th=[ 32], | 30.00th=[ 36], 40.00th=[ 41], 50.00th=[ 45], 60.00th=[ 51], | 70.00th=[ 56], 80.00th=[ 65], 90.00th=[ 78], 95.00th=[ 91], | 99.00th=[ 121], 99.50th=[ 130], 99.90th=[ 176], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 31, max= 144, per=24.75%, avg=69.81, stdev=24.93, samples=177 iops : min= 7, max= 36, avg=17.27, stdev= 6.27, samples=177 lat (msec) : 2=0.10%, 4=15.59%, 10=21.22%, 20=10.64%, 50=31.61% lat (msec) : 100=19.27%, 250=1.53%, 500=0.03% cpu : usr=0.07%, sys=3.79%, ctx=3856, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=270KiB/s (276kB/s), 66.7KiB/s-68.3KiB/s (68.3kB/s-69.9kB/s), io=23.4MiB (24.6MB), run=88275-89058msec WRITE: bw=282KiB/s (289kB/s), 70.2KiB/s-71.3KiB/s (71.9kB/s-72.0kB/s), io=24.6MiB (25.7MB), run=88275-89058msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8173: Wed May 29 18:55:51 2024 read: IOPS=40, BW=651KiB/s (667kB/s)(5920KiB/9095msec) clat (usec): min=54, max=77083, avg=18077.90, stdev=14463.16 lat (usec): min=54, max=77083, avg=18079.04, stdev=14463.17 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 84], 10.00th=[ 95], 20.00th=[ 110], | 30.00th=[ 215], 40.00th=[17695], 50.00th=[20579], 60.00th=[23200], | 70.00th=[26346], 80.00th=[30802], 90.00th=[36439], 95.00th=[40633], | 99.00th=[47449], 99.50th=[51119], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 416, max= 1117, per=24.79%, avg=642.89, stdev=202.88, samples=18 iops : min= 26, max= 69, avg=39.94, stdev=12.53, samples=18 write: IOPS=43, BW=700KiB/s (717kB/s)(6368KiB/9095msec) clat (usec): min=634, max=55041, avg=5998.97, stdev=6961.34 lat (usec): min=635, max=55057, avg=6000.66, stdev=6961.68 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 947], 10.00th=[ 1045], 20.00th=[ 1188], | 30.00th=[ 1319], 40.00th=[ 1647], 50.00th=[ 2507], 60.00th=[ 5080], | 70.00th=[ 7570], 80.00th=[10159], 90.00th=[15008], 95.00th=[20841], | 99.00th=[27395], 99.50th=[41157], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 383, max= 1469, per=25.53%, avg=696.33, stdev=276.58, samples=18 iops : min= 23, max= 91, avg=43.28, stdev=17.16, samples=18 lat (usec) : 100=6.51%, 250=8.20%, 500=0.39%, 750=0.78%, 1000=3.91% lat (msec) : 2=19.14%, 4=6.12%, 10=12.24%, 20=14.71%, 50=27.60% lat (msec) : 100=0.39% cpu : usr=0.16%, sys=15.43%, ctx=1768, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8175: Wed May 29 18:55:51 2024 read: IOPS=39, BW=638KiB/s (654kB/s)(5744KiB/8998msec) clat (usec): min=48, max=52484, avg=18294.12, stdev=14397.37 lat (usec): min=48, max=52486, avg=18295.25, stdev=14397.43 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 82], 10.00th=[ 92], 20.00th=[ 106], | 30.00th=[ 217], 40.00th=[17957], 50.00th=[20841], 60.00th=[23987], | 70.00th=[27132], 80.00th=[30540], 90.00th=[35914], 95.00th=[40109], | 99.00th=[46924], 99.50th=[51119], 99.90th=[52691], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 480, max= 990, per=24.30%, avg=630.00, stdev=148.16, samples=17 iops : min= 30, max= 61, avg=39.12, stdev= 9.03, samples=17 write: IOPS=45, BW=727KiB/s (745kB/s)(6544KiB/8998msec) clat (usec): min=683, max=63567, avg=5906.22, stdev=7618.55 lat (usec): min=683, max=63569, avg=5908.53, stdev=7618.49 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 955], 10.00th=[ 1012], 20.00th=[ 1123], | 30.00th=[ 1287], 40.00th=[ 1582], 50.00th=[ 2409], 60.00th=[ 4113], | 70.00th=[ 6783], 80.00th=[ 9896], 90.00th=[15270], 95.00th=[19006], | 99.00th=[35390], 99.50th=[44827], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 384, max= 1245, per=26.20%, avg=714.65, stdev=224.53, samples=17 iops : min= 24, max= 77, avg=44.41, stdev=13.85, samples=17 lat (usec) : 50=0.13%, 100=7.81%, 250=6.25%, 500=0.52%, 750=0.65% lat (usec) : 1000=4.56% lat (msec) : 2=19.40%, 4=7.42%, 10=11.33%, 20=14.97%, 50=26.43% lat (msec) : 100=0.52% cpu : usr=0.10%, sys=15.17%, ctx=1693, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8177: Wed May 29 18:55:51 2024 read: IOPS=42, BW=677KiB/s (694kB/s)(6256KiB/9235msec) clat (usec): min=46, max=83262, avg=17506.10, stdev=15071.70 lat (usec): min=46, max=83264, avg=17507.37, stdev=15071.51 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 85], 10.00th=[ 94], 20.00th=[ 108], | 30.00th=[ 155], 40.00th=[16712], 50.00th=[20317], 60.00th=[22676], | 70.00th=[25822], 80.00th=[29754], 90.00th=[34866], 95.00th=[40633], | 99.00th=[55313], 99.50th=[63177], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 448, max= 1120, per=25.88%, avg=671.00, stdev=171.86, samples=18 iops : min= 28, max= 70, avg=41.67, stdev=10.78, samples=18 write: IOPS=40, BW=653KiB/s (669kB/s)(6032KiB/9235msec) clat (usec): min=670, max=61040, avg=6295.91, stdev=7578.76 lat (usec): min=671, max=61042, avg=6301.72, stdev=7576.52 clat percentiles (usec): | 1.00th=[ 848], 5.00th=[ 988], 10.00th=[ 1037], 20.00th=[ 1156], | 30.00th=[ 1385], 40.00th=[ 1647], 50.00th=[ 2573], 60.00th=[ 5473], | 70.00th=[ 7635], 80.00th=[10683], 90.00th=[15533], 95.00th=[20317], | 99.00th=[37487], 99.50th=[44303], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 192, max= 1120, per=23.37%, avg=637.67, stdev=239.47, samples=18 iops : min= 12, max= 70, avg=39.61, stdev=14.97, samples=18 lat (usec) : 50=0.26%, 100=6.90%, 250=9.24%, 500=1.30%, 750=0.13% lat (usec) : 1000=3.26% lat (msec) : 2=18.62%, 4=5.47%, 10=11.46%, 20=15.23%, 50=27.08% lat (msec) : 100=1.04% cpu : usr=0.21%, sys=14.75%, ctx=1766, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8179: Wed May 29 18:55:51 2024 read: IOPS=42, BW=678KiB/s (694kB/s)(6032KiB/8897msec) clat (usec): min=54, max=66088, avg=17758.37, stdev=14234.57 lat (usec): min=55, max=66089, avg=17759.90, stdev=14235.23 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 86], 10.00th=[ 91], 20.00th=[ 113], | 30.00th=[ 223], 40.00th=[17171], 50.00th=[20579], 60.00th=[23200], | 70.00th=[25822], 80.00th=[30278], 90.00th=[35390], 95.00th=[39060], | 99.00th=[50594], 99.50th=[51119], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 512, max= 894, per=25.75%, avg=667.65, stdev=93.41, samples=17 iops : min= 32, max= 55, avg=41.47, stdev= 5.64, samples=17 write: IOPS=43, BW=703KiB/s (720kB/s)(6256KiB/8897msec) clat (usec): min=754, max=62440, avg=5592.59, stdev=6988.88 lat (usec): min=755, max=62441, avg=5594.40, stdev=6988.88 clat percentiles (usec): | 1.00th=[ 816], 5.00th=[ 963], 10.00th=[ 1037], 20.00th=[ 1172], | 30.00th=[ 1303], 40.00th=[ 1532], 50.00th=[ 2114], 60.00th=[ 3982], | 70.00th=[ 6980], 80.00th=[ 9372], 90.00th=[14091], 95.00th=[17695], | 99.00th=[33817], 99.50th=[39584], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 352, max= 1021, per=25.51%, avg=695.82, stdev=190.04, samples=17 iops : min= 22, max= 63, avg=43.24, stdev=11.73, samples=17 lat (usec) : 100=6.90%, 250=8.07%, 500=0.91%, 750=0.26%, 1000=4.04% lat (msec) : 2=21.48%, 4=5.34%, 10=11.07%, 20=14.19%, 50=27.08% lat (msec) : 100=0.65% cpu : usr=0.12%, sys=14.89%, ctx=1721, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2594KiB/s (2656kB/s), 638KiB/s-678KiB/s (654kB/s-694kB/s), io=23.4MiB (24.5MB), run=8897-9235msec WRITE: bw=2729KiB/s (2794kB/s), 653KiB/s-727KiB/s (669kB/s-745kB/s), io=24.6MiB (25.8MB), run=8897-9235msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8174: Wed May 29 18:56:10 2024 read: IOPS=13, BW=211KiB/s (216kB/s)(5920KiB/28096msec) clat (usec): min=3341, max=85338, avg=10649.81, stdev=9069.31 lat (usec): min=3341, max=85339, avg=10651.65, stdev=9069.33 clat percentiles (usec): | 1.00th=[ 3556], 5.00th=[ 3949], 10.00th=[ 4228], 20.00th=[ 4883], | 30.00th=[ 5473], 40.00th=[ 6259], 50.00th=[ 7373], 60.00th=[ 9372], | 70.00th=[11731], 80.00th=[14746], 90.00th=[19268], 95.00th=[27395], | 99.00th=[49546], 99.50th=[62653], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 32, max= 704, per=25.40%, avg=213.91, stdev=109.42, samples=55 iops : min= 2, max= 44, avg=13.13, stdev= 6.91, samples=55 write: IOPS=14, BW=227KiB/s (232kB/s)(6368KiB/28096msec) clat (msec): min=24, max=152, avg=60.66, stdev=23.68 lat (msec): min=24, max=152, avg=60.66, stdev=23.68 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 32], 10.00th=[ 35], 20.00th=[ 41], | 30.00th=[ 46], 40.00th=[ 52], 50.00th=[ 57], 60.00th=[ 64], | 70.00th=[ 69], 80.00th=[ 75], 90.00th=[ 92], 95.00th=[ 112], | 99.00th=[ 134], 99.50th=[ 153], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 96, max= 352, per=25.49%, avg=225.84, stdev=65.94, samples=56 iops : min= 6, max= 22, avg=13.86, stdev= 4.11, samples=56 lat (msec) : 4=2.60%, 10=27.60%, 20=13.67%, 50=23.70%, 100=28.65% lat (msec) : 250=3.78% cpu : usr=0.00%, sys=3.70%, ctx=1024, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8176: Wed May 29 18:56:10 2024 read: IOPS=12, BW=202KiB/s (207kB/s)(5744KiB/28436msec) clat (usec): min=2930, max=83029, avg=11051.47, stdev=10083.54 lat (usec): min=2931, max=83030, avg=11052.85, stdev=10083.47 clat percentiles (usec): | 1.00th=[ 3195], 5.00th=[ 3818], 10.00th=[ 4178], 20.00th=[ 4883], | 30.00th=[ 5604], 40.00th=[ 6718], 50.00th=[ 7701], 60.00th=[ 9241], | 70.00th=[11994], 80.00th=[14746], 90.00th=[20055], 95.00th=[32375], | 99.00th=[56361], 99.50th=[71828], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 31, max= 608, per=24.21%, avg=203.87, stdev=114.37, samples=55 iops : min= 1, max= 38, avg=12.45, stdev= 7.22, samples=55 write: IOPS=14, BW=230KiB/s (236kB/s)(6544KiB/28436msec) clat (msec): min=23, max=149, avg=59.78, stdev=23.37 lat (msec): min=24, max=149, avg=59.79, stdev=23.37 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 31], 10.00th=[ 36], 20.00th=[ 41], | 30.00th=[ 45], 40.00th=[ 49], 50.00th=[ 55], 60.00th=[ 62], | 70.00th=[ 68], 80.00th=[ 78], 90.00th=[ 92], 95.00th=[ 107], | 99.00th=[ 136], 99.50th=[ 140], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 95, max= 352, per=25.67%, avg=227.41, stdev=65.83, samples=56 iops : min= 5, max= 22, avg=13.95, stdev= 4.19, samples=56 lat (msec) : 4=2.99%, 10=26.43%, 20=12.76%, 50=26.17%, 100=27.73% lat (msec) : 250=3.91% cpu : usr=0.08%, sys=3.57%, ctx=1036, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8178: Wed May 29 18:56:10 2024 read: IOPS=14, BW=238KiB/s (244kB/s)(6256KiB/26233msec) clat (usec): min=2960, max=75317, avg=10045.83, stdev=8416.37 lat (usec): min=2962, max=75319, avg=10048.06, stdev=8416.36 clat percentiles (usec): | 1.00th=[ 3359], 5.00th=[ 3818], 10.00th=[ 4146], 20.00th=[ 4686], | 30.00th=[ 5276], 40.00th=[ 5866], 50.00th=[ 6915], 60.00th=[ 8356], | 70.00th=[11338], 80.00th=[14091], 90.00th=[18220], 95.00th=[25560], | 99.00th=[49546], 99.50th=[53216], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 63, max= 672, per=29.11%, avg=245.08, stdev=124.24, samples=51 iops : min= 3, max= 42, avg=15.08, stdev= 7.80, samples=51 write: IOPS=14, BW=230KiB/s (235kB/s)(6032KiB/26233msec) clat (msec): min=21, max=140, avg=59.11, stdev=21.85 lat (msec): min=21, max=140, avg=59.12, stdev=21.85 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 32], 10.00th=[ 35], 20.00th=[ 39], | 30.00th=[ 47], 40.00th=[ 53], 50.00th=[ 57], 60.00th=[ 62], | 70.00th=[ 66], 80.00th=[ 73], 90.00th=[ 89], 95.00th=[ 107], | 99.00th=[ 122], 99.50th=[ 133], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 126, max= 384, per=25.88%, avg=229.27, stdev=65.74, samples=52 iops : min= 7, max= 24, avg=14.10, stdev= 4.07, samples=52 lat (msec) : 4=3.91%, 10=29.95%, 20=12.76%, 50=21.74%, 100=28.78% lat (msec) : 250=2.86% cpu : usr=0.00%, sys=3.82%, ctx=1035, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8180: Wed May 29 18:56:10 2024 read: IOPS=13, BW=219KiB/s (224kB/s)(6032KiB/27580msec) clat (usec): min=2909, max=71221, avg=10424.48, stdev=8583.41 lat (usec): min=2910, max=71222, avg=10426.49, stdev=8583.79 clat percentiles (usec): | 1.00th=[ 3294], 5.00th=[ 3654], 10.00th=[ 4146], 20.00th=[ 4948], | 30.00th=[ 5604], 40.00th=[ 6259], 50.00th=[ 7504], 60.00th=[ 9634], | 70.00th=[11469], 80.00th=[14877], 90.00th=[18744], 95.00th=[22414], | 99.00th=[52167], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 32, max= 512, per=25.75%, avg=216.80, stdev=104.50, samples=55 iops : min= 2, max= 32, avg=13.38, stdev= 6.58, samples=55 write: IOPS=14, BW=227KiB/s (232kB/s)(6256KiB/27580msec) clat (msec): min=25, max=182, avg=60.44, stdev=23.75 lat (msec): min=25, max=182, avg=60.45, stdev=23.75 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 32], 10.00th=[ 36], 20.00th=[ 42], | 30.00th=[ 47], 40.00th=[ 52], 50.00th=[ 58], 60.00th=[ 62], | 70.00th=[ 67], 80.00th=[ 77], 90.00th=[ 89], 95.00th=[ 107], | 99.00th=[ 142], 99.50th=[ 171], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 96, max= 383, per=25.59%, avg=226.71, stdev=66.45, samples=55 iops : min= 6, max= 23, avg=14.00, stdev= 4.19, samples=55 lat (msec) : 4=3.91%, 10=26.56%, 20=15.23%, 50=21.22%, 100=30.08% lat (msec) : 250=2.99% cpu : usr=0.05%, sys=3.62%, ctx=1016, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=842KiB/s (863kB/s), 202KiB/s-238KiB/s (207kB/s-244kB/s), io=23.4MiB (24.5MB), run=26233-28436msec WRITE: bw=886KiB/s (907kB/s), 227KiB/s-230KiB/s (232kB/s-236kB/s), io=24.6MiB (25.8MB), run=26233-28436msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8209: Wed May 29 18:56:19 2024 read: IOPS=2, BW=2159KiB/s (2211kB/s)(7168KiB/3320msec) clat (usec): min=444, max=1583.0k, avg=434120.64, stdev=640507.28 lat (usec): min=446, max=1583.0k, avg=434122.08, stdev=640507.60 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 445], 10.00th=[ 445], | 20.00th=[ 537], 30.00th=[ 766], 40.00th=[ 766], | 50.00th=[ 56361], 60.00th=[ 316670], 70.00th=[ 316670], | 80.00th=[1082131], 90.00th=[1585447], 95.00th=[1585447], | 99.00th=[1585447], 99.50th=[1585447], 99.90th=[1585447], | 99.95th=[1585447], 99.99th=[1585447] bw ( KiB/s): min= 2043, max= 2048, per=81.49%, avg=2046.33, stdev= 2.89, samples=3 iops : min= 1, max= 2, avg= 1.67, stdev= 0.58, samples=3 write: IOPS=1, BW=1542KiB/s (1579kB/s)(5120KiB/3320msec) clat (usec): min=30610, max=96590, avg=53622.07, stdev=27067.13 lat (usec): min=30627, max=96671, avg=53685.97, stdev=27079.74 clat percentiles (usec): | 1.00th=[30540], 5.00th=[30540], 10.00th=[30540], 20.00th=[30540], | 30.00th=[31327], 40.00th=[31327], 50.00th=[49546], 60.00th=[49546], | 70.00th=[60031], 80.00th=[60031], 90.00th=[96994], 95.00th=[96994], | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 2048, max= 2048, per=58.26%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33%, 1000=8.33% lat (msec) : 50=25.00%, 100=25.00%, 500=8.33% cpu : usr=0.03%, sys=35.19%, ctx=564, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8212: Wed May 29 18:56:20 2024 read: IOPS=0, BW=634KiB/s (649kB/s)(2048KiB/3229msec) clat (msec): min=54, max=388, avg=221.47, stdev=236.48 lat (msec): min=54, max=388, avg=221.47, stdev=236.48 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 55], 10.00th=[ 55], 20.00th=[ 55], | 30.00th=[ 55], 40.00th=[ 55], 50.00th=[ 55], 60.00th=[ 388], | 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 2043, max= 2048, per=81.46%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3171KiB/s (3247kB/s)(10.0MiB/3229msec) clat (msec): min=37, max=1227, avg=277.69, stdev=450.40 lat (msec): min=37, max=1227, avg=277.77, stdev=450.39 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 39], | 30.00th=[ 40], 40.00th=[ 48], 50.00th=[ 80], 60.00th=[ 83], | 70.00th=[ 105], 80.00th=[ 105], 90.00th=[ 1028], 95.00th=[ 1234], | 99.00th=[ 1234], 99.50th=[ 1234], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 4096, max= 6144, per=100.00%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 50=33.33%, 100=33.33%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=14.00%, ctx=132, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8214: Wed May 29 18:56:20 2024 read: IOPS=0, BW=753KiB/s (771kB/s)(6144KiB/8156msec) clat (msec): min=2, max=1435, avg=700.04, stdev=630.95 lat (msec): min=2, max=1435, avg=700.04, stdev=630.95 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 73], | 30.00th=[ 73], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 1150], | 70.00th=[ 1200], 80.00th=[ 1200], 90.00th=[ 1435], 95.00th=[ 1435], | 99.00th=[ 1435], 99.50th=[ 1435], 99.90th=[ 1435], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 2048, max= 4096, per=100.00%, avg=3410.33, stdev=1179.82, samples=3 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=3 write: IOPS=0, BW=753KiB/s (771kB/s)(6144KiB/8156msec) clat (msec): min=40, max=3124, avg=658.29, stdev=1212.87 lat (msec): min=40, max=3124, avg=658.67, stdev=1212.70 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 85], | 30.00th=[ 85], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 184], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 3138], 95.00th=[ 3138], | 99.00th=[ 3138], 99.50th=[ 3138], 99.90th=[ 3138], 99.95th=[ 3138], | 99.99th=[ 3138] bw ( KiB/s): min= 2043, max= 4096, per=69.89%, avg=2456.60, stdev=916.46, samples=5 iops : min= 1, max= 4, avg= 2.20, stdev= 1.10, samples=5 lat (msec) : 4=8.33%, 50=8.33%, 100=16.67%, 250=16.67%, 500=16.67% cpu : usr=0.10%, sys=16.09%, ctx=836, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8216: Wed May 29 18:56:20 2024 read: IOPS=0, BW=629KiB/s (644kB/s)(5120KiB/8140msec) clat (msec): min=8, max=1538, avg=677.81, stdev=593.37 lat (msec): min=8, max=1538, avg=677.81, stdev=593.37 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 9], 10.00th=[ 9], 20.00th=[ 9], | 30.00th=[ 284], 40.00th=[ 284], 50.00th=[ 625], 60.00th=[ 625], | 70.00th=[ 927], 80.00th=[ 927], 90.00th=[ 1536], 95.00th=[ 1536], | 99.00th=[ 1536], 99.50th=[ 1536], 99.90th=[ 1536], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 2043, max= 4063, per=100.00%, avg=2716.33, stdev=1166.25, samples=3 iops : min= 1, max= 3, avg= 1.67, stdev= 1.15, samples=3 write: IOPS=0, BW=881KiB/s (902kB/s)(7168KiB/8140msec) clat (msec): min=46, max=3012, avg=678.37, stdev=1105.91 lat (msec): min=46, max=3012, avg=678.44, stdev=1105.92 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 79], | 30.00th=[ 132], 40.00th=[ 132], 50.00th=[ 136], 60.00th=[ 153], | 70.00th=[ 153], 80.00th=[ 1183], 90.00th=[ 3004], 95.00th=[ 3004], | 99.00th=[ 3004], 99.50th=[ 3004], 99.90th=[ 3004], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 2048, max= 6131, per=100.00%, avg=4770.00, stdev=2357.32, samples=3 iops : min= 2, max= 5, avg= 4.00, stdev= 1.73, samples=3 lat (msec) : 10=8.33%, 50=8.33%, 100=8.33%, 250=25.00%, 500=8.33% lat (msec) : 750=8.33%, 1000=8.33% cpu : usr=0.10%, sys=8.02%, ctx=628, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2511KiB/s (2571kB/s), 629KiB/s-2159KiB/s (644kB/s-2211kB/s), io=20.0MiB (20.0MB), run=3229-8156msec WRITE: bw=3515KiB/s (3600kB/s), 753KiB/s-3171KiB/s (771kB/s-3247kB/s), io=28.0MiB (29.4MB), run=3229-8156msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8210: Wed May 29 18:56:20 2024 read: IOPS=1, BW=1815KiB/s (1859kB/s)(7168KiB/3949msec) clat (msec): min=24, max=1594, avg=476.11, stdev=616.08 lat (msec): min=24, max=1594, avg=476.11, stdev=616.08 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 78], | 30.00th=[ 96], 40.00th=[ 96], 50.00th=[ 106], 60.00th=[ 355], | 70.00th=[ 355], 80.00th=[ 1083], 90.00th=[ 1603], 95.00th=[ 1603], | 99.00th=[ 1603], 99.50th=[ 1603], 99.90th=[ 1603], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min= 2043, max= 4087, per=100.00%, avg=2556.50, stdev=1020.34, samples=4 iops : min= 1, max= 3, avg= 2.00, stdev= 0.82, samples=4 write: IOPS=1, BW=1297KiB/s (1328kB/s)(5120KiB/3949msec) clat (msec): min=83, max=214, avg=122.52, stdev=53.66 lat (msec): min=83, max=214, avg=122.60, stdev=53.66 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 84], 10.00th=[ 84], 20.00th=[ 84], | 30.00th=[ 90], 40.00th=[ 90], 50.00th=[ 103], 60.00th=[ 103], | 70.00th=[ 123], 80.00th=[ 123], 90.00th=[ 215], 95.00th=[ 215], | 99.00th=[ 215], 99.50th=[ 215], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 2043, max= 2048, per=58.46%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 50=8.33%, 100=33.33%, 250=33.33%, 500=8.33% cpu : usr=0.03%, sys=2.99%, ctx=47, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8211: Wed May 29 18:56:20 2024 read: IOPS=0, BW=368KiB/s (377kB/s)(2048KiB/5561msec) clat (msec): min=36, max=1877, avg=956.96, stdev=1301.86 lat (msec): min=36, max=1877, avg=956.96, stdev=1301.86 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 37], 60.00th=[ 1871], | 70.00th=[ 1871], 80.00th=[ 1871], 90.00th=[ 1871], 95.00th=[ 1871], | 99.00th=[ 1871], 99.50th=[ 1871], 99.90th=[ 1871], 99.95th=[ 1871], | 99.99th=[ 1871] bw ( KiB/s): min= 2048, max= 2048, per=81.95%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=1, BW=1841KiB/s (1886kB/s)(10.0MiB/5561msec) clat (msec): min=65, max=1576, avg=364.28, stdev=553.48 lat (msec): min=65, max=1577, avg=364.37, stdev=553.49 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 66], 10.00th=[ 66], 20.00th=[ 72], | 30.00th=[ 84], 40.00th=[ 93], 50.00th=[ 100], 60.00th=[ 129], | 70.00th=[ 150], 80.00th=[ 150], 90.00th=[ 1234], 95.00th=[ 1569], | 99.00th=[ 1569], 99.50th=[ 1569], 99.90th=[ 1569], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 4096, max= 8126, per=100.00%, avg=6122.00, stdev=2015.09, samples=3 iops : min= 4, max= 7, avg= 5.67, stdev= 1.53, samples=3 lat (msec) : 50=8.33%, 100=41.67%, 250=25.00% cpu : usr=0.02%, sys=2.25%, ctx=57, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8213: Wed May 29 18:56:20 2024 read: IOPS=0, BW=750KiB/s (768kB/s)(6144KiB/8193msec) clat (msec): min=25, max=1460, avg=490.31, stdev=586.59 lat (msec): min=25, max=1460, avg=490.31, stdev=586.59 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 59], | 30.00th=[ 59], 40.00th=[ 95], 50.00th=[ 95], 60.00th=[ 359], | 70.00th=[ 944], 80.00th=[ 944], 90.00th=[ 1469], 95.00th=[ 1469], | 99.00th=[ 1469], 99.50th=[ 1469], 99.90th=[ 1469], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 1809, max= 6131, per=100.00%, avg=3249.67, stdev=2495.31, samples=3 iops : min= 1, max= 5, avg= 2.33, stdev= 2.31, samples=3 write: IOPS=0, BW=750KiB/s (768kB/s)(6144KiB/8193msec) clat (msec): min=80, max=3122, avg=874.82, stdev=1252.64 lat (msec): min=80, max=3122, avg=874.90, stdev=1252.65 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 81], 10.00th=[ 81], 20.00th=[ 91], | 30.00th=[ 91], 40.00th=[ 159], 50.00th=[ 159], 60.00th=[ 180], | 70.00th=[ 1620], 80.00th=[ 1620], 90.00th=[ 3138], 95.00th=[ 3138], | 99.00th=[ 3138], 99.50th=[ 3138], 99.90th=[ 3138], 99.95th=[ 3138], | 99.99th=[ 3138] bw ( KiB/s): min= 2043, max= 4096, per=84.31%, avg=2950.00, stdev=1065.34, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.50, samples=4 lat (msec) : 50=8.33%, 100=33.33%, 250=16.67%, 500=8.33%, 1000=8.33% cpu : usr=0.02%, sys=1.44%, ctx=46, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8215: Wed May 29 18:56:20 2024 read: IOPS=1, BW=1207KiB/s (1236kB/s)(5120KiB/4241msec) clat (msec): min=22, max=300, avg=84.73, stdev=120.95 lat (msec): min=22, max=300, avg=84.73, stdev=120.95 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 23], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 29], 60.00th=[ 29], | 70.00th=[ 44], 80.00th=[ 44], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 8110, max= 8110, per=100.00%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1690KiB/s (1731kB/s)(7168KiB/4241msec) clat (msec): min=75, max=2997, avg=544.75, stdev=1081.91 lat (msec): min=75, max=2997, avg=544.84, stdev=1081.90 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 77], 10.00th=[ 77], 20.00th=[ 131], | 30.00th=[ 131], 40.00th=[ 131], 50.00th=[ 153], 60.00th=[ 159], | 70.00th=[ 159], 80.00th=[ 169], 90.00th=[ 3004], 95.00th=[ 3004], | 99.00th=[ 3004], 99.50th=[ 3004], 99.90th=[ 3004], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 2027, max= 6144, per=97.35%, avg=3406.33, stdev=2370.91, samples=3 iops : min= 1, max= 6, avg= 3.00, stdev= 2.65, samples=3 lat (msec) : 50=33.33%, 100=8.33%, 250=41.67%, 500=8.33% cpu : usr=0.00%, sys=2.74%, ctx=45, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2500KiB/s (2560kB/s), 368KiB/s-1815KiB/s (377kB/s-1859kB/s), io=20.0MiB (20.0MB), run=3949-8193msec WRITE: bw=3500KiB/s (3584kB/s), 750KiB/s-1841KiB/s (768kB/s-1886kB/s), io=28.0MiB (29.4MB), run=3949-8193msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8229: Wed May 29 18:56:24 2024 read: IOPS=1, BW=6923KiB/s (7089kB/s)(12.0MiB/1775msec) clat (msec): min=2, max=1389, avg=566.95, stdev=728.63 lat (msec): min=2, max=1389, avg=566.96, stdev=728.63 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 3], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 1385], 80.00th=[ 1385], 90.00th=[ 1385], 95.00th=[ 1385], | 99.00th=[ 1385], 99.50th=[ 1385], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 4=33.33%, 500=33.33% cpu : usr=0.11%, sys=51.97%, ctx=351, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8230: Wed May 29 18:56:24 2024 read: IOPS=0, BW=1344KiB/s (1376kB/s)(4096KiB/3048msec) clat (nsec): min=309758k, max=309758k, avg=309758015.00, stdev= 0.00 lat (nsec): min=309760k, max=309760k, avg=309760485.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 309], 10.00th=[ 309], 20.00th=[ 309], | 30.00th=[ 309], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2688KiB/s (2752kB/s)(8192KiB/3048msec) clat (msec): min=440, max=2224, avg=1332.19, stdev=1261.28 lat (msec): min=440, max=2224, avg=1332.54, stdev=1261.26 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 439], 10.00th=[ 439], 20.00th=[ 439], | 30.00th=[ 439], 40.00th=[ 439], 50.00th=[ 439], 60.00th=[ 2232], | 70.00th=[ 2232], 80.00th=[ 2232], 90.00th=[ 2232], 95.00th=[ 2232], | 99.00th=[ 2232], 99.50th=[ 2232], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 8175, max= 8175, per=76.05%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=10.53%, ctx=160, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8231: Wed May 29 18:56:24 2024 write: IOPS=1, BW=4583KiB/s (4693kB/s)(12.0MiB/2681msec) clat (msec): min=190, max=2037, avg=885.63, stdev=1004.31 lat (msec): min=191, max=2037, avg=885.95, stdev=1004.38 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 192], 10.00th=[ 192], 20.00th=[ 192], | 30.00th=[ 192], 40.00th=[ 430], 50.00th=[ 430], 60.00th=[ 430], | 70.00th=[ 2039], 80.00th=[ 2039], 90.00th=[ 2039], 95.00th=[ 2039], | 99.00th=[ 2039], 99.50th=[ 2039], 99.90th=[ 2039], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 8192, max= 8192, per=76.20%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=9.93%, ctx=94, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8232: Wed May 29 18:56:24 2024 write: IOPS=1, BW=4611KiB/s (4722kB/s)(12.0MiB/2665msec) clat (msec): min=264, max=1745, avg=882.42, stdev=769.92 lat (msec): min=265, max=1745, avg=882.83, stdev=769.79 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 266], 10.00th=[ 266], 20.00th=[ 266], | 30.00th=[ 266], 40.00th=[ 634], 50.00th=[ 634], 60.00th=[ 634], | 70.00th=[ 1754], 80.00th=[ 1754], 90.00th=[ 1754], 95.00th=[ 1754], | 99.00th=[ 1754], 99.50th=[ 1754], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 8143, max= 8192, per=75.98%, avg=8167.50, stdev=34.65, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.08%, sys=10.92%, ctx=92, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5375KiB/s (5504kB/s), 1344KiB/s-6923KiB/s (1376kB/s-7089kB/s), io=16.0MiB (16.8MB), run=1775-3048msec WRITE: bw=10.5MiB/s (11.0MB/s), 2688KiB/s-4611KiB/s (2752kB/s-4722kB/s), io=32.0MiB (33.6MB), run=2665-3048msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8233: Wed May 29 18:56:24 2024 read: IOPS=0, BW=3890KiB/s (3983kB/s)(12.0MiB/3159msec) clat (msec): min=90, max=1956, avg=1051.90, stdev=934.18 lat (msec): min=90, max=1956, avg=1051.90, stdev=934.18 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 91], 10.00th=[ 91], 20.00th=[ 91], | 30.00th=[ 91], 40.00th=[ 1116], 50.00th=[ 1116], 60.00th=[ 1116], | 70.00th=[ 1955], 80.00th=[ 1955], 90.00th=[ 1955], 95.00th=[ 1955], | 99.00th=[ 1955], 99.50th=[ 1955], 99.90th=[ 1955], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33% cpu : usr=0.03%, sys=2.50%, ctx=33, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8234: Wed May 29 18:56:24 2024 read: IOPS=0, BW=1430KiB/s (1464kB/s)(4096KiB/2865msec) clat (nsec): min=77640k, max=77640k, avg=77639607.00, stdev= 0.00 lat (nsec): min=77642k, max=77642k, avg=77642314.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[78119], 5.00th=[78119], 10.00th=[78119], 20.00th=[78119], | 30.00th=[78119], 40.00th=[78119], 50.00th=[78119], 60.00th=[78119], | 70.00th=[78119], 80.00th=[78119], 90.00th=[78119], 95.00th=[78119], | 99.00th=[78119], 99.50th=[78119], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2859KiB/s (2928kB/s)(8192KiB/2865msec) clat (msec): min=544, max=2217, avg=1380.63, stdev=1183.03 lat (msec): min=544, max=2217, avg=1381.08, stdev=1183.07 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 542], 10.00th=[ 542], 20.00th=[ 542], | 30.00th=[ 542], 40.00th=[ 542], 50.00th=[ 542], 60.00th=[ 2232], | 70.00th=[ 2232], 80.00th=[ 2232], 90.00th=[ 2232], 95.00th=[ 2232], | 99.00th=[ 2232], 99.50th=[ 2232], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 8192, max= 8192, per=83.13%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.10%, sys=3.56%, ctx=37, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8235: Wed May 29 18:56:24 2024 write: IOPS=0, BW=3696KiB/s (3784kB/s)(12.0MiB/3325msec) clat (msec): min=116, max=2454, avg=1106.66, stdev=1209.11 lat (msec): min=116, max=2454, avg=1106.94, stdev=1209.05 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 117], 10.00th=[ 117], 20.00th=[ 117], | 30.00th=[ 117], 40.00th=[ 751], 50.00th=[ 751], 60.00th=[ 751], | 70.00th=[ 2467], 80.00th=[ 2467], 90.00th=[ 2467], 95.00th=[ 2467], | 99.00th=[ 2467], 99.50th=[ 2467], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 5542, max= 8192, per=69.68%, avg=6867.00, stdev=1873.83, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=2.53%, ctx=39, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8236: Wed May 29 18:56:24 2024 write: IOPS=1, BW=4309KiB/s (4412kB/s)(12.0MiB/2852msec) clat (msec): min=158, max=2166, avg=943.92, stdev=1072.98 lat (msec): min=158, max=2166, avg=944.26, stdev=1072.98 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 506], 50.00th=[ 506], 60.00th=[ 506], | 70.00th=[ 2165], 80.00th=[ 2165], 90.00th=[ 2165], 95.00th=[ 2165], | 99.00th=[ 2165], 99.50th=[ 2165], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 8159, max= 8159, per=82.79%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=5.44%, ctx=58, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5186KiB/s (5311kB/s), 1430KiB/s-3890KiB/s (1464kB/s-3983kB/s), io=16.0MiB (16.8MB), run=2865-3159msec WRITE: bw=9855KiB/s (10.1MB/s), 2859KiB/s-4309KiB/s (2928kB/s-4412kB/s), io=32.0MiB (33.6MB), run=2852-3325msec