== sanity test 398b: DIO and buffer IO race ============== 17:48:44 (1713217724) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.55425 s, 32.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15288: Mon Apr 15 17:48:59 2024 read: IOPS=128, BW=513KiB/s (526kB/s)(5992KiB/11675msec) clat (usec): min=45, max=166349, avg=3366.62, stdev=5217.86 lat (usec): min=45, max=166350, avg=3367.25, stdev=5218.03 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 61], 10.00th=[ 69], 20.00th=[ 77], | 30.00th=[ 2008], 40.00th=[ 2311], 50.00th=[ 2704], 60.00th=[ 3163], | 70.00th=[ 3851], 80.00th=[ 5145], 90.00th=[ 7046], 95.00th=[ 8979], | 99.00th=[ 12911], 99.50th=[ 14222], 99.90th=[ 36963], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 216, max= 752, per=25.22%, avg=512.00, stdev=103.33, samples=23 iops : min= 54, max= 188, avg=128.00, stdev=25.83, samples=23 write: IOPS=134, BW=539KiB/s (552kB/s)(6296KiB/11675msec) clat (usec): min=411, max=64713, avg=4133.24, stdev=4224.99 lat (usec): min=412, max=64714, avg=4133.89, stdev=4224.98 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 506], 10.00th=[ 529], 20.00th=[ 594], | 30.00th=[ 873], 40.00th=[ 2147], 50.00th=[ 3425], 60.00th=[ 4228], | 70.00th=[ 5342], 80.00th=[ 6718], 90.00th=[ 9110], 95.00th=[11469], | 99.00th=[17433], 99.50th=[21103], 99.90th=[42206], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 240, max= 680, per=25.21%, avg=536.00, stdev=94.41, samples=23 iops : min= 60, max= 170, avg=134.00, stdev=23.60, samples=23 lat (usec) : 50=0.94%, 100=9.90%, 250=0.98%, 500=2.21%, 750=11.98% lat (usec) : 1000=2.28% lat (msec) : 2=6.35%, 4=29.39%, 10=30.73%, 20=4.79%, 50=0.39% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.16%, sys=22.69%, ctx=5625, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15289: Mon Apr 15 17:48:59 2024 read: IOPS=125, BW=502KiB/s (514kB/s)(5940KiB/11825msec) clat (usec): min=46, max=40461, avg=3197.15, stdev=3202.44 lat (usec): min=46, max=40461, avg=3197.90, stdev=3202.42 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 60], 10.00th=[ 69], 20.00th=[ 73], | 30.00th=[ 1975], 40.00th=[ 2311], 50.00th=[ 2704], 60.00th=[ 3163], | 70.00th=[ 3818], 80.00th=[ 5014], 90.00th=[ 6849], 95.00th=[ 8225], | 99.00th=[12780], 99.50th=[16712], 99.90th=[32375], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 240, max= 792, per=24.62%, avg=499.83, stdev=124.45, samples=23 iops : min= 60, max= 198, avg=124.83, stdev=31.10, samples=23 write: IOPS=134, BW=537KiB/s (550kB/s)(6348KiB/11825msec) clat (usec): min=419, max=165192, avg=4380.82, stdev=6136.07 lat (usec): min=420, max=165193, avg=4381.94, stdev=6135.94 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 502], 10.00th=[ 523], 20.00th=[ 578], | 30.00th=[ 807], 40.00th=[ 2057], 50.00th=[ 3261], 60.00th=[ 4293], | 70.00th=[ 5342], 80.00th=[ 6783], 90.00th=[ 9765], 95.00th=[ 12518], | 99.00th=[ 20841], 99.50th=[ 31065], 99.90th=[ 52167], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 288, max= 784, per=24.87%, avg=528.70, stdev=116.74, samples=23 iops : min= 72, max= 196, avg=132.04, stdev=29.21, samples=23 lat (usec) : 50=1.11%, 100=10.16%, 250=0.91%, 500=2.47%, 750=12.66% lat (usec) : 1000=2.15% lat (msec) : 2=5.70%, 4=29.43%, 10=29.46%, 20=5.18%, 50=0.72% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.08%, sys=22.01%, ctx=5594, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15290: Mon Apr 15 17:48:59 2024 read: IOPS=127, BW=511KiB/s (524kB/s)(6028KiB/11786msec) clat (usec): min=45, max=28063, avg=3248.89, stdev=2891.73 lat (usec): min=46, max=28063, avg=3249.46, stdev=2891.73 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 60], 10.00th=[ 69], 20.00th=[ 73], | 30.00th=[ 1958], 40.00th=[ 2311], 50.00th=[ 2704], 60.00th=[ 3228], | 70.00th=[ 4113], 80.00th=[ 5407], 90.00th=[ 6980], 95.00th=[ 8586], | 99.00th=[11469], 99.50th=[13829], 99.90th=[24773], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 168, max= 704, per=24.94%, avg=506.30, stdev=121.71, samples=23 iops : min= 42, max= 176, avg=126.52, stdev=30.43, samples=23 write: IOPS=132, BW=531KiB/s (544kB/s)(6260KiB/11786msec) clat (usec): min=423, max=182471, avg=4314.15, stdev=6550.61 lat (usec): min=423, max=182473, avg=4314.93, stdev=6550.70 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 498], 10.00th=[ 519], 20.00th=[ 570], | 30.00th=[ 750], 40.00th=[ 1582], 50.00th=[ 3032], 60.00th=[ 4113], | 70.00th=[ 5211], 80.00th=[ 6980], 90.00th=[ 9765], 95.00th=[ 12387], | 99.00th=[ 18220], 99.50th=[ 31327], 99.90th=[ 60556], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 208, max= 696, per=24.61%, avg=523.30, stdev=108.11, samples=23 iops : min= 52, max= 174, avg=130.78, stdev=27.02, samples=23 lat (usec) : 50=1.17%, 100=10.25%, 250=0.85%, 500=2.90%, 750=12.47% lat (usec) : 1000=2.93% lat (msec) : 2=6.22%, 4=26.86%, 10=30.47%, 20=5.37%, 50=0.46% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.20%, sys=22.37%, ctx=5500, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15292: Mon Apr 15 17:48:59 2024 read: IOPS=128, BW=514KiB/s (526kB/s)(6052KiB/11780msec) clat (usec): min=46, max=171080, avg=3462.16, stdev=5606.17 lat (usec): min=47, max=171081, avg=3462.71, stdev=5606.17 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 61], 10.00th=[ 69], 20.00th=[ 77], | 30.00th=[ 2057], 40.00th=[ 2343], 50.00th=[ 2704], 60.00th=[ 3195], | 70.00th=[ 4015], 80.00th=[ 5276], 90.00th=[ 7046], 95.00th=[ 8455], | 99.00th=[ 14484], 99.50th=[ 20579], 99.90th=[ 46400], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 264, max= 760, per=25.04%, avg=508.35, stdev=100.06, samples=23 iops : min= 66, max= 190, avg=127.04, stdev=25.01, samples=23 write: IOPS=132, BW=529KiB/s (542kB/s)(6236KiB/11780msec) clat (usec): min=401, max=40358, avg=4114.78, stdev=4239.84 lat (usec): min=401, max=40358, avg=4115.60, stdev=4239.74 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 502], 10.00th=[ 523], 20.00th=[ 578], | 30.00th=[ 750], 40.00th=[ 1762], 50.00th=[ 3261], 60.00th=[ 4113], | 70.00th=[ 5145], 80.00th=[ 6521], 90.00th=[ 9634], 95.00th=[12518], | 99.00th=[18220], 99.50th=[23462], 99.90th=[35390], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 344, max= 696, per=24.65%, avg=524.09, stdev=83.29, samples=23 iops : min= 86, max= 174, avg=130.96, stdev=20.86, samples=23 lat (usec) : 50=0.94%, 100=10.16%, 250=0.65%, 500=2.47%, 750=12.83% lat (usec) : 1000=1.95% lat (msec) : 2=5.89%, 4=29.07%, 10=30.27%, 20=5.08%, 50=0.65% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=22.34%, ctx=5587, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2031KiB/s (2079kB/s), 502KiB/s-514KiB/s (514kB/s-526kB/s), io=23.4MiB (24.6MB), run=11675-11825msec WRITE: bw=2126KiB/s (2177kB/s), 529KiB/s-539KiB/s (542kB/s-552kB/s), io=24.6MiB (25.7MB), run=11675-11825msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15291: Mon Apr 15 17:49:29 2024 read: IOPS=35, BW=143KiB/s (147kB/s)(5992KiB/41848msec) clat (usec): min=1707, max=196423, avg=4028.58, stdev=5871.20 lat (usec): min=1707, max=196424, avg=4029.22, stdev=5871.20 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1844], 10.00th=[ 1893], 20.00th=[ 1991], | 30.00th=[ 2089], 40.00th=[ 2278], 50.00th=[ 2573], 60.00th=[ 3359], | 70.00th=[ 4359], 80.00th=[ 5735], 90.00th=[ 6980], 95.00th=[ 9241], | 99.00th=[ 14091], 99.50th=[ 18744], 99.90th=[ 50594], 99.95th=[196084], | 99.99th=[196084] bw ( KiB/s): min= 47, max= 272, per=24.90%, avg=142.70, stdev=55.94, samples=83 iops : min= 11, max= 68, avg=35.57, stdev=14.04, samples=83 write: IOPS=37, BW=150KiB/s (154kB/s)(6296KiB/41848msec) clat (usec): min=12085, max=93935, avg=22674.20, stdev=8271.46 lat (usec): min=12086, max=93936, avg=22674.94, stdev=8271.57 clat percentiles (usec): | 1.00th=[12911], 5.00th=[13960], 10.00th=[14615], 20.00th=[16057], | 30.00th=[17695], 40.00th=[19530], 50.00th=[21365], 60.00th=[22938], | 70.00th=[24249], 80.00th=[26346], 90.00th=[33162], 95.00th=[40109], | 99.00th=[51119], 99.50th=[57410], 99.90th=[69731], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 40, max= 200, per=24.97%, avg=149.83, stdev=39.29, samples=83 iops : min= 10, max= 50, avg=37.35, stdev= 9.80, samples=83 lat (msec) : 2=10.22%, 4=22.72%, 10=13.87%, 20=23.50%, 50=29.00% lat (msec) : 100=0.65%, 250=0.03% cpu : usr=0.05%, sys=4.51%, ctx=3855, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15293: Mon Apr 15 17:49:29 2024 read: IOPS=35, BW=142KiB/s (145kB/s)(5940KiB/41837msec) clat (usec): min=1614, max=173184, avg=4029.44, stdev=5465.43 lat (usec): min=1614, max=173185, avg=4030.06, stdev=5465.45 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1844], 10.00th=[ 1893], 20.00th=[ 1991], | 30.00th=[ 2114], 40.00th=[ 2278], 50.00th=[ 2573], 60.00th=[ 3392], | 70.00th=[ 4293], 80.00th=[ 5669], 90.00th=[ 6783], 95.00th=[ 8586], | 99.00th=[ 16319], 99.50th=[ 21890], 99.90th=[ 44303], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 16, max= 280, per=24.75%, avg=141.82, stdev=59.90, samples=83 iops : min= 4, max= 70, avg=35.34, stdev=14.97, samples=83 write: IOPS=37, BW=152KiB/s (155kB/s)(6348KiB/41837msec) clat (usec): min=11878, max=84975, avg=22515.38, stdev=8361.11 lat (usec): min=11878, max=84976, avg=22516.11, stdev=8361.11 clat percentiles (usec): | 1.00th=[12911], 5.00th=[13698], 10.00th=[14484], 20.00th=[15926], | 30.00th=[17171], 40.00th=[19006], 50.00th=[21103], 60.00th=[22676], | 70.00th=[24249], 80.00th=[26346], 90.00th=[33162], 95.00th=[40109], | 99.00th=[52691], 99.50th=[56886], 99.90th=[73925], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 40, max= 200, per=25.18%, avg=151.07, stdev=40.32, samples=83 iops : min= 10, max= 50, avg=37.65, stdev=10.08, samples=83 lat (msec) : 2=10.12%, 4=22.33%, 10=13.80%, 20=24.87%, 50=28.19% lat (msec) : 100=0.65%, 250=0.03% cpu : usr=0.08%, sys=4.47%, ctx=3869, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15294: Mon Apr 15 17:49:29 2024 read: IOPS=36, BW=146KiB/s (150kB/s)(6028KiB/41186msec) clat (usec): min=1577, max=42984, avg=4019.71, stdev=3101.61 lat (usec): min=1577, max=42985, avg=4020.31, stdev=3101.61 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1844], 10.00th=[ 1909], 20.00th=[ 2008], | 30.00th=[ 2147], 40.00th=[ 2311], 50.00th=[ 2704], 60.00th=[ 3556], | 70.00th=[ 4817], 80.00th=[ 5866], 90.00th=[ 7046], 95.00th=[ 9110], | 99.00th=[14615], 99.50th=[19792], 99.90th=[30802], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 47, max= 296, per=25.54%, avg=146.32, stdev=55.68, samples=82 iops : min= 11, max= 74, avg=36.49, stdev=13.96, samples=82 write: IOPS=37, BW=152KiB/s (156kB/s)(6260KiB/41186msec) clat (msec): min=11, max=192, avg=22.37, stdev= 9.33 lat (msec): min=11, max=192, avg=22.37, stdev= 9.33 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 21], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 26], 90.00th=[ 33], 95.00th=[ 40], | 99.00th=[ 54], 99.50th=[ 57], 99.90th=[ 84], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 47, max= 200, per=25.30%, avg=151.78, stdev=40.91, samples=82 iops : min= 11, max= 50, avg=37.85, stdev=10.28, samples=82 lat (msec) : 2=9.54%, 4=22.17%, 10=15.46%, 20=25.10%, 50=26.89% lat (msec) : 100=0.81%, 250=0.03% cpu : usr=0.07%, sys=4.62%, ctx=3904, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15295: Mon Apr 15 17:49:29 2024 read: IOPS=36, BW=146KiB/s (150kB/s)(6052KiB/41420msec) clat (usec): min=1599, max=50720, avg=4076.29, stdev=3560.69 lat (usec): min=1600, max=50721, avg=4076.90, stdev=3560.69 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1844], 10.00th=[ 1893], 20.00th=[ 1991], | 30.00th=[ 2114], 40.00th=[ 2278], 50.00th=[ 2573], 60.00th=[ 3392], | 70.00th=[ 4752], 80.00th=[ 5997], 90.00th=[ 7242], 95.00th=[10028], | 99.00th=[14877], 99.50th=[23200], 99.90th=[44303], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 32, max= 280, per=25.40%, avg=145.52, stdev=56.48, samples=82 iops : min= 8, max= 70, avg=36.35, stdev=14.13, samples=82 write: IOPS=37, BW=151KiB/s (154kB/s)(6236KiB/41420msec) clat (msec): min=11, max=205, avg=22.53, stdev= 9.55 lat (msec): min=11, max=205, avg=22.53, stdev= 9.55 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 27], 90.00th=[ 33], 95.00th=[ 42], | 99.00th=[ 54], 99.50th=[ 56], 99.90th=[ 94], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 56, max= 200, per=25.07%, avg=150.40, stdev=41.12, samples=82 iops : min= 14, max= 50, avg=37.57, stdev=10.30, samples=82 lat (msec) : 2=9.99%, 4=22.53%, 10=14.26%, 20=24.93%, 50=27.47% lat (msec) : 100=0.78%, 250=0.03% cpu : usr=0.09%, sys=4.56%, ctx=3936, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=574KiB/s (588kB/s), 142KiB/s-146KiB/s (145kB/s-150kB/s), io=23.4MiB (24.6MB), run=41186-41848msec WRITE: bw=601KiB/s (615kB/s), 150KiB/s-152KiB/s (154kB/s-156kB/s), io=24.6MiB (25.7MB), run=41186-41848msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15340: Mon Apr 15 17:49:34 2024 read: IOPS=87, BW=1403KiB/s (1437kB/s)(5920KiB/4220msec) clat (usec): min=51, max=26295, avg=8038.99, stdev=5461.21 lat (usec): min=51, max=26296, avg=8039.51, stdev=5461.21 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 63], 10.00th=[ 71], 20.00th=[ 76], | 30.00th=[ 7242], 40.00th=[ 8455], 50.00th=[ 9503], 60.00th=[10421], | 70.00th=[11207], 80.00th=[12125], 90.00th=[13435], 95.00th=[14353], | 99.00th=[21890], 99.50th=[23462], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 1120, max= 1504, per=23.82%, avg=1351.62, stdev=134.34, samples=8 iops : min= 70, max= 94, avg=84.38, stdev= 8.38, samples=8 write: IOPS=94, BW=1509KiB/s (1545kB/s)(6368KiB/4220msec) clat (usec): min=643, max=33708, avg=3115.28, stdev=4180.69 lat (usec): min=643, max=33709, avg=3116.08, stdev=4180.69 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 717], 10.00th=[ 742], 20.00th=[ 783], | 30.00th=[ 824], 40.00th=[ 873], 50.00th=[ 1401], 60.00th=[ 2540], | 70.00th=[ 3326], 80.00th=[ 4555], 90.00th=[ 6587], 95.00th=[ 9765], | 99.00th=[23200], 99.50th=[31851], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 1024, max= 1952, per=24.58%, avg=1467.50, stdev=301.31, samples=8 iops : min= 64, max= 122, avg=91.62, stdev=18.82, samples=8 lat (usec) : 100=12.24%, 250=0.78%, 750=6.25%, 1000=17.58% lat (msec) : 2=3.91%, 4=12.89%, 10=22.27%, 20=22.79%, 50=1.30% cpu : usr=0.12%, sys=21.33%, ctx=1849, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15341: Mon Apr 15 17:49:34 2024 read: IOPS=87, BW=1397KiB/s (1430kB/s)(5744KiB/4112msec) clat (usec): min=50, max=31370, avg=8191.15, stdev=5499.27 lat (usec): min=50, max=31370, avg=8193.52, stdev=5501.14 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 64], 10.00th=[ 73], 20.00th=[ 86], | 30.00th=[ 7832], 40.00th=[ 8848], 50.00th=[ 9372], 60.00th=[10290], | 70.00th=[11207], 80.00th=[11994], 90.00th=[13566], 95.00th=[14746], | 99.00th=[23200], 99.50th=[31327], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 1056, max= 1856, per=24.66%, avg=1399.62, stdev=259.82, samples=8 iops : min= 66, max= 116, avg=87.38, stdev=16.24, samples=8 write: IOPS=99, BW=1591KiB/s (1630kB/s)(6544KiB/4112msec) clat (usec): min=643, max=39266, avg=2846.15, stdev=3687.91 lat (usec): min=644, max=39267, avg=2847.00, stdev=3687.92 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 709], 10.00th=[ 734], 20.00th=[ 783], | 30.00th=[ 807], 40.00th=[ 889], 50.00th=[ 1205], 60.00th=[ 2540], | 70.00th=[ 3195], 80.00th=[ 4359], 90.00th=[ 6521], 95.00th=[ 7832], | 99.00th=[19792], 99.50th=[25035], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 1216, max= 2240, per=26.25%, avg=1567.62, stdev=317.84, samples=8 iops : min= 76, max= 140, avg=97.88, stdev=19.90, samples=8 lat (usec) : 100=11.59%, 250=0.13%, 750=7.55%, 1000=17.84% lat (msec) : 2=4.69%, 4=11.59%, 10=25.26%, 20=20.31%, 50=1.04% cpu : usr=0.02%, sys=21.80%, ctx=1753, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15342: Mon Apr 15 17:49:34 2024 read: IOPS=92, BW=1482KiB/s (1518kB/s)(6256KiB/4220msec) clat (usec): min=47, max=36638, avg=7959.36, stdev=5823.15 lat (usec): min=48, max=36639, avg=7959.88, stdev=5823.16 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 62], 10.00th=[ 71], 20.00th=[ 75], | 30.00th=[ 6783], 40.00th=[ 8586], 50.00th=[ 9372], 60.00th=[10028], | 70.00th=[10945], 80.00th=[11994], 90.00th=[13566], 95.00th=[15401], | 99.00th=[23462], 99.50th=[35390], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 1056, max= 2139, per=26.35%, avg=1495.38, stdev=403.18, samples=8 iops : min= 66, max= 133, avg=93.38, stdev=25.04, samples=8 write: IOPS=89, BW=1429KiB/s (1464kB/s)(6032KiB/4220msec) clat (usec): min=640, max=35129, avg=2921.52, stdev=3559.94 lat (usec): min=641, max=35129, avg=2922.34, stdev=3559.95 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 701], 10.00th=[ 734], 20.00th=[ 791], | 30.00th=[ 832], 40.00th=[ 889], 50.00th=[ 1467], 60.00th=[ 2704], | 70.00th=[ 3458], 80.00th=[ 4752], 90.00th=[ 6194], 95.00th=[ 7701], | 99.00th=[17695], 99.50th=[31851], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 704, max= 2208, per=23.44%, avg=1399.62, stdev=427.18, samples=8 iops : min= 44, max= 138, avg=87.38, stdev=26.70, samples=8 lat (usec) : 50=0.13%, 100=12.89%, 250=1.04%, 500=0.39%, 750=6.77% lat (usec) : 1000=15.89% lat (msec) : 2=2.60%, 4=12.24%, 10=25.52%, 20=21.35%, 50=1.17% cpu : usr=0.07%, sys=20.98%, ctx=1851, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15343: Mon Apr 15 17:49:34 2024 read: IOPS=90, BW=1452KiB/s (1487kB/s)(6032KiB/4155msec) clat (usec): min=48, max=23726, avg=8001.87, stdev=5242.26 lat (usec): min=49, max=23727, avg=8002.39, stdev=5242.26 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 63], 10.00th=[ 72], 20.00th=[ 82], | 30.00th=[ 7570], 40.00th=[ 8586], 50.00th=[ 9241], 60.00th=[10159], | 70.00th=[10814], 80.00th=[11994], 90.00th=[13173], 95.00th=[14877], | 99.00th=[20579], 99.50th=[21890], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 1184, max= 1692, per=25.30%, avg=1435.50, stdev=176.46, samples=8 iops : min= 74, max= 105, avg=89.62, stdev=10.88, samples=8 write: IOPS=94, BW=1506KiB/s (1542kB/s)(6256KiB/4155msec) clat (usec): min=668, max=35426, avg=2895.35, stdev=3366.07 lat (usec): min=669, max=35427, avg=2896.20, stdev=3366.06 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 709], 10.00th=[ 742], 20.00th=[ 783], | 30.00th=[ 816], 40.00th=[ 873], 50.00th=[ 1106], 60.00th=[ 2573], | 70.00th=[ 3326], 80.00th=[ 4752], 90.00th=[ 6849], 95.00th=[ 8586], | 99.00th=[16188], 99.50th=[17171], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 1056, max= 1888, per=25.11%, avg=1499.50, stdev=285.67, samples=8 iops : min= 66, max= 118, avg=93.63, stdev=17.81, samples=8 lat (usec) : 50=0.26%, 100=11.20%, 250=1.17%, 750=5.86%, 1000=17.84% lat (msec) : 2=4.43%, 4=10.81%, 10=26.43%, 20=21.35%, 50=0.65% cpu : usr=0.14%, sys=21.64%, ctx=1891, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5676KiB/s (5812kB/s), 1397KiB/s-1482KiB/s (1430kB/s-1518kB/s), io=23.4MiB (24.5MB), run=4112-4220msec WRITE: bw=5972KiB/s (6115kB/s), 1429KiB/s-1591KiB/s (1464kB/s-1630kB/s), io=24.6MiB (25.8MB), run=4112-4220msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15344: Mon Apr 15 17:49:40 2024 read: IOPS=35, BW=571KiB/s (585kB/s)(5920KiB/10368msec) clat (usec): min=1906, max=25793, avg=3795.49, stdev=3193.92 lat (usec): min=1907, max=25794, avg=3796.07, stdev=3193.92 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2057], 10.00th=[ 2114], 20.00th=[ 2212], | 30.00th=[ 2311], 40.00th=[ 2409], 50.00th=[ 2540], 60.00th=[ 2802], | 70.00th=[ 3195], 80.00th=[ 4490], 90.00th=[ 6718], 95.00th=[11207], | 99.00th=[18220], 99.50th=[23725], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 352, max= 864, per=24.38%, avg=558.20, stdev=152.95, samples=20 iops : min= 22, max= 54, avg=34.80, stdev= 9.50, samples=20 write: IOPS=38, BW=614KiB/s (629kB/s)(6368KiB/10368msec) clat (usec): min=12519, max=48955, avg=22502.07, stdev=6517.31 lat (usec): min=12520, max=48956, avg=22503.05, stdev=6517.29 clat percentiles (usec): | 1.00th=[13042], 5.00th=[13829], 10.00th=[14615], 20.00th=[16319], | 30.00th=[18482], 40.00th=[20579], 50.00th=[22152], 60.00th=[23462], | 70.00th=[25035], 80.00th=[27132], 90.00th=[30016], 95.00th=[34341], | 99.00th=[47449], 99.50th=[48497], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 416, max= 768, per=25.23%, avg=607.85, stdev=135.41, samples=20 iops : min= 26, max= 48, avg=37.90, stdev= 8.52, samples=20 lat (msec) : 2=0.65%, 4=36.59%, 10=8.07%, 20=21.09%, 50=33.59% cpu : usr=0.06%, sys=4.82%, ctx=852, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15345: Mon Apr 15 17:49:40 2024 read: IOPS=34, BW=551KiB/s (564kB/s)(5744KiB/10421msec) clat (usec): min=1935, max=25219, avg=3872.91, stdev=3416.68 lat (usec): min=1936, max=25219, avg=3873.49, stdev=3416.68 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2057], 10.00th=[ 2114], 20.00th=[ 2212], | 30.00th=[ 2278], 40.00th=[ 2376], 50.00th=[ 2507], 60.00th=[ 2769], | 70.00th=[ 3195], 80.00th=[ 4490], 90.00th=[ 6980], 95.00th=[11600], | 99.00th=[20317], 99.50th=[24249], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 192, max= 926, per=24.10%, avg=551.90, stdev=211.03, samples=20 iops : min= 12, max= 57, avg=34.45, stdev=13.11, samples=20 write: IOPS=39, BW=628KiB/s (643kB/s)(6544KiB/10421msec) clat (usec): min=12695, max=63898, avg=22061.83, stdev=6616.30 lat (usec): min=12697, max=63899, avg=22062.89, stdev=6616.46 clat percentiles (usec): | 1.00th=[12911], 5.00th=[13829], 10.00th=[14484], 20.00th=[15926], | 30.00th=[18220], 40.00th=[20055], 50.00th=[21627], 60.00th=[22938], | 70.00th=[23987], 80.00th=[25822], 90.00th=[29492], 95.00th=[34866], | 99.00th=[42206], 99.50th=[50070], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 448, max= 768, per=25.77%, avg=620.70, stdev=119.84, samples=20 iops : min= 28, max= 48, avg=38.75, stdev= 7.46, samples=20 lat (msec) : 2=1.17%, 4=35.03%, 10=7.29%, 20=23.83%, 50=32.29% lat (msec) : 100=0.39% cpu : usr=0.08%, sys=4.84%, ctx=867, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15346: Mon Apr 15 17:49:40 2024 read: IOPS=38, BW=616KiB/s (631kB/s)(6256KiB/10156msec) clat (usec): min=1871, max=31241, avg=3774.17, stdev=3176.15 lat (usec): min=1872, max=31242, avg=3774.78, stdev=3176.15 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2089], 10.00th=[ 2147], 20.00th=[ 2212], | 30.00th=[ 2311], 40.00th=[ 2409], 50.00th=[ 2540], 60.00th=[ 2737], | 70.00th=[ 3130], 80.00th=[ 4424], 90.00th=[ 6718], 95.00th=[11207], | 99.00th=[16057], 99.50th=[19268], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 256, max= 1152, per=27.17%, avg=622.30, stdev=233.83, samples=20 iops : min= 16, max= 72, avg=38.85, stdev=14.59, samples=20 write: IOPS=37, BW=594KiB/s (608kB/s)(6032KiB/10156msec) clat (usec): min=12916, max=48907, avg=23004.25, stdev=6796.34 lat (usec): min=12917, max=48908, avg=23005.38, stdev=6796.53 clat percentiles (usec): | 1.00th=[13173], 5.00th=[14222], 10.00th=[15008], 20.00th=[16188], | 30.00th=[18482], 40.00th=[20841], 50.00th=[22414], 60.00th=[23987], | 70.00th=[25297], 80.00th=[27919], 90.00th=[32375], 95.00th=[35914], | 99.00th=[42730], 99.50th=[45351], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 352, max= 768, per=24.44%, avg=588.70, stdev=129.75, samples=20 iops : min= 22, max= 48, avg=36.75, stdev= 8.07, samples=20 lat (msec) : 2=0.78%, 4=38.67%, 10=8.07%, 20=21.22%, 50=31.25% cpu : usr=0.06%, sys=4.93%, ctx=838, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15347: Mon Apr 15 17:49:40 2024 read: IOPS=36, BW=577KiB/s (591kB/s)(6032KiB/10457msec) clat (usec): min=1899, max=31220, avg=3959.35, stdev=3566.93 lat (usec): min=1900, max=31221, avg=3959.96, stdev=3566.91 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2073], 10.00th=[ 2147], 20.00th=[ 2245], | 30.00th=[ 2343], 40.00th=[ 2442], 50.00th=[ 2606], 60.00th=[ 2835], | 70.00th=[ 3163], 80.00th=[ 4621], 90.00th=[ 8094], 95.00th=[10814], | 99.00th=[17695], 99.50th=[30016], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 288, max= 832, per=24.87%, avg=569.50, stdev=178.65, samples=20 iops : min= 18, max= 52, avg=35.55, stdev=11.10, samples=20 write: IOPS=37, BW=598KiB/s (613kB/s)(6256KiB/10457msec) clat (usec): min=12630, max=51333, avg=22911.66, stdev=7032.37 lat (usec): min=12631, max=51334, avg=22912.62, stdev=7032.40 clat percentiles (usec): | 1.00th=[12911], 5.00th=[13698], 10.00th=[14615], 20.00th=[16319], | 30.00th=[18220], 40.00th=[20579], 50.00th=[22414], 60.00th=[23725], | 70.00th=[25297], 80.00th=[27657], 90.00th=[32375], 95.00th=[35914], | 99.00th=[44827], 99.50th=[47973], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 384, max= 768, per=24.64%, avg=593.50, stdev=136.29, samples=20 iops : min= 24, max= 48, avg=37.05, stdev= 8.49, samples=20 lat (msec) : 2=0.26%, 4=37.63%, 10=8.07%, 20=21.35%, 50=32.55% lat (msec) : 100=0.13% cpu : usr=0.07%, sys=4.77%, ctx=864, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2291KiB/s (2345kB/s), 551KiB/s-616KiB/s (564kB/s-631kB/s), io=23.4MiB (24.5MB), run=10156-10457msec WRITE: bw=2410KiB/s (2468kB/s), 594KiB/s-628KiB/s (608kB/s-643kB/s), io=24.6MiB (25.8MB), run=10156-10457msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15366: Mon Apr 15 17:49:45 2024 read: IOPS=1, BW=1781KiB/s (1824kB/s)(7168KiB/4024msec) clat (msec): min=20, max=969, avg=171.02, stdev=352.26 lat (msec): min=20, max=969, avg=171.02, stdev=352.26 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 29], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 44], 60.00th=[ 47], | 70.00th=[ 47], 80.00th=[ 49], 90.00th=[ 969], 95.00th=[ 969], | 99.00th=[ 969], 99.50th=[ 969], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 2048, max= 6144, per=84.45%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 write: IOPS=1, BW=1272KiB/s (1303kB/s)(5120KiB/4024msec) clat (msec): min=44, max=1687, avg=565.13, stdev=746.87 lat (msec): min=44, max=1687, avg=565.19, stdev=746.90 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 45], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 58], 60.00th=[ 58], | 70.00th=[ 986], 80.00th=[ 986], 90.00th=[ 1687], 95.00th=[ 1687], | 99.00th=[ 1687], 99.50th=[ 1687], 99.90th=[ 1687], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 2048, max= 4096, per=50.26%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 lat (msec) : 50=58.33%, 100=16.67%, 1000=16.67% cpu : usr=0.00%, sys=2.93%, ctx=64, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15368: Mon Apr 15 17:49:45 2024 read: IOPS=0, BW=485KiB/s (497kB/s)(2048KiB/4222msec) clat (msec): min=49, max=855, avg=452.23, stdev=569.84 lat (msec): min=49, max=855, avg=452.23, stdev=569.84 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 852], | 70.00th=[ 852], 80.00th=[ 852], 90.00th=[ 852], 95.00th=[ 852], | 99.00th=[ 852], 99.50th=[ 852], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 2048, max= 2048, per=42.23%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2425KiB/s (2484kB/s)(10.0MiB/4222msec) clat (msec): min=43, max=1127, avg=331.59, stdev=446.44 lat (msec): min=43, max=1127, avg=331.65, stdev=446.44 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 47], | 30.00th=[ 51], 40.00th=[ 61], 50.00th=[ 75], 60.00th=[ 91], | 70.00th=[ 592], 80.00th=[ 592], 90.00th=[ 1116], 95.00th=[ 1133], | 99.00th=[ 1133], 99.50th=[ 1133], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 2048, max= 6131, per=45.19%, avg=3068.75, stdev=2041.50, samples=4 iops : min= 2, max= 5, avg= 2.75, stdev= 1.50, samples=4 lat (msec) : 50=25.00%, 100=33.33%, 250=8.33%, 750=8.33%, 1000=8.33% cpu : usr=0.05%, sys=3.51%, ctx=113, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15370: Mon Apr 15 17:49:45 2024 read: IOPS=1, BW=1493KiB/s (1529kB/s)(6144KiB/4116msec) clat (msec): min=17, max=957, avg=199.24, stdev=372.42 lat (msec): min=17, max=957, avg=199.24, stdev=372.42 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 41], 50.00th=[ 41], 60.00th=[ 58], | 70.00th=[ 93], 80.00th=[ 93], 90.00th=[ 961], 95.00th=[ 961], | 99.00th=[ 961], 99.50th=[ 961], 99.90th=[ 961], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 4096, max= 4096, per=84.45%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=1, BW=1493KiB/s (1529kB/s)(6144KiB/4116msec) clat (msec): min=34, max=1548, avg=486.14, stdev=650.47 lat (msec): min=34, max=1548, avg=486.18, stdev=650.47 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 51], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 53], 60.00th=[ 186], | 70.00th=[ 1045], 80.00th=[ 1045], 90.00th=[ 1552], 95.00th=[ 1552], | 99.00th=[ 1552], 99.50th=[ 1552], 99.90th=[ 1552], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 2048, max= 4096, per=50.26%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 lat (msec) : 20=8.33%, 50=33.33%, 100=25.00%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=3.06%, ctx=40, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15371: Mon Apr 15 17:49:45 2024 read: IOPS=1, BW=1222KiB/s (1251kB/s)(5120KiB/4191msec) clat (msec): min=17, max=639, avg=150.86, stdev=273.29 lat (msec): min=17, max=639, avg=150.86, stdev=273.29 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 26], 40.00th=[ 26], 50.00th=[ 35], 60.00th=[ 35], | 70.00th=[ 37], 80.00th=[ 37], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 4096, max= 4096, per=84.45%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=1, BW=1710KiB/s (1751kB/s)(7168KiB/4191msec) clat (msec): min=49, max=1039, avg=490.17, stdev=497.86 lat (msec): min=49, max=1039, avg=490.23, stdev=497.86 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 51], 10.00th=[ 51], 20.00th=[ 53], | 30.00th=[ 65], 40.00th=[ 65], 50.00th=[ 207], 60.00th=[ 995], | 70.00th=[ 995], 80.00th=[ 1028], 90.00th=[ 1036], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 2048, max= 4096, per=40.21%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 lat (msec) : 20=8.33%, 50=33.33%, 100=16.67%, 250=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.02%, sys=3.25%, ctx=50, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4851KiB/s (4967kB/s), 485KiB/s-1781KiB/s (497kB/s-1824kB/s), io=20.0MiB (20.0MB), run=4024-4222msec WRITE: bw=6791KiB/s (6954kB/s), 1272KiB/s-2425KiB/s (1303kB/s-2484kB/s), io=28.0MiB (29.4MB), run=4024-4222msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15364: Mon Apr 15 17:49:47 2024 read: IOPS=1, BW=1199KiB/s (1228kB/s)(7168KiB/5976msec) clat (msec): min=35, max=963, avg=749.48, stdev=323.70 lat (msec): min=35, max=963, avg=749.48, stdev=323.70 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 760], | 30.00th=[ 810], 40.00th=[ 810], 50.00th=[ 827], 60.00th=[ 902], | 70.00th=[ 902], 80.00th=[ 953], 90.00th=[ 961], 95.00th=[ 961], | 99.00th=[ 961], 99.50th=[ 961], 99.90th=[ 961], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 1741, max= 2048, per=56.77%, avg=1945.67, stdev=158.53, samples=6 iops : min= 1, max= 2, avg= 1.67, stdev= 0.52, samples=6 write: IOPS=0, BW=857KiB/s (877kB/s)(5120KiB/5976msec) clat (msec): min=23, max=595, avg=142.95, stdev=253.09 lat (msec): min=23, max=595, avg=143.01, stdev=253.11 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 24], | 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 36], 60.00th=[ 36], | 70.00th=[ 38], 80.00th=[ 38], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 1741, max= 5224, per=62.63%, avg=3004.33, stdev=1928.41, samples=3 iops : min= 1, max= 5, avg= 2.67, stdev= 2.08, samples=3 lat (msec) : 50=41.67%, 750=8.33%, 1000=50.00% cpu : usr=0.00%, sys=15.31%, ctx=1727, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15365: Mon Apr 15 17:49:47 2024 read: IOPS=0, BW=496KiB/s (508kB/s)(2048KiB/4126msec) clat (msec): min=31, max=877, avg=454.54, stdev=598.20 lat (msec): min=31, max=877, avg=454.54, stdev=598.20 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 877], | 70.00th=[ 877], 80.00th=[ 877], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 2039, max= 2048, per=59.63%, avg=2043.50, stdev= 6.36, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2482KiB/s (2541kB/s)(10.0MiB/4126msec) clat (msec): min=41, max=1061, avg=320.31, stdev=429.94 lat (msec): min=42, max=1061, avg=320.36, stdev=429.94 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 50], 50.00th=[ 77], 60.00th=[ 82], | 70.00th=[ 667], 80.00th=[ 667], 90.00th=[ 1045], 95.00th=[ 1062], | 99.00th=[ 1062], 99.50th=[ 1062], 99.90th=[ 1062], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 2043, max= 6144, per=74.60%, avg=3578.50, stdev=1960.65, samples=4 iops : min= 1, max= 6, avg= 3.00, stdev= 2.16, samples=4 lat (msec) : 50=41.67%, 100=25.00%, 750=8.33%, 1000=8.33% cpu : usr=0.05%, sys=9.50%, ctx=362, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15367: Mon Apr 15 17:49:47 2024 read: IOPS=1, BW=2027KiB/s (2076kB/s)(6144KiB/3031msec) clat (usec): min=353, max=908372, avg=299602.70, stdev=435487.26 lat (usec): min=354, max=908373, avg=299603.67, stdev=435487.27 clat percentiles (usec): | 1.00th=[ 355], 5.00th=[ 355], 10.00th=[ 355], 20.00th=[ 363], | 30.00th=[ 363], 40.00th=[ 742], 50.00th=[ 742], 60.00th=[ 78119], | 70.00th=[809501], 80.00th=[809501], 90.00th=[910164], 95.00th=[910164], | 99.00th=[910164], 99.50th=[910164], 99.90th=[910164], 99.95th=[910164], | 99.99th=[910164] bw ( KiB/s): min= 4096, max= 4096, per=100.00%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=1, BW=2027KiB/s (2076kB/s)(6144KiB/3031msec) clat (msec): min=21, max=1031, avg=203.00, stdev=406.38 lat (msec): min=21, max=1031, avg=203.04, stdev=406.37 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 44], | 70.00th=[ 69], 80.00th=[ 69], 90.00th=[ 1028], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 2048, max= 4096, per=71.16%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 50=33.33%, 100=16.67%, 1000=16.67% cpu : usr=0.00%, sys=12.48%, ctx=597, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15369: Mon Apr 15 17:49:47 2024 read: IOPS=1, BW=1248KiB/s (1278kB/s)(5120KiB/4102msec) clat (usec): min=336, max=908658, avg=344850.74, stdev=453280.28 lat (usec): min=337, max=908659, avg=344851.85, stdev=453280.15 clat percentiles (usec): | 1.00th=[ 338], 5.00th=[ 338], 10.00th=[ 338], 20.00th=[ 338], | 30.00th=[ 562], 40.00th=[ 562], 50.00th=[ 47449], 60.00th=[ 47449], | 70.00th=[767558], 80.00th=[767558], 90.00th=[910164], 95.00th=[910164], | 99.00th=[910164], 99.50th=[910164], 99.90th=[910164], 99.95th=[910164], | 99.99th=[910164] bw ( KiB/s): min= 2048, max= 4096, per=79.68%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 write: IOPS=1, BW=1747KiB/s (1789kB/s)(7168KiB/4102msec) clat (msec): min=32, max=1038, avg=337.77, stdev=474.22 lat (msec): min=33, max=1038, avg=339.55, stdev=473.06 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 60], | 30.00th=[ 66], 40.00th=[ 66], 50.00th=[ 67], 60.00th=[ 77], | 70.00th=[ 77], 80.00th=[ 1028], 90.00th=[ 1036], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 2039, max= 6144, per=71.09%, avg=3410.33, stdev=2367.43, samples=3 iops : min= 1, max= 6, avg= 3.00, stdev= 2.65, samples=3 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=16.67%, 100=33.33%, 1000=16.67% cpu : usr=0.00%, sys=9.78%, ctx=613, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3427KiB/s (3509kB/s), 496KiB/s-2027KiB/s (508kB/s-2076kB/s), io=20.0MiB (20.0MB), run=3031-5976msec WRITE: bw=4798KiB/s (4913kB/s), 857KiB/s-2482KiB/s (877kB/s-2541kB/s), io=28.0MiB (29.4MB), run=3031-5976msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15381: Mon Apr 15 17:49:51 2024 read: IOPS=1, BW=6282KiB/s (6433kB/s)(12.0MiB/1956msec) clat (usec): min=1470, max=1759.9k, avg=646173.82, stdev=968506.58 lat (usec): min=1471, max=1759.9k, avg=646174.92, stdev=968506.31 clat percentiles (usec): | 1.00th=[ 1467], 5.00th=[ 1467], 10.00th=[ 1467], | 20.00th=[ 1467], 30.00th=[ 1467], 40.00th=[ 177210], | 50.00th=[ 177210], 60.00th=[ 177210], 70.00th=[1753220], | 80.00th=[1753220], 90.00th=[1753220], 95.00th=[1753220], | 99.00th=[1753220], 99.50th=[1753220], 99.90th=[1753220], | 99.95th=[1753220], 99.99th=[1753220] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 2=33.33%, 250=33.33% cpu : usr=0.05%, sys=43.73%, ctx=550, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15383: Mon Apr 15 17:49:51 2024 read: IOPS=0, BW=1564KiB/s (1601kB/s)(4096KiB/2619msec) clat (nsec): min=178946k, max=178946k, avg=178946271.00, stdev= 0.00 lat (nsec): min=178947k, max=178947k, avg=178947358.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 180], 10.00th=[ 180], 20.00th=[ 180], | 30.00th=[ 180], 40.00th=[ 180], 50.00th=[ 180], 60.00th=[ 180], | 70.00th=[ 180], 80.00th=[ 180], 90.00th=[ 180], 95.00th=[ 180], | 99.00th=[ 180], 99.50th=[ 180], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3128KiB/s (3203kB/s)(8192KiB/2619msec) clat (msec): min=1179, max=1253, avg=1216.15, stdev=52.22 lat (msec): min=1179, max=1253, avg=1216.38, stdev=52.23 clat percentiles (msec): | 1.00th=[ 1183], 5.00th=[ 1183], 10.00th=[ 1183], 20.00th=[ 1183], | 30.00th=[ 1183], 40.00th=[ 1183], 50.00th=[ 1183], 60.00th=[ 1250], | 70.00th=[ 1250], 80.00th=[ 1250], 90.00th=[ 1250], 95.00th=[ 1250], | 99.00th=[ 1250], 99.50th=[ 1250], 99.90th=[ 1250], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 8192, max= 8192, per=65.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=8.44%, ctx=104, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15385: Mon Apr 15 17:49:51 2024 write: IOPS=1, BW=4669KiB/s (4781kB/s)(12.0MiB/2632msec) clat (msec): min=85, max=1278, avg=874.48, stdev=683.44 lat (msec): min=85, max=1278, avg=874.73, stdev=683.45 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 86], 10.00th=[ 86], 20.00th=[ 86], | 30.00th=[ 86], 40.00th=[ 1267], 50.00th=[ 1267], 60.00th=[ 1267], | 70.00th=[ 1284], 80.00th=[ 1284], 90.00th=[ 1284], 95.00th=[ 1284], | 99.00th=[ 1284], 99.50th=[ 1284], 99.90th=[ 1284], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 8192, max= 8192, per=65.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33% cpu : usr=0.04%, sys=11.74%, ctx=55, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15388: Mon Apr 15 17:49:51 2024 write: IOPS=1, BW=5555KiB/s (5688kB/s)(12.0MiB/2212msec) clat (msec): min=96, max=2014, avg=737.03, stdev=1106.05 lat (msec): min=96, max=2014, avg=737.21, stdev=1106.11 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 97], 10.00th=[ 97], 20.00th=[ 97], | 30.00th=[ 97], 40.00th=[ 101], 50.00th=[ 101], 60.00th=[ 101], | 70.00th=[ 2022], 80.00th=[ 2022], 90.00th=[ 2022], 95.00th=[ 2022], | 99.00th=[ 2022], 99.50th=[ 2022], 99.90th=[ 2022], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 8192, max= 8192, per=65.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=11.57%, ctx=81, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6256KiB/s (6406kB/s), 1564KiB/s-6282KiB/s (1601kB/s-6433kB/s), io=16.0MiB (16.8MB), run=1956-2619msec WRITE: bw=12.2MiB/s (12.7MB/s), 3128KiB/s-5555KiB/s (3203kB/s-5688kB/s), io=32.0MiB (33.6MB), run=2212-2632msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15382: Mon Apr 15 17:49:51 2024 read: IOPS=2, BW=9630KiB/s (9861kB/s)(12.0MiB/1276msec) clat (msec): min=49, max=1115, avg=418.94, stdev=604.00 lat (msec): min=49, max=1115, avg=418.95, stdev=604.00 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 92], 50.00th=[ 92], 60.00th=[ 92], | 70.00th=[ 1116], 80.00th=[ 1116], 90.00th=[ 1116], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=33.33% cpu : usr=0.00%, sys=7.29%, ctx=47, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15384: Mon Apr 15 17:49:51 2024 read: IOPS=0, BW=1499KiB/s (1535kB/s)(4096KiB/2733msec) clat (nsec): min=2091.0M, max=2091.0M, avg=2091970685.00, stdev= 0.00 lat (nsec): min=2091.0M, max=2091.0M, avg=2091971570.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 2089], 5.00th=[ 2089], 10.00th=[ 2089], 20.00th=[ 2089], | 30.00th=[ 2089], 40.00th=[ 2089], 50.00th=[ 2089], 60.00th=[ 2089], | 70.00th=[ 2089], 80.00th=[ 2089], 90.00th=[ 2089], 95.00th=[ 2089], | 99.00th=[ 2089], 99.50th=[ 2089], 99.90th=[ 2089], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 8143, max= 8143, per=100.00%, avg=8143.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2997KiB/s (3069kB/s)(8192KiB/2733msec) clat (msec): min=86, max=549, avg=318.47, stdev=327.42 lat (msec): min=87, max=550, avg=318.71, stdev=327.45 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 87], 10.00th=[ 87], 20.00th=[ 87], | 30.00th=[ 87], 40.00th=[ 87], 50.00th=[ 87], 60.00th=[ 550], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 8143, max= 8143, per=67.92%, avg=8143.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.00%, sys=3.73%, ctx=106, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15387: Mon Apr 15 17:49:51 2024 write: IOPS=1, BW=4971KiB/s (5090kB/s)(12.0MiB/2472msec) clat (msec): min=95, max=1190, avg=807.90, stdev=617.27 lat (msec): min=96, max=1191, avg=808.23, stdev=617.25 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 96], 10.00th=[ 96], 20.00th=[ 96], | 30.00th=[ 96], 40.00th=[ 1133], 50.00th=[ 1133], 60.00th=[ 1133], | 70.00th=[ 1183], 80.00th=[ 1183], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 8175, max= 8175, per=68.19%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33% cpu : usr=0.04%, sys=6.64%, ctx=68, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15389: Mon Apr 15 17:49:51 2024 write: IOPS=1, BW=4800KiB/s (4915kB/s)(12.0MiB/2560msec) clat (msec): min=145, max=1263, avg=852.55, stdev=615.19 lat (msec): min=145, max=1263, avg=852.81, stdev=615.13 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 146], 10.00th=[ 146], 20.00th=[ 146], | 30.00th=[ 146], 40.00th=[ 1150], 50.00th=[ 1150], 60.00th=[ 1150], | 70.00th=[ 1267], 80.00th=[ 1267], 90.00th=[ 1267], 95.00th=[ 1267], | 99.00th=[ 1267], 99.50th=[ 1267], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 8192, max= 8192, per=68.33%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=5.47%, ctx=37, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5995KiB/s (6139kB/s), 1499KiB/s-9630KiB/s (1535kB/s-9861kB/s), io=16.0MiB (16.8MB), run=1276-2733msec WRITE: bw=11.7MiB/s (12.3MB/s), 2997KiB/s-4971KiB/s (3069kB/s-5090kB/s), io=32.0MiB (33.6MB), run=2472-2733msec