== sanity test 398b: DIO and buffer IO race ============== 01:34:27 (1721885667) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.45017 s, 34.7 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23571: Thu Jul 25 01:34:43 2024 read: IOPS=118, BW=472KiB/s (484kB/s)(5992KiB/12689msec) clat (usec): min=49, max=64173, avg=3832.79, stdev=3518.53 lat (usec): min=49, max=64173, avg=3833.40, stdev=3518.53 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 62], 10.00th=[ 71], 20.00th=[ 208], | 30.00th=[ 2278], 40.00th=[ 2704], 50.00th=[ 3163], 60.00th=[ 3851], | 70.00th=[ 4883], 80.00th=[ 6063], 90.00th=[ 7963], 95.00th=[ 9503], | 99.00th=[12256], 99.50th=[14877], 99.90th=[31327], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 168, max= 672, per=25.03%, avg=470.32, stdev=111.20, samples=25 iops : min= 42, max= 168, avg=117.52, stdev=27.81, samples=25 write: IOPS=124, BW=496KiB/s (508kB/s)(6296KiB/12689msec) clat (usec): min=459, max=131540, avg=4339.04, stdev=5281.15 lat (usec): min=460, max=131541, avg=4339.88, stdev=5281.26 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 545], 10.00th=[ 570], 20.00th=[ 652], | 30.00th=[ 963], 40.00th=[ 2008], 50.00th=[ 3359], 60.00th=[ 4293], | 70.00th=[ 5407], 80.00th=[ 7111], 90.00th=[ 9765], 95.00th=[ 11994], | 99.00th=[ 19268], 99.50th=[ 22414], 99.90th=[ 43254], 99.95th=[131597], | 99.99th=[131597] bw ( KiB/s): min= 184, max= 680, per=25.03%, avg=492.36, stdev=119.89, samples=25 iops : min= 46, max= 170, avg=123.04, stdev=29.97, samples=25 lat (usec) : 50=0.03%, 100=8.92%, 250=0.85%, 500=0.55%, 750=12.17% lat (usec) : 1000=3.35% lat (msec) : 2=5.99%, 4=27.51%, 10=33.72%, 20=6.32%, 50=0.52% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.21%, sys=25.08%, ctx=6179, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23572: Thu Jul 25 01:34:43 2024 read: IOPS=116, BW=465KiB/s (476kB/s)(5940KiB/12775msec) clat (usec): min=48, max=39586, avg=3591.16, stdev=3035.57 lat (usec): min=48, max=39587, avg=3591.78, stdev=3035.58 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 62], 10.00th=[ 72], 20.00th=[ 145], | 30.00th=[ 2278], 40.00th=[ 2606], 50.00th=[ 3032], 60.00th=[ 3621], | 70.00th=[ 4490], 80.00th=[ 5735], 90.00th=[ 7373], 95.00th=[ 8848], | 99.00th=[12780], 99.50th=[13829], 99.90th=[23987], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 176, max= 648, per=24.50%, avg=460.40, stdev=102.73, samples=25 iops : min= 44, max= 162, avg=115.04, stdev=25.68, samples=25 write: IOPS=124, BW=497KiB/s (509kB/s)(6348KiB/12775msec) clat (usec): min=461, max=171107, avg=4614.97, stdev=6253.36 lat (usec): min=461, max=171108, avg=4616.20, stdev=6253.39 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 545], 10.00th=[ 578], 20.00th=[ 685], | 30.00th=[ 988], 40.00th=[ 2474], 50.00th=[ 3556], 60.00th=[ 4424], | 70.00th=[ 5604], 80.00th=[ 7242], 90.00th=[ 10421], 95.00th=[ 12911], | 99.00th=[ 20841], 99.50th=[ 26870], 99.90th=[ 61604], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 232, max= 712, per=24.74%, avg=486.64, stdev=109.92, samples=25 iops : min= 58, max= 178, avg=121.60, stdev=27.51, samples=25 lat (usec) : 50=0.20%, 100=8.24%, 250=1.60%, 500=0.72%, 750=11.43% lat (usec) : 1000=3.61% lat (msec) : 2=5.34%, 4=28.55%, 10=33.30%, 20=6.41%, 50=0.55% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.16%, sys=24.69%, ctx=6214, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23573: Thu Jul 25 01:34:43 2024 read: IOPS=118, BW=476KiB/s (487kB/s)(6028KiB/12665msec) clat (usec): min=47, max=34625, avg=3767.69, stdev=3307.56 lat (usec): min=48, max=34626, avg=3768.38, stdev=3307.56 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 61], 10.00th=[ 68], 20.00th=[ 113], | 30.00th=[ 2278], 40.00th=[ 2638], 50.00th=[ 3064], 60.00th=[ 3818], | 70.00th=[ 4817], 80.00th=[ 6063], 90.00th=[ 7898], 95.00th=[ 9372], | 99.00th=[14877], 99.50th=[19792], 99.90th=[24249], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 120, max= 688, per=25.11%, avg=471.84, stdev=126.81, samples=25 iops : min= 30, max= 172, avg=117.88, stdev=31.72, samples=25 write: IOPS=123, BW=494KiB/s (506kB/s)(6260KiB/12665msec) clat (usec): min=459, max=193639, avg=4391.73, stdev=6540.62 lat (usec): min=459, max=193641, avg=4392.86, stdev=6540.86 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 545], 10.00th=[ 570], 20.00th=[ 644], | 30.00th=[ 840], 40.00th=[ 1795], 50.00th=[ 3359], 60.00th=[ 4293], | 70.00th=[ 5473], 80.00th=[ 7177], 90.00th=[ 9503], 95.00th=[ 12125], | 99.00th=[ 19268], 99.50th=[ 23725], 99.90th=[ 55313], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 160, max= 728, per=24.85%, avg=488.80, stdev=122.15, samples=25 iops : min= 40, max= 182, avg=122.12, stdev=30.59, samples=25 lat (usec) : 50=0.16%, 100=9.44%, 250=0.85%, 500=0.65%, 750=13.51% lat (usec) : 1000=2.86% lat (msec) : 2=5.34%, 4=26.20%, 10=34.57%, 20=5.79%, 50=0.55% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.22%, sys=25.29%, ctx=6033, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23574: Thu Jul 25 01:34:43 2024 read: IOPS=119, BW=479KiB/s (491kB/s)(6052KiB/12634msec) clat (usec): min=48, max=39714, avg=3829.26, stdev=3297.12 lat (usec): min=48, max=39715, avg=3830.00, stdev=3296.98 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 63], 10.00th=[ 73], 20.00th=[ 424], | 30.00th=[ 2278], 40.00th=[ 2671], 50.00th=[ 3097], 60.00th=[ 3818], | 70.00th=[ 4883], 80.00th=[ 6128], 90.00th=[ 7832], 95.00th=[ 9503], | 99.00th=[13698], 99.50th=[18482], 99.90th=[24511], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 192, max= 680, per=25.44%, avg=478.00, stdev=111.37, samples=25 iops : min= 48, max= 170, avg=119.44, stdev=27.85, samples=25 write: IOPS=123, BW=494KiB/s (505kB/s)(6236KiB/12634msec) clat (usec): min=455, max=172343, avg=4312.06, stdev=6133.05 lat (usec): min=456, max=172344, avg=4313.84, stdev=6132.74 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 537], 10.00th=[ 570], 20.00th=[ 635], | 30.00th=[ 832], 40.00th=[ 1745], 50.00th=[ 3163], 60.00th=[ 4228], | 70.00th=[ 5473], 80.00th=[ 7111], 90.00th=[ 9765], 95.00th=[ 11994], | 99.00th=[ 17957], 99.50th=[ 21890], 99.90th=[ 62653], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 208, max= 688, per=25.10%, avg=493.64, stdev=121.00, samples=25 iops : min= 52, max= 172, avg=123.36, stdev=30.27, samples=25 lat (usec) : 50=0.16%, 100=8.79%, 250=0.88%, 500=0.55%, 750=13.31% lat (usec) : 1000=3.52% lat (msec) : 2=4.88%, 4=27.96%, 10=33.50%, 20=5.92%, 50=0.46% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.14%, sys=24.78%, ctx=6139, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1880KiB/s (1925kB/s), 465KiB/s-479KiB/s (476kB/s-491kB/s), io=23.4MiB (24.6MB), run=12634-12775msec WRITE: bw=1968KiB/s (2015kB/s), 494KiB/s-497KiB/s (505kB/s-509kB/s), io=24.6MiB (25.7MB), run=12634-12775msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23576: Thu Jul 25 01:35:10 2024 read: IOPS=37, BW=151KiB/s (154kB/s)(5992KiB/39776msec) clat (usec): min=1597, max=222773, avg=3829.42, stdev=6320.24 lat (usec): min=1598, max=222775, avg=3830.40, stdev=6320.22 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1942], 10.00th=[ 2114], 20.00th=[ 2343], | 30.00th=[ 2507], 40.00th=[ 2606], 50.00th=[ 2737], 60.00th=[ 2900], | 70.00th=[ 3130], 80.00th=[ 3785], 90.00th=[ 7111], 95.00th=[ 9765], | 99.00th=[ 15533], 99.50th=[ 18482], 99.90th=[ 38536], 99.95th=[223347], | 99.99th=[223347] bw ( KiB/s): min= 56, max= 248, per=25.17%, avg=150.53, stdev=53.72, samples=79 iops : min= 14, max= 62, avg=37.59, stdev=13.38, samples=79 write: IOPS=39, BW=158KiB/s (162kB/s)(6296KiB/39776msec) clat (usec): min=11628, max=80011, avg=21561.46, stdev=6970.27 lat (usec): min=11629, max=80011, avg=21562.55, stdev=6970.25 clat percentiles (usec): | 1.00th=[12649], 5.00th=[13698], 10.00th=[14222], 20.00th=[15270], | 30.00th=[17171], 40.00th=[19268], 50.00th=[20841], 60.00th=[22152], | 70.00th=[23462], 80.00th=[25560], 90.00th=[30540], 95.00th=[34866], | 99.00th=[42730], 99.50th=[47449], 99.90th=[64750], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 40, max= 208, per=25.26%, avg=158.13, stdev=39.01, samples=79 iops : min= 10, max= 52, avg=39.49, stdev= 9.73, samples=79 lat (msec) : 2=3.03%, 4=36.56%, 10=7.00%, 20=24.64%, 50=28.52% lat (msec) : 100=0.23%, 250=0.03% cpu : usr=0.11%, sys=5.74%, ctx=3422, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23577: Thu Jul 25 01:35:10 2024 read: IOPS=36, BW=148KiB/s (151kB/s)(5940KiB/40151msec) clat (usec): min=1643, max=175583, avg=3754.57, stdev=5416.57 lat (usec): min=1643, max=175585, avg=3755.40, stdev=5416.58 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1942], 10.00th=[ 2057], 20.00th=[ 2311], | 30.00th=[ 2442], 40.00th=[ 2573], 50.00th=[ 2704], 60.00th=[ 2868], | 70.00th=[ 3130], 80.00th=[ 3654], 90.00th=[ 6652], 95.00th=[ 9241], | 99.00th=[ 15139], 99.50th=[ 20579], 99.90th=[ 44303], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 16, max= 312, per=24.74%, avg=147.93, stdev=62.36, samples=80 iops : min= 4, max= 78, avg=36.93, stdev=15.62, samples=80 write: IOPS=39, BW=158KiB/s (162kB/s)(6348KiB/40151msec) clat (msec): min=11, max=111, avg=21.73, stdev= 7.37 lat (msec): min=11, max=111, avg=21.73, stdev= 7.37 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 21], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 26], 90.00th=[ 32], 95.00th=[ 37], | 99.00th=[ 45], 99.50th=[ 49], 99.90th=[ 59], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 40, max= 208, per=25.21%, avg=157.81, stdev=39.33, samples=80 iops : min= 10, max= 52, avg=39.39, stdev= 9.82, samples=80 lat (msec) : 2=3.45%, 4=36.43%, 10=6.64%, 20=25.13%, 50=28.19% lat (msec) : 100=0.10%, 250=0.07% cpu : usr=0.07%, sys=5.71%, ctx=3396, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23578: Thu Jul 25 01:35:10 2024 read: IOPS=37, BW=151KiB/s (155kB/s)(6028KiB/39902msec) clat (usec): min=1705, max=50716, avg=3823.93, stdev=3408.93 lat (usec): min=1706, max=50717, avg=3824.71, stdev=3408.90 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1942], 10.00th=[ 2057], 20.00th=[ 2311], | 30.00th=[ 2474], 40.00th=[ 2606], 50.00th=[ 2769], 60.00th=[ 2933], | 70.00th=[ 3195], 80.00th=[ 3851], 90.00th=[ 7111], 95.00th=[10028], | 99.00th=[16909], 99.50th=[22414], 99.90th=[44303], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 47, max= 295, per=25.23%, avg=150.86, stdev=53.65, samples=79 iops : min= 11, max= 73, avg=37.62, stdev=13.40, samples=79 write: IOPS=39, BW=157KiB/s (161kB/s)(6260KiB/39902msec) clat (msec): min=11, max=190, avg=21.76, stdev= 8.09 lat (msec): min=11, max=190, avg=21.76, stdev= 8.09 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 26], 90.00th=[ 31], 95.00th=[ 36], | 99.00th=[ 45], 99.50th=[ 49], 99.90th=[ 65], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 47, max= 200, per=24.97%, avg=156.33, stdev=40.21, samples=79 iops : min= 11, max= 50, avg=38.99, stdev=10.05, samples=79 lat (msec) : 2=3.94%, 4=35.48%, 10=7.13%, 20=23.83%, 50=29.43% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.07%, sys=5.84%, ctx=3436, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23579: Thu Jul 25 01:35:10 2024 read: IOPS=38, BW=152KiB/s (156kB/s)(6052KiB/39689msec) clat (usec): min=1635, max=45718, avg=3703.58, stdev=3211.94 lat (usec): min=1636, max=45718, avg=3704.55, stdev=3211.89 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1942], 10.00th=[ 2089], 20.00th=[ 2343], | 30.00th=[ 2474], 40.00th=[ 2606], 50.00th=[ 2737], 60.00th=[ 2900], | 70.00th=[ 3064], 80.00th=[ 3687], 90.00th=[ 6915], 95.00th=[ 9896], | 99.00th=[15664], 99.50th=[18744], 99.90th=[42206], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 32, max= 296, per=25.53%, avg=152.66, stdev=56.81, samples=79 iops : min= 8, max= 74, avg=38.13, stdev=14.22, samples=79 write: IOPS=39, BW=157KiB/s (161kB/s)(6236KiB/39689msec) clat (msec): min=11, max=201, avg=21.80, stdev= 8.56 lat (msec): min=11, max=201, avg=21.80, stdev= 8.56 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 21], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 26], 90.00th=[ 32], 95.00th=[ 37], | 99.00th=[ 46], 99.50th=[ 50], 99.90th=[ 81], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 48, max= 216, per=25.05%, avg=156.81, stdev=39.62, samples=79 iops : min= 12, max= 54, avg=39.16, stdev= 9.90, samples=79 lat (msec) : 2=3.22%, 4=37.43%, 10=6.25%, 20=25.42%, 50=27.51% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.10%, sys=5.78%, ctx=3423, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=598KiB/s (612kB/s), 148KiB/s-152KiB/s (151kB/s-156kB/s), io=23.4MiB (24.6MB), run=39689-40151msec WRITE: bw=626KiB/s (641kB/s), 157KiB/s-158KiB/s (161kB/s-162kB/s), io=24.6MiB (25.7MB), run=39689-40151msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23621: Thu Jul 25 01:35:16 2024 read: IOPS=83, BW=1339KiB/s (1371kB/s)(5920KiB/4421msec) clat (usec): min=51, max=24828, avg=8783.71, stdev=4578.55 lat (usec): min=52, max=24829, avg=8784.29, stdev=4578.57 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 62], 10.00th=[ 74], 20.00th=[ 7046], | 30.00th=[ 7963], 40.00th=[ 8848], 50.00th=[ 9634], 60.00th=[10421], | 70.00th=[11076], 80.00th=[11863], 90.00th=[13173], 95.00th=[15008], | 99.00th=[19268], 99.50th=[20841], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 1024, max= 1440, per=25.49%, avg=1296.00, stdev=124.52, samples=8 iops : min= 64, max= 90, avg=81.00, stdev= 7.78, samples=8 write: IOPS=90, BW=1440KiB/s (1475kB/s)(6368KiB/4421msec) clat (usec): min=629, max=42020, avg=2845.18, stdev=3625.53 lat (usec): min=630, max=42021, avg=2846.15, stdev=3625.55 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 709], 10.00th=[ 750], 20.00th=[ 791], | 30.00th=[ 832], 40.00th=[ 898], 50.00th=[ 1237], 60.00th=[ 2507], | 70.00th=[ 3294], 80.00th=[ 4555], 90.00th=[ 6063], 95.00th=[ 7767], | 99.00th=[16057], 99.50th=[29754], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 1120, max= 1920, per=26.47%, avg=1416.00, stdev=276.20, samples=8 iops : min= 70, max= 120, avg=88.50, stdev=17.26, samples=8 lat (usec) : 100=6.77%, 250=1.17%, 750=5.21%, 1000=18.10% lat (msec) : 2=4.56%, 4=11.20%, 10=30.21%, 20=22.27%, 50=0.52% cpu : usr=0.09%, sys=22.04%, ctx=2012, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23622: Thu Jul 25 01:35:16 2024 read: IOPS=83, BW=1339KiB/s (1371kB/s)(5744KiB/4291msec) clat (usec): min=53, max=34219, avg=8763.88, stdev=4889.93 lat (usec): min=53, max=34220, avg=8764.49, stdev=4889.96 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 62], 10.00th=[ 74], 20.00th=[ 6915], | 30.00th=[ 8094], 40.00th=[ 8848], 50.00th=[ 9634], 60.00th=[10421], | 70.00th=[11207], 80.00th=[11863], 90.00th=[12911], 95.00th=[14091], | 99.00th=[19006], 99.50th=[32113], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 1024, max= 1632, per=26.20%, avg=1332.00, stdev=184.17, samples=8 iops : min= 64, max= 102, avg=83.25, stdev=11.51, samples=8 write: IOPS=95, BW=1525KiB/s (1562kB/s)(6544KiB/4291msec) clat (usec): min=615, max=25775, avg=2760.13, stdev=3291.50 lat (usec): min=616, max=25776, avg=2762.50, stdev=3290.82 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 709], 10.00th=[ 758], 20.00th=[ 799], | 30.00th=[ 865], 40.00th=[ 979], 50.00th=[ 1516], 60.00th=[ 2442], | 70.00th=[ 2999], 80.00th=[ 3949], 90.00th=[ 5932], 95.00th=[ 7963], | 99.00th=[20579], 99.50th=[21627], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 1056, max= 1984, per=27.89%, avg=1492.00, stdev=287.21, samples=8 iops : min= 66, max= 124, avg=93.25, stdev=17.95, samples=8 lat (usec) : 100=7.16%, 250=0.78%, 500=0.13%, 750=4.95%, 1000=16.80% lat (msec) : 2=6.51%, 4=14.58%, 10=26.69%, 20=21.35%, 50=1.04% cpu : usr=0.14%, sys=22.66%, ctx=1958, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23624: Thu Jul 25 01:35:16 2024 read: IOPS=82, BW=1328KiB/s (1360kB/s)(6256KiB/4711msec) clat (usec): min=48, max=30023, avg=8604.89, stdev=5041.31 lat (usec): min=48, max=30023, avg=8605.49, stdev=5041.33 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 62], 10.00th=[ 65], 20.00th=[ 5997], | 30.00th=[ 7767], 40.00th=[ 8717], 50.00th=[ 9503], 60.00th=[10159], | 70.00th=[11076], 80.00th=[11863], 90.00th=[13698], 95.00th=[15401], | 99.00th=[21627], 99.50th=[22938], 99.90th=[30016], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 416, max= 1920, per=25.95%, avg=1319.11, stdev=399.61, samples=9 iops : min= 26, max= 120, avg=82.44, stdev=24.98, samples=9 write: IOPS=80, BW=1280KiB/s (1311kB/s)(6032KiB/4711msec) clat (usec): min=621, max=37889, avg=3324.73, stdev=5024.77 lat (usec): min=622, max=37890, avg=3325.59, stdev=5024.80 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 693], 10.00th=[ 742], 20.00th=[ 783], | 30.00th=[ 832], 40.00th=[ 898], 50.00th=[ 1369], 60.00th=[ 2474], | 70.00th=[ 2999], 80.00th=[ 4113], 90.00th=[ 6652], 95.00th=[14222], | 99.00th=[32113], 99.50th=[35914], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 480, max= 1600, per=23.20%, avg=1240.89, stdev=359.83, samples=9 iops : min= 30, max= 100, avg=77.56, stdev=22.49, samples=9 lat (usec) : 50=0.13%, 100=9.37%, 250=0.52%, 750=5.86%, 1000=16.28% lat (msec) : 2=4.17%, 4=12.11%, 10=26.82%, 20=23.05%, 50=1.69% cpu : usr=0.13%, sys=20.36%, ctx=2031, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23626: Thu Jul 25 01:35:16 2024 read: IOPS=86, BW=1387KiB/s (1420kB/s)(6032KiB/4349msec) clat (usec): min=49, max=24974, avg=8419.73, stdev=4957.56 lat (usec): min=49, max=24974, avg=8420.32, stdev=4957.49 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 62], 10.00th=[ 65], 20.00th=[ 117], | 30.00th=[ 8029], 40.00th=[ 8717], 50.00th=[ 9634], 60.00th=[10290], | 70.00th=[10945], 80.00th=[11731], 90.00th=[13042], 95.00th=[15270], | 99.00th=[18744], 99.50th=[20579], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 896, max= 1756, per=26.82%, avg=1363.50, stdev=280.75, samples=8 iops : min= 56, max= 109, avg=85.13, stdev=17.40, samples=8 write: IOPS=89, BW=1438KiB/s (1473kB/s)(6256KiB/4349msec) clat (usec): min=656, max=29877, avg=2698.67, stdev=3224.16 lat (usec): min=657, max=29878, avg=2699.54, stdev=3224.17 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 734], 10.00th=[ 758], 20.00th=[ 799], | 30.00th=[ 832], 40.00th=[ 938], 50.00th=[ 1336], 60.00th=[ 2409], | 70.00th=[ 2900], 80.00th=[ 3916], 90.00th=[ 5735], 95.00th=[ 7570], | 99.00th=[17433], 99.50th=[24511], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 992, max= 1728, per=26.91%, avg=1439.62, stdev=223.27, samples=8 iops : min= 62, max= 108, avg=89.88, stdev=14.02, samples=8 lat (usec) : 50=0.13%, 100=9.64%, 250=0.39%, 500=0.13%, 750=3.52% lat (usec) : 1000=19.14% lat (msec) : 2=4.69%, 4=13.80%, 10=25.00%, 20=22.79%, 50=0.78% cpu : usr=0.07%, sys=21.76%, ctx=1987, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5084KiB/s (5206kB/s), 1328KiB/s-1387KiB/s (1360kB/s-1420kB/s), io=23.4MiB (24.5MB), run=4291-4711msec WRITE: bw=5349KiB/s (5478kB/s), 1280KiB/s-1525KiB/s (1311kB/s-1562kB/s), io=24.6MiB (25.8MB), run=4291-4711msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23623: Thu Jul 25 01:35:21 2024 read: IOPS=37, BW=593KiB/s (607kB/s)(5920KiB/9987msec) clat (usec): min=1967, max=36829, avg=4335.85, stdev=3745.11 lat (usec): min=1968, max=36830, avg=4336.56, stdev=3745.10 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2180], 10.00th=[ 2278], 20.00th=[ 2442], | 30.00th=[ 2573], 40.00th=[ 2704], 50.00th=[ 2933], 60.00th=[ 3195], | 70.00th=[ 4015], 80.00th=[ 4883], 90.00th=[ 7832], 95.00th=[12518], | 99.00th=[19268], 99.50th=[24511], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 352, max= 896, per=25.04%, avg=587.79, stdev=156.84, samples=19 iops : min= 22, max= 56, avg=36.74, stdev= 9.80, samples=19 write: IOPS=39, BW=638KiB/s (653kB/s)(6368KiB/9987msec) clat (usec): min=11601, max=55160, avg=20938.78, stdev=5976.38 lat (usec): min=11602, max=55174, avg=20939.99, stdev=5976.51 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13566], 10.00th=[14091], 20.00th=[15533], | 30.00th=[17171], 40.00th=[19006], 50.00th=[20317], 60.00th=[21890], | 70.00th=[23200], 80.00th=[24773], 90.00th=[28181], 95.00th=[31589], | 99.00th=[41681], 99.50th=[44827], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 416, max= 800, per=25.58%, avg=631.58, stdev=134.01, samples=19 iops : min= 26, max= 50, avg=39.47, stdev= 8.38, samples=19 lat (msec) : 2=0.26%, 4=33.20%, 10=10.94%, 20=28.26%, 50=27.21% lat (msec) : 100=0.13% cpu : usr=0.12%, sys=6.15%, ctx=913, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23625: Thu Jul 25 01:35:21 2024 read: IOPS=35, BW=563KiB/s (576kB/s)(5744KiB/10203msec) clat (usec): min=1981, max=24335, avg=4122.04, stdev=3104.43 lat (usec): min=1982, max=24335, avg=4122.75, stdev=3104.41 clat percentiles (usec): | 1.00th=[ 2073], 5.00th=[ 2212], 10.00th=[ 2343], 20.00th=[ 2474], | 30.00th=[ 2606], 40.00th=[ 2737], 50.00th=[ 2900], 60.00th=[ 3097], | 70.00th=[ 3752], 80.00th=[ 4817], 90.00th=[ 6980], 95.00th=[11863], | 99.00th=[16909], 99.50th=[19006], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 192, max= 928, per=24.00%, avg=563.20, stdev=199.54, samples=20 iops : min= 12, max= 58, avg=35.20, stdev=12.47, samples=20 write: IOPS=40, BW=641KiB/s (657kB/s)(6544KiB/10203msec) clat (usec): min=12492, max=56824, avg=21092.63, stdev=6236.96 lat (usec): min=12493, max=56826, avg=21093.89, stdev=6236.91 clat percentiles (usec): | 1.00th=[12780], 5.00th=[13698], 10.00th=[14222], 20.00th=[15401], | 30.00th=[16909], 40.00th=[18482], 50.00th=[20317], 60.00th=[21890], | 70.00th=[23200], 80.00th=[25560], 90.00th=[28705], 95.00th=[32113], | 99.00th=[42206], 99.50th=[45876], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 384, max= 864, per=25.86%, avg=638.40, stdev=129.04, samples=20 iops : min= 24, max= 54, avg=39.90, stdev= 8.06, samples=20 lat (msec) : 2=0.26%, 4=33.33%, 10=10.03%, 20=28.12%, 50=28.12% lat (msec) : 100=0.13% cpu : usr=0.09%, sys=6.08%, ctx=929, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23627: Thu Jul 25 01:35:21 2024 read: IOPS=39, BW=638KiB/s (654kB/s)(6256KiB/9802msec) clat (usec): min=2039, max=24338, avg=4520.97, stdev=3474.14 lat (usec): min=2040, max=24338, avg=4522.05, stdev=3473.94 clat percentiles (usec): | 1.00th=[ 2147], 5.00th=[ 2343], 10.00th=[ 2442], 20.00th=[ 2540], | 30.00th=[ 2704], 40.00th=[ 2835], 50.00th=[ 3032], 60.00th=[ 3392], | 70.00th=[ 4359], 80.00th=[ 5669], 90.00th=[ 8029], 95.00th=[12518], | 99.00th=[19006], 99.50th=[20317], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 352, max= 1248, per=27.41%, avg=643.37, stdev=239.44, samples=19 iops : min= 22, max= 78, avg=40.21, stdev=14.97, samples=19 write: IOPS=38, BW=615KiB/s (630kB/s)(6032KiB/9802msec) clat (usec): min=12105, max=46194, avg=21266.25, stdev=6064.89 lat (usec): min=12106, max=46196, avg=21267.48, stdev=6064.84 clat percentiles (usec): | 1.00th=[12780], 5.00th=[13435], 10.00th=[14353], 20.00th=[15926], | 30.00th=[17695], 40.00th=[19006], 50.00th=[20579], 60.00th=[21890], | 70.00th=[23200], 80.00th=[25297], 90.00th=[29754], 95.00th=[32900], | 99.00th=[40109], 99.50th=[45351], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 416, max= 800, per=24.76%, avg=611.37, stdev=127.95, samples=19 iops : min= 26, max= 50, avg=38.21, stdev= 8.00, samples=19 lat (msec) : 4=33.98%, 10=13.02%, 20=26.69%, 50=26.30% cpu : usr=0.06%, sys=6.39%, ctx=925, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23628: Thu Jul 25 01:35:21 2024 read: IOPS=37, BW=601KiB/s (616kB/s)(6032KiB/10031msec) clat (usec): min=1873, max=31566, avg=4379.03, stdev=3759.13 lat (usec): min=1874, max=31566, avg=4379.72, stdev=3759.09 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2212], 10.00th=[ 2311], 20.00th=[ 2474], | 30.00th=[ 2606], 40.00th=[ 2737], 50.00th=[ 2933], 60.00th=[ 3097], | 70.00th=[ 3884], 80.00th=[ 5211], 90.00th=[ 8717], 95.00th=[11994], | 99.00th=[20841], 99.50th=[23462], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 288, max= 928, per=25.43%, avg=596.80, stdev=178.70, samples=20 iops : min= 18, max= 58, avg=37.30, stdev=11.17, samples=20 write: IOPS=38, BW=624KiB/s (639kB/s)(6256KiB/10031msec) clat (usec): min=12284, max=53045, avg=21124.00, stdev=6220.56 lat (usec): min=12285, max=53045, avg=21125.18, stdev=6220.58 clat percentiles (usec): | 1.00th=[12649], 5.00th=[13566], 10.00th=[14484], 20.00th=[15664], | 30.00th=[17171], 40.00th=[18482], 50.00th=[20317], 60.00th=[21627], | 70.00th=[22938], 80.00th=[24773], 90.00th=[29754], 95.00th=[33162], | 99.00th=[41681], 99.50th=[45351], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 416, max= 832, per=25.27%, avg=624.00, stdev=135.96, samples=20 iops : min= 26, max= 52, avg=39.00, stdev= 8.50, samples=20 lat (msec) : 2=0.26%, 4=34.24%, 10=10.81%, 20=27.21%, 50=27.34% lat (msec) : 100=0.13% cpu : usr=0.07%, sys=6.21%, ctx=932, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2348KiB/s (2404kB/s), 563KiB/s-638KiB/s (576kB/s-654kB/s), io=23.4MiB (24.5MB), run=9802-10203msec WRITE: bw=2470KiB/s (2529kB/s), 615KiB/s-641KiB/s (630kB/s-657kB/s), io=24.6MiB (25.8MB), run=9802-10203msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23644: Thu Jul 25 01:35:26 2024 read: IOPS=1, BW=2042KiB/s (2091kB/s)(7168KiB/3511msec) clat (msec): min=18, max=957, avg=192.31, stdev=342.60 lat (msec): min=18, max=957, avg=192.31, stdev=342.60 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 23], | 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 61], 60.00th=[ 77], | 70.00th=[ 77], 80.00th=[ 188], 90.00th=[ 961], 95.00th=[ 961], | 99.00th=[ 961], 99.50th=[ 961], 99.90th=[ 961], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 2048, max= 6144, per=76.55%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 write: IOPS=1, BW=1458KiB/s (1493kB/s)(5120KiB/3511msec) clat (msec): min=49, max=991, avg=430.48, stdev=499.71 lat (msec): min=49, max=991, avg=430.52, stdev=499.72 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 50], | 30.00th=[ 73], 40.00th=[ 73], 50.00th=[ 75], 60.00th=[ 75], | 70.00th=[ 961], 80.00th=[ 961], 90.00th=[ 995], 95.00th=[ 995], | 99.00th=[ 995], 99.50th=[ 995], 99.90th=[ 995], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 2048, max= 4096, per=45.56%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 lat (msec) : 20=8.33%, 50=25.00%, 100=33.33%, 250=8.33%, 1000=25.00% cpu : usr=0.00%, sys=2.85%, ctx=43, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23646: Thu Jul 25 01:35:26 2024 read: IOPS=0, BW=556KiB/s (569kB/s)(2048KiB/3683msec) clat (msec): min=75, max=1094, avg=584.68, stdev=720.65 lat (msec): min=75, max=1094, avg=584.69, stdev=720.65 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 75], 10.00th=[ 75], 20.00th=[ 75], | 30.00th=[ 75], 40.00th=[ 75], 50.00th=[ 75], 60.00th=[ 1099], | 70.00th=[ 1099], 80.00th=[ 1099], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 1796, max= 2048, per=35.92%, avg=1922.00, stdev=178.19, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2780KiB/s (2847kB/s)(10.0MiB/3683msec) clat (msec): min=35, max=1977, avg=249.52, stdev=607.23 lat (msec): min=35, max=1977, avg=249.58, stdev=607.23 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 45], | 30.00th=[ 46], 40.00th=[ 47], 50.00th=[ 54], 60.00th=[ 58], | 70.00th=[ 74], 80.00th=[ 74], 90.00th=[ 96], 95.00th=[ 1972], | 99.00th=[ 1972], 99.50th=[ 1972], 99.90th=[ 1972], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 5389, max=10219, per=100.00%, avg=7804.00, stdev=3415.33, samples=2 iops : min= 5, max= 9, avg= 7.00, stdev= 2.83, samples=2 lat (msec) : 50=33.33%, 100=50.00% cpu : usr=0.05%, sys=3.88%, ctx=56, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23648: Thu Jul 25 01:35:26 2024 read: IOPS=1, BW=1605KiB/s (1644kB/s)(6144KiB/3827msec) clat (msec): min=20, max=163, avg=64.88, stdev=54.58 lat (msec): min=20, max=163, avg=64.88, stdev=54.58 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 29], 50.00th=[ 29], 60.00th=[ 66], | 70.00th=[ 86], 80.00th=[ 86], 90.00th=[ 163], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 3518, max= 3518, per=65.74%, avg=3518.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1605KiB/s (1644kB/s)(6144KiB/3827msec) clat (msec): min=38, max=3024, avg=569.68, stdev=1203.38 lat (msec): min=38, max=3024, avg=569.74, stdev=1203.37 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 46], | 30.00th=[ 46], 40.00th=[ 53], 50.00th=[ 53], 60.00th=[ 111], | 70.00th=[ 146], 80.00th=[ 146], 90.00th=[ 3037], 95.00th=[ 3037], | 99.00th=[ 3037], 99.50th=[ 3037], 99.90th=[ 3037], 99.95th=[ 3037], | 99.99th=[ 3037] bw ( KiB/s): min= 1759, max= 4087, per=39.01%, avg=2923.00, stdev=1646.14, samples=2 iops : min= 1, max= 3, avg= 2.00, stdev= 1.41, samples=2 lat (msec) : 50=41.67%, 100=25.00%, 250=25.00% cpu : usr=0.05%, sys=3.03%, ctx=37, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23650: Thu Jul 25 01:35:26 2024 read: IOPS=1, BW=1524KiB/s (1561kB/s)(5120KiB/3359msec) clat (usec): min=22533, max=47899, avg=32067.63, stdev=10875.25 lat (usec): min=22534, max=47900, avg=32068.59, stdev=10875.21 clat percentiles (usec): | 1.00th=[22414], 5.00th=[22414], 10.00th=[22414], 20.00th=[22414], | 30.00th=[23462], 40.00th=[23462], 50.00th=[27919], 60.00th=[27919], | 70.00th=[38536], 80.00th=[38536], 90.00th=[47973], 95.00th=[47973], | 99.00th=[47973], 99.50th=[47973], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2134KiB/s (2185kB/s)(7168KiB/3359msec) clat (msec): min=45, max=1951, avg=450.80, stdev=713.79 lat (msec): min=45, max=1951, avg=450.87, stdev=713.79 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 61], | 30.00th=[ 78], 40.00th=[ 78], 50.00th=[ 90], 60.00th=[ 136], | 70.00th=[ 136], 80.00th=[ 793], 90.00th=[ 1955], 95.00th=[ 1955], | 99.00th=[ 1955], 99.50th=[ 1955], 99.90th=[ 1955], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 2048, max= 8192, per=68.34%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (msec) : 50=50.00%, 100=25.00%, 250=8.33%, 1000=8.33% cpu : usr=0.03%, sys=4.79%, ctx=48, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5351KiB/s (5480kB/s), 556KiB/s-2042KiB/s (569kB/s-2091kB/s), io=20.0MiB (20.0MB), run=3359-3827msec WRITE: bw=7492KiB/s (7672kB/s), 1458KiB/s-2780KiB/s (1493kB/s-2847kB/s), io=28.0MiB (29.4MB), run=3359-3827msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23645: Thu Jul 25 01:35:27 2024 read: IOPS=1, BW=1962KiB/s (2009kB/s)(7168KiB/3653msec) clat (msec): min=18, max=1065, avg=486.02, stdev=478.30 lat (msec): min=18, max=1065, avg=486.02, stdev=478.30 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 47], | 30.00th=[ 153], 40.00th=[ 153], 50.00th=[ 211], 60.00th=[ 936], | 70.00th=[ 936], 80.00th=[ 969], 90.00th=[ 1062], 95.00th=[ 1062], | 99.00th=[ 1062], 99.50th=[ 1062], 99.90th=[ 1062], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 2043, max= 4096, per=57.63%, avg=2558.75, stdev=1024.84, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 write: IOPS=1, BW=1402KiB/s (1435kB/s)(5120KiB/3653msec) clat (usec): min=16062, max=85834, avg=38775.44, stdev=27631.96 lat (usec): min=16118, max=85895, avg=38837.94, stdev=27631.43 clat percentiles (usec): | 1.00th=[16057], 5.00th=[16057], 10.00th=[16057], 20.00th=[16057], | 30.00th=[21627], 40.00th=[21627], 50.00th=[33817], 60.00th=[33817], | 70.00th=[36439], 80.00th=[36439], 90.00th=[85459], 95.00th=[85459], | 99.00th=[85459], 99.50th=[85459], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 2043, max= 6144, per=65.85%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (msec) : 20=16.67%, 50=33.33%, 100=8.33%, 250=16.67%, 1000=16.67% cpu : usr=0.03%, sys=21.28%, ctx=1087, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23647: Thu Jul 25 01:35:27 2024 read: IOPS=0, BW=578KiB/s (592kB/s)(2048KiB/3544msec) clat (msec): min=77, max=1824, avg=950.97, stdev=1234.70 lat (msec): min=77, max=1824, avg=950.97, stdev=1234.70 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 79], 10.00th=[ 79], 20.00th=[ 79], | 30.00th=[ 79], 40.00th=[ 79], 50.00th=[ 79], 60.00th=[ 1821], | 70.00th=[ 1821], 80.00th=[ 1821], 90.00th=[ 1821], 95.00th=[ 1821], | 99.00th=[ 1821], 99.50th=[ 1821], 99.90th=[ 1821], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 2048, max= 2048, per=46.13%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2889KiB/s (2959kB/s)(10.0MiB/3544msec) clat (msec): min=24, max=1077, avg=163.26, stdev=324.97 lat (msec): min=24, max=1077, avg=163.31, stdev=324.98 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 27], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 48], 60.00th=[ 60], | 70.00th=[ 90], 80.00th=[ 90], 90.00th=[ 190], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 8192, max=10240, per=100.00%, avg=9216.00, stdev=1448.15, samples=2 iops : min= 8, max= 10, avg= 9.00, stdev= 1.41, samples=2 lat (msec) : 50=41.67%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=10.53%, ctx=372, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23649: Thu Jul 25 01:35:27 2024 read: IOPS=1, BW=1677KiB/s (1717kB/s)(6144KiB/3664msec) clat (usec): min=276, max=69693, avg=41988.30, stdev=32943.27 lat (usec): min=276, max=69694, avg=41989.15, stdev=32943.52 clat percentiles (usec): | 1.00th=[ 277], 5.00th=[ 277], 10.00th=[ 277], 20.00th=[ 441], | 30.00th=[ 441], 40.00th=[50070], 50.00th=[50070], 60.00th=[64750], | 70.00th=[66847], 80.00th=[66847], 90.00th=[69731], 95.00th=[69731], | 99.00th=[69731], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1677KiB/s (1717kB/s)(6144KiB/3664msec) clat (msec): min=18, max=3053, avg=558.37, stdev=1223.41 lat (msec): min=18, max=3053, avg=558.42, stdev=1223.41 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 20], | 30.00th=[ 20], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 77], | 70.00th=[ 153], 80.00th=[ 153], 90.00th=[ 3071], 95.00th=[ 3071], | 99.00th=[ 3071], 99.50th=[ 3071], 99.90th=[ 3071], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min= 4096, max= 4096, per=65.89%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 lat (usec) : 500=16.67% lat (msec) : 20=16.67%, 50=8.33%, 100=41.67%, 250=8.33% cpu : usr=0.00%, sys=7.04%, ctx=91, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23651: Thu Jul 25 01:35:27 2024 read: IOPS=1, BW=1110KiB/s (1137kB/s)(5120KiB/4612msec) clat (usec): min=281, max=827541, avg=205192.58, stdev=358188.51 lat (usec): min=282, max=827542, avg=205193.35, stdev=358188.42 clat percentiles (usec): | 1.00th=[ 281], 5.00th=[ 281], 10.00th=[ 281], 20.00th=[ 281], | 30.00th=[ 351], 40.00th=[ 351], 50.00th=[ 586], 60.00th=[ 586], | 70.00th=[198181], 80.00th=[198181], 90.00th=[826278], 95.00th=[826278], | 99.00th=[826278], 99.50th=[826278], 99.90th=[826278], 99.95th=[826278], | 99.99th=[826278] bw ( KiB/s): min= 2048, max= 4096, per=61.50%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 write: IOPS=1, BW=1554KiB/s (1592kB/s)(7168KiB/4612msec) clat (msec): min=26, max=3058, avg=507.24, stdev=1126.52 lat (msec): min=26, max=3058, avg=507.29, stdev=1126.52 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 39], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 66], 60.00th=[ 124], | 70.00th=[ 124], 80.00th=[ 188], 90.00th=[ 3071], 95.00th=[ 3071], | 99.00th=[ 3071], 99.50th=[ 3071], 99.90th=[ 3071], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min= 2043, max= 6144, per=65.85%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 50=16.67%, 100=16.67%, 250=25.00%, 1000=8.33% cpu : usr=0.02%, sys=10.39%, ctx=351, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4441KiB/s (4547kB/s), 578KiB/s-1962KiB/s (592kB/s-2009kB/s), io=20.0MiB (20.0MB), run=3544-4612msec WRITE: bw=6217KiB/s (6366kB/s), 1402KiB/s-2889KiB/s (1435kB/s-2959kB/s), io=28.0MiB (29.4MB), run=3544-4612msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23661: Thu Jul 25 01:35:30 2024 read: IOPS=2, BW=10.3MiB/s (10.8MB/s)(12.0MiB/1166msec) clat (msec): min=3, max=999, avg=377.77, stdev=542.30 lat (msec): min=3, max=999, avg=377.77, stdev=542.30 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 131], 50.00th=[ 131], 60.00th=[ 131], | 70.00th=[ 1003], 80.00th=[ 1003], 90.00th=[ 1003], 95.00th=[ 1003], | 99.00th=[ 1003], 99.50th=[ 1003], 99.90th=[ 1003], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 8192, max= 8192, per=87.10%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 4=33.33%, 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=56.91%, ctx=165, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23663: Thu Jul 25 01:35:30 2024 read: IOPS=0, BW=2351KiB/s (2408kB/s)(4096KiB/1742msec) clat (nsec): min=143501k, max=143501k, avg=143501171.00, stdev= 0.00 lat (nsec): min=143504k, max=143504k, avg=143503778.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 144], 50.00th=[ 144], 60.00th=[ 144], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 8175, max= 8175, per=86.92%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4703KiB/s (4816kB/s)(8192KiB/1742msec) clat (msec): min=245, max=1330, avg=788.02, stdev=767.71 lat (msec): min=245, max=1331, avg=788.25, stdev=767.70 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 245], 10.00th=[ 245], 20.00th=[ 245], | 30.00th=[ 245], 40.00th=[ 245], 50.00th=[ 245], 60.00th=[ 1334], | 70.00th=[ 1334], 80.00th=[ 1334], 90.00th=[ 1334], 95.00th=[ 1334], | 99.00th=[ 1334], 99.50th=[ 1334], 99.90th=[ 1334], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 8175, max= 8175, per=43.46%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.06%, sys=13.79%, ctx=318, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23665: Thu Jul 25 01:35:30 2024 write: IOPS=1, BW=7336KiB/s (7512kB/s)(12.0MiB/1675msec) clat (msec): min=66, max=1349, avg=540.47, stdev=704.24 lat (msec): min=66, max=1349, avg=540.74, stdev=704.26 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 67], | 30.00th=[ 67], 40.00th=[ 205], 50.00th=[ 205], 60.00th=[ 205], | 70.00th=[ 1351], 80.00th=[ 1351], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 8192, max= 8192, per=43.55%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=14.52%, ctx=42, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23666: Thu Jul 25 01:35:30 2024 write: IOPS=1, BW=7938KiB/s (8128kB/s)(12.0MiB/1548msec) clat (msec): min=105, max=1266, avg=498.17, stdev=665.04 lat (msec): min=105, max=1266, avg=498.44, stdev=665.04 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 106], 10.00th=[ 106], 20.00th=[ 106], | 30.00th=[ 106], 40.00th=[ 124], 50.00th=[ 124], 60.00th=[ 124], | 70.00th=[ 1267], 80.00th=[ 1267], 90.00th=[ 1267], 95.00th=[ 1267], | 99.00th=[ 1267], 99.50th=[ 1267], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 8192, max= 8192, per=43.55%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=18.55%, ctx=102, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9405KiB/s (9631kB/s), 2351KiB/s-10.3MiB/s (2408kB/s-10.8MB/s), io=16.0MiB (16.8MB), run=1166-1742msec WRITE: bw=18.4MiB/s (19.3MB/s), 4703KiB/s-7938KiB/s (4816kB/s-8128kB/s), io=32.0MiB (33.6MB), run=1548-1742msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23660: Thu Jul 25 01:35:30 2024 read: IOPS=1, BW=6578KiB/s (6736kB/s)(12.0MiB/1868msec) clat (msec): min=54, max=1120, avg=611.85, stdev=534.23 lat (msec): min=54, max=1120, avg=611.86, stdev=534.23 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 55], 10.00th=[ 55], 20.00th=[ 55], | 30.00th=[ 55], 40.00th=[ 659], 50.00th=[ 659], 60.00th=[ 659], | 70.00th=[ 1116], 80.00th=[ 1116], 90.00th=[ 1116], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 8192, max= 8192, per=98.76%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.05%, sys=4.23%, ctx=40, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23662: Thu Jul 25 01:35:30 2024 read: IOPS=0, BW=2074KiB/s (2124kB/s)(4096KiB/1975msec) clat (nsec): min=66593k, max=66593k, avg=66593216.00, stdev= 0.00 lat (nsec): min=66595k, max=66595k, avg=66595068.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[66847], 5.00th=[66847], 10.00th=[66847], 20.00th=[66847], | 30.00th=[66847], 40.00th=[66847], 50.00th=[66847], 60.00th=[66847], | 70.00th=[66847], 80.00th=[66847], 90.00th=[66847], 95.00th=[66847], | 99.00th=[66847], 99.50th=[66847], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 8192, max= 8192, per=98.76%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4148KiB/s (4247kB/s)(8192KiB/1975msec) clat (msec): min=209, max=1671, avg=940.70, stdev=1033.45 lat (msec): min=210, max=1671, avg=940.99, stdev=1033.49 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 211], 10.00th=[ 211], 20.00th=[ 211], | 30.00th=[ 211], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 1670], | 70.00th=[ 1670], 80.00th=[ 1670], 90.00th=[ 1670], 95.00th=[ 1670], | 99.00th=[ 1670], 99.50th=[ 1670], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 8192, max= 8192, per=49.38%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=4.05%, ctx=47, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23664: Thu Jul 25 01:35:30 2024 write: IOPS=1, BW=6488KiB/s (6644kB/s)(12.0MiB/1894msec) clat (msec): min=61, max=1116, avg=611.84, stdev=528.96 lat (msec): min=61, max=1116, avg=612.12, stdev=529.05 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 62], 10.00th=[ 62], 20.00th=[ 62], | 30.00th=[ 62], 40.00th=[ 659], 50.00th=[ 659], 60.00th=[ 659], | 70.00th=[ 1116], 80.00th=[ 1116], 90.00th=[ 1116], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 8192, max= 8192, per=49.38%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.05%, sys=4.60%, ctx=28, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23667: Thu Jul 25 01:35:30 2024 write: IOPS=1, BW=7553KiB/s (7734kB/s)(12.0MiB/1627msec) clat (msec): min=156, max=1270, avg=538.64, stdev=634.24 lat (msec): min=157, max=1271, avg=538.94, stdev=634.19 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 188], 50.00th=[ 188], 60.00th=[ 188], | 70.00th=[ 1267], 80.00th=[ 1267], 90.00th=[ 1267], 95.00th=[ 1267], | 99.00th=[ 1267], 99.50th=[ 1267], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 8192, max= 8192, per=49.38%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=66.67% cpu : usr=0.06%, sys=7.56%, ctx=116, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8296KiB/s (8495kB/s), 2074KiB/s-6578KiB/s (2124kB/s-6736kB/s), io=16.0MiB (16.8MB), run=1868-1975msec WRITE: bw=16.2MiB/s (16.0MB/s), 4148KiB/s-7553KiB/s (4247kB/s-7734kB/s), io=32.0MiB (33.6MB), run=1627-1975msec