== sanity test 398b: DIO and buffer IO race ============== 07:29:12 (1715340552) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.26528 s, 39.8 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17644: Fri May 10 07:29:28 2024 read: IOPS=115, BW=462KiB/s (473kB/s)(5992KiB/12965msec) clat (usec): min=50, max=174869, avg=4853.66, stdev=5210.33 lat (usec): min=51, max=174870, avg=4855.08, stdev=5210.43 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 118], 10.00th=[ 2311], 20.00th=[ 2704], | 30.00th=[ 3130], 40.00th=[ 3556], 50.00th=[ 4228], 60.00th=[ 4883], | 70.00th=[ 5604], 80.00th=[ 6587], 90.00th=[ 7963], 95.00th=[ 9372], | 99.00th=[ 15008], 99.50th=[ 18220], 99.90th=[ 21627], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 232, max= 592, per=25.30%, avg=457.92, stdev=68.01, samples=25 iops : min= 58, max= 148, avg=114.48, stdev=17.00, samples=25 write: IOPS=121, BW=486KiB/s (497kB/s)(6296KiB/12965msec) clat (usec): min=448, max=20426, avg=3576.57, stdev=3178.99 lat (usec): min=449, max=20427, avg=3577.49, stdev=3179.32 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 545], 10.00th=[ 586], 20.00th=[ 685], | 30.00th=[ 906], 40.00th=[ 2073], 50.00th=[ 2900], 60.00th=[ 3621], | 70.00th=[ 4555], 80.00th=[ 5932], 90.00th=[ 7832], 95.00th=[ 9896], | 99.00th=[13829], 99.50th=[15664], 99.90th=[18744], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 296, max= 648, per=25.28%, avg=479.04, stdev=72.05, samples=25 iops : min= 74, max= 162, avg=119.76, stdev=18.01, samples=25 lat (usec) : 100=2.08%, 250=0.49%, 500=0.20%, 750=12.11%, 1000=4.33% lat (msec) : 2=4.10%, 4=32.81%, 10=39.36%, 20=4.43%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=23.53%, ctx=6088, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17645: Fri May 10 07:29:28 2024 read: IOPS=111, BW=448KiB/s (459kB/s)(5940KiB/13263msec) clat (usec): min=54, max=20124, avg=4861.60, stdev=2594.84 lat (usec): min=55, max=20124, avg=4862.22, stdev=2594.85 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 1958], 10.00th=[ 2442], 20.00th=[ 2900], | 30.00th=[ 3359], 40.00th=[ 3851], 50.00th=[ 4490], 60.00th=[ 5080], | 70.00th=[ 5800], 80.00th=[ 6587], 90.00th=[ 7963], 95.00th=[ 9372], | 99.00th=[13698], 99.50th=[15139], 99.90th=[19006], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 248, max= 592, per=24.60%, avg=445.19, stdev=64.09, samples=26 iops : min= 62, max= 148, avg=111.27, stdev=16.03, samples=26 write: IOPS=119, BW=479KiB/s (490kB/s)(6348KiB/13263msec) clat (usec): min=428, max=173283, avg=3768.43, stdev=5370.47 lat (usec): min=429, max=173284, avg=3769.13, stdev=5370.50 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 545], 10.00th=[ 578], 20.00th=[ 676], | 30.00th=[ 857], 40.00th=[ 1926], 50.00th=[ 2900], 60.00th=[ 3621], | 70.00th=[ 4883], 80.00th=[ 6456], 90.00th=[ 8455], 95.00th=[ 10159], | 99.00th=[ 13304], 99.50th=[ 14484], 99.90th=[ 26870], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 288, max= 592, per=24.79%, avg=469.77, stdev=65.42, samples=26 iops : min= 72, max= 148, avg=117.42, stdev=16.33, samples=26 lat (usec) : 100=1.76%, 250=0.42%, 500=0.33%, 750=12.96%, 1000=4.00% lat (msec) : 2=3.97%, 4=30.05%, 10=41.96%, 20=4.46%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=23.26%, ctx=6172, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17646: Fri May 10 07:29:28 2024 read: IOPS=114, BW=459KiB/s (470kB/s)(6028KiB/13120msec) clat (usec): min=55, max=172514, avg=5033.70, stdev=5169.70 lat (usec): min=56, max=172515, avg=5034.32, stdev=5169.69 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 578], 10.00th=[ 2376], 20.00th=[ 2835], | 30.00th=[ 3294], 40.00th=[ 3785], 50.00th=[ 4424], 60.00th=[ 5211], | 70.00th=[ 5866], 80.00th=[ 6718], 90.00th=[ 8160], 95.00th=[ 9896], | 99.00th=[ 14353], 99.50th=[ 17957], 99.90th=[ 33424], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 232, max= 680, per=25.33%, avg=458.38, stdev=82.44, samples=26 iops : min= 58, max= 170, avg=114.58, stdev=20.59, samples=26 write: IOPS=119, BW=477KiB/s (489kB/s)(6260KiB/13120msec) clat (usec): min=430, max=18792, avg=3494.88, stdev=3061.22 lat (usec): min=431, max=18792, avg=3496.55, stdev=3061.62 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 537], 10.00th=[ 570], 20.00th=[ 668], | 30.00th=[ 865], 40.00th=[ 2008], 50.00th=[ 2868], 60.00th=[ 3621], | 70.00th=[ 4555], 80.00th=[ 5800], 90.00th=[ 7898], 95.00th=[ 9241], | 99.00th=[13173], 99.50th=[14484], 99.90th=[17957], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 312, max= 688, per=25.07%, avg=475.00, stdev=82.25, samples=26 iops : min= 78, max= 172, avg=118.73, stdev=20.56, samples=26 lat (usec) : 100=1.89%, 250=0.49%, 500=0.29%, 750=12.83%, 1000=4.10% lat (msec) : 2=3.58%, 4=30.83%, 10=41.83%, 20=4.07%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=23.61%, ctx=6308, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17647: Fri May 10 07:29:28 2024 read: IOPS=114, BW=458KiB/s (469kB/s)(6052KiB/13217msec) clat (usec): min=47, max=27266, avg=4869.46, stdev=2772.24 lat (usec): min=47, max=27267, avg=4871.53, stdev=2772.60 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 1893], 10.00th=[ 2343], 20.00th=[ 2868], | 30.00th=[ 3261], 40.00th=[ 3785], 50.00th=[ 4424], 60.00th=[ 5080], | 70.00th=[ 5735], 80.00th=[ 6718], 90.00th=[ 8029], 95.00th=[ 9372], | 99.00th=[14615], 99.50th=[18220], 99.90th=[25297], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 208, max= 592, per=25.06%, avg=453.50, stdev=76.41, samples=26 iops : min= 52, max= 148, avg=113.35, stdev=19.09, samples=26 write: IOPS=117, BW=472KiB/s (483kB/s)(6236KiB/13217msec) clat (usec): min=432, max=189734, avg=3710.60, stdev=5727.81 lat (usec): min=432, max=189736, avg=3712.11, stdev=5728.18 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 545], 10.00th=[ 578], 20.00th=[ 676], | 30.00th=[ 873], 40.00th=[ 1958], 50.00th=[ 2933], 60.00th=[ 3589], | 70.00th=[ 4490], 80.00th=[ 5997], 90.00th=[ 8160], 95.00th=[ 10028], | 99.00th=[ 14484], 99.50th=[ 16188], 99.90th=[ 22414], 99.95th=[189793], | 99.99th=[189793] bw ( KiB/s): min= 224, max= 672, per=24.65%, avg=467.04, stdev=85.68, samples=26 iops : min= 56, max= 168, avg=116.73, stdev=21.42, samples=26 lat (usec) : 50=0.07%, 100=1.60%, 250=0.55%, 500=0.39%, 750=12.04% lat (usec) : 1000=4.49% lat (msec) : 2=3.94%, 4=31.64%, 10=40.89%, 20=4.23%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=23.37%, ctx=6117, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1810KiB/s (1854kB/s), 448KiB/s-462KiB/s (459kB/s-473kB/s), io=23.4MiB (24.6MB), run=12965-13263msec WRITE: bw=1895KiB/s (1941kB/s), 472KiB/s-486KiB/s (483kB/s-497kB/s), io=24.6MiB (25.7MB), run=12965-13263msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17648: Fri May 10 07:29:33 2024 read: IOPS=84, BW=340KiB/s (348kB/s)(5992KiB/17633msec) clat (usec): min=1654, max=177660, avg=4768.83, stdev=5522.36 lat (usec): min=1655, max=177661, avg=4769.53, stdev=5522.36 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2057], 10.00th=[ 2180], 20.00th=[ 2442], | 30.00th=[ 2671], 40.00th=[ 2966], 50.00th=[ 3392], 60.00th=[ 3916], | 70.00th=[ 4752], 80.00th=[ 6390], 90.00th=[ 9503], 95.00th=[ 11994], | 99.00th=[ 15795], 99.50th=[ 16909], 99.90th=[ 30278], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 184, max= 664, per=25.01%, avg=338.60, stdev=124.19, samples=35 iops : min= 46, max= 166, avg=84.54, stdev=31.06, samples=35 write: IOPS=89, BW=357KiB/s (366kB/s)(6296KiB/17633msec) clat (usec): min=2338, max=33498, avg=6632.75, stdev=3812.95 lat (usec): min=2338, max=33499, avg=6633.66, stdev=3812.92 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3228], 10.00th=[ 3425], 20.00th=[ 3785], | 30.00th=[ 4228], 40.00th=[ 4752], 50.00th=[ 5276], 60.00th=[ 6063], | 70.00th=[ 7242], 80.00th=[ 8979], 90.00th=[11863], 95.00th=[14353], | 99.00th=[20841], 99.50th=[23200], 99.90th=[28705], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 200, max= 680, per=25.07%, avg=355.49, stdev=124.53, samples=35 iops : min= 50, max= 170, avg=88.77, stdev=31.17, samples=35 lat (msec) : 2=1.79%, 4=41.11%, 10=44.60%, 20=11.78%, 50=0.68% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=11.41%, ctx=3685, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17649: Fri May 10 07:29:33 2024 read: IOPS=84, BW=338KiB/s (346kB/s)(5940KiB/17575msec) clat (usec): min=1704, max=187767, avg=4684.46, stdev=5761.96 lat (usec): min=1705, max=187769, avg=4685.17, stdev=5761.98 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2040], 10.00th=[ 2212], 20.00th=[ 2442], | 30.00th=[ 2704], 40.00th=[ 2999], 50.00th=[ 3294], 60.00th=[ 3687], | 70.00th=[ 4490], 80.00th=[ 5997], 90.00th=[ 9372], 95.00th=[ 11469], | 99.00th=[ 18482], 99.50th=[ 20317], 99.90th=[ 27132], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 136, max= 696, per=24.98%, avg=338.29, stdev=130.43, samples=35 iops : min= 34, max= 174, avg=84.57, stdev=32.61, samples=35 write: IOPS=90, BW=361KiB/s (370kB/s)(6348KiB/17575msec) clat (usec): min=2322, max=30183, avg=6660.24, stdev=3782.48 lat (usec): min=2322, max=30184, avg=6661.05, stdev=3782.46 clat percentiles (usec): | 1.00th=[ 2802], 5.00th=[ 3195], 10.00th=[ 3458], 20.00th=[ 3851], | 30.00th=[ 4293], 40.00th=[ 4752], 50.00th=[ 5342], 60.00th=[ 6259], | 70.00th=[ 7373], 80.00th=[ 8848], 90.00th=[11731], 95.00th=[14484], | 99.00th=[20317], 99.50th=[23987], 99.90th=[28181], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 184, max= 672, per=25.40%, avg=360.23, stdev=121.32, samples=35 iops : min= 46, max= 168, avg=90.06, stdev=30.33, samples=35 lat (msec) : 2=1.76%, 4=41.67%, 10=44.53%, 20=11.20%, 50=0.81% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=11.63%, ctx=3653, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17650: Fri May 10 07:29:33 2024 read: IOPS=85, BW=343KiB/s (351kB/s)(6028KiB/17595msec) clat (usec): min=1602, max=25808, avg=4709.72, stdev=3200.07 lat (usec): min=1603, max=25809, avg=4710.43, stdev=3200.16 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2089], 10.00th=[ 2212], 20.00th=[ 2474], | 30.00th=[ 2737], 40.00th=[ 3064], 50.00th=[ 3359], 60.00th=[ 3949], | 70.00th=[ 4883], 80.00th=[ 6390], 90.00th=[ 9634], 95.00th=[11994], | 99.00th=[15270], 99.50th=[17695], 99.90th=[20579], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 176, max= 656, per=25.23%, avg=341.60, stdev=118.96, samples=35 iops : min= 44, max= 164, avg=85.31, stdev=29.74, samples=35 write: IOPS=88, BW=356KiB/s (364kB/s)(6260KiB/17595msec) clat (msec): min=2, max=187, avg= 6.68, stdev= 5.81 lat (msec): min=2, max=187, avg= 6.68, stdev= 5.81 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 20], 99.50th=[ 23], 99.90th=[ 33], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 199, max= 752, per=24.99%, avg=354.40, stdev=124.68, samples=35 iops : min= 49, max= 188, avg=88.51, stdev=31.19, samples=35 lat (msec) : 2=1.43%, 4=40.82%, 10=45.44%, 20=11.75%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=11.50%, ctx=3761, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17651: Fri May 10 07:29:33 2024 read: IOPS=85, BW=341KiB/s (350kB/s)(6052KiB/17729msec) clat (usec): min=1668, max=22532, avg=4802.63, stdev=3347.02 lat (usec): min=1669, max=22533, avg=4803.33, stdev=3347.04 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2073], 10.00th=[ 2212], 20.00th=[ 2507], | 30.00th=[ 2769], 40.00th=[ 3064], 50.00th=[ 3425], 60.00th=[ 4047], | 70.00th=[ 5014], 80.00th=[ 6521], 90.00th=[10028], 95.00th=[12256], | 99.00th=[16581], 99.50th=[18744], 99.90th=[21890], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 152, max= 712, per=25.03%, avg=338.86, stdev=126.70, samples=35 iops : min= 38, max= 178, avg=84.63, stdev=31.67, samples=35 write: IOPS=87, BW=352KiB/s (360kB/s)(6236KiB/17729msec) clat (msec): min=2, max=187, avg= 6.68, stdev= 5.85 lat (msec): min=2, max=187, avg= 6.68, stdev= 5.85 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 20], 99.50th=[ 24], 99.90th=[ 34], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 176, max= 672, per=24.62%, avg=349.11, stdev=121.54, samples=35 iops : min= 44, max= 168, avg=87.20, stdev=30.41, samples=35 lat (msec) : 2=1.63%, 4=39.65%, 10=46.03%, 20=12.01%, 50=0.65% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.46%, ctx=3701, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1354KiB/s (1387kB/s), 338KiB/s-343KiB/s (346kB/s-351kB/s), io=23.4MiB (24.6MB), run=17575-17729msec WRITE: bw=1418KiB/s (1452kB/s), 352KiB/s-361KiB/s (360kB/s-370kB/s), io=24.6MiB (25.7MB), run=17575-17729msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17678: Fri May 10 07:29:40 2024 read: IOPS=57, BW=919KiB/s (941kB/s)(5920KiB/6444msec) clat (usec): min=60, max=35588, avg=11414.71, stdev=3709.85 lat (usec): min=60, max=35589, avg=11415.36, stdev=3709.83 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 7242], 10.00th=[ 8029], 20.00th=[ 9110], | 30.00th=[ 9896], 40.00th=[10552], 50.00th=[11207], 60.00th=[11994], | 70.00th=[12780], 80.00th=[13698], 90.00th=[15270], 95.00th=[16909], | 99.00th=[23725], 99.50th=[27132], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 800, max= 992, per=24.88%, avg=912.00, stdev=70.24, samples=12 iops : min= 50, max= 62, avg=57.00, stdev= 4.39, samples=12 write: IOPS=61, BW=988KiB/s (1012kB/s)(6368KiB/6444msec) clat (usec): min=655, max=51728, avg=5555.60, stdev=6083.06 lat (usec): min=656, max=51728, avg=5556.68, stdev=6083.04 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 807], 10.00th=[ 906], 20.00th=[ 1205], | 30.00th=[ 2376], 40.00th=[ 2835], 50.00th=[ 3392], 60.00th=[ 4817], | 70.00th=[ 5997], 80.00th=[ 7963], 90.00th=[12780], 95.00th=[17433], | 99.00th=[28705], 99.50th=[47449], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 768, max= 1376, per=25.66%, avg=989.33, stdev=183.80, samples=12 iops : min= 48, max= 86, avg=61.83, stdev=11.49, samples=12 lat (usec) : 100=1.04%, 250=0.26%, 750=0.26%, 1000=7.81% lat (msec) : 2=4.95%, 4=15.23%, 10=29.30%, 20=38.41%, 50=2.60% lat (msec) : 100=0.13% cpu : usr=0.12%, sys=19.68%, ctx=2855, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17679: Fri May 10 07:29:40 2024 read: IOPS=56, BW=896KiB/s (918kB/s)(5744KiB/6409msec) clat (usec): min=61, max=35588, avg=11725.92, stdev=3966.26 lat (usec): min=62, max=35588, avg=11726.53, stdev=3966.27 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 7570], 10.00th=[ 8356], 20.00th=[ 9110], | 30.00th=[ 9896], 40.00th=[10552], 50.00th=[11338], 60.00th=[12125], | 70.00th=[12911], 80.00th=[13829], 90.00th=[15926], 95.00th=[18220], | 99.00th=[24249], 99.50th=[33817], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 672, max= 1184, per=24.45%, avg=896.00, stdev=146.96, samples=12 iops : min= 42, max= 74, avg=56.00, stdev= 9.18, samples=12 write: IOPS=63, BW=1021KiB/s (1046kB/s)(6544KiB/6409msec) clat (usec): min=713, max=53174, avg=5359.50, stdev=5939.82 lat (usec): min=714, max=53175, avg=5360.50, stdev=5939.83 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 840], 10.00th=[ 930], 20.00th=[ 1500], | 30.00th=[ 2573], 40.00th=[ 2868], 50.00th=[ 3359], 60.00th=[ 4293], | 70.00th=[ 5669], 80.00th=[ 7373], 90.00th=[11994], 95.00th=[16188], | 99.00th=[30016], 99.50th=[37487], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 736, max= 1440, per=26.35%, avg=1016.00, stdev=212.32, samples=12 iops : min= 46, max= 90, avg=63.50, stdev=13.27, samples=12 lat (usec) : 100=0.65%, 250=0.39%, 750=0.13%, 1000=6.77% lat (msec) : 2=4.82%, 4=19.01%, 10=28.65%, 20=37.37%, 50=2.08% lat (msec) : 100=0.13% cpu : usr=0.08%, sys=19.62%, ctx=2803, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17680: Fri May 10 07:29:40 2024 read: IOPS=59, BW=958KiB/s (981kB/s)(6256KiB/6531msec) clat (usec): min=61, max=29840, avg=11436.42, stdev=3908.66 lat (usec): min=61, max=29840, avg=11437.03, stdev=3908.67 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 6849], 10.00th=[ 7898], 20.00th=[ 9110], | 30.00th=[ 9765], 40.00th=[10552], 50.00th=[11076], 60.00th=[11863], | 70.00th=[12518], 80.00th=[13698], 90.00th=[15664], 95.00th=[17695], | 99.00th=[23987], 99.50th=[26608], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 768, max= 1088, per=26.12%, avg=957.38, stdev=90.88, samples=13 iops : min= 48, max= 68, avg=59.77, stdev= 5.66, samples=13 write: IOPS=57, BW=924KiB/s (946kB/s)(6032KiB/6531msec) clat (usec): min=678, max=49909, avg=5442.01, stdev=6270.66 lat (usec): min=679, max=49910, avg=5443.12, stdev=6270.61 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 873], 10.00th=[ 963], 20.00th=[ 1401], | 30.00th=[ 2442], 40.00th=[ 2835], 50.00th=[ 3490], 60.00th=[ 4424], | 70.00th=[ 5604], 80.00th=[ 7767], 90.00th=[11731], 95.00th=[15270], | 99.00th=[41157], 99.50th=[46400], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 544, max= 1216, per=23.62%, avg=910.62, stdev=227.22, samples=13 iops : min= 34, max= 76, avg=56.85, stdev=14.19, samples=13 lat (usec) : 100=0.78%, 250=0.39%, 500=0.26%, 750=0.26%, 1000=5.08% lat (msec) : 2=6.38%, 4=16.41%, 10=29.69%, 20=38.02%, 50=2.73% cpu : usr=0.12%, sys=19.53%, ctx=2878, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17681: Fri May 10 07:29:40 2024 read: IOPS=57, BW=923KiB/s (945kB/s)(6032KiB/6535msec) clat (usec): min=60, max=37176, avg=11802.54, stdev=4692.98 lat (usec): min=60, max=37177, avg=11803.14, stdev=4692.98 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 6915], 10.00th=[ 8225], 20.00th=[ 9241], | 30.00th=[ 9896], 40.00th=[10421], 50.00th=[11076], 60.00th=[11731], | 70.00th=[12387], 80.00th=[13304], 90.00th=[15926], 95.00th=[19792], | 99.00th=[31327], 99.50th=[33162], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 704, max= 1024, per=25.05%, avg=917.92, stdev=108.58, samples=13 iops : min= 44, max= 64, avg=57.31, stdev= 6.73, samples=13 write: IOPS=59, BW=957KiB/s (980kB/s)(6256KiB/6535msec) clat (usec): min=714, max=47967, avg=5312.95, stdev=5770.58 lat (usec): min=715, max=47968, avg=5313.96, stdev=5770.58 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 832], 10.00th=[ 938], 20.00th=[ 1418], | 30.00th=[ 2409], 40.00th=[ 2933], 50.00th=[ 3458], 60.00th=[ 4621], | 70.00th=[ 5604], 80.00th=[ 7439], 90.00th=[11731], 95.00th=[15401], | 99.00th=[33424], 99.50th=[42730], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 704, max= 1472, per=24.95%, avg=962.23, stdev=210.77, samples=13 iops : min= 44, max= 92, avg=60.08, stdev=13.16, samples=13 lat (usec) : 100=0.91%, 250=0.26%, 750=0.13%, 1000=6.38% lat (msec) : 2=4.69%, 4=17.32%, 10=29.95%, 20=36.98%, 50=3.39% cpu : usr=0.14%, sys=20.71%, ctx=2889, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3665KiB/s (3753kB/s), 896KiB/s-958KiB/s (918kB/s-981kB/s), io=23.4MiB (24.5MB), run=6409-6535msec WRITE: bw=3856KiB/s (3949kB/s), 924KiB/s-1021KiB/s (946kB/s-1046kB/s), io=24.6MiB (25.8MB), run=6409-6535msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17682: Fri May 10 07:29:40 2024 read: IOPS=57, BW=919KiB/s (941kB/s)(5920KiB/6444msec) clat (usec): min=1792, max=53441, avg=6675.23, stdev=6339.53 lat (usec): min=1792, max=53442, avg=6675.84, stdev=6339.54 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2245], 10.00th=[ 2409], 20.00th=[ 2802], | 30.00th=[ 3032], 40.00th=[ 3458], 50.00th=[ 3884], 60.00th=[ 5080], | 70.00th=[ 7832], 80.00th=[10028], 90.00th=[12518], 95.00th=[17433], | 99.00th=[30802], 99.50th=[49021], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 704, max= 1085, per=24.90%, avg=909.08, stdev=106.52, samples=12 iops : min= 44, max= 67, avg=56.75, stdev= 6.54, samples=12 write: IOPS=61, BW=988KiB/s (1012kB/s)(6368KiB/6444msec) clat (usec): min=2834, max=82515, avg=9967.22, stdev=7151.66 lat (usec): min=2835, max=82516, avg=9968.27, stdev=7151.67 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3425], 10.00th=[ 3785], 20.00th=[ 4752], | 30.00th=[ 6194], 40.00th=[ 7111], 50.00th=[ 8586], 60.00th=[ 9765], | 70.00th=[11207], 80.00th=[13435], 90.00th=[17433], 95.00th=[21103], | 99.00th=[39060], 99.50th=[47449], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 768, max= 1216, per=25.68%, avg=986.50, stdev=163.78, samples=12 iops : min= 48, max= 76, avg=61.58, stdev=10.30, samples=12 lat (msec) : 2=0.39%, 4=30.47%, 10=39.32%, 20=24.35%, 50=5.21% lat (msec) : 100=0.26% cpu : usr=0.08%, sys=9.61%, ctx=1183, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17683: Fri May 10 07:29:40 2024 read: IOPS=55, BW=890KiB/s (911kB/s)(5744KiB/6457msec) clat (usec): min=1811, max=53160, avg=7150.95, stdev=6769.76 lat (usec): min=1811, max=53160, avg=7151.55, stdev=6769.76 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2147], 10.00th=[ 2376], 20.00th=[ 2671], | 30.00th=[ 2933], 40.00th=[ 3556], 50.00th=[ 4178], 60.00th=[ 6521], | 70.00th=[ 8586], 80.00th=[10814], 90.00th=[14877], 95.00th=[18482], | 99.00th=[40633], 99.50th=[48497], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 704, max= 1149, per=24.46%, avg=892.92, stdev=143.52, samples=12 iops : min= 44, max= 71, avg=55.67, stdev= 8.82, samples=12 write: IOPS=63, BW=1013KiB/s (1038kB/s)(6544KiB/6457msec) clat (usec): min=2616, max=66245, avg=9494.39, stdev=6071.53 lat (usec): min=2617, max=66246, avg=9495.41, stdev=6071.52 clat percentiles (usec): | 1.00th=[ 3097], 5.00th=[ 3359], 10.00th=[ 3752], 20.00th=[ 4752], | 30.00th=[ 6194], 40.00th=[ 6980], 50.00th=[ 7898], 60.00th=[ 9372], | 70.00th=[10945], 80.00th=[12911], 90.00th=[16581], 95.00th=[20317], | 99.00th=[26084], 99.50th=[26870], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 736, max= 1280, per=26.37%, avg=1013.00, stdev=175.03, samples=12 iops : min= 46, max= 80, avg=63.17, stdev=10.99, samples=12 lat (msec) : 2=0.39%, 4=28.65%, 10=41.80%, 20=24.48%, 50=4.43% lat (msec) : 100=0.26% cpu : usr=0.06%, sys=9.56%, ctx=1182, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17684: Fri May 10 07:29:40 2024 read: IOPS=59, BW=954KiB/s (977kB/s)(6256KiB/6560msec) clat (usec): min=1871, max=36170, avg=6970.17, stdev=5497.09 lat (usec): min=1871, max=36171, avg=6970.90, stdev=5497.04 clat percentiles (usec): | 1.00th=[ 2057], 5.00th=[ 2278], 10.00th=[ 2409], 20.00th=[ 2704], | 30.00th=[ 3032], 40.00th=[ 3523], 50.00th=[ 4948], 60.00th=[ 6521], | 70.00th=[ 8848], 80.00th=[10683], 90.00th=[13960], 95.00th=[18220], | 99.00th=[27395], 99.50th=[30016], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 736, max= 1184, per=26.15%, avg=954.85, stdev=114.30, samples=13 iops : min= 46, max= 74, avg=59.62, stdev= 7.09, samples=13 write: IOPS=57, BW=920KiB/s (942kB/s)(6032KiB/6560msec) clat (usec): min=2793, max=50538, avg=10154.13, stdev=7334.15 lat (usec): min=2794, max=50540, avg=10155.20, stdev=7334.12 clat percentiles (usec): | 1.00th=[ 3163], 5.00th=[ 3621], 10.00th=[ 4080], 20.00th=[ 5276], | 30.00th=[ 6325], 40.00th=[ 7177], 50.00th=[ 8094], 60.00th=[ 9110], | 70.00th=[10814], 80.00th=[14091], 90.00th=[17433], 95.00th=[21627], | 99.00th=[49021], 99.50th=[50070], 99.90th=[50594], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 576, max= 1184, per=23.58%, avg=905.62, stdev=203.17, samples=13 iops : min= 36, max= 74, avg=56.54, stdev=12.64, samples=13 lat (msec) : 2=0.26%, 4=26.04%, 10=44.14%, 20=25.00%, 50=4.30% lat (msec) : 100=0.26% cpu : usr=0.11%, sys=9.32%, ctx=1209, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17685: Fri May 10 07:29:40 2024 read: IOPS=57, BW=921KiB/s (943kB/s)(6032KiB/6550msec) clat (usec): min=1954, max=47666, avg=7007.18, stdev=6131.88 lat (usec): min=1955, max=47666, avg=7007.85, stdev=6131.88 clat percentiles (usec): | 1.00th=[ 2089], 5.00th=[ 2245], 10.00th=[ 2409], 20.00th=[ 2737], | 30.00th=[ 3163], 40.00th=[ 3621], 50.00th=[ 4359], 60.00th=[ 6128], | 70.00th=[ 8455], 80.00th=[10421], 90.00th=[14484], 95.00th=[18220], | 99.00th=[30802], 99.50th=[46400], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 704, max= 1120, per=25.01%, avg=913.08, stdev=123.57, samples=13 iops : min= 44, max= 70, avg=57.00, stdev= 7.68, samples=13 write: IOPS=59, BW=955KiB/s (978kB/s)(6256KiB/6550msec) clat (usec): min=2735, max=57094, avg=9978.38, stdev=6944.37 lat (usec): min=2736, max=57095, avg=9979.47, stdev=6944.36 clat percentiles (usec): | 1.00th=[ 2999], 5.00th=[ 3490], 10.00th=[ 3916], 20.00th=[ 5014], | 30.00th=[ 5997], 40.00th=[ 7046], 50.00th=[ 8094], 60.00th=[ 9634], | 70.00th=[11338], 80.00th=[13829], 90.00th=[17171], 95.00th=[20841], | 99.00th=[47973], 99.50th=[49546], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 736, max= 1248, per=24.86%, avg=954.92, stdev=151.21, samples=13 iops : min= 46, max= 78, avg=59.62, stdev= 9.48, samples=13 lat (msec) : 2=0.13%, 4=27.99%, 10=41.15%, 20=26.43%, 50=4.17% lat (msec) : 100=0.13% cpu : usr=0.14%, sys=9.34%, ctx=1200, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3651KiB/s (3739kB/s), 890KiB/s-954KiB/s (911kB/s-977kB/s), io=23.4MiB (24.5MB), run=6444-6560msec WRITE: bw=3841KiB/s (3934kB/s), 920KiB/s-1013KiB/s (942kB/s-1038kB/s), io=24.6MiB (25.8MB), run=6444-6560msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17695: Fri May 10 07:29:46 2024 read: IOPS=1, BW=1583KiB/s (1621kB/s)(7168KiB/4527msec) clat (msec): min=40, max=710, avg=536.24, stdev=234.55 lat (msec): min=40, max=710, avg=536.24, stdev=234.55 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 464], | 30.00th=[ 558], 40.00th=[ 558], 50.00th=[ 625], 60.00th=[ 659], | 70.00th=[ 659], 80.00th=[ 693], 90.00th=[ 709], 95.00th=[ 709], | 99.00th=[ 709], 99.50th=[ 709], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 1673, max= 2048, per=41.10%, avg=1859.17, stdev=203.21, samples=6 iops : min= 1, max= 2, avg= 1.17, stdev= 0.41, samples=6 write: IOPS=1, BW=1131KiB/s (1158kB/s)(5120KiB/4527msec) clat (msec): min=47, max=356, avg=154.45, stdev=139.34 lat (msec): min=47, max=356, avg=154.53, stdev=139.33 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 48], | 30.00th=[ 55], 40.00th=[ 55], 50.00th=[ 69], 60.00th=[ 69], | 70.00th=[ 245], 80.00th=[ 245], 90.00th=[ 355], 95.00th=[ 355], | 99.00th=[ 355], 99.50th=[ 355], 99.90th=[ 355], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 1673, max= 5019, per=46.00%, avg=2913.33, stdev=1833.17, samples=3 iops : min= 1, max= 4, avg= 2.33, stdev= 1.53, samples=3 lat (msec) : 50=16.67%, 100=16.67%, 250=8.33%, 500=16.67%, 750=41.67% cpu : usr=0.00%, sys=30.80%, ctx=1391, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17696: Fri May 10 07:29:46 2024 read: IOPS=0, BW=879KiB/s (900kB/s)(2048KiB/2330msec) clat (msec): min=47, max=490, avg=269.15, stdev=313.59 lat (msec): min=47, max=490, avg=269.15, stdev=313.59 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 48], 50.00th=[ 48], 60.00th=[ 493], | 70.00th=[ 493], 80.00th=[ 493], 90.00th=[ 493], 95.00th=[ 493], | 99.00th=[ 493], 99.50th=[ 493], 99.90th=[ 493], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 2043, max= 2043, per=45.17%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4395KiB/s (4500kB/s)(10.0MiB/2330msec) clat (msec): min=28, max=724, avg=179.03, stdev=270.30 lat (msec): min=28, max=724, avg=179.09, stdev=270.29 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 31], | 30.00th=[ 34], 40.00th=[ 40], 50.00th=[ 41], 60.00th=[ 62], | 70.00th=[ 107], 80.00th=[ 107], 90.00th=[ 651], 95.00th=[ 726], | 99.00th=[ 726], 99.50th=[ 726], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 4096, max= 6144, per=80.85%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 50=50.00%, 100=16.67%, 250=8.33%, 500=8.33%, 750=16.67% cpu : usr=0.00%, sys=15.59%, ctx=344, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17697: Fri May 10 07:29:46 2024 read: IOPS=2, BW=2977KiB/s (3048kB/s)(6144KiB/2064msec) clat (usec): min=937, max=553034, avg=189446.78, stdev=264687.99 lat (usec): min=938, max=553034, avg=189447.94, stdev=264688.16 clat percentiles (usec): | 1.00th=[ 938], 5.00th=[ 938], 10.00th=[ 938], 20.00th=[ 2999], | 30.00th=[ 2999], 40.00th=[ 20055], 50.00th=[ 20055], 60.00th=[ 53216], | 70.00th=[505414], 80.00th=[505414], 90.00th=[549454], 95.00th=[549454], | 99.00th=[549454], 99.50th=[549454], 99.90th=[549454], 99.95th=[549454], | 99.99th=[549454] bw ( KiB/s): min= 4096, max= 4096, per=90.56%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2977KiB/s (3048kB/s)(6144KiB/2064msec) clat (msec): min=22, max=727, avg=154.20, stdev=281.44 lat (msec): min=22, max=727, avg=154.25, stdev=281.45 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 51], | 70.00th=[ 62], 80.00th=[ 62], 90.00th=[ 726], 95.00th=[ 726], | 99.00th=[ 726], 99.50th=[ 726], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 2043, max= 4096, per=53.87%, avg=3411.67, stdev=1185.30, samples=3 iops : min= 1, max= 4, avg= 3.00, stdev= 1.73, samples=3 lat (usec) : 1000=8.33% lat (msec) : 4=8.33%, 50=33.33%, 100=25.00%, 750=25.00% cpu : usr=0.05%, sys=16.72%, ctx=613, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17698: Fri May 10 07:29:46 2024 read: IOPS=2, BW=2281KiB/s (2335kB/s)(5120KiB/2245msec) clat (usec): min=345, max=563421, avg=222858.84, stdev=280452.72 lat (usec): min=346, max=563422, avg=222860.02, stdev=280452.64 clat percentiles (usec): | 1.00th=[ 347], 5.00th=[ 347], 10.00th=[ 347], 20.00th=[ 347], | 30.00th=[ 10552], 40.00th=[ 10552], 50.00th=[ 46924], 60.00th=[ 46924], | 70.00th=[492831], 80.00th=[492831], 90.00th=[566232], 95.00th=[566232], | 99.00th=[566232], 99.50th=[566232], 99.90th=[566232], 99.95th=[566232], | 99.99th=[566232] bw ( KiB/s): min= 2048, max= 4087, per=60.31%, avg=2727.67, stdev=1177.22, samples=3 iops : min= 2, max= 3, avg= 2.33, stdev= 0.58, samples=3 write: IOPS=3, BW=3193KiB/s (3270kB/s)(7168KiB/2245msec) clat (msec): min=28, max=739, avg=161.30, stdev=262.43 lat (msec): min=28, max=739, avg=161.36, stdev=262.43 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 31], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 43], 60.00th=[ 44], | 70.00th=[ 44], 80.00th=[ 203], 90.00th=[ 743], 95.00th=[ 743], | 99.00th=[ 743], 99.50th=[ 743], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 2048, max= 6144, per=64.68%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (usec) : 500=8.33% lat (msec) : 20=8.33%, 50=50.00%, 250=8.33%, 500=8.33%, 750=16.67% cpu : usr=0.00%, sys=16.22%, ctx=603, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4524KiB/s (4633kB/s), 879KiB/s-2977KiB/s (900kB/s-3048kB/s), io=20.0MiB (20.0MB), run=2064-4527msec WRITE: bw=6334KiB/s (6486kB/s), 1131KiB/s-4395KiB/s (1158kB/s-4500kB/s), io=28.0MiB (29.4MB), run=2064-4527msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17700: Fri May 10 07:29:46 2024 read: IOPS=1, BW=1567KiB/s (1605kB/s)(7168KiB/4574msec) clat (msec): min=16, max=685, avg=342.12, stdev=280.62 lat (msec): min=16, max=685, avg=342.12, stdev=280.62 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 18], | 30.00th=[ 174], 40.00th=[ 174], 50.00th=[ 388], 60.00th=[ 456], | 70.00th=[ 456], 80.00th=[ 659], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 1686, max= 4087, per=53.10%, avg=2377.50, stdev=1152.27, samples=4 iops : min= 1, max= 3, avg= 1.75, stdev= 0.96, samples=4 write: IOPS=1, BW=1119KiB/s (1146kB/s)(5120KiB/4574msec) clat (msec): min=94, max=984, avg=434.43, stdev=333.16 lat (msec): min=94, max=984, avg=434.49, stdev=333.18 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 95], 10.00th=[ 95], 20.00th=[ 95], | 30.00th=[ 279], 40.00th=[ 279], 50.00th=[ 388], 60.00th=[ 388], | 70.00th=[ 426], 80.00th=[ 426], 90.00th=[ 986], 95.00th=[ 986], | 99.00th=[ 986], 99.50th=[ 986], 99.90th=[ 986], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 1686, max= 2048, per=28.84%, avg=1807.67, stdev=208.14, samples=3 iops : min= 1, max= 2, avg= 1.33, stdev= 0.58, samples=3 lat (msec) : 20=16.67%, 100=8.33%, 250=8.33%, 500=41.67%, 750=16.67% lat (msec) : 1000=8.33% cpu : usr=0.04%, sys=3.89%, ctx=49, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17701: Fri May 10 07:29:46 2024 read: IOPS=0, BW=870KiB/s (891kB/s)(2048KiB/2355msec) clat (msec): min=56, max=487, avg=271.63, stdev=304.61 lat (msec): min=56, max=487, avg=271.63, stdev=304.61 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 57], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 57], 60.00th=[ 489], | 70.00th=[ 489], 80.00th=[ 489], 90.00th=[ 489], 95.00th=[ 489], | 99.00th=[ 489], 99.50th=[ 489], 99.90th=[ 489], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 2043, max= 2043, per=45.63%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4348KiB/s (4453kB/s)(10.0MiB/2355msec) clat (msec): min=39, max=707, avg=180.76, stdev=245.66 lat (msec): min=39, max=707, avg=180.81, stdev=245.65 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 50], 60.00th=[ 91], | 70.00th=[ 112], 80.00th=[ 112], 90.00th=[ 575], 95.00th=[ 709], | 99.00th=[ 709], 99.50th=[ 709], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 4087, max= 6144, per=81.61%, avg=5115.50, stdev=1454.52, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 lat (msec) : 50=41.67%, 100=16.67%, 250=16.67%, 500=8.33%, 750=16.67% cpu : usr=0.04%, sys=7.26%, ctx=72, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17702: Fri May 10 07:29:46 2024 read: IOPS=2, BW=2802KiB/s (2869kB/s)(6144KiB/2193msec) clat (msec): min=19, max=566, avg=128.75, stdev=215.21 lat (msec): min=19, max=566, avg=128.75, stdev=215.21 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 45], 50.00th=[ 45], 60.00th=[ 55], | 70.00th=[ 59], 80.00th=[ 59], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 4096, max= 4096, per=91.49%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2802KiB/s (2869kB/s)(6144KiB/2193msec) clat (msec): min=17, max=727, avg=235.31, stdev=311.64 lat (msec): min=17, max=727, avg=235.35, stdev=311.64 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 42], 50.00th=[ 42], 60.00th=[ 69], | 70.00th=[ 531], 80.00th=[ 531], 90.00th=[ 726], 95.00th=[ 726], | 99.00th=[ 726], 99.50th=[ 726], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 4096, max= 4096, per=65.35%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 lat (msec) : 20=16.67%, 50=33.33%, 100=25.00%, 750=25.00% cpu : usr=0.00%, sys=6.02%, ctx=47, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17703: Fri May 10 07:29:46 2024 read: IOPS=2, BW=2236KiB/s (2289kB/s)(5120KiB/2290msec) clat (msec): min=18, max=459, avg=202.61, stdev=229.69 lat (msec): min=18, max=459, avg=202.61, stdev=229.69 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 47], 60.00th=[ 47], | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 460], 95.00th=[ 460], | 99.00th=[ 460], 99.50th=[ 460], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 2048, max= 4096, per=60.99%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 write: IOPS=3, BW=3130KiB/s (3205kB/s)(7168KiB/2290msec) clat (msec): min=35, max=719, avg=181.41, stdev=245.14 lat (msec): min=35, max=719, avg=181.47, stdev=245.15 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 70], | 30.00th=[ 72], 40.00th=[ 72], 50.00th=[ 73], 60.00th=[ 74], | 70.00th=[ 74], 80.00th=[ 228], 90.00th=[ 718], 95.00th=[ 718], | 99.00th=[ 718], 99.50th=[ 718], 99.90th=[ 718], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 2048, max= 6144, per=65.35%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=8.33%, 50=25.00%, 100=33.33%, 250=8.33%, 500=16.67% lat (msec) : 750=8.33% cpu : usr=0.04%, sys=5.68%, ctx=65, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4477KiB/s (4585kB/s), 870KiB/s-2802KiB/s (891kB/s-2869kB/s), io=20.0MiB (20.0MB), run=2193-4574msec WRITE: bw=6268KiB/s (6419kB/s), 1119KiB/s-4348KiB/s (1146kB/s-4453kB/s), io=28.0MiB (29.4MB), run=2193-4574msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17712: Fri May 10 07:29:53 2024 read: IOPS=0, BW=2257KiB/s (2311kB/s)(12.0MiB/5444msec) clat (msec): min=187, max=3519, avg=1811.81, stdev=1667.91 lat (msec): min=187, max=3519, avg=1811.82, stdev=1667.91 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 188], 10.00th=[ 188], 20.00th=[ 188], | 30.00th=[ 188], 40.00th=[ 1737], 50.00th=[ 1737], 60.00th=[ 1737], | 70.00th=[ 3507], 80.00th=[ 3507], 90.00th=[ 3507], 95.00th=[ 3507], | 99.00th=[ 3507], 99.50th=[ 3507], 99.90th=[ 3507], 99.95th=[ 3507], | 99.99th=[ 3507] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.02%, sys=31.31%, ctx=1628, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17713: Fri May 10 07:29:53 2024 read: IOPS=0, BW=655KiB/s (670kB/s)(4096KiB/6257msec) clat (nsec): min=218690k, max=218690k, avg=218689526.00, stdev= 0.00 lat (nsec): min=218691k, max=218691k, avg=218691050.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 220], 10.00th=[ 220], 20.00th=[ 220], | 30.00th=[ 220], 40.00th=[ 220], 50.00th=[ 220], 60.00th=[ 220], | 70.00th=[ 220], 80.00th=[ 220], 90.00th=[ 220], 95.00th=[ 220], | 99.00th=[ 220], 99.50th=[ 220], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1309KiB/s (1341kB/s)(8192KiB/6257msec) clat (msec): min=490, max=5544, avg=3017.39, stdev=3573.75 lat (msec): min=490, max=5544, avg=3017.59, stdev=3573.65 clat percentiles (msec): | 1.00th=[ 489], 5.00th=[ 489], 10.00th=[ 489], 20.00th=[ 489], | 30.00th=[ 489], 40.00th=[ 489], 50.00th=[ 489], 60.00th=[ 5537], | 70.00th=[ 5537], 80.00th=[ 5537], 90.00th=[ 5537], 95.00th=[ 5537], | 99.00th=[ 5537], 99.50th=[ 5537], 99.90th=[ 5537], 99.95th=[ 5537], | 99.99th=[ 5537] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=4.99%, ctx=105, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17715: Fri May 10 07:29:53 2024 write: IOPS=0, BW=1978KiB/s (2026kB/s)(12.0MiB/6211msec) clat (msec): min=356, max=4615, avg=2069.74, stdev=2248.26 lat (msec): min=357, max=4615, avg=2070.00, stdev=2248.29 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 359], | 30.00th=[ 359], 40.00th=[ 1234], 50.00th=[ 1234], 60.00th=[ 1234], | 70.00th=[ 4597], 80.00th=[ 4597], 90.00th=[ 4597], 95.00th=[ 4597], | 99.00th=[ 4597], 99.50th=[ 4597], 99.90th=[ 4597], 99.95th=[ 4597], | 99.99th=[ 4597] bw ( KiB/s): min= 7953, max= 8192, per=100.00%, avg=8072.50, stdev=169.00, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33% cpu : usr=0.02%, sys=5.14%, ctx=52, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17717: Fri May 10 07:29:53 2024 write: IOPS=0, BW=2096KiB/s (2147kB/s)(12.0MiB/5862msec) clat (msec): min=100, max=4551, avg=1952.88, stdev=2317.28 lat (msec): min=101, max=4551, avg=1953.16, stdev=2317.31 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 102], 10.00th=[ 102], 20.00th=[ 102], | 30.00th=[ 102], 40.00th=[ 1200], 50.00th=[ 1200], 60.00th=[ 1200], | 70.00th=[ 4530], 80.00th=[ 4530], 90.00th=[ 4530], 95.00th=[ 4530], | 99.00th=[ 4530], 99.50th=[ 4530], 99.90th=[ 4530], 99.95th=[ 4530], | 99.99th=[ 4530] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=5.31%, ctx=71, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2619KiB/s (2681kB/s), 655KiB/s-2257KiB/s (670kB/s-2311kB/s), io=16.0MiB (16.8MB), run=5444-6257msec WRITE: bw=5237KiB/s (5363kB/s), 1309KiB/s-2096KiB/s (1341kB/s-2147kB/s), io=32.0MiB (33.6MB), run=5862-6257msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17714: Fri May 10 07:29:53 2024 read: IOPS=0, BW=1935KiB/s (1981kB/s)(12.0MiB/6352msec) clat (msec): min=304, max=4714, avg=2117.12, stdev=2307.39 lat (msec): min=304, max=4714, avg=2117.12, stdev=2307.39 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 305], 10.00th=[ 305], 20.00th=[ 305], | 30.00th=[ 305], 40.00th=[ 1334], 50.00th=[ 1334], 60.00th=[ 1334], | 70.00th=[ 4732], 80.00th=[ 4732], 90.00th=[ 4732], 95.00th=[ 4732], | 99.00th=[ 4732], 99.50th=[ 4732], 99.90th=[ 4732], 99.95th=[ 4732], | 99.99th=[ 4732] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=2.08%, ctx=48, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17716: Fri May 10 07:29:53 2024 read: IOPS=0, BW=645KiB/s (661kB/s)(4096KiB/6349msec) clat (nsec): min=1328.4M, max=1328.4M, avg=1328370763.00, stdev= 0.00 lat (nsec): min=1328.4M, max=1328.4M, avg=1328373216.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1334], 5.00th=[ 1334], 10.00th=[ 1334], 20.00th=[ 1334], | 30.00th=[ 1334], 40.00th=[ 1334], 50.00th=[ 1334], 60.00th=[ 1334], | 70.00th=[ 1334], 80.00th=[ 1334], 90.00th=[ 1334], 95.00th=[ 1334], | 99.00th=[ 1334], 99.50th=[ 1334], 99.90th=[ 1334], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1290KiB/s (1321kB/s)(8192KiB/6349msec) clat (msec): min=321, max=4665, avg=2493.45, stdev=3071.11 lat (msec): min=321, max=4665, avg=2493.65, stdev=3071.18 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 321], 10.00th=[ 321], 20.00th=[ 321], | 30.00th=[ 321], 40.00th=[ 321], 50.00th=[ 321], 60.00th=[ 4665], | 70.00th=[ 4665], 80.00th=[ 4665], 90.00th=[ 4665], 95.00th=[ 4665], | 99.00th=[ 4665], 99.50th=[ 4665], 99.90th=[ 4665], 99.95th=[ 4665], | 99.99th=[ 4665] lat (msec) : 500=33.33% cpu : usr=0.00%, sys=2.79%, ctx=82, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17718: Fri May 10 07:29:53 2024 write: IOPS=0, BW=2012KiB/s (2061kB/s)(12.0MiB/6106msec) clat (msec): min=212, max=3647, avg=2034.55, stdev=1727.24 lat (msec): min=212, max=3648, avg=2034.82, stdev=1727.33 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 213], 10.00th=[ 213], 20.00th=[ 213], | 30.00th=[ 213], 40.00th=[ 2232], 50.00th=[ 2232], 60.00th=[ 2232], | 70.00th=[ 3641], 80.00th=[ 3641], 90.00th=[ 3641], 95.00th=[ 3641], | 99.00th=[ 3641], 99.50th=[ 3641], 99.90th=[ 3641], 99.95th=[ 3641], | 99.99th=[ 3641] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=2.64%, ctx=223, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17719: Fri May 10 07:29:53 2024 write: IOPS=0, BW=1982KiB/s (2029kB/s)(12.0MiB/6201msec) clat (msec): min=87, max=3843, avg=2066.43, stdev=1886.42 lat (msec): min=87, max=3843, avg=2066.67, stdev=1886.32 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 88], 10.00th=[ 88], 20.00th=[ 88], | 30.00th=[ 88], 40.00th=[ 2265], 50.00th=[ 2265], 60.00th=[ 2265], | 70.00th=[ 3842], 80.00th=[ 3842], 90.00th=[ 3842], 95.00th=[ 3842], | 99.00th=[ 3842], 99.50th=[ 3842], 99.90th=[ 3842], 99.95th=[ 3842], | 99.99th=[ 3842] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33% cpu : usr=0.02%, sys=2.32%, ctx=57, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2579KiB/s (2641kB/s), 645KiB/s-1935KiB/s (661kB/s-1981kB/s), io=16.0MiB (16.8MB), run=6349-6352msec WRITE: bw=5161KiB/s (5285kB/s), 1290KiB/s-2012KiB/s (1321kB/s-2061kB/s), io=32.0MiB (33.6MB), run=6106-6349msec