== sanity test 398b: DIO and buffer IO race ============== 17:28:26 (1713216506) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.3893 s, 36.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29191: Mon Apr 15 17:28:41 2024 read: IOPS=125, BW=500KiB/s (512kB/s)(5992KiB/11976msec) clat (usec): min=47, max=159015, avg=3502.75, stdev=5132.66 lat (usec): min=48, max=159016, avg=3503.81, stdev=5132.92 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 64], 10.00th=[ 72], 20.00th=[ 85], | 30.00th=[ 2024], 40.00th=[ 2376], 50.00th=[ 2802], 60.00th=[ 3326], | 70.00th=[ 4228], 80.00th=[ 5473], 90.00th=[ 7635], 95.00th=[ 9241], | 99.00th=[ 12911], 99.50th=[ 17695], 99.90th=[ 40109], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 231, max= 728, per=24.90%, avg=495.91, stdev=106.40, samples=23 iops : min= 57, max= 182, avg=123.91, stdev=26.70, samples=23 write: IOPS=131, BW=526KiB/s (538kB/s)(6296KiB/11976msec) clat (usec): min=446, max=59869, avg=4211.38, stdev=4436.73 lat (usec): min=447, max=59870, avg=4212.04, stdev=4436.73 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 611], | 30.00th=[ 807], 40.00th=[ 1860], 50.00th=[ 3228], 60.00th=[ 4228], | 70.00th=[ 5342], 80.00th=[ 6915], 90.00th=[ 9634], 95.00th=[12518], | 99.00th=[17171], 99.50th=[20317], 99.90th=[48497], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 263, max= 784, per=24.90%, avg=519.22, stdev=111.28, samples=23 iops : min= 65, max= 196, avg=129.74, stdev=27.91, samples=23 lat (usec) : 50=0.55%, 100=10.51%, 250=1.07%, 500=1.14%, 750=13.51% lat (usec) : 1000=2.90% lat (msec) : 2=5.76%, 4=27.18%, 10=31.09%, 20=5.83%, 50=0.39% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.23%, sys=22.99%, ctx=5538, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29192: Mon Apr 15 17:28:41 2024 read: IOPS=123, BW=496KiB/s (508kB/s)(5940KiB/11983msec) clat (usec): min=47, max=201507, avg=3316.05, stdev=6007.11 lat (usec): min=47, max=201508, avg=3316.74, stdev=6007.15 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 63], 10.00th=[ 71], 20.00th=[ 80], | 30.00th=[ 1909], 40.00th=[ 2278], 50.00th=[ 2671], 60.00th=[ 3228], | 70.00th=[ 4113], 80.00th=[ 5276], 90.00th=[ 6915], 95.00th=[ 8848], | 99.00th=[ 11338], 99.50th=[ 13304], 99.90th=[ 55313], 99.95th=[202376], | 99.99th=[202376] bw ( KiB/s): min= 232, max= 680, per=24.74%, avg=492.91, stdev=108.40, samples=23 iops : min= 58, max= 170, avg=123.13, stdev=27.03, samples=23 write: IOPS=132, BW=530KiB/s (542kB/s)(6348KiB/11983msec) clat (usec): min=475, max=41599, avg=4384.84, stdev=4408.74 lat (usec): min=476, max=41599, avg=4385.75, stdev=4408.78 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 611], | 30.00th=[ 848], 40.00th=[ 2180], 50.00th=[ 3392], 60.00th=[ 4293], | 70.00th=[ 5473], 80.00th=[ 7177], 90.00th=[10290], 95.00th=[12911], | 99.00th=[17695], 99.50th=[20841], 99.90th=[38011], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 288, max= 776, per=24.93%, avg=519.74, stdev=94.85, samples=23 iops : min= 72, max= 194, avg=129.83, stdev=23.70, samples=23 lat (usec) : 50=0.46%, 100=11.10%, 250=1.30%, 500=1.33%, 750=12.96% lat (usec) : 1000=2.47% lat (msec) : 2=5.96%, 4=27.25%, 10=30.37%, 20=6.45%, 50=0.29% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.17%, sys=22.97%, ctx=5597, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29193: Mon Apr 15 17:28:41 2024 read: IOPS=125, BW=500KiB/s (512kB/s)(6028KiB/12054msec) clat (usec): min=48, max=65532, avg=3378.37, stdev=3376.63 lat (usec): min=49, max=65533, avg=3379.92, stdev=3376.37 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 63], 10.00th=[ 71], 20.00th=[ 85], | 30.00th=[ 2008], 40.00th=[ 2376], 50.00th=[ 2802], 60.00th=[ 3359], | 70.00th=[ 4146], 80.00th=[ 5538], 90.00th=[ 7439], 95.00th=[ 9110], | 99.00th=[12256], 99.50th=[14615], 99.90th=[20579], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 192, max= 704, per=24.93%, avg=496.58, stdev=107.59, samples=24 iops : min= 48, max= 176, avg=124.21, stdev=26.92, samples=24 write: IOPS=129, BW=519KiB/s (532kB/s)(6260KiB/12054msec) clat (usec): min=404, max=211053, avg=4384.02, stdev=6720.98 lat (usec): min=404, max=211055, avg=4385.17, stdev=6720.80 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 619], | 30.00th=[ 873], 40.00th=[ 2040], 50.00th=[ 3326], 60.00th=[ 4293], | 70.00th=[ 5342], 80.00th=[ 6915], 90.00th=[ 9896], 95.00th=[ 12518], | 99.00th=[ 17957], 99.50th=[ 20317], 99.90th=[ 44303], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 264, max= 696, per=24.70%, avg=514.92, stdev=100.47, samples=24 iops : min= 66, max= 174, avg=128.71, stdev=25.10, samples=24 lat (usec) : 50=0.39%, 100=10.71%, 250=1.37%, 500=1.01%, 750=12.70% lat (usec) : 1000=2.86% lat (msec) : 2=5.76%, 4=27.60%, 10=30.79%, 20=6.45%, 50=0.29% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.22%, sys=23.11%, ctx=5623, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29194: Mon Apr 15 17:28:41 2024 read: IOPS=125, BW=504KiB/s (516kB/s)(6052KiB/12013msec) clat (usec): min=46, max=24253, avg=3237.53, stdev=2940.57 lat (usec): min=46, max=24254, avg=3238.45, stdev=2940.35 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 62], 10.00th=[ 71], 20.00th=[ 79], | 30.00th=[ 1958], 40.00th=[ 2376], 50.00th=[ 2737], 60.00th=[ 3326], | 70.00th=[ 4015], 80.00th=[ 5342], 90.00th=[ 7046], 95.00th=[ 8586], | 99.00th=[12256], 99.50th=[13960], 99.90th=[22938], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 192, max= 696, per=25.23%, avg=502.67, stdev=113.03, samples=24 iops : min= 48, max= 174, avg=125.67, stdev=28.26, samples=24 write: IOPS=129, BW=519KiB/s (532kB/s)(6236KiB/12013msec) clat (usec): min=479, max=201819, avg=4499.40, stdev=6703.80 lat (usec): min=479, max=201820, avg=4500.37, stdev=6703.74 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 523], 10.00th=[ 537], 20.00th=[ 603], | 30.00th=[ 881], 40.00th=[ 2311], 50.00th=[ 3490], 60.00th=[ 4359], | 70.00th=[ 5473], 80.00th=[ 7046], 90.00th=[ 10290], 95.00th=[ 12780], | 99.00th=[ 17695], 99.50th=[ 24249], 99.90th=[ 47449], 99.95th=[202376], | 99.99th=[202376] bw ( KiB/s): min= 200, max= 728, per=24.76%, avg=516.33, stdev=105.84, samples=24 iops : min= 50, max= 182, avg=129.08, stdev=26.46, samples=24 lat (usec) : 50=0.75%, 100=11.10%, 250=1.20%, 500=0.68%, 750=13.28% lat (usec) : 1000=2.38% lat (msec) : 2=5.37%, 4=27.73%, 10=30.79%, 20=6.22%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=23.12%, ctx=5581, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1992KiB/s (2040kB/s), 496KiB/s-504KiB/s (508kB/s-516kB/s), io=23.4MiB (24.6MB), run=11976-12054msec WRITE: bw=2086KiB/s (2136kB/s), 519KiB/s-530KiB/s (532kB/s-542kB/s), io=24.6MiB (25.7MB), run=11976-12054msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29196: Mon Apr 15 17:29:09 2024 read: IOPS=37, BW=150KiB/s (153kB/s)(5992KiB/40064msec) clat (usec): min=1664, max=160475, avg=3499.58, stdev=5158.09 lat (usec): min=1665, max=160478, avg=3500.30, stdev=5158.14 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1844], 10.00th=[ 1893], 20.00th=[ 1975], | 30.00th=[ 2057], 40.00th=[ 2147], 50.00th=[ 2278], 60.00th=[ 2409], | 70.00th=[ 2671], 80.00th=[ 3326], 90.00th=[ 6783], 95.00th=[ 10159], | 99.00th=[ 16057], 99.50th=[ 23200], 99.90th=[ 41157], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 55, max= 272, per=25.31%, avg=149.60, stdev=59.90, samples=80 iops : min= 13, max= 68, avg=37.33, stdev=15.03, samples=80 write: IOPS=39, BW=157KiB/s (161kB/s)(6296KiB/40064msec) clat (usec): min=11563, max=65323, avg=22079.49, stdev=7764.09 lat (usec): min=11564, max=65324, avg=22080.34, stdev=7764.15 clat percentiles (usec): | 1.00th=[12780], 5.00th=[13566], 10.00th=[14222], 20.00th=[15401], | 30.00th=[16909], 40.00th=[19268], 50.00th=[21103], 60.00th=[22676], | 70.00th=[23725], 80.00th=[25560], 90.00th=[32375], 95.00th=[39584], | 99.00th=[48497], 99.50th=[51119], 99.90th=[62653], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 56, max= 208, per=25.35%, avg=156.90, stdev=42.36, samples=80 iops : min= 14, max= 52, avg=39.15, stdev=10.64, samples=80 lat (msec) : 2=10.97%, 4=29.30%, 10=6.05%, 20=24.64%, 50=28.71% lat (msec) : 100=0.29%, 250=0.03% cpu : usr=0.06%, sys=4.50%, ctx=3307, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29197: Mon Apr 15 17:29:09 2024 read: IOPS=36, BW=146KiB/s (150kB/s)(5940KiB/40606msec) clat (usec): min=1604, max=203887, avg=3567.90, stdev=6214.32 lat (usec): min=1605, max=203889, avg=3568.58, stdev=6214.32 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1844], 10.00th=[ 1893], 20.00th=[ 1975], | 30.00th=[ 2057], 40.00th=[ 2147], 50.00th=[ 2278], 60.00th=[ 2442], | 70.00th=[ 2671], 80.00th=[ 3392], 90.00th=[ 7046], 95.00th=[ 10159], | 99.00th=[ 17171], 99.50th=[ 21627], 99.90th=[ 55313], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 16, max= 304, per=24.75%, avg=146.29, stdev=61.63, samples=80 iops : min= 4, max= 76, avg=36.49, stdev=15.40, samples=80 write: IOPS=39, BW=156KiB/s (160kB/s)(6348KiB/40606msec) clat (usec): min=12049, max=74164, avg=22202.75, stdev=7949.15 lat (usec): min=12049, max=74165, avg=22203.56, stdev=7949.12 clat percentiles (usec): | 1.00th=[12649], 5.00th=[13566], 10.00th=[14222], 20.00th=[15401], | 30.00th=[17171], 40.00th=[19530], 50.00th=[21103], 60.00th=[22414], | 70.00th=[23987], 80.00th=[25822], 90.00th=[32637], 95.00th=[39060], | 99.00th=[48497], 99.50th=[58459], 99.90th=[70779], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 56, max= 208, per=25.10%, avg=155.39, stdev=42.24, samples=80 iops : min= 14, max= 52, avg=38.76, stdev=10.57, samples=80 lat (msec) : 2=11.13%, 4=29.20%, 10=5.47%, 20=24.48%, 50=29.20% lat (msec) : 100=0.49%, 250=0.03% cpu : usr=0.10%, sys=4.40%, ctx=3287, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29198: Mon Apr 15 17:29:09 2024 read: IOPS=37, BW=151KiB/s (154kB/s)(6028KiB/40022msec) clat (usec): min=1640, max=41399, avg=3436.17, stdev=3187.92 lat (usec): min=1641, max=41400, avg=3436.84, stdev=3187.95 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1860], 10.00th=[ 1926], 20.00th=[ 1991], | 30.00th=[ 2073], 40.00th=[ 2180], 50.00th=[ 2278], 60.00th=[ 2442], | 70.00th=[ 2737], 80.00th=[ 3490], 90.00th=[ 6783], 95.00th=[ 9896], | 99.00th=[15795], 99.50th=[19006], 99.90th=[38011], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 40, max= 288, per=25.48%, avg=150.59, stdev=58.03, samples=80 iops : min= 10, max= 72, avg=37.56, stdev=14.51, samples=80 write: IOPS=39, BW=156KiB/s (160kB/s)(6260KiB/40022msec) clat (msec): min=12, max=203, avg=22.22, stdev= 9.23 lat (msec): min=12, max=203, avg=22.22, stdev= 9.23 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 21], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 26], 90.00th=[ 33], 95.00th=[ 40], | 99.00th=[ 51], 99.50th=[ 55], 99.90th=[ 70], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 56, max= 208, per=25.23%, avg=156.19, stdev=43.72, samples=80 iops : min= 14, max= 52, avg=38.96, stdev=10.95, samples=80 lat (msec) : 2=10.29%, 4=30.05%, 10=6.28%, 20=24.45%, 50=28.48% lat (msec) : 100=0.42%, 250=0.03% cpu : usr=0.05%, sys=4.48%, ctx=3287, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29199: Mon Apr 15 17:29:09 2024 read: IOPS=37, BW=151KiB/s (154kB/s)(6052KiB/40209msec) clat (usec): min=1654, max=55173, avg=3407.87, stdev=3269.70 lat (usec): min=1654, max=55174, avg=3408.55, stdev=3269.73 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1876], 10.00th=[ 1926], 20.00th=[ 2008], | 30.00th=[ 2089], 40.00th=[ 2180], 50.00th=[ 2278], 60.00th=[ 2442], | 70.00th=[ 2704], 80.00th=[ 3326], 90.00th=[ 6652], 95.00th=[ 9896], | 99.00th=[15139], 99.50th=[21890], 99.90th=[36963], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 16, max= 280, per=25.40%, avg=150.10, stdev=60.02, samples=80 iops : min= 4, max= 70, avg=37.45, stdev=15.04, samples=80 write: IOPS=38, BW=155KiB/s (159kB/s)(6236KiB/40209msec) clat (msec): min=11, max=203, avg=22.44, stdev= 9.08 lat (msec): min=11, max=203, avg=22.44, stdev= 9.08 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 26], 90.00th=[ 34], 95.00th=[ 40], | 99.00th=[ 49], 99.50th=[ 55], 99.90th=[ 65], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 56, max= 207, per=24.98%, avg=154.60, stdev=42.36, samples=80 iops : min= 14, max= 51, avg=38.57, stdev=10.65, samples=80 lat (msec) : 2=9.51%, 4=31.48%, 10=5.92%, 20=23.18%, 50=29.52% lat (msec) : 100=0.36%, 250=0.03% cpu : usr=0.12%, sys=4.46%, ctx=3292, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=591KiB/s (606kB/s), 146KiB/s-151KiB/s (150kB/s-154kB/s), io=23.4MiB (24.6MB), run=40022-40606msec WRITE: bw=619KiB/s (634kB/s), 155KiB/s-157KiB/s (159kB/s-161kB/s), io=24.6MiB (25.7MB), run=40022-40606msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29245: Mon Apr 15 17:29:15 2024 read: IOPS=82, BW=1314KiB/s (1345kB/s)(5920KiB/4506msec) clat (usec): min=53, max=61408, avg=9044.50, stdev=6647.67 lat (usec): min=53, max=61409, avg=9045.25, stdev=6647.63 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 74], 10.00th=[ 78], 20.00th=[ 95], | 30.00th=[ 8160], 40.00th=[ 9503], 50.00th=[10159], 60.00th=[11338], | 70.00th=[11994], 80.00th=[12911], 90.00th=[14222], 95.00th=[16319], | 99.00th=[26084], 99.50th=[46924], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 800, max= 1628, per=24.61%, avg=1308.00, stdev=236.88, samples=9 iops : min= 50, max= 101, avg=81.67, stdev=14.68, samples=9 write: IOPS=88, BW=1413KiB/s (1447kB/s)(6368KiB/4506msec) clat (usec): min=683, max=30003, avg=2895.27, stdev=3354.39 lat (usec): min=684, max=30004, avg=2896.24, stdev=3354.36 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 725], 10.00th=[ 766], 20.00th=[ 816], | 30.00th=[ 857], 40.00th=[ 930], 50.00th=[ 1205], 60.00th=[ 2737], | 70.00th=[ 3523], 80.00th=[ 4752], 90.00th=[ 6390], 95.00th=[ 7898], | 99.00th=[16319], 99.50th=[27395], 99.90th=[30016], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 928, max= 1952, per=25.23%, avg=1411.11, stdev=357.62, samples=9 iops : min= 58, max= 122, avg=88.11, stdev=22.30, samples=9 lat (usec) : 100=10.55%, 250=1.30%, 750=4.30%, 1000=18.75% lat (msec) : 2=5.21%, 4=11.33%, 10=22.14%, 20=24.87%, 50=1.43% lat (msec) : 100=0.13% cpu : usr=0.18%, sys=22.86%, ctx=1949, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29246: Mon Apr 15 17:29:15 2024 read: IOPS=80, BW=1282KiB/s (1313kB/s)(5744KiB/4480msec) clat (usec): min=54, max=22545, avg=7990.80, stdev=5787.26 lat (usec): min=54, max=22545, avg=7991.38, stdev=5787.28 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 69], 10.00th=[ 75], 20.00th=[ 81], | 30.00th=[ 660], 40.00th=[ 8717], 50.00th=[ 9634], 60.00th=[10290], | 70.00th=[11338], 80.00th=[12387], 90.00th=[14484], 95.00th=[16057], | 99.00th=[20579], 99.50th=[22152], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 192, max= 1504, per=22.57%, avg=1199.62, stdev=433.78, samples=8 iops : min= 12, max= 94, avg=74.88, stdev=27.13, samples=8 write: IOPS=91, BW=1461KiB/s (1496kB/s)(6544KiB/4480msec) clat (usec): min=663, max=98951, avg=3913.21, stdev=6970.08 lat (usec): min=664, max=98953, avg=3917.27, stdev=6969.07 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 725], 10.00th=[ 758], 20.00th=[ 824], | 30.00th=[ 873], 40.00th=[ 996], 50.00th=[ 1827], 60.00th=[ 2999], | 70.00th=[ 3720], 80.00th=[ 5407], 90.00th=[ 8160], 95.00th=[11863], | 99.00th=[31065], 99.50th=[34341], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 448, max= 1888, per=24.24%, avg=1355.62, stdev=474.30, samples=8 iops : min= 28, max= 118, avg=84.62, stdev=29.73, samples=8 lat (usec) : 100=12.50%, 250=1.17%, 500=0.26%, 750=4.56%, 1000=17.71% lat (msec) : 2=4.95%, 4=11.72%, 10=22.53%, 20=22.53%, 50=1.95% lat (msec) : 100=0.13% cpu : usr=0.20%, sys=21.08%, ctx=1846, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29247: Mon Apr 15 17:29:15 2024 read: IOPS=89, BW=1434KiB/s (1468kB/s)(6256KiB/4364msec) clat (usec): min=50, max=28116, avg=8092.34, stdev=5779.34 lat (usec): min=50, max=28116, avg=8093.06, stdev=5779.15 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 57], 10.00th=[ 74], 20.00th=[ 81], | 30.00th=[ 408], 40.00th=[ 8586], 50.00th=[ 9634], 60.00th=[10945], | 70.00th=[11731], 80.00th=[12649], 90.00th=[14353], 95.00th=[15664], | 99.00th=[19530], 99.50th=[22938], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 800, max= 1852, per=25.58%, avg=1359.50, stdev=345.53, samples=8 iops : min= 50, max= 115, avg=84.88, stdev=21.44, samples=8 write: IOPS=86, BW=1382KiB/s (1415kB/s)(6032KiB/4364msec) clat (usec): min=673, max=79831, avg=3160.22, stdev=5462.31 lat (usec): min=674, max=79833, avg=3161.22, stdev=5462.38 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 734], 10.00th=[ 758], 20.00th=[ 816], | 30.00th=[ 865], 40.00th=[ 947], 50.00th=[ 1385], 60.00th=[ 2638], | 70.00th=[ 3261], 80.00th=[ 4686], 90.00th=[ 6718], 95.00th=[ 8455], | 99.00th=[20317], 99.50th=[44303], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 864, max= 1980, per=23.81%, avg=1331.50, stdev=353.20, samples=8 iops : min= 54, max= 123, avg=83.12, stdev=21.88, samples=8 lat (usec) : 100=13.02%, 250=2.08%, 500=0.26%, 750=4.56%, 1000=16.28% lat (msec) : 2=5.99%, 4=11.07%, 10=20.83%, 20=25.00%, 50=0.78% lat (msec) : 100=0.13% cpu : usr=0.05%, sys=22.03%, ctx=1948, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29248: Mon Apr 15 17:29:15 2024 read: IOPS=85, BW=1370KiB/s (1403kB/s)(6032KiB/4402msec) clat (usec): min=51, max=32255, avg=8411.51, stdev=5750.57 lat (usec): min=51, max=32255, avg=8412.16, stdev=5750.56 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 68], 10.00th=[ 76], 20.00th=[ 88], | 30.00th=[ 7111], 40.00th=[ 8848], 50.00th=[10159], 60.00th=[11076], | 70.00th=[11863], 80.00th=[12649], 90.00th=[14353], 95.00th=[15795], | 99.00th=[19268], 99.50th=[27657], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 672, max= 1728, per=24.91%, avg=1324.00, stdev=328.54, samples=8 iops : min= 42, max= 108, avg=82.75, stdev=20.53, samples=8 write: IOPS=88, BW=1421KiB/s (1455kB/s)(6256KiB/4402msec) clat (usec): min=668, max=123044, avg=3124.86, stdev=6811.65 lat (usec): min=669, max=123046, avg=3126.39, stdev=6811.55 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 734], 10.00th=[ 766], 20.00th=[ 816], | 30.00th=[ 873], 40.00th=[ 947], 50.00th=[ 1172], 60.00th=[ 2507], | 70.00th=[ 3228], 80.00th=[ 4817], 90.00th=[ 6849], 95.00th=[ 8291], | 99.00th=[ 15270], 99.50th=[ 28443], 99.90th=[123208], 99.95th=[123208], | 99.99th=[123208] bw ( KiB/s): min= 768, max= 2112, per=24.89%, avg=1392.00, stdev=414.06, samples=8 iops : min= 48, max= 132, avg=87.00, stdev=25.88, samples=8 lat (usec) : 100=11.46%, 250=1.69%, 500=0.13%, 750=4.30%, 1000=17.84% lat (msec) : 2=7.42%, 4=9.77%, 10=20.31%, 20=26.56%, 50=0.39% lat (msec) : 250=0.13% cpu : usr=0.25%, sys=21.97%, ctx=1938, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5316KiB/s (5443kB/s), 1282KiB/s-1434KiB/s (1313kB/s-1468kB/s), io=23.4MiB (24.5MB), run=4364-4506msec WRITE: bw=5593KiB/s (5727kB/s), 1382KiB/s-1461KiB/s (1415kB/s-1496kB/s), io=24.6MiB (25.8MB), run=4364-4506msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29240: Mon Apr 15 17:29:21 2024 read: IOPS=34, BW=553KiB/s (567kB/s)(5920KiB/10697msec) clat (usec): min=1950, max=31533, avg=4026.43, stdev=3883.11 lat (usec): min=1951, max=31533, avg=4027.11, stdev=3883.11 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2057], 10.00th=[ 2147], 20.00th=[ 2245], | 30.00th=[ 2311], 40.00th=[ 2442], 50.00th=[ 2606], 60.00th=[ 2933], | 70.00th=[ 3392], 80.00th=[ 4359], 90.00th=[ 6915], 95.00th=[13566], | 99.00th=[20841], 99.50th=[28705], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 320, max= 1024, per=24.32%, avg=544.00, stdev=170.53, samples=21 iops : min= 20, max= 64, avg=34.00, stdev=10.66, samples=21 write: IOPS=37, BW=595KiB/s (610kB/s)(6368KiB/10697msec) clat (usec): min=12636, max=65883, avg=23096.14, stdev=7932.68 lat (usec): min=12637, max=65886, avg=23097.53, stdev=7932.74 clat percentiles (usec): | 1.00th=[12911], 5.00th=[14091], 10.00th=[14615], 20.00th=[16188], | 30.00th=[18482], 40.00th=[20055], 50.00th=[21627], 60.00th=[23462], | 70.00th=[25297], 80.00th=[27919], 90.00th=[33162], 95.00th=[38536], | 99.00th=[51643], 99.50th=[63177], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 352, max= 800, per=25.06%, avg=589.71, stdev=151.64, samples=21 iops : min= 22, max= 50, avg=36.86, stdev= 9.48, samples=21 lat (msec) : 2=0.78%, 4=36.59%, 10=7.68%, 20=21.87%, 50=32.55% lat (msec) : 100=0.52% cpu : usr=0.04%, sys=5.10%, ctx=858, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29241: Mon Apr 15 17:29:21 2024 read: IOPS=33, BW=536KiB/s (549kB/s)(5744KiB/10707msec) clat (usec): min=1825, max=20584, avg=3706.05, stdev=2827.22 lat (usec): min=1826, max=20584, avg=3706.78, stdev=2827.21 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2073], 10.00th=[ 2147], 20.00th=[ 2212], | 30.00th=[ 2311], 40.00th=[ 2442], 50.00th=[ 2573], 60.00th=[ 2835], | 70.00th=[ 3228], 80.00th=[ 4113], 90.00th=[ 7046], 95.00th=[10552], | 99.00th=[15926], 99.50th=[18220], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 192, max= 896, per=23.97%, avg=536.29, stdev=182.29, samples=21 iops : min= 12, max= 56, avg=33.48, stdev=11.35, samples=21 write: IOPS=38, BW=611KiB/s (626kB/s)(6544KiB/10707msec) clat (msec): min=12, max=103, avg=22.86, stdev= 7.81 lat (msec): min=12, max=103, avg=22.86, stdev= 7.81 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 28], 90.00th=[ 33], 95.00th=[ 36], | 99.00th=[ 44], 99.50th=[ 45], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 416, max= 768, per=25.71%, avg=604.86, stdev=123.40, samples=21 iops : min= 26, max= 48, avg=37.76, stdev= 7.68, samples=21 lat (msec) : 2=0.78%, 4=36.20%, 10=7.42%, 20=23.05%, 50=32.42% lat (msec) : 250=0.13% cpu : usr=0.07%, sys=5.01%, ctx=853, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29243: Mon Apr 15 17:29:21 2024 read: IOPS=37, BW=607KiB/s (622kB/s)(6256KiB/10307msec) clat (usec): min=1945, max=27897, avg=3912.16, stdev=3185.57 lat (usec): min=1946, max=27898, avg=3912.87, stdev=3185.56 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2114], 10.00th=[ 2180], 20.00th=[ 2278], | 30.00th=[ 2376], 40.00th=[ 2507], 50.00th=[ 2671], 60.00th=[ 2900], | 70.00th=[ 3458], 80.00th=[ 4686], 90.00th=[ 7111], 95.00th=[10421], | 99.00th=[17433], 99.50th=[19268], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 320, max= 1248, per=27.32%, avg=611.20, stdev=279.91, samples=20 iops : min= 20, max= 78, avg=38.20, stdev=17.49, samples=20 write: IOPS=36, BW=585KiB/s (599kB/s)(6032KiB/10307msec) clat (usec): min=12168, max=57310, avg=23214.19, stdev=7394.22 lat (usec): min=12169, max=57313, avg=23215.57, stdev=7394.30 clat percentiles (usec): | 1.00th=[12649], 5.00th=[13960], 10.00th=[14877], 20.00th=[16712], | 30.00th=[19006], 40.00th=[20841], 50.00th=[22414], 60.00th=[23725], | 70.00th=[25035], 80.00th=[28443], 90.00th=[32900], 95.00th=[38011], | 99.00th=[49546], 99.50th=[51643], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 352, max= 768, per=24.62%, avg=579.20, stdev=136.12, samples=20 iops : min= 22, max= 48, avg=36.20, stdev= 8.51, samples=20 lat (msec) : 2=0.65%, 4=38.02%, 10=9.51%, 20=19.66%, 50=31.77% lat (msec) : 100=0.39% cpu : usr=0.07%, sys=5.23%, ctx=877, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29244: Mon Apr 15 17:29:21 2024 read: IOPS=35, BW=568KiB/s (581kB/s)(6032KiB/10628msec) clat (usec): min=1985, max=31741, avg=4154.33, stdev=3670.69 lat (usec): min=1986, max=31742, avg=4154.99, stdev=3670.70 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2089], 10.00th=[ 2147], 20.00th=[ 2278], | 30.00th=[ 2409], 40.00th=[ 2540], 50.00th=[ 2704], 60.00th=[ 3064], | 70.00th=[ 3589], 80.00th=[ 5014], 90.00th=[ 7767], 95.00th=[12256], | 99.00th=[20317], 99.50th=[25035], 99.90th=[31851], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 224, max= 864, per=25.20%, avg=563.81, stdev=157.25, samples=21 iops : min= 14, max= 54, avg=35.24, stdev= 9.83, samples=21 write: IOPS=36, BW=589KiB/s (603kB/s)(6256KiB/10628msec) clat (usec): min=12821, max=98739, avg=23109.86, stdev=7608.19 lat (usec): min=12822, max=98742, avg=23111.35, stdev=7608.41 clat percentiles (usec): | 1.00th=[12911], 5.00th=[14091], 10.00th=[14877], 20.00th=[17171], | 30.00th=[19268], 40.00th=[20841], 50.00th=[22414], 60.00th=[23725], | 70.00th=[25035], 80.00th=[27657], 90.00th=[32113], 95.00th=[35914], | 99.00th=[42206], 99.50th=[50594], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 352, max= 768, per=25.00%, avg=588.19, stdev=129.38, samples=21 iops : min= 22, max= 48, avg=36.76, stdev= 8.09, samples=21 lat (msec) : 2=0.26%, 4=35.81%, 10=9.51%, 20=21.74%, 50=32.42% lat (msec) : 100=0.26% cpu : usr=0.08%, sys=4.98%, ctx=849, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2237KiB/s (2291kB/s), 536KiB/s-607KiB/s (549kB/s-622kB/s), io=23.4MiB (24.5MB), run=10307-10707msec WRITE: bw=2354KiB/s (2410kB/s), 585KiB/s-611KiB/s (599kB/s-626kB/s), io=24.6MiB (25.8MB), run=10307-10707msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29263: Mon Apr 15 17:29:27 2024 read: IOPS=1, BW=1872KiB/s (1916kB/s)(7168KiB/3830msec) clat (msec): min=3, max=1220, avg=490.77, stdev=480.29 lat (msec): min=3, max=1220, avg=490.77, stdev=480.29 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 54], 40.00th=[ 54], 50.00th=[ 575], 60.00th=[ 760], | 70.00th=[ 760], 80.00th=[ 818], 90.00th=[ 1217], 95.00th=[ 1217], | 99.00th=[ 1217], 99.50th=[ 1217], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 2043, max= 6144, per=72.07%, avg=3070.75, stdev=2048.83, samples=4 iops : min= 1, max= 6, avg= 2.75, stdev= 2.22, samples=4 write: IOPS=1, BW=1337KiB/s (1369kB/s)(5120KiB/3830msec) clat (msec): min=17, max=234, avg=78.36, stdev=92.92 lat (msec): min=17, max=234, avg=78.42, stdev=92.92 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 29], 60.00th=[ 29], | 70.00th=[ 93], 80.00th=[ 93], 90.00th=[ 234], 95.00th=[ 234], | 99.00th=[ 234], 99.50th=[ 234], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 2043, max= 4096, per=42.90%, avg=2558.75, stdev=1024.84, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 lat (msec) : 4=16.67%, 20=16.67%, 50=8.33%, 100=16.67%, 250=8.33% lat (msec) : 750=8.33%, 1000=16.67% cpu : usr=0.00%, sys=33.98%, ctx=850, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29265: Mon Apr 15 17:29:27 2024 read: IOPS=0, BW=426KiB/s (436kB/s)(2048KiB/4806msec) clat (msec): min=36, max=681, avg=359.07, stdev=456.25 lat (msec): min=36, max=681, avg=359.07, stdev=456.25 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 37], 60.00th=[ 684], | 70.00th=[ 684], 80.00th=[ 684], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 2048, max= 2048, per=48.06%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2131KiB/s (2182kB/s)(10.0MiB/4806msec) clat (msec): min=26, max=1898, avg=407.58, stdev=631.15 lat (msec): min=27, max=1898, avg=407.64, stdev=631.15 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 32], | 30.00th=[ 33], 40.00th=[ 44], 50.00th=[ 55], 60.00th=[ 116], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 1099], 95.00th=[ 1905], | 99.00th=[ 1905], 99.50th=[ 1905], 99.90th=[ 1905], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 2048, max= 8192, per=68.67%, avg=4096.00, stdev=2896.31, samples=4 iops : min= 2, max= 8, avg= 4.00, stdev= 2.83, samples=4 lat (msec) : 50=41.67%, 100=8.33%, 250=16.67%, 750=16.67% cpu : usr=0.00%, sys=9.26%, ctx=317, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29267: Mon Apr 15 17:29:27 2024 read: IOPS=1, BW=1327KiB/s (1359kB/s)(6144KiB/4630msec) clat (msec): min=5, max=1430, avg=545.85, stdev=495.62 lat (msec): min=5, max=1430, avg=545.85, stdev=495.62 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 230], | 30.00th=[ 230], 40.00th=[ 363], 50.00th=[ 363], 60.00th=[ 558], | 70.00th=[ 693], 80.00th=[ 693], 90.00th=[ 1435], 95.00th=[ 1435], | 99.00th=[ 1435], 99.50th=[ 1435], 99.90th=[ 1435], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 2043, max= 2048, per=48.04%, avg=2047.00, stdev= 2.24, samples=5 iops : min= 1, max= 2, avg= 1.80, stdev= 0.45, samples=5 write: IOPS=1, BW=1327KiB/s (1359kB/s)(6144KiB/4630msec) clat (msec): min=16, max=1046, avg=225.49, stdev=404.29 lat (msec): min=16, max=1046, avg=225.53, stdev=404.30 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 49], 50.00th=[ 49], 60.00th=[ 103], | 70.00th=[ 120], 80.00th=[ 120], 90.00th=[ 1045], 95.00th=[ 1045], | 99.00th=[ 1045], 99.50th=[ 1045], 99.90th=[ 1045], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 2043, max= 4096, per=41.18%, avg=2456.60, stdev=916.46, samples=5 iops : min= 1, max= 4, avg= 2.20, stdev= 1.10, samples=5 lat (msec) : 10=8.33%, 20=8.33%, 50=16.67%, 250=25.00%, 500=8.33% lat (msec) : 750=16.67% cpu : usr=0.00%, sys=15.32%, ctx=851, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29269: Mon Apr 15 17:29:27 2024 read: IOPS=1, BW=1104KiB/s (1130kB/s)(5120KiB/4639msec) clat (usec): min=527, max=587462, avg=254648.37, stdev=276550.95 lat (usec): min=528, max=587463, avg=254649.16, stdev=276550.75 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 529], 10.00th=[ 529], 20.00th=[ 529], | 30.00th=[ 1057], 40.00th=[ 1057], 50.00th=[183501], 60.00th=[183501], | 70.00th=[501220], 80.00th=[501220], 90.00th=[591397], 95.00th=[591397], | 99.00th=[591397], 99.50th=[591397], 99.90th=[591397], 99.95th=[591397], | 99.99th=[591397] bw ( KiB/s): min= 2048, max= 6144, per=96.13%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=1, BW=1545KiB/s (1582kB/s)(7168KiB/4639msec) clat (msec): min=36, max=1953, avg=480.68, stdev=740.90 lat (msec): min=36, max=1953, avg=480.74, stdev=740.91 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 41], | 30.00th=[ 68], 40.00th=[ 68], 50.00th=[ 118], 60.00th=[ 120], | 70.00th=[ 120], 80.00th=[ 1028], 90.00th=[ 1955], 95.00th=[ 1955], | 99.00th=[ 1955], 99.50th=[ 1955], 99.90th=[ 1955], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 2048, max= 4096, per=48.07%, avg=2867.20, stdev=1121.74, samples=5 iops : min= 2, max= 4, avg= 2.80, stdev= 1.10, samples=5 lat (usec) : 750=8.33% lat (msec) : 2=8.33%, 50=16.67%, 100=8.33%, 250=25.00%, 750=16.67% cpu : usr=0.00%, sys=8.06%, ctx=547, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4261KiB/s (4364kB/s), 426KiB/s-1872KiB/s (436kB/s-1916kB/s), io=20.0MiB (20.0MB), run=3830-4806msec WRITE: bw=5966KiB/s (6109kB/s), 1327KiB/s-2131KiB/s (1359kB/s-2182kB/s), io=28.0MiB (29.4MB), run=3830-4806msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29262: Mon Apr 15 17:29:27 2024 read: IOPS=1, BW=1815KiB/s (1858kB/s)(7168KiB/3950msec) clat (msec): min=26, max=812, avg=168.96, stdev=287.08 lat (msec): min=26, max=812, avg=168.96, stdev=287.08 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 34], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 50], 60.00th=[ 73], | 70.00th=[ 73], 80.00th=[ 150], 90.00th=[ 810], 95.00th=[ 810], | 99.00th=[ 810], 99.50th=[ 810], 99.90th=[ 810], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 2043, max= 4096, per=72.16%, avg=3069.50, stdev=1185.30, samples=4 iops : min= 1, max= 4, avg= 2.50, stdev= 1.73, samples=4 write: IOPS=1, BW=1296KiB/s (1327kB/s)(5120KiB/3950msec) clat (msec): min=57, max=1054, avg=553.11, stdev=447.53 lat (msec): min=57, max=1054, avg=553.19, stdev=447.52 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 58], 10.00th=[ 58], 20.00th=[ 58], | 30.00th=[ 104], 40.00th=[ 104], 50.00th=[ 726], 60.00th=[ 726], | 70.00th=[ 818], 80.00th=[ 818], 90.00th=[ 1053], 95.00th=[ 1053], | 99.00th=[ 1053], 99.50th=[ 1053], 99.90th=[ 1053], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 4087, max= 4096, per=68.71%, avg=4091.50, stdev= 6.36, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 lat (msec) : 50=33.33%, 100=16.67%, 250=16.67%, 750=8.33%, 1000=16.67% cpu : usr=0.00%, sys=4.28%, ctx=35, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29264: Mon Apr 15 17:29:27 2024 read: IOPS=0, BW=429KiB/s (440kB/s)(2048KiB/4769msec) clat (usec): min=24607, max=71059, avg=47833.16, stdev=32846.67 lat (usec): min=24608, max=71060, avg=47834.28, stdev=32846.17 clat percentiles (usec): | 1.00th=[24511], 5.00th=[24511], 10.00th=[24511], 20.00th=[24511], | 30.00th=[24511], 40.00th=[24511], 50.00th=[24511], 60.00th=[70779], | 70.00th=[70779], 80.00th=[70779], 90.00th=[70779], 95.00th=[70779], | 99.00th=[70779], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 2048, max= 2048, per=48.14%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2147KiB/s (2199kB/s)(10.0MiB/4769msec) clat (msec): min=36, max=1342, avg=467.12, stdev=477.07 lat (msec): min=36, max=1342, avg=467.20, stdev=477.09 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 40], | 30.00th=[ 64], 40.00th=[ 84], 50.00th=[ 100], 60.00th=[ 625], | 70.00th=[ 667], 80.00th=[ 667], 90.00th=[ 1083], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 2048, max= 6144, per=55.00%, avg=3275.00, stdev=1830.78, samples=5 iops : min= 2, max= 6, avg= 3.00, stdev= 1.73, samples=5 lat (msec) : 50=25.00%, 100=33.33%, 750=25.00% cpu : usr=0.06%, sys=3.19%, ctx=120, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29266: Mon Apr 15 17:29:27 2024 read: IOPS=1, BW=1299KiB/s (1330kB/s)(6144KiB/4730msec) clat (msec): min=33, max=1346, avg=536.25, stdev=467.60 lat (msec): min=33, max=1346, avg=536.26, stdev=467.59 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 456], 50.00th=[ 456], 60.00th=[ 592], | 70.00th=[ 651], 80.00th=[ 651], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 2048, max= 4096, per=80.17%, avg=3410.33, stdev=1179.82, samples=3 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=3 write: IOPS=1, BW=1299KiB/s (1330kB/s)(6144KiB/4730msec) clat (msec): min=55, max=1040, avg=251.09, stdev=387.61 lat (msec): min=55, max=1040, avg=251.16, stdev=387.60 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 56], 10.00th=[ 56], 20.00th=[ 86], | 30.00th=[ 86], 40.00th=[ 89], 50.00th=[ 89], 60.00th=[ 107], | 70.00th=[ 130], 80.00th=[ 130], 90.00th=[ 1045], 95.00th=[ 1045], | 99.00th=[ 1045], 99.50th=[ 1045], 99.90th=[ 1045], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 2043, max= 4096, per=41.25%, avg=2456.60, stdev=916.46, samples=5 iops : min= 1, max= 4, avg= 2.20, stdev= 1.10, samples=5 lat (msec) : 50=8.33%, 100=25.00%, 250=25.00%, 500=8.33%, 750=16.67% cpu : usr=0.00%, sys=3.02%, ctx=53, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29268: Mon Apr 15 17:29:27 2024 read: IOPS=1, BW=1064KiB/s (1089kB/s)(5120KiB/4814msec) clat (msec): min=30, max=731, avg=289.81, stdev=354.43 lat (msec): min=30, max=731, avg=289.81, stdev=354.43 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 35], 60.00th=[ 35], | 70.00th=[ 617], 80.00th=[ 617], 90.00th=[ 735], 95.00th=[ 735], | 99.00th=[ 735], 99.50th=[ 735], 99.90th=[ 735], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 4096, max= 4096, per=96.29%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=1, BW=1489KiB/s (1525kB/s)(7168KiB/4814msec) clat (msec): min=34, max=1955, avg=480.34, stdev=740.13 lat (msec): min=34, max=1955, avg=480.43, stdev=740.14 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 48], | 30.00th=[ 68], 40.00th=[ 68], 50.00th=[ 97], 60.00th=[ 140], | 70.00th=[ 140], 80.00th=[ 1020], 90.00th=[ 1955], 95.00th=[ 1955], | 99.00th=[ 1955], 99.50th=[ 1955], 99.90th=[ 1955], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 2048, max= 6144, per=60.18%, avg=3584.00, stdev=1960.81, samples=4 iops : min= 2, max= 6, avg= 3.50, stdev= 1.91, samples=4 lat (msec) : 50=41.67%, 100=16.67%, 250=8.33%, 750=16.67% cpu : usr=0.02%, sys=3.43%, ctx=97, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4254KiB/s (4356kB/s), 429KiB/s-1815KiB/s (440kB/s-1858kB/s), io=20.0MiB (20.0MB), run=3950-4814msec WRITE: bw=5956KiB/s (6099kB/s), 1296KiB/s-2147KiB/s (1327kB/s-2199kB/s), io=28.0MiB (29.4MB), run=3950-4814msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29279: Mon Apr 15 17:29:32 2024 read: IOPS=0, BW=3433KiB/s (3516kB/s)(12.0MiB/3579msec) clat (msec): min=136, max=1952, avg=1178.86, stdev=937.18 lat (msec): min=136, max=1952, avg=1178.86, stdev=937.18 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 138], 10.00th=[ 138], 20.00th=[ 138], | 30.00th=[ 138], 40.00th=[ 1452], 50.00th=[ 1452], 60.00th=[ 1452], | 70.00th=[ 1955], 80.00th=[ 1955], 90.00th=[ 1955], 95.00th=[ 1955], | 99.00th=[ 1955], 99.50th=[ 1955], 99.90th=[ 1955], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=45.70%, ctx=1076, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29280: Mon Apr 15 17:29:32 2024 read: IOPS=0, BW=1027KiB/s (1051kB/s)(4096KiB/3990msec) clat (nsec): min=125114k, max=125114k, avg=125114294.00, stdev= 0.00 lat (nsec): min=125116k, max=125116k, avg=125115656.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 126], 10.00th=[ 126], 20.00th=[ 126], | 30.00th=[ 126], 40.00th=[ 126], 50.00th=[ 126], 60.00th=[ 126], | 70.00th=[ 126], 80.00th=[ 126], 90.00th=[ 126], 95.00th=[ 126], | 99.00th=[ 126], 99.50th=[ 126], 99.90th=[ 126], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2053KiB/s (2102kB/s)(8192KiB/3990msec) clat (msec): min=254, max=3609, avg=1931.68, stdev=2372.32 lat (msec): min=254, max=3609, avg=1931.96, stdev=2372.17 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 255], 10.00th=[ 255], 20.00th=[ 255], | 30.00th=[ 255], 40.00th=[ 255], 50.00th=[ 255], 60.00th=[ 3608], | 70.00th=[ 3608], 80.00th=[ 3608], 90.00th=[ 3608], 95.00th=[ 3608], | 99.00th=[ 3608], 99.50th=[ 3608], 99.90th=[ 3608], 99.95th=[ 3608], | 99.99th=[ 3608] lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=5.79%, ctx=112, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29281: Mon Apr 15 17:29:32 2024 write: IOPS=0, BW=2991KiB/s (3062kB/s)(12.0MiB/4109msec) clat (msec): min=209, max=2341, avg=1368.28, stdev=1078.12 lat (msec): min=209, max=2341, avg=1368.46, stdev=1078.17 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 209], 10.00th=[ 209], 20.00th=[ 209], | 30.00th=[ 209], 40.00th=[ 1552], 50.00th=[ 1552], 60.00th=[ 1552], | 70.00th=[ 2333], 80.00th=[ 2333], 90.00th=[ 2333], 95.00th=[ 2333], | 99.00th=[ 2333], 99.50th=[ 2333], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.02%, sys=6.04%, ctx=31, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29282: Mon Apr 15 17:29:32 2024 write: IOPS=1, BW=5322KiB/s (5450kB/s)(12.0MiB/2309msec) clat (msec): min=212, max=1627, avg=756.10, stdev=762.60 lat (msec): min=213, max=1628, avg=756.42, stdev=762.52 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 213], 10.00th=[ 213], 20.00th=[ 213], | 30.00th=[ 213], 40.00th=[ 430], 50.00th=[ 430], 60.00th=[ 430], | 70.00th=[ 1636], 80.00th=[ 1636], 90.00th=[ 1636], 95.00th=[ 1636], | 99.00th=[ 1636], 99.50th=[ 1636], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.04%, sys=11.22%, ctx=49, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4106KiB/s (4205kB/s), 1027KiB/s-3433KiB/s (1051kB/s-3516kB/s), io=16.0MiB (16.8MB), run=3579-3990msec WRITE: bw=7975KiB/s (8166kB/s), 2053KiB/s-5322KiB/s (2102kB/s-5450kB/s), io=32.0MiB (33.6MB), run=2309-4109msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29283: Mon Apr 15 17:29:32 2024 read: IOPS=0, BW=2965KiB/s (3036kB/s)(12.0MiB/4145msec) clat (msec): min=372, max=2130, avg=1381.29, stdev=906.90 lat (msec): min=372, max=2130, avg=1381.29, stdev=906.90 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 372], 10.00th=[ 372], 20.00th=[ 372], | 30.00th=[ 372], 40.00th=[ 1636], 50.00th=[ 1636], 60.00th=[ 1636], | 70.00th=[ 2123], 80.00th=[ 2123], 90.00th=[ 2123], 95.00th=[ 2123], | 99.00th=[ 2123], 99.50th=[ 2123], 99.90th=[ 2123], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=2.15%, ctx=29, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29284: Mon Apr 15 17:29:32 2024 read: IOPS=0, BW=1006KiB/s (1030kB/s)(4096KiB/4071msec) clat (nsec): min=419399k, max=419399k, avg=419398888.00, stdev= 0.00 lat (nsec): min=419400k, max=419400k, avg=419400275.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 418], 10.00th=[ 418], 20.00th=[ 418], | 30.00th=[ 418], 40.00th=[ 418], 50.00th=[ 418], 60.00th=[ 418], | 70.00th=[ 418], 80.00th=[ 418], 90.00th=[ 418], 95.00th=[ 418], | 99.00th=[ 418], 99.50th=[ 418], 99.90th=[ 418], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2012KiB/s (2061kB/s)(8192KiB/4071msec) clat (msec): min=90, max=3558, avg=1824.24, stdev=2452.05 lat (msec): min=90, max=3558, avg=1824.62, stdev=2451.91 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 91], 10.00th=[ 91], 20.00th=[ 91], | 30.00th=[ 91], 40.00th=[ 91], 50.00th=[ 91], 60.00th=[ 3574], | 70.00th=[ 3574], 80.00th=[ 3574], 90.00th=[ 3574], 95.00th=[ 3574], | 99.00th=[ 3574], 99.50th=[ 3574], 99.90th=[ 3574], 99.95th=[ 3574], | 99.99th=[ 3574] bw ( KiB/s): min= 7772, max= 7772, per=99.22%, avg=7772.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=3.59%, ctx=38, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29285: Mon Apr 15 17:29:32 2024 write: IOPS=0, BW=2938KiB/s (3008kB/s)(12.0MiB/4183msec) clat (msec): min=468, max=2100, avg=1392.48, stdev=837.33 lat (msec): min=468, max=2100, avg=1392.72, stdev=837.45 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 468], 10.00th=[ 468], 20.00th=[ 468], | 30.00th=[ 468], 40.00th=[ 1603], 50.00th=[ 1603], 60.00th=[ 1603], | 70.00th=[ 2106], 80.00th=[ 2106], 90.00th=[ 2106], 95.00th=[ 2106], | 99.00th=[ 2106], 99.50th=[ 2106], 99.90th=[ 2106], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 7984, max= 8192, per=100.00%, avg=8088.00, stdev=147.08, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33% cpu : usr=0.02%, sys=2.08%, ctx=20, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29286: Mon Apr 15 17:29:32 2024 write: IOPS=1, BW=5251KiB/s (5377kB/s)(12.0MiB/2340msec) clat (msec): min=199, max=1708, avg=777.75, stdev=814.29 lat (msec): min=199, max=1709, avg=778.05, stdev=814.32 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 201], 10.00th=[ 201], 20.00th=[ 201], | 30.00th=[ 201], 40.00th=[ 426], 50.00th=[ 426], 60.00th=[ 426], | 70.00th=[ 1703], 80.00th=[ 1703], 90.00th=[ 1703], 95.00th=[ 1703], | 99.00th=[ 1703], 99.50th=[ 1703], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=4.87%, ctx=148, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3953KiB/s (4048kB/s), 1006KiB/s-2965KiB/s (1030kB/s-3036kB/s), io=16.0MiB (16.8MB), run=4071-4145msec WRITE: bw=7834KiB/s (8022kB/s), 2012KiB/s-5251KiB/s (2061kB/s-5377kB/s), io=32.0MiB (33.6MB), run=2340-4183msec