== sanity test 398b: DIO and buffer IO race ============== 16:17:57 (1709068677) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.4846 s, 33.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15577: Tue Feb 27 16:18:30 2024 read: IOPS=51, BW=205KiB/s (210kB/s)(5992KiB/29163msec) clat (usec): min=41, max=256216, avg=7545.23, stdev=10136.39 lat (usec): min=41, max=256217, avg=7546.59, stdev=10136.35 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 72], 10.00th=[ 85], 20.00th=[ 101], | 30.00th=[ 137], 40.00th=[ 4555], 50.00th=[ 5932], 60.00th=[ 7504], | 70.00th=[ 9372], 80.00th=[ 12518], 90.00th=[ 18482], 95.00th=[ 22676], | 99.00th=[ 32637], 99.50th=[ 41681], 99.90th=[ 57934], 99.95th=[256902], | 99.99th=[256902] bw ( KiB/s): min= 96, max= 304, per=25.21%, avg=204.45, stdev=56.33, samples=58 iops : min= 24, max= 76, avg=50.86, stdev=14.18, samples=58 write: IOPS=53, BW=216KiB/s (221kB/s)(6296KiB/29163msec) clat (usec): min=386, max=111459, avg=11228.73, stdev=11542.44 lat (usec): min=387, max=111460, avg=11230.90, stdev=11543.15 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 627], 10.00th=[ 676], 20.00th=[ 914], | 30.00th=[ 2474], 40.00th=[ 6063], 50.00th=[ 8979], 60.00th=[ 11469], | 70.00th=[ 14222], 80.00th=[ 18482], 90.00th=[ 25560], 95.00th=[ 32637], | 99.00th=[ 52167], 99.50th=[ 61080], 99.90th=[ 95945], 99.95th=[111674], | 99.99th=[111674] bw ( KiB/s): min= 64, max= 344, per=25.27%, avg=214.52, stdev=59.41, samples=58 iops : min= 16, max= 86, avg=53.38, stdev=14.93, samples=58 lat (usec) : 50=0.13%, 100=9.44%, 250=5.89%, 500=0.75%, 750=7.71% lat (usec) : 1000=3.26% lat (msec) : 2=3.65%, 4=4.65%, 10=28.12%, 20=24.25%, 50=11.46% lat (msec) : 100=0.62%, 250=0.03%, 500=0.03% cpu : usr=0.11%, sys=13.94%, ctx=4813, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15578: Tue Feb 27 16:18:30 2024 read: IOPS=50, BW=203KiB/s (208kB/s)(5940KiB/29301msec) clat (usec): min=42, max=53058, avg=7383.67, stdev=7510.60 lat (usec): min=42, max=53059, avg=7385.21, stdev=7511.53 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 76], 10.00th=[ 89], 20.00th=[ 105], | 30.00th=[ 273], 40.00th=[ 4621], 50.00th=[ 5932], 60.00th=[ 7504], | 70.00th=[ 9372], 80.00th=[12518], 90.00th=[17957], 95.00th=[22676], | 99.00th=[31851], 99.50th=[34341], 99.90th=[39584], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 80, max= 320, per=24.98%, avg=202.60, stdev=59.46, samples=58 iops : min= 20, max= 80, avg=50.47, stdev=14.87, samples=58 write: IOPS=54, BW=217KiB/s (222kB/s)(6348KiB/29301msec) clat (usec): min=438, max=250898, avg=11444.88, stdev=13714.38 lat (usec): min=439, max=250899, avg=11449.05, stdev=13716.30 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 627], 10.00th=[ 676], 20.00th=[ 922], | 30.00th=[ 2180], 40.00th=[ 5669], 50.00th=[ 8455], 60.00th=[ 11207], | 70.00th=[ 14091], 80.00th=[ 19006], 90.00th=[ 26608], 95.00th=[ 34341], | 99.00th=[ 50070], 99.50th=[ 67634], 99.90th=[139461], 99.95th=[250610], | 99.99th=[250610] bw ( KiB/s): min= 104, max= 336, per=25.40%, avg=215.67, stdev=56.54, samples=58 iops : min= 26, max= 84, avg=53.74, stdev=14.15, samples=58 lat (usec) : 50=0.07%, 100=8.01%, 250=6.09%, 500=0.94%, 750=8.11% lat (usec) : 1000=3.19% lat (msec) : 2=4.17%, 4=4.82%, 10=28.29%, 20=23.21%, 50=12.57% lat (msec) : 100=0.46%, 250=0.07%, 500=0.03% cpu : usr=0.17%, sys=13.71%, ctx=4794, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15579: Tue Feb 27 16:18:30 2024 read: IOPS=50, BW=204KiB/s (209kB/s)(6028KiB/29582msec) clat (usec): min=51, max=255365, avg=7794.91, stdev=10303.70 lat (usec): min=51, max=255365, avg=7797.46, stdev=10305.29 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 75], 10.00th=[ 88], 20.00th=[ 105], | 30.00th=[ 461], 40.00th=[ 4686], 50.00th=[ 6063], 60.00th=[ 7635], | 70.00th=[ 9765], 80.00th=[ 12911], 90.00th=[ 18482], 95.00th=[ 22938], | 99.00th=[ 33424], 99.50th=[ 39060], 99.90th=[ 78119], 99.95th=[254804], | 99.99th=[254804] bw ( KiB/s): min= 47, max= 368, per=24.62%, avg=199.69, stdev=65.13, samples=58 iops : min= 11, max= 92, avg=49.71, stdev=16.37, samples=58 write: IOPS=52, BW=212KiB/s (217kB/s)(6260KiB/29582msec) clat (usec): min=411, max=164810, avg=11285.20, stdev=12166.09 lat (usec): min=411, max=164811, avg=11288.57, stdev=12164.91 clat percentiles (usec): | 1.00th=[ 578], 5.00th=[ 635], 10.00th=[ 685], 20.00th=[ 930], | 30.00th=[ 2147], 40.00th=[ 5604], 50.00th=[ 8586], 60.00th=[ 11338], | 70.00th=[ 14222], 80.00th=[ 19006], 90.00th=[ 26346], 95.00th=[ 33424], | 99.00th=[ 48497], 99.50th=[ 56361], 99.90th=[ 98042], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 120, max= 360, per=24.30%, avg=206.31, stdev=51.38, samples=58 iops : min= 30, max= 90, avg=51.36, stdev=12.86, samples=58 lat (usec) : 100=8.56%, 250=5.76%, 500=0.62%, 750=7.62%, 1000=3.58% lat (msec) : 2=4.56%, 4=5.18%, 10=27.02%, 20=23.80%, 50=12.73% lat (msec) : 100=0.52%, 250=0.03%, 500=0.03% cpu : usr=0.11%, sys=14.16%, ctx=4754, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15580: Tue Feb 27 16:18:30 2024 read: IOPS=51, BW=208KiB/s (213kB/s)(6052KiB/29134msec) clat (usec): min=41, max=213386, avg=7641.59, stdev=9735.57 lat (usec): min=41, max=213386, avg=7643.10, stdev=9735.31 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 74], 10.00th=[ 87], 20.00th=[ 102], | 30.00th=[ 229], 40.00th=[ 4490], 50.00th=[ 5800], 60.00th=[ 7635], | 70.00th=[ 9503], 80.00th=[ 12649], 90.00th=[ 18220], 95.00th=[ 23200], | 99.00th=[ 33817], 99.50th=[ 40109], 99.90th=[100140], 99.95th=[212861], | 99.99th=[212861] bw ( KiB/s): min= 95, max= 359, per=25.50%, avg=206.83, stdev=56.32, samples=58 iops : min= 23, max= 89, avg=51.47, stdev=14.12, samples=58 write: IOPS=53, BW=214KiB/s (219kB/s)(6236KiB/29134msec) clat (usec): min=407, max=203896, avg=11154.24, stdev=12512.11 lat (usec): min=408, max=203897, avg=11155.51, stdev=12512.10 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 627], 10.00th=[ 676], 20.00th=[ 848], | 30.00th=[ 2008], 40.00th=[ 5473], 50.00th=[ 8455], 60.00th=[ 11207], | 70.00th=[ 14353], 80.00th=[ 18482], 90.00th=[ 25822], 95.00th=[ 33424], | 99.00th=[ 49546], 99.50th=[ 57934], 99.90th=[105382], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 79, max= 328, per=25.09%, avg=213.02, stdev=54.14, samples=58 iops : min= 19, max= 82, avg=53.02, stdev=13.62, samples=58 lat (usec) : 50=0.20%, 100=9.18%, 250=5.57%, 500=0.68%, 750=8.30% lat (usec) : 1000=3.68% lat (msec) : 2=3.55%, 4=5.86%, 10=26.46%, 20=23.60%, 50=12.34% lat (msec) : 100=0.46%, 250=0.13% cpu : usr=0.13%, sys=13.66%, ctx=4836, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=812KiB/s (831kB/s), 203KiB/s-208KiB/s (208kB/s-213kB/s), io=23.4MiB (24.6MB), run=29134-29582msec WRITE: bw=850KiB/s (870kB/s), 212KiB/s-217KiB/s (217kB/s-222kB/s), io=24.6MiB (25.7MB), run=29134-29582msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15573: Tue Feb 27 16:20:06 2024 read: IOPS=12, BW=48.4KiB/s (49.5kB/s)(5992KiB/123833msec) clat (msec): min=2, max=256, avg= 8.89, stdev= 9.94 lat (msec): min=2, max=256, avg= 8.89, stdev= 9.94 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 17], 95.00th=[ 23], | 99.00th=[ 36], 99.50th=[ 52], 99.90th=[ 108], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 8, max= 184, per=25.92%, avg=49.51, stdev=28.96, samples=240 iops : min= 2, max= 46, avg=12.14, stdev= 7.25, samples=240 write: IOPS=12, BW=50.8KiB/s (52.1kB/s)(6296KiB/123833msec) clat (msec): min=24, max=203, avg=70.09, stdev=26.67 lat (msec): min=24, max=203, avg=70.10, stdev=26.67 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 37], 10.00th=[ 42], 20.00th=[ 50], | 30.00th=[ 55], 40.00th=[ 60], 50.00th=[ 66], 60.00th=[ 72], | 70.00th=[ 79], 80.00th=[ 88], 90.00th=[ 104], 95.00th=[ 122], | 99.00th=[ 163], 99.50th=[ 174], 99.90th=[ 201], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 8, max= 87, per=25.26%, avg=50.53, stdev=14.70, samples=247 iops : min= 2, max= 21, avg=12.39, stdev= 3.69, samples=247 lat (msec) : 4=4.23%, 10=34.18%, 20=6.93%, 50=14.32%, 100=34.37% lat (msec) : 250=5.92%, 500=0.03% cpu : usr=0.02%, sys=2.56%, ctx=3282, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15574: Tue Feb 27 16:20:06 2024 read: IOPS=11, BW=47.4KiB/s (48.5kB/s)(5940KiB/125302msec) clat (msec): min=2, max=215, avg= 9.07, stdev= 9.65 lat (msec): min=2, max=215, avg= 9.08, stdev= 9.65 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 17], 95.00th=[ 24], | 99.00th=[ 42], 99.50th=[ 60], 99.90th=[ 108], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 7, max= 152, per=25.09%, avg=47.93, stdev=27.16, samples=246 iops : min= 1, max= 38, avg=11.81, stdev= 6.81, samples=246 write: IOPS=12, BW=50.7KiB/s (51.9kB/s)(6348KiB/125302msec) clat (msec): min=26, max=237, avg=70.34, stdev=26.85 lat (msec): min=26, max=237, avg=70.34, stdev=26.86 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 37], 10.00th=[ 41], 20.00th=[ 48], | 30.00th=[ 54], 40.00th=[ 61], 50.00th=[ 66], 60.00th=[ 72], | 70.00th=[ 81], 80.00th=[ 90], 90.00th=[ 106], 95.00th=[ 120], | 99.00th=[ 157], 99.50th=[ 174], 99.90th=[ 201], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 8, max= 88, per=25.17%, avg=50.34, stdev=14.86, samples=250 iops : min= 2, max= 22, avg=12.42, stdev= 3.72, samples=250 lat (msec) : 4=3.22%, 10=34.83%, 20=6.77%, 50=15.40%, 100=33.20% lat (msec) : 250=6.58% cpu : usr=0.06%, sys=2.51%, ctx=3294, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15575: Tue Feb 27 16:20:06 2024 read: IOPS=12, BW=48.4KiB/s (49.6kB/s)(6028KiB/124433msec) clat (usec): min=2609, max=75928, avg=9136.36, stdev=7859.90 lat (usec): min=2610, max=75929, avg=9139.34, stdev=7859.42 clat percentiles (usec): | 1.00th=[ 2999], 5.00th=[ 3687], 10.00th=[ 4080], 20.00th=[ 4686], | 30.00th=[ 5276], 40.00th=[ 5800], 50.00th=[ 6456], 60.00th=[ 7373], | 70.00th=[ 8717], 80.00th=[10945], 90.00th=[17957], 95.00th=[25035], | 99.00th=[40633], 99.50th=[52167], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 7, max= 152, per=25.56%, avg=48.82, stdev=28.43, samples=245 iops : min= 1, max= 38, avg=12.04, stdev= 7.14, samples=245 write: IOPS=12, BW=50.3KiB/s (51.5kB/s)(6260KiB/124433msec) clat (msec): min=23, max=248, avg=70.58, stdev=27.63 lat (msec): min=23, max=248, avg=70.59, stdev=27.63 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 36], 10.00th=[ 41], 20.00th=[ 48], | 30.00th=[ 55], 40.00th=[ 61], 50.00th=[ 66], 60.00th=[ 72], | 70.00th=[ 81], 80.00th=[ 90], 90.00th=[ 105], 95.00th=[ 120], | 99.00th=[ 163], 99.50th=[ 180], 99.90th=[ 241], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min= 16, max= 88, per=25.03%, avg=50.06, stdev=15.10, samples=248 iops : min= 4, max= 22, avg=12.35, stdev= 3.77, samples=248 lat (msec) : 4=4.43%, 10=33.24%, 20=7.39%, 50=15.43%, 100=33.30% lat (msec) : 250=6.22% cpu : usr=0.03%, sys=2.53%, ctx=3302, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15576: Tue Feb 27 16:20:06 2024 read: IOPS=12, BW=48.7KiB/s (49.8kB/s)(6052KiB/124391msec) clat (msec): min=2, max=115, avg= 9.11, stdev= 8.81 lat (msec): min=2, max=115, avg= 9.11, stdev= 8.81 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 18], 95.00th=[ 25], | 99.00th=[ 46], 99.50th=[ 59], 99.90th=[ 100], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 7, max= 160, per=25.81%, avg=49.30, stdev=27.14, samples=244 iops : min= 1, max= 40, avg=12.14, stdev= 6.79, samples=244 write: IOPS=12, BW=50.1KiB/s (51.3kB/s)(6236KiB/124391msec) clat (msec): min=20, max=284, avg=70.82, stdev=27.05 lat (msec): min=20, max=284, avg=70.83, stdev=27.05 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 37], 10.00th=[ 42], 20.00th=[ 50], | 30.00th=[ 55], 40.00th=[ 60], 50.00th=[ 66], 60.00th=[ 73], | 70.00th=[ 81], 80.00th=[ 90], 90.00th=[ 106], 95.00th=[ 122], | 99.00th=[ 157], 99.50th=[ 180], 99.90th=[ 194], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 15, max= 95, per=24.96%, avg=49.92, stdev=14.43, samples=248 iops : min= 3, max= 23, avg=12.30, stdev= 3.66, samples=248 lat (msec) : 4=3.97%, 10=34.83%, 20=6.80%, 50=14.13%, 100=33.69% lat (msec) : 250=6.54%, 500=0.03% cpu : usr=0.03%, sys=2.54%, ctx=3315, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=192KiB/s (196kB/s), 47.4KiB/s-48.7KiB/s (48.5kB/s-49.8kB/s), io=23.4MiB (24.6MB), run=123833-125302msec WRITE: bw=201KiB/s (205kB/s), 50.1KiB/s-50.8KiB/s (51.3kB/s-52.1kB/s), io=24.6MiB (25.7MB), run=123833-125302msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15695: Tue Feb 27 16:20:12 2024 read: IOPS=68, BW=1092KiB/s (1118kB/s)(5920KiB/5421msec) clat (usec): min=43, max=53516, avg=7199.15, stdev=7909.32 lat (usec): min=43, max=53517, avg=7206.47, stdev=7911.90 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 74], 10.00th=[ 82], 20.00th=[ 98], | 30.00th=[ 120], 40.00th=[ 709], 50.00th=[ 6587], 60.00th=[ 8160], | 70.00th=[10421], 80.00th=[12911], 90.00th=[16712], 95.00th=[21365], | 99.00th=[33817], 99.50th=[41157], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 736, max= 1696, per=24.46%, avg=1042.20, stdev=314.31, samples=10 iops : min= 46, max= 106, avg=64.80, stdev=19.59, samples=10 write: IOPS=73, BW=1175KiB/s (1203kB/s)(6368KiB/5421msec) clat (usec): min=696, max=70424, avg=6866.32, stdev=8983.75 lat (usec): min=698, max=70425, avg=6868.20, stdev=8983.72 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 889], 10.00th=[ 979], 20.00th=[ 1123], | 30.00th=[ 1287], 40.00th=[ 1549], 50.00th=[ 2278], 60.00th=[ 4490], | 70.00th=[ 7767], 80.00th=[12387], 90.00th=[18482], 95.00th=[25035], | 99.00th=[36439], 99.50th=[50594], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 798, max= 2075, per=25.67%, avg=1151.00, stdev=409.46, samples=10 iops : min= 49, max= 129, avg=71.60, stdev=25.57, samples=10 lat (usec) : 50=0.39%, 100=9.90%, 250=8.20%, 500=0.52%, 750=0.52% lat (usec) : 1000=6.38% lat (msec) : 2=17.84%, 4=6.38%, 10=22.40%, 20=19.92%, 50=7.03% lat (msec) : 100=0.52% cpu : usr=0.24%, sys=18.01%, ctx=997, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15698: Tue Feb 27 16:20:12 2024 read: IOPS=65, BW=1048KiB/s (1073kB/s)(5744KiB/5480msec) clat (usec): min=41, max=100855, avg=6861.84, stdev=8998.47 lat (usec): min=41, max=100855, avg=6862.93, stdev=8998.52 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 78], 10.00th=[ 88], 20.00th=[ 100], | 30.00th=[ 114], 40.00th=[ 200], 50.00th=[ 5932], 60.00th=[ 7177], | 70.00th=[ 9110], 80.00th=[ 12518], 90.00th=[ 17433], 95.00th=[ 22414], | 99.00th=[ 27132], 99.50th=[ 52167], 99.90th=[101188], 99.95th=[101188], | 99.99th=[101188] bw ( KiB/s): min= 576, max= 1600, per=23.15%, avg=986.30, stdev=298.47, samples=10 iops : min= 36, max= 100, avg=61.30, stdev=18.72, samples=10 write: IOPS=74, BW=1194KiB/s (1223kB/s)(6544KiB/5480msec) clat (usec): min=555, max=105984, avg=7284.34, stdev=9434.52 lat (usec): min=556, max=105986, avg=7286.32, stdev=9434.60 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 914], 10.00th=[ 996], 20.00th=[ 1205], | 30.00th=[ 1467], 40.00th=[ 2057], 50.00th=[ 3326], 60.00th=[ 5669], | 70.00th=[ 8586], 80.00th=[ 12780], 90.00th=[ 17695], 95.00th=[ 23200], | 99.00th=[ 38011], 99.50th=[ 43779], 99.90th=[106431], 99.95th=[106431], | 99.99th=[106431] bw ( KiB/s): min= 593, max= 1564, per=24.93%, avg=1117.50, stdev=266.16, samples=10 iops : min= 37, max= 97, avg=69.60, stdev=16.57, samples=10 lat (usec) : 50=0.13%, 100=9.77%, 250=9.11%, 500=1.04%, 750=0.52% lat (usec) : 1000=5.21% lat (msec) : 2=15.76%, 4=7.29%, 10=24.74%, 20=19.27%, 50=6.64% lat (msec) : 100=0.26%, 250=0.26% cpu : usr=0.16%, sys=17.70%, ctx=1013, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15700: Tue Feb 27 16:20:12 2024 read: IOPS=69, BW=1113KiB/s (1140kB/s)(6256KiB/5620msec) clat (usec): min=59, max=142256, avg=6994.73, stdev=10281.43 lat (usec): min=60, max=142257, avg=6997.23, stdev=10282.59 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 73], 10.00th=[ 81], 20.00th=[ 96], | 30.00th=[ 111], 40.00th=[ 139], 50.00th=[ 5473], 60.00th=[ 7701], | 70.00th=[ 10159], 80.00th=[ 12780], 90.00th=[ 16581], 95.00th=[ 20055], | 99.00th=[ 41157], 99.50th=[ 44303], 99.90th=[141558], 99.95th=[141558], | 99.99th=[141558] bw ( KiB/s): min= 415, max= 2075, per=25.82%, avg=1100.27, stdev=481.13, samples=11 iops : min= 25, max= 129, avg=68.45, stdev=30.04, samples=11 write: IOPS=67, BW=1073KiB/s (1099kB/s)(6032KiB/5620msec) clat (usec): min=706, max=93831, avg=7566.78, stdev=11489.93 lat (usec): min=708, max=93834, avg=7568.48, stdev=11490.04 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 881], 10.00th=[ 955], 20.00th=[ 1090], | 30.00th=[ 1287], 40.00th=[ 1680], 50.00th=[ 3064], 60.00th=[ 6128], | 70.00th=[ 8291], 80.00th=[11731], 90.00th=[17957], 95.00th=[24511], | 99.00th=[66847], 99.50th=[86508], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 510, max= 1583, per=23.41%, avg=1049.64, stdev=411.57, samples=11 iops : min= 31, max= 98, avg=65.27, stdev=25.71, samples=11 lat (usec) : 100=11.72%, 250=9.90%, 500=0.65%, 750=0.65%, 1000=6.12% lat (msec) : 2=16.41%, 4=5.08%, 10=22.14%, 20=20.70%, 50=5.60% lat (msec) : 100=0.91%, 250=0.13% cpu : usr=0.16%, sys=16.96%, ctx=1008, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15702: Tue Feb 27 16:20:12 2024 read: IOPS=70, BW=1126KiB/s (1153kB/s)(6032KiB/5355msec) clat (usec): min=46, max=43057, avg=6541.87, stdev=7600.10 lat (usec): min=47, max=43058, avg=6542.92, stdev=7600.14 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 71], 10.00th=[ 79], 20.00th=[ 95], | 30.00th=[ 109], 40.00th=[ 143], 50.00th=[ 5145], 60.00th=[ 7701], | 70.00th=[ 9241], 80.00th=[12256], 90.00th=[17695], 95.00th=[21365], | 99.00th=[27132], 99.50th=[35914], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 224, max= 1756, per=25.54%, avg=1088.30, stdev=432.50, samples=10 iops : min= 14, max= 109, avg=67.70, stdev=27.04, samples=10 write: IOPS=73, BW=1168KiB/s (1196kB/s)(6256KiB/5355msec) clat (usec): min=712, max=109322, avg=7299.36, stdev=11943.41 lat (usec): min=714, max=109324, avg=7301.63, stdev=11943.82 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 873], 10.00th=[ 947], 20.00th=[ 1074], | 30.00th=[ 1287], 40.00th=[ 1582], 50.00th=[ 2311], 60.00th=[ 3949], | 70.00th=[ 7701], 80.00th=[ 12780], 90.00th=[ 19268], 95.00th=[ 22938], | 99.00th=[ 57410], 99.50th=[104334], 99.90th=[109577], 99.95th=[109577], | 99.99th=[109577] bw ( KiB/s): min= 288, max= 1696, per=25.64%, avg=1149.30, stdev=460.34, samples=10 iops : min= 18, max= 106, avg=71.60, stdev=28.77, samples=10 lat (usec) : 50=0.13%, 100=11.33%, 250=9.77%, 500=1.30%, 750=0.39% lat (usec) : 1000=6.77% lat (msec) : 2=16.67%, 4=7.55%, 10=19.79%, 20=19.01%, 50=6.77% lat (msec) : 100=0.13%, 250=0.39% cpu : usr=0.06%, sys=17.26%, ctx=971, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4262KiB/s (4364kB/s), 1048KiB/s-1126KiB/s (1073kB/s-1153kB/s), io=23.4MiB (24.5MB), run=5355-5620msec WRITE: bw=4484KiB/s (4592kB/s), 1073KiB/s-1194KiB/s (1099kB/s-1223kB/s), io=24.6MiB (25.8MB), run=5355-5620msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15696: Tue Feb 27 16:20:43 2024 read: IOPS=10, BW=164KiB/s (168kB/s)(5920KiB/36018msec) clat (usec): min=3404, max=80874, avg=10246.63, stdev=8076.09 lat (usec): min=3405, max=80876, avg=10248.65, stdev=8076.03 clat percentiles (usec): | 1.00th=[ 4113], 5.00th=[ 4752], 10.00th=[ 5407], 20.00th=[ 6194], | 30.00th=[ 6652], 40.00th=[ 7242], 50.00th=[ 8029], 60.00th=[ 8848], | 70.00th=[10159], 80.00th=[11863], 90.00th=[16909], 95.00th=[21627], | 99.00th=[57410], 99.50th=[69731], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 31, max= 416, per=24.52%, avg=161.82, stdev=85.28, samples=71 iops : min= 1, max= 26, avg= 9.63, stdev= 5.36, samples=71 write: IOPS=11, BW=177KiB/s (181kB/s)(6368KiB/36018msec) clat (msec): min=27, max=227, avg=80.90, stdev=29.94 lat (msec): min=27, max=227, avg=80.91, stdev=29.94 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 44], 10.00th=[ 49], 20.00th=[ 57], | 30.00th=[ 64], 40.00th=[ 69], 50.00th=[ 77], 60.00th=[ 83], | 70.00th=[ 91], 80.00th=[ 99], 90.00th=[ 117], 95.00th=[ 140], | 99.00th=[ 180], 99.50th=[ 203], 99.90th=[ 228], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 63, max= 287, per=25.18%, avg=174.76, stdev=46.50, samples=71 iops : min= 3, max= 17, avg=10.45, stdev= 2.95, samples=71 lat (msec) : 4=0.39%, 10=33.20%, 20=11.72%, 50=8.20%, 100=36.72% lat (msec) : 250=9.77% cpu : usr=0.09%, sys=2.67%, ctx=832, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15697: Tue Feb 27 16:20:43 2024 read: IOPS=9, BW=158KiB/s (162kB/s)(5744KiB/36272msec) clat (usec): min=3672, max=85470, avg=10462.29, stdev=9223.78 lat (usec): min=3673, max=85494, avg=10464.34, stdev=9224.36 clat percentiles (usec): | 1.00th=[ 4015], 5.00th=[ 4817], 10.00th=[ 5080], 20.00th=[ 5866], | 30.00th=[ 6390], 40.00th=[ 6915], 50.00th=[ 7635], 60.00th=[ 8586], | 70.00th=[ 9896], 80.00th=[11994], 90.00th=[17433], 95.00th=[25560], | 99.00th=[47449], 99.50th=[77071], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 32, max= 447, per=24.66%, avg=162.78, stdev=100.00, samples=69 iops : min= 2, max= 27, avg= 9.77, stdev= 6.19, samples=69 write: IOPS=11, BW=180KiB/s (185kB/s)(6544KiB/36272msec) clat (msec): min=26, max=213, avg=79.43, stdev=28.56 lat (msec): min=26, max=213, avg=79.43, stdev=28.56 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 42], 10.00th=[ 48], 20.00th=[ 55], | 30.00th=[ 62], 40.00th=[ 68], 50.00th=[ 77], 60.00th=[ 84], | 70.00th=[ 91], 80.00th=[ 102], 90.00th=[ 116], 95.00th=[ 132], | 99.00th=[ 169], 99.50th=[ 180], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 95, max= 256, per=25.68%, avg=178.19, stdev=39.37, samples=72 iops : min= 5, max= 16, avg=10.69, stdev= 2.51, samples=72 lat (msec) : 4=0.39%, 10=32.68%, 20=10.16%, 50=9.51%, 100=36.46% lat (msec) : 250=10.81% cpu : usr=0.07%, sys=2.61%, ctx=845, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15699: Tue Feb 27 16:20:43 2024 read: IOPS=11, BW=182KiB/s (186kB/s)(6256KiB/34379msec) clat (usec): min=3179, max=86217, avg=10562.17, stdev=9930.01 lat (usec): min=3179, max=86218, avg=10564.07, stdev=9929.94 clat percentiles (usec): | 1.00th=[ 3851], 5.00th=[ 4686], 10.00th=[ 5211], 20.00th=[ 5866], | 30.00th=[ 6521], 40.00th=[ 7111], 50.00th=[ 7898], 60.00th=[ 8979], | 70.00th=[10028], 80.00th=[11469], 90.00th=[16909], 95.00th=[25297], | 99.00th=[62129], 99.50th=[81265], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 31, max= 448, per=27.76%, avg=183.19, stdev=100.42, samples=68 iops : min= 1, max= 28, avg=11.13, stdev= 6.33, samples=68 write: IOPS=10, BW=175KiB/s (180kB/s)(6032KiB/34379msec) clat (msec): min=32, max=209, avg=80.15, stdev=29.89 lat (msec): min=32, max=209, avg=80.15, stdev=29.89 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 42], 10.00th=[ 48], 20.00th=[ 55], | 30.00th=[ 61], 40.00th=[ 68], 50.00th=[ 77], 60.00th=[ 84], | 70.00th=[ 93], 80.00th=[ 103], 90.00th=[ 122], 95.00th=[ 138], | 99.00th=[ 171], 99.50th=[ 205], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 64, max= 287, per=25.17%, avg=174.71, stdev=46.10, samples=68 iops : min= 4, max= 17, avg=10.60, stdev= 2.91, samples=68 lat (msec) : 4=0.65%, 10=34.77%, 20=11.98%, 50=8.72%, 100=33.33% lat (msec) : 250=10.55% cpu : usr=0.03%, sys=2.80%, ctx=843, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15701: Tue Feb 27 16:20:43 2024 read: IOPS=10, BW=167KiB/s (172kB/s)(6032KiB/36015msec) clat (usec): min=3573, max=72215, avg=10414.78, stdev=9543.44 lat (usec): min=3575, max=72216, avg=10417.18, stdev=9543.29 clat percentiles (usec): | 1.00th=[ 3818], 5.00th=[ 4686], 10.00th=[ 5014], 20.00th=[ 5866], | 30.00th=[ 6521], 40.00th=[ 7046], 50.00th=[ 7898], 60.00th=[ 8455], | 70.00th=[ 9765], 80.00th=[11207], 90.00th=[16712], 95.00th=[22414], | 99.00th=[63701], 99.50th=[66323], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 31, max= 416, per=25.92%, avg=171.07, stdev=88.29, samples=70 iops : min= 1, max= 26, avg=10.37, stdev= 5.48, samples=70 write: IOPS=10, BW=174KiB/s (178kB/s)(6256KiB/36015msec) clat (msec): min=37, max=213, avg=82.02, stdev=28.12 lat (msec): min=37, max=213, avg=82.02, stdev=28.12 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 47], 10.00th=[ 51], 20.00th=[ 59], | 30.00th=[ 66], 40.00th=[ 71], 50.00th=[ 79], 60.00th=[ 86], | 70.00th=[ 92], 80.00th=[ 101], 90.00th=[ 118], 95.00th=[ 133], | 99.00th=[ 188], 99.50th=[ 203], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 94, max= 256, per=24.99%, avg=173.40, stdev=39.08, samples=72 iops : min= 5, max= 16, avg=10.51, stdev= 2.51, samples=72 lat (msec) : 4=0.78%, 10=34.64%, 20=10.55%, 50=7.03%, 100=36.33% lat (msec) : 250=10.68% cpu : usr=0.21%, sys=2.45%, ctx=843, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=660KiB/s (676kB/s), 158KiB/s-182KiB/s (162kB/s-186kB/s), io=23.4MiB (24.5MB), run=34379-36272msec WRITE: bw=695KiB/s (711kB/s), 174KiB/s-180KiB/s (178kB/s-185kB/s), io=24.6MiB (25.8MB), run=34379-36272msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15736: Tue Feb 27 16:20:45 2024 read: IOPS=10, BW=10.3MiB/s (10.8MB/s)(7168KiB/679msec) clat (usec): min=551, max=105702, avg=63975.05, stdev=34968.24 lat (usec): min=552, max=105704, avg=63976.88, stdev=34968.54 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 553], 10.00th=[ 553], 20.00th=[ 45876], | 30.00th=[ 57410], 40.00th=[ 57410], 50.00th=[ 67634], 60.00th=[ 73925], | 70.00th=[ 73925], 80.00th=[ 96994], 90.00th=[105382], 95.00th=[105382], | 99.00th=[105382], 99.50th=[105382], 99.90th=[105382], 99.95th=[105382], | 99.99th=[105382] bw ( KiB/s): min=10240, max=10240, per=59.40%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7541KiB/s (7721kB/s)(5120KiB/679msec) clat (usec): min=22675, max=62922, avg=44376.67, stdev=16661.11 lat (usec): min=22830, max=62941, avg=44464.64, stdev=16624.52 clat percentiles (usec): | 1.00th=[22676], 5.00th=[22676], 10.00th=[22676], 20.00th=[22676], | 30.00th=[33162], 40.00th=[33162], 50.00th=[45876], 60.00th=[45876], | 70.00th=[57410], 80.00th=[57410], 90.00th=[63177], 95.00th=[63177], | 99.00th=[63177], 99.50th=[63177], 99.90th=[63177], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 6144, max= 6144, per=25.46%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=33.33%, 100=50.00%, 250=8.33% cpu : usr=0.00%, sys=36.43%, ctx=131, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15737: Tue Feb 27 16:20:45 2024 read: IOPS=1, BW=1829KiB/s (1872kB/s)(2048KiB/1120msec) clat (msec): min=89, max=133, avg=111.57, stdev=30.61 lat (msec): min=89, max=133, avg=111.58, stdev=30.61 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 90], 10.00th=[ 90], 20.00th=[ 90], | 30.00th=[ 90], 40.00th=[ 90], 50.00th=[ 90], 60.00th=[ 134], | 70.00th=[ 134], 80.00th=[ 134], 90.00th=[ 134], 95.00th=[ 134], | 99.00th=[ 134], 99.50th=[ 134], 99.90th=[ 134], 99.95th=[ 134], | 99.99th=[ 134] bw ( KiB/s): min= 1723, max= 2043, per=10.92%, avg=1883.00, stdev=226.27, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=8, BW=9143KiB/s (9362kB/s)(10.0MiB/1120msec) clat (msec): min=41, max=187, avg=89.34, stdev=50.29 lat (msec): min=41, max=188, avg=89.44, stdev=50.28 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 51], | 30.00th=[ 51], 40.00th=[ 59], 50.00th=[ 68], 60.00th=[ 81], | 70.00th=[ 104], 80.00th=[ 104], 90.00th=[ 167], 95.00th=[ 188], | 99.00th=[ 188], 99.50th=[ 188], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 6131, max=10343, per=34.13%, avg=8237.00, stdev=2978.33, samples=2 iops : min= 5, max= 10, avg= 7.50, stdev= 3.54, samples=2 lat (msec) : 50=8.33%, 100=58.33%, 250=33.33% cpu : usr=0.18%, sys=22.70%, ctx=86, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15738: Tue Feb 27 16:20:45 2024 read: IOPS=5, BW=5172KiB/s (5296kB/s)(6144KiB/1188msec) clat (usec): min=612, max=125741, avg=68989.52, stdev=59253.48 lat (usec): min=614, max=125742, avg=68991.37, stdev=59253.41 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 611], 10.00th=[ 611], 20.00th=[ 627], | 30.00th=[ 627], 40.00th=[ 52691], 50.00th=[ 52691], 60.00th=[110625], | 70.00th=[123208], 80.00th=[123208], 90.00th=[125305], 95.00th=[125305], | 99.00th=[125305], 99.50th=[125305], 99.90th=[125305], 99.95th=[125305], | 99.99th=[125305] bw ( KiB/s): min= 8707, max= 8707, per=50.51%, avg=8707.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5172KiB/s (5296kB/s)(6144KiB/1188msec) clat (msec): min=25, max=207, avg=128.15, stdev=73.91 lat (msec): min=25, max=207, avg=128.25, stdev=73.92 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 140], 50.00th=[ 140], 60.00th=[ 165], | 70.00th=[ 182], 80.00th=[ 182], 90.00th=[ 207], 95.00th=[ 207], | 99.00th=[ 207], 99.50th=[ 207], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 3482, max= 6131, per=19.92%, avg=4806.50, stdev=1873.13, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 lat (usec) : 750=16.67% lat (msec) : 50=16.67%, 100=8.33%, 250=58.33% cpu : usr=0.25%, sys=21.80%, ctx=82, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15739: Tue Feb 27 16:20:45 2024 read: IOPS=5, BW=5356KiB/s (5484kB/s)(5120KiB/956msec) clat (usec): min=671, max=120092, avg=73453.27, stdev=45706.91 lat (usec): min=672, max=120093, avg=73455.08, stdev=45706.62 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 668], 10.00th=[ 668], 20.00th=[ 668], | 30.00th=[ 70779], 40.00th=[ 70779], 50.00th=[ 72877], 60.00th=[ 72877], | 70.00th=[103285], 80.00th=[103285], 90.00th=[120062], 95.00th=[120062], | 99.00th=[120062], 99.50th=[120062], 99.90th=[120062], 99.95th=[120062], | 99.99th=[120062] bw ( KiB/s): min= 4063, max= 4063, per=23.57%, avg=4063.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7498KiB/s (7678kB/s)(7168KiB/956msec) clat (msec): min=25, max=199, avg=83.85, stdev=70.62 lat (msec): min=25, max=199, avg=83.94, stdev=70.61 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 26], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 35], 60.00th=[ 128], | 70.00th=[ 128], 80.00th=[ 140], 90.00th=[ 201], 95.00th=[ 201], | 99.00th=[ 201], 99.50th=[ 201], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 8126, max= 8126, per=33.67%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=33.33%, 100=16.67%, 250=41.67% cpu : usr=0.10%, sys=23.33%, ctx=187, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.8MiB/s (17.7MB/s), 1829KiB/s-10.3MiB/s (1872kB/s-10.8MB/s), io=20.0MiB (20.0MB), run=679-1188msec WRITE: bw=23.6MiB/s (24.7MB/s), 5172KiB/s-9143KiB/s (5296kB/s-9362kB/s), io=28.0MiB (29.4MB), run=679-1188msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15741: Tue Feb 27 16:20:46 2024 read: IOPS=5, BW=5373KiB/s (5502kB/s)(7168KiB/1334msec) clat (msec): min=44, max=184, avg=81.58, stdev=50.59 lat (msec): min=44, max=184, avg=81.59, stdev=50.59 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 53], 50.00th=[ 58], 60.00th=[ 72], | 70.00th=[ 72], 80.00th=[ 111], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 4087, max= 6144, per=39.99%, avg=5115.50, stdev=1454.52, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 write: IOPS=3, BW=3838KiB/s (3930kB/s)(5120KiB/1334msec) clat (msec): min=80, max=152, avg=118.99, stdev=25.89 lat (msec): min=80, max=152, avg=119.10, stdev=25.90 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 81], 10.00th=[ 81], 20.00th=[ 81], | 30.00th=[ 116], 40.00th=[ 116], 50.00th=[ 120], 60.00th=[ 120], | 70.00th=[ 126], 80.00th=[ 126], 90.00th=[ 153], 95.00th=[ 153], | 99.00th=[ 153], 99.50th=[ 153], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 4096, max= 4096, per=22.87%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=16.67%, 100=33.33%, 250=50.00% cpu : usr=0.08%, sys=12.68%, ctx=55, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15742: Tue Feb 27 16:20:46 2024 read: IOPS=1, BW=1279KiB/s (1310kB/s)(2048KiB/1601msec) clat (msec): min=40, max=152, avg=96.72, stdev=79.41 lat (msec): min=40, max=152, avg=96.72, stdev=79.41 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 41], 60.00th=[ 153], | 70.00th=[ 153], 80.00th=[ 153], 90.00th=[ 153], 95.00th=[ 153], | 99.00th=[ 153], 99.50th=[ 153], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 2048, max= 2048, per=16.01%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=6, BW=6396KiB/s (6550kB/s)(10.0MiB/1601msec) clat (msec): min=59, max=291, avg=126.36, stdev=63.51 lat (msec): min=59, max=291, avg=126.45, stdev=63.52 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 77], | 30.00th=[ 97], 40.00th=[ 103], 50.00th=[ 112], 60.00th=[ 122], | 70.00th=[ 138], 80.00th=[ 138], 90.00th=[ 138], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 8192, max=10240, per=51.46%, avg=9216.00, stdev=1448.15, samples=2 iops : min= 8, max= 10, avg= 9.00, stdev= 1.41, samples=2 lat (msec) : 50=8.33%, 100=25.00%, 250=58.33%, 500=8.33% cpu : usr=0.19%, sys=9.06%, ctx=58, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15743: Tue Feb 27 16:20:46 2024 read: IOPS=3, BW=4024KiB/s (4120kB/s)(6144KiB/1527msec) clat (msec): min=35, max=132, avg=65.74, stdev=36.10 lat (msec): min=35, max=132, avg=65.74, stdev=36.10 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 45], | 30.00th=[ 45], 40.00th=[ 48], 50.00th=[ 48], 60.00th=[ 54], | 70.00th=[ 82], 80.00th=[ 82], 90.00th=[ 132], 95.00th=[ 132], | 99.00th=[ 132], 99.50th=[ 132], 99.90th=[ 132], 99.95th=[ 132], | 99.99th=[ 132] bw ( KiB/s): min= 2048, max= 8192, per=40.03%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=3, BW=4024KiB/s (4120kB/s)(6144KiB/1527msec) clat (msec): min=82, max=309, avg=166.29, stdev=93.93 lat (msec): min=82, max=309, avg=166.44, stdev=93.96 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 83], 10.00th=[ 83], 20.00th=[ 84], | 30.00th=[ 84], 40.00th=[ 111], 50.00th=[ 111], 60.00th=[ 163], | 70.00th=[ 249], 80.00th=[ 249], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 2048, max= 6144, per=22.87%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (msec) : 50=25.00%, 100=33.33%, 250=33.33%, 500=8.33% cpu : usr=0.00%, sys=9.50%, ctx=64, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15744: Tue Feb 27 16:20:46 2024 read: IOPS=3, BW=3238KiB/s (3316kB/s)(5120KiB/1581msec) clat (usec): min=39348, max=70998, avg=54968.71, stdev=13481.86 lat (usec): min=39349, max=71000, avg=54973.44, stdev=13482.98 clat percentiles (usec): | 1.00th=[39584], 5.00th=[39584], 10.00th=[39584], 20.00th=[39584], | 30.00th=[42730], 40.00th=[42730], 50.00th=[58983], 60.00th=[58983], | 70.00th=[62653], 80.00th=[62653], 90.00th=[70779], 95.00th=[70779], | 99.00th=[70779], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 8175, max= 8175, per=63.91%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4534KiB/s (4643kB/s)(7168KiB/1581msec) clat (msec): min=99, max=251, avg=163.45, stdev=68.53 lat (msec): min=99, max=251, avg=163.56, stdev=68.51 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 101], 10.00th=[ 101], 20.00th=[ 102], | 30.00th=[ 114], 40.00th=[ 114], 50.00th=[ 123], 60.00th=[ 218], | 70.00th=[ 218], 80.00th=[ 239], 90.00th=[ 251], 95.00th=[ 251], | 99.00th=[ 251], 99.50th=[ 251], 99.90th=[ 251], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 2048, max= 6144, per=22.86%, avg=4093.00, stdev=2048.01, samples=3 iops : min= 2, max= 6, avg= 3.67, stdev= 2.08, samples=3 lat (msec) : 50=16.67%, 100=33.33%, 250=41.67%, 500=8.33% cpu : usr=0.13%, sys=10.25%, ctx=59, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.5MiB/s (13.1MB/s), 1279KiB/s-5373KiB/s (1310kB/s-5502kB/s), io=20.0MiB (20.0MB), run=1334-1601msec WRITE: bw=17.5MiB/s (18.3MB/s), 3838KiB/s-6396KiB/s (3930kB/s-6550kB/s), io=28.0MiB (29.4MB), run=1334-1601msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15751: Tue Feb 27 16:20:49 2024 read: IOPS=4, BW=19.7MiB/s (20.7MB/s)(12.0MiB/608msec) clat (msec): min=99, max=337, avg=200.41, stdev=123.15 lat (msec): min=99, max=337, avg=200.42, stdev=123.15 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 100], 10.00th=[ 100], 20.00th=[ 100], | 30.00th=[ 100], 40.00th=[ 165], 50.00th=[ 165], 60.00th=[ 165], | 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min=16384, max=16384, per=88.50%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.16%, sys=34.76%, ctx=124, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15752: Tue Feb 27 16:20:49 2024 read: IOPS=1, BW=4628KiB/s (4739kB/s)(4096KiB/885msec) clat (nsec): min=249064k, max=249064k, avg=249064026.00, stdev= 0.00 lat (nsec): min=249066k, max=249066k, avg=249065563.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 249], 10.00th=[ 249], 20.00th=[ 249], | 30.00th=[ 249], 40.00th=[ 249], 50.00th=[ 249], 60.00th=[ 249], | 70.00th=[ 249], 80.00th=[ 249], 90.00th=[ 249], 95.00th=[ 249], | 99.00th=[ 249], 99.50th=[ 249], 99.90th=[ 249], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min= 8159, max= 8159, per=44.07%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9256KiB/s (9479kB/s)(8192KiB/885msec) clat (msec): min=182, max=451, avg=316.84, stdev=189.95 lat (msec): min=182, max=451, avg=317.24, stdev=189.89 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 451], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=27.94%, ctx=93, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15753: Tue Feb 27 16:20:49 2024 write: IOPS=2, BW=8474KiB/s (8678kB/s)(12.0MiB/1450msec) clat (msec): min=295, max=716, avg=482.64, stdev=214.62 lat (msec): min=295, max=717, avg=483.00, stdev=214.75 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 296], 10.00th=[ 296], 20.00th=[ 296], | 30.00th=[ 296], 40.00th=[ 435], 50.00th=[ 435], 60.00th=[ 435], | 70.00th=[ 718], 80.00th=[ 718], 90.00th=[ 718], 95.00th=[ 718], | 99.00th=[ 718], 99.50th=[ 718], 99.90th=[ 718], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 8175, max= 8175, per=36.18%, avg=8175.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.00%, sys=16.36%, ctx=71, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15756: Tue Feb 27 16:20:49 2024 write: IOPS=2, BW=11.5MiB/s (12.1MB/s)(12.0MiB/1039msec) clat (msec): min=237, max=429, avg=344.37, stdev=97.90 lat (msec): min=238, max=429, avg=344.78, stdev=97.77 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 239], 10.00th=[ 239], 20.00th=[ 239], | 30.00th=[ 239], 40.00th=[ 368], 50.00th=[ 368], 60.00th=[ 368], | 70.00th=[ 430], 80.00th=[ 430], 90.00th=[ 430], 95.00th=[ 430], | 99.00th=[ 430], 99.50th=[ 430], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 8192, max= 8192, per=36.25%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.19%, sys=25.24%, ctx=76, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.1MiB/s (18.0MB/s), 4628KiB/s-19.7MiB/s (4739kB/s-20.7MB/s), io=16.0MiB (16.8MB), run=608-885msec WRITE: bw=22.1MiB/s (23.1MB/s), 8474KiB/s-11.5MiB/s (8678kB/s-12.1MB/s), io=32.0MiB (33.6MB), run=885-1450msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15754: Tue Feb 27 16:20:49 2024 read: IOPS=3, BW=15.3MiB/s (16.0MB/s)(12.0MiB/785msec) clat (msec): min=207, max=331, avg=260.85, stdev=63.46 lat (msec): min=207, max=331, avg=260.85, stdev=63.46 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 209], 10.00th=[ 209], 20.00th=[ 209], | 30.00th=[ 209], 40.00th=[ 245], 50.00th=[ 245], 60.00th=[ 245], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=14.41%, ctx=41, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15755: Tue Feb 27 16:20:49 2024 read: IOPS=0, BW=2800KiB/s (2867kB/s)(4096KiB/1463msec) clat (nsec): min=690454k, max=690454k, avg=690454175.00, stdev= 0.00 lat (nsec): min=690456k, max=690456k, avg=690455730.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 693], 5.00th=[ 693], 10.00th=[ 693], 20.00th=[ 693], | 30.00th=[ 693], 40.00th=[ 693], 50.00th=[ 693], 60.00th=[ 693], | 70.00th=[ 693], 80.00th=[ 693], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 8192, max= 8192, per=73.16%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5599KiB/s (5734kB/s)(8192KiB/1463msec) clat (msec): min=177, max=593, avg=385.23, stdev=293.85 lat (msec): min=177, max=593, avg=385.54, stdev=294.00 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 178], 10.00th=[ 178], 20.00th=[ 178], | 30.00th=[ 178], 40.00th=[ 178], 50.00th=[ 178], 60.00th=[ 592], | 70.00th=[ 592], 80.00th=[ 592], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] lat (msec) : 250=33.33%, 750=66.67% cpu : usr=0.27%, sys=7.05%, ctx=47, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15757: Tue Feb 27 16:20:49 2024 write: IOPS=1, BW=7797KiB/s (7984kB/s)(12.0MiB/1576msec) clat (msec): min=191, max=873, avg=524.17, stdev=341.71 lat (msec): min=192, max=874, avg=524.98, stdev=341.08 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 192], 10.00th=[ 192], 20.00th=[ 192], | 30.00th=[ 192], 40.00th=[ 510], 50.00th=[ 510], 60.00th=[ 510], | 70.00th=[ 877], 80.00th=[ 877], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 8094, max= 8192, per=39.74%, avg=8143.00, stdev=69.30, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.06%, sys=8.70%, ctx=48, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15758: Tue Feb 27 16:20:49 2024 write: IOPS=1, BW=7685KiB/s (7869kB/s)(12.0MiB/1599msec) clat (msec): min=138, max=1120, avg=532.23, stdev=519.01 lat (msec): min=138, max=1120, avg=532.57, stdev=518.99 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 138], 10.00th=[ 138], 20.00th=[ 138], | 30.00th=[ 138], 40.00th=[ 338], 50.00th=[ 338], 60.00th=[ 338], | 70.00th=[ 1116], 80.00th=[ 1116], 90.00th=[ 1116], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min=16384, max=16384, per=79.95%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.19%, sys=8.51%, ctx=52, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.9MiB/s (11.5MB/s), 2800KiB/s-15.3MiB/s (2867kB/s-16.0MB/s), io=16.0MiB (16.8MB), run=785-1463msec WRITE: bw=20.0MiB/s (20.0MB/s), 5599KiB/s-7797KiB/s (5734kB/s-7984kB/s), io=32.0MiB (33.6MB), run=1463-1599msec