== sanity test 398b: DIO and buffer IO race ============== 22:18:07 (1746152287) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 3.81419 s, 13.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24178: Thu May 1 22:19:19 2025 read: IOPS=23, BW=93.0KiB/s (96.2kB/s)(5992KiB/63766msec) clat (usec): min=66, max=531206, avg=24062.84, stdev=20447.20 lat (usec): min=67, max=531207, avg=24064.68, stdev=20447.53 clat percentiles (usec): | 1.00th=[ 100], 5.00th=[ 145], 10.00th=[ 7767], 20.00th=[ 11863], | 30.00th=[ 14615], 40.00th=[ 17957], 50.00th=[ 21103], 60.00th=[ 24773], | 70.00th=[ 29492], 80.00th=[ 34866], 90.00th=[ 43254], 95.00th=[ 50594], | 99.00th=[ 84411], 99.50th=[ 91751], 99.90th=[121111], 99.95th=[530580], | 99.99th=[530580] bw ( KiB/s): min= 31, max= 208, per=25.04%, avg=93.15, stdev=31.51, samples=127 iops : min= 7, max= 52, avg=22.89, stdev= 7.90, samples=127 write: IOPS=24, BW=98.7KiB/s (101kB/s)(6296KiB/63766msec) clat (usec): min=545, max=147866, avg=17410.28, stdev=18871.86 lat (usec): min=547, max=147868, avg=17415.78, stdev=18870.94 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 881], 10.00th=[ 1020], 20.00th=[ 1401], | 30.00th=[ 2966], 40.00th=[ 5997], 50.00th=[ 11994], 60.00th=[ 17433], | 70.00th=[ 22414], 80.00th=[ 30278], 90.00th=[ 43779], 95.00th=[ 55313], | 99.00th=[ 82314], 99.50th=[ 89654], 99.90th=[131597], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 23, max= 231, per=25.20%, avg=98.02, stdev=36.83, samples=127 iops : min= 5, max= 57, avg=24.10, stdev= 9.22, samples=127 lat (usec) : 100=0.49%, 250=2.64%, 500=0.59%, 750=0.42%, 1000=4.62% lat (msec) : 2=7.94%, 4=5.05%, 10=9.34%, 20=25.46%, 50=37.34% lat (msec) : 100=5.92%, 250=0.16%, 750=0.03% cpu : usr=0.13%, sys=8.93%, ctx=5290, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24179: Thu May 1 22:19:19 2025 read: IOPS=23, BW=92.5KiB/s (94.7kB/s)(5940KiB/64205msec) clat (usec): min=56, max=538498, avg=23893.59, stdev=20959.77 lat (usec): min=57, max=538499, avg=23895.16, stdev=20959.87 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 147], 10.00th=[ 8455], 20.00th=[ 11731], | 30.00th=[ 14484], 40.00th=[ 17695], 50.00th=[ 20317], 60.00th=[ 23725], | 70.00th=[ 28967], 80.00th=[ 34341], 90.00th=[ 42730], 95.00th=[ 51119], | 99.00th=[ 78119], 99.50th=[101188], 99.90th=[158335], 99.95th=[541066], | 99.99th=[541066] bw ( KiB/s): min= 24, max= 160, per=24.68%, avg=91.80, stdev=30.12, samples=128 iops : min= 6, max= 40, avg=22.53, stdev= 7.54, samples=128 write: IOPS=24, BW=98.9KiB/s (101kB/s)(6348KiB/64205msec) clat (usec): min=484, max=137869, avg=17910.06, stdev=19423.06 lat (usec): min=484, max=137870, avg=17915.19, stdev=19422.01 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 873], 10.00th=[ 996], 20.00th=[ 1385], | 30.00th=[ 2704], 40.00th=[ 5473], 50.00th=[ 12518], 60.00th=[ 17957], | 70.00th=[ 22938], 80.00th=[ 32113], 90.00th=[ 45351], 95.00th=[ 54789], | 99.00th=[ 82314], 99.50th=[ 93848], 99.90th=[132645], 99.95th=[137364], | 99.99th=[137364] bw ( KiB/s): min= 31, max= 248, per=25.15%, avg=97.85, stdev=37.33, samples=128 iops : min= 7, max= 62, avg=24.05, stdev= 9.35, samples=128 lat (usec) : 100=0.39%, 250=2.47%, 500=0.59%, 750=0.68%, 1000=4.79% lat (msec) : 2=8.11%, 4=5.08%, 10=8.33%, 20=26.60%, 50=36.65% lat (msec) : 100=5.83%, 250=0.46%, 750=0.03% cpu : usr=0.14%, sys=8.83%, ctx=5280, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24181: Thu May 1 22:19:19 2025 read: IOPS=23, BW=93.5KiB/s (95.8kB/s)(6028KiB/64465msec) clat (usec): min=54, max=186378, avg=23960.18, stdev=16170.27 lat (usec): min=55, max=186378, avg=23966.24, stdev=16168.90 clat percentiles (usec): | 1.00th=[ 109], 5.00th=[ 217], 10.00th=[ 8455], 20.00th=[ 12387], | 30.00th=[ 15270], 40.00th=[ 18220], 50.00th=[ 21103], 60.00th=[ 24773], | 70.00th=[ 28967], 80.00th=[ 34866], 90.00th=[ 42206], 95.00th=[ 50070], | 99.00th=[ 79168], 99.50th=[101188], 99.90th=[127402], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 15, max= 175, per=24.83%, avg=92.36, stdev=29.55, samples=128 iops : min= 3, max= 43, avg=22.61, stdev= 7.42, samples=128 write: IOPS=24, BW=97.1KiB/s (99.4kB/s)(6260KiB/64465msec) clat (usec): min=588, max=531332, avg=17915.44, stdev=22786.63 lat (usec): min=590, max=531333, avg=17921.44, stdev=22786.00 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 873], 10.00th=[ 1012], 20.00th=[ 1352], | 30.00th=[ 3097], 40.00th=[ 6128], 50.00th=[ 12518], 60.00th=[ 17433], | 70.00th=[ 22676], 80.00th=[ 30802], 90.00th=[ 44303], 95.00th=[ 55837], | 99.00th=[ 79168], 99.50th=[ 87557], 99.90th=[124257], 99.95th=[530580], | 99.99th=[530580] bw ( KiB/s): min= 31, max= 183, per=24.64%, avg=95.84, stdev=32.05, samples=128 iops : min= 7, max= 45, avg=23.48, stdev= 8.04, samples=128 lat (usec) : 100=0.20%, 250=2.34%, 500=0.20%, 750=0.55%, 1000=4.46% lat (msec) : 2=8.14%, 4=4.82%, 10=9.47%, 20=25.98%, 50=37.79% lat (msec) : 100=5.63%, 250=0.39%, 750=0.03% cpu : usr=0.11%, sys=8.95%, ctx=5412, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24182: Thu May 1 22:19:19 2025 read: IOPS=23, BW=94.0KiB/s (96.3kB/s)(6052KiB/64367msec) clat (usec): min=54, max=164996, avg=23834.62, stdev=16646.97 lat (usec): min=54, max=164998, avg=23837.27, stdev=16647.95 clat percentiles (usec): | 1.00th=[ 98], 5.00th=[ 141], 10.00th=[ 8717], 20.00th=[ 11994], | 30.00th=[ 14746], 40.00th=[ 17171], 50.00th=[ 20841], 60.00th=[ 24511], | 70.00th=[ 28967], 80.00th=[ 34341], 90.00th=[ 43254], 95.00th=[ 50594], | 99.00th=[ 84411], 99.50th=[101188], 99.90th=[139461], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 23, max= 199, per=25.05%, avg=93.18, stdev=32.80, samples=128 iops : min= 5, max= 49, avg=22.83, stdev= 8.27, samples=128 write: IOPS=24, BW=96.9KiB/s (99.2kB/s)(6236KiB/64368msec) clat (usec): min=657, max=527163, avg=17958.00, stdev=23389.05 lat (usec): min=659, max=527165, avg=17962.82, stdev=23388.81 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 881], 10.00th=[ 996], 20.00th=[ 1287], | 30.00th=[ 2540], 40.00th=[ 5473], 50.00th=[ 11731], 60.00th=[ 16909], | 70.00th=[ 22938], 80.00th=[ 31851], 90.00th=[ 44827], 95.00th=[ 57410], | 99.00th=[ 79168], 99.50th=[ 94897], 99.90th=[149947], 99.95th=[526386], | 99.99th=[526386] bw ( KiB/s): min= 31, max= 206, per=24.69%, avg=96.04, stdev=34.16, samples=128 iops : min= 7, max= 51, avg=23.55, stdev= 8.58, samples=128 lat (usec) : 100=0.62%, 250=2.51%, 500=0.23%, 750=0.39%, 1000=4.85% lat (msec) : 2=8.69%, 4=4.46%, 10=8.59%, 20=26.14%, 50=37.30% lat (msec) : 100=5.73%, 250=0.46%, 750=0.03% cpu : usr=0.14%, sys=8.92%, ctx=5304, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=372KiB/s (381kB/s), 92.5KiB/s-94.0KiB/s (94.7kB/s-96.3kB/s), io=23.4MiB (24.6MB), run=63766-64465msec WRITE: bw=390KiB/s (399kB/s), 96.9KiB/s-98.9KiB/s (99.2kB/s-101kB/s), io=24.6MiB (25.7MB), run=63766-64465msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24183: Thu May 1 22:19:42 2025 read: IOPS=17, BW=68.1KiB/s (69.8kB/s)(5992KiB/87963msec) clat (msec): min=3, max=466, avg=22.32, stdev=24.28 lat (msec): min=3, max=466, avg=22.33, stdev=24.28 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 17], | 70.00th=[ 22], 80.00th=[ 31], 90.00th=[ 52], 95.00th=[ 68], | 99.00th=[ 113], 99.50th=[ 130], 99.90th=[ 178], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 7, max= 208, per=25.03%, avg=67.83, stdev=45.61, samples=174 iops : min= 1, max= 52, avg=16.63, stdev=11.44, samples=174 write: IOPS=17, BW=71.6KiB/s (73.3kB/s)(6296KiB/87963msec) clat (msec): min=5, max=212, avg=34.40, stdev=24.04 lat (msec): min=5, max=212, avg=34.40, stdev=24.04 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 22], 50.00th=[ 26], 60.00th=[ 32], | 70.00th=[ 40], 80.00th=[ 51], 90.00th=[ 67], 95.00th=[ 82], | 99.00th=[ 114], 99.50th=[ 140], 99.90th=[ 205], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 8, max= 207, per=25.08%, avg=71.23, stdev=44.94, samples=174 iops : min= 2, max= 51, avg=17.48, stdev=11.29, samples=174 lat (msec) : 4=0.13%, 10=15.62%, 20=34.77%, 50=33.69%, 100=14.19% lat (msec) : 250=1.56%, 500=0.03% cpu : usr=0.12%, sys=4.35%, ctx=3562, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24184: Thu May 1 22:19:42 2025 read: IOPS=16, BW=67.3KiB/s (68.9kB/s)(5940KiB/88282msec) clat (msec): min=2, max=509, avg=22.29, stdev=24.01 lat (msec): min=2, max=509, avg=22.29, stdev=24.01 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 17], | 70.00th=[ 23], 80.00th=[ 32], 90.00th=[ 50], 95.00th=[ 67], | 99.00th=[ 101], 99.50th=[ 117], 99.90th=[ 140], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 7, max= 272, per=24.73%, avg=67.01, stdev=46.43, samples=175 iops : min= 1, max= 68, avg=16.34, stdev=11.64, samples=175 write: IOPS=17, BW=71.9KiB/s (73.6kB/s)(6348KiB/88282msec) clat (msec): min=6, max=209, avg=34.58, stdev=24.08 lat (msec): min=6, max=209, avg=34.58, stdev=24.08 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 17], | 30.00th=[ 20], 40.00th=[ 22], 50.00th=[ 26], 60.00th=[ 32], | 70.00th=[ 41], 80.00th=[ 52], 90.00th=[ 67], 95.00th=[ 82], | 99.00th=[ 121], 99.50th=[ 136], 99.90th=[ 192], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 15, max= 184, per=25.17%, avg=71.47, stdev=42.91, samples=175 iops : min= 3, max= 46, avg=17.46, stdev=10.77, samples=175 lat (msec) : 4=0.13%, 10=14.49%, 20=35.03%, 50=34.73%, 100=14.06% lat (msec) : 250=1.53%, 750=0.03% cpu : usr=0.11%, sys=4.32%, ctx=3559, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24185: Thu May 1 22:19:42 2025 read: IOPS=17, BW=68.2KiB/s (69.9kB/s)(6028KiB/88333msec) clat (msec): min=3, max=221, avg=22.18, stdev=21.58 lat (msec): min=3, max=221, avg=22.19, stdev=21.58 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 18], | 70.00th=[ 22], 80.00th=[ 31], 90.00th=[ 50], 95.00th=[ 68], | 99.00th=[ 109], 99.50th=[ 122], 99.90th=[ 161], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 8, max= 216, per=25.05%, avg=67.88, stdev=44.89, samples=175 iops : min= 2, max= 54, avg=16.58, stdev=11.29, samples=175 write: IOPS=17, BW=70.9KiB/s (72.6kB/s)(6260KiB/88333msec) clat (msec): min=6, max=438, avg=34.83, stdev=26.82 lat (msec): min=6, max=438, avg=34.83, stdev=26.82 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 19], 40.00th=[ 22], 50.00th=[ 26], 60.00th=[ 32], | 70.00th=[ 41], 80.00th=[ 53], 90.00th=[ 68], 95.00th=[ 83], | 99.00th=[ 124], 99.50th=[ 148], 99.90th=[ 222], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 7, max= 184, per=24.78%, avg=70.37, stdev=44.87, samples=175 iops : min= 1, max= 46, avg=17.21, stdev=11.28, samples=175 lat (msec) : 4=0.10%, 10=14.23%, 20=36.49%, 50=33.20%, 100=14.19% lat (msec) : 250=1.76%, 500=0.03% cpu : usr=0.09%, sys=4.35%, ctx=3545, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24186: Thu May 1 22:19:42 2025 read: IOPS=17, BW=68.8KiB/s (70.5kB/s)(6052KiB/87949msec) clat (msec): min=3, max=214, avg=21.54, stdev=21.37 lat (msec): min=3, max=214, avg=21.55, stdev=21.38 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 15], 60.00th=[ 17], | 70.00th=[ 22], 80.00th=[ 29], 90.00th=[ 46], 95.00th=[ 69], | 99.00th=[ 114], 99.50th=[ 128], 99.90th=[ 150], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 7, max= 224, per=25.19%, avg=68.28, stdev=44.94, samples=174 iops : min= 1, max= 56, avg=16.67, stdev=11.32, samples=174 write: IOPS=17, BW=70.9KiB/s (72.6kB/s)(6236KiB/87949msec) clat (msec): min=5, max=592, avg=35.35, stdev=27.88 lat (msec): min=5, max=592, avg=35.35, stdev=27.89 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 17], | 30.00th=[ 20], 40.00th=[ 23], 50.00th=[ 27], 60.00th=[ 33], | 70.00th=[ 42], 80.00th=[ 53], 90.00th=[ 68], 95.00th=[ 83], | 99.00th=[ 126], 99.50th=[ 140], 99.90th=[ 176], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 16, max= 224, per=24.83%, avg=70.53, stdev=43.16, samples=174 iops : min= 4, max= 56, avg=17.23, stdev=10.87, samples=174 lat (msec) : 4=0.03%, 10=15.46%, 20=34.60%, 50=34.37%, 100=13.64% lat (msec) : 250=1.86%, 750=0.03% cpu : usr=0.10%, sys=4.43%, ctx=3520, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=272KiB/s (278kB/s), 67.3KiB/s-68.8KiB/s (68.9kB/s-70.5kB/s), io=23.4MiB (24.6MB), run=87949-88333msec WRITE: bw=285KiB/s (291kB/s), 70.9KiB/s-71.9KiB/s (72.6kB/s-73.6kB/s), io=24.6MiB (25.7MB), run=87949-88333msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24275: Thu May 1 22:19:59 2025 read: IOPS=26, BW=421KiB/s (431kB/s)(5920KiB/14072msec) clat (usec): min=74, max=100048, avg=20323.44, stdev=14890.04 lat (usec): min=75, max=100050, avg=20325.34, stdev=14889.72 clat percentiles (usec): | 1.00th=[ 100], 5.00th=[ 127], 10.00th=[ 289], 20.00th=[ 9634], | 30.00th=[ 13042], 40.00th=[ 16450], 50.00th=[ 18220], 60.00th=[ 20841], | 70.00th=[ 24249], 80.00th=[ 29492], 90.00th=[ 36963], 95.00th=[ 45351], | 99.00th=[ 77071], 99.50th=[ 89654], 99.90th=[100140], 99.95th=[100140], | 99.99th=[100140] bw ( KiB/s): min= 224, max= 672, per=25.75%, avg=416.96, stdev=125.71, samples=28 iops : min= 14, max= 42, avg=25.68, stdev= 7.84, samples=28 write: IOPS=28, BW=453KiB/s (463kB/s)(6368KiB/14072msec) clat (usec): min=1132, max=123528, avg=16110.77, stdev=18651.62 lat (usec): min=1133, max=123531, avg=16124.89, stdev=18654.00 clat percentiles (usec): | 1.00th=[ 1221], 5.00th=[ 1369], 10.00th=[ 1631], 20.00th=[ 2212], | 30.00th=[ 3654], 40.00th=[ 5932], 50.00th=[ 9110], 60.00th=[ 14353], | 70.00th=[ 18482], 80.00th=[ 26084], 90.00th=[ 40109], 95.00th=[ 54264], | 99.00th=[ 98042], 99.50th=[102237], 99.90th=[123208], 99.95th=[123208], | 99.99th=[123208] bw ( KiB/s): min= 192, max= 926, per=26.43%, avg=450.14, stdev=182.18, samples=28 iops : min= 12, max= 57, avg=27.75, stdev=11.32, samples=28 lat (usec) : 100=0.52%, 250=4.17%, 500=0.91%, 750=0.39%, 1000=0.39% lat (msec) : 2=9.11%, 4=8.59%, 10=12.37%, 20=28.39%, 50=30.73% lat (msec) : 100=3.91%, 250=0.52% cpu : usr=0.10%, sys=10.81%, ctx=1385, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24276: Thu May 1 22:19:59 2025 read: IOPS=25, BW=409KiB/s (418kB/s)(5744KiB/14060msec) clat (usec): min=74, max=109532, avg=20470.04, stdev=16993.07 lat (usec): min=75, max=109533, avg=20478.76, stdev=16991.83 clat percentiles (usec): | 1.00th=[ 99], 5.00th=[ 116], 10.00th=[ 127], 20.00th=[ 8979], | 30.00th=[ 11731], 40.00th=[ 14353], 50.00th=[ 17695], 60.00th=[ 21627], | 70.00th=[ 26084], 80.00th=[ 31851], 90.00th=[ 39584], 95.00th=[ 50594], | 99.00th=[ 88605], 99.50th=[102237], 99.90th=[109577], 99.95th=[109577], | 99.99th=[109577] bw ( KiB/s): min= 63, max= 672, per=24.98%, avg=404.39, stdev=149.30, samples=28 iops : min= 3, max= 42, avg=24.79, stdev= 9.34, samples=28 write: IOPS=29, BW=465KiB/s (477kB/s)(6544KiB/14060msec) clat (usec): min=800, max=107782, avg=16168.76, stdev=17198.98 lat (usec): min=801, max=107785, avg=16177.30, stdev=17200.64 clat percentiles (usec): | 1.00th=[ 1123], 5.00th=[ 1319], 10.00th=[ 1532], 20.00th=[ 2147], | 30.00th=[ 3556], 40.00th=[ 6259], 50.00th=[ 10683], 60.00th=[ 14746], | 70.00th=[ 19530], 80.00th=[ 28443], 90.00th=[ 40633], 95.00th=[ 53216], | 99.00th=[ 76022], 99.50th=[ 79168], 99.90th=[107480], 99.95th=[107480], | 99.99th=[107480] bw ( KiB/s): min= 223, max= 701, per=27.03%, avg=460.29, stdev=149.22, samples=28 iops : min= 13, max= 43, avg=28.29, stdev= 9.40, samples=28 lat (usec) : 100=0.52%, 250=6.38%, 500=0.91%, 750=0.26%, 1000=0.52% lat (msec) : 2=9.24%, 4=7.81%, 10=11.33%, 20=26.82%, 50=30.60% lat (msec) : 100=5.21%, 250=0.39% cpu : usr=0.14%, sys=10.70%, ctx=1358, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24277: Thu May 1 22:19:59 2025 read: IOPS=26, BW=423KiB/s (433kB/s)(6256KiB/14793msec) clat (usec): min=74, max=97293, avg=19354.17, stdev=14843.85 lat (usec): min=75, max=97294, avg=19355.81, stdev=14843.79 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 125], 10.00th=[ 143], 20.00th=[ 8455], | 30.00th=[11338], 40.00th=[14222], 50.00th=[17433], 60.00th=[20579], | 70.00th=[24511], 80.00th=[29754], 90.00th=[38011], 95.00th=[46400], | 99.00th=[70779], 99.50th=[88605], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 63, max= 767, per=25.69%, avg=415.97, stdev=162.70, samples=29 iops : min= 3, max= 47, avg=25.55, stdev=10.21, samples=29 write: IOPS=25, BW=408KiB/s (418kB/s)(6032KiB/14793msec) clat (usec): min=674, max=179478, avg=18610.25, stdev=20724.07 lat (usec): min=676, max=179481, avg=18619.35, stdev=20722.41 clat percentiles (usec): | 1.00th=[ 1106], 5.00th=[ 1336], 10.00th=[ 1582], 20.00th=[ 2474], | 30.00th=[ 3654], 40.00th=[ 7046], 50.00th=[ 13042], 60.00th=[ 16188], | 70.00th=[ 23462], 80.00th=[ 32900], 90.00th=[ 46400], 95.00th=[ 61080], | 99.00th=[ 73925], 99.50th=[117965], 99.90th=[179307], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 127, max= 672, per=23.17%, avg=394.59, stdev=142.15, samples=29 iops : min= 7, max= 42, avg=24.24, stdev= 8.95, samples=29 lat (usec) : 100=0.65%, 250=6.12%, 500=1.30%, 750=0.13% lat (msec) : 2=8.20%, 4=8.07%, 10=10.03%, 20=27.47%, 50=32.55% lat (msec) : 100=5.08%, 250=0.39% cpu : usr=0.18%, sys=10.38%, ctx=1376, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24278: Thu May 1 22:19:59 2025 read: IOPS=26, BW=422KiB/s (432kB/s)(6032KiB/14307msec) clat (usec): min=78, max=88997, avg=19854.00, stdev=16145.82 lat (usec): min=78, max=88999, avg=19859.55, stdev=16150.13 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 122], 10.00th=[ 143], 20.00th=[ 7111], | 30.00th=[11994], 40.00th=[15139], 50.00th=[17433], 60.00th=[20579], | 70.00th=[24249], 80.00th=[30278], 90.00th=[37487], 95.00th=[50070], | 99.00th=[84411], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 128, max= 800, per=25.29%, avg=409.43, stdev=148.92, samples=28 iops : min= 8, max= 50, avg=25.18, stdev= 9.35, samples=28 write: IOPS=27, BW=437KiB/s (448kB/s)(6256KiB/14307msec) clat (usec): min=731, max=114807, avg=16855.11, stdev=17647.72 lat (usec): min=732, max=114822, avg=16857.70, stdev=17647.52 clat percentiles (usec): | 1.00th=[ 1029], 5.00th=[ 1303], 10.00th=[ 1500], 20.00th=[ 2147], | 30.00th=[ 3654], 40.00th=[ 5866], 50.00th=[ 11076], 60.00th=[ 15664], | 70.00th=[ 22414], 80.00th=[ 29492], 90.00th=[ 41157], 95.00th=[ 52167], | 99.00th=[ 80217], 99.50th=[ 86508], 99.90th=[114820], 99.95th=[114820], | 99.99th=[114820] bw ( KiB/s): min= 127, max= 733, per=25.12%, avg=427.82, stdev=156.98, samples=28 iops : min= 7, max= 45, avg=26.32, stdev= 9.88, samples=28 lat (usec) : 100=0.39%, 250=6.38%, 500=1.17%, 750=0.39%, 1000=0.26% lat (msec) : 2=9.11%, 4=7.94%, 10=10.94%, 20=26.30%, 50=31.90% lat (msec) : 100=5.08%, 250=0.13% cpu : usr=0.13%, sys=10.56%, ctx=1382, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1619KiB/s (1658kB/s), 409KiB/s-423KiB/s (418kB/s-433kB/s), io=23.4MiB (24.5MB), run=14060-14793msec WRITE: bw=1704KiB/s (1744kB/s), 408KiB/s-465KiB/s (418kB/s-477kB/s), io=24.6MiB (25.8MB), run=14060-14793msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24270: Thu May 1 22:20:08 2025 read: IOPS=16, BW=256KiB/s (263kB/s)(5920KiB/23086msec) clat (msec): min=4, max=102, avg=23.15, stdev=18.31 lat (msec): min=4, max=102, avg=23.16, stdev=18.31 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 20], | 70.00th=[ 23], 80.00th=[ 31], 90.00th=[ 51], 95.00th=[ 69], | 99.00th=[ 95], 99.50th=[ 102], 99.90th=[ 103], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min= 64, max= 544, per=25.00%, avg=255.22, stdev=116.99, samples=46 iops : min= 4, max= 34, avg=15.52, stdev= 7.33, samples=46 write: IOPS=17, BW=276KiB/s (282kB/s)(6368KiB/23086msec) clat (msec): min=8, max=129, avg=36.11, stdev=21.39 lat (msec): min=8, max=129, avg=36.12, stdev=21.40 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 16], 10.00th=[ 18], 20.00th=[ 21], | 30.00th=[ 24], 40.00th=[ 27], 50.00th=[ 30], 60.00th=[ 33], | 70.00th=[ 38], 80.00th=[ 46], 90.00th=[ 67], 95.00th=[ 85], | 99.00th=[ 122], 99.50th=[ 128], 99.90th=[ 130], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 64, max= 512, per=25.58%, avg=274.70, stdev=118.77, samples=46 iops : min= 4, max= 32, avg=16.74, stdev= 7.44, samples=46 lat (msec) : 10=8.46%, 20=30.08%, 50=47.40%, 100=12.50%, 250=1.56% cpu : usr=0.06%, sys=4.44%, ctx=879, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24271: Thu May 1 22:20:08 2025 read: IOPS=15, BW=245KiB/s (251kB/s)(5744KiB/23447msec) clat (msec): min=5, max=116, avg=22.13, stdev=18.19 lat (msec): min=5, max=116, avg=22.13, stdev=18.19 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 13], 40.00th=[ 14], 50.00th=[ 16], 60.00th=[ 18], | 70.00th=[ 23], 80.00th=[ 28], 90.00th=[ 47], 95.00th=[ 65], | 99.00th=[ 100], 99.50th=[ 107], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 31, max= 448, per=23.78%, avg=242.80, stdev=118.00, samples=46 iops : min= 1, max= 28, avg=14.74, stdev= 7.38, samples=46 write: IOPS=17, BW=279KiB/s (286kB/s)(6544KiB/23447msec) clat (msec): min=9, max=153, avg=37.44, stdev=21.27 lat (msec): min=9, max=153, avg=37.44, stdev=21.27 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 16], 10.00th=[ 18], 20.00th=[ 22], | 30.00th=[ 25], 40.00th=[ 28], 50.00th=[ 32], 60.00th=[ 36], | 70.00th=[ 42], 80.00th=[ 51], 90.00th=[ 70], 95.00th=[ 84], | 99.00th=[ 101], 99.50th=[ 117], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 127, max= 512, per=25.58%, avg=274.76, stdev=109.39, samples=46 iops : min= 7, max= 32, avg=16.74, stdev= 6.91, samples=46 lat (msec) : 10=7.68%, 20=31.38%, 50=46.48%, 100=13.54%, 250=0.91% cpu : usr=0.07%, sys=4.38%, ctx=891, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24272: Thu May 1 22:20:08 2025 read: IOPS=16, BW=269KiB/s (276kB/s)(6256KiB/23247msec) clat (msec): min=5, max=104, avg=22.54, stdev=17.44 lat (msec): min=5, max=104, avg=22.54, stdev=17.44 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 19], | 70.00th=[ 22], 80.00th=[ 31], 90.00th=[ 48], 95.00th=[ 65], | 99.00th=[ 84], 99.50th=[ 101], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 63, max= 672, per=26.50%, avg=270.59, stdev=138.14, samples=46 iops : min= 3, max= 42, avg=16.43, stdev= 8.78, samples=46 write: IOPS=16, BW=259KiB/s (266kB/s)(6032KiB/23247msec) clat (msec): min=7, max=146, avg=38.04, stdev=21.65 lat (msec): min=7, max=146, avg=38.04, stdev=21.65 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 18], 10.00th=[ 20], 20.00th=[ 23], | 30.00th=[ 25], 40.00th=[ 28], 50.00th=[ 32], 60.00th=[ 36], | 70.00th=[ 42], 80.00th=[ 51], 90.00th=[ 66], 95.00th=[ 82], | 99.00th=[ 128], 99.50th=[ 136], 99.90th=[ 146], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 63, max= 448, per=23.90%, avg=256.74, stdev=93.80, samples=46 iops : min= 3, max= 28, avg=15.57, stdev= 6.02, samples=46 lat (msec) : 10=7.55%, 20=32.03%, 50=45.44%, 100=13.41%, 250=1.56% cpu : usr=0.13%, sys=4.19%, ctx=909, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24273: Thu May 1 22:20:08 2025 read: IOPS=16, BW=261KiB/s (268kB/s)(6032KiB/23090msec) clat (msec): min=5, max=115, avg=21.99, stdev=17.71 lat (msec): min=5, max=115, avg=22.00, stdev=17.71 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 11], | 30.00th=[ 13], 40.00th=[ 14], 50.00th=[ 17], 60.00th=[ 19], | 70.00th=[ 22], 80.00th=[ 29], 90.00th=[ 43], 95.00th=[ 66], | 99.00th=[ 94], 99.50th=[ 114], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 63, max= 544, per=25.35%, avg=258.85, stdev=114.24, samples=46 iops : min= 3, max= 34, avg=15.72, stdev= 7.23, samples=46 write: IOPS=16, BW=271KiB/s (277kB/s)(6256KiB/23090msec) clat (msec): min=7, max=164, avg=37.74, stdev=23.16 lat (msec): min=7, max=164, avg=37.75, stdev=23.17 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 16], 10.00th=[ 19], 20.00th=[ 22], | 30.00th=[ 25], 40.00th=[ 28], 50.00th=[ 31], 60.00th=[ 35], | 70.00th=[ 40], 80.00th=[ 50], 90.00th=[ 68], 95.00th=[ 87], | 99.00th=[ 126], 99.50th=[ 130], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 96, max= 512, per=25.13%, avg=269.91, stdev=113.74, samples=46 iops : min= 6, max= 32, avg=16.41, stdev= 7.13, samples=46 lat (msec) : 10=7.81%, 20=31.64%, 50=46.61%, 100=11.85%, 250=2.08% cpu : usr=0.13%, sys=4.34%, ctx=889, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1022KiB/s (1046kB/s), 245KiB/s-269KiB/s (251kB/s-276kB/s), io=23.4MiB (24.5MB), run=23086-23447msec WRITE: bw=1075KiB/s (1101kB/s), 259KiB/s-279KiB/s (266kB/s-286kB/s), io=24.6MiB (25.8MB), run=23086-23447msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24307: Thu May 1 22:20:14 2025 read: IOPS=2, BW=2594KiB/s (2657kB/s)(7168KiB/2763msec) clat (usec): min=618, max=1522.5k, avg=320330.86, stdev=538195.60 lat (usec): min=619, max=1522.6k, avg=320332.88, stdev=538195.56 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 619], 10.00th=[ 619], | 20.00th=[ 2704], 30.00th=[ 114820], 40.00th=[ 114820], | 50.00th=[ 152044], 60.00th=[ 206570], 70.00th=[ 206570], | 80.00th=[ 242222], 90.00th=[1518339], 95.00th=[1518339], | 99.00th=[1518339], 99.50th=[1518339], 99.90th=[1518339], | 99.95th=[1518339], 99.99th=[1518339] bw ( KiB/s): min= 2039, max= 4071, per=59.97%, avg=3385.67, stdev=1166.31, samples=3 iops : min= 1, max= 3, avg= 2.33, stdev= 1.15, samples=3 write: IOPS=1, BW=1853KiB/s (1898kB/s)(5120KiB/2763msec) clat (msec): min=74, max=126, avg=94.43, stdev=20.91 lat (msec): min=74, max=126, avg=94.53, stdev=20.90 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 75], 10.00th=[ 75], 20.00th=[ 75], | 30.00th=[ 81], 40.00th=[ 81], 50.00th=[ 89], 60.00th=[ 89], | 70.00th=[ 103], 80.00th=[ 103], 90.00th=[ 127], 95.00th=[ 127], | 99.00th=[ 127], 99.50th=[ 127], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 4047, max= 4047, per=51.20%, avg=4047.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 4=8.33%, 100=25.00%, 250=50.00% cpu : usr=0.14%, sys=13.43%, ctx=440, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24309: Thu May 1 22:20:14 2025 read: IOPS=0, BW=565KiB/s (578kB/s)(2048KiB/3627msec) clat (msec): min=100, max=365, avg=232.75, stdev=187.24 lat (msec): min=100, max=365, avg=232.75, stdev=187.24 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 101], 10.00th=[ 101], 20.00th=[ 101], | 30.00th=[ 101], 40.00th=[ 101], 50.00th=[ 101], 60.00th=[ 368], | 70.00th=[ 368], 80.00th=[ 368], 90.00th=[ 368], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 2043, max= 2043, per=36.18%, avg=2043.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2823KiB/s (2891kB/s)(10.0MiB/3627msec) clat (msec): min=56, max=1386, avg=308.06, stdev=399.69 lat (msec): min=56, max=1386, avg=308.53, stdev=399.62 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 73], | 30.00th=[ 92], 40.00th=[ 142], 50.00th=[ 146], 60.00th=[ 194], | 70.00th=[ 268], 80.00th=[ 268], 90.00th=[ 493], 95.00th=[ 1385], | 99.00th=[ 1385], 99.50th=[ 1385], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 2019, max= 5616, per=49.57%, avg=3918.75, stdev=1474.13, samples=4 iops : min= 1, max= 5, avg= 3.00, stdev= 1.63, samples=4 lat (msec) : 100=25.00%, 250=41.67%, 500=25.00% cpu : usr=0.00%, sys=10.07%, ctx=259, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24310: Thu May 1 22:20:14 2025 read: IOPS=1, BW=1734KiB/s (1775kB/s)(6144KiB/3544msec) clat (usec): min=657, max=390063, avg=129375.81, stdev=141378.69 lat (usec): min=659, max=390065, avg=129377.38, stdev=141378.93 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 660], 10.00th=[ 660], 20.00th=[ 6915], | 30.00th=[ 6915], 40.00th=[112722], 50.00th=[112722], 60.00th=[125305], | 70.00th=[141558], 80.00th=[141558], 90.00th=[387974], 95.00th=[387974], | 99.00th=[387974], 99.50th=[387974], 99.90th=[387974], 99.95th=[387974], | 99.99th=[387974] bw ( KiB/s): min= 2039, max= 4087, per=59.19%, avg=3342.00, stdev=1132.30, samples=3 iops : min= 1, max= 3, avg= 2.33, stdev= 1.15, samples=3 write: IOPS=1, BW=1734KiB/s (1775kB/s)(6144KiB/3544msec) clat (msec): min=94, max=1664, avg=448.04, stdev=605.31 lat (msec): min=94, max=1664, avg=448.15, stdev=605.33 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 94], 10.00th=[ 94], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 207], 50.00th=[ 207], 60.00th=[ 218], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 1670], 95.00th=[ 1670], | 99.00th=[ 1670], 99.50th=[ 1670], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 1950, max= 4096, per=38.49%, avg=3043.00, stdev=1211.25, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.50, samples=4 lat (usec) : 750=8.33% lat (msec) : 10=8.33%, 100=8.33%, 250=50.00%, 500=16.67% cpu : usr=0.03%, sys=8.64%, ctx=286, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24311: Thu May 1 22:20:14 2025 read: IOPS=1, BW=1412KiB/s (1446kB/s)(5120KiB/3627msec) clat (usec): min=648, max=77257, avg=19021.43, stdev=32903.08 lat (usec): min=649, max=77259, avg=19022.90, stdev=32903.18 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 652], 10.00th=[ 652], 20.00th=[ 652], | 30.00th=[ 734], 40.00th=[ 734], 50.00th=[ 4113], 60.00th=[ 4113], | 70.00th=[12387], 80.00th=[12387], 90.00th=[77071], 95.00th=[77071], | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 8078, max= 8078, per=100.00%, avg=8078.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1976KiB/s (2024kB/s)(7168KiB/3627msec) clat (msec): min=63, max=1547, avg=471.44, stdev=498.35 lat (msec): min=63, max=1547, avg=471.53, stdev=498.37 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 215], | 30.00th=[ 262], 40.00th=[ 262], 50.00th=[ 317], 60.00th=[ 326], | 70.00th=[ 326], 80.00th=[ 567], 90.00th=[ 1552], 95.00th=[ 1552], | 99.00th=[ 1552], 99.50th=[ 1552], 99.90th=[ 1552], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 2043, max= 4087, per=38.64%, avg=3054.25, stdev=1164.97, samples=4 iops : min= 1, max= 3, avg= 2.25, stdev= 0.96, samples=4 lat (usec) : 750=16.67% lat (msec) : 10=8.33%, 20=8.33%, 100=16.67%, 250=8.33%, 500=25.00% lat (msec) : 750=8.33% cpu : usr=0.00%, sys=7.64%, ctx=150, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5647KiB/s (5782kB/s), 565KiB/s-2594KiB/s (578kB/s-2657kB/s), io=20.0MiB (20.0MB), run=2763-3627msec WRITE: bw=7905KiB/s (8095kB/s), 1734KiB/s-2823KiB/s (1775kB/s-2891kB/s), io=28.0MiB (29.4MB), run=2763-3627msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24304: Thu May 1 22:20:14 2025 read: IOPS=1, BW=1817KiB/s (1861kB/s)(7168KiB/3944msec) clat (msec): min=44, max=1583, avg=331.62, stdev=556.30 lat (msec): min=44, max=1583, avg=331.63, stdev=556.30 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 54], | 30.00th=[ 108], 40.00th=[ 108], 50.00th=[ 122], 60.00th=[ 165], | 70.00th=[ 165], 80.00th=[ 247], 90.00th=[ 1586], 95.00th=[ 1586], | 99.00th=[ 1586], 99.50th=[ 1586], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 2039, max= 6144, per=68.16%, avg=3408.67, stdev=2368.87, samples=3 iops : min= 1, max= 6, avg= 2.67, stdev= 2.89, samples=3 write: IOPS=1, BW=1298KiB/s (1329kB/s)(5120KiB/3944msec) clat (msec): min=81, max=500, avg=283.92, stdev=150.48 lat (msec): min=81, max=500, avg=284.02, stdev=150.49 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 82], 10.00th=[ 82], 20.00th=[ 82], | 30.00th=[ 239], 40.00th=[ 239], 50.00th=[ 292], 60.00th=[ 292], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 2035, max= 4079, per=38.86%, avg=2720.67, stdev=1176.37, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 lat (msec) : 50=8.33%, 100=16.67%, 250=41.67%, 500=16.67%, 750=8.33% cpu : usr=0.00%, sys=2.16%, ctx=46, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24305: Thu May 1 22:20:14 2025 read: IOPS=0, BW=621KiB/s (636kB/s)(2048KiB/3296msec) clat (msec): min=110, max=319, avg=214.91, stdev=147.70 lat (msec): min=110, max=319, avg=214.92, stdev=147.70 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 111], 50.00th=[ 111], 60.00th=[ 321], | 70.00th=[ 321], 80.00th=[ 321], 90.00th=[ 321], 95.00th=[ 321], | 99.00th=[ 321], 99.50th=[ 321], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 2043, max= 2048, per=40.90%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3107KiB/s (3181kB/s)(10.0MiB/3296msec) clat (msec): min=84, max=1360, avg=277.86, stdev=384.23 lat (msec): min=84, max=1360, avg=278.00, stdev=384.20 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 85], 10.00th=[ 85], 20.00th=[ 106], | 30.00th=[ 117], 40.00th=[ 120], 50.00th=[ 138], 60.00th=[ 169], | 70.00th=[ 234], 80.00th=[ 234], 90.00th=[ 241], 95.00th=[ 1368], | 99.00th=[ 1368], 99.50th=[ 1368], 99.90th=[ 1368], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min= 4096, max= 6144, per=77.95%, avg=5457.00, stdev=1178.68, samples=3 iops : min= 4, max= 6, avg= 5.00, stdev= 1.00, samples=3 lat (msec) : 100=8.33%, 250=75.00%, 500=8.33% cpu : usr=0.00%, sys=3.82%, ctx=45, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24306: Thu May 1 22:20:14 2025 read: IOPS=1, BW=1500KiB/s (1536kB/s)(6144KiB/4095msec) clat (msec): min=66, max=591, avg=247.62, stdev=232.86 lat (msec): min=66, max=591, avg=247.62, stdev=232.86 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 73], | 30.00th=[ 73], 40.00th=[ 75], 50.00th=[ 75], 60.00th=[ 194], | 70.00th=[ 485], 80.00th=[ 485], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 1408, max= 2048, per=37.71%, avg=1885.75, stdev=318.53, samples=4 iops : min= 1, max= 2, avg= 1.50, stdev= 0.58, samples=4 write: IOPS=1, BW=1500KiB/s (1536kB/s)(6144KiB/4095msec) clat (msec): min=30, max=1635, avg=427.39, stdev=604.91 lat (msec): min=30, max=1636, avg=427.54, stdev=604.90 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 86], | 30.00th=[ 86], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 271], | 70.00th=[ 376], 80.00th=[ 376], 90.00th=[ 1636], 95.00th=[ 1636], | 99.00th=[ 1636], 99.50th=[ 1636], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 1408, max= 4087, per=34.22%, avg=2395.50, stdev=1166.79, samples=4 iops : min= 1, max= 3, avg= 1.75, stdev= 0.96, samples=4 lat (msec) : 50=8.33%, 100=33.33%, 250=16.67%, 500=25.00%, 750=8.33% cpu : usr=0.07%, sys=2.25%, ctx=50, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24308: Thu May 1 22:20:14 2025 read: IOPS=1, BW=1528KiB/s (1565kB/s)(5120KiB/3350msec) clat (msec): min=31, max=215, avg=110.37, stdev=76.09 lat (msec): min=31, max=215, avg=110.37, stdev=76.09 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 64], 40.00th=[ 64], 50.00th=[ 80], 60.00th=[ 80], | 70.00th=[ 163], 80.00th=[ 163], 90.00th=[ 215], 95.00th=[ 215], | 99.00th=[ 215], 99.50th=[ 215], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 8143, max= 8143, per=100.00%, avg=8143.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2140KiB/s (2191kB/s)(7168KiB/3350msec) clat (msec): min=49, max=1601, avg=378.82, stdev=556.93 lat (msec): min=49, max=1601, avg=379.29, stdev=556.71 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 69], | 30.00th=[ 96], 40.00th=[ 96], 50.00th=[ 117], 60.00th=[ 284], | 70.00th=[ 284], 80.00th=[ 439], 90.00th=[ 1603], 95.00th=[ 1603], | 99.00th=[ 1603], 99.50th=[ 1603], 99.90th=[ 1603], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min= 2031, max= 6119, per=58.31%, avg=4082.00, stdev=2044.04, samples=3 iops : min= 1, max= 5, avg= 3.33, stdev= 2.08, samples=3 lat (msec) : 50=16.67%, 100=33.33%, 250=25.00%, 500=16.67% cpu : usr=0.06%, sys=2.00%, ctx=38, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5001KiB/s (5121kB/s), 621KiB/s-1817KiB/s (636kB/s-1861kB/s), io=20.0MiB (20.0MB), run=3296-4095msec WRITE: bw=7002KiB/s (7170kB/s), 1298KiB/s-3107KiB/s (1329kB/s-3181kB/s), io=28.0MiB (29.4MB), run=3296-4095msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24321: Thu May 1 22:20:19 2025 read: IOPS=1, BW=7136KiB/s (7307kB/s)(12.0MiB/1722msec) clat (msec): min=330, max=753, avg=504.82, stdev=221.38 lat (msec): min=330, max=753, avg=504.82, stdev=221.39 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 330], | 30.00th=[ 330], 40.00th=[ 430], 50.00th=[ 430], 60.00th=[ 430], | 70.00th=[ 751], 80.00th=[ 751], 90.00th=[ 751], 95.00th=[ 751], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 8126, max= 8159, per=100.00%, avg=8142.50, stdev=23.33, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67%, 1000=33.33% cpu : usr=0.00%, sys=2.61%, ctx=23, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24323: Thu May 1 22:20:19 2025 read: IOPS=0, BW=1542KiB/s (1579kB/s)(4096KiB/2657msec) clat (nsec): min=501127k, max=501127k, avg=501127411.00, stdev= 0.00 lat (nsec): min=501131k, max=501131k, avg=501130593.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 502], 10.00th=[ 502], 20.00th=[ 502], | 30.00th=[ 502], 40.00th=[ 502], 50.00th=[ 502], 60.00th=[ 502], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 8126, max= 8126, per=100.00%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3083KiB/s (3157kB/s)(8192KiB/2657msec) clat (msec): min=569, max=1361, avg=965.13, stdev=560.00 lat (msec): min=569, max=1361, avg=965.78, stdev=559.96 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 567], 10.00th=[ 567], 20.00th=[ 567], | 30.00th=[ 567], 40.00th=[ 567], 50.00th=[ 567], 60.00th=[ 1368], | 70.00th=[ 1368], 80.00th=[ 1368], 90.00th=[ 1368], 95.00th=[ 1368], | 99.00th=[ 1368], 99.50th=[ 1368], 99.90th=[ 1368], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min= 8192, max= 8192, per=66.43%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 750=66.67% cpu : usr=0.00%, sys=2.79%, ctx=22, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24324: Thu May 1 22:20:19 2025 write: IOPS=1, BW=6172KiB/s (6320kB/s)(12.0MiB/1991msec) clat (msec): min=471, max=822, avg=600.68, stdev=193.38 lat (msec): min=471, max=823, avg=601.09, stdev=193.39 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 472], 10.00th=[ 472], 20.00th=[ 472], | 30.00th=[ 472], 40.00th=[ 510], 50.00th=[ 510], 60.00th=[ 510], | 70.00th=[ 827], 80.00th=[ 827], 90.00th=[ 827], 95.00th=[ 827], | 99.00th=[ 827], 99.50th=[ 827], 99.90th=[ 827], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 7861, max= 7861, per=63.74%, avg=7861.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=7.29%, ctx=79, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24326: Thu May 1 22:20:19 2025 write: IOPS=1, BW=5274KiB/s (5400kB/s)(12.0MiB/2330msec) clat (msec): min=612, max=846, avg=718.30, stdev=118.98 lat (msec): min=620, max=848, avg=721.97, stdev=116.18 clat percentiles (msec): | 1.00th=[ 609], 5.00th=[ 609], 10.00th=[ 609], 20.00th=[ 609], | 30.00th=[ 609], 40.00th=[ 693], 50.00th=[ 693], 60.00th=[ 693], | 70.00th=[ 844], 80.00th=[ 844], 90.00th=[ 844], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 8192, max= 8192, per=66.43%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 750=66.67%, 1000=33.33% cpu : usr=0.00%, sys=4.94%, ctx=88, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6166KiB/s (6314kB/s), 1542KiB/s-7136KiB/s (1579kB/s-7307kB/s), io=16.0MiB (16.8MB), run=1722-2657msec WRITE: bw=12.0MiB/s (12.6MB/s), 3083KiB/s-6172KiB/s (3157kB/s-6320kB/s), io=32.0MiB (33.6MB), run=1991-2657msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24322: Thu May 1 22:20:20 2025 read: IOPS=1, BW=6543KiB/s (6700kB/s)(12.0MiB/1878msec) clat (msec): min=434, max=862, avg=616.19, stdev=221.50 lat (msec): min=434, max=862, avg=616.19, stdev=221.50 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 435], 10.00th=[ 435], 20.00th=[ 435], | 30.00th=[ 435], 40.00th=[ 550], 50.00th=[ 550], 60.00th=[ 550], | 70.00th=[ 860], 80.00th=[ 860], 90.00th=[ 860], 95.00th=[ 860], | 99.00th=[ 860], 99.50th=[ 860], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=14.12%, ctx=646, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24325: Thu May 1 22:20:20 2025 read: IOPS=0, BW=1624KiB/s (1663kB/s)(4096KiB/2522msec) clat (nsec): min=747727k, max=747727k, avg=747727440.00, stdev= 0.00 lat (nsec): min=747732k, max=747732k, avg=747731691.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 751], 5.00th=[ 751], 10.00th=[ 751], 20.00th=[ 751], | 30.00th=[ 751], 40.00th=[ 751], 50.00th=[ 751], 60.00th=[ 751], | 70.00th=[ 751], 80.00th=[ 751], 90.00th=[ 751], 95.00th=[ 751], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3248KiB/s (3326kB/s)(8192KiB/2522msec) clat (msec): min=768, max=889, avg=829.25, stdev=85.47 lat (msec): min=777, max=890, avg=833.67, stdev=79.91 clat percentiles (msec): | 1.00th=[ 768], 5.00th=[ 768], 10.00th=[ 768], 20.00th=[ 768], | 30.00th=[ 768], 40.00th=[ 768], 50.00th=[ 768], 60.00th=[ 894], | 70.00th=[ 894], 80.00th=[ 894], 90.00th=[ 894], 95.00th=[ 894], | 99.00th=[ 894], 99.50th=[ 894], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 6059, max= 6059, per=46.64%, avg=6059.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 750=33.33%, 1000=66.67% cpu : usr=0.00%, sys=13.65%, ctx=244, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24327: Thu May 1 22:20:20 2025 write: IOPS=1, BW=5466KiB/s (5597kB/s)(12.0MiB/2248msec) clat (msec): min=335, max=948, avg=727.71, stdev=340.44 lat (msec): min=336, max=949, avg=728.16, stdev=340.49 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 338], 10.00th=[ 338], 20.00th=[ 338], | 30.00th=[ 338], 40.00th=[ 902], 50.00th=[ 902], 60.00th=[ 902], | 70.00th=[ 953], 80.00th=[ 953], 90.00th=[ 953], 95.00th=[ 953], | 99.00th=[ 953], 99.50th=[ 953], 99.90th=[ 953], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 7571, max= 8192, per=60.66%, avg=7881.50, stdev=439.11, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 1000=66.67% cpu : usr=0.00%, sys=15.80%, ctx=129, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24328: Thu May 1 22:20:20 2025 write: IOPS=1, BW=5684KiB/s (5820kB/s)(12.0MiB/2162msec) clat (msec): min=539, max=743, avg=658.45, stdev=105.74 lat (msec): min=540, max=771, avg=668.26, stdev=117.69 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 542], 10.00th=[ 542], 20.00th=[ 542], | 30.00th=[ 542], 40.00th=[ 693], 50.00th=[ 693], 60.00th=[ 693], | 70.00th=[ 743], 80.00th=[ 743], 90.00th=[ 743], 95.00th=[ 743], | 99.00th=[ 743], 99.50th=[ 743], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 8159, max= 8192, per=62.93%, avg=8175.50, stdev=23.33, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 750=100.00% cpu : usr=0.00%, sys=16.34%, ctx=104, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6496KiB/s (6652kB/s), 1624KiB/s-6543KiB/s (1663kB/s-6700kB/s), io=16.0MiB (16.8MB), run=1878-2522msec WRITE: bw=12.7MiB/s (13.3MB/s), 3248KiB/s-5684KiB/s (3326kB/s-5820kB/s), io=32.0MiB (33.6MB), run=2162-2522msec