== sanity test 398b: DIO and buffer IO race ============== 10:32:16 (1749479536) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.14412 s, 44.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=18544: Mon Jun 9 10:33:24 2025 read: IOPS=22, BW=91.4KiB/s (93.6kB/s)(5992KiB/65526msec) clat (usec): min=56, max=178804, avg=4558.70, stdev=6399.08 lat (usec): min=56, max=178805, avg=4559.62, stdev=6399.04 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 78], 10.00th=[ 102], 20.00th=[ 186], | 30.00th=[ 2671], 40.00th=[ 3130], 50.00th=[ 3621], 60.00th=[ 4293], | 70.00th=[ 5342], 80.00th=[ 6915], 90.00th=[ 9372], 95.00th=[ 10945], | 99.00th=[ 16188], 99.50th=[ 20055], 99.90th=[ 84411], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 152, max= 534, per=100.00%, avg=374.13, stdev=83.59, samples=32 iops : min= 38, max= 133, avg=93.47, stdev=20.86, samples=32 write: IOPS=24, BW=96.1KiB/s (98.4kB/s)(6296KiB/65526msec) clat (usec): min=506, max=50016k, avg=37223.20, stdev=1260551.98 lat (usec): min=507, max=50016k, avg=37226.10, stdev=1260551.96 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 594], 10.00th=[ 652], | 20.00th=[ 840], 30.00th=[ 1221], 40.00th=[ 2900], | 50.00th=[ 4293], 60.00th=[ 5669], 70.00th=[ 7177], | 80.00th=[ 8979], 90.00th=[ 12518], 95.00th=[ 15664], | 99.00th=[ 21365], 99.50th=[ 23987], 99.90th=[ 54789], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 216, max= 566, per=100.00%, avg=392.37, stdev=76.55, samples=32 iops : min= 54, max= 141, avg=98.03, stdev=19.09, samples=32 lat (usec) : 100=4.62%, 250=5.40%, 500=0.03%, 750=8.17%, 1000=5.60% lat (msec) : 2=4.75%, 4=23.11%, 10=36.20%, 20=11.26%, 50=0.68% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.04%, sys=5.39%, ctx=5772, majf=0, minf=37 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18546: Mon Jun 9 10:33:24 2025 read: IOPS=22, BW=90.5KiB/s (92.7kB/s)(5940KiB/65626msec) clat (usec): min=54, max=31074, avg=4421.85, stdev=3590.73 lat (usec): min=54, max=31074, avg=4423.18, stdev=3590.61 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 81], 10.00th=[ 104], 20.00th=[ 2040], | 30.00th=[ 2769], 40.00th=[ 3195], 50.00th=[ 3654], 60.00th=[ 4359], | 70.00th=[ 5473], 80.00th=[ 6783], 90.00th=[ 9503], 95.00th=[10945], | 99.00th=[15926], 99.50th=[17695], 99.90th=[26346], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 184, max= 582, per=100.00%, avg=367.12, stdev=83.58, samples=32 iops : min= 46, max= 145, avg=91.72, stdev=20.83, samples=32 write: IOPS=24, BW=96.7KiB/s (99.1kB/s)(6348KiB/65626msec) clat (usec): min=481, max=49969k, avg=37148.03, stdev=1254207.41 lat (usec): min=482, max=49969k, avg=37149.98, stdev=1254207.38 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 586], 10.00th=[ 652], | 20.00th=[ 832], 30.00th=[ 1156], 40.00th=[ 2671], | 50.00th=[ 4424], 60.00th=[ 5735], 70.00th=[ 7242], | 80.00th=[ 9503], 90.00th=[ 12649], 95.00th=[ 15795], | 99.00th=[ 24249], 99.50th=[ 28181], 99.90th=[ 177210], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 248, max= 592, per=100.00%, avg=390.62, stdev=74.84, samples=32 iops : min= 62, max= 148, avg=97.59, stdev=18.68, samples=32 lat (usec) : 100=4.65%, 250=4.39%, 500=0.29%, 750=8.17%, 1000=5.92% lat (msec) : 2=4.85%, 4=23.31%, 10=35.32%, 20=11.69%, 50=1.33% lat (msec) : 250=0.03% cpu : usr=0.04%, sys=5.29%, ctx=5727, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18548: Mon Jun 9 10:33:24 2025 read: IOPS=22, BW=91.3KiB/s (93.5kB/s)(6028KiB/66005msec) clat (usec): min=57, max=21414, avg=4533.92, stdev=3727.64 lat (usec): min=57, max=21415, avg=4535.62, stdev=3727.37 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 79], 10.00th=[ 98], 20.00th=[ 167], | 30.00th=[ 2606], 40.00th=[ 3163], 50.00th=[ 3720], 60.00th=[ 4424], | 70.00th=[ 5866], 80.00th=[ 7439], 90.00th=[ 9765], 95.00th=[11731], | 99.00th=[16057], 99.50th=[17957], 99.90th=[20055], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 48, max= 544, per=97.49%, avg=353.87, stdev=95.50, samples=32 iops : min= 12, max= 136, avg=88.41, stdev=23.82, samples=32 write: IOPS=23, BW=94.8KiB/s (97.1kB/s)(6260KiB/66005msec) clat (usec): min=496, max=49984k, avg=37740.13, stdev=1263382.37 lat (usec): min=497, max=49984k, avg=37741.52, stdev=1263382.36 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 594], 10.00th=[ 652], | 20.00th=[ 832], 30.00th=[ 1205], 40.00th=[ 3064], | 50.00th=[ 4555], 60.00th=[ 5669], 70.00th=[ 7177], | 80.00th=[ 9241], 90.00th=[ 12780], 95.00th=[ 16581], | 99.00th=[ 25035], 99.50th=[ 34341], 99.90th=[ 231736], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 48, max= 504, per=96.49%, avg=366.66, stdev=96.02, samples=32 iops : min= 12, max= 126, avg=91.59, stdev=23.96, samples=32 lat (usec) : 100=5.27%, 250=4.75%, 500=0.07%, 750=8.43%, 1000=5.60% lat (msec) : 2=3.71%, 4=22.43%, 10=36.88%, 20=11.65%, 50=1.11% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.05%, sys=5.36%, ctx=5831, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18549: Mon Jun 9 10:33:24 2025 read: IOPS=23, BW=92.1KiB/s (94.3kB/s)(6052KiB/65712msec) clat (usec): min=56, max=26853, avg=4430.13, stdev=3765.54 lat (usec): min=56, max=26853, avg=4431.36, stdev=3765.41 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 80], 10.00th=[ 100], 20.00th=[ 277], | 30.00th=[ 2638], 40.00th=[ 3064], 50.00th=[ 3556], 60.00th=[ 4293], | 70.00th=[ 5342], 80.00th=[ 6980], 90.00th=[ 9634], 95.00th=[11731], | 99.00th=[16581], 99.50th=[18744], 99.90th=[25560], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 184, max= 512, per=100.00%, avg=368.72, stdev=80.68, samples=32 iops : min= 46, max= 128, avg=92.16, stdev=20.18, samples=32 write: IOPS=23, BW=94.9KiB/s (97.2kB/s)(6236KiB/65712msec) clat (usec): min=453, max=50033k, avg=37781.36, stdev=1267036.89 lat (usec): min=453, max=50033k, avg=37782.68, stdev=1267036.87 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 586], 10.00th=[ 644], | 20.00th=[ 816], 30.00th=[ 1139], 40.00th=[ 2540], | 50.00th=[ 4080], 60.00th=[ 5342], 70.00th=[ 7177], | 80.00th=[ 9372], 90.00th=[ 13042], 95.00th=[ 16909], | 99.00th=[ 24249], 99.50th=[ 31327], 99.90th=[ 202376], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 184, max= 584, per=99.93%, avg=379.72, stdev=83.44, samples=32 iops : min= 46, max= 146, avg=94.91, stdev=20.85, samples=32 lat (usec) : 100=4.88%, 250=4.95%, 500=0.23%, 750=8.46%, 1000=5.40% lat (msec) : 2=5.27%, 4=23.83%, 10=33.69%, 20=11.78%, 50=1.40% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.06%, sys=5.22%, ctx=5697, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=364KiB/s (373kB/s), 90.5KiB/s-92.1KiB/s (92.7kB/s-94.3kB/s), io=23.4MiB (24.6MB), run=65526-66005msec WRITE: bw=381KiB/s (390kB/s), 94.8KiB/s-96.7KiB/s (97.1kB/s-99.1kB/s), io=24.6MiB (25.7MB), run=65526-66005msec rand-rw: (groupid=0, jobs=1): err= 0: pid=18542: Mon Jun 9 10:33:45 2025 read: IOPS=17, BW=68.8KiB/s (70.5kB/s)(5992KiB/87090msec) clat (usec): min=1745, max=50009k, avg=37877.83, stdev=1291976.89 lat (usec): min=1746, max=50009k, avg=37878.77, stdev=1291976.88 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2073], 10.00th=[ 2180], | 20.00th=[ 2311], 30.00th=[ 2442], 40.00th=[ 2573], | 50.00th=[ 2737], 60.00th=[ 2933], 70.00th=[ 3359], | 80.00th=[ 5014], 90.00th=[ 9372], 95.00th=[ 12911], | 99.00th=[ 22414], 99.50th=[ 29492], 99.90th=[ 183501], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 40, max= 304, per=57.94%, avg=159.35, stdev=72.98, samples=75 iops : min= 10, max= 76, avg=39.77, stdev=18.23, samples=75 write: IOPS=18, BW=72.3KiB/s (74.0kB/s)(6296KiB/87090msec) clat (usec): min=8565, max=60559, avg=19213.59, stdev=8227.27 lat (usec): min=8566, max=60561, avg=19214.72, stdev=8227.33 clat percentiles (usec): | 1.00th=[ 9765], 5.00th=[10814], 10.00th=[11469], 20.00th=[12649], | 30.00th=[13960], 40.00th=[15664], 50.00th=[16909], 60.00th=[18482], | 70.00th=[20579], 80.00th=[23725], 90.00th=[31065], 95.00th=[36439], | 99.00th=[46924], 99.50th=[51643], 99.90th=[57410], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min= 64, max= 256, per=58.05%, avg=167.17, stdev=62.10, samples=75 iops : min= 16, max= 64, avg=41.73, stdev=15.52, samples=75 lat (msec) : 2=1.17%, 4=35.84%, 10=8.17%, 20=37.53%, 50=16.80% lat (msec) : 100=0.42%, 250=0.03% cpu : usr=0.04%, sys=2.45%, ctx=3451, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18543: Mon Jun 9 10:33:45 2025 read: IOPS=17, BW=68.2KiB/s (69.9kB/s)(5940KiB/87059msec) clat (usec): min=1740, max=50012k, avg=38046.00, stdev=1297706.34 lat (usec): min=1740, max=50012k, avg=38046.90, stdev=1297706.33 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2040], 10.00th=[ 2147], | 20.00th=[ 2311], 30.00th=[ 2442], 40.00th=[ 2573], | 50.00th=[ 2704], 60.00th=[ 2900], 70.00th=[ 3359], | 80.00th=[ 4752], 90.00th=[ 8979], 95.00th=[ 12387], | 99.00th=[ 21627], 99.50th=[ 26084], 99.90th=[ 202376], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 16, max= 312, per=57.54%, avg=158.24, stdev=75.38, samples=74 iops : min= 4, max= 78, avg=39.54, stdev=18.86, samples=74 write: IOPS=18, BW=72.9KiB/s (74.7kB/s)(6348KiB/87059msec) clat (usec): min=8668, max=94223, avg=19188.49, stdev=8564.63 lat (usec): min=8669, max=94224, avg=19189.58, stdev=8564.62 clat percentiles (usec): | 1.00th=[ 9634], 5.00th=[10552], 10.00th=[11207], 20.00th=[12649], | 30.00th=[14091], 40.00th=[15533], 50.00th=[16909], 60.00th=[18482], | 70.00th=[20579], 80.00th=[23725], 90.00th=[30802], 95.00th=[37487], | 99.00th=[49546], 99.50th=[51119], 99.90th=[76022], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 48, max= 264, per=58.40%, avg=168.19, stdev=62.89, samples=74 iops : min= 12, max= 66, avg=42.03, stdev=15.72, samples=74 lat (msec) : 2=1.53%, 4=35.38%, 10=8.46%, 20=37.17%, 50=16.96% lat (msec) : 100=0.42%, 250=0.03% cpu : usr=0.06%, sys=2.45%, ctx=3399, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18545: Mon Jun 9 10:33:45 2025 read: IOPS=17, BW=69.4KiB/s (71.0kB/s)(6028KiB/86901msec) clat (usec): min=1802, max=80593, avg=4566.71, stdev=4557.22 lat (usec): min=1803, max=80594, avg=4567.64, stdev=4557.21 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2040], 10.00th=[ 2147], 20.00th=[ 2311], | 30.00th=[ 2442], 40.00th=[ 2606], 50.00th=[ 2769], 60.00th=[ 2999], | 70.00th=[ 3621], 80.00th=[ 5866], 90.00th=[10290], 95.00th=[13173], | 99.00th=[21103], 99.50th=[23725], 99.90th=[46400], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 24, max= 320, per=58.66%, avg=161.31, stdev=68.47, samples=74 iops : min= 6, max= 80, avg=40.26, stdev=17.07, samples=74 write: IOPS=18, BW=72.0KiB/s (73.8kB/s)(6260KiB/86901msec) clat (msec): min=8, max=50061, avg=51.06, stdev=1265.02 lat (msec): min=8, max=50061, avg=51.06, stdev=1265.02 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 19], | 70.00th=[ 21], 80.00th=[ 24], 90.00th=[ 31], 95.00th=[ 37], | 99.00th=[ 45], 99.50th=[ 54], 99.90th=[ 232], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 40, max= 256, per=57.96%, avg=166.93, stdev=64.93, samples=74 iops : min= 10, max= 64, avg=41.66, stdev=16.23, samples=74 lat (msec) : 2=1.43%, 4=34.18%, 10=9.41%, 20=38.57%, 50=16.08% lat (msec) : 100=0.26%, 250=0.03% cpu : usr=0.05%, sys=2.49%, ctx=3463, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18547: Mon Jun 9 10:33:45 2025 read: IOPS=17, BW=69.7KiB/s (71.4kB/s)(6052KiB/86788msec) clat (usec): min=1721, max=49047, avg=4359.95, stdev=4163.08 lat (usec): min=1721, max=49048, avg=4360.91, stdev=4163.05 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2057], 10.00th=[ 2147], 20.00th=[ 2311], | 30.00th=[ 2442], 40.00th=[ 2573], 50.00th=[ 2737], 60.00th=[ 2966], | 70.00th=[ 3359], 80.00th=[ 5145], 90.00th=[ 9634], 95.00th=[12780], | 99.00th=[20055], 99.50th=[25822], 99.90th=[44827], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 40, max= 328, per=58.76%, avg=161.59, stdev=71.33, samples=74 iops : min= 10, max= 82, avg=40.30, stdev=17.83, samples=74 write: IOPS=17, BW=71.9KiB/s (73.6kB/s)(6236KiB/86788msec) clat (msec): min=8, max=50057, avg=51.37, stdev=1267.33 lat (msec): min=8, max=50057, avg=51.37, stdev=1267.33 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 19], | 70.00th=[ 21], 80.00th=[ 25], 90.00th=[ 32], 95.00th=[ 36], | 99.00th=[ 50], 99.50th=[ 57], 99.90th=[ 218], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 48, max= 256, per=57.95%, avg=166.89, stdev=63.51, samples=74 iops : min= 12, max= 64, avg=41.62, stdev=15.87, samples=74 lat (msec) : 2=1.56%, 4=35.58%, 10=9.02%, 20=37.17%, 50=16.24% lat (msec) : 100=0.36%, 250=0.03% cpu : usr=0.07%, sys=2.47%, ctx=3417, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=276KiB/s (282kB/s), 68.2KiB/s-69.7KiB/s (69.9kB/s-71.4kB/s), io=23.4MiB (24.6MB), run=86788-87090msec WRITE: bw=289KiB/s (296kB/s), 71.9KiB/s-72.9KiB/s (73.6kB/s-74.7kB/s), io=24.6MiB (25.7MB), run=86788-87090msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=18626: Mon Jun 9 10:33:49 2025 read: IOPS=124, BW=1987KiB/s (2035kB/s)(5920KiB/2979msec) clat (usec): min=60, max=24654, avg=3861.55, stdev=3256.49 lat (usec): min=60, max=24655, avg=3862.36, stdev=3256.50 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 80], 10.00th=[ 104], 20.00th=[ 135], | 30.00th=[ 1745], 40.00th=[ 3130], 50.00th=[ 3621], 60.00th=[ 4178], | 70.00th=[ 5211], 80.00th=[ 6390], 90.00th=[ 8225], 95.00th=[ 9372], | 99.00th=[11731], 99.50th=[13042], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 1532, max= 2336, per=25.30%, avg=2014.40, stdev=299.34, samples=5 iops : min= 95, max= 146, avg=125.60, stdev=19.03, samples=5 write: IOPS=133, BW=2138KiB/s (2189kB/s)(6368KiB/2979msec) clat (usec): min=711, max=35745, avg=3862.91, stdev=4229.60 lat (usec): min=712, max=35746, avg=3864.82, stdev=4229.14 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 865], 10.00th=[ 938], 20.00th=[ 1106], | 30.00th=[ 1254], 40.00th=[ 1418], 50.00th=[ 1811], 60.00th=[ 3195], | 70.00th=[ 4359], 80.00th=[ 6521], 90.00th=[ 9241], 95.00th=[11338], | 99.00th=[20055], 99.50th=[31065], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 1756, max= 2746, per=26.18%, avg=2193.20, stdev=382.03, samples=5 iops : min= 109, max= 171, avg=136.80, stdev=23.87, samples=5 lat (usec) : 100=4.30%, 250=8.98%, 500=0.52%, 750=0.26%, 1000=6.38% lat (msec) : 2=20.70%, 4=21.48%, 10=31.90%, 20=4.82%, 50=0.65% cpu : usr=0.13%, sys=29.72%, ctx=1449, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18627: Mon Jun 9 10:33:49 2025 read: IOPS=123, BW=1975KiB/s (2022kB/s)(5744KiB/2909msec) clat (usec): min=71, max=15545, avg=3722.41, stdev=3055.13 lat (usec): min=71, max=15546, avg=3723.36, stdev=3054.89 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 84], 10.00th=[ 96], 20.00th=[ 124], | 30.00th=[ 2278], 40.00th=[ 3228], 50.00th=[ 3752], 60.00th=[ 4293], | 70.00th=[ 5014], 80.00th=[ 6063], 90.00th=[ 7242], 95.00th=[ 9372], | 99.00th=[13435], 99.50th=[15270], 99.90th=[15533], 99.95th=[15533], | 99.99th=[15533] bw ( KiB/s): min= 1504, max= 2464, per=25.88%, avg=2060.80, stdev=374.28, samples=5 iops : min= 94, max= 154, avg=128.80, stdev=23.39, samples=5 write: IOPS=140, BW=2250KiB/s (2304kB/s)(6544KiB/2909msec) clat (usec): min=660, max=22542, avg=3781.64, stdev=3702.60 lat (usec): min=660, max=22542, avg=3784.77, stdev=3701.83 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 832], 10.00th=[ 898], 20.00th=[ 1057], | 30.00th=[ 1270], 40.00th=[ 1549], 50.00th=[ 2073], 60.00th=[ 3032], | 70.00th=[ 4686], 80.00th=[ 6390], 90.00th=[ 9241], 95.00th=[11076], | 99.00th=[15926], 99.50th=[20317], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 2016, max= 2528, per=27.73%, avg=2323.20, stdev=209.35, samples=5 iops : min= 126, max= 158, avg=145.20, stdev=13.08, samples=5 lat (usec) : 100=5.73%, 250=7.55%, 500=0.26%, 750=0.91%, 1000=8.33% lat (msec) : 2=16.93%, 4=20.96%, 10=33.33%, 20=5.60%, 50=0.39% cpu : usr=0.24%, sys=29.85%, ctx=1522, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18628: Mon Jun 9 10:33:49 2025 read: IOPS=129, BW=2080KiB/s (2130kB/s)(6256KiB/3008msec) clat (usec): min=58, max=22136, avg=3770.34, stdev=3519.73 lat (usec): min=58, max=22137, avg=3771.88, stdev=3519.16 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 75], 10.00th=[ 89], 20.00th=[ 119], | 30.00th=[ 147], 40.00th=[ 3130], 50.00th=[ 3589], 60.00th=[ 4228], | 70.00th=[ 5080], 80.00th=[ 6652], 90.00th=[ 7898], 95.00th=[ 9503], | 99.00th=[16712], 99.50th=[19792], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 1856, max= 2304, per=26.19%, avg=2085.33, stdev=152.01, samples=6 iops : min= 116, max= 144, avg=130.33, stdev= 9.50, samples=6 write: IOPS=125, BW=2005KiB/s (2053kB/s)(6032KiB/3008msec) clat (usec): min=693, max=26783, avg=4003.54, stdev=3876.94 lat (usec): min=694, max=26784, avg=4004.67, stdev=3877.01 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 865], 10.00th=[ 979], 20.00th=[ 1188], | 30.00th=[ 1352], 40.00th=[ 1696], 50.00th=[ 2442], 60.00th=[ 3458], | 70.00th=[ 4555], 80.00th=[ 6849], 90.00th=[ 9241], 95.00th=[11338], | 99.00th=[18220], 99.50th=[23987], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 1760, max= 2272, per=23.75%, avg=1989.33, stdev=218.37, samples=6 iops : min= 110, max= 142, avg=124.33, stdev=13.65, samples=6 lat (usec) : 100=7.29%, 250=9.64%, 500=0.39%, 750=0.39%, 1000=5.08% lat (msec) : 2=17.06%, 4=21.48%, 10=32.42%, 20=5.86%, 50=0.39% cpu : usr=0.17%, sys=28.87%, ctx=1511, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18629: Mon Jun 9 10:33:49 2025 read: IOPS=128, BW=2060KiB/s (2110kB/s)(6032KiB/2928msec) clat (usec): min=62, max=23044, avg=3811.13, stdev=3532.18 lat (usec): min=62, max=23045, avg=3812.39, stdev=3531.69 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 80], 10.00th=[ 95], 20.00th=[ 119], | 30.00th=[ 176], 40.00th=[ 3130], 50.00th=[ 3589], 60.00th=[ 4146], | 70.00th=[ 4948], 80.00th=[ 6390], 90.00th=[ 8291], 95.00th=[ 9896], | 99.00th=[16450], 99.50th=[21103], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 1760, max= 2304, per=25.40%, avg=2022.40, stdev=212.75, samples=5 iops : min= 110, max= 144, avg=126.40, stdev=13.30, samples=5 write: IOPS=133, BW=2137KiB/s (2188kB/s)(6256KiB/2928msec) clat (usec): min=716, max=19851, avg=3735.08, stdev=3515.61 lat (usec): min=717, max=19853, avg=3736.34, stdev=3515.63 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 881], 10.00th=[ 979], 20.00th=[ 1139], | 30.00th=[ 1319], 40.00th=[ 1549], 50.00th=[ 1860], 60.00th=[ 2966], | 70.00th=[ 4621], 80.00th=[ 6652], 90.00th=[ 8979], 95.00th=[10552], | 99.00th=[15533], 99.50th=[18482], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 1856, max= 2464, per=25.29%, avg=2118.40, stdev=224.46, samples=5 iops : min= 116, max= 154, avg=132.40, stdev=14.03, samples=5 lat (usec) : 100=5.86%, 250=9.37%, 500=0.52%, 750=0.39%, 1000=5.73% lat (msec) : 2=20.57%, 4=18.88%, 10=32.81%, 20=5.60%, 50=0.26% cpu : usr=0.27%, sys=29.24%, ctx=1421, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7963KiB/s (8154kB/s), 1975KiB/s-2080KiB/s (2022kB/s-2130kB/s), io=23.4MiB (24.5MB), run=2909-3008msec WRITE: bw=8378KiB/s (8579kB/s), 2005KiB/s-2250KiB/s (2053kB/s-2304kB/s), io=24.6MiB (25.8MB), run=2909-3008msec rand-rw: (groupid=0, jobs=1): err= 0: pid=18631: Mon Jun 9 10:33:55 2025 read: IOPS=41, BW=668KiB/s (684kB/s)(5920KiB/8865msec) clat (usec): min=2004, max=27785, avg=4336.04, stdev=3946.75 lat (usec): min=2005, max=27785, avg=4337.06, stdev=3946.70 clat percentiles (usec): | 1.00th=[ 2073], 5.00th=[ 2245], 10.00th=[ 2376], 20.00th=[ 2474], | 30.00th=[ 2638], 40.00th=[ 2769], 50.00th=[ 2900], 60.00th=[ 3064], | 70.00th=[ 3425], 80.00th=[ 4359], 90.00th=[ 8225], 95.00th=[12780], | 99.00th=[22676], 99.50th=[25297], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 319, max= 1216, per=24.85%, avg=656.88, stdev=259.59, samples=17 iops : min= 19, max= 76, avg=41.00, stdev=16.30, samples=17 write: IOPS=44, BW=718KiB/s (736kB/s)(6368KiB/8865msec) clat (usec): min=9310, max=56359, avg=18149.79, stdev=7116.78 lat (usec): min=9311, max=56361, avg=18151.76, stdev=7116.96 clat percentiles (usec): | 1.00th=[ 9503], 5.00th=[10552], 10.00th=[11338], 20.00th=[12780], | 30.00th=[13960], 40.00th=[15139], 50.00th=[16450], 60.00th=[17957], | 70.00th=[19530], 80.00th=[21890], 90.00th=[26870], 95.00th=[31589], | 99.00th=[46924], 99.50th=[53216], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 384, max= 960, per=25.58%, avg=711.47, stdev=219.60, samples=17 iops : min= 24, max= 60, avg=44.41, stdev=13.79, samples=17 lat (msec) : 4=36.85%, 10=9.51%, 20=37.76%, 50=15.49%, 100=0.39% cpu : usr=0.15%, sys=7.10%, ctx=886, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18632: Mon Jun 9 10:33:55 2025 read: IOPS=39, BW=634KiB/s (649kB/s)(5744KiB/9060msec) clat (usec): min=2059, max=25904, avg=4419.07, stdev=3867.20 lat (usec): min=2059, max=25905, avg=4420.02, stdev=3867.18 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2245], 10.00th=[ 2376], 20.00th=[ 2507], | 30.00th=[ 2606], 40.00th=[ 2704], 50.00th=[ 2868], 60.00th=[ 3064], | 70.00th=[ 3556], 80.00th=[ 5276], 90.00th=[ 8455], 95.00th=[12256], | 99.00th=[23462], 99.50th=[24249], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 192, max= 1152, per=23.87%, avg=630.94, stdev=249.82, samples=18 iops : min= 12, max= 72, avg=39.33, stdev=15.68, samples=18 write: IOPS=45, BW=722KiB/s (740kB/s)(6544KiB/9060msec) clat (usec): min=8837, max=44802, avg=18106.93, stdev=6714.68 lat (usec): min=8839, max=44804, avg=18108.79, stdev=6714.73 clat percentiles (usec): | 1.00th=[ 9765], 5.00th=[10814], 10.00th=[11600], 20.00th=[12649], | 30.00th=[13960], 40.00th=[15270], 50.00th=[16712], 60.00th=[17695], | 70.00th=[19268], 80.00th=[21627], 90.00th=[27395], 95.00th=[33424], | 99.00th=[41157], 99.50th=[43254], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 384, max= 990, per=25.88%, avg=719.83, stdev=218.15, samples=18 iops : min= 24, max= 61, avg=44.89, stdev=13.65, samples=18 lat (msec) : 4=35.16%, 10=8.85%, 20=40.89%, 50=15.10% cpu : usr=0.22%, sys=6.94%, ctx=906, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18633: Mon Jun 9 10:33:55 2025 read: IOPS=44, BW=719KiB/s (736kB/s)(6256KiB/8703msec) clat (usec): min=1965, max=26667, avg=4254.31, stdev=3661.82 lat (usec): min=1965, max=26668, avg=4255.31, stdev=3661.81 clat percentiles (usec): | 1.00th=[ 2057], 5.00th=[ 2278], 10.00th=[ 2409], 20.00th=[ 2507], | 30.00th=[ 2638], 40.00th=[ 2769], 50.00th=[ 2900], 60.00th=[ 3130], | 70.00th=[ 3392], 80.00th=[ 4359], 90.00th=[ 7504], 95.00th=[12911], | 99.00th=[20579], 99.50th=[23462], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 320, max= 1280, per=27.28%, avg=720.94, stdev=316.40, samples=17 iops : min= 20, max= 80, avg=45.06, stdev=19.78, samples=17 write: IOPS=43, BW=693KiB/s (710kB/s)(6032KiB/8703msec) clat (usec): min=9346, max=56421, avg=18534.13, stdev=6909.04 lat (usec): min=9348, max=56423, avg=18535.94, stdev=6909.08 clat percentiles (usec): | 1.00th=[ 9896], 5.00th=[11207], 10.00th=[11863], 20.00th=[13435], | 30.00th=[14746], 40.00th=[15533], 50.00th=[17171], 60.00th=[18220], | 70.00th=[19530], 80.00th=[21627], 90.00th=[28443], 95.00th=[33817], | 99.00th=[40109], 99.50th=[53740], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 384, max= 864, per=24.64%, avg=685.18, stdev=181.75, samples=17 iops : min= 24, max= 54, avg=42.82, stdev=11.36, samples=17 lat (msec) : 2=0.13%, 4=39.58%, 10=8.46%, 20=37.63%, 50=13.93% lat (msec) : 100=0.26% cpu : usr=0.16%, sys=7.17%, ctx=901, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18634: Mon Jun 9 10:33:55 2025 read: IOPS=42, BW=677KiB/s (694kB/s)(6032KiB/8905msec) clat (usec): min=1894, max=27614, avg=4195.03, stdev=3551.00 lat (usec): min=1895, max=27615, avg=4195.94, stdev=3551.00 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2180], 10.00th=[ 2311], 20.00th=[ 2474], | 30.00th=[ 2606], 40.00th=[ 2704], 50.00th=[ 2868], 60.00th=[ 3097], | 70.00th=[ 3359], 80.00th=[ 4359], 90.00th=[ 8717], 95.00th=[11600], | 99.00th=[19792], 99.50th=[23725], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 288, max= 1088, per=24.93%, avg=658.82, stdev=232.15, samples=17 iops : min= 18, max= 68, avg=41.18, stdev=14.51, samples=17 write: IOPS=43, BW=703KiB/s (719kB/s)(6256KiB/8905msec) clat (usec): min=9595, max=67116, avg=18561.60, stdev=7185.33 lat (usec): min=9598, max=67119, avg=18563.24, stdev=7185.41 clat percentiles (usec): | 1.00th=[10028], 5.00th=[11207], 10.00th=[11994], 20.00th=[13304], | 30.00th=[14484], 40.00th=[15664], 50.00th=[16909], 60.00th=[18220], | 70.00th=[19530], 80.00th=[21890], 90.00th=[26870], 95.00th=[32900], | 99.00th=[45351], 99.50th=[53740], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 384, max= 992, per=24.91%, avg=692.71, stdev=211.94, samples=17 iops : min= 24, max= 62, avg=43.29, stdev=13.25, samples=17 lat (msec) : 2=0.26%, 4=37.89%, 10=7.68%, 20=39.45%, 50=14.45% lat (msec) : 100=0.26% cpu : usr=0.09%, sys=6.90%, ctx=894, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2644KiB/s (2707kB/s), 634KiB/s-719KiB/s (649kB/s-736kB/s), io=23.4MiB (24.5MB), run=8703-9060msec WRITE: bw=2781KiB/s (2848kB/s), 693KiB/s-722KiB/s (710kB/s-740kB/s), io=24.6MiB (25.8MB), run=8703-9060msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=18648: Mon Jun 9 10:33:57 2025 read: IOPS=15, BW=15.6MiB/s (16.4MB/s)(7168KiB/448msec) clat (usec): min=522, max=40569, avg=29027.94, stdev=13296.06 lat (usec): min=523, max=40571, avg=29029.09, stdev=13296.19 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 523], 10.00th=[ 523], 20.00th=[28443], | 30.00th=[30540], 40.00th=[30540], 50.00th=[31065], 60.00th=[33424], | 70.00th=[33424], 80.00th=[38536], 90.00th=[40633], 95.00th=[40633], | 99.00th=[40633], 99.50th=[40633], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] write: IOPS=11, BW=11.2MiB/s (11.7MB/s)(5120KiB/448msec) clat (usec): min=26645, max=60207, avg=42515.46, stdev=15550.17 lat (usec): min=26687, max=60279, avg=42560.09, stdev=15547.92 clat percentiles (usec): | 1.00th=[26608], 5.00th=[26608], 10.00th=[26608], 20.00th=[26608], | 30.00th=[27919], 40.00th=[27919], 50.00th=[41681], 60.00th=[41681], | 70.00th=[56361], 80.00th=[56361], 90.00th=[60031], 95.00th=[60031], | 99.00th=[60031], 99.50th=[60031], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] lat (usec) : 750=8.33% lat (msec) : 50=75.00%, 100=16.67% cpu : usr=0.00%, sys=39.15%, ctx=1126, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18649: Mon Jun 9 10:33:57 2025 read: IOPS=2, BW=3016KiB/s (3089kB/s)(2048KiB/679msec) clat (usec): min=33252, max=40342, avg=36797.39, stdev=5013.89 lat (usec): min=33253, max=40343, avg=36798.37, stdev=5013.66 clat percentiles (usec): | 1.00th=[33162], 5.00th=[33162], 10.00th=[33162], 20.00th=[33162], | 30.00th=[33162], 40.00th=[33162], 50.00th=[33162], 60.00th=[40109], | 70.00th=[40109], 80.00th=[40109], 90.00th=[40109], 95.00th=[40109], | 99.00th=[40109], 99.50th=[40109], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 4087, max= 4087, per=13.55%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=14, BW=14.7MiB/s (15.4MB/s)(10.0MiB/679msec) clat (msec): min=21, max=119, avg=58.90, stdev=34.38 lat (msec): min=21, max=119, avg=58.96, stdev=34.37 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 26], | 30.00th=[ 37], 40.00th=[ 43], 50.00th=[ 43], 60.00th=[ 45], | 70.00th=[ 90], 80.00th=[ 90], 90.00th=[ 107], 95.00th=[ 120], | 99.00th=[ 120], 99.50th=[ 120], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min=10219, max=10219, per=24.20%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 50=66.67%, 100=16.67%, 250=16.67% cpu : usr=0.00%, sys=36.28%, ctx=244, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18651: Mon Jun 9 10:33:57 2025 read: IOPS=11, BW=11.8MiB/s (12.4MB/s)(6144KiB/508msec) clat (usec): min=2023, max=46950, avg=34131.75, stdev=16434.91 lat (usec): min=2025, max=46951, avg=34133.22, stdev=16434.71 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2024], 10.00th=[ 2024], 20.00th=[32637], | 30.00th=[32637], 40.00th=[39060], 50.00th=[39060], 60.00th=[40633], | 70.00th=[43254], 80.00th=[43254], 90.00th=[46924], 95.00th=[46924], | 99.00th=[46924], 99.50th=[46924], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min=10240, max=10240, per=33.95%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.8MiB/s (12.4MB/s)(6144KiB/509msec) clat (msec): min=24, max=112, avg=49.13, stdev=36.71 lat (msec): min=24, max=113, avg=49.18, stdev=36.72 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 28], 50.00th=[ 28], 60.00th=[ 29], | 70.00th=[ 75], 80.00th=[ 75], 90.00th=[ 113], 95.00th=[ 113], | 99.00th=[ 113], 99.50th=[ 113], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min=12288, max=12288, per=29.10%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (msec) : 4=8.33%, 50=75.00%, 100=8.33%, 250=8.33% cpu : usr=0.00%, sys=34.25%, ctx=302, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18653: Mon Jun 9 10:33:57 2025 read: IOPS=7, BW=8063KiB/s (8257kB/s)(5120KiB/635msec) clat (usec): min=373, max=60962, avg=25374.93, stdev=25591.63 lat (usec): min=375, max=60963, avg=25375.92, stdev=25591.51 clat percentiles (usec): | 1.00th=[ 375], 5.00th=[ 375], 10.00th=[ 375], 20.00th=[ 375], | 30.00th=[ 388], 40.00th=[ 388], 50.00th=[32637], 60.00th=[32637], | 70.00th=[32637], 80.00th=[32637], 90.00th=[61080], 95.00th=[61080], | 99.00th=[61080], 99.50th=[61080], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 8159, max= 8159, per=27.05%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.0MiB/s (11.6MB/s)(7168KiB/635msec) clat (msec): min=22, max=178, avg=66.77, stdev=57.38 lat (msec): min=22, max=178, avg=66.82, stdev=57.38 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 27], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 37], 60.00th=[ 74], | 70.00th=[ 74], 80.00th=[ 103], 90.00th=[ 178], 95.00th=[ 178], | 99.00th=[ 178], 99.50th=[ 178], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min=10199, max=10199, per=24.15%, avg=10199.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 50=50.00%, 100=16.67%, 250=16.67% cpu : usr=0.00%, sys=30.13%, ctx=258, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=29.5MiB/s (30.9MB/s), 3016KiB/s-15.6MiB/s (3089kB/s-16.4MB/s), io=20.0MiB (20.0MB), run=448-679msec WRITE: bw=41.2MiB/s (43.2MB/s), 11.0MiB/s-14.7MiB/s (11.6MB/s-15.4MB/s), io=28.0MiB (29.4MB), run=448-679msec rand-rw: (groupid=0, jobs=1): err= 0: pid=18647: Mon Jun 9 10:33:57 2025 read: IOPS=12, BW=12.7MiB/s (13.3MB/s)(7168KiB/552msec) clat (msec): min=15, max=142, avg=42.47, stdev=44.61 lat (msec): min=15, max=142, avg=42.47, stdev=44.61 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 22], | 30.00th=[ 23], 40.00th=[ 23], 50.00th=[ 23], 60.00th=[ 34], | 70.00th=[ 34], 80.00th=[ 39], 90.00th=[ 142], 95.00th=[ 142], | 99.00th=[ 142], 99.50th=[ 142], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min=10240, max=10240, per=41.25%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9275KiB/s (9498kB/s)(5120KiB/552msec) clat (usec): min=25810, max=53990, avg=44088.40, stdev=11029.43 lat (usec): min=25851, max=54053, avg=44139.08, stdev=11036.33 clat percentiles (usec): | 1.00th=[25822], 5.00th=[25822], 10.00th=[25822], 20.00th=[25822], | 30.00th=[42730], 40.00th=[42730], 50.00th=[47449], 60.00th=[47449], | 70.00th=[50594], 80.00th=[50594], 90.00th=[53740], 95.00th=[53740], | 99.00th=[53740], 99.50th=[53740], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 8192, max= 8192, per=23.57%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=66.67%, 100=16.67%, 250=8.33% cpu : usr=0.00%, sys=16.33%, ctx=35, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18650: Mon Jun 9 10:33:57 2025 read: IOPS=2, BW=2535KiB/s (2595kB/s)(2048KiB/808msec) clat (msec): min=25, max=105, avg=65.46, stdev=57.20 lat (msec): min=25, max=105, avg=65.46, stdev=57.20 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 26], | 30.00th=[ 26], 40.00th=[ 26], 50.00th=[ 26], 60.00th=[ 106], | 70.00th=[ 106], 80.00th=[ 106], 90.00th=[ 106], 95.00th=[ 106], | 99.00th=[ 106], 99.50th=[ 106], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 2043, max= 2043, per=8.23%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.4MiB/s (12.0MB/s)(10.0MiB/808msec) clat (msec): min=21, max=214, avg=66.49, stdev=62.42 lat (msec): min=21, max=214, avg=66.55, stdev=62.43 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 23], | 30.00th=[ 24], 40.00th=[ 35], 50.00th=[ 40], 60.00th=[ 43], | 70.00th=[ 72], 80.00th=[ 72], 90.00th=[ 138], 95.00th=[ 215], | 99.00th=[ 215], 99.50th=[ 215], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 6131, max= 6131, per=17.64%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=16.67%, 250=25.00% cpu : usr=0.12%, sys=16.60%, ctx=37, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18652: Mon Jun 9 10:33:57 2025 read: IOPS=7, BW=8138KiB/s (8333kB/s)(6144KiB/755msec) clat (usec): min=23958, max=93113, avg=40095.73, stdev=26432.93 lat (usec): min=23961, max=93114, avg=40097.44, stdev=26432.51 clat percentiles (usec): | 1.00th=[23987], 5.00th=[23987], 10.00th=[23987], 20.00th=[24249], | 30.00th=[24249], 40.00th=[28967], 50.00th=[28967], 60.00th=[34866], | 70.00th=[35390], 80.00th=[35390], 90.00th=[92799], 95.00th=[92799], | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 4096, max= 4096, per=16.50%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=7, BW=8138KiB/s (8333kB/s)(6144KiB/755msec) clat (msec): min=25, max=188, avg=84.48, stdev=62.26 lat (msec): min=25, max=188, avg=84.54, stdev=62.26 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 51], | 30.00th=[ 51], 40.00th=[ 55], 50.00th=[ 55], 60.00th=[ 57], | 70.00th=[ 133], 80.00th=[ 133], 90.00th=[ 188], 95.00th=[ 188], | 99.00th=[ 188], 99.50th=[ 188], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 6144, max= 6144, per=17.68%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=33.33%, 250=16.67% cpu : usr=0.00%, sys=11.14%, ctx=41, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18654: Mon Jun 9 10:33:57 2025 read: IOPS=6, BW=6206KiB/s (6355kB/s)(5120KiB/825msec) clat (usec): min=16509, max=66918, avg=34524.99, stdev=24053.51 lat (usec): min=16510, max=66919, avg=34526.35, stdev=24053.20 clat percentiles (usec): | 1.00th=[16450], 5.00th=[16450], 10.00th=[16450], 20.00th=[16450], | 30.00th=[17171], 40.00th=[17171], 50.00th=[18220], 60.00th=[18220], | 70.00th=[53740], 80.00th=[53740], 90.00th=[66847], 95.00th=[66847], | 99.00th=[66847], 99.50th=[66847], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] write: IOPS=8, BW=8688KiB/s (8897kB/s)(7168KiB/825msec) clat (msec): min=24, max=199, avg=84.63, stdev=76.20 lat (msec): min=24, max=199, avg=84.69, stdev=76.21 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 27], 40.00th=[ 27], 50.00th=[ 45], 60.00th=[ 90], | 70.00th=[ 90], 80.00th=[ 182], 90.00th=[ 201], 95.00th=[ 201], | 99.00th=[ 201], 99.50th=[ 201], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 4079, max= 4079, per=11.74%, avg=4079.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=33.33%, 100=25.00%, 250=16.67% cpu : usr=0.00%, sys=13.35%, ctx=77, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=24.2MiB/s (25.4MB/s), 2535KiB/s-12.7MiB/s (2595kB/s-13.3MB/s), io=20.0MiB (20.0MB), run=552-825msec WRITE: bw=33.9MiB/s (35.6MB/s), 8138KiB/s-12.4MiB/s (8333kB/s-12.0MB/s), io=28.0MiB (29.4MB), run=552-825msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=18660: Mon Jun 9 10:33:59 2025 read: IOPS=7, BW=28.6MiB/s (29.0MB/s)(12.0MiB/420msec) clat (msec): min=95, max=184, avg=136.28, stdev=44.95 lat (msec): min=95, max=184, avg=136.28, stdev=44.95 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 96], 10.00th=[ 96], 20.00th=[ 96], | 30.00th=[ 96], 40.00th=[ 129], 50.00th=[ 129], 60.00th=[ 129], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=36.28%, ctx=1597, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18661: Mon Jun 9 10:33:59 2025 read: IOPS=1, BW=4618KiB/s (4729kB/s)(4096KiB/887msec) clat (nsec): min=125005k, max=125005k, avg=125005455.00, stdev= 0.00 lat (nsec): min=125009k, max=125009k, avg=125008825.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 126], 10.00th=[ 126], 20.00th=[ 126], | 30.00th=[ 126], 40.00th=[ 126], 50.00th=[ 126], 60.00th=[ 126], | 70.00th=[ 126], 80.00th=[ 126], 90.00th=[ 126], 95.00th=[ 126], | 99.00th=[ 126], 99.50th=[ 126], 99.90th=[ 126], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 8192, max= 8192, per=44.35%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9236KiB/s (9457kB/s)(8192KiB/887msec) clat (msec): min=362, max=377, avg=370.03, stdev=10.78 lat (msec): min=362, max=377, avg=370.29, stdev=10.75 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 363], | 30.00th=[ 363], 40.00th=[ 363], 50.00th=[ 363], 60.00th=[ 380], | 70.00th=[ 380], 80.00th=[ 380], 90.00th=[ 380], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=25.17%, ctx=664, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18662: Mon Jun 9 10:33:59 2025 write: IOPS=3, BW=15.7MiB/s (16.4MB/s)(12.0MiB/765msec) clat (msec): min=119, max=339, avg=246.97, stdev=113.80 lat (msec): min=120, max=339, avg=247.24, stdev=113.74 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 121], 10.00th=[ 121], 20.00th=[ 121], | 30.00th=[ 121], 40.00th=[ 284], 50.00th=[ 284], 60.00th=[ 284], | 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 8192, max= 8192, per=22.18%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=36.52%, ctx=86, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18664: Mon Jun 9 10:33:59 2025 write: IOPS=5, BW=20.2MiB/s (21.2MB/s)(12.0MiB/593msec) clat (msec): min=110, max=276, avg=194.99, stdev=83.05 lat (msec): min=110, max=276, avg=195.32, stdev=83.06 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 199], 50.00th=[ 199], 60.00th=[ 199], | 70.00th=[ 275], 80.00th=[ 275], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min=16286, max=16286, per=44.09%, avg=16286.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=47.47%, ctx=67, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.0MiB/s (18.9MB/s), 4618KiB/s-28.6MiB/s (4729kB/s-29.0MB/s), io=16.0MiB (16.8MB), run=420-887msec WRITE: bw=36.1MiB/s (37.8MB/s), 9236KiB/s-20.2MiB/s (9457kB/s-21.2MB/s), io=32.0MiB (33.6MB), run=593-887msec rand-rw: (groupid=0, jobs=1): err= 0: pid=18663: Mon Jun 9 10:33:59 2025 read: IOPS=4, BW=19.0MiB/s (20.9MB/s)(12.0MiB/601msec) clat (msec): min=127, max=282, avg=197.54, stdev=78.52 lat (msec): min=127, max=282, avg=197.54, stdev=78.52 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 129], 10.00th=[ 129], 20.00th=[ 129], | 30.00th=[ 129], 40.00th=[ 182], 50.00th=[ 182], 60.00th=[ 182], | 70.00th=[ 284], 80.00th=[ 284], 90.00th=[ 284], 95.00th=[ 284], | 99.00th=[ 284], 99.50th=[ 284], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min=16384, max=16384, per=97.40%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.17%, sys=12.17%, ctx=40, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18665: Mon Jun 9 10:33:59 2025 read: IOPS=1, BW=4205KiB/s (4306kB/s)(4096KiB/974msec) clat (nsec): min=138373k, max=138373k, avg=138373101.00, stdev= 0.00 lat (nsec): min=138375k, max=138375k, avg=138375430.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 138], 10.00th=[ 138], 20.00th=[ 138], | 30.00th=[ 138], 40.00th=[ 138], 50.00th=[ 138], 60.00th=[ 138], | 70.00th=[ 138], 80.00th=[ 138], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 8192, max= 8192, per=48.70%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=8411KiB/s (8613kB/s)(8192KiB/974msec) clat (msec): min=317, max=488, avg=402.83, stdev=120.68 lat (msec): min=317, max=488, avg=403.16, stdev=120.62 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 317], 10.00th=[ 317], 20.00th=[ 317], | 30.00th=[ 317], 40.00th=[ 317], 50.00th=[ 317], 60.00th=[ 489], | 70.00th=[ 489], 80.00th=[ 489], 90.00th=[ 489], 95.00th=[ 489], | 99.00th=[ 489], 99.50th=[ 489], 99.90th=[ 489], 99.95th=[ 489], | 99.99th=[ 489] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=9.56%, ctx=80, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18666: Mon Jun 9 10:33:59 2025 write: IOPS=3, BW=12.4MiB/s (13.0MB/s)(12.0MiB/967msec) clat (msec): min=66, max=447, avg=305.80, stdev=208.59 lat (msec): min=66, max=447, avg=306.09, stdev=208.44 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 67], | 30.00th=[ 67], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 405], | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 8192, max= 8192, per=24.35%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.10%, sys=14.18%, ctx=61, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18667: Mon Jun 9 10:33:59 2025 write: IOPS=4, BW=16.3MiB/s (17.1MB/s)(12.0MiB/734msec) clat (msec): min=177, max=270, avg=232.48, stdev=48.78 lat (msec): min=177, max=270, avg=232.78, stdev=48.72 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 178], 10.00th=[ 178], 20.00th=[ 178], | 30.00th=[ 178], 40.00th=[ 251], 50.00th=[ 251], 60.00th=[ 251], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 8192, max= 8192, per=24.35%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.14%, sys=18.42%, ctx=64, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.4MiB/s (17.2MB/s), 4205KiB/s-19.0MiB/s (4306kB/s-20.9MB/s), io=16.0MiB (16.8MB), run=601-974msec WRITE: bw=32.9MiB/s (34.5MB/s), 8411KiB/s-16.3MiB/s (8613kB/s-17.1MB/s), io=32.0MiB (33.6MB), run=734-974msec