== sanity test 398b: DIO and buffer IO race ============== 22:22:39 (1746152559) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.46166 s, 20.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=697: Thu May 1 22:23:43 2025 read: IOPS=25, BW=102KiB/s (105kB/s)(5992KiB/58589msec) clat (usec): min=54, max=360393, avg=17026.64, stdev=19388.14 lat (usec): min=54, max=360394, avg=17029.63, stdev=19387.90 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 104], 10.00th=[ 119], 20.00th=[ 178], | 30.00th=[ 8225], 40.00th=[ 11207], 50.00th=[ 13698], 60.00th=[ 16909], | 70.00th=[ 20579], 80.00th=[ 26608], 90.00th=[ 36963], 95.00th=[ 46400], | 99.00th=[ 73925], 99.50th=[ 91751], 99.90th=[193987], 99.95th=[358613], | 99.99th=[358613] bw ( KiB/s): min= 32, max= 224, per=25.05%, avg=102.45, stdev=38.25, samples=116 iops : min= 8, max= 56, avg=25.28, stdev= 9.58, samples=116 write: IOPS=26, BW=107KiB/s (110kB/s)(6296KiB/58589msec) clat (usec): min=599, max=196639, avg=20821.11, stdev=21746.27 lat (usec): min=600, max=196640, avg=20824.75, stdev=21746.97 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 865], 10.00th=[ 1037], 20.00th=[ 1975], | 30.00th=[ 4424], 40.00th=[ 9634], 50.00th=[ 15401], 60.00th=[ 20841], | 70.00th=[ 27657], 80.00th=[ 35390], 90.00th=[ 50070], 95.00th=[ 62653], | 99.00th=[ 88605], 99.50th=[108528], 99.90th=[185598], 99.95th=[196084], | 99.99th=[196084] bw ( KiB/s): min= 40, max= 272, per=25.05%, avg=107.47, stdev=41.72, samples=116 iops : min= 10, max= 68, avg=26.53, stdev=10.48, samples=116 lat (usec) : 100=1.89%, 250=8.17%, 500=1.11%, 750=0.65%, 1000=4.20% lat (msec) : 2=6.09%, 4=4.59%, 10=11.33%, 20=25.07%, 50=29.85% lat (msec) : 100=6.48%, 250=0.55%, 500=0.03% cpu : usr=0.13%, sys=9.83%, ctx=5077, majf=0, minf=37 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=698: Thu May 1 22:23:43 2025 read: IOPS=25, BW=102KiB/s (104kB/s)(5940KiB/58484msec) clat (usec): min=56, max=141358, avg=17259.40, stdev=15505.45 lat (usec): min=57, max=141360, avg=17264.35, stdev=15502.67 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 108], 10.00th=[ 124], 20.00th=[ 343], | 30.00th=[ 9241], 40.00th=[ 11731], 50.00th=[ 14484], 60.00th=[ 17433], | 70.00th=[ 21627], 80.00th=[ 27919], 90.00th=[ 36963], 95.00th=[ 46924], | 99.00th=[ 68682], 99.50th=[ 76022], 99.90th=[ 99091], 99.95th=[141558], | 99.99th=[141558] bw ( KiB/s): min= 8, max= 199, per=24.53%, avg=100.33, stdev=38.53, samples=116 iops : min= 2, max= 49, avg=24.71, stdev= 9.61, samples=116 write: IOPS=27, BW=109KiB/s (111kB/s)(6348KiB/58484msec) clat (usec): min=638, max=345841, avg=20506.13, stdev=24039.55 lat (usec): min=639, max=345843, avg=20508.99, stdev=24040.00 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 873], 10.00th=[ 1045], 20.00th=[ 1729], | 30.00th=[ 3916], 40.00th=[ 8291], 50.00th=[ 14091], 60.00th=[ 19268], | 70.00th=[ 25560], 80.00th=[ 34866], 90.00th=[ 50594], 95.00th=[ 62653], | 99.00th=[102237], 99.50th=[114820], 99.90th=[261096], 99.95th=[346031], | 99.99th=[346031] bw ( KiB/s): min= 39, max= 263, per=25.11%, avg=107.73, stdev=40.88, samples=115 iops : min= 9, max= 65, avg=26.54, stdev=10.27, samples=115 lat (usec) : 100=1.33%, 250=7.68%, 500=1.01%, 750=0.65%, 1000=4.10% lat (msec) : 2=7.52%, 4=4.33%, 10=11.17%, 20=26.11%, 50=29.07% lat (msec) : 100=6.48%, 250=0.49%, 500=0.07% cpu : usr=0.13%, sys=10.02%, ctx=5073, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=699: Thu May 1 22:23:43 2025 read: IOPS=25, BW=104KiB/s (106kB/s)(6028KiB/58202msec) clat (usec): min=65, max=133126, avg=16366.30, stdev=15248.36 lat (usec): min=65, max=133128, avg=16371.02, stdev=15246.46 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 105], 10.00th=[ 122], 20.00th=[ 310], | 30.00th=[ 8160], 40.00th=[ 10945], 50.00th=[ 14091], 60.00th=[ 16909], | 70.00th=[ 20579], 80.00th=[ 26084], 90.00th=[ 35914], 95.00th=[ 44827], | 99.00th=[ 62129], 99.50th=[ 70779], 99.90th=[129500], 99.95th=[132645], | 99.99th=[132645] bw ( KiB/s): min= 32, max= 232, per=25.36%, avg=103.73, stdev=40.24, samples=115 iops : min= 8, max= 58, avg=25.53, stdev=10.07, samples=115 write: IOPS=26, BW=108KiB/s (110kB/s)(6260KiB/58202msec) clat (usec): min=556, max=358724, avg=21220.74, stdev=24068.13 lat (usec): min=557, max=358725, avg=21227.93, stdev=24070.51 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 873], 10.00th=[ 1037], 20.00th=[ 1729], | 30.00th=[ 4146], 40.00th=[ 9634], 50.00th=[ 15139], 60.00th=[ 20579], | 70.00th=[ 26870], 80.00th=[ 36439], 90.00th=[ 50070], 95.00th=[ 66323], | 99.00th=[ 95945], 99.50th=[117965], 99.90th=[240124], 99.95th=[358613], | 99.99th=[358613] bw ( KiB/s): min= 16, max= 256, per=24.88%, avg=106.73, stdev=37.63, samples=116 iops : min= 4, max= 64, avg=26.29, stdev= 9.40, samples=116 lat (usec) : 100=1.60%, 250=7.88%, 500=1.11%, 750=0.62%, 1000=4.23% lat (msec) : 2=7.10%, 4=4.49%, 10=11.26%, 20=25.39%, 50=29.36% lat (msec) : 100=6.35%, 250=0.59%, 500=0.03% cpu : usr=0.12%, sys=9.86%, ctx=5039, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=701: Thu May 1 22:23:43 2025 read: IOPS=26, BW=105KiB/s (107kB/s)(6052KiB/57861msec) clat (usec): min=57, max=403584, avg=16235.69, stdev=18353.81 lat (usec): min=58, max=403584, avg=16239.24, stdev=18353.36 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 103], 10.00th=[ 119], 20.00th=[ 281], | 30.00th=[ 7898], 40.00th=[ 10945], 50.00th=[ 13829], 60.00th=[ 16450], | 70.00th=[ 19792], 80.00th=[ 25297], 90.00th=[ 34341], 95.00th=[ 44303], | 99.00th=[ 65799], 99.50th=[ 84411], 99.90th=[179307], 99.95th=[404751], | 99.99th=[404751] bw ( KiB/s): min= 32, max= 248, per=25.66%, avg=104.96, stdev=39.80, samples=114 iops : min= 8, max= 62, avg=25.84, stdev= 9.97, samples=114 write: IOPS=26, BW=108KiB/s (110kB/s)(6236KiB/57861msec) clat (usec): min=545, max=196681, avg=21175.22, stdev=22718.50 lat (usec): min=546, max=196683, avg=21176.94, stdev=22718.54 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 889], 10.00th=[ 1012], 20.00th=[ 1663], | 30.00th=[ 3752], 40.00th=[ 8848], 50.00th=[ 15008], 60.00th=[ 20841], | 70.00th=[ 27657], 80.00th=[ 36963], 90.00th=[ 50594], 95.00th=[ 63701], | 99.00th=[ 95945], 99.50th=[114820], 99.90th=[149947], 99.95th=[196084], | 99.99th=[196084] bw ( KiB/s): min= 32, max= 239, per=24.96%, avg=107.10, stdev=41.11, samples=115 iops : min= 8, max= 59, avg=26.38, stdev=10.28, samples=115 lat (usec) : 100=1.92%, 250=7.88%, 500=1.24%, 750=0.59%, 1000=4.33% lat (msec) : 2=7.03%, 4=4.79%, 10=11.23%, 20=25.23%, 50=29.00% lat (msec) : 100=6.25%, 250=0.49%, 500=0.03% cpu : usr=0.13%, sys=9.74%, ctx=5019, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=410KiB/s (420kB/s), 102KiB/s-105KiB/s (104kB/s-107kB/s), io=23.4MiB (24.6MB), run=57861-58589msec WRITE: bw=429KiB/s (439kB/s), 107KiB/s-109KiB/s (110kB/s-111kB/s), io=24.6MiB (25.7MB), run=57861-58589msec rand-rw: (groupid=0, jobs=1): err= 0: pid=700: Thu May 1 22:26:25 2025 read: IOPS=6, BW=27.1KiB/s (27.8kB/s)(5992KiB/220989msec) clat (msec): min=3, max=364, avg=20.68, stdev=19.08 lat (msec): min=3, max=364, avg=20.68, stdev=19.08 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 11], | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 20], | 70.00th=[ 23], 80.00th=[ 27], 90.00th=[ 34], 95.00th=[ 47], | 99.00th=[ 96], 99.50th=[ 118], 99.90th=[ 213], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 7, max= 96, per=27.38%, avg=29.57, stdev=18.79, samples=398 iops : min= 1, max= 24, avg= 7.03, stdev= 4.76, samples=398 write: IOPS=7, BW=28.5KiB/s (29.2kB/s)(6296KiB/220989msec) clat (msec): min=31, max=422, avg=120.54, stdev=50.60 lat (msec): min=31, max=422, avg=120.54, stdev=50.60 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 64], 10.00th=[ 70], 20.00th=[ 82], | 30.00th=[ 92], 40.00th=[ 101], 50.00th=[ 110], 60.00th=[ 121], | 70.00th=[ 134], 80.00th=[ 153], 90.00th=[ 184], 95.00th=[ 220], | 99.00th=[ 305], 99.50th=[ 330], 99.90th=[ 409], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 7, max= 64, per=24.85%, avg=28.08, stdev= 9.67, samples=439 iops : min= 1, max= 16, avg= 6.65, stdev= 2.49, samples=439 lat (msec) : 4=0.10%, 10=9.64%, 20=19.99%, 50=17.42%, 100=21.81% lat (msec) : 250=29.75%, 500=1.30% cpu : usr=0.05%, sys=2.07%, ctx=4098, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=702: Thu May 1 22:26:25 2025 read: IOPS=6, BW=26.8KiB/s (27.5kB/s)(5940KiB/221293msec) clat (msec): min=3, max=406, avg=20.37, stdev=18.86 lat (msec): min=3, max=406, avg=20.37, stdev=18.86 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 12], 40.00th=[ 14], 50.00th=[ 17], 60.00th=[ 20], | 70.00th=[ 22], 80.00th=[ 26], 90.00th=[ 35], 95.00th=[ 49], | 99.00th=[ 86], 99.50th=[ 106], 99.90th=[ 213], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 7, max= 103, per=26.88%, avg=29.03, stdev=18.90, samples=403 iops : min= 1, max= 25, avg= 6.99, stdev= 4.80, samples=403 write: IOPS=7, BW=28.7KiB/s (29.4kB/s)(6348KiB/221293msec) clat (msec): min=40, max=412, avg=120.19, stdev=50.33 lat (msec): min=40, max=412, avg=120.20, stdev=50.33 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 61], 10.00th=[ 69], 20.00th=[ 82], | 30.00th=[ 91], 40.00th=[ 100], 50.00th=[ 110], 60.00th=[ 121], | 70.00th=[ 136], 80.00th=[ 150], 90.00th=[ 188], 95.00th=[ 222], | 99.00th=[ 300], 99.50th=[ 321], 99.90th=[ 347], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 7, max= 64, per=25.14%, avg=28.40, stdev= 9.64, samples=440 iops : min= 1, max= 16, avg= 6.84, stdev= 2.47, samples=440 lat (msec) : 4=0.03%, 10=10.64%, 20=19.92%, 50=16.15%, 100=22.40% lat (msec) : 250=29.46%, 500=1.40% cpu : usr=0.04%, sys=2.06%, ctx=4041, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=703: Thu May 1 22:26:25 2025 read: IOPS=6, BW=27.3KiB/s (27.0kB/s)(6028KiB/220605msec) clat (msec): min=3, max=188, avg=19.97, stdev=15.39 lat (msec): min=3, max=188, avg=19.98, stdev=15.39 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 12], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 19], | 70.00th=[ 22], 80.00th=[ 26], 90.00th=[ 35], 95.00th=[ 47], | 99.00th=[ 85], 99.50th=[ 106], 99.90th=[ 127], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 7, max= 127, per=27.15%, avg=29.33, stdev=19.69, samples=405 iops : min= 1, max= 31, avg= 7.02, stdev= 4.94, samples=405 write: IOPS=7, BW=28.4KiB/s (29.1kB/s)(6260KiB/220605msec) clat (msec): min=36, max=491, avg=121.52, stdev=50.33 lat (msec): min=36, max=491, avg=121.53, stdev=50.33 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 64], 10.00th=[ 70], 20.00th=[ 83], | 30.00th=[ 93], 40.00th=[ 103], 50.00th=[ 112], 60.00th=[ 123], | 70.00th=[ 136], 80.00th=[ 155], 90.00th=[ 184], 95.00th=[ 215], | 99.00th=[ 296], 99.50th=[ 321], 99.90th=[ 409], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 7, max= 56, per=24.77%, avg=27.99, stdev= 9.67, samples=440 iops : min= 1, max= 14, avg= 6.69, stdev= 2.50, samples=440 lat (msec) : 4=0.03%, 10=10.29%, 20=21.81%, 50=15.53%, 100=20.61% lat (msec) : 250=30.40%, 500=1.33% cpu : usr=0.05%, sys=2.05%, ctx=4010, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=704: Thu May 1 22:26:25 2025 read: IOPS=6, BW=27.7KiB/s (28.3kB/s)(6052KiB/218765msec) clat (msec): min=4, max=190, avg=19.69, stdev=15.30 lat (msec): min=4, max=190, avg=19.69, stdev=15.30 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 12], 40.00th=[ 14], 50.00th=[ 17], 60.00th=[ 19], | 70.00th=[ 22], 80.00th=[ 27], 90.00th=[ 34], 95.00th=[ 46], | 99.00th=[ 75], 99.50th=[ 104], 99.90th=[ 184], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 7, max= 119, per=28.10%, avg=30.35, stdev=19.53, samples=393 iops : min= 1, max= 29, avg= 7.27, stdev= 4.90, samples=393 write: IOPS=7, BW=28.5KiB/s (29.2kB/s)(6236KiB/218765msec) clat (msec): min=41, max=491, avg=121.02, stdev=51.77 lat (msec): min=41, max=491, avg=121.03, stdev=51.77 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 63], 10.00th=[ 70], 20.00th=[ 82], | 30.00th=[ 91], 40.00th=[ 101], 50.00th=[ 110], 60.00th=[ 123], | 70.00th=[ 136], 80.00th=[ 153], 90.00th=[ 180], 95.00th=[ 215], | 99.00th=[ 330], 99.50th=[ 355], 99.90th=[ 418], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 7, max= 56, per=24.90%, avg=28.14, stdev= 9.69, samples=436 iops : min= 1, max= 14, avg= 6.72, stdev= 2.50, samples=436 lat (msec) : 10=11.23%, 20=20.67%, 50=15.98%, 100=21.48%, 250=29.33% lat (msec) : 500=1.30% cpu : usr=0.06%, sys=2.05%, ctx=3974, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=109KiB/s (111kB/s), 26.8KiB/s-27.7KiB/s (27.5kB/s-28.3kB/s), io=23.4MiB (24.6MB), run=218765-221293msec WRITE: bw=114KiB/s (116kB/s), 28.4KiB/s-28.7KiB/s (29.1kB/s-29.4kB/s), io=24.6MiB (25.7MB), run=218765-221293msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=929: Thu May 1 22:26:43 2025 read: IOPS=24, BW=398KiB/s (407kB/s)(5920KiB/14878msec) clat (usec): min=83, max=371376, avg=18520.70, stdev=25769.06 lat (usec): min=84, max=371377, avg=18522.16, stdev=25769.05 clat percentiles (usec): | 1.00th=[ 96], 5.00th=[ 110], 10.00th=[ 121], 20.00th=[ 145], | 30.00th=[ 178], 40.00th=[ 10028], 50.00th=[ 16057], 60.00th=[ 20579], | 70.00th=[ 24511], 80.00th=[ 30016], 90.00th=[ 41157], 95.00th=[ 54264], | 99.00th=[ 86508], 99.50th=[102237], 99.90th=[371196], 99.95th=[371196], | 99.99th=[371196] bw ( KiB/s): min= 63, max= 798, per=24.38%, avg=387.59, stdev=167.14, samples=29 iops : min= 3, max= 49, avg=23.72, stdev=10.48, samples=29 write: IOPS=26, BW=428KiB/s (438kB/s)(6368KiB/14878msec) clat (usec): min=1010, max=204760, avg=20078.99, stdev=27446.64 lat (usec): min=1012, max=204763, avg=20083.03, stdev=27446.07 clat percentiles (usec): | 1.00th=[ 1020], 5.00th=[ 1254], 10.00th=[ 1532], 20.00th=[ 1860], | 30.00th=[ 3490], 40.00th=[ 5735], 50.00th=[ 8029], 60.00th=[ 13435], | 70.00th=[ 23200], 80.00th=[ 35914], 90.00th=[ 51119], 95.00th=[ 67634], | 99.00th=[152044], 99.50th=[173016], 99.90th=[204473], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 63, max= 672, per=25.14%, avg=420.31, stdev=142.89, samples=29 iops : min= 3, max= 42, avg=25.79, stdev= 9.06, samples=29 lat (usec) : 100=1.30%, 250=13.80%, 500=1.43%, 750=0.13%, 1000=0.39% lat (msec) : 2=11.46%, 4=5.60%, 10=12.63%, 20=16.15%, 50=28.65% lat (msec) : 100=7.29%, 250=1.04%, 500=0.13% cpu : usr=0.11%, sys=9.61%, ctx=1075, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=930: Thu May 1 22:26:43 2025 read: IOPS=24, BW=399KiB/s (409kB/s)(5744KiB/14387msec) clat (usec): min=78, max=108255, avg=17131.82, stdev=17103.72 lat (usec): min=78, max=108257, avg=17133.28, stdev=17103.78 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 109], 10.00th=[ 122], 20.00th=[ 147], | 30.00th=[ 277], 40.00th=[ 11076], 50.00th=[ 15664], 60.00th=[ 19792], | 70.00th=[ 25035], 80.00th=[ 31065], 90.00th=[ 39060], 95.00th=[ 45351], | 99.00th=[ 62653], 99.50th=[105382], 99.90th=[108528], 99.95th=[108528], | 99.99th=[108528] bw ( KiB/s): min= 31, max= 717, per=24.88%, avg=395.61, stdev=153.12, samples=28 iops : min= 1, max= 44, avg=24.11, stdev= 9.71, samples=28 write: IOPS=28, BW=455KiB/s (466kB/s)(6544KiB/14387msec) clat (usec): min=940, max=203688, avg=20054.67, stdev=27676.06 lat (usec): min=941, max=203691, avg=20058.53, stdev=27675.85 clat percentiles (usec): | 1.00th=[ 1074], 5.00th=[ 1254], 10.00th=[ 1450], 20.00th=[ 1860], | 30.00th=[ 3032], 40.00th=[ 5145], 50.00th=[ 7570], 60.00th=[ 13698], | 70.00th=[ 21890], 80.00th=[ 34866], 90.00th=[ 56361], 95.00th=[ 73925], | 99.00th=[143655], 99.50th=[177210], 99.90th=[204473], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 126, max= 827, per=26.58%, avg=444.43, stdev=157.93, samples=28 iops : min= 7, max= 51, avg=27.18, stdev= 9.86, samples=28 lat (usec) : 100=1.69%, 250=12.11%, 500=2.08%, 1000=0.39% lat (msec) : 2=11.59%, 4=7.68%, 10=10.81%, 20=18.10%, 50=26.69% lat (msec) : 100=7.81%, 250=1.04% cpu : usr=0.19%, sys=9.91%, ctx=1059, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=931: Thu May 1 22:26:43 2025 read: IOPS=25, BW=415KiB/s (425kB/s)(6256KiB/15064msec) clat (usec): min=77, max=160610, avg=17175.70, stdev=19640.62 lat (usec): min=78, max=160610, avg=17177.12, stdev=19640.56 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 103], 10.00th=[ 118], 20.00th=[ 145], | 30.00th=[ 180], 40.00th=[ 8455], 50.00th=[ 15533], 60.00th=[ 19530], | 70.00th=[ 23462], 80.00th=[ 29492], 90.00th=[ 40109], 95.00th=[ 50594], | 99.00th=[ 80217], 99.50th=[158335], 99.90th=[160433], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 31, max= 768, per=25.72%, avg=409.00, stdev=155.55, samples=29 iops : min= 1, max= 48, avg=25.10, stdev= 9.78, samples=29 write: IOPS=25, BW=400KiB/s (410kB/s)(6032KiB/15064msec) clat (usec): min=987, max=232839, avg=22041.24, stdev=29880.48 lat (usec): min=988, max=232842, avg=22044.68, stdev=29879.98 clat percentiles (usec): | 1.00th=[ 1074], 5.00th=[ 1303], 10.00th=[ 1483], 20.00th=[ 1893], | 30.00th=[ 3884], 40.00th=[ 6849], 50.00th=[ 10421], 60.00th=[ 16188], | 70.00th=[ 25560], 80.00th=[ 38011], 90.00th=[ 53216], 95.00th=[ 81265], | 99.00th=[139461], 99.50th=[219153], 99.90th=[233833], 99.95th=[233833], | 99.99th=[233833] bw ( KiB/s): min= 95, max= 704, per=22.95%, avg=383.66, stdev=126.65, samples=29 iops : min= 5, max= 44, avg=23.52, stdev= 7.97, samples=29 lat (usec) : 100=1.95%, 250=14.45%, 500=1.95%, 750=0.13%, 1000=0.39% lat (msec) : 2=10.94%, 4=4.82%, 10=10.42%, 20=17.97%, 50=28.12% lat (msec) : 100=7.16%, 250=1.69% cpu : usr=0.17%, sys=9.44%, ctx=1048, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=932: Thu May 1 22:26:43 2025 read: IOPS=25, BW=409KiB/s (419kB/s)(6032KiB/14754msec) clat (usec): min=71, max=256869, avg=18894.42, stdev=21932.65 lat (usec): min=72, max=256870, avg=18895.87, stdev=21932.71 clat percentiles (usec): | 1.00th=[ 96], 5.00th=[ 108], 10.00th=[ 124], 20.00th=[ 147], | 30.00th=[ 326], 40.00th=[ 10290], 50.00th=[ 17433], 60.00th=[ 22938], | 70.00th=[ 26608], 80.00th=[ 32375], 90.00th=[ 41681], 95.00th=[ 50594], | 99.00th=[ 78119], 99.50th=[119014], 99.90th=[256902], 99.95th=[256902], | 99.99th=[256902] bw ( KiB/s): min= 190, max= 896, per=26.26%, avg=417.61, stdev=153.68, samples=28 iops : min= 11, max= 56, avg=25.57, stdev= 9.81, samples=28 write: IOPS=26, BW=424KiB/s (434kB/s)(6256KiB/14754msec) clat (usec): min=1048, max=175099, avg=19354.71, stdev=27045.31 lat (usec): min=1052, max=175103, avg=19359.73, stdev=27044.66 clat percentiles (usec): | 1.00th=[ 1090], 5.00th=[ 1319], 10.00th=[ 1483], 20.00th=[ 2008], | 30.00th=[ 3097], 40.00th=[ 5276], 50.00th=[ 8291], 60.00th=[ 13566], | 70.00th=[ 21103], 80.00th=[ 32113], 90.00th=[ 49021], 95.00th=[ 64226], | 99.00th=[149947], 99.50th=[168821], 99.90th=[175113], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 95, max= 672, per=25.07%, avg=419.24, stdev=172.83, samples=29 iops : min= 5, max= 42, avg=25.69, stdev=10.97, samples=29 lat (usec) : 100=1.17%, 250=12.76%, 500=2.08%, 750=0.39%, 1000=0.13% lat (msec) : 2=10.94%, 4=8.20%, 10=10.94%, 20=15.23%, 50=31.12% lat (msec) : 100=5.47%, 250=1.43%, 500=0.13% cpu : usr=0.18%, sys=9.54%, ctx=1049, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1590KiB/s (1628kB/s), 398KiB/s-415KiB/s (407kB/s-425kB/s), io=23.4MiB (24.5MB), run=14387-15064msec WRITE: bw=1673KiB/s (1713kB/s), 400KiB/s-455KiB/s (410kB/s-466kB/s), io=24.6MiB (25.8MB), run=14387-15064msec rand-rw: (groupid=0, jobs=1): err= 0: pid=924: Thu May 1 22:27:29 2025 read: IOPS=6, BW=97.0KiB/s (99.3kB/s)(5920KiB/61020msec) clat (msec): min=5, max=196, avg=19.87, stdev=22.44 lat (msec): min=5, max=196, avg=19.87, stdev=22.45 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 13], 50.00th=[ 14], 60.00th=[ 16], | 70.00th=[ 17], 80.00th=[ 22], 90.00th=[ 33], 95.00th=[ 62], | 99.00th=[ 130], 99.50th=[ 176], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 31, max= 415, per=28.36%, avg=110.03, stdev=68.91, samples=107 iops : min= 1, max= 25, avg= 6.35, stdev= 4.32, samples=107 write: IOPS=6, BW=104KiB/s (107kB/s)(6368KiB/61020msec) clat (msec): min=47, max=363, avg=134.47, stdev=58.22 lat (msec): min=47, max=363, avg=134.48, stdev=58.22 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 67], 10.00th=[ 80], 20.00th=[ 89], | 30.00th=[ 100], 40.00th=[ 110], 50.00th=[ 123], 60.00th=[ 134], | 70.00th=[ 150], 80.00th=[ 167], 90.00th=[ 203], 95.00th=[ 264], | 99.00th=[ 347], 99.50th=[ 363], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 31, max= 192, per=25.37%, avg=103.50, stdev=38.31, samples=122 iops : min= 1, max= 12, avg= 5.92, stdev= 2.50, samples=122 lat (msec) : 10=9.77%, 20=27.99%, 50=7.68%, 100=17.45%, 250=34.24% lat (msec) : 500=2.86% cpu : usr=0.04%, sys=2.02%, ctx=932, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=925: Thu May 1 22:27:29 2025 read: IOPS=5, BW=93.1KiB/s (95.3kB/s)(5744KiB/61700msec) clat (msec): min=4, max=184, avg=18.80, stdev=19.21 lat (msec): min=4, max=184, avg=18.80, stdev=19.21 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 15], | 70.00th=[ 17], 80.00th=[ 21], 90.00th=[ 34], 95.00th=[ 57], | 99.00th=[ 121], 99.50th=[ 128], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 31, max= 288, per=27.46%, avg=106.54, stdev=68.24, samples=107 iops : min= 1, max= 18, avg= 6.24, stdev= 4.38, samples=107 write: IOPS=6, BW=106KiB/s (109kB/s)(6544KiB/61700msec) clat (msec): min=52, max=451, avg=134.21, stdev=59.29 lat (msec): min=52, max=451, avg=134.21, stdev=59.29 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 67], 10.00th=[ 74], 20.00th=[ 90], | 30.00th=[ 104], 40.00th=[ 113], 50.00th=[ 123], 60.00th=[ 134], | 70.00th=[ 148], 80.00th=[ 163], 90.00th=[ 207], 95.00th=[ 249], | 99.00th=[ 359], 99.50th=[ 388], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 31, max= 192, per=25.98%, avg=106.01, stdev=39.81, samples=122 iops : min= 1, max= 12, avg= 6.20, stdev= 2.59, samples=122 lat (msec) : 10=11.85%, 20=24.87%, 50=7.29%, 100=17.58%, 250=35.94% lat (msec) : 500=2.47% cpu : usr=0.04%, sys=1.98%, ctx=903, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=926: Thu May 1 22:27:29 2025 read: IOPS=6, BW=107KiB/s (109kB/s)(6256KiB/58528msec) clat (msec): min=3, max=174, avg=17.94, stdev=18.84 lat (msec): min=3, max=174, avg=17.94, stdev=18.84 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 13], 50.00th=[ 14], 60.00th=[ 15], | 70.00th=[ 17], 80.00th=[ 20], 90.00th=[ 27], 95.00th=[ 43], | 99.00th=[ 127], 99.50th=[ 157], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 31, max= 415, per=30.85%, avg=119.71, stdev=87.75, samples=104 iops : min= 1, max= 25, avg= 6.97, stdev= 5.55, samples=104 write: IOPS=6, BW=103KiB/s (106kB/s)(6032KiB/58528msec) clat (msec): min=53, max=579, avg=136.24, stdev=64.81 lat (msec): min=53, max=579, avg=136.25, stdev=64.81 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 68], 10.00th=[ 74], 20.00th=[ 89], | 30.00th=[ 102], 40.00th=[ 110], 50.00th=[ 124], 60.00th=[ 133], | 70.00th=[ 148], 80.00th=[ 169], 90.00th=[ 224], 95.00th=[ 268], | 99.00th=[ 388], 99.50th=[ 414], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 31, max= 192, per=25.06%, avg=102.23, stdev=38.33, samples=117 iops : min= 1, max= 12, avg= 5.85, stdev= 2.55, samples=117 lat (msec) : 4=0.13%, 10=12.37%, 20=29.04%, 50=7.55%, 100=15.36% lat (msec) : 250=32.55%, 500=2.86%, 750=0.13% cpu : usr=0.04%, sys=2.04%, ctx=929, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=927: Thu May 1 22:27:29 2025 read: IOPS=6, BW=99.2KiB/s (102kB/s)(6032KiB/60836msec) clat (msec): min=5, max=180, avg=20.16, stdev=22.95 lat (msec): min=5, max=180, avg=20.17, stdev=22.95 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 15], | 70.00th=[ 17], 80.00th=[ 22], 90.00th=[ 37], 95.00th=[ 62], | 99.00th=[ 146], 99.50th=[ 176], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 31, max= 319, per=28.48%, avg=110.49, stdev=71.66, samples=107 iops : min= 1, max= 19, avg= 6.51, stdev= 4.50, samples=107 write: IOPS=6, BW=103KiB/s (105kB/s)(6256KiB/60836msec) clat (msec): min=46, max=501, avg=135.79, stdev=54.12 lat (msec): min=46, max=501, avg=135.80, stdev=54.12 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 75], 10.00th=[ 81], 20.00th=[ 93], | 30.00th=[ 104], 40.00th=[ 115], 50.00th=[ 126], 60.00th=[ 136], | 70.00th=[ 153], 80.00th=[ 169], 90.00th=[ 203], 95.00th=[ 243], | 99.00th=[ 342], 99.50th=[ 368], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 31, max= 190, per=25.04%, avg=102.15, stdev=34.53, samples=121 iops : min= 1, max= 11, avg= 5.98, stdev= 2.23, samples=121 lat (msec) : 10=11.72%, 20=25.78%, 50=8.46%, 100=15.23%, 250=36.59% lat (msec) : 500=2.08%, 750=0.13% cpu : usr=0.07%, sys=1.95%, ctx=926, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=388KiB/s (398kB/s), 93.1KiB/s-107KiB/s (95.3kB/s-109kB/s), io=23.4MiB (24.5MB), run=58528-61700msec WRITE: bw=408KiB/s (418kB/s), 103KiB/s-106KiB/s (105kB/s-109kB/s), io=24.6MiB (25.8MB), run=58528-61700msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=994: Thu May 1 22:27:36 2025 read: IOPS=2, BW=2107KiB/s (2158kB/s)(7168KiB/3402msec) clat (msec): min=14, max=2013, avg=355.68, stdev=731.89 lat (msec): min=14, max=2013, avg=355.68, stdev=731.89 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 59], | 30.00th=[ 82], 40.00th=[ 82], 50.00th=[ 86], 60.00th=[ 107], | 70.00th=[ 107], 80.00th=[ 130], 90.00th=[ 2022], 95.00th=[ 2022], | 99.00th=[ 2022], 99.50th=[ 2022], 99.90th=[ 2022], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 4096, max= 6023, per=100.00%, avg=5059.50, stdev=1362.59, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=1, BW=1505KiB/s (1541kB/s)(5120KiB/3402msec) clat (msec): min=53, max=357, avg=170.29, stdev=116.54 lat (msec): min=53, max=357, avg=170.53, stdev=116.85 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 54], 10.00th=[ 54], 20.00th=[ 54], | 30.00th=[ 95], 40.00th=[ 95], 50.00th=[ 165], 60.00th=[ 165], | 70.00th=[ 182], 80.00th=[ 182], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 2007, max= 6083, per=66.82%, avg=4045.00, stdev=2882.17, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 lat (msec) : 20=8.33%, 100=41.67%, 250=33.33%, 500=8.33% cpu : usr=0.18%, sys=13.32%, ctx=684, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=995: Thu May 1 22:27:36 2025 read: IOPS=0, BW=742KiB/s (760kB/s)(2048KiB/2761msec) clat (msec): min=89, max=1250, avg=669.87, stdev=821.46 lat (msec): min=89, max=1250, avg=669.87, stdev=821.46 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 89], 10.00th=[ 89], 20.00th=[ 89], | 30.00th=[ 89], 40.00th=[ 89], 50.00th=[ 89], 60.00th=[ 1250], | 70.00th=[ 1250], 80.00th=[ 1250], 90.00th=[ 1250], 95.00th=[ 1250], | 99.00th=[ 1250], 99.50th=[ 1250], 99.90th=[ 1250], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 2043, max= 2048, per=47.31%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3709KiB/s (3798kB/s)(10.0MiB/2761msec) clat (msec): min=42, max=277, avg=124.00, stdev=77.31 lat (msec): min=42, max=277, avg=124.09, stdev=77.32 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 53], | 30.00th=[ 66], 40.00th=[ 90], 50.00th=[ 99], 60.00th=[ 107], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 232], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 2043, max= 8175, per=67.54%, avg=4088.67, stdev=3538.87, samples=3 iops : min= 1, max= 7, avg= 3.33, stdev= 3.21, samples=3 lat (msec) : 50=8.33%, 100=41.67%, 250=33.33%, 500=8.33% cpu : usr=0.00%, sys=11.81%, ctx=255, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=996: Thu May 1 22:27:36 2025 read: IOPS=1, BW=1899KiB/s (1945kB/s)(6144KiB/3235msec) clat (usec): min=790, max=256943, avg=108285.99, stdev=96097.57 lat (usec): min=792, max=256945, avg=108287.53, stdev=96097.59 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 791], 10.00th=[ 791], 20.00th=[ 7177], | 30.00th=[ 7177], 40.00th=[108528], 50.00th=[108528], 60.00th=[122160], | 70.00th=[154141], 80.00th=[154141], 90.00th=[256902], 95.00th=[256902], | 99.00th=[256902], 99.50th=[256902], 99.90th=[256902], 99.95th=[256902], | 99.99th=[256902] bw ( KiB/s): min= 4063, max= 6144, per=100.00%, avg=5103.50, stdev=1471.49, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 write: IOPS=1, BW=1899KiB/s (1945kB/s)(6144KiB/3235msec) clat (msec): min=41, max=2107, avg=417.26, stdev=828.94 lat (msec): min=41, max=2107, avg=417.41, stdev=828.92 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 57], | 30.00th=[ 57], 40.00th=[ 66], 50.00th=[ 66], 60.00th=[ 77], | 70.00th=[ 159], 80.00th=[ 159], 90.00th=[ 2106], 95.00th=[ 2106], | 99.00th=[ 2106], 99.50th=[ 2106], 99.90th=[ 2106], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 2031, max= 4096, per=56.29%, avg=3407.67, stdev=1192.23, samples=3 iops : min= 1, max= 4, avg= 3.00, stdev= 1.73, samples=3 lat (usec) : 1000=8.33% lat (msec) : 10=8.33%, 50=8.33%, 100=25.00%, 250=33.33%, 500=8.33% cpu : usr=0.00%, sys=7.58%, ctx=174, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=997: Thu May 1 22:27:36 2025 read: IOPS=1, BW=1081KiB/s (1107kB/s)(5120KiB/4736msec) clat (msec): min=4, max=1177, avg=299.77, stdev=493.24 lat (msec): min=4, max=1177, avg=299.77, stdev=493.24 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 83], 40.00th=[ 83], 50.00th=[ 84], 60.00th=[ 84], | 70.00th=[ 153], 80.00th=[ 153], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1514KiB/s (1550kB/s)(7168KiB/4736msec) clat (msec): min=51, max=2133, avg=441.09, stdev=753.91 lat (msec): min=51, max=2133, avg=441.17, stdev=753.90 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 54], | 30.00th=[ 78], 40.00th=[ 78], 50.00th=[ 186], 60.00th=[ 259], | 70.00th=[ 259], 80.00th=[ 330], 90.00th=[ 2140], 95.00th=[ 2140], | 99.00th=[ 2140], 99.50th=[ 2140], 99.90th=[ 2140], 99.95th=[ 2140], | 99.99th=[ 2140] bw ( KiB/s): min= 2048, max= 4096, per=59.13%, avg=3579.50, stdev=1021.01, samples=4 iops : min= 2, max= 4, avg= 3.00, stdev= 0.82, samples=4 lat (msec) : 10=8.33%, 100=41.67%, 250=16.67%, 500=16.67% cpu : usr=0.00%, sys=8.79%, ctx=471, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4324KiB/s (4428kB/s), 742KiB/s-2107KiB/s (760kB/s-2158kB/s), io=20.0MiB (20.0MB), run=2761-4736msec WRITE: bw=6054KiB/s (6199kB/s), 1505KiB/s-3709KiB/s (1541kB/s-3798kB/s), io=28.0MiB (29.4MB), run=2761-4736msec rand-rw: (groupid=0, jobs=1): err= 0: pid=990: Thu May 1 22:27:36 2025 read: IOPS=2, BW=2161KiB/s (2213kB/s)(7168KiB/3317msec) clat (msec): min=63, max=1734, avg=320.76, stdev=623.88 lat (msec): min=63, max=1734, avg=320.76, stdev=623.88 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 75], | 30.00th=[ 80], 40.00th=[ 80], 50.00th=[ 83], 60.00th=[ 85], | 70.00th=[ 85], 80.00th=[ 127], 90.00th=[ 1737], 95.00th=[ 1737], | 99.00th=[ 1737], 99.50th=[ 1737], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 4031, max= 6144, per=100.00%, avg=5087.50, stdev=1494.12, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 write: IOPS=1, BW=1544KiB/s (1581kB/s)(5120KiB/3317msec) clat (msec): min=159, max=269, avg=202.90, stdev=40.08 lat (msec): min=160, max=269, avg=203.02, stdev=40.07 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 194], 40.00th=[ 194], 50.00th=[ 194], 60.00th=[ 194], | 70.00th=[ 197], 80.00th=[ 197], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 2048, max= 6144, per=72.68%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 100=41.67%, 250=41.67%, 500=8.33% cpu : usr=0.06%, sys=1.60%, ctx=28, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=991: Thu May 1 22:27:36 2025 read: IOPS=0, BW=403KiB/s (412kB/s)(2048KiB/5087msec) clat (msec): min=69, max=1848, avg=959.02, stdev=1258.36 lat (msec): min=69, max=1848, avg=959.02, stdev=1258.36 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 70], 10.00th=[ 70], 20.00th=[ 70], | 30.00th=[ 70], 40.00th=[ 70], 50.00th=[ 70], 60.00th=[ 1854], | 70.00th=[ 1854], 80.00th=[ 1854], 90.00th=[ 1854], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 2031, max= 2048, per=50.67%, avg=2039.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=1, BW=2013KiB/s (2061kB/s)(10.0MiB/5087msec) clat (msec): min=85, max=1356, avg=309.73, stdev=387.42 lat (msec): min=85, max=1356, avg=309.81, stdev=387.43 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 86], 10.00th=[ 86], 20.00th=[ 95], | 30.00th=[ 105], 40.00th=[ 112], 50.00th=[ 132], 60.00th=[ 176], | 70.00th=[ 355], 80.00th=[ 355], 90.00th=[ 451], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 2031, max= 6144, per=46.68%, avg=2630.71, stdev=1549.23, samples=7 iops : min= 1, max= 6, avg= 2.43, stdev= 1.62, samples=7 lat (msec) : 100=25.00%, 250=41.67%, 500=16.67% cpu : usr=0.10%, sys=1.95%, ctx=46, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=992: Thu May 1 22:27:36 2025 read: IOPS=1, BW=1235KiB/s (1265kB/s)(6144KiB/4974msec) clat (msec): min=30, max=1162, avg=279.42, stdev=438.01 lat (msec): min=30, max=1162, avg=279.42, stdev=438.01 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 62], | 30.00th=[ 62], 40.00th=[ 86], 50.00th=[ 86], 60.00th=[ 111], | 70.00th=[ 228], 80.00th=[ 228], 90.00th=[ 1167], 95.00th=[ 1167], | 99.00th=[ 1167], 99.50th=[ 1167], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 2048, max= 6107, per=100.00%, avg=4077.50, stdev=2870.15, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=1, BW=1235KiB/s (1265kB/s)(6144KiB/4974msec) clat (msec): min=117, max=2157, avg=533.98, stdev=803.31 lat (msec): min=117, max=2158, avg=534.20, stdev=803.65 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 118], 10.00th=[ 118], 20.00th=[ 127], | 30.00th=[ 127], 40.00th=[ 150], 50.00th=[ 150], 60.00th=[ 234], | 70.00th=[ 418], 80.00th=[ 418], 90.00th=[ 2165], 95.00th=[ 2165], | 99.00th=[ 2165], 99.50th=[ 2165], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 2035, max= 4096, per=45.36%, avg=2556.75, stdev=1026.18, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 lat (msec) : 50=8.33%, 100=16.67%, 250=50.00%, 500=8.33% cpu : usr=0.00%, sys=1.71%, ctx=48, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=993: Thu May 1 22:27:36 2025 read: IOPS=0, BW=1012KiB/s (1036kB/s)(5120KiB/5061msec) clat (msec): min=37, max=243, avg=93.52, stdev=87.46 lat (msec): min=37, max=243, avg=93.53, stdev=87.46 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 38], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 44], 60.00th=[ 44], | 70.00th=[ 100], 80.00th=[ 100], 90.00th=[ 245], 95.00th=[ 245], | 99.00th=[ 245], 99.50th=[ 245], 99.90th=[ 245], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1416KiB/s (1450kB/s)(7168KiB/5061msec) clat (msec): min=87, max=2119, avg=629.14, stdev=744.80 lat (msec): min=87, max=2119, avg=629.28, stdev=744.76 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 88], 10.00th=[ 88], 20.00th=[ 211], | 30.00th=[ 251], 40.00th=[ 251], 50.00th=[ 279], 60.00th=[ 309], | 70.00th=[ 309], 80.00th=[ 1150], 90.00th=[ 2123], 95.00th=[ 2123], | 99.00th=[ 2123], 99.50th=[ 2123], 99.90th=[ 2123], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 2039, max= 4096, per=54.25%, avg=3057.50, stdev=1171.04, samples=4 iops : min= 1, max= 4, avg= 2.50, stdev= 1.29, samples=4 lat (msec) : 50=25.00%, 100=16.67%, 250=16.67%, 500=25.00% cpu : usr=0.00%, sys=1.60%, ctx=44, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4026KiB/s (4123kB/s), 403KiB/s-2161KiB/s (412kB/s-2213kB/s), io=20.0MiB (20.0MB), run=3317-5087msec WRITE: bw=5636KiB/s (5772kB/s), 1235KiB/s-2013KiB/s (1265kB/s-2061kB/s), io=28.0MiB (29.4MB), run=3317-5087msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1006: Thu May 1 22:27:41 2025 read: IOPS=1, BW=7411KiB/s (7589kB/s)(12.0MiB/1658msec) clat (msec): min=391, max=582, avg=492.45, stdev=95.73 lat (msec): min=391, max=582, avg=492.45, stdev=95.73 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 393], 10.00th=[ 393], 20.00th=[ 393], | 30.00th=[ 393], 40.00th=[ 506], 50.00th=[ 506], 60.00th=[ 506], | 70.00th=[ 584], 80.00th=[ 584], 90.00th=[ 584], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.36%, sys=13.04%, ctx=367, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1007: Thu May 1 22:27:41 2025 read: IOPS=0, BW=1723KiB/s (1765kB/s)(4096KiB/2377msec) clat (nsec): min=668338k, max=668338k, avg=668337694.00, stdev= 0.00 lat (nsec): min=668340k, max=668340k, avg=668340089.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 667], 5.00th=[ 667], 10.00th=[ 667], 20.00th=[ 667], | 30.00th=[ 667], 40.00th=[ 667], 50.00th=[ 667], 60.00th=[ 667], | 70.00th=[ 667], 80.00th=[ 667], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3446KiB/s (3529kB/s)(8192KiB/2377msec) clat (msec): min=610, max=1030, avg=820.23, stdev=297.08 lat (msec): min=610, max=1030, avg=820.56, stdev=297.05 clat percentiles (msec): | 1.00th=[ 609], 5.00th=[ 609], 10.00th=[ 609], 20.00th=[ 609], | 30.00th=[ 609], 40.00th=[ 609], 50.00th=[ 609], 60.00th=[ 1028], | 70.00th=[ 1028], 80.00th=[ 1028], 90.00th=[ 1028], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 6617, max= 6617, per=53.31%, avg=6617.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 750=66.67% cpu : usr=0.00%, sys=12.45%, ctx=117, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1008: Thu May 1 22:27:41 2025 write: IOPS=1, BW=4655KiB/s (4766kB/s)(12.0MiB/2640msec) clat (msec): min=859, max=871, avg=864.55, stdev= 5.99 lat (msec): min=860, max=871, avg=864.95, stdev= 5.87 clat percentiles (msec): | 1.00th=[ 860], 5.00th=[ 860], 10.00th=[ 860], 20.00th=[ 860], | 30.00th=[ 860], 40.00th=[ 860], 50.00th=[ 860], 60.00th=[ 860], | 70.00th=[ 869], 80.00th=[ 869], 90.00th=[ 869], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 6532, max= 8110, per=58.98%, avg=7321.00, stdev=1115.81, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 1000=100.00% cpu : usr=0.00%, sys=12.73%, ctx=149, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1009: Thu May 1 22:27:41 2025 write: IOPS=1, BW=7026KiB/s (7194kB/s)(12.0MiB/1749msec) clat (msec): min=352, max=715, avg=562.02, stdev=187.49 lat (msec): min=355, max=715, avg=563.47, stdev=186.67 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 355], 10.00th=[ 355], 20.00th=[ 355], | 30.00th=[ 355], 40.00th=[ 617], 50.00th=[ 617], 60.00th=[ 617], | 70.00th=[ 718], 80.00th=[ 718], 90.00th=[ 718], 95.00th=[ 718], | 99.00th=[ 718], 99.50th=[ 718], 99.90th=[ 718], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 8159, max= 8175, per=65.80%, avg=8167.00, stdev=11.31, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.11%, sys=22.88%, ctx=100, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6893KiB/s (7058kB/s), 1723KiB/s-7411KiB/s (1765kB/s-7589kB/s), io=16.0MiB (16.8MB), run=1658-2377msec WRITE: bw=12.1MiB/s (12.7MB/s), 3446KiB/s-7026KiB/s (3529kB/s-7194kB/s), io=32.0MiB (33.6MB), run=1749-2640msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1012: Thu May 1 22:27:42 2025 read: IOPS=1, BW=4658KiB/s (4770kB/s)(12.0MiB/2638msec) clat (msec): min=869, max=873, avg=871.52, stdev= 1.99 lat (msec): min=869, max=873, avg=871.52, stdev= 1.99 clat percentiles (msec): | 1.00th=[ 869], 5.00th=[ 869], 10.00th=[ 869], 20.00th=[ 869], | 30.00th=[ 869], 40.00th=[ 869], 50.00th=[ 869], 60.00th=[ 869], | 70.00th=[ 877], 80.00th=[ 877], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 1000=100.00% cpu : usr=0.04%, sys=0.95%, ctx=18, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1013: Thu May 1 22:27:42 2025 read: IOPS=0, BW=1379KiB/s (1412kB/s)(4096KiB/2971msec) clat (nsec): min=852909k, max=852909k, avg=852908671.00, stdev= 0.00 lat (nsec): min=852912k, max=852912k, avg=852911628.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 852], 5.00th=[ 852], 10.00th=[ 852], 20.00th=[ 852], | 30.00th=[ 852], 40.00th=[ 852], 50.00th=[ 852], 60.00th=[ 852], | 70.00th=[ 852], 80.00th=[ 852], 90.00th=[ 852], 95.00th=[ 852], | 99.00th=[ 852], 99.50th=[ 852], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 8143, max= 8143, per=100.00%, avg=8143.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2757KiB/s (2823kB/s)(8192KiB/2971msec) clat (msec): min=248, max=1813, avg=1031.05, stdev=1106.75 lat (msec): min=248, max=1815, avg=1032.13, stdev=1107.65 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 249], 10.00th=[ 249], 20.00th=[ 249], | 30.00th=[ 249], 40.00th=[ 249], 50.00th=[ 249], 60.00th=[ 1821], | 70.00th=[ 1821], 80.00th=[ 1821], 90.00th=[ 1821], 95.00th=[ 1821], | 99.00th=[ 1821], 99.50th=[ 1821], 99.90th=[ 1821], 99.95th=[ 1821], | 99.99th=[ 1821] lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.17%, sys=1.89%, ctx=38, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1014: Thu May 1 22:27:42 2025 write: IOPS=0, BW=4080KiB/s (4178kB/s)(12.0MiB/3012msec) clat (msec): min=741, max=1101, avg=938.04, stdev=182.10 lat (msec): min=742, max=1101, avg=938.36, stdev=182.12 clat percentiles (msec): | 1.00th=[ 743], 5.00th=[ 743], 10.00th=[ 743], 20.00th=[ 743], | 30.00th=[ 743], 40.00th=[ 969], 50.00th=[ 969], 60.00th=[ 969], | 70.00th=[ 1099], 80.00th=[ 1099], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 8175, max= 8192, per=79.10%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=1.46%, ctx=21, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1015: Thu May 1 22:27:42 2025 write: IOPS=0, BW=3880KiB/s (3973kB/s)(12.0MiB/3167msec) clat (msec): min=529, max=2030, avg=1044.89, stdev=853.47 lat (msec): min=529, max=2030, avg=1045.26, stdev=853.60 clat percentiles (msec): | 1.00th=[ 531], 5.00th=[ 531], 10.00th=[ 531], 20.00th=[ 531], | 30.00th=[ 531], 40.00th=[ 575], 50.00th=[ 575], 60.00th=[ 575], | 70.00th=[ 2022], 80.00th=[ 2022], 90.00th=[ 2022], 95.00th=[ 2022], | 99.00th=[ 2022], 99.50th=[ 2022], 99.90th=[ 2022], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 8192, max= 8192, per=79.18%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 750=66.67% cpu : usr=0.00%, sys=2.91%, ctx=46, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5515KiB/s (5647kB/s), 1379KiB/s-4658KiB/s (1412kB/s-4770kB/s), io=16.0MiB (16.8MB), run=2638-2971msec WRITE: bw=10.1MiB/s (10.6MB/s), 2757KiB/s-4080KiB/s (2823kB/s-4178kB/s), io=32.0MiB (33.6MB), run=2971-3167msec