== sanity test 398b: DIO and buffer IO race ============== 19:52:27 (1746143547) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.45668 s, 20.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2379: Thu May 1 19:53:18 2025 read: IOPS=33, BW=135KiB/s (138kB/s)(5992KiB/44485msec) clat (usec): min=61, max=401312, avg=13522.85, stdev=15579.59 lat (usec): min=62, max=401315, avg=13524.84, stdev=15580.09 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 102], 10.00th=[ 118], 20.00th=[ 1237], | 30.00th=[ 7177], 40.00th=[ 9110], 50.00th=[ 10945], 60.00th=[ 13566], | 70.00th=[ 16581], 80.00th=[ 20841], 90.00th=[ 27919], 95.00th=[ 35390], | 99.00th=[ 54789], 99.50th=[ 61080], 99.90th=[127402], 99.95th=[400557], | 99.99th=[400557] bw ( KiB/s): min= 47, max= 256, per=25.71%, avg=134.99, stdev=45.18, samples=87 iops : min= 11, max= 64, avg=33.48, stdev=11.33, samples=87 write: IOPS=35, BW=142KiB/s (145kB/s)(6296KiB/44485msec) clat (usec): min=531, max=177440, avg=15156.84, stdev=17232.93 lat (usec): min=531, max=177441, avg=15160.50, stdev=17235.17 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 865], 10.00th=[ 1029], 20.00th=[ 1483], | 30.00th=[ 2966], 40.00th=[ 5932], 50.00th=[ 10945], 60.00th=[ 14615], | 70.00th=[ 19006], 80.00th=[ 25297], 90.00th=[ 34341], 95.00th=[ 47973], | 99.00th=[ 73925], 99.50th=[ 89654], 99.90th=[154141], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 47, max= 271, per=25.71%, avg=141.43, stdev=47.23, samples=87 iops : min= 11, max= 67, avg=35.09, stdev=11.84, samples=87 lat (usec) : 100=2.02%, 250=6.35%, 500=0.98%, 750=1.17%, 1000=3.84% lat (msec) : 2=7.94%, 4=5.79%, 10=18.36%, 20=28.74%, 50=21.87% lat (msec) : 100=2.70%, 250=0.20%, 500=0.03% cpu : usr=0.08%, sys=11.96%, ctx=5279, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2380: Thu May 1 19:53:18 2025 read: IOPS=32, BW=130KiB/s (133kB/s)(5940KiB/45673msec) clat (usec): min=49, max=344362, avg=12710.14, stdev=15116.52 lat (usec): min=50, max=344365, avg=12714.57, stdev=15115.01 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 102], 10.00th=[ 115], 20.00th=[ 330], | 30.00th=[ 6325], 40.00th=[ 8586], 50.00th=[ 10552], 60.00th=[ 12780], | 70.00th=[ 15401], 80.00th=[ 19792], 90.00th=[ 25822], 95.00th=[ 32637], | 99.00th=[ 56361], 99.50th=[ 74974], 99.90th=[158335], 99.95th=[346031], | 99.99th=[346031] bw ( KiB/s): min= 7, max= 350, per=25.30%, avg=132.84, stdev=58.31, samples=88 iops : min= 1, max= 87, avg=32.91, stdev=14.60, samples=88 write: IOPS=34, BW=139KiB/s (142kB/s)(6348KiB/45673msec) clat (usec): min=443, max=266924, avg=16653.36, stdev=22352.05 lat (usec): min=444, max=266926, avg=16657.14, stdev=22351.46 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 832], 10.00th=[ 963], 20.00th=[ 1467], | 30.00th=[ 2966], 40.00th=[ 7570], 50.00th=[ 11600], 60.00th=[ 15401], | 70.00th=[ 19792], 80.00th=[ 25822], 90.00th=[ 35914], 95.00th=[ 49546], | 99.00th=[106431], 99.50th=[162530], 99.90th=[258999], 99.95th=[267387], | 99.99th=[267387] bw ( KiB/s): min= 15, max= 304, per=25.15%, avg=138.30, stdev=58.75, samples=90 iops : min= 3, max= 76, avg=34.27, stdev=14.72, samples=90 lat (usec) : 50=0.07%, 100=2.12%, 250=6.71%, 500=1.56%, 750=1.30% lat (usec) : 1000=5.01% lat (msec) : 2=7.10%, 4=4.69%, 10=18.55%, 20=28.22%, 50=21.55% lat (msec) : 100=2.44%, 250=0.59%, 500=0.10% cpu : usr=0.11%, sys=11.46%, ctx=5259, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2381: Thu May 1 19:53:18 2025 read: IOPS=33, BW=134KiB/s (137kB/s)(6028KiB/45052msec) clat (usec): min=65, max=189704, avg=13452.80, stdev=13373.79 lat (usec): min=65, max=189704, avg=13454.06, stdev=13373.87 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 105], 10.00th=[ 124], 20.00th=[ 1020], | 30.00th=[ 6915], 40.00th=[ 8848], 50.00th=[ 10945], 60.00th=[ 13566], | 70.00th=[ 16581], 80.00th=[ 21103], 90.00th=[ 28443], 95.00th=[ 34866], | 99.00th=[ 52167], 99.50th=[ 63701], 99.90th=[170918], 99.95th=[189793], | 99.99th=[189793] bw ( KiB/s): min= 24, max= 264, per=25.67%, avg=134.76, stdev=46.01, samples=87 iops : min= 6, max= 66, avg=33.36, stdev=11.57, samples=87 write: IOPS=34, BW=139KiB/s (142kB/s)(6260KiB/45052msec) clat (usec): min=516, max=573008, avg=15663.37, stdev=23182.04 lat (usec): min=517, max=573010, avg=15666.25, stdev=23181.09 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 824], 10.00th=[ 971], 20.00th=[ 1369], | 30.00th=[ 2606], 40.00th=[ 5735], 50.00th=[ 10683], 60.00th=[ 15139], | 70.00th=[ 19792], 80.00th=[ 25560], 90.00th=[ 35390], 95.00th=[ 44827], | 99.00th=[ 82314], 99.50th=[130548], 99.90th=[202376], 99.95th=[574620], | 99.99th=[574620] bw ( KiB/s): min= 16, max= 272, per=25.14%, avg=138.27, stdev=51.19, samples=88 iops : min= 4, max= 68, avg=34.23, stdev=12.87, samples=88 lat (usec) : 100=1.56%, 250=6.28%, 500=1.46%, 750=1.20%, 1000=4.75% lat (msec) : 2=7.94%, 4=5.44%, 10=18.03%, 20=27.51%, 50=23.40% lat (msec) : 100=2.02%, 250=0.36%, 750=0.03% cpu : usr=0.12%, sys=11.63%, ctx=5221, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2382: Thu May 1 19:53:18 2025 read: IOPS=33, BW=135KiB/s (138kB/s)(6052KiB/44747msec) clat (usec): min=60, max=448027, avg=14068.77, stdev=16615.02 lat (usec): min=61, max=448028, avg=14074.71, stdev=16613.84 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 110], 10.00th=[ 130], 20.00th=[ 5211], | 30.00th=[ 7439], 40.00th=[ 9503], 50.00th=[ 11600], 60.00th=[ 13960], | 70.00th=[ 16909], 80.00th=[ 20841], 90.00th=[ 28443], 95.00th=[ 35390], | 99.00th=[ 54789], 99.50th=[ 71828], 99.90th=[147850], 99.95th=[446694], | 99.99th=[446694] bw ( KiB/s): min= 56, max= 304, per=25.76%, avg=135.23, stdev=43.16, samples=88 iops : min= 14, max= 76, avg=33.49, stdev=10.83, samples=88 write: IOPS=34, BW=139KiB/s (143kB/s)(6236KiB/44747msec) clat (usec): min=461, max=227346, avg=14872.90, stdev=17398.71 lat (usec): min=463, max=227348, avg=14876.08, stdev=17397.42 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 840], 10.00th=[ 1020], 20.00th=[ 1500], | 30.00th=[ 3097], 40.00th=[ 6259], 50.00th=[ 10945], 60.00th=[ 14484], | 70.00th=[ 19006], 80.00th=[ 25035], 90.00th=[ 33424], 95.00th=[ 43254], | 99.00th=[ 70779], 99.50th=[114820], 99.90th=[175113], 99.95th=[227541], | 99.99th=[227541] bw ( KiB/s): min= 32, max= 247, per=25.13%, avg=138.21, stdev=48.02, samples=89 iops : min= 8, max= 61, avg=34.24, stdev=12.05, samples=89 lat (usec) : 100=1.37%, 250=5.34%, 500=1.33%, 750=1.04%, 1000=4.07% lat (msec) : 2=7.85%, 4=4.95%, 10=18.85%, 20=29.88%, 50=23.37% lat (msec) : 100=1.56%, 250=0.36%, 500=0.03% cpu : usr=0.17%, sys=11.99%, ctx=5264, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=526KiB/s (538kB/s), 130KiB/s-135KiB/s (133kB/s-138kB/s), io=23.4MiB (24.6MB), run=44485-45673msec WRITE: bw=550KiB/s (564kB/s), 139KiB/s-142KiB/s (142kB/s-145kB/s), io=24.6MiB (25.7MB), run=44485-45673msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2384: Thu May 1 19:55:17 2025 read: IOPS=9, BW=36.4KiB/s (37.2kB/s)(5992KiB/164724msec) clat (msec): min=3, max=486, avg=14.48, stdev=16.31 lat (msec): min=3, max=486, avg=14.49, stdev=16.31 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 13], | 70.00th=[ 16], 80.00th=[ 19], 90.00th=[ 27], 95.00th=[ 35], | 99.00th=[ 59], 99.50th=[ 70], 99.90th=[ 121], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 7, max= 120, per=26.23%, avg=38.04, stdev=23.37, samples=312 iops : min= 1, max= 30, avg= 9.29, stdev= 5.87, samples=312 write: IOPS=9, BW=38.2KiB/s (39.1kB/s)(6296KiB/164724msec) clat (msec): min=23, max=319, avg=90.71, stdev=37.27 lat (msec): min=23, max=319, avg=90.72, stdev=37.27 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 44], 10.00th=[ 51], 20.00th=[ 61], | 30.00th=[ 68], 40.00th=[ 75], 50.00th=[ 85], 60.00th=[ 94], | 70.00th=[ 104], 80.00th=[ 118], 90.00th=[ 140], 95.00th=[ 161], | 99.00th=[ 213], 99.50th=[ 232], 99.90th=[ 279], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 7, max= 96, per=24.97%, avg=37.95, stdev=13.07, samples=328 iops : min= 1, max= 24, avg= 9.26, stdev= 3.32, samples=328 lat (msec) : 4=0.29%, 10=21.35%, 20=18.46%, 50=12.92%, 100=29.92% lat (msec) : 250=16.89%, 500=0.16% cpu : usr=0.04%, sys=2.57%, ctx=3613, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2385: Thu May 1 19:55:17 2025 read: IOPS=9, BW=36.1KiB/s (36.9kB/s)(5940KiB/164665msec) clat (msec): min=3, max=439, avg=14.38, stdev=15.58 lat (msec): min=3, max=439, avg=14.39, stdev=15.58 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 13], | 70.00th=[ 15], 80.00th=[ 19], 90.00th=[ 25], 95.00th=[ 35], | 99.00th=[ 63], 99.50th=[ 75], 99.90th=[ 142], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 7, max= 120, per=26.03%, avg=37.74, stdev=23.13, samples=311 iops : min= 1, max= 30, avg= 9.14, stdev= 5.83, samples=311 write: IOPS=9, BW=38.6KiB/s (39.5kB/s)(6348KiB/164665msec) clat (msec): min=20, max=272, avg=90.15, stdev=36.63 lat (msec): min=20, max=272, avg=90.15, stdev=36.63 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 46], 10.00th=[ 52], 20.00th=[ 60], | 30.00th=[ 68], 40.00th=[ 75], 50.00th=[ 84], 60.00th=[ 92], | 70.00th=[ 103], 80.00th=[ 114], 90.00th=[ 142], 95.00th=[ 161], | 99.00th=[ 209], 99.50th=[ 234], 99.90th=[ 259], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 7, max= 80, per=25.13%, avg=38.20, stdev=12.86, samples=328 iops : min= 1, max= 20, avg= 9.25, stdev= 3.30, samples=328 lat (msec) : 4=0.29%, 10=20.54%, 20=19.53%, 50=11.43%, 100=31.67% lat (msec) : 250=16.37%, 500=0.16% cpu : usr=0.07%, sys=2.56%, ctx=3696, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2386: Thu May 1 19:55:17 2025 read: IOPS=9, BW=36.7KiB/s (37.6kB/s)(6028KiB/164087msec) clat (msec): min=3, max=232, avg=14.76, stdev=13.60 lat (msec): min=3, max=232, avg=14.76, stdev=13.60 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 14], | 70.00th=[ 16], 80.00th=[ 20], 90.00th=[ 27], 95.00th=[ 36], | 99.00th=[ 59], 99.50th=[ 93], 99.90th=[ 180], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 7, max= 136, per=26.40%, avg=38.28, stdev=23.95, samples=311 iops : min= 1, max= 34, avg= 9.34, stdev= 6.02, samples=311 write: IOPS=9, BW=38.2KiB/s (39.1kB/s)(6260KiB/164087msec) clat (msec): min=21, max=560, avg=90.49, stdev=38.98 lat (msec): min=21, max=560, avg=90.49, stdev=38.98 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 44], 10.00th=[ 51], 20.00th=[ 61], | 30.00th=[ 69], 40.00th=[ 75], 50.00th=[ 85], 60.00th=[ 92], | 70.00th=[ 103], 80.00th=[ 115], 90.00th=[ 138], 95.00th=[ 161], | 99.00th=[ 215], 99.50th=[ 239], 99.90th=[ 321], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 15, max= 80, per=24.93%, avg=37.89, stdev=12.87, samples=326 iops : min= 3, max= 20, avg= 9.24, stdev= 3.26, samples=326 lat (msec) : 4=0.29%, 10=21.48%, 20=17.77%, 50=13.74%, 100=30.37% lat (msec) : 250=16.24%, 500=0.07%, 750=0.03% cpu : usr=0.05%, sys=2.59%, ctx=3705, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2387: Thu May 1 19:55:17 2025 read: IOPS=9, BW=37.1KiB/s (38.0kB/s)(6052KiB/162956msec) clat (msec): min=3, max=249, avg=15.10, stdev=14.51 lat (msec): min=3, max=249, avg=15.10, stdev=14.51 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 13], | 70.00th=[ 16], 80.00th=[ 20], 90.00th=[ 27], 95.00th=[ 37], | 99.00th=[ 77], 99.50th=[ 90], 99.90th=[ 180], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 7, max= 128, per=26.81%, avg=38.88, stdev=23.32, samples=308 iops : min= 1, max= 32, avg= 9.46, stdev= 5.86, samples=308 write: IOPS=9, BW=38.3KiB/s (39.2kB/s)(6236KiB/162956msec) clat (msec): min=21, max=434, avg=89.68, stdev=38.24 lat (msec): min=21, max=434, avg=89.68, stdev=38.24 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 43], 10.00th=[ 51], 20.00th=[ 59], | 30.00th=[ 68], 40.00th=[ 77], 50.00th=[ 83], 60.00th=[ 91], | 70.00th=[ 102], 80.00th=[ 116], 90.00th=[ 138], 95.00th=[ 159], | 99.00th=[ 218], 99.50th=[ 241], 99.90th=[ 284], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 7, max= 88, per=25.01%, avg=38.02, stdev=13.51, samples=324 iops : min= 1, max= 22, avg= 9.24, stdev= 3.42, samples=324 lat (msec) : 4=0.42%, 10=20.77%, 20=18.95%, 50=12.92%, 100=30.92% lat (msec) : 250=15.85%, 500=0.16% cpu : usr=0.04%, sys=2.60%, ctx=3674, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=146KiB/s (149kB/s), 36.1KiB/s-37.1KiB/s (36.9kB/s-38.0kB/s), io=23.4MiB (24.6MB), run=162956-164724msec WRITE: bw=153KiB/s (156kB/s), 38.2KiB/s-38.6KiB/s (39.1kB/s-39.5kB/s), io=24.6MiB (25.7MB), run=162956-164724msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2535: Thu May 1 19:55:30 2025 read: IOPS=35, BW=563KiB/s (577kB/s)(5920KiB/10512msec) clat (usec): min=64, max=95468, avg=13650.19, stdev=14734.75 lat (usec): min=65, max=95470, avg=13651.95, stdev=14734.27 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 100], 10.00th=[ 112], 20.00th=[ 129], | 30.00th=[ 363], 40.00th=[ 8848], 50.00th=[12125], 60.00th=[14877], | 70.00th=[18482], 80.00th=[22152], 90.00th=[29492], 95.00th=[36439], | 99.00th=[85459], 99.50th=[91751], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 286, max= 795, per=24.96%, avg=558.86, stdev=119.85, samples=21 iops : min= 17, max= 49, avg=34.48, stdev= 7.56, samples=21 write: IOPS=37, BW=606KiB/s (620kB/s)(6368KiB/10512msec) clat (usec): min=938, max=202800, avg=13559.96, stdev=17154.38 lat (usec): min=940, max=202801, avg=13566.31, stdev=17157.83 clat percentiles (usec): | 1.00th=[ 1004], 5.00th=[ 1172], 10.00th=[ 1385], 20.00th=[ 1729], | 30.00th=[ 2442], 40.00th=[ 3916], 50.00th=[ 7308], 60.00th=[ 11469], | 70.00th=[ 16319], 80.00th=[ 23725], 90.00th=[ 34866], 95.00th=[ 42730], | 99.00th=[ 70779], 99.50th=[ 74974], 99.90th=[202376], 99.95th=[202376], | 99.99th=[202376] bw ( KiB/s): min= 159, max= 1024, per=25.60%, avg=603.10, stdev=212.93, samples=21 iops : min= 9, max= 64, avg=37.24, stdev=13.45, samples=21 lat (usec) : 100=2.60%, 250=10.94%, 500=1.56%, 750=0.39%, 1000=0.39% lat (msec) : 2=12.63%, 4=8.59%, 10=11.98%, 20=25.26%, 50=23.31% lat (msec) : 100=2.21%, 250=0.13% cpu : usr=0.25%, sys=12.48%, ctx=1146, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2536: Thu May 1 19:55:30 2025 read: IOPS=33, BW=537KiB/s (550kB/s)(5744KiB/10693msec) clat (usec): min=80, max=243248, avg=12104.61, stdev=17304.53 lat (usec): min=80, max=243250, avg=12105.79, stdev=17304.51 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 100], 10.00th=[ 106], 20.00th=[ 126], | 30.00th=[ 153], 40.00th=[ 1045], 50.00th=[ 10159], 60.00th=[ 14091], | 70.00th=[ 16909], 80.00th=[ 21890], 90.00th=[ 28181], 95.00th=[ 33424], | 99.00th=[ 60031], 99.50th=[ 65799], 99.90th=[242222], 99.95th=[242222], | 99.99th=[242222] bw ( KiB/s): min= 31, max= 1056, per=23.76%, avg=532.05, stdev=297.44, samples=21 iops : min= 1, max= 66, avg=32.90, stdev=18.65, samples=21 write: IOPS=38, BW=612KiB/s (627kB/s)(6544KiB/10693msec) clat (usec): min=924, max=224589, avg=15308.67, stdev=25457.32 lat (usec): min=926, max=224592, avg=15310.48, stdev=25457.32 clat percentiles (usec): | 1.00th=[ 1090], 5.00th=[ 1172], 10.00th=[ 1287], 20.00th=[ 1614], | 30.00th=[ 2409], 40.00th=[ 3884], 50.00th=[ 6128], 60.00th=[ 10421], | 70.00th=[ 15926], 80.00th=[ 22414], 90.00th=[ 36963], 95.00th=[ 47449], | 99.00th=[135267], 99.50th=[168821], 99.90th=[225444], 99.95th=[225444], | 99.99th=[225444] bw ( KiB/s): min= 95, max= 1120, per=25.71%, avg=605.62, stdev=276.87, samples=21 iops : min= 5, max= 70, avg=37.52, stdev=17.38, samples=21 lat (usec) : 100=2.08%, 250=12.89%, 500=2.47%, 750=0.65%, 1000=0.39% lat (msec) : 2=14.19%, 4=8.85%, 10=12.89%, 20=22.79%, 50=19.53% lat (msec) : 100=1.95%, 250=1.30% cpu : usr=0.11%, sys=11.58%, ctx=1080, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2537: Thu May 1 19:55:30 2025 read: IOPS=37, BW=604KiB/s (618kB/s)(6256KiB/10358msec) clat (usec): min=55, max=118282, avg=12366.98, stdev=13986.93 lat (usec): min=55, max=118282, avg=12368.28, stdev=13987.01 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 101], 10.00th=[ 110], 20.00th=[ 122], | 30.00th=[ 147], 40.00th=[ 1221], 50.00th=[ 10814], 60.00th=[ 14353], | 70.00th=[ 17957], 80.00th=[ 22152], 90.00th=[ 28967], 95.00th=[ 35390], | 99.00th=[ 53740], 99.50th=[ 71828], 99.90th=[117965], 99.95th=[117965], | 99.99th=[117965] bw ( KiB/s): min= 384, max= 1149, per=27.54%, avg=616.60, stdev=219.70, samples=20 iops : min= 24, max= 71, avg=38.25, stdev=13.76, samples=20 write: IOPS=36, BW=582KiB/s (596kB/s)(6032KiB/10358msec) clat (usec): min=922, max=91237, avg=14347.42, stdev=15341.74 lat (usec): min=922, max=91237, avg=14349.94, stdev=15343.79 clat percentiles (usec): | 1.00th=[ 979], 5.00th=[ 1188], 10.00th=[ 1467], 20.00th=[ 1975], | 30.00th=[ 2999], 40.00th=[ 5014], 50.00th=[ 8979], 60.00th=[14222], | 70.00th=[18482], 80.00th=[23987], 90.00th=[33817], 95.00th=[43779], | 99.00th=[67634], 99.50th=[86508], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 223, max= 1184, per=24.89%, avg=586.35, stdev=207.55, samples=20 iops : min= 13, max= 74, avg=36.35, stdev=13.15, samples=20 lat (usec) : 100=2.08%, 250=14.45%, 500=2.86%, 750=0.65%, 1000=0.65% lat (msec) : 2=10.16%, 4=7.42%, 10=11.98%, 20=24.22%, 50=22.79% lat (msec) : 100=2.60%, 250=0.13% cpu : usr=0.18%, sys=12.58%, ctx=1109, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2538: Thu May 1 19:55:30 2025 read: IOPS=36, BW=578KiB/s (592kB/s)(6032KiB/10435msec) clat (usec): min=68, max=57974, avg=12411.04, stdev=12943.30 lat (usec): min=68, max=57975, avg=12412.73, stdev=12942.77 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 102], 10.00th=[ 113], 20.00th=[ 125], | 30.00th=[ 167], 40.00th=[ 4621], 50.00th=[10683], 60.00th=[13698], | 70.00th=[17957], 80.00th=[23200], 90.00th=[30540], 95.00th=[36963], | 99.00th=[49546], 99.50th=[56361], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 320, max= 928, per=26.96%, avg=603.74, stdev=164.63, samples=19 iops : min= 20, max= 58, avg=37.47, stdev=10.27, samples=19 write: IOPS=37, BW=600KiB/s (614kB/s)(6256KiB/10435msec) clat (usec): min=925, max=134812, avg=14182.16, stdev=17115.04 lat (usec): min=927, max=134814, avg=14184.10, stdev=17115.33 clat percentiles (usec): | 1.00th=[ 1057], 5.00th=[ 1287], 10.00th=[ 1450], 20.00th=[ 1876], | 30.00th=[ 2606], 40.00th=[ 4686], 50.00th=[ 7504], 60.00th=[ 11863], | 70.00th=[ 17171], 80.00th=[ 24249], 90.00th=[ 37487], 95.00th=[ 44827], | 99.00th=[ 76022], 99.50th=[126354], 99.90th=[135267], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 96, max= 1120, per=25.44%, avg=599.25, stdev=236.55, samples=20 iops : min= 6, max= 70, avg=37.20, stdev=14.80, samples=20 lat (usec) : 100=2.08%, 250=13.54%, 500=1.82%, 750=0.39%, 1000=0.65% lat (msec) : 2=11.72%, 4=8.46%, 10=13.54%, 20=21.61%, 50=24.35% lat (msec) : 100=1.43%, 250=0.39% cpu : usr=0.08%, sys=12.27%, ctx=1151, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2240KiB/s (2294kB/s), 537KiB/s-604KiB/s (550kB/s-618kB/s), io=23.4MiB (24.5MB), run=10358-10693msec WRITE: bw=2357KiB/s (2413kB/s), 582KiB/s-612KiB/s (596kB/s-627kB/s), io=24.6MiB (25.8MB), run=10358-10693msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2531: Thu May 1 19:56:04 2025 read: IOPS=8, BW=132KiB/s (135kB/s)(5920KiB/44898msec) clat (msec): min=3, max=126, avg=13.93, stdev=11.85 lat (msec): min=3, max=126, avg=13.93, stdev=11.85 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 12], | 70.00th=[ 14], 80.00th=[ 16], 90.00th=[ 24], 95.00th=[ 36], | 99.00th=[ 61], 99.50th=[ 102], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 31, max= 576, per=26.35%, avg=139.12, stdev=84.91, samples=83 iops : min= 1, max= 36, avg= 8.20, stdev= 5.40, samples=83 write: IOPS=8, BW=142KiB/s (145kB/s)(6368KiB/44898msec) clat (msec): min=29, max=289, avg=99.37, stdev=38.99 lat (msec): min=29, max=289, avg=99.38, stdev=38.99 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 52], 10.00th=[ 57], 20.00th=[ 68], | 30.00th=[ 77], 40.00th=[ 84], 50.00th=[ 91], 60.00th=[ 101], | 70.00th=[ 112], 80.00th=[ 131], 90.00th=[ 150], 95.00th=[ 174], | 99.00th=[ 228], 99.50th=[ 279], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 62, max= 256, per=25.30%, avg=140.64, stdev=45.26, samples=89 iops : min= 3, max= 16, avg= 8.30, stdev= 2.89, samples=89 lat (msec) : 4=0.39%, 10=19.27%, 20=22.40%, 50=7.16%, 100=29.56% lat (msec) : 250=20.96%, 500=0.26% cpu : usr=0.05%, sys=2.51%, ctx=890, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2532: Thu May 1 19:56:04 2025 read: IOPS=7, BW=127KiB/s (130kB/s)(5744KiB/45084msec) clat (usec): min=4011, max=90152, avg=15871.54, stdev=14354.25 lat (usec): min=4013, max=90153, avg=15874.19, stdev=14353.53 clat percentiles (usec): | 1.00th=[ 4752], 5.00th=[ 5932], 10.00th=[ 6718], 20.00th=[ 7832], | 30.00th=[ 8979], 40.00th=[10028], 50.00th=[11076], 60.00th=[12256], | 70.00th=[14484], 80.00th=[18482], 90.00th=[32375], 95.00th=[50070], | 99.00th=[79168], 99.50th=[85459], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 31, max= 448, per=24.83%, avg=131.09, stdev=88.37, samples=87 iops : min= 1, max= 28, avg= 7.76, stdev= 5.55, samples=87 write: IOPS=9, BW=145KiB/s (149kB/s)(6544KiB/45084msec) clat (msec): min=35, max=271, avg=96.20, stdev=36.05 lat (msec): min=35, max=271, avg=96.20, stdev=36.05 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 51], 10.00th=[ 57], 20.00th=[ 65], | 30.00th=[ 75], 40.00th=[ 83], 50.00th=[ 90], 60.00th=[ 99], | 70.00th=[ 107], 80.00th=[ 120], 90.00th=[ 148], 95.00th=[ 169], | 99.00th=[ 203], 99.50th=[ 230], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 62, max= 224, per=25.91%, avg=144.04, stdev=42.35, samples=90 iops : min= 3, max= 14, avg= 8.56, stdev= 2.75, samples=90 lat (msec) : 10=18.75%, 20=19.53%, 50=8.46%, 100=33.07%, 250=20.05% lat (msec) : 500=0.13% cpu : usr=0.04%, sys=2.51%, ctx=883, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2533: Thu May 1 19:56:04 2025 read: IOPS=9, BW=146KiB/s (149kB/s)(6256KiB/42905msec) clat (msec): min=3, max=167, avg=15.52, stdev=14.70 lat (msec): min=3, max=167, avg=15.52, stdev=14.70 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 13], | 70.00th=[ 15], 80.00th=[ 18], 90.00th=[ 28], 95.00th=[ 41], | 99.00th=[ 82], 99.50th=[ 94], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 31, max= 384, per=28.74%, avg=151.73, stdev=87.56, samples=82 iops : min= 1, max= 24, avg= 9.12, stdev= 5.51, samples=82 write: IOPS=8, BW=141KiB/s (144kB/s)(6032KiB/42905msec) clat (msec): min=24, max=331, avg=97.14, stdev=38.43 lat (msec): min=24, max=331, avg=97.15, stdev=38.43 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 48], 10.00th=[ 56], 20.00th=[ 66], | 30.00th=[ 73], 40.00th=[ 82], 50.00th=[ 90], 60.00th=[ 100], | 70.00th=[ 111], 80.00th=[ 125], 90.00th=[ 148], 95.00th=[ 171], | 99.00th=[ 197], 99.50th=[ 230], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 63, max= 256, per=25.11%, avg=139.60, stdev=45.28, samples=85 iops : min= 3, max= 16, avg= 8.35, stdev= 2.88, samples=85 lat (msec) : 4=0.26%, 10=19.40%, 20=23.18%, 50=9.37%, 100=28.78% lat (msec) : 250=18.88%, 500=0.13% cpu : usr=0.04%, sys=2.63%, ctx=919, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2534: Thu May 1 19:56:04 2025 read: IOPS=8, BW=133KiB/s (136kB/s)(6032KiB/45289msec) clat (msec): min=4, max=168, avg=15.95, stdev=16.42 lat (msec): min=4, max=168, avg=15.95, stdev=16.42 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 13], | 70.00th=[ 15], 80.00th=[ 18], 90.00th=[ 29], 95.00th=[ 43], | 99.00th=[ 102], 99.50th=[ 112], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 31, max= 416, per=26.04%, avg=137.50, stdev=77.52, samples=86 iops : min= 1, max= 26, avg= 8.15, stdev= 4.93, samples=86 write: IOPS=8, BW=138KiB/s (141kB/s)(6256KiB/45289msec) clat (msec): min=34, max=233, avg=99.97, stdev=35.55 lat (msec): min=34, max=233, avg=99.97, stdev=35.55 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 53], 10.00th=[ 63], 20.00th=[ 72], | 30.00th=[ 79], 40.00th=[ 86], 50.00th=[ 92], 60.00th=[ 101], | 70.00th=[ 112], 80.00th=[ 128], 90.00th=[ 148], 95.00th=[ 171], | 99.00th=[ 201], 99.50th=[ 220], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 63, max= 255, per=24.72%, avg=137.44, stdev=41.71, samples=90 iops : min= 3, max= 15, avg= 8.13, stdev= 2.66, samples=90 lat (msec) : 10=18.10%, 20=23.57%, 50=7.55%, 100=29.69%, 250=21.09% cpu : usr=0.05%, sys=2.72%, ctx=930, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=529KiB/s (542kB/s), 127KiB/s-146KiB/s (130kB/s-149kB/s), io=23.4MiB (24.5MB), run=42905-45289msec WRITE: bw=556KiB/s (570kB/s), 138KiB/s-145KiB/s (141kB/s-149kB/s), io=24.6MiB (25.8MB), run=42905-45289msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2587: Thu May 1 19:56:11 2025 read: IOPS=1, BW=1931KiB/s (1977kB/s)(7168KiB/3712msec) clat (usec): min=758, max=1952.7k, avg=349213.65, stdev=709993.05 lat (usec): min=760, max=1952.7k, avg=349215.44, stdev=709993.44 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 758], 10.00th=[ 758], | 20.00th=[ 1156], 30.00th=[ 80217], 40.00th=[ 80217], | 50.00th=[ 107480], 60.00th=[ 123208], 70.00th=[ 123208], | 80.00th=[ 179307], 90.00th=[1954546], 95.00th=[1954546], | 99.00th=[1954546], 99.50th=[1954546], 99.90th=[1954546], | 99.95th=[1954546], 99.99th=[1954546] bw ( KiB/s): min= 4096, max= 6119, per=100.00%, avg=5107.50, stdev=1430.48, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=1, BW=1379KiB/s (1412kB/s)(5120KiB/3712msec) clat (msec): min=67, max=449, avg=239.37, stdev=143.04 lat (msec): min=67, max=449, avg=239.43, stdev=143.06 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 68], | 30.00th=[ 163], 40.00th=[ 163], 50.00th=[ 230], 60.00th=[ 230], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 2039, max= 2048, per=32.05%, avg=2042.00, stdev= 5.20, samples=3 iops : min= 1, max= 2, avg= 1.33, stdev= 0.58, samples=3 lat (usec) : 1000=8.33% lat (msec) : 2=8.33%, 100=16.67%, 250=41.67%, 500=16.67% cpu : usr=0.00%, sys=11.53%, ctx=651, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2588: Thu May 1 19:56:11 2025 read: IOPS=0, BW=455KiB/s (466kB/s)(2048KiB/4499msec) clat (msec): min=142, max=177, avg=160.25, stdev=24.80 lat (msec): min=142, max=177, avg=160.25, stdev=24.81 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 144], 50.00th=[ 144], 60.00th=[ 178], | 70.00th=[ 178], 80.00th=[ 178], 90.00th=[ 178], 95.00th=[ 178], | 99.00th=[ 178], 99.50th=[ 178], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 2048, max= 2048, per=44.99%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2276KiB/s (2331kB/s)(10.0MiB/4499msec) clat (msec): min=50, max=2036, avg=400.63, stdev=594.16 lat (msec): min=50, max=2037, avg=400.71, stdev=594.15 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 51], 10.00th=[ 51], 20.00th=[ 87], | 30.00th=[ 106], 40.00th=[ 159], 50.00th=[ 169], 60.00th=[ 171], | 70.00th=[ 460], 80.00th=[ 460], 90.00th=[ 493], 95.00th=[ 2039], | 99.00th=[ 2039], 99.50th=[ 2039], 99.90th=[ 2039], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 2027, max= 6144, per=48.09%, avg=3064.50, stdev=2053.02, samples=4 iops : min= 1, max= 6, avg= 2.50, stdev= 2.38, samples=4 lat (msec) : 100=16.67%, 250=50.00%, 500=25.00% cpu : usr=0.00%, sys=7.78%, ctx=117, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2589: Thu May 1 19:56:11 2025 read: IOPS=1, BW=1735KiB/s (1776kB/s)(6144KiB/3542msec) clat (msec): min=41, max=237, avg=124.80, stdev=64.56 lat (msec): min=41, max=237, avg=124.80, stdev=64.56 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 96], | 30.00th=[ 96], 40.00th=[ 112], 50.00th=[ 112], 60.00th=[ 127], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 239], 95.00th=[ 239], | 99.00th=[ 239], 99.50th=[ 239], 99.90th=[ 239], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 4087, max= 6144, per=100.00%, avg=5115.50, stdev=1454.52, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 write: IOPS=1, BW=1735KiB/s (1776kB/s)(6144KiB/3542msec) clat (msec): min=53, max=2102, avg=423.52, stdev=823.04 lat (msec): min=54, max=2102, avg=423.60, stdev=823.06 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 54], 10.00th=[ 54], 20.00th=[ 78], | 30.00th=[ 78], 40.00th=[ 79], 50.00th=[ 79], 60.00th=[ 111], | 70.00th=[ 118], 80.00th=[ 118], 90.00th=[ 2106], 95.00th=[ 2106], | 99.00th=[ 2106], 99.50th=[ 2106], 99.90th=[ 2106], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 4087, max= 4096, per=64.19%, avg=4090.00, stdev= 5.20, samples=3 iops : min= 3, max= 4, avg= 3.33, stdev= 0.58, samples=3 lat (msec) : 50=8.33%, 100=33.33%, 250=50.00% cpu : usr=0.03%, sys=8.11%, ctx=223, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2590: Thu May 1 19:56:11 2025 read: IOPS=1, BW=1315KiB/s (1346kB/s)(5120KiB/3895msec) clat (usec): min=758, max=152003, avg=50597.30, stdev=70398.97 lat (usec): min=760, max=152003, avg=50598.67, stdev=70398.47 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 758], 10.00th=[ 758], 20.00th=[ 758], | 30.00th=[ 783], 40.00th=[ 783], 50.00th=[ 1893], 60.00th=[ 1893], | 70.00th=[ 98042], 80.00th=[ 98042], 90.00th=[152044], 95.00th=[152044], | 99.00th=[152044], 99.50th=[152044], 99.90th=[152044], 99.95th=[152044], | 99.99th=[152044] bw ( KiB/s): min= 2048, max= 6119, per=89.71%, avg=4083.50, stdev=2878.63, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=1, BW=1840KiB/s (1884kB/s)(7168KiB/3895msec) clat (msec): min=100, max=2048, avg=486.92, stdev=697.00 lat (msec): min=100, max=2048, avg=486.98, stdev=697.01 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 101], 10.00th=[ 101], 20.00th=[ 109], | 30.00th=[ 153], 40.00th=[ 153], 50.00th=[ 300], 60.00th=[ 330], | 70.00th=[ 330], 80.00th=[ 368], 90.00th=[ 2056], 95.00th=[ 2056], | 99.00th=[ 2056], 99.50th=[ 2056], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 2039, max= 4096, per=40.10%, avg=2555.25, stdev=1027.17, samples=4 iops : min= 1, max= 4, avg= 1.75, stdev= 1.50, samples=4 lat (usec) : 1000=16.67% lat (msec) : 2=8.33%, 100=8.33%, 250=33.33%, 500=25.00% cpu : usr=0.00%, sys=6.24%, ctx=200, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4552KiB/s (4661kB/s), 455KiB/s-1931KiB/s (466kB/s-1977kB/s), io=20.0MiB (20.0MB), run=3542-4499msec WRITE: bw=6373KiB/s (6526kB/s), 1379KiB/s-2276KiB/s (1412kB/s-2331kB/s), io=28.0MiB (29.4MB), run=3542-4499msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2583: Thu May 1 19:56:11 2025 read: IOPS=1, BW=1982KiB/s (2029kB/s)(7168KiB/3617msec) clat (msec): min=36, max=1965, avg=376.43, stdev=702.61 lat (msec): min=36, max=1965, avg=376.44, stdev=702.61 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 89], | 30.00th=[ 105], 40.00th=[ 105], 50.00th=[ 110], 60.00th=[ 118], | 70.00th=[ 118], 80.00th=[ 213], 90.00th=[ 1972], 95.00th=[ 1972], | 99.00th=[ 1972], 99.50th=[ 1972], 99.90th=[ 1972], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 2035, max= 4096, per=69.12%, avg=3064.25, stdev=1186.17, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.50, samples=4 write: IOPS=1, BW=1416KiB/s (1450kB/s)(5120KiB/3617msec) clat (msec): min=126, max=216, avg=188.12, stdev=35.58 lat (msec): min=126, max=216, avg=188.20, stdev=35.59 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 127], 10.00th=[ 127], 20.00th=[ 127], | 30.00th=[ 194], 40.00th=[ 194], 50.00th=[ 201], 60.00th=[ 201], | 70.00th=[ 203], 80.00th=[ 203], 90.00th=[ 218], 95.00th=[ 218], | 99.00th=[ 218], 99.50th=[ 218], 99.90th=[ 218], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 2043, max= 4071, per=43.83%, avg=2720.67, stdev=1169.43, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 lat (msec) : 50=8.33%, 100=8.33%, 250=75.00% cpu : usr=0.00%, sys=2.52%, ctx=34, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2584: Thu May 1 19:56:11 2025 read: IOPS=0, BW=462KiB/s (473kB/s)(2048KiB/4435msec) clat (msec): min=73, max=121, avg=97.21, stdev=34.21 lat (msec): min=73, max=121, avg=97.21, stdev=34.21 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 73], 10.00th=[ 73], 20.00th=[ 73], | 30.00th=[ 73], 40.00th=[ 73], 50.00th=[ 73], 60.00th=[ 122], | 70.00th=[ 122], 80.00th=[ 122], 90.00th=[ 122], 95.00th=[ 122], | 99.00th=[ 122], 99.50th=[ 122], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 2039, max= 2043, per=46.04%, avg=2041.00, stdev= 2.83, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2309KiB/s (2364kB/s)(10.0MiB/4435msec) clat (msec): min=145, max=2071, avg=408.49, stdev=589.41 lat (msec): min=145, max=2071, avg=408.62, stdev=589.38 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 146], 10.00th=[ 146], 20.00th=[ 157], | 30.00th=[ 165], 40.00th=[ 171], 50.00th=[ 180], 60.00th=[ 232], | 70.00th=[ 355], 80.00th=[ 355], 90.00th=[ 359], 95.00th=[ 2072], | 99.00th=[ 2072], 99.50th=[ 2072], 99.90th=[ 2072], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 2043, max= 6131, per=57.63%, avg=3577.25, stdev=1955.68, samples=4 iops : min= 1, max= 5, avg= 2.75, stdev= 1.71, samples=4 lat (msec) : 100=8.33%, 250=58.33%, 500=25.00% cpu : usr=0.02%, sys=3.72%, ctx=68, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2585: Thu May 1 19:56:11 2025 read: IOPS=1, BW=1425KiB/s (1459kB/s)(6144KiB/4313msec) clat (msec): min=61, max=491, avg=221.89, stdev=192.71 lat (msec): min=61, max=491, avg=221.90, stdev=192.70 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 62], 10.00th=[ 62], 20.00th=[ 81], | 30.00th=[ 81], 40.00th=[ 100], 50.00th=[ 100], 60.00th=[ 159], | 70.00th=[ 443], 80.00th=[ 443], 90.00th=[ 493], 95.00th=[ 493], | 99.00th=[ 493], 99.50th=[ 493], 99.90th=[ 493], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 2048, max= 4096, per=61.60%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 write: IOPS=1, BW=1424KiB/s (1458kB/s)(6144KiB/4314msec) clat (msec): min=153, max=1989, avg=491.68, stdev=734.70 lat (msec): min=153, max=1989, avg=491.78, stdev=734.69 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 180], | 30.00th=[ 180], 40.00th=[ 186], 50.00th=[ 186], 60.00th=[ 192], | 70.00th=[ 253], 80.00th=[ 253], 90.00th=[ 1989], 95.00th=[ 1989], | 99.00th=[ 1989], 99.50th=[ 1989], 99.90th=[ 1989], 99.95th=[ 1989], | 99.99th=[ 1989] bw ( KiB/s): min= 2048, max= 4096, per=41.24%, avg=2560.00, stdev=1024.00, samples=4 iops : min= 2, max= 4, avg= 2.50, stdev= 1.00, samples=4 lat (msec) : 100=25.00%, 250=41.67%, 500=25.00% cpu : usr=0.02%, sys=2.50%, ctx=55, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2586: Thu May 1 19:56:11 2025 read: IOPS=1, BW=1108KiB/s (1135kB/s)(5120KiB/4619msec) clat (msec): min=28, max=157, avg=75.19, stdev=50.27 lat (msec): min=28, max=157, avg=75.19, stdev=50.27 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 29], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 57], 60.00th=[ 57], | 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 159], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1552KiB/s (1589kB/s)(7168KiB/4619msec) clat (msec): min=74, max=2552, avg=592.86, stdev=871.97 lat (msec): min=75, max=2552, avg=592.94, stdev=871.96 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 75], 10.00th=[ 75], 20.00th=[ 155], | 30.00th=[ 262], 40.00th=[ 262], 50.00th=[ 342], 60.00th=[ 359], | 70.00th=[ 359], 80.00th=[ 405], 90.00th=[ 2567], 95.00th=[ 2567], | 99.00th=[ 2567], 99.50th=[ 2567], 99.90th=[ 2567], 99.95th=[ 2567], | 99.99th=[ 2567] bw ( KiB/s): min= 2019, max= 4096, per=46.02%, avg=2856.40, stdev=1131.65, samples=5 iops : min= 1, max= 4, avg= 2.40, stdev= 1.52, samples=5 lat (msec) : 50=16.67%, 100=25.00%, 250=16.67%, 500=33.33% cpu : usr=0.00%, sys=3.57%, ctx=66, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4434KiB/s (4540kB/s), 462KiB/s-1982KiB/s (473kB/s-2029kB/s), io=20.0MiB (20.0MB), run=3617-4619msec WRITE: bw=6207KiB/s (6356kB/s), 1416KiB/s-2309KiB/s (1450kB/s-2364kB/s), io=28.0MiB (29.4MB), run=3617-4619msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2603: Thu May 1 19:56:19 2025 read: IOPS=1, BW=4781KiB/s (4896kB/s)(12.0MiB/2570msec) clat (msec): min=279, max=1832, avg=833.66, stdev=867.04 lat (msec): min=279, max=1832, avg=833.67, stdev=867.04 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 279], 10.00th=[ 279], 20.00th=[ 279], | 30.00th=[ 279], 40.00th=[ 388], 50.00th=[ 388], 60.00th=[ 388], | 70.00th=[ 1838], 80.00th=[ 1838], 90.00th=[ 1838], 95.00th=[ 1838], | 99.00th=[ 1838], 99.50th=[ 1838], 99.90th=[ 1838], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=10.43%, ctx=946, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2605: Thu May 1 19:56:19 2025 read: IOPS=0, BW=885KiB/s (906kB/s)(4096KiB/4628msec) clat (nsec): min=3835.4M, max=3835.4M, avg=3835406184.00, stdev= 0.00 lat (nsec): min=3835.4M, max=3835.4M, avg=3835408710.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 3842], 5.00th=[ 3842], 10.00th=[ 3842], 20.00th=[ 3842], | 30.00th=[ 3842], 40.00th=[ 3842], 50.00th=[ 3842], 60.00th=[ 3842], | 70.00th=[ 3842], 80.00th=[ 3842], 90.00th=[ 3842], 95.00th=[ 3842], | 99.00th=[ 3842], 99.50th=[ 3842], 99.90th=[ 3842], 99.95th=[ 3842], | 99.99th=[ 3842] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1770KiB/s (1813kB/s)(8192KiB/4628msec) clat (msec): min=313, max=391, avg=352.51, stdev=55.37 lat (msec): min=313, max=391, avg=352.73, stdev=55.28 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 313], 10.00th=[ 313], 20.00th=[ 313], | 30.00th=[ 313], 40.00th=[ 313], 50.00th=[ 313], 60.00th=[ 393], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 8110, max= 8110, per=100.00%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.45%, sys=13.12%, ctx=713, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2606: Thu May 1 19:56:19 2025 write: IOPS=0, BW=2341KiB/s (2398kB/s)(12.0MiB/5248msec) clat (msec): min=385, max=4348, avg=1715.77, stdev=2279.79 lat (msec): min=385, max=4348, avg=1716.21, stdev=2279.68 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 384], 10.00th=[ 384], 20.00th=[ 384], | 30.00th=[ 384], 40.00th=[ 414], 50.00th=[ 414], 60.00th=[ 414], | 70.00th=[ 4329], 80.00th=[ 4329], 90.00th=[ 4329], 95.00th=[ 4329], | 99.00th=[ 4329], 99.50th=[ 4329], 99.90th=[ 4329], 99.95th=[ 4329], | 99.99th=[ 4329] bw ( KiB/s): min= 7876, max= 8143, per=100.00%, avg=8009.50, stdev=188.80, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=5.77%, ctx=94, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2607: Thu May 1 19:56:19 2025 write: IOPS=0, BW=2300KiB/s (2355kB/s)(12.0MiB/5343msec) clat (msec): min=232, max=2632, avg=1729.19, stdev=1305.16 lat (msec): min=233, max=2633, avg=1729.74, stdev=1305.33 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 234], 10.00th=[ 234], 20.00th=[ 234], | 30.00th=[ 234], 40.00th=[ 2333], 50.00th=[ 2333], 60.00th=[ 2333], | 70.00th=[ 2635], 80.00th=[ 2635], 90.00th=[ 2635], 95.00th=[ 2635], | 99.00th=[ 2635], 99.50th=[ 2635], 99.90th=[ 2635], 99.95th=[ 2635], | 99.99th=[ 2635] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.04%, sys=6.31%, ctx=108, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3540KiB/s (3625kB/s), 885KiB/s-4781KiB/s (906kB/s-4896kB/s), io=16.0MiB (16.8MB), run=2570-4628msec WRITE: bw=6133KiB/s (6280kB/s), 1770KiB/s-2341KiB/s (1813kB/s-2398kB/s), io=32.0MiB (33.6MB), run=4628-5343msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2600: Thu May 1 19:56:19 2025 read: IOPS=0, BW=2649KiB/s (2713kB/s)(12.0MiB/4638msec) clat (msec): min=182, max=4017, avg=1479.13, stdev=2198.57 lat (msec): min=182, max=4017, avg=1479.13, stdev=2198.57 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 182], 10.00th=[ 182], 20.00th=[ 182], | 30.00th=[ 182], 40.00th=[ 239], 50.00th=[ 239], 60.00th=[ 239], | 70.00th=[ 4010], 80.00th=[ 4010], 90.00th=[ 4010], 95.00th=[ 4010], | 99.00th=[ 4010], 99.50th=[ 4010], 99.90th=[ 4010], 99.95th=[ 4010], | 99.99th=[ 4010] bw ( KiB/s): min=16351, max=16351, per=100.00%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.09%, sys=1.64%, ctx=39, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2601: Thu May 1 19:56:19 2025 read: IOPS=0, BW=775KiB/s (793kB/s)(4096KiB/5288msec) clat (nsec): min=4413.7M, max=4413.7M, avg=4413746962.00, stdev= 0.00 lat (nsec): min=4413.7M, max=4413.7M, avg=4413749680.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 4396], 5.00th=[ 4396], 10.00th=[ 4396], 20.00th=[ 4396], | 30.00th=[ 4396], 40.00th=[ 4396], 50.00th=[ 4396], 60.00th=[ 4396], | 70.00th=[ 4396], 80.00th=[ 4396], 90.00th=[ 4396], 95.00th=[ 4396], | 99.00th=[ 4396], 99.50th=[ 4396], 99.90th=[ 4396], 99.95th=[ 4396], | 99.99th=[ 4396] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1549KiB/s (1586kB/s)(8192KiB/5288msec) clat (usec): min=375503, max=376824, avg=376164.09, stdev=933.74 lat (msec): min=382, max=387, avg=384.76, stdev= 3.56 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 376], 10.00th=[ 376], 20.00th=[ 376], | 30.00th=[ 376], 40.00th=[ 376], 50.00th=[ 376], 60.00th=[ 376], | 70.00th=[ 376], 80.00th=[ 376], 90.00th=[ 376], 95.00th=[ 376], | 99.00th=[ 376], 99.50th=[ 376], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=1.68%, ctx=31, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2602: Thu May 1 19:56:19 2025 write: IOPS=0, BW=2210KiB/s (2264kB/s)(12.0MiB/5559msec) clat (msec): min=277, max=4829, avg=1817.99, stdev=2608.10 lat (msec): min=278, max=4829, avg=1818.39, stdev=2607.96 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 279], 10.00th=[ 279], 20.00th=[ 279], | 30.00th=[ 279], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 347], | 70.00th=[ 4799], 80.00th=[ 4799], 90.00th=[ 4799], 95.00th=[ 4799], | 99.00th=[ 4799], 99.50th=[ 4799], 99.90th=[ 4799], 99.95th=[ 4799], | 99.99th=[ 4799] bw ( KiB/s): min= 7728, max= 8192, per=100.00%, avg=7960.00, stdev=328.10, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=3.11%, ctx=89, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2604: Thu May 1 19:56:19 2025 write: IOPS=0, BW=2417KiB/s (2475kB/s)(12.0MiB/5085msec) clat (msec): min=203, max=2538, avg=1664.92, stdev=1273.87 lat (msec): min=204, max=2538, avg=1665.64, stdev=1273.69 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 203], 10.00th=[ 203], 20.00th=[ 203], | 30.00th=[ 203], 40.00th=[ 2265], 50.00th=[ 2265], 60.00th=[ 2265], | 70.00th=[ 2534], 80.00th=[ 2534], 90.00th=[ 2534], 95.00th=[ 2534], | 99.00th=[ 2534], 99.50th=[ 2534], 99.90th=[ 2534], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=2.04%, ctx=50, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3098KiB/s (3173kB/s), 775KiB/s-2649KiB/s (793kB/s-2713kB/s), io=16.0MiB (16.8MB), run=4638-5288msec WRITE: bw=5895KiB/s (6036kB/s), 1549KiB/s-2417KiB/s (1586kB/s-2475kB/s), io=32.0MiB (33.6MB), run=5085-5559msec