== sanity test 398b: DIO and buffer IO race ============== 13:21:36 (1750180896) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.84961 s, 27.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=3586: Tue Jun 17 13:22:23 2025 read: IOPS=35, BW=142KiB/s (146kB/s)(5992KiB/42065msec) clat (usec): min=54, max=391299, avg=13207.75, stdev=15092.60 lat (usec): min=54, max=391319, avg=13209.85, stdev=15092.72 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 106], 10.00th=[ 122], 20.00th=[ 379], | 30.00th=[ 7177], 40.00th=[ 9372], 50.00th=[ 11338], 60.00th=[ 13304], | 70.00th=[ 16319], 80.00th=[ 20317], 90.00th=[ 26870], 95.00th=[ 33424], | 99.00th=[ 47973], 99.50th=[ 59507], 99.90th=[137364], 99.95th=[392168], | 99.99th=[392168] bw ( KiB/s): min= 23, max= 262, per=25.63%, avg=142.25, stdev=44.39, samples=84 iops : min= 5, max= 65, avg=35.27, stdev=11.11, samples=84 write: IOPS=37, BW=150KiB/s (153kB/s)(6296KiB/42065msec) clat (usec): min=534, max=145480, avg=14059.27, stdev=15236.69 lat (usec): min=535, max=145482, avg=14062.42, stdev=15235.59 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 799], 10.00th=[ 881], 20.00th=[ 1352], | 30.00th=[ 2802], 40.00th=[ 5473], 50.00th=[ 9765], 60.00th=[ 13566], | 70.00th=[ 18482], 80.00th=[ 25035], 90.00th=[ 33162], 95.00th=[ 41681], | 99.00th=[ 67634], 99.50th=[ 79168], 99.90th=[109577], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 7, max= 263, per=25.68%, avg=149.20, stdev=50.98, samples=84 iops : min= 1, max= 65, avg=37.01, stdev=12.68, samples=84 lat (usec) : 100=1.76%, 250=7.71%, 500=0.49%, 750=1.04%, 1000=6.90% lat (msec) : 2=5.73%, 4=4.92%, 10=18.42%, 20=28.81%, 50=22.36% lat (msec) : 100=1.66%, 250=0.16%, 500=0.03% cpu : usr=0.14%, sys=12.06%, ctx=5000, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3587: Tue Jun 17 13:22:23 2025 read: IOPS=34, BW=137KiB/s (141kB/s)(5940KiB/43208msec) clat (usec): min=55, max=118620, avg=12902.80, stdev=11756.48 lat (usec): min=55, max=118621, avg=12904.55, stdev=11756.56 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 105], 10.00th=[ 123], 20.00th=[ 412], | 30.00th=[ 6915], 40.00th=[ 8979], 50.00th=[ 10683], 60.00th=[ 13173], | 70.00th=[ 16319], 80.00th=[ 20579], 90.00th=[ 26608], 95.00th=[ 33162], | 99.00th=[ 51119], 99.50th=[ 60031], 99.90th=[115868], 99.95th=[119014], | 99.99th=[119014] bw ( KiB/s): min= 7, max= 248, per=24.67%, avg=136.92, stdev=49.07, samples=86 iops : min= 1, max= 62, avg=34.02, stdev=12.35, samples=86 write: IOPS=36, BW=147KiB/s (150kB/s)(6348KiB/43208msec) clat (usec): min=504, max=399489, avg=15080.09, stdev=18905.35 lat (usec): min=505, max=399491, avg=15084.11, stdev=18904.83 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 807], 10.00th=[ 889], 20.00th=[ 1319], | 30.00th=[ 2802], 40.00th=[ 6259], 50.00th=[ 11076], 60.00th=[ 14222], | 70.00th=[ 19006], 80.00th=[ 25822], 90.00th=[ 35390], 95.00th=[ 43254], | 99.00th=[ 67634], 99.50th=[ 85459], 99.90th=[181404], 99.95th=[400557], | 99.99th=[400557] bw ( KiB/s): min= 39, max= 272, per=25.09%, avg=145.76, stdev=43.33, samples=86 iops : min= 9, max= 68, avg=36.23, stdev=10.88, samples=86 lat (usec) : 100=1.95%, 250=7.29%, 500=0.52%, 750=1.27%, 1000=6.80% lat (msec) : 2=5.57%, 4=5.01%, 10=18.16%, 20=28.22%, 50=22.98% lat (msec) : 100=1.99%, 250=0.20%, 500=0.03% cpu : usr=0.17%, sys=11.96%, ctx=5131, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3588: Tue Jun 17 13:22:23 2025 read: IOPS=34, BW=139KiB/s (143kB/s)(6028KiB/43241msec) clat (usec): min=51, max=147290, avg=12702.06, stdev=12320.93 lat (usec): min=52, max=147292, avg=12703.46, stdev=12320.97 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 101], 10.00th=[ 119], 20.00th=[ 239], | 30.00th=[ 6390], 40.00th=[ 8586], 50.00th=[ 10683], 60.00th=[ 13042], | 70.00th=[ 16057], 80.00th=[ 20317], 90.00th=[ 26870], 95.00th=[ 32900], | 99.00th=[ 52167], 99.50th=[ 63177], 99.90th=[132645], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 32, max= 256, per=25.18%, avg=139.76, stdev=50.17, samples=85 iops : min= 8, max= 64, avg=34.58, stdev=12.53, samples=85 write: IOPS=36, BW=145KiB/s (148kB/s)(6260KiB/43241msec) clat (usec): min=525, max=451077, avg=15289.46, stdev=19050.54 lat (usec): min=526, max=451080, avg=15291.12, stdev=19050.56 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 799], 10.00th=[ 881], 20.00th=[ 1385], | 30.00th=[ 3097], 40.00th=[ 6915], 50.00th=[ 11469], 60.00th=[ 14746], | 70.00th=[ 19792], 80.00th=[ 25560], 90.00th=[ 35914], 95.00th=[ 45351], | 99.00th=[ 62653], 99.50th=[ 82314], 99.90th=[137364], 99.95th=[450888], | 99.99th=[450888] bw ( KiB/s): min= 63, max= 293, per=24.95%, avg=144.96, stdev=42.57, samples=85 iops : min= 15, max= 73, avg=35.87, stdev=10.64, samples=85 lat (usec) : 100=2.31%, 250=7.62%, 500=0.49%, 750=1.33%, 1000=6.97% lat (msec) : 2=5.53%, 4=4.52%, 10=17.61%, 20=28.61%, 50=22.82% lat (msec) : 100=1.92%, 250=0.23%, 500=0.03% cpu : usr=0.17%, sys=11.75%, ctx=5130, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3589: Tue Jun 17 13:22:23 2025 read: IOPS=35, BW=141KiB/s (144kB/s)(6052KiB/42988msec) clat (usec): min=67, max=230973, avg=12799.88, stdev=13810.82 lat (usec): min=68, max=230975, avg=12803.04, stdev=13811.14 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 104], 10.00th=[ 121], 20.00th=[ 326], | 30.00th=[ 6980], 40.00th=[ 8717], 50.00th=[ 10814], 60.00th=[ 12911], | 70.00th=[ 15795], 80.00th=[ 19530], 90.00th=[ 25822], 95.00th=[ 32113], | 99.00th=[ 56886], 99.50th=[ 80217], 99.90th=[181404], 99.95th=[231736], | 99.99th=[231736] bw ( KiB/s): min= 63, max= 264, per=25.37%, avg=140.81, stdev=42.20, samples=84 iops : min= 15, max= 66, avg=34.79, stdev=10.63, samples=84 write: IOPS=36, BW=145KiB/s (149kB/s)(6236KiB/42988msec) clat (usec): min=548, max=412591, avg=15020.48, stdev=18735.14 lat (usec): min=548, max=412594, avg=15023.01, stdev=18734.61 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 799], 10.00th=[ 889], 20.00th=[ 1336], | 30.00th=[ 2835], 40.00th=[ 7046], 50.00th=[ 11338], 60.00th=[ 14484], | 70.00th=[ 19268], 80.00th=[ 24511], 90.00th=[ 34341], 95.00th=[ 43254], | 99.00th=[ 68682], 99.50th=[ 89654], 99.90th=[152044], 99.95th=[413139], | 99.99th=[413139] bw ( KiB/s): min= 47, max= 248, per=24.99%, avg=145.18, stdev=46.28, samples=84 iops : min= 11, max= 62, avg=35.88, stdev=11.66, samples=84 lat (usec) : 100=1.95%, 250=7.71%, 500=0.59%, 750=1.86%, 1000=6.35% lat (msec) : 2=5.86%, 4=4.72%, 10=17.35%, 20=30.08%, 50=21.13% lat (msec) : 100=2.15%, 250=0.23%, 500=0.03% cpu : usr=0.23%, sys=11.69%, ctx=5086, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=555KiB/s (569kB/s), 137KiB/s-142KiB/s (141kB/s-146kB/s), io=23.4MiB (24.6MB), run=42065-43241msec WRITE: bw=581KiB/s (595kB/s), 145KiB/s-150KiB/s (148kB/s-153kB/s), io=24.6MiB (25.7MB), run=42065-43241msec rand-rw: (groupid=0, jobs=1): err= 0: pid=3581: Tue Jun 17 13:24:06 2025 read: IOPS=10, BW=41.0KiB/s (42.0kB/s)(5992KiB/145995msec) clat (msec): min=3, max=416, avg=16.06, stdev=15.22 lat (msec): min=3, max=416, avg=16.06, stdev=15.22 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 16], | 70.00th=[ 18], 80.00th=[ 21], 90.00th=[ 28], 95.00th=[ 35], | 99.00th=[ 58], 99.50th=[ 68], 99.90th=[ 184], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 7, max= 136, per=25.26%, avg=41.42, stdev=22.95, samples=285 iops : min= 1, max= 34, avg=10.10, stdev= 5.76, samples=285 write: IOPS=10, BW=43.1KiB/s (44.2kB/s)(6296KiB/145995msec) clat (msec): min=27, max=280, avg=77.35, stdev=30.09 lat (msec): min=27, max=280, avg=77.35, stdev=30.09 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 42], 10.00th=[ 47], 20.00th=[ 54], | 30.00th=[ 59], 40.00th=[ 65], 50.00th=[ 72], 60.00th=[ 79], | 70.00th=[ 86], 80.00th=[ 97], 90.00th=[ 117], 95.00th=[ 132], | 99.00th=[ 178], 99.50th=[ 213], 99.90th=[ 239], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 15, max= 80, per=25.05%, avg=42.83, stdev=12.72, samples=290 iops : min= 3, max= 20, avg=10.45, stdev= 3.24, samples=290 lat (msec) : 4=0.23%, 10=14.58%, 20=23.57%, 50=17.25%, 100=35.03% lat (msec) : 250=9.28%, 500=0.07% cpu : usr=0.10%, sys=2.77%, ctx=4029, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3582: Tue Jun 17 13:24:06 2025 read: IOPS=10, BW=40.6KiB/s (41.6kB/s)(5940KiB/146217msec) clat (msec): min=3, max=397, avg=16.13, stdev=15.73 lat (msec): min=3, max=397, avg=16.13, stdev=15.73 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 16], | 70.00th=[ 18], 80.00th=[ 20], 90.00th=[ 27], 95.00th=[ 35], | 99.00th=[ 63], 99.50th=[ 96], 99.90th=[ 171], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 7, max= 144, per=25.18%, avg=41.29, stdev=23.79, samples=284 iops : min= 1, max= 36, avg= 9.99, stdev= 6.01, samples=284 write: IOPS=10, BW=43.4KiB/s (44.5kB/s)(6348KiB/146217msec) clat (msec): min=25, max=294, avg=76.91, stdev=31.30 lat (msec): min=25, max=294, avg=76.92, stdev=31.30 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 41], 10.00th=[ 46], 20.00th=[ 53], | 30.00th=[ 59], 40.00th=[ 65], 50.00th=[ 71], 60.00th=[ 78], | 70.00th=[ 86], 80.00th=[ 95], 90.00th=[ 117], 95.00th=[ 136], | 99.00th=[ 190], 99.50th=[ 222], 99.90th=[ 247], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 8, max= 80, per=25.13%, avg=42.98, stdev=13.05, samples=291 iops : min= 2, max= 20, avg=10.40, stdev= 3.32, samples=291 lat (msec) : 4=0.07%, 10=14.42%, 20=24.32%, 50=17.12%, 100=35.22% lat (msec) : 250=8.79%, 500=0.07% cpu : usr=0.06%, sys=2.85%, ctx=4047, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3583: Tue Jun 17 13:24:06 2025 read: IOPS=10, BW=41.4KiB/s (42.4kB/s)(6028KiB/145490msec) clat (msec): min=2, max=170, avg=15.78, stdev=10.93 lat (msec): min=2, max=170, avg=15.78, stdev=10.93 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 16], | 70.00th=[ 18], 80.00th=[ 21], 90.00th=[ 27], 95.00th=[ 35], | 99.00th=[ 55], 99.50th=[ 60], 99.90th=[ 131], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 7, max= 143, per=25.35%, avg=41.58, stdev=23.73, samples=286 iops : min= 1, max= 35, avg=10.09, stdev= 5.97, samples=286 write: IOPS=10, BW=43.0KiB/s (44.1kB/s)(6260KiB/145490msec) clat (msec): min=23, max=504, avg=77.64, stdev=32.00 lat (msec): min=23, max=504, avg=77.65, stdev=32.00 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 40], 10.00th=[ 45], 20.00th=[ 53], | 30.00th=[ 59], 40.00th=[ 66], 50.00th=[ 72], 60.00th=[ 80], | 70.00th=[ 88], 80.00th=[ 99], 90.00th=[ 118], 95.00th=[ 136], | 99.00th=[ 176], 99.50th=[ 190], 99.90th=[ 257], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 7, max= 80, per=24.95%, avg=42.67, stdev=13.05, samples=289 iops : min= 1, max= 20, avg=10.36, stdev= 3.31, samples=289 lat (msec) : 4=0.16%, 10=14.78%, 20=23.18%, 50=18.62%, 100=33.59% lat (msec) : 250=9.60%, 500=0.03%, 750=0.03% cpu : usr=0.06%, sys=2.90%, ctx=4006, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3584: Tue Jun 17 13:24:06 2025 read: IOPS=10, BW=41.5KiB/s (42.5kB/s)(6052KiB/145900msec) clat (msec): min=3, max=249, avg=15.99, stdev=12.94 lat (msec): min=3, max=249, avg=15.99, stdev=12.94 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 12], 50.00th=[ 13], 60.00th=[ 16], | 70.00th=[ 18], 80.00th=[ 21], 90.00th=[ 28], 95.00th=[ 36], | 99.00th=[ 58], 99.50th=[ 67], 99.90th=[ 184], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min= 7, max= 136, per=25.70%, avg=42.14, stdev=24.30, samples=284 iops : min= 1, max= 34, avg=10.29, stdev= 6.11, samples=284 write: IOPS=10, BW=42.7KiB/s (43.8kB/s)(6236KiB/145900msec) clat (msec): min=28, max=501, avg=77.94, stdev=31.83 lat (msec): min=28, max=501, avg=77.95, stdev=31.84 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 42], 10.00th=[ 47], 20.00th=[ 53], | 30.00th=[ 59], 40.00th=[ 65], 50.00th=[ 72], 60.00th=[ 80], | 70.00th=[ 88], 80.00th=[ 100], 90.00th=[ 116], 95.00th=[ 138], | 99.00th=[ 180], 99.50th=[ 192], 99.90th=[ 253], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 15, max= 80, per=24.84%, avg=42.47, stdev=12.64, samples=290 iops : min= 3, max= 20, avg=10.36, stdev= 3.19, samples=290 lat (msec) : 4=0.07%, 10=15.59%, 20=23.54%, 50=16.96%, 100=33.82% lat (msec) : 250=9.96%, 500=0.03%, 750=0.03% cpu : usr=0.05%, sys=2.88%, ctx=4000, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=164KiB/s (168kB/s), 40.6KiB/s-41.5KiB/s (41.6kB/s-42.5kB/s), io=23.4MiB (24.6MB), run=145490-146217msec WRITE: bw=172KiB/s (176kB/s), 42.7KiB/s-43.4KiB/s (43.8kB/s-44.5kB/s), io=24.6MiB (25.7MB), run=145490-146217msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=3725: Tue Jun 17 13:24:14 2025 read: IOPS=56, BW=905KiB/s (926kB/s)(5920KiB/6545msec) clat (usec): min=79, max=60241, avg=8111.01, stdev=8190.07 lat (usec): min=80, max=60242, avg=8112.19, stdev=8190.14 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 96], 10.00th=[ 114], 20.00th=[ 130], | 30.00th=[ 412], 40.00th=[ 5735], 50.00th=[ 7242], 60.00th=[ 8717], | 70.00th=[10945], 80.00th=[13829], 90.00th=[17171], 95.00th=[21627], | 99.00th=[43779], 99.50th=[54789], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 480, max= 1117, per=24.71%, avg=903.00, stdev=184.31, samples=12 iops : min= 30, max= 69, avg=56.17, stdev=11.30, samples=12 write: IOPS=60, BW=973KiB/s (996kB/s)(6368KiB/6545msec) clat (usec): min=816, max=136928, avg=8676.78, stdev=13060.13 lat (usec): min=818, max=136931, avg=8678.53, stdev=13060.15 clat percentiles (usec): | 1.00th=[ 865], 5.00th=[ 1074], 10.00th=[ 1156], 20.00th=[ 1287], | 30.00th=[ 1582], 40.00th=[ 2040], 50.00th=[ 3818], 60.00th=[ 7046], | 70.00th=[ 9634], 80.00th=[ 13698], 90.00th=[ 20841], 95.00th=[ 27395], | 99.00th=[ 62653], 99.50th=[ 99091], 99.90th=[137364], 99.95th=[137364], | 99.99th=[137364] bw ( KiB/s): min= 480, max= 1660, per=25.49%, avg=980.17, stdev=332.86, samples=12 iops : min= 30, max= 103, avg=61.00, stdev=20.62, samples=12 lat (usec) : 100=2.99%, 250=10.81%, 500=1.30%, 1000=1.17% lat (msec) : 2=18.88%, 4=7.03%, 10=26.82%, 20=22.27%, 50=7.68% lat (msec) : 100=0.91%, 250=0.13% cpu : usr=0.15%, sys=16.69%, ctx=1142, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3727: Tue Jun 17 13:24:14 2025 read: IOPS=54, BW=877KiB/s (898kB/s)(5744KiB/6553msec) clat (usec): min=63, max=68947, avg=8673.20, stdev=9620.40 lat (usec): min=63, max=68949, avg=8674.38, stdev=9620.47 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 102], 10.00th=[ 114], 20.00th=[ 133], | 30.00th=[ 196], 40.00th=[ 5669], 50.00th=[ 7373], 60.00th=[ 9110], | 70.00th=[11469], 80.00th=[14746], 90.00th=[19006], 95.00th=[25035], | 99.00th=[50594], 99.50th=[61604], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 479, max= 1216, per=23.67%, avg=865.23, stdev=252.86, samples=13 iops : min= 29, max= 76, avg=53.77, stdev=15.97, samples=13 write: IOPS=62, BW=999KiB/s (1023kB/s)(6544KiB/6553msec) clat (usec): min=638, max=88515, avg=8340.12, stdev=10636.51 lat (usec): min=638, max=88516, avg=8342.16, stdev=10636.52 clat percentiles (usec): | 1.00th=[ 865], 5.00th=[ 1057], 10.00th=[ 1156], 20.00th=[ 1336], | 30.00th=[ 1811], 40.00th=[ 2474], 50.00th=[ 4080], 60.00th=[ 6587], | 70.00th=[ 9634], 80.00th=[14222], 90.00th=[19268], 95.00th=[26346], | 99.00th=[50594], 99.50th=[67634], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 448, max= 1312, per=25.39%, avg=976.08, stdev=260.44, samples=13 iops : min= 28, max= 82, avg=60.69, stdev=16.40, samples=13 lat (usec) : 100=2.08%, 250=12.50%, 500=1.04%, 750=0.52%, 1000=1.43% lat (msec) : 2=16.54%, 4=8.85%, 10=24.22%, 20=24.09%, 50=7.55% lat (msec) : 100=1.17% cpu : usr=0.46%, sys=16.70%, ctx=1098, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3728: Tue Jun 17 13:24:14 2025 read: IOPS=60, BW=968KiB/s (991kB/s)(6256KiB/6462msec) clat (usec): min=51, max=108334, avg=8913.19, stdev=10706.91 lat (usec): min=52, max=108335, avg=8914.63, stdev=10706.86 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 96], 10.00th=[ 111], 20.00th=[ 130], | 30.00th=[ 239], 40.00th=[ 5800], 50.00th=[ 7242], 60.00th=[ 9241], | 70.00th=[ 11469], 80.00th=[ 13829], 90.00th=[ 18220], 95.00th=[ 24249], | 99.00th=[ 56361], 99.50th=[ 72877], 99.90th=[108528], 99.95th=[108528], | 99.99th=[108528] bw ( KiB/s): min= 608, max= 1376, per=26.54%, avg=970.00, stdev=237.89, samples=12 iops : min= 38, max= 86, avg=60.42, stdev=14.95, samples=12 write: IOPS=58, BW=933KiB/s (956kB/s)(6032KiB/6462msec) clat (usec): min=806, max=57091, avg=7559.77, stdev=8541.55 lat (usec): min=807, max=57092, avg=7561.99, stdev=8541.81 clat percentiles (usec): | 1.00th=[ 881], 5.00th=[ 1074], 10.00th=[ 1172], 20.00th=[ 1303], | 30.00th=[ 1696], 40.00th=[ 2507], 50.00th=[ 4047], 60.00th=[ 5800], | 70.00th=[ 9372], 80.00th=[13173], 90.00th=[17957], 95.00th=[25297], | 99.00th=[44303], 99.50th=[50070], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 576, max= 1600, per=23.84%, avg=916.58, stdev=276.11, samples=12 iops : min= 36, max= 100, avg=57.08, stdev=17.26, samples=12 lat (usec) : 100=2.99%, 250=12.50%, 500=0.52%, 750=0.52%, 1000=1.69% lat (msec) : 2=15.10%, 4=7.42%, 10=26.30%, 20=24.61%, 50=7.55% lat (msec) : 100=0.65%, 250=0.13% cpu : usr=0.17%, sys=17.29%, ctx=1190, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3730: Tue Jun 17 13:24:14 2025 read: IOPS=59, BW=948KiB/s (970kB/s)(6032KiB/6366msec) clat (usec): min=73, max=108136, avg=7872.55, stdev=9171.93 lat (usec): min=74, max=108138, avg=7874.98, stdev=9172.71 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 101], 10.00th=[ 115], 20.00th=[ 139], | 30.00th=[ 392], 40.00th=[ 5604], 50.00th=[ 6718], 60.00th=[ 8356], | 70.00th=[ 10159], 80.00th=[ 12387], 90.00th=[ 16581], 95.00th=[ 21103], | 99.00th=[ 42206], 99.50th=[ 48497], 99.90th=[108528], 99.95th=[108528], | 99.99th=[108528] bw ( KiB/s): min= 544, max= 1472, per=25.88%, avg=945.92, stdev=321.60, samples=12 iops : min= 34, max= 92, avg=58.83, stdev=20.17, samples=12 write: IOPS=61, BW=983KiB/s (1006kB/s)(6256KiB/6366msec) clat (usec): min=627, max=59992, avg=8280.60, stdev=9668.25 lat (usec): min=628, max=59994, avg=8282.50, stdev=9668.44 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 1057], 10.00th=[ 1188], 20.00th=[ 1352], | 30.00th=[ 1827], 40.00th=[ 2442], 50.00th=[ 3556], 60.00th=[ 6325], | 70.00th=[ 9896], 80.00th=[14484], 90.00th=[20841], 95.00th=[29754], | 99.00th=[43779], 99.50th=[46924], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 480, max= 1405, per=25.64%, avg=985.75, stdev=268.12, samples=12 iops : min= 30, max= 87, avg=61.33, stdev=16.52, samples=12 lat (usec) : 100=2.21%, 250=11.72%, 500=0.91%, 750=0.91%, 1000=1.43% lat (msec) : 2=16.15%, 4=9.64%, 10=26.82%, 20=21.48%, 50=8.46% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.31%, sys=16.65%, ctx=1168, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3655KiB/s (3743kB/s), 877KiB/s-968KiB/s (898kB/s-991kB/s), io=23.4MiB (24.5MB), run=6366-6553msec WRITE: bw=3846KiB/s (3938kB/s), 933KiB/s-999KiB/s (956kB/s-1023kB/s), io=24.6MiB (25.8MB), run=6366-6553msec rand-rw: (groupid=0, jobs=1): err= 0: pid=3723: Tue Jun 17 13:24:37 2025 read: IOPS=12, BW=194KiB/s (199kB/s)(5920KiB/30466msec) clat (msec): min=4, max=112, avg=12.74, stdev=12.05 lat (msec): min=4, max=112, avg=12.74, stdev=12.05 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 8], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 12], 80.00th=[ 16], 90.00th=[ 22], 95.00th=[ 32], | 99.00th=[ 74], 99.50th=[ 111], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 63, max= 509, per=24.63%, avg=193.58, stdev=95.73, samples=59 iops : min= 3, max= 31, avg=11.66, stdev= 6.00, samples=59 write: IOPS=13, BW=209KiB/s (214kB/s)(6368KiB/30466msec) clat (msec): min=25, max=205, avg=64.41, stdev=20.93 lat (msec): min=25, max=205, avg=64.41, stdev=20.94 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 38], 10.00th=[ 42], 20.00th=[ 47], | 30.00th=[ 52], 40.00th=[ 57], 50.00th=[ 62], 60.00th=[ 68], | 70.00th=[ 73], 80.00th=[ 80], 90.00th=[ 90], 95.00th=[ 102], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 205], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 127, max= 320, per=25.08%, avg=207.45, stdev=42.22, samples=60 iops : min= 7, max= 20, avg=12.52, stdev= 2.68, samples=60 lat (msec) : 10=26.69%, 20=16.15%, 50=18.88%, 100=35.29%, 250=2.99% cpu : usr=0.07%, sys=3.52%, ctx=957, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3724: Tue Jun 17 13:24:37 2025 read: IOPS=11, BW=189KiB/s (194kB/s)(5744KiB/30371msec) clat (usec): min=3880, max=66524, avg=11450.26, stdev=8427.16 lat (usec): min=3882, max=66526, avg=11452.16, stdev=8427.02 clat percentiles (usec): | 1.00th=[ 4293], 5.00th=[ 5080], 10.00th=[ 5473], 20.00th=[ 6325], | 30.00th=[ 7177], 40.00th=[ 8160], 50.00th=[ 9241], 60.00th=[ 9896], | 70.00th=[11076], 80.00th=[13698], 90.00th=[19530], 95.00th=[30278], | 99.00th=[43779], 99.50th=[55313], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 31, max= 480, per=24.25%, avg=190.59, stdev=104.85, samples=59 iops : min= 1, max= 30, avg=11.64, stdev= 6.62, samples=59 write: IOPS=13, BW=215KiB/s (221kB/s)(6544KiB/30371msec) clat (msec): min=27, max=155, avg=64.10, stdev=19.81 lat (msec): min=27, max=155, avg=64.10, stdev=19.81 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 48], | 30.00th=[ 53], 40.00th=[ 57], 50.00th=[ 61], 60.00th=[ 66], | 70.00th=[ 72], 80.00th=[ 79], 90.00th=[ 91], 95.00th=[ 100], | 99.00th=[ 123], 99.50th=[ 138], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 127, max= 319, per=25.82%, avg=213.50, stdev=41.81, samples=60 iops : min= 7, max= 19, avg=13.07, stdev= 2.67, samples=60 lat (msec) : 4=0.13%, 10=27.99%, 20=14.06%, 50=16.80%, 100=38.54% lat (msec) : 250=2.47% cpu : usr=0.09%, sys=3.46%, ctx=938, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3726: Tue Jun 17 13:24:37 2025 read: IOPS=13, BW=213KiB/s (218kB/s)(6256KiB/29436msec) clat (usec): min=3493, max=90117, avg=12149.67, stdev=10093.91 lat (usec): min=3494, max=90119, avg=12151.59, stdev=10093.84 clat percentiles (usec): | 1.00th=[ 4490], 5.00th=[ 5342], 10.00th=[ 5800], 20.00th=[ 6718], | 30.00th=[ 7504], 40.00th=[ 8455], 50.00th=[ 8979], 60.00th=[10421], | 70.00th=[11469], 80.00th=[13829], 90.00th=[20055], 95.00th=[30278], | 99.00th=[61080], 99.50th=[61604], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 32, max= 574, per=27.26%, avg=214.29, stdev=115.97, samples=58 iops : min= 2, max= 35, avg=13.14, stdev= 7.11, samples=58 write: IOPS=12, BW=205KiB/s (210kB/s)(6032KiB/29436msec) clat (msec): min=27, max=154, avg=65.08, stdev=20.76 lat (msec): min=27, max=154, avg=65.08, stdev=20.76 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 39], 10.00th=[ 43], 20.00th=[ 48], | 30.00th=[ 54], 40.00th=[ 58], 50.00th=[ 63], 60.00th=[ 67], | 70.00th=[ 72], 80.00th=[ 81], 90.00th=[ 90], 95.00th=[ 104], | 99.00th=[ 146], 99.50th=[ 150], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 95, max= 320, per=24.52%, avg=202.74, stdev=47.04, samples=58 iops : min= 5, max= 20, avg=12.41, stdev= 2.98, samples=58 lat (msec) : 4=0.13%, 10=29.17%, 20=16.54%, 50=15.76%, 100=35.16% lat (msec) : 250=3.26% cpu : usr=0.00%, sys=3.76%, ctx=956, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3729: Tue Jun 17 13:24:37 2025 read: IOPS=12, BW=201KiB/s (206kB/s)(6032KiB/30022msec) clat (usec): min=4433, max=61127, avg=10898.11, stdev=7897.22 lat (usec): min=4435, max=61129, avg=10900.05, stdev=7897.12 clat percentiles (usec): | 1.00th=[ 4686], 5.00th=[ 5145], 10.00th=[ 5669], 20.00th=[ 6390], | 30.00th=[ 7046], 40.00th=[ 7963], 50.00th=[ 8717], 60.00th=[ 9634], | 70.00th=[10552], 80.00th=[12387], 90.00th=[17695], 95.00th=[26870], | 99.00th=[48497], 99.50th=[50594], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 32, max= 480, per=25.77%, avg=202.56, stdev=106.44, samples=59 iops : min= 2, max= 30, avg=12.44, stdev= 6.58, samples=59 write: IOPS=13, BW=208KiB/s (213kB/s)(6256KiB/30022msec) clat (msec): min=27, max=161, avg=65.58, stdev=22.07 lat (msec): min=27, max=161, avg=65.58, stdev=22.07 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 47], | 30.00th=[ 53], 40.00th=[ 58], 50.00th=[ 62], 60.00th=[ 67], | 70.00th=[ 73], 80.00th=[ 83], 90.00th=[ 93], 95.00th=[ 109], | 99.00th=[ 140], 99.50th=[ 142], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 95, max= 288, per=25.18%, avg=208.27, stdev=46.59, samples=60 iops : min= 5, max= 18, avg=12.80, stdev= 2.97, samples=60 lat (msec) : 10=31.51%, 20=13.67%, 50=16.93%, 100=33.72%, 250=4.17% cpu : usr=0.04%, sys=3.63%, ctx=923, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=786KiB/s (805kB/s), 189KiB/s-213KiB/s (194kB/s-218kB/s), io=23.4MiB (24.5MB), run=29436-30466msec WRITE: bw=827KiB/s (847kB/s), 205KiB/s-215KiB/s (210kB/s-221kB/s), io=24.6MiB (25.8MB), run=29436-30466msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=3760: Tue Jun 17 13:24:40 2025 read: IOPS=6, BW=6522KiB/s (6679kB/s)(7168KiB/1099msec) clat (msec): min=7, max=119, avg=86.19, stdev=38.92 lat (msec): min=7, max=119, avg=86.19, stdev=38.92 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 8], 10.00th=[ 8], 20.00th=[ 74], | 30.00th=[ 82], 40.00th=[ 82], 50.00th=[ 96], 60.00th=[ 108], | 70.00th=[ 108], 80.00th=[ 118], 90.00th=[ 120], 95.00th=[ 120], | 99.00th=[ 120], 99.50th=[ 120], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 4039, max= 8159, per=39.07%, avg=6099.00, stdev=2913.28, samples=2 iops : min= 3, max= 7, avg= 5.00, stdev= 2.83, samples=2 write: IOPS=4, BW=4659KiB/s (4771kB/s)(5120KiB/1099msec) clat (msec): min=43, max=148, avg=94.57, stdev=40.18 lat (msec): min=43, max=148, avg=94.66, stdev=40.17 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 44], | 30.00th=[ 80], 40.00th=[ 80], 50.00th=[ 85], 60.00th=[ 85], | 70.00th=[ 118], 80.00th=[ 118], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 2039, max= 6059, per=18.53%, avg=4049.00, stdev=2842.57, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 lat (msec) : 10=8.33%, 50=8.33%, 100=41.67%, 250=41.67% cpu : usr=0.00%, sys=22.13%, ctx=447, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3761: Tue Jun 17 13:24:40 2025 read: IOPS=1, BW=1561KiB/s (1598kB/s)(2048KiB/1312msec) clat (usec): min=113579, max=114793, avg=114186.60, stdev=858.08 lat (usec): min=113595, max=114794, avg=114194.86, stdev=847.86 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 114], 10.00th=[ 114], 20.00th=[ 114], | 30.00th=[ 114], 40.00th=[ 114], 50.00th=[ 114], 60.00th=[ 115], | 70.00th=[ 115], 80.00th=[ 115], 90.00th=[ 115], 95.00th=[ 115], | 99.00th=[ 115], 99.50th=[ 115], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 2048, max= 2048, per=13.12%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=7, BW=7805KiB/s (7992kB/s)(10.0MiB/1312msec) clat (msec): min=32, max=166, avg=92.78, stdev=41.87 lat (msec): min=32, max=166, avg=92.87, stdev=41.86 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 35], | 30.00th=[ 79], 40.00th=[ 81], 50.00th=[ 81], 60.00th=[ 95], | 70.00th=[ 111], 80.00th=[ 111], 90.00th=[ 142], 95.00th=[ 167], | 99.00th=[ 167], 99.50th=[ 167], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 2048, max= 8192, per=23.43%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (msec) : 50=16.67%, 100=33.33%, 250=50.00% cpu : usr=0.00%, sys=17.01%, ctx=158, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3762: Tue Jun 17 13:24:40 2025 read: IOPS=5, BW=5560KiB/s (5694kB/s)(6144KiB/1105msec) clat (usec): min=475, max=107152, avg=35072.27, stdev=38959.14 lat (usec): min=476, max=107153, avg=35073.93, stdev=38958.98 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 474], 10.00th=[ 474], 20.00th=[ 881], | 30.00th=[ 881], 40.00th=[ 30278], 50.00th=[ 30278], 60.00th=[ 34866], | 70.00th=[ 36963], 80.00th=[ 36963], 90.00th=[107480], 95.00th=[107480], | 99.00th=[107480], 99.50th=[107480], 99.90th=[107480], 99.95th=[107480], | 99.99th=[107480] bw ( KiB/s): min=10219, max=10219, per=65.47%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5560KiB/s (5694kB/s)(6144KiB/1105msec) clat (msec): min=44, max=290, avg=133.62, stdev=95.62 lat (msec): min=44, max=290, avg=133.68, stdev=95.62 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 69], | 30.00th=[ 69], 40.00th=[ 79], 50.00th=[ 79], 60.00th=[ 112], | 70.00th=[ 207], 80.00th=[ 207], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 4096, max= 6131, per=23.40%, avg=5113.50, stdev=1438.96, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 lat (usec) : 500=8.33%, 1000=8.33% lat (msec) : 50=33.33%, 100=16.67%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=16.39%, ctx=298, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3763: Tue Jun 17 13:24:40 2025 read: IOPS=4, BW=4613KiB/s (4723kB/s)(5120KiB/1110msec) clat (usec): min=786, max=119486, avg=79367.01, stdev=47731.08 lat (usec): min=789, max=119487, avg=79368.75, stdev=47730.62 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 791], 10.00th=[ 791], 20.00th=[ 791], | 30.00th=[ 76022], 40.00th=[ 76022], 50.00th=[ 84411], 60.00th=[ 84411], | 70.00th=[114820], 80.00th=[114820], 90.00th=[119014], 95.00th=[119014], | 99.00th=[119014], 99.50th=[119014], 99.90th=[119014], 99.95th=[119014], | 99.99th=[119014] bw ( KiB/s): min= 8192, max= 8192, per=52.48%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6458KiB/s (6613kB/s)(7168KiB/1110msec) clat (msec): min=23, max=264, avg=96.61, stdev=80.43 lat (msec): min=23, max=264, avg=96.78, stdev=80.39 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 52], | 30.00th=[ 56], 40.00th=[ 56], 50.00th=[ 72], 60.00th=[ 86], | 70.00th=[ 86], 80.00th=[ 125], 90.00th=[ 266], 95.00th=[ 266], | 99.00th=[ 266], 99.50th=[ 266], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 6144, max= 8192, per=32.80%, avg=7168.00, stdev=1448.15, samples=2 iops : min= 6, max= 8, avg= 7.00, stdev= 1.41, samples=2 lat (usec) : 1000=8.33% lat (msec) : 50=8.33%, 100=50.00%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=20.47%, ctx=249, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.2MiB/s (15.0MB/s), 1561KiB/s-6522KiB/s (1598kB/s-6679kB/s), io=20.0MiB (20.0MB), run=1099-1312msec WRITE: bw=21.3MiB/s (22.4MB/s), 4659KiB/s-7805KiB/s (4771kB/s-7992kB/s), io=28.0MiB (29.4MB), run=1099-1312msec rand-rw: (groupid=0, jobs=1): err= 0: pid=3765: Tue Jun 17 13:24:41 2025 read: IOPS=3, BW=3862KiB/s (3955kB/s)(7168KiB/1856msec) clat (msec): min=59, max=303, avg=151.02, stdev=85.26 lat (msec): min=59, max=303, avg=151.02, stdev=85.26 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 85], | 30.00th=[ 111], 40.00th=[ 111], 50.00th=[ 123], 60.00th=[ 157], | 70.00th=[ 157], 80.00th=[ 222], 90.00th=[ 305], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 2023, max= 4087, per=30.95%, avg=3385.67, stdev=1180.27, samples=3 iops : min= 1, max= 3, avg= 2.33, stdev= 1.15, samples=3 write: IOPS=2, BW=2759KiB/s (2825kB/s)(5120KiB/1856msec) clat (msec): min=71, max=156, avg=104.47, stdev=40.80 lat (msec): min=71, max=156, avg=104.57, stdev=40.78 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 72], 10.00th=[ 72], 20.00th=[ 72], | 30.00th=[ 73], 40.00th=[ 73], 50.00th=[ 81], 60.00th=[ 81], | 70.00th=[ 140], 80.00th=[ 140], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 2023, max= 2043, per=13.27%, avg=2033.00, stdev=14.14, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 100=41.67%, 250=50.00%, 500=8.33% cpu : usr=0.05%, sys=5.23%, ctx=47, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3766: Tue Jun 17 13:24:41 2025 read: IOPS=1, BW=1134KiB/s (1161kB/s)(2048KiB/1806msec) clat (msec): min=48, max=115, avg=82.11, stdev=47.74 lat (msec): min=48, max=115, avg=82.11, stdev=47.74 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 49], 10.00th=[ 49], 20.00th=[ 49], | 30.00th=[ 49], 40.00th=[ 49], 50.00th=[ 49], 60.00th=[ 116], | 70.00th=[ 116], 80.00th=[ 116], 90.00th=[ 116], 95.00th=[ 116], | 99.00th=[ 116], 99.50th=[ 116], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 2043, max= 2043, per=18.67%, avg=2043.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5670KiB/s (5806kB/s)(10.0MiB/1806msec) clat (msec): min=68, max=223, avg=137.11, stdev=51.87 lat (msec): min=68, max=223, avg=137.20, stdev=51.87 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 69], 10.00th=[ 69], 20.00th=[ 82], | 30.00th=[ 92], 40.00th=[ 105], 50.00th=[ 128], 60.00th=[ 146], | 70.00th=[ 180], 80.00th=[ 180], 90.00th=[ 199], 95.00th=[ 224], | 99.00th=[ 224], 99.50th=[ 224], 99.90th=[ 224], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 6144, max= 8175, per=46.75%, avg=7159.50, stdev=1436.13, samples=2 iops : min= 6, max= 7, avg= 6.50, stdev= 0.71, samples=2 lat (msec) : 50=8.33%, 100=25.00%, 250=66.67% cpu : usr=0.00%, sys=7.04%, ctx=70, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3767: Tue Jun 17 13:24:41 2025 read: IOPS=3, BW=3383KiB/s (3464kB/s)(6144KiB/1816msec) clat (msec): min=39, max=110, avg=60.33, stdev=27.40 lat (msec): min=39, max=110, avg=60.34, stdev=27.39 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 53], | 70.00th=[ 73], 80.00th=[ 73], 90.00th=[ 111], 95.00th=[ 111], | 99.00th=[ 111], 99.50th=[ 111], 99.90th=[ 111], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 8175, max= 8175, per=74.73%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3383KiB/s (3464kB/s)(6144KiB/1816msec) clat (msec): min=115, max=337, avg=197.45, stdev=95.34 lat (msec): min=115, max=338, avg=197.54, stdev=95.32 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 116], 10.00th=[ 116], 20.00th=[ 121], | 30.00th=[ 121], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 161], | 70.00th=[ 296], 80.00th=[ 296], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 2019, max= 4087, per=17.68%, avg=2708.33, stdev=1193.96, samples=3 iops : min= 1, max= 3, avg= 1.67, stdev= 1.15, samples=3 lat (msec) : 50=25.00%, 100=16.67%, 250=41.67%, 500=16.67% cpu : usr=0.06%, sys=7.33%, ctx=52, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3768: Tue Jun 17 13:24:41 2025 read: IOPS=2, BW=2735KiB/s (2801kB/s)(5120KiB/1872msec) clat (usec): min=31750, max=73544, avg=57174.24, stdev=17378.09 lat (usec): min=31752, max=73546, avg=57176.06, stdev=17377.93 clat percentiles (usec): | 1.00th=[31851], 5.00th=[31851], 10.00th=[31851], 20.00th=[31851], | 30.00th=[46924], 40.00th=[46924], 50.00th=[65274], 60.00th=[65274], | 70.00th=[67634], 80.00th=[67634], 90.00th=[73925], 95.00th=[73925], | 99.00th=[73925], 99.50th=[73925], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 8192, max= 8192, per=74.88%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3829KiB/s (3921kB/s)(7168KiB/1872msec) clat (msec): min=83, max=310, avg=209.51, stdev=91.22 lat (msec): min=83, max=310, avg=209.59, stdev=91.22 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 84], 10.00th=[ 84], 20.00th=[ 87], | 30.00th=[ 218], 40.00th=[ 218], 50.00th=[ 220], 60.00th=[ 271], | 70.00th=[ 271], 80.00th=[ 279], 90.00th=[ 313], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 2048, max= 4055, per=17.74%, avg=2717.00, stdev=1158.74, samples=3 iops : min= 2, max= 3, avg= 2.33, stdev= 0.58, samples=3 lat (msec) : 50=16.67%, 100=41.67%, 250=16.67%, 500=25.00% cpu : usr=0.00%, sys=6.79%, ctx=51, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.7MiB/s (11.2MB/s), 1134KiB/s-3862KiB/s (1161kB/s-3955kB/s), io=20.0MiB (20.0MB), run=1806-1872msec WRITE: bw=14.0MiB/s (15.7MB/s), 2759KiB/s-5670KiB/s (2825kB/s-5806kB/s), io=28.0MiB (29.4MB), run=1806-1872msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=3774: Tue Jun 17 13:24:44 2025 read: IOPS=1, BW=7054KiB/s (7223kB/s)(12.0MiB/1742msec) clat (msec): min=435, max=607, avg=537.13, stdev=90.48 lat (msec): min=435, max=607, avg=537.13, stdev=90.49 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 435], 10.00th=[ 435], 20.00th=[ 435], | 30.00th=[ 435], 40.00th=[ 567], 50.00th=[ 567], 60.00th=[ 567], | 70.00th=[ 609], 80.00th=[ 609], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 8175, max= 8192, per=89.47%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.00%, sys=5.17%, ctx=41, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3775: Tue Jun 17 13:24:44 2025 read: IOPS=0, BW=2287KiB/s (2342kB/s)(4096KiB/1791msec) clat (nsec): min=477101k, max=477101k, avg=477100781.00, stdev= 0.00 lat (nsec): min=477104k, max=477104k, avg=477103807.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 477], 10.00th=[ 477], 20.00th=[ 477], | 30.00th=[ 477], 40.00th=[ 477], 50.00th=[ 477], 60.00th=[ 477], | 70.00th=[ 477], 80.00th=[ 477], 90.00th=[ 477], 95.00th=[ 477], | 99.00th=[ 477], 99.50th=[ 477], 99.90th=[ 477], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 8110, max= 8110, per=88.66%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4574KiB/s (4684kB/s)(8192KiB/1791msec) clat (msec): min=403, max=819, avg=611.12, stdev=294.31 lat (msec): min=403, max=819, avg=611.45, stdev=294.39 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 405], 10.00th=[ 405], 20.00th=[ 405], | 30.00th=[ 405], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 818], | 70.00th=[ 818], 80.00th=[ 818], 90.00th=[ 818], 95.00th=[ 818], | 99.00th=[ 818], 99.50th=[ 818], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 8110, max= 8110, per=44.33%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 1000=33.33% cpu : usr=0.00%, sys=6.92%, ctx=40, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3776: Tue Jun 17 13:24:44 2025 write: IOPS=2, BW=9.96MiB/s (10.4MB/s)(12.0MiB/1205msec) clat (msec): min=199, max=499, avg=386.42, stdev=162.94 lat (msec): min=200, max=499, avg=386.78, stdev=162.81 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 201], 10.00th=[ 201], 20.00th=[ 201], | 30.00th=[ 201], 40.00th=[ 460], 50.00th=[ 460], 60.00th=[ 460], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 8159, max= 8175, per=44.64%, avg=8167.00, stdev=11.31, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.08%, sys=7.48%, ctx=28, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3777: Tue Jun 17 13:24:44 2025 write: IOPS=2, BW=8924KiB/s (9138kB/s)(12.0MiB/1377msec) clat (msec): min=220, max=870, avg=449.81, stdev=364.89 lat (msec): min=220, max=870, avg=450.10, stdev=364.89 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 222], 10.00th=[ 222], 20.00th=[ 222], | 30.00th=[ 222], 40.00th=[ 259], 50.00th=[ 259], 60.00th=[ 259], | 70.00th=[ 869], 80.00th=[ 869], 90.00th=[ 869], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 8175, max= 8175, per=44.68%, avg=8175.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.15%, sys=7.70%, ctx=58, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9148KiB/s (9368kB/s), 2287KiB/s-7054KiB/s (2342kB/s-7223kB/s), io=16.0MiB (16.8MB), run=1742-1791msec WRITE: bw=17.9MiB/s (18.7MB/s), 4574KiB/s-9.96MiB/s (4684kB/s-10.4MB/s), io=32.0MiB (33.6MB), run=1205-1791msec rand-rw: (groupid=0, jobs=1): err= 0: pid=3779: Tue Jun 17 13:24:45 2025 read: IOPS=2, BW=8569KiB/s (8775kB/s)(12.0MiB/1434msec) clat (msec): min=288, max=558, avg=430.63, stdev=135.55 lat (msec): min=288, max=558, avg=430.63, stdev=135.55 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 288], 10.00th=[ 288], 20.00th=[ 288], | 30.00th=[ 288], 40.00th=[ 447], 50.00th=[ 447], 60.00th=[ 447], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 8192, max= 8192, per=74.75%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.00%, sys=11.93%, ctx=460, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3780: Tue Jun 17 13:24:45 2025 read: IOPS=0, BW=2740KiB/s (2806kB/s)(4096KiB/1495msec) clat (nsec): min=350597k, max=350597k, avg=350596683.00, stdev= 0.00 lat (nsec): min=350600k, max=350600k, avg=350599697.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 351], 10.00th=[ 351], 20.00th=[ 351], | 30.00th=[ 351], 40.00th=[ 351], 50.00th=[ 351], 60.00th=[ 351], | 70.00th=[ 351], 80.00th=[ 351], 90.00th=[ 351], 95.00th=[ 351], | 99.00th=[ 351], 99.50th=[ 351], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 8175, max= 8175, per=74.60%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5480KiB/s (5611kB/s)(8192KiB/1495msec) clat (msec): min=198, max=849, avg=524.11, stdev=460.36 lat (msec): min=198, max=850, avg=524.51, stdev=460.36 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 199], 10.00th=[ 199], 20.00th=[ 199], | 30.00th=[ 199], 40.00th=[ 199], 50.00th=[ 199], 60.00th=[ 852], | 70.00th=[ 852], 80.00th=[ 852], 90.00th=[ 852], 95.00th=[ 852], | 99.00th=[ 852], 99.50th=[ 852], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 8192, max= 8192, per=56.43%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=14.19%, ctx=140, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3781: Tue Jun 17 13:24:45 2025 write: IOPS=1, BW=5444KiB/s (5575kB/s)(12.0MiB/2257msec) clat (msec): min=239, max=1030, avg=722.66, stdev=423.59 lat (msec): min=240, max=1033, avg=724.00, stdev=424.39 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 241], 10.00th=[ 241], 20.00th=[ 241], | 30.00th=[ 241], 40.00th=[ 902], 50.00th=[ 902], 60.00th=[ 902], | 70.00th=[ 1028], 80.00th=[ 1028], 90.00th=[ 1028], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 7460, max= 8175, per=53.85%, avg=7817.50, stdev=505.58, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=13.34%, ctx=94, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3782: Tue Jun 17 13:24:45 2025 write: IOPS=2, BW=8214KiB/s (8411kB/s)(12.0MiB/1496msec) clat (msec): min=304, max=585, avg=443.63, stdev=140.40 lat (msec): min=305, max=585, avg=444.03, stdev=140.35 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 305], 10.00th=[ 305], 20.00th=[ 305], | 30.00th=[ 305], 40.00th=[ 443], 50.00th=[ 443], 60.00th=[ 443], | 70.00th=[ 584], 80.00th=[ 584], 90.00th=[ 584], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 8110, max= 8110, per=55.86%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.00%, sys=20.60%, ctx=68, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.7MiB/s (11.2MB/s), 2740KiB/s-8569KiB/s (2806kB/s-8775kB/s), io=16.0MiB (16.8MB), run=1434-1495msec WRITE: bw=14.2MiB/s (14.9MB/s), 5444KiB/s-8214KiB/s (5575kB/s-8411kB/s), io=32.0MiB (33.6MB), run=1495-2257msec