== sanity test 398b: DIO and buffer IO race ============== 18:30:47 (1741905047) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.24396 s, 40.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4663: Thu Mar 13 18:31:03 2025 read: IOPS=112, BW=448KiB/s (459kB/s)(5992KiB/13367msec) clat (usec): min=51, max=173447, avg=4070.29, stdev=5796.61 lat (usec): min=52, max=173447, avg=4071.36, stdev=5796.58 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 74], 10.00th=[ 90], 20.00th=[ 1942], | 30.00th=[ 2442], 40.00th=[ 2802], 50.00th=[ 3228], 60.00th=[ 3818], | 70.00th=[ 4883], 80.00th=[ 6128], 90.00th=[ 8029], 95.00th=[ 9241], | 99.00th=[ 14746], 99.50th=[ 22414], 99.90th=[ 82314], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 232, max= 680, per=24.86%, avg=442.42, stdev=86.47, samples=26 iops : min= 58, max= 170, avg=110.58, stdev=21.63, samples=26 write: IOPS=117, BW=471KiB/s (482kB/s)(6296KiB/13367msec) clat (usec): min=469, max=45366, avg=4541.05, stdev=4523.17 lat (usec): min=470, max=45367, avg=4541.97, stdev=4523.39 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 537], 10.00th=[ 578], 20.00th=[ 701], | 30.00th=[ 930], 40.00th=[ 2114], 50.00th=[ 3490], 60.00th=[ 4490], | 70.00th=[ 5866], 80.00th=[ 7570], 90.00th=[10421], 95.00th=[13042], | 99.00th=[17957], 99.50th=[20841], 99.90th=[44827], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 296, max= 624, per=24.82%, avg=462.73, stdev=82.26, samples=26 iops : min= 74, max= 156, avg=115.65, stdev=20.57, samples=26 lat (usec) : 100=5.63%, 250=3.09%, 500=0.75%, 750=10.97%, 1000=4.65% lat (msec) : 2=5.21%, 4=28.45%, 10=33.43%, 20=7.23%, 50=0.52% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.18%, sys=22.89%, ctx=5518, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4664: Thu Mar 13 18:31:03 2025 read: IOPS=110, BW=443KiB/s (453kB/s)(5940KiB/13418msec) clat (usec): min=50, max=44198, avg=3837.29, stdev=3531.77 lat (usec): min=51, max=44199, avg=3838.16, stdev=3531.69 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 73], 10.00th=[ 87], 20.00th=[ 469], | 30.00th=[ 2343], 40.00th=[ 2737], 50.00th=[ 3130], 60.00th=[ 3621], | 70.00th=[ 4555], 80.00th=[ 5932], 90.00th=[ 7963], 95.00th=[ 9634], | 99.00th=[14222], 99.50th=[21890], 99.90th=[33162], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 208, max= 664, per=24.77%, avg=440.88, stdev=94.56, samples=26 iops : min= 52, max= 166, avg=110.19, stdev=23.66, samples=26 write: IOPS=118, BW=473KiB/s (484kB/s)(6348KiB/13418msec) clat (usec): min=468, max=187463, avg=4789.22, stdev=6673.36 lat (usec): min=469, max=187463, avg=4790.14, stdev=6673.36 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 545], 10.00th=[ 586], 20.00th=[ 693], | 30.00th=[ 938], 40.00th=[ 2008], 50.00th=[ 3654], 60.00th=[ 4686], | 70.00th=[ 5866], 80.00th=[ 7635], 90.00th=[ 10683], 95.00th=[ 13829], | 99.00th=[ 20317], 99.50th=[ 24511], 99.90th=[ 62653], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 264, max= 704, per=24.92%, avg=464.58, stdev=95.19, samples=26 iops : min= 66, max= 176, avg=116.12, stdev=23.80, samples=26 lat (usec) : 100=5.92%, 250=3.48%, 500=0.52%, 750=11.56%, 1000=4.43% lat (msec) : 2=5.60%, 4=26.86%, 10=33.46%, 20=7.26%, 50=0.85% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.17%, sys=22.78%, ctx=5597, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4665: Thu Mar 13 18:31:03 2025 read: IOPS=111, BW=447KiB/s (458kB/s)(6028KiB/13483msec) clat (usec): min=52, max=88495, avg=3882.83, stdev=3970.04 lat (usec): min=52, max=88496, avg=3883.59, stdev=3970.02 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 72], 10.00th=[ 85], 20.00th=[ 206], | 30.00th=[ 2376], 40.00th=[ 2802], 50.00th=[ 3195], 60.00th=[ 3785], | 70.00th=[ 4686], 80.00th=[ 6063], 90.00th=[ 7832], 95.00th=[ 9241], | 99.00th=[14746], 99.50th=[19792], 99.90th=[40109], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 120, max= 656, per=24.81%, avg=441.54, stdev=104.72, samples=26 iops : min= 30, max= 164, avg=110.38, stdev=26.18, samples=26 write: IOPS=116, BW=464KiB/s (475kB/s)(6260KiB/13483msec) clat (usec): min=476, max=212536, avg=4798.24, stdev=6964.49 lat (usec): min=477, max=212537, avg=4799.34, stdev=6964.54 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 545], 10.00th=[ 586], 20.00th=[ 725], | 30.00th=[ 1020], 40.00th=[ 2245], 50.00th=[ 3752], 60.00th=[ 4883], | 70.00th=[ 6063], 80.00th=[ 7701], 90.00th=[ 10552], 95.00th=[ 13304], | 99.00th=[ 20841], 99.50th=[ 26084], 99.90th=[ 43254], 99.95th=[212861], | 99.99th=[212861] bw ( KiB/s): min= 176, max= 680, per=24.48%, avg=456.31, stdev=96.12, samples=26 iops : min= 44, max= 170, avg=114.08, stdev=24.03, samples=26 lat (usec) : 100=6.71%, 250=3.16%, 500=0.68%, 750=10.55%, 1000=4.30% lat (msec) : 2=5.57%, 4=26.79%, 10=34.41%, 20=7.10%, 50=0.68% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.20%, sys=23.04%, ctx=5433, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4666: Thu Mar 13 18:31:03 2025 read: IOPS=113, BW=452KiB/s (463kB/s)(6052KiB/13388msec) clat (usec): min=51, max=41899, avg=3827.43, stdev=3284.00 lat (usec): min=51, max=41900, avg=3828.14, stdev=3284.00 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 75], 10.00th=[ 93], 20.00th=[ 668], | 30.00th=[ 2376], 40.00th=[ 2769], 50.00th=[ 3195], 60.00th=[ 3785], | 70.00th=[ 4752], 80.00th=[ 5997], 90.00th=[ 7701], 95.00th=[ 9503], | 99.00th=[12780], 99.50th=[16188], 99.90th=[34866], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 224, max= 600, per=25.25%, avg=449.46, stdev=77.56, samples=26 iops : min= 56, max= 150, avg=112.31, stdev=19.41, samples=26 write: IOPS=116, BW=466KiB/s (477kB/s)(6236KiB/13388msec) clat (usec): min=429, max=191730, avg=4794.53, stdev=6532.39 lat (usec): min=430, max=191731, avg=4795.30, stdev=6532.40 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 553], 10.00th=[ 586], 20.00th=[ 725], | 30.00th=[ 1057], 40.00th=[ 2507], 50.00th=[ 3818], 60.00th=[ 4752], | 70.00th=[ 5997], 80.00th=[ 7701], 90.00th=[ 10683], 95.00th=[ 13042], | 99.00th=[ 19006], 99.50th=[ 25035], 99.90th=[ 40633], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 248, max= 624, per=24.84%, avg=463.00, stdev=77.52, samples=26 iops : min= 62, max= 156, avg=115.69, stdev=19.41, samples=26 lat (usec) : 100=5.86%, 250=3.81%, 500=0.49%, 750=10.38%, 1000=4.04% lat (msec) : 2=5.79%, 4=27.25%, 10=34.28%, 20=7.49%, 50=0.59% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=22.73%, ctx=5527, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1781KiB/s (1824kB/s), 443KiB/s-452KiB/s (453kB/s-463kB/s), io=23.4MiB (24.6MB), run=13367-13483msec WRITE: bw=1865KiB/s (1909kB/s), 464KiB/s-473KiB/s (475kB/s-484kB/s), io=24.6MiB (25.7MB), run=13367-13483msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4667: Thu Mar 13 18:31:28 2025 read: IOPS=39, BW=159KiB/s (163kB/s)(5992KiB/37722msec) clat (usec): min=1630, max=176122, avg=4261.58, stdev=5625.05 lat (usec): min=1631, max=176123, avg=4262.40, stdev=5625.07 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1909], 10.00th=[ 1991], 20.00th=[ 2114], | 30.00th=[ 2245], 40.00th=[ 2409], 50.00th=[ 2638], 60.00th=[ 3195], | 70.00th=[ 4424], 80.00th=[ 5932], 90.00th=[ 7701], 95.00th=[ 10814], | 99.00th=[ 17957], 99.50th=[ 21365], 99.90th=[ 47449], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 56, max= 304, per=25.31%, avg=158.47, stdev=63.85, samples=75 iops : min= 14, max= 76, avg=39.51, stdev=15.99, samples=75 write: IOPS=41, BW=167KiB/s (171kB/s)(6296KiB/37722msec) clat (usec): min=9040, max=92013, avg=19844.75, stdev=7731.34 lat (usec): min=9041, max=92014, avg=19845.82, stdev=7731.30 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11994], 10.00th=[12649], 20.00th=[13829], | 30.00th=[15270], 40.00th=[16581], 50.00th=[18220], 60.00th=[19268], | 70.00th=[21103], 80.00th=[23725], 90.00th=[30278], 95.00th=[36439], | 99.00th=[46924], 99.50th=[51643], 99.90th=[68682], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 48, max= 248, per=25.39%, avg=166.57, stdev=50.23, samples=75 iops : min= 12, max= 62, avg=41.53, stdev=12.58, samples=75 lat (msec) : 2=5.21%, 4=27.67%, 10=13.15%, 20=35.03%, 50=18.55% lat (msec) : 100=0.36%, 250=0.03% cpu : usr=0.13%, sys=5.68%, ctx=3863, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4668: Thu Mar 13 18:31:28 2025 read: IOPS=38, BW=155KiB/s (159kB/s)(5940KiB/38297msec) clat (usec): min=1577, max=193303, avg=4382.64, stdev=6031.52 lat (usec): min=1578, max=193306, avg=4383.44, stdev=6031.57 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1909], 10.00th=[ 2008], 20.00th=[ 2147], | 30.00th=[ 2278], 40.00th=[ 2409], 50.00th=[ 2704], 60.00th=[ 3392], | 70.00th=[ 4686], 80.00th=[ 6194], 90.00th=[ 7701], 95.00th=[ 10945], | 99.00th=[ 17957], 99.50th=[ 21365], 99.90th=[ 44827], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 16, max= 336, per=24.72%, avg=154.76, stdev=69.80, samples=76 iops : min= 4, max= 84, avg=38.63, stdev=17.47, samples=76 write: IOPS=41, BW=166KiB/s (170kB/s)(6348KiB/38297msec) clat (usec): min=9206, max=78638, avg=19965.92, stdev=7632.37 lat (usec): min=9207, max=78639, avg=19966.94, stdev=7632.42 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11731], 10.00th=[12518], 20.00th=[13960], | 30.00th=[15270], 40.00th=[16909], 50.00th=[18220], 60.00th=[19792], | 70.00th=[21365], 80.00th=[23725], 90.00th=[30802], 95.00th=[35390], | 99.00th=[45876], 99.50th=[51643], 99.90th=[58983], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 64, max= 232, per=25.12%, avg=164.76, stdev=48.57, samples=76 iops : min= 16, max= 58, avg=41.13, stdev=12.20, samples=76 lat (msec) : 2=4.69%, 4=26.60%, 10=14.19%, 20=34.21%, 50=19.92% lat (msec) : 100=0.36%, 250=0.03% cpu : usr=0.14%, sys=5.67%, ctx=3966, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4669: Thu Mar 13 18:31:28 2025 read: IOPS=39, BW=159KiB/s (163kB/s)(6028KiB/37916msec) clat (usec): min=1585, max=43857, avg=4382.07, stdev=3611.28 lat (usec): min=1586, max=43857, avg=4382.88, stdev=3611.28 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1909], 10.00th=[ 1991], 20.00th=[ 2114], | 30.00th=[ 2278], 40.00th=[ 2442], 50.00th=[ 2802], 60.00th=[ 3752], | 70.00th=[ 4883], 80.00th=[ 6194], 90.00th=[ 8225], 95.00th=[10814], | 99.00th=[17695], 99.50th=[22414], 99.90th=[43254], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 40, max= 280, per=25.37%, avg=158.80, stdev=54.05, samples=75 iops : min= 10, max= 70, avg=39.60, stdev=13.62, samples=75 write: IOPS=41, BW=165KiB/s (169kB/s)(6260KiB/37916msec) clat (msec): min=9, max=209, avg=19.94, stdev= 8.91 lat (msec): min=9, max=209, avg=19.94, stdev= 8.91 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 20], | 70.00th=[ 22], 80.00th=[ 24], 90.00th=[ 31], 95.00th=[ 36], | 99.00th=[ 46], 99.50th=[ 51], 99.90th=[ 92], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 56, max= 232, per=25.09%, avg=164.56, stdev=50.28, samples=75 iops : min= 14, max= 58, avg=41.04, stdev=12.66, samples=75 lat (msec) : 2=5.18%, 4=25.39%, 10=15.36%, 20=34.18%, 50=19.63% lat (msec) : 100=0.23%, 250=0.03% cpu : usr=0.08%, sys=5.77%, ctx=3906, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4670: Thu Mar 13 18:31:28 2025 read: IOPS=39, BW=160KiB/s (164kB/s)(6052KiB/37898msec) clat (usec): min=1578, max=63641, avg=4186.06, stdev=3773.46 lat (usec): min=1579, max=63642, avg=4186.85, stdev=3773.46 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1893], 10.00th=[ 1991], 20.00th=[ 2114], | 30.00th=[ 2245], 40.00th=[ 2442], 50.00th=[ 2671], 60.00th=[ 3326], | 70.00th=[ 4490], 80.00th=[ 5997], 90.00th=[ 7308], 95.00th=[10290], | 99.00th=[17695], 99.50th=[22414], 99.90th=[43254], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 31, max= 320, per=25.39%, avg=158.95, stdev=63.88, samples=75 iops : min= 7, max= 80, avg=39.59, stdev=15.97, samples=75 write: IOPS=41, BW=165KiB/s (168kB/s)(6236KiB/37898msec) clat (msec): min=9, max=221, avg=20.18, stdev= 9.03 lat (msec): min=9, max=221, avg=20.18, stdev= 9.03 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 21], | 70.00th=[ 22], 80.00th=[ 25], 90.00th=[ 32], 95.00th=[ 36], | 99.00th=[ 45], 99.50th=[ 47], 99.90th=[ 69], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 56, max= 232, per=25.03%, avg=164.17, stdev=49.03, samples=75 iops : min= 14, max= 58, avg=40.89, stdev=12.23, samples=75 lat (msec) : 2=5.27%, 4=27.51%, 10=13.96%, 20=32.65%, 50=20.44% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.11%, sys=5.71%, ctx=3921, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=627KiB/s (642kB/s), 155KiB/s-160KiB/s (159kB/s-164kB/s), io=23.4MiB (24.6MB), run=37722-38297msec WRITE: bw=656KiB/s (672kB/s), 165KiB/s-167KiB/s (168kB/s-171kB/s), io=24.6MiB (25.7MB), run=37722-38297msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4712: Thu Mar 13 18:31:32 2025 read: IOPS=139, BW=2232KiB/s (2286kB/s)(5920KiB/2652msec) clat (usec): min=54, max=28355, avg=3419.32, stdev=3443.15 lat (usec): min=54, max=28356, avg=3420.21, stdev=3442.94 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 70], 10.00th=[ 72], 20.00th=[ 95], | 30.00th=[ 217], 40.00th=[ 2573], 50.00th=[ 3064], 60.00th=[ 3654], | 70.00th=[ 4490], 80.00th=[ 5538], 90.00th=[ 7046], 95.00th=[ 8356], | 99.00th=[17433], 99.50th=[23725], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 1472, max= 2688, per=24.82%, avg=2182.40, stdev=513.70, samples=5 iops : min= 92, max= 168, avg=136.40, stdev=32.11, samples=5 write: IOPS=150, BW=2401KiB/s (2459kB/s)(6368KiB/2652msec) clat (usec): min=620, max=16637, avg=3413.79, stdev=3214.86 lat (usec): min=621, max=16638, avg=3414.77, stdev=3214.87 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 889], | 30.00th=[ 1090], 40.00th=[ 1303], 50.00th=[ 1811], 60.00th=[ 2999], | 70.00th=[ 4293], 80.00th=[ 6063], 90.00th=[ 8029], 95.00th=[10159], | 99.00th=[13960], 99.50th=[14353], 99.90th=[16581], 99.95th=[16581], | 99.99th=[16581] bw ( KiB/s): min= 1760, max= 3104, per=25.46%, avg=2355.20, stdev=546.63, samples=5 iops : min= 110, max= 194, avg=147.20, stdev=34.16, samples=5 lat (usec) : 100=10.16%, 250=4.43%, 500=0.26%, 750=3.26%, 1000=10.29% lat (msec) : 2=13.67%, 4=24.09%, 10=29.95%, 20=3.52%, 50=0.39% cpu : usr=0.23%, sys=28.63%, ctx=1454, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4713: Thu Mar 13 18:31:32 2025 read: IOPS=131, BW=2109KiB/s (2159kB/s)(5744KiB/2724msec) clat (usec): min=55, max=17173, avg=2967.99, stdev=2561.38 lat (usec): min=55, max=17174, avg=2968.88, stdev=2561.42 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 71], 10.00th=[ 74], 20.00th=[ 108], | 30.00th=[ 186], 40.00th=[ 2507], 50.00th=[ 2999], 60.00th=[ 3359], | 70.00th=[ 3949], 80.00th=[ 4752], 90.00th=[ 6194], 95.00th=[ 7177], | 99.00th=[10552], 99.50th=[15008], 99.90th=[17171], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 1408, max= 2464, per=23.80%, avg=2092.80, stdev=435.01, samples=5 iops : min= 88, max= 154, avg=130.80, stdev=27.19, samples=5 write: IOPS=150, BW=2402KiB/s (2460kB/s)(6544KiB/2724msec) clat (usec): min=641, max=31407, avg=3749.42, stdev=4093.13 lat (usec): min=642, max=31407, avg=3755.08, stdev=4094.63 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 889], | 30.00th=[ 1045], 40.00th=[ 1319], 50.00th=[ 1795], 60.00th=[ 3326], | 70.00th=[ 4490], 80.00th=[ 6325], 90.00th=[ 8586], 95.00th=[10421], | 99.00th=[16450], 99.50th=[27132], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 1568, max= 2912, per=25.39%, avg=2348.80, stdev=486.67, samples=5 iops : min= 98, max= 182, avg=146.80, stdev=30.42, samples=5 lat (usec) : 100=8.59%, 250=5.60%, 500=0.26%, 750=3.39%, 1000=11.20% lat (msec) : 2=13.28%, 4=25.91%, 10=27.60%, 20=3.65%, 50=0.52% cpu : usr=0.22%, sys=27.14%, ctx=1471, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4714: Thu Mar 13 18:31:32 2025 read: IOPS=145, BW=2320KiB/s (2376kB/s)(6256KiB/2696msec) clat (usec): min=55, max=20635, avg=3302.08, stdev=2939.45 lat (usec): min=55, max=20636, avg=3302.68, stdev=2939.46 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 69], 10.00th=[ 72], 20.00th=[ 103], | 30.00th=[ 143], 40.00th=[ 2638], 50.00th=[ 3228], 60.00th=[ 3818], | 70.00th=[ 4555], 80.00th=[ 5538], 90.00th=[ 6849], 95.00th=[ 8586], | 99.00th=[10945], 99.50th=[12911], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 1660, max= 2810, per=27.06%, avg=2378.80, stdev=440.43, samples=5 iops : min= 103, max= 175, avg=148.40, stdev=27.68, samples=5 write: IOPS=139, BW=2237KiB/s (2291kB/s)(6032KiB/2696msec) clat (usec): min=608, max=31473, avg=3457.01, stdev=3878.68 lat (usec): min=609, max=31475, avg=3459.08, stdev=3879.37 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 709], 10.00th=[ 766], 20.00th=[ 881], | 30.00th=[ 1029], 40.00th=[ 1270], 50.00th=[ 1614], 60.00th=[ 2933], | 70.00th=[ 4293], 80.00th=[ 5473], 90.00th=[ 8029], 95.00th=[ 9634], | 99.00th=[16450], 99.50th=[30540], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 1920, max= 2592, per=24.12%, avg=2231.80, stdev=300.28, samples=5 iops : min= 120, max= 162, avg=139.20, stdev=18.78, samples=5 lat (usec) : 100=9.90%, 250=6.25%, 500=0.65%, 750=4.43%, 1000=9.64% lat (msec) : 2=12.37%, 4=21.22%, 10=32.42%, 20=2.60%, 50=0.52% cpu : usr=0.26%, sys=27.12%, ctx=1479, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4715: Thu Mar 13 18:31:32 2025 read: IOPS=145, BW=2329KiB/s (2385kB/s)(6032KiB/2590msec) clat (usec): min=59, max=18866, avg=3389.08, stdev=2887.73 lat (usec): min=59, max=18866, avg=3389.68, stdev=2887.74 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 70], 10.00th=[ 75], 20.00th=[ 110], | 30.00th=[ 2278], 40.00th=[ 2704], 50.00th=[ 3130], 60.00th=[ 3654], | 70.00th=[ 4424], 80.00th=[ 5342], 90.00th=[ 6783], 95.00th=[ 8586], | 99.00th=[13566], 99.50th=[15401], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 2011, max= 2624, per=26.11%, avg=2295.60, stdev=266.99, samples=5 iops : min= 125, max= 164, avg=143.20, stdev=16.93, samples=5 write: IOPS=150, BW=2415KiB/s (2473kB/s)(6256KiB/2590msec) clat (usec): min=628, max=25270, avg=3304.52, stdev=3116.03 lat (usec): min=629, max=25271, avg=3305.44, stdev=3116.08 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 873], | 30.00th=[ 1090], 40.00th=[ 1319], 50.00th=[ 1876], 60.00th=[ 3228], | 70.00th=[ 4047], 80.00th=[ 5604], 90.00th=[ 7373], 95.00th=[ 9241], | 99.00th=[13173], 99.50th=[15664], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 1824, max= 2810, per=25.99%, avg=2404.20, stdev=393.66, samples=5 iops : min= 114, max= 175, avg=150.00, stdev=24.51, samples=5 lat (usec) : 100=8.59%, 250=4.56%, 750=2.99%, 1000=11.33% lat (msec) : 2=11.85%, 4=27.86%, 10=29.17%, 20=3.52%, 50=0.13% cpu : usr=0.27%, sys=29.08%, ctx=1532, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8793KiB/s (9004kB/s), 2109KiB/s-2329KiB/s (2159kB/s-2385kB/s), io=23.4MiB (24.5MB), run=2590-2724msec WRITE: bw=9251KiB/s (9473kB/s), 2237KiB/s-2415KiB/s (2291kB/s-2473kB/s), io=24.6MiB (25.8MB), run=2590-2724msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4716: Thu Mar 13 18:31:38 2025 read: IOPS=42, BW=684KiB/s (700kB/s)(5920KiB/8655msec) clat (usec): min=1762, max=36146, avg=3664.38, stdev=3467.34 lat (usec): min=1763, max=36147, avg=3665.13, stdev=3467.34 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1942], 10.00th=[ 1991], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2311], 50.00th=[ 2409], 60.00th=[ 2606], | 70.00th=[ 2933], 80.00th=[ 4080], 90.00th=[ 6718], 95.00th=[10683], | 99.00th=[19530], 99.50th=[21890], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 320, max= 1120, per=24.76%, avg=673.82, stdev=228.45, samples=17 iops : min= 20, max= 70, avg=42.06, stdev=14.35, samples=17 write: IOPS=45, BW=736KiB/s (753kB/s)(6368KiB/8655msec) clat (usec): min=10158, max=65766, avg=18072.37, stdev=6403.98 lat (usec): min=10160, max=65767, avg=18073.82, stdev=6404.01 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11338], 10.00th=[11731], 20.00th=[12911], | 30.00th=[14353], 40.00th=[15664], 50.00th=[16909], 60.00th=[17957], | 70.00th=[19268], 80.00th=[21365], 90.00th=[25822], 95.00th=[29230], | 99.00th=[40633], 99.50th=[46400], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 383, max= 992, per=25.51%, avg=730.29, stdev=212.14, samples=17 iops : min= 23, max= 62, avg=45.59, stdev=13.36, samples=17 lat (msec) : 2=4.95%, 4=33.33%, 10=7.42%, 20=40.36%, 50=13.80% lat (msec) : 100=0.13% cpu : usr=0.06%, sys=6.48%, ctx=893, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4717: Thu Mar 13 18:31:38 2025 read: IOPS=40, BW=653KiB/s (668kB/s)(5744KiB/8801msec) clat (usec): min=1723, max=29445, avg=3828.48, stdev=3564.67 lat (usec): min=1723, max=29446, avg=3829.22, stdev=3564.67 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1958], 10.00th=[ 2008], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2638], | 70.00th=[ 3097], 80.00th=[ 4883], 90.00th=[ 6980], 95.00th=[10290], | 99.00th=[20841], 99.50th=[27919], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 287, max= 1088, per=23.86%, avg=649.24, stdev=242.85, samples=17 iops : min= 17, max= 68, avg=40.47, stdev=15.23, samples=17 write: IOPS=46, BW=744KiB/s (761kB/s)(6544KiB/8801msec) clat (usec): min=9601, max=40913, avg=18098.15, stdev=5957.58 lat (usec): min=9602, max=40915, avg=18099.63, stdev=5957.64 clat percentiles (usec): | 1.00th=[10159], 5.00th=[11338], 10.00th=[11863], 20.00th=[13042], | 30.00th=[13960], 40.00th=[15664], 50.00th=[17171], 60.00th=[18220], | 70.00th=[19792], 80.00th=[22152], 90.00th=[25822], 95.00th=[30016], | 99.00th=[37487], 99.50th=[40633], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 384, max= 992, per=25.70%, avg=735.76, stdev=204.59, samples=17 iops : min= 24, max= 62, avg=45.88, stdev=12.79, samples=17 lat (msec) : 2=3.78%, 4=32.42%, 10=8.33%, 20=39.58%, 50=15.89% cpu : usr=0.06%, sys=6.24%, ctx=925, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4718: Thu Mar 13 18:31:38 2025 read: IOPS=45, BW=732KiB/s (750kB/s)(6256KiB/8541msec) clat (usec): min=1802, max=36123, avg=3460.53, stdev=3098.81 lat (usec): min=1803, max=36124, avg=3461.31, stdev=3098.80 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 1991], 10.00th=[ 2073], 20.00th=[ 2147], | 30.00th=[ 2212], 40.00th=[ 2343], 50.00th=[ 2442], 60.00th=[ 2606], | 70.00th=[ 2868], 80.00th=[ 3425], 90.00th=[ 5997], 95.00th=[ 9372], | 99.00th=[15926], 99.50th=[20579], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 320, max= 1440, per=27.04%, avg=735.82, stdev=342.27, samples=17 iops : min= 20, max= 90, avg=45.88, stdev=21.44, samples=17 write: IOPS=44, BW=706KiB/s (723kB/s)(6032KiB/8541msec) clat (usec): min=9599, max=56716, avg=18784.01, stdev=7294.76 lat (usec): min=9601, max=56717, avg=18785.39, stdev=7294.95 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11076], 10.00th=[11731], 20.00th=[13173], | 30.00th=[14615], 40.00th=[15926], 50.00th=[17433], 60.00th=[18482], | 70.00th=[19792], 80.00th=[21890], 90.00th=[27657], 95.00th=[33424], | 99.00th=[45876], 99.50th=[50070], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 383, max= 960, per=24.58%, avg=703.82, stdev=202.78, samples=17 iops : min= 23, max= 60, avg=43.88, stdev=12.76, samples=17 lat (msec) : 2=2.86%, 4=40.10%, 10=5.73%, 20=37.37%, 50=13.67% lat (msec) : 100=0.26% cpu : usr=0.07%, sys=6.49%, ctx=894, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4719: Thu Mar 13 18:31:38 2025 read: IOPS=43, BW=698KiB/s (715kB/s)(6032KiB/8636msec) clat (usec): min=1738, max=16606, avg=3533.96, stdev=2574.75 lat (usec): min=1739, max=16607, avg=3534.71, stdev=2574.75 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 1958], 10.00th=[ 2040], 20.00th=[ 2147], | 30.00th=[ 2245], 40.00th=[ 2376], 50.00th=[ 2507], 60.00th=[ 2737], | 70.00th=[ 3032], 80.00th=[ 4015], 90.00th=[ 6783], 95.00th=[ 9765], | 99.00th=[14353], 99.50th=[16450], 99.90th=[16581], 99.95th=[16581], | 99.99th=[16581] bw ( KiB/s): min= 288, max= 1024, per=25.53%, avg=694.59, stdev=229.30, samples=17 iops : min= 18, max= 64, avg=43.41, stdev=14.33, samples=17 write: IOPS=45, BW=724KiB/s (742kB/s)(6256KiB/8636msec) clat (usec): min=9183, max=46597, avg=18421.05, stdev=6129.19 lat (usec): min=9184, max=46598, avg=18422.57, stdev=6129.25 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11207], 10.00th=[11994], 20.00th=[13566], | 30.00th=[14877], 40.00th=[16057], 50.00th=[17433], 60.00th=[18744], | 70.00th=[19792], 80.00th=[21627], 90.00th=[26608], 95.00th=[31589], | 99.00th=[40109], 99.50th=[45876], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 416, max= 928, per=25.25%, avg=722.82, stdev=188.32, samples=17 iops : min= 26, max= 58, avg=45.18, stdev=11.77, samples=17 lat (msec) : 2=2.86%, 4=36.20%, 10=8.07%, 20=38.41%, 50=14.45% cpu : usr=0.15%, sys=6.36%, ctx=901, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2722KiB/s (2787kB/s), 653KiB/s-732KiB/s (668kB/s-750kB/s), io=23.4MiB (24.5MB), run=8541-8801msec WRITE: bw=2863KiB/s (2932kB/s), 706KiB/s-744KiB/s (723kB/s-761kB/s), io=24.6MiB (25.8MB), run=8541-8801msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4738: Thu Mar 13 18:31:39 2025 read: IOPS=17, BW=17.2MiB/s (17.0MB/s)(7168KiB/408msec) clat (usec): min=478, max=62236, avg=26870.72, stdev=22674.37 lat (usec): min=479, max=62237, avg=26871.78, stdev=22674.30 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 478], 10.00th=[ 478], 20.00th=[ 693], | 30.00th=[16450], 40.00th=[16450], 50.00th=[28705], 60.00th=[37487], | 70.00th=[37487], 80.00th=[41681], 90.00th=[62129], 95.00th=[62129], | 99.00th=[62129], 99.50th=[62129], 99.90th=[62129], 99.95th=[62129], | 99.99th=[62129] write: IOPS=12, BW=12.3MiB/s (12.8MB/s)(5120KiB/408msec) clat (usec): min=27135, max=51700, avg=39565.00, stdev=10634.35 lat (usec): min=27214, max=51769, avg=39606.25, stdev=10631.93 clat percentiles (usec): | 1.00th=[27132], 5.00th=[27132], 10.00th=[27132], 20.00th=[27132], | 30.00th=[32900], 40.00th=[32900], 50.00th=[36439], 60.00th=[36439], | 70.00th=[49546], 80.00th=[49546], 90.00th=[51643], 95.00th=[51643], | 99.00th=[51643], 99.50th=[51643], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 20=8.33%, 50=58.33%, 100=16.67% cpu : usr=0.00%, sys=31.94%, ctx=262, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4739: Thu Mar 13 18:31:39 2025 read: IOPS=3, BW=3580KiB/s (3666kB/s)(2048KiB/572msec) clat (usec): min=29778, max=30500, avg=30139.51, stdev=511.15 lat (usec): min=29778, max=30502, avg=30140.67, stdev=511.86 clat percentiles (usec): | 1.00th=[29754], 5.00th=[29754], 10.00th=[29754], 20.00th=[29754], | 30.00th=[29754], 40.00th=[29754], 50.00th=[29754], 60.00th=[30540], | 70.00th=[30540], 80.00th=[30540], 90.00th=[30540], 95.00th=[30540], | 99.00th=[30540], 99.50th=[30540], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] write: IOPS=17, BW=17.5MiB/s (18.3MB/s)(10.0MiB/572msec) clat (usec): min=25531, max=84117, avg=49832.88, stdev=20573.47 lat (usec): min=25590, max=84184, avg=49879.93, stdev=20564.81 clat percentiles (usec): | 1.00th=[25560], 5.00th=[25560], 10.00th=[25560], 20.00th=[33424], | 30.00th=[35390], 40.00th=[35914], 50.00th=[36439], 60.00th=[43779], | 70.00th=[65799], 80.00th=[65799], 90.00th=[78119], 95.00th=[84411], | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] lat (msec) : 50=66.67%, 100=33.33% cpu : usr=0.00%, sys=40.63%, ctx=146, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4740: Thu Mar 13 18:31:39 2025 read: IOPS=17, BW=17.4MiB/s (18.2MB/s)(6144KiB/345msec) clat (usec): min=350, max=36591, avg=15524.88, stdev=16865.16 lat (usec): min=352, max=36592, avg=15525.74, stdev=16865.03 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 351], 10.00th=[ 351], 20.00th=[ 404], | 30.00th=[ 404], 40.00th=[ 510], 50.00th=[ 510], 60.00th=[27395], | 70.00th=[27919], 80.00th=[27919], 90.00th=[36439], 95.00th=[36439], | 99.00th=[36439], 99.50th=[36439], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] write: IOPS=17, BW=17.4MiB/s (18.2MB/s)(6144KiB/345msec) clat (usec): min=24022, max=58255, avg=41036.39, stdev=14968.73 lat (usec): min=24040, max=58323, avg=41111.97, stdev=15012.02 clat percentiles (usec): | 1.00th=[23987], 5.00th=[23987], 10.00th=[23987], 20.00th=[29230], | 30.00th=[29230], 40.00th=[29492], 50.00th=[29492], 60.00th=[51119], | 70.00th=[54264], 80.00th=[54264], 90.00th=[58459], 95.00th=[58459], | 99.00th=[58459], 99.50th=[58459], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 50=50.00%, 100=25.00% cpu : usr=0.00%, sys=37.21%, ctx=517, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4741: Thu Mar 13 18:31:39 2025 read: IOPS=11, BW=11.4MiB/s (11.9MB/s)(5120KiB/440msec) clat (usec): min=408, max=34986, avg=23252.34, stdev=14320.62 lat (usec): min=409, max=34986, avg=23253.21, stdev=14320.79 clat percentiles (usec): | 1.00th=[ 408], 5.00th=[ 408], 10.00th=[ 408], 20.00th=[ 408], | 30.00th=[19006], 40.00th=[19006], 50.00th=[27132], 60.00th=[27132], | 70.00th=[34341], 80.00th=[34341], 90.00th=[34866], 95.00th=[34866], | 99.00th=[34866], 99.50th=[34866], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] write: IOPS=15, BW=15.9MiB/s (16.7MB/s)(7168KiB/440msec) clat (usec): min=23854, max=86450, avg=41374.34, stdev=22527.62 lat (usec): min=23946, max=86512, avg=41441.14, stdev=22519.15 clat percentiles (usec): | 1.00th=[23725], 5.00th=[23725], 10.00th=[23725], 20.00th=[23987], | 30.00th=[26870], 40.00th=[26870], 50.00th=[34866], 60.00th=[39584], | 70.00th=[39584], 80.00th=[53740], 90.00th=[86508], 95.00th=[86508], | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] lat (usec) : 500=8.33% lat (msec) : 20=8.33%, 50=66.67%, 100=16.67% cpu : usr=0.00%, sys=33.18%, ctx=306, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=34.0MiB/s (36.7MB/s), 3580KiB/s-17.4MiB/s (3666kB/s-18.2MB/s), io=20.0MiB (20.0MB), run=345-572msec WRITE: bw=48.0MiB/s (51.3MB/s), 12.3MiB/s-17.5MiB/s (12.8MB/s-18.3MB/s), io=28.0MiB (29.4MB), run=345-572msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4743: Thu Mar 13 18:31:39 2025 read: IOPS=10, BW=10.8MiB/s (11.3MB/s)(7168KiB/647msec) clat (msec): min=15, max=117, avg=42.77, stdev=34.88 lat (msec): min=15, max=117, avg=42.77, stdev=34.88 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 21], | 30.00th=[ 29], 40.00th=[ 29], 50.00th=[ 33], 60.00th=[ 33], | 70.00th=[ 33], 80.00th=[ 52], 90.00th=[ 118], 95.00th=[ 118], | 99.00th=[ 118], 99.50th=[ 118], 99.90th=[ 118], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min=10240, max=10240, per=33.95%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7913KiB/s (8103kB/s)(5120KiB/647msec) clat (msec): min=33, max=129, avg=64.37, stdev=40.41 lat (msec): min=33, max=129, avg=64.44, stdev=40.41 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34], | 30.00th=[ 38], 40.00th=[ 38], 50.00th=[ 44], 60.00th=[ 44], | 70.00th=[ 79], 80.00th=[ 79], 90.00th=[ 130], 95.00th=[ 130], | 99.00th=[ 130], 99.50th=[ 130], 99.90th=[ 130], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 6144, max= 6144, per=14.55%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=58.33%, 100=16.67%, 250=16.67% cpu : usr=0.00%, sys=14.09%, ctx=52, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4744: Thu Mar 13 18:31:39 2025 read: IOPS=2, BW=3016KiB/s (3089kB/s)(2048KiB/679msec) clat (usec): min=15406, max=52662, avg=34034.71, stdev=26344.20 lat (usec): min=15407, max=52665, avg=34036.51, stdev=26345.78 clat percentiles (usec): | 1.00th=[15401], 5.00th=[15401], 10.00th=[15401], 20.00th=[15401], | 30.00th=[15401], 40.00th=[15401], 50.00th=[15401], 60.00th=[52691], | 70.00th=[52691], 80.00th=[52691], 90.00th=[52691], 95.00th=[52691], | 99.00th=[52691], 99.50th=[52691], 99.90th=[52691], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 4096, max= 4096, per=13.58%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=14, BW=14.7MiB/s (15.4MB/s)(10.0MiB/679msec) clat (msec): min=21, max=129, avg=58.48, stdev=33.43 lat (msec): min=21, max=129, avg=58.55, stdev=33.43 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 25], | 30.00th=[ 36], 40.00th=[ 40], 50.00th=[ 51], 60.00th=[ 54], | 70.00th=[ 86], 80.00th=[ 86], 90.00th=[ 87], 95.00th=[ 130], | 99.00th=[ 130], 99.50th=[ 130], 99.90th=[ 130], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min=12288, max=12288, per=29.10%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=50.00%, 250=8.33% cpu : usr=0.29%, sys=13.27%, ctx=63, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4745: Thu Mar 13 18:31:39 2025 read: IOPS=11, BW=11.6MiB/s (12.1MB/s)(6144KiB/519msec) clat (usec): min=12230, max=48971, avg=25569.31, stdev=14748.24 lat (usec): min=12231, max=48972, avg=25570.17, stdev=14748.29 clat percentiles (usec): | 1.00th=[12256], 5.00th=[12256], 10.00th=[12256], 20.00th=[14353], | 30.00th=[14353], 40.00th=[15139], 50.00th=[15139], 60.00th=[25822], | 70.00th=[36963], 80.00th=[36963], 90.00th=[49021], 95.00th=[49021], | 99.00th=[49021], 99.50th=[49021], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min=10240, max=10240, per=33.95%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.6MiB/s (12.1MB/s)(6144KiB/519msec) clat (msec): min=22, max=114, avg=54.82, stdev=34.72 lat (msec): min=22, max=114, avg=54.87, stdev=34.72 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 61], | 70.00th=[ 71], 80.00th=[ 71], 90.00th=[ 114], 95.00th=[ 114], | 99.00th=[ 114], 99.50th=[ 114], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min=12288, max=12288, per=29.10%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=50.00%, 100=16.67%, 250=8.33% cpu : usr=0.00%, sys=12.93%, ctx=52, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4746: Thu Mar 13 18:31:39 2025 read: IOPS=8, BW=8312KiB/s (8511kB/s)(5120KiB/616msec) clat (usec): min=15691, max=78131, avg=29826.05, stdev=27045.46 lat (usec): min=15693, max=78132, avg=29827.22, stdev=27045.36 clat percentiles (usec): | 1.00th=[15664], 5.00th=[15664], 10.00th=[15664], 20.00th=[15664], | 30.00th=[17433], 40.00th=[17433], 50.00th=[18220], 60.00th=[18220], | 70.00th=[19792], 80.00th=[19792], 90.00th=[78119], 95.00th=[78119], | 99.00th=[78119], 99.50th=[78119], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 8192, max= 8192, per=27.16%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.4MiB/s (11.9MB/s)(7168KiB/616msec) clat (msec): min=28, max=111, avg=61.24, stdev=35.21 lat (msec): min=28, max=111, avg=61.29, stdev=35.20 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 32], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 49], 60.00th=[ 68], | 70.00th=[ 68], 80.00th=[ 107], 90.00th=[ 112], 95.00th=[ 112], | 99.00th=[ 112], 99.50th=[ 112], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 8192, max= 8192, per=19.40%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=33.33%, 100=16.67%, 250=16.67% cpu : usr=0.16%, sys=10.73%, ctx=50, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=29.5MiB/s (30.9MB/s), 3016KiB/s-11.6MiB/s (3089kB/s-12.1MB/s), io=20.0MiB (20.0MB), run=519-679msec WRITE: bw=41.2MiB/s (43.2MB/s), 7913KiB/s-14.7MiB/s (8103kB/s-15.4MB/s), io=28.0MiB (29.4MB), run=519-679msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4752: Thu Mar 13 18:31:41 2025 read: IOPS=5, BW=22.0MiB/s (24.1MB/s)(12.0MiB/522msec) clat (msec): min=74, max=269, avg=166.69, stdev=98.33 lat (msec): min=74, max=269, avg=166.70, stdev=98.33 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 74], 20.00th=[ 74], | 30.00th=[ 74], 40.00th=[ 157], 50.00th=[ 157], 60.00th=[ 157], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min=16384, max=16384, per=80.70%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.19%, sys=26.87%, ctx=1531, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4753: Thu Mar 13 18:31:41 2025 read: IOPS=1, BW=5076KiB/s (5197kB/s)(4096KiB/807msec) clat (nsec): min=324641k, max=324641k, avg=324640812.00, stdev= 0.00 lat (nsec): min=324643k, max=324643k, avg=324643463.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 326], 10.00th=[ 326], 20.00th=[ 326], | 30.00th=[ 326], 40.00th=[ 326], 50.00th=[ 326], 60.00th=[ 326], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 8192, max= 8192, per=40.35%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9.91MiB/s (10.4MB/s)(8192KiB/807msec) clat (msec): min=214, max=259, avg=237.10, stdev=31.44 lat (msec): min=215, max=259, avg=237.35, stdev=31.41 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 215], 10.00th=[ 215], 20.00th=[ 215], | 30.00th=[ 215], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 259], | 70.00th=[ 259], 80.00th=[ 259], 90.00th=[ 259], 95.00th=[ 259], | 99.00th=[ 259], 99.50th=[ 259], 99.90th=[ 259], 99.95th=[ 259], | 99.99th=[ 259] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=26.18%, ctx=610, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4754: Thu Mar 13 18:31:41 2025 write: IOPS=3, BW=15.4MiB/s (16.2MB/s)(12.0MiB/779msec) clat (msec): min=173, max=324, avg=255.19, stdev=76.04 lat (msec): min=174, max=324, avg=255.43, stdev=76.00 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 268], 50.00th=[ 268], 60.00th=[ 268], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 8192, max= 8192, per=20.18%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=29.69%, ctx=71, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4755: Thu Mar 13 18:31:41 2025 write: IOPS=4, BW=18.4MiB/s (19.3MB/s)(12.0MiB/653msec) clat (msec): min=126, max=308, avg=211.31, stdev=91.22 lat (msec): min=127, max=308, avg=211.59, stdev=91.25 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 128], 10.00th=[ 128], 20.00th=[ 128], | 30.00th=[ 128], 40.00th=[ 199], 50.00th=[ 199], 60.00th=[ 199], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 8192, max= 8192, per=20.18%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=36.66%, ctx=57, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.8MiB/s (20.8MB/s), 5076KiB/s-22.0MiB/s (5197kB/s-24.1MB/s), io=16.0MiB (16.8MB), run=522-807msec WRITE: bw=39.7MiB/s (41.6MB/s), 9.91MiB/s-18.4MiB/s (10.4MB/s-19.3MB/s), io=32.0MiB (33.6MB), run=653-807msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4756: Thu Mar 13 18:31:41 2025 read: IOPS=3, BW=14.9MiB/s (15.6MB/s)(12.0MiB/808msec) clat (msec): min=152, max=430, avg=261.72, stdev=148.46 lat (msec): min=152, max=430, avg=261.72, stdev=148.46 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 153], 10.00th=[ 153], 20.00th=[ 153], | 30.00th=[ 153], 40.00th=[ 203], 50.00th=[ 203], 60.00th=[ 203], | 70.00th=[ 430], 80.00th=[ 430], 90.00th=[ 430], 95.00th=[ 430], | 99.00th=[ 430], 99.50th=[ 430], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 8192, max= 8192, per=45.75%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=7.56%, ctx=45, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4757: Thu Mar 13 18:31:41 2025 read: IOPS=1, BW=4477KiB/s (4584kB/s)(4096KiB/915msec) clat (nsec): min=447806k, max=447806k, avg=447806383.00, stdev= 0.00 lat (nsec): min=447808k, max=447808k, avg=447808023.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 447], 10.00th=[ 447], 20.00th=[ 447], | 30.00th=[ 447], 40.00th=[ 447], 50.00th=[ 447], 60.00th=[ 447], | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 8192, max= 8192, per=45.75%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=8953KiB/s (9168kB/s)(8192KiB/915msec) clat (msec): min=186, max=258, avg=222.13, stdev=50.82 lat (msec): min=186, max=258, avg=222.41, stdev=50.93 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 186], 10.00th=[ 186], 20.00th=[ 186], | 30.00th=[ 186], 40.00th=[ 186], 50.00th=[ 186], 60.00th=[ 259], | 70.00th=[ 259], 80.00th=[ 259], 90.00th=[ 259], 95.00th=[ 259], | 99.00th=[ 259], 99.50th=[ 259], 99.90th=[ 259], 99.95th=[ 259], | 99.99th=[ 259] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=7.88%, ctx=45, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4758: Thu Mar 13 18:31:41 2025 write: IOPS=3, BW=13.3MiB/s (13.0MB/s)(12.0MiB/900msec) clat (msec): min=47, max=548, avg=293.06, stdev=250.57 lat (msec): min=47, max=548, avg=293.43, stdev=250.80 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 284], 50.00th=[ 284], 60.00th=[ 284], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] lat (msec) : 50=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.11%, sys=9.34%, ctx=53, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4759: Thu Mar 13 18:31:41 2025 write: IOPS=3, BW=12.9MiB/s (13.5MB/s)(12.0MiB/932msec) clat (msec): min=61, max=571, avg=300.13, stdev=256.68 lat (msec): min=61, max=571, avg=300.36, stdev=256.71 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 62], 10.00th=[ 62], 20.00th=[ 62], | 30.00th=[ 62], 40.00th=[ 268], 50.00th=[ 268], 60.00th=[ 268], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] lat (msec) : 100=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=11.39%, ctx=48, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.5MiB/s (18.3MB/s), 4477KiB/s-14.9MiB/s (4584kB/s-15.6MB/s), io=16.0MiB (16.8MB), run=808-915msec WRITE: bw=34.3MiB/s (36.0MB/s), 8953KiB/s-13.3MiB/s (9168kB/s-13.0MB/s), io=32.0MiB (33.6MB), run=900-932msec