== sanity test 398b: DIO and buffer IO race ============== 13:11:57 (1746119517) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.05929 s, 47.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12479: Thu May 1 13:12:12 2025 read: IOPS=117, BW=469KiB/s (481kB/s)(5992KiB/12767msec) clat (usec): min=57, max=167777, avg=3893.22, stdev=5318.18 lat (usec): min=57, max=167778, avg=3893.93, stdev=5318.16 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 75], 10.00th=[ 93], 20.00th=[ 2024], | 30.00th=[ 2409], 40.00th=[ 2737], 50.00th=[ 3064], 60.00th=[ 3654], | 70.00th=[ 4555], 80.00th=[ 5800], 90.00th=[ 7439], 95.00th=[ 9241], | 99.00th=[ 13042], 99.50th=[ 19792], 99.90th=[ 39584], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 224, max= 696, per=24.76%, avg=464.28, stdev=104.52, samples=25 iops : min= 56, max= 174, avg=116.04, stdev=26.18, samples=25 write: IOPS=123, BW=493KiB/s (505kB/s)(6296KiB/12767msec) clat (usec): min=468, max=48107, avg=4312.64, stdev=4230.95 lat (usec): min=469, max=48108, avg=4313.53, stdev=4230.99 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 562], 10.00th=[ 594], 20.00th=[ 693], | 30.00th=[ 963], 40.00th=[ 2212], 50.00th=[ 3458], 60.00th=[ 4490], | 70.00th=[ 5473], 80.00th=[ 7046], 90.00th=[ 9765], 95.00th=[11863], | 99.00th=[17957], 99.50th=[21103], 99.90th=[35390], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 256, max= 680, per=24.86%, avg=488.28, stdev=102.22, samples=25 iops : min= 64, max= 170, avg=122.04, stdev=25.60, samples=25 lat (usec) : 100=5.31%, 250=2.86%, 500=0.52%, 750=11.49%, 1000=4.00% lat (msec) : 2=5.27%, 4=30.08%, 10=33.63%, 20=6.18%, 50=0.62% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=23.16%, ctx=5673, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12480: Thu May 1 13:12:12 2025 read: IOPS=116, BW=466KiB/s (477kB/s)(5940KiB/12745msec) clat (usec): min=55, max=43746, avg=3755.61, stdev=3122.38 lat (usec): min=55, max=43747, avg=3756.32, stdev=3122.35 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 74], 10.00th=[ 91], 20.00th=[ 2057], | 30.00th=[ 2442], 40.00th=[ 2769], 50.00th=[ 3163], 60.00th=[ 3720], | 70.00th=[ 4555], 80.00th=[ 5669], 90.00th=[ 7177], 95.00th=[ 8848], | 99.00th=[14091], 99.50th=[15401], 99.90th=[31327], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 232, max= 632, per=24.86%, avg=466.20, stdev=90.78, samples=25 iops : min= 58, max= 158, avg=116.52, stdev=22.71, samples=25 write: IOPS=124, BW=498KiB/s (510kB/s)(6348KiB/12745msec) clat (usec): min=483, max=166312, avg=4424.37, stdev=5941.75 lat (usec): min=484, max=166313, avg=4425.23, stdev=5941.68 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 562], 10.00th=[ 603], 20.00th=[ 709], | 30.00th=[ 971], 40.00th=[ 2311], 50.00th=[ 3326], 60.00th=[ 4359], | 70.00th=[ 5473], 80.00th=[ 7111], 90.00th=[ 9503], 95.00th=[ 12256], | 99.00th=[ 20841], 99.50th=[ 23987], 99.90th=[ 50070], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 288, max= 648, per=25.17%, avg=494.36, stdev=83.11, samples=25 iops : min= 72, max= 162, avg=123.56, stdev=20.85, samples=25 lat (usec) : 100=5.70%, 250=2.15%, 500=0.42%, 750=11.46%, 1000=4.30% lat (msec) : 2=4.82%, 4=31.38%, 10=33.66%, 20=5.44%, 50=0.65% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=23.24%, ctx=5590, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12482: Thu May 1 13:12:12 2025 read: IOPS=117, BW=471KiB/s (482kB/s)(6028KiB/12800msec) clat (usec): min=55, max=41293, avg=3706.94, stdev=3304.22 lat (usec): min=55, max=41293, avg=3707.73, stdev=3304.25 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 72], 10.00th=[ 84], 20.00th=[ 1680], | 30.00th=[ 2311], 40.00th=[ 2704], 50.00th=[ 3097], 60.00th=[ 3654], | 70.00th=[ 4555], 80.00th=[ 5669], 90.00th=[ 7308], 95.00th=[ 8979], | 99.00th=[13435], 99.50th=[17957], 99.90th=[38536], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 56, max= 648, per=24.71%, avg=463.32, stdev=126.71, samples=25 iops : min= 14, max= 162, avg=115.80, stdev=31.69, samples=25 write: IOPS=122, BW=489KiB/s (501kB/s)(6260KiB/12800msec) clat (usec): min=434, max=216806, avg=4514.37, stdev=7068.42 lat (usec): min=434, max=216808, avg=4515.78, stdev=7068.76 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 553], 10.00th=[ 594], 20.00th=[ 701], | 30.00th=[ 930], 40.00th=[ 2073], 50.00th=[ 3425], 60.00th=[ 4424], | 70.00th=[ 5538], 80.00th=[ 7046], 90.00th=[ 9634], 95.00th=[ 12518], | 99.00th=[ 19268], 99.50th=[ 27657], 99.90th=[ 50594], 99.95th=[217056], | 99.99th=[217056] bw ( KiB/s): min= 80, max= 712, per=24.34%, avg=478.04, stdev=126.38, samples=25 iops : min= 20, max= 178, avg=119.48, stdev=31.62, samples=25 lat (usec) : 100=6.64%, 250=2.86%, 500=0.46%, 750=11.59%, 1000=4.20% lat (msec) : 2=5.89%, 4=28.58%, 10=33.53%, 20=5.60%, 50=0.59% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.20%, sys=23.10%, ctx=5615, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12484: Thu May 1 13:12:12 2025 read: IOPS=118, BW=475KiB/s (486kB/s)(6052KiB/12748msec) clat (usec): min=53, max=65831, avg=3606.82, stdev=3398.21 lat (usec): min=53, max=65831, avg=3607.48, stdev=3398.23 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 75], 10.00th=[ 89], 20.00th=[ 1778], | 30.00th=[ 2343], 40.00th=[ 2638], 50.00th=[ 2999], 60.00th=[ 3556], | 70.00th=[ 4228], 80.00th=[ 5407], 90.00th=[ 7308], 95.00th=[ 8848], | 99.00th=[12780], 99.50th=[17433], 99.90th=[29230], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 80, max= 616, per=25.23%, avg=473.08, stdev=107.22, samples=25 iops : min= 20, max= 154, avg=118.20, stdev=26.81, samples=25 write: IOPS=122, BW=489KiB/s (501kB/s)(6236KiB/12748msec) clat (usec): min=469, max=217681, avg=4580.74, stdev=6990.22 lat (usec): min=470, max=217683, avg=4582.44, stdev=6991.19 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 562], 10.00th=[ 611], 20.00th=[ 734], | 30.00th=[ 1037], 40.00th=[ 2180], 50.00th=[ 3326], 60.00th=[ 4359], | 70.00th=[ 5538], 80.00th=[ 7308], 90.00th=[ 10290], 95.00th=[ 13042], | 99.00th=[ 19268], 99.50th=[ 25560], 99.90th=[ 39584], 99.95th=[217056], | 99.99th=[217056] bw ( KiB/s): min= 96, max= 704, per=24.76%, avg=486.24, stdev=117.11, samples=25 iops : min= 24, max= 176, avg=121.48, stdev=29.31, samples=25 lat (usec) : 100=5.89%, 250=3.52%, 500=0.36%, 750=10.61%, 1000=4.10% lat (msec) : 2=6.09%, 4=30.79%, 10=31.41%, 20=6.61%, 50=0.55% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.23%, sys=22.70%, ctx=5512, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1876KiB/s (1921kB/s), 466KiB/s-475KiB/s (477kB/s-486kB/s), io=23.4MiB (24.6MB), run=12745-12800msec WRITE: bw=1964KiB/s (2011kB/s), 489KiB/s-498KiB/s (501kB/s-510kB/s), io=24.6MiB (25.7MB), run=12745-12800msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12481: Thu May 1 13:12:36 2025 read: IOPS=41, BW=164KiB/s (168kB/s)(5992KiB/36516msec) clat (usec): min=1608, max=170626, avg=3498.82, stdev=5369.81 lat (usec): min=1608, max=170627, avg=3499.58, stdev=5369.83 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1844], 10.00th=[ 1909], 20.00th=[ 1975], | 30.00th=[ 2057], 40.00th=[ 2147], 50.00th=[ 2245], 60.00th=[ 2409], | 70.00th=[ 2671], 80.00th=[ 3359], 90.00th=[ 6783], 95.00th=[ 10028], | 99.00th=[ 16450], 99.50th=[ 21890], 99.90th=[ 39584], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 55, max= 288, per=25.12%, avg=164.05, stdev=66.10, samples=73 iops : min= 13, max= 72, avg=40.93, stdev=16.58, samples=73 write: IOPS=43, BW=172KiB/s (177kB/s)(6296KiB/36516msec) clat (usec): min=9618, max=67138, avg=19777.96, stdev=7426.73 lat (usec): min=9619, max=67139, avg=19778.89, stdev=7426.75 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11731], 10.00th=[12387], 20.00th=[13566], | 30.00th=[15270], 40.00th=[16909], 50.00th=[18482], 60.00th=[19792], | 70.00th=[21365], 80.00th=[23200], 90.00th=[29230], 95.00th=[34866], | 99.00th=[45351], 99.50th=[50070], 99.90th=[62653], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 40, max= 232, per=25.22%, avg=172.27, stdev=50.41, samples=73 iops : min= 10, max= 58, avg=42.99, stdev=12.67, samples=73 lat (msec) : 2=10.68%, 4=29.92%, 10=5.83%, 20=32.98%, 50=20.28% lat (msec) : 100=0.29%, 250=0.03% cpu : usr=0.08%, sys=5.03%, ctx=3448, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12483: Thu May 1 13:12:36 2025 read: IOPS=40, BW=162KiB/s (166kB/s)(5940KiB/36727msec) clat (usec): min=1603, max=186343, avg=3559.47, stdev=5845.41 lat (usec): min=1603, max=186344, avg=3560.15, stdev=5845.42 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1844], 10.00th=[ 1909], 20.00th=[ 1975], | 30.00th=[ 2040], 40.00th=[ 2147], 50.00th=[ 2245], 60.00th=[ 2409], | 70.00th=[ 2704], 80.00th=[ 3392], 90.00th=[ 6980], 95.00th=[ 9896], | 99.00th=[ 16712], 99.50th=[ 22152], 99.90th=[ 54264], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 16, max= 312, per=24.78%, avg=161.79, stdev=67.56, samples=73 iops : min= 4, max= 78, avg=40.40, stdev=16.87, samples=73 write: IOPS=43, BW=173KiB/s (177kB/s)(6348KiB/36727msec) clat (usec): min=9721, max=80159, avg=19719.56, stdev=7282.90 lat (usec): min=9721, max=80160, avg=19720.45, stdev=7282.92 clat percentiles (usec): | 1.00th=[10945], 5.00th=[11994], 10.00th=[12518], 20.00th=[13698], | 30.00th=[15270], 40.00th=[17171], 50.00th=[18482], 60.00th=[19530], | 70.00th=[21103], 80.00th=[23462], 90.00th=[29754], 95.00th=[34341], | 99.00th=[43779], 99.50th=[48497], 99.90th=[62653], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 48, max= 232, per=25.26%, avg=172.53, stdev=50.71, samples=73 iops : min= 12, max= 58, avg=43.08, stdev=12.68, samples=73 lat (msec) : 2=11.10%, 4=28.97%, 10=6.02%, 20=34.18%, 50=19.43% lat (msec) : 100=0.26%, 250=0.03% cpu : usr=0.11%, sys=5.03%, ctx=3429, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12485: Thu May 1 13:12:36 2025 read: IOPS=40, BW=164KiB/s (168kB/s)(6028KiB/36769msec) clat (usec): min=1624, max=48781, avg=3633.41, stdev=3658.14 lat (usec): min=1625, max=48782, avg=3634.18, stdev=3658.14 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1844], 10.00th=[ 1909], 20.00th=[ 2008], | 30.00th=[ 2089], 40.00th=[ 2180], 50.00th=[ 2311], 60.00th=[ 2507], | 70.00th=[ 2868], 80.00th=[ 4146], 90.00th=[ 7242], 95.00th=[10421], | 99.00th=[17171], 99.50th=[21627], 99.90th=[45876], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 40, max= 304, per=25.08%, avg=163.77, stdev=58.54, samples=73 iops : min= 10, max= 76, avg=40.89, stdev=14.65, samples=73 write: IOPS=42, BW=170KiB/s (174kB/s)(6260KiB/36769msec) clat (msec): min=9, max=214, avg=19.90, stdev= 8.75 lat (msec): min=9, max=214, avg=19.91, stdev= 8.75 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 19], 60.00th=[ 21], | 70.00th=[ 22], 80.00th=[ 24], 90.00th=[ 29], 95.00th=[ 34], | 99.00th=[ 47], 99.50th=[ 52], 99.90th=[ 82], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 40, max= 248, per=24.86%, avg=169.79, stdev=50.12, samples=73 iops : min= 10, max= 62, avg=42.40, stdev=12.55, samples=73 lat (msec) : 2=9.51%, 4=29.43%, 10=7.55%, 20=32.78%, 50=20.38% lat (msec) : 100=0.33%, 250=0.03% cpu : usr=0.10%, sys=4.98%, ctx=3480, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12486: Thu May 1 13:12:36 2025 read: IOPS=41, BW=166KiB/s (170kB/s)(6052KiB/36543msec) clat (usec): min=1624, max=53842, avg=3522.82, stdev=3692.89 lat (usec): min=1625, max=53843, avg=3523.59, stdev=3692.85 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1827], 10.00th=[ 1909], 20.00th=[ 1991], | 30.00th=[ 2073], 40.00th=[ 2147], 50.00th=[ 2278], 60.00th=[ 2442], | 70.00th=[ 2737], 80.00th=[ 3556], 90.00th=[ 7046], 95.00th=[10159], | 99.00th=[16909], 99.50th=[23200], 99.90th=[48497], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 16, max= 344, per=25.37%, avg=165.64, stdev=65.33, samples=73 iops : min= 4, max= 86, avg=41.37, stdev=16.37, samples=73 write: IOPS=42, BW=171KiB/s (175kB/s)(6236KiB/36543msec) clat (msec): min=9, max=215, avg=19.93, stdev= 8.72 lat (msec): min=9, max=215, avg=19.93, stdev= 8.72 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 19], 60.00th=[ 21], | 70.00th=[ 22], 80.00th=[ 24], 90.00th=[ 30], 95.00th=[ 35], | 99.00th=[ 46], 99.50th=[ 51], 99.90th=[ 74], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 40, max= 232, per=24.96%, avg=170.47, stdev=49.61, samples=73 iops : min= 10, max= 58, avg=42.58, stdev=12.42, samples=73 lat (msec) : 2=10.61%, 4=29.88%, 10=6.32%, 20=32.71%, 50=20.18% lat (msec) : 100=0.26%, 250=0.03% cpu : usr=0.11%, sys=4.93%, ctx=3438, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=653KiB/s (669kB/s), 162KiB/s-166KiB/s (166kB/s-170kB/s), io=23.4MiB (24.6MB), run=36516-36769msec WRITE: bw=684KiB/s (700kB/s), 170KiB/s-173KiB/s (174kB/s-177kB/s), io=24.6MiB (25.7MB), run=36516-36769msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12541: Thu May 1 13:12:40 2025 read: IOPS=146, BW=2343KiB/s (2399kB/s)(5920KiB/2527msec) clat (usec): min=59, max=13452, avg=3042.09, stdev=2440.64 lat (usec): min=59, max=13452, avg=3042.72, stdev=2440.67 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 75], 10.00th=[ 85], 20.00th=[ 110], | 30.00th=[ 359], 40.00th=[ 2737], 50.00th=[ 3195], 60.00th=[ 3523], | 70.00th=[ 3949], 80.00th=[ 4817], 90.00th=[ 6194], 95.00th=[ 7308], | 99.00th=[ 9241], 99.50th=[12387], 99.90th=[13435], 99.95th=[13435], | 99.99th=[13435] bw ( KiB/s): min= 2048, max= 2752, per=26.77%, avg=2348.80, stdev=299.85, samples=5 iops : min= 128, max= 172, avg=146.80, stdev=18.74, samples=5 write: IOPS=157, BW=2520KiB/s (2580kB/s)(6368KiB/2527msec) clat (usec): min=658, max=15753, avg=3319.71, stdev=3227.23 lat (usec): min=659, max=15754, avg=3320.71, stdev=3227.21 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 775], 10.00th=[ 840], 20.00th=[ 938], | 30.00th=[ 1057], 40.00th=[ 1254], 50.00th=[ 1680], 60.00th=[ 3032], | 70.00th=[ 4146], 80.00th=[ 5276], 90.00th=[ 8094], 95.00th=[10028], | 99.00th=[15533], 99.50th=[15795], 99.90th=[15795], 99.95th=[15795], | 99.99th=[15795] bw ( KiB/s): min= 1952, max= 3200, per=27.39%, avg=2528.00, stdev=487.93, samples=5 iops : min= 122, max= 200, avg=158.00, stdev=30.50, samples=5 lat (usec) : 100=8.98%, 250=4.82%, 500=0.65%, 750=2.08%, 1000=10.94% lat (msec) : 2=14.84%, 4=27.21%, 10=27.60%, 20=2.86% cpu : usr=0.48%, sys=29.77%, ctx=1323, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12542: Thu May 1 13:12:40 2025 read: IOPS=131, BW=2104KiB/s (2155kB/s)(5744KiB/2730msec) clat (usec): min=61, max=10494, avg=2977.30, stdev=2638.02 lat (usec): min=61, max=10495, avg=2977.87, stdev=2638.02 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 74], 10.00th=[ 78], 20.00th=[ 89], | 30.00th=[ 121], 40.00th=[ 2507], 50.00th=[ 2999], 60.00th=[ 3490], | 70.00th=[ 4359], 80.00th=[ 5342], 90.00th=[ 6587], 95.00th=[ 7832], | 99.00th=[10028], 99.50th=[10421], 99.90th=[10552], 99.95th=[10552], | 99.99th=[10552] bw ( KiB/s): min= 960, max= 2880, per=23.27%, avg=2041.60, stdev=697.57, samples=5 iops : min= 60, max= 180, avg=127.60, stdev=43.60, samples=5 write: IOPS=149, BW=2397KiB/s (2455kB/s)(6544KiB/2730msec) clat (usec): min=672, max=27432, avg=3894.27, stdev=4484.48 lat (usec): min=673, max=27434, avg=3895.17, stdev=4484.53 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 775], 10.00th=[ 832], 20.00th=[ 914], | 30.00th=[ 1029], 40.00th=[ 1270], 50.00th=[ 1762], 60.00th=[ 3392], | 70.00th=[ 4686], 80.00th=[ 6194], 90.00th=[ 9241], 95.00th=[11600], | 99.00th=[24511], 99.50th=[26608], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 1152, max= 2784, per=24.89%, avg=2297.60, stdev=651.65, samples=5 iops : min= 72, max= 174, avg=143.60, stdev=40.73, samples=5 lat (usec) : 100=11.33%, 250=4.69%, 500=0.52%, 750=1.95%, 1000=13.67% lat (msec) : 2=12.24%, 4=21.09%, 10=30.21%, 20=3.12%, 50=1.17% cpu : usr=0.26%, sys=27.70%, ctx=1335, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12543: Thu May 1 13:12:40 2025 read: IOPS=148, BW=2378KiB/s (2435kB/s)(6256KiB/2631msec) clat (usec): min=59, max=16432, avg=3167.17, stdev=2890.28 lat (usec): min=59, max=16433, avg=3167.75, stdev=2890.29 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 71], 10.00th=[ 78], 20.00th=[ 92], | 30.00th=[ 135], 40.00th=[ 2573], 50.00th=[ 2999], 60.00th=[ 3523], | 70.00th=[ 4555], 80.00th=[ 5538], 90.00th=[ 6849], 95.00th=[ 8848], | 99.00th=[10814], 99.50th=[12911], 99.90th=[16450], 99.95th=[16450], | 99.99th=[16450] bw ( KiB/s): min= 1856, max= 2688, per=27.72%, avg=2432.00, stdev=333.32, samples=5 iops : min= 116, max= 168, avg=152.00, stdev=20.83, samples=5 write: IOPS=143, BW=2293KiB/s (2348kB/s)(6032KiB/2631msec) clat (usec): min=637, max=25881, avg=3634.15, stdev=3503.73 lat (usec): min=637, max=25882, avg=3635.75, stdev=3505.17 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 775], 10.00th=[ 824], 20.00th=[ 938], | 30.00th=[ 1123], 40.00th=[ 1369], 50.00th=[ 1926], 60.00th=[ 3359], | 70.00th=[ 4424], 80.00th=[ 6194], 90.00th=[ 8717], 95.00th=[10290], | 99.00th=[15008], 99.50th=[16581], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 1728, max= 2880, per=24.75%, avg=2284.80, stdev=496.57, samples=5 iops : min= 108, max= 180, avg=142.80, stdev=31.04, samples=5 lat (usec) : 100=11.72%, 250=5.21%, 500=0.65%, 750=1.95%, 1000=10.68% lat (msec) : 2=12.63%, 4=22.53%, 10=31.12%, 20=3.39%, 50=0.13% cpu : usr=0.34%, sys=28.97%, ctx=1455, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12544: Thu May 1 13:12:40 2025 read: IOPS=141, BW=2271KiB/s (2326kB/s)(6032KiB/2656msec) clat (usec): min=61, max=17220, avg=2911.35, stdev=2936.22 lat (usec): min=61, max=17221, avg=2912.11, stdev=2936.37 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 73], 10.00th=[ 78], 20.00th=[ 89], | 30.00th=[ 121], 40.00th=[ 2245], 50.00th=[ 2769], 60.00th=[ 3228], | 70.00th=[ 3949], 80.00th=[ 5014], 90.00th=[ 6456], 95.00th=[ 8225], | 99.00th=[13173], 99.50th=[16909], 99.90th=[17171], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 1728, max= 2560, per=25.53%, avg=2240.00, stdev=314.35, samples=5 iops : min= 108, max= 160, avg=140.00, stdev=19.65, samples=5 write: IOPS=147, BW=2355KiB/s (2412kB/s)(6256KiB/2656msec) clat (usec): min=653, max=27752, avg=3757.83, stdev=4013.63 lat (usec): min=653, max=27755, avg=3758.78, stdev=4013.70 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 799], 10.00th=[ 848], 20.00th=[ 947], | 30.00th=[ 1074], 40.00th=[ 1336], 50.00th=[ 2180], 60.00th=[ 3294], | 70.00th=[ 4555], 80.00th=[ 5997], 90.00th=[ 8225], 95.00th=[11207], | 99.00th=[23200], 99.50th=[24773], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 1888, max= 2944, per=25.45%, avg=2348.80, stdev=435.60, samples=5 iops : min= 118, max= 184, avg=146.80, stdev=27.22, samples=5 lat (usec) : 100=12.37%, 250=5.86%, 500=0.39%, 750=1.69%, 1000=10.68% lat (msec) : 2=13.15%, 4=24.22%, 10=26.82%, 20=4.17%, 50=0.65% cpu : usr=0.19%, sys=27.76%, ctx=1330, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8774KiB/s (8984kB/s), 2104KiB/s-2378KiB/s (2155kB/s-2435kB/s), io=23.4MiB (24.5MB), run=2527-2730msec WRITE: bw=9231KiB/s (9452kB/s), 2293KiB/s-2520KiB/s (2348kB/s-2580kB/s), io=24.6MiB (25.8MB), run=2527-2730msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12537: Thu May 1 13:12:47 2025 read: IOPS=38, BW=623KiB/s (638kB/s)(5920KiB/9497msec) clat (usec): min=1870, max=23380, avg=4259.15, stdev=3094.74 lat (usec): min=1870, max=23380, avg=4259.89, stdev=3094.79 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2073], 10.00th=[ 2147], 20.00th=[ 2278], | 30.00th=[ 2376], 40.00th=[ 2540], 50.00th=[ 2835], 60.00th=[ 3458], | 70.00th=[ 4686], 80.00th=[ 6063], 90.00th=[ 7111], 95.00th=[10683], | 99.00th=[17171], 99.50th=[19792], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 256, max= 1085, per=25.09%, avg=620.28, stdev=191.61, samples=18 iops : min= 16, max= 67, avg=38.72, stdev=11.86, samples=18 write: IOPS=41, BW=671KiB/s (687kB/s)(6368KiB/9497msec) clat (usec): min=11362, max=51722, avg=19673.74, stdev=5908.33 lat (usec): min=11364, max=51725, avg=19675.22, stdev=5908.40 clat percentiles (usec): | 1.00th=[12125], 5.00th=[12911], 10.00th=[13304], 20.00th=[14615], | 30.00th=[15664], 40.00th=[17171], 50.00th=[18482], 60.00th=[20317], | 70.00th=[21627], 80.00th=[23200], 90.00th=[27657], 95.00th=[32113], | 99.00th=[38536], 99.50th=[42206], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 384, max= 896, per=25.49%, avg=663.00, stdev=160.84, samples=18 iops : min= 24, max= 56, avg=41.39, stdev=10.04, samples=18 lat (msec) : 2=0.65%, 4=29.82%, 10=15.23%, 20=32.42%, 50=21.74% lat (msec) : 100=0.13% cpu : usr=0.13%, sys=6.38%, ctx=1011, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12538: Thu May 1 13:12:47 2025 read: IOPS=37, BW=593KiB/s (607kB/s)(5744KiB/9688msec) clat (usec): min=1873, max=25942, avg=4252.66, stdev=3131.37 lat (usec): min=1874, max=25942, avg=4253.53, stdev=3131.51 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2057], 10.00th=[ 2114], 20.00th=[ 2245], | 30.00th=[ 2376], 40.00th=[ 2540], 50.00th=[ 2868], 60.00th=[ 3425], | 70.00th=[ 4555], 80.00th=[ 6128], 90.00th=[ 7570], 95.00th=[10421], | 99.00th=[18744], 99.50th=[21103], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 288, max= 1024, per=23.98%, avg=592.68, stdev=202.52, samples=19 iops : min= 18, max= 64, avg=36.95, stdev=12.70, samples=19 write: IOPS=42, BW=675KiB/s (692kB/s)(6544KiB/9688msec) clat (usec): min=10303, max=53687, avg=19911.52, stdev=6298.63 lat (usec): min=10304, max=53689, avg=19913.12, stdev=6298.71 clat percentiles (usec): | 1.00th=[12256], 5.00th=[12780], 10.00th=[13435], 20.00th=[15008], | 30.00th=[16057], 40.00th=[17433], 50.00th=[18744], 60.00th=[20317], | 70.00th=[21890], 80.00th=[23462], 90.00th=[27657], 95.00th=[32113], | 99.00th=[42206], 99.50th=[44303], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 416, max= 862, per=25.96%, avg=675.16, stdev=154.89, samples=19 iops : min= 26, max= 53, avg=42.11, stdev= 9.68, samples=19 lat (msec) : 2=0.78%, 4=29.30%, 10=14.06%, 20=33.98%, 50=21.74% lat (msec) : 100=0.13% cpu : usr=0.14%, sys=6.09%, ctx=987, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12539: Thu May 1 13:12:47 2025 read: IOPS=43, BW=693KiB/s (710kB/s)(6256KiB/9022msec) clat (usec): min=1835, max=34312, avg=4098.12, stdev=3648.61 lat (usec): min=1836, max=34313, avg=4098.82, stdev=3648.62 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2073], 10.00th=[ 2147], 20.00th=[ 2212], | 30.00th=[ 2343], 40.00th=[ 2442], 50.00th=[ 2606], 60.00th=[ 2802], | 70.00th=[ 3425], 80.00th=[ 5604], 90.00th=[ 7177], 95.00th=[10814], | 99.00th=[19792], 99.50th=[23200], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 320, max= 1248, per=28.11%, avg=694.94, stdev=287.37, samples=18 iops : min= 20, max= 78, avg=43.33, stdev=18.06, samples=18 write: IOPS=41, BW=669KiB/s (685kB/s)(6032KiB/9022msec) clat (usec): min=11056, max=50363, avg=19456.37, stdev=5953.72 lat (usec): min=11057, max=50364, avg=19457.82, stdev=5953.82 clat percentiles (usec): | 1.00th=[11863], 5.00th=[12780], 10.00th=[13304], 20.00th=[14222], | 30.00th=[15270], 40.00th=[16909], 50.00th=[18482], 60.00th=[20055], | 70.00th=[21627], 80.00th=[23200], 90.00th=[27395], 95.00th=[31851], | 99.00th=[40109], 99.50th=[42206], 99.90th=[50594], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 416, max= 832, per=25.62%, avg=666.44, stdev=156.16, samples=18 iops : min= 26, max= 52, avg=41.56, stdev= 9.79, samples=18 lat (msec) : 2=0.65%, 4=35.94%, 10=11.20%, 20=31.64%, 50=20.44% lat (msec) : 100=0.13% cpu : usr=0.13%, sys=6.42%, ctx=984, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12540: Thu May 1 13:12:47 2025 read: IOPS=39, BW=637KiB/s (653kB/s)(6032KiB/9462msec) clat (usec): min=1910, max=34200, avg=4421.87, stdev=3974.77 lat (usec): min=1910, max=34201, avg=4422.58, stdev=3974.79 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2073], 10.00th=[ 2180], 20.00th=[ 2278], | 30.00th=[ 2376], 40.00th=[ 2507], 50.00th=[ 2769], 60.00th=[ 3294], | 70.00th=[ 4359], 80.00th=[ 5997], 90.00th=[ 8094], 95.00th=[12256], | 99.00th=[22938], 99.50th=[30016], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 223, max= 1152, per=25.31%, avg=625.61, stdev=219.80, samples=18 iops : min= 13, max= 72, avg=39.00, stdev=13.77, samples=18 write: IOPS=41, BW=661KiB/s (677kB/s)(6256KiB/9462msec) clat (usec): min=11258, max=49863, avg=19702.37, stdev=5887.15 lat (usec): min=11259, max=49865, avg=19703.92, stdev=5887.31 clat percentiles (usec): | 1.00th=[11994], 5.00th=[12911], 10.00th=[13566], 20.00th=[14484], | 30.00th=[15664], 40.00th=[17171], 50.00th=[19006], 60.00th=[20317], | 70.00th=[21627], 80.00th=[22938], 90.00th=[26608], 95.00th=[31589], | 99.00th=[39060], 99.50th=[47449], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 384, max= 832, per=25.21%, avg=655.78, stdev=157.27, samples=18 iops : min= 24, max= 52, avg=40.89, stdev= 9.85, samples=18 lat (msec) : 2=0.91%, 4=31.90%, 10=12.50%, 20=31.64%, 50=23.05% cpu : usr=0.10%, sys=6.25%, ctx=990, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2472KiB/s (2532kB/s), 593KiB/s-693KiB/s (607kB/s-710kB/s), io=23.4MiB (24.5MB), run=9022-9688msec WRITE: bw=2601KiB/s (2664kB/s), 661KiB/s-675KiB/s (677kB/s-692kB/s), io=24.6MiB (25.8MB), run=9022-9688msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12561: Thu May 1 13:12:49 2025 read: IOPS=8, BW=8564KiB/s (8769kB/s)(7168KiB/837msec) clat (usec): min=587, max=57589, avg=36193.02, stdev=20777.70 lat (usec): min=588, max=57590, avg=36194.15, stdev=20777.88 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 586], 10.00th=[ 586], 20.00th=[16909], | 30.00th=[38536], 40.00th=[38536], 50.00th=[39060], 60.00th=[43779], | 70.00th=[43779], 80.00th=[56886], 90.00th=[57410], 95.00th=[57410], | 99.00th=[57410], 99.50th=[57410], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min=10240, max=10240, per=41.85%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=5, BW=6117KiB/s (6264kB/s)(5120KiB/837msec) clat (msec): min=20, max=304, avg=109.12, stdev=113.08 lat (msec): min=20, max=304, avg=109.18, stdev=113.09 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 45], 40.00th=[ 45], 50.00th=[ 83], 60.00th=[ 83], | 70.00th=[ 94], 80.00th=[ 94], 90.00th=[ 305], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 4096, max= 4096, per=11.96%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 20=8.33%, 50=41.67%, 100=33.33%, 500=8.33% cpu : usr=0.00%, sys=23.33%, ctx=658, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12562: Thu May 1 13:12:49 2025 read: IOPS=2, BW=2501KiB/s (2561kB/s)(2048KiB/819msec) clat (usec): min=25802, max=33134, avg=29468.04, stdev=5184.56 lat (usec): min=25803, max=33135, avg=29469.18, stdev=5184.74 clat percentiles (usec): | 1.00th=[25822], 5.00th=[25822], 10.00th=[25822], 20.00th=[25822], | 30.00th=[25822], 40.00th=[25822], 50.00th=[25822], 60.00th=[33162], | 70.00th=[33162], 80.00th=[33162], 90.00th=[33162], 95.00th=[33162], | 99.00th=[33162], 99.50th=[33162], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 4096, max= 4096, per=16.74%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.2MiB/s (12.8MB/s)(10.0MiB/819msec) clat (msec): min=26, max=171, avg=71.18, stdev=43.63 lat (msec): min=26, max=171, avg=71.24, stdev=43.63 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 33], | 30.00th=[ 42], 40.00th=[ 43], 50.00th=[ 57], 60.00th=[ 61], | 70.00th=[ 97], 80.00th=[ 97], 90.00th=[ 100], 95.00th=[ 171], | 99.00th=[ 171], 99.50th=[ 171], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min=14336, max=14336, per=41.85%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=41.67%, 250=8.33% cpu : usr=0.12%, sys=31.05%, ctx=270, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12563: Thu May 1 13:12:49 2025 read: IOPS=8, BW=8752KiB/s (8962kB/s)(6144KiB/702msec) clat (usec): min=614, max=299553, avg=63782.42, stdev=116358.68 lat (usec): min=615, max=299554, avg=63783.53, stdev=116358.40 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 611], 10.00th=[ 611], 20.00th=[ 1106], | 30.00th=[ 1106], 40.00th=[ 16712], 50.00th=[ 16712], 60.00th=[ 31589], | 70.00th=[ 33162], 80.00th=[ 33162], 90.00th=[299893], 95.00th=[299893], | 99.00th=[299893], 99.50th=[299893], 99.90th=[299893], 99.95th=[299893], | 99.99th=[299893] bw ( KiB/s): min= 6144, max= 6144, per=25.11%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8752KiB/s (8962kB/s)(6144KiB/702msec) clat (usec): min=27392, max=79651, avg=51103.15, stdev=19792.17 lat (usec): min=27486, max=79676, avg=51174.97, stdev=19764.08 clat percentiles (usec): | 1.00th=[27395], 5.00th=[27395], 10.00th=[27395], 20.00th=[36963], | 30.00th=[36963], 40.00th=[39060], 50.00th=[39060], 60.00th=[61604], | 70.00th=[62129], 80.00th=[62129], 90.00th=[79168], 95.00th=[79168], | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 8192, max= 8192, per=23.91%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 2=8.33%, 20=8.33%, 50=41.67%, 100=25.00%, 500=8.33% cpu : usr=0.00%, sys=28.10%, ctx=467, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12564: Thu May 1 13:12:49 2025 read: IOPS=8, BW=9143KiB/s (9362kB/s)(5120KiB/560msec) clat (usec): min=450, max=32076, avg=17961.89, stdev=16051.01 lat (usec): min=450, max=32078, avg=17963.02, stdev=16051.35 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 449], 10.00th=[ 449], 20.00th=[ 449], | 30.00th=[ 478], 40.00th=[ 478], 50.00th=[27657], 60.00th=[27657], | 70.00th=[28967], 80.00th=[28967], 90.00th=[32113], 95.00th=[32113], | 99.00th=[32113], 99.50th=[32113], 99.90th=[32113], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 8192, max= 8192, per=33.48%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.5MiB/s (13.1MB/s)(7168KiB/560msec) clat (msec): min=30, max=103, avg=61.57, stdev=31.87 lat (msec): min=30, max=103, avg=61.64, stdev=31.84 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 33], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 57], 60.00th=[ 70], | 70.00th=[ 70], 80.00th=[ 104], 90.00th=[ 104], 95.00th=[ 104], | 99.00th=[ 104], 99.50th=[ 104], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min=12288, max=12288, per=35.87%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 50=50.00%, 100=16.67%, 250=16.67% cpu : usr=0.00%, sys=32.38%, ctx=661, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=23.9MiB/s (25.1MB/s), 2501KiB/s-9143KiB/s (2561kB/s-9362kB/s), io=20.0MiB (20.0MB), run=560-837msec WRITE: bw=33.5MiB/s (35.1MB/s), 6117KiB/s-12.5MiB/s (6264kB/s-13.1MB/s), io=28.0MiB (29.4MB), run=560-837msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12556: Thu May 1 13:12:49 2025 read: IOPS=7, BW=7436KiB/s (7614kB/s)(7168KiB/964msec) clat (usec): min=11854, max=78839, avg=46442.50, stdev=27312.08 lat (usec): min=11855, max=78840, avg=46443.61, stdev=27312.19 clat percentiles (usec): | 1.00th=[11863], 5.00th=[11863], 10.00th=[11863], 20.00th=[22938], | 30.00th=[24773], 40.00th=[24773], 50.00th=[44827], 60.00th=[70779], | 70.00th=[70779], 80.00th=[71828], 90.00th=[79168], 95.00th=[79168], | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min=10240, max=10240, per=48.20%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5311KiB/s (5439kB/s)(5120KiB/964msec) clat (msec): min=46, max=347, avg=125.39, stdev=124.79 lat (msec): min=46, max=347, avg=125.47, stdev=124.80 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 68], 40.00th=[ 68], 50.00th=[ 81], 60.00th=[ 81], | 70.00th=[ 85], 80.00th=[ 85], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 4096, max= 4096, per=13.77%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=50.00%, 500=8.33% cpu : usr=0.00%, sys=9.55%, ctx=117, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12557: Thu May 1 13:12:49 2025 read: IOPS=2, BW=2306KiB/s (2362kB/s)(2048KiB/888msec) clat (usec): min=14457, max=46683, avg=30570.47, stdev=22787.68 lat (usec): min=14458, max=46684, avg=30571.65, stdev=22786.93 clat percentiles (usec): | 1.00th=[14484], 5.00th=[14484], 10.00th=[14484], 20.00th=[14484], | 30.00th=[14484], 40.00th=[14484], 50.00th=[14484], 60.00th=[46924], | 70.00th=[46924], 80.00th=[46924], 90.00th=[46924], 95.00th=[46924], | 99.00th=[46924], 99.50th=[46924], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 4087, max= 4087, per=19.24%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.3MiB/s (11.8MB/s)(10.0MiB/888msec) clat (msec): min=30, max=277, avg=81.83, stdev=74.88 lat (msec): min=30, max=277, avg=81.90, stdev=74.88 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 33], | 30.00th=[ 34], 40.00th=[ 36], 50.00th=[ 56], 60.00th=[ 65], | 70.00th=[ 109], 80.00th=[ 109], 90.00th=[ 109], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min=14307, max=14307, per=48.10%, avg=14307.00, stdev= 0.00, samples=1 iops : min= 13, max= 13, avg=13.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=25.00%, 250=16.67%, 500=8.33% cpu : usr=0.00%, sys=11.61%, ctx=39, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12558: Thu May 1 13:12:49 2025 read: IOPS=8, BW=8521KiB/s (8726kB/s)(6144KiB/721msec) clat (usec): min=12695, max=77206, avg=34868.59, stdev=24097.33 lat (usec): min=12696, max=77207, avg=34869.62, stdev=24097.08 clat percentiles (usec): | 1.00th=[12649], 5.00th=[12649], 10.00th=[12649], 20.00th=[14484], | 30.00th=[14484], 40.00th=[24249], 50.00th=[24249], 60.00th=[36439], | 70.00th=[44303], 80.00th=[44303], 90.00th=[77071], 95.00th=[77071], | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min=10240, max=10240, per=48.20%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8521KiB/s (8726kB/s)(6144KiB/721msec) clat (msec): min=27, max=258, avg=81.90, stdev=89.51 lat (msec): min=27, max=258, avg=81.98, stdev=89.48 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 35], 50.00th=[ 35], 60.00th=[ 52], | 70.00th=[ 91], 80.00th=[ 91], 90.00th=[ 259], 95.00th=[ 259], | 99.00th=[ 259], 99.50th=[ 259], 99.90th=[ 259], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min=10240, max=10240, per=34.43%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=50.00%, 100=25.00%, 500=8.33% cpu : usr=0.14%, sys=9.44%, ctx=30, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12559: Thu May 1 13:12:49 2025 read: IOPS=8, BW=8967KiB/s (9182kB/s)(5120KiB/571msec) clat (usec): min=12026, max=32133, avg=22601.56, stdev=7326.95 lat (usec): min=12028, max=32134, avg=22602.72, stdev=7326.85 clat percentiles (usec): | 1.00th=[11994], 5.00th=[11994], 10.00th=[11994], 20.00th=[11994], | 30.00th=[20317], 40.00th=[20317], 50.00th=[23987], 60.00th=[23987], | 70.00th=[24773], 80.00th=[24773], 90.00th=[32113], 95.00th=[32113], | 99.00th=[32113], 99.50th=[32113], 99.90th=[32113], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 8192, max= 8192, per=38.56%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.3MiB/s (12.9MB/s)(7168KiB/571msec) clat (msec): min=38, max=123, avg=63.03, stdev=31.09 lat (msec): min=38, max=123, avg=63.10, stdev=31.08 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 44], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 45], 60.00th=[ 64], | 70.00th=[ 64], 80.00th=[ 85], 90.00th=[ 124], 95.00th=[ 124], | 99.00th=[ 124], 99.50th=[ 124], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min=12288, max=12288, per=41.32%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=66.67%, 100=16.67%, 250=8.33% cpu : usr=0.18%, sys=13.33%, ctx=45, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=20.7MiB/s (21.8MB/s), 2306KiB/s-8967KiB/s (2362kB/s-9182kB/s), io=20.0MiB (20.0MB), run=571-964msec WRITE: bw=29.0MiB/s (30.5MB/s), 5311KiB/s-12.3MiB/s (5439kB/s-12.9MB/s), io=28.0MiB (29.4MB), run=571-964msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12572: Thu May 1 13:12:51 2025 read: IOPS=7, BW=30.8MiB/s (32.3MB/s)(12.0MiB/390msec) clat (msec): min=111, max=133, avg=120.91, stdev=11.03 lat (msec): min=111, max=133, avg=120.91, stdev=11.03 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 118], 50.00th=[ 118], 60.00th=[ 118], | 70.00th=[ 134], 80.00th=[ 134], 90.00th=[ 134], 95.00th=[ 134], | 99.00th=[ 134], 99.50th=[ 134], 99.90th=[ 134], 99.95th=[ 134], | 99.99th=[ 134] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=38.56%, ctx=1140, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12574: Thu May 1 13:12:51 2025 read: IOPS=1, BW=5810KiB/s (5949kB/s)(4096KiB/705msec) clat (nsec): min=135828k, max=135828k, avg=135828165.00, stdev= 0.00 lat (nsec): min=135829k, max=135829k, avg=135829170.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 136], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 8175, max= 8175, per=35.18%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=11.3MiB/s (11.9MB/s)(8192KiB/705msec) clat (msec): min=251, max=296, avg=274.24, stdev=31.66 lat (msec): min=252, max=296, avg=274.59, stdev=31.69 clat percentiles (msec): | 1.00th=[ 253], 5.00th=[ 253], 10.00th=[ 253], 20.00th=[ 253], | 30.00th=[ 253], 40.00th=[ 253], 50.00th=[ 253], 60.00th=[ 296], | 70.00th=[ 296], 80.00th=[ 296], 90.00th=[ 296], 95.00th=[ 296], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 8175, max= 8175, per=18.09%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.28%, sys=25.14%, ctx=513, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12576: Thu May 1 13:12:51 2025 write: IOPS=4, BW=16.6MiB/s (17.4MB/s)(12.0MiB/725msec) clat (msec): min=183, max=285, avg=236.61, stdev=51.38 lat (msec): min=183, max=285, avg=236.94, stdev=51.42 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 241], 50.00th=[ 241], 60.00th=[ 241], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 8192, max= 8192, per=18.13%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=38.12%, ctx=107, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12578: Thu May 1 13:12:51 2025 write: IOPS=5, BW=21.0MiB/s (22.0MB/s)(12.0MiB/571msec) clat (msec): min=170, max=206, avg=186.45, stdev=18.06 lat (msec): min=170, max=206, avg=186.76, stdev=18.11 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 184], | 70.00th=[ 207], 80.00th=[ 207], 90.00th=[ 207], 95.00th=[ 207], | 99.00th=[ 207], 99.50th=[ 207], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min=16384, max=16384, per=36.25%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=43.33%, ctx=100, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.7MiB/s (23.8MB/s), 5810KiB/s-30.8MiB/s (5949kB/s-32.3MB/s), io=16.0MiB (16.8MB), run=390-705msec WRITE: bw=44.1MiB/s (46.3MB/s), 11.3MiB/s-21.0MiB/s (11.9MB/s-22.0MB/s), io=32.0MiB (33.6MB), run=571-725msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12571: Thu May 1 13:12:51 2025 read: IOPS=11, BW=45.3MiB/s (47.5MB/s)(12.0MiB/265msec) clat (msec): min=31, max=144, avg=77.29, stdev=59.32 lat (msec): min=31, max=144, avg=77.29, stdev=59.32 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 56], 50.00th=[ 56], 60.00th=[ 56], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] lat (msec) : 50=33.33%, 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=25.76%, ctx=50, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12573: Thu May 1 13:12:51 2025 read: IOPS=1, BW=4989KiB/s (5109kB/s)(4096KiB/821msec) clat (nsec): min=192644k, max=192644k, avg=192644162.00, stdev= 0.00 lat (nsec): min=192647k, max=192647k, avg=192646810.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 192], 10.00th=[ 192], 20.00th=[ 192], | 30.00th=[ 192], 40.00th=[ 192], 50.00th=[ 192], 60.00th=[ 192], | 70.00th=[ 192], 80.00th=[ 192], 90.00th=[ 192], 95.00th=[ 192], | 99.00th=[ 192], 99.50th=[ 192], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 5927, max= 5927, per=29.70%, avg=5927.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9978KiB/s (10.2MB/s)(8192KiB/821msec) clat (msec): min=297, max=311, avg=304.53, stdev=10.10 lat (msec): min=297, max=312, avg=304.89, stdev=10.09 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 296], 10.00th=[ 296], 20.00th=[ 296], | 30.00th=[ 296], 40.00th=[ 296], 50.00th=[ 296], 60.00th=[ 313], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 313], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 5927, max= 5927, per=14.85%, avg=5927.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.24%, sys=10.61%, ctx=57, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12575: Thu May 1 13:12:51 2025 write: IOPS=4, BW=17.4MiB/s (18.3MB/s)(12.0MiB/688msec) clat (msec): min=172, max=277, avg=210.16, stdev=58.68 lat (msec): min=172, max=278, avg=210.50, stdev=58.68 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 180], 50.00th=[ 180], 60.00th=[ 180], | 70.00th=[ 279], 80.00th=[ 279], 90.00th=[ 279], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 8192, max= 8192, per=20.53%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.15%, sys=17.18%, ctx=83, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12577: Thu May 1 13:12:51 2025 write: IOPS=6, BW=25.4MiB/s (26.7MB/s)(12.0MiB/472msec) clat (msec): min=66, max=269, avg=150.98, stdev=105.70 lat (msec): min=66, max=269, avg=151.29, stdev=105.74 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 66], 10.00th=[ 66], 20.00th=[ 66], | 30.00th=[ 66], 40.00th=[ 118], 50.00th=[ 118], 60.00th=[ 118], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=15.92%, ctx=53, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.5MiB/s (20.4MB/s), 4989KiB/s-45.3MiB/s (5109kB/s-47.5MB/s), io=16.0MiB (16.8MB), run=265-821msec WRITE: bw=38.0MiB/s (40.9MB/s), 9978KiB/s-25.4MiB/s (10.2MB/s-26.7MB/s), io=32.0MiB (33.6MB), run=472-821msec