== sanity test 398b: DIO and buffer IO race ============== 14:13:22 (1744136002) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.24353 s, 40.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11184: Tue Apr 8 14:13:37 2025 read: IOPS=122, BW=491KiB/s (503kB/s)(5992KiB/12205msec) clat (usec): min=56, max=182375, avg=3820.44, stdev=5421.43 lat (usec): min=57, max=182376, avg=3821.19, stdev=5421.44 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 78], 10.00th=[ 92], 20.00th=[ 2024], | 30.00th=[ 2376], 40.00th=[ 2737], 50.00th=[ 3163], 60.00th=[ 3589], | 70.00th=[ 4686], 80.00th=[ 5800], 90.00th=[ 7111], 95.00th=[ 8356], | 99.00th=[ 12387], 99.50th=[ 15270], 99.90th=[ 34866], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 240, max= 736, per=24.98%, avg=485.58, stdev=103.75, samples=24 iops : min= 60, max= 184, avg=121.33, stdev=26.02, samples=24 write: IOPS=128, BW=516KiB/s (528kB/s)(6296KiB/12205msec) clat (usec): min=457, max=42624, avg=4033.92, stdev=3906.09 lat (usec): min=457, max=42625, avg=4035.04, stdev=3906.11 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 553], 10.00th=[ 586], 20.00th=[ 676], | 30.00th=[ 955], 40.00th=[ 1844], 50.00th=[ 3163], 60.00th=[ 4146], | 70.00th=[ 5211], 80.00th=[ 6718], 90.00th=[ 8979], 95.00th=[11207], | 99.00th=[17171], 99.50th=[19792], 99.90th=[33424], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 296, max= 816, per=25.04%, avg=509.58, stdev=109.97, samples=24 iops : min= 74, max= 204, avg=127.33, stdev=27.54, samples=24 lat (usec) : 100=5.47%, 250=2.38%, 500=0.20%, 750=11.98%, 1000=4.10% lat (msec) : 2=6.32%, 4=30.27%, 10=34.67%, 20=4.26%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=24.27%, ctx=5808, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11186: Tue Apr 8 14:13:37 2025 read: IOPS=120, BW=481KiB/s (492kB/s)(5940KiB/12351msec) clat (usec): min=52, max=165764, avg=3597.42, stdev=5070.00 lat (usec): min=52, max=165765, avg=3598.15, stdev=5070.11 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 71], 10.00th=[ 83], 20.00th=[ 1795], | 30.00th=[ 2245], 40.00th=[ 2606], 50.00th=[ 2933], 60.00th=[ 3425], | 70.00th=[ 4293], 80.00th=[ 5473], 90.00th=[ 6980], 95.00th=[ 8356], | 99.00th=[ 11863], 99.50th=[ 16319], 99.90th=[ 29230], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 16, max= 672, per=24.48%, avg=475.96, stdev=123.46, samples=24 iops : min= 4, max= 168, avg=118.96, stdev=30.86, samples=24 write: IOPS=128, BW=514KiB/s (526kB/s)(6348KiB/12351msec) clat (usec): min=428, max=45949, avg=4330.72, stdev=4509.52 lat (usec): min=429, max=45949, avg=4331.75, stdev=4509.50 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 545], 10.00th=[ 570], 20.00th=[ 676], | 30.00th=[ 922], 40.00th=[ 2008], 50.00th=[ 3392], 60.00th=[ 4293], | 70.00th=[ 5407], 80.00th=[ 6980], 90.00th=[ 9503], 95.00th=[12518], | 99.00th=[20841], 99.50th=[27657], 99.90th=[36963], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 56, max= 784, per=24.63%, avg=501.29, stdev=130.85, samples=24 iops : min= 14, max= 196, avg=125.29, stdev=32.73, samples=24 lat (usec) : 100=6.80%, 250=2.15%, 500=0.72%, 750=11.95%, 1000=3.97% lat (msec) : 2=6.54%, 4=29.43%, 10=32.62%, 20=5.18%, 50=0.62% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=23.89%, ctx=5552, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11188: Tue Apr 8 14:13:37 2025 read: IOPS=122, BW=491KiB/s (502kB/s)(6028KiB/12284msec) clat (usec): min=54, max=180236, avg=3828.25, stdev=5656.45 lat (usec): min=54, max=180237, avg=3828.87, stdev=5656.46 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 73], 10.00th=[ 86], 20.00th=[ 1909], | 30.00th=[ 2311], 40.00th=[ 2671], 50.00th=[ 3097], 60.00th=[ 3654], | 70.00th=[ 4555], 80.00th=[ 5735], 90.00th=[ 7177], 95.00th=[ 8586], | 99.00th=[ 13960], 99.50th=[ 20841], 99.90th=[ 58459], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 232, max= 680, per=24.94%, avg=484.92, stdev=106.68, samples=24 iops : min= 58, max= 170, avg=121.17, stdev=26.73, samples=24 write: IOPS=127, BW=510KiB/s (522kB/s)(6260KiB/12284msec) clat (usec): min=461, max=24985, avg=4077.64, stdev=3827.07 lat (usec): min=462, max=24986, avg=4078.40, stdev=3827.08 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 545], 10.00th=[ 578], 20.00th=[ 676], | 30.00th=[ 898], 40.00th=[ 1647], 50.00th=[ 3261], 60.00th=[ 4293], | 70.00th=[ 5342], 80.00th=[ 6849], 90.00th=[ 9241], 95.00th=[11469], | 99.00th=[16450], 99.50th=[19792], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 312, max= 664, per=24.57%, avg=499.92, stdev=92.44, samples=24 iops : min= 78, max= 166, avg=124.92, stdev=23.19, samples=24 lat (usec) : 100=6.38%, 250=2.47%, 500=0.29%, 750=12.17%, 1000=4.23% lat (msec) : 2=6.25%, 4=28.94%, 10=33.72%, 20=5.08%, 50=0.39% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.20%, sys=23.60%, ctx=5658, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11189: Tue Apr 8 14:13:37 2025 read: IOPS=122, BW=491KiB/s (503kB/s)(6052KiB/12316msec) clat (usec): min=52, max=65771, avg=3543.84, stdev=3324.86 lat (usec): min=53, max=65771, avg=3544.46, stdev=3324.87 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 74], 10.00th=[ 86], 20.00th=[ 1745], | 30.00th=[ 2278], 40.00th=[ 2573], 50.00th=[ 2966], 60.00th=[ 3490], | 70.00th=[ 4293], 80.00th=[ 5538], 90.00th=[ 6980], 95.00th=[ 8356], | 99.00th=[11600], 99.50th=[14877], 99.90th=[44303], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 160, max= 774, per=24.96%, avg=485.21, stdev=108.02, samples=24 iops : min= 40, max= 193, avg=121.25, stdev=26.97, samples=24 write: IOPS=126, BW=506KiB/s (518kB/s)(6236KiB/12316msec) clat (usec): min=440, max=199662, avg=4375.12, stdev=6455.15 lat (usec): min=440, max=199664, avg=4375.89, stdev=6455.21 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 553], 10.00th=[ 578], 20.00th=[ 668], | 30.00th=[ 947], 40.00th=[ 1991], 50.00th=[ 3359], 60.00th=[ 4359], | 70.00th=[ 5473], 80.00th=[ 7046], 90.00th=[ 9765], 95.00th=[ 11994], | 99.00th=[ 18220], 99.50th=[ 21627], 99.90th=[ 49021], 99.95th=[200279], | 99.99th=[200279] bw ( KiB/s): min= 192, max= 726, per=24.60%, avg=500.54, stdev=118.84, samples=24 iops : min= 48, max= 181, avg=125.08, stdev=29.67, samples=24 lat (usec) : 100=6.61%, 250=2.77%, 500=0.36%, 750=12.08%, 1000=3.45% lat (msec) : 2=6.32%, 4=30.11%, 10=32.49%, 20=5.40%, 50=0.36% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.20%, sys=23.54%, ctx=5546, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1944KiB/s (1991kB/s), 481KiB/s-491KiB/s (492kB/s-503kB/s), io=23.4MiB (24.6MB), run=12205-12351msec WRITE: bw=2035KiB/s (2084kB/s), 506KiB/s-516KiB/s (518kB/s-528kB/s), io=24.6MiB (25.7MB), run=12205-12351msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11182: Tue Apr 8 14:14:01 2025 read: IOPS=41, BW=166KiB/s (170kB/s)(5992KiB/36109msec) clat (usec): min=1685, max=184418, avg=4385.07, stdev=5573.19 lat (usec): min=1686, max=184419, avg=4386.12, stdev=5573.12 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2147], 10.00th=[ 2311], 20.00th=[ 2507], | 30.00th=[ 2671], 40.00th=[ 2868], 50.00th=[ 3097], 60.00th=[ 3589], | 70.00th=[ 4293], 80.00th=[ 5211], 90.00th=[ 7832], 95.00th=[ 10028], | 99.00th=[ 16450], 99.50th=[ 21365], 99.90th=[ 39060], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 64, max= 280, per=24.98%, avg=165.88, stdev=58.20, samples=72 iops : min= 16, max= 70, avg=41.39, stdev=14.52, samples=72 write: IOPS=43, BW=174KiB/s (179kB/s)(6296KiB/36109msec) clat (usec): min=9587, max=62931, avg=18677.49, stdev=6310.84 lat (usec): min=9588, max=62932, avg=18678.60, stdev=6310.83 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11469], 10.00th=[12256], 20.00th=[13435], | 30.00th=[14615], 40.00th=[16057], 50.00th=[17433], 60.00th=[19006], | 70.00th=[20317], 80.00th=[22152], 90.00th=[27395], 95.00th=[31589], | 99.00th=[38536], 99.50th=[42730], 99.90th=[60031], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 48, max= 232, per=25.00%, avg=173.99, stdev=44.08, samples=72 iops : min= 12, max= 58, avg=43.42, stdev=11.01, samples=72 lat (msec) : 2=0.68%, 4=31.05%, 10=14.88%, 20=36.72%, 50=16.50% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.13%, sys=7.02%, ctx=3806, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11183: Tue Apr 8 14:14:01 2025 read: IOPS=41, BW=166KiB/s (170kB/s)(5940KiB/35861msec) clat (usec): min=1684, max=165979, avg=4309.15, stdev=5242.85 lat (usec): min=1685, max=165981, avg=4310.02, stdev=5242.83 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2147], 10.00th=[ 2311], 20.00th=[ 2507], | 30.00th=[ 2671], 40.00th=[ 2868], 50.00th=[ 3130], 60.00th=[ 3589], | 70.00th=[ 4228], 80.00th=[ 5014], 90.00th=[ 7308], 95.00th=[ 9241], | 99.00th=[ 18220], 99.50th=[ 26084], 99.90th=[ 36963], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 16, max= 296, per=24.96%, avg=165.76, stdev=61.31, samples=71 iops : min= 4, max= 74, avg=41.37, stdev=15.31, samples=71 write: IOPS=44, BW=177KiB/s (181kB/s)(6348KiB/35861msec) clat (usec): min=9699, max=66139, avg=18475.26, stdev=6194.90 lat (usec): min=9700, max=66139, avg=18476.59, stdev=6194.79 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11731], 10.00th=[12387], 20.00th=[13435], | 30.00th=[14615], 40.00th=[15926], 50.00th=[17433], 60.00th=[18744], | 70.00th=[20055], 80.00th=[21890], 90.00th=[26608], 95.00th=[30540], | 99.00th=[39584], 99.50th=[43254], 99.90th=[62129], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 56, max= 232, per=25.37%, avg=176.58, stdev=44.39, samples=71 iops : min= 14, max= 58, avg=44.07, stdev=11.08, samples=71 lat (msec) : 2=0.91%, 4=31.12%, 10=14.42%, 20=37.96%, 50=15.46% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.14%, sys=7.06%, ctx=3794, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11185: Tue Apr 8 14:14:01 2025 read: IOPS=41, BW=167KiB/s (171kB/s)(6028KiB/36009msec) clat (usec): min=1675, max=49121, avg=4367.48, stdev=3050.88 lat (usec): min=1675, max=49122, avg=4368.39, stdev=3050.85 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2147], 10.00th=[ 2343], 20.00th=[ 2540], | 30.00th=[ 2737], 40.00th=[ 2933], 50.00th=[ 3261], 60.00th=[ 3851], | 70.00th=[ 4555], 80.00th=[ 5342], 90.00th=[ 7898], 95.00th=[10290], | 99.00th=[15533], 99.50th=[17957], 99.90th=[30016], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 39, max= 296, per=25.17%, avg=167.11, stdev=57.79, samples=71 iops : min= 9, max= 74, avg=41.70, stdev=14.45, samples=71 write: IOPS=43, BW=174KiB/s (178kB/s)(6260KiB/36009msec) clat (msec): min=9, max=178, avg=18.71, stdev= 7.58 lat (msec): min=9, max=178, avg=18.71, stdev= 7.58 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 27], 95.00th=[ 32], | 99.00th=[ 42], 99.50th=[ 47], 99.90th=[ 66], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 64, max= 239, per=24.87%, avg=173.08, stdev=45.60, samples=71 iops : min= 16, max= 59, avg=43.20, stdev=11.38, samples=71 lat (msec) : 2=0.78%, 4=29.92%, 10=16.02%, 20=37.63%, 50=15.49% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.12%, sys=7.10%, ctx=3785, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11187: Tue Apr 8 14:14:01 2025 read: IOPS=42, BW=170KiB/s (174kB/s)(6052KiB/35646msec) clat (usec): min=1608, max=49173, avg=4213.41, stdev=3189.79 lat (usec): min=1608, max=49174, avg=4214.40, stdev=3189.72 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2147], 10.00th=[ 2278], 20.00th=[ 2507], | 30.00th=[ 2638], 40.00th=[ 2835], 50.00th=[ 3032], 60.00th=[ 3490], | 70.00th=[ 4293], 80.00th=[ 5080], 90.00th=[ 7439], 95.00th=[ 9896], | 99.00th=[15401], 99.50th=[19268], 99.90th=[36963], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 32, max= 335, per=25.57%, avg=169.76, stdev=61.36, samples=71 iops : min= 8, max= 83, avg=42.32, stdev=15.35, samples=71 write: IOPS=43, BW=175KiB/s (179kB/s)(6236KiB/35646msec) clat (msec): min=8, max=190, avg=18.68, stdev= 7.69 lat (msec): min=8, max=190, avg=18.69, stdev= 7.69 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 28], 95.00th=[ 32], | 99.00th=[ 40], 99.50th=[ 43], 99.90th=[ 55], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 48, max= 224, per=25.07%, avg=174.49, stdev=44.44, samples=71 iops : min= 12, max= 56, avg=43.51, stdev=11.11, samples=71 lat (msec) : 2=0.91%, 4=31.51%, 10=14.62%, 20=36.26%, 50=16.63% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.13%, sys=7.15%, ctx=3795, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=665KiB/s (681kB/s), 166KiB/s-170KiB/s (170kB/s-174kB/s), io=23.4MiB (24.6MB), run=35646-36109msec WRITE: bw=696KiB/s (713kB/s), 174KiB/s-177KiB/s (178kB/s-181kB/s), io=24.6MiB (25.7MB), run=35646-36109msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11233: Tue Apr 8 14:14:05 2025 read: IOPS=131, BW=2109KiB/s (2160kB/s)(5920KiB/2807msec) clat (usec): min=56, max=36572, avg=3295.51, stdev=3386.34 lat (usec): min=57, max=36573, avg=3296.53, stdev=3386.43 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 72], 10.00th=[ 77], 20.00th=[ 91], | 30.00th=[ 147], 40.00th=[ 2507], 50.00th=[ 2999], 60.00th=[ 3621], | 70.00th=[ 4555], 80.00th=[ 5407], 90.00th=[ 6652], 95.00th=[ 8094], | 99.00th=[15270], 99.50th=[18220], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 1088, max= 2624, per=24.30%, avg=2073.60, stdev=596.27, samples=5 iops : min= 68, max= 164, avg=129.60, stdev=37.27, samples=5 write: IOPS=141, BW=2269KiB/s (2323kB/s)(6368KiB/2807msec) clat (usec): min=653, max=38697, avg=3645.03, stdev=4061.15 lat (usec): min=653, max=38698, avg=3650.23, stdev=4061.80 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 750], 10.00th=[ 791], 20.00th=[ 889], | 30.00th=[ 996], 40.00th=[ 1270], 50.00th=[ 2040], 60.00th=[ 3195], | 70.00th=[ 4555], 80.00th=[ 5932], 90.00th=[ 8291], 95.00th=[10683], | 99.00th=[21890], 99.50th=[31851], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 1408, max= 2912, per=25.10%, avg=2252.80, stdev=565.05, samples=5 iops : min= 88, max= 182, avg=140.80, stdev=35.32, samples=5 lat (usec) : 100=11.59%, 250=3.52%, 500=0.39%, 750=2.47%, 1000=13.15% lat (msec) : 2=10.16%, 4=23.31%, 10=31.25%, 20=3.52%, 50=0.65% cpu : usr=0.25%, sys=26.05%, ctx=1426, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11234: Tue Apr 8 14:14:05 2025 read: IOPS=144, BW=2315KiB/s (2371kB/s)(5744KiB/2481msec) clat (usec): min=62, max=44466, avg=3421.21, stdev=3498.61 lat (usec): min=62, max=44467, avg=3423.04, stdev=3499.59 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 73], 10.00th=[ 82], 20.00th=[ 95], | 30.00th=[ 2089], 40.00th=[ 2704], 50.00th=[ 3163], 60.00th=[ 3851], | 70.00th=[ 4490], 80.00th=[ 5538], 90.00th=[ 6915], 95.00th=[ 7963], | 99.00th=[11338], 99.50th=[17433], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 1824, max= 2688, per=27.75%, avg=2368.00, stdev=377.73, samples=4 iops : min= 114, max= 168, avg=148.00, stdev=23.61, samples=4 write: IOPS=164, BW=2638KiB/s (2701kB/s)(6544KiB/2481msec) clat (usec): min=657, max=16604, avg=3014.69, stdev=2855.67 lat (usec): min=657, max=16605, avg=3015.64, stdev=2855.69 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 750], 10.00th=[ 783], 20.00th=[ 848], | 30.00th=[ 930], 40.00th=[ 1090], 50.00th=[ 1582], 60.00th=[ 2835], | 70.00th=[ 3752], 80.00th=[ 5080], 90.00th=[ 7504], 95.00th=[ 8979], | 99.00th=[11600], 99.50th=[13698], 99.90th=[16581], 99.95th=[16581], | 99.99th=[16581] bw ( KiB/s): min= 2144, max= 3008, per=28.87%, avg=2592.00, stdev=364.86, samples=4 iops : min= 134, max= 188, avg=162.00, stdev=22.80, samples=4 lat (usec) : 100=10.03%, 250=3.12%, 500=0.52%, 750=2.60%, 1000=15.76% lat (msec) : 2=10.94%, 4=25.26%, 10=29.17%, 20=2.47%, 50=0.13% cpu : usr=0.28%, sys=29.31%, ctx=1419, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11235: Tue Apr 8 14:14:05 2025 read: IOPS=154, BW=2479KiB/s (2538kB/s)(6256KiB/2524msec) clat (usec): min=57, max=28428, avg=2976.47, stdev=3139.45 lat (usec): min=57, max=28428, avg=2977.10, stdev=3139.47 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 67], 10.00th=[ 73], 20.00th=[ 86], | 30.00th=[ 109], 40.00th=[ 2311], 50.00th=[ 2737], 60.00th=[ 3294], | 70.00th=[ 4146], 80.00th=[ 5080], 90.00th=[ 6259], 95.00th=[ 7242], | 99.00th=[12518], 99.50th=[24249], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 1728, max= 3008, per=29.18%, avg=2489.60, stdev=480.21, samples=5 iops : min= 108, max= 188, avg=155.60, stdev=30.01, samples=5 write: IOPS=149, BW=2390KiB/s (2447kB/s)(6032KiB/2524msec) clat (usec): min=655, max=42619, avg=3316.46, stdev=3695.31 lat (usec): min=656, max=42620, avg=3317.33, stdev=3695.32 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 775], 10.00th=[ 816], 20.00th=[ 889], | 30.00th=[ 1020], 40.00th=[ 1221], 50.00th=[ 1876], 60.00th=[ 2999], | 70.00th=[ 4113], 80.00th=[ 5407], 90.00th=[ 7504], 95.00th=[ 8717], | 99.00th=[16450], 99.50th=[23462], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 2048, max= 2720, per=26.38%, avg=2368.00, stdev=287.11, samples=5 iops : min= 128, max= 170, avg=148.00, stdev=17.94, samples=5 lat (usec) : 100=13.67%, 250=3.26%, 500=0.91%, 750=1.43%, 1000=13.28% lat (msec) : 2=10.55%, 4=25.52%, 10=28.91%, 20=1.82%, 50=0.65% cpu : usr=0.36%, sys=27.67%, ctx=1318, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11236: Tue Apr 8 14:14:05 2025 read: IOPS=155, BW=2480KiB/s (2540kB/s)(6032KiB/2432msec) clat (usec): min=63, max=45270, avg=3111.73, stdev=3341.98 lat (usec): min=63, max=45270, avg=3113.08, stdev=3341.85 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 73], 10.00th=[ 80], 20.00th=[ 89], | 30.00th=[ 375], 40.00th=[ 2507], 50.00th=[ 2933], 60.00th=[ 3392], | 70.00th=[ 4047], 80.00th=[ 5080], 90.00th=[ 6521], 95.00th=[ 7504], | 99.00th=[11076], 99.50th=[14222], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 1664, max= 2810, per=28.86%, avg=2462.50, stdev=536.07, samples=4 iops : min= 104, max= 175, avg=153.75, stdev=33.37, samples=4 write: IOPS=160, BW=2572KiB/s (2634kB/s)(6256KiB/2432msec) clat (usec): min=658, max=16468, avg=2880.27, stdev=2820.89 lat (usec): min=659, max=16468, avg=2881.87, stdev=2821.01 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 742], 10.00th=[ 783], 20.00th=[ 857], | 30.00th=[ 930], 40.00th=[ 1057], 50.00th=[ 1287], 60.00th=[ 2573], | 70.00th=[ 3261], 80.00th=[ 5276], 90.00th=[ 7308], 95.00th=[ 9110], | 99.00th=[10814], 99.50th=[16188], 99.90th=[16450], 99.95th=[16450], | 99.99th=[16450] bw ( KiB/s): min= 1824, max= 3168, per=29.22%, avg=2622.75, stdev=615.42, samples=4 iops : min= 114, max= 198, avg=163.75, stdev=38.53, samples=4 lat (usec) : 100=11.85%, 250=2.73%, 500=0.52%, 750=2.99%, 1000=15.10% lat (msec) : 2=11.85%, 4=27.60%, 10=25.91%, 20=1.30%, 50=0.13% cpu : usr=0.29%, sys=28.88%, ctx=1314, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8533KiB/s (8738kB/s), 2109KiB/s-2480KiB/s (2160kB/s-2540kB/s), io=23.4MiB (24.5MB), run=2432-2807msec WRITE: bw=8978KiB/s (9193kB/s), 2269KiB/s-2638KiB/s (2323kB/s-2701kB/s), io=24.6MiB (25.8MB), run=2432-2807msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11237: Tue Apr 8 14:14:11 2025 read: IOPS=43, BW=699KiB/s (716kB/s)(5920KiB/8472msec) clat (usec): min=1806, max=44727, avg=3784.60, stdev=4052.66 lat (usec): min=1807, max=44728, avg=3785.41, stdev=4052.77 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2008], 10.00th=[ 2073], 20.00th=[ 2180], | 30.00th=[ 2245], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2540], | 70.00th=[ 2835], 80.00th=[ 3687], 90.00th=[ 7046], 95.00th=[11338], | 99.00th=[21627], 99.50th=[27657], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 382, max= 1152, per=24.45%, avg=681.44, stdev=233.07, samples=16 iops : min= 23, max= 72, avg=42.31, stdev=14.76, samples=16 write: IOPS=46, BW=752KiB/s (770kB/s)(6368KiB/8472msec) clat (usec): min=10370, max=76415, avg=17709.60, stdev=6097.29 lat (usec): min=10371, max=76417, avg=17710.95, stdev=6097.34 clat percentiles (usec): | 1.00th=[10814], 5.00th=[11469], 10.00th=[11863], 20.00th=[12911], | 30.00th=[13829], 40.00th=[15664], 50.00th=[16909], 60.00th=[18220], | 70.00th=[19006], 80.00th=[20579], 90.00th=[24773], 95.00th=[28181], | 99.00th=[38536], 99.50th=[39584], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 350, max= 928, per=25.15%, avg=737.44, stdev=201.14, samples=16 iops : min= 21, max= 58, avg=45.81, stdev=12.76, samples=16 lat (msec) : 2=1.30%, 4=38.41%, 10=5.47%, 20=42.71%, 50=11.98% lat (msec) : 100=0.13% cpu : usr=0.05%, sys=6.74%, ctx=896, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11238: Tue Apr 8 14:14:11 2025 read: IOPS=41, BW=668KiB/s (684kB/s)(5744KiB/8593msec) clat (usec): min=1723, max=24424, avg=3675.81, stdev=3061.73 lat (usec): min=1724, max=24427, avg=3676.53, stdev=3061.76 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2008], 10.00th=[ 2057], 20.00th=[ 2147], | 30.00th=[ 2245], 40.00th=[ 2343], 50.00th=[ 2442], 60.00th=[ 2606], | 70.00th=[ 2999], 80.00th=[ 4424], 90.00th=[ 7570], 95.00th=[ 9765], | 99.00th=[17171], 99.50th=[17695], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 287, max= 1056, per=23.97%, avg=668.00, stdev=212.94, samples=17 iops : min= 17, max= 66, avg=41.65, stdev=13.42, samples=17 write: IOPS=47, BW=762KiB/s (780kB/s)(6544KiB/8593msec) clat (usec): min=10288, max=48577, avg=17659.02, stdev=5416.13 lat (usec): min=10289, max=48580, avg=17660.54, stdev=5416.53 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11338], 10.00th=[11863], 20.00th=[12911], | 30.00th=[14222], 40.00th=[16057], 50.00th=[17171], 60.00th=[17957], | 70.00th=[19268], 80.00th=[20317], 90.00th=[24249], 95.00th=[28705], | 99.00th=[34341], 99.50th=[40633], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 447, max= 956, per=25.93%, avg=760.18, stdev=174.93, samples=17 iops : min= 27, max= 59, avg=47.41, stdev=10.99, samples=17 lat (msec) : 2=2.21%, 4=34.24%, 10=8.07%, 20=42.97%, 50=12.50% cpu : usr=0.14%, sys=6.35%, ctx=876, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11239: Tue Apr 8 14:14:11 2025 read: IOPS=47, BW=759KiB/s (777kB/s)(6256KiB/8245msec) clat (usec): min=1830, max=27735, avg=3369.16, stdev=2959.80 lat (usec): min=1831, max=27736, avg=3369.91, stdev=2959.78 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2024], 10.00th=[ 2057], 20.00th=[ 2147], | 30.00th=[ 2212], 40.00th=[ 2278], 50.00th=[ 2376], 60.00th=[ 2507], | 70.00th=[ 2737], 80.00th=[ 3163], 90.00th=[ 5997], 95.00th=[ 9241], | 99.00th=[15664], 99.50th=[27132], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 320, max= 1312, per=27.40%, avg=763.75, stdev=324.33, samples=16 iops : min= 20, max= 82, avg=47.56, stdev=20.40, samples=16 write: IOPS=45, BW=732KiB/s (749kB/s)(6032KiB/8245msec) clat (usec): min=10651, max=46664, avg=17991.42, stdev=5874.37 lat (usec): min=10652, max=46669, avg=17992.84, stdev=5874.42 clat percentiles (usec): | 1.00th=[10945], 5.00th=[11600], 10.00th=[11994], 20.00th=[12649], | 30.00th=[14091], 40.00th=[15795], 50.00th=[17433], 60.00th=[18482], | 70.00th=[19530], 80.00th=[20841], 90.00th=[25560], 95.00th=[30278], | 99.00th=[38536], 99.50th=[45876], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 351, max= 928, per=24.75%, avg=725.69, stdev=179.93, samples=16 iops : min= 21, max= 58, avg=45.19, stdev=11.39, samples=16 lat (msec) : 2=2.21%, 4=40.62%, 10=5.73%, 20=38.93%, 50=12.50% cpu : usr=0.06%, sys=6.70%, ctx=900, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11240: Tue Apr 8 14:14:11 2025 read: IOPS=45, BW=722KiB/s (740kB/s)(6032KiB/8349msec) clat (usec): min=1798, max=19924, avg=3278.79, stdev=2522.46 lat (usec): min=1799, max=19924, avg=3279.75, stdev=2522.51 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2008], 10.00th=[ 2057], 20.00th=[ 2147], | 30.00th=[ 2212], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2540], | 70.00th=[ 2737], 80.00th=[ 3326], 90.00th=[ 5866], 95.00th=[ 9241], | 99.00th=[15270], 99.50th=[19268], 99.90th=[20055], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 320, max= 1120, per=25.55%, avg=712.00, stdev=235.01, samples=16 iops : min= 20, max= 70, avg=44.50, stdev=14.69, samples=16 write: IOPS=46, BW=749KiB/s (767kB/s)(6256KiB/8349msec) clat (usec): min=10479, max=46334, avg=17830.49, stdev=5701.56 lat (usec): min=10481, max=46338, avg=17831.94, stdev=5701.66 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11207], 10.00th=[11600], 20.00th=[12649], | 30.00th=[14091], 40.00th=[15664], 50.00th=[17171], 60.00th=[18482], | 70.00th=[19792], 80.00th=[21103], 90.00th=[25035], 95.00th=[29492], | 99.00th=[37487], 99.50th=[45351], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 384, max= 928, per=25.24%, avg=740.00, stdev=185.44, samples=16 iops : min= 24, max= 58, avg=46.25, stdev=11.59, samples=16 lat (msec) : 2=2.34%, 4=40.23%, 10=4.43%, 20=38.93%, 50=14.06% cpu : usr=0.04%, sys=6.64%, ctx=892, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2787KiB/s (2854kB/s), 668KiB/s-759KiB/s (684kB/s-777kB/s), io=23.4MiB (24.5MB), run=8245-8593msec WRITE: bw=2933KiB/s (3003kB/s), 732KiB/s-762KiB/s (749kB/s-780kB/s), io=24.6MiB (25.8MB), run=8245-8593msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11252: Tue Apr 8 14:14:13 2025 read: IOPS=5, BW=5259KiB/s (5385kB/s)(7168KiB/1363msec) clat (msec): min=16, max=867, avg=148.98, stdev=317.02 lat (msec): min=16, max=867, avg=148.98, stdev=317.02 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 19], | 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 27], 60.00th=[ 42], | 70.00th=[ 42], 80.00th=[ 53], 90.00th=[ 869], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 2043, max= 6144, per=30.74%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 write: IOPS=3, BW=3756KiB/s (3847kB/s)(5120KiB/1363msec) clat (usec): min=38295, max=99514, avg=61312.58, stdev=26451.85 lat (usec): min=38369, max=99600, avg=61379.64, stdev=26463.03 clat percentiles (usec): | 1.00th=[38536], 5.00th=[38536], 10.00th=[38536], 20.00th=[38536], | 30.00th=[40109], 40.00th=[40109], 50.00th=[51119], 60.00th=[51119], | 70.00th=[77071], 80.00th=[77071], 90.00th=[99091], 95.00th=[99091], | 99.00th=[99091], 99.50th=[99091], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 2048, max= 2048, per=10.99%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=33.33%, 100=33.33%, 1000=8.33% cpu : usr=0.00%, sys=7.56%, ctx=61, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11253: Tue Apr 8 14:14:13 2025 read: IOPS=1, BW=1376KiB/s (1409kB/s)(2048KiB/1488msec) clat (usec): min=17766, max=49380, avg=33573.07, stdev=22354.44 lat (usec): min=17768, max=49380, avg=33574.73, stdev=22353.06 clat percentiles (usec): | 1.00th=[17695], 5.00th=[17695], 10.00th=[17695], 20.00th=[17695], | 30.00th=[17695], 40.00th=[17695], 50.00th=[17695], 60.00th=[49546], | 70.00th=[49546], 80.00th=[49546], 90.00th=[49546], 95.00th=[49546], | 99.00th=[49546], 99.50th=[49546], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 2048, max= 2048, per=15.38%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6882KiB/s (7047kB/s)(10.0MiB/1488msec) clat (msec): min=23, max=868, avg=140.58, stdev=257.12 lat (msec): min=23, max=868, avg=140.67, stdev=257.12 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 38], | 30.00th=[ 42], 40.00th=[ 47], 50.00th=[ 55], 60.00th=[ 65], | 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 114], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 4096, max= 4096, per=21.97%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=33.33%, 250=8.33%, 1000=8.33% cpu : usr=0.20%, sys=7.20%, ctx=64, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11254: Tue Apr 8 14:14:13 2025 read: IOPS=3, BW=4016KiB/s (4112kB/s)(6144KiB/1530msec) clat (usec): min=14024, max=86797, avg=38099.77, stdev=26706.89 lat (usec): min=14025, max=86798, avg=38100.64, stdev=26706.76 clat percentiles (usec): | 1.00th=[13960], 5.00th=[13960], 10.00th=[13960], 20.00th=[14353], | 30.00th=[14353], 40.00th=[33162], 50.00th=[33162], 60.00th=[37487], | 70.00th=[43254], 80.00th=[43254], 90.00th=[86508], 95.00th=[86508], | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min=10240, max=10240, per=76.91%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=3, BW=4016KiB/s (4112kB/s)(6144KiB/1530msec) clat (msec): min=25, max=934, avg=211.91, stdev=355.83 lat (msec): min=25, max=934, avg=211.96, stdev=355.84 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 95], | 70.00th=[ 111], 80.00th=[ 111], 90.00th=[ 936], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 4096, max= 8192, per=32.96%, avg=6144.00, stdev=2896.31, samples=2 iops : min= 4, max= 8, avg= 6.00, stdev= 2.83, samples=2 lat (msec) : 20=16.67%, 50=41.67%, 100=25.00%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=4.58%, ctx=44, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11255: Tue Apr 8 14:14:13 2025 read: IOPS=3, BW=3329KiB/s (3409kB/s)(5120KiB/1538msec) clat (usec): min=22873, max=92650, avg=50727.70, stdev=33351.34 lat (usec): min=22874, max=92651, avg=50728.93, stdev=33350.99 clat percentiles (usec): | 1.00th=[22938], 5.00th=[22938], 10.00th=[22938], 20.00th=[22938], | 30.00th=[26084], 40.00th=[26084], 50.00th=[31065], 60.00th=[31065], | 70.00th=[81265], 80.00th=[81265], 90.00th=[92799], 95.00th=[92799], | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 8192, max= 8192, per=61.52%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4661KiB/s (4772kB/s)(7168KiB/1538msec) clat (msec): min=29, max=913, avg=177.49, stdev=325.76 lat (msec): min=29, max=913, avg=177.56, stdev=325.76 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 31], | 30.00th=[ 38], 40.00th=[ 38], 50.00th=[ 50], 60.00th=[ 78], | 70.00th=[ 78], 80.00th=[ 106], 90.00th=[ 911], 95.00th=[ 911], | 99.00th=[ 911], 99.50th=[ 911], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 2048, max=12288, per=38.45%, avg=7168.00, stdev=7240.77, samples=2 iops : min= 2, max= 12, avg= 7.00, stdev= 7.07, samples=2 lat (msec) : 50=58.33%, 100=25.00%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=5.92%, ctx=34, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.0MiB/s (13.6MB/s), 1376KiB/s-5259KiB/s (1409kB/s-5385kB/s), io=20.0MiB (20.0MB), run=1363-1538msec WRITE: bw=18.2MiB/s (19.1MB/s), 3756KiB/s-6882KiB/s (3847kB/s-7047kB/s), io=28.0MiB (29.4MB), run=1363-1538msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11256: Tue Apr 8 14:14:13 2025 read: IOPS=5, BW=5649KiB/s (5784kB/s)(7168KiB/1269msec) clat (usec): min=552, max=815918, avg=140327.03, stdev=298575.32 lat (usec): min=554, max=815919, avg=140328.35, stdev=298575.15 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 553], 10.00th=[ 553], 20.00th=[ 611], | 30.00th=[ 35390], 40.00th=[ 35390], 50.00th=[ 38536], 60.00th=[ 39060], | 70.00th=[ 39060], 80.00th=[ 52691], 90.00th=[817890], 95.00th=[817890], | 99.00th=[817890], 99.50th=[817890], 99.90th=[817890], 99.95th=[817890], | 99.99th=[817890] bw ( KiB/s): min= 2048, max= 4096, per=23.70%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 write: IOPS=3, BW=4035KiB/s (4132kB/s)(5120KiB/1269msec) clat (usec): min=34444, max=53522, avg=45821.33, stdev=8688.26 lat (usec): min=34524, max=53592, avg=45877.36, stdev=8695.01 clat percentiles (usec): | 1.00th=[34341], 5.00th=[34341], 10.00th=[34341], 20.00th=[34341], | 30.00th=[39060], 40.00th=[39060], 50.00th=[48497], 60.00th=[48497], | 70.00th=[53216], 80.00th=[53216], 90.00th=[53740], 95.00th=[53740], | 99.00th=[53740], 99.50th=[53740], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] lat (usec) : 750=16.67% lat (msec) : 50=50.00%, 100=25.00%, 1000=8.33% cpu : usr=0.00%, sys=24.84%, ctx=748, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11257: Tue Apr 8 14:14:13 2025 read: IOPS=1, BW=1364KiB/s (1396kB/s)(2048KiB/1502msec) clat (usec): min=29899, max=37817, avg=33858.66, stdev=5599.16 lat (usec): min=29900, max=37819, avg=33859.80, stdev=5599.86 clat percentiles (usec): | 1.00th=[30016], 5.00th=[30016], 10.00th=[30016], 20.00th=[30016], | 30.00th=[30016], 40.00th=[30016], 50.00th=[30016], 60.00th=[38011], | 70.00th=[38011], 80.00th=[38011], 90.00th=[38011], 95.00th=[38011], | 99.00th=[38011], 99.50th=[38011], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 2048, max= 2048, per=15.80%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=6, BW=6818KiB/s (6981kB/s)(10.0MiB/1502msec) clat (msec): min=29, max=859, avg=141.21, stdev=254.71 lat (msec): min=29, max=859, avg=141.26, stdev=254.70 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 32], | 30.00th=[ 34], 40.00th=[ 36], 50.00th=[ 42], 60.00th=[ 57], | 70.00th=[ 104], 80.00th=[ 104], 90.00th=[ 126], 95.00th=[ 860], | 99.00th=[ 860], 99.50th=[ 860], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 4096, max=14336, per=50.79%, avg=9216.00, stdev=7240.77, samples=2 iops : min= 4, max= 14, avg= 9.00, stdev= 7.07, samples=2 lat (msec) : 50=58.33%, 100=16.67%, 250=16.67%, 1000=8.33% cpu : usr=0.00%, sys=17.12%, ctx=407, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11258: Tue Apr 8 14:14:13 2025 read: IOPS=3, BW=3889KiB/s (3982kB/s)(6144KiB/1580msec) clat (usec): min=25143, max=90034, avg=49515.42, stdev=26755.50 lat (usec): min=25144, max=90035, avg=49516.42, stdev=26755.37 clat percentiles (usec): | 1.00th=[25035], 5.00th=[25035], 10.00th=[25035], 20.00th=[30278], | 30.00th=[30278], 40.00th=[37487], 50.00th=[37487], 60.00th=[38011], | 70.00th=[76022], 80.00th=[76022], 90.00th=[89654], 95.00th=[89654], | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 8192, max= 8192, per=63.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3889KiB/s (3982kB/s)(6144KiB/1580msec) clat (msec): min=19, max=934, avg=208.27, stdev=359.34 lat (msec): min=19, max=934, avg=208.30, stdev=359.35 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 26], | 30.00th=[ 26], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 88], | 70.00th=[ 150], 80.00th=[ 150], 90.00th=[ 936], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 4096, max= 6144, per=28.22%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 20=8.33%, 50=50.00%, 100=25.00%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=14.12%, ctx=603, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11259: Tue Apr 8 14:14:13 2025 read: IOPS=3, BW=3464KiB/s (3547kB/s)(5120KiB/1478msec) clat (usec): min=353, max=31340, avg=15476.59, stdev=15045.59 lat (usec): min=354, max=31341, avg=15477.36, stdev=15045.59 clat percentiles (usec): | 1.00th=[ 355], 5.00th=[ 355], 10.00th=[ 355], 20.00th=[ 355], | 30.00th=[ 445], 40.00th=[ 445], 50.00th=[15664], 60.00th=[15664], | 70.00th=[29492], 80.00th=[29492], 90.00th=[31327], 95.00th=[31327], | 99.00th=[31327], 99.50th=[31327], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] write: IOPS=4, BW=4850KiB/s (4966kB/s)(7168KiB/1478msec) clat (msec): min=26, max=956, avg=189.11, stdev=339.15 lat (msec): min=26, max=956, avg=189.28, stdev=339.11 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 41], | 30.00th=[ 55], 40.00th=[ 55], 50.00th=[ 77], 60.00th=[ 80], | 70.00th=[ 80], 80.00th=[ 91], 90.00th=[ 961], 95.00th=[ 961], | 99.00th=[ 961], 99.50th=[ 961], 99.90th=[ 961], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 2048, max= 2048, per=11.29%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 20=8.33%, 50=33.33%, 100=33.33%, 1000=8.33% cpu : usr=0.00%, sys=11.17%, ctx=320, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.7MiB/s (13.3MB/s), 1364KiB/s-5649KiB/s (1396kB/s-5784kB/s), io=20.0MiB (20.0MB), run=1269-1580msec WRITE: bw=17.7MiB/s (18.6MB/s), 3889KiB/s-6818KiB/s (3982kB/s-6981kB/s), io=28.0MiB (29.4MB), run=1269-1580msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11270: Tue Apr 8 14:14:15 2025 read: IOPS=7, BW=28.3MiB/s (29.7MB/s)(12.0MiB/424msec) clat (msec): min=68, max=181, avg=134.18, stdev=58.52 lat (msec): min=68, max=181, avg=134.19, stdev=58.52 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 69], 10.00th=[ 69], 20.00th=[ 69], | 30.00th=[ 69], 40.00th=[ 153], 50.00th=[ 153], 60.00th=[ 153], | 70.00th=[ 182], 80.00th=[ 182], 90.00th=[ 182], 95.00th=[ 182], | 99.00th=[ 182], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=24.82%, ctx=1322, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11271: Tue Apr 8 14:14:15 2025 read: IOPS=1, BW=7199KiB/s (7371kB/s)(4096KiB/569msec) clat (nsec): min=187186k, max=187186k, avg=187185602.00, stdev= 0.00 lat (nsec): min=187187k, max=187187k, avg=187186733.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 188], 10.00th=[ 188], 20.00th=[ 188], | 30.00th=[ 188], 40.00th=[ 188], 50.00th=[ 188], 60.00th=[ 188], | 70.00th=[ 188], 80.00th=[ 188], 90.00th=[ 188], 95.00th=[ 188], | 99.00th=[ 188], 99.50th=[ 188], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 8192, max= 8192, per=28.45%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=14.1MiB/s (14.7MB/s)(8192KiB/569msec) clat (msec): min=135, max=224, avg=179.87, stdev=62.46 lat (msec): min=136, max=224, avg=180.12, stdev=62.32 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 224], | 70.00th=[ 224], 80.00th=[ 224], 90.00th=[ 224], 95.00th=[ 224], | 99.00th=[ 224], 99.50th=[ 224], 99.90th=[ 224], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 8192, max= 8192, per=16.35%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=32.04%, ctx=620, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11272: Tue Apr 8 14:14:15 2025 write: IOPS=4, BW=18.3MiB/s (19.2MB/s)(12.0MiB/654msec) clat (msec): min=172, max=236, avg=213.60, stdev=35.68 lat (msec): min=172, max=236, avg=213.89, stdev=35.69 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 232], 50.00th=[ 232], 60.00th=[ 232], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 236], 95.00th=[ 236], | 99.00th=[ 236], 99.50th=[ 236], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min=16384, max=16384, per=32.70%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=36.24%, ctx=43, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11274: Tue Apr 8 14:14:15 2025 write: IOPS=5, BW=22.4MiB/s (23.5MB/s)(12.0MiB/535msec) clat (msec): min=138, max=191, avg=170.30, stdev=28.31 lat (msec): min=138, max=191, avg=170.80, stdev=28.37 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 138], 10.00th=[ 138], 20.00th=[ 138], | 30.00th=[ 138], 40.00th=[ 182], 50.00th=[ 182], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 192], 90.00th=[ 192], 95.00th=[ 192], | 99.00th=[ 192], 99.50th=[ 192], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min=16384, max=16384, per=32.70%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=44.38%, ctx=118, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=28.1MiB/s (29.5MB/s), 7199KiB/s-28.3MiB/s (7371kB/s-29.7MB/s), io=16.0MiB (16.8MB), run=424-569msec WRITE: bw=48.9MiB/s (51.3MB/s), 14.1MiB/s-22.4MiB/s (14.7MB/s-23.5MB/s), io=32.0MiB (33.6MB), run=535-654msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11265: Tue Apr 8 14:14:15 2025 read: IOPS=6, BW=27.1MiB/s (28.4MB/s)(12.0MiB/443msec) clat (msec): min=85, max=197, avg=138.21, stdev=55.86 lat (msec): min=85, max=197, avg=138.21, stdev=55.86 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 86], 10.00th=[ 86], 20.00th=[ 86], | 30.00th=[ 86], 40.00th=[ 132], 50.00th=[ 132], 60.00th=[ 132], | 70.00th=[ 197], 80.00th=[ 197], 90.00th=[ 197], 95.00th=[ 197], | 99.00th=[ 197], 99.50th=[ 197], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.45%, sys=16.52%, ctx=68, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11266: Tue Apr 8 14:14:15 2025 read: IOPS=3, BW=13.2MiB/s (13.8MB/s)(4096KiB/304msec) clat (nsec): min=35506k, max=35506k, avg=35505595.00, stdev= 0.00 lat (nsec): min=35508k, max=35508k, avg=35507707.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[35390], 5.00th=[35390], 10.00th=[35390], 20.00th=[35390], | 30.00th=[35390], 40.00th=[35390], 50.00th=[35390], 60.00th=[35390], | 70.00th=[35390], 80.00th=[35390], 90.00th=[35390], 95.00th=[35390], | 99.00th=[35390], 99.50th=[35390], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] write: IOPS=6, BW=26.3MiB/s (27.6MB/s)(8192KiB/304msec) clat (msec): min=102, max=127, avg=115.15, stdev=18.05 lat (msec): min=102, max=128, avg=115.30, stdev=18.07 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 103], 10.00th=[ 103], 20.00th=[ 103], | 30.00th=[ 103], 40.00th=[ 103], 50.00th=[ 103], 60.00th=[ 128], | 70.00th=[ 128], 80.00th=[ 128], 90.00th=[ 128], 95.00th=[ 128], | 99.00th=[ 128], 99.50th=[ 128], 99.90th=[ 128], 99.95th=[ 128], | 99.99th=[ 128] lat (msec) : 50=33.33%, 250=66.67% cpu : usr=0.00%, sys=27.39%, ctx=66, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11267: Tue Apr 8 14:14:15 2025 write: IOPS=3, BW=14.1MiB/s (14.8MB/s)(12.0MiB/852msec) clat (msec): min=143, max=469, avg=277.34, stdev=170.70 lat (msec): min=143, max=469, avg=277.54, stdev=170.64 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 220], 50.00th=[ 220], 60.00th=[ 220], | 70.00th=[ 468], 80.00th=[ 468], 90.00th=[ 468], 95.00th=[ 468], | 99.00th=[ 468], 99.50th=[ 468], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min=15693, max=15693, per=40.80%, avg=15693.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.12%, sys=11.63%, ctx=140, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11268: Tue Apr 8 14:14:15 2025 write: IOPS=12, BW=50.4MiB/s (52.9MB/s)(12.0MiB/238msec) clat (usec): min=48736, max=83306, avg=66659.53, stdev=17320.28 lat (usec): min=49005, max=83697, avg=66981.34, stdev=17380.16 clat percentiles (usec): | 1.00th=[48497], 5.00th=[48497], 10.00th=[48497], 20.00th=[48497], | 30.00th=[48497], 40.00th=[67634], 50.00th=[67634], 60.00th=[67634], | 70.00th=[83362], 80.00th=[83362], 90.00th=[83362], 95.00th=[83362], | 99.00th=[83362], 99.50th=[83362], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] lat (msec) : 50=33.33%, 100=66.67% cpu : usr=0.42%, sys=37.55%, ctx=64, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=36.1MiB/s (37.9MB/s), 13.2MiB/s-27.1MiB/s (13.8MB/s-28.4MB/s), io=16.0MiB (16.8MB), run=304-443msec WRITE: bw=37.6MiB/s (39.4MB/s), 14.1MiB/s-50.4MiB/s (14.8MB/s-52.9MB/s), io=32.0MiB (33.6MB), run=238-852msec