== sanity test 398b: DIO and buffer IO race ============== 18:11:54 (1748470314) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.16954 s, 43.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=18623: Wed May 28 18:12:09 2025 read: IOPS=121, BW=486KiB/s (498kB/s)(5992KiB/12331msec) clat (usec): min=54, max=174665, avg=3779.56, stdev=5261.36 lat (usec): min=55, max=174666, avg=3780.36, stdev=5261.44 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 72], 10.00th=[ 82], 20.00th=[ 1926], | 30.00th=[ 2442], 40.00th=[ 2769], 50.00th=[ 3163], 60.00th=[ 3654], | 70.00th=[ 4424], 80.00th=[ 5669], 90.00th=[ 7439], 95.00th=[ 8979], | 99.00th=[ 12256], 99.50th=[ 16450], 99.90th=[ 26084], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 232, max= 632, per=25.21%, avg=480.17, stdev=91.94, samples=24 iops : min= 58, max= 158, avg=119.96, stdev=23.01, samples=24 write: IOPS=127, BW=511KiB/s (523kB/s)(6296KiB/12331msec) clat (usec): min=434, max=27911, avg=4154.40, stdev=4007.95 lat (usec): min=435, max=27912, avg=4155.37, stdev=4008.07 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 668], | 30.00th=[ 906], 40.00th=[ 1811], 50.00th=[ 3228], 60.00th=[ 4293], | 70.00th=[ 5538], 80.00th=[ 7046], 90.00th=[ 9372], 95.00th=[11994], | 99.00th=[17433], 99.50th=[22414], 99.90th=[26346], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 288, max= 704, per=25.29%, avg=504.50, stdev=95.93, samples=24 iops : min= 72, max= 176, avg=126.04, stdev=24.03, samples=24 lat (usec) : 100=6.87%, 250=1.95%, 500=0.55%, 750=11.82%, 1000=4.26% lat (msec) : 2=6.02%, 4=29.98%, 10=32.91%, 20=5.24%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.29%, sys=23.41%, ctx=5455, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18624: Wed May 28 18:12:09 2025 read: IOPS=120, BW=480KiB/s (492kB/s)(5940KiB/12373msec) clat (usec): min=51, max=39252, avg=3669.09, stdev=2951.30 lat (usec): min=52, max=39253, avg=3669.76, stdev=2951.30 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 72], 10.00th=[ 85], 20.00th=[ 1926], | 30.00th=[ 2343], 40.00th=[ 2737], 50.00th=[ 3097], 60.00th=[ 3523], | 70.00th=[ 4424], 80.00th=[ 5800], 90.00th=[ 7570], 95.00th=[ 8848], | 99.00th=[12125], 99.50th=[15401], 99.90th=[20579], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 296, max= 704, per=25.07%, avg=477.62, stdev=84.91, samples=24 iops : min= 74, max= 176, avg=119.38, stdev=21.25, samples=24 write: IOPS=128, BW=513KiB/s (525kB/s)(6348KiB/12373msec) clat (usec): min=480, max=172940, avg=4281.79, stdev=5872.59 lat (usec): min=480, max=172941, avg=4283.02, stdev=5872.65 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 660], | 30.00th=[ 881], 40.00th=[ 1893], 50.00th=[ 3261], 60.00th=[ 4359], | 70.00th=[ 5407], 80.00th=[ 6915], 90.00th=[ 9241], 95.00th=[ 11731], | 99.00th=[ 18220], 99.50th=[ 22938], 99.90th=[ 34866], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 352, max= 704, per=25.23%, avg=503.29, stdev=95.22, samples=24 iops : min= 88, max= 176, avg=125.79, stdev=23.83, samples=24 lat (usec) : 100=6.15%, 250=2.54%, 500=0.33%, 750=12.89%, 1000=3.78% lat (msec) : 2=5.37%, 4=30.18%, 10=33.46%, 20=4.92%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.29%, sys=23.67%, ctx=5394, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18625: Wed May 28 18:12:09 2025 read: IOPS=121, BW=485KiB/s (497kB/s)(6028KiB/12428msec) clat (usec): min=53, max=55797, avg=3554.41, stdev=3114.33 lat (usec): min=53, max=55797, avg=3555.46, stdev=3114.19 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 71], 10.00th=[ 83], 20.00th=[ 725], | 30.00th=[ 2311], 40.00th=[ 2671], 50.00th=[ 3032], 60.00th=[ 3490], | 70.00th=[ 4228], 80.00th=[ 5473], 90.00th=[ 7308], 95.00th=[ 8717], | 99.00th=[12125], 99.50th=[15533], 99.90th=[22676], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 255, max= 656, per=25.00%, avg=476.33, stdev=95.77, samples=24 iops : min= 63, max= 164, avg=118.96, stdev=23.98, samples=24 write: IOPS=125, BW=504KiB/s (516kB/s)(6260KiB/12428msec) clat (usec): min=456, max=175896, avg=4434.68, stdev=6058.76 lat (usec): min=457, max=175896, avg=4435.72, stdev=6058.71 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 537], 10.00th=[ 570], 20.00th=[ 660], | 30.00th=[ 930], 40.00th=[ 2147], 50.00th=[ 3458], 60.00th=[ 4555], | 70.00th=[ 5669], 80.00th=[ 7308], 90.00th=[ 9634], 95.00th=[ 11863], | 99.00th=[ 18744], 99.50th=[ 21103], 99.90th=[ 50070], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 319, max= 664, per=24.68%, avg=492.42, stdev=87.63, samples=24 iops : min= 79, max= 166, avg=122.96, stdev=21.97, samples=24 lat (usec) : 100=7.39%, 250=2.25%, 500=0.68%, 750=11.59%, 1000=3.78% lat (msec) : 2=5.21%, 4=30.11%, 10=33.40%, 20=5.14%, 50=0.36% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.17%, sys=23.44%, ctx=5531, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18626: Wed May 28 18:12:09 2025 read: IOPS=120, BW=480KiB/s (492kB/s)(6052KiB/12599msec) clat (usec): min=54, max=51452, avg=3600.81, stdev=3170.56 lat (usec): min=54, max=51452, avg=3602.54, stdev=3171.54 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 71], 10.00th=[ 83], 20.00th=[ 141], | 30.00th=[ 2311], 40.00th=[ 2638], 50.00th=[ 3032], 60.00th=[ 3523], | 70.00th=[ 4424], 80.00th=[ 5735], 90.00th=[ 7373], 95.00th=[ 8848], | 99.00th=[12911], 99.50th=[15270], 99.90th=[28705], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 152, max= 646, per=24.94%, avg=475.04, stdev=112.45, samples=25 iops : min= 38, max= 161, avg=118.68, stdev=28.15, samples=25 write: IOPS=123, BW=495KiB/s (507kB/s)(6236KiB/12599msec) clat (usec): min=451, max=206972, avg=4501.94, stdev=6701.63 lat (usec): min=452, max=206973, avg=4503.14, stdev=6701.50 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 537], 10.00th=[ 570], 20.00th=[ 660], | 30.00th=[ 938], 40.00th=[ 2114], 50.00th=[ 3392], 60.00th=[ 4424], | 70.00th=[ 5735], 80.00th=[ 7439], 90.00th=[ 10028], 95.00th=[ 12125], | 99.00th=[ 18220], 99.50th=[ 23200], 99.90th=[ 50070], 99.95th=[206570], | 99.99th=[206570] bw ( KiB/s): min= 192, max= 752, per=24.60%, avg=490.72, stdev=114.41, samples=25 iops : min= 48, max= 188, avg=122.60, stdev=28.61, samples=25 lat (usec) : 100=7.00%, 250=3.32%, 500=0.26%, 750=11.82%, 1000=3.78% lat (msec) : 2=5.40%, 4=29.20%, 10=32.88%, 20=5.89%, 50=0.36% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.16%, sys=23.28%, ctx=5459, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1906KiB/s (1952kB/s), 480KiB/s-486KiB/s (492kB/s-498kB/s), io=23.4MiB (24.6MB), run=12331-12599msec WRITE: bw=1995KiB/s (2043kB/s), 495KiB/s-513KiB/s (507kB/s-525kB/s), io=24.6MiB (25.7MB), run=12331-12599msec rand-rw: (groupid=0, jobs=1): err= 0: pid=18627: Wed May 28 18:12:32 2025 read: IOPS=42, BW=169KiB/s (174kB/s)(5992KiB/35361msec) clat (usec): min=1661, max=179282, avg=4088.11, stdev=5455.25 lat (usec): min=1662, max=179286, avg=4088.92, stdev=5455.31 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1975], 10.00th=[ 2089], 20.00th=[ 2245], | 30.00th=[ 2409], 40.00th=[ 2540], 50.00th=[ 2769], 60.00th=[ 3228], | 70.00th=[ 4113], 80.00th=[ 5080], 90.00th=[ 7242], 95.00th=[ 9634], | 99.00th=[ 15139], 99.50th=[ 19006], 99.90th=[ 47973], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 48, max= 296, per=25.01%, avg=169.33, stdev=63.56, samples=70 iops : min= 12, max= 74, avg=42.30, stdev=15.90, samples=70 write: IOPS=44, BW=178KiB/s (182kB/s)(6296KiB/35361msec) clat (usec): min=8725, max=73884, avg=18497.04, stdev=6933.69 lat (usec): min=8726, max=73885, avg=18498.00, stdev=6933.68 clat percentiles (usec): | 1.00th=[ 9503], 5.00th=[10683], 10.00th=[11600], 20.00th=[12911], | 30.00th=[14222], 40.00th=[15664], 50.00th=[17171], 60.00th=[18744], | 70.00th=[20317], 80.00th=[22414], 90.00th=[27132], 95.00th=[32113], | 99.00th=[41157], 99.50th=[45351], 99.90th=[71828], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 56, max= 264, per=25.08%, avg=177.56, stdev=50.28, samples=70 iops : min= 14, max= 66, avg=44.36, stdev=12.57, samples=70 lat (msec) : 2=2.73%, 4=30.63%, 10=14.52%, 20=35.61%, 50=16.31% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.13%, sys=5.88%, ctx=3711, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18628: Wed May 28 18:12:32 2025 read: IOPS=42, BW=168KiB/s (172kB/s)(5940KiB/35325msec) clat (usec): min=1707, max=192049, avg=4128.99, stdev=5762.90 lat (usec): min=1707, max=192051, avg=4129.81, stdev=5762.90 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1958], 10.00th=[ 2057], 20.00th=[ 2212], | 30.00th=[ 2376], 40.00th=[ 2540], 50.00th=[ 2868], 60.00th=[ 3392], | 70.00th=[ 4080], 80.00th=[ 5145], 90.00th=[ 7177], 95.00th=[ 9634], | 99.00th=[ 16450], 99.50th=[ 19530], 99.90th=[ 41157], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 16, max= 335, per=24.84%, avg=168.16, stdev=69.68, samples=70 iops : min= 4, max= 83, avg=41.99, stdev=17.41, samples=70 write: IOPS=44, BW=180KiB/s (184kB/s)(6348KiB/35325msec) clat (usec): min=8140, max=63467, avg=18317.81, stdev=6755.86 lat (usec): min=8141, max=63468, avg=18318.75, stdev=6755.85 clat percentiles (usec): | 1.00th=[ 9503], 5.00th=[10552], 10.00th=[11338], 20.00th=[12780], | 30.00th=[13960], 40.00th=[15533], 50.00th=[16909], 60.00th=[18482], | 70.00th=[20055], 80.00th=[22414], 90.00th=[26870], 95.00th=[32375], | 99.00th=[40109], 99.50th=[43779], 99.90th=[52167], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 64, max= 256, per=25.30%, avg=179.13, stdev=50.19, samples=70 iops : min= 16, max= 64, avg=44.73, stdev=12.54, samples=70 lat (msec) : 2=3.39%, 4=30.01%, 10=14.13%, 20=36.98%, 50=15.33% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.16%, sys=5.86%, ctx=3729, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18629: Wed May 28 18:12:32 2025 read: IOPS=42, BW=170KiB/s (174kB/s)(6028KiB/35467msec) clat (usec): min=1652, max=43736, avg=4083.06, stdev=2974.41 lat (usec): min=1653, max=43737, avg=4083.85, stdev=2974.40 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1975], 10.00th=[ 2114], 20.00th=[ 2278], | 30.00th=[ 2442], 40.00th=[ 2638], 50.00th=[ 2933], 60.00th=[ 3490], | 70.00th=[ 4178], 80.00th=[ 5276], 90.00th=[ 7570], 95.00th=[10159], | 99.00th=[14746], 99.50th=[19006], 99.90th=[28443], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 48, max= 336, per=25.07%, avg=169.70, stdev=59.58, samples=70 iops : min= 12, max= 84, avg=42.33, stdev=14.94, samples=70 write: IOPS=44, BW=177KiB/s (181kB/s)(6260KiB/35467msec) clat (msec): min=8, max=217, avg=18.65, stdev= 8.44 lat (msec): min=8, max=217, avg=18.65, stdev= 8.44 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 28], 95.00th=[ 33], | 99.00th=[ 42], 99.50th=[ 51], 99.90th=[ 64], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 56, max= 248, per=24.81%, avg=175.64, stdev=50.71, samples=70 iops : min= 14, max= 62, avg=43.81, stdev=12.71, samples=70 lat (msec) : 2=3.03%, 4=29.98%, 10=14.39%, 20=36.78%, 50=15.53% lat (msec) : 100=0.26%, 250=0.03% cpu : usr=0.13%, sys=5.94%, ctx=3767, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18630: Wed May 28 18:12:32 2025 read: IOPS=43, BW=172KiB/s (176kB/s)(6052KiB/35157msec) clat (usec): min=1571, max=48180, avg=4005.25, stdev=3168.81 lat (usec): min=1572, max=48180, avg=4006.17, stdev=3168.87 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1991], 10.00th=[ 2089], 20.00th=[ 2245], | 30.00th=[ 2409], 40.00th=[ 2573], 50.00th=[ 2835], 60.00th=[ 3392], | 70.00th=[ 4113], 80.00th=[ 5080], 90.00th=[ 7111], 95.00th=[ 9765], | 99.00th=[14484], 99.50th=[17695], 99.90th=[43779], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 39, max= 320, per=25.44%, avg=172.23, stdev=66.33, samples=70 iops : min= 9, max= 80, avg=42.97, stdev=16.64, samples=70 write: IOPS=44, BW=177KiB/s (182kB/s)(6236KiB/35157msec) clat (msec): min=8, max=205, avg=18.59, stdev= 7.99 lat (msec): min=8, max=205, avg=18.59, stdev= 7.99 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 28], 95.00th=[ 32], | 99.00th=[ 39], 99.50th=[ 47], 99.90th=[ 55], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 56, max= 240, per=24.99%, avg=176.91, stdev=47.81, samples=70 iops : min= 14, max= 60, avg=44.14, stdev=12.01, samples=70 lat (msec) : 2=2.90%, 4=30.96%, 10=13.93%, 20=36.43%, 50=15.62% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.12%, sys=5.91%, ctx=3743, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=677KiB/s (693kB/s), 168KiB/s-172KiB/s (172kB/s-176kB/s), io=23.4MiB (24.6MB), run=35157-35467msec WRITE: bw=709KiB/s (726kB/s), 177KiB/s-180KiB/s (181kB/s-184kB/s), io=24.6MiB (25.7MB), run=35157-35467msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=18668: Wed May 28 18:12:36 2025 read: IOPS=140, BW=2252KiB/s (2306kB/s)(5920KiB/2629msec) clat (usec): min=56, max=22285, avg=3363.33, stdev=2704.93 lat (usec): min=57, max=22286, avg=3363.98, stdev=2704.96 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 74], 10.00th=[ 84], 20.00th=[ 128], | 30.00th=[ 2409], 40.00th=[ 2802], 50.00th=[ 3261], 60.00th=[ 3818], | 70.00th=[ 4490], 80.00th=[ 5538], 90.00th=[ 6718], 95.00th=[ 7570], | 99.00th=[10421], 99.50th=[12256], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 1728, max= 2816, per=24.52%, avg=2233.60, stdev=390.87, samples=5 iops : min= 108, max= 176, avg=139.60, stdev=24.43, samples=5 write: IOPS=151, BW=2422KiB/s (2480kB/s)(6368KiB/2629msec) clat (usec): min=646, max=21228, avg=3435.39, stdev=3224.00 lat (usec): min=646, max=21229, avg=3436.39, stdev=3224.00 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 750], 10.00th=[ 807], 20.00th=[ 922], | 30.00th=[ 1074], 40.00th=[ 1287], 50.00th=[ 1909], 60.00th=[ 3261], | 70.00th=[ 4359], 80.00th=[ 6128], 90.00th=[ 8029], 95.00th=[ 9503], | 99.00th=[15008], 99.50th=[17695], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 1824, max= 2944, per=25.17%, avg=2412.80, stdev=452.89, samples=5 iops : min= 114, max= 184, avg=150.80, stdev=28.31, samples=5 lat (usec) : 100=7.16%, 250=5.08%, 500=0.78%, 750=2.86%, 1000=10.55% lat (msec) : 2=13.28%, 4=25.39%, 10=32.42%, 20=2.21%, 50=0.26% cpu : usr=0.53%, sys=29.03%, ctx=1564, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18669: Wed May 28 18:12:36 2025 read: IOPS=137, BW=2194KiB/s (2247kB/s)(5744KiB/2618msec) clat (usec): min=59, max=16149, avg=3192.57, stdev=2806.27 lat (usec): min=59, max=16150, avg=3194.53, stdev=2806.66 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 73], 10.00th=[ 78], 20.00th=[ 102], | 30.00th=[ 330], 40.00th=[ 2671], 50.00th=[ 3032], 60.00th=[ 3458], | 70.00th=[ 4293], 80.00th=[ 5407], 90.00th=[ 6783], 95.00th=[ 7963], | 99.00th=[13042], 99.50th=[13566], 99.90th=[16188], 99.95th=[16188], | 99.99th=[16188] bw ( KiB/s): min= 1920, max= 2592, per=24.17%, avg=2201.60, stdev=274.72, samples=5 iops : min= 120, max= 162, avg=137.60, stdev=17.17, samples=5 write: IOPS=156, BW=2500KiB/s (2560kB/s)(6544KiB/2618msec) clat (usec): min=626, max=24541, avg=3539.61, stdev=3334.25 lat (usec): min=627, max=24542, avg=3540.56, stdev=3334.25 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 766], 10.00th=[ 832], 20.00th=[ 947], | 30.00th=[ 1172], 40.00th=[ 1500], 50.00th=[ 2442], 60.00th=[ 3458], | 70.00th=[ 4490], 80.00th=[ 5604], 90.00th=[ 7898], 95.00th=[10159], | 99.00th=[16450], 99.50th=[19530], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 2208, max= 2752, per=25.71%, avg=2464.00, stdev=206.15, samples=5 iops : min= 138, max= 172, avg=154.00, stdev=12.88, samples=5 lat (usec) : 100=9.11%, 250=4.82%, 500=0.52%, 750=2.47%, 1000=10.29% lat (msec) : 2=12.24%, 4=26.95%, 10=29.95%, 20=3.39%, 50=0.26% cpu : usr=0.23%, sys=28.85%, ctx=1483, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18670: Wed May 28 18:12:36 2025 read: IOPS=151, BW=2424KiB/s (2482kB/s)(6256KiB/2581msec) clat (usec): min=58, max=24052, avg=3108.78, stdev=3065.22 lat (usec): min=58, max=24053, avg=3109.54, stdev=3065.28 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 71], 10.00th=[ 74], 20.00th=[ 103], | 30.00th=[ 137], 40.00th=[ 2573], 50.00th=[ 3064], 60.00th=[ 3556], | 70.00th=[ 4293], 80.00th=[ 5211], 90.00th=[ 6325], 95.00th=[ 7767], | 99.00th=[14484], 99.50th=[18482], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 1856, max= 2976, per=26.84%, avg=2444.80, stdev=538.61, samples=5 iops : min= 116, max= 186, avg=152.80, stdev=33.66, samples=5 write: IOPS=146, BW=2337KiB/s (2393kB/s)(6032KiB/2581msec) clat (usec): min=678, max=24804, avg=3588.21, stdev=3441.09 lat (usec): min=679, max=24805, avg=3593.03, stdev=3439.07 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 758], 10.00th=[ 807], 20.00th=[ 979], | 30.00th=[ 1156], 40.00th=[ 1434], 50.00th=[ 2212], 60.00th=[ 3294], | 70.00th=[ 4686], 80.00th=[ 5932], 90.00th=[ 8094], 95.00th=[10421], | 99.00th=[14877], 99.50th=[21365], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 1952, max= 2560, per=24.24%, avg=2323.20, stdev=237.96, samples=5 iops : min= 122, max= 160, avg=145.20, stdev=14.87, samples=5 lat (usec) : 100=10.03%, 250=7.42%, 500=0.52%, 750=1.95%, 1000=8.85% lat (msec) : 2=13.28%, 4=24.48%, 10=29.82%, 20=3.26%, 50=0.39% cpu : usr=0.23%, sys=28.22%, ctx=1454, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18671: Wed May 28 18:12:36 2025 read: IOPS=146, BW=2352KiB/s (2408kB/s)(6032KiB/2565msec) clat (usec): min=60, max=15263, avg=3005.05, stdev=2789.11 lat (usec): min=60, max=15264, avg=3005.65, stdev=2789.11 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 72], 10.00th=[ 74], 20.00th=[ 91], | 30.00th=[ 135], 40.00th=[ 2442], 50.00th=[ 3032], 60.00th=[ 3458], | 70.00th=[ 4047], 80.00th=[ 5211], 90.00th=[ 6587], 95.00th=[ 7832], | 99.00th=[11076], 99.50th=[15008], 99.90th=[15270], 99.95th=[15270], | 99.99th=[15270] bw ( KiB/s): min= 1888, max= 3040, per=25.36%, avg=2310.40, stdev=452.21, samples=5 iops : min= 118, max= 190, avg=144.40, stdev=28.26, samples=5 write: IOPS=152, BW=2439KiB/s (2498kB/s)(6256KiB/2565msec) clat (usec): min=625, max=36079, avg=3539.31, stdev=3864.61 lat (usec): min=626, max=36080, avg=3540.25, stdev=3864.60 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 742], 10.00th=[ 824], 20.00th=[ 930], | 30.00th=[ 1123], 40.00th=[ 1336], 50.00th=[ 1795], 60.00th=[ 2900], | 70.00th=[ 4178], 80.00th=[ 6063], 90.00th=[ 8160], 95.00th=[10552], | 99.00th=[17171], 99.50th=[28181], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 1984, max= 2880, per=25.44%, avg=2438.40, stdev=360.20, samples=5 iops : min= 124, max= 180, avg=152.40, stdev=22.51, samples=5 lat (usec) : 100=10.55%, 250=6.12%, 500=0.65%, 750=2.99%, 1000=8.85% lat (msec) : 2=15.10%, 4=25.26%, 10=26.30%, 20=3.91%, 50=0.26% cpu : usr=0.20%, sys=28.86%, ctx=1441, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9111KiB/s (9329kB/s), 2194KiB/s-2424KiB/s (2247kB/s-2482kB/s), io=23.4MiB (24.5MB), run=2565-2629msec WRITE: bw=9585KiB/s (9815kB/s), 2337KiB/s-2500KiB/s (2393kB/s-2560kB/s), io=24.6MiB (25.8MB), run=2565-2629msec rand-rw: (groupid=0, jobs=1): err= 0: pid=18672: Wed May 28 18:12:42 2025 read: IOPS=40, BW=648KiB/s (663kB/s)(5920KiB/9139msec) clat (usec): min=1920, max=22903, avg=4262.40, stdev=3164.28 lat (usec): min=1921, max=22904, avg=4263.19, stdev=3164.29 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2180], 10.00th=[ 2278], 20.00th=[ 2442], | 30.00th=[ 2638], 40.00th=[ 2835], 50.00th=[ 3064], 60.00th=[ 3392], | 70.00th=[ 3949], 80.00th=[ 5145], 90.00th=[ 7767], 95.00th=[10814], | 99.00th=[20579], 99.50th=[21627], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 256, max= 1184, per=24.68%, avg=641.56, stdev=209.02, samples=18 iops : min= 16, max= 74, avg=40.00, stdev=13.09, samples=18 write: IOPS=43, BW=697KiB/s (714kB/s)(6368KiB/9139msec) clat (usec): min=10043, max=50045, avg=18819.33, stdev=6187.18 lat (usec): min=10045, max=50047, avg=18821.03, stdev=6187.14 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11338], 10.00th=[12125], 20.00th=[13566], | 30.00th=[15008], 40.00th=[16581], 50.00th=[17957], 60.00th=[19268], | 70.00th=[20579], 80.00th=[22414], 90.00th=[26870], 95.00th=[31589], | 99.00th=[41157], 99.50th=[42206], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 384, max= 864, per=25.41%, avg=694.94, stdev=169.95, samples=18 iops : min= 24, max= 54, avg=43.33, stdev=10.66, samples=18 lat (msec) : 2=0.39%, 4=33.59%, 10=11.59%, 20=36.85%, 50=17.45% lat (msec) : 100=0.13% cpu : usr=0.08%, sys=6.91%, ctx=931, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18673: Wed May 28 18:12:42 2025 read: IOPS=38, BW=623KiB/s (638kB/s)(5744KiB/9213msec) clat (usec): min=2003, max=29268, avg=4324.67, stdev=3576.72 lat (usec): min=2003, max=29269, avg=4325.49, stdev=3576.74 clat percentiles (usec): | 1.00th=[ 2089], 5.00th=[ 2245], 10.00th=[ 2376], 20.00th=[ 2507], | 30.00th=[ 2671], 40.00th=[ 2802], 50.00th=[ 2999], 60.00th=[ 3326], | 70.00th=[ 3720], 80.00th=[ 4817], 90.00th=[ 7570], 95.00th=[12911], | 99.00th=[19006], 99.50th=[27919], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 288, max= 1024, per=23.93%, avg=622.00, stdev=212.86, samples=18 iops : min= 18, max= 64, avg=38.78, stdev=13.29, samples=18 write: IOPS=44, BW=710KiB/s (727kB/s)(6544KiB/9213msec) clat (usec): min=9943, max=49232, avg=18618.68, stdev=5961.01 lat (usec): min=9944, max=49233, avg=18620.28, stdev=5961.03 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11600], 10.00th=[12125], 20.00th=[13435], | 30.00th=[14877], 40.00th=[16581], 50.00th=[17957], 60.00th=[18744], | 70.00th=[20579], 80.00th=[22414], 90.00th=[26346], 95.00th=[30540], | 99.00th=[38011], 99.50th=[39584], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 384, max= 896, per=25.67%, avg=702.06, stdev=168.04, samples=18 iops : min= 24, max= 56, avg=43.78, stdev=10.54, samples=18 lat (msec) : 4=34.77%, 10=9.11%, 20=38.67%, 50=17.45% cpu : usr=0.15%, sys=6.75%, ctx=920, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18674: Wed May 28 18:12:42 2025 read: IOPS=43, BW=704KiB/s (721kB/s)(6256KiB/8888msec) clat (usec): min=1872, max=29001, avg=3993.35, stdev=3004.00 lat (usec): min=1872, max=29002, avg=3994.12, stdev=3004.01 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2147], 10.00th=[ 2278], 20.00th=[ 2474], | 30.00th=[ 2671], 40.00th=[ 2835], 50.00th=[ 3032], 60.00th=[ 3261], | 70.00th=[ 3556], 80.00th=[ 4228], 90.00th=[ 6980], 95.00th=[ 8979], | 99.00th=[16319], 99.50th=[25035], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 320, max= 1280, per=27.51%, avg=715.00, stdev=298.61, samples=17 iops : min= 20, max= 80, avg=44.59, stdev=18.59, samples=17 write: IOPS=42, BW=679KiB/s (695kB/s)(6032KiB/8888msec) clat (usec): min=9116, max=56420, avg=19401.41, stdev=6925.73 lat (usec): min=9118, max=56423, avg=19403.18, stdev=6925.82 clat percentiles (usec): | 1.00th=[ 9896], 5.00th=[11207], 10.00th=[12256], 20.00th=[13829], | 30.00th=[15270], 40.00th=[16712], 50.00th=[17957], 60.00th=[19530], | 70.00th=[21365], 80.00th=[23462], 90.00th=[27919], 95.00th=[32637], | 99.00th=[43779], 99.50th=[50070], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 416, max= 864, per=24.63%, avg=673.65, stdev=152.23, samples=17 iops : min= 26, max= 54, avg=42.00, stdev= 9.44, samples=17 lat (msec) : 2=0.52%, 4=38.93%, 10=9.90%, 20=32.03%, 50=18.36% lat (msec) : 100=0.26% cpu : usr=0.17%, sys=6.93%, ctx=937, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18675: Wed May 28 18:12:42 2025 read: IOPS=41, BW=660KiB/s (675kB/s)(6032KiB/9145msec) clat (usec): min=1873, max=29034, avg=4270.70, stdev=3106.34 lat (usec): min=1874, max=29035, avg=4271.73, stdev=3106.27 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2245], 10.00th=[ 2343], 20.00th=[ 2573], | 30.00th=[ 2769], 40.00th=[ 2966], 50.00th=[ 3195], 60.00th=[ 3458], | 70.00th=[ 3884], 80.00th=[ 5473], 90.00th=[ 7373], 95.00th=[10159], | 99.00th=[17957], 99.50th=[23725], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 224, max= 1024, per=25.24%, avg=655.89, stdev=228.80, samples=18 iops : min= 14, max= 64, avg=40.94, stdev=14.33, samples=18 write: IOPS=42, BW=684KiB/s (701kB/s)(6256KiB/9145msec) clat (usec): min=8916, max=56860, avg=19086.26, stdev=6446.02 lat (usec): min=8917, max=56861, avg=19087.82, stdev=6446.04 clat percentiles (usec): | 1.00th=[10028], 5.00th=[11207], 10.00th=[12649], 20.00th=[13960], | 30.00th=[15139], 40.00th=[16450], 50.00th=[17957], 60.00th=[19268], | 70.00th=[20841], 80.00th=[22414], 90.00th=[27919], 95.00th=[32900], | 99.00th=[39584], 99.50th=[43779], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 384, max= 896, per=25.02%, avg=684.39, stdev=171.59, samples=18 iops : min= 24, max= 56, avg=42.72, stdev=10.81, samples=18 lat (msec) : 2=0.26%, 4=35.03%, 10=11.85%, 20=33.72%, 50=19.01% lat (msec) : 100=0.13% cpu : usr=0.10%, sys=6.81%, ctx=936, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2600KiB/s (2662kB/s), 623KiB/s-704KiB/s (638kB/s-721kB/s), io=23.4MiB (24.5MB), run=8888-9213msec WRITE: bw=2735KiB/s (2801kB/s), 679KiB/s-710KiB/s (695kB/s-727kB/s), io=24.6MiB (25.8MB), run=8888-9213msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=18695: Wed May 28 18:12:44 2025 read: IOPS=7, BW=7506KiB/s (7686kB/s)(7168KiB/955msec) clat (usec): min=437, max=347800, avg=74849.67, stdev=121177.69 lat (usec): min=438, max=347801, avg=74851.05, stdev=121177.39 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 437], 10.00th=[ 437], 20.00th=[ 24511], | 30.00th=[ 33162], 40.00th=[ 33162], 50.00th=[ 36439], 60.00th=[ 40633], | 70.00th=[ 40633], 80.00th=[ 41157], 90.00th=[346031], 95.00th=[346031], | 99.00th=[346031], 99.50th=[346031], 99.90th=[346031], 99.95th=[346031], | 99.99th=[346031] bw ( KiB/s): min= 6095, max= 6095, per=31.34%, avg=6095.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5361KiB/s (5490kB/s)(5120KiB/955msec) clat (msec): min=29, max=112, avg=80.52, stdev=31.53 lat (msec): min=29, max=112, avg=80.60, stdev=31.53 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 30], | 30.00th=[ 75], 40.00th=[ 75], 50.00th=[ 87], 60.00th=[ 87], | 70.00th=[ 99], 80.00th=[ 99], 90.00th=[ 113], 95.00th=[ 113], | 99.00th=[ 113], 99.50th=[ 113], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 2031, max= 2031, per=7.46%, avg=2031.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=50.00%, 100=25.00%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=25.16%, ctx=615, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18697: Wed May 28 18:12:44 2025 read: IOPS=1, BW=1945KiB/s (1992kB/s)(2048KiB/1053msec) clat (usec): min=588, max=30265, avg=15427.35, stdev=20984.73 lat (usec): min=590, max=30266, avg=15428.78, stdev=20984.24 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 586], 10.00th=[ 586], 20.00th=[ 586], | 30.00th=[ 586], 40.00th=[ 586], 50.00th=[ 586], 60.00th=[30278], | 70.00th=[30278], 80.00th=[30278], 90.00th=[30278], 95.00th=[30278], | 99.00th=[30278], 99.50th=[30278], 99.90th=[30278], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 2048, max= 2048, per=10.53%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=9, BW=9725KiB/s (9958kB/s)(10.0MiB/1053msec) clat (msec): min=32, max=437, avg=100.28, stdev=121.53 lat (msec): min=32, max=437, avg=100.35, stdev=121.52 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 34], | 30.00th=[ 44], 40.00th=[ 50], 50.00th=[ 52], 60.00th=[ 58], | 70.00th=[ 104], 80.00th=[ 104], 90.00th=[ 110], 95.00th=[ 439], | 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 2048, max=14336, per=30.09%, avg=8192.00, stdev=8688.93, samples=2 iops : min= 2, max= 14, avg= 8.00, stdev= 8.49, samples=2 lat (usec) : 750=8.33% lat (msec) : 50=41.67%, 100=25.00%, 250=16.67%, 500=8.33% cpu : usr=0.00%, sys=20.63%, ctx=179, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18699: Wed May 28 18:12:44 2025 read: IOPS=6, BW=6686KiB/s (6846kB/s)(6144KiB/919msec) clat (usec): min=277, max=36449, avg=19055.21, stdev=17449.07 lat (usec): min=278, max=36451, avg=19056.45, stdev=17449.29 clat percentiles (usec): | 1.00th=[ 277], 5.00th=[ 277], 10.00th=[ 277], 20.00th=[ 482], | 30.00th=[ 482], 40.00th=[ 9634], 50.00th=[ 9634], 60.00th=[33162], | 70.00th=[34341], 80.00th=[34341], 90.00th=[36439], 95.00th=[36439], | 99.00th=[36439], 99.50th=[36439], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] write: IOPS=6, BW=6686KiB/s (6846kB/s)(6144KiB/919msec) clat (msec): min=24, max=452, avg=126.97, stdev=161.47 lat (msec): min=24, max=452, avg=127.02, stdev=161.48 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 78], 50.00th=[ 78], 60.00th=[ 82], | 70.00th=[ 90], 80.00th=[ 90], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 4096, max= 4096, per=15.04%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 10=8.33%, 50=41.67%, 100=25.00%, 500=8.33% cpu : usr=0.00%, sys=19.28%, ctx=532, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18701: Wed May 28 18:12:44 2025 read: IOPS=5, BW=5645KiB/s (5780kB/s)(5120KiB/907msec) clat (usec): min=368, max=49058, avg=13768.03, stdev=20618.89 lat (usec): min=369, max=49059, avg=13768.89, stdev=20619.20 clat percentiles (usec): | 1.00th=[ 367], 5.00th=[ 367], 10.00th=[ 367], 20.00th=[ 367], | 30.00th=[ 449], 40.00th=[ 449], 50.00th=[ 3949], 60.00th=[ 3949], | 70.00th=[15008], 80.00th=[15008], 90.00th=[49021], 95.00th=[49021], | 99.00th=[49021], 99.50th=[49021], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] write: IOPS=7, BW=7903KiB/s (8093kB/s)(7168KiB/907msec) clat (msec): min=19, max=432, avg=115.88, stdev=144.12 lat (msec): min=19, max=432, avg=115.93, stdev=144.12 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 28], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 69], 60.00th=[ 93], | 70.00th=[ 93], 80.00th=[ 120], 90.00th=[ 435], 95.00th=[ 435], | 99.00th=[ 435], 99.50th=[ 435], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 4079, max= 4079, per=14.98%, avg=4079.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 4=8.33%, 20=16.67%, 50=25.00%, 100=16.67%, 250=8.33% lat (msec) : 500=8.33% cpu : usr=0.00%, sys=17.00%, ctx=348, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.0MiB/s (19.9MB/s), 1945KiB/s-7506KiB/s (1992kB/s-7686kB/s), io=20.0MiB (20.0MB), run=907-1053msec WRITE: bw=26.6MiB/s (27.9MB/s), 5361KiB/s-9725KiB/s (5490kB/s-9958kB/s), io=28.0MiB (29.4MB), run=907-1053msec rand-rw: (groupid=0, jobs=1): err= 0: pid=18694: Wed May 28 18:12:44 2025 read: IOPS=7, BW=7868KiB/s (8057kB/s)(7168KiB/911msec) clat (msec): min=14, max=279, avg=63.42, stdev=96.32 lat (msec): min=14, max=279, avg=63.42, stdev=96.32 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 18], | 30.00th=[ 22], 40.00th=[ 22], 50.00th=[ 28], 60.00th=[ 31], | 70.00th=[ 31], 80.00th=[ 53], 90.00th=[ 279], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min=10098, max=10098, per=56.41%, avg=10098.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5620KiB/s (5755kB/s)(5120KiB/911msec) clat (msec): min=43, max=146, avg=91.55, stdev=41.68 lat (msec): min=43, max=147, avg=91.62, stdev=41.68 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 44], | 30.00th=[ 64], 40.00th=[ 64], 50.00th=[ 86], 60.00th=[ 86], | 70.00th=[ 120], 80.00th=[ 120], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 2019, max= 2019, per=8.06%, avg=2019.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=33.33%, 100=25.00%, 250=16.67%, 500=8.33% cpu : usr=0.11%, sys=8.24%, ctx=55, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18696: Wed May 28 18:12:44 2025 read: IOPS=1, BW=1790KiB/s (1833kB/s)(2048KiB/1144msec) clat (usec): min=22247, max=36853, avg=29550.77, stdev=10327.98 lat (usec): min=22248, max=36854, avg=29551.85, stdev=10327.94 clat percentiles (usec): | 1.00th=[22152], 5.00th=[22152], 10.00th=[22152], 20.00th=[22152], | 30.00th=[22152], 40.00th=[22152], 50.00th=[22152], 60.00th=[36963], | 70.00th=[36963], 80.00th=[36963], 90.00th=[36963], 95.00th=[36963], | 99.00th=[36963], 99.50th=[36963], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 2039, max= 2048, per=11.41%, avg=2043.50, stdev= 6.36, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=8, BW=8951KiB/s (9166kB/s)(10.0MiB/1144msec) clat (msec): min=21, max=511, avg=105.61, stdev=147.27 lat (msec): min=21, max=511, avg=105.69, stdev=147.26 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 30], | 30.00th=[ 33], 40.00th=[ 37], 50.00th=[ 44], 60.00th=[ 55], | 70.00th=[ 115], 80.00th=[ 115], 90.00th=[ 128], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min=12288, max=12288, per=49.03%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=16.67%, 250=16.67%, 750=8.33% cpu : usr=0.17%, sys=10.06%, ctx=59, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18698: Wed May 28 18:12:44 2025 read: IOPS=5, BW=5699KiB/s (5836kB/s)(6144KiB/1078msec) clat (msec): min=14, max=147, avg=55.07, stdev=48.90 lat (msec): min=14, max=147, avg=55.08, stdev=48.90 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 26], | 30.00th=[ 26], 40.00th=[ 33], 50.00th=[ 33], 60.00th=[ 41], | 70.00th=[ 70], 80.00th=[ 70], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 8192, max= 8192, per=45.76%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5699KiB/s (5836kB/s)(6144KiB/1078msec) clat (msec): min=28, max=446, avg=122.31, stdev=161.52 lat (msec): min=29, max=446, avg=122.38, stdev=161.52 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 37], 50.00th=[ 37], 60.00th=[ 88], | 70.00th=[ 99], 80.00th=[ 99], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 4047, max= 6144, per=20.33%, avg=5095.50, stdev=1482.80, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 lat (msec) : 20=8.33%, 50=50.00%, 100=25.00%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=8.64%, ctx=57, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18700: Wed May 28 18:12:44 2025 read: IOPS=4, BW=4872KiB/s (4988kB/s)(5120KiB/1051msec) clat (usec): min=13689, max=61356, avg=33595.29, stdev=19702.17 lat (usec): min=13690, max=61357, avg=33596.34, stdev=19702.54 clat percentiles (usec): | 1.00th=[13698], 5.00th=[13698], 10.00th=[13698], 20.00th=[13698], | 30.00th=[16909], 40.00th=[16909], 50.00th=[32113], 60.00th=[32113], | 70.00th=[43779], 80.00th=[43779], 90.00th=[61604], 95.00th=[61604], | 99.00th=[61604], 99.50th=[61604], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 8192, max= 8192, per=45.76%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6820KiB/s (6984kB/s)(7168KiB/1051msec) clat (msec): min=29, max=439, avg=124.88, stdev=142.80 lat (msec): min=29, max=439, avg=124.96, stdev=142.79 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 43], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 92], 60.00th=[ 95], | 70.00th=[ 95], 80.00th=[ 126], 90.00th=[ 439], 95.00th=[ 439], | 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 4096, max= 8192, per=24.52%, avg=6144.00, stdev=2896.31, samples=2 iops : min= 4, max= 8, avg= 6.00, stdev= 2.83, samples=2 lat (msec) : 20=16.67%, 50=33.33%, 100=33.33%, 250=8.33%, 500=8.33% cpu : usr=0.38%, sys=10.48%, ctx=72, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.5MiB/s (18.3MB/s), 1790KiB/s-7868KiB/s (1833kB/s-8057kB/s), io=20.0MiB (20.0MB), run=911-1144msec WRITE: bw=24.5MiB/s (25.7MB/s), 5620KiB/s-8951KiB/s (5755kB/s-9166kB/s), io=28.0MiB (29.4MB), run=911-1144msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=18711: Wed May 28 18:12:47 2025 read: IOPS=3, BW=15.8MiB/s (16.5MB/s)(12.0MiB/761msec) clat (msec): min=42, max=624, avg=243.65, stdev=330.28 lat (msec): min=42, max=624, avg=243.66, stdev=330.28 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 43], | 30.00th=[ 43], 40.00th=[ 64], 50.00th=[ 64], 60.00th=[ 64], | 70.00th=[ 625], 80.00th=[ 625], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 8192, max= 8192, per=58.10%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=33.33%, 750=33.33% cpu : usr=0.13%, sys=23.03%, ctx=1201, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18712: Wed May 28 18:12:47 2025 read: IOPS=0, BW=3525KiB/s (3610kB/s)(4096KiB/1162msec) clat (nsec): min=97865k, max=97865k, avg=97865074.00, stdev= 0.00 lat (nsec): min=97868k, max=97868k, avg=97867955.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[98042], 5.00th=[98042], 10.00th=[98042], 20.00th=[98042], | 30.00th=[98042], 40.00th=[98042], 50.00th=[98042], 60.00th=[98042], | 70.00th=[98042], 80.00th=[98042], 90.00th=[98042], 95.00th=[98042], | 99.00th=[98042], 99.50th=[98042], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 8192, max= 8192, per=58.10%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=7050KiB/s (7219kB/s)(8192KiB/1162msec) clat (msec): min=208, max=844, avg=526.80, stdev=449.62 lat (msec): min=210, max=844, avg=527.49, stdev=448.79 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 209], 10.00th=[ 209], 20.00th=[ 209], | 30.00th=[ 209], 40.00th=[ 209], 50.00th=[ 209], 60.00th=[ 844], | 70.00th=[ 844], 80.00th=[ 844], 90.00th=[ 844], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 8175, max= 8175, per=28.99%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 1000=33.33% cpu : usr=0.17%, sys=18.00%, ctx=232, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18713: Wed May 28 18:12:47 2025 write: IOPS=2, BW=10.4MiB/s (10.0MB/s)(12.0MiB/1149msec) clat (msec): min=190, max=752, avg=380.45, stdev=322.64 lat (msec): min=190, max=753, avg=380.74, stdev=322.66 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 190], 10.00th=[ 190], 20.00th=[ 190], | 30.00th=[ 190], 40.00th=[ 199], 50.00th=[ 199], 60.00th=[ 199], | 70.00th=[ 751], 80.00th=[ 751], 90.00th=[ 751], 95.00th=[ 751], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min=16286, max=16286, per=57.75%, avg=16286.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.00%, sys=22.04%, ctx=178, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18714: Wed May 28 18:12:47 2025 write: IOPS=2, BW=11.4MiB/s (11.9MB/s)(12.0MiB/1053msec) clat (msec): min=112, max=783, avg=341.50, stdev=382.54 lat (msec): min=112, max=783, avg=341.72, stdev=382.48 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 113], 10.00th=[ 113], 20.00th=[ 113], | 30.00th=[ 113], 40.00th=[ 129], 50.00th=[ 129], 60.00th=[ 129], | 70.00th=[ 785], 80.00th=[ 785], 90.00th=[ 785], 95.00th=[ 785], | 99.00th=[ 785], 99.50th=[ 785], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min=16286, max=16286, per=57.75%, avg=16286.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.00%, sys=22.43%, ctx=69, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.8MiB/s (14.4MB/s), 3525KiB/s-15.8MiB/s (3610kB/s-16.5MB/s), io=16.0MiB (16.8MB), run=761-1162msec WRITE: bw=27.5MiB/s (28.9MB/s), 7050KiB/s-11.4MiB/s (7219kB/s-11.9MB/s), io=32.0MiB (33.6MB), run=1053-1162msec rand-rw: (groupid=0, jobs=1): err= 0: pid=18707: Wed May 28 18:12:47 2025 read: IOPS=4, BW=16.4MiB/s (17.2MB/s)(12.0MiB/732msec) clat (msec): min=37, max=617, avg=241.77, stdev=325.52 lat (msec): min=37, max=617, avg=241.77, stdev=325.52 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 71], 50.00th=[ 71], 60.00th=[ 71], | 70.00th=[ 617], 80.00th=[ 617], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min=16351, max=16351, per=100.00%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=33.33%, 750=33.33% cpu : usr=0.00%, sys=8.21%, ctx=43, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18708: Wed May 28 18:12:47 2025 read: IOPS=0, BW=3180KiB/s (3256kB/s)(4096KiB/1288msec) clat (nsec): min=830034k, max=830034k, avg=830033902.00, stdev= 0.00 lat (nsec): min=830036k, max=830036k, avg=830035972.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 827], 5.00th=[ 827], 10.00th=[ 827], 20.00th=[ 827], | 30.00th=[ 827], 40.00th=[ 827], 50.00th=[ 827], 60.00th=[ 827], | 70.00th=[ 827], 80.00th=[ 827], 90.00th=[ 827], 95.00th=[ 827], | 99.00th=[ 827], 99.50th=[ 827], 99.90th=[ 827], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 8192, max= 8192, per=64.40%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6360KiB/s (6513kB/s)(8192KiB/1288msec) clat (msec): min=83, max=363, avg=223.21, stdev=197.95 lat (msec): min=83, max=363, avg=223.59, stdev=198.00 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 84], 10.00th=[ 84], 20.00th=[ 84], | 30.00th=[ 84], 40.00th=[ 84], 50.00th=[ 84], 60.00th=[ 363], | 70.00th=[ 363], 80.00th=[ 363], 90.00th=[ 363], 95.00th=[ 363], | 99.00th=[ 363], 99.50th=[ 363], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] lat (msec) : 100=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.08%, sys=7.85%, ctx=51, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18709: Wed May 28 18:12:47 2025 write: IOPS=2, BW=9548KiB/s (9777kB/s)(12.0MiB/1287msec) clat (msec): min=172, max=895, avg=424.67, stdev=408.17 lat (msec): min=172, max=895, avg=424.88, stdev=408.20 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 207], 50.00th=[ 207], 60.00th=[ 207], | 70.00th=[ 894], 80.00th=[ 894], 90.00th=[ 894], 95.00th=[ 894], | 99.00th=[ 894], 99.50th=[ 894], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 7953, max= 7953, per=31.26%, avg=7953.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.00%, sys=8.16%, ctx=43, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18710: Wed May 28 18:12:47 2025 write: IOPS=2, BW=11.9MiB/s (12.5MB/s)(12.0MiB/1009msec) clat (msec): min=121, max=623, avg=328.86, stdev=262.18 lat (msec): min=126, max=624, avg=330.72, stdev=260.43 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 123], 10.00th=[ 123], 20.00th=[ 123], | 30.00th=[ 123], 40.00th=[ 241], 50.00th=[ 241], 60.00th=[ 241], | 70.00th=[ 625], 80.00th=[ 625], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 8175, max= 8192, per=32.17%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=8.13%, ctx=51, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.4MiB/s (13.0MB/s), 3180KiB/s-16.4MiB/s (3256kB/s-17.2MB/s), io=16.0MiB (16.8MB), run=732-1288msec WRITE: bw=24.8MiB/s (26.1MB/s), 6360KiB/s-11.9MiB/s (6513kB/s-12.5MB/s), io=32.0MiB (33.6MB), run=1009-1288msec