== sanity test 398b: DIO and buffer IO race ============== 11:52:58 (1746633178) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.0679 s, 47.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8750: Wed May 7 11:53:12 2025 read: IOPS=127, BW=510KiB/s (523kB/s)(5992KiB/11743msec) clat (usec): min=55, max=158607, avg=3569.84, stdev=5074.07 lat (usec): min=55, max=158607, avg=3570.65, stdev=5074.13 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 70], 10.00th=[ 80], 20.00th=[ 1762], | 30.00th=[ 2114], 40.00th=[ 2409], 50.00th=[ 2769], 60.00th=[ 3261], | 70.00th=[ 4293], 80.00th=[ 5538], 90.00th=[ 7177], 95.00th=[ 8455], | 99.00th=[ 13173], 99.50th=[ 17171], 99.90th=[ 52167], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 272, max= 712, per=24.57%, avg=502.26, stdev=106.38, samples=23 iops : min= 68, max= 178, avg=125.57, stdev=26.59, samples=23 write: IOPS=134, BW=536KiB/s (549kB/s)(6296KiB/11743msec) clat (usec): min=448, max=36232, avg=3963.32, stdev=3964.24 lat (usec): min=448, max=36232, avg=3964.53, stdev=3964.38 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 519], 10.00th=[ 545], 20.00th=[ 627], | 30.00th=[ 857], 40.00th=[ 1795], 50.00th=[ 3163], 60.00th=[ 4080], | 70.00th=[ 5145], 80.00th=[ 6456], 90.00th=[ 8848], 95.00th=[10683], | 99.00th=[17171], 99.50th=[24511], 99.90th=[34866], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 352, max= 680, per=24.69%, avg=528.35, stdev=87.35, samples=23 iops : min= 88, max= 170, avg=132.09, stdev=21.84, samples=23 lat (usec) : 100=7.58%, 250=1.50%, 500=1.01%, 750=12.92%, 1000=3.06% lat (msec) : 2=7.58%, 4=29.65%, 10=32.16%, 20=4.00%, 50=0.46% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.19%, sys=23.10%, ctx=5707, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8752: Wed May 7 11:53:12 2025 read: IOPS=128, BW=515KiB/s (527kB/s)(5940KiB/11538msec) clat (usec): min=55, max=24703, avg=3262.58, stdev=2748.94 lat (usec): min=55, max=24704, avg=3263.22, stdev=2748.98 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 69], 10.00th=[ 79], 20.00th=[ 302], | 30.00th=[ 2073], 40.00th=[ 2376], 50.00th=[ 2704], 60.00th=[ 3163], | 70.00th=[ 3949], 80.00th=[ 5080], 90.00th=[ 6652], 95.00th=[ 8029], | 99.00th=[11600], 99.50th=[16057], 99.90th=[24249], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 256, max= 776, per=25.17%, avg=514.43, stdev=109.38, samples=23 iops : min= 64, max= 194, avg=128.61, stdev=27.35, samples=23 write: IOPS=137, BW=550KiB/s (563kB/s)(6348KiB/11538msec) clat (usec): min=433, max=157910, avg=4116.62, stdev=5585.67 lat (usec): min=433, max=157911, avg=4117.94, stdev=5585.43 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 523], 10.00th=[ 553], 20.00th=[ 635], | 30.00th=[ 832], 40.00th=[ 1827], 50.00th=[ 3130], 60.00th=[ 4047], | 70.00th=[ 5145], 80.00th=[ 6718], 90.00th=[ 9110], 95.00th=[ 11600], | 99.00th=[ 17171], 99.50th=[ 20055], 99.90th=[ 54264], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 296, max= 832, per=25.62%, avg=548.17, stdev=122.68, samples=23 iops : min= 74, max= 208, avg=137.04, stdev=30.67, samples=23 lat (usec) : 100=8.20%, 250=1.43%, 500=0.94%, 750=13.15%, 1000=3.29% lat (msec) : 2=7.62%, 4=30.27%, 10=30.27%, 20=4.46%, 50=0.29% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.21%, sys=22.92%, ctx=5562, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8753: Wed May 7 11:53:12 2025 read: IOPS=128, BW=515KiB/s (527kB/s)(6028KiB/11703msec) clat (usec): min=50, max=16350, avg=3153.85, stdev=2528.98 lat (usec): min=50, max=16350, avg=3154.75, stdev=2529.17 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 69], 10.00th=[ 76], 20.00th=[ 130], | 30.00th=[ 2024], 40.00th=[ 2311], 50.00th=[ 2671], 60.00th=[ 3130], | 70.00th=[ 3851], 80.00th=[ 5080], 90.00th=[ 6587], 95.00th=[ 8029], | 99.00th=[10683], 99.50th=[12780], 99.90th=[15008], 99.95th=[16319], | 99.99th=[16319] bw ( KiB/s): min= 160, max= 672, per=24.88%, avg=508.48, stdev=110.64, samples=23 iops : min= 40, max= 168, avg=127.09, stdev=27.68, samples=23 write: IOPS=133, BW=535KiB/s (548kB/s)(6260KiB/11703msec) clat (usec): min=422, max=194313, avg=4337.78, stdev=6498.04 lat (usec): min=422, max=194315, avg=4338.68, stdev=6498.08 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 515], 10.00th=[ 537], 20.00th=[ 603], | 30.00th=[ 832], 40.00th=[ 2180], 50.00th=[ 3359], 60.00th=[ 4293], | 70.00th=[ 5473], 80.00th=[ 6980], 90.00th=[ 9241], 95.00th=[ 11600], | 99.00th=[ 18744], 99.50th=[ 24511], 99.90th=[ 58459], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 184, max= 732, per=24.65%, avg=527.43, stdev=111.52, samples=23 iops : min= 46, max= 183, avg=131.83, stdev=27.90, samples=23 lat (usec) : 100=8.98%, 250=1.27%, 500=1.30%, 750=12.96%, 1000=2.31% lat (msec) : 2=7.19%, 4=30.31%, 10=30.76%, 20=4.49%, 50=0.36% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.18%, sys=22.27%, ctx=5471, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8755: Wed May 7 11:53:12 2025 read: IOPS=132, BW=529KiB/s (542kB/s)(6052KiB/11434msec) clat (usec): min=54, max=161167, avg=3355.99, stdev=5010.56 lat (usec): min=54, max=161168, avg=3356.81, stdev=5010.49 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 70], 10.00th=[ 80], 20.00th=[ 1762], | 30.00th=[ 2114], 40.00th=[ 2409], 50.00th=[ 2737], 60.00th=[ 3163], | 70.00th=[ 3884], 80.00th=[ 5014], 90.00th=[ 6390], 95.00th=[ 7635], | 99.00th=[ 11600], 99.50th=[ 13829], 99.90th=[ 46924], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 280, max= 670, per=25.52%, avg=521.73, stdev=87.24, samples=22 iops : min= 70, max= 167, avg=130.32, stdev=21.75, samples=22 write: IOPS=136, BW=545KiB/s (558kB/s)(6236KiB/11434msec) clat (usec): min=461, max=65153, avg=3975.94, stdev=4351.81 lat (usec): min=462, max=65154, avg=3976.73, stdev=4351.78 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 611], | 30.00th=[ 848], 40.00th=[ 1631], 50.00th=[ 3064], 60.00th=[ 3949], | 70.00th=[ 5080], 80.00th=[ 6652], 90.00th=[ 8979], 95.00th=[11338], | 99.00th=[16909], 99.50th=[17957], 99.90th=[64750], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 352, max= 656, per=25.23%, avg=539.95, stdev=74.40, samples=22 iops : min= 88, max= 164, avg=134.86, stdev=18.58, samples=22 lat (usec) : 100=7.91%, 250=1.24%, 500=1.14%, 750=12.92%, 1000=2.96% lat (msec) : 2=8.11%, 4=31.67%, 10=29.10%, 20=4.62%, 50=0.23% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.20%, sys=23.09%, ctx=5470, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2045KiB/s (2094kB/s), 510KiB/s-529KiB/s (523kB/s-542kB/s), io=23.4MiB (24.6MB), run=11434-11743msec WRITE: bw=2141KiB/s (2192kB/s), 535KiB/s-550KiB/s (548kB/s-563kB/s), io=24.6MiB (25.7MB), run=11434-11743msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8748: Wed May 7 11:53:37 2025 read: IOPS=41, BW=168KiB/s (172kB/s)(5992KiB/35687msec) clat (usec): min=1530, max=164566, avg=4072.45, stdev=5197.14 lat (usec): min=1531, max=164567, avg=4073.11, stdev=5197.15 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1778], 10.00th=[ 1876], 20.00th=[ 2024], | 30.00th=[ 2180], 40.00th=[ 2606], 50.00th=[ 3326], 60.00th=[ 3589], | 70.00th=[ 3982], 80.00th=[ 5080], 90.00th=[ 6718], 95.00th=[ 9241], | 99.00th=[ 19006], 99.50th=[ 22414], 99.90th=[ 40109], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 56, max= 296, per=25.18%, avg=167.42, stdev=62.18, samples=71 iops : min= 14, max= 74, avg=41.76, stdev=15.54, samples=71 write: IOPS=44, BW=176KiB/s (181kB/s)(6296KiB/35687msec) clat (usec): min=9322, max=67297, avg=18696.69, stdev=6747.61 lat (usec): min=9322, max=67298, avg=18697.47, stdev=6747.60 clat percentiles (usec): | 1.00th=[ 9896], 5.00th=[11207], 10.00th=[12125], 20.00th=[13173], | 30.00th=[14353], 40.00th=[15664], 50.00th=[17171], 60.00th=[19006], | 70.00th=[20579], 80.00th=[22676], 90.00th=[27395], 95.00th=[32113], | 99.00th=[41157], 99.50th=[43779], 99.90th=[55837], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 48, max= 256, per=25.27%, avg=176.10, stdev=48.44, samples=71 iops : min= 12, max= 64, avg=43.93, stdev=12.09, samples=71 lat (msec) : 2=9.15%, 4=25.13%, 10=13.18%, 20=34.80%, 50=17.64% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.07%, sys=5.41%, ctx=4151, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8749: Wed May 7 11:53:37 2025 read: IOPS=41, BW=165KiB/s (169kB/s)(5940KiB/36067msec) clat (usec): min=1471, max=158254, avg=4086.16, stdev=5274.34 lat (usec): min=1471, max=158255, avg=4086.99, stdev=5274.29 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1795], 10.00th=[ 1876], 20.00th=[ 2008], | 30.00th=[ 2212], 40.00th=[ 2638], 50.00th=[ 3294], 60.00th=[ 3589], | 70.00th=[ 3949], 80.00th=[ 5080], 90.00th=[ 6521], 95.00th=[ 9372], | 99.00th=[ 17433], 99.50th=[ 27132], 99.90th=[ 54264], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 16, max= 328, per=24.87%, avg=165.37, stdev=68.19, samples=71 iops : min= 4, max= 82, avg=41.31, stdev=17.06, samples=71 write: IOPS=44, BW=176KiB/s (180kB/s)(6348KiB/36067msec) clat (usec): min=9161, max=66032, avg=18798.94, stdev=6896.24 lat (usec): min=9162, max=66033, avg=18799.77, stdev=6896.24 clat percentiles (usec): | 1.00th=[ 9765], 5.00th=[11338], 10.00th=[12125], 20.00th=[13304], | 30.00th=[14353], 40.00th=[15795], 50.00th=[17171], 60.00th=[19006], | 70.00th=[20579], 80.00th=[22676], 90.00th=[28443], 95.00th=[33817], | 99.00th=[41681], 99.50th=[44303], 99.90th=[51643], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 56, max= 240, per=25.23%, avg=175.85, stdev=48.88, samples=71 iops : min= 14, max= 60, avg=43.93, stdev=12.25, samples=71 lat (msec) : 2=9.47%, 4=24.58%, 10=12.79%, 20=35.61%, 50=17.42% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.09%, sys=5.31%, ctx=4117, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8751: Wed May 7 11:53:37 2025 read: IOPS=42, BW=170KiB/s (174kB/s)(6028KiB/35551msec) clat (usec): min=1455, max=38020, avg=3940.26, stdev=3172.47 lat (usec): min=1455, max=38021, avg=3940.89, stdev=3172.48 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1778], 10.00th=[ 1893], 20.00th=[ 2040], | 30.00th=[ 2245], 40.00th=[ 2638], 50.00th=[ 3294], 60.00th=[ 3556], | 70.00th=[ 3884], 80.00th=[ 5014], 90.00th=[ 6456], 95.00th=[ 9110], | 99.00th=[17171], 99.50th=[24249], 99.90th=[32637], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 40, max= 304, per=25.49%, avg=169.54, stdev=60.91, samples=71 iops : min= 10, max= 76, avg=42.35, stdev=15.26, samples=71 write: IOPS=44, BW=176KiB/s (180kB/s)(6260KiB/35551msec) clat (msec): min=9, max=194, avg=18.82, stdev= 8.35 lat (msec): min=9, max=194, avg=18.82, stdev= 8.35 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 19], | 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 28], 95.00th=[ 34], | 99.00th=[ 44], 99.50th=[ 46], 99.90th=[ 68], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 40, max= 248, per=25.25%, avg=175.96, stdev=50.96, samples=71 iops : min= 10, max= 62, avg=43.96, stdev=12.76, samples=71 lat (msec) : 2=9.18%, 4=25.78%, 10=12.47%, 20=35.97%, 50=16.44% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.13%, sys=5.37%, ctx=4153, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8754: Wed May 7 11:53:37 2025 read: IOPS=42, BW=170KiB/s (174kB/s)(6052KiB/35617msec) clat (usec): min=1465, max=29123, avg=3943.85, stdev=2833.09 lat (usec): min=1466, max=29124, avg=3944.58, stdev=2833.08 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1811], 10.00th=[ 1893], 20.00th=[ 2024], | 30.00th=[ 2180], 40.00th=[ 2606], 50.00th=[ 3261], 60.00th=[ 3589], | 70.00th=[ 4047], 80.00th=[ 5211], 90.00th=[ 6980], 95.00th=[ 9503], | 99.00th=[14484], 99.50th=[20579], 99.90th=[28443], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 32, max= 336, per=25.54%, avg=169.83, stdev=65.23, samples=71 iops : min= 8, max= 84, avg=42.39, stdev=16.29, samples=71 write: IOPS=43, BW=175KiB/s (179kB/s)(6236KiB/35617msec) clat (msec): min=8, max=178, avg=18.92, stdev= 7.99 lat (msec): min=8, max=178, avg=18.92, stdev= 7.99 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 28], 95.00th=[ 34], | 99.00th=[ 43], 99.50th=[ 47], 99.90th=[ 68], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 48, max= 240, per=25.08%, avg=174.79, stdev=47.68, samples=71 iops : min= 12, max= 60, avg=43.63, stdev=11.93, samples=71 lat (msec) : 2=9.31%, 4=24.74%, 10=13.74%, 20=35.22%, 50=16.86% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.10%, sys=5.45%, ctx=4129, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=666KiB/s (682kB/s), 165KiB/s-170KiB/s (169kB/s-174kB/s), io=23.4MiB (24.6MB), run=35551-36067msec WRITE: bw=697KiB/s (714kB/s), 175KiB/s-176KiB/s (179kB/s-181kB/s), io=24.6MiB (25.7MB), run=35551-36067msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8798: Wed May 7 11:53:40 2025 read: IOPS=169, BW=2707KiB/s (2772kB/s)(5920KiB/2187msec) clat (usec): min=57, max=15650, avg=2573.22, stdev=2434.30 lat (usec): min=58, max=15651, avg=2573.81, stdev=2434.35 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 70], 10.00th=[ 74], 20.00th=[ 79], | 30.00th=[ 92], 40.00th=[ 2180], 50.00th=[ 2507], 60.00th=[ 2933], | 70.00th=[ 3621], 80.00th=[ 4686], 90.00th=[ 5735], 95.00th=[ 6980], | 99.00th=[ 8291], 99.50th=[12387], 99.90th=[15664], 99.95th=[15664], | 99.99th=[15664] bw ( KiB/s): min= 1824, max= 3232, per=24.81%, avg=2608.00, stdev=584.53, samples=4 iops : min= 114, max= 202, avg=163.00, stdev=36.53, samples=4 write: IOPS=181, BW=2912KiB/s (2982kB/s)(6368KiB/2187msec) clat (usec): min=670, max=14763, avg=2835.84, stdev=2651.01 lat (usec): min=671, max=14764, avg=2836.97, stdev=2652.11 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 725], 10.00th=[ 766], 20.00th=[ 857], | 30.00th=[ 914], 40.00th=[ 1057], 50.00th=[ 1450], 60.00th=[ 2638], | 70.00th=[ 3556], 80.00th=[ 4948], 90.00th=[ 6521], 95.00th=[ 8291], | 99.00th=[12256], 99.50th=[13435], 99.90th=[14746], 99.95th=[14746], | 99.99th=[14746] bw ( KiB/s): min= 1952, max= 3456, per=25.75%, avg=2848.00, stdev=658.40, samples=4 iops : min= 122, max= 216, avg=178.00, stdev=41.15, samples=4 lat (usec) : 100=15.10%, 250=2.73%, 500=0.26%, 750=3.91%, 1000=15.62% lat (msec) : 2=9.90%, 4=25.91%, 10=25.13%, 20=1.43% cpu : usr=0.05%, sys=30.27%, ctx=1341, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8799: Wed May 7 11:53:40 2025 read: IOPS=157, BW=2522KiB/s (2582kB/s)(5744KiB/2278msec) clat (usec): min=56, max=23937, avg=2797.02, stdev=3053.85 lat (usec): min=56, max=23938, avg=2798.09, stdev=3053.67 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 70], 10.00th=[ 75], 20.00th=[ 84], | 30.00th=[ 93], 40.00th=[ 2089], 50.00th=[ 2540], 60.00th=[ 3097], | 70.00th=[ 3916], 80.00th=[ 4817], 90.00th=[ 6325], 95.00th=[ 7504], | 99.00th=[11994], 99.50th=[22676], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 1952, max= 2880, per=23.51%, avg=2472.00, stdev=385.22, samples=4 iops : min= 122, max= 180, avg=154.50, stdev=24.08, samples=4 write: IOPS=179, BW=2873KiB/s (2942kB/s)(6544KiB/2278msec) clat (usec): min=661, max=17541, avg=3080.92, stdev=2726.01 lat (usec): min=662, max=17542, avg=3081.89, stdev=2725.94 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 734], 10.00th=[ 775], 20.00th=[ 857], | 30.00th=[ 971], 40.00th=[ 1237], 50.00th=[ 1745], 60.00th=[ 3032], | 70.00th=[ 4228], 80.00th=[ 5276], 90.00th=[ 6783], 95.00th=[ 8225], | 99.00th=[11863], 99.50th=[12780], 99.90th=[17433], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 2208, max= 3360, per=25.31%, avg=2800.00, stdev=471.39, samples=4 iops : min= 138, max= 210, avg=175.00, stdev=29.46, samples=4 lat (usec) : 100=15.62%, 250=1.95%, 500=0.39%, 750=4.69%, 1000=12.50% lat (msec) : 2=11.20%, 4=23.44%, 10=28.26%, 20=1.69%, 50=0.26% cpu : usr=0.26%, sys=29.64%, ctx=1358, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8800: Wed May 7 11:53:40 2025 read: IOPS=173, BW=2774KiB/s (2841kB/s)(6256KiB/2255msec) clat (usec): min=59, max=12549, avg=2553.72, stdev=2489.77 lat (usec): min=59, max=12550, avg=2554.26, stdev=2489.81 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 68], 10.00th=[ 73], 20.00th=[ 77], | 30.00th=[ 92], 40.00th=[ 1893], 50.00th=[ 2474], 60.00th=[ 2933], | 70.00th=[ 3621], 80.00th=[ 4555], 90.00th=[ 5866], 95.00th=[ 7046], | 99.00th=[10552], 99.50th=[11076], 99.90th=[12518], 99.95th=[12518], | 99.99th=[12518] bw ( KiB/s): min= 1852, max= 3136, per=25.94%, avg=2727.00, stdev=596.26, samples=4 iops : min= 115, max= 196, avg=170.25, stdev=37.63, samples=4 write: IOPS=167, BW=2675KiB/s (2739kB/s)(6032KiB/2255msec) clat (usec): min=647, max=27476, avg=3179.14, stdev=3143.33 lat (usec): min=648, max=27477, avg=3183.20, stdev=3141.95 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 840], | 30.00th=[ 922], 40.00th=[ 1090], 50.00th=[ 1582], 60.00th=[ 3032], | 70.00th=[ 4015], 80.00th=[ 5538], 90.00th=[ 7308], 95.00th=[ 8979], | 99.00th=[12518], 99.50th=[15795], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 2240, max= 3136, per=24.14%, avg=2670.75, stdev=483.52, samples=4 iops : min= 140, max= 196, avg=166.75, stdev=30.41, samples=4 lat (usec) : 100=17.06%, 250=2.08%, 500=0.78%, 750=3.78%, 1000=13.67% lat (msec) : 2=9.11%, 4=25.00%, 10=26.17%, 20=2.21%, 50=0.13% cpu : usr=0.22%, sys=29.33%, ctx=1324, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8801: Wed May 7 11:53:40 2025 read: IOPS=166, BW=2668KiB/s (2732kB/s)(6032KiB/2261msec) clat (usec): min=62, max=15176, avg=2397.67, stdev=2476.01 lat (usec): min=62, max=15176, avg=2399.09, stdev=2476.87 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 70], 10.00th=[ 74], 20.00th=[ 78], | 30.00th=[ 87], 40.00th=[ 141], 50.00th=[ 2376], 60.00th=[ 2769], | 70.00th=[ 3294], 80.00th=[ 4424], 90.00th=[ 5669], 95.00th=[ 6652], | 99.00th=[10814], 99.50th=[11994], 99.90th=[15139], 99.95th=[15139], | 99.99th=[15139] bw ( KiB/s): min= 1760, max= 3232, per=24.88%, avg=2616.00, stdev=617.95, samples=4 iops : min= 110, max= 202, avg=163.50, stdev=38.62, samples=4 write: IOPS=172, BW=2767KiB/s (2833kB/s)(6256KiB/2261msec) clat (usec): min=643, max=18857, avg=3144.99, stdev=2995.72 lat (usec): min=644, max=18858, avg=3145.84, stdev=2995.72 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 848], | 30.00th=[ 963], 40.00th=[ 1139], 50.00th=[ 1598], 60.00th=[ 3064], | 70.00th=[ 4047], 80.00th=[ 5276], 90.00th=[ 6915], 95.00th=[ 9241], | 99.00th=[13435], 99.50th=[16909], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 1888, max= 3136, per=24.66%, avg=2728.00, stdev=588.53, samples=4 iops : min= 118, max= 196, avg=170.50, stdev=36.78, samples=4 lat (usec) : 100=18.23%, 250=2.21%, 500=0.13%, 750=3.26%, 1000=13.93% lat (msec) : 2=10.16%, 4=25.26%, 10=24.22%, 20=2.60% cpu : usr=0.27%, sys=28.89%, ctx=1352, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.3MiB/s (10.8MB/s), 2522KiB/s-2774KiB/s (2582kB/s-2841kB/s), io=23.4MiB (24.5MB), run=2187-2278msec WRITE: bw=10.8MiB/s (11.3MB/s), 2675KiB/s-2912KiB/s (2739kB/s-2982kB/s), io=24.6MiB (25.8MB), run=2187-2278msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8802: Wed May 7 11:53:47 2025 read: IOPS=42, BW=677KiB/s (693kB/s)(5920KiB/8749msec) clat (usec): min=1761, max=24328, avg=3558.50, stdev=2716.97 lat (usec): min=1762, max=24329, avg=3559.13, stdev=2716.99 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 1975], 10.00th=[ 2057], 20.00th=[ 2147], | 30.00th=[ 2212], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2638], | 70.00th=[ 3326], 80.00th=[ 4424], 90.00th=[ 6063], 95.00th=[ 9896], | 99.00th=[15139], 99.50th=[16188], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 320, max= 1088, per=24.71%, avg=662.59, stdev=202.94, samples=17 iops : min= 20, max= 68, avg=41.41, stdev=12.68, samples=17 write: IOPS=45, BW=728KiB/s (745kB/s)(6368KiB/8749msec) clat (usec): min=10581, max=68304, avg=18597.00, stdev=6627.87 lat (usec): min=10582, max=68306, avg=18598.50, stdev=6627.97 clat percentiles (usec): | 1.00th=[11207], 5.00th=[11863], 10.00th=[12387], 20.00th=[13173], | 30.00th=[14484], 40.00th=[15795], 50.00th=[17433], 60.00th=[19006], | 70.00th=[20317], 80.00th=[21890], 90.00th=[26346], 95.00th=[30278], | 99.00th=[40633], 99.50th=[53740], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 352, max= 928, per=25.62%, avg=722.82, stdev=182.10, samples=17 iops : min= 22, max= 58, avg=45.18, stdev=11.38, samples=17 lat (msec) : 2=3.12%, 4=34.51%, 10=8.33%, 20=37.24%, 50=16.41% lat (msec) : 100=0.39% cpu : usr=0.06%, sys=6.07%, ctx=943, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8803: Wed May 7 11:53:47 2025 read: IOPS=40, BW=643KiB/s (658kB/s)(5744KiB/8933msec) clat (usec): min=1709, max=21049, avg=3579.07, stdev=2818.68 lat (usec): min=1710, max=21050, avg=3579.89, stdev=2818.82 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1991], 10.00th=[ 2057], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2343], 50.00th=[ 2507], 60.00th=[ 2737], | 70.00th=[ 3261], 80.00th=[ 4047], 90.00th=[ 6063], 95.00th=[10683], | 99.00th=[15664], 99.50th=[17695], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 288, max= 1088, per=24.22%, avg=649.24, stdev=226.99, samples=17 iops : min= 18, max= 68, avg=40.47, stdev=14.20, samples=17 write: IOPS=45, BW=733KiB/s (750kB/s)(6544KiB/8933msec) clat (usec): min=10692, max=57460, avg=18658.35, stdev=6046.96 lat (usec): min=10693, max=57462, avg=18659.55, stdev=6047.03 clat percentiles (usec): | 1.00th=[11207], 5.00th=[11731], 10.00th=[12125], 20.00th=[13173], | 30.00th=[14615], 40.00th=[16450], 50.00th=[17957], 60.00th=[19792], | 70.00th=[20579], 80.00th=[21890], 90.00th=[25297], 95.00th=[31065], | 99.00th=[37487], 99.50th=[40633], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 448, max= 928, per=25.81%, avg=728.24, stdev=163.19, samples=17 iops : min= 28, max= 58, avg=45.41, stdev=10.12, samples=17 lat (msec) : 2=2.60%, 4=34.77%, 10=7.03%, 20=36.20%, 50=19.27% lat (msec) : 100=0.13% cpu : usr=0.06%, sys=5.83%, ctx=895, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8804: Wed May 7 11:53:47 2025 read: IOPS=44, BW=720KiB/s (737kB/s)(6256KiB/8689msec) clat (usec): min=1799, max=26381, avg=3488.10, stdev=2976.27 lat (usec): min=1800, max=26381, avg=3488.77, stdev=2976.28 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1975], 10.00th=[ 2008], 20.00th=[ 2089], | 30.00th=[ 2147], 40.00th=[ 2245], 50.00th=[ 2409], 60.00th=[ 2540], | 70.00th=[ 2933], 80.00th=[ 3916], 90.00th=[ 6063], 95.00th=[ 9241], | 99.00th=[17171], 99.50th=[20841], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 287, max= 1312, per=27.02%, avg=724.41, stdev=307.53, samples=17 iops : min= 17, max= 82, avg=45.06, stdev=19.42, samples=17 write: IOPS=43, BW=694KiB/s (711kB/s)(6032KiB/8689msec) clat (usec): min=10993, max=47099, avg=19113.52, stdev=5880.94 lat (usec): min=10994, max=47101, avg=19114.78, stdev=5881.02 clat percentiles (usec): | 1.00th=[11076], 5.00th=[11994], 10.00th=[12780], 20.00th=[14091], | 30.00th=[15795], 40.00th=[17171], 50.00th=[18482], 60.00th=[19530], | 70.00th=[20841], 80.00th=[21890], 90.00th=[26084], 95.00th=[31065], | 99.00th=[42730], 99.50th=[46400], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 351, max= 864, per=24.48%, avg=690.53, stdev=159.49, samples=17 iops : min= 21, max= 54, avg=42.94, stdev=10.24, samples=17 lat (msec) : 2=4.04%, 4=36.85%, 10=7.68%, 20=34.11%, 50=17.32% cpu : usr=0.06%, sys=6.02%, ctx=927, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8805: Wed May 7 11:53:47 2025 read: IOPS=42, BW=687KiB/s (703kB/s)(6032KiB/8783msec) clat (usec): min=1752, max=26461, avg=3462.63, stdev=2733.10 lat (usec): min=1753, max=26461, avg=3463.30, stdev=2733.09 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1975], 10.00th=[ 2057], 20.00th=[ 2147], | 30.00th=[ 2212], 40.00th=[ 2278], 50.00th=[ 2442], 60.00th=[ 2606], | 70.00th=[ 3097], 80.00th=[ 3818], 90.00th=[ 6259], 95.00th=[ 9241], | 99.00th=[15533], 99.50th=[18220], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 224, max= 1024, per=25.34%, avg=679.41, stdev=221.26, samples=17 iops : min= 14, max= 64, avg=42.41, stdev=13.83, samples=17 write: IOPS=44, BW=712KiB/s (729kB/s)(6256KiB/8783msec) clat (usec): min=10693, max=54071, avg=18816.37, stdev=6162.98 lat (usec): min=10694, max=54075, avg=18817.52, stdev=6163.09 clat percentiles (usec): | 1.00th=[11338], 5.00th=[11863], 10.00th=[12256], 20.00th=[13304], | 30.00th=[14877], 40.00th=[16909], 50.00th=[17957], 60.00th=[19530], | 70.00th=[20841], 80.00th=[22152], 90.00th=[25822], 95.00th=[31327], | 99.00th=[41157], 99.50th=[44827], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 320, max= 896, per=25.22%, avg=711.41, stdev=177.31, samples=17 iops : min= 20, max= 56, avg=44.41, stdev=11.10, samples=17 lat (msec) : 2=3.12%, 4=36.85%, 10=6.90%, 20=35.03%, 50=17.97% lat (msec) : 100=0.13% cpu : usr=0.10%, sys=5.78%, ctx=902, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2681KiB/s (2746kB/s), 643KiB/s-720KiB/s (658kB/s-737kB/s), io=23.4MiB (24.5MB), run=8689-8933msec WRITE: bw=2821KiB/s (2889kB/s), 694KiB/s-733KiB/s (711kB/s-750kB/s), io=24.6MiB (25.8MB), run=8689-8933msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8819: Wed May 7 11:53:48 2025 read: IOPS=14, BW=14.5MiB/s (15.2MB/s)(7168KiB/484msec) clat (usec): min=13116, max=55684, avg=28897.45, stdev=17218.83 lat (usec): min=13116, max=55685, avg=28898.37, stdev=17218.94 clat percentiles (usec): | 1.00th=[13173], 5.00th=[13173], 10.00th=[13173], 20.00th=[17171], | 30.00th=[17957], 40.00th=[17957], 50.00th=[19268], 60.00th=[28443], | 70.00th=[28443], 80.00th=[50594], 90.00th=[55837], 95.00th=[55837], | 99.00th=[55837], 99.50th=[55837], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] write: IOPS=10, BW=10.3MiB/s (10.8MB/s)(5120KiB/484msec) clat (usec): min=29880, max=71725, avg=46043.58, stdev=16852.78 lat (usec): min=29910, max=71747, avg=46090.18, stdev=16844.61 clat percentiles (usec): | 1.00th=[29754], 5.00th=[29754], 10.00th=[29754], 20.00th=[29754], | 30.00th=[31851], 40.00th=[31851], 50.00th=[46924], 60.00th=[46924], | 70.00th=[49546], 80.00th=[49546], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] lat (msec) : 20=33.33%, 50=41.67%, 100=25.00% cpu : usr=0.00%, sys=13.66%, ctx=41, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8820: Wed May 7 11:53:48 2025 read: IOPS=2, BW=2702KiB/s (2767kB/s)(2048KiB/758msec) clat (usec): min=22416, max=70613, avg=46514.76, stdev=34080.69 lat (usec): min=22417, max=70614, avg=46515.65, stdev=34080.36 clat percentiles (usec): | 1.00th=[22414], 5.00th=[22414], 10.00th=[22414], 20.00th=[22414], | 30.00th=[22414], 40.00th=[22414], 50.00th=[22414], 60.00th=[70779], | 70.00th=[70779], 80.00th=[70779], 90.00th=[70779], 95.00th=[70779], | 99.00th=[70779], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 4087, max= 4087, per=15.13%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.2MiB/s (13.8MB/s)(10.0MiB/758msec) clat (usec): min=45899, max=94526, avg=64550.12, stdev=19483.83 lat (usec): min=45961, max=94599, avg=64616.49, stdev=19488.27 clat percentiles (usec): | 1.00th=[45876], 5.00th=[45876], 10.00th=[45876], 20.00th=[46400], | 30.00th=[47973], 40.00th=[49021], 50.00th=[53740], 60.00th=[62129], | 70.00th=[81265], 80.00th=[81265], 90.00th=[93848], 95.00th=[94897], | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min=10219, max=10219, per=27.02%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=58.33% cpu : usr=0.00%, sys=16.51%, ctx=41, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8821: Wed May 7 11:53:48 2025 read: IOPS=8, BW=8303KiB/s (8502kB/s)(6144KiB/740msec) clat (msec): min=28, max=110, avg=51.69, stdev=30.61 lat (msec): min=28, max=110, avg=51.69, stdev=30.61 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 42], 50.00th=[ 42], 60.00th=[ 42], | 70.00th=[ 58], 80.00th=[ 58], 90.00th=[ 111], 95.00th=[ 111], | 99.00th=[ 111], 99.50th=[ 111], 99.90th=[ 111], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 4096, max= 4096, per=15.16%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8303KiB/s (8502kB/s)(6144KiB/740msec) clat (msec): min=38, max=120, avg=69.82, stdev=37.54 lat (msec): min=38, max=120, avg=69.87, stdev=37.55 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 45], | 30.00th=[ 45], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 51], | 70.00th=[ 116], 80.00th=[ 116], 90.00th=[ 121], 95.00th=[ 121], | 99.00th=[ 121], 99.50th=[ 121], 99.90th=[ 121], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 8192, max= 8192, per=21.66%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=16.67%, 250=25.00% cpu : usr=0.00%, sys=9.74%, ctx=41, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8823: Wed May 7 11:53:48 2025 read: IOPS=8, BW=9159KiB/s (9379kB/s)(5120KiB/559msec) clat (usec): min=13311, max=58021, avg=28096.67, stdev=19230.00 lat (usec): min=13312, max=58021, avg=28097.30, stdev=19229.99 clat percentiles (usec): | 1.00th=[13304], 5.00th=[13304], 10.00th=[13304], 20.00th=[13304], | 30.00th=[15533], 40.00th=[15533], 50.00th=[16712], 60.00th=[16712], | 70.00th=[36963], 80.00th=[36963], 90.00th=[57934], 95.00th=[57934], | 99.00th=[57934], 99.50th=[57934], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 8192, max= 8192, per=30.32%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.5MiB/s (13.1MB/s)(7168KiB/559msec) clat (usec): min=27944, max=92458, avg=56467.15, stdev=26365.36 lat (usec): min=27964, max=92523, avg=56539.00, stdev=26365.06 clat percentiles (usec): | 1.00th=[27919], 5.00th=[27919], 10.00th=[27919], 20.00th=[31327], | 30.00th=[39584], 40.00th=[39584], 50.00th=[45876], 60.00th=[74974], | 70.00th=[74974], 80.00th=[82314], 90.00th=[92799], 95.00th=[92799], | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min=12288, max=12288, per=32.49%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=41.67%, 100=33.33% cpu : usr=0.00%, sys=12.90%, ctx=54, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=26.4MiB/s (27.7MB/s), 2702KiB/s-14.5MiB/s (2767kB/s-15.2MB/s), io=20.0MiB (20.0MB), run=484-758msec WRITE: bw=36.9MiB/s (38.7MB/s), 8303KiB/s-13.2MiB/s (8502kB/s-13.8MB/s), io=28.0MiB (29.4MB), run=484-758msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8822: Wed May 7 11:53:48 2025 read: IOPS=12, BW=12.6MiB/s (13.2MB/s)(7168KiB/554msec) clat (usec): min=407, max=103522, avg=35911.77, stdev=34582.40 lat (usec): min=408, max=103523, avg=35912.62, stdev=34582.48 clat percentiles (usec): | 1.00th=[ 408], 5.00th=[ 408], 10.00th=[ 408], 20.00th=[ 453], | 30.00th=[ 30016], 40.00th=[ 30016], 50.00th=[ 34866], 60.00th=[ 40109], | 70.00th=[ 40109], 80.00th=[ 41681], 90.00th=[103285], 95.00th=[103285], | 99.00th=[103285], 99.50th=[103285], 99.90th=[103285], 99.95th=[103285], | 99.99th=[103285] bw ( KiB/s): min=10199, max=10199, per=39.34%, avg=10199.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9242KiB/s (9464kB/s)(5120KiB/554msec) clat (msec): min=19, max=107, avg=46.19, stdev=35.58 lat (msec): min=19, max=107, avg=46.22, stdev=35.59 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 40], 60.00th=[ 40], | 70.00th=[ 41], 80.00th=[ 41], 90.00th=[ 108], 95.00th=[ 108], | 99.00th=[ 108], 99.50th=[ 108], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 6119, max= 6119, per=16.86%, avg=6119.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 20=8.33%, 50=58.33%, 250=16.67% cpu : usr=0.00%, sys=29.29%, ctx=602, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8824: Wed May 7 11:53:48 2025 read: IOPS=2, BW=2592KiB/s (2655kB/s)(2048KiB/790msec) clat (usec): min=34126, max=36717, avg=35421.92, stdev=1832.15 lat (usec): min=34127, max=36718, avg=35423.24, stdev=1832.14 clat percentiles (usec): | 1.00th=[34341], 5.00th=[34341], 10.00th=[34341], 20.00th=[34341], | 30.00th=[34341], 40.00th=[34341], 50.00th=[34341], 60.00th=[36963], | 70.00th=[36963], 80.00th=[36963], 90.00th=[36963], 95.00th=[36963], | 99.00th=[36963], 99.50th=[36963], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 2048, max= 2048, per=7.90%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.7MiB/s (13.3MB/s)(10.0MiB/790msec) clat (msec): min=26, max=134, avg=67.11, stdev=32.78 lat (msec): min=26, max=134, avg=67.16, stdev=32.78 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 32], | 30.00th=[ 34], 40.00th=[ 57], 50.00th=[ 62], 60.00th=[ 77], | 70.00th=[ 83], 80.00th=[ 83], 90.00th=[ 89], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min=10240, max=10240, per=28.21%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=50.00%, 250=8.33% cpu : usr=0.00%, sys=32.19%, ctx=93, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8825: Wed May 7 11:53:48 2025 read: IOPS=11, BW=11.5MiB/s (12.0MB/s)(6144KiB/524msec) clat (usec): min=369, max=32303, avg=19553.68, stdev=15043.55 lat (usec): min=370, max=32304, avg=19554.27, stdev=15043.52 clat percentiles (usec): | 1.00th=[ 371], 5.00th=[ 371], 10.00th=[ 371], 20.00th=[ 537], | 30.00th=[ 537], 40.00th=[24249], 50.00th=[24249], 60.00th=[29230], | 70.00th=[30540], 80.00th=[30540], 90.00th=[32375], 95.00th=[32375], | 99.00th=[32375], 99.50th=[32375], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min=10240, max=10240, per=39.50%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.5MiB/s (12.0MB/s)(6144KiB/524msec) clat (msec): min=34, max=138, avg=65.46, stdev=38.33 lat (msec): min=34, max=138, avg=65.50, stdev=38.34 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 59], | 70.00th=[ 72], 80.00th=[ 72], 90.00th=[ 140], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min=12288, max=12288, per=33.86%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=58.33%, 100=16.67%, 250=8.33% cpu : usr=0.00%, sys=28.30%, ctx=406, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8826: Wed May 7 11:53:48 2025 read: IOPS=8, BW=9143KiB/s (9362kB/s)(5120KiB/560msec) clat (usec): min=417, max=35701, avg=16165.09, stdev=16651.94 lat (usec): min=418, max=35701, avg=16166.06, stdev=16651.73 clat percentiles (usec): | 1.00th=[ 416], 5.00th=[ 416], 10.00th=[ 416], 20.00th=[ 416], | 30.00th=[ 474], 40.00th=[ 474], 50.00th=[13042], 60.00th=[13042], | 70.00th=[31065], 80.00th=[31065], 90.00th=[35914], 95.00th=[35914], | 99.00th=[35914], 99.50th=[35914], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 8175, max= 8175, per=31.53%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.5MiB/s (13.1MB/s)(7168KiB/560msec) clat (msec): min=19, max=127, avg=59.72, stdev=43.38 lat (msec): min=19, max=127, avg=59.78, stdev=43.39 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 29], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 34], 60.00th=[ 66], | 70.00th=[ 66], 80.00th=[ 111], 90.00th=[ 128], 95.00th=[ 128], | 99.00th=[ 128], 99.50th=[ 128], 99.90th=[ 128], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min=12263, max=12263, per=33.79%, avg=12263.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 20=16.67%, 50=41.67%, 100=8.33%, 250=16.67% cpu : usr=0.00%, sys=24.64%, ctx=282, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.3MiB/s (26.5MB/s), 2592KiB/s-12.6MiB/s (2655kB/s-13.2MB/s), io=20.0MiB (20.0MB), run=524-790msec WRITE: bw=35.4MiB/s (37.2MB/s), 9242KiB/s-12.7MiB/s (9464kB/s-13.3MB/s), io=28.0MiB (29.4MB), run=524-790msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8832: Wed May 7 11:53:50 2025 read: IOPS=5, BW=22.1MiB/s (23.2MB/s)(12.0MiB/543msec) clat (msec): min=90, max=222, avg=176.56, stdev=74.24 lat (msec): min=90, max=222, avg=176.56, stdev=74.24 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 91], 10.00th=[ 91], 20.00th=[ 91], | 30.00th=[ 91], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 215], | 70.00th=[ 224], 80.00th=[ 224], 90.00th=[ 224], 95.00th=[ 224], | 99.00th=[ 224], 99.50th=[ 224], 99.90th=[ 224], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min=15845, max=15845, per=54.45%, avg=15845.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=9.96%, ctx=36, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8834: Wed May 7 11:53:50 2025 read: IOPS=1, BW=7275KiB/s (7450kB/s)(4096KiB/563msec) clat (nsec): min=88806k, max=88806k, avg=88806343.00, stdev= 0.00 lat (nsec): min=88807k, max=88807k, avg=88807288.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[88605], 5.00th=[88605], 10.00th=[88605], 20.00th=[88605], | 30.00th=[88605], 40.00th=[88605], 50.00th=[88605], 60.00th=[88605], | 70.00th=[88605], 80.00th=[88605], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 8192, max= 8192, per=28.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=14.2MiB/s (14.9MB/s)(8192KiB/563msec) clat (msec): min=161, max=280, avg=220.86, stdev=84.52 lat (msec): min=161, max=280, avg=221.12, stdev=84.54 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 279], | 70.00th=[ 279], 80.00th=[ 279], 90.00th=[ 279], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 8192, max= 8192, per=14.08%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=12.99%, ctx=54, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8835: Wed May 7 11:53:50 2025 write: IOPS=5, BW=23.7MiB/s (24.8MB/s)(12.0MiB/507msec) clat (msec): min=110, max=182, avg=150.29, stdev=36.30 lat (msec): min=111, max=182, avg=150.61, stdev=36.32 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 159], 50.00th=[ 159], 60.00th=[ 159], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min=16384, max=16384, per=28.15%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.40%, sys=21.74%, ctx=97, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8837: Wed May 7 11:53:50 2025 write: IOPS=5, BW=23.9MiB/s (25.1MB/s)(12.0MiB/502msec) clat (msec): min=109, max=210, avg=160.58, stdev=50.47 lat (msec): min=110, max=211, avg=160.88, stdev=50.50 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 110], 10.00th=[ 110], 20.00th=[ 110], | 30.00th=[ 110], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 161], | 70.00th=[ 211], 80.00th=[ 211], 90.00th=[ 211], 95.00th=[ 211], | 99.00th=[ 211], 99.50th=[ 211], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min=16384, max=16384, per=28.15%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=18.56%, ctx=45, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=28.4MiB/s (29.8MB/s), 7275KiB/s-22.1MiB/s (7450kB/s-23.2MB/s), io=16.0MiB (16.8MB), run=543-563msec WRITE: bw=56.8MiB/s (59.6MB/s), 14.2MiB/s-23.9MiB/s (14.9MB/s-25.1MB/s), io=32.0MiB (33.6MB), run=502-563msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8833: Wed May 7 11:53:50 2025 read: IOPS=8, BW=33.5MiB/s (35.1MB/s)(12.0MiB/358msec) clat (msec): min=81, max=156, avg=116.96, stdev=37.56 lat (msec): min=81, max=156, avg=116.96, stdev=37.56 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 82], 10.00th=[ 82], 20.00th=[ 82], | 30.00th=[ 82], 40.00th=[ 114], 50.00th=[ 114], 60.00th=[ 114], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=36.69%, ctx=1018, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8836: Wed May 7 11:53:50 2025 read: IOPS=1, BW=6024KiB/s (6168kB/s)(4096KiB/680msec) clat (nsec): min=99026k, max=99026k, avg=99026346.00, stdev= 0.00 lat (nsec): min=99027k, max=99027k, avg=99027211.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[99091], 5.00th=[99091], 10.00th=[99091], 20.00th=[99091], | 30.00th=[99091], 40.00th=[99091], 50.00th=[99091], 60.00th=[99091], | 70.00th=[99091], 80.00th=[99091], 90.00th=[99091], 95.00th=[99091], | 99.00th=[99091], 99.50th=[99091], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] write: IOPS=2, BW=11.8MiB/s (12.3MB/s)(8192KiB/680msec) clat (msec): min=241, max=330, avg=286.02, stdev=63.55 lat (msec): min=241, max=331, avg=286.27, stdev=63.57 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 241], 10.00th=[ 241], 20.00th=[ 241], | 30.00th=[ 241], 40.00th=[ 241], 50.00th=[ 241], 60.00th=[ 330], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=15.91%, ctx=581, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8838: Wed May 7 11:53:50 2025 write: IOPS=4, BW=16.6MiB/s (17.4MB/s)(12.0MiB/723msec) clat (msec): min=86, max=312, avg=233.29, stdev=127.55 lat (msec): min=86, max=312, avg=233.63, stdev=127.61 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 87], 10.00th=[ 87], 20.00th=[ 87], | 30.00th=[ 87], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 313], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.00%, sys=33.24%, ctx=60, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8839: Wed May 7 11:53:50 2025 write: IOPS=7, BW=30.2MiB/s (31.7MB/s)(12.0MiB/397msec) clat (msec): min=95, max=164, avg=125.09, stdev=35.21 lat (msec): min=96, max=164, avg=125.41, stdev=35.19 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 96], 10.00th=[ 96], 20.00th=[ 96], | 30.00th=[ 96], 40.00th=[ 116], 50.00th=[ 116], 60.00th=[ 116], | 70.00th=[ 165], 80.00th=[ 165], 90.00th=[ 165], 95.00th=[ 165], | 99.00th=[ 165], 99.50th=[ 165], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=50.00%, ctx=71, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=23.5MiB/s (24.7MB/s), 6024KiB/s-33.5MiB/s (6168kB/s-35.1MB/s), io=16.0MiB (16.8MB), run=358-680msec WRITE: bw=44.3MiB/s (46.4MB/s), 11.8MiB/s-30.2MiB/s (12.3MB/s-31.7MB/s), io=32.0MiB (33.6MB), run=397-723msec