== sanity test 398b: DIO and buffer IO race ============== 14:34:40 (1744137280) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.11331 s, 45.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30225: Tue Apr 8 14:34:56 2025 read: IOPS=108, BW=434KiB/s (444kB/s)(5992KiB/13807msec) clat (usec): min=67, max=165228, avg=5165.52, stdev=5079.58 lat (usec): min=68, max=165229, avg=5166.24, stdev=5079.56 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 277], 10.00th=[ 2311], 20.00th=[ 2900], | 30.00th=[ 3425], 40.00th=[ 3884], 50.00th=[ 4490], 60.00th=[ 5211], | 70.00th=[ 5997], 80.00th=[ 6980], 90.00th=[ 8356], 95.00th=[ 10159], | 99.00th=[ 16188], 99.50th=[ 17957], 99.90th=[ 26870], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 248, max= 544, per=25.17%, avg=430.93, stdev=68.44, samples=27 iops : min= 62, max= 136, avg=107.63, stdev=17.15, samples=27 write: IOPS=114, BW=456KiB/s (467kB/s)(6296KiB/13807msec) clat (usec): min=432, max=43749, avg=3811.37, stdev=3597.90 lat (usec): min=432, max=43750, avg=3812.17, stdev=3597.93 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 570], 10.00th=[ 635], 20.00th=[ 775], | 30.00th=[ 930], 40.00th=[ 1631], 50.00th=[ 2966], 60.00th=[ 3949], | 70.00th=[ 4883], 80.00th=[ 6456], 90.00th=[ 8586], 95.00th=[10683], | 99.00th=[14484], 99.50th=[17433], 99.90th=[24511], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 296, max= 560, per=25.27%, avg=452.89, stdev=70.12, samples=27 iops : min= 74, max= 140, avg=113.11, stdev=17.55, samples=27 lat (usec) : 100=1.43%, 250=0.98%, 500=0.29%, 750=8.92%, 1000=7.55% lat (msec) : 2=5.60%, 4=26.92%, 10=42.48%, 20=5.60%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=22.33%, ctx=5682, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30226: Tue Apr 8 14:34:56 2025 read: IOPS=108, BW=433KiB/s (443kB/s)(5940KiB/13727msec) clat (usec): min=62, max=37364, avg=4930.98, stdev=3059.62 lat (usec): min=63, max=37365, avg=4932.53, stdev=3059.35 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 133], 10.00th=[ 2311], 20.00th=[ 2835], | 30.00th=[ 3294], 40.00th=[ 3785], 50.00th=[ 4424], 60.00th=[ 5080], | 70.00th=[ 5735], 80.00th=[ 6783], 90.00th=[ 8291], 95.00th=[ 9896], | 99.00th=[15270], 99.50th=[18482], 99.90th=[33817], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 272, max= 544, per=25.19%, avg=431.33, stdev=70.08, samples=27 iops : min= 68, max= 136, avg=107.78, stdev=17.54, samples=27 write: IOPS=115, BW=462KiB/s (474kB/s)(6348KiB/13727msec) clat (usec): min=438, max=164003, avg=3991.83, stdev=5309.95 lat (usec): min=438, max=164004, avg=3994.25, stdev=5310.46 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 570], 10.00th=[ 619], 20.00th=[ 791], | 30.00th=[ 996], 40.00th=[ 2114], 50.00th=[ 3097], 60.00th=[ 3916], | 70.00th=[ 5080], 80.00th=[ 6456], 90.00th=[ 8848], 95.00th=[ 10683], | 99.00th=[ 15270], 99.50th=[ 16909], 99.90th=[ 23200], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 304, max= 624, per=25.46%, avg=456.19, stdev=81.17, samples=27 iops : min= 76, max= 156, avg=114.00, stdev=20.25, samples=27 lat (usec) : 100=1.60%, 250=0.98%, 500=0.33%, 750=8.69%, 1000=6.58% lat (msec) : 2=5.47%, 4=28.55%, 10=42.32%, 20=5.21%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=22.45%, ctx=5837, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30227: Tue Apr 8 14:34:56 2025 read: IOPS=107, BW=431KiB/s (442kB/s)(6028KiB/13979msec) clat (usec): min=54, max=164195, avg=5238.88, stdev=5105.52 lat (usec): min=54, max=164195, avg=5239.90, stdev=5105.61 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 1647], 10.00th=[ 2376], 20.00th=[ 2900], | 30.00th=[ 3490], 40.00th=[ 4015], 50.00th=[ 4621], 60.00th=[ 5276], | 70.00th=[ 5997], 80.00th=[ 7177], 90.00th=[ 8717], 95.00th=[ 10028], | 99.00th=[ 16712], 99.50th=[ 19792], 99.90th=[ 28967], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 248, max= 552, per=25.03%, avg=428.59, stdev=66.50, samples=27 iops : min= 62, max= 138, avg=107.04, stdev=16.69, samples=27 write: IOPS=111, BW=448KiB/s (459kB/s)(6260KiB/13979msec) clat (usec): min=423, max=26227, avg=3842.51, stdev=3583.77 lat (usec): min=423, max=26228, avg=3843.58, stdev=3583.78 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 570], 10.00th=[ 627], 20.00th=[ 766], | 30.00th=[ 922], 40.00th=[ 1860], 50.00th=[ 2966], 60.00th=[ 3851], | 70.00th=[ 4948], 80.00th=[ 6390], 90.00th=[ 8848], 95.00th=[10814], | 99.00th=[15795], 99.50th=[17695], 99.90th=[21103], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 272, max= 608, per=24.68%, avg=442.19, stdev=83.05, samples=27 iops : min= 68, max= 152, avg=110.44, stdev=20.74, samples=27 lat (usec) : 100=1.37%, 250=1.04%, 500=0.42%, 750=9.08%, 1000=7.26% lat (msec) : 2=4.82%, 4=26.86%, 10=43.26%, 20=5.53%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=22.22%, ctx=5714, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30228: Tue Apr 8 14:34:56 2025 read: IOPS=107, BW=432KiB/s (442kB/s)(6052KiB/14025msec) clat (usec): min=59, max=34798, avg=5078.64, stdev=3051.60 lat (usec): min=59, max=34799, avg=5079.56, stdev=3051.78 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 322], 10.00th=[ 2311], 20.00th=[ 2868], | 30.00th=[ 3359], 40.00th=[ 3982], 50.00th=[ 4555], 60.00th=[ 5211], | 70.00th=[ 6063], 80.00th=[ 6915], 90.00th=[ 8586], 95.00th=[10290], | 99.00th=[14746], 99.50th=[19006], 99.90th=[29754], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 296, max= 608, per=25.16%, avg=430.75, stdev=70.54, samples=28 iops : min= 74, max= 152, avg=107.64, stdev=17.59, samples=28 write: IOPS=111, BW=445KiB/s (455kB/s)(6236KiB/14025msec) clat (usec): min=411, max=152935, avg=4027.82, stdev=5244.08 lat (usec): min=412, max=152936, avg=4028.76, stdev=5244.16 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 570], 10.00th=[ 619], 20.00th=[ 775], | 30.00th=[ 947], 40.00th=[ 1893], 50.00th=[ 3064], 60.00th=[ 3916], | 70.00th=[ 5211], 80.00th=[ 6849], 90.00th=[ 8717], 95.00th=[ 10945], | 99.00th=[ 15401], 99.50th=[ 19792], 99.90th=[ 25297], 99.95th=[152044], | 99.99th=[152044] bw ( KiB/s): min= 288, max= 600, per=24.72%, avg=443.07, stdev=75.67, samples=28 iops : min= 72, max= 150, avg=110.71, stdev=18.91, samples=28 lat (usec) : 100=1.46%, 250=0.94%, 500=0.42%, 750=8.72%, 1000=7.19% lat (msec) : 2=5.14%, 4=27.08%, 10=43.10%, 20=5.50%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=22.17%, ctx=5891, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1712KiB/s (1753kB/s), 431KiB/s-434KiB/s (442kB/s-444kB/s), io=23.4MiB (24.6MB), run=13727-14025msec WRITE: bw=1793KiB/s (1836kB/s), 445KiB/s-462KiB/s (455kB/s-474kB/s), io=24.6MiB (25.7MB), run=13727-14025msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30230: Tue Apr 8 14:35:01 2025 read: IOPS=77, BW=312KiB/s (319kB/s)(5992KiB/19213msec) clat (usec): min=1546, max=166928, avg=5142.55, stdev=5819.27 lat (usec): min=1546, max=166929, avg=5143.33, stdev=5819.27 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1860], 10.00th=[ 2008], 20.00th=[ 2245], | 30.00th=[ 2573], 40.00th=[ 3032], 50.00th=[ 3523], 60.00th=[ 4359], | 70.00th=[ 5276], 80.00th=[ 7177], 90.00th=[ 10290], 95.00th=[ 13042], | 99.00th=[ 19530], 99.50th=[ 24773], 99.90th=[ 44303], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 144, max= 848, per=25.05%, avg=311.61, stdev=155.25, samples=38 iops : min= 36, max= 212, avg=77.79, stdev=38.83, samples=38 write: IOPS=81, BW=328KiB/s (336kB/s)(6296KiB/19213msec) clat (usec): min=2157, max=38250, avg=7291.74, stdev=4393.33 lat (usec): min=2158, max=38251, avg=7292.80, stdev=4393.42 clat percentiles (usec): | 1.00th=[ 2507], 5.00th=[ 2933], 10.00th=[ 3195], 20.00th=[ 3720], | 30.00th=[ 4359], 40.00th=[ 5211], 50.00th=[ 6063], 60.00th=[ 7111], | 70.00th=[ 8586], 80.00th=[10290], 90.00th=[12780], 95.00th=[15533], | 99.00th=[22938], 99.50th=[27919], 99.90th=[32375], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 152, max= 752, per=25.10%, avg=326.76, stdev=151.44, samples=38 iops : min= 38, max= 188, avg=81.58, stdev=37.86, samples=38 lat (msec) : 2=4.82%, 4=35.68%, 10=43.42%, 20=14.75%, 50=1.30% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=10.91%, ctx=3741, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30231: Tue Apr 8 14:35:01 2025 read: IOPS=77, BW=309KiB/s (316kB/s)(5940KiB/19242msec) clat (usec): min=1553, max=164764, avg=5096.38, stdev=5611.57 lat (usec): min=1553, max=164765, avg=5097.25, stdev=5611.60 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1909], 10.00th=[ 2008], 20.00th=[ 2311], | 30.00th=[ 2606], 40.00th=[ 3032], 50.00th=[ 3556], 60.00th=[ 4293], | 70.00th=[ 5407], 80.00th=[ 7046], 90.00th=[ 10159], 95.00th=[ 13042], | 99.00th=[ 19792], 99.50th=[ 21627], 99.90th=[ 30016], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 128, max= 808, per=24.88%, avg=309.53, stdev=159.53, samples=38 iops : min= 32, max= 202, avg=77.29, stdev=39.87, samples=38 write: IOPS=82, BW=330KiB/s (338kB/s)(6348KiB/19242msec) clat (usec): min=2220, max=48078, avg=7320.16, stdev=4543.72 lat (usec): min=2221, max=48079, avg=7321.01, stdev=4543.78 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2933], 10.00th=[ 3195], 20.00th=[ 3687], | 30.00th=[ 4178], 40.00th=[ 5080], 50.00th=[ 6063], 60.00th=[ 7111], | 70.00th=[ 8455], 80.00th=[10290], 90.00th=[13304], 95.00th=[16057], | 99.00th=[23200], 99.50th=[26346], 99.90th=[37487], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 168, max= 790, per=25.28%, avg=329.11, stdev=149.42, samples=38 iops : min= 42, max= 197, avg=82.18, stdev=37.34, samples=38 lat (msec) : 2=4.65%, 4=36.00%, 10=43.10%, 20=14.84%, 50=1.37% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=10.90%, ctx=3760, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30232: Tue Apr 8 14:35:01 2025 read: IOPS=78, BW=312KiB/s (320kB/s)(6028KiB/19300msec) clat (usec): min=1619, max=27578, avg=5227.51, stdev=3928.12 lat (usec): min=1620, max=27579, avg=5228.23, stdev=3928.18 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1926], 10.00th=[ 2040], 20.00th=[ 2278], | 30.00th=[ 2638], 40.00th=[ 3032], 50.00th=[ 3621], 60.00th=[ 4490], | 70.00th=[ 5735], 80.00th=[ 7832], 90.00th=[11076], 95.00th=[13566], | 99.00th=[18744], 99.50th=[20841], 99.90th=[26084], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 152, max= 848, per=25.14%, avg=312.68, stdev=147.91, samples=38 iops : min= 38, max= 212, avg=78.08, stdev=36.99, samples=38 write: IOPS=81, BW=324KiB/s (332kB/s)(6260KiB/19300msec) clat (msec): min=2, max=173, avg= 7.27, stdev= 6.08 lat (msec): min=2, max=173, avg= 7.27, stdev= 6.08 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 13], 95.00th=[ 16], | 99.00th=[ 24], 99.50th=[ 26], 99.90th=[ 48], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 152, max= 768, per=24.84%, avg=323.42, stdev=152.90, samples=38 iops : min= 38, max= 192, avg=80.76, stdev=38.23, samples=38 lat (msec) : 2=3.87%, 4=35.74%, 10=43.85%, 20=15.40%, 50=1.11% lat (msec) : 250=0.03% cpu : usr=0.10%, sys=11.02%, ctx=3844, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30233: Tue Apr 8 14:35:01 2025 read: IOPS=78, BW=314KiB/s (321kB/s)(6052KiB/19280msec) clat (usec): min=1517, max=40392, avg=5107.40, stdev=4043.55 lat (usec): min=1517, max=40393, avg=5108.18, stdev=4043.57 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1893], 10.00th=[ 2024], 20.00th=[ 2278], | 30.00th=[ 2638], 40.00th=[ 3064], 50.00th=[ 3523], 60.00th=[ 4359], | 70.00th=[ 5604], 80.00th=[ 7111], 90.00th=[10552], 95.00th=[12911], | 99.00th=[19530], 99.50th=[23462], 99.90th=[32375], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 168, max= 808, per=25.18%, avg=313.18, stdev=150.79, samples=38 iops : min= 42, max= 202, avg=78.24, stdev=37.70, samples=38 write: IOPS=80, BW=323KiB/s (331kB/s)(6236KiB/19280msec) clat (msec): min=2, max=176, avg= 7.38, stdev= 6.16 lat (msec): min=2, max=176, avg= 7.38, stdev= 6.16 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 6], 60.00th=[ 8], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 13], 95.00th=[ 16], | 99.00th=[ 25], 99.50th=[ 29], 99.90th=[ 36], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 175, max= 728, per=24.83%, avg=323.29, stdev=146.86, samples=38 iops : min= 43, max= 182, avg=80.76, stdev=36.74, samples=38 lat (msec) : 2=4.46%, 4=35.81%, 10=43.42%, 20=14.68%, 50=1.60% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=10.90%, ctx=3850, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1244KiB/s (1274kB/s), 309KiB/s-314KiB/s (316kB/s-321kB/s), io=23.4MiB (24.6MB), run=19213-19300msec WRITE: bw=1303KiB/s (1334kB/s), 323KiB/s-330KiB/s (331kB/s-338kB/s), io=24.6MiB (25.7MB), run=19213-19300msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30267: Tue Apr 8 14:35:06 2025 read: IOPS=113, BW=1809KiB/s (1852kB/s)(5920KiB/3273msec) clat (usec): min=58, max=17465, avg=4819.56, stdev=2927.65 lat (usec): min=58, max=17466, avg=4820.15, stdev=2927.67 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 97], 10.00th=[ 1909], 20.00th=[ 2606], | 30.00th=[ 3228], 40.00th=[ 3752], 50.00th=[ 4359], 60.00th=[ 5145], | 70.00th=[ 5800], 80.00th=[ 6915], 90.00th=[ 8455], 95.00th=[10552], | 99.00th=[13042], 99.50th=[16909], 99.90th=[17433], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 1120, max= 2144, per=23.90%, avg=1712.00, stdev=340.32, samples=6 iops : min= 70, max= 134, avg=107.00, stdev=21.27, samples=6 write: IOPS=121, BW=1946KiB/s (1992kB/s)(6368KiB/3273msec) clat (usec): min=618, max=19471, avg=3604.33, stdev=3328.08 lat (usec): min=619, max=19473, avg=3609.48, stdev=3326.51 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 709], 10.00th=[ 758], 20.00th=[ 947], | 30.00th=[ 1205], 40.00th=[ 1614], 50.00th=[ 2474], 60.00th=[ 3326], | 70.00th=[ 4359], 80.00th=[ 5604], 90.00th=[ 8225], 95.00th=[10421], | 99.00th=[16319], 99.50th=[18482], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 1280, max= 2496, per=24.76%, avg=1866.67, stdev=535.72, samples=6 iops : min= 80, max= 156, avg=116.67, stdev=33.48, samples=6 lat (usec) : 100=2.73%, 250=1.56%, 750=4.69%, 1000=6.64% lat (msec) : 2=12.24%, 4=28.12%, 10=38.15%, 20=5.86% cpu : usr=0.12%, sys=24.66%, ctx=1583, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30269: Tue Apr 8 14:35:06 2025 read: IOPS=109, BW=1750KiB/s (1792kB/s)(5744KiB/3282msec) clat (usec): min=52, max=21746, avg=4694.06, stdev=2893.91 lat (usec): min=52, max=21747, avg=4694.97, stdev=2893.82 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 100], 10.00th=[ 1909], 20.00th=[ 2638], | 30.00th=[ 3195], 40.00th=[ 3589], 50.00th=[ 4228], 60.00th=[ 4948], | 70.00th=[ 5669], 80.00th=[ 6652], 90.00th=[ 8291], 95.00th=[ 9372], | 99.00th=[15401], 99.50th=[17171], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 1248, max= 2400, per=23.82%, avg=1706.67, stdev=440.00, samples=6 iops : min= 78, max= 150, avg=106.67, stdev=27.50, samples=6 write: IOPS=124, BW=1994KiB/s (2042kB/s)(6544KiB/3282msec) clat (usec): min=593, max=14805, avg=3861.15, stdev=3277.12 lat (usec): min=594, max=14807, avg=3862.33, stdev=3277.22 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 709], 10.00th=[ 758], 20.00th=[ 1020], | 30.00th=[ 1254], 40.00th=[ 2057], 50.00th=[ 2835], 60.00th=[ 3752], | 70.00th=[ 4883], 80.00th=[ 6456], 90.00th=[ 8848], 95.00th=[10552], | 99.00th=[13829], 99.50th=[14353], 99.90th=[14746], 99.95th=[14746], | 99.99th=[14746] bw ( KiB/s): min= 1472, max= 2528, per=25.54%, avg=1925.33, stdev=408.51, samples=6 iops : min= 92, max= 158, avg=120.33, stdev=25.53, samples=6 lat (usec) : 100=2.34%, 250=0.91%, 750=4.56%, 1000=5.86% lat (msec) : 2=11.98%, 4=29.30%, 10=40.49%, 20=4.43%, 50=0.13% cpu : usr=0.15%, sys=24.44%, ctx=1739, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30271: Tue Apr 8 14:35:06 2025 read: IOPS=116, BW=1871KiB/s (1916kB/s)(6256KiB/3343msec) clat (usec): min=55, max=22270, avg=4642.53, stdev=2977.29 lat (usec): min=55, max=22271, avg=4643.41, stdev=2977.57 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 91], 10.00th=[ 1713], 20.00th=[ 2606], | 30.00th=[ 3195], 40.00th=[ 3621], 50.00th=[ 4113], 60.00th=[ 4948], | 70.00th=[ 5604], 80.00th=[ 6456], 90.00th=[ 7963], 95.00th=[10028], | 99.00th=[14091], 99.50th=[20317], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 960, max= 2240, per=24.02%, avg=1720.83, stdev=498.19, samples=6 iops : min= 60, max= 140, avg=107.33, stdev=31.08, samples=6 write: IOPS=112, BW=1804KiB/s (1848kB/s)(6032KiB/3343msec) clat (usec): min=597, max=21151, avg=3861.09, stdev=3549.91 lat (usec): min=598, max=21153, avg=3862.09, stdev=3549.91 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 693], 10.00th=[ 742], 20.00th=[ 971], | 30.00th=[ 1237], 40.00th=[ 1860], 50.00th=[ 2900], 60.00th=[ 3556], | 70.00th=[ 4555], 80.00th=[ 5997], 90.00th=[ 8848], 95.00th=[12125], | 99.00th=[15008], 99.50th=[15926], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 1248, max= 2363, per=22.54%, avg=1699.33, stdev=429.85, samples=6 iops : min= 78, max= 147, avg=106.00, stdev=26.69, samples=6 lat (usec) : 100=2.99%, 250=1.43%, 500=0.13%, 750=5.34%, 1000=4.82% lat (msec) : 2=11.85%, 4=29.95%, 10=36.85%, 20=6.25%, 50=0.39% cpu : usr=0.15%, sys=23.52%, ctx=1746, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30273: Tue Apr 8 14:35:06 2025 read: IOPS=116, BW=1865KiB/s (1909kB/s)(6032KiB/3235msec) clat (usec): min=54, max=23024, avg=4604.15, stdev=3287.12 lat (usec): min=54, max=23024, avg=4605.48, stdev=3287.08 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 81], 10.00th=[ 118], 20.00th=[ 2343], | 30.00th=[ 2966], 40.00th=[ 3523], 50.00th=[ 4228], 60.00th=[ 4817], | 70.00th=[ 5538], 80.00th=[ 6521], 90.00th=[ 8586], 95.00th=[10290], | 99.00th=[17433], 99.50th=[19006], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 1216, max= 2240, per=24.63%, avg=1764.67, stdev=373.42, samples=6 iops : min= 76, max= 140, avg=110.17, stdev=23.45, samples=6 write: IOPS=120, BW=1934KiB/s (1980kB/s)(6256KiB/3235msec) clat (usec): min=552, max=18274, avg=3628.67, stdev=3142.75 lat (usec): min=552, max=18275, avg=3630.73, stdev=3144.76 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 701], 10.00th=[ 766], 20.00th=[ 963], | 30.00th=[ 1188], 40.00th=[ 1663], 50.00th=[ 2835], 60.00th=[ 3818], | 70.00th=[ 4686], 80.00th=[ 5735], 90.00th=[ 8160], 95.00th=[ 9503], | 99.00th=[14877], 99.50th=[15926], 99.90th=[18220], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 1245, max= 2528, per=24.76%, avg=1866.17, stdev=527.56, samples=6 iops : min= 77, max= 158, avg=116.50, stdev=33.16, samples=6 lat (usec) : 100=3.39%, 250=2.73%, 750=4.69%, 1000=6.25% lat (msec) : 2=13.02%, 4=24.61%, 10=40.49%, 20=4.69%, 50=0.13% cpu : usr=0.25%, sys=23.93%, ctx=1663, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7165KiB/s (7337kB/s), 1750KiB/s-1871KiB/s (1792kB/s-1916kB/s), io=23.4MiB (24.5MB), run=3235-3343msec WRITE: bw=7538KiB/s (7719kB/s), 1804KiB/s-1994KiB/s (1848kB/s-2042kB/s), io=24.6MiB (25.8MB), run=3235-3343msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30266: Tue Apr 8 14:35:07 2025 read: IOPS=81, BW=1297KiB/s (1329kB/s)(5920KiB/4563msec) clat (usec): min=1499, max=27987, avg=4563.82, stdev=3499.60 lat (usec): min=1500, max=27988, avg=4564.60, stdev=3499.60 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1975], 10.00th=[ 2147], 20.00th=[ 2343], | 30.00th=[ 2573], 40.00th=[ 2933], 50.00th=[ 3392], 60.00th=[ 3916], | 70.00th=[ 4555], 80.00th=[ 6194], 90.00th=[ 8094], 95.00th=[11600], | 99.00th=[17433], 99.50th=[24773], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 640, max= 2656, per=24.46%, avg=1284.00, stdev=633.21, samples=8 iops : min= 40, max= 166, avg=80.25, stdev=39.58, samples=8 write: IOPS=87, BW=1396KiB/s (1429kB/s)(6368KiB/4563msec) clat (usec): min=2349, max=29705, avg=7107.95, stdev=4698.63 lat (usec): min=2350, max=29706, avg=7109.16, stdev=4698.80 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2966], 10.00th=[ 3130], 20.00th=[ 3458], | 30.00th=[ 3785], 40.00th=[ 4228], 50.00th=[ 5538], 60.00th=[ 6718], | 70.00th=[ 8586], 80.00th=[10552], 90.00th=[12911], 95.00th=[16581], | 99.00th=[25560], 99.50th=[29230], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 736, max= 2624, per=25.35%, avg=1400.00, stdev=686.91, samples=8 iops : min= 46, max= 164, avg=87.50, stdev=42.93, samples=8 lat (msec) : 2=2.60%, 4=45.31%, 10=36.98%, 20=13.54%, 50=1.56% cpu : usr=0.18%, sys=11.79%, ctx=956, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30268: Tue Apr 8 14:35:07 2025 read: IOPS=79, BW=1280KiB/s (1311kB/s)(5744KiB/4488msec) clat (usec): min=1617, max=24394, avg=4531.86, stdev=3499.57 lat (usec): min=1617, max=24395, avg=4532.58, stdev=3499.65 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1909], 10.00th=[ 2114], 20.00th=[ 2311], | 30.00th=[ 2540], 40.00th=[ 2868], 50.00th=[ 3294], 60.00th=[ 3785], | 70.00th=[ 4359], 80.00th=[ 6128], 90.00th=[ 8717], 95.00th=[12518], | 99.00th=[20317], 99.50th=[23462], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 702, max= 2464, per=24.53%, avg=1287.50, stdev=606.56, samples=8 iops : min= 43, max= 154, avg=80.25, stdev=38.10, samples=8 write: IOPS=91, BW=1458KiB/s (1493kB/s)(6544KiB/4488msec) clat (usec): min=2424, max=22473, avg=6929.70, stdev=3990.15 lat (usec): min=2425, max=22474, avg=6931.65, stdev=3991.42 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2868], 10.00th=[ 3195], 20.00th=[ 3458], | 30.00th=[ 3818], 40.00th=[ 4686], 50.00th=[ 5866], 60.00th=[ 6915], | 70.00th=[ 8094], 80.00th=[10290], 90.00th=[12911], 95.00th=[15533], | 99.00th=[18220], 99.50th=[19530], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 894, max= 2848, per=26.28%, avg=1451.25, stdev=663.00, samples=8 iops : min= 55, max= 178, avg=90.50, stdev=41.52, samples=8 lat (msec) : 2=3.65%, 4=44.27%, 10=37.50%, 20=13.80%, 50=0.78% cpu : usr=0.18%, sys=11.68%, ctx=951, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30270: Tue Apr 8 14:35:07 2025 read: IOPS=85, BW=1374KiB/s (1407kB/s)(6256KiB/4554msec) clat (usec): min=1696, max=19346, avg=4670.80, stdev=3384.80 lat (usec): min=1697, max=19347, avg=4671.62, stdev=3384.71 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 1991], 10.00th=[ 2089], 20.00th=[ 2278], | 30.00th=[ 2409], 40.00th=[ 2769], 50.00th=[ 3359], 60.00th=[ 3916], | 70.00th=[ 4817], 80.00th=[ 6980], 90.00th=[ 9110], 95.00th=[12125], | 99.00th=[17695], 99.50th=[19268], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 640, max= 2688, per=26.41%, avg=1386.44, stdev=656.39, samples=9 iops : min= 40, max= 168, avg=86.56, stdev=41.11, samples=9 write: IOPS=82, BW=1325KiB/s (1356kB/s)(6032KiB/4554msec) clat (usec): min=2361, max=21779, avg=7069.92, stdev=4254.60 lat (usec): min=2362, max=21780, avg=7071.28, stdev=4254.71 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2900], 10.00th=[ 3097], 20.00th=[ 3490], | 30.00th=[ 3916], 40.00th=[ 4490], 50.00th=[ 5735], 60.00th=[ 6915], | 70.00th=[ 8586], 80.00th=[10421], 90.00th=[13304], 95.00th=[16188], | 99.00th=[20055], 99.50th=[21103], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 800, max= 2752, per=23.82%, avg=1315.33, stdev=626.28, samples=9 iops : min= 50, max= 172, avg=82.11, stdev=39.21, samples=9 lat (msec) : 2=2.99%, 4=43.36%, 10=38.93%, 20=14.32%, 50=0.39% cpu : usr=0.24%, sys=11.38%, ctx=966, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30272: Tue Apr 8 14:35:07 2025 read: IOPS=83, BW=1331KiB/s (1363kB/s)(6032KiB/4533msec) clat (usec): min=1707, max=23675, avg=4658.72, stdev=3475.97 lat (usec): min=1707, max=23675, avg=4659.42, stdev=3476.05 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 1991], 10.00th=[ 2114], 20.00th=[ 2245], | 30.00th=[ 2442], 40.00th=[ 2737], 50.00th=[ 3228], 60.00th=[ 3785], | 70.00th=[ 4752], 80.00th=[ 6783], 90.00th=[ 9634], 95.00th=[12387], | 99.00th=[16319], 99.50th=[23725], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 672, max= 2528, per=25.27%, avg=1326.22, stdev=572.68, samples=9 iops : min= 42, max= 158, avg=82.89, stdev=35.79, samples=9 write: IOPS=86, BW=1380KiB/s (1413kB/s)(6256KiB/4533msec) clat (usec): min=2164, max=30082, avg=6927.00, stdev=4680.87 lat (usec): min=2165, max=30083, avg=6928.22, stdev=4680.97 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2868], 10.00th=[ 3163], 20.00th=[ 3425], | 30.00th=[ 3785], 40.00th=[ 4228], 50.00th=[ 5145], 60.00th=[ 6718], | 70.00th=[ 7898], 80.00th=[ 9765], 90.00th=[12911], 95.00th=[15926], | 99.00th=[27395], 99.50th=[29754], 99.90th=[30016], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 768, max= 2848, per=25.05%, avg=1383.11, stdev=657.13, samples=9 iops : min= 48, max= 178, avg=86.44, stdev=41.07, samples=9 lat (msec) : 2=2.73%, 4=46.09%, 10=37.11%, 20=12.76%, 50=1.30% cpu : usr=0.13%, sys=11.89%, ctx=968, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5249KiB/s (5375kB/s), 1280KiB/s-1374KiB/s (1311kB/s-1407kB/s), io=23.4MiB (24.5MB), run=4488-4563msec WRITE: bw=5523KiB/s (5655kB/s), 1325KiB/s-1458KiB/s (1356kB/s-1493kB/s), io=24.6MiB (25.8MB), run=4488-4563msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30283: Tue Apr 8 14:35:09 2025 read: IOPS=9, BW=9.83MiB/s (10.3MB/s)(7168KiB/712msec) clat (usec): min=26222, max=55971, avg=39507.76, stdev=9198.83 lat (usec): min=26222, max=55972, avg=39509.02, stdev=9198.91 clat percentiles (usec): | 1.00th=[26346], 5.00th=[26346], 10.00th=[26346], 20.00th=[33817], | 30.00th=[35914], 40.00th=[35914], 50.00th=[39584], 60.00th=[42206], | 70.00th=[42206], 80.00th=[42730], 90.00th=[55837], 95.00th=[55837], | 99.00th=[55837], 99.50th=[55837], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min=10240, max=10240, per=41.05%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7191KiB/s (7364kB/s)(5120KiB/712msec) clat (msec): min=34, max=153, avg=69.91, stdev=47.91 lat (msec): min=34, max=153, avg=70.97, stdev=47.67 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 59], 60.00th=[ 59], | 70.00th=[ 61], 80.00th=[ 61], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 6144, max= 6144, per=17.59%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 50=66.67%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=34.32%, ctx=609, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30286: Tue Apr 8 14:35:09 2025 read: IOPS=2, BW=2495KiB/s (2554kB/s)(2048KiB/821msec) clat (usec): min=47056, max=73118, avg=60087.96, stdev=18428.63 lat (usec): min=47058, max=73120, avg=60089.23, stdev=18428.41 clat percentiles (usec): | 1.00th=[46924], 5.00th=[46924], 10.00th=[46924], 20.00th=[46924], | 30.00th=[46924], 40.00th=[46924], 50.00th=[46924], 60.00th=[72877], | 70.00th=[72877], 80.00th=[72877], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 2048, max= 2048, per=8.21%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.2MiB/s (12.8MB/s)(10.0MiB/821msec) clat (msec): min=23, max=140, avg=62.75, stdev=39.44 lat (msec): min=23, max=140, avg=62.82, stdev=39.44 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 36], | 30.00th=[ 37], 40.00th=[ 42], 50.00th=[ 47], 60.00th=[ 47], | 70.00th=[ 83], 80.00th=[ 83], 90.00th=[ 123], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 6144, max= 6144, per=17.59%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=25.00%, 250=16.67% cpu : usr=0.00%, sys=31.95%, ctx=253, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30288: Tue Apr 8 14:35:09 2025 read: IOPS=8, BW=8192KiB/s (8389kB/s)(6144KiB/750msec) clat (usec): min=900, max=68046, avg=31459.61, stdev=27020.17 lat (usec): min=901, max=68047, avg=31460.65, stdev=27019.98 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 898], 10.00th=[ 898], 20.00th=[ 2212], | 30.00th=[ 2212], 40.00th=[30016], 50.00th=[30016], 60.00th=[33424], | 70.00th=[54264], 80.00th=[54264], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 4079, max= 4079, per=16.35%, avg=4079.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8192KiB/s (8389kB/s)(6144KiB/750msec) clat (msec): min=58, max=184, avg=90.36, stdev=47.25 lat (msec): min=58, max=184, avg=90.42, stdev=47.27 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 58], 10.00th=[ 58], 20.00th=[ 67], | 30.00th=[ 67], 40.00th=[ 70], 50.00th=[ 70], 60.00th=[ 72], | 70.00th=[ 91], 80.00th=[ 91], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 6119, max= 6119, per=17.52%, avg=6119.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (usec) : 1000=8.33% lat (msec) : 4=8.33%, 50=16.67%, 100=58.33%, 250=8.33% cpu : usr=0.00%, sys=28.70%, ctx=575, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30290: Tue Apr 8 14:35:09 2025 read: IOPS=6, BW=6854KiB/s (7019kB/s)(5120KiB/747msec) clat (usec): min=389, max=48056, avg=9982.97, stdev=21283.96 lat (usec): min=389, max=48058, avg=9983.85, stdev=21284.29 clat percentiles (usec): | 1.00th=[ 392], 5.00th=[ 392], 10.00th=[ 392], 20.00th=[ 392], | 30.00th=[ 392], 40.00th=[ 392], 50.00th=[ 537], 60.00th=[ 537], | 70.00th=[ 545], 80.00th=[ 545], 90.00th=[47973], 95.00th=[47973], | 99.00th=[47973], 99.50th=[47973], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] write: IOPS=9, BW=9596KiB/s (9826kB/s)(7168KiB/747msec) clat (msec): min=21, max=154, avg=90.94, stdev=44.09 lat (msec): min=21, max=155, avg=91.01, stdev=44.11 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 72], | 30.00th=[ 72], 40.00th=[ 72], 50.00th=[ 81], 60.00th=[ 104], | 70.00th=[ 104], 80.00th=[ 133], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 6144, max= 6144, per=17.59%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67%, 750=16.67% lat (msec) : 50=16.67%, 100=25.00%, 250=25.00% cpu : usr=0.00%, sys=23.86%, ctx=187, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=24.4MiB/s (25.5MB/s), 2495KiB/s-9.83MiB/s (2554kB/s-10.3MB/s), io=20.0MiB (20.0MB), run=712-821msec WRITE: bw=34.1MiB/s (35.8MB/s), 7191KiB/s-12.2MiB/s (7364kB/s-12.8MB/s), io=28.0MiB (29.4MB), run=712-821msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30284: Tue Apr 8 14:35:09 2025 read: IOPS=12, BW=12.7MiB/s (13.3MB/s)(7168KiB/550msec) clat (usec): min=17346, max=48930, avg=25892.95, stdev=10712.87 lat (usec): min=17347, max=48931, avg=25893.93, stdev=10712.87 clat percentiles (usec): | 1.00th=[17433], 5.00th=[17433], 10.00th=[17433], 20.00th=[20317], | 30.00th=[20317], 40.00th=[20317], 50.00th=[21627], 60.00th=[24773], | 70.00th=[24773], 80.00th=[27919], 90.00th=[49021], 95.00th=[49021], | 99.00th=[49021], 99.50th=[49021], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min=12047, max=12047, per=51.59%, avg=12047.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9309KiB/s (9533kB/s)(5120KiB/550msec) clat (msec): min=35, max=185, avg=71.52, stdev=64.12 lat (msec): min=35, max=185, avg=71.59, stdev=64.12 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 42], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 46], | 70.00th=[ 50], 80.00th=[ 50], 90.00th=[ 186], 95.00th=[ 186], | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 8031, max= 8031, per=24.56%, avg=8031.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=83.33%, 250=8.33% cpu : usr=0.00%, sys=17.49%, ctx=52, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30285: Tue Apr 8 14:35:09 2025 read: IOPS=2, BW=2764KiB/s (2830kB/s)(2048KiB/741msec) clat (usec): min=27287, max=54729, avg=41008.43, stdev=19404.76 lat (usec): min=27289, max=54731, avg=41010.42, stdev=19404.17 clat percentiles (usec): | 1.00th=[27395], 5.00th=[27395], 10.00th=[27395], 20.00th=[27395], | 30.00th=[27395], 40.00th=[27395], 50.00th=[27395], 60.00th=[54789], | 70.00th=[54789], 80.00th=[54789], 90.00th=[54789], 95.00th=[54789], | 99.00th=[54789], 99.50th=[54789], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 4031, max= 4031, per=17.26%, avg=4031.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.5MiB/s (14.1MB/s)(10.0MiB/741msec) clat (msec): min=20, max=158, avg=64.10, stdev=43.12 lat (msec): min=20, max=159, avg=64.18, stdev=43.12 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 27], | 30.00th=[ 30], 40.00th=[ 34], 50.00th=[ 52], 60.00th=[ 63], | 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 106], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min=12094, max=12094, per=36.99%, avg=12094.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=41.67%, 250=16.67% cpu : usr=0.00%, sys=17.57%, ctx=59, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30287: Tue Apr 8 14:35:09 2025 read: IOPS=7, BW=7237KiB/s (7410kB/s)(6144KiB/849msec) clat (msec): min=32, max=114, avg=70.46, stdev=34.72 lat (msec): min=32, max=114, avg=70.47, stdev=34.72 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 65], 50.00th=[ 65], 60.00th=[ 66], | 70.00th=[ 109], 80.00th=[ 109], 90.00th=[ 115], 95.00th=[ 115], | 99.00th=[ 115], 99.50th=[ 115], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 6144, max= 6144, per=26.31%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7237KiB/s (7410kB/s)(6144KiB/849msec) clat (msec): min=37, max=118, avg=69.90, stdev=33.94 lat (msec): min=37, max=118, avg=70.00, stdev=33.94 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 45], 50.00th=[ 45], 60.00th=[ 87], | 70.00th=[ 93], 80.00th=[ 93], 90.00th=[ 118], 95.00th=[ 118], | 99.00th=[ 118], 99.50th=[ 118], 99.90th=[ 118], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 8192, max= 8192, per=25.06%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=33.33%, 250=25.00% cpu : usr=0.00%, sys=17.81%, ctx=87, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30289: Tue Apr 8 14:35:09 2025 read: IOPS=5, BW=5838KiB/s (5978kB/s)(5120KiB/877msec) clat (usec): min=13768, max=95247, avg=56914.91, stdev=36715.25 lat (usec): min=13769, max=95248, avg=56916.13, stdev=36715.13 clat percentiles (usec): | 1.00th=[13829], 5.00th=[13829], 10.00th=[13829], 20.00th=[13829], | 30.00th=[24249], 40.00th=[24249], 50.00th=[63701], 60.00th=[63701], | 70.00th=[87557], 80.00th=[87557], 90.00th=[94897], 95.00th=[94897], | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 2039, max= 2039, per=8.73%, avg=2039.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=7, BW=8173KiB/s (8369kB/s)(7168KiB/877msec) clat (msec): min=16, max=137, avg=77.19, stdev=47.90 lat (msec): min=16, max=137, avg=77.25, stdev=47.90 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 20], | 30.00th=[ 78], 40.00th=[ 78], 50.00th=[ 80], 60.00th=[ 80], | 70.00th=[ 80], 80.00th=[ 133], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 8159, max= 8159, per=24.96%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=8.33%, 100=50.00%, 250=16.67% cpu : usr=0.23%, sys=15.18%, ctx=57, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.8MiB/s (23.9MB/s), 2764KiB/s-12.7MiB/s (2830kB/s-13.3MB/s), io=20.0MiB (20.0MB), run=550-877msec WRITE: bw=31.9MiB/s (33.5MB/s), 7237KiB/s-13.5MiB/s (7410kB/s-14.1MB/s), io=28.0MiB (29.4MB), run=550-877msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=30296: Tue Apr 8 14:35:11 2025 read: IOPS=4, BW=18.7MiB/s (19.6MB/s)(12.0MiB/641msec) clat (msec): min=83, max=327, avg=200.49, stdev=122.11 lat (msec): min=83, max=327, avg=200.49, stdev=122.11 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 85], 10.00th=[ 85], 20.00th=[ 85], | 30.00th=[ 85], 40.00th=[ 190], 50.00th=[ 190], 60.00th=[ 190], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 8192, max= 8192, per=41.10%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.16%, sys=23.75%, ctx=1641, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30297: Tue Apr 8 14:35:11 2025 read: IOPS=1, BW=4983KiB/s (5103kB/s)(4096KiB/822msec) clat (nsec): min=368177k, max=368177k, avg=368177341.00, stdev= 0.00 lat (nsec): min=368180k, max=368180k, avg=368179766.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 368], 10.00th=[ 368], 20.00th=[ 368], | 30.00th=[ 368], 40.00th=[ 368], 50.00th=[ 368], 60.00th=[ 368], | 70.00th=[ 368], 80.00th=[ 368], 90.00th=[ 368], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 8192, max= 8192, per=41.10%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9966KiB/s (10.2MB/s)(8192KiB/822msec) clat (msec): min=214, max=226, avg=220.06, stdev= 8.45 lat (msec): min=214, max=226, avg=220.32, stdev= 8.43 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 215], 10.00th=[ 215], 20.00th=[ 215], | 30.00th=[ 215], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 226], | 70.00th=[ 226], 80.00th=[ 226], 90.00th=[ 226], 95.00th=[ 226], | 99.00th=[ 226], 99.50th=[ 226], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 226] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=35.69%, ctx=839, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30298: Tue Apr 8 14:35:11 2025 write: IOPS=3, BW=12.6MiB/s (13.2MB/s)(12.0MiB/956msec) clat (msec): min=157, max=549, avg=305.08, stdev=213.20 lat (msec): min=157, max=554, avg=306.86, stdev=215.82 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 209], 50.00th=[ 209], 60.00th=[ 209], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=27.75%, ctx=75, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30299: Tue Apr 8 14:35:11 2025 write: IOPS=3, BW=14.3MiB/s (14.0MB/s)(12.0MiB/841msec) clat (msec): min=116, max=572, avg=275.24, stdev=257.47 lat (msec): min=117, max=572, avg=275.52, stdev=257.39 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 117], 10.00th=[ 117], 20.00th=[ 117], | 30.00th=[ 117], 40.00th=[ 138], 50.00th=[ 138], 60.00th=[ 138], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min=16384, max=16384, per=47.80%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.12%, sys=35.71%, ctx=66, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.5MiB/s (20.4MB/s), 4983KiB/s-18.7MiB/s (5103kB/s-19.6MB/s), io=16.0MiB (16.8MB), run=641-822msec WRITE: bw=33.5MiB/s (35.1MB/s), 9966KiB/s-14.3MiB/s (10.2MB/s-14.0MB/s), io=32.0MiB (33.6MB), run=822-956msec rand-rw: (groupid=0, jobs=1): err= 0: pid=30300: Tue Apr 8 14:35:11 2025 read: IOPS=3, BW=12.6MiB/s (13.2MB/s)(12.0MiB/953msec) clat (msec): min=83, max=608, avg=307.81, stdev=270.86 lat (msec): min=83, max=608, avg=307.81, stdev=270.86 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 84], 10.00th=[ 84], 20.00th=[ 84], | 30.00th=[ 84], 40.00th=[ 232], 50.00th=[ 232], 60.00th=[ 232], | 70.00th=[ 609], 80.00th=[ 609], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] lat (msec) : 100=33.33%, 250=33.33%, 750=33.33% cpu : usr=0.11%, sys=8.40%, ctx=48, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30301: Tue Apr 8 14:35:11 2025 read: IOPS=1, BW=4276KiB/s (4378kB/s)(4096KiB/958msec) clat (nsec): min=641092k, max=641092k, avg=641092415.00, stdev= 0.00 lat (nsec): min=641094k, max=641094k, avg=641093552.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 642], 5.00th=[ 642], 10.00th=[ 642], 20.00th=[ 642], | 30.00th=[ 642], 40.00th=[ 642], 50.00th=[ 642], 60.00th=[ 642], | 70.00th=[ 642], 80.00th=[ 642], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] write: IOPS=2, BW=8551KiB/s (8756kB/s)(8192KiB/958msec) clat (msec): min=134, max=140, avg=137.48, stdev= 3.74 lat (msec): min=138, max=140, avg=139.50, stdev= 1.27 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 140], | 70.00th=[ 140], 80.00th=[ 140], 90.00th=[ 140], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=6.58%, ctx=52, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30302: Tue Apr 8 14:35:11 2025 write: IOPS=4, BW=16.3MiB/s (17.1MB/s)(12.0MiB/734msec) clat (msec): min=112, max=395, avg=238.95, stdev=143.58 lat (msec): min=112, max=395, avg=239.38, stdev=143.81 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 113], 10.00th=[ 113], 20.00th=[ 113], | 30.00th=[ 113], 40.00th=[ 209], 50.00th=[ 209], 60.00th=[ 209], | 70.00th=[ 397], 80.00th=[ 397], 90.00th=[ 397], 95.00th=[ 397], | 99.00th=[ 397], 99.50th=[ 397], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 8192, max= 8192, per=23.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.27%, sys=20.87%, ctx=56, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=30303: Tue Apr 8 14:35:11 2025 write: IOPS=3, BW=13.7MiB/s (14.4MB/s)(12.0MiB/874msec) clat (msec): min=112, max=522, avg=281.18, stdev=214.62 lat (msec): min=112, max=522, avg=281.50, stdev=214.56 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 209], 50.00th=[ 209], 60.00th=[ 209], | 70.00th=[ 523], 80.00th=[ 523], 90.00th=[ 523], 95.00th=[ 523], | 99.00th=[ 523], 99.50th=[ 523], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min=16351, max=16351, per=47.80%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.23%, sys=17.87%, ctx=112, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.7MiB/s (17.5MB/s), 4276KiB/s-12.6MiB/s (4378kB/s-13.2MB/s), io=16.0MiB (16.8MB), run=953-958msec WRITE: bw=33.4MiB/s (35.0MB/s), 8551KiB/s-16.3MiB/s (8756kB/s-17.1MB/s), io=32.0MiB (33.6MB), run=734-958msec