== sanity test 398b: DIO and buffer IO race ============== 08:13:05 (1744891985) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.13401 s, 44.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28681: Thu Apr 17 08:13:19 2025 read: IOPS=126, BW=506KiB/s (518kB/s)(5992KiB/11840msec) clat (usec): min=64, max=166699, avg=4403.63, stdev=4799.81 lat (usec): min=64, max=166700, avg=4404.32, stdev=4799.84 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 1713], 10.00th=[ 2073], 20.00th=[ 2507], | 30.00th=[ 2868], 40.00th=[ 3326], 50.00th=[ 3916], 60.00th=[ 4555], | 70.00th=[ 5211], 80.00th=[ 5932], 90.00th=[ 6980], 95.00th=[ 7963], | 99.00th=[ 12780], 99.50th=[ 14484], 99.90th=[ 19792], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 344, max= 646, per=24.78%, avg=499.35, stdev=62.63, samples=23 iops : min= 86, max= 161, avg=124.78, stdev=15.61, samples=23 write: IOPS=132, BW=532KiB/s (545kB/s)(6296KiB/11840msec) clat (usec): min=421, max=19035, avg=3292.30, stdev=2857.25 lat (usec): min=422, max=19035, avg=3292.98, stdev=2857.25 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 553], 10.00th=[ 570], 20.00th=[ 635], | 30.00th=[ 881], 40.00th=[ 2147], 50.00th=[ 2737], 60.00th=[ 3261], | 70.00th=[ 4047], 80.00th=[ 5473], 90.00th=[ 7308], 95.00th=[ 8848], | 99.00th=[12649], 99.50th=[14484], 99.90th=[19006], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 408, max= 680, per=24.75%, avg=522.26, stdev=65.28, samples=23 iops : min= 102, max= 170, avg=130.52, stdev=16.29, samples=23 lat (usec) : 100=1.63%, 250=0.49%, 500=0.23%, 750=13.15%, 1000=2.83% lat (msec) : 2=5.44%, 4=36.75%, 10=36.56%, 20=2.90%, 250=0.03% cpu : usr=0.18%, sys=22.79%, ctx=5733, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28682: Thu Apr 17 08:13:19 2025 read: IOPS=125, BW=502KiB/s (514kB/s)(5940KiB/11840msec) clat (usec): min=59, max=177666, avg=4377.23, stdev=5044.75 lat (usec): min=59, max=177668, avg=4377.84, stdev=5044.77 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 1844], 10.00th=[ 2147], 20.00th=[ 2474], | 30.00th=[ 2835], 40.00th=[ 3294], 50.00th=[ 3884], 60.00th=[ 4424], | 70.00th=[ 5080], 80.00th=[ 5866], 90.00th=[ 7046], 95.00th=[ 8455], | 99.00th=[ 11994], 99.50th=[ 13698], 99.90th=[ 18220], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 240, max= 680, per=24.58%, avg=495.30, stdev=82.64, samples=23 iops : min= 60, max= 170, avg=123.83, stdev=20.66, samples=23 write: IOPS=134, BW=536KiB/s (549kB/s)(6348KiB/11840msec) clat (usec): min=482, max=17183, avg=3322.59, stdev=2831.80 lat (usec): min=483, max=17183, avg=3323.48, stdev=2831.89 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 545], 10.00th=[ 570], 20.00th=[ 627], | 30.00th=[ 873], 40.00th=[ 1991], 50.00th=[ 2737], 60.00th=[ 3392], | 70.00th=[ 4359], 80.00th=[ 5604], 90.00th=[ 7570], 95.00th=[ 8586], | 99.00th=[11994], 99.50th=[13829], 99.90th=[16319], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 288, max= 632, per=24.71%, avg=521.39, stdev=73.51, samples=23 iops : min= 72, max= 158, avg=130.35, stdev=18.38, samples=23 lat (usec) : 100=1.60%, 250=0.29%, 500=0.13%, 750=13.93%, 1000=2.38% lat (msec) : 2=5.76%, 4=35.90%, 10=37.47%, 20=2.51%, 250=0.03% cpu : usr=0.17%, sys=22.59%, ctx=5826, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28683: Thu Apr 17 08:13:19 2025 read: IOPS=126, BW=506KiB/s (518kB/s)(6028KiB/11913msec) clat (usec): min=54, max=23031, avg=4310.93, stdev=2375.98 lat (usec): min=54, max=23032, avg=4311.80, stdev=2376.07 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 1795], 10.00th=[ 2089], 20.00th=[ 2474], | 30.00th=[ 2868], 40.00th=[ 3294], 50.00th=[ 3851], 60.00th=[ 4555], | 70.00th=[ 5211], 80.00th=[ 6063], 90.00th=[ 7046], 95.00th=[ 8586], | 99.00th=[12125], 99.50th=[14353], 99.90th=[16319], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 264, max= 616, per=24.73%, avg=498.26, stdev=73.23, samples=23 iops : min= 66, max= 154, avg=124.52, stdev=18.27, samples=23 write: IOPS=131, BW=525KiB/s (538kB/s)(6260KiB/11913msec) clat (usec): min=454, max=170172, avg=3412.30, stdev=5083.48 lat (usec): min=455, max=170174, avg=3413.08, stdev=5083.52 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 553], 10.00th=[ 570], 20.00th=[ 635], | 30.00th=[ 873], 40.00th=[ 2057], 50.00th=[ 2704], 60.00th=[ 3326], | 70.00th=[ 4228], 80.00th=[ 5538], 90.00th=[ 7439], 95.00th=[ 8717], | 99.00th=[ 12780], 99.50th=[ 14353], 99.90th=[ 16712], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 320, max= 696, per=24.44%, avg=515.70, stdev=85.85, samples=23 iops : min= 80, max= 174, avg=128.87, stdev=21.50, samples=23 lat (usec) : 100=1.89%, 250=0.16%, 500=0.23%, 750=13.90%, 1000=2.47% lat (msec) : 2=5.18%, 4=36.33%, 10=37.24%, 20=2.54%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=22.55%, ctx=5625, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28684: Thu Apr 17 08:13:19 2025 read: IOPS=129, BW=516KiB/s (529kB/s)(6052KiB/11719msec) clat (usec): min=65, max=24028, avg=4229.82, stdev=2333.13 lat (usec): min=66, max=24029, avg=4230.42, stdev=2333.13 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 1745], 10.00th=[ 2057], 20.00th=[ 2442], | 30.00th=[ 2802], 40.00th=[ 3294], 50.00th=[ 3884], 60.00th=[ 4424], | 70.00th=[ 5080], 80.00th=[ 5800], 90.00th=[ 6980], 95.00th=[ 8225], | 99.00th=[11076], 99.50th=[12780], 99.90th=[21103], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 304, max= 632, per=25.41%, avg=512.00, stdev=76.60, samples=23 iops : min= 76, max= 158, avg=127.91, stdev=19.12, samples=23 write: IOPS=133, BW=532KiB/s (545kB/s)(6236KiB/11719msec) clat (usec): min=442, max=170400, avg=3361.90, stdev=5078.78 lat (usec): min=443, max=170401, avg=3364.40, stdev=5079.94 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 545], 10.00th=[ 570], 20.00th=[ 619], | 30.00th=[ 791], 40.00th=[ 1909], 50.00th=[ 2704], 60.00th=[ 3326], | 70.00th=[ 4228], 80.00th=[ 5604], 90.00th=[ 7242], 95.00th=[ 8586], | 99.00th=[ 12387], 99.50th=[ 13304], 99.90th=[ 17171], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 384, max= 728, per=24.99%, avg=527.35, stdev=82.46, samples=23 iops : min= 96, max= 182, avg=131.74, stdev=20.59, samples=23 lat (usec) : 100=1.46%, 250=0.39%, 500=0.23%, 750=14.55%, 1000=2.38% lat (msec) : 2=6.05%, 4=35.35%, 10=37.21%, 20=2.28%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=22.55%, ctx=5803, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2016KiB/s (2064kB/s), 502KiB/s-516KiB/s (514kB/s-529kB/s), io=23.4MiB (24.6MB), run=11719-11913msec WRITE: bw=2110KiB/s (2161kB/s), 525KiB/s-536KiB/s (538kB/s-549kB/s), io=24.6MiB (25.7MB), run=11719-11913msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28677: Thu Apr 17 08:13:23 2025 read: IOPS=97, BW=390KiB/s (400kB/s)(5992KiB/15353msec) clat (usec): min=1471, max=171109, avg=4247.90, stdev=5324.49 lat (usec): min=1472, max=171111, avg=4248.50, stdev=5324.52 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1729], 10.00th=[ 1827], 20.00th=[ 1991], | 30.00th=[ 2212], 40.00th=[ 2507], 50.00th=[ 2999], 60.00th=[ 3490], | 70.00th=[ 4146], 80.00th=[ 5735], 90.00th=[ 8586], 95.00th=[ 10552], | 99.00th=[ 16909], 99.50th=[ 17957], 99.90th=[ 23462], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 200, max= 864, per=24.57%, avg=384.10, stdev=166.93, samples=30 iops : min= 50, max= 216, avg=95.93, stdev=41.74, samples=30 write: IOPS=102, BW=410KiB/s (420kB/s)(6296KiB/15353msec) clat (usec): min=2054, max=27974, avg=5672.08, stdev=3226.90 lat (usec): min=2055, max=27975, avg=5672.86, stdev=3226.90 clat percentiles (usec): | 1.00th=[ 2343], 5.00th=[ 2638], 10.00th=[ 2835], 20.00th=[ 3130], | 30.00th=[ 3458], 40.00th=[ 3982], 50.00th=[ 4555], 60.00th=[ 5407], | 70.00th=[ 6652], 80.00th=[ 7963], 90.00th=[10028], 95.00th=[11600], | 99.00th=[16450], 99.50th=[20055], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 224, max= 776, per=24.72%, avg=404.63, stdev=159.79, samples=30 iops : min= 56, max= 194, avg=101.07, stdev=39.96, samples=30 lat (msec) : 2=10.06%, 4=43.82%, 10=37.89%, 20=7.81%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=11.00%, ctx=3611, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28678: Thu Apr 17 08:13:23 2025 read: IOPS=97, BW=389KiB/s (398kB/s)(5940KiB/15289msec) clat (usec): min=1485, max=179124, avg=4242.53, stdev=5472.57 lat (usec): min=1486, max=179127, avg=4243.17, stdev=5472.62 clat percentiles (usec): | 1.00th=[ 1598], 5.00th=[ 1745], 10.00th=[ 1844], 20.00th=[ 2008], | 30.00th=[ 2245], 40.00th=[ 2573], 50.00th=[ 2999], 60.00th=[ 3458], | 70.00th=[ 4113], 80.00th=[ 5604], 90.00th=[ 8586], 95.00th=[ 10814], | 99.00th=[ 15926], 99.50th=[ 17171], 99.90th=[ 22938], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 184, max= 792, per=24.77%, avg=387.17, stdev=169.03, samples=30 iops : min= 46, max= 198, avg=96.77, stdev=42.26, samples=30 write: IOPS=103, BW=415KiB/s (425kB/s)(6348KiB/15289msec) clat (usec): min=2047, max=27478, avg=5624.65, stdev=3188.37 lat (usec): min=2048, max=27479, avg=5625.51, stdev=3188.29 clat percentiles (usec): | 1.00th=[ 2343], 5.00th=[ 2671], 10.00th=[ 2868], 20.00th=[ 3130], | 30.00th=[ 3392], 40.00th=[ 3818], 50.00th=[ 4424], 60.00th=[ 5276], | 70.00th=[ 6521], 80.00th=[ 7963], 90.00th=[10290], 95.00th=[11600], | 99.00th=[16319], 99.50th=[18482], 99.90th=[23987], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 240, max= 784, per=25.00%, avg=409.30, stdev=157.04, samples=30 iops : min= 60, max= 196, avg=102.30, stdev=39.27, samples=30 lat (msec) : 2=9.11%, 4=46.48%, 10=35.22%, 20=8.95%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=11.28%, ctx=3663, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28679: Thu Apr 17 08:13:23 2025 read: IOPS=99, BW=397KiB/s (407kB/s)(6028KiB/15181msec) clat (usec): min=1522, max=23577, avg=4152.59, stdev=3155.65 lat (usec): min=1522, max=23578, avg=4153.19, stdev=3155.66 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 1762], 10.00th=[ 1860], 20.00th=[ 2008], | 30.00th=[ 2245], 40.00th=[ 2573], 50.00th=[ 2999], 60.00th=[ 3490], | 70.00th=[ 4178], 80.00th=[ 5538], 90.00th=[ 8455], 95.00th=[10945], | 99.00th=[15795], 99.50th=[18482], 99.90th=[21627], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 168, max= 856, per=25.27%, avg=395.03, stdev=157.56, samples=30 iops : min= 42, max= 214, avg=98.67, stdev=39.42, samples=30 write: IOPS=103, BW=412KiB/s (422kB/s)(6260KiB/15181msec) clat (msec): min=2, max=180, avg= 5.66, stdev= 5.44 lat (msec): min=2, max=180, avg= 5.66, stdev= 5.44 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 11], 95.00th=[ 12], | 99.00th=[ 18], 99.50th=[ 20], 99.90th=[ 24], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 216, max= 872, per=25.01%, avg=409.43, stdev=163.34, samples=30 iops : min= 54, max= 218, avg=102.27, stdev=40.85, samples=30 lat (msec) : 2=9.51%, 4=45.80%, 10=36.17%, 20=8.17%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=11.48%, ctx=3625, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28680: Thu Apr 17 08:13:23 2025 read: IOPS=98, BW=395KiB/s (404kB/s)(6052KiB/15323msec) clat (usec): min=1495, max=22804, avg=4155.62, stdev=3086.69 lat (usec): min=1495, max=22805, avg=4156.26, stdev=3086.68 clat percentiles (usec): | 1.00th=[ 1582], 5.00th=[ 1713], 10.00th=[ 1811], 20.00th=[ 1991], | 30.00th=[ 2245], 40.00th=[ 2606], 50.00th=[ 3032], 60.00th=[ 3523], | 70.00th=[ 4178], 80.00th=[ 5538], 90.00th=[ 8586], 95.00th=[11207], | 99.00th=[15139], 99.50th=[16909], 99.90th=[19530], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 200, max= 864, per=24.92%, avg=389.43, stdev=166.52, samples=30 iops : min= 50, max= 216, avg=97.27, stdev=41.66, samples=30 write: IOPS=101, BW=407KiB/s (417kB/s)(6236KiB/15323msec) clat (msec): min=2, max=179, avg= 5.76, stdev= 5.54 lat (msec): min=2, max=179, avg= 5.76, stdev= 5.54 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 12], | 99.00th=[ 19], 99.50th=[ 21], 99.90th=[ 28], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 216, max= 808, per=24.51%, avg=401.17, stdev=165.12, samples=30 iops : min= 54, max= 202, avg=100.20, stdev=41.29, samples=30 lat (msec) : 2=10.09%, 4=45.05%, 10=35.61%, 20=8.95%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.11%, sys=11.22%, ctx=3645, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1564KiB/s (1602kB/s), 389KiB/s-397KiB/s (398kB/s-407kB/s), io=23.4MiB (24.6MB), run=15181-15353msec WRITE: bw=1637KiB/s (1677kB/s), 407KiB/s-415KiB/s (417kB/s-425kB/s), io=24.6MiB (25.7MB), run=15181-15353msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28712: Thu Apr 17 08:13:27 2025 read: IOPS=117, BW=1886KiB/s (1931kB/s)(5920KiB/3139msec) clat (usec): min=67, max=19571, avg=4386.73, stdev=2644.88 lat (usec): min=67, max=19572, avg=4387.30, stdev=2644.88 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 81], 10.00th=[ 198], 20.00th=[ 2606], | 30.00th=[ 3064], 40.00th=[ 3490], 50.00th=[ 4047], 60.00th=[ 4752], | 70.00th=[ 5473], 80.00th=[ 6128], 90.00th=[ 7242], 95.00th=[ 8979], | 99.00th=[14091], 99.50th=[15139], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 1568, max= 2304, per=24.64%, avg=1850.67, stdev=295.66, samples=6 iops : min= 98, max= 144, avg=115.67, stdev=18.48, samples=6 write: IOPS=126, BW=2029KiB/s (2077kB/s)(6368KiB/3139msec) clat (usec): min=669, max=18870, avg=3722.27, stdev=3180.46 lat (usec): min=670, max=18870, avg=3723.99, stdev=3179.71 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 775], 10.00th=[ 807], 20.00th=[ 873], | 30.00th=[ 1012], 40.00th=[ 1680], 50.00th=[ 3032], 60.00th=[ 3720], | 70.00th=[ 4883], 80.00th=[ 6259], 90.00th=[ 8356], 95.00th=[ 9765], | 99.00th=[13829], 99.50th=[15008], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 1504, max= 2336, per=25.18%, avg=1989.33, stdev=354.27, samples=6 iops : min= 94, max= 146, avg=124.33, stdev=22.14, samples=6 lat (usec) : 100=4.17%, 250=0.65%, 500=0.13%, 750=1.69%, 1000=13.67% lat (msec) : 2=6.64%, 4=29.82%, 10=39.58%, 20=3.65% cpu : usr=0.29%, sys=23.74%, ctx=1586, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28714: Thu Apr 17 08:13:27 2025 read: IOPS=116, BW=1856KiB/s (1901kB/s)(5744KiB/3094msec) clat (usec): min=62, max=21223, avg=4589.15, stdev=3027.59 lat (usec): min=62, max=21224, avg=4589.81, stdev=3027.73 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 80], 10.00th=[ 424], 20.00th=[ 2737], | 30.00th=[ 3163], 40.00th=[ 3556], 50.00th=[ 4146], 60.00th=[ 4686], | 70.00th=[ 5342], 80.00th=[ 6390], 90.00th=[ 7832], 95.00th=[10421], | 99.00th=[14877], 99.50th=[20579], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 1376, max= 2400, per=24.86%, avg=1866.67, stdev=374.09, samples=6 iops : min= 86, max= 150, avg=116.67, stdev=23.38, samples=6 write: IOPS=132, BW=2115KiB/s (2166kB/s)(6544KiB/3094msec) clat (usec): min=638, max=15720, avg=3504.32, stdev=3063.32 lat (usec): min=639, max=15721, avg=3505.20, stdev=3063.29 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 775], 10.00th=[ 807], 20.00th=[ 832], | 30.00th=[ 979], 40.00th=[ 1483], 50.00th=[ 2573], 60.00th=[ 3425], | 70.00th=[ 4686], 80.00th=[ 6259], 90.00th=[ 8094], 95.00th=[10159], | 99.00th=[12518], 99.50th=[13304], 99.90th=[15664], 99.95th=[15664], | 99.99th=[15664] bw ( KiB/s): min= 1856, max= 2848, per=26.73%, avg=2112.00, stdev=373.18, samples=6 iops : min= 116, max= 178, avg=132.00, stdev=23.32, samples=6 lat (usec) : 100=4.17%, 250=0.39%, 500=0.26%, 750=2.34%, 1000=14.19% lat (msec) : 2=7.81%, 4=28.52%, 10=37.11%, 20=4.95%, 50=0.26% cpu : usr=0.16%, sys=24.15%, ctx=1486, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28716: Thu Apr 17 08:13:27 2025 read: IOPS=122, BW=1962KiB/s (2009kB/s)(6256KiB/3189msec) clat (usec): min=59, max=23845, avg=4540.83, stdev=3074.77 lat (usec): min=59, max=23845, avg=4544.30, stdev=3076.05 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 86], 10.00th=[ 2073], 20.00th=[ 2671], | 30.00th=[ 3064], 40.00th=[ 3458], 50.00th=[ 4015], 60.00th=[ 4621], | 70.00th=[ 5276], 80.00th=[ 6325], 90.00th=[ 7635], 95.00th=[ 9372], | 99.00th=[17171], 99.50th=[21365], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 1472, max= 2336, per=25.49%, avg=1914.67, stdev=325.97, samples=6 iops : min= 92, max= 146, avg=119.67, stdev=20.37, samples=6 write: IOPS=118, BW=1892KiB/s (1937kB/s)(6032KiB/3189msec) clat (usec): min=680, max=13821, avg=3675.67, stdev=3003.17 lat (usec): min=682, max=13822, avg=3676.54, stdev=3003.19 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 791], 10.00th=[ 824], 20.00th=[ 914], | 30.00th=[ 1287], 40.00th=[ 2180], 50.00th=[ 2868], 60.00th=[ 3490], | 70.00th=[ 4424], 80.00th=[ 6194], 90.00th=[ 8356], 95.00th=[ 9634], | 99.00th=[13304], 99.50th=[13435], 99.90th=[13829], 99.95th=[13829], | 99.99th=[13829] bw ( KiB/s): min= 1664, max= 1920, per=22.95%, avg=1813.33, stdev=92.01, samples=6 iops : min= 104, max= 120, avg=113.33, stdev= 5.75, samples=6 lat (usec) : 100=3.78%, 250=1.04%, 750=0.91%, 1000=10.94% lat (msec) : 2=6.90%, 4=33.98%, 10=38.02%, 20=4.04%, 50=0.39% cpu : usr=0.13%, sys=23.43%, ctx=1663, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28718: Thu Apr 17 08:13:27 2025 read: IOPS=120, BW=1934KiB/s (1980kB/s)(6032KiB/3119msec) clat (usec): min=56, max=16896, avg=4409.60, stdev=2797.45 lat (usec): min=56, max=16897, avg=4410.19, stdev=2797.44 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 84], 10.00th=[ 108], 20.00th=[ 2606], | 30.00th=[ 2933], 40.00th=[ 3458], 50.00th=[ 4015], 60.00th=[ 4817], | 70.00th=[ 5407], 80.00th=[ 6259], 90.00th=[ 7504], 95.00th=[ 8848], | 99.00th=[15139], 99.50th=[16581], 99.90th=[16909], 99.95th=[16909], | 99.99th=[16909] bw ( KiB/s): min= 1216, max= 2368, per=25.42%, avg=1909.33, stdev=392.79, samples=6 iops : min= 76, max= 148, avg=119.33, stdev=24.55, samples=6 write: IOPS=125, BW=2006KiB/s (2054kB/s)(6256KiB/3119msec) clat (usec): min=682, max=25236, avg=3593.88, stdev=3176.26 lat (usec): min=683, max=25237, avg=3594.70, stdev=3176.26 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 775], 10.00th=[ 824], 20.00th=[ 889], | 30.00th=[ 1123], 40.00th=[ 1795], 50.00th=[ 2704], 60.00th=[ 3425], | 70.00th=[ 4555], 80.00th=[ 6063], 90.00th=[ 7832], 95.00th=[ 9634], | 99.00th=[13566], 99.50th=[16581], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 1568, max= 2592, per=25.24%, avg=1994.67, stdev=364.67, samples=6 iops : min= 98, max= 162, avg=124.67, stdev=22.79, samples=6 lat (usec) : 100=4.30%, 250=1.04%, 500=0.13%, 750=1.82%, 1000=11.72% lat (msec) : 2=8.20%, 4=30.21%, 10=38.67%, 20=3.78%, 50=0.13% cpu : usr=0.16%, sys=23.70%, ctx=1601, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7511KiB/s (7691kB/s), 1856KiB/s-1962KiB/s (1901kB/s-2009kB/s), io=23.4MiB (24.5MB), run=3094-3189msec WRITE: bw=7902KiB/s (8092kB/s), 1892KiB/s-2115KiB/s (1937kB/s-2166kB/s), io=24.6MiB (25.8MB), run=3094-3189msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28711: Thu Apr 17 08:13:28 2025 read: IOPS=81, BW=1302KiB/s (1333kB/s)(5920KiB/4548msec) clat (usec): min=2116, max=18964, avg=4805.97, stdev=3256.40 lat (usec): min=2116, max=18965, avg=4806.67, stdev=3256.36 clat percentiles (usec): | 1.00th=[ 2180], 5.00th=[ 2376], 10.00th=[ 2507], 20.00th=[ 2704], | 30.00th=[ 2900], 40.00th=[ 3064], 50.00th=[ 3294], 60.00th=[ 3785], | 70.00th=[ 4490], 80.00th=[ 6194], 90.00th=[10683], 95.00th=[12911], | 99.00th=[14615], 99.50th=[16188], 99.90th=[19006], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 928, max= 1952, per=24.85%, avg=1297.78, stdev=381.70, samples=9 iops : min= 58, max= 122, avg=81.11, stdev=23.86, samples=9 write: IOPS=87, BW=1400KiB/s (1434kB/s)(6368KiB/4548msec) clat (usec): min=2648, max=23081, avg=6850.78, stdev=3378.45 lat (usec): min=2649, max=23082, avg=6851.96, stdev=3378.40 clat percentiles (usec): | 1.00th=[ 3195], 5.00th=[ 3720], 10.00th=[ 3949], 20.00th=[ 4424], | 30.00th=[ 4752], 40.00th=[ 5145], 50.00th=[ 5538], 60.00th=[ 6259], | 70.00th=[ 7439], 80.00th=[ 8848], 90.00th=[11863], 95.00th=[13566], | 99.00th=[19792], 99.50th=[21103], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 992, max= 2048, per=25.50%, avg=1400.89, stdev=387.91, samples=9 iops : min= 62, max= 128, avg=87.56, stdev=24.24, samples=9 lat (msec) : 4=36.33%, 10=50.26%, 20=13.02%, 50=0.39% cpu : usr=0.29%, sys=11.41%, ctx=904, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28713: Thu Apr 17 08:13:28 2025 read: IOPS=78, BW=1253KiB/s (1283kB/s)(5744KiB/4586msec) clat (usec): min=1952, max=18054, avg=4580.67, stdev=2954.36 lat (usec): min=1953, max=18054, avg=4581.32, stdev=2954.35 clat percentiles (usec): | 1.00th=[ 2089], 5.00th=[ 2278], 10.00th=[ 2409], 20.00th=[ 2704], | 30.00th=[ 2900], 40.00th=[ 3097], 50.00th=[ 3359], 60.00th=[ 3687], | 70.00th=[ 4424], 80.00th=[ 5735], 90.00th=[ 9634], 95.00th=[11731], | 99.00th=[14353], 99.50th=[16909], 99.90th=[17957], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 768, max= 1792, per=24.03%, avg=1254.67, stdev=385.81, samples=9 iops : min= 48, max= 112, avg=78.33, stdev=24.03, samples=9 write: IOPS=89, BW=1427KiB/s (1461kB/s)(6544KiB/4586msec) clat (usec): min=2766, max=28660, avg=7070.29, stdev=3656.50 lat (usec): min=2768, max=28662, avg=7071.69, stdev=3656.33 clat percentiles (usec): | 1.00th=[ 3261], 5.00th=[ 3654], 10.00th=[ 4015], 20.00th=[ 4490], | 30.00th=[ 4883], 40.00th=[ 5145], 50.00th=[ 5604], 60.00th=[ 6390], | 70.00th=[ 7898], 80.00th=[ 9765], 90.00th=[11600], 95.00th=[14091], | 99.00th=[21365], 99.50th=[21627], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 1024, max= 2144, per=26.01%, avg=1428.89, stdev=400.35, samples=9 iops : min= 64, max= 134, avg=89.22, stdev=24.91, samples=9 lat (msec) : 2=0.13%, 4=35.29%, 10=50.26%, 20=13.67%, 50=0.65% cpu : usr=0.15%, sys=11.47%, ctx=876, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28715: Thu Apr 17 08:13:28 2025 read: IOPS=87, BW=1394KiB/s (1428kB/s)(6256KiB/4487msec) clat (usec): min=2065, max=16199, avg=4675.33, stdev=2924.38 lat (usec): min=2066, max=16200, avg=4676.16, stdev=2924.80 clat percentiles (usec): | 1.00th=[ 2147], 5.00th=[ 2311], 10.00th=[ 2409], 20.00th=[ 2671], | 30.00th=[ 2900], 40.00th=[ 3130], 50.00th=[ 3392], 60.00th=[ 3752], | 70.00th=[ 4555], 80.00th=[ 6521], 90.00th=[ 9765], 95.00th=[11207], | 99.00th=[14746], 99.50th=[15926], 99.90th=[16188], 99.95th=[16188], | 99.99th=[16188] bw ( KiB/s): min= 864, max= 1984, per=25.28%, avg=1320.00, stdev=395.91, samples=8 iops : min= 54, max= 124, avg=82.50, stdev=24.74, samples=8 write: IOPS=84, BW=1344KiB/s (1377kB/s)(6032KiB/4487msec) clat (usec): min=2836, max=28722, avg=6945.79, stdev=3650.43 lat (usec): min=2837, max=28723, avg=6947.05, stdev=3650.41 clat percentiles (usec): | 1.00th=[ 3326], 5.00th=[ 3687], 10.00th=[ 3949], 20.00th=[ 4293], | 30.00th=[ 4752], 40.00th=[ 5211], 50.00th=[ 5604], 60.00th=[ 6325], | 70.00th=[ 7570], 80.00th=[ 8979], 90.00th=[11731], 95.00th=[13960], | 99.00th=[21365], 99.50th=[27395], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 992, max= 2048, per=23.95%, avg=1316.00, stdev=334.94, samples=8 iops : min= 62, max= 128, avg=82.25, stdev=20.93, samples=8 lat (msec) : 4=38.67%, 10=48.96%, 20=11.85%, 50=0.52% cpu : usr=0.16%, sys=11.70%, ctx=871, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28717: Thu Apr 17 08:13:28 2025 read: IOPS=85, BW=1372KiB/s (1404kB/s)(6032KiB/4398msec) clat (usec): min=1847, max=18148, avg=4594.23, stdev=3093.24 lat (usec): min=1847, max=18149, avg=4594.88, stdev=3093.22 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2376], 10.00th=[ 2507], 20.00th=[ 2704], | 30.00th=[ 2900], 40.00th=[ 3032], 50.00th=[ 3294], 60.00th=[ 3556], | 70.00th=[ 4178], 80.00th=[ 5932], 90.00th=[ 9241], 95.00th=[12125], | 99.00th=[15664], 99.50th=[16581], 99.90th=[18220], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 928, max= 2043, per=25.27%, avg=1319.37, stdev=353.12, samples=8 iops : min= 58, max= 127, avg=82.37, stdev=21.87, samples=8 write: IOPS=88, BW=1422KiB/s (1457kB/s)(6256KiB/4398msec) clat (usec): min=2624, max=27640, avg=6780.30, stdev=3317.27 lat (usec): min=2625, max=27641, avg=6781.48, stdev=3317.23 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3687], 10.00th=[ 4015], 20.00th=[ 4490], | 30.00th=[ 4817], 40.00th=[ 5145], 50.00th=[ 5473], 60.00th=[ 6063], | 70.00th=[ 7308], 80.00th=[ 8979], 90.00th=[11207], 95.00th=[13829], | 99.00th=[18482], 99.50th=[22152], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 1056, max= 1916, per=25.18%, avg=1383.50, stdev=327.08, samples=8 iops : min= 66, max= 119, avg=86.38, stdev=20.27, samples=8 lat (msec) : 2=0.13%, 4=38.28%, 10=49.61%, 20=11.72%, 50=0.26% cpu : usr=0.09%, sys=12.12%, ctx=886, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5223KiB/s (5348kB/s), 1253KiB/s-1394KiB/s (1283kB/s-1428kB/s), io=23.4MiB (24.5MB), run=4398-4586msec WRITE: bw=5495KiB/s (5627kB/s), 1344KiB/s-1427KiB/s (1377kB/s-1461kB/s), io=24.6MiB (25.8MB), run=4398-4586msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28732: Thu Apr 17 08:13:30 2025 read: IOPS=13, BW=13.8MiB/s (14.4MB/s)(7168KiB/508msec) clat (usec): min=422, max=82445, avg=28434.80, stdev=28012.46 lat (usec): min=423, max=82446, avg=28435.57, stdev=28012.50 clat percentiles (usec): | 1.00th=[ 424], 5.00th=[ 424], 10.00th=[ 424], 20.00th=[ 433], | 30.00th=[16581], 40.00th=[16581], 50.00th=[31065], 60.00th=[31327], | 70.00th=[31327], 80.00th=[36439], 90.00th=[82314], 95.00th=[82314], | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min=12288, max=12288, per=41.88%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9.84MiB/s (10.3MB/s)(5120KiB/508msec) clat (usec): min=26137, max=81416, avg=52721.41, stdev=21361.55 lat (usec): min=26157, max=81466, avg=52756.66, stdev=21381.79 clat percentiles (usec): | 1.00th=[26084], 5.00th=[26084], 10.00th=[26084], 20.00th=[26084], | 30.00th=[41681], 40.00th=[41681], 50.00th=[49021], 60.00th=[49021], | 70.00th=[65274], 80.00th=[65274], 90.00th=[81265], 95.00th=[81265], | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 8192, max= 8192, per=19.94%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 20=8.33%, 50=50.00%, 100=25.00% cpu : usr=0.00%, sys=28.60%, ctx=349, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28733: Thu Apr 17 08:13:30 2025 read: IOPS=2, BW=2934KiB/s (3005kB/s)(2048KiB/698msec) clat (usec): min=27371, max=74020, avg=50695.96, stdev=32986.24 lat (usec): min=27371, max=74022, avg=50696.98, stdev=32986.70 clat percentiles (usec): | 1.00th=[27395], 5.00th=[27395], 10.00th=[27395], 20.00th=[27395], | 30.00th=[27395], 40.00th=[27395], 50.00th=[27395], 60.00th=[73925], | 70.00th=[73925], 80.00th=[73925], 90.00th=[73925], 95.00th=[73925], | 99.00th=[73925], 99.50th=[73925], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 4055, max= 4055, per=13.82%, avg=4055.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=14, BW=14.3MiB/s (15.0MB/s)(10.0MiB/698msec) clat (usec): min=17692, max=99024, avg=54010.25, stdev=29682.66 lat (usec): min=17762, max=99071, avg=54073.53, stdev=29680.46 clat percentiles (usec): | 1.00th=[17695], 5.00th=[17695], 10.00th=[17695], 20.00th=[24511], | 30.00th=[28967], 40.00th=[38011], 50.00th=[44827], 60.00th=[48497], | 70.00th=[84411], 80.00th=[84411], 90.00th=[95945], 95.00th=[99091], | 99.00th=[99091], 99.50th=[99091], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min=10138, max=10138, per=24.68%, avg=10138.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=41.67% cpu : usr=0.00%, sys=30.56%, ctx=267, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28734: Thu Apr 17 08:13:30 2025 read: IOPS=12, BW=12.6MiB/s (13.2MB/s)(6144KiB/477msec) clat (usec): min=10371, max=46175, avg=29061.88, stdev=12056.96 lat (usec): min=10372, max=46176, avg=29062.77, stdev=12056.94 clat percentiles (usec): | 1.00th=[10421], 5.00th=[10421], 10.00th=[10421], 20.00th=[22938], | 30.00th=[22938], 40.00th=[28967], 50.00th=[28967], 60.00th=[30278], | 70.00th=[35914], 80.00th=[35914], 90.00th=[46400], 95.00th=[46400], | 99.00th=[46400], 99.50th=[46400], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] write: IOPS=12, BW=12.6MiB/s (13.2MB/s)(6144KiB/477msec) clat (msec): min=22, max=129, avg=47.29, stdev=40.95 lat (msec): min=22, max=129, avg=47.34, stdev=40.93 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 26], | 30.00th=[ 26], 40.00th=[ 28], 50.00th=[ 28], 60.00th=[ 40], | 70.00th=[ 40], 80.00th=[ 40], 90.00th=[ 130], 95.00th=[ 130], | 99.00th=[ 130], 99.50th=[ 130], 99.90th=[ 130], 99.95th=[ 130], | 99.99th=[ 130] lat (msec) : 20=8.33%, 50=83.33%, 250=8.33% cpu : usr=0.00%, sys=38.24%, ctx=626, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28735: Thu Apr 17 08:13:30 2025 read: IOPS=8, BW=8752KiB/s (8962kB/s)(5120KiB/585msec) clat (usec): min=485, max=33658, avg=21082.40, stdev=12327.03 lat (usec): min=486, max=33659, avg=21083.16, stdev=12327.13 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 486], 10.00th=[ 486], 20.00th=[ 486], | 30.00th=[22676], 40.00th=[22676], 50.00th=[23462], 60.00th=[23462], | 70.00th=[25297], 80.00th=[25297], 90.00th=[33817], 95.00th=[33817], | 99.00th=[33817], 99.50th=[33817], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 8192, max= 8192, per=27.92%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.0MiB/s (12.5MB/s)(7168KiB/585msec) clat (msec): min=19, max=102, avg=61.04, stdev=36.60 lat (msec): min=19, max=102, avg=61.10, stdev=36.59 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 22], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 82], 60.00th=[ 87], | 70.00th=[ 87], 80.00th=[ 89], 90.00th=[ 103], 95.00th=[ 103], | 99.00th=[ 103], 99.50th=[ 103], 99.90th=[ 103], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min=12288, max=12288, per=29.91%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 20=8.33%, 50=50.00%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=31.68%, ctx=320, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=28.7MiB/s (30.0MB/s), 2934KiB/s-13.8MiB/s (3005kB/s-14.4MB/s), io=20.0MiB (20.0MB), run=477-698msec WRITE: bw=40.1MiB/s (42.1MB/s), 9.84MiB/s-14.3MiB/s (10.3MB/s-15.0MB/s), io=28.0MiB (29.4MB), run=477-698msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28727: Thu Apr 17 08:13:30 2025 read: IOPS=13, BW=13.4MiB/s (14.0MB/s)(7168KiB/523msec) clat (usec): min=12535, max=67239, avg=23805.25, stdev=19586.75 lat (usec): min=12536, max=67240, avg=23806.16, stdev=19586.76 clat percentiles (usec): | 1.00th=[12518], 5.00th=[12518], 10.00th=[12518], 20.00th=[13566], | 30.00th=[14222], 40.00th=[14222], 50.00th=[14615], 60.00th=[22152], | 70.00th=[22152], 80.00th=[22414], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min=12263, max=12263, per=45.09%, avg=12263.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9790KiB/s (10.0MB/s)(5120KiB/523msec) clat (usec): min=52258, max=86277, avg=67816.94, stdev=16220.69 lat (usec): min=52357, max=86306, avg=67884.68, stdev=16187.54 clat percentiles (usec): | 1.00th=[52167], 5.00th=[52167], 10.00th=[52167], 20.00th=[52167], | 30.00th=[52167], 40.00th=[52167], 50.00th=[65274], 60.00th=[65274], | 70.00th=[82314], 80.00th=[82314], 90.00th=[86508], 95.00th=[86508], | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 8175, max= 8175, per=21.47%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=16.67%, 100=50.00% cpu : usr=0.19%, sys=13.41%, ctx=55, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28728: Thu Apr 17 08:13:30 2025 read: IOPS=2, BW=2889KiB/s (2958kB/s)(2048KiB/709msec) clat (usec): min=17364, max=39746, avg=28555.59, stdev=15826.84 lat (usec): min=17365, max=39747, avg=28556.57, stdev=15826.72 clat percentiles (usec): | 1.00th=[17433], 5.00th=[17433], 10.00th=[17433], 20.00th=[17433], | 30.00th=[17433], 40.00th=[17433], 50.00th=[17433], 60.00th=[39584], | 70.00th=[39584], 80.00th=[39584], 90.00th=[39584], 95.00th=[39584], | 99.00th=[39584], 99.50th=[39584], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 4087, max= 4087, per=15.03%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=14, BW=14.1MiB/s (14.8MB/s)(10.0MiB/709msec) clat (msec): min=13, max=135, avg=60.86, stdev=39.96 lat (msec): min=13, max=135, avg=60.93, stdev=39.97 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 24], 40.00th=[ 45], 50.00th=[ 57], 60.00th=[ 62], | 70.00th=[ 74], 80.00th=[ 74], 90.00th=[ 112], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min=12263, max=12263, per=32.21%, avg=12263.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=25.00%, 100=33.33%, 250=16.67% cpu : usr=0.00%, sys=12.27%, ctx=123, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28729: Thu Apr 17 08:13:30 2025 read: IOPS=8, BW=9184KiB/s (9404kB/s)(6144KiB/669msec) clat (usec): min=28197, max=96523, avg=59129.65, stdev=23479.31 lat (usec): min=28197, max=96524, avg=59130.51, stdev=23479.18 clat percentiles (usec): | 1.00th=[28181], 5.00th=[28181], 10.00th=[28181], 20.00th=[43779], | 30.00th=[43779], 40.00th=[55837], 50.00th=[55837], 60.00th=[58983], | 70.00th=[71828], 80.00th=[71828], 90.00th=[96994], 95.00th=[96994], | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 8192, max= 8192, per=30.12%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=8, BW=9184KiB/s (9404kB/s)(6144KiB/669msec) clat (usec): min=19697, max=86571, avg=51172.78, stdev=26865.28 lat (usec): min=19798, max=86745, avg=51236.79, stdev=26886.05 clat percentiles (usec): | 1.00th=[19792], 5.00th=[19792], 10.00th=[19792], 20.00th=[30540], | 30.00th=[30540], 40.00th=[33817], 50.00th=[33817], 60.00th=[62129], | 70.00th=[73925], 80.00th=[73925], 90.00th=[86508], 95.00th=[86508], | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min=10240, max=10240, per=26.89%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=58.33% cpu : usr=0.00%, sys=15.57%, ctx=52, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28730: Thu Apr 17 08:13:30 2025 read: IOPS=6, BW=6799KiB/s (6963kB/s)(5120KiB/753msec) clat (usec): min=16107, max=94451, avg=53245.07, stdev=30802.33 lat (usec): min=16108, max=94452, avg=53248.94, stdev=30801.76 clat percentiles (usec): | 1.00th=[16057], 5.00th=[16057], 10.00th=[16057], 20.00th=[16057], | 30.00th=[34866], 40.00th=[34866], 50.00th=[49021], 60.00th=[49021], | 70.00th=[71828], 80.00th=[71828], 90.00th=[94897], 95.00th=[94897], | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 8175, max= 8175, per=30.06%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9519KiB/s (9748kB/s)(7168KiB/753msec) clat (msec): min=18, max=121, avg=66.49, stdev=42.12 lat (msec): min=18, max=121, avg=66.56, stdev=42.10 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 19], | 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 83], 60.00th=[ 91], | 70.00th=[ 91], 80.00th=[ 101], 90.00th=[ 122], 95.00th=[ 122], | 99.00th=[ 122], 99.50th=[ 122], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 8175, max= 8175, per=21.47%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=25.00%, 100=41.67%, 250=8.33% cpu : usr=0.13%, sys=12.77%, ctx=122, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=26.6MiB/s (27.8MB/s), 2889KiB/s-13.4MiB/s (2958kB/s-14.0MB/s), io=20.0MiB (20.0MB), run=523-753msec WRITE: bw=37.2MiB/s (38.0MB/s), 9184KiB/s-14.1MiB/s (9404kB/s-14.8MB/s), io=28.0MiB (29.4MB), run=523-753msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28744: Thu Apr 17 08:13:32 2025 read: IOPS=8, BW=32.8MiB/s (34.4MB/s)(12.0MiB/366msec) clat (msec): min=104, max=139, avg=117.31, stdev=19.21 lat (msec): min=104, max=139, avg=117.31, stdev=19.21 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 105], 10.00th=[ 105], 20.00th=[ 105], | 30.00th=[ 105], 40.00th=[ 109], 50.00th=[ 109], 60.00th=[ 109], | 70.00th=[ 140], 80.00th=[ 140], 90.00th=[ 140], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=46.03%, ctx=2768, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28746: Thu Apr 17 08:13:32 2025 read: IOPS=1, BW=4847KiB/s (4964kB/s)(4096KiB/845msec) clat (nsec): min=95954k, max=95954k, avg=95954265.00, stdev= 0.00 lat (nsec): min=95956k, max=95956k, avg=95955590.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[95945], 5.00th=[95945], 10.00th=[95945], 20.00th=[95945], | 30.00th=[95945], 40.00th=[95945], 50.00th=[95945], 60.00th=[95945], | 70.00th=[95945], 80.00th=[95945], 90.00th=[95945], 95.00th=[95945], | 99.00th=[95945], 99.50th=[95945], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 8192, max= 8192, per=42.25%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9695KiB/s (9927kB/s)(8192KiB/845msec) clat (msec): min=347, max=372, avg=359.77, stdev=17.64 lat (msec): min=347, max=372, avg=360.02, stdev=17.66 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 347], 10.00th=[ 347], 20.00th=[ 347], | 30.00th=[ 347], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 372], | 70.00th=[ 372], 80.00th=[ 372], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 8192, max= 8192, per=21.13%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.24%, sys=25.12%, ctx=1020, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28747: Thu Apr 17 08:13:32 2025 write: IOPS=4, BW=17.3MiB/s (18.2MB/s)(12.0MiB/692msec) clat (msec): min=94, max=340, avg=225.97, stdev=123.48 lat (msec): min=95, max=340, avg=226.18, stdev=123.44 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 95], 10.00th=[ 95], 20.00th=[ 95], | 30.00th=[ 95], 40.00th=[ 243], 50.00th=[ 243], 60.00th=[ 243], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 342], 95.00th=[ 342], | 99.00th=[ 342], 99.50th=[ 342], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 8175, max= 8175, per=21.08%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.14%, sys=35.17%, ctx=32, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28748: Thu Apr 17 08:13:32 2025 write: IOPS=5, BW=20.9MiB/s (21.9MB/s)(12.0MiB/575msec) clat (msec): min=96, max=298, avg=187.48, stdev=102.73 lat (msec): min=96, max=298, avg=187.76, stdev=102.73 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 96], 10.00th=[ 96], 20.00th=[ 96], | 30.00th=[ 96], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 169], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min=16351, max=16351, per=42.17%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=49.48%, ctx=47, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.9MiB/s (19.9MB/s), 4847KiB/s-32.8MiB/s (4964kB/s-34.4MB/s), io=16.0MiB (16.8MB), run=366-845msec WRITE: bw=37.9MiB/s (39.7MB/s), 9695KiB/s-20.9MiB/s (9927kB/s-21.9MB/s), io=32.0MiB (33.6MB), run=575-845msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28741: Thu Apr 17 08:13:32 2025 read: IOPS=5, BW=20.5MiB/s (21.5MB/s)(12.0MiB/586msec) clat (msec): min=109, max=342, avg=189.41, stdev=132.48 lat (msec): min=109, max=342, avg=189.41, stdev=132.48 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 110], 10.00th=[ 110], 20.00th=[ 110], | 30.00th=[ 110], 40.00th=[ 116], 50.00th=[ 116], 60.00th=[ 116], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 342], 95.00th=[ 342], | 99.00th=[ 342], 99.50th=[ 342], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.17%, sys=14.02%, ctx=113, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28742: Thu Apr 17 08:13:32 2025 read: IOPS=1, BW=4267KiB/s (4369kB/s)(4096KiB/960msec) clat (nsec): min=42002k, max=42002k, avg=42001977.00, stdev= 0.00 lat (nsec): min=42003k, max=42003k, avg=42003099.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[42206], 5.00th=[42206], 10.00th=[42206], 20.00th=[42206], | 30.00th=[42206], 40.00th=[42206], 50.00th=[42206], 60.00th=[42206], | 70.00th=[42206], 80.00th=[42206], 90.00th=[42206], 95.00th=[42206], | 99.00th=[42206], 99.50th=[42206], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 8192, max= 8192, per=48.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=8533KiB/s (8738kB/s)(8192KiB/960msec) clat (msec): min=319, max=547, avg=433.94, stdev=161.27 lat (msec): min=320, max=548, avg=434.24, stdev=161.25 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 321], 10.00th=[ 321], 20.00th=[ 321], | 30.00th=[ 321], 40.00th=[ 321], 50.00th=[ 321], 60.00th=[ 550], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 8192, max= 8192, per=24.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=16.27%, ctx=195, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28743: Thu Apr 17 08:13:32 2025 write: IOPS=3, BW=13.5MiB/s (14.2MB/s)(12.0MiB/888msec) clat (msec): min=119, max=411, avg=291.27, stdev=152.85 lat (msec): min=119, max=411, avg=291.56, stdev=152.90 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 342], 50.00th=[ 342], 60.00th=[ 342], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 8000, max= 8000, per=23.44%, avg=8000.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=14.21%, ctx=126, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28745: Thu Apr 17 08:13:32 2025 write: IOPS=6, BW=25.4MiB/s (26.7MB/s)(12.0MiB/472msec) clat (msec): min=46, max=319, avg=151.59, stdev=146.87 lat (msec): min=46, max=319, avg=151.87, stdev=146.86 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 89], 50.00th=[ 89], 60.00th=[ 89], | 70.00th=[ 321], 80.00th=[ 321], 90.00th=[ 321], 95.00th=[ 321], | 99.00th=[ 321], 99.50th=[ 321], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] lat (msec) : 50=33.33%, 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=16.14%, ctx=77, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.7MiB/s (17.5MB/s), 4267KiB/s-20.5MiB/s (4369kB/s-21.5MB/s), io=16.0MiB (16.8MB), run=586-960msec WRITE: bw=33.3MiB/s (34.0MB/s), 8533KiB/s-25.4MiB/s (8738kB/s-26.7MB/s), io=32.0MiB (33.6MB), run=472-960msec