== sanity test 398b: DIO and buffer IO race ============== 13:32:05 (1746120725) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.27393 s, 39.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=588: Thu May 1 13:32:21 2025 read: IOPS=121, BW=488KiB/s (499kB/s)(5992KiB/12285msec) clat (usec): min=64, max=154782, avg=4432.90, stdev=4573.11 lat (usec): min=64, max=154783, avg=4433.52, stdev=4573.12 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 115], 10.00th=[ 2147], 20.00th=[ 2507], | 30.00th=[ 2900], 40.00th=[ 3294], 50.00th=[ 3851], 60.00th=[ 4621], | 70.00th=[ 5342], 80.00th=[ 6063], 90.00th=[ 7111], 95.00th=[ 8455], | 99.00th=[ 12780], 99.50th=[ 13566], 99.90th=[ 17171], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 240, max= 600, per=25.28%, avg=482.29, stdev=73.23, samples=24 iops : min= 60, max= 150, avg=120.54, stdev=18.31, samples=24 write: IOPS=128, BW=512KiB/s (525kB/s)(6296KiB/12285msec) clat (usec): min=494, max=30025, avg=3538.01, stdev=3142.62 lat (usec): min=495, max=30025, avg=3538.85, stdev=3142.53 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 562], 10.00th=[ 594], 20.00th=[ 668], | 30.00th=[ 889], 40.00th=[ 2180], 50.00th=[ 2900], 60.00th=[ 3523], | 70.00th=[ 4490], 80.00th=[ 5997], 90.00th=[ 7963], 95.00th=[ 9634], | 99.00th=[13042], 99.50th=[15270], 99.90th=[17695], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 296, max= 656, per=25.44%, avg=508.29, stdev=84.12, samples=24 iops : min= 74, max= 164, avg=127.04, stdev=21.07, samples=24 lat (usec) : 100=2.05%, 250=0.72%, 500=0.13%, 750=13.05%, 1000=3.42% lat (msec) : 2=4.43%, 4=35.32%, 10=37.53%, 20=3.29%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=23.02%, ctx=5664, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=589: Thu May 1 13:32:21 2025 read: IOPS=120, BW=480KiB/s (492kB/s)(5940KiB/12369msec) clat (usec): min=56, max=190474, avg=4595.18, stdev=5530.44 lat (usec): min=56, max=190475, avg=4596.13, stdev=5530.42 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 123], 10.00th=[ 2147], 20.00th=[ 2606], | 30.00th=[ 2933], 40.00th=[ 3359], 50.00th=[ 3982], 60.00th=[ 4686], | 70.00th=[ 5276], 80.00th=[ 6128], 90.00th=[ 7570], 95.00th=[ 8848], | 99.00th=[ 14877], 99.50th=[ 16909], 99.90th=[ 22414], 99.95th=[189793], | 99.99th=[189793] bw ( KiB/s): min= 216, max= 672, per=24.89%, avg=474.96, stdev=81.61, samples=24 iops : min= 54, max= 168, avg=118.71, stdev=20.41, samples=24 write: IOPS=128, BW=513KiB/s (526kB/s)(6348KiB/12369msec) clat (usec): min=498, max=22662, avg=3458.39, stdev=3058.87 lat (usec): min=499, max=22663, avg=3459.62, stdev=3058.45 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 570], 10.00th=[ 603], 20.00th=[ 660], | 30.00th=[ 840], 40.00th=[ 2040], 50.00th=[ 2835], 60.00th=[ 3490], | 70.00th=[ 4424], 80.00th=[ 5669], 90.00th=[ 7832], 95.00th=[ 9372], | 99.00th=[12911], 99.50th=[14484], 99.90th=[19792], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 272, max= 672, per=25.01%, avg=499.62, stdev=81.67, samples=24 iops : min= 68, max= 168, avg=124.87, stdev=20.43, samples=24 lat (usec) : 100=2.05%, 250=0.78%, 500=0.13%, 750=14.00%, 1000=3.35% lat (msec) : 2=4.13%, 4=34.11%, 10=37.63%, 20=3.71%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=22.85%, ctx=5609, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=590: Thu May 1 13:32:21 2025 read: IOPS=119, BW=479KiB/s (491kB/s)(6028KiB/12582msec) clat (usec): min=60, max=24662, avg=4509.60, stdev=2634.28 lat (usec): min=60, max=24663, avg=4510.21, stdev=2634.26 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 114], 10.00th=[ 2114], 20.00th=[ 2573], | 30.00th=[ 2966], 40.00th=[ 3458], 50.00th=[ 4080], 60.00th=[ 4817], | 70.00th=[ 5473], 80.00th=[ 6325], 90.00th=[ 7439], 95.00th=[ 8848], | 99.00th=[13566], 99.50th=[16188], 99.90th=[18744], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 232, max= 608, per=24.61%, avg=469.62, stdev=76.88, samples=24 iops : min= 58, max= 152, avg=117.38, stdev=19.24, samples=24 write: IOPS=124, BW=498KiB/s (509kB/s)(6260KiB/12582msec) clat (usec): min=476, max=168698, avg=3663.14, stdev=5261.19 lat (usec): min=476, max=168698, avg=3665.07, stdev=5261.58 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 553], 10.00th=[ 586], 20.00th=[ 652], | 30.00th=[ 840], 40.00th=[ 2008], 50.00th=[ 2900], 60.00th=[ 3589], | 70.00th=[ 4555], 80.00th=[ 6128], 90.00th=[ 8029], 95.00th=[ 9503], | 99.00th=[ 13829], 99.50th=[ 16188], 99.90th=[ 25297], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 312, max= 664, per=24.32%, avg=485.96, stdev=70.82, samples=24 iops : min= 78, max= 166, avg=121.46, stdev=17.70, samples=24 lat (usec) : 100=2.28%, 250=0.59%, 500=0.16%, 750=13.67%, 1000=3.39% lat (msec) : 2=4.26%, 4=32.71%, 10=39.32%, 20=3.45%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=22.44%, ctx=5732, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=591: Thu May 1 13:32:21 2025 read: IOPS=122, BW=490KiB/s (502kB/s)(6052KiB/12343msec) clat (usec): min=59, max=18900, avg=4478.92, stdev=2512.35 lat (usec): min=59, max=18900, avg=4479.55, stdev=2512.40 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 145], 10.00th=[ 2147], 20.00th=[ 2638], | 30.00th=[ 3032], 40.00th=[ 3458], 50.00th=[ 4015], 60.00th=[ 4817], | 70.00th=[ 5407], 80.00th=[ 6194], 90.00th=[ 7242], 95.00th=[ 8586], | 99.00th=[13435], 99.50th=[15664], 99.90th=[17695], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 216, max= 608, per=25.50%, avg=486.54, stdev=82.72, samples=24 iops : min= 54, max= 152, avg=121.58, stdev=20.71, samples=24 write: IOPS=126, BW=505KiB/s (517kB/s)(6236KiB/12343msec) clat (usec): min=463, max=174388, avg=3521.07, stdev=5332.17 lat (usec): min=464, max=174390, avg=3521.85, stdev=5332.20 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 562], 10.00th=[ 594], 20.00th=[ 668], | 30.00th=[ 840], 40.00th=[ 1680], 50.00th=[ 2737], 60.00th=[ 3392], | 70.00th=[ 4293], 80.00th=[ 5866], 90.00th=[ 7767], 95.00th=[ 9634], | 99.00th=[ 13042], 99.50th=[ 16581], 99.90th=[ 21627], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 248, max= 688, per=25.07%, avg=500.92, stdev=85.10, samples=24 iops : min= 62, max= 172, avg=125.17, stdev=21.34, samples=24 lat (usec) : 100=2.08%, 250=0.46%, 500=0.16%, 750=13.64%, 1000=3.78% lat (msec) : 2=4.75%, 4=33.92%, 10=37.73%, 20=3.39%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=22.74%, ctx=5705, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1908KiB/s (1954kB/s), 479KiB/s-490KiB/s (491kB/s-502kB/s), io=23.4MiB (24.6MB), run=12285-12582msec WRITE: bw=1998KiB/s (2046kB/s), 498KiB/s-513KiB/s (509kB/s-526kB/s), io=24.6MiB (25.7MB), run=12285-12582msec rand-rw: (groupid=0, jobs=1): err= 0: pid=583: Thu May 1 13:32:27 2025 read: IOPS=80, BW=323KiB/s (331kB/s)(5992KiB/18541msec) clat (usec): min=1729, max=177915, avg=4943.98, stdev=5321.20 lat (usec): min=1729, max=177917, avg=4944.83, stdev=5321.19 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2212], 10.00th=[ 2376], 20.00th=[ 2835], | 30.00th=[ 3261], 40.00th=[ 3687], 50.00th=[ 3949], 60.00th=[ 4228], | 70.00th=[ 4752], 80.00th=[ 5997], 90.00th=[ 9241], 95.00th=[ 11207], | 99.00th=[ 15139], 99.50th=[ 17433], 99.90th=[ 23200], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 111, max= 576, per=25.39%, avg=323.22, stdev=88.00, samples=37 iops : min= 27, max= 144, avg=80.78, stdev=22.05, samples=37 write: IOPS=84, BW=340KiB/s (348kB/s)(6296KiB/18541msec) clat (usec): min=2619, max=33530, avg=7016.78, stdev=3290.52 lat (usec): min=2620, max=33531, avg=7017.77, stdev=3290.49 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3425], 10.00th=[ 3785], 20.00th=[ 4359], | 30.00th=[ 5145], 40.00th=[ 5866], 50.00th=[ 6390], 60.00th=[ 6783], | 70.00th=[ 7635], 80.00th=[ 8979], 90.00th=[11207], 95.00th=[13173], | 99.00th=[18744], 99.50th=[23462], 99.90th=[25822], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 135, max= 640, per=25.45%, avg=339.22, stdev=87.12, samples=37 iops : min= 33, max= 160, avg=84.78, stdev=21.83, samples=37 lat (msec) : 2=0.72%, 4=31.80%, 10=56.22%, 20=10.77%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=12.25%, ctx=3544, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=584: Thu May 1 13:32:27 2025 read: IOPS=78, BW=315KiB/s (323kB/s)(5940KiB/18851msec) clat (usec): min=1664, max=194777, avg=5202.34, stdev=5949.15 lat (usec): min=1664, max=194779, avg=5203.14, stdev=5949.16 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2212], 10.00th=[ 2409], 20.00th=[ 2769], | 30.00th=[ 3228], 40.00th=[ 3687], 50.00th=[ 4015], 60.00th=[ 4293], | 70.00th=[ 4883], 80.00th=[ 6390], 90.00th=[ 9896], 95.00th=[ 12518], | 99.00th=[ 18744], 99.50th=[ 20579], 99.90th=[ 26346], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 79, max= 624, per=24.85%, avg=316.30, stdev=99.66, samples=37 iops : min= 19, max= 156, avg=79.05, stdev=24.96, samples=37 write: IOPS=84, BW=337KiB/s (345kB/s)(6348KiB/18851msec) clat (usec): min=2350, max=30619, avg=6953.56, stdev=3193.16 lat (usec): min=2351, max=30620, avg=6954.60, stdev=3193.10 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3425], 10.00th=[ 3785], 20.00th=[ 4424], | 30.00th=[ 5211], 40.00th=[ 5866], 50.00th=[ 6456], 60.00th=[ 6849], | 70.00th=[ 7439], 80.00th=[ 8717], 90.00th=[11076], 95.00th=[12649], | 99.00th=[18744], 99.50th=[23200], 99.90th=[30278], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 151, max= 568, per=25.14%, avg=335.11, stdev=81.60, samples=37 iops : min= 37, max= 142, avg=83.76, stdev=20.45, samples=37 lat (msec) : 2=0.65%, 4=30.08%, 10=57.45%, 20=11.07%, 50=0.72% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=12.14%, ctx=3570, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=585: Thu May 1 13:32:27 2025 read: IOPS=80, BW=322KiB/s (330kB/s)(6028KiB/18729msec) clat (usec): min=1782, max=26419, avg=4866.48, stdev=2953.59 lat (usec): min=1783, max=26420, avg=4867.32, stdev=2953.55 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2245], 10.00th=[ 2442], 20.00th=[ 2868], | 30.00th=[ 3294], 40.00th=[ 3720], 50.00th=[ 4015], 60.00th=[ 4293], | 70.00th=[ 4752], 80.00th=[ 6194], 90.00th=[ 8848], 95.00th=[11338], | 99.00th=[15664], 99.50th=[19006], 99.90th=[25035], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 119, max= 568, per=25.29%, avg=321.92, stdev=84.40, samples=37 iops : min= 29, max= 142, avg=80.46, stdev=21.15, samples=37 write: IOPS=83, BW=334KiB/s (342kB/s)(6260KiB/18729msec) clat (msec): min=2, max=198, avg= 7.22, stdev= 5.87 lat (msec): min=2, max=198, avg= 7.22, stdev= 5.87 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 21], 99.50th=[ 26], 99.90th=[ 34], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 151, max= 656, per=25.03%, avg=333.59, stdev=99.00, samples=37 iops : min= 37, max= 164, avg=83.38, stdev=24.79, samples=37 lat (msec) : 2=0.59%, 4=30.01%, 10=58.95%, 20=9.73%, 50=0.68% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=12.27%, ctx=3570, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=586: Thu May 1 13:32:27 2025 read: IOPS=80, BW=323KiB/s (331kB/s)(6052KiB/18741msec) clat (usec): min=1680, max=22389, avg=4953.14, stdev=3167.11 lat (usec): min=1681, max=22390, avg=4953.93, stdev=3167.06 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2212], 10.00th=[ 2409], 20.00th=[ 2835], | 30.00th=[ 3228], 40.00th=[ 3654], 50.00th=[ 3949], 60.00th=[ 4293], | 70.00th=[ 4752], 80.00th=[ 6128], 90.00th=[ 9372], 95.00th=[11994], | 99.00th=[16581], 99.50th=[19530], 99.90th=[21890], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 111, max= 552, per=25.27%, avg=321.65, stdev=86.79, samples=37 iops : min= 27, max= 138, avg=80.35, stdev=21.75, samples=37 write: IOPS=83, BW=333KiB/s (341kB/s)(6236KiB/18741msec) clat (msec): min=2, max=198, avg= 7.16, stdev= 5.85 lat (msec): min=2, max=198, avg= 7.16, stdev= 5.85 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 20], 99.50th=[ 25], 99.90th=[ 28], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 151, max= 600, per=24.96%, avg=332.68, stdev=85.82, samples=37 iops : min= 37, max= 150, avg=83.11, stdev=21.50, samples=37 lat (msec) : 2=0.52%, 4=31.84%, 10=56.09%, 20=10.90%, 50=0.62% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=12.23%, ctx=3545, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1274KiB/s (1304kB/s), 315KiB/s-323KiB/s (323kB/s-331kB/s), io=23.4MiB (24.6MB), run=18541-18851msec WRITE: bw=1334KiB/s (1366kB/s), 333KiB/s-340KiB/s (341kB/s-348kB/s), io=24.6MiB (25.7MB), run=18541-18851msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=620: Thu May 1 13:32:31 2025 read: IOPS=107, BW=1722KiB/s (1763kB/s)(5920KiB/3438msec) clat (usec): min=66, max=20799, avg=4817.21, stdev=2781.84 lat (usec): min=66, max=20800, avg=4817.80, stdev=2781.87 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 91], 10.00th=[ 2114], 20.00th=[ 2737], | 30.00th=[ 3195], 40.00th=[ 3949], 50.00th=[ 4424], 60.00th=[ 5014], | 70.00th=[ 5997], 80.00th=[ 6915], 90.00th=[ 8455], 95.00th=[ 9372], | 99.00th=[12387], 99.50th=[14353], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 926, max= 2112, per=23.42%, avg=1631.67, stdev=406.98, samples=6 iops : min= 57, max= 132, avg=101.83, stdev=25.74, samples=6 write: IOPS=115, BW=1852KiB/s (1897kB/s)(6368KiB/3438msec) clat (usec): min=669, max=15574, avg=3983.32, stdev=3069.86 lat (usec): min=670, max=15575, avg=3984.24, stdev=3069.83 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 799], 10.00th=[ 832], 20.00th=[ 922], | 30.00th=[ 1270], 40.00th=[ 2606], 50.00th=[ 3425], 60.00th=[ 4178], | 70.00th=[ 5211], 80.00th=[ 6849], 90.00th=[ 8586], 95.00th=[ 9765], | 99.00th=[12780], 99.50th=[15008], 99.90th=[15533], 99.95th=[15533], | 99.99th=[15533] bw ( KiB/s): min= 1053, max= 2144, per=24.66%, avg=1807.50, stdev=386.63, samples=6 iops : min= 65, max= 134, avg=112.83, stdev=24.48, samples=6 lat (usec) : 100=2.60%, 250=1.43%, 500=0.26%, 750=0.91%, 1000=11.07% lat (msec) : 2=6.51%, 4=27.21%, 10=46.22%, 20=3.65%, 50=0.13% cpu : usr=0.26%, sys=23.95%, ctx=1493, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=622: Thu May 1 13:32:31 2025 read: IOPS=113, BW=1820KiB/s (1864kB/s)(5744KiB/3156msec) clat (usec): min=69, max=21198, avg=4524.75, stdev=2696.94 lat (usec): min=69, max=21199, avg=4525.37, stdev=2696.94 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 85], 10.00th=[ 145], 20.00th=[ 2638], | 30.00th=[ 3097], 40.00th=[ 3621], 50.00th=[ 4178], 60.00th=[ 4817], | 70.00th=[ 5604], 80.00th=[ 6718], 90.00th=[ 7832], 95.00th=[ 9372], | 99.00th=[12256], 99.50th=[13435], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 1504, max= 2112, per=26.40%, avg=1839.33, stdev=230.04, samples=6 iops : min= 94, max= 132, avg=114.83, stdev=14.46, samples=6 write: IOPS=129, BW=2074KiB/s (2123kB/s)(6544KiB/3156msec) clat (usec): min=661, max=18722, avg=3708.41, stdev=3118.33 lat (usec): min=662, max=18723, avg=3713.45, stdev=3117.54 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 807], 10.00th=[ 840], 20.00th=[ 922], | 30.00th=[ 1123], 40.00th=[ 1958], 50.00th=[ 2868], 60.00th=[ 3556], | 70.00th=[ 4883], 80.00th=[ 6325], 90.00th=[ 7832], 95.00th=[10159], | 99.00th=[14091], 99.50th=[14877], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 1568, max= 2496, per=28.22%, avg=2068.50, stdev=345.97, samples=6 iops : min= 98, max= 156, avg=129.17, stdev=21.51, samples=6 lat (usec) : 100=3.65%, 250=1.30%, 750=0.78%, 1000=11.72% lat (msec) : 2=8.85%, 4=29.04%, 10=40.49%, 20=4.04%, 50=0.13% cpu : usr=0.41%, sys=26.02%, ctx=1418, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=624: Thu May 1 13:32:31 2025 read: IOPS=120, BW=1921KiB/s (1967kB/s)(6256KiB/3256msec) clat (usec): min=60, max=17089, avg=4619.22, stdev=2826.05 lat (usec): min=61, max=17090, avg=4624.74, stdev=2825.61 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 77], 10.00th=[ 114], 20.00th=[ 2802], | 30.00th=[ 3261], 40.00th=[ 3752], 50.00th=[ 4293], 60.00th=[ 5014], | 70.00th=[ 5604], 80.00th=[ 6718], 90.00th=[ 7963], 95.00th=[ 9241], | 99.00th=[14484], 99.50th=[16057], 99.90th=[17171], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 1280, max= 2400, per=27.64%, avg=1925.33, stdev=371.76, samples=6 iops : min= 80, max= 150, avg=120.33, stdev=23.24, samples=6 write: IOPS=115, BW=1853KiB/s (1897kB/s)(6032KiB/3256msec) clat (usec): min=654, max=19341, avg=3707.07, stdev=3101.93 lat (usec): min=655, max=19343, avg=3708.95, stdev=3101.96 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 799], 10.00th=[ 848], 20.00th=[ 971], | 30.00th=[ 1123], 40.00th=[ 2057], 50.00th=[ 2933], 60.00th=[ 3621], | 70.00th=[ 4752], 80.00th=[ 6128], 90.00th=[ 7898], 95.00th=[ 9765], | 99.00th=[14353], 99.50th=[15139], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 1440, max= 2176, per=24.74%, avg=1813.33, stdev=269.00, samples=6 iops : min= 90, max= 136, avg=113.33, stdev=16.81, samples=6 lat (usec) : 100=4.69%, 250=1.30%, 500=0.13%, 750=1.69%, 1000=9.90% lat (msec) : 2=8.07%, 4=28.65%, 10=41.93%, 20=3.65% cpu : usr=0.15%, sys=25.04%, ctx=1509, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=626: Thu May 1 13:32:31 2025 read: IOPS=116, BW=1861KiB/s (1905kB/s)(6032KiB/3242msec) clat (usec): min=57, max=17835, avg=4441.18, stdev=2889.57 lat (usec): min=57, max=17835, avg=4443.18, stdev=2890.22 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 74], 10.00th=[ 87], 20.00th=[ 2507], | 30.00th=[ 3097], 40.00th=[ 3589], 50.00th=[ 4228], 60.00th=[ 4817], | 70.00th=[ 5342], 80.00th=[ 6390], 90.00th=[ 8225], 95.00th=[ 9634], | 99.00th=[12911], 99.50th=[14615], 99.90th=[17957], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 1312, max= 2363, per=26.48%, avg=1844.50, stdev=359.84, samples=6 iops : min= 82, max= 147, avg=115.17, stdev=22.29, samples=6 write: IOPS=120, BW=1930KiB/s (1976kB/s)(6256KiB/3242msec) clat (usec): min=648, max=27783, avg=3780.02, stdev=3425.85 lat (usec): min=649, max=27784, avg=3781.70, stdev=3425.47 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 766], 10.00th=[ 816], 20.00th=[ 889], | 30.00th=[ 1057], 40.00th=[ 1795], 50.00th=[ 2638], 60.00th=[ 3589], | 70.00th=[ 4883], 80.00th=[ 6652], 90.00th=[ 8717], 95.00th=[10159], | 99.00th=[13960], 99.50th=[16188], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 1632, max= 2240, per=26.55%, avg=1945.83, stdev=267.32, samples=6 iops : min= 102, max= 140, avg=121.50, stdev=16.61, samples=6 lat (usec) : 100=6.12%, 250=0.78%, 750=1.82%, 1000=12.24% lat (msec) : 2=7.94%, 4=26.30%, 10=40.23%, 20=4.43%, 50=0.13% cpu : usr=0.22%, sys=24.44%, ctx=1470, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6967KiB/s (7134kB/s), 1722KiB/s-1921KiB/s (1763kB/s-1967kB/s), io=23.4MiB (24.5MB), run=3156-3438msec WRITE: bw=7330KiB/s (7506kB/s), 1852KiB/s-2074KiB/s (1897kB/s-2123kB/s), io=24.6MiB (25.8MB), run=3156-3438msec rand-rw: (groupid=0, jobs=1): err= 0: pid=619: Thu May 1 13:32:32 2025 read: IOPS=88, BW=1419KiB/s (1453kB/s)(5920KiB/4172msec) clat (usec): min=1776, max=18849, avg=4469.18, stdev=3085.76 lat (usec): min=1777, max=18849, avg=4470.07, stdev=3085.60 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2089], 10.00th=[ 2180], 20.00th=[ 2376], | 30.00th=[ 2573], 40.00th=[ 2769], 50.00th=[ 3130], 60.00th=[ 3621], | 70.00th=[ 4686], 80.00th=[ 5997], 90.00th=[ 9110], 95.00th=[11863], | 99.00th=[16319], 99.50th=[16909], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 926, max= 2304, per=25.20%, avg=1391.50, stdev=467.75, samples=8 iops : min= 57, max= 144, avg=86.75, stdev=29.48, samples=8 write: IOPS=95, BW=1526KiB/s (1563kB/s)(6368KiB/4172msec) clat (usec): min=2532, max=21186, avg=6282.64, stdev=3352.40 lat (usec): min=2533, max=21188, avg=6283.82, stdev=3352.45 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 3163], 10.00th=[ 3359], 20.00th=[ 3720], | 30.00th=[ 4047], 40.00th=[ 4359], 50.00th=[ 4686], 60.00th=[ 5669], | 70.00th=[ 7177], 80.00th=[ 8848], 90.00th=[11338], 95.00th=[13566], | 99.00th=[16909], 99.50th=[19268], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 1085, max= 2560, per=25.67%, avg=1491.25, stdev=462.87, samples=8 iops : min= 67, max= 160, avg=93.00, stdev=29.05, samples=8 lat (msec) : 2=0.78%, 4=44.79%, 10=42.32%, 20=11.98%, 50=0.13% cpu : usr=0.14%, sys=12.23%, ctx=893, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=621: Thu May 1 13:32:32 2025 read: IOPS=82, BW=1326KiB/s (1358kB/s)(5744KiB/4332msec) clat (usec): min=1708, max=21457, avg=4541.93, stdev=3318.90 lat (usec): min=1708, max=21458, avg=4542.55, stdev=3318.90 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2057], 10.00th=[ 2180], 20.00th=[ 2376], | 30.00th=[ 2573], 40.00th=[ 2802], 50.00th=[ 3228], 60.00th=[ 3687], | 70.00th=[ 4686], 80.00th=[ 5997], 90.00th=[ 9241], 95.00th=[12387], | 99.00th=[16319], 99.50th=[18220], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 832, max= 2272, per=23.11%, avg=1275.75, stdev=461.21, samples=8 iops : min= 52, max= 142, avg=79.62, stdev=28.91, samples=8 write: IOPS=94, BW=1511KiB/s (1547kB/s)(6544KiB/4332msec) clat (usec): min=2475, max=26351, avg=6403.93, stdev=3687.91 lat (usec): min=2477, max=26352, avg=6405.13, stdev=3687.94 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3130], 10.00th=[ 3326], 20.00th=[ 3720], | 30.00th=[ 4047], 40.00th=[ 4359], 50.00th=[ 4817], 60.00th=[ 5604], | 70.00th=[ 7111], 80.00th=[ 9110], 90.00th=[11600], 95.00th=[13960], | 99.00th=[17957], 99.50th=[20055], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 1056, max= 2656, per=24.85%, avg=1443.62, stdev=525.79, samples=8 iops : min= 66, max= 166, avg=90.12, stdev=32.90, samples=8 lat (msec) : 2=1.30%, 4=44.53%, 10=42.71%, 20=10.94%, 50=0.52% cpu : usr=0.25%, sys=11.61%, ctx=888, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=623: Thu May 1 13:32:32 2025 read: IOPS=90, BW=1442KiB/s (1477kB/s)(6256KiB/4338msec) clat (usec): min=1812, max=21319, avg=4542.84, stdev=3388.73 lat (usec): min=1812, max=21320, avg=4543.46, stdev=3388.72 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2057], 10.00th=[ 2212], 20.00th=[ 2409], | 30.00th=[ 2573], 40.00th=[ 2737], 50.00th=[ 3064], 60.00th=[ 3523], | 70.00th=[ 4621], 80.00th=[ 6259], 90.00th=[ 8848], 95.00th=[12387], | 99.00th=[16909], 99.50th=[20317], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 768, max= 2496, per=24.63%, avg=1360.00, stdev=566.78, samples=8 iops : min= 48, max= 156, avg=85.00, stdev=35.42, samples=8 write: IOPS=86, BW=1391KiB/s (1424kB/s)(6032KiB/4338msec) clat (usec): min=2546, max=34060, avg=6685.66, stdev=4126.12 lat (usec): min=2547, max=34061, avg=6686.87, stdev=4126.18 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3195], 10.00th=[ 3392], 20.00th=[ 3654], | 30.00th=[ 3949], 40.00th=[ 4359], 50.00th=[ 4883], 60.00th=[ 6063], | 70.00th=[ 7767], 80.00th=[ 9503], 90.00th=[12518], 95.00th=[14222], | 99.00th=[20317], 99.50th=[25560], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 800, max= 2560, per=22.93%, avg=1332.00, stdev=528.57, samples=8 iops : min= 50, max= 160, avg=83.25, stdev=33.04, samples=8 lat (msec) : 2=1.82%, 4=46.87%, 10=38.41%, 20=12.11%, 50=0.78% cpu : usr=0.14%, sys=11.55%, ctx=905, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=625: Thu May 1 13:32:32 2025 read: IOPS=88, BW=1421KiB/s (1455kB/s)(6032KiB/4246msec) clat (usec): min=1813, max=20297, avg=4732.57, stdev=3461.97 lat (usec): min=1813, max=20297, avg=4733.18, stdev=3461.96 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2040], 10.00th=[ 2114], 20.00th=[ 2343], | 30.00th=[ 2540], 40.00th=[ 2835], 50.00th=[ 3195], 60.00th=[ 3818], | 70.00th=[ 4883], 80.00th=[ 6980], 90.00th=[ 9896], 95.00th=[12780], | 99.00th=[16581], 99.50th=[19006], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 896, max= 2528, per=24.63%, avg=1359.62, stdev=492.80, samples=8 iops : min= 56, max= 158, avg=84.88, stdev=30.81, samples=8 write: IOPS=92, BW=1473KiB/s (1509kB/s)(6256KiB/4246msec) clat (usec): min=2679, max=34075, avg=6095.37, stdev=3470.03 lat (usec): min=2680, max=34087, avg=6096.58, stdev=3470.33 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3195], 10.00th=[ 3359], 20.00th=[ 3654], | 30.00th=[ 4015], 40.00th=[ 4228], 50.00th=[ 4752], 60.00th=[ 5407], | 70.00th=[ 6849], 80.00th=[ 8225], 90.00th=[10683], 95.00th=[12518], | 99.00th=[16712], 99.50th=[23200], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 1056, max= 2560, per=24.64%, avg=1431.62, stdev=488.18, samples=8 iops : min= 66, max= 160, avg=89.38, stdev=30.58, samples=8 lat (msec) : 2=1.56%, 4=44.14%, 10=43.49%, 20=10.29%, 50=0.52% cpu : usr=0.09%, sys=11.87%, ctx=930, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5521KiB/s (5654kB/s), 1326KiB/s-1442KiB/s (1358kB/s-1477kB/s), io=23.4MiB (24.5MB), run=4172-4338msec WRITE: bw=5809KiB/s (5949kB/s), 1391KiB/s-1526KiB/s (1424kB/s-1563kB/s), io=24.6MiB (25.8MB), run=4172-4338msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=636: Thu May 1 13:32:34 2025 read: IOPS=14, BW=14.8MiB/s (15.5MB/s)(7168KiB/473msec) clat (usec): min=445, max=50259, avg=25539.40, stdev=18793.08 lat (usec): min=446, max=50259, avg=25540.46, stdev=18793.15 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 445], 10.00th=[ 445], 20.00th=[ 490], | 30.00th=[26084], 40.00th=[26084], 50.00th=[28705], 60.00th=[35390], | 70.00th=[35390], 80.00th=[37487], 90.00th=[50070], 95.00th=[50070], | 99.00th=[50070], 99.50th=[50070], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] write: IOPS=10, BW=10.6MiB/s (11.1MB/s)(5120KiB/473msec) clat (usec): min=34507, max=89104, avg=54567.19, stdev=21879.24 lat (usec): min=34549, max=89165, avg=54609.55, stdev=21892.84 clat percentiles (usec): | 1.00th=[34341], 5.00th=[34341], 10.00th=[34341], 20.00th=[34341], | 30.00th=[39584], 40.00th=[39584], 50.00th=[47973], 60.00th=[47973], | 70.00th=[61604], 80.00th=[61604], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] lat (usec) : 500=16.67% lat (msec) : 50=58.33%, 100=25.00% cpu : usr=0.00%, sys=47.15%, ctx=1090, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=637: Thu May 1 13:32:34 2025 read: IOPS=3, BW=3272KiB/s (3350kB/s)(2048KiB/626msec) clat (usec): min=40571, max=51142, avg=45857.08, stdev=7474.49 lat (usec): min=40572, max=51145, avg=45858.93, stdev=7475.97 clat percentiles (usec): | 1.00th=[40633], 5.00th=[40633], 10.00th=[40633], 20.00th=[40633], | 30.00th=[40633], 40.00th=[40633], 50.00th=[40633], 60.00th=[51119], | 70.00th=[51119], 80.00th=[51119], 90.00th=[51119], 95.00th=[51119], | 99.00th=[51119], 99.50th=[51119], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 3471, max= 3471, per=10.61%, avg=3471.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=15, BW=15.0MiB/s (16.8MB/s)(10.0MiB/626msec) clat (usec): min=18868, max=96529, avg=50003.39, stdev=22123.10 lat (usec): min=18896, max=96546, avg=50059.78, stdev=22115.89 clat percentiles (usec): | 1.00th=[18744], 5.00th=[18744], 10.00th=[18744], 20.00th=[20579], | 30.00th=[42730], 40.00th=[43254], 50.00th=[49021], 60.00th=[50594], | 70.00th=[61604], 80.00th=[61604], 90.00th=[62129], 95.00th=[96994], | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min=15620, max=15620, per=34.10%, avg=15620.00, stdev= 0.00, samples=1 iops : min= 15, max= 15, avg=15.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=50.00% cpu : usr=0.00%, sys=37.76%, ctx=371, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=638: Thu May 1 13:32:34 2025 read: IOPS=13, BW=13.6MiB/s (14.3MB/s)(6144KiB/441msec) clat (usec): min=552, max=46240, avg=29106.05, stdev=16459.54 lat (usec): min=553, max=46241, avg=29106.77, stdev=16459.69 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 553], 10.00th=[ 553], 20.00th=[25035], | 30.00th=[25035], 40.00th=[28705], 50.00th=[28705], 60.00th=[30016], | 70.00th=[44303], 80.00th=[44303], 90.00th=[46400], 95.00th=[46400], | 99.00th=[46400], 99.50th=[46400], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] write: IOPS=13, BW=13.6MiB/s (14.3MB/s)(6144KiB/441msec) clat (msec): min=21, max=113, avg=43.12, stdev=34.52 lat (msec): min=21, max=113, avg=43.19, stdev=34.54 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 30], | 30.00th=[ 30], 40.00th=[ 30], 50.00th=[ 30], 60.00th=[ 31], | 70.00th=[ 35], 80.00th=[ 35], 90.00th=[ 113], 95.00th=[ 113], | 99.00th=[ 113], 99.50th=[ 113], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] lat (usec) : 750=8.33% lat (msec) : 50=83.33%, 250=8.33% cpu : usr=0.00%, sys=39.55%, ctx=291, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=639: Thu May 1 13:32:34 2025 read: IOPS=10, BW=10.0MiB/s (10.5MB/s)(5120KiB/500msec) clat (usec): min=348, max=34579, avg=12522.92, stdev=16740.63 lat (usec): min=349, max=34581, avg=12523.78, stdev=16740.98 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 351], 10.00th=[ 351], 20.00th=[ 351], | 30.00th=[ 388], 40.00th=[ 388], 50.00th=[ 676], 60.00th=[ 676], | 70.00th=[26608], 80.00th=[26608], 90.00th=[34341], 95.00th=[34341], | 99.00th=[34341], 99.50th=[34341], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] write: IOPS=14, BW=14.0MiB/s (14.7MB/s)(7168KiB/500msec) clat (msec): min=19, max=107, avg=52.35, stdev=33.66 lat (msec): min=19, max=108, avg=52.40, stdev=33.66 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 23], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 37], 60.00th=[ 66], | 70.00th=[ 66], 80.00th=[ 83], 90.00th=[ 108], 95.00th=[ 108], | 99.00th=[ 108], 99.50th=[ 108], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 20=8.33%, 50=41.67%, 100=16.67%, 250=8.33% cpu : usr=0.00%, sys=38.68%, ctx=165, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=31.9MiB/s (33.5MB/s), 3272KiB/s-14.8MiB/s (3350kB/s-15.5MB/s), io=20.0MiB (20.0MB), run=441-626msec WRITE: bw=44.7MiB/s (46.9MB/s), 10.6MiB/s-15.0MiB/s (11.1MB/s-16.8MB/s), io=28.0MiB (29.4MB), run=441-626msec rand-rw: (groupid=0, jobs=1): err= 0: pid=640: Thu May 1 13:32:34 2025 read: IOPS=10, BW=10.7MiB/s (11.2MB/s)(7168KiB/657msec) clat (msec): min=19, max=146, avg=57.43, stdev=42.13 lat (msec): min=19, max=146, avg=57.43, stdev=42.13 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 39], | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 40], 60.00th=[ 51], | 70.00th=[ 51], 80.00th=[ 70], 90.00th=[ 146], 95.00th=[ 146], | 99.00th=[ 146], 99.50th=[ 146], 99.90th=[ 146], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min=10240, max=10240, per=33.70%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7793KiB/s (7980kB/s)(5120KiB/657msec) clat (usec): min=11751, max=62212, avg=40684.31, stdev=22665.87 lat (usec): min=11787, max=62242, avg=40720.62, stdev=22669.08 clat percentiles (usec): | 1.00th=[11731], 5.00th=[11731], 10.00th=[11731], 20.00th=[11731], | 30.00th=[21103], 40.00th=[21103], 50.00th=[51643], 60.00th=[51643], | 70.00th=[56361], 80.00th=[56361], 90.00th=[62129], 95.00th=[62129], | 99.00th=[62129], 99.50th=[62129], 99.90th=[62129], 99.95th=[62129], | 99.99th=[62129] bw ( KiB/s): min= 6144, max= 6144, per=14.44%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=41.67%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=14.18%, ctx=71, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=641: Thu May 1 13:32:34 2025 read: IOPS=2, BW=3039KiB/s (3112kB/s)(2048KiB/674msec) clat (usec): min=18565, max=57947, avg=38256.27, stdev=27847.53 lat (usec): min=18566, max=57948, avg=38257.48, stdev=27847.19 clat percentiles (usec): | 1.00th=[18482], 5.00th=[18482], 10.00th=[18482], 20.00th=[18482], | 30.00th=[18482], 40.00th=[18482], 50.00th=[18482], 60.00th=[57934], | 70.00th=[57934], 80.00th=[57934], 90.00th=[57934], 95.00th=[57934], | 99.00th=[57934], 99.50th=[57934], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 4096, max= 4096, per=13.48%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=14, BW=14.8MiB/s (15.6MB/s)(10.0MiB/674msec) clat (msec): min=19, max=131, avg=54.09, stdev=33.84 lat (msec): min=19, max=131, avg=54.15, stdev=33.83 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 21], | 30.00th=[ 32], 40.00th=[ 43], 50.00th=[ 46], 60.00th=[ 49], | 70.00th=[ 63], 80.00th=[ 63], 90.00th=[ 89], 95.00th=[ 131], | 99.00th=[ 131], 99.50th=[ 131], 99.90th=[ 131], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min=12288, max=12288, per=28.89%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=41.67%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=17.98%, ctx=123, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=642: Thu May 1 13:32:34 2025 read: IOPS=9, BW=9409KiB/s (9635kB/s)(6144KiB/653msec) clat (usec): min=9484, max=66231, avg=30233.59, stdev=21231.79 lat (usec): min=9485, max=66231, avg=30234.36, stdev=21231.83 clat percentiles (usec): | 1.00th=[ 9503], 5.00th=[ 9503], 10.00th=[ 9503], 20.00th=[16057], | 30.00th=[16057], 40.00th=[16909], 50.00th=[16909], 60.00th=[30540], | 70.00th=[42206], 80.00th=[42206], 90.00th=[66323], 95.00th=[66323], | 99.00th=[66323], 99.50th=[66323], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 4096, max= 4096, per=13.48%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9409KiB/s (9635kB/s)(6144KiB/653msec) clat (msec): min=13, max=121, avg=73.56, stdev=44.78 lat (msec): min=13, max=121, avg=73.62, stdev=44.79 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 93], 50.00th=[ 93], 60.00th=[ 95], | 70.00th=[ 99], 80.00th=[ 99], 90.00th=[ 122], 95.00th=[ 122], | 99.00th=[ 122], 99.50th=[ 122], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 8192, max= 8192, per=19.26%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 10=8.33%, 20=25.00%, 50=25.00%, 100=33.33%, 250=8.33% cpu : usr=0.15%, sys=10.28%, ctx=61, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=643: Thu May 1 13:32:34 2025 read: IOPS=8, BW=8312KiB/s (8511kB/s)(5120KiB/616msec) clat (usec): min=10578, max=39479, avg=21592.18, stdev=12224.04 lat (usec): min=10579, max=39480, avg=21593.00, stdev=12224.03 clat percentiles (usec): | 1.00th=[10552], 5.00th=[10552], 10.00th=[10552], 20.00th=[10552], | 30.00th=[13435], 40.00th=[13435], 50.00th=[15664], 60.00th=[15664], | 70.00th=[28967], 80.00th=[28967], 90.00th=[39584], 95.00th=[39584], | 99.00th=[39584], 99.50th=[39584], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 8192, max= 8192, per=26.96%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.4MiB/s (11.9MB/s)(7168KiB/616msec) clat (msec): min=16, max=193, avg=64.80, stdev=60.21 lat (msec): min=16, max=193, avg=64.86, stdev=60.20 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 27], | 30.00th=[ 30], 40.00th=[ 30], 50.00th=[ 57], 60.00th=[ 63], | 70.00th=[ 63], 80.00th=[ 70], 90.00th=[ 194], 95.00th=[ 194], | 99.00th=[ 194], 99.50th=[ 194], 99.90th=[ 194], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 8192, max= 8192, per=19.26%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=33.33%, 100=25.00%, 250=8.33% cpu : usr=0.16%, sys=10.57%, ctx=63, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=29.7MiB/s (31.1MB/s), 3039KiB/s-10.7MiB/s (3112kB/s-11.2MB/s), io=20.0MiB (20.0MB), run=616-674msec WRITE: bw=41.5MiB/s (43.6MB/s), 7793KiB/s-14.8MiB/s (7980kB/s-15.6MB/s), io=28.0MiB (29.4MB), run=616-674msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=664: Thu May 1 13:32:36 2025 read: IOPS=8, BW=32.0MiB/s (33.6MB/s)(12.0MiB/375msec) clat (msec): min=73, max=180, avg=121.10, stdev=54.23 lat (msec): min=73, max=180, avg=121.10, stdev=54.23 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 74], 20.00th=[ 74], | 30.00th=[ 74], 40.00th=[ 110], 50.00th=[ 110], 60.00th=[ 110], | 70.00th=[ 180], 80.00th=[ 180], 90.00th=[ 180], 95.00th=[ 180], | 99.00th=[ 180], 99.50th=[ 180], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.27%, sys=22.99%, ctx=1139, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=666: Thu May 1 13:32:36 2025 read: IOPS=1, BW=5596KiB/s (5730kB/s)(4096KiB/732msec) clat (nsec): min=184059k, max=184059k, avg=184059261.00, stdev= 0.00 lat (nsec): min=184060k, max=184060k, avg=184060276.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 184], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] write: IOPS=2, BW=10.9MiB/s (11.5MB/s)(8192KiB/732msec) clat (msec): min=221, max=306, avg=264.06, stdev=59.98 lat (msec): min=221, max=306, avg=264.37, stdev=60.02 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 222], 10.00th=[ 222], 20.00th=[ 222], | 30.00th=[ 222], 40.00th=[ 222], 50.00th=[ 222], 60.00th=[ 309], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=34.20%, ctx=729, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=668: Thu May 1 13:32:36 2025 write: IOPS=5, BW=20.7MiB/s (21.7MB/s)(12.0MiB/581msec) clat (msec): min=88, max=306, avg=187.97, stdev=110.44 lat (msec): min=88, max=306, avg=188.85, stdev=110.29 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 89], 10.00th=[ 89], 20.00th=[ 89], | 30.00th=[ 89], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 169], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.52%, sys=47.76%, ctx=84, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=670: Thu May 1 13:32:36 2025 write: IOPS=6, BW=24.1MiB/s (25.3MB/s)(12.0MiB/497msec) clat (msec): min=132, max=185, avg=161.52, stdev=26.67 lat (msec): min=132, max=185, avg=161.81, stdev=26.69 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 133], | 30.00th=[ 133], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 167], | 70.00th=[ 186], 80.00th=[ 186], 90.00th=[ 186], 95.00th=[ 186], | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=53.02%, ctx=70, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=21.9MiB/s (22.9MB/s), 5596KiB/s-32.0MiB/s (5730kB/s-33.6MB/s), io=16.0MiB (16.8MB), run=375-732msec WRITE: bw=43.7MiB/s (45.8MB/s), 10.9MiB/s-24.1MiB/s (11.5MB/s-25.3MB/s), io=32.0MiB (33.6MB), run=497-732msec rand-rw: (groupid=0, jobs=1): err= 0: pid=663: Thu May 1 13:32:36 2025 read: IOPS=8, BW=35.6MiB/s (37.3MB/s)(12.0MiB/337msec) clat (msec): min=29, max=188, avg=110.26, stdev=79.91 lat (msec): min=29, max=188, avg=110.27, stdev=79.91 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 30], | 30.00th=[ 30], 40.00th=[ 113], 50.00th=[ 113], 60.00th=[ 113], | 70.00th=[ 190], 80.00th=[ 190], 90.00th=[ 190], 95.00th=[ 190], | 99.00th=[ 190], 99.50th=[ 190], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] lat (msec) : 50=33.33%, 250=66.67% cpu : usr=0.00%, sys=13.39%, ctx=39, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=665: Thu May 1 13:32:36 2025 read: IOPS=1, BW=7136KiB/s (7307kB/s)(4096KiB/574msec) clat (nsec): min=69577k, max=69577k, avg=69576730.00, stdev= 0.00 lat (nsec): min=69578k, max=69578k, avg=69578454.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[69731], 5.00th=[69731], 10.00th=[69731], 20.00th=[69731], | 30.00th=[69731], 40.00th=[69731], 50.00th=[69731], 60.00th=[69731], | 70.00th=[69731], 80.00th=[69731], 90.00th=[69731], 95.00th=[69731], | 99.00th=[69731], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 8192, max= 8192, per=28.70%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=13.9MiB/s (14.6MB/s)(8192KiB/574msec) clat (msec): min=172, max=300, avg=236.56, stdev=90.44 lat (msec): min=172, max=300, avg=236.80, stdev=90.46 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 174], 50.00th=[ 174], 60.00th=[ 300], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 8192, max= 8192, per=18.48%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.17%, sys=18.50%, ctx=123, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=667: Thu May 1 13:32:36 2025 write: IOPS=4, BW=16.2MiB/s (17.0MB/s)(12.0MiB/739msec) clat (msec): min=111, max=362, avg=240.19, stdev=125.42 lat (msec): min=112, max=363, avg=240.49, stdev=125.43 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 247], 50.00th=[ 247], 60.00th=[ 247], | 70.00th=[ 363], 80.00th=[ 363], 90.00th=[ 363], 95.00th=[ 363], | 99.00th=[ 363], 99.50th=[ 363], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 8192, max= 8192, per=18.48%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.27%, sys=16.67%, ctx=182, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=669: Thu May 1 13:32:36 2025 write: IOPS=4, BW=18.2MiB/s (19.1MB/s)(12.0MiB/658msec) clat (msec): min=101, max=278, avg=207.21, stdev=93.60 lat (msec): min=101, max=278, avg=207.51, stdev=93.57 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 102], 10.00th=[ 102], 20.00th=[ 102], | 30.00th=[ 102], 40.00th=[ 243], 50.00th=[ 243], 60.00th=[ 243], | 70.00th=[ 279], 80.00th=[ 279], 90.00th=[ 279], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min=16384, max=16384, per=36.95%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.30%, sys=16.13%, ctx=260, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=27.9MiB/s (29.2MB/s), 7136KiB/s-35.6MiB/s (7307kB/s-37.3MB/s), io=16.0MiB (16.8MB), run=337-574msec WRITE: bw=43.3MiB/s (45.4MB/s), 13.9MiB/s-18.2MiB/s (14.6MB/s-19.1MB/s), io=32.0MiB (33.6MB), run=574-739msec