== sanity test 398b: DIO and buffer IO race ============== 09:04:54 (1748869494) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.20052 s, 41.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8382: Mon Jun 2 09:05:10 2025 read: IOPS=111, BW=446KiB/s (457kB/s)(5992KiB/13428msec) clat (usec): min=62, max=202841, avg=4949.54, stdev=5733.75 lat (usec): min=63, max=202842, avg=4950.52, stdev=5733.87 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 1713], 10.00th=[ 2376], 20.00th=[ 2802], | 30.00th=[ 3326], 40.00th=[ 3884], 50.00th=[ 4424], 60.00th=[ 5080], | 70.00th=[ 5800], 80.00th=[ 6587], 90.00th=[ 7963], 95.00th=[ 9372], | 99.00th=[ 13566], 99.50th=[ 15664], 99.90th=[ 25822], 99.95th=[202376], | 99.99th=[202376] bw ( KiB/s): min= 232, max= 582, per=24.60%, avg=439.04, stdev=70.50, samples=26 iops : min= 58, max= 145, avg=109.58, stdev=17.59, samples=26 write: IOPS=117, BW=469KiB/s (480kB/s)(6296KiB/13428msec) clat (usec): min=473, max=24684, avg=3781.21, stdev=3268.20 lat (usec): min=474, max=24685, avg=3782.60, stdev=3268.77 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 578], 10.00th=[ 635], 20.00th=[ 799], | 30.00th=[ 1090], 40.00th=[ 2442], 50.00th=[ 3163], 60.00th=[ 3785], | 70.00th=[ 4752], 80.00th=[ 6259], 90.00th=[ 8094], 95.00th=[ 9896], | 99.00th=[13566], 99.50th=[17433], 99.90th=[23462], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 256, max= 624, per=24.58%, avg=459.38, stdev=69.27, samples=26 iops : min= 64, max= 156, avg=114.65, stdev=17.33, samples=26 lat (usec) : 100=1.17%, 250=1.20%, 500=0.03%, 750=8.59%, 1000=5.76% lat (msec) : 2=4.69%, 4=31.25%, 10=43.46%, 20=3.55%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=23.02%, ctx=5907, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8383: Mon Jun 2 09:05:10 2025 read: IOPS=111, BW=447KiB/s (458kB/s)(5940KiB/13279msec) clat (usec): min=54, max=140908, avg=4942.33, stdev=4438.94 lat (usec): min=54, max=140909, avg=4943.81, stdev=4438.72 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 2040], 10.00th=[ 2409], 20.00th=[ 2868], | 30.00th=[ 3261], 40.00th=[ 3752], 50.00th=[ 4424], 60.00th=[ 5080], | 70.00th=[ 5800], 80.00th=[ 6652], 90.00th=[ 7963], 95.00th=[ 9241], | 99.00th=[ 15401], 99.50th=[ 18220], 99.90th=[ 25035], 99.95th=[141558], | 99.99th=[141558] bw ( KiB/s): min= 160, max= 560, per=24.91%, avg=444.73, stdev=71.27, samples=26 iops : min= 40, max= 140, avg=111.04, stdev=17.79, samples=26 write: IOPS=119, BW=478KiB/s (490kB/s)(6348KiB/13279msec) clat (usec): min=492, max=47613, avg=3693.07, stdev=3405.04 lat (usec): min=493, max=47613, avg=3693.90, stdev=3405.12 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 570], 10.00th=[ 619], 20.00th=[ 766], | 30.00th=[ 947], 40.00th=[ 2147], 50.00th=[ 3032], 60.00th=[ 3720], | 70.00th=[ 4752], 80.00th=[ 5997], 90.00th=[ 8160], 95.00th=[ 9896], | 99.00th=[14091], 99.50th=[15401], 99.90th=[25297], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 224, max= 584, per=25.11%, avg=469.23, stdev=76.74, samples=26 iops : min= 56, max= 146, avg=117.19, stdev=19.16, samples=26 lat (usec) : 100=1.20%, 250=0.78%, 500=0.13%, 750=9.83%, 1000=6.54% lat (msec) : 2=4.07%, 4=31.09%, 10=41.96%, 20=4.20%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=22.97%, ctx=5598, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8384: Mon Jun 2 09:05:10 2025 read: IOPS=112, BW=450KiB/s (461kB/s)(6028KiB/13403msec) clat (usec): min=61, max=25910, avg=4875.67, stdev=2881.09 lat (usec): min=61, max=25911, avg=4876.61, stdev=2880.73 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 123], 10.00th=[ 2278], 20.00th=[ 2802], | 30.00th=[ 3261], 40.00th=[ 3785], 50.00th=[ 4490], 60.00th=[ 5211], | 70.00th=[ 5932], 80.00th=[ 6783], 90.00th=[ 7898], 95.00th=[ 9372], | 99.00th=[15139], 99.50th=[18220], 99.90th=[25560], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 248, max= 544, per=24.91%, avg=444.58, stdev=70.93, samples=26 iops : min= 62, max= 136, avg=111.12, stdev=17.77, samples=26 write: IOPS=116, BW=467KiB/s (478kB/s)(6260KiB/13403msec) clat (usec): min=509, max=202290, avg=3826.31, stdev=5972.70 lat (usec): min=509, max=202291, avg=3828.28, stdev=5972.69 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 578], 10.00th=[ 644], 20.00th=[ 799], | 30.00th=[ 971], 40.00th=[ 2057], 50.00th=[ 2966], 60.00th=[ 3621], | 70.00th=[ 4686], 80.00th=[ 6194], 90.00th=[ 8455], 95.00th=[ 10290], | 99.00th=[ 13435], 99.50th=[ 14484], 99.90th=[ 19792], 99.95th=[202376], | 99.99th=[202376] bw ( KiB/s): min= 320, max= 584, per=24.56%, avg=459.04, stdev=66.51, samples=26 iops : min= 80, max= 146, avg=114.73, stdev=16.63, samples=26 lat (usec) : 100=1.66%, 250=1.27%, 500=0.03%, 750=8.98%, 1000=6.80% lat (msec) : 2=4.95%, 4=30.31%, 10=41.24%, 20=4.52%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=22.74%, ctx=5765, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8385: Mon Jun 2 09:05:10 2025 read: IOPS=112, BW=450KiB/s (461kB/s)(6052KiB/13451msec) clat (usec): min=58, max=23076, avg=4879.37, stdev=2730.56 lat (usec): min=58, max=23077, avg=4880.32, stdev=2730.60 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 594], 10.00th=[ 2311], 20.00th=[ 2835], | 30.00th=[ 3261], 40.00th=[ 3785], 50.00th=[ 4424], 60.00th=[ 5014], | 70.00th=[ 5800], 80.00th=[ 6652], 90.00th=[ 8291], 95.00th=[ 9896], | 99.00th=[13960], 99.50th=[15664], 99.90th=[17957], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 224, max= 544, per=24.85%, avg=443.65, stdev=67.92, samples=26 iops : min= 56, max= 136, avg=110.73, stdev=16.99, samples=26 write: IOPS=115, BW=464KiB/s (475kB/s)(6236KiB/13451msec) clat (usec): min=423, max=147025, avg=3841.46, stdev=5034.24 lat (usec): min=423, max=147027, avg=3842.42, stdev=5034.29 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 578], 10.00th=[ 619], 20.00th=[ 775], | 30.00th=[ 963], 40.00th=[ 2147], 50.00th=[ 3032], 60.00th=[ 3785], | 70.00th=[ 4752], 80.00th=[ 6325], 90.00th=[ 8356], 95.00th=[ 9896], | 99.00th=[ 14222], 99.50th=[ 16712], 99.90th=[ 46400], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 256, max= 632, per=24.56%, avg=459.04, stdev=80.57, samples=26 iops : min= 64, max= 158, avg=114.58, stdev=20.22, samples=26 lat (usec) : 100=1.50%, 250=0.88%, 500=0.23%, 750=9.47%, 1000=6.25% lat (msec) : 2=4.23%, 4=30.47%, 10=42.32%, 20=4.43%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=22.41%, ctx=5806, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1785KiB/s (1828kB/s), 446KiB/s-450KiB/s (457kB/s-461kB/s), io=23.4MiB (24.6MB), run=13279-13451msec WRITE: bw=1869KiB/s (1914kB/s), 464KiB/s-478KiB/s (475kB/s-490kB/s), io=24.6MiB (25.7MB), run=13279-13451msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8378: Mon Jun 2 09:05:14 2025 read: IOPS=86, BW=348KiB/s (356kB/s)(5992KiB/17239msec) clat (usec): min=1639, max=203927, avg=4728.26, stdev=6255.69 lat (usec): min=1640, max=203929, avg=4728.99, stdev=6255.73 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1909], 10.00th=[ 2008], 20.00th=[ 2212], | 30.00th=[ 2409], 40.00th=[ 2671], 50.00th=[ 3064], 60.00th=[ 3654], | 70.00th=[ 4555], 80.00th=[ 6652], 90.00th=[ 10028], 95.00th=[ 12387], | 99.00th=[ 17433], 99.50th=[ 19530], 99.90th=[ 27919], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 152, max= 760, per=24.76%, avg=344.44, stdev=171.80, samples=34 iops : min= 38, max= 190, avg=86.09, stdev=42.96, samples=34 write: IOPS=91, BW=365KiB/s (374kB/s)(6296KiB/17239msec) clat (usec): min=2054, max=36711, avg=6413.53, stdev=4021.45 lat (usec): min=2055, max=36711, avg=6414.37, stdev=4021.47 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2900], 10.00th=[ 3097], 20.00th=[ 3359], | 30.00th=[ 3621], 40.00th=[ 4080], 50.00th=[ 4883], 60.00th=[ 5997], | 70.00th=[ 7570], 80.00th=[ 9372], 90.00th=[11731], 95.00th=[13960], | 99.00th=[20841], 99.50th=[25035], 99.90th=[29754], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 160, max= 784, per=24.84%, avg=361.85, stdev=177.28, samples=34 iops : min= 40, max= 196, avg=90.44, stdev=44.34, samples=34 lat (msec) : 2=4.62%, 4=47.07%, 10=34.90%, 20=12.63%, 50=0.75% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=11.50%, ctx=3692, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8379: Mon Jun 2 09:05:14 2025 read: IOPS=86, BW=344KiB/s (353kB/s)(5940KiB/17254msec) clat (usec): min=1523, max=155715, avg=4528.89, stdev=5164.49 lat (usec): min=1524, max=155716, avg=4529.57, stdev=5164.49 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1926], 10.00th=[ 2040], 20.00th=[ 2212], | 30.00th=[ 2376], 40.00th=[ 2671], 50.00th=[ 3130], 60.00th=[ 3556], | 70.00th=[ 4424], 80.00th=[ 5932], 90.00th=[ 9241], 95.00th=[ 12387], | 99.00th=[ 17171], 99.50th=[ 17695], 99.90th=[ 23462], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 104, max= 736, per=24.64%, avg=342.79, stdev=177.75, samples=34 iops : min= 26, max= 184, avg=85.68, stdev=44.44, samples=34 write: IOPS=91, BW=368KiB/s (377kB/s)(6348KiB/17254msec) clat (usec): min=2266, max=46986, avg=6593.76, stdev=4196.06 lat (usec): min=2266, max=46987, avg=6594.65, stdev=4196.08 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2900], 10.00th=[ 3130], 20.00th=[ 3359], | 30.00th=[ 3720], 40.00th=[ 4228], 50.00th=[ 4948], 60.00th=[ 6128], | 70.00th=[ 7767], 80.00th=[ 9634], 90.00th=[12256], 95.00th=[14746], | 99.00th=[20579], 99.50th=[23725], 99.90th=[36439], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 168, max= 720, per=24.93%, avg=363.26, stdev=168.12, samples=34 iops : min= 42, max= 180, avg=90.79, stdev=42.04, samples=34 lat (msec) : 2=3.71%, 4=47.20%, 10=35.51%, 20=12.86%, 50=0.68% lat (msec) : 250=0.03% cpu : usr=0.10%, sys=11.28%, ctx=3707, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8380: Mon Jun 2 09:05:14 2025 read: IOPS=87, BW=350KiB/s (358kB/s)(6028KiB/17247msec) clat (usec): min=1643, max=29163, avg=4616.11, stdev=3711.06 lat (usec): min=1644, max=29163, avg=4616.83, stdev=3711.04 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1909], 10.00th=[ 2024], 20.00th=[ 2180], | 30.00th=[ 2376], 40.00th=[ 2638], 50.00th=[ 2999], 60.00th=[ 3523], | 70.00th=[ 4490], 80.00th=[ 6456], 90.00th=[10552], 95.00th=[12649], | 99.00th=[17695], 99.50th=[20841], 99.90th=[27657], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 120, max= 768, per=24.91%, avg=346.56, stdev=171.76, samples=34 iops : min= 30, max= 192, avg=86.62, stdev=42.96, samples=34 write: IOPS=90, BW=363KiB/s (372kB/s)(6260KiB/17247msec) clat (msec): min=2, max=160, avg= 6.54, stdev= 5.63 lat (msec): min=2, max=160, avg= 6.54, stdev= 5.63 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 20], 99.50th=[ 23], 99.90th=[ 45], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 176, max= 768, per=24.66%, avg=359.26, stdev=180.57, samples=34 iops : min= 44, max= 192, avg=89.79, stdev=45.16, samples=34 lat (msec) : 2=4.65%, 4=47.56%, 10=33.33%, 20=13.80%, 50=0.62% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=11.21%, ctx=3748, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8381: Mon Jun 2 09:05:14 2025 read: IOPS=87, BW=352KiB/s (360kB/s)(6052KiB/17207msec) clat (usec): min=1559, max=23154, avg=4556.95, stdev=3483.44 lat (usec): min=1559, max=23154, avg=4557.60, stdev=3483.45 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1942], 10.00th=[ 2057], 20.00th=[ 2212], | 30.00th=[ 2376], 40.00th=[ 2704], 50.00th=[ 3097], 60.00th=[ 3654], | 70.00th=[ 4424], 80.00th=[ 6390], 90.00th=[10159], 95.00th=[12256], | 99.00th=[16909], 99.50th=[18744], 99.90th=[22152], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 136, max= 720, per=25.05%, avg=348.47, stdev=168.52, samples=34 iops : min= 34, max= 180, avg=87.12, stdev=42.13, samples=34 write: IOPS=90, BW=362KiB/s (371kB/s)(6236KiB/17207msec) clat (msec): min=2, max=149, avg= 6.57, stdev= 5.53 lat (msec): min=2, max=149, avg= 6.57, stdev= 5.53 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 22], 99.50th=[ 27], 99.90th=[ 46], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 160, max= 720, per=24.66%, avg=359.29, stdev=170.55, samples=34 iops : min= 40, max= 180, avg=89.82, stdev=42.64, samples=34 lat (msec) : 2=3.61%, 4=47.66%, 10=34.70%, 20=13.28%, 50=0.72% lat (msec) : 250=0.03% cpu : usr=0.12%, sys=11.34%, ctx=3727, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1392KiB/s (1425kB/s), 344KiB/s-352KiB/s (353kB/s-360kB/s), io=23.4MiB (24.6MB), run=17207-17254msec WRITE: bw=1457KiB/s (1492kB/s), 362KiB/s-368KiB/s (371kB/s-377kB/s), io=24.6MiB (25.7MB), run=17207-17254msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8415: Mon Jun 2 09:05:18 2025 read: IOPS=117, BW=1875KiB/s (1920kB/s)(5920KiB/3157msec) clat (usec): min=61, max=15420, avg=4602.45, stdev=2927.38 lat (usec): min=61, max=15421, avg=4607.47, stdev=2921.96 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 93], 10.00th=[ 126], 20.00th=[ 2638], | 30.00th=[ 3097], 40.00th=[ 3589], 50.00th=[ 4080], 60.00th=[ 4752], | 70.00th=[ 5604], 80.00th=[ 6652], 90.00th=[ 8455], 95.00th=[10028], | 99.00th=[13960], 99.50th=[15270], 99.90th=[15401], 99.95th=[15401], | 99.99th=[15401] bw ( KiB/s): min= 1536, max= 2112, per=26.16%, avg=1849.17, stdev=258.22, samples=6 iops : min= 96, max= 132, avg=115.33, stdev=15.95, samples=6 write: IOPS=126, BW=2017KiB/s (2066kB/s)(6368KiB/3157msec) clat (usec): min=637, max=25233, avg=3590.71, stdev=3215.71 lat (usec): min=638, max=25233, avg=3592.90, stdev=3214.82 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 807], 10.00th=[ 873], 20.00th=[ 1037], | 30.00th=[ 1237], 40.00th=[ 1614], 50.00th=[ 2540], 60.00th=[ 3294], | 70.00th=[ 4178], 80.00th=[ 6194], 90.00th=[ 8356], 95.00th=[ 9372], | 99.00th=[13960], 99.50th=[18744], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 1568, max= 2176, per=26.65%, avg=1982.33, stdev=212.49, samples=6 iops : min= 98, max= 136, avg=123.67, stdev=13.19, samples=6 lat (usec) : 100=2.86%, 250=2.60%, 500=0.13%, 750=0.78%, 1000=8.59% lat (msec) : 2=14.97%, 4=28.65%, 10=37.24%, 20=4.04%, 50=0.13% cpu : usr=0.29%, sys=26.90%, ctx=1666, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8416: Mon Jun 2 09:05:18 2025 read: IOPS=107, BW=1727KiB/s (1768kB/s)(5744KiB/3326msec) clat (usec): min=70, max=18572, avg=4785.56, stdev=2790.25 lat (usec): min=71, max=18572, avg=4786.18, stdev=2790.26 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 103], 10.00th=[ 2114], 20.00th=[ 2737], | 30.00th=[ 3359], 40.00th=[ 3916], 50.00th=[ 4424], 60.00th=[ 5014], | 70.00th=[ 5800], 80.00th=[ 6652], 90.00th=[ 8291], 95.00th=[ 9241], | 99.00th=[13829], 99.50th=[15008], 99.90th=[18482], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 1280, max= 1952, per=24.07%, avg=1701.33, stdev=250.68, samples=6 iops : min= 80, max= 122, avg=106.33, stdev=15.67, samples=6 write: IOPS=122, BW=1968KiB/s (2015kB/s)(6544KiB/3326msec) clat (usec): min=729, max=17446, avg=3899.35, stdev=2982.73 lat (usec): min=730, max=17447, avg=3900.27, stdev=2982.74 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 840], 10.00th=[ 914], 20.00th=[ 1139], | 30.00th=[ 1483], 40.00th=[ 2573], 50.00th=[ 3163], 60.00th=[ 3916], | 70.00th=[ 4948], 80.00th=[ 6390], 90.00th=[ 8356], 95.00th=[ 9372], | 99.00th=[12518], 99.50th=[14877], 99.90th=[17433], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 1632, max= 2368, per=25.81%, avg=1920.00, stdev=310.25, samples=6 iops : min= 102, max= 148, avg=120.00, stdev=19.39, samples=6 lat (usec) : 100=2.21%, 250=1.95%, 750=0.39%, 1000=6.90% lat (msec) : 2=12.11%, 4=28.26%, 10=44.40%, 20=3.78% cpu : usr=0.21%, sys=26.59%, ctx=1673, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8417: Mon Jun 2 09:05:18 2025 read: IOPS=115, BW=1847KiB/s (1891kB/s)(6256KiB/3388msec) clat (usec): min=62, max=19122, avg=4950.01, stdev=2882.29 lat (usec): min=62, max=19122, avg=4953.23, stdev=2879.55 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 83], 10.00th=[ 1729], 20.00th=[ 2835], | 30.00th=[ 3359], 40.00th=[ 4015], 50.00th=[ 4686], 60.00th=[ 5538], | 70.00th=[ 6128], 80.00th=[ 7111], 90.00th=[ 8225], 95.00th=[ 9765], | 99.00th=[13698], 99.50th=[14877], 99.90th=[19006], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 1600, max= 2043, per=25.18%, avg=1779.83, stdev=193.68, samples=6 iops : min= 100, max= 127, avg=111.00, stdev=11.76, samples=6 write: IOPS=111, BW=1780KiB/s (1823kB/s)(6032KiB/3388msec) clat (usec): min=717, max=19973, avg=3800.09, stdev=3200.35 lat (usec): min=718, max=19974, avg=3804.97, stdev=3202.24 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 832], 10.00th=[ 881], 20.00th=[ 1057], | 30.00th=[ 1369], 40.00th=[ 2147], 50.00th=[ 2933], 60.00th=[ 3752], | 70.00th=[ 4686], 80.00th=[ 6325], 90.00th=[ 8094], 95.00th=[10683], | 99.00th=[13566], 99.50th=[15795], 99.90th=[20055], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 1376, max= 2075, per=23.64%, avg=1758.50, stdev=273.09, samples=6 iops : min= 86, max= 129, avg=109.67, stdev=17.00, samples=6 lat (usec) : 100=3.39%, 250=1.56%, 500=0.13%, 750=0.65%, 1000=7.55% lat (msec) : 2=11.33%, 4=27.34%, 10=42.71%, 20=5.34% cpu : usr=0.15%, sys=25.33%, ctx=1760, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8418: Mon Jun 2 09:05:18 2025 read: IOPS=119, BW=1919KiB/s (1965kB/s)(6032KiB/3144msec) clat (usec): min=60, max=15617, avg=4488.42, stdev=2782.32 lat (usec): min=60, max=15618, avg=4489.00, stdev=2782.33 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 79], 10.00th=[ 96], 20.00th=[ 2474], | 30.00th=[ 3163], 40.00th=[ 3589], 50.00th=[ 4293], 60.00th=[ 4883], | 70.00th=[ 5800], 80.00th=[ 6587], 90.00th=[ 7963], 95.00th=[ 9503], | 99.00th=[12387], 99.50th=[13960], 99.90th=[15664], 99.95th=[15664], | 99.99th=[15664] bw ( KiB/s): min= 1568, max= 2400, per=26.85%, avg=1898.00, stdev=295.43, samples=6 iops : min= 98, max= 150, avg=118.50, stdev=18.52, samples=6 write: IOPS=124, BW=1990KiB/s (2038kB/s)(6256KiB/3144msec) clat (usec): min=642, max=26105, avg=3624.17, stdev=3209.06 lat (usec): min=642, max=26106, avg=3625.08, stdev=3209.07 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 807], 10.00th=[ 881], 20.00th=[ 1037], | 30.00th=[ 1287], 40.00th=[ 1696], 50.00th=[ 2573], 60.00th=[ 3458], | 70.00th=[ 4555], 80.00th=[ 5735], 90.00th=[ 8094], 95.00th=[ 9896], | 99.00th=[13566], 99.50th=[15664], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 1632, max= 2363, per=26.73%, avg=1988.50, stdev=257.26, samples=6 iops : min= 102, max= 147, avg=124.17, stdev=15.88, samples=6 lat (usec) : 100=5.21%, 250=1.43%, 500=0.13%, 750=0.91%, 1000=7.94% lat (msec) : 2=14.32%, 4=25.78%, 10=40.10%, 20=4.04%, 50=0.13% cpu : usr=0.10%, sys=26.03%, ctx=1779, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7070KiB/s (7239kB/s), 1727KiB/s-1919KiB/s (1768kB/s-1965kB/s), io=23.4MiB (24.5MB), run=3144-3388msec WRITE: bw=7438KiB/s (7617kB/s), 1780KiB/s-2017KiB/s (1823kB/s-2066kB/s), io=24.6MiB (25.8MB), run=3144-3388msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8420: Mon Jun 2 09:05:19 2025 read: IOPS=85, BW=1366KiB/s (1399kB/s)(5920KiB/4333msec) clat (usec): min=1885, max=23408, avg=4689.45, stdev=3821.34 lat (usec): min=1885, max=23408, avg=4690.08, stdev=3821.35 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2147], 10.00th=[ 2245], 20.00th=[ 2376], | 30.00th=[ 2540], 40.00th=[ 2704], 50.00th=[ 3032], 60.00th=[ 3556], | 70.00th=[ 4359], 80.00th=[ 6325], 90.00th=[10028], 95.00th=[13698], | 99.00th=[20317], 99.50th=[22938], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 832, max= 2432, per=23.01%, avg=1271.62, stdev=527.28, samples=8 iops : min= 52, max= 152, avg=79.37, stdev=32.92, samples=8 write: IOPS=91, BW=1470KiB/s (1505kB/s)(6368KiB/4333msec) clat (usec): min=2633, max=23261, avg=6488.26, stdev=3816.58 lat (usec): min=2634, max=23262, avg=6489.50, stdev=3816.63 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3326], 10.00th=[ 3523], 20.00th=[ 3752], | 30.00th=[ 4015], 40.00th=[ 4293], 50.00th=[ 4686], 60.00th=[ 5342], | 70.00th=[ 7177], 80.00th=[ 9110], 90.00th=[12387], 95.00th=[14746], | 99.00th=[19530], 99.50th=[22676], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 928, max= 2496, per=23.86%, avg=1387.50, stdev=528.59, samples=8 iops : min= 58, max= 156, avg=86.62, stdev=32.94, samples=8 lat (msec) : 2=0.26%, 4=46.87%, 10=38.80%, 20=13.15%, 50=0.91% cpu : usr=0.14%, sys=12.19%, ctx=958, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8421: Mon Jun 2 09:05:19 2025 read: IOPS=82, BW=1326KiB/s (1357kB/s)(5744KiB/4333msec) clat (usec): min=1909, max=25417, avg=4618.94, stdev=3657.93 lat (usec): min=1909, max=25417, avg=4619.61, stdev=3657.92 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2114], 10.00th=[ 2212], 20.00th=[ 2376], | 30.00th=[ 2573], 40.00th=[ 2802], 50.00th=[ 3130], 60.00th=[ 3654], | 70.00th=[ 4490], 80.00th=[ 5800], 90.00th=[ 9896], 95.00th=[12911], | 99.00th=[17695], 99.50th=[22676], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 832, max= 2240, per=22.51%, avg=1244.00, stdev=473.54, samples=8 iops : min= 52, max= 140, avg=77.75, stdev=29.60, samples=8 write: IOPS=94, BW=1510KiB/s (1547kB/s)(6544KiB/4333msec) clat (usec): min=2888, max=25513, avg=6456.05, stdev=3860.31 lat (usec): min=2889, max=25515, avg=6457.25, stdev=3860.36 clat percentiles (usec): | 1.00th=[ 3097], 5.00th=[ 3294], 10.00th=[ 3458], 20.00th=[ 3654], | 30.00th=[ 3916], 40.00th=[ 4228], 50.00th=[ 4621], 60.00th=[ 5473], | 70.00th=[ 7242], 80.00th=[ 9372], 90.00th=[12256], 95.00th=[13960], | 99.00th=[19530], 99.50th=[23462], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 1088, max= 2624, per=24.14%, avg=1404.00, stdev=528.16, samples=8 iops : min= 68, max= 164, avg=87.75, stdev=33.01, samples=8 lat (msec) : 2=0.65%, 4=47.14%, 10=38.67%, 20=12.63%, 50=0.91% cpu : usr=0.05%, sys=12.60%, ctx=946, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8422: Mon Jun 2 09:05:19 2025 read: IOPS=90, BW=1446KiB/s (1481kB/s)(6256KiB/4325msec) clat (usec): min=1881, max=21356, avg=4362.25, stdev=3248.96 lat (usec): min=1882, max=21357, avg=4362.99, stdev=3249.08 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2147], 10.00th=[ 2212], 20.00th=[ 2343], | 30.00th=[ 2474], 40.00th=[ 2638], 50.00th=[ 3032], 60.00th=[ 3359], | 70.00th=[ 4080], 80.00th=[ 5604], 90.00th=[ 8717], 95.00th=[12518], | 99.00th=[16188], 99.50th=[17171], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 733, max= 2432, per=24.30%, avg=1343.25, stdev=589.17, samples=8 iops : min= 45, max= 152, avg=83.75, stdev=36.99, samples=8 write: IOPS=87, BW=1395KiB/s (1428kB/s)(6032KiB/4325msec) clat (usec): min=2569, max=24264, avg=6857.07, stdev=4287.42 lat (usec): min=2570, max=24266, avg=6858.32, stdev=4287.30 clat percentiles (usec): | 1.00th=[ 2802], 5.00th=[ 3261], 10.00th=[ 3458], 20.00th=[ 3720], | 30.00th=[ 3982], 40.00th=[ 4228], 50.00th=[ 4817], 60.00th=[ 5866], | 70.00th=[ 7767], 80.00th=[10290], 90.00th=[13566], 95.00th=[16319], | 99.00th=[20317], 99.50th=[20841], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 864, max= 2560, per=22.75%, avg=1323.12, stdev=544.93, samples=8 iops : min= 54, max= 160, avg=82.50, stdev=34.18, samples=8 lat (msec) : 2=0.39%, 4=49.48%, 10=35.55%, 20=13.93%, 50=0.65% cpu : usr=0.19%, sys=12.16%, ctx=985, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8423: Mon Jun 2 09:05:19 2025 read: IOPS=88, BW=1411KiB/s (1445kB/s)(6032KiB/4275msec) clat (usec): min=1841, max=22312, avg=4698.99, stdev=3565.45 lat (usec): min=1841, max=22313, avg=4699.81, stdev=3565.42 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2114], 10.00th=[ 2212], 20.00th=[ 2376], | 30.00th=[ 2540], 40.00th=[ 2769], 50.00th=[ 3130], 60.00th=[ 3589], | 70.00th=[ 4621], 80.00th=[ 6718], 90.00th=[10290], 95.00th=[12911], | 99.00th=[17695], 99.50th=[20579], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 896, max= 2336, per=23.95%, avg=1323.50, stdev=477.55, samples=8 iops : min= 56, max= 146, avg=82.62, stdev=29.94, samples=8 write: IOPS=91, BW=1463KiB/s (1499kB/s)(6256KiB/4275msec) clat (usec): min=2541, max=25864, avg=6261.59, stdev=3863.76 lat (usec): min=2542, max=25865, avg=6262.63, stdev=3863.78 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3130], 10.00th=[ 3458], 20.00th=[ 3654], | 30.00th=[ 3884], 40.00th=[ 4228], 50.00th=[ 4686], 60.00th=[ 5276], | 70.00th=[ 6390], 80.00th=[ 8586], 90.00th=[11731], 95.00th=[14222], | 99.00th=[21890], 99.50th=[25297], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 896, max= 2592, per=23.86%, avg=1387.38, stdev=580.58, samples=8 iops : min= 56, max= 162, avg=86.62, stdev=36.33, samples=8 lat (msec) : 2=0.52%, 4=47.53%, 10=39.45%, 20=11.46%, 50=1.04% cpu : usr=0.12%, sys=12.33%, ctx=966, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5528KiB/s (5660kB/s), 1326KiB/s-1446KiB/s (1357kB/s-1481kB/s), io=23.4MiB (24.5MB), run=4275-4333msec WRITE: bw=5816KiB/s (5955kB/s), 1395KiB/s-1510KiB/s (1428kB/s-1547kB/s), io=24.6MiB (25.8MB), run=4275-4333msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8433: Mon Jun 2 09:05:20 2025 read: IOPS=14, BW=14.7MiB/s (15.5MB/s)(7168KiB/475msec) clat (usec): min=373, max=49973, avg=22399.39, stdev=21332.56 lat (usec): min=373, max=49974, avg=22400.47, stdev=21332.66 clat percentiles (usec): | 1.00th=[ 375], 5.00th=[ 375], 10.00th=[ 375], 20.00th=[ 478], | 30.00th=[ 482], 40.00th=[ 482], 50.00th=[30802], 60.00th=[35390], | 70.00th=[35390], 80.00th=[39060], 90.00th=[50070], 95.00th=[50070], | 99.00th=[50070], 99.50th=[50070], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] write: IOPS=10, BW=10.5MiB/s (11.0MB/s)(5120KiB/475msec) clat (usec): min=36120, max=86452, avg=59312.47, stdev=22734.99 lat (usec): min=36192, max=86470, avg=59352.13, stdev=22734.75 clat percentiles (usec): | 1.00th=[35914], 5.00th=[35914], 10.00th=[35914], 20.00th=[35914], | 30.00th=[46400], 40.00th=[46400], 50.00th=[46924], 60.00th=[46924], | 70.00th=[81265], 80.00th=[81265], 90.00th=[86508], 95.00th=[86508], | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] lat (usec) : 500=25.00% lat (msec) : 50=58.33%, 100=16.67% cpu : usr=0.00%, sys=38.82%, ctx=449, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8435: Mon Jun 2 09:05:20 2025 read: IOPS=2, BW=2943KiB/s (3013kB/s)(2048KiB/696msec) clat (usec): min=1483, max=29547, avg=15515.48, stdev=19844.41 lat (usec): min=1484, max=29549, avg=15516.76, stdev=19845.15 clat percentiles (usec): | 1.00th=[ 1483], 5.00th=[ 1483], 10.00th=[ 1483], 20.00th=[ 1483], | 30.00th=[ 1483], 40.00th=[ 1483], 50.00th=[ 1483], 60.00th=[29492], | 70.00th=[29492], 80.00th=[29492], 90.00th=[29492], 95.00th=[29492], | 99.00th=[29492], 99.50th=[29492], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 4096, max= 4096, per=13.92%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=14, BW=14.4MiB/s (15.1MB/s)(10.0MiB/696msec) clat (msec): min=30, max=107, avg=60.76, stdev=27.18 lat (msec): min=30, max=107, avg=60.81, stdev=27.19 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 41], | 30.00th=[ 42], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 53], | 70.00th=[ 92], 80.00th=[ 92], 90.00th=[ 97], 95.00th=[ 108], | 99.00th=[ 108], 99.50th=[ 108], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min=12288, max=12288, per=29.83%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (msec) : 2=8.33%, 50=50.00%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=38.27%, ctx=211, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8437: Mon Jun 2 09:05:20 2025 read: IOPS=10, BW=10.7MiB/s (11.2MB/s)(6144KiB/561msec) clat (usec): min=365, max=56348, avg=15330.54, stdev=22443.33 lat (usec): min=365, max=56349, avg=15333.01, stdev=22444.51 clat percentiles (usec): | 1.00th=[ 367], 5.00th=[ 367], 10.00th=[ 367], 20.00th=[ 433], | 30.00th=[ 433], 40.00th=[ 453], 50.00th=[ 453], 60.00th=[ 8225], | 70.00th=[26084], 80.00th=[26084], 90.00th=[56361], 95.00th=[56361], | 99.00th=[56361], 99.50th=[56361], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 8192, max= 8192, per=27.84%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.7MiB/s (11.2MB/s)(6144KiB/561msec) clat (msec): min=30, max=156, avg=68.95, stdev=45.26 lat (msec): min=30, max=156, avg=68.99, stdev=45.26 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 45], | 30.00th=[ 45], 40.00th=[ 49], 50.00th=[ 49], 60.00th=[ 64], | 70.00th=[ 70], 80.00th=[ 70], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min=10240, max=10240, per=24.86%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (usec) : 500=25.00% lat (msec) : 10=8.33%, 50=33.33%, 100=25.00%, 250=8.33% cpu : usr=0.18%, sys=24.29%, ctx=174, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8439: Mon Jun 2 09:05:20 2025 read: IOPS=7, BW=7938KiB/s (8128kB/s)(5120KiB/645msec) clat (usec): min=557, max=50296, avg=20253.65, stdev=20895.89 lat (usec): min=558, max=50297, avg=20254.86, stdev=20895.90 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 562], 10.00th=[ 562], 20.00th=[ 562], | 30.00th=[ 1434], 40.00th=[ 1434], 50.00th=[18744], 60.00th=[18744], | 70.00th=[30278], 80.00th=[30278], 90.00th=[50070], 95.00th=[50070], | 99.00th=[50070], 99.50th=[50070], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 8192, max= 8192, per=27.84%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.9MiB/s (11.4MB/s)(7168KiB/645msec) clat (msec): min=31, max=115, avg=70.45, stdev=30.47 lat (msec): min=31, max=115, avg=70.52, stdev=30.48 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 37], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 77], 60.00th=[ 81], | 70.00th=[ 81], 80.00th=[ 95], 90.00th=[ 116], 95.00th=[ 116], | 99.00th=[ 116], 99.50th=[ 116], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min=10240, max=10240, per=24.86%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 2=8.33%, 20=8.33%, 50=25.00%, 100=41.67%, 250=8.33% cpu : usr=0.00%, sys=27.33%, ctx=431, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=28.7MiB/s (30.1MB/s), 2943KiB/s-14.7MiB/s (3013kB/s-15.5MB/s), io=20.0MiB (20.0MB), run=475-696msec WRITE: bw=40.2MiB/s (42.2MB/s), 10.5MiB/s-14.4MiB/s (11.0MB/s-15.1MB/s), io=28.0MiB (29.4MB), run=475-696msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8432: Mon Jun 2 09:05:21 2025 read: IOPS=12, BW=12.1MiB/s (12.7MB/s)(7168KiB/578msec) clat (usec): min=17798, max=79334, avg=38515.20, stdev=20900.99 lat (usec): min=17799, max=79335, avg=38516.52, stdev=20900.84 clat percentiles (usec): | 1.00th=[17695], 5.00th=[17695], 10.00th=[17695], 20.00th=[21627], | 30.00th=[25035], 40.00th=[25035], 50.00th=[39584], 60.00th=[41681], | 70.00th=[41681], 80.00th=[44827], 90.00th=[79168], 95.00th=[79168], | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min=12288, max=12288, per=46.38%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8858KiB/s (9071kB/s)(5120KiB/578msec) clat (usec): min=42763, max=82069, avg=58626.25, stdev=15919.93 lat (usec): min=42843, max=82139, avg=58687.74, stdev=15916.15 clat percentiles (usec): | 1.00th=[42730], 5.00th=[42730], 10.00th=[42730], 20.00th=[42730], | 30.00th=[45876], 40.00th=[45876], 50.00th=[56886], 60.00th=[56886], | 70.00th=[65799], 80.00th=[65799], 90.00th=[82314], 95.00th=[82314], | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 8192, max= 8192, per=22.09%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=58.33%, 100=33.33% cpu : usr=0.00%, sys=20.62%, ctx=78, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8434: Mon Jun 2 09:05:21 2025 read: IOPS=2, BW=2649KiB/s (2713kB/s)(2048KiB/773msec) clat (usec): min=16257, max=91117, avg=53687.41, stdev=52933.69 lat (usec): min=16260, max=91118, avg=53689.49, stdev=52933.00 clat percentiles (usec): | 1.00th=[16319], 5.00th=[16319], 10.00th=[16319], 20.00th=[16319], | 30.00th=[16319], 40.00th=[16319], 50.00th=[16319], 60.00th=[90702], | 70.00th=[90702], 80.00th=[90702], 90.00th=[90702], 95.00th=[90702], | 99.00th=[90702], 99.50th=[90702], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 4039, max= 4039, per=15.24%, avg=4039.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.9MiB/s (13.6MB/s)(10.0MiB/773msec) clat (usec): min=25195, max=97800, avg=65498.37, stdev=23507.72 lat (usec): min=25297, max=97892, avg=65567.02, stdev=23496.92 clat percentiles (usec): | 1.00th=[25297], 5.00th=[25297], 10.00th=[25297], 20.00th=[43779], | 30.00th=[50594], 40.00th=[52167], 50.00th=[60556], 60.00th=[63701], | 70.00th=[87557], 80.00th=[87557], 90.00th=[91751], 95.00th=[98042], | 99.00th=[98042], 99.50th=[98042], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min=10098, max=10098, per=27.22%, avg=10098.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=16.67%, 100=75.00% cpu : usr=0.00%, sys=20.21%, ctx=91, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8436: Mon Jun 2 09:05:21 2025 read: IOPS=7, BW=7969KiB/s (8160kB/s)(6144KiB/771msec) clat (msec): min=14, max=100, avg=57.23, stdev=36.13 lat (msec): min=14, max=100, avg=57.23, stdev=36.13 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 18], | 30.00th=[ 18], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 70], | 70.00th=[ 90], 80.00th=[ 90], 90.00th=[ 102], 95.00th=[ 102], | 99.00th=[ 102], 99.50th=[ 102], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 4055, max= 4055, per=15.31%, avg=4055.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7969KiB/s (8160kB/s)(6144KiB/771msec) clat (msec): min=23, max=135, avg=62.20, stdev=42.08 lat (msec): min=23, max=135, avg=62.25, stdev=42.09 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 35], 50.00th=[ 35], 60.00th=[ 73], | 70.00th=[ 77], 80.00th=[ 77], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 6083, max= 6083, per=16.40%, avg=6083.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=25.00%, 100=41.67%, 250=16.67% cpu : usr=0.00%, sys=16.75%, ctx=77, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8438: Mon Jun 2 09:05:21 2025 read: IOPS=6, BW=6781KiB/s (6944kB/s)(5120KiB/755msec) clat (usec): min=15331, max=74907, avg=51659.70, stdev=23985.91 lat (usec): min=15333, max=74908, avg=51660.70, stdev=23985.52 clat percentiles (usec): | 1.00th=[15270], 5.00th=[15270], 10.00th=[15270], 20.00th=[15270], | 30.00th=[40109], 40.00th=[40109], 50.00th=[63701], 60.00th=[63701], | 70.00th=[64226], 80.00th=[64226], 90.00th=[74974], 95.00th=[74974], | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 4096, max= 4096, per=15.46%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9494KiB/s (9722kB/s)(7168KiB/755msec) clat (msec): min=22, max=120, avg=64.97, stdev=33.73 lat (msec): min=22, max=120, avg=65.03, stdev=33.73 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 33], | 30.00th=[ 49], 40.00th=[ 49], 50.00th=[ 70], 60.00th=[ 77], | 70.00th=[ 77], 80.00th=[ 87], 90.00th=[ 121], 95.00th=[ 121], | 99.00th=[ 121], 99.50th=[ 121], 99.90th=[ 121], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 8192, max= 8192, per=22.09%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=50.00%, 250=8.33% cpu : usr=0.00%, sys=13.93%, ctx=75, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.9MiB/s (27.1MB/s), 2649KiB/s-12.1MiB/s (2713kB/s-12.7MB/s), io=20.0MiB (20.0MB), run=578-773msec WRITE: bw=36.2MiB/s (37.0MB/s), 7969KiB/s-12.9MiB/s (8160kB/s-13.6MB/s), io=28.0MiB (29.4MB), run=578-773msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8444: Mon Jun 2 09:05:22 2025 read: IOPS=4, BW=19.3MiB/s (20.3MB/s)(12.0MiB/621msec) clat (msec): min=156, max=225, avg=193.24, stdev=34.90 lat (msec): min=156, max=225, avg=193.24, stdev=34.90 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 199], 50.00th=[ 199], 60.00th=[ 199], | 70.00th=[ 226], 80.00th=[ 226], 90.00th=[ 226], 95.00th=[ 226], | 99.00th=[ 226], 99.50th=[ 226], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min=16253, max=16253, per=61.60%, avg=16253.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=10.97%, ctx=52, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8445: Mon Jun 2 09:05:22 2025 read: IOPS=2, BW=9918KiB/s (10.2MB/s)(4096KiB/413msec) clat (nsec): min=42105k, max=42105k, avg=42104950.00, stdev= 0.00 lat (nsec): min=42108k, max=42108k, avg=42107500.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[42206], 5.00th=[42206], 10.00th=[42206], 20.00th=[42206], | 30.00th=[42206], 40.00th=[42206], 50.00th=[42206], 60.00th=[42206], | 70.00th=[42206], 80.00th=[42206], 90.00th=[42206], 95.00th=[42206], | 99.00th=[42206], 99.50th=[42206], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] write: IOPS=4, BW=19.4MiB/s (20.3MB/s)(8192KiB/413msec) clat (msec): min=126, max=211, avg=169.08, stdev=60.67 lat (msec): min=126, max=212, avg=169.39, stdev=60.61 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 127], 10.00th=[ 127], 20.00th=[ 127], | 30.00th=[ 127], 40.00th=[ 127], 50.00th=[ 127], 60.00th=[ 213], | 70.00th=[ 213], 80.00th=[ 213], 90.00th=[ 213], 95.00th=[ 213], | 99.00th=[ 213], 99.50th=[ 213], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 213] lat (msec) : 50=33.33%, 250=66.67% cpu : usr=0.24%, sys=20.39%, ctx=44, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8446: Mon Jun 2 09:05:22 2025 write: IOPS=4, BW=18.8MiB/s (19.7MB/s)(12.0MiB/639msec) clat (msec): min=111, max=310, avg=199.02, stdev=101.54 lat (msec): min=111, max=310, avg=199.31, stdev=101.45 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 176], 50.00th=[ 176], 60.00th=[ 176], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min=16384, max=16384, per=31.95%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.16%, sys=13.64%, ctx=77, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8447: Mon Jun 2 09:05:22 2025 write: IOPS=5, BW=21.7MiB/s (22.8MB/s)(12.0MiB/552msec) clat (msec): min=64, max=352, avg=177.03, stdev=153.90 lat (msec): min=66, max=352, avg=177.78, stdev=153.39 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 65], 10.00th=[ 65], 20.00th=[ 65], | 30.00th=[ 65], 40.00th=[ 114], 50.00th=[ 114], 60.00th=[ 114], | 70.00th=[ 355], 80.00th=[ 355], 90.00th=[ 355], 95.00th=[ 355], | 99.00th=[ 355], 99.50th=[ 355], 99.90th=[ 355], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min=16384, max=16384, per=31.95%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.18%, sys=17.79%, ctx=115, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.8MiB/s (27.0MB/s), 9918KiB/s-19.3MiB/s (10.2MB/s-20.3MB/s), io=16.0MiB (16.8MB), run=413-621msec WRITE: bw=50.1MiB/s (52.5MB/s), 18.8MiB/s-21.7MiB/s (19.7MB/s-22.8MB/s), io=32.0MiB (33.6MB), run=413-639msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8449: Mon Jun 2 09:05:22 2025 read: IOPS=9, BW=37.6MiB/s (39.4MB/s)(12.0MiB/319msec) clat (usec): min=1548, max=150592, avg=88246.26, stdev=77448.74 lat (usec): min=1549, max=150594, avg=88248.12, stdev=77449.02 clat percentiles (usec): | 1.00th=[ 1549], 5.00th=[ 1549], 10.00th=[ 1549], 20.00th=[ 1549], | 30.00th=[ 1549], 40.00th=[112722], 50.00th=[112722], 60.00th=[112722], | 70.00th=[149947], 80.00th=[149947], 90.00th=[149947], 95.00th=[149947], | 99.00th=[149947], 99.50th=[149947], 99.90th=[149947], 99.95th=[149947], | 99.99th=[149947] lat (msec) : 2=33.33%, 250=66.67% cpu : usr=0.00%, sys=34.91%, ctx=1609, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8450: Mon Jun 2 09:05:22 2025 read: IOPS=1, BW=5988KiB/s (6132kB/s)(4096KiB/684msec) clat (nsec): min=134850k, max=134850k, avg=134849973.00, stdev= 0.00 lat (nsec): min=134852k, max=134852k, avg=134851851.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 136], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 8192, max= 8192, per=34.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=11.7MiB/s (12.3MB/s)(8192KiB/684msec) clat (msec): min=239, max=290, avg=264.89, stdev=36.01 lat (msec): min=239, max=290, avg=265.19, stdev=36.05 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 241], 10.00th=[ 241], 20.00th=[ 241], | 30.00th=[ 241], 40.00th=[ 241], 50.00th=[ 241], 60.00th=[ 292], | 70.00th=[ 292], 80.00th=[ 292], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 8192, max= 8192, per=18.28%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.29%, sys=33.09%, ctx=1001, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8451: Mon Jun 2 09:05:22 2025 write: IOPS=4, BW=17.8MiB/s (18.7MB/s)(12.0MiB/674msec) clat (msec): min=131, max=250, avg=208.36, stdev=66.41 lat (msec): min=132, max=250, avg=208.65, stdev=66.42 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 132], 10.00th=[ 132], 20.00th=[ 132], | 30.00th=[ 132], 40.00th=[ 243], 50.00th=[ 243], 60.00th=[ 243], | 70.00th=[ 251], 80.00th=[ 251], 90.00th=[ 251], 95.00th=[ 251], | 99.00th=[ 251], 99.50th=[ 251], 99.90th=[ 251], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 8175, max= 8175, per=18.24%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.45%, sys=49.63%, ctx=30, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8452: Mon Jun 2 09:05:22 2025 write: IOPS=4, BW=16.4MiB/s (17.2MB/s)(12.0MiB/731msec) clat (msec): min=100, max=326, avg=235.38, stdev=119.31 lat (msec): min=100, max=326, avg=235.71, stdev=119.35 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 101], 10.00th=[ 101], 20.00th=[ 101], | 30.00th=[ 101], 40.00th=[ 279], 50.00th=[ 279], 60.00th=[ 279], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 8192, max= 8192, per=18.28%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=35.48%, ctx=53, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=23.4MiB/s (24.5MB/s), 5988KiB/s-37.6MiB/s (6132kB/s-39.4MB/s), io=16.0MiB (16.8MB), run=319-684msec WRITE: bw=43.8MiB/s (45.9MB/s), 11.7MiB/s-17.8MiB/s (12.3MB/s-18.7MB/s), io=32.0MiB (33.6MB), run=674-731msec