== sanity test 398b: DIO and buffer IO race ============== 14:00:24 (1747850424) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.48135 s, 34.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8175: Wed May 21 14:00:43 2025 read: IOPS=96, BW=384KiB/s (393kB/s)(5992KiB/15596msec) clat (usec): min=58, max=22210, avg=5774.00, stdev=3238.20 lat (usec): min=58, max=22210, avg=5775.00, stdev=3238.72 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 206], 10.00th=[ 2769], 20.00th=[ 3326], | 30.00th=[ 3851], 40.00th=[ 4490], 50.00th=[ 5276], 60.00th=[ 6063], | 70.00th=[ 6915], 80.00th=[ 7963], 90.00th=[ 9634], 95.00th=[11207], | 99.00th=[17433], 99.50th=[20317], 99.90th=[21890], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 184, max= 520, per=25.03%, avg=382.03, stdev=63.99, samples=31 iops : min= 46, max= 130, avg=95.39, stdev=16.02, samples=31 write: IOPS=100, BW=404KiB/s (413kB/s)(6296KiB/15596msec) clat (usec): min=467, max=206561, avg=4362.00, stdev=6456.92 lat (usec): min=467, max=206562, avg=4362.87, stdev=6456.93 clat percentiles (usec): | 1.00th=[ 578], 5.00th=[ 676], 10.00th=[ 717], 20.00th=[ 799], | 30.00th=[ 963], 40.00th=[ 1598], 50.00th=[ 3392], 60.00th=[ 4228], | 70.00th=[ 5342], 80.00th=[ 7046], 90.00th=[ 10159], 95.00th=[ 12256], | 99.00th=[ 17171], 99.50th=[ 19268], 99.90th=[ 26084], 99.95th=[206570], | 99.99th=[206570] bw ( KiB/s): min= 216, max= 600, per=25.07%, avg=400.61, stdev=72.63, samples=31 iops : min= 54, max= 150, avg=100.03, stdev=18.14, samples=31 lat (usec) : 100=1.11%, 250=1.33%, 500=0.23%, 750=7.45%, 1000=8.46% lat (msec) : 2=5.83%, 4=20.15%, 10=46.29%, 20=8.79%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=21.60%, ctx=5695, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8177: Wed May 21 14:00:43 2025 read: IOPS=94, BW=379KiB/s (388kB/s)(5940KiB/15685msec) clat (usec): min=58, max=24132, avg=5636.65, stdev=3141.70 lat (usec): min=58, max=24133, avg=5637.58, stdev=3141.73 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 159], 10.00th=[ 2769], 20.00th=[ 3359], | 30.00th=[ 3785], 40.00th=[ 4359], 50.00th=[ 5080], 60.00th=[ 5866], | 70.00th=[ 6783], 80.00th=[ 7832], 90.00th=[ 9503], 95.00th=[10945], | 99.00th=[16909], 99.50th=[18744], 99.90th=[23987], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 184, max= 496, per=24.68%, avg=376.65, stdev=64.74, samples=31 iops : min= 46, max= 124, avg=94.06, stdev=16.14, samples=31 write: IOPS=101, BW=405KiB/s (414kB/s)(6348KiB/15685msec) clat (usec): min=462, max=211146, avg=4555.34, stdev=6583.07 lat (usec): min=462, max=211147, avg=4557.11, stdev=6582.61 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 676], 10.00th=[ 725], 20.00th=[ 799], | 30.00th=[ 947], 40.00th=[ 1729], 50.00th=[ 3556], 60.00th=[ 4555], | 70.00th=[ 5735], 80.00th=[ 7635], 90.00th=[ 10421], 95.00th=[ 12256], | 99.00th=[ 16319], 99.50th=[ 19530], 99.90th=[ 26346], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 256, max= 496, per=24.99%, avg=399.35, stdev=65.68, samples=31 iops : min= 64, max= 124, avg=99.74, stdev=16.48, samples=31 lat (usec) : 100=1.50%, 250=1.01%, 500=0.23%, 750=6.97%, 1000=9.05% lat (msec) : 2=5.14%, 4=20.77%, 10=45.57%, 20=9.41%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=21.30%, ctx=5548, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8179: Wed May 21 14:00:43 2025 read: IOPS=95, BW=383KiB/s (392kB/s)(6028KiB/15728msec) clat (usec): min=53, max=26721, avg=5698.60, stdev=3445.89 lat (usec): min=53, max=26722, avg=5699.40, stdev=3445.88 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 117], 10.00th=[ 2671], 20.00th=[ 3359], | 30.00th=[ 3818], 40.00th=[ 4293], 50.00th=[ 5014], 60.00th=[ 5932], | 70.00th=[ 6718], 80.00th=[ 7767], 90.00th=[ 9765], 95.00th=[11994], | 99.00th=[18220], 99.50th=[21103], 99.90th=[25035], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 112, max= 488, per=24.85%, avg=379.19, stdev=74.21, samples=31 iops : min= 28, max= 122, avg=94.68, stdev=18.57, samples=31 write: IOPS=99, BW=398KiB/s (408kB/s)(6260KiB/15728msec) clat (usec): min=465, max=223670, avg=4511.18, stdev=6886.86 lat (usec): min=465, max=223672, avg=4512.34, stdev=6886.91 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 668], 10.00th=[ 717], 20.00th=[ 791], | 30.00th=[ 930], 40.00th=[ 1729], 50.00th=[ 3556], 60.00th=[ 4424], | 70.00th=[ 5604], 80.00th=[ 7635], 90.00th=[ 10028], 95.00th=[ 12387], | 99.00th=[ 17433], 99.50th=[ 20579], 99.90th=[ 30278], 99.95th=[223347], | 99.99th=[223347] bw ( KiB/s): min= 152, max= 560, per=24.59%, avg=392.87, stdev=77.08, samples=31 iops : min= 38, max= 140, avg=98.10, stdev=19.22, samples=31 lat (usec) : 100=1.66%, 250=1.37%, 500=0.16%, 750=7.55%, 1000=8.85% lat (msec) : 2=4.75%, 4=20.08%, 10=46.06%, 20=8.85%, 50=0.62% lat (msec) : 250=0.03% cpu : usr=0.30%, sys=20.78%, ctx=5591, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8181: Wed May 21 14:00:43 2025 read: IOPS=97, BW=391KiB/s (400kB/s)(6052KiB/15480msec) clat (usec): min=58, max=36655, avg=5788.85, stdev=3328.86 lat (usec): min=59, max=36655, avg=5789.65, stdev=3328.83 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 2180], 10.00th=[ 2868], 20.00th=[ 3425], | 30.00th=[ 3884], 40.00th=[ 4424], 50.00th=[ 5080], 60.00th=[ 5932], | 70.00th=[ 6980], 80.00th=[ 7898], 90.00th=[ 9634], 95.00th=[11207], | 99.00th=[16712], 99.50th=[20317], 99.90th=[31589], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 200, max= 520, per=25.45%, avg=388.37, stdev=59.84, samples=30 iops : min= 50, max= 130, avg=96.97, stdev=15.01, samples=30 write: IOPS=100, BW=403KiB/s (413kB/s)(6236KiB/15480msec) clat (usec): min=463, max=212486, avg=4257.87, stdev=6525.14 lat (usec): min=464, max=212486, avg=4258.99, stdev=6525.18 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 685], 10.00th=[ 717], 20.00th=[ 783], | 30.00th=[ 889], 40.00th=[ 1500], 50.00th=[ 3392], 60.00th=[ 4228], | 70.00th=[ 5145], 80.00th=[ 7111], 90.00th=[ 9634], 95.00th=[ 11731], | 99.00th=[ 16909], 99.50th=[ 20055], 99.90th=[ 24249], 99.95th=[212861], | 99.99th=[212861] bw ( KiB/s): min= 216, max= 528, per=25.05%, avg=400.37, stdev=70.90, samples=30 iops : min= 54, max= 132, avg=99.97, stdev=17.70, samples=30 lat (usec) : 100=1.11%, 250=1.11%, 500=0.13%, 750=7.42%, 1000=9.93% lat (msec) : 2=3.97%, 4=21.39%, 10=46.06%, 20=8.37%, 50=0.49% lat (msec) : 250=0.03% cpu : usr=0.30%, sys=21.49%, ctx=5481, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1527KiB/s (1563kB/s), 379KiB/s-391KiB/s (388kB/s-400kB/s), io=23.4MiB (24.6MB), run=15480-15728msec WRITE: bw=1598KiB/s (1637kB/s), 398KiB/s-405KiB/s (408kB/s-414kB/s), io=24.6MiB (25.7MB), run=15480-15728msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8176: Wed May 21 14:00:49 2025 read: IOPS=69, BW=277KiB/s (284kB/s)(5992KiB/21603msec) clat (usec): min=1976, max=223504, avg=5623.43, stdev=7048.63 lat (usec): min=1976, max=223506, avg=5624.99, stdev=7048.38 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 3], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 5], | 70.00th=[ 6], 80.00th=[ 8], 90.00th=[ 12], 95.00th=[ 16], | 99.00th=[ 22], 99.50th=[ 25], 99.90th=[ 35], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 112, max= 558, per=25.04%, avg=276.21, stdev=130.34, samples=43 iops : min= 28, max= 139, avg=68.91, stdev=32.58, samples=43 write: IOPS=72, BW=291KiB/s (298kB/s)(6296KiB/21603msec) clat (usec): min=3096, max=38678, avg=8320.83, stdev=4846.18 lat (usec): min=3097, max=38680, avg=8322.04, stdev=4846.14 clat percentiles (usec): | 1.00th=[ 3392], 5.00th=[ 3982], 10.00th=[ 4293], 20.00th=[ 4752], | 30.00th=[ 5211], 40.00th=[ 5735], 50.00th=[ 6325], 60.00th=[ 7373], | 70.00th=[ 9503], 80.00th=[11863], 90.00th=[15008], 95.00th=[17171], | 99.00th=[25822], 99.50th=[29492], 99.90th=[36963], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 104, max= 574, per=25.14%, avg=290.37, stdev=132.00, samples=43 iops : min= 26, max= 143, avg=72.44, stdev=32.99, samples=43 lat (msec) : 2=0.03%, 4=29.56%, 10=49.74%, 20=18.55%, 50=2.08% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=10.33%, ctx=3614, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8178: Wed May 21 14:00:49 2025 read: IOPS=68, BW=273KiB/s (280kB/s)(5940KiB/21753msec) clat (usec): min=1944, max=193626, avg=5546.83, stdev=6470.81 lat (usec): min=1945, max=193628, avg=5547.64, stdev=6470.83 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 3], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 5], | 70.00th=[ 6], 80.00th=[ 8], 90.00th=[ 12], 95.00th=[ 15], | 99.00th=[ 23], 99.50th=[ 26], 99.90th=[ 29], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 72, max= 552, per=24.69%, avg=272.28, stdev=131.14, samples=43 iops : min= 18, max= 138, avg=68.00, stdev=32.76, samples=43 write: IOPS=72, BW=292KiB/s (299kB/s)(6348KiB/21753msec) clat (usec): min=3068, max=40779, avg=8452.90, stdev=4919.93 lat (usec): min=3069, max=40780, avg=8454.00, stdev=4919.90 clat percentiles (usec): | 1.00th=[ 3490], 5.00th=[ 4080], 10.00th=[ 4424], 20.00th=[ 4883], | 30.00th=[ 5342], 40.00th=[ 5800], 50.00th=[ 6456], 60.00th=[ 7635], | 70.00th=[ 9503], 80.00th=[11600], 90.00th=[15139], 95.00th=[18220], | 99.00th=[26084], 99.50th=[31589], 99.90th=[39060], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 120, max= 518, per=24.97%, avg=288.44, stdev=126.17, samples=43 iops : min= 30, max= 129, avg=72.05, stdev=31.52, samples=43 lat (msec) : 2=0.07%, 4=29.43%, 10=49.64%, 20=18.52%, 50=2.31% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=10.21%, ctx=3628, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8180: Wed May 21 14:00:49 2025 read: IOPS=69, BW=278KiB/s (285kB/s)(6028KiB/21688msec) clat (usec): min=2012, max=30730, avg=5575.59, stdev=4348.27 lat (usec): min=2012, max=30731, avg=5576.54, stdev=4348.22 clat percentiles (usec): | 1.00th=[ 2180], 5.00th=[ 2409], 10.00th=[ 2540], 20.00th=[ 2769], | 30.00th=[ 3032], 40.00th=[ 3294], 50.00th=[ 3687], 60.00th=[ 4359], | 70.00th=[ 5342], 80.00th=[ 7767], 90.00th=[12256], 95.00th=[15664], | 99.00th=[20841], 99.50th=[23462], 99.90th=[27395], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 96, max= 550, per=25.09%, avg=276.77, stdev=124.28, samples=43 iops : min= 24, max= 137, avg=69.05, stdev=31.03, samples=43 write: IOPS=72, BW=289KiB/s (296kB/s)(6260KiB/21688msec) clat (msec): min=2, max=220, avg= 8.44, stdev= 7.24 lat (msec): min=2, max=220, avg= 8.44, stdev= 7.24 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 10], 80.00th=[ 12], 90.00th=[ 16], 95.00th=[ 19], | 99.00th=[ 27], 99.50th=[ 30], 99.90th=[ 37], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 104, max= 568, per=24.79%, avg=286.28, stdev=133.21, samples=43 iops : min= 26, max= 142, avg=71.42, stdev=33.32, samples=43 lat (msec) : 4=30.01%, 10=49.77%, 20=18.03%, 50=2.15%, 250=0.03% cpu : usr=0.18%, sys=10.22%, ctx=3641, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8182: Wed May 21 14:00:49 2025 read: IOPS=69, BW=280KiB/s (286kB/s)(6052KiB/21645msec) clat (usec): min=2008, max=33786, avg=5349.43, stdev=4232.75 lat (usec): min=2009, max=33787, avg=5350.22, stdev=4232.76 clat percentiles (usec): | 1.00th=[ 2212], 5.00th=[ 2409], 10.00th=[ 2540], 20.00th=[ 2737], | 30.00th=[ 2933], 40.00th=[ 3195], 50.00th=[ 3556], 60.00th=[ 4113], | 70.00th=[ 5211], 80.00th=[ 7177], 90.00th=[11600], 95.00th=[15008], | 99.00th=[21103], 99.50th=[25560], 99.90th=[32375], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 96, max= 560, per=25.16%, avg=277.49, stdev=130.05, samples=43 iops : min= 24, max= 140, avg=69.30, stdev=32.55, samples=43 write: IOPS=72, BW=288KiB/s (295kB/s)(6236KiB/21645msec) clat (msec): min=3, max=202, avg= 8.63, stdev= 6.98 lat (msec): min=3, max=202, avg= 8.63, stdev= 6.98 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 10], 80.00th=[ 12], 90.00th=[ 16], 95.00th=[ 19], | 99.00th=[ 27], 99.50th=[ 31], 99.90th=[ 41], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 104, max= 536, per=24.83%, avg=286.79, stdev=127.78, samples=43 iops : min= 26, max= 134, avg=71.63, stdev=31.98, samples=43 lat (msec) : 4=30.83%, 10=48.37%, 20=18.23%, 50=2.54%, 250=0.03% cpu : usr=0.18%, sys=10.26%, ctx=3615, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1104KiB/s (1130kB/s), 273KiB/s-280KiB/s (280kB/s-286kB/s), io=23.4MiB (24.6MB), run=21603-21753msec WRITE: bw=1156KiB/s (1183kB/s), 288KiB/s-292KiB/s (295kB/s-299kB/s), io=24.6MiB (25.7MB), run=21603-21753msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8211: Wed May 21 14:00:54 2025 read: IOPS=86, BW=1388KiB/s (1421kB/s)(5920KiB/4265msec) clat (usec): min=61, max=27251, avg=6238.76, stdev=4143.48 lat (usec): min=62, max=27251, avg=6244.94, stdev=4145.40 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 99], 10.00th=[ 127], 20.00th=[ 3556], | 30.00th=[ 4228], 40.00th=[ 4752], 50.00th=[ 5604], 60.00th=[ 6587], | 70.00th=[ 7635], 80.00th=[ 8848], 90.00th=[10683], 95.00th=[13829], | 99.00th=[21103], 99.50th=[26084], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 1088, max= 1728, per=25.19%, avg=1394.37, stdev=212.39, samples=8 iops : min= 68, max= 108, avg=86.88, stdev=13.16, samples=8 write: IOPS=93, BW=1493KiB/s (1529kB/s)(6368KiB/4265msec) clat (usec): min=837, max=28097, avg=4816.55, stdev=4394.17 lat (usec): min=838, max=28098, avg=4818.49, stdev=4393.64 clat percentiles (usec): | 1.00th=[ 857], 5.00th=[ 979], 10.00th=[ 1057], 20.00th=[ 1156], | 30.00th=[ 1336], 40.00th=[ 1975], 50.00th=[ 3458], 60.00th=[ 4752], | 70.00th=[ 5997], 80.00th=[ 8160], 90.00th=[11076], 95.00th=[13173], | 99.00th=[19792], 99.50th=[22152], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 1152, max= 1756, per=25.59%, avg=1490.38, stdev=207.71, samples=8 iops : min= 72, max= 109, avg=92.88, stdev=12.92, samples=8 lat (usec) : 100=2.60%, 250=2.86%, 1000=2.86% lat (msec) : 2=18.49%, 4=13.54%, 10=45.96%, 20=12.63%, 50=1.04% cpu : usr=0.47%, sys=22.73%, ctx=1523, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8212: Wed May 21 14:00:54 2025 read: IOPS=85, BW=1360KiB/s (1393kB/s)(5744KiB/4223msec) clat (usec): min=76, max=23779, avg=6315.10, stdev=3914.65 lat (usec): min=76, max=23780, avg=6315.96, stdev=3914.68 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 104], 10.00th=[ 693], 20.00th=[ 3752], | 30.00th=[ 4293], 40.00th=[ 4883], 50.00th=[ 5735], 60.00th=[ 6849], | 70.00th=[ 7504], 80.00th=[ 8848], 90.00th=[11207], 95.00th=[12649], | 99.00th=[20055], 99.50th=[23725], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 1152, max= 1664, per=24.75%, avg=1370.25, stdev=180.45, samples=8 iops : min= 72, max= 104, avg=85.25, stdev=11.41, samples=8 write: IOPS=96, BW=1550KiB/s (1587kB/s)(6544KiB/4223msec) clat (usec): min=749, max=29959, avg=4716.87, stdev=4381.96 lat (usec): min=750, max=29960, avg=4718.14, stdev=4381.89 clat percentiles (usec): | 1.00th=[ 832], 5.00th=[ 1012], 10.00th=[ 1074], 20.00th=[ 1172], | 30.00th=[ 1385], 40.00th=[ 2040], 50.00th=[ 3359], 60.00th=[ 4424], | 70.00th=[ 5866], 80.00th=[ 7635], 90.00th=[10552], 95.00th=[13698], | 99.00th=[17957], 99.50th=[22414], 99.90th=[30016], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 1309, max= 2080, per=26.33%, avg=1533.87, stdev=287.12, samples=8 iops : min= 81, max= 130, avg=95.50, stdev=18.07, samples=8 lat (usec) : 100=1.82%, 250=2.73%, 750=0.26%, 1000=1.69% lat (msec) : 2=19.27%, 4=16.28%, 10=44.01%, 20=13.02%, 50=0.91% cpu : usr=0.31%, sys=22.95%, ctx=1500, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8213: Wed May 21 14:00:54 2025 read: IOPS=90, BW=1446KiB/s (1481kB/s)(6256KiB/4326msec) clat (usec): min=60, max=27985, avg=6103.64, stdev=4128.83 lat (usec): min=60, max=27986, avg=6104.43, stdev=4128.83 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 101], 10.00th=[ 141], 20.00th=[ 3458], | 30.00th=[ 4047], 40.00th=[ 4752], 50.00th=[ 5276], 60.00th=[ 6194], | 70.00th=[ 7242], 80.00th=[ 8586], 90.00th=[10945], 95.00th=[13173], | 99.00th=[21103], 99.50th=[23987], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 1149, max= 1792, per=25.54%, avg=1414.00, stdev=220.18, samples=8 iops : min= 71, max= 112, avg=88.00, stdev=13.94, samples=8 write: IOPS=87, BW=1394KiB/s (1428kB/s)(6032KiB/4326msec) clat (usec): min=628, max=25888, avg=5062.74, stdev=4610.51 lat (usec): min=629, max=25889, avg=5065.13, stdev=4609.99 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 988], 10.00th=[ 1074], 20.00th=[ 1172], | 30.00th=[ 1352], 40.00th=[ 2114], 50.00th=[ 3916], 60.00th=[ 4686], | 70.00th=[ 6259], 80.00th=[ 8455], 90.00th=[11863], 95.00th=[14353], | 99.00th=[20055], 99.50th=[23987], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 1117, max= 1728, per=23.11%, avg=1346.12, stdev=230.19, samples=8 iops : min= 69, max= 108, avg=83.75, stdev=14.48, samples=8 lat (usec) : 100=2.34%, 250=3.12%, 750=0.39%, 1000=2.34% lat (msec) : 2=16.54%, 4=15.62%, 10=45.18%, 20=13.28%, 50=1.17% cpu : usr=0.28%, sys=21.92%, ctx=1590, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8214: Wed May 21 14:00:54 2025 read: IOPS=89, BW=1427KiB/s (1461kB/s)(6032KiB/4227msec) clat (usec): min=62, max=34239, avg=5968.10, stdev=4259.32 lat (usec): min=63, max=34239, avg=5968.91, stdev=4259.33 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 99], 10.00th=[ 128], 20.00th=[ 3458], | 30.00th=[ 4047], 40.00th=[ 4490], 50.00th=[ 5276], 60.00th=[ 6128], | 70.00th=[ 7111], 80.00th=[ 8455], 90.00th=[10421], 95.00th=[12780], | 99.00th=[22152], 99.50th=[32375], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 1021, max= 1824, per=25.25%, avg=1398.00, stdev=234.25, samples=8 iops : min= 63, max= 114, avg=87.00, stdev=14.89, samples=8 write: IOPS=92, BW=1480KiB/s (1516kB/s)(6256KiB/4227msec) clat (usec): min=795, max=24042, avg=4997.37, stdev=4501.81 lat (usec): min=796, max=24043, avg=4999.93, stdev=4502.04 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 1020], 10.00th=[ 1090], 20.00th=[ 1188], | 30.00th=[ 1401], 40.00th=[ 2212], 50.00th=[ 3916], 60.00th=[ 4817], | 70.00th=[ 5997], 80.00th=[ 7963], 90.00th=[11338], 95.00th=[14484], | 99.00th=[19530], 99.50th=[20317], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 1088, max= 1632, per=25.23%, avg=1469.75, stdev=180.96, samples=8 iops : min= 68, max= 102, avg=91.50, stdev=11.20, samples=8 lat (usec) : 100=2.73%, 250=2.60%, 500=0.39%, 1000=2.21% lat (msec) : 2=17.84%, 4=14.45%, 10=46.61%, 20=12.11%, 50=1.04% cpu : usr=0.31%, sys=22.46%, ctx=1534, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5537KiB/s (5670kB/s), 1360KiB/s-1446KiB/s (1393kB/s-1481kB/s), io=23.4MiB (24.5MB), run=4223-4326msec WRITE: bw=5825KiB/s (5965kB/s), 1394KiB/s-1550KiB/s (1428kB/s-1587kB/s), io=24.6MiB (25.8MB), run=4223-4326msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8216: Wed May 21 14:00:56 2025 read: IOPS=60, BW=974KiB/s (997kB/s)(5920KiB/6078msec) clat (usec): min=2432, max=32467, avg=6111.40, stdev=4700.07 lat (usec): min=2433, max=32468, avg=6112.24, stdev=4700.07 clat percentiles (usec): | 1.00th=[ 2507], 5.00th=[ 2769], 10.00th=[ 2933], 20.00th=[ 3261], | 30.00th=[ 3523], 40.00th=[ 3851], 50.00th=[ 4228], 60.00th=[ 4752], | 70.00th=[ 5473], 80.00th=[ 7504], 90.00th=[12911], 95.00th=[17171], | 99.00th=[23462], 99.50th=[25297], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 512, max= 1632, per=24.74%, avg=967.50, stdev=407.43, samples=12 iops : min= 32, max= 102, avg=60.33, stdev=25.44, samples=12 write: IOPS=65, BW=1048KiB/s (1073kB/s)(6368KiB/6078msec) clat (usec): min=3330, max=36766, avg=9397.75, stdev=5307.14 lat (usec): min=3331, max=36768, avg=9399.31, stdev=5307.13 clat percentiles (usec): | 1.00th=[ 3720], 5.00th=[ 4621], 10.00th=[ 5145], 20.00th=[ 5604], | 30.00th=[ 6128], 40.00th=[ 6521], 50.00th=[ 7111], 60.00th=[ 8356], | 70.00th=[10421], 80.00th=[13435], 90.00th=[16909], 95.00th=[20055], | 99.00th=[27657], 99.50th=[34866], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 638, max= 1760, per=25.46%, avg=1047.50, stdev=436.48, samples=12 iops : min= 39, max= 110, avg=65.33, stdev=27.25, samples=12 lat (msec) : 4=22.27%, 10=53.91%, 20=19.92%, 50=3.91% cpu : usr=0.10%, sys=10.58%, ctx=929, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8217: Wed May 21 14:00:56 2025 read: IOPS=59, BW=958KiB/s (981kB/s)(5744KiB/5996msec) clat (usec): min=2526, max=25211, avg=5935.25, stdev=4432.55 lat (usec): min=2527, max=25211, avg=5936.14, stdev=4432.51 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2769], 10.00th=[ 2933], 20.00th=[ 3261], | 30.00th=[ 3490], 40.00th=[ 3818], 50.00th=[ 4228], 60.00th=[ 4621], | 70.00th=[ 5473], 80.00th=[ 7635], 90.00th=[12256], 95.00th=[16581], | 99.00th=[22676], 99.50th=[24773], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 510, max= 1760, per=23.13%, avg=904.55, stdev=381.29, samples=11 iops : min= 31, max= 110, avg=56.45, stdev=23.92, samples=11 write: IOPS=68, BW=1091KiB/s (1118kB/s)(6544KiB/5996msec) clat (usec): min=4133, max=42513, avg=9323.19, stdev=5540.18 lat (usec): min=4135, max=42514, avg=9324.81, stdev=5540.13 clat percentiles (usec): | 1.00th=[ 4359], 5.00th=[ 4621], 10.00th=[ 5080], 20.00th=[ 5604], | 30.00th=[ 6063], 40.00th=[ 6456], 50.00th=[ 7046], 60.00th=[ 8160], | 70.00th=[10028], 80.00th=[12256], 90.00th=[16581], 95.00th=[20317], | 99.00th=[30540], 99.50th=[32375], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 704, max= 1792, per=24.95%, avg=1026.73, stdev=379.94, samples=11 iops : min= 44, max= 112, avg=64.09, stdev=23.79, samples=11 lat (msec) : 4=21.09%, 10=56.38%, 20=18.36%, 50=4.17% cpu : usr=0.12%, sys=10.71%, ctx=952, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8218: Wed May 21 14:00:56 2025 read: IOPS=64, BW=1028KiB/s (1052kB/s)(6256KiB/6088msec) clat (usec): min=2448, max=32579, avg=6329.54, stdev=5022.79 lat (usec): min=2448, max=32579, avg=6330.50, stdev=5022.69 clat percentiles (usec): | 1.00th=[ 2474], 5.00th=[ 2704], 10.00th=[ 2868], 20.00th=[ 3097], | 30.00th=[ 3392], 40.00th=[ 3785], 50.00th=[ 4113], 60.00th=[ 4817], | 70.00th=[ 6128], 80.00th=[ 8979], 90.00th=[13698], 95.00th=[17695], | 99.00th=[25035], 99.50th=[26084], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 574, max= 2080, per=26.44%, avg=1034.17, stdev=531.87, samples=12 iops : min= 35, max= 130, avg=64.50, stdev=33.25, samples=12 write: IOPS=61, BW=991KiB/s (1015kB/s)(6032KiB/6088msec) clat (usec): min=3759, max=33503, avg=9454.35, stdev=5405.26 lat (usec): min=3761, max=33504, avg=9455.89, stdev=5405.23 clat percentiles (usec): | 1.00th=[ 3916], 5.00th=[ 4621], 10.00th=[ 5145], 20.00th=[ 5604], | 30.00th=[ 5997], 40.00th=[ 6587], 50.00th=[ 7308], 60.00th=[ 8225], | 70.00th=[10421], 80.00th=[12911], 90.00th=[17433], 95.00th=[21627], | 99.00th=[28705], 99.50th=[32637], 99.90th=[33424], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 672, max= 1756, per=23.77%, avg=978.17, stdev=384.23, samples=12 iops : min= 42, max= 109, avg=61.00, stdev=23.94, samples=12 lat (msec) : 4=25.00%, 10=51.17%, 20=18.49%, 50=5.34% cpu : usr=0.07%, sys=10.79%, ctx=943, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8219: Wed May 21 14:00:56 2025 read: IOPS=61, BW=985KiB/s (1009kB/s)(6032KiB/6123msec) clat (usec): min=2372, max=26256, avg=6110.99, stdev=4866.22 lat (usec): min=2373, max=26257, avg=6111.78, stdev=4866.24 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2769], 10.00th=[ 2966], 20.00th=[ 3163], | 30.00th=[ 3392], 40.00th=[ 3687], 50.00th=[ 4015], 60.00th=[ 4490], | 70.00th=[ 5538], 80.00th=[ 8094], 90.00th=[12780], 95.00th=[17957], | 99.00th=[25297], 99.50th=[26084], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 446, max= 1728, per=24.93%, avg=975.17, stdev=431.19, samples=12 iops : min= 27, max= 108, avg=60.67, stdev=27.00, samples=12 write: IOPS=63, BW=1022KiB/s (1046kB/s)(6256KiB/6123msec) clat (usec): min=3585, max=36857, avg=9718.21, stdev=6074.06 lat (usec): min=3586, max=36859, avg=9719.87, stdev=6074.04 clat percentiles (usec): | 1.00th=[ 3916], 5.00th=[ 4424], 10.00th=[ 5211], 20.00th=[ 5604], | 30.00th=[ 6063], 40.00th=[ 6521], 50.00th=[ 7242], 60.00th=[ 8291], | 70.00th=[10290], 80.00th=[13435], 90.00th=[17171], 95.00th=[22676], | 99.00th=[33817], 99.50th=[36439], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 605, max= 1820, per=24.67%, avg=1015.08, stdev=468.41, samples=12 iops : min= 37, max= 113, avg=63.17, stdev=29.34, samples=12 lat (msec) : 4=24.74%, 10=52.86%, 20=17.06%, 50=5.34% cpu : usr=0.28%, sys=10.37%, ctx=903, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3912KiB/s (4006kB/s), 958KiB/s-1028KiB/s (981kB/s-1052kB/s), io=23.4MiB (24.5MB), run=5996-6123msec WRITE: bw=4116KiB/s (4214kB/s), 991KiB/s-1091KiB/s (1015kB/s-1118kB/s), io=24.6MiB (25.8MB), run=5996-6123msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8234: Wed May 21 14:00:59 2025 read: IOPS=6, BW=6211KiB/s (6361kB/s)(7168KiB/1154msec) clat (usec): min=1677, max=641215, avg=121822.90, stdev=229624.67 lat (usec): min=1678, max=641216, avg=121824.38, stdev=229624.62 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1680], 10.00th=[ 1680], 20.00th=[ 34341], | 30.00th=[ 38011], 40.00th=[ 38011], 50.00th=[ 40633], 60.00th=[ 40633], | 70.00th=[ 40633], 80.00th=[ 56361], 90.00th=[641729], 95.00th=[641729], | 99.00th=[641729], 99.50th=[641729], 99.90th=[641729], 99.95th=[641729], | 99.99th=[641729] bw ( KiB/s): min= 2048, max= 8192, per=54.60%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=4, BW=4437KiB/s (4543kB/s)(5120KiB/1154msec) clat (usec): min=28543, max=92777, avg=58158.64, stdev=24763.87 lat (usec): min=28559, max=92829, avg=58205.66, stdev=24770.60 clat percentiles (usec): | 1.00th=[28443], 5.00th=[28443], 10.00th=[28443], 20.00th=[28443], | 30.00th=[44827], 40.00th=[44827], 50.00th=[53216], 60.00th=[53216], | 70.00th=[71828], 80.00th=[71828], 90.00th=[92799], 95.00th=[92799], | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 8192, max= 8192, per=62.40%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 2=8.33%, 50=50.00%, 100=33.33%, 750=8.33% cpu : usr=0.09%, sys=27.47%, ctx=1171, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8236: Wed May 21 14:00:59 2025 read: IOPS=0, BW=938KiB/s (960kB/s)(2048KiB/2184msec) clat (usec): min=41675, max=46176, avg=43925.91, stdev=3182.41 lat (usec): min=41677, max=46177, avg=43927.39, stdev=3182.41 clat percentiles (usec): | 1.00th=[41681], 5.00th=[41681], 10.00th=[41681], 20.00th=[41681], | 30.00th=[41681], 40.00th=[41681], 50.00th=[41681], 60.00th=[46400], | 70.00th=[46400], 80.00th=[46400], 90.00th=[46400], 95.00th=[46400], | 99.00th=[46400], 99.50th=[46400], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 2048, max= 2048, per=21.84%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4689KiB/s (4801kB/s)(10.0MiB/2184msec) clat (msec): min=33, max=777, avg=202.39, stdev=273.44 lat (msec): min=33, max=777, avg=202.46, stdev=273.45 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 41], | 30.00th=[ 47], 40.00th=[ 54], 50.00th=[ 71], 60.00th=[ 92], | 70.00th=[ 142], 80.00th=[ 142], 90.00th=[ 651], 95.00th=[ 776], | 99.00th=[ 776], 99.50th=[ 776], 99.90th=[ 776], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 2048, max= 6144, per=31.20%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (msec) : 50=41.67%, 100=25.00%, 250=16.67%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=14.25%, ctx=447, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8237: Wed May 21 14:00:59 2025 read: IOPS=3, BW=3280KiB/s (3359kB/s)(6144KiB/1873msec) clat (usec): min=796, max=67994, avg=29717.36, stdev=25735.76 lat (usec): min=796, max=67994, avg=29718.33, stdev=25735.80 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 799], 10.00th=[ 799], 20.00th=[ 1012], | 30.00th=[ 1012], 40.00th=[30540], 50.00th=[30540], 60.00th=[36439], | 70.00th=[41681], 80.00th=[41681], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 4096, max= 6144, per=54.60%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=3, BW=3280KiB/s (3359kB/s)(6144KiB/1873msec) clat (msec): min=29, max=759, avg=268.18, stdev=355.95 lat (msec): min=29, max=759, avg=268.25, stdev=355.94 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 38], 50.00th=[ 38], 60.00th=[ 51], | 70.00th=[ 693], 80.00th=[ 693], 90.00th=[ 760], 95.00th=[ 760], | 99.00th=[ 760], 99.50th=[ 760], 99.90th=[ 760], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 2048, max= 4096, per=26.00%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 lat (usec) : 1000=8.33% lat (msec) : 2=8.33%, 50=50.00%, 100=16.67%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=11.91%, ctx=166, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8238: Wed May 21 14:00:59 2025 read: IOPS=2, BW=2726KiB/s (2792kB/s)(5120KiB/1878msec) clat (usec): min=513, max=738352, avg=154726.44, stdev=326561.76 lat (usec): min=515, max=738352, avg=154727.85, stdev=326561.31 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 515], 10.00th=[ 515], 20.00th=[ 515], | 30.00th=[ 693], 40.00th=[ 693], 50.00th=[ 816], 60.00th=[ 816], | 70.00th=[ 33162], 80.00th=[ 33162], 90.00th=[742392], 95.00th=[742392], | 99.00th=[742392], 99.50th=[742392], 99.90th=[742392], 99.95th=[742392], | 99.99th=[742392] bw ( KiB/s): min= 4096, max= 4096, per=43.68%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3817KiB/s (3908kB/s)(7168KiB/1878msec) clat (msec): min=24, max=683, avg=145.72, stdev=241.12 lat (msec): min=24, max=683, avg=145.80, stdev=241.13 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 27], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 42], 60.00th=[ 52], | 70.00th=[ 52], 80.00th=[ 153], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 2048, max= 4096, per=20.80%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 lat (usec) : 750=16.67%, 1000=8.33% lat (msec) : 50=41.67%, 100=8.33%, 250=8.33%, 750=16.67% cpu : usr=0.05%, sys=15.50%, ctx=433, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9377KiB/s (9602kB/s), 938KiB/s-6211KiB/s (960kB/s-6361kB/s), io=20.0MiB (20.0MB), run=1154-2184msec WRITE: bw=12.8MiB/s (13.4MB/s), 3280KiB/s-4689KiB/s (3359kB/s-4801kB/s), io=28.0MiB (29.4MB), run=1154-2184msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8230: Wed May 21 14:00:59 2025 read: IOPS=3, BW=3167KiB/s (3243kB/s)(7168KiB/2263msec) clat (msec): min=20, max=840, avg=267.61, stdev=335.79 lat (msec): min=20, max=840, avg=267.61, stdev=335.79 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 36], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 52], 60.00th=[ 236], | 70.00th=[ 236], 80.00th=[ 642], 90.00th=[ 844], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 2048, max= 6095, per=37.54%, avg=3397.00, stdev=2336.54, samples=3 iops : min= 2, max= 5, avg= 3.00, stdev= 1.73, samples=3 write: IOPS=2, BW=2262KiB/s (2317kB/s)(5120KiB/2263msec) clat (usec): min=36024, max=92955, avg=60976.88, stdev=24523.84 lat (usec): min=36074, max=93054, avg=61060.76, stdev=24541.27 clat percentiles (usec): | 1.00th=[35914], 5.00th=[35914], 10.00th=[35914], 20.00th=[35914], | 30.00th=[41681], 40.00th=[41681], 50.00th=[54789], 60.00th=[54789], | 70.00th=[79168], 80.00th=[79168], 90.00th=[92799], 95.00th=[92799], | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 2048, max= 2048, per=16.17%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=33.33%, 250=8.33%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=5.31%, ctx=60, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8231: Wed May 21 14:00:59 2025 read: IOPS=0, BW=947KiB/s (970kB/s)(2048KiB/2163msec) clat (usec): min=20043, max=27415, avg=23729.56, stdev=5212.60 lat (usec): min=20045, max=27417, avg=23731.46, stdev=5212.37 clat percentiles (usec): | 1.00th=[20055], 5.00th=[20055], 10.00th=[20055], 20.00th=[20055], | 30.00th=[20055], 40.00th=[20055], 50.00th=[20055], 60.00th=[27395], | 70.00th=[27395], 80.00th=[27395], 90.00th=[27395], 95.00th=[27395], | 99.00th=[27395], 99.50th=[27395], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 2048, max= 2048, per=22.63%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4734KiB/s (4848kB/s)(10.0MiB/2163msec) clat (msec): min=27, max=787, avg=210.13, stdev=284.70 lat (msec): min=27, max=787, avg=210.20, stdev=284.69 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 40], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 80], 60.00th=[ 80], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 701], 95.00th=[ 785], | 99.00th=[ 785], 99.50th=[ 785], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 2043, max= 6119, per=32.25%, avg=4086.00, stdev=2038.02, samples=3 iops : min= 1, max= 5, avg= 3.33, stdev= 2.08, samples=3 lat (msec) : 50=33.33%, 100=41.67%, 250=8.33%, 750=8.33%, 1000=8.33% cpu : usr=0.05%, sys=6.98%, ctx=77, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8232: Wed May 21 14:00:59 2025 read: IOPS=2, BW=2991KiB/s (3063kB/s)(6144KiB/2054msec) clat (msec): min=22, max=654, avg=163.67, stdev=243.93 lat (msec): min=22, max=654, avg=163.67, stdev=243.93 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 30], | 30.00th=[ 30], 40.00th=[ 60], 50.00th=[ 60], 60.00th=[ 83], | 70.00th=[ 134], 80.00th=[ 134], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 4096, max= 6144, per=56.58%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2991KiB/s (3063kB/s)(6144KiB/2054msec) clat (msec): min=21, max=829, avg=170.90, stdev=323.43 lat (msec): min=21, max=829, avg=170.98, stdev=323.43 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 37], | 70.00th=[ 78], 80.00th=[ 78], 90.00th=[ 827], 95.00th=[ 827], | 99.00th=[ 827], 99.50th=[ 827], 99.90th=[ 827], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 2031, max= 4096, per=20.17%, avg=2555.75, stdev=1026.86, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 lat (msec) : 50=50.00%, 100=25.00%, 250=8.33%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=7.70%, ctx=73, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8233: Wed May 21 14:00:59 2025 read: IOPS=2, BW=2315KiB/s (2370kB/s)(5120KiB/2212msec) clat (msec): min=23, max=772, avg=208.54, stdev=316.54 lat (msec): min=23, max=772, avg=208.54, stdev=316.54 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 23], | 30.00th=[ 56], 40.00th=[ 56], 50.00th=[ 88], 60.00th=[ 88], | 70.00th=[ 105], 80.00th=[ 105], 90.00th=[ 776], 95.00th=[ 776], | 99.00th=[ 776], 99.50th=[ 776], 99.90th=[ 776], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 2031, max= 4096, per=30.11%, avg=2725.00, stdev=1187.35, samples=3 iops : min= 1, max= 4, avg= 2.33, stdev= 1.53, samples=3 write: IOPS=3, BW=3241KiB/s (3318kB/s)(7168KiB/2212msec) clat (msec): min=24, max=684, avg=155.95, stdev=235.95 lat (msec): min=24, max=684, avg=156.02, stdev=235.96 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 30], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 81], 60.00th=[ 100], | 70.00th=[ 100], 80.00th=[ 125], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 2048, max= 6095, per=26.81%, avg=3397.00, stdev=2336.54, samples=3 iops : min= 2, max= 5, avg= 3.00, stdev= 1.73, samples=3 lat (msec) : 50=33.33%, 100=33.33%, 250=16.67%, 750=8.33%, 1000=8.33% cpu : usr=0.09%, sys=5.07%, ctx=59, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9050KiB/s (9267kB/s), 947KiB/s-3167KiB/s (970kB/s-3243kB/s), io=20.0MiB (20.0MB), run=2054-2263msec WRITE: bw=12.4MiB/s (12.0MB/s), 2262KiB/s-4734KiB/s (2317kB/s-4848kB/s), io=28.0MiB (29.4MB), run=2054-2263msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8244: Wed May 21 14:01:01 2025 read: IOPS=6, BW=24.5MiB/s (25.7MB/s)(12.0MiB/489msec) clat (msec): min=31, max=223, avg=147.74, stdev=102.17 lat (msec): min=31, max=223, avg=147.74, stdev=102.17 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 188], 50.00th=[ 188], 60.00th=[ 188], | 70.00th=[ 224], 80.00th=[ 224], 90.00th=[ 224], 95.00th=[ 224], | 99.00th=[ 224], 99.50th=[ 224], 99.90th=[ 224], 99.95th=[ 224], | 99.99th=[ 224] lat (msec) : 50=33.33%, 250=66.67% cpu : usr=0.20%, sys=26.43%, ctx=1630, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8245: Wed May 21 14:01:01 2025 read: IOPS=0, BW=3590KiB/s (3676kB/s)(4096KiB/1141msec) clat (nsec): min=265074k, max=265074k, avg=265074107.00, stdev= 0.00 lat (nsec): min=265076k, max=265076k, avg=265075860.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 266], 10.00th=[ 266], 20.00th=[ 266], | 30.00th=[ 266], 40.00th=[ 266], 50.00th=[ 266], 60.00th=[ 266], | 70.00th=[ 266], 80.00th=[ 266], 90.00th=[ 266], 95.00th=[ 266], | 99.00th=[ 266], 99.50th=[ 266], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 8192, max= 8192, per=57.05%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=7180KiB/s (7352kB/s)(8192KiB/1141msec) clat (msec): min=400, max=470, avg=435.27, stdev=49.17 lat (msec): min=400, max=470, avg=435.55, stdev=49.17 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 401], 10.00th=[ 401], 20.00th=[ 401], | 30.00th=[ 401], 40.00th=[ 401], 50.00th=[ 401], 60.00th=[ 472], | 70.00th=[ 472], 80.00th=[ 472], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 7086, max= 7086, per=24.67%, avg=7086.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=100.00% cpu : usr=0.00%, sys=25.70%, ctx=947, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8246: Wed May 21 14:01:01 2025 write: IOPS=3, BW=12.7MiB/s (13.3MB/s)(12.0MiB/947msec) clat (msec): min=213, max=422, avg=296.54, stdev=111.07 lat (msec): min=213, max=423, avg=296.85, stdev=111.17 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 213], 10.00th=[ 213], 20.00th=[ 213], | 30.00th=[ 213], 40.00th=[ 253], 50.00th=[ 253], 60.00th=[ 253], | 70.00th=[ 422], 80.00th=[ 422], 90.00th=[ 422], 95.00th=[ 422], | 99.00th=[ 422], 99.50th=[ 422], 99.90th=[ 422], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 8159, max= 8159, per=28.41%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=37.10%, ctx=53, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8247: Wed May 21 14:01:01 2025 write: IOPS=3, BW=14.4MiB/s (15.1MB/s)(12.0MiB/831msec) clat (msec): min=204, max=369, avg=266.25, stdev=90.14 lat (msec): min=204, max=370, avg=266.71, stdev=90.44 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 205], 10.00th=[ 205], 20.00th=[ 205], | 30.00th=[ 205], 40.00th=[ 226], 50.00th=[ 226], 60.00th=[ 226], | 70.00th=[ 372], 80.00th=[ 372], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 8126, max= 8126, per=28.30%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.12%, sys=41.69%, ctx=38, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=14.0MiB/s (14.7MB/s), 3590KiB/s-24.5MiB/s (3676kB/s-25.7MB/s), io=16.0MiB (16.8MB), run=489-1141msec WRITE: bw=28.0MiB/s (29.4MB/s), 7180KiB/s-14.4MiB/s (7352kB/s-15.1MB/s), io=32.0MiB (33.6MB), run=831-1141msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8249: Wed May 21 14:01:02 2025 read: IOPS=2, BW=11.1MiB/s (11.6MB/s)(12.0MiB/1084msec) clat (msec): min=232, max=536, avg=358.55, stdev=158.77 lat (msec): min=232, max=536, avg=358.55, stdev=158.77 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 234], 10.00th=[ 234], 20.00th=[ 234], | 30.00th=[ 234], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 305], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.09%, sys=13.11%, ctx=65, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8250: Wed May 21 14:01:02 2025 read: IOPS=0, BW=3368KiB/s (3449kB/s)(4096KiB/1216msec) clat (nsec): min=602872k, max=602872k, avg=602871917.00, stdev= 0.00 lat (nsec): min=602875k, max=602875k, avg=602874691.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 600], 5.00th=[ 600], 10.00th=[ 600], 20.00th=[ 600], | 30.00th=[ 600], 40.00th=[ 600], 50.00th=[ 600], 60.00th=[ 600], | 70.00th=[ 600], 80.00th=[ 600], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 8192, max= 8192, per=60.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6737KiB/s (6899kB/s)(8192KiB/1216msec) clat (msec): min=173, max=419, avg=296.74, stdev=174.24 lat (msec): min=173, max=420, avg=297.02, stdev=174.20 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 174], 50.00th=[ 174], 60.00th=[ 422], | 70.00th=[ 422], 80.00th=[ 422], 90.00th=[ 422], 95.00th=[ 422], | 99.00th=[ 422], 99.50th=[ 422], 99.90th=[ 422], 99.95th=[ 422], | 99.99th=[ 422] lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.08%, sys=12.18%, ctx=130, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8251: Wed May 21 14:01:02 2025 write: IOPS=2, BW=10.7MiB/s (11.2MB/s)(12.0MiB/1120msec) clat (msec): min=186, max=567, avg=370.18, stdev=190.84 lat (msec): min=186, max=567, avg=370.56, stdev=190.92 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 186], 10.00th=[ 186], 20.00th=[ 186], | 30.00th=[ 186], 40.00th=[ 359], 50.00th=[ 359], 60.00th=[ 359], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min=14628, max=14628, per=54.28%, avg=14628.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.09%, sys=6.70%, ctx=16, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8252: Wed May 21 14:01:02 2025 write: IOPS=3, BW=12.1MiB/s (12.7MB/s)(12.0MiB/989msec) clat (msec): min=111, max=519, avg=324.71, stdev=204.78 lat (msec): min=111, max=519, avg=324.98, stdev=204.79 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 342], 50.00th=[ 342], 60.00th=[ 342], | 70.00th=[ 518], 80.00th=[ 518], 90.00th=[ 518], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.10%, sys=13.16%, ctx=57, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.2MiB/s (13.8MB/s), 3368KiB/s-11.1MiB/s (3449kB/s-11.6MB/s), io=16.0MiB (16.8MB), run=1084-1216msec WRITE: bw=26.3MiB/s (27.6MB/s), 6737KiB/s-12.1MiB/s (6899kB/s-12.7MB/s), io=32.0MiB (33.6MB), run=989-1216msec