== sanity test 398b: DIO and buffer IO race ============== 03:52:16 (1749541936) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.13674 s, 44.3 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17424: Tue Jun 10 03:52:32 2025 read: IOPS=105, BW=423KiB/s (433kB/s)(5992KiB/14174msec) clat (usec): min=57, max=35493, avg=4248.58, stdev=3454.72 lat (usec): min=57, max=35495, avg=4250.50, stdev=3454.58 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 84], 10.00th=[ 111], 20.00th=[ 2278], | 30.00th=[ 2704], 40.00th=[ 3064], 50.00th=[ 3458], 60.00th=[ 4146], | 70.00th=[ 5276], 80.00th=[ 6456], 90.00th=[ 8225], 95.00th=[ 9765], | 99.00th=[15401], 99.50th=[21890], 99.90th=[33424], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 176, max= 528, per=24.70%, avg=417.93, stdev=80.20, samples=28 iops : min= 44, max= 132, avg=104.43, stdev=20.12, samples=28 write: IOPS=111, BW=444KiB/s (455kB/s)(6296KiB/14174msec) clat (usec): min=491, max=218061, avg=4890.76, stdev=6899.87 lat (usec): min=492, max=218064, avg=4891.77, stdev=6899.86 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 594], 10.00th=[ 668], 20.00th=[ 857], | 30.00th=[ 1237], 40.00th=[ 2671], 50.00th=[ 3949], 60.00th=[ 4948], | 70.00th=[ 6194], 80.00th=[ 7898], 90.00th=[ 10290], 95.00th=[ 12649], | 99.00th=[ 19792], 99.50th=[ 22414], 99.90th=[ 44827], 99.95th=[217056], | 99.99th=[217056] bw ( KiB/s): min= 192, max= 592, per=24.76%, avg=438.79, stdev=81.47, samples=28 iops : min= 48, max= 148, avg=109.64, stdev=20.36, samples=28 lat (usec) : 100=4.04%, 250=3.81%, 500=0.10%, 750=8.40%, 1000=4.92% lat (msec) : 2=5.83%, 4=27.21%, 10=37.70%, 20=7.32%, 50=0.65% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=23.12%, ctx=5941, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17425: Tue Jun 10 03:52:32 2025 read: IOPS=105, BW=424KiB/s (434kB/s)(5940KiB/14011msec) clat (usec): min=56, max=37242, avg=4282.58, stdev=3426.32 lat (usec): min=57, max=37243, avg=4283.50, stdev=3426.27 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 85], 10.00th=[ 117], 20.00th=[ 2245], | 30.00th=[ 2737], 40.00th=[ 3130], 50.00th=[ 3556], 60.00th=[ 4228], | 70.00th=[ 5407], 80.00th=[ 6587], 90.00th=[ 8029], 95.00th=[ 9372], | 99.00th=[14615], 99.50th=[18744], 99.90th=[35390], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 184, max= 632, per=25.02%, avg=423.36, stdev=87.06, samples=28 iops : min= 46, max= 158, avg=105.79, stdev=21.80, samples=28 write: IOPS=113, BW=453KiB/s (464kB/s)(6348KiB/14011msec) clat (usec): min=490, max=226512, avg=4748.12, stdev=6910.53 lat (usec): min=491, max=226513, avg=4749.71, stdev=6910.59 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 578], 10.00th=[ 644], 20.00th=[ 848], | 30.00th=[ 1156], 40.00th=[ 2540], 50.00th=[ 3818], 60.00th=[ 4948], | 70.00th=[ 6128], 80.00th=[ 7635], 90.00th=[ 10028], 95.00th=[ 12256], | 99.00th=[ 18220], 99.50th=[ 19530], 99.90th=[ 30016], 99.95th=[227541], | 99.99th=[227541] bw ( KiB/s): min= 232, max= 688, per=25.55%, avg=452.79, stdev=98.71, samples=28 iops : min= 58, max= 172, avg=113.14, stdev=24.71, samples=28 lat (usec) : 100=3.78%, 250=3.42%, 500=0.26%, 750=8.43%, 1000=5.31% lat (msec) : 2=6.18%, 4=27.34%, 10=38.05%, 20=6.77%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=24.13%, ctx=5889, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17426: Tue Jun 10 03:52:32 2025 read: IOPS=106, BW=425KiB/s (435kB/s)(6028KiB/14184msec) clat (usec): min=54, max=48356, avg=4188.25, stdev=3403.98 lat (usec): min=54, max=48358, avg=4189.07, stdev=3404.00 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 83], 10.00th=[ 103], 20.00th=[ 2212], | 30.00th=[ 2606], 40.00th=[ 3032], 50.00th=[ 3490], 60.00th=[ 4146], | 70.00th=[ 5211], 80.00th=[ 6521], 90.00th=[ 8455], 95.00th=[10028], | 99.00th=[13829], 99.50th=[16909], 99.90th=[34341], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 104, max= 576, per=24.95%, avg=422.21, stdev=95.33, samples=28 iops : min= 26, max= 144, avg=105.54, stdev=23.81, samples=28 write: IOPS=110, BW=441KiB/s (452kB/s)(6260KiB/14184msec) clat (usec): min=494, max=258889, avg=4958.75, stdev=7990.49 lat (usec): min=495, max=258890, avg=4959.88, stdev=7990.49 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 594], 10.00th=[ 652], 20.00th=[ 840], | 30.00th=[ 1188], 40.00th=[ 2802], 50.00th=[ 3949], 60.00th=[ 4883], | 70.00th=[ 6128], 80.00th=[ 7767], 90.00th=[ 10421], 95.00th=[ 13435], | 99.00th=[ 19006], 99.50th=[ 26084], 99.90th=[ 78119], 99.95th=[258999], | 99.99th=[258999] bw ( KiB/s): min= 120, max= 600, per=24.73%, avg=438.25, stdev=98.15, samples=28 iops : min= 30, max= 150, avg=109.54, stdev=24.53, samples=28 lat (usec) : 100=4.43%, 250=4.04%, 500=0.16%, 750=8.27%, 1000=5.40% lat (msec) : 2=4.75%, 4=27.41%, 10=37.04%, 20=7.98%, 50=0.46% lat (msec) : 100=0.03%, 500=0.03% cpu : usr=0.18%, sys=23.30%, ctx=5867, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17427: Tue Jun 10 03:52:32 2025 read: IOPS=106, BW=428KiB/s (438kB/s)(6052KiB/14145msec) clat (usec): min=56, max=233129, avg=4402.68, stdev=6896.13 lat (usec): min=56, max=233129, avg=4404.65, stdev=6896.10 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 84], 10.00th=[ 117], 20.00th=[ 2245], | 30.00th=[ 2704], 40.00th=[ 3097], 50.00th=[ 3589], 60.00th=[ 4113], | 70.00th=[ 5145], 80.00th=[ 6456], 90.00th=[ 8225], 95.00th=[ 9634], | 99.00th=[ 15795], 99.50th=[ 21627], 99.90th=[ 63177], 99.95th=[233833], | 99.99th=[233833] bw ( KiB/s): min= 208, max= 592, per=25.07%, avg=424.18, stdev=80.02, samples=28 iops : min= 52, max= 148, avg=105.96, stdev=20.03, samples=28 write: IOPS=110, BW=441KiB/s (451kB/s)(6236KiB/14145msec) clat (usec): min=510, max=35268, avg=4727.03, stdev=4375.07 lat (usec): min=511, max=35270, avg=4728.11, stdev=4375.30 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 586], 10.00th=[ 635], 20.00th=[ 816], | 30.00th=[ 1156], 40.00th=[ 2540], 50.00th=[ 3884], 60.00th=[ 4883], | 70.00th=[ 6194], 80.00th=[ 7767], 90.00th=[10683], 95.00th=[12649], | 99.00th=[18220], 99.50th=[22676], 99.90th=[34341], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 216, max= 648, per=24.68%, avg=437.29, stdev=93.47, samples=28 iops : min= 54, max= 162, avg=109.25, stdev=23.39, samples=28 lat (usec) : 100=3.87%, 250=3.48%, 500=0.36%, 750=8.56%, 1000=5.44% lat (msec) : 2=5.60%, 4=27.25%, 10=37.34%, 20=7.42%, 50=0.62% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.25%, sys=23.29%, ctx=5884, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1693KiB/s (1734kB/s), 423KiB/s-428KiB/s (433kB/s-438kB/s), io=23.4MiB (24.6MB), run=14011-14184msec WRITE: bw=1772KiB/s (1815kB/s), 441KiB/s-453KiB/s (451kB/s-464kB/s), io=24.6MiB (25.7MB), run=14011-14184msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17428: Tue Jun 10 03:52:54 2025 read: IOPS=42, BW=170KiB/s (174kB/s)(5992KiB/35255msec) clat (usec): min=1814, max=239155, avg=4112.22, stdev=6785.13 lat (usec): min=1814, max=239157, avg=4113.14, stdev=6785.15 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2089], 10.00th=[ 2180], 20.00th=[ 2343], | 30.00th=[ 2474], 40.00th=[ 2606], 50.00th=[ 2769], 60.00th=[ 2999], | 70.00th=[ 3425], 80.00th=[ 4621], 90.00th=[ 7832], 95.00th=[ 10421], | 99.00th=[ 16188], 99.50th=[ 19268], 99.90th=[ 29230], 99.95th=[240124], | 99.99th=[240124] bw ( KiB/s): min= 64, max= 288, per=25.15%, avg=169.79, stdev=62.64, samples=70 iops : min= 16, max= 72, avg=42.41, stdev=15.66, samples=70 write: IOPS=44, BW=179KiB/s (183kB/s)(6296KiB/35255msec) clat (usec): min=8886, max=80466, avg=18415.50, stdev=6658.76 lat (usec): min=8888, max=80468, avg=18416.67, stdev=6658.76 clat percentiles (usec): | 1.00th=[ 9896], 5.00th=[11076], 10.00th=[11863], 20.00th=[13042], | 30.00th=[14353], 40.00th=[15533], 50.00th=[16909], 60.00th=[18482], | 70.00th=[20317], 80.00th=[22676], 90.00th=[26870], 95.00th=[30802], | 99.00th=[39584], 99.50th=[45351], 99.90th=[67634], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 48, max= 248, per=25.24%, avg=178.47, stdev=50.42, samples=70 iops : min= 12, max= 62, avg=44.59, stdev=12.60, samples=70 lat (msec) : 2=1.24%, 4=35.90%, 10=9.60%, 20=37.01%, 50=16.02% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.15%, sys=5.93%, ctx=3490, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17429: Tue Jun 10 03:52:54 2025 read: IOPS=41, BW=167KiB/s (171kB/s)(5940KiB/35535msec) clat (usec): min=1719, max=232491, avg=4236.53, stdev=6956.83 lat (usec): min=1720, max=232495, avg=4237.47, stdev=6956.89 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2089], 10.00th=[ 2180], 20.00th=[ 2311], | 30.00th=[ 2474], 40.00th=[ 2606], 50.00th=[ 2769], 60.00th=[ 3032], | 70.00th=[ 3425], 80.00th=[ 4686], 90.00th=[ 7832], 95.00th=[ 11076], | 99.00th=[ 19006], 99.50th=[ 22152], 99.90th=[ 56361], 99.95th=[231736], | 99.99th=[231736] bw ( KiB/s): min= 16, max= 312, per=24.72%, avg=166.89, stdev=69.17, samples=71 iops : min= 4, max= 78, avg=41.65, stdev=17.32, samples=71 write: IOPS=44, BW=179KiB/s (183kB/s)(6348KiB/35535msec) clat (usec): min=8871, max=68359, avg=18362.26, stdev=6639.77 lat (usec): min=8872, max=68360, avg=18363.35, stdev=6639.75 clat percentiles (usec): | 1.00th=[ 9634], 5.00th=[10683], 10.00th=[11469], 20.00th=[12649], | 30.00th=[14091], 40.00th=[15664], 50.00th=[17171], 60.00th=[18744], | 70.00th=[20055], 80.00th=[22676], 90.00th=[27132], 95.00th=[31327], | 99.00th=[39584], 99.50th=[44303], 99.90th=[61604], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 56, max= 264, per=25.23%, avg=178.38, stdev=51.00, samples=71 iops : min= 14, max= 66, avg=44.52, stdev=12.79, samples=71 lat (msec) : 2=0.94%, 4=35.35%, 10=9.90%, 20=37.60%, 50=16.05% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.09%, sys=5.97%, ctx=3469, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17430: Tue Jun 10 03:52:54 2025 read: IOPS=42, BW=170KiB/s (174kB/s)(6028KiB/35415msec) clat (usec): min=1834, max=37012, avg=4154.35, stdev=3406.46 lat (usec): min=1835, max=37013, avg=4155.27, stdev=3406.44 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2073], 10.00th=[ 2180], 20.00th=[ 2343], | 30.00th=[ 2507], 40.00th=[ 2638], 50.00th=[ 2835], 60.00th=[ 3064], | 70.00th=[ 3621], 80.00th=[ 5080], 90.00th=[ 8225], 95.00th=[11076], | 99.00th=[17695], 99.50th=[21627], 99.90th=[34341], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 40, max= 296, per=25.22%, avg=170.24, stdev=58.87, samples=70 iops : min= 10, max= 74, avg=42.53, stdev=14.72, samples=70 write: IOPS=44, BW=177KiB/s (181kB/s)(6260KiB/35415msec) clat (msec): min=8, max=256, avg=18.56, stdev= 8.93 lat (msec): min=8, max=256, avg=18.56, stdev= 8.93 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 28], 95.00th=[ 32], | 99.00th=[ 40], 99.50th=[ 44], 99.90th=[ 81], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 48, max= 256, per=24.95%, avg=176.41, stdev=52.23, samples=70 iops : min= 12, max= 64, avg=44.07, stdev=13.06, samples=70 lat (msec) : 2=1.01%, 4=35.55%, 10=9.93%, 20=37.01%, 50=16.37% lat (msec) : 100=0.10%, 500=0.03% cpu : usr=0.13%, sys=6.01%, ctx=3495, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17431: Tue Jun 10 03:52:54 2025 read: IOPS=42, BW=171KiB/s (175kB/s)(6052KiB/35439msec) clat (usec): min=1769, max=60575, avg=4142.74, stdev=3708.36 lat (usec): min=1769, max=60576, avg=4143.67, stdev=3708.35 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2114], 10.00th=[ 2212], 20.00th=[ 2376], | 30.00th=[ 2507], 40.00th=[ 2638], 50.00th=[ 2802], 60.00th=[ 3064], | 70.00th=[ 3523], 80.00th=[ 4817], 90.00th=[ 8291], 95.00th=[11338], | 99.00th=[17957], 99.50th=[21890], 99.90th=[46924], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min= 32, max= 328, per=25.13%, avg=169.66, stdev=64.32, samples=70 iops : min= 8, max= 82, avg=42.29, stdev=16.12, samples=70 write: IOPS=43, BW=176KiB/s (180kB/s)(6236KiB/35439msec) clat (msec): min=8, max=223, avg=18.63, stdev= 8.21 lat (msec): min=8, max=223, avg=18.63, stdev= 8.21 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 28], 95.00th=[ 32], | 99.00th=[ 41], 99.50th=[ 42], 99.90th=[ 63], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 56, max= 264, per=24.81%, avg=175.37, stdev=50.36, samples=70 iops : min= 14, max= 66, avg=43.71, stdev=12.60, samples=70 lat (msec) : 2=1.07%, 4=35.94%, 10=9.80%, 20=36.52%, 50=16.54% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.12%, sys=5.97%, ctx=3483, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=676KiB/s (692kB/s), 167KiB/s-171KiB/s (171kB/s-175kB/s), io=23.4MiB (24.6MB), run=35255-35535msec WRITE: bw=707KiB/s (724kB/s), 176KiB/s-179KiB/s (180kB/s-183kB/s), io=24.6MiB (25.7MB), run=35255-35535msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17469: Tue Jun 10 03:52:57 2025 read: IOPS=126, BW=2026KiB/s (2075kB/s)(5920KiB/2922msec) clat (usec): min=57, max=20986, avg=3647.38, stdev=3023.64 lat (usec): min=57, max=20987, avg=3648.10, stdev=3023.65 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 75], 10.00th=[ 91], 20.00th=[ 131], | 30.00th=[ 2638], 40.00th=[ 3032], 50.00th=[ 3490], 60.00th=[ 4015], | 70.00th=[ 4490], 80.00th=[ 5407], 90.00th=[ 7242], 95.00th=[ 8455], | 99.00th=[16319], 99.50th=[19006], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 1920, max= 2176, per=25.25%, avg=2028.80, stdev=102.70, samples=5 iops : min= 120, max= 136, avg=126.80, stdev= 6.42, samples=5 write: IOPS=136, BW=2179KiB/s (2232kB/s)(6368KiB/2922msec) clat (usec): min=694, max=30272, avg=3881.71, stdev=3781.77 lat (usec): min=695, max=30272, avg=3882.71, stdev=3781.76 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 865], 10.00th=[ 938], 20.00th=[ 1139], | 30.00th=[ 1319], 40.00th=[ 1516], 50.00th=[ 2180], 60.00th=[ 3523], | 70.00th=[ 4883], 80.00th=[ 6390], 90.00th=[ 8979], 95.00th=[11076], | 99.00th=[17171], 99.50th=[21890], 99.90th=[30278], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 1888, max= 2528, per=26.12%, avg=2208.00, stdev=266.77, samples=5 iops : min= 118, max= 158, avg=138.00, stdev=16.67, samples=5 lat (usec) : 100=5.60%, 250=6.12%, 500=0.52%, 750=0.65%, 1000=6.51% lat (msec) : 2=18.23%, 4=24.35%, 10=33.85%, 20=3.78%, 50=0.39% cpu : usr=0.27%, sys=28.65%, ctx=1707, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17470: Tue Jun 10 03:52:57 2025 read: IOPS=120, BW=1927KiB/s (1973kB/s)(5744KiB/2981msec) clat (usec): min=61, max=33732, avg=4093.29, stdev=3450.72 lat (usec): min=62, max=33733, avg=4093.98, stdev=3450.73 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 85], 10.00th=[ 109], 20.00th=[ 151], | 30.00th=[ 2900], 40.00th=[ 3392], 50.00th=[ 3818], 60.00th=[ 4293], | 70.00th=[ 5080], 80.00th=[ 6194], 90.00th=[ 7701], 95.00th=[ 9241], | 99.00th=[16188], 99.50th=[23462], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 1664, max= 2336, per=24.14%, avg=1939.20, stdev=250.54, samples=5 iops : min= 104, max= 146, avg=121.20, stdev=15.66, samples=5 write: IOPS=137, BW=2195KiB/s (2248kB/s)(6544KiB/2981msec) clat (usec): min=651, max=20037, avg=3647.64, stdev=3472.42 lat (usec): min=652, max=20037, avg=3648.84, stdev=3472.30 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 840], 10.00th=[ 938], 20.00th=[ 1139], | 30.00th=[ 1254], 40.00th=[ 1450], 50.00th=[ 1762], 60.00th=[ 3490], | 70.00th=[ 4490], 80.00th=[ 6063], 90.00th=[ 8225], 95.00th=[10945], | 99.00th=[16909], 99.50th=[17433], 99.90th=[20055], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 2016, max= 2400, per=25.97%, avg=2195.20, stdev=179.03, samples=5 iops : min= 126, max= 150, avg=137.20, stdev=11.19, samples=5 lat (usec) : 100=3.65%, 250=6.25%, 500=0.65%, 750=0.52%, 1000=6.64% lat (msec) : 2=20.96%, 4=21.48%, 10=35.03%, 20=4.43%, 50=0.39% cpu : usr=0.30%, sys=29.73%, ctx=1671, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17471: Tue Jun 10 03:52:57 2025 read: IOPS=133, BW=2142KiB/s (2193kB/s)(6256KiB/2921msec) clat (usec): min=59, max=20975, avg=3909.67, stdev=3128.23 lat (usec): min=59, max=20976, avg=3910.66, stdev=3128.25 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 75], 10.00th=[ 102], 20.00th=[ 143], | 30.00th=[ 2704], 40.00th=[ 3294], 50.00th=[ 3720], 60.00th=[ 4293], | 70.00th=[ 5211], 80.00th=[ 6259], 90.00th=[ 7701], 95.00th=[ 8848], | 99.00th=[14615], 99.50th=[15533], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 1724, max= 2400, per=26.28%, avg=2111.20, stdev=250.45, samples=5 iops : min= 107, max= 150, avg=131.80, stdev=15.94, samples=5 write: IOPS=129, BW=2065KiB/s (2115kB/s)(6032KiB/2921msec) clat (usec): min=699, max=16752, avg=3631.66, stdev=3105.43 lat (usec): min=700, max=16753, avg=3632.77, stdev=3105.47 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 898], 10.00th=[ 996], 20.00th=[ 1139], | 30.00th=[ 1303], 40.00th=[ 1532], 50.00th=[ 2245], 60.00th=[ 3392], | 70.00th=[ 4424], 80.00th=[ 6390], 90.00th=[ 8455], 95.00th=[ 9765], | 99.00th=[12911], 99.50th=[15401], 99.90th=[16712], 99.95th=[16712], | 99.99th=[16712] bw ( KiB/s): min= 1888, max= 2432, per=24.90%, avg=2104.60, stdev=201.60, samples=5 iops : min= 118, max= 152, avg=131.40, stdev=12.60, samples=5 lat (usec) : 100=4.95%, 250=6.90%, 500=1.04%, 750=0.65%, 1000=4.69% lat (msec) : 2=18.62%, 4=24.22%, 10=35.16%, 20=3.65%, 50=0.13% cpu : usr=0.51%, sys=28.77%, ctx=1771, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17472: Tue Jun 10 03:52:57 2025 read: IOPS=135, BW=2170KiB/s (2222kB/s)(6032KiB/2780msec) clat (usec): min=67, max=20119, avg=3510.51, stdev=2958.71 lat (usec): min=67, max=20119, avg=3512.22, stdev=2957.59 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 78], 10.00th=[ 90], 20.00th=[ 123], | 30.00th=[ 2474], 40.00th=[ 3032], 50.00th=[ 3392], 60.00th=[ 3884], | 70.00th=[ 4490], 80.00th=[ 5473], 90.00th=[ 7177], 95.00th=[ 8455], | 99.00th=[13698], 99.50th=[19530], 99.90th=[20055], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 1980, max= 2496, per=26.36%, avg=2117.60, stdev=222.78, samples=5 iops : min= 123, max= 156, avg=132.20, stdev=14.04, samples=5 write: IOPS=140, BW=2250KiB/s (2304kB/s)(6256KiB/2780msec) clat (usec): min=677, max=22461, avg=3688.30, stdev=3515.84 lat (usec): min=678, max=22462, avg=3692.72, stdev=3514.06 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 840], 10.00th=[ 971], 20.00th=[ 1123], | 30.00th=[ 1270], 40.00th=[ 1418], 50.00th=[ 1876], 60.00th=[ 3097], | 70.00th=[ 4817], 80.00th=[ 6259], 90.00th=[ 8717], 95.00th=[10159], | 99.00th=[16581], 99.50th=[19006], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 1856, max= 2656, per=26.49%, avg=2239.00, stdev=287.40, samples=5 iops : min= 116, max= 166, avg=139.80, stdev=18.01, samples=5 lat (usec) : 100=5.73%, 250=7.42%, 500=0.13%, 750=0.65%, 1000=5.21% lat (msec) : 2=20.96%, 4=24.35%, 10=31.64%, 20=3.65%, 50=0.26% cpu : usr=0.11%, sys=29.18%, ctx=1711, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8035KiB/s (8228kB/s), 1927KiB/s-2170KiB/s (1973kB/s-2222kB/s), io=23.4MiB (24.5MB), run=2780-2981msec WRITE: bw=8454KiB/s (8656kB/s), 2065KiB/s-2250KiB/s (2115kB/s-2304kB/s), io=24.6MiB (25.8MB), run=2780-2981msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17474: Tue Jun 10 03:53:03 2025 read: IOPS=41, BW=660KiB/s (676kB/s)(5920KiB/8965msec) clat (usec): min=2037, max=28866, avg=4327.28, stdev=4009.78 lat (usec): min=2038, max=28868, avg=4328.17, stdev=4009.78 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2212], 10.00th=[ 2343], 20.00th=[ 2507], | 30.00th=[ 2638], 40.00th=[ 2769], 50.00th=[ 2900], 60.00th=[ 3064], | 70.00th=[ 3425], 80.00th=[ 4424], 90.00th=[ 7504], 95.00th=[12387], | 99.00th=[25560], 99.50th=[28705], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 319, max= 1152, per=24.24%, avg=647.35, stdev=220.22, samples=17 iops : min= 19, max= 72, avg=40.35, stdev=13.86, samples=17 write: IOPS=44, BW=710KiB/s (727kB/s)(6368KiB/8965msec) clat (usec): min=9313, max=47516, avg=18453.73, stdev=6039.19 lat (usec): min=9315, max=47518, avg=18455.33, stdev=6039.20 clat percentiles (usec): | 1.00th=[ 9765], 5.00th=[11207], 10.00th=[12256], 20.00th=[13173], | 30.00th=[14877], 40.00th=[16581], 50.00th=[17695], 60.00th=[19006], | 70.00th=[20055], 80.00th=[21627], 90.00th=[25035], 95.00th=[31065], | 99.00th=[40109], 99.50th=[44827], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 447, max= 896, per=24.85%, avg=698.18, stdev=174.23, samples=17 iops : min= 27, max= 56, avg=43.53, stdev=10.93, samples=17 lat (msec) : 4=37.11%, 10=8.59%, 20=37.24%, 50=17.06% cpu : usr=0.16%, sys=6.79%, ctx=896, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17475: Tue Jun 10 03:53:03 2025 read: IOPS=40, BW=643KiB/s (659kB/s)(5744KiB/8930msec) clat (usec): min=2042, max=29334, avg=4323.77, stdev=3717.33 lat (usec): min=2043, max=29335, avg=4324.74, stdev=3717.28 clat percentiles (usec): | 1.00th=[ 2147], 5.00th=[ 2278], 10.00th=[ 2343], 20.00th=[ 2507], | 30.00th=[ 2671], 40.00th=[ 2802], 50.00th=[ 2900], 60.00th=[ 3097], | 70.00th=[ 3425], 80.00th=[ 5276], 90.00th=[ 7832], 95.00th=[12518], | 99.00th=[20579], 99.50th=[25822], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 288, max= 1056, per=24.23%, avg=647.29, stdev=235.64, samples=17 iops : min= 18, max= 66, avg=40.35, stdev=14.76, samples=17 write: IOPS=45, BW=733KiB/s (750kB/s)(6544KiB/8930msec) clat (usec): min=10353, max=39788, avg=17904.89, stdev=5309.17 lat (usec): min=10354, max=39790, avg=17906.54, stdev=5309.36 clat percentiles (usec): | 1.00th=[10552], 5.00th=[11207], 10.00th=[12125], 20.00th=[13304], | 30.00th=[14615], 40.00th=[15664], 50.00th=[17171], 60.00th=[18482], | 70.00th=[19792], 80.00th=[21103], 90.00th=[25035], 95.00th=[28967], | 99.00th=[35390], 99.50th=[38011], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 448, max= 928, per=25.71%, avg=722.59, stdev=171.09, samples=17 iops : min= 28, max= 58, avg=45.06, stdev=10.80, samples=17 lat (msec) : 4=35.68%, 10=7.81%, 20=41.15%, 50=15.36% cpu : usr=0.08%, sys=6.85%, ctx=895, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17476: Tue Jun 10 03:53:03 2025 read: IOPS=45, BW=721KiB/s (739kB/s)(6256KiB/8671msec) clat (usec): min=1905, max=31946, avg=4214.34, stdev=3666.56 lat (usec): min=1906, max=31947, avg=4215.30, stdev=3666.54 clat percentiles (usec): | 1.00th=[ 2057], 5.00th=[ 2278], 10.00th=[ 2311], 20.00th=[ 2474], | 30.00th=[ 2606], 40.00th=[ 2737], 50.00th=[ 2900], 60.00th=[ 3097], | 70.00th=[ 3392], 80.00th=[ 4359], 90.00th=[ 7504], 95.00th=[11600], | 99.00th=[20317], 99.50th=[24511], 99.90th=[31851], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 320, max= 1344, per=27.41%, avg=732.18, stdev=330.48, samples=17 iops : min= 20, max= 84, avg=45.71, stdev=20.72, samples=17 write: IOPS=43, BW=696KiB/s (712kB/s)(6032KiB/8671msec) clat (usec): min=9858, max=44777, avg=18448.81, stdev=5989.61 lat (usec): min=9860, max=44778, avg=18450.63, stdev=5989.51 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11731], 10.00th=[12518], 20.00th=[13829], | 30.00th=[14877], 40.00th=[15795], 50.00th=[17171], 60.00th=[18744], | 70.00th=[19792], 80.00th=[21627], 90.00th=[25822], 95.00th=[32900], | 99.00th=[39060], 99.50th=[41681], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 416, max= 864, per=24.65%, avg=692.59, stdev=163.68, samples=17 iops : min= 26, max= 54, avg=43.24, stdev=10.29, samples=17 lat (msec) : 2=0.26%, 4=39.32%, 10=8.33%, 20=37.11%, 50=14.97% cpu : usr=0.18%, sys=7.02%, ctx=888, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17477: Tue Jun 10 03:53:03 2025 read: IOPS=42, BW=678KiB/s (694kB/s)(6032KiB/8894msec) clat (usec): min=1940, max=33608, avg=4405.65, stdev=4073.12 lat (usec): min=1941, max=33608, avg=4406.60, stdev=4073.08 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2245], 10.00th=[ 2343], 20.00th=[ 2474], | 30.00th=[ 2671], 40.00th=[ 2769], 50.00th=[ 2933], 60.00th=[ 3163], | 70.00th=[ 3458], 80.00th=[ 4293], 90.00th=[ 8717], 95.00th=[14091], | 99.00th=[22676], 99.50th=[23987], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 288, max= 1056, per=25.15%, avg=671.82, stdev=242.56, samples=17 iops : min= 18, max= 66, avg=41.88, stdev=15.15, samples=17 write: IOPS=43, BW=703KiB/s (720kB/s)(6256KiB/8894msec) clat (usec): min=9645, max=60292, avg=18401.47, stdev=6300.93 lat (usec): min=9646, max=60295, avg=18403.45, stdev=6300.83 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11207], 10.00th=[11994], 20.00th=[13698], | 30.00th=[14746], 40.00th=[16319], 50.00th=[17433], 60.00th=[18482], | 70.00th=[19530], 80.00th=[21627], 90.00th=[25035], 95.00th=[30016], | 99.00th=[41681], 99.50th=[51119], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 416, max= 960, per=24.98%, avg=701.94, stdev=191.46, samples=17 iops : min= 26, max= 60, avg=43.76, stdev=11.99, samples=17 lat (msec) : 2=0.26%, 4=38.02%, 10=7.03%, 20=39.71%, 50=14.71% lat (msec) : 100=0.26% cpu : usr=0.12%, sys=6.97%, ctx=913, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2672KiB/s (2736kB/s), 643KiB/s-721KiB/s (659kB/s-739kB/s), io=23.4MiB (24.5MB), run=8671-8965msec WRITE: bw=2811KiB/s (2878kB/s), 696KiB/s-733KiB/s (712kB/s-750kB/s), io=24.6MiB (25.8MB), run=8671-8965msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17492: Tue Jun 10 03:53:05 2025 read: IOPS=16, BW=16.7MiB/s (17.5MB/s)(7168KiB/420msec) clat (usec): min=396, max=64646, avg=21581.29, stdev=24621.93 lat (usec): min=397, max=64648, avg=21582.33, stdev=24622.13 clat percentiles (usec): | 1.00th=[ 396], 5.00th=[ 396], 10.00th=[ 396], 20.00th=[ 465], | 30.00th=[ 478], 40.00th=[ 478], 50.00th=[14877], 60.00th=[31065], | 70.00th=[31065], 80.00th=[39060], 90.00th=[64750], 95.00th=[64750], | 99.00th=[64750], 99.50th=[64750], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] write: IOPS=11, BW=11.9MiB/s (12.5MB/s)(5120KiB/420msec) clat (usec): min=25408, max=76979, avg=44795.68, stdev=20017.85 lat (usec): min=25419, max=77009, avg=44834.64, stdev=20025.02 clat percentiles (usec): | 1.00th=[25297], 5.00th=[25297], 10.00th=[25297], 20.00th=[25297], | 30.00th=[31589], 40.00th=[31589], 50.00th=[41681], 60.00th=[41681], | 70.00th=[47973], 80.00th=[47973], 90.00th=[77071], 95.00th=[77071], | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] lat (usec) : 500=25.00% lat (msec) : 20=8.33%, 50=50.00%, 100=16.67% cpu : usr=0.00%, sys=31.74%, ctx=615, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17494: Tue Jun 10 03:53:05 2025 read: IOPS=2, BW=2720KiB/s (2785kB/s)(2048KiB/753msec) clat (msec): min=81, max=104, avg=92.65, stdev=16.45 lat (msec): min=81, max=104, avg=92.65, stdev=16.45 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 82], 10.00th=[ 82], 20.00th=[ 82], | 30.00th=[ 82], 40.00th=[ 82], 50.00th=[ 82], 60.00th=[ 105], | 70.00th=[ 105], 80.00th=[ 105], 90.00th=[ 105], 95.00th=[ 105], | 99.00th=[ 105], 99.50th=[ 105], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 4079, max= 4079, per=15.00%, avg=4079.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.3MiB/s (13.9MB/s)(10.0MiB/753msec) clat (usec): min=26670, max=81772, avg=50856.68, stdev=21493.94 lat (usec): min=26745, max=81796, avg=50921.54, stdev=21491.20 clat percentiles (usec): | 1.00th=[26608], 5.00th=[26608], 10.00th=[26608], 20.00th=[27395], | 30.00th=[27919], 40.00th=[31851], 50.00th=[51119], 60.00th=[53740], | 70.00th=[71828], 80.00th=[71828], 90.00th=[76022], 95.00th=[81265], | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min=10199, max=10199, per=26.79%, avg=10199.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=58.33%, 250=8.33% cpu : usr=0.00%, sys=34.31%, ctx=374, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17496: Tue Jun 10 03:53:05 2025 read: IOPS=12, BW=12.0MiB/s (12.6MB/s)(6144KiB/498msec) clat (usec): min=330, max=27794, avg=7289.26, stdev=11467.43 lat (usec): min=331, max=27796, avg=7290.23, stdev=11467.56 clat percentiles (usec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 416], | 30.00th=[ 416], 40.00th=[ 457], 50.00th=[ 457], 60.00th=[ 490], | 70.00th=[14222], 80.00th=[14222], 90.00th=[27919], 95.00th=[27919], | 99.00th=[27919], 99.50th=[27919], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] write: IOPS=12, BW=12.0MiB/s (12.6MB/s)(6144KiB/498msec) clat (msec): min=20, max=188, avg=66.57, stdev=61.87 lat (msec): min=20, max=188, avg=66.62, stdev=61.87 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 34], | 30.00th=[ 34], 40.00th=[ 35], 50.00th=[ 35], 60.00th=[ 55], | 70.00th=[ 69], 80.00th=[ 69], 90.00th=[ 188], 95.00th=[ 188], | 99.00th=[ 188], 99.50th=[ 188], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] lat (usec) : 500=33.33% lat (msec) : 20=8.33%, 50=33.33%, 100=16.67%, 250=8.33% cpu : usr=0.00%, sys=28.97%, ctx=460, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17498: Tue Jun 10 03:53:05 2025 read: IOPS=9, BW=9.90MiB/s (10.4MB/s)(5120KiB/505msec) clat (usec): min=514, max=42054, avg=24771.68, stdev=22124.65 lat (usec): min=514, max=42055, avg=24772.84, stdev=22124.68 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 515], 10.00th=[ 515], 20.00th=[ 515], | 30.00th=[ 578], 40.00th=[ 578], 50.00th=[40109], 60.00th=[40109], | 70.00th=[40633], 80.00th=[40633], 90.00th=[42206], 95.00th=[42206], | 99.00th=[42206], 99.50th=[42206], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 8192, max= 8192, per=30.12%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.9MiB/s (14.5MB/s)(7168KiB/505msec) clat (usec): min=25647, max=92558, avg=53411.42, stdev=27391.86 lat (usec): min=25720, max=92619, avg=53474.00, stdev=27393.09 clat percentiles (usec): | 1.00th=[25560], 5.00th=[25560], 10.00th=[25560], 20.00th=[31327], | 30.00th=[31589], 40.00th=[31589], 50.00th=[42730], 60.00th=[65799], | 70.00th=[65799], 80.00th=[84411], 90.00th=[92799], 95.00th=[92799], | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min=14336, max=14336, per=37.65%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 50=58.33%, 100=25.00% cpu : usr=0.00%, sys=40.28%, ctx=580, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=26.6MiB/s (27.8MB/s), 2720KiB/s-16.7MiB/s (2785kB/s-17.5MB/s), io=20.0MiB (20.0MB), run=420-753msec WRITE: bw=37.2MiB/s (38.0MB/s), 11.9MiB/s-13.9MiB/s (12.5MB/s-14.5MB/s), io=28.0MiB (29.4MB), run=420-753msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17491: Tue Jun 10 03:53:05 2025 read: IOPS=11, BW=11.5MiB/s (12.1MB/s)(7168KiB/608msec) clat (msec): min=14, max=114, avg=45.82, stdev=36.22 lat (msec): min=14, max=114, avg=45.82, stdev=36.22 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 22], | 30.00th=[ 27], 40.00th=[ 27], 50.00th=[ 28], 60.00th=[ 41], | 70.00th=[ 41], 80.00th=[ 75], 90.00th=[ 115], 95.00th=[ 115], | 99.00th=[ 115], 99.50th=[ 115], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min=10240, max=10240, per=39.55%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8421KiB/s (8623kB/s)(5120KiB/608msec) clat (usec): min=24799, max=69623, avg=50983.80, stdev=16803.63 lat (usec): min=24836, max=69675, avg=51041.79, stdev=16810.38 clat percentiles (usec): | 1.00th=[24773], 5.00th=[24773], 10.00th=[24773], 20.00th=[24773], | 30.00th=[47973], 40.00th=[47973], 50.00th=[52167], 60.00th=[52167], | 70.00th=[60556], 80.00th=[60556], 90.00th=[69731], 95.00th=[69731], | 99.00th=[69731], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 6144, max= 6144, per=16.95%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=15.82%, ctx=59, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17493: Tue Jun 10 03:53:05 2025 read: IOPS=2, BW=2809KiB/s (2877kB/s)(2048KiB/729msec) clat (usec): min=17308, max=79730, avg=48519.69, stdev=44139.35 lat (usec): min=17310, max=79732, avg=48521.44, stdev=44138.80 clat percentiles (usec): | 1.00th=[17433], 5.00th=[17433], 10.00th=[17433], 20.00th=[17433], | 30.00th=[17433], 40.00th=[17433], 50.00th=[17433], 60.00th=[80217], | 70.00th=[80217], 80.00th=[80217], 90.00th=[80217], 95.00th=[80217], | 99.00th=[80217], 99.50th=[80217], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 4087, max= 4087, per=15.79%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.7MiB/s (14.4MB/s)(10.0MiB/729msec) clat (msec): min=23, max=130, avg=61.60, stdev=31.69 lat (msec): min=23, max=130, avg=61.66, stdev=31.69 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 32], | 30.00th=[ 41], 40.00th=[ 44], 50.00th=[ 51], 60.00th=[ 66], | 70.00th=[ 79], 80.00th=[ 79], 90.00th=[ 87], 95.00th=[ 131], | 99.00th=[ 131], 99.50th=[ 131], 99.90th=[ 131], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min=10219, max=10219, per=28.19%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=50.00%, 250=8.33% cpu : usr=0.00%, sys=15.52%, ctx=63, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17495: Tue Jun 10 03:53:05 2025 read: IOPS=7, BW=8042KiB/s (8235kB/s)(6144KiB/764msec) clat (msec): min=16, max=106, avg=52.06, stdev=35.17 lat (msec): min=16, max=106, avg=52.06, stdev=35.17 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 25], | 30.00th=[ 25], 40.00th=[ 34], 50.00th=[ 34], 60.00th=[ 50], | 70.00th=[ 83], 80.00th=[ 83], 90.00th=[ 107], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 4096, max= 4096, per=15.82%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=7, BW=8042KiB/s (8235kB/s)(6144KiB/764msec) clat (msec): min=27, max=162, avg=73.75, stdev=51.42 lat (msec): min=27, max=162, avg=73.87, stdev=51.41 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 30], | 30.00th=[ 30], 40.00th=[ 58], 50.00th=[ 58], 60.00th=[ 64], | 70.00th=[ 104], 80.00th=[ 104], 90.00th=[ 163], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 6144, max= 6144, per=16.95%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=25.00%, 250=25.00% cpu : usr=0.13%, sys=12.19%, ctx=64, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17497: Tue Jun 10 03:53:05 2025 read: IOPS=6, BW=6473KiB/s (6628kB/s)(5120KiB/791msec) clat (usec): min=13329, max=37012, avg=22458.27, stdev=10585.76 lat (usec): min=13330, max=37013, avg=22459.36, stdev=10585.84 clat percentiles (usec): | 1.00th=[13304], 5.00th=[13304], 10.00th=[13304], 20.00th=[13304], | 30.00th=[14746], 40.00th=[14746], 50.00th=[16909], 60.00th=[16909], | 70.00th=[30278], 80.00th=[30278], 90.00th=[36963], 95.00th=[36963], | 99.00th=[36963], 99.50th=[36963], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] write: IOPS=8, BW=9062KiB/s (9279kB/s)(7168KiB/791msec) clat (msec): min=27, max=181, avg=89.75, stdev=63.26 lat (msec): min=27, max=181, avg=89.81, stdev=63.26 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 28], | 30.00th=[ 61], 40.00th=[ 61], 50.00th=[ 66], 60.00th=[ 99], | 70.00th=[ 99], 80.00th=[ 169], 90.00th=[ 182], 95.00th=[ 182], | 99.00th=[ 182], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 8175, max= 8175, per=22.55%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=33.33%, 100=25.00%, 250=16.67% cpu : usr=0.13%, sys=10.25%, ctx=57, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.3MiB/s (26.5MB/s), 2809KiB/s-11.5MiB/s (2877kB/s-12.1MB/s), io=20.0MiB (20.0MB), run=608-791msec WRITE: bw=35.4MiB/s (37.1MB/s), 8042KiB/s-13.7MiB/s (8235kB/s-14.4MB/s), io=28.0MiB (29.4MB), run=608-791msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17506: Tue Jun 10 03:53:07 2025 read: IOPS=3, BW=13.1MiB/s (13.8MB/s)(12.0MiB/914msec) clat (msec): min=86, max=649, avg=301.00, stdev=304.16 lat (msec): min=86, max=649, avg=301.01, stdev=304.16 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 87], 10.00th=[ 87], 20.00th=[ 87], | 30.00th=[ 87], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 167], | 70.00th=[ 651], 80.00th=[ 651], 90.00th=[ 651], 95.00th=[ 651], | 99.00th=[ 651], 99.50th=[ 651], 99.90th=[ 651], 99.95th=[ 651], | 99.99th=[ 651] lat (msec) : 100=33.33%, 250=33.33%, 750=33.33% cpu : usr=0.11%, sys=23.00%, ctx=1483, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17507: Tue Jun 10 03:53:07 2025 read: IOPS=0, BW=3908KiB/s (4002kB/s)(4096KiB/1048msec) clat (nsec): min=627469k, max=627469k, avg=627468935.00, stdev= 0.00 lat (nsec): min=627472k, max=627472k, avg=627471596.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 625], 5.00th=[ 625], 10.00th=[ 625], 20.00th=[ 625], | 30.00th=[ 625], 40.00th=[ 625], 50.00th=[ 625], 60.00th=[ 625], | 70.00th=[ 625], 80.00th=[ 625], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 8192, max= 8192, per=52.40%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=7817KiB/s (8004kB/s)(8192KiB/1048msec) clat (msec): min=134, max=256, avg=195.68, stdev=85.93 lat (msec): min=135, max=256, avg=195.82, stdev=85.96 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 257], | 70.00th=[ 257], 80.00th=[ 257], 90.00th=[ 257], 95.00th=[ 257], | 99.00th=[ 257], 99.50th=[ 257], 99.90th=[ 257], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 8192, max= 8192, per=27.98%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=13.28%, ctx=555, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17509: Tue Jun 10 03:53:07 2025 write: IOPS=2, BW=11.0MiB/s (11.5MB/s)(12.0MiB/1090msec) clat (msec): min=127, max=705, avg=358.59, stdev=305.63 lat (msec): min=128, max=705, avg=358.85, stdev=305.64 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 128], 10.00th=[ 128], 20.00th=[ 128], | 30.00th=[ 128], 40.00th=[ 243], 50.00th=[ 243], 60.00th=[ 243], | 70.00th=[ 709], 80.00th=[ 709], 90.00th=[ 709], 95.00th=[ 709], | 99.00th=[ 709], 99.50th=[ 709], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min=16318, max=16318, per=55.73%, avg=16318.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=24.24%, ctx=179, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17511: Tue Jun 10 03:53:07 2025 write: IOPS=2, BW=10.7MiB/s (11.2MB/s)(12.0MiB/1119msec) clat (msec): min=99, max=841, avg=360.04, stdev=417.26 lat (msec): min=99, max=841, avg=360.29, stdev=417.25 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 100], 10.00th=[ 100], 20.00th=[ 100], | 30.00th=[ 100], 40.00th=[ 140], 50.00th=[ 140], 60.00th=[ 140], | 70.00th=[ 844], 80.00th=[ 844], 90.00th=[ 844], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min=16384, max=16384, per=55.95%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=24.33%, ctx=81, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.3MiB/s (16.0MB/s), 3908KiB/s-13.1MiB/s (4002kB/s-13.8MB/s), io=16.0MiB (16.8MB), run=914-1048msec WRITE: bw=28.6MiB/s (29.0MB/s), 7817KiB/s-11.0MiB/s (8004kB/s-11.5MB/s), io=32.0MiB (33.6MB), run=1048-1119msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17505: Tue Jun 10 03:53:07 2025 read: IOPS=2, BW=10.2MiB/s (10.7MB/s)(12.0MiB/1173msec) clat (msec): min=69, max=817, avg=383.90, stdev=388.23 lat (msec): min=69, max=817, avg=383.91, stdev=388.23 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 70], 10.00th=[ 70], 20.00th=[ 70], | 30.00th=[ 70], 40.00th=[ 264], 50.00th=[ 264], 60.00th=[ 264], | 70.00th=[ 818], 80.00th=[ 818], 90.00th=[ 818], 95.00th=[ 818], | 99.00th=[ 818], 99.50th=[ 818], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 8047, max= 8047, per=61.15%, avg=8047.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=5.80%, ctx=47, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17508: Tue Jun 10 03:53:07 2025 read: IOPS=0, BW=3290KiB/s (3369kB/s)(4096KiB/1245msec) clat (nsec): min=808641k, max=808641k, avg=808640592.00, stdev= 0.00 lat (nsec): min=808643k, max=808643k, avg=808642908.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 810], 5.00th=[ 810], 10.00th=[ 810], 20.00th=[ 810], | 30.00th=[ 810], 40.00th=[ 810], 50.00th=[ 810], 60.00th=[ 810], | 70.00th=[ 810], 80.00th=[ 810], 90.00th=[ 810], 95.00th=[ 810], | 99.00th=[ 810], 99.50th=[ 810], 99.90th=[ 810], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 8126, max= 8126, per=61.75%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6580KiB/s (6738kB/s)(8192KiB/1245msec) clat (msec): min=132, max=246, avg=189.50, stdev=81.23 lat (msec): min=132, max=247, avg=189.79, stdev=81.18 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 132], 10.00th=[ 132], 20.00th=[ 132], | 30.00th=[ 132], 40.00th=[ 132], 50.00th=[ 132], 60.00th=[ 247], | 70.00th=[ 247], 80.00th=[ 247], 90.00th=[ 247], 95.00th=[ 247], | 99.00th=[ 247], 99.50th=[ 247], 99.90th=[ 247], 99.95th=[ 247], | 99.99th=[ 247] lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.00%, sys=8.36%, ctx=69, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17510: Tue Jun 10 03:53:07 2025 write: IOPS=2, BW=9.84MiB/s (10.3MB/s)(12.0MiB/1220msec) clat (msec): min=66, max=879, avg=391.11, stdev=430.69 lat (msec): min=66, max=880, avg=391.38, stdev=430.68 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 67], | 30.00th=[ 67], 40.00th=[ 228], 50.00th=[ 228], 60.00th=[ 228], | 70.00th=[ 877], 80.00th=[ 877], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 8192, max= 8192, per=31.13%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=7.22%, ctx=54, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17512: Tue Jun 10 03:53:07 2025 write: IOPS=2, BW=9.79MiB/s (10.3MB/s)(12.0MiB/1226msec) clat (msec): min=69, max=568, avg=389.95, stdev=277.80 lat (msec): min=70, max=568, avg=390.25, stdev=277.81 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 70], 10.00th=[ 70], 20.00th=[ 70], | 30.00th=[ 70], 40.00th=[ 531], 50.00th=[ 531], 60.00th=[ 531], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min=16286, max=16286, per=61.88%, avg=16286.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=66.67% cpu : usr=0.24%, sys=9.96%, ctx=58, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.9MiB/s (13.5MB/s), 3290KiB/s-10.2MiB/s (3369kB/s-10.7MB/s), io=16.0MiB (16.8MB), run=1173-1245msec WRITE: bw=25.7MiB/s (26.0MB/s), 6580KiB/s-9.84MiB/s (6738kB/s-10.3MB/s), io=32.0MiB (33.6MB), run=1220-1245msec