== sanity test 398b: DIO and buffer IO race ============== 09:58:19 (1746799099) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.13476 s, 44.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7402: Fri May 9 09:58:34 2025 read: IOPS=130, BW=522KiB/s (535kB/s)(5992KiB/11478msec) clat (usec): min=55, max=165078, avg=3446.89, stdev=5220.64 lat (usec): min=55, max=165079, avg=3448.16, stdev=5220.31 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 71], 10.00th=[ 80], 20.00th=[ 1745], | 30.00th=[ 2114], 40.00th=[ 2409], 50.00th=[ 2704], 60.00th=[ 3130], | 70.00th=[ 4080], 80.00th=[ 5211], 90.00th=[ 6521], 95.00th=[ 8094], | 99.00th=[ 12649], 99.50th=[ 15270], 99.90th=[ 60031], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 312, max= 824, per=25.05%, avg=519.55, stdev=105.53, samples=22 iops : min= 78, max= 206, avg=129.86, stdev=26.38, samples=22 write: IOPS=137, BW=549KiB/s (562kB/s)(6296KiB/11478msec) clat (usec): min=454, max=47594, avg=3936.63, stdev=3897.39 lat (usec): min=455, max=47595, avg=3937.48, stdev=3897.42 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 537], 10.00th=[ 553], 20.00th=[ 619], | 30.00th=[ 889], 40.00th=[ 2073], 50.00th=[ 3228], 60.00th=[ 4047], | 70.00th=[ 5014], 80.00th=[ 6325], 90.00th=[ 8717], 95.00th=[10945], | 99.00th=[16450], 99.50th=[21890], 99.90th=[34341], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 368, max= 712, per=25.02%, avg=543.18, stdev=88.37, samples=22 iops : min= 92, max= 178, avg=135.77, stdev=22.09, samples=22 lat (usec) : 100=8.27%, 250=0.52%, 500=0.62%, 750=13.25%, 1000=2.60% lat (msec) : 2=8.07%, 4=30.86%, 10=31.35%, 20=4.04%, 50=0.36% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.20%, sys=23.46%, ctx=5362, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7403: Fri May 9 09:58:34 2025 read: IOPS=128, BW=513KiB/s (525kB/s)(5940KiB/11576msec) clat (usec): min=51, max=134942, avg=3333.27, stdev=4361.33 lat (usec): min=52, max=134944, avg=3334.24, stdev=4361.34 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 70], 10.00th=[ 79], 20.00th=[ 1647], | 30.00th=[ 2089], 40.00th=[ 2376], 50.00th=[ 2704], 60.00th=[ 3097], | 70.00th=[ 3851], 80.00th=[ 5145], 90.00th=[ 6849], 95.00th=[ 7963], | 99.00th=[ 10814], 99.50th=[ 14091], 99.90th=[ 26608], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 16, max= 664, per=24.66%, avg=511.48, stdev=133.79, samples=23 iops : min= 4, max= 166, avg=127.78, stdev=33.47, samples=23 write: IOPS=137, BW=548KiB/s (562kB/s)(6348KiB/11576msec) clat (usec): min=481, max=36408, avg=4075.99, stdev=4089.10 lat (usec): min=482, max=36409, avg=4077.26, stdev=4088.64 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 603], | 30.00th=[ 865], 40.00th=[ 2114], 50.00th=[ 3326], 60.00th=[ 4146], | 70.00th=[ 5145], 80.00th=[ 6652], 90.00th=[ 8848], 95.00th=[10814], | 99.00th=[19268], 99.50th=[25297], 99.90th=[32900], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 64, max= 752, per=25.08%, avg=544.43, stdev=131.99, samples=23 iops : min= 16, max= 188, avg=136.04, stdev=32.99, samples=23 lat (usec) : 100=8.37%, 250=0.91%, 500=0.72%, 750=13.44%, 1000=2.64% lat (msec) : 2=7.58%, 4=30.73%, 10=31.38%, 20=3.55%, 50=0.65% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=22.62%, ctx=5332, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7404: Fri May 9 09:58:34 2025 read: IOPS=130, BW=521KiB/s (534kB/s)(6028KiB/11563msec) clat (usec): min=54, max=34255, avg=3211.45, stdev=2743.93 lat (usec): min=54, max=34255, avg=3212.03, stdev=2743.94 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 69], 10.00th=[ 74], 20.00th=[ 133], | 30.00th=[ 2040], 40.00th=[ 2343], 50.00th=[ 2737], 60.00th=[ 3195], | 70.00th=[ 3916], 80.00th=[ 5145], 90.00th=[ 6652], 95.00th=[ 7701], | 99.00th=[10290], 99.50th=[11731], 99.90th=[28967], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 184, max= 744, per=25.14%, avg=521.39, stdev=125.12, samples=23 iops : min= 46, max= 186, avg=130.35, stdev=31.28, samples=23 write: IOPS=135, BW=541KiB/s (554kB/s)(6260KiB/11563msec) clat (usec): min=454, max=216451, avg=4221.08, stdev=6707.94 lat (usec): min=454, max=216453, avg=4221.79, stdev=6707.97 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 586], | 30.00th=[ 791], 40.00th=[ 1909], 50.00th=[ 3228], 60.00th=[ 4080], | 70.00th=[ 5211], 80.00th=[ 6980], 90.00th=[ 9372], 95.00th=[ 11469], | 99.00th=[ 17433], 99.50th=[ 22938], 99.90th=[ 34341], 99.95th=[217056], | 99.99th=[217056] bw ( KiB/s): min= 248, max= 736, per=24.88%, avg=540.17, stdev=103.97, samples=23 iops : min= 62, max= 184, avg=135.04, stdev=25.99, samples=23 lat (usec) : 100=9.34%, 250=0.68%, 500=0.55%, 750=14.45%, 1000=2.34% lat (msec) : 2=7.16%, 4=30.14%, 10=30.53%, 20=4.30%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=22.63%, ctx=5396, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7405: Fri May 9 09:58:34 2025 read: IOPS=132, BW=530KiB/s (543kB/s)(6052KiB/11423msec) clat (usec): min=55, max=171705, avg=3438.04, stdev=5173.74 lat (usec): min=55, max=171706, avg=3438.70, stdev=5173.69 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 70], 10.00th=[ 79], 20.00th=[ 1778], | 30.00th=[ 2114], 40.00th=[ 2409], 50.00th=[ 2769], 60.00th=[ 3294], | 70.00th=[ 3916], 80.00th=[ 5080], 90.00th=[ 6521], 95.00th=[ 8225], | 99.00th=[ 12649], 99.50th=[ 19268], 99.90th=[ 29492], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 296, max= 808, per=25.16%, avg=521.82, stdev=114.82, samples=22 iops : min= 74, max= 202, avg=130.45, stdev=28.70, samples=22 write: IOPS=136, BW=546KiB/s (559kB/s)(6236KiB/11423msec) clat (usec): min=446, max=56997, avg=3914.86, stdev=4298.87 lat (usec): min=447, max=56997, avg=3916.07, stdev=4298.83 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 594], | 30.00th=[ 791], 40.00th=[ 1680], 50.00th=[ 3032], 60.00th=[ 3851], | 70.00th=[ 5014], 80.00th=[ 6521], 90.00th=[ 8848], 95.00th=[11076], | 99.00th=[16581], 99.50th=[23462], 99.90th=[52691], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 344, max= 752, per=24.87%, avg=540.00, stdev=103.83, samples=22 iops : min= 86, max= 188, avg=135.00, stdev=25.96, samples=22 lat (usec) : 100=7.98%, 250=0.94%, 500=0.72%, 750=14.06%, 1000=2.64% lat (msec) : 2=8.04%, 4=31.67%, 10=29.43%, 20=4.00%, 50=0.42% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.24%, sys=22.92%, ctx=5376, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2074KiB/s (2124kB/s), 513KiB/s-530KiB/s (525kB/s-543kB/s), io=23.4MiB (24.6MB), run=11423-11576msec WRITE: bw=2172KiB/s (2224kB/s), 541KiB/s-549KiB/s (554kB/s-562kB/s), io=24.6MiB (25.7MB), run=11423-11576msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7397: Fri May 9 09:58:55 2025 read: IOPS=46, BW=184KiB/s (189kB/s)(5992KiB/32506msec) clat (usec): min=1499, max=169555, avg=3299.16, stdev=5237.49 lat (usec): min=1499, max=169557, avg=3299.82, stdev=5237.51 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1729], 10.00th=[ 1778], 20.00th=[ 1860], | 30.00th=[ 1926], 40.00th=[ 2008], 50.00th=[ 2114], 60.00th=[ 2245], | 70.00th=[ 2507], 80.00th=[ 3228], 90.00th=[ 6325], 95.00th=[ 8979], | 99.00th=[ 15533], 99.50th=[ 20317], 99.90th=[ 35914], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 56, max= 320, per=25.24%, avg=184.28, stdev=70.89, samples=65 iops : min= 14, max= 80, avg=46.00, stdev=17.74, samples=65 write: IOPS=48, BW=194KiB/s (198kB/s)(6296KiB/32506msec) clat (usec): min=9073, max=80437, avg=17421.26, stdev=6291.61 lat (usec): min=9074, max=80438, avg=17422.02, stdev=6291.61 clat percentiles (usec): | 1.00th=[ 9896], 5.00th=[10683], 10.00th=[11076], 20.00th=[11994], | 30.00th=[13304], 40.00th=[15139], 50.00th=[16450], 60.00th=[17695], | 70.00th=[18744], 80.00th=[20317], 90.00th=[26608], 95.00th=[30278], | 99.00th=[36963], 99.50th=[39060], 99.90th=[44827], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 48, max= 272, per=25.33%, avg=193.51, stdev=57.64, samples=65 iops : min= 12, max= 68, avg=48.31, stdev=14.40, samples=65 lat (msec) : 2=19.27%, 4=21.29%, 10=6.80%, 20=41.47%, 50=11.10% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.09%, sys=5.39%, ctx=3538, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7398: Fri May 9 09:58:55 2025 read: IOPS=45, BW=181KiB/s (185kB/s)(5940KiB/32878msec) clat (usec): min=1411, max=170990, avg=3252.54, stdev=5205.60 lat (usec): min=1411, max=170991, avg=3253.20, stdev=5205.61 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 1713], 10.00th=[ 1778], 20.00th=[ 1860], | 30.00th=[ 1942], 40.00th=[ 2024], 50.00th=[ 2114], 60.00th=[ 2245], | 70.00th=[ 2507], 80.00th=[ 3458], 90.00th=[ 6456], 95.00th=[ 8586], | 99.00th=[ 12911], 99.50th=[ 15795], 99.90th=[ 50070], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 16, max= 328, per=24.82%, avg=181.15, stdev=76.73, samples=65 iops : min= 4, max= 82, avg=45.18, stdev=19.21, samples=65 write: IOPS=48, BW=193KiB/s (198kB/s)(6348KiB/32878msec) clat (usec): min=9265, max=81101, avg=17581.27, stdev=6416.74 lat (usec): min=9266, max=81102, avg=17582.54, stdev=6416.66 clat percentiles (usec): | 1.00th=[ 9896], 5.00th=[10552], 10.00th=[11076], 20.00th=[12125], | 30.00th=[13698], 40.00th=[15401], 50.00th=[16712], 60.00th=[17695], | 70.00th=[18744], 80.00th=[20579], 90.00th=[26084], 95.00th=[30540], | 99.00th=[36963], 99.50th=[41681], 99.90th=[72877], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 56, max= 264, per=25.26%, avg=192.97, stdev=55.81, samples=65 iops : min= 14, max= 66, avg=48.14, stdev=13.92, samples=65 lat (msec) : 2=17.58%, 4=22.62%, 10=7.49%, 20=40.53%, 50=11.65% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.09%, sys=5.31%, ctx=3514, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7399: Fri May 9 09:58:55 2025 read: IOPS=46, BW=186KiB/s (191kB/s)(6028KiB/32331msec) clat (usec): min=1534, max=51266, avg=3335.51, stdev=3418.15 lat (usec): min=1534, max=51267, avg=3336.40, stdev=3418.08 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 1696], 10.00th=[ 1762], 20.00th=[ 1860], | 30.00th=[ 1942], 40.00th=[ 2008], 50.00th=[ 2114], 60.00th=[ 2278], | 70.00th=[ 2638], 80.00th=[ 3687], 90.00th=[ 6849], 95.00th=[ 9503], | 99.00th=[15008], 99.50th=[22676], 99.90th=[50070], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 48, max= 336, per=25.50%, avg=186.17, stdev=68.18, samples=64 iops : min= 12, max= 84, avg=46.48, stdev=17.01, samples=64 write: IOPS=48, BW=194KiB/s (198kB/s)(6260KiB/32331msec) clat (msec): min=8, max=215, avg=17.36, stdev= 7.82 lat (msec): min=8, max=215, avg=17.36, stdev= 7.82 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 27], 95.00th=[ 30], | 99.00th=[ 39], 99.50th=[ 40], 99.90th=[ 55], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 48, max= 280, per=25.25%, avg=192.92, stdev=58.69, samples=64 iops : min= 12, max= 70, avg=48.17, stdev=14.69, samples=64 lat (msec) : 2=18.78%, 4=21.26%, 10=7.58%, 20=41.83%, 50=10.45% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.09%, sys=5.34%, ctx=3520, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7400: Fri May 9 09:58:55 2025 read: IOPS=46, BW=186KiB/s (191kB/s)(6052KiB/32497msec) clat (usec): min=1527, max=39644, avg=3217.99, stdev=3046.39 lat (usec): min=1528, max=39645, avg=3218.62, stdev=3046.38 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 1729], 10.00th=[ 1778], 20.00th=[ 1860], | 30.00th=[ 1926], 40.00th=[ 2008], 50.00th=[ 2089], 60.00th=[ 2245], | 70.00th=[ 2540], 80.00th=[ 3490], 90.00th=[ 6390], 95.00th=[ 9372], | 99.00th=[13435], 99.50th=[19006], 99.90th=[35914], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 32, max= 344, per=25.41%, avg=185.48, stdev=70.20, samples=64 iops : min= 8, max= 86, avg=46.27, stdev=17.58, samples=64 write: IOPS=47, BW=192KiB/s (197kB/s)(6236KiB/32497msec) clat (msec): min=9, max=195, avg=17.63, stdev= 7.79 lat (msec): min=9, max=195, avg=17.63, stdev= 7.79 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 27], 95.00th=[ 31], | 99.00th=[ 37], 99.50th=[ 42], 99.90th=[ 73], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 56, max= 256, per=25.01%, avg=191.11, stdev=56.58, samples=64 iops : min= 14, max= 64, avg=47.67, stdev=14.17, samples=64 lat (msec) : 2=19.56%, 4=20.83%, 10=7.39%, 20=40.92%, 50=11.17% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.07%, sys=5.41%, ctx=3500, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=730KiB/s (748kB/s), 181KiB/s-186KiB/s (185kB/s-191kB/s), io=23.4MiB (24.6MB), run=32331-32878msec WRITE: bw=765KiB/s (783kB/s), 192KiB/s-194KiB/s (197kB/s-198kB/s), io=24.6MiB (25.7MB), run=32331-32878msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7450: Fri May 9 09:58:58 2025 read: IOPS=146, BW=2343KiB/s (2399kB/s)(5920KiB/2527msec) clat (usec): min=59, max=34593, avg=3039.56, stdev=3553.26 lat (usec): min=59, max=34593, avg=3040.11, stdev=3553.32 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 72], 10.00th=[ 74], 20.00th=[ 86], | 30.00th=[ 149], 40.00th=[ 2442], 50.00th=[ 2802], 60.00th=[ 3294], | 70.00th=[ 3949], 80.00th=[ 4883], 90.00th=[ 6128], 95.00th=[ 7111], | 99.00th=[10683], 99.50th=[33424], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 1568, max= 2848, per=24.80%, avg=2336.00, stdev=533.55, samples=5 iops : min= 98, max= 178, avg=146.00, stdev=33.35, samples=5 write: IOPS=157, BW=2520KiB/s (2580kB/s)(6368KiB/2527msec) clat (usec): min=651, max=33958, avg=3400.57, stdev=3702.20 lat (usec): min=652, max=33959, avg=3402.15, stdev=3701.75 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 750], 10.00th=[ 799], 20.00th=[ 848], | 30.00th=[ 938], 40.00th=[ 1090], 50.00th=[ 2024], 60.00th=[ 3163], | 70.00th=[ 4146], 80.00th=[ 5669], 90.00th=[ 7635], 95.00th=[ 9634], | 99.00th=[20579], 99.50th=[23725], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 1792, max= 3360, per=25.45%, avg=2521.60, stdev=725.98, samples=5 iops : min= 112, max= 210, avg=157.60, stdev=45.37, samples=5 lat (usec) : 100=12.76%, 250=2.47%, 500=0.52%, 750=2.73%, 1000=16.28% lat (msec) : 2=7.03%, 4=27.86%, 10=27.47%, 20=1.95%, 50=0.91% cpu : usr=0.00%, sys=27.59%, ctx=1377, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7451: Fri May 9 09:58:58 2025 read: IOPS=141, BW=2259KiB/s (2313kB/s)(5744KiB/2543msec) clat (usec): min=59, max=26838, avg=3199.73, stdev=3139.42 lat (usec): min=60, max=26839, avg=3200.32, stdev=3139.45 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 72], 10.00th=[ 75], 20.00th=[ 88], | 30.00th=[ 1975], 40.00th=[ 2409], 50.00th=[ 2868], 60.00th=[ 3359], | 70.00th=[ 4293], 80.00th=[ 5080], 90.00th=[ 6718], 95.00th=[ 7898], | 99.00th=[13566], 99.50th=[22152], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 1152, max= 3104, per=23.78%, avg=2240.00, stdev=745.33, samples=5 iops : min= 72, max= 194, avg=140.00, stdev=46.58, samples=5 write: IOPS=160, BW=2573KiB/s (2635kB/s)(6544KiB/2543msec) clat (usec): min=645, max=42105, avg=3373.42, stdev=4584.92 lat (usec): min=646, max=42107, avg=3374.39, stdev=4584.93 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 717], 10.00th=[ 750], 20.00th=[ 824], | 30.00th=[ 922], 40.00th=[ 1057], 50.00th=[ 1434], 60.00th=[ 2737], | 70.00th=[ 3589], 80.00th=[ 4817], 90.00th=[ 7111], 95.00th=[10028], | 99.00th=[23987], 99.50th=[24511], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 1408, max= 3008, per=25.51%, avg=2528.00, stdev=654.24, samples=5 iops : min= 88, max= 188, avg=158.00, stdev=40.89, samples=5 lat (usec) : 100=11.46%, 250=1.30%, 500=0.78%, 750=5.47%, 1000=13.54% lat (msec) : 2=11.07%, 4=27.08%, 10=25.78%, 20=1.95%, 50=1.56% cpu : usr=0.04%, sys=26.95%, ctx=1362, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7452: Fri May 9 09:58:58 2025 read: IOPS=157, BW=2525KiB/s (2585kB/s)(6256KiB/2478msec) clat (usec): min=60, max=30710, avg=3293.89, stdev=3301.90 lat (usec): min=60, max=30710, avg=3294.43, stdev=3301.91 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 69], 10.00th=[ 74], 20.00th=[ 85], | 30.00th=[ 2024], 40.00th=[ 2474], 50.00th=[ 2933], 60.00th=[ 3392], | 70.00th=[ 4178], 80.00th=[ 5211], 90.00th=[ 6718], 95.00th=[ 8455], | 99.00th=[16450], 99.50th=[21627], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 1600, max= 3168, per=26.67%, avg=2512.00, stdev=678.57, samples=4 iops : min= 100, max= 198, avg=157.00, stdev=42.41, samples=4 write: IOPS=152, BW=2434KiB/s (2493kB/s)(6032KiB/2478msec) clat (usec): min=630, max=30682, avg=3011.06, stdev=3035.36 lat (usec): min=630, max=30684, avg=3012.62, stdev=3035.42 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 717], 10.00th=[ 766], 20.00th=[ 824], | 30.00th=[ 898], 40.00th=[ 1106], 50.00th=[ 1549], 60.00th=[ 2769], | 70.00th=[ 3720], 80.00th=[ 5145], 90.00th=[ 7373], 95.00th=[ 8586], | 99.00th=[11076], 99.50th=[12780], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 1952, max= 2944, per=25.03%, avg=2480.00, stdev=505.63, samples=4 iops : min= 122, max= 184, avg=155.00, stdev=31.60, samples=4 lat (usec) : 100=13.15%, 250=1.30%, 500=0.26%, 750=4.17%, 1000=13.02% lat (msec) : 2=9.64%, 4=29.04%, 10=26.17%, 20=2.86%, 50=0.39% cpu : usr=0.08%, sys=28.26%, ctx=1419, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7453: Fri May 9 09:58:58 2025 read: IOPS=150, BW=2414KiB/s (2472kB/s)(6032KiB/2499msec) clat (usec): min=56, max=15286, avg=2754.03, stdev=2547.48 lat (usec): min=56, max=15286, avg=2754.91, stdev=2547.42 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 69], 10.00th=[ 73], 20.00th=[ 84], | 30.00th=[ 104], 40.00th=[ 2278], 50.00th=[ 2606], 60.00th=[ 3130], | 70.00th=[ 3818], 80.00th=[ 4817], 90.00th=[ 5866], 95.00th=[ 7308], | 99.00th=[ 9896], 99.50th=[13829], 99.90th=[15270], 99.95th=[15270], | 99.99th=[15270] bw ( KiB/s): min= 992, max= 3072, per=24.89%, avg=2344.00, stdev=935.65, samples=4 iops : min= 62, max= 192, avg=146.50, stdev=58.48, samples=4 write: IOPS=156, BW=2503KiB/s (2563kB/s)(6256KiB/2499msec) clat (usec): min=644, max=32224, avg=3569.76, stdev=4451.53 lat (usec): min=645, max=32227, avg=3574.59, stdev=4452.52 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 742], 10.00th=[ 775], 20.00th=[ 824], | 30.00th=[ 898], 40.00th=[ 1012], 50.00th=[ 1352], 60.00th=[ 2835], | 70.00th=[ 3982], 80.00th=[ 5669], 90.00th=[ 7898], 95.00th=[11076], | 99.00th=[24773], 99.50th=[28443], 99.90th=[32113], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 1216, max= 3072, per=25.59%, avg=2536.00, stdev=883.48, samples=4 iops : min= 76, max= 192, avg=158.50, stdev=55.22, samples=4 lat (usec) : 100=14.58%, 250=2.08%, 500=0.52%, 750=3.52%, 1000=16.54% lat (msec) : 2=7.81%, 4=26.30%, 10=25.39%, 20=2.34%, 50=0.91% cpu : usr=0.08%, sys=26.58%, ctx=1331, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9419KiB/s (9645kB/s), 2259KiB/s-2525KiB/s (2313kB/s-2585kB/s), io=23.4MiB (24.5MB), run=2478-2543msec WRITE: bw=9910KiB/s (10.1MB/s), 2434KiB/s-2573KiB/s (2493kB/s-2635kB/s), io=24.6MiB (25.8MB), run=2478-2543msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7445: Fri May 9 09:59:04 2025 read: IOPS=45, BW=727KiB/s (744kB/s)(5920KiB/8144msec) clat (usec): min=1602, max=32946, avg=3326.32, stdev=3147.46 lat (usec): min=1603, max=32948, avg=3326.94, stdev=3147.50 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1893], 10.00th=[ 1942], 20.00th=[ 2024], | 30.00th=[ 2089], 40.00th=[ 2147], 50.00th=[ 2245], 60.00th=[ 2376], | 70.00th=[ 2704], 80.00th=[ 3687], 90.00th=[ 5669], 95.00th=[ 9372], | 99.00th=[14615], 99.50th=[31327], 99.90th=[32900], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 384, max= 1248, per=24.49%, avg=717.75, stdev=257.67, samples=16 iops : min= 24, max= 78, avg=44.75, stdev=16.14, samples=16 write: IOPS=48, BW=782KiB/s (801kB/s)(6368KiB/8144msec) clat (usec): min=9449, max=45346, avg=17304.01, stdev=5983.55 lat (usec): min=9451, max=45348, avg=17305.14, stdev=5983.62 clat percentiles (usec): | 1.00th=[ 9765], 5.00th=[10814], 10.00th=[11207], 20.00th=[11994], | 30.00th=[13435], 40.00th=[15270], 50.00th=[16712], 60.00th=[17695], | 70.00th=[18482], 80.00th=[20055], 90.00th=[24511], 95.00th=[28443], | 99.00th=[41681], 99.50th=[44303], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 447, max= 1056, per=25.29%, avg=779.81, stdev=208.30, samples=16 iops : min= 27, max= 66, avg=48.62, stdev=13.06, samples=16 lat (msec) : 2=8.33%, 4=30.99%, 10=7.68%, 20=42.06%, 50=10.94% cpu : usr=0.09%, sys=5.94%, ctx=904, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7446: Fri May 9 09:59:04 2025 read: IOPS=43, BW=703KiB/s (720kB/s)(5744KiB/8171msec) clat (usec): min=1789, max=15682, avg=3151.08, stdev=2456.79 lat (usec): min=1790, max=15683, avg=3151.68, stdev=2456.81 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1909], 10.00th=[ 1942], 20.00th=[ 2008], | 30.00th=[ 2073], 40.00th=[ 2147], 50.00th=[ 2245], 60.00th=[ 2343], | 70.00th=[ 2638], 80.00th=[ 3064], 90.00th=[ 6259], 95.00th=[ 9765], | 99.00th=[13698], 99.50th=[15270], 99.90th=[15664], 99.95th=[15664], | 99.99th=[15664] bw ( KiB/s): min= 320, max= 1056, per=23.95%, avg=701.94, stdev=228.61, samples=16 iops : min= 20, max= 66, avg=43.81, stdev=14.38, samples=16 write: IOPS=50, BW=801KiB/s (820kB/s)(6544KiB/8171msec) clat (usec): min=9236, max=52173, avg=17174.79, stdev=5615.91 lat (usec): min=9237, max=52175, avg=17175.91, stdev=5615.97 clat percentiles (usec): | 1.00th=[ 9634], 5.00th=[10945], 10.00th=[11338], 20.00th=[12256], | 30.00th=[13566], 40.00th=[15139], 50.00th=[16450], 60.00th=[17695], | 70.00th=[18744], 80.00th=[20055], 90.00th=[24773], 95.00th=[28443], | 99.00th=[36439], 99.50th=[39584], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 480, max= 1024, per=25.74%, avg=793.87, stdev=191.85, samples=16 iops : min= 30, max= 64, avg=49.56, stdev=12.05, samples=16 lat (msec) : 2=8.98%, 4=31.12%, 10=5.73%, 20=42.71%, 50=11.33% lat (msec) : 100=0.13% cpu : usr=0.04%, sys=6.00%, ctx=877, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7447: Fri May 9 09:59:04 2025 read: IOPS=49, BW=794KiB/s (813kB/s)(6256KiB/7883msec) clat (usec): min=1759, max=26216, avg=3281.32, stdev=2879.41 lat (usec): min=1759, max=26216, avg=3281.97, stdev=2879.41 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1893], 10.00th=[ 1942], 20.00th=[ 2024], | 30.00th=[ 2114], 40.00th=[ 2147], 50.00th=[ 2245], 60.00th=[ 2376], | 70.00th=[ 2573], 80.00th=[ 3261], 90.00th=[ 5800], 95.00th=[ 8717], | 99.00th=[16712], 99.50th=[18220], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 416, max= 1536, per=27.51%, avg=806.27, stdev=383.58, samples=15 iops : min= 26, max= 96, avg=50.33, stdev=24.02, samples=15 write: IOPS=47, BW=765KiB/s (784kB/s)(6032KiB/7883msec) clat (usec): min=9957, max=52094, avg=17415.10, stdev=5911.39 lat (usec): min=9958, max=52096, avg=17416.31, stdev=5911.47 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11076], 10.00th=[11338], 20.00th=[12387], | 30.00th=[13698], 40.00th=[15533], 50.00th=[16712], 60.00th=[17695], | 70.00th=[19006], 80.00th=[20317], 90.00th=[23725], 95.00th=[29754], | 99.00th=[40109], 99.50th=[44827], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 448, max= 928, per=24.41%, avg=752.93, stdev=171.59, samples=15 iops : min= 28, max= 58, avg=47.00, stdev=10.80, samples=15 lat (msec) : 2=8.59%, 4=33.33%, 10=7.03%, 20=40.49%, 50=10.42% lat (msec) : 100=0.13% cpu : usr=0.11%, sys=6.33%, ctx=881, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7448: Fri May 9 09:59:04 2025 read: IOPS=47, BW=752KiB/s (770kB/s)(6032KiB/8019msec) clat (usec): min=1729, max=26197, avg=3121.59, stdev=2526.73 lat (usec): min=1730, max=26198, avg=3122.19, stdev=2526.74 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1876], 10.00th=[ 1942], 20.00th=[ 2008], | 30.00th=[ 2089], 40.00th=[ 2147], 50.00th=[ 2212], 60.00th=[ 2376], | 70.00th=[ 2573], 80.00th=[ 3163], 90.00th=[ 5276], 95.00th=[ 8455], | 99.00th=[15139], 99.50th=[17433], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 384, max= 1053, per=25.44%, avg=745.69, stdev=225.37, samples=16 iops : min= 24, max= 65, avg=46.50, stdev=14.04, samples=16 write: IOPS=48, BW=780KiB/s (799kB/s)(6256KiB/8019msec) clat (usec): min=9880, max=42486, avg=17389.92, stdev=5755.64 lat (usec): min=9881, max=42488, avg=17391.18, stdev=5755.79 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11076], 10.00th=[11469], 20.00th=[12256], | 30.00th=[13698], 40.00th=[15401], 50.00th=[16712], 60.00th=[17695], | 70.00th=[18744], 80.00th=[20055], 90.00th=[24511], 95.00th=[28181], | 99.00th=[40109], 99.50th=[41681], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 448, max= 960, per=25.28%, avg=779.75, stdev=193.10, samples=16 iops : min= 28, max= 60, avg=48.62, stdev=12.11, samples=16 lat (msec) : 2=9.24%, 4=32.16%, 10=6.51%, 20=41.54%, 50=10.55% cpu : usr=0.07%, sys=6.15%, ctx=874, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2931KiB/s (3002kB/s), 703KiB/s-794KiB/s (720kB/s-813kB/s), io=23.4MiB (24.5MB), run=7883-8171msec WRITE: bw=3084KiB/s (3158kB/s), 765KiB/s-801KiB/s (784kB/s-820kB/s), io=24.6MiB (25.8MB), run=7883-8171msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7469: Fri May 9 09:59:07 2025 read: IOPS=5, BW=5510KiB/s (5642kB/s)(7168KiB/1301msec) clat (msec): min=9, max=878, avg=155.58, stdev=319.10 lat (msec): min=9, max=878, avg=155.58, stdev=319.10 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 10], 10.00th=[ 10], 20.00th=[ 31], | 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 42], 60.00th=[ 48], | 70.00th=[ 48], 80.00th=[ 48], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 2048, max= 4096, per=22.05%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 write: IOPS=3, BW=3935KiB/s (4030kB/s)(5120KiB/1301msec) clat (usec): min=25020, max=63828, avg=36347.50, stdev=15707.98 lat (usec): min=25079, max=63846, avg=36386.50, stdev=15694.12 clat percentiles (usec): | 1.00th=[25035], 5.00th=[25035], 10.00th=[25035], 20.00th=[25035], | 30.00th=[28705], 40.00th=[28705], 50.00th=[30016], 60.00th=[30016], | 70.00th=[34341], 80.00th=[34341], 90.00th=[63701], 95.00th=[63701], | 99.00th=[63701], 99.50th=[63701], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] lat (msec) : 10=8.33%, 50=75.00%, 100=8.33%, 1000=8.33% cpu : usr=0.00%, sys=24.08%, ctx=1133, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7470: Fri May 9 09:59:07 2025 read: IOPS=1, BW=1504KiB/s (1540kB/s)(2048KiB/1362msec) clat (usec): min=30237, max=33432, avg=31835.12, stdev=2259.41 lat (usec): min=30238, max=33434, avg=31836.47, stdev=2260.42 clat percentiles (usec): | 1.00th=[30278], 5.00th=[30278], 10.00th=[30278], 20.00th=[30278], | 30.00th=[30278], 40.00th=[30278], 50.00th=[30278], 60.00th=[33424], | 70.00th=[33424], 80.00th=[33424], 90.00th=[33424], 95.00th=[33424], | 99.00th=[33424], 99.50th=[33424], 99.90th=[33424], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 2048, max= 2048, per=14.70%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7518KiB/s (7699kB/s)(10.0MiB/1362msec) clat (msec): min=19, max=867, avg=118.75, stdev=263.18 lat (msec): min=19, max=867, avg=118.81, stdev=263.17 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 24], | 30.00th=[ 26], 40.00th=[ 32], 50.00th=[ 36], 60.00th=[ 37], | 70.00th=[ 46], 80.00th=[ 46], 90.00th=[ 57], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] lat (msec) : 20=8.33%, 50=75.00%, 100=8.33%, 1000=8.33% cpu : usr=0.00%, sys=15.80%, ctx=271, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7471: Fri May 9 09:59:07 2025 read: IOPS=4, BW=4180KiB/s (4280kB/s)(6144KiB/1470msec) clat (usec): min=362, max=52047, avg=18617.00, stdev=20962.42 lat (usec): min=363, max=52048, avg=18617.78, stdev=20962.30 clat percentiles (usec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 498], | 30.00th=[ 498], 40.00th=[ 2966], 50.00th=[ 2966], 60.00th=[27395], | 70.00th=[28443], 80.00th=[28443], 90.00th=[52167], 95.00th=[52167], | 99.00th=[52167], 99.50th=[52167], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] write: IOPS=4, BW=4180KiB/s (4280kB/s)(6144KiB/1470msec) clat (msec): min=24, max=968, avg=208.94, stdev=373.67 lat (msec): min=24, max=968, avg=208.99, stdev=373.67 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 80], | 70.00th=[ 110], 80.00th=[ 110], 90.00th=[ 969], 95.00th=[ 969], | 99.00th=[ 969], 99.50th=[ 969], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 4096, max= 4096, per=21.00%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 4=8.33%, 50=41.67%, 100=16.67%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=14.84%, ctx=428, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7472: Fri May 9 09:59:07 2025 read: IOPS=3, BW=3689KiB/s (3777kB/s)(5120KiB/1388msec) clat (usec): min=306, max=40208, avg=15721.19, stdev=17591.07 lat (usec): min=306, max=40209, avg=15721.87, stdev=17590.98 clat percentiles (usec): | 1.00th=[ 306], 5.00th=[ 306], 10.00th=[ 306], 20.00th=[ 306], | 30.00th=[ 1418], 40.00th=[ 1418], 50.00th=[ 8848], 60.00th=[ 8848], | 70.00th=[27919], 80.00th=[27919], 90.00th=[40109], 95.00th=[40109], | 99.00th=[40109], 99.50th=[40109], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] write: IOPS=5, BW=5164KiB/s (5288kB/s)(7168KiB/1388msec) clat (msec): min=23, max=951, avg=183.33, stdev=340.38 lat (msec): min=23, max=951, avg=183.40, stdev=340.38 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 34], | 30.00th=[ 45], 40.00th=[ 45], 50.00th=[ 53], 60.00th=[ 53], | 70.00th=[ 53], 80.00th=[ 125], 90.00th=[ 953], 95.00th=[ 953], | 99.00th=[ 953], 99.50th=[ 953], 99.90th=[ 953], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 2048, max= 2048, per=10.50%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 2=8.33%, 10=8.33%, 50=41.67%, 100=16.67%, 250=8.33% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=12.47%, ctx=249, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.6MiB/s (14.3MB/s), 1504KiB/s-5510KiB/s (1540kB/s-5642kB/s), io=20.0MiB (20.0MB), run=1301-1470msec WRITE: bw=19.0MiB/s (19.0MB/s), 3935KiB/s-7518KiB/s (4030kB/s-7699kB/s), io=28.0MiB (29.4MB), run=1301-1470msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7473: Fri May 9 09:59:07 2025 read: IOPS=4, BW=4592KiB/s (4702kB/s)(7168KiB/1561msec) clat (msec): min=15, max=989, avg=177.49, stdev=359.27 lat (msec): min=15, max=989, avg=177.49, stdev=359.27 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 36], 60.00th=[ 55], | 70.00th=[ 55], 80.00th=[ 99], 90.00th=[ 986], 95.00th=[ 986], | 99.00th=[ 986], 99.50th=[ 986], 99.90th=[ 986], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 2048, max= 8175, per=39.64%, avg=5111.50, stdev=4332.44, samples=2 iops : min= 2, max= 7, avg= 4.50, stdev= 3.54, samples=2 write: IOPS=3, BW=3280KiB/s (3359kB/s)(5120KiB/1561msec) clat (usec): min=26021, max=82097, avg=54748.33, stdev=24532.73 lat (usec): min=26080, max=82122, avg=54788.29, stdev=24524.86 clat percentiles (usec): | 1.00th=[26084], 5.00th=[26084], 10.00th=[26084], 20.00th=[26084], | 30.00th=[35914], 40.00th=[35914], 50.00th=[52691], 60.00th=[52691], | 70.00th=[77071], 80.00th=[77071], 90.00th=[82314], 95.00th=[82314], | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 8175, max= 8175, per=45.28%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=33.33%, 100=41.67%, 1000=8.33% cpu : usr=0.00%, sys=5.96%, ctx=56, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7474: Fri May 9 09:59:07 2025 read: IOPS=1, BW=1290KiB/s (1321kB/s)(2048KiB/1588msec) clat (usec): min=30430, max=79628, avg=55029.72, stdev=34788.42 lat (usec): min=30431, max=79630, avg=55030.81, stdev=34789.12 clat percentiles (usec): | 1.00th=[30540], 5.00th=[30540], 10.00th=[30540], 20.00th=[30540], | 30.00th=[30540], 40.00th=[30540], 50.00th=[30540], 60.00th=[79168], | 70.00th=[79168], 80.00th=[79168], 90.00th=[79168], 95.00th=[79168], | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 2048, max= 2048, per=15.88%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=6, BW=6448KiB/s (6603kB/s)(10.0MiB/1588msec) clat (msec): min=27, max=900, avg=138.04, stdev=268.98 lat (msec): min=27, max=900, avg=138.11, stdev=268.97 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 35], | 30.00th=[ 35], 40.00th=[ 39], 50.00th=[ 40], 60.00th=[ 50], | 70.00th=[ 96], 80.00th=[ 96], 90.00th=[ 97], 95.00th=[ 902], | 99.00th=[ 902], 99.50th=[ 902], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min=14336, max=14336, per=79.40%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=33.33%, 1000=8.33% cpu : usr=0.19%, sys=6.55%, ctx=46, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7475: Fri May 9 09:59:07 2025 read: IOPS=3, BW=3928KiB/s (4023kB/s)(6144KiB/1564msec) clat (msec): min=14, max=100, avg=36.66, stdev=32.05 lat (msec): min=14, max=100, avg=36.66, stdev=32.05 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 17], 40.00th=[ 26], 50.00th=[ 26], 60.00th=[ 30], | 70.00th=[ 34], 80.00th=[ 34], 90.00th=[ 101], 95.00th=[ 101], | 99.00th=[ 101], 99.50th=[ 101], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min=10240, max=10240, per=79.40%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3928KiB/s (4023kB/s)(6144KiB/1564msec) clat (msec): min=29, max=940, avg=208.44, stdev=360.25 lat (msec): min=29, max=940, avg=208.50, stdev=360.25 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 57], 50.00th=[ 57], 60.00th=[ 64], | 70.00th=[ 124], 80.00th=[ 124], 90.00th=[ 944], 95.00th=[ 944], | 99.00th=[ 944], 99.50th=[ 944], 99.90th=[ 944], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 4096, max= 6144, per=28.36%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 20=16.67%, 50=41.67%, 100=16.67%, 250=16.67%, 1000=8.33% cpu : usr=0.00%, sys=6.85%, ctx=52, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7476: Fri May 9 09:59:07 2025 read: IOPS=3, BW=3243KiB/s (3320kB/s)(5120KiB/1579msec) clat (usec): min=12845, max=61551, avg=34743.84, stdev=17931.43 lat (usec): min=12846, max=61552, avg=34744.58, stdev=17931.48 clat percentiles (usec): | 1.00th=[12911], 5.00th=[12911], 10.00th=[12911], 20.00th=[12911], | 30.00th=[25822], 40.00th=[25822], 50.00th=[35914], 60.00th=[35914], | 70.00th=[37487], 80.00th=[37487], 90.00th=[61604], 95.00th=[61604], | 99.00th=[61604], 99.50th=[61604], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 8192, max= 8192, per=63.52%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4540KiB/s (4649kB/s)(7168KiB/1579msec) clat (msec): min=23, max=907, avg=187.52, stdev=319.89 lat (msec): min=23, max=907, avg=187.60, stdev=319.89 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 25], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 82], 60.00th=[ 109], | 70.00th=[ 109], 80.00th=[ 128], 90.00th=[ 911], 95.00th=[ 911], | 99.00th=[ 911], 99.50th=[ 911], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 2048, max= 8192, per=28.36%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (msec) : 20=8.33%, 50=50.00%, 100=16.67%, 250=16.67%, 1000=8.33% cpu : usr=0.00%, sys=4.25%, ctx=31, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.6MiB/s (13.2MB/s), 1290KiB/s-4592KiB/s (1321kB/s-4702kB/s), io=20.0MiB (20.0MB), run=1561-1588msec WRITE: bw=17.6MiB/s (18.5MB/s), 3280KiB/s-6448KiB/s (3359kB/s-6603kB/s), io=28.0MiB (29.4MB), run=1561-1588msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7485: Fri May 9 09:59:09 2025 read: IOPS=2, BW=10.2MiB/s (10.7MB/s)(12.0MiB/1171msec) clat (msec): min=2, max=971, avg=386.53, stdev=514.63 lat (msec): min=2, max=971, avg=386.53, stdev=514.63 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 3], 40.00th=[ 186], 50.00th=[ 186], 60.00th=[ 186], | 70.00th=[ 969], 80.00th=[ 969], 90.00th=[ 969], 95.00th=[ 969], | 99.00th=[ 969], 99.50th=[ 969], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min=16351, max=16351, per=100.00%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 4=33.33%, 250=33.33%, 1000=33.33% cpu : usr=0.17%, sys=9.15%, ctx=1212, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7487: Fri May 9 09:59:09 2025 read: IOPS=0, BW=3052KiB/s (3125kB/s)(4096KiB/1342msec) clat (nsec): min=988021k, max=988021k, avg=988020971.00, stdev= 0.00 lat (nsec): min=988023k, max=988023k, avg=988022633.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 986], 5.00th=[ 986], 10.00th=[ 986], 20.00th=[ 986], | 30.00th=[ 986], 40.00th=[ 986], 50.00th=[ 986], 60.00th=[ 986], | 70.00th=[ 986], 80.00th=[ 986], 90.00th=[ 986], 95.00th=[ 986], | 99.00th=[ 986], 99.50th=[ 986], 99.90th=[ 986], 99.95th=[ 986], | 99.99th=[ 986] write: IOPS=1, BW=6104KiB/s (6251kB/s)(8192KiB/1342msec) clat (msec): min=143, max=195, avg=169.66, stdev=36.34 lat (msec): min=144, max=195, avg=170.03, stdev=36.37 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 144], 50.00th=[ 144], 60.00th=[ 197], | 70.00th=[ 197], 80.00th=[ 197], 90.00th=[ 197], 95.00th=[ 197], | 99.00th=[ 197], 99.50th=[ 197], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.15%, sys=22.67%, ctx=1019, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7489: Fri May 9 09:59:09 2025 write: IOPS=2, BW=9555KiB/s (9785kB/s)(12.0MiB/1286msec) clat (msec): min=68, max=947, avg=414.44, stdev=468.71 lat (msec): min=68, max=948, avg=414.77, stdev=468.79 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 69], 10.00th=[ 69], 20.00th=[ 69], | 30.00th=[ 69], 40.00th=[ 228], 50.00th=[ 228], 60.00th=[ 228], | 70.00th=[ 944], 80.00th=[ 944], 90.00th=[ 944], 95.00th=[ 944], | 99.00th=[ 944], 99.50th=[ 944], 99.90th=[ 944], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 8192, max= 8192, per=33.55%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=20.39%, ctx=55, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7490: Fri May 9 09:59:09 2025 write: IOPS=2, BW=9330KiB/s (9554kB/s)(12.0MiB/1317msec) clat (msec): min=164, max=943, avg=433.07, stdev=442.58 lat (msec): min=164, max=944, avg=433.36, stdev=442.60 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 165], 10.00th=[ 165], 20.00th=[ 165], | 30.00th=[ 165], 40.00th=[ 192], 50.00th=[ 192], 60.00th=[ 192], | 70.00th=[ 944], 80.00th=[ 944], 90.00th=[ 944], 95.00th=[ 944], | 99.00th=[ 944], 99.50th=[ 944], 99.90th=[ 944], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min=16384, max=16384, per=67.10%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.15%, sys=19.45%, ctx=70, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.9MiB/s (12.5MB/s), 3052KiB/s-10.2MiB/s (3125kB/s-10.7MB/s), io=16.0MiB (16.8MB), run=1171-1342msec WRITE: bw=23.8MiB/s (25.0MB/s), 6104KiB/s-9555KiB/s (6251kB/s-9785kB/s), io=32.0MiB (33.6MB), run=1286-1342msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7483: Fri May 9 09:59:09 2025 read: IOPS=2, BW=9136KiB/s (9355kB/s)(12.0MiB/1345msec) clat (msec): min=63, max=1095, avg=431.80, stdev=575.52 lat (msec): min=63, max=1095, avg=431.80, stdev=575.52 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 64], | 30.00th=[ 64], 40.00th=[ 138], 50.00th=[ 138], 60.00th=[ 138], | 70.00th=[ 1099], 80.00th=[ 1099], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min=11025, max=11025, per=90.51%, avg=11025.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=4.46%, ctx=52, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7484: Fri May 9 09:59:09 2025 read: IOPS=3, BW=13.7MiB/s (14.4MB/s)(4096KiB/292msec) clat (nsec): min=44085k, max=44085k, avg=44084838.00, stdev= 0.00 lat (nsec): min=44087k, max=44087k, avg=44086925.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[44303], 5.00th=[44303], 10.00th=[44303], 20.00th=[44303], | 30.00th=[44303], 40.00th=[44303], 50.00th=[44303], 60.00th=[44303], | 70.00th=[44303], 80.00th=[44303], 90.00th=[44303], 95.00th=[44303], | 99.00th=[44303], 99.50th=[44303], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] write: IOPS=6, BW=27.4MiB/s (28.7MB/s)(8192KiB/292msec) clat (msec): min=60, max=178, avg=119.39, stdev=83.02 lat (msec): min=60, max=178, avg=119.66, stdev=83.03 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 61], 10.00th=[ 61], 20.00th=[ 61], | 30.00th=[ 61], 40.00th=[ 61], 50.00th=[ 61], 60.00th=[ 178], | 70.00th=[ 178], 80.00th=[ 178], 90.00th=[ 178], 95.00th=[ 178], | 99.00th=[ 178], 99.50th=[ 178], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] lat (msec) : 50=33.33%, 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=19.24%, ctx=32, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7486: Fri May 9 09:59:09 2025 write: IOPS=7, BW=30.0MiB/s (31.5MB/s)(12.0MiB/400msec) clat (msec): min=57, max=235, avg=127.06, stdev=95.30 lat (msec): min=57, max=235, avg=127.35, stdev=95.32 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 58], 10.00th=[ 58], 20.00th=[ 58], | 30.00th=[ 58], 40.00th=[ 89], 50.00th=[ 89], 60.00th=[ 89], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 236], 95.00th=[ 236], | 99.00th=[ 236], 99.50th=[ 236], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] lat (msec) : 100=66.67%, 250=33.33% cpu : usr=0.50%, sys=16.04%, ctx=108, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7488: Fri May 9 09:59:09 2025 write: IOPS=2, BW=9.85MiB/s (10.3MB/s)(12.0MiB/1218msec) clat (msec): min=101, max=916, avg=403.19, stdev=446.85 lat (msec): min=101, max=916, avg=403.42, stdev=446.91 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 102], 10.00th=[ 102], 20.00th=[ 102], | 30.00th=[ 102], 40.00th=[ 192], 50.00th=[ 192], 60.00th=[ 192], | 70.00th=[ 919], 80.00th=[ 919], 90.00th=[ 919], 95.00th=[ 919], | 99.00th=[ 919], 99.50th=[ 919], 99.90th=[ 919], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min=16384, max=16384, per=60.90%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.00%, sys=5.34%, ctx=37, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.9MiB/s (12.5MB/s), 9136KiB/s-13.7MiB/s (9355kB/s-14.4MB/s), io=16.0MiB (16.8MB), run=292-1345msec WRITE: bw=26.3MiB/s (27.5MB/s), 9.85MiB/s-30.0MiB/s (10.3MB/s-31.5MB/s), io=32.0MiB (33.6MB), run=292-1218msec