== sanity test 398b: DIO and buffer IO race ============== 09:38:33 (1741700313) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.05389 s, 47.8 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7760: Tue Mar 11 09:38:49 2025 read: IOPS=111, BW=446KiB/s (457kB/s)(5992KiB/13421msec) clat (usec): min=54, max=174312, avg=3882.93, stdev=5395.99 lat (usec): min=55, max=174313, avg=3883.64, stdev=5395.99 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 72], 10.00th=[ 83], 20.00th=[ 204], | 30.00th=[ 2343], 40.00th=[ 2704], 50.00th=[ 3130], 60.00th=[ 3654], | 70.00th=[ 4555], 80.00th=[ 5997], 90.00th=[ 7963], 95.00th=[ 9503], | 99.00th=[ 14091], 99.50th=[ 15926], 99.90th=[ 20579], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 224, max= 720, per=24.64%, avg=440.85, stdev=103.68, samples=26 iops : min= 56, max= 180, avg=110.19, stdev=25.92, samples=26 write: IOPS=117, BW=469KiB/s (480kB/s)(6296KiB/13421msec) clat (usec): min=460, max=47892, avg=4777.90, stdev=4576.67 lat (usec): min=460, max=47893, avg=4778.84, stdev=4576.54 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 562], 10.00th=[ 611], 20.00th=[ 750], | 30.00th=[ 1221], 40.00th=[ 2606], 50.00th=[ 3785], 60.00th=[ 4817], | 70.00th=[ 5997], 80.00th=[ 7898], 90.00th=[10683], 95.00th=[13698], | 99.00th=[19792], 99.50th=[22676], 99.90th=[28705], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 352, max= 656, per=24.60%, avg=460.85, stdev=88.49, samples=26 iops : min= 88, max= 164, avg=115.19, stdev=22.14, samples=26 lat (usec) : 100=6.38%, 250=3.48%, 500=0.39%, 750=9.99%, 1000=3.61% lat (msec) : 2=5.86%, 4=28.35%, 10=33.85%, 20=7.49%, 50=0.55% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=23.11%, ctx=6053, majf=0, minf=37 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7761: Tue Mar 11 09:38:49 2025 read: IOPS=110, BW=443KiB/s (454kB/s)(5940KiB/13409msec) clat (usec): min=50, max=33090, avg=3615.50, stdev=3125.44 lat (usec): min=50, max=33090, avg=3616.41, stdev=3125.49 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 71], 10.00th=[ 83], 20.00th=[ 128], | 30.00th=[ 2245], 40.00th=[ 2671], 50.00th=[ 3097], 60.00th=[ 3556], | 70.00th=[ 4359], 80.00th=[ 5735], 90.00th=[ 7635], 95.00th=[ 9372], | 99.00th=[14222], 99.50th=[16188], 99.90th=[23462], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 288, max= 640, per=24.52%, avg=438.69, stdev=89.17, samples=26 iops : min= 72, max= 160, avg=109.62, stdev=22.32, samples=26 write: IOPS=118, BW=473KiB/s (485kB/s)(6348KiB/13409msec) clat (usec): min=472, max=170504, avg=5012.35, stdev=6493.36 lat (usec): min=472, max=170505, avg=5013.46, stdev=6493.23 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 553], 10.00th=[ 611], 20.00th=[ 750], | 30.00th=[ 1057], 40.00th=[ 2507], 50.00th=[ 3884], 60.00th=[ 4948], | 70.00th=[ 6325], 80.00th=[ 8225], 90.00th=[ 10814], 95.00th=[ 13698], | 99.00th=[ 20841], 99.50th=[ 25560], 99.90th=[ 76022], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 352, max= 776, per=24.62%, avg=461.12, stdev=85.03, samples=26 iops : min= 88, max= 194, avg=115.23, stdev=21.24, samples=26 lat (usec) : 100=7.26%, 250=3.39%, 500=0.39%, 750=10.29%, 1000=4.36% lat (msec) : 2=5.79%, 4=26.99%, 10=33.24%, 20=7.58%, 50=0.65% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.17%, sys=22.62%, ctx=5903, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7762: Tue Mar 11 09:38:49 2025 read: IOPS=112, BW=450KiB/s (461kB/s)(6028KiB/13392msec) clat (usec): min=54, max=45647, avg=3929.22, stdev=3706.98 lat (usec): min=54, max=45648, avg=3930.60, stdev=3707.13 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 71], 10.00th=[ 85], 20.00th=[ 145], | 30.00th=[ 2343], 40.00th=[ 2769], 50.00th=[ 3163], 60.00th=[ 3720], | 70.00th=[ 4621], 80.00th=[ 6128], 90.00th=[ 8586], 95.00th=[10552], | 99.00th=[14746], 99.50th=[18220], 99.90th=[37487], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 311, max= 688, per=24.93%, avg=445.92, stdev=90.62, samples=26 iops : min= 77, max= 172, avg=111.35, stdev=22.73, samples=26 write: IOPS=116, BW=467KiB/s (479kB/s)(6260KiB/13392msec) clat (usec): min=460, max=169594, avg=4717.61, stdev=6065.62 lat (usec): min=461, max=169596, avg=4718.75, stdev=6065.38 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 545], 10.00th=[ 586], 20.00th=[ 742], | 30.00th=[ 1045], 40.00th=[ 2278], 50.00th=[ 3621], 60.00th=[ 4686], | 70.00th=[ 5866], 80.00th=[ 7635], 90.00th=[ 10552], 95.00th=[ 13566], | 99.00th=[ 19792], 99.50th=[ 25297], 99.90th=[ 29230], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 304, max= 680, per=24.58%, avg=460.42, stdev=81.66, samples=26 iops : min= 76, max= 170, avg=114.96, stdev=20.45, samples=26 lat (usec) : 100=6.97%, 250=3.29%, 500=0.85%, 750=9.83%, 1000=4.65% lat (msec) : 2=5.14%, 4=27.47%, 10=33.24%, 20=7.91%, 50=0.62% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=22.72%, ctx=5900, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7763: Tue Mar 11 09:38:49 2025 read: IOPS=113, BW=454KiB/s (465kB/s)(6052KiB/13333msec) clat (usec): min=53, max=169738, avg=3872.95, stdev=5488.71 lat (usec): min=54, max=169739, avg=3873.90, stdev=5488.49 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 71], 10.00th=[ 85], 20.00th=[ 186], | 30.00th=[ 2311], 40.00th=[ 2671], 50.00th=[ 3064], 60.00th=[ 3589], | 70.00th=[ 4490], 80.00th=[ 5866], 90.00th=[ 8029], 95.00th=[ 9765], | 99.00th=[ 13829], 99.50th=[ 16909], 99.90th=[ 55837], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 295, max= 736, per=25.24%, avg=451.50, stdev=101.03, samples=26 iops : min= 73, max= 184, avg=112.77, stdev=25.34, samples=26 write: IOPS=116, BW=468KiB/s (479kB/s)(6236KiB/13333msec) clat (usec): min=465, max=60292, avg=4739.87, stdev=5152.86 lat (usec): min=466, max=60293, avg=4741.45, stdev=5153.42 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 553], 10.00th=[ 611], 20.00th=[ 734], | 30.00th=[ 955], 40.00th=[ 1958], 50.00th=[ 3523], 60.00th=[ 4621], | 70.00th=[ 5932], 80.00th=[ 7832], 90.00th=[10683], 95.00th=[13960], | 99.00th=[21365], 99.50th=[31327], 99.90th=[52691], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 304, max= 648, per=24.84%, avg=465.31, stdev=87.82, samples=26 iops : min= 76, max= 162, avg=116.23, stdev=22.03, samples=26 lat (usec) : 100=6.77%, 250=3.22%, 500=0.68%, 750=10.35%, 1000=4.85% lat (msec) : 2=5.66%, 4=28.65%, 10=31.51%, 20=7.42%, 50=0.75% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.18%, sys=22.85%, ctx=5823, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1789KiB/s (1832kB/s), 443KiB/s-454KiB/s (454kB/s-465kB/s), io=23.4MiB (24.6MB), run=13333-13421msec WRITE: bw=1873KiB/s (1918kB/s), 467KiB/s-473KiB/s (479kB/s-485kB/s), io=24.6MiB (25.7MB), run=13333-13421msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7755: Tue Mar 11 09:39:14 2025 read: IOPS=38, BW=156KiB/s (159kB/s)(5992KiB/38531msec) clat (usec): min=1696, max=177231, avg=3938.93, stdev=5710.06 lat (usec): min=1696, max=177233, avg=3939.72, stdev=5710.07 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 1975], 10.00th=[ 2073], 20.00th=[ 2212], | 30.00th=[ 2311], 40.00th=[ 2442], 50.00th=[ 2573], 60.00th=[ 2769], | 70.00th=[ 3130], 80.00th=[ 4080], 90.00th=[ 7570], 95.00th=[ 11338], | 99.00th=[ 18220], 99.50th=[ 25297], 99.90th=[ 46924], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 56, max= 304, per=25.09%, avg=155.53, stdev=62.93, samples=77 iops : min= 14, max= 76, avg=38.81, stdev=15.75, samples=77 write: IOPS=40, BW=163KiB/s (167kB/s)(6296KiB/38531msec) clat (usec): min=9834, max=65764, avg=20650.02, stdev=7919.52 lat (usec): min=9835, max=65765, avg=20650.98, stdev=7919.54 clat percentiles (usec): | 1.00th=[10945], 5.00th=[12256], 10.00th=[12911], 20.00th=[14091], | 30.00th=[15664], 40.00th=[17433], 50.00th=[19268], 60.00th=[20579], | 70.00th=[21890], 80.00th=[24249], 90.00th=[31851], 95.00th=[38011], | 99.00th=[46924], 99.50th=[52167], 99.90th=[60031], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 48, max= 232, per=25.15%, avg=163.22, stdev=49.32, samples=77 iops : min= 12, max= 58, avg=40.73, stdev=12.33, samples=77 lat (msec) : 2=2.90%, 4=35.87%, 10=7.00%, 20=31.15%, 50=22.75% lat (msec) : 100=0.29%, 250=0.03% cpu : usr=0.11%, sys=5.78%, ctx=3415, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7756: Tue Mar 11 09:39:14 2025 read: IOPS=38, BW=153KiB/s (157kB/s)(5940KiB/38698msec) clat (usec): min=1698, max=175546, avg=3936.88, stdev=5828.28 lat (usec): min=1698, max=175548, avg=3937.64, stdev=5828.28 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 1991], 10.00th=[ 2073], 20.00th=[ 2180], | 30.00th=[ 2311], 40.00th=[ 2409], 50.00th=[ 2573], 60.00th=[ 2769], | 70.00th=[ 3032], 80.00th=[ 3916], 90.00th=[ 7570], 95.00th=[ 10421], | 99.00th=[ 19268], 99.50th=[ 27919], 99.90th=[ 49021], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 16, max= 312, per=24.79%, avg=153.70, stdev=66.89, samples=77 iops : min= 4, max= 78, avg=38.38, stdev=16.73, samples=77 write: IOPS=41, BW=164KiB/s (168kB/s)(6348KiB/38698msec) clat (usec): min=10618, max=81334, avg=20621.50, stdev=8054.44 lat (usec): min=10619, max=81335, avg=20622.50, stdev=8054.38 clat percentiles (usec): | 1.00th=[11469], 5.00th=[12387], 10.00th=[13042], 20.00th=[14091], | 30.00th=[15401], 40.00th=[17433], 50.00th=[19006], 60.00th=[20579], | 70.00th=[21890], 80.00th=[23987], 90.00th=[31851], 95.00th=[37487], | 99.00th=[50594], 99.50th=[52167], 99.90th=[71828], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 56, max= 224, per=25.22%, avg=163.68, stdev=49.10, samples=77 iops : min= 14, max= 56, avg=40.87, stdev=12.28, samples=77 lat (msec) : 2=2.73%, 4=36.26%, 10=6.51%, 20=31.54%, 50=22.40% lat (msec) : 100=0.52%, 250=0.03% cpu : usr=0.15%, sys=5.70%, ctx=3476, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7757: Tue Mar 11 09:39:14 2025 read: IOPS=39, BW=157KiB/s (160kB/s)(6028KiB/38499msec) clat (usec): min=1761, max=44623, avg=4022.67, stdev=3889.34 lat (usec): min=1761, max=44624, avg=4023.47, stdev=3889.35 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 1975], 10.00th=[ 2057], 20.00th=[ 2212], | 30.00th=[ 2311], 40.00th=[ 2474], 50.00th=[ 2638], 60.00th=[ 2835], | 70.00th=[ 3163], 80.00th=[ 4359], 90.00th=[ 8356], 95.00th=[11600], | 99.00th=[19530], 99.50th=[25035], 99.90th=[39060], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 40, max= 288, per=25.26%, avg=156.63, stdev=59.08, samples=76 iops : min= 10, max= 72, avg=39.08, stdev=14.76, samples=76 write: IOPS=40, BW=163KiB/s (167kB/s)(6260KiB/38499msec) clat (msec): min=9, max=192, avg=20.65, stdev= 8.99 lat (msec): min=9, max=192, avg=20.65, stdev= 8.99 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 21], | 70.00th=[ 23], 80.00th=[ 25], 90.00th=[ 32], 95.00th=[ 36], | 99.00th=[ 48], 99.50th=[ 56], 99.90th=[ 82], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 56, max= 224, per=24.99%, avg=162.21, stdev=50.05, samples=76 iops : min= 14, max= 56, avg=40.47, stdev=12.52, samples=76 lat (msec) : 2=2.99%, 4=35.64%, 10=6.84%, 20=32.10%, 50=22.07% lat (msec) : 100=0.33%, 250=0.03% cpu : usr=0.12%, sys=5.71%, ctx=3485, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7758: Tue Mar 11 09:39:14 2025 read: IOPS=39, BW=157KiB/s (161kB/s)(6052KiB/38603msec) clat (usec): min=1687, max=70617, avg=3690.60, stdev=3521.11 lat (usec): min=1688, max=70618, avg=3691.35, stdev=3521.09 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 1958], 10.00th=[ 2040], 20.00th=[ 2180], | 30.00th=[ 2311], 40.00th=[ 2442], 50.00th=[ 2573], 60.00th=[ 2769], | 70.00th=[ 3064], 80.00th=[ 3785], 90.00th=[ 7046], 95.00th=[10159], | 99.00th=[15926], 99.50th=[20317], 99.90th=[38011], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 32, max= 312, per=25.27%, avg=156.70, stdev=64.64, samples=77 iops : min= 8, max= 78, avg=39.12, stdev=16.16, samples=77 write: IOPS=40, BW=162KiB/s (165kB/s)(6236KiB/38603msec) clat (msec): min=9, max=191, avg=21.10, stdev= 9.14 lat (msec): min=9, max=191, avg=21.10, stdev= 9.14 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 21], | 70.00th=[ 23], 80.00th=[ 25], 90.00th=[ 33], 95.00th=[ 39], | 99.00th=[ 51], 99.50th=[ 56], 99.90th=[ 67], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 56, max= 216, per=24.87%, avg=161.38, stdev=48.35, samples=77 iops : min= 14, max= 54, avg=40.29, stdev=12.09, samples=77 lat (msec) : 2=3.81%, 4=36.07%, 10=6.90%, 20=29.26%, 50=23.40% lat (msec) : 100=0.52%, 250=0.03% cpu : usr=0.09%, sys=5.69%, ctx=3495, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=620KiB/s (635kB/s), 153KiB/s-157KiB/s (157kB/s-161kB/s), io=23.4MiB (24.6MB), run=38499-38698msec WRITE: bw=650KiB/s (665kB/s), 162KiB/s-164KiB/s (165kB/s-168kB/s), io=24.6MiB (25.7MB), run=38499-38698msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7808: Tue Mar 11 09:39:18 2025 read: IOPS=131, BW=2106KiB/s (2157kB/s)(5920KiB/2811msec) clat (usec): min=59, max=37943, avg=2977.99, stdev=3272.89 lat (usec): min=59, max=37944, avg=2978.58, stdev=3272.91 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 69], 10.00th=[ 72], 20.00th=[ 87], | 30.00th=[ 112], 40.00th=[ 2409], 50.00th=[ 2868], 60.00th=[ 3261], | 70.00th=[ 4047], 80.00th=[ 5014], 90.00th=[ 6259], 95.00th=[ 7439], | 99.00th=[12780], 99.50th=[17433], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 448, max= 2592, per=24.23%, avg=2064.20, stdev=914.38, samples=5 iops : min= 28, max= 162, avg=129.00, stdev=57.14, samples=5 write: IOPS=141, BW=2265KiB/s (2320kB/s)(6368KiB/2811msec) clat (usec): min=617, max=33604, avg=4184.75, stdev=5359.57 lat (usec): min=618, max=33605, avg=4187.14, stdev=5359.78 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 889], | 30.00th=[ 1045], 40.00th=[ 1237], 50.00th=[ 2311], 60.00th=[ 3359], | 70.00th=[ 4686], 80.00th=[ 6325], 90.00th=[ 8455], 95.00th=[15401], | 99.00th=[28967], 99.50th=[30540], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 512, max= 2912, per=25.10%, avg=2249.80, stdev=982.85, samples=5 iops : min= 32, max= 182, avg=140.60, stdev=61.42, samples=5 lat (usec) : 100=12.63%, 250=4.69%, 500=0.13%, 750=3.26%, 1000=11.33% lat (msec) : 2=10.81%, 4=24.22%, 10=28.26%, 20=2.21%, 50=2.47% cpu : usr=0.25%, sys=26.37%, ctx=1391, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7810: Tue Mar 11 09:39:18 2025 read: IOPS=144, BW=2319KiB/s (2375kB/s)(5744KiB/2477msec) clat (usec): min=56, max=18569, avg=2701.05, stdev=2554.40 lat (usec): min=57, max=18570, avg=2701.78, stdev=2554.31 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 68], 10.00th=[ 70], 20.00th=[ 81], | 30.00th=[ 112], 40.00th=[ 2245], 50.00th=[ 2704], 60.00th=[ 3163], | 70.00th=[ 3687], 80.00th=[ 4621], 90.00th=[ 6063], 95.00th=[ 7046], | 99.00th=[ 9765], 99.50th=[14222], 99.90th=[18482], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 1373, max= 2816, per=27.60%, avg=2351.25, stdev=662.72, samples=4 iops : min= 85, max= 176, avg=146.75, stdev=41.82, samples=4 write: IOPS=165, BW=2642KiB/s (2705kB/s)(6544KiB/2477msec) clat (usec): min=662, max=32533, avg=3545.30, stdev=4239.58 lat (usec): min=663, max=32534, avg=3546.16, stdev=4239.58 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 758], 10.00th=[ 799], 20.00th=[ 873], | 30.00th=[ 947], 40.00th=[ 1156], 50.00th=[ 1795], 60.00th=[ 3097], | 70.00th=[ 4080], 80.00th=[ 5866], 90.00th=[ 7635], 95.00th=[ 9110], | 99.00th=[23725], 99.50th=[29230], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 1469, max= 3296, per=28.55%, avg=2559.25, stdev=776.04, samples=4 iops : min= 91, max= 206, avg=159.75, stdev=48.88, samples=4 lat (usec) : 100=13.15%, 250=3.39%, 500=0.13%, 750=2.34%, 1000=16.41% lat (msec) : 2=9.11%, 4=26.82%, 10=25.91%, 20=1.95%, 50=0.78% cpu : usr=0.32%, sys=27.18%, ctx=1461, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7812: Tue Mar 11 09:39:18 2025 read: IOPS=163, BW=2618KiB/s (2680kB/s)(6256KiB/2390msec) clat (usec): min=54, max=37856, avg=3187.98, stdev=3338.84 lat (usec): min=55, max=37856, avg=3188.53, stdev=3338.87 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 64], 10.00th=[ 71], 20.00th=[ 84], | 30.00th=[ 133], 40.00th=[ 2442], 50.00th=[ 2966], 60.00th=[ 3490], | 70.00th=[ 4293], 80.00th=[ 5211], 90.00th=[ 6652], 95.00th=[ 8094], | 99.00th=[14484], 99.50th=[16909], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 1984, max= 2784, per=29.58%, avg=2520.00, stdev=366.14, samples=4 iops : min= 124, max= 174, avg=157.50, stdev=22.88, samples=4 write: IOPS=157, BW=2524KiB/s (2584kB/s)(6032KiB/2390msec) clat (usec): min=644, max=21082, avg=3001.59, stdev=2764.62 lat (usec): min=644, max=21083, avg=3002.43, stdev=2764.63 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 750], 10.00th=[ 807], 20.00th=[ 889], | 30.00th=[ 1020], 40.00th=[ 1156], 50.00th=[ 1532], 60.00th=[ 2933], | 70.00th=[ 3884], 80.00th=[ 5276], 90.00th=[ 6783], 95.00th=[ 8291], | 99.00th=[12911], 99.50th=[15008], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 2112, max= 2784, per=27.84%, avg=2496.00, stdev=282.62, samples=4 iops : min= 132, max= 174, avg=156.00, stdev=17.66, samples=4 lat (usec) : 100=12.37%, 250=4.17%, 500=0.13%, 750=2.60%, 1000=11.72% lat (msec) : 2=12.24%, 4=25.39%, 10=29.95%, 20=1.17%, 50=0.26% cpu : usr=0.13%, sys=28.97%, ctx=1517, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7813: Tue Mar 11 09:39:18 2025 read: IOPS=154, BW=2469KiB/s (2528kB/s)(6032KiB/2443msec) clat (usec): min=58, max=15649, avg=2733.93, stdev=2268.56 lat (usec): min=58, max=15650, avg=2735.12, stdev=2269.12 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 69], 10.00th=[ 71], 20.00th=[ 87], | 30.00th=[ 273], 40.00th=[ 2442], 50.00th=[ 2802], 60.00th=[ 3228], | 70.00th=[ 3687], 80.00th=[ 4293], 90.00th=[ 5669], 95.00th=[ 6390], | 99.00th=[ 8455], 99.50th=[12256], 99.90th=[15664], 99.95th=[15664], | 99.99th=[15664] bw ( KiB/s): min= 1980, max= 2656, per=28.81%, avg=2455.00, stdev=319.17, samples=4 iops : min= 123, max= 166, avg=153.25, stdev=20.32, samples=4 write: IOPS=160, BW=2561KiB/s (2622kB/s)(6256KiB/2443msec) clat (usec): min=633, max=46181, avg=3442.19, stdev=4144.31 lat (usec): min=634, max=46182, avg=3444.94, stdev=4144.34 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 766], 10.00th=[ 799], 20.00th=[ 881], | 30.00th=[ 963], 40.00th=[ 1139], 50.00th=[ 1532], 60.00th=[ 2933], | 70.00th=[ 4080], 80.00th=[ 5735], 90.00th=[ 8291], 95.00th=[ 9634], | 99.00th=[20317], 99.50th=[26608], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 2171, max= 3136, per=29.26%, avg=2622.75, stdev=425.49, samples=4 iops : min= 135, max= 196, avg=163.75, stdev=26.84, samples=4 lat (usec) : 100=11.33%, 250=3.26%, 500=0.39%, 750=2.21%, 1000=15.23% lat (msec) : 2=10.81%, 4=29.17%, 10=25.26%, 20=1.82%, 50=0.52% cpu : usr=0.20%, sys=28.42%, ctx=1474, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8521KiB/s (8725kB/s), 2106KiB/s-2618KiB/s (2157kB/s-2680kB/s), io=23.4MiB (24.5MB), run=2390-2811msec WRITE: bw=8965KiB/s (9180kB/s), 2265KiB/s-2642KiB/s (2320kB/s-2705kB/s), io=24.6MiB (25.8MB), run=2390-2811msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7806: Tue Mar 11 09:39:25 2025 read: IOPS=37, BW=595KiB/s (609kB/s)(5920KiB/9949msec) clat (usec): min=2110, max=43475, avg=4199.79, stdev=2872.77 lat (usec): min=2110, max=43476, avg=4200.88, stdev=2872.68 clat percentiles (usec): | 1.00th=[ 2245], 5.00th=[ 2638], 10.00th=[ 2835], 20.00th=[ 3032], | 30.00th=[ 3163], 40.00th=[ 3294], 50.00th=[ 3458], 60.00th=[ 3654], | 70.00th=[ 3884], 80.00th=[ 4293], 90.00th=[ 6325], 95.00th=[ 8455], | 99.00th=[13698], 99.50th=[14746], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 384, max= 928, per=24.72%, avg=587.58, stdev=165.59, samples=19 iops : min= 24, max= 58, avg=36.63, stdev=10.32, samples=19 write: IOPS=40, BW=640KiB/s (655kB/s)(6368KiB/9949msec) clat (usec): min=12390, max=49285, avg=21000.57, stdev=5817.49 lat (usec): min=12391, max=49286, avg=21002.60, stdev=5817.22 clat percentiles (usec): | 1.00th=[12780], 5.00th=[13960], 10.00th=[14484], 20.00th=[15664], | 30.00th=[17433], 40.00th=[18744], 50.00th=[20317], 60.00th=[21890], | 70.00th=[23200], 80.00th=[25035], 90.00th=[27657], 95.00th=[31327], | 99.00th=[41681], 99.50th=[43779], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 416, max= 832, per=25.45%, avg=636.42, stdev=111.18, samples=19 iops : min= 26, max= 52, avg=39.68, stdev= 6.90, samples=19 lat (msec) : 4=35.81%, 10=10.81%, 20=26.69%, 50=26.69% cpu : usr=0.13%, sys=8.03%, ctx=896, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7807: Tue Mar 11 09:39:25 2025 read: IOPS=35, BW=570KiB/s (584kB/s)(5744KiB/10075msec) clat (usec): min=2082, max=27159, avg=4414.51, stdev=2706.31 lat (usec): min=2082, max=27160, avg=4415.46, stdev=2706.26 clat percentiles (usec): | 1.00th=[ 2245], 5.00th=[ 2573], 10.00th=[ 2900], 20.00th=[ 3097], | 30.00th=[ 3261], 40.00th=[ 3392], 50.00th=[ 3556], 60.00th=[ 3785], | 70.00th=[ 4047], 80.00th=[ 4752], 90.00th=[ 7046], 95.00th=[ 9503], | 99.00th=[16057], 99.50th=[20841], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 320, max= 864, per=23.89%, avg=567.80, stdev=179.55, samples=20 iops : min= 20, max= 54, avg=35.40, stdev=11.12, samples=20 write: IOPS=40, BW=650KiB/s (665kB/s)(6544KiB/10075msec) clat (usec): min=12327, max=56128, avg=20683.17, stdev=5919.07 lat (usec): min=12329, max=56129, avg=20684.84, stdev=5918.94 clat percentiles (usec): | 1.00th=[13042], 5.00th=[13698], 10.00th=[14353], 20.00th=[15270], | 30.00th=[16581], 40.00th=[18482], 50.00th=[20055], 60.00th=[21627], | 70.00th=[22938], 80.00th=[24511], 90.00th=[27657], 95.00th=[31589], | 99.00th=[40109], 99.50th=[44827], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 416, max= 768, per=25.90%, avg=647.80, stdev=107.72, samples=20 iops : min= 26, max= 48, avg=40.40, stdev= 6.69, samples=20 lat (msec) : 4=32.29%, 10=12.37%, 20=27.99%, 50=27.21%, 100=0.13% cpu : usr=0.10%, sys=8.00%, ctx=911, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7809: Tue Mar 11 09:39:25 2025 read: IOPS=40, BW=650KiB/s (665kB/s)(6256KiB/9629msec) clat (usec): min=2093, max=43320, avg=4171.39, stdev=3044.12 lat (usec): min=2094, max=43321, avg=4172.56, stdev=3044.31 clat percentiles (usec): | 1.00th=[ 2245], 5.00th=[ 2638], 10.00th=[ 2802], 20.00th=[ 2999], | 30.00th=[ 3163], 40.00th=[ 3261], 50.00th=[ 3425], 60.00th=[ 3654], | 70.00th=[ 3851], 80.00th=[ 4228], 90.00th=[ 5669], 95.00th=[ 8455], | 99.00th=[16057], 99.50th=[25035], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 256, max= 1088, per=27.62%, avg=656.63, stdev=235.69, samples=19 iops : min= 16, max= 68, avg=40.95, stdev=14.72, samples=19 write: IOPS=39, BW=626KiB/s (641kB/s)(6032KiB/9629msec) clat (usec): min=11721, max=53700, avg=21062.26, stdev=6684.23 lat (usec): min=11722, max=53701, avg=21063.83, stdev=6684.12 clat percentiles (usec): | 1.00th=[12649], 5.00th=[13566], 10.00th=[14222], 20.00th=[15139], | 30.00th=[16450], 40.00th=[18482], 50.00th=[20317], 60.00th=[21890], | 70.00th=[23200], 80.00th=[25035], 90.00th=[29230], 95.00th=[34341], | 99.00th=[45876], 99.50th=[51643], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 352, max= 766, per=24.98%, avg=624.63, stdev=120.62, samples=19 iops : min= 22, max= 47, avg=38.95, stdev= 7.44, samples=19 lat (msec) : 4=38.15%, 10=10.94%, 20=25.78%, 50=24.87%, 100=0.26% cpu : usr=0.09%, sys=8.41%, ctx=926, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7811: Tue Mar 11 09:39:25 2025 read: IOPS=37, BW=601KiB/s (616kB/s)(6032KiB/10033msec) clat (usec): min=2097, max=43644, avg=4638.10, stdev=4042.83 lat (usec): min=2097, max=43645, avg=4639.19, stdev=4042.76 clat percentiles (usec): | 1.00th=[ 2343], 5.00th=[ 2638], 10.00th=[ 2769], 20.00th=[ 3032], | 30.00th=[ 3228], 40.00th=[ 3359], 50.00th=[ 3490], 60.00th=[ 3687], | 70.00th=[ 3884], 80.00th=[ 4293], 90.00th=[ 7373], 95.00th=[12649], | 99.00th=[26870], 99.50th=[32375], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 320, max= 832, per=25.10%, avg=596.60, stdev=167.10, samples=20 iops : min= 20, max= 52, avg=37.20, stdev=10.42, samples=20 write: IOPS=38, BW=624KiB/s (639kB/s)(6256KiB/10033msec) clat (usec): min=11350, max=60790, avg=21142.14, stdev=6032.68 lat (usec): min=11351, max=60791, avg=21144.10, stdev=6032.53 clat percentiles (usec): | 1.00th=[12649], 5.00th=[13566], 10.00th=[14353], 20.00th=[15401], | 30.00th=[17171], 40.00th=[18744], 50.00th=[20841], 60.00th=[22152], | 70.00th=[23462], 80.00th=[25297], 90.00th=[28181], 95.00th=[31589], | 99.00th=[41157], 99.50th=[42730], 99.90th=[60556], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min= 384, max= 768, per=24.94%, avg=623.80, stdev=113.82, samples=20 iops : min= 24, max= 48, avg=38.90, stdev= 7.06, samples=20 lat (msec) : 4=36.85%, 10=8.59%, 20=26.04%, 50=28.39%, 100=0.13% cpu : usr=0.10%, sys=7.95%, ctx=913, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2377KiB/s (2434kB/s), 570KiB/s-650KiB/s (584kB/s-665kB/s), io=23.4MiB (24.5MB), run=9629-10075msec WRITE: bw=2501KiB/s (2561kB/s), 624KiB/s-650KiB/s (639kB/s-665kB/s), io=24.6MiB (25.8MB), run=9629-10075msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7827: Tue Mar 11 09:39:28 2025 read: IOPS=6, BW=6712KiB/s (6873kB/s)(7168KiB/1068msec) clat (msec): min=27, max=587, avg=120.41, stdev=206.40 lat (msec): min=27, max=587, avg=120.42, stdev=206.40 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 35], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 42], 60.00th=[ 45], | 70.00th=[ 45], 80.00th=[ 70], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min=12263, max=12263, per=73.23%, avg=12263.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4794KiB/s (4909kB/s)(5120KiB/1068msec) clat (usec): min=20987, max=63878, avg=42401.04, stdev=19838.64 lat (usec): min=21039, max=63924, avg=42441.42, stdev=19831.01 clat percentiles (usec): | 1.00th=[21103], 5.00th=[21103], 10.00th=[21103], 20.00th=[21103], | 30.00th=[31851], 40.00th=[31851], 50.00th=[31851], 60.00th=[31851], | 70.00th=[63177], 80.00th=[63177], 90.00th=[63701], 95.00th=[63701], | 99.00th=[63701], 99.50th=[63701], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 8175, max= 8175, per=34.87%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=66.67%, 100=25.00%, 750=8.33% cpu : usr=0.00%, sys=27.37%, ctx=1489, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7828: Tue Mar 11 09:39:28 2025 read: IOPS=1, BW=1675KiB/s (1715kB/s)(2048KiB/1223msec) clat (usec): min=32246, max=33381, avg=32814.12, stdev=802.46 lat (usec): min=32247, max=33382, avg=32815.39, stdev=802.48 clat percentiles (usec): | 1.00th=[32375], 5.00th=[32375], 10.00th=[32375], 20.00th=[32375], | 30.00th=[32375], 40.00th=[32375], 50.00th=[32375], 60.00th=[33424], | 70.00th=[33424], 80.00th=[33424], 90.00th=[33424], 95.00th=[33424], | 99.00th=[33424], 99.50th=[33424], 99.90th=[33424], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 1700, max= 2048, per=11.19%, avg=1874.00, stdev=246.07, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=8, BW=8373KiB/s (8574kB/s)(10.0MiB/1223msec) clat (msec): min=20, max=628, avg=111.02, stdev=185.31 lat (msec): min=20, max=628, avg=111.06, stdev=185.30 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 23], | 30.00th=[ 30], 40.00th=[ 40], 50.00th=[ 44], 60.00th=[ 50], | 70.00th=[ 92], 80.00th=[ 92], 90.00th=[ 136], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min=10205, max=10205, per=43.53%, avg=10205.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 50=75.00%, 100=8.33%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=21.03%, ctx=432, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7829: Tue Mar 11 09:39:28 2025 read: IOPS=6, BW=6150KiB/s (6298kB/s)(6144KiB/999msec) clat (usec): min=355, max=40118, avg=19476.16, stdev=16111.17 lat (usec): min=355, max=40118, avg=19476.77, stdev=16111.14 clat percentiles (usec): | 1.00th=[ 355], 5.00th=[ 355], 10.00th=[ 355], 20.00th=[ 586], | 30.00th=[ 586], 40.00th=[19792], 50.00th=[19792], 60.00th=[26870], | 70.00th=[29230], 80.00th=[29230], 90.00th=[40109], 95.00th=[40109], | 99.00th=[40109], 99.50th=[40109], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] write: IOPS=6, BW=6150KiB/s (6298kB/s)(6144KiB/999msec) clat (msec): min=21, max=629, avg=144.36, stdev=240.35 lat (msec): min=21, max=629, avg=144.40, stdev=240.35 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 23], | 30.00th=[ 23], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 45], | 70.00th=[ 117], 80.00th=[ 117], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 4096, max= 4096, per=17.47%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 20=8.33%, 50=58.33%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=17.74%, ctx=296, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7830: Tue Mar 11 09:39:28 2025 read: IOPS=5, BW=5361KiB/s (5490kB/s)(5120KiB/955msec) clat (usec): min=356, max=39117, avg=13090.21, stdev=18005.00 lat (usec): min=357, max=39118, avg=13090.89, stdev=18005.09 clat percentiles (usec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 359], | 30.00th=[ 490], 40.00th=[ 490], 50.00th=[ 537], 60.00th=[ 537], | 70.00th=[25035], 80.00th=[25035], 90.00th=[39060], 95.00th=[39060], | 99.00th=[39060], 99.50th=[39060], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] write: IOPS=7, BW=7506KiB/s (7686kB/s)(7168KiB/955msec) clat (msec): min=15, max=640, avg=119.43, stdev=230.36 lat (msec): min=15, max=640, avg=119.48, stdev=230.36 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 18], | 30.00th=[ 23], 40.00th=[ 23], 50.00th=[ 28], 60.00th=[ 54], | 70.00th=[ 54], 80.00th=[ 59], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 2048, max= 2048, per=8.74%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 20=16.67%, 50=33.33%, 100=16.67%, 750=8.33% cpu : usr=0.10%, sys=14.47%, ctx=352, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.4MiB/s (17.1MB/s), 1675KiB/s-6712KiB/s (1715kB/s-6873kB/s), io=20.0MiB (20.0MB), run=955-1223msec WRITE: bw=22.9MiB/s (24.0MB/s), 4794KiB/s-8373KiB/s (4909kB/s-8574kB/s), io=28.0MiB (29.4MB), run=955-1223msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7831: Tue Mar 11 09:39:28 2025 read: IOPS=5, BW=5609KiB/s (5743kB/s)(7168KiB/1278msec) clat (msec): min=12, max=577, avg=135.82, stdev=213.15 lat (msec): min=12, max=577, avg=135.82, stdev=213.15 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 19], | 30.00th=[ 26], 40.00th=[ 26], 50.00th=[ 29], 60.00th=[ 34], | 70.00th=[ 34], 80.00th=[ 255], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 6144, max= 6144, per=39.93%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=3, BW=4006KiB/s (4102kB/s)(5120KiB/1278msec) clat (msec): min=32, max=160, avg=61.85, stdev=55.65 lat (msec): min=32, max=160, avg=61.90, stdev=55.64 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33], | 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 35], 60.00th=[ 35], | 70.00th=[ 48], 80.00th=[ 48], 90.00th=[ 161], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] lat (msec) : 20=16.67%, 50=58.33%, 250=8.33%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=5.56%, ctx=36, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7832: Tue Mar 11 09:39:28 2025 read: IOPS=1, BW=1539KiB/s (1576kB/s)(2048KiB/1331msec) clat (usec): min=17867, max=19218, avg=18543.01, stdev=955.14 lat (usec): min=17868, max=19219, avg=18543.89, stdev=955.42 clat percentiles (usec): | 1.00th=[17957], 5.00th=[17957], 10.00th=[17957], 20.00th=[17957], | 30.00th=[17957], 40.00th=[17957], 50.00th=[17957], 60.00th=[19268], | 70.00th=[19268], 80.00th=[19268], 90.00th=[19268], 95.00th=[19268], | 99.00th=[19268], 99.50th=[19268], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 2048, max= 2048, per=13.31%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7693KiB/s (7878kB/s)(10.0MiB/1331msec) clat (msec): min=25, max=693, avg=120.66, stdev=203.71 lat (msec): min=25, max=693, avg=120.72, stdev=203.70 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 31], | 30.00th=[ 32], 40.00th=[ 42], 50.00th=[ 44], 60.00th=[ 50], | 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 132], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 6131, max= 6131, per=28.46%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=50.00%, 100=16.67%, 250=8.33%, 750=8.33% cpu : usr=0.08%, sys=6.99%, ctx=35, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7833: Tue Mar 11 09:39:28 2025 read: IOPS=4, BW=4808KiB/s (4923kB/s)(6144KiB/1278msec) clat (usec): min=20765, max=85271, avg=48245.66, stdev=29412.03 lat (usec): min=20766, max=85272, avg=48246.40, stdev=29412.05 clat percentiles (usec): | 1.00th=[20841], 5.00th=[20841], 10.00th=[20841], 20.00th=[25297], | 30.00th=[25297], 40.00th=[32637], 50.00th=[32637], 60.00th=[40109], | 70.00th=[85459], 80.00th=[85459], 90.00th=[85459], 95.00th=[85459], | 99.00th=[85459], 99.50th=[85459], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 4087, max= 4087, per=26.56%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4808KiB/s (4923kB/s)(6144KiB/1278msec) clat (msec): min=43, max=637, avg=147.81, stdev=240.20 lat (msec): min=43, max=638, avg=147.87, stdev=240.20 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 44], | 30.00th=[ 44], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 57], | 70.00th=[ 57], 80.00th=[ 57], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 4087, max= 4096, per=18.99%, avg=4091.50, stdev= 6.36, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 lat (msec) : 50=58.33%, 100=33.33%, 750=8.33% cpu : usr=0.08%, sys=5.72%, ctx=43, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7834: Tue Mar 11 09:39:28 2025 read: IOPS=3, BW=3879KiB/s (3972kB/s)(5120KiB/1320msec) clat (usec): min=13839, max=85747, avg=38710.94, stdev=29247.54 lat (usec): min=13840, max=85748, avg=38711.59, stdev=29247.52 clat percentiles (usec): | 1.00th=[13829], 5.00th=[13829], 10.00th=[13829], 20.00th=[13829], | 30.00th=[14615], 40.00th=[14615], 50.00th=[37487], 60.00th=[37487], | 70.00th=[41681], 80.00th=[41681], 90.00th=[85459], 95.00th=[85459], | 99.00th=[85459], 99.50th=[85459], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] write: IOPS=5, BW=5430KiB/s (5561kB/s)(7168KiB/1320msec) clat (msec): min=30, max=655, avg=147.82, stdev=228.20 lat (msec): min=30, max=655, avg=147.88, stdev=228.19 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 32], | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 48], 60.00th=[ 73], | 70.00th=[ 73], 80.00th=[ 159], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 2048, max= 4079, per=14.22%, avg=3063.50, stdev=1436.13, samples=2 iops : min= 2, max= 3, avg= 2.50, stdev= 0.71, samples=2 lat (msec) : 20=16.67%, 50=50.00%, 100=16.67%, 250=8.33%, 750=8.33% cpu : usr=0.08%, sys=6.29%, ctx=65, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.0MiB/s (15.8MB/s), 1539KiB/s-5609KiB/s (1576kB/s-5743kB/s), io=20.0MiB (20.0MB), run=1278-1331msec WRITE: bw=21.0MiB/s (22.1MB/s), 4006KiB/s-7693KiB/s (4102kB/s-7878kB/s), io=28.0MiB (29.4MB), run=1278-1331msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7842: Tue Mar 11 09:39:30 2025 read: IOPS=3, BW=12.2MiB/s (12.8MB/s)(12.0MiB/981msec) clat (usec): min=1738, max=958126, avg=320679.75, stdev=552044.87 lat (usec): min=1739, max=958127, avg=320681.13, stdev=552044.61 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1745], 10.00th=[ 1745], 20.00th=[ 1745], | 30.00th=[ 1745], 40.00th=[ 2180], 50.00th=[ 2180], 60.00th=[ 2180], | 70.00th=[960496], 80.00th=[960496], 90.00th=[960496], 95.00th=[960496], | 99.00th=[960496], 99.50th=[960496], 99.90th=[960496], 99.95th=[960496], | 99.99th=[960496] lat (msec) : 2=33.33%, 4=33.33%, 1000=33.33% cpu : usr=0.10%, sys=16.43%, ctx=784, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7844: Tue Mar 11 09:39:30 2025 read: IOPS=2, BW=11.4MiB/s (11.0MB/s)(4096KiB/350msec) clat (nsec): min=114160k, max=114160k, avg=114160266.00, stdev= 0.00 lat (nsec): min=114163k, max=114163k, avg=114162940.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 114], 10.00th=[ 114], 20.00th=[ 114], | 30.00th=[ 114], 40.00th=[ 114], 50.00th=[ 114], 60.00th=[ 114], | 70.00th=[ 114], 80.00th=[ 114], 90.00th=[ 114], 95.00th=[ 114], | 99.00th=[ 114], 99.50th=[ 114], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] write: IOPS=5, BW=22.9MiB/s (23.0MB/s)(8192KiB/350msec) clat (msec): min=84, max=140, avg=112.67, stdev=39.92 lat (msec): min=84, max=141, avg=112.87, stdev=39.81 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 85], 10.00th=[ 85], 20.00th=[ 85], | 30.00th=[ 85], 40.00th=[ 85], 50.00th=[ 85], 60.00th=[ 142], | 70.00th=[ 142], 80.00th=[ 142], 90.00th=[ 142], 95.00th=[ 142], | 99.00th=[ 142], 99.50th=[ 142], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=54.15%, ctx=773, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7846: Tue Mar 11 09:39:30 2025 write: IOPS=1, BW=7647KiB/s (7830kB/s)(12.0MiB/1607msec) clat (msec): min=77, max=1311, avg=517.96, stdev=688.72 lat (msec): min=77, max=1312, avg=518.25, stdev=688.80 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 79], 10.00th=[ 79], 20.00th=[ 79], | 30.00th=[ 79], 40.00th=[ 165], 50.00th=[ 165], 60.00th=[ 165], | 70.00th=[ 1318], 80.00th=[ 1318], 90.00th=[ 1318], 95.00th=[ 1318], | 99.00th=[ 1318], 99.50th=[ 1318], 99.90th=[ 1318], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 8192, max= 8192, per=40.18%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=19.05%, ctx=65, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7848: Tue Mar 11 09:39:30 2025 write: IOPS=1, BW=7918KiB/s (8108kB/s)(12.0MiB/1552msec) clat (msec): min=81, max=1305, avg=499.74, stdev=698.08 lat (msec): min=82, max=1305, avg=500.04, stdev=698.03 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 82], 10.00th=[ 82], 20.00th=[ 82], | 30.00th=[ 82], 40.00th=[ 112], 50.00th=[ 112], 60.00th=[ 112], | 70.00th=[ 1301], 80.00th=[ 1301], 90.00th=[ 1301], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min=16384, max=16384, per=80.35%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=16.38%, ctx=49, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.3MiB/s (17.1MB/s), 11.4MiB/s-12.2MiB/s (11.0MB/s-12.8MB/s), io=16.0MiB (16.8MB), run=350-981msec WRITE: bw=19.9MiB/s (20.9MB/s), 7647KiB/s-22.9MiB/s (7830kB/s-23.0MB/s), io=32.0MiB (33.6MB), run=350-1607msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7841: Tue Mar 11 09:39:31 2025 read: IOPS=2, BW=9.91MiB/s (10.4MB/s)(12.0MiB/1211msec) clat (msec): min=53, max=1063, avg=400.35, stdev=574.46 lat (msec): min=53, max=1063, avg=400.36, stdev=574.46 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 54], 10.00th=[ 54], 20.00th=[ 54], | 30.00th=[ 54], 40.00th=[ 85], 50.00th=[ 85], 60.00th=[ 85], | 70.00th=[ 1062], 80.00th=[ 1062], 90.00th=[ 1062], 95.00th=[ 1062], | 99.00th=[ 1062], 99.50th=[ 1062], 99.90th=[ 1062], 99.95th=[ 1062], | 99.99th=[ 1062] lat (msec) : 100=66.67% cpu : usr=0.00%, sys=6.86%, ctx=59, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7843: Tue Mar 11 09:39:31 2025 read: IOPS=0, BW=2864KiB/s (2933kB/s)(4096KiB/1430msec) clat (nsec): min=1176.5M, max=1176.5M, avg=1176456975.00, stdev= 0.00 lat (nsec): min=1176.5M, max=1176.5M, avg=1176458085.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1183], 5.00th=[ 1183], 10.00th=[ 1183], 20.00th=[ 1183], | 30.00th=[ 1183], 40.00th=[ 1183], 50.00th=[ 1183], 60.00th=[ 1183], | 70.00th=[ 1183], 80.00th=[ 1183], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] write: IOPS=1, BW=5729KiB/s (5866kB/s)(8192KiB/1430msec) clat (msec): min=81, max=135, avg=108.86, stdev=38.08 lat (msec): min=82, max=136, avg=109.19, stdev=38.04 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 83], 10.00th=[ 83], 20.00th=[ 83], | 30.00th=[ 83], 40.00th=[ 83], 50.00th=[ 83], 60.00th=[ 136], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=6.37%, ctx=35, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7845: Tue Mar 11 09:39:31 2025 write: IOPS=1, BW=6575KiB/s (6732kB/s)(12.0MiB/1869msec) clat (msec): min=76, max=1642, avg=615.66, stdev=889.26 lat (msec): min=76, max=1643, avg=616.47, stdev=890.18 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 77], 10.00th=[ 77], 20.00th=[ 77], | 30.00th=[ 77], 40.00th=[ 129], 50.00th=[ 129], 60.00th=[ 129], | 70.00th=[ 1636], 80.00th=[ 1636], 90.00th=[ 1636], 95.00th=[ 1636], | 99.00th=[ 1636], 99.50th=[ 1636], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.05%, sys=6.26%, ctx=84, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7847: Tue Mar 11 09:39:31 2025 write: IOPS=1, BW=6958KiB/s (7125kB/s)(12.0MiB/1766msec) clat (msec): min=66, max=1611, avg=585.93, stdev=888.10 lat (msec): min=66, max=1611, avg=586.18, stdev=888.11 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 67], | 30.00th=[ 67], 40.00th=[ 81], 50.00th=[ 81], 60.00th=[ 81], | 70.00th=[ 1620], 80.00th=[ 1620], 90.00th=[ 1620], 95.00th=[ 1620], | 99.00th=[ 1620], 99.50th=[ 1620], 99.90th=[ 1620], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min=16384, max=16384, per=93.45%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=66.67% cpu : usr=0.06%, sys=3.97%, ctx=139, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.2MiB/s (11.7MB/s), 2864KiB/s-9.91MiB/s (2933kB/s-10.4MB/s), io=16.0MiB (16.8MB), run=1211-1430msec WRITE: bw=17.1MiB/s (17.0MB/s), 5729KiB/s-6958KiB/s (5866kB/s-7125kB/s), io=32.0MiB (33.6MB), run=1430-1869msec