== sanity test 398b: DIO and buffer IO race ============== 23:40:57 (1708836057) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.22346 s, 41.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6755: Sat Feb 24 23:41:14 2024 read: IOPS=109, BW=439KiB/s (449kB/s)(5992KiB/13651msec) clat (usec): min=51, max=25652, avg=4875.81, stdev=3001.09 lat (usec): min=52, max=25653, avg=4877.77, stdev=3000.49 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 112], 10.00th=[ 2180], 20.00th=[ 2704], | 30.00th=[ 3130], 40.00th=[ 3556], 50.00th=[ 4293], 60.00th=[ 5145], | 70.00th=[ 5866], 80.00th=[ 6783], 90.00th=[ 8291], 95.00th=[10028], | 99.00th=[15139], 99.50th=[17433], 99.90th=[24773], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 320, max= 504, per=24.90%, avg=436.33, stdev=47.31, samples=27 iops : min= 80, max= 126, avg=109.00, stdev=11.80, samples=27 write: IOPS=115, BW=461KiB/s (472kB/s)(6296KiB/13651msec) clat (usec): min=428, max=143832, avg=3994.98, stdev=4892.76 lat (usec): min=429, max=143833, avg=3995.89, stdev=4892.75 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 537], 10.00th=[ 594], 20.00th=[ 734], | 30.00th=[ 914], 40.00th=[ 2343], 50.00th=[ 3261], 60.00th=[ 4047], | 70.00th=[ 5211], 80.00th=[ 6587], 90.00th=[ 8717], 95.00th=[ 10552], | 99.00th=[ 14091], 99.50th=[ 15795], 99.90th=[ 21103], 99.95th=[143655], | 99.99th=[143655] bw ( KiB/s): min= 368, max= 600, per=24.91%, avg=456.78, stdev=57.00, samples=27 iops : min= 92, max= 150, avg=114.11, stdev=14.24, samples=27 lat (usec) : 100=1.95%, 250=0.78%, 500=1.66%, 750=9.51%, 1000=5.37% lat (msec) : 2=4.17%, 4=29.62%, 10=41.18%, 20=5.57%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.28%, sys=22.29%, ctx=5721, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6756: Sat Feb 24 23:41:14 2024 read: IOPS=109, BW=437KiB/s (447kB/s)(5940KiB/13598msec) clat (usec): min=50, max=142137, avg=4897.54, stdev=4638.13 lat (usec): min=50, max=142138, avg=4898.43, stdev=4638.09 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 110], 10.00th=[ 2212], 20.00th=[ 2671], | 30.00th=[ 3097], 40.00th=[ 3490], 50.00th=[ 4146], 60.00th=[ 4883], | 70.00th=[ 5866], 80.00th=[ 6849], 90.00th=[ 8356], 95.00th=[ 10290], | 99.00th=[ 15926], 99.50th=[ 17957], 99.90th=[ 23987], 99.95th=[141558], | 99.99th=[141558] bw ( KiB/s): min= 336, max= 520, per=24.90%, avg=436.33, stdev=56.56, samples=27 iops : min= 84, max= 130, avg=109.04, stdev=14.13, samples=27 write: IOPS=116, BW=467KiB/s (478kB/s)(6348KiB/13598msec) clat (usec): min=378, max=28607, avg=3949.54, stdev=3565.05 lat (usec): min=379, max=28608, avg=3950.53, stdev=3565.12 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 545], 10.00th=[ 603], 20.00th=[ 725], | 30.00th=[ 922], 40.00th=[ 2212], 50.00th=[ 3228], 60.00th=[ 3982], | 70.00th=[ 5145], 80.00th=[ 6718], 90.00th=[ 8979], 95.00th=[10814], | 99.00th=[15533], 99.50th=[16909], 99.90th=[26870], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 376, max= 640, per=25.34%, avg=464.81, stdev=64.45, samples=27 iops : min= 94, max= 160, avg=116.15, stdev=16.13, samples=27 lat (usec) : 100=1.92%, 250=0.88%, 500=1.01%, 750=10.48%, 1000=5.14% lat (msec) : 2=4.39%, 4=30.50%, 10=39.32%, 20=6.22%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=22.17%, ctx=5725, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6757: Sat Feb 24 23:41:14 2024 read: IOPS=109, BW=440KiB/s (451kB/s)(6028KiB/13701msec) clat (usec): min=40, max=145326, avg=4907.39, stdev=4696.76 lat (usec): min=40, max=145327, avg=4909.16, stdev=4696.64 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 101], 10.00th=[ 2147], 20.00th=[ 2671], | 30.00th=[ 3064], 40.00th=[ 3589], 50.00th=[ 4293], 60.00th=[ 5145], | 70.00th=[ 5932], 80.00th=[ 6849], 90.00th=[ 8160], 95.00th=[ 9765], | 99.00th=[ 16319], 99.50th=[ 18220], 99.90th=[ 22676], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 296, max= 736, per=24.99%, avg=437.78, stdev=82.36, samples=27 iops : min= 74, max= 184, avg=109.37, stdev=20.58, samples=27 write: IOPS=114, BW=457KiB/s (468kB/s)(6260KiB/13701msec) clat (usec): min=388, max=33093, avg=3992.42, stdev=3707.15 lat (usec): min=389, max=33094, avg=3994.06, stdev=3706.71 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 553], 10.00th=[ 611], 20.00th=[ 709], | 30.00th=[ 881], 40.00th=[ 2024], 50.00th=[ 3228], 60.00th=[ 3982], | 70.00th=[ 5211], 80.00th=[ 6783], 90.00th=[ 9241], 95.00th=[10945], | 99.00th=[15270], 99.50th=[17957], 99.90th=[30802], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 304, max= 768, per=24.76%, avg=454.11, stdev=88.67, samples=27 iops : min= 76, max= 192, avg=113.44, stdev=22.20, samples=27 lat (usec) : 50=0.10%, 100=2.21%, 250=1.20%, 500=1.17%, 750=10.68% lat (usec) : 1000=4.72% lat (msec) : 2=4.49%, 4=28.52%, 10=40.89%, 20=5.73%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=21.92%, ctx=5688, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6758: Sat Feb 24 23:41:14 2024 read: IOPS=112, BW=448KiB/s (459kB/s)(6052KiB/13503msec) clat (usec): min=51, max=25074, avg=4716.89, stdev=2906.60 lat (usec): min=51, max=25074, avg=4719.63, stdev=2906.04 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 109], 10.00th=[ 2114], 20.00th=[ 2606], | 30.00th=[ 2966], 40.00th=[ 3458], 50.00th=[ 4113], 60.00th=[ 4948], | 70.00th=[ 5735], 80.00th=[ 6783], 90.00th=[ 8455], 95.00th=[10028], | 99.00th=[14484], 99.50th=[16188], 99.90th=[19530], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 280, max= 688, per=25.58%, avg=448.07, stdev=74.95, samples=27 iops : min= 70, max= 172, avg=111.89, stdev=18.74, samples=27 write: IOPS=115, BW=462KiB/s (473kB/s)(6236KiB/13503msec) clat (usec): min=428, max=145191, avg=4043.35, stdev=5214.99 lat (usec): min=429, max=145192, avg=4044.25, stdev=5214.99 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 537], 10.00th=[ 603], 20.00th=[ 709], | 30.00th=[ 865], 40.00th=[ 1647], 50.00th=[ 3163], 60.00th=[ 3982], | 70.00th=[ 5145], 80.00th=[ 6718], 90.00th=[ 8848], 95.00th=[ 11207], | 99.00th=[ 16057], 99.50th=[ 18220], 99.90th=[ 34341], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 352, max= 696, per=25.16%, avg=461.41, stdev=73.78, samples=27 iops : min= 88, max= 174, avg=115.22, stdev=18.50, samples=27 lat (usec) : 100=2.02%, 250=1.27%, 500=1.20%, 750=11.04%, 1000=4.98% lat (msec) : 2=4.79%, 4=29.13%, 10=39.55%, 20=5.79%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=22.12%, ctx=5631, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1753KiB/s (1795kB/s), 437KiB/s-448KiB/s (447kB/s-459kB/s), io=23.4MiB (24.6MB), run=13503-13701msec WRITE: bw=1835KiB/s (1879kB/s), 457KiB/s-467KiB/s (468kB/s-478kB/s), io=24.6MiB (25.7MB), run=13503-13701msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6760: Sat Feb 24 23:41:17 2024 read: IOPS=86, BW=345KiB/s (353kB/s)(5992KiB/17359msec) clat (usec): min=1698, max=148465, avg=4703.36, stdev=5193.49 lat (usec): min=1698, max=148466, avg=4704.24, stdev=5193.51 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1926], 10.00th=[ 2024], 20.00th=[ 2180], | 30.00th=[ 2343], 40.00th=[ 2638], 50.00th=[ 3064], 60.00th=[ 3687], | 70.00th=[ 4752], 80.00th=[ 6652], 90.00th=[ 9765], 95.00th=[ 12518], | 99.00th=[ 16712], 99.50th=[ 21627], 99.90th=[ 33817], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 184, max= 768, per=24.62%, avg=338.50, stdev=170.77, samples=34 iops : min= 46, max= 192, avg=84.56, stdev=42.71, samples=34 write: IOPS=90, BW=363KiB/s (371kB/s)(6296KiB/17359msec) clat (usec): min=2229, max=35507, avg=6534.99, stdev=4218.48 lat (usec): min=2230, max=35508, avg=6536.01, stdev=4218.44 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2966], 10.00th=[ 3195], 20.00th=[ 3425], | 30.00th=[ 3654], 40.00th=[ 4015], 50.00th=[ 4686], 60.00th=[ 5866], | 70.00th=[ 7701], 80.00th=[ 9503], 90.00th=[12256], 95.00th=[15008], | 99.00th=[21890], 99.50th=[26608], 99.90th=[29230], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 199, max= 744, per=24.62%, avg=354.50, stdev=174.15, samples=34 iops : min= 49, max= 186, avg=88.56, stdev=43.58, samples=34 lat (msec) : 2=4.23%, 4=47.01%, 10=34.80%, 20=13.05%, 50=0.88% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=10.71%, ctx=3456, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6761: Sat Feb 24 23:41:17 2024 read: IOPS=85, BW=341KiB/s (349kB/s)(5940KiB/17430msec) clat (usec): min=1653, max=147775, avg=4670.67, stdev=5256.63 lat (usec): min=1654, max=147776, avg=4671.55, stdev=5256.64 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1926], 10.00th=[ 2008], 20.00th=[ 2180], | 30.00th=[ 2311], 40.00th=[ 2540], 50.00th=[ 2966], 60.00th=[ 3556], | 70.00th=[ 4424], 80.00th=[ 6259], 90.00th=[ 10552], 95.00th=[ 12780], | 99.00th=[ 18220], 99.50th=[ 21103], 99.90th=[ 26608], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 168, max= 784, per=24.38%, avg=335.24, stdev=183.03, samples=34 iops : min= 42, max= 196, avg=83.76, stdev=45.79, samples=34 write: IOPS=91, BW=364KiB/s (373kB/s)(6348KiB/17430msec) clat (usec): min=2325, max=35158, avg=6595.30, stdev=4334.66 lat (usec): min=2326, max=35159, avg=6596.32, stdev=4334.69 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2900], 10.00th=[ 3130], 20.00th=[ 3392], | 30.00th=[ 3621], 40.00th=[ 3982], 50.00th=[ 4686], 60.00th=[ 5997], | 70.00th=[ 7898], 80.00th=[ 9765], 90.00th=[12518], 95.00th=[14746], | 99.00th=[22676], 99.50th=[26084], 99.90th=[32637], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 224, max= 744, per=24.49%, avg=352.65, stdev=170.28, samples=34 iops : min= 56, max= 186, avg=88.12, stdev=42.59, samples=34 lat (msec) : 2=4.23%, 4=48.40%, 10=32.52%, 20=13.87%, 50=0.94% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=10.80%, ctx=3458, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6762: Sat Feb 24 23:41:17 2024 read: IOPS=86, BW=348KiB/s (356kB/s)(6028KiB/17346msec) clat (usec): min=1633, max=26879, avg=4650.85, stdev=3674.57 lat (usec): min=1634, max=26880, avg=4652.02, stdev=3674.72 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1958], 10.00th=[ 2040], 20.00th=[ 2212], | 30.00th=[ 2376], 40.00th=[ 2638], 50.00th=[ 3032], 60.00th=[ 3556], | 70.00th=[ 4621], 80.00th=[ 6849], 90.00th=[10421], 95.00th=[12518], | 99.00th=[17433], 99.50th=[20055], 99.90th=[26084], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 184, max= 720, per=24.84%, avg=341.56, stdev=170.12, samples=34 iops : min= 46, max= 180, avg=85.32, stdev=42.57, samples=34 write: IOPS=90, BW=361KiB/s (370kB/s)(6260KiB/17346msec) clat (msec): min=2, max=156, avg= 6.59, stdev= 5.66 lat (msec): min=2, max=156, avg= 6.59, stdev= 5.66 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 23], 99.50th=[ 25], 99.90th=[ 36], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 215, max= 784, per=24.49%, avg=352.62, stdev=176.02, samples=34 iops : min= 53, max= 196, avg=88.09, stdev=44.05, samples=34 lat (msec) : 2=3.52%, 4=48.50%, 10=33.50%, 20=13.51%, 50=0.94% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=10.85%, ctx=3506, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6763: Sat Feb 24 23:41:17 2024 read: IOPS=86, BW=347KiB/s (355kB/s)(6052KiB/17456msec) clat (usec): min=1589, max=27522, avg=4655.99, stdev=3759.14 lat (usec): min=1589, max=27523, avg=4656.79, stdev=3759.14 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1942], 10.00th=[ 2024], 20.00th=[ 2180], | 30.00th=[ 2343], 40.00th=[ 2606], 50.00th=[ 2999], 60.00th=[ 3523], | 70.00th=[ 4621], 80.00th=[ 7046], 90.00th=[10159], 95.00th=[12256], | 99.00th=[19268], 99.50th=[21365], 99.90th=[26870], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 184, max= 712, per=24.46%, avg=336.35, stdev=170.47, samples=34 iops : min= 46, max= 178, avg=84.03, stdev=42.60, samples=34 write: IOPS=89, BW=357KiB/s (366kB/s)(6236KiB/17456msec) clat (msec): min=2, max=154, avg= 6.66, stdev= 5.67 lat (msec): min=2, max=154, avg= 6.66, stdev= 5.67 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 23], 99.50th=[ 27], 99.90th=[ 36], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 208, max= 768, per=24.11%, avg=347.18, stdev=176.92, samples=34 iops : min= 52, max= 192, avg=86.74, stdev=44.22, samples=34 lat (msec) : 2=4.07%, 4=48.31%, 10=32.85%, 20=13.57%, 50=1.17% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=10.77%, ctx=3465, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1376KiB/s (1409kB/s), 341KiB/s-348KiB/s (349kB/s-356kB/s), io=23.4MiB (24.6MB), run=17346-17456msec WRITE: bw=1440KiB/s (1475kB/s), 357KiB/s-364KiB/s (366kB/s-373kB/s), io=24.6MiB (25.7MB), run=17346-17456msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6799: Sat Feb 24 23:41:22 2024 read: IOPS=101, BW=1626KiB/s (1665kB/s)(5920KiB/3641msec) clat (usec): min=52, max=19829, avg=5250.21, stdev=3000.12 lat (usec): min=52, max=19831, avg=5254.71, stdev=3005.22 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 86], 10.00th=[ 127], 20.00th=[ 3294], | 30.00th=[ 3884], 40.00th=[ 4555], 50.00th=[ 5145], 60.00th=[ 5604], | 70.00th=[ 6325], 80.00th=[ 7111], 90.00th=[ 8848], 95.00th=[ 9896], | 99.00th=[15533], 99.50th=[17171], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 1088, max= 1920, per=25.34%, avg=1586.29, stdev=265.08, samples=7 iops : min= 68, max= 120, avg=99.14, stdev=16.57, samples=7 write: IOPS=109, BW=1749KiB/s (1791kB/s)(6368KiB/3641msec) clat (usec): min=623, max=16891, avg=4236.10, stdev=3421.43 lat (usec): min=623, max=16892, avg=4237.60, stdev=3421.77 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 832], 10.00th=[ 914], 20.00th=[ 1074], | 30.00th=[ 1352], 40.00th=[ 2180], 50.00th=[ 3326], 60.00th=[ 4359], | 70.00th=[ 5669], 80.00th=[ 7046], 90.00th=[ 9503], 95.00th=[10683], | 99.00th=[14091], 99.50th=[16057], 99.90th=[16909], 99.95th=[16909], | 99.99th=[16909] bw ( KiB/s): min= 1248, max= 2368, per=25.82%, avg=1700.57, stdev=372.46, samples=7 iops : min= 78, max= 148, avg=106.29, stdev=23.28, samples=7 lat (usec) : 100=2.86%, 250=2.34%, 500=0.13%, 750=0.65%, 1000=7.16% lat (msec) : 2=12.11%, 4=19.14%, 10=49.48%, 20=6.12% cpu : usr=0.33%, sys=26.43%, ctx=1669, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6800: Sat Feb 24 23:41:22 2024 read: IOPS=93, BW=1501KiB/s (1537kB/s)(5744KiB/3826msec) clat (usec): min=57, max=21375, avg=5528.17, stdev=3209.38 lat (usec): min=57, max=21376, avg=5529.54, stdev=3209.61 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 97], 10.00th=[ 243], 20.00th=[ 3359], | 30.00th=[ 3949], 40.00th=[ 4686], 50.00th=[ 5407], 60.00th=[ 5932], | 70.00th=[ 6652], 80.00th=[ 7635], 90.00th=[ 9110], 95.00th=[10683], | 99.00th=[17171], 99.50th=[21103], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 1152, max= 1792, per=23.28%, avg=1457.29, stdev=209.30, samples=7 iops : min= 72, max= 112, avg=90.86, stdev=13.02, samples=7 write: IOPS=106, BW=1710KiB/s (1751kB/s)(6544KiB/3826msec) clat (usec): min=621, max=21062, avg=4472.55, stdev=3763.88 lat (usec): min=622, max=21063, avg=4473.73, stdev=3763.91 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 840], 10.00th=[ 922], 20.00th=[ 1106], | 30.00th=[ 1369], 40.00th=[ 2606], 50.00th=[ 3621], 60.00th=[ 4424], | 70.00th=[ 5604], 80.00th=[ 7373], 90.00th=[ 9634], 95.00th=[11863], | 99.00th=[17695], 99.50th=[18482], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 1309, max= 2011, per=24.83%, avg=1635.43, stdev=222.88, samples=7 iops : min= 81, max= 125, avg=102.00, stdev=13.94, samples=7 lat (usec) : 100=2.60%, 250=2.08%, 750=0.65%, 1000=6.38% lat (msec) : 2=11.59%, 4=21.09%, 10=48.05%, 20=7.03%, 50=0.52% cpu : usr=0.21%, sys=25.54%, ctx=1665, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6801: Sat Feb 24 23:41:22 2024 read: IOPS=105, BW=1687KiB/s (1728kB/s)(6256KiB/3708msec) clat (usec): min=62, max=18819, avg=5271.47, stdev=2843.09 lat (usec): min=63, max=18820, avg=5272.27, stdev=2843.09 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 98], 10.00th=[ 2409], 20.00th=[ 3425], | 30.00th=[ 3884], 40.00th=[ 4424], 50.00th=[ 4948], 60.00th=[ 5800], | 70.00th=[ 6521], 80.00th=[ 7242], 90.00th=[ 8979], 95.00th=[10421], | 99.00th=[13042], 99.50th=[15926], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 1248, max= 2176, per=27.02%, avg=1691.43, stdev=295.85, samples=7 iops : min= 78, max= 136, avg=105.71, stdev=18.49, samples=7 write: IOPS=101, BW=1627KiB/s (1666kB/s)(6032KiB/3708msec) clat (usec): min=684, max=20937, avg=4341.42, stdev=3600.94 lat (usec): min=686, max=20938, avg=4344.66, stdev=3599.45 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 816], 10.00th=[ 914], 20.00th=[ 1123], | 30.00th=[ 1516], 40.00th=[ 2802], 50.00th=[ 3425], 60.00th=[ 4359], | 70.00th=[ 5342], 80.00th=[ 6783], 90.00th=[ 8979], 95.00th=[11863], | 99.00th=[15664], 99.50th=[20317], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 1248, max= 1952, per=24.16%, avg=1590.86, stdev=307.88, samples=7 iops : min= 78, max= 122, avg=99.43, stdev=19.24, samples=7 lat (usec) : 100=2.60%, 250=1.95%, 500=0.13%, 750=0.65%, 1000=6.38% lat (msec) : 2=9.64%, 4=22.66%, 10=49.61%, 20=6.12%, 50=0.26% cpu : usr=0.22%, sys=26.22%, ctx=1769, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6802: Sat Feb 24 23:41:22 2024 read: IOPS=101, BW=1629KiB/s (1668kB/s)(6032KiB/3704msec) clat (usec): min=59, max=23793, avg=5399.74, stdev=3112.17 lat (usec): min=60, max=23794, avg=5400.70, stdev=3112.12 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 93], 10.00th=[ 149], 20.00th=[ 3359], | 30.00th=[ 3916], 40.00th=[ 4424], 50.00th=[ 5145], 60.00th=[ 5932], | 70.00th=[ 6587], 80.00th=[ 7635], 90.00th=[ 8848], 95.00th=[10552], | 99.00th=[14091], 99.50th=[19006], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 1376, max= 1920, per=26.06%, avg=1631.43, stdev=199.85, samples=7 iops : min= 86, max= 120, avg=101.86, stdev=12.33, samples=7 write: IOPS=105, BW=1689KiB/s (1730kB/s)(6256KiB/3704msec) clat (usec): min=655, max=18931, avg=4239.94, stdev=3436.23 lat (usec): min=656, max=18932, avg=4241.07, stdev=3436.26 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 799], 10.00th=[ 881], 20.00th=[ 1090], | 30.00th=[ 1352], 40.00th=[ 2212], 50.00th=[ 3294], 60.00th=[ 4228], | 70.00th=[ 5800], 80.00th=[ 7308], 90.00th=[ 9110], 95.00th=[10945], | 99.00th=[13698], 99.50th=[16712], 99.90th=[19006], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 1408, max= 1856, per=25.95%, avg=1709.14, stdev=156.13, samples=7 iops : min= 88, max= 116, avg=106.71, stdev= 9.67, samples=7 lat (usec) : 100=2.99%, 250=2.08%, 500=0.13%, 750=1.04%, 1000=7.16% lat (msec) : 2=11.33%, 4=20.18%, 10=48.18%, 20=6.77%, 50=0.13% cpu : usr=0.11%, sys=26.55%, ctx=1729, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6260KiB/s (6411kB/s), 1501KiB/s-1687KiB/s (1537kB/s-1728kB/s), io=23.4MiB (24.5MB), run=3641-3826msec WRITE: bw=6587KiB/s (6745kB/s), 1627KiB/s-1749KiB/s (1666kB/s-1791kB/s), io=24.6MiB (25.8MB), run=3641-3826msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6795: Sat Feb 24 23:41:23 2024 read: IOPS=78, BW=1248KiB/s (1278kB/s)(5920KiB/4743msec) clat (usec): min=2020, max=23434, avg=5092.18, stdev=4188.87 lat (usec): min=2021, max=23434, avg=5092.99, stdev=4188.88 clat percentiles (usec): | 1.00th=[ 2073], 5.00th=[ 2180], 10.00th=[ 2311], 20.00th=[ 2474], | 30.00th=[ 2671], 40.00th=[ 2900], 50.00th=[ 3294], 60.00th=[ 3818], | 70.00th=[ 4555], 80.00th=[ 6849], 90.00th=[11994], 95.00th=[15401], | 99.00th=[21103], 99.50th=[22414], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 830, max= 2432, per=23.38%, avg=1180.22, stdev=510.78, samples=9 iops : min= 51, max= 152, avg=73.67, stdev=32.00, samples=9 write: IOPS=83, BW=1343KiB/s (1375kB/s)(6368KiB/4743msec) clat (usec): min=2676, max=22723, avg=7153.63, stdev=4289.47 lat (usec): min=2678, max=22724, avg=7155.30, stdev=4289.52 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3228], 10.00th=[ 3458], 20.00th=[ 3752], | 30.00th=[ 4113], 40.00th=[ 4621], 50.00th=[ 5407], 60.00th=[ 6718], | 70.00th=[ 8160], 80.00th=[10421], 90.00th=[14222], 95.00th=[16909], | 99.00th=[21365], 99.50th=[22152], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 832, max= 2656, per=24.15%, avg=1283.22, stdev=556.98, samples=9 iops : min= 52, max= 166, avg=80.11, stdev=34.84, samples=9 lat (msec) : 4=44.27%, 10=38.15%, 20=16.41%, 50=1.17% cpu : usr=0.21%, sys=12.21%, ctx=923, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6796: Sat Feb 24 23:41:23 2024 read: IOPS=76, BW=1219KiB/s (1249kB/s)(5744KiB/4711msec) clat (usec): min=2038, max=24232, avg=5160.10, stdev=3816.30 lat (usec): min=2038, max=24233, avg=5160.96, stdev=3816.30 clat percentiles (usec): | 1.00th=[ 2089], 5.00th=[ 2212], 10.00th=[ 2311], 20.00th=[ 2540], | 30.00th=[ 2802], 40.00th=[ 3195], 50.00th=[ 3523], 60.00th=[ 4080], | 70.00th=[ 5211], 80.00th=[ 7308], 90.00th=[11338], 95.00th=[13173], | 99.00th=[19006], 99.50th=[21103], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 832, max= 2144, per=23.37%, avg=1180.11, stdev=440.95, samples=9 iops : min= 52, max= 134, avg=73.67, stdev=27.57, samples=9 write: IOPS=86, BW=1389KiB/s (1422kB/s)(6544KiB/4711msec) clat (usec): min=2667, max=24185, avg=6946.87, stdev=4120.20 lat (usec): min=2668, max=24186, avg=6948.46, stdev=4120.37 clat percentiles (usec): | 1.00th=[ 2802], 5.00th=[ 3195], 10.00th=[ 3458], 20.00th=[ 3720], | 30.00th=[ 3982], 40.00th=[ 4359], 50.00th=[ 5211], 60.00th=[ 6521], | 70.00th=[ 8160], 80.00th=[10159], 90.00th=[12780], 95.00th=[15401], | 99.00th=[20579], 99.50th=[21890], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 960, max= 2784, per=24.89%, avg=1322.33, stdev=561.34, samples=9 iops : min= 60, max= 174, avg=82.56, stdev=35.14, samples=9 lat (msec) : 4=43.36%, 10=39.45%, 20=16.15%, 50=1.04% cpu : usr=0.15%, sys=12.10%, ctx=928, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6797: Sat Feb 24 23:41:23 2024 read: IOPS=82, BW=1322KiB/s (1354kB/s)(6256KiB/4733msec) clat (usec): min=1828, max=22617, avg=5182.45, stdev=3954.46 lat (usec): min=1829, max=22618, avg=5183.33, stdev=3954.61 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2180], 10.00th=[ 2343], 20.00th=[ 2507], | 30.00th=[ 2802], 40.00th=[ 3130], 50.00th=[ 3523], 60.00th=[ 4113], | 70.00th=[ 5276], 80.00th=[ 7046], 90.00th=[11600], 95.00th=[13566], | 99.00th=[20055], 99.50th=[20317], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 800, max= 2752, per=25.07%, avg=1265.78, stdev=590.29, samples=9 iops : min= 50, max= 172, avg=79.11, stdev=36.89, samples=9 write: IOPS=79, BW=1274KiB/s (1305kB/s)(6032KiB/4733msec) clat (usec): min=2666, max=22597, avg=7132.64, stdev=4113.17 lat (usec): min=2666, max=22598, avg=7134.22, stdev=4113.22 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3294], 10.00th=[ 3458], 20.00th=[ 3752], | 30.00th=[ 4047], 40.00th=[ 4490], 50.00th=[ 5211], 60.00th=[ 6915], | 70.00th=[ 8717], 80.00th=[10421], 90.00th=[13698], 95.00th=[15533], | 99.00th=[19006], 99.50th=[20317], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 960, max= 2400, per=22.75%, avg=1208.89, stdev=460.89, samples=9 iops : min= 60, max= 150, avg=75.56, stdev=28.81, samples=9 lat (msec) : 2=0.65%, 4=42.97%, 10=38.67%, 20=17.06%, 50=0.65% cpu : usr=0.19%, sys=12.13%, ctx=901, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6798: Sat Feb 24 23:41:23 2024 read: IOPS=80, BW=1291KiB/s (1322kB/s)(6032KiB/4674msec) clat (usec): min=2016, max=22570, avg=5224.37, stdev=3955.17 lat (usec): min=2016, max=22571, avg=5225.14, stdev=3955.19 clat percentiles (usec): | 1.00th=[ 2057], 5.00th=[ 2245], 10.00th=[ 2376], 20.00th=[ 2540], | 30.00th=[ 2704], 40.00th=[ 2999], 50.00th=[ 3458], 60.00th=[ 4047], | 70.00th=[ 5538], 80.00th=[ 7373], 90.00th=[10683], 95.00th=[14353], | 99.00th=[20317], 99.50th=[21890], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 830, max= 2560, per=24.43%, avg=1233.22, stdev=521.80, samples=9 iops : min= 51, max= 160, avg=76.89, stdev=32.68, samples=9 write: IOPS=83, BW=1338KiB/s (1371kB/s)(6256KiB/4674msec) clat (usec): min=2511, max=22504, avg=6874.75, stdev=3903.79 lat (usec): min=2512, max=22505, avg=6876.30, stdev=3903.84 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 3130], 10.00th=[ 3523], 20.00th=[ 3752], | 30.00th=[ 4178], 40.00th=[ 4686], 50.00th=[ 5538], 60.00th=[ 6652], | 70.00th=[ 7898], 80.00th=[ 9241], 90.00th=[12387], 95.00th=[15008], | 99.00th=[20841], 99.50th=[22414], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 928, max= 2464, per=24.41%, avg=1297.00, stdev=505.62, samples=9 iops : min= 58, max= 154, avg=80.89, stdev=31.54, samples=9 lat (msec) : 4=42.19%, 10=43.62%, 20=12.89%, 50=1.30% cpu : usr=0.06%, sys=12.48%, ctx=893, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5050KiB/s (5171kB/s), 1219KiB/s-1322KiB/s (1249kB/s-1354kB/s), io=23.4MiB (24.5MB), run=4674-4743msec WRITE: bw=5313KiB/s (5441kB/s), 1274KiB/s-1389KiB/s (1305kB/s-1422kB/s), io=24.6MiB (25.8MB), run=4674-4743msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6811: Sat Feb 24 23:41:25 2024 read: IOPS=8, BW=8720KiB/s (8929kB/s)(7168KiB/822msec) clat (usec): min=25167, max=95137, avg=56471.24, stdev=24405.39 lat (usec): min=25169, max=95139, avg=56472.96, stdev=24405.41 clat percentiles (usec): | 1.00th=[25297], 5.00th=[25297], 10.00th=[25297], 20.00th=[40633], | 30.00th=[41157], 40.00th=[41157], 50.00th=[47973], 60.00th=[69731], | 70.00th=[69731], 80.00th=[74974], 90.00th=[94897], 95.00th=[94897], | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min=10240, max=10240, per=45.90%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6229KiB/s (6378kB/s)(5120KiB/822msec) clat (msec): min=32, max=128, avg=84.11, stdev=34.59 lat (msec): min=32, max=129, avg=84.20, stdev=34.59 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33], | 30.00th=[ 79], 40.00th=[ 79], 50.00th=[ 90], 60.00th=[ 90], | 70.00th=[ 92], 80.00th=[ 92], 90.00th=[ 129], 95.00th=[ 129], | 99.00th=[ 129], 99.50th=[ 129], 99.90th=[ 129], 99.95th=[ 129], | 99.99th=[ 129] bw ( KiB/s): min= 4096, max= 4096, per=13.11%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=50.00%, 250=8.33% cpu : usr=0.12%, sys=20.10%, ctx=52, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6813: Sat Feb 24 23:41:25 2024 read: IOPS=2, BW=2231KiB/s (2284kB/s)(2048KiB/918msec) clat (usec): min=25005, max=67668, avg=46337.00, stdev=30166.99 lat (usec): min=25007, max=67669, avg=46338.42, stdev=30166.48 clat percentiles (usec): | 1.00th=[25035], 5.00th=[25035], 10.00th=[25035], 20.00th=[25035], | 30.00th=[25035], 40.00th=[25035], 50.00th=[25035], 60.00th=[67634], | 70.00th=[67634], 80.00th=[67634], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 4096, max= 4096, per=18.36%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.9MiB/s (11.4MB/s)(10.0MiB/918msec) clat (msec): min=30, max=167, avg=81.86, stdev=41.84 lat (msec): min=30, max=168, avg=81.93, stdev=41.82 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 39], | 30.00th=[ 53], 40.00th=[ 55], 50.00th=[ 74], 60.00th=[ 86], | 70.00th=[ 103], 80.00th=[ 103], 90.00th=[ 123], 95.00th=[ 169], | 99.00th=[ 169], 99.50th=[ 169], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min=10240, max=10240, per=32.79%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=25.00%, 100=50.00%, 250=25.00% cpu : usr=0.00%, sys=17.12%, ctx=64, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6814: Sat Feb 24 23:41:25 2024 read: IOPS=6, BW=6919KiB/s (7085kB/s)(6144KiB/888msec) clat (msec): min=36, max=123, avg=67.99, stdev=32.95 lat (msec): min=36, max=123, avg=67.99, stdev=32.95 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 74], | 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 124], 95.00th=[ 124], | 99.00th=[ 124], 99.50th=[ 124], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 4096, max= 4096, per=18.36%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6919KiB/s (7085kB/s)(6144KiB/888msec) clat (msec): min=22, max=149, avg=79.02, stdev=55.58 lat (msec): min=22, max=149, avg=79.10, stdev=55.57 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 46], 50.00th=[ 46], 60.00th=[ 69], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 150], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 6144, max= 6144, per=19.67%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=33.33%, 250=25.00% cpu : usr=0.34%, sys=15.99%, ctx=59, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6815: Sat Feb 24 23:41:25 2024 read: IOPS=6, BW=6573KiB/s (6730kB/s)(5120KiB/779msec) clat (usec): min=24968, max=91432, avg=43791.22, stdev=27647.16 lat (usec): min=24970, max=91433, avg=43792.52, stdev=27647.08 clat percentiles (usec): | 1.00th=[25035], 5.00th=[25035], 10.00th=[25035], 20.00th=[25035], | 30.00th=[25560], 40.00th=[25560], 50.00th=[33817], 60.00th=[33817], | 70.00th=[43254], 80.00th=[43254], 90.00th=[91751], 95.00th=[91751], | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 8192, max= 8192, per=36.72%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=8, BW=9202KiB/s (9422kB/s)(7168KiB/779msec) clat (msec): min=22, max=161, avg=78.65, stdev=54.47 lat (msec): min=22, max=161, avg=78.73, stdev=54.46 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 34], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 64], 60.00th=[ 87], | 70.00th=[ 87], 80.00th=[ 142], 90.00th=[ 163], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min=10240, max=10240, per=32.79%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=25.00%, 250=16.67% cpu : usr=0.00%, sys=18.38%, ctx=44, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=21.8MiB/s (22.8MB/s), 2231KiB/s-8720KiB/s (2284kB/s-8929kB/s), io=20.0MiB (20.0MB), run=779-918msec WRITE: bw=30.5MiB/s (31.0MB/s), 6229KiB/s-10.9MiB/s (6378kB/s-11.4MB/s), io=28.0MiB (29.4MB), run=779-918msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6816: Sat Feb 24 23:41:25 2024 read: IOPS=9, BW=9752KiB/s (9986kB/s)(7168KiB/735msec) clat (usec): min=464, max=109805, avg=46731.53, stdev=36129.46 lat (usec): min=465, max=109807, avg=46733.10, stdev=36129.59 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 465], 10.00th=[ 465], 20.00th=[ 15008], | 30.00th=[ 39060], 40.00th=[ 39060], 50.00th=[ 43254], 60.00th=[ 47449], | 70.00th=[ 47449], 80.00th=[ 71828], 90.00th=[109577], 95.00th=[109577], | 99.00th=[109577], 99.50th=[109577], 99.90th=[109577], 99.95th=[109577], | 99.99th=[109577] bw ( KiB/s): min=10240, max=10240, per=45.70%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6966KiB/s (7133kB/s)(5120KiB/735msec) clat (msec): min=33, max=124, avg=78.56, stdev=34.94 lat (msec): min=33, max=124, avg=78.63, stdev=34.95 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34], | 30.00th=[ 58], 40.00th=[ 58], 50.00th=[ 84], 60.00th=[ 84], | 70.00th=[ 95], 80.00th=[ 95], 90.00th=[ 125], 95.00th=[ 125], | 99.00th=[ 125], 99.50th=[ 125], 99.90th=[ 125], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 4096, max= 4096, per=13.06%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 20=8.33%, 50=33.33%, 100=33.33%, 250=16.67% cpu : usr=0.00%, sys=34.47%, ctx=121, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6817: Sat Feb 24 23:41:25 2024 read: IOPS=2, BW=2241KiB/s (2294kB/s)(2048KiB/914msec) clat (msec): min=50, max=102, avg=76.30, stdev=36.48 lat (msec): min=50, max=102, avg=76.30, stdev=36.48 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 51], 10.00th=[ 51], 20.00th=[ 51], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 103], | 70.00th=[ 103], 80.00th=[ 103], 90.00th=[ 103], 95.00th=[ 103], | 99.00th=[ 103], 99.50th=[ 103], 99.90th=[ 103], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min= 4087, max= 4087, per=18.24%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.9MiB/s (11.5MB/s)(10.0MiB/914msec) clat (msec): min=25, max=165, avg=74.57, stdev=47.59 lat (msec): min=25, max=165, avg=74.65, stdev=47.58 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 30], | 30.00th=[ 43], 40.00th=[ 45], 50.00th=[ 61], 60.00th=[ 64], | 70.00th=[ 102], 80.00th=[ 102], 90.00th=[ 144], 95.00th=[ 165], | 99.00th=[ 165], 99.50th=[ 165], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min=10219, max=10219, per=32.58%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=33.33%, 250=33.33% cpu : usr=0.11%, sys=36.25%, ctx=75, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6818: Sat Feb 24 23:41:25 2024 read: IOPS=9, BW=9926KiB/s (10.2MB/s)(6144KiB/619msec) clat (usec): min=361, max=53965, avg=13350.88, stdev=22055.09 lat (usec): min=362, max=53966, avg=13352.17, stdev=22055.21 clat percentiles (usec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 412], | 30.00th=[ 412], 40.00th=[ 553], 50.00th=[ 553], 60.00th=[ 562], | 70.00th=[24249], 80.00th=[24249], 90.00th=[53740], 95.00th=[53740], | 99.00th=[53740], 99.50th=[53740], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 8159, max= 8159, per=36.41%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9926KiB/s (10.2MB/s)(6144KiB/619msec) clat (msec): min=39, max=128, avg=87.16, stdev=35.19 lat (msec): min=39, max=128, avg=87.22, stdev=35.17 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 94], 50.00th=[ 94], 60.00th=[ 100], | 70.00th=[ 112], 80.00th=[ 112], 90.00th=[ 129], 95.00th=[ 129], | 99.00th=[ 129], 99.50th=[ 129], 99.90th=[ 129], 99.95th=[ 129], | 99.99th=[ 129] bw ( KiB/s): min= 8159, max= 8159, per=26.01%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67%, 750=16.67% lat (msec) : 50=25.00%, 100=25.00%, 250=16.67% cpu : usr=0.00%, sys=27.99%, ctx=59, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6819: Sat Feb 24 23:41:25 2024 read: IOPS=9, BW=9981KiB/s (10.2MB/s)(5120KiB/513msec) clat (usec): min=405, max=46148, avg=24434.84, stdev=22165.24 lat (usec): min=406, max=46149, avg=24435.88, stdev=22165.41 clat percentiles (usec): | 1.00th=[ 404], 5.00th=[ 404], 10.00th=[ 404], 20.00th=[ 404], | 30.00th=[ 529], 40.00th=[ 529], 50.00th=[36963], 60.00th=[36963], | 70.00th=[38011], 80.00th=[38011], 90.00th=[46400], 95.00th=[46400], | 99.00th=[46400], 99.50th=[46400], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 8143, max= 8143, per=36.34%, avg=8143.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.6MiB/s (14.3MB/s)(7168KiB/513msec) clat (usec): min=27268, max=97398, avg=52753.76, stdev=22505.43 lat (usec): min=27344, max=97487, avg=52826.28, stdev=22506.38 clat percentiles (usec): | 1.00th=[27395], 5.00th=[27395], 10.00th=[27395], 20.00th=[38536], | 30.00th=[40109], 40.00th=[40109], 50.00th=[53740], 60.00th=[55837], | 70.00th=[55837], 80.00th=[56886], 90.00th=[96994], 95.00th=[96994], | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min=14250, max=14250, per=45.43%, avg=14250.00, stdev= 0.00, samples=1 iops : min= 13, max= 13, avg=13.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=50.00%, 100=33.33% cpu : usr=0.00%, sys=47.66%, ctx=75, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=21.9MiB/s (22.9MB/s), 2241KiB/s-9981KiB/s (2294kB/s-10.2MB/s), io=20.0MiB (20.0MB), run=513-914msec WRITE: bw=30.6MiB/s (32.1MB/s), 6966KiB/s-13.6MiB/s (7133kB/s-14.3MB/s), io=28.0MiB (29.4MB), run=513-914msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6825: Sat Feb 24 23:41:27 2024 read: IOPS=7, BW=28.7MiB/s (30.1MB/s)(12.0MiB/418msec) clat (msec): min=11, max=194, avg=132.32, stdev=104.64 lat (msec): min=11, max=194, avg=132.33, stdev=104.64 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 12], | 30.00th=[ 12], 40.00th=[ 192], 50.00th=[ 192], 60.00th=[ 192], | 70.00th=[ 194], 80.00th=[ 194], 90.00th=[ 194], 95.00th=[ 194], | 99.00th=[ 194], 99.50th=[ 194], 99.90th=[ 194], 99.95th=[ 194], | 99.99th=[ 194] lat (msec) : 20=33.33%, 250=66.67% cpu : usr=0.00%, sys=46.52%, ctx=131, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6826: Sat Feb 24 23:41:27 2024 read: IOPS=1, BW=5665KiB/s (5801kB/s)(4096KiB/723msec) clat (nsec): min=152544k, max=152544k, avg=152544122.00, stdev= 0.00 lat (nsec): min=152545k, max=152545k, avg=152544988.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 153], 10.00th=[ 153], 20.00th=[ 153], | 30.00th=[ 153], 40.00th=[ 153], 50.00th=[ 153], 60.00th=[ 153], | 70.00th=[ 153], 80.00th=[ 153], 90.00th=[ 153], 95.00th=[ 153], | 99.00th=[ 153], 99.50th=[ 153], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 8192, max= 8192, per=36.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=11.1MiB/s (11.6MB/s)(8192KiB/723msec) clat (msec): min=262, max=305, avg=283.88, stdev=30.39 lat (msec): min=262, max=305, avg=284.09, stdev=30.49 clat percentiles (msec): | 1.00th=[ 264], 5.00th=[ 264], 10.00th=[ 264], 20.00th=[ 264], | 30.00th=[ 264], 40.00th=[ 264], 50.00th=[ 264], 60.00th=[ 305], | 70.00th=[ 305], 80.00th=[ 305], 90.00th=[ 305], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 8192, max= 8192, per=18.08%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=27.01%, ctx=125, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6828: Sat Feb 24 23:41:27 2024 write: IOPS=5, BW=21.2MiB/s (22.2MB/s)(12.0MiB/566msec) clat (msec): min=120, max=285, avg=181.80, stdev=90.28 lat (msec): min=121, max=285, avg=182.13, stdev=90.31 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 122], 10.00th=[ 122], 20.00th=[ 122], | 30.00th=[ 122], 40.00th=[ 140], 50.00th=[ 140], 60.00th=[ 140], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min=16384, max=16384, per=36.15%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=50.27%, ctx=64, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6830: Sat Feb 24 23:41:27 2024 write: IOPS=5, BW=23.2MiB/s (24.3MB/s)(12.0MiB/518msec) clat (msec): min=119, max=252, avg=171.97, stdev=71.15 lat (msec): min=119, max=253, avg=172.27, stdev=71.15 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 144], 50.00th=[ 144], 60.00th=[ 144], | 70.00th=[ 253], 80.00th=[ 253], 90.00th=[ 253], 95.00th=[ 253], | 99.00th=[ 253], 99.50th=[ 253], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min=16384, max=16384, per=36.15%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=54.93%, ctx=32, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.1MiB/s (23.2MB/s), 5665KiB/s-28.7MiB/s (5801kB/s-30.1MB/s), io=16.0MiB (16.8MB), run=418-723msec WRITE: bw=44.3MiB/s (46.4MB/s), 11.1MiB/s-23.2MiB/s (11.6MB/s-24.3MB/s), io=32.0MiB (33.6MB), run=518-723msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6827: Sat Feb 24 23:41:27 2024 read: IOPS=3, BW=14.8MiB/s (15.5MB/s)(12.0MiB/812msec) clat (msec): min=194, max=381, avg=264.83, stdev=101.99 lat (msec): min=194, max=381, avg=264.84, stdev=101.99 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 194], 10.00th=[ 194], 20.00th=[ 194], | 30.00th=[ 194], 40.00th=[ 220], 50.00th=[ 220], 60.00th=[ 220], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 8175, max= 8175, per=44.21%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=12.45%, ctx=54, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6829: Sat Feb 24 23:41:27 2024 read: IOPS=1, BW=4623KiB/s (4734kB/s)(4096KiB/886msec) clat (nsec): min=277885k, max=277885k, avg=277885441.00, stdev= 0.00 lat (nsec): min=277887k, max=277887k, avg=277887067.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 279], 10.00th=[ 279], 20.00th=[ 279], | 30.00th=[ 279], 40.00th=[ 279], 50.00th=[ 279], 60.00th=[ 279], | 70.00th=[ 279], 80.00th=[ 279], 90.00th=[ 279], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 8192, max= 8192, per=44.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9246KiB/s (9468kB/s)(8192KiB/886msec) clat (msec): min=166, max=440, avg=303.07, stdev=193.68 lat (msec): min=166, max=440, avg=303.38, stdev=193.67 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 167], | 30.00th=[ 167], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 439], | 70.00th=[ 439], 80.00th=[ 439], 90.00th=[ 439], 95.00th=[ 439], | 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=16.14%, ctx=70, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6831: Sat Feb 24 23:41:27 2024 write: IOPS=3, BW=15.9MiB/s (16.7MB/s)(12.0MiB/755msec) clat (msec): min=192, max=335, avg=247.74, stdev=76.49 lat (msec): min=192, max=335, avg=248.03, stdev=76.50 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 192], 10.00th=[ 192], 20.00th=[ 192], | 30.00th=[ 192], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 215], | 70.00th=[ 334], 80.00th=[ 334], 90.00th=[ 334], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 8192, max= 8192, per=22.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.13%, sys=14.70%, ctx=33, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6832: Sat Feb 24 23:41:27 2024 write: IOPS=4, BW=17.6MiB/s (18.5MB/s)(12.0MiB/682msec) clat (msec): min=180, max=248, avg=221.54, stdev=36.12 lat (msec): min=180, max=249, avg=221.82, stdev=36.12 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 182], 10.00th=[ 182], 20.00th=[ 182], | 30.00th=[ 182], 40.00th=[ 236], 50.00th=[ 236], 60.00th=[ 236], | 70.00th=[ 249], 80.00th=[ 249], 90.00th=[ 249], 95.00th=[ 249], | 99.00th=[ 249], 99.50th=[ 249], 99.90th=[ 249], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min=16351, max=16351, per=44.21%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=31.86%, ctx=344, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.1MiB/s (18.9MB/s), 4623KiB/s-14.8MiB/s (4734kB/s-15.5MB/s), io=16.0MiB (16.8MB), run=812-886msec WRITE: bw=36.1MiB/s (37.9MB/s), 9246KiB/s-17.6MiB/s (9468kB/s-18.5MB/s), io=32.0MiB (33.6MB), run=682-886msec