== sanity test 398b: DIO and buffer IO race ============== 23:40:21 (1708836021) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.23055 s, 40.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5442: Sat Feb 24 23:40:39 2024 read: IOPS=101, BW=406KiB/s (416kB/s)(5992KiB/14757msec) clat (usec): min=43, max=173953, avg=5332.55, stdev=5437.53 lat (usec): min=43, max=173954, avg=5333.70, stdev=5437.49 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 104], 10.00th=[ 2278], 20.00th=[ 2868], | 30.00th=[ 3392], 40.00th=[ 3851], 50.00th=[ 4490], 60.00th=[ 5473], | 70.00th=[ 6521], 80.00th=[ 7570], 90.00th=[ 9110], 95.00th=[ 10683], | 99.00th=[ 17171], 99.50th=[ 20841], 99.90th=[ 24773], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 168, max= 560, per=24.72%, avg=402.14, stdev=75.55, samples=29 iops : min= 42, max= 140, avg=100.48, stdev=18.88, samples=29 write: IOPS=106, BW=427KiB/s (437kB/s)(6296KiB/14757msec) clat (usec): min=435, max=25717, avg=4254.34, stdev=4041.01 lat (usec): min=436, max=25719, avg=4257.98, stdev=4044.68 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 594], 10.00th=[ 676], 20.00th=[ 783], | 30.00th=[ 938], 40.00th=[ 1598], 50.00th=[ 3359], 60.00th=[ 4359], | 70.00th=[ 5473], 80.00th=[ 7242], 90.00th=[ 9503], 95.00th=[12256], | 99.00th=[17957], 99.50th=[20579], 99.90th=[25035], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 184, max= 544, per=24.78%, avg=422.00, stdev=67.50, samples=29 iops : min= 46, max= 136, avg=105.45, stdev=16.85, samples=29 lat (usec) : 50=0.03%, 100=2.18%, 250=1.30%, 500=0.65%, 750=8.27% lat (usec) : 1000=7.91% lat (msec) : 2=5.05%, 4=24.06%, 10=42.68%, 20=7.32%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.34%, sys=22.25%, ctx=5791, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5444: Sat Feb 24 23:40:39 2024 read: IOPS=100, BW=403KiB/s (412kB/s)(5940KiB/14752msec) clat (usec): min=42, max=171070, avg=5280.41, stdev=5320.91 lat (usec): min=43, max=171072, avg=5282.43, stdev=5320.94 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 102], 10.00th=[ 2212], 20.00th=[ 2900], | 30.00th=[ 3425], 40.00th=[ 3982], 50.00th=[ 4621], 60.00th=[ 5473], | 70.00th=[ 6390], 80.00th=[ 7439], 90.00th=[ 9110], 95.00th=[ 10814], | 99.00th=[ 15795], 99.50th=[ 17171], 99.90th=[ 23200], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 176, max= 560, per=24.60%, avg=400.21, stdev=69.73, samples=29 iops : min= 44, max= 140, avg=100.00, stdev=17.41, samples=29 write: IOPS=107, BW=430KiB/s (441kB/s)(6348KiB/14752msec) clat (usec): min=466, max=34131, avg=4303.25, stdev=3852.01 lat (usec): min=467, max=34132, avg=4304.40, stdev=3852.03 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 594], 10.00th=[ 676], 20.00th=[ 791], | 30.00th=[ 1020], 40.00th=[ 2245], 50.00th=[ 3556], 60.00th=[ 4424], | 70.00th=[ 5604], 80.00th=[ 7439], 90.00th=[ 9765], 95.00th=[11731], | 99.00th=[15926], 99.50th=[17695], 99.90th=[25297], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 232, max= 528, per=24.83%, avg=422.83, stdev=70.09, samples=29 iops : min= 58, max= 132, avg=105.66, stdev=17.52, samples=29 lat (usec) : 50=0.03%, 100=2.02%, 250=1.86%, 500=0.59%, 750=7.78% lat (usec) : 1000=6.97% lat (msec) : 2=5.31%, 4=23.76%, 10=44.01%, 20=7.42%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.30%, sys=22.44%, ctx=5719, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5446: Sat Feb 24 23:40:39 2024 read: IOPS=102, BW=411KiB/s (421kB/s)(6028KiB/14678msec) clat (usec): min=47, max=23998, avg=5290.58, stdev=3285.69 lat (usec): min=47, max=23998, avg=5291.77, stdev=3286.07 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 109], 10.00th=[ 2212], 20.00th=[ 2835], | 30.00th=[ 3326], 40.00th=[ 3949], 50.00th=[ 4686], 60.00th=[ 5604], | 70.00th=[ 6521], 80.00th=[ 7570], 90.00th=[ 9110], 95.00th=[10814], | 99.00th=[16319], 99.50th=[19530], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 264, max= 592, per=25.07%, avg=407.93, stdev=78.45, samples=29 iops : min= 66, max= 148, avg=101.93, stdev=19.64, samples=29 write: IOPS=106, BW=426KiB/s (437kB/s)(6260KiB/14678msec) clat (usec): min=387, max=175888, avg=4239.14, stdev=5795.63 lat (usec): min=388, max=175890, avg=4240.82, stdev=5795.72 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 594], 10.00th=[ 668], 20.00th=[ 783], | 30.00th=[ 922], 40.00th=[ 1762], 50.00th=[ 3392], 60.00th=[ 4146], | 70.00th=[ 5342], 80.00th=[ 7111], 90.00th=[ 9634], 95.00th=[ 11600], | 99.00th=[ 16188], 99.50th=[ 19268], 99.90th=[ 30016], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 320, max= 616, per=24.86%, avg=423.38, stdev=73.33, samples=29 iops : min= 80, max= 154, avg=105.79, stdev=18.34, samples=29 lat (usec) : 50=0.07%, 100=1.82%, 250=1.33%, 500=0.81%, 750=8.01% lat (usec) : 1000=7.62% lat (msec) : 2=5.01%, 4=24.90%, 10=42.64%, 20=7.36%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.33%, sys=22.55%, ctx=5758, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5447: Sat Feb 24 23:40:39 2024 read: IOPS=103, BW=414KiB/s (424kB/s)(6052KiB/14622msec) clat (usec): min=42, max=171976, avg=5272.87, stdev=5339.12 lat (usec): min=42, max=171977, avg=5274.42, stdev=5339.27 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 108], 10.00th=[ 2278], 20.00th=[ 2868], | 30.00th=[ 3294], 40.00th=[ 3884], 50.00th=[ 4555], 60.00th=[ 5407], | 70.00th=[ 6325], 80.00th=[ 7373], 90.00th=[ 8979], 95.00th=[ 10421], | 99.00th=[ 16712], 99.50th=[ 19530], 99.90th=[ 22938], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 279, max= 518, per=25.23%, avg=410.52, stdev=54.42, samples=29 iops : min= 69, max= 129, avg=102.48, stdev=13.73, samples=29 write: IOPS=106, BW=426KiB/s (437kB/s)(6236KiB/14622msec) clat (usec): min=413, max=30093, avg=4215.94, stdev=3929.85 lat (usec): min=413, max=30094, avg=4216.99, stdev=3929.98 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 578], 10.00th=[ 652], 20.00th=[ 775], | 30.00th=[ 914], 40.00th=[ 1483], 50.00th=[ 3359], 60.00th=[ 4293], | 70.00th=[ 5669], 80.00th=[ 7439], 90.00th=[ 9896], 95.00th=[12125], | 99.00th=[15926], 99.50th=[16581], 99.90th=[20579], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 312, max= 592, per=24.92%, avg=424.31, stdev=68.70, samples=29 iops : min= 78, max= 148, avg=105.93, stdev=17.22, samples=29 lat (usec) : 50=0.03%, 100=2.08%, 250=1.17%, 500=0.98%, 750=8.27% lat (usec) : 1000=7.98% lat (msec) : 2=4.65%, 4=24.41%, 10=42.35%, 20=7.78%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=22.20%, ctx=5676, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1627KiB/s (1666kB/s), 403KiB/s-414KiB/s (412kB/s-424kB/s), io=23.4MiB (24.6MB), run=14622-14757msec WRITE: bw=1704KiB/s (1744kB/s), 426KiB/s-430KiB/s (437kB/s-441kB/s), io=24.6MiB (25.7MB), run=14622-14757msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5440: Sat Feb 24 23:40:43 2024 read: IOPS=79, BW=318KiB/s (326kB/s)(5992KiB/18826msec) clat (usec): min=1671, max=174742, avg=5060.17, stdev=6001.53 lat (usec): min=1671, max=174744, avg=5061.13, stdev=6001.58 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1942], 10.00th=[ 2040], 20.00th=[ 2212], | 30.00th=[ 2409], 40.00th=[ 2704], 50.00th=[ 3228], 60.00th=[ 3949], | 70.00th=[ 4948], 80.00th=[ 6980], 90.00th=[ 11207], 95.00th=[ 13829], | 99.00th=[ 21103], 99.50th=[ 22414], 99.90th=[ 31589], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 152, max= 776, per=24.54%, avg=312.14, stdev=174.72, samples=37 iops : min= 38, max= 194, avg=77.97, stdev=43.71, samples=37 write: IOPS=83, BW=334KiB/s (342kB/s)(6296KiB/18826msec) clat (usec): min=2280, max=39167, avg=7102.61, stdev=4781.16 lat (usec): min=2280, max=39168, avg=7103.81, stdev=4781.33 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 2999], 10.00th=[ 3228], 20.00th=[ 3523], | 30.00th=[ 3785], 40.00th=[ 4228], 50.00th=[ 5014], 60.00th=[ 6521], | 70.00th=[ 8455], 80.00th=[10683], 90.00th=[13829], 95.00th=[16319], | 99.00th=[22676], 99.50th=[26870], 99.90th=[38536], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 160, max= 776, per=24.65%, avg=328.35, stdev=181.52, samples=37 iops : min= 40, max= 194, avg=82.03, stdev=45.40, samples=37 lat (msec) : 2=3.68%, 4=44.56%, 10=34.21%, 20=16.05%, 50=1.46% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=10.44%, ctx=3605, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5441: Sat Feb 24 23:40:43 2024 read: IOPS=78, BW=315KiB/s (322kB/s)(5940KiB/18869msec) clat (usec): min=1610, max=172074, avg=4996.74, stdev=6003.75 lat (usec): min=1610, max=172076, avg=4997.81, stdev=6003.79 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1975], 10.00th=[ 2057], 20.00th=[ 2212], | 30.00th=[ 2409], 40.00th=[ 2638], 50.00th=[ 3064], 60.00th=[ 3785], | 70.00th=[ 4883], 80.00th=[ 6849], 90.00th=[ 10814], 95.00th=[ 13829], | 99.00th=[ 21103], 99.50th=[ 23987], 99.90th=[ 34341], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 112, max= 744, per=24.39%, avg=310.22, stdev=184.61, samples=37 iops : min= 28, max= 186, avg=77.51, stdev=46.18, samples=37 write: IOPS=84, BW=336KiB/s (344kB/s)(6348KiB/18869msec) clat (usec): min=2308, max=38919, avg=7173.48, stdev=4756.02 lat (usec): min=2309, max=38921, avg=7174.62, stdev=4756.00 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 2999], 10.00th=[ 3195], 20.00th=[ 3490], | 30.00th=[ 3818], 40.00th=[ 4228], 50.00th=[ 5211], 60.00th=[ 6587], | 70.00th=[ 8586], 80.00th=[10814], 90.00th=[13960], 95.00th=[16450], | 99.00th=[22938], 99.50th=[26608], 99.90th=[35390], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 184, max= 776, per=24.59%, avg=327.51, stdev=174.00, samples=37 iops : min= 46, max= 194, avg=81.84, stdev=43.53, samples=37 lat (msec) : 2=3.03%, 4=45.08%, 10=33.66%, 20=16.54%, 50=1.66% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=10.44%, ctx=3659, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5443: Sat Feb 24 23:40:43 2024 read: IOPS=80, BW=320KiB/s (328kB/s)(6028KiB/18828msec) clat (usec): min=1679, max=33477, avg=5051.60, stdev=4369.49 lat (usec): min=1679, max=33478, avg=5052.71, stdev=4369.47 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 1975], 10.00th=[ 2073], 20.00th=[ 2212], | 30.00th=[ 2409], 40.00th=[ 2671], 50.00th=[ 3130], 60.00th=[ 3818], | 70.00th=[ 4883], 80.00th=[ 7373], 90.00th=[11600], 95.00th=[14746], | 99.00th=[21103], 99.50th=[22414], 99.90th=[32375], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 160, max= 720, per=24.76%, avg=314.95, stdev=171.83, samples=37 iops : min= 40, max= 180, avg=78.68, stdev=42.99, samples=37 write: IOPS=83, BW=332KiB/s (340kB/s)(6260KiB/18828msec) clat (msec): min=2, max=173, avg= 7.12, stdev= 6.25 lat (msec): min=2, max=173, avg= 7.13, stdev= 6.25 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 14], 95.00th=[ 17], | 99.00th=[ 24], 99.50th=[ 27], 99.90th=[ 39], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 168, max= 688, per=24.46%, avg=325.76, stdev=179.93, samples=37 iops : min= 42, max= 172, avg=81.38, stdev=45.01, samples=37 lat (msec) : 2=3.19%, 4=45.57%, 10=33.33%, 20=16.18%, 50=1.69% lat (msec) : 250=0.03% cpu : usr=0.31%, sys=10.31%, ctx=3582, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5445: Sat Feb 24 23:40:43 2024 read: IOPS=80, BW=322KiB/s (330kB/s)(6052KiB/18793msec) clat (usec): min=1637, max=32482, avg=4901.66, stdev=4144.57 lat (usec): min=1638, max=32483, avg=4902.71, stdev=4144.61 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1958], 10.00th=[ 2073], 20.00th=[ 2245], | 30.00th=[ 2409], 40.00th=[ 2704], 50.00th=[ 3195], 60.00th=[ 3851], | 70.00th=[ 4883], 80.00th=[ 6652], 90.00th=[10421], 95.00th=[14484], | 99.00th=[21103], 99.50th=[22938], 99.90th=[28443], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 144, max= 744, per=24.90%, avg=316.68, stdev=176.07, samples=37 iops : min= 36, max= 186, avg=79.11, stdev=44.03, samples=37 write: IOPS=82, BW=332KiB/s (340kB/s)(6236KiB/18793msec) clat (msec): min=2, max=173, avg= 7.26, stdev= 6.45 lat (msec): min=2, max=173, avg= 7.26, stdev= 6.45 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 14], 95.00th=[ 18], | 99.00th=[ 24], 99.50th=[ 28], 99.90th=[ 40], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 168, max= 736, per=24.52%, avg=326.62, stdev=180.14, samples=37 iops : min= 42, max= 184, avg=81.59, stdev=45.07, samples=37 lat (msec) : 2=3.03%, 4=45.74%, 10=34.28%, 20=15.20%, 50=1.73% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=10.40%, ctx=3643, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1273KiB/s (1303kB/s), 315KiB/s-322KiB/s (322kB/s-330kB/s), io=23.4MiB (24.6MB), run=18793-18869msec WRITE: bw=1332KiB/s (1364kB/s), 332KiB/s-336KiB/s (340kB/s-344kB/s), io=24.6MiB (25.7MB), run=18793-18869msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5477: Sat Feb 24 23:40:47 2024 read: IOPS=99, BW=1586KiB/s (1624kB/s)(5920KiB/3733msec) clat (usec): min=47, max=21445, avg=5521.48, stdev=3243.61 lat (usec): min=48, max=21446, avg=5522.49, stdev=3243.67 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 94], 10.00th=[ 128], 20.00th=[ 3425], | 30.00th=[ 4047], 40.00th=[ 4621], 50.00th=[ 5211], 60.00th=[ 5932], | 70.00th=[ 6718], 80.00th=[ 7504], 90.00th=[ 8848], 95.00th=[10814], | 99.00th=[17171], 99.50th=[19006], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 1280, max= 1824, per=24.30%, avg=1558.86, stdev=189.96, samples=7 iops : min= 80, max= 114, avg=97.43, stdev=11.87, samples=7 write: IOPS=106, BW=1706KiB/s (1747kB/s)(6368KiB/3733msec) clat (usec): min=737, max=17223, avg=4219.52, stdev=3312.60 lat (usec): min=738, max=17224, avg=4223.70, stdev=3310.08 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 906], 10.00th=[ 1029], 20.00th=[ 1172], | 30.00th=[ 1418], 40.00th=[ 2212], 50.00th=[ 3326], 60.00th=[ 4293], | 70.00th=[ 5342], 80.00th=[ 7046], 90.00th=[ 9372], 95.00th=[10814], | 99.00th=[13304], 99.50th=[17171], 99.90th=[17171], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 1344, max= 1888, per=24.99%, avg=1686.86, stdev=192.63, samples=7 iops : min= 84, max= 118, avg=105.43, stdev=12.04, samples=7 lat (usec) : 50=0.13%, 100=2.47%, 250=2.47%, 750=0.39%, 1000=4.17% lat (msec) : 2=15.23%, 4=17.32%, 10=50.52%, 20=7.16%, 50=0.13% cpu : usr=0.21%, sys=26.90%, ctx=1668, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5478: Sat Feb 24 23:40:47 2024 read: IOPS=97, BW=1555KiB/s (1593kB/s)(5744KiB/3693msec) clat (usec): min=62, max=22523, avg=5531.73, stdev=3182.08 lat (usec): min=62, max=22524, avg=5533.19, stdev=3182.26 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 100], 10.00th=[ 141], 20.00th=[ 3425], | 30.00th=[ 4080], 40.00th=[ 4686], 50.00th=[ 5342], 60.00th=[ 6063], | 70.00th=[ 6783], 80.00th=[ 7898], 90.00th=[ 9372], 95.00th=[10552], | 99.00th=[15401], 99.50th=[18220], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 1408, max= 1760, per=24.08%, avg=1545.14, stdev=134.14, samples=7 iops : min= 88, max= 110, avg=96.57, stdev= 8.38, samples=7 write: IOPS=110, BW=1772KiB/s (1815kB/s)(6544KiB/3693msec) clat (usec): min=662, max=16123, avg=4143.48, stdev=3417.08 lat (usec): min=663, max=16124, avg=4147.31, stdev=3415.30 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 840], 10.00th=[ 955], 20.00th=[ 1139], | 30.00th=[ 1418], 40.00th=[ 2212], 50.00th=[ 3359], 60.00th=[ 3949], | 70.00th=[ 4883], 80.00th=[ 6783], 90.00th=[ 9634], 95.00th=[11338], | 99.00th=[13566], 99.50th=[15008], 99.90th=[16188], 99.95th=[16188], | 99.99th=[16188] bw ( KiB/s): min= 1504, max= 2176, per=25.87%, avg=1746.29, stdev=251.19, samples=7 iops : min= 94, max= 136, avg=109.14, stdev=15.70, samples=7 lat (usec) : 100=2.47%, 250=2.34%, 500=0.26%, 750=1.56%, 1000=4.82% lat (msec) : 2=13.54%, 4=20.44%, 10=46.74%, 20=7.68%, 50=0.13% cpu : usr=0.27%, sys=26.84%, ctx=1724, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5479: Sat Feb 24 23:40:47 2024 read: IOPS=104, BW=1677KiB/s (1717kB/s)(6256KiB/3731msec) clat (usec): min=64, max=22002, avg=5439.70, stdev=3322.55 lat (usec): min=65, max=22003, avg=5441.43, stdev=3322.66 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 104], 10.00th=[ 163], 20.00th=[ 3294], | 30.00th=[ 3884], 40.00th=[ 4490], 50.00th=[ 5080], 60.00th=[ 5735], | 70.00th=[ 6652], 80.00th=[ 7635], 90.00th=[ 9110], 95.00th=[10421], | 99.00th=[17433], 99.50th=[20841], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 1245, max= 2144, per=25.92%, avg=1663.00, stdev=332.93, samples=7 iops : min= 77, max= 134, avg=103.71, stdev=20.93, samples=7 write: IOPS=101, BW=1617KiB/s (1656kB/s)(6032KiB/3731msec) clat (usec): min=658, max=20384, avg=4214.79, stdev=3588.71 lat (usec): min=659, max=20386, avg=4218.10, stdev=3586.95 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 840], 10.00th=[ 963], 20.00th=[ 1139], | 30.00th=[ 1369], 40.00th=[ 2180], 50.00th=[ 3195], 60.00th=[ 4178], | 70.00th=[ 4948], 80.00th=[ 6849], 90.00th=[ 9634], 95.00th=[11600], | 99.00th=[15664], 99.50th=[19268], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 1213, max= 2080, per=23.28%, avg=1571.71, stdev=259.16, samples=7 iops : min= 75, max= 130, avg=98.00, stdev=16.44, samples=7 lat (usec) : 100=2.08%, 250=3.39%, 500=0.26%, 750=0.78%, 1000=5.08% lat (msec) : 2=12.76%, 4=20.83%, 10=47.01%, 20=7.42%, 50=0.39% cpu : usr=0.13%, sys=26.51%, ctx=1847, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5480: Sat Feb 24 23:40:47 2024 read: IOPS=103, BW=1652KiB/s (1691kB/s)(6032KiB/3652msec) clat (usec): min=67, max=24869, avg=5484.77, stdev=3306.23 lat (usec): min=67, max=24870, avg=5485.61, stdev=3306.24 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 109], 10.00th=[ 2376], 20.00th=[ 3392], | 30.00th=[ 3818], 40.00th=[ 4424], 50.00th=[ 5145], 60.00th=[ 5735], | 70.00th=[ 6521], 80.00th=[ 7504], 90.00th=[ 8848], 95.00th=[11600], | 99.00th=[16909], 99.50th=[22414], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 1440, max= 1920, per=25.44%, avg=1632.00, stdev=171.33, samples=7 iops : min= 90, max= 120, avg=102.00, stdev=10.71, samples=7 write: IOPS=107, BW=1713KiB/s (1754kB/s)(6256KiB/3652msec) clat (usec): min=634, max=16225, avg=4021.64, stdev=3429.08 lat (usec): min=635, max=16227, avg=4023.05, stdev=3429.19 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 791], 10.00th=[ 930], 20.00th=[ 1074], | 30.00th=[ 1237], 40.00th=[ 1926], 50.00th=[ 3032], 60.00th=[ 4047], | 70.00th=[ 5014], 80.00th=[ 6521], 90.00th=[ 8979], 95.00th=[11469], | 99.00th=[15139], 99.50th=[15795], 99.90th=[16188], 99.95th=[16188], | 99.99th=[16188] bw ( KiB/s): min= 1568, max= 1984, per=25.33%, avg=1709.71, stdev=153.31, samples=7 iops : min= 98, max= 124, avg=106.86, stdev= 9.58, samples=7 lat (usec) : 100=1.82%, 250=2.47%, 750=1.56%, 1000=6.51% lat (msec) : 2=12.89%, 4=20.70%, 10=47.53%, 20=6.25%, 50=0.26% cpu : usr=0.33%, sys=27.33%, ctx=1777, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6416KiB/s (6570kB/s), 1555KiB/s-1677KiB/s (1593kB/s-1717kB/s), io=23.4MiB (24.5MB), run=3652-3733msec WRITE: bw=6751KiB/s (6913kB/s), 1617KiB/s-1772KiB/s (1656kB/s-1815kB/s), io=24.6MiB (25.8MB), run=3652-3733msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5481: Sat Feb 24 23:40:48 2024 read: IOPS=78, BW=1256KiB/s (1287kB/s)(5920KiB/4712msec) clat (usec): min=1748, max=20498, avg=4943.12, stdev=3592.33 lat (usec): min=1749, max=20498, avg=4944.03, stdev=3592.32 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2245], 10.00th=[ 2343], 20.00th=[ 2540], | 30.00th=[ 2704], 40.00th=[ 2999], 50.00th=[ 3425], 60.00th=[ 3851], | 70.00th=[ 4752], 80.00th=[ 6980], 90.00th=[10552], 95.00th=[13566], | 99.00th=[16319], 99.50th=[20317], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 896, max= 2336, per=23.63%, avg=1187.11, stdev=478.66, samples=9 iops : min= 56, max= 146, avg=74.11, stdev=29.84, samples=9 write: IOPS=84, BW=1351KiB/s (1384kB/s)(6368KiB/4712msec) clat (usec): min=2807, max=25456, avg=7222.30, stdev=4172.52 lat (usec): min=2808, max=25457, avg=7224.17, stdev=4172.91 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3195], 10.00th=[ 3458], 20.00th=[ 3785], | 30.00th=[ 4113], 40.00th=[ 4686], 50.00th=[ 5800], 60.00th=[ 7242], | 70.00th=[ 8717], 80.00th=[10290], 90.00th=[12780], 95.00th=[15926], | 99.00th=[19792], 99.50th=[23725], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 768, max= 2624, per=24.61%, avg=1300.89, stdev=567.09, samples=9 iops : min= 48, max= 164, avg=81.22, stdev=35.37, samples=9 lat (msec) : 2=0.52%, 4=43.62%, 10=39.45%, 20=15.76%, 50=0.65% cpu : usr=0.17%, sys=12.42%, ctx=946, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5482: Sat Feb 24 23:40:48 2024 read: IOPS=75, BW=1211KiB/s (1240kB/s)(5744KiB/4744msec) clat (usec): min=1962, max=20808, avg=4921.44, stdev=3658.80 lat (usec): min=1962, max=20808, avg=4922.49, stdev=3658.71 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2245], 10.00th=[ 2343], 20.00th=[ 2474], | 30.00th=[ 2638], 40.00th=[ 2933], 50.00th=[ 3359], 60.00th=[ 3916], | 70.00th=[ 5014], 80.00th=[ 6718], 90.00th=[10290], 95.00th=[13435], | 99.00th=[17695], 99.50th=[20317], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 736, max= 2176, per=23.00%, avg=1155.22, stdev=478.23, samples=9 iops : min= 46, max= 136, avg=72.11, stdev=29.88, samples=9 write: IOPS=86, BW=1379KiB/s (1413kB/s)(6544KiB/4744msec) clat (usec): min=2616, max=28583, avg=7256.17, stdev=4709.76 lat (usec): min=2617, max=28585, avg=7257.88, stdev=4709.86 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3228], 10.00th=[ 3490], 20.00th=[ 3752], | 30.00th=[ 4015], 40.00th=[ 4424], 50.00th=[ 5080], 60.00th=[ 6194], | 70.00th=[ 8455], 80.00th=[10814], 90.00th=[14746], 95.00th=[17433], | 99.00th=[22414], 99.50th=[23462], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 798, max= 2720, per=24.69%, avg=1304.67, stdev=581.26, samples=9 iops : min= 49, max= 170, avg=81.44, stdev=36.42, samples=9 lat (msec) : 2=0.39%, 4=43.62%, 10=38.80%, 20=15.89%, 50=1.30% cpu : usr=0.19%, sys=12.29%, ctx=922, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5483: Sat Feb 24 23:40:48 2024 read: IOPS=82, BW=1323KiB/s (1355kB/s)(6256KiB/4729msec) clat (usec): min=2003, max=23783, avg=5262.98, stdev=3945.78 lat (usec): min=2003, max=23784, avg=5263.82, stdev=3945.81 clat percentiles (usec): | 1.00th=[ 2057], 5.00th=[ 2212], 10.00th=[ 2311], 20.00th=[ 2474], | 30.00th=[ 2704], 40.00th=[ 2999], 50.00th=[ 3425], 60.00th=[ 4228], | 70.00th=[ 5669], 80.00th=[ 7439], 90.00th=[11863], 95.00th=[14091], | 99.00th=[18220], 99.50th=[22152], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 800, max= 2624, per=25.19%, avg=1265.44, stdev=614.02, samples=9 iops : min= 50, max= 164, avg=79.00, stdev=38.41, samples=9 write: IOPS=79, BW=1276KiB/s (1306kB/s)(6032KiB/4729msec) clat (usec): min=2738, max=27692, avg=7063.77, stdev=4302.88 lat (usec): min=2739, max=27694, avg=7065.60, stdev=4303.10 clat percentiles (usec): | 1.00th=[ 2802], 5.00th=[ 3228], 10.00th=[ 3425], 20.00th=[ 3687], | 30.00th=[ 3982], 40.00th=[ 4490], 50.00th=[ 5342], 60.00th=[ 6587], | 70.00th=[ 8291], 80.00th=[10159], 90.00th=[13304], 95.00th=[15139], | 99.00th=[20841], 99.50th=[25297], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 704, max= 2368, per=22.80%, avg=1205.00, stdev=491.61, samples=9 iops : min= 44, max= 148, avg=75.22, stdev=30.73, samples=9 lat (msec) : 4=44.40%, 10=38.93%, 20=15.49%, 50=1.17% cpu : usr=0.17%, sys=12.39%, ctx=925, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5484: Sat Feb 24 23:40:48 2024 read: IOPS=79, BW=1265KiB/s (1295kB/s)(6032KiB/4768msec) clat (usec): min=1907, max=23629, avg=5343.12, stdev=4436.31 lat (usec): min=1907, max=23630, avg=5344.28, stdev=4436.24 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2147], 10.00th=[ 2245], 20.00th=[ 2442], | 30.00th=[ 2606], 40.00th=[ 2933], 50.00th=[ 3392], 60.00th=[ 4047], | 70.00th=[ 5211], 80.00th=[ 7308], 90.00th=[12518], 95.00th=[15533], | 99.00th=[22152], 99.50th=[23462], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 864, max= 2336, per=23.43%, avg=1176.89, stdev=495.68, samples=9 iops : min= 54, max= 146, avg=73.56, stdev=30.98, samples=9 write: IOPS=82, BW=1312KiB/s (1344kB/s)(6256KiB/4768msec) clat (usec): min=2641, max=28743, avg=7021.05, stdev=4534.94 lat (usec): min=2642, max=28744, avg=7022.85, stdev=4535.02 clat percentiles (usec): | 1.00th=[ 2802], 5.00th=[ 3130], 10.00th=[ 3392], 20.00th=[ 3654], | 30.00th=[ 3982], 40.00th=[ 4359], 50.00th=[ 5276], 60.00th=[ 6325], | 70.00th=[ 7767], 80.00th=[ 9634], 90.00th=[13698], 95.00th=[16712], | 99.00th=[24773], 99.50th=[25035], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 864, max= 2592, per=23.34%, avg=1233.78, stdev=587.90, samples=9 iops : min= 54, max= 162, avg=77.11, stdev=36.74, samples=9 lat (msec) : 2=0.39%, 4=44.27%, 10=38.67%, 20=14.97%, 50=1.69% cpu : usr=0.31%, sys=11.96%, ctx=975, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5023KiB/s (5144kB/s), 1211KiB/s-1323KiB/s (1240kB/s-1355kB/s), io=23.4MiB (24.5MB), run=4712-4768msec WRITE: bw=5285KiB/s (5412kB/s), 1276KiB/s-1379KiB/s (1306kB/s-1413kB/s), io=24.6MiB (25.8MB), run=4712-4768msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5493: Sat Feb 24 23:40:50 2024 read: IOPS=10, BW=10.6MiB/s (11.1MB/s)(7168KiB/663msec) clat (usec): min=22758, max=81789, avg=43657.36, stdev=20770.00 lat (usec): min=22760, max=81790, avg=43659.06, stdev=20769.93 clat percentiles (usec): | 1.00th=[22676], 5.00th=[22676], 10.00th=[22676], 20.00th=[25822], | 30.00th=[30802], 40.00th=[30802], 50.00th=[42730], 60.00th=[43779], | 70.00th=[43779], 80.00th=[58459], 90.00th=[82314], 95.00th=[82314], | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min=12288, max=12288, per=52.98%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7722KiB/s (7908kB/s)(5120KiB/663msec) clat (msec): min=24, max=142, avg=71.20, stdev=45.27 lat (msec): min=24, max=142, avg=71.29, stdev=45.27 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 42], 40.00th=[ 42], 50.00th=[ 66], 60.00th=[ 66], | 70.00th=[ 82], 80.00th=[ 82], 90.00th=[ 142], 95.00th=[ 142], | 99.00th=[ 142], 99.50th=[ 142], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 8192, max= 8192, per=25.23%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=17.82%, ctx=44, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5494: Sat Feb 24 23:40:50 2024 read: IOPS=2, BW=2444KiB/s (2503kB/s)(2048KiB/838msec) clat (msec): min=27, max=104, avg=66.04, stdev=53.94 lat (msec): min=27, max=104, avg=66.04, stdev=53.94 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 28], 60.00th=[ 105], | 70.00th=[ 105], 80.00th=[ 105], 90.00th=[ 105], 95.00th=[ 105], | 99.00th=[ 105], 99.50th=[ 105], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 2031, max= 2031, per=8.76%, avg=2031.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.9MiB/s (12.5MB/s)(10.0MiB/838msec) clat (msec): min=28, max=132, avg=70.36, stdev=35.09 lat (msec): min=28, max=132, avg=70.46, stdev=35.09 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 35], | 30.00th=[ 48], 40.00th=[ 51], 50.00th=[ 59], 60.00th=[ 70], | 70.00th=[ 88], 80.00th=[ 88], 90.00th=[ 124], 95.00th=[ 133], | 99.00th=[ 133], 99.50th=[ 133], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min=10158, max=10158, per=31.28%, avg=10158.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=41.67%, 250=25.00% cpu : usr=0.24%, sys=25.93%, ctx=82, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5495: Sat Feb 24 23:40:50 2024 read: IOPS=6, BW=6958KiB/s (7125kB/s)(6144KiB/883msec) clat (msec): min=39, max=144, avg=87.97, stdev=34.26 lat (msec): min=39, max=144, avg=87.97, stdev=34.26 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 78], | 30.00th=[ 78], 40.00th=[ 78], 50.00th=[ 78], 60.00th=[ 91], | 70.00th=[ 100], 80.00th=[ 100], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 6144, max= 6144, per=26.49%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6958KiB/s (7125kB/s)(6144KiB/883msec) clat (msec): min=23, max=139, avg=58.89, stdev=43.98 lat (msec): min=24, max=139, avg=58.97, stdev=43.97 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 35], 50.00th=[ 35], 60.00th=[ 46], | 70.00th=[ 79], 80.00th=[ 79], 90.00th=[ 140], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 8192, max= 8192, per=25.23%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=41.67%, 250=16.67% cpu : usr=0.23%, sys=16.21%, ctx=50, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5496: Sat Feb 24 23:40:50 2024 read: IOPS=6, BW=6221KiB/s (6370kB/s)(5120KiB/823msec) clat (usec): min=29168, max=76937, avg=54109.76, stdev=17674.19 lat (usec): min=29170, max=76938, avg=54111.49, stdev=17674.41 clat percentiles (usec): | 1.00th=[29230], 5.00th=[29230], 10.00th=[29230], 20.00th=[29230], | 30.00th=[49546], 40.00th=[49546], 50.00th=[51643], 60.00th=[51643], | 70.00th=[63177], 80.00th=[63177], 90.00th=[77071], 95.00th=[77071], | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 8126, max= 8126, per=35.04%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8710KiB/s (8919kB/s)(7168KiB/823msec) clat (msec): min=41, max=116, avg=78.75, stdev=24.91 lat (msec): min=41, max=116, avg=78.83, stdev=24.92 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 57], | 30.00th=[ 74], 40.00th=[ 74], 50.00th=[ 81], 60.00th=[ 91], | 70.00th=[ 91], 80.00th=[ 92], 90.00th=[ 117], 95.00th=[ 117], | 99.00th=[ 117], 99.50th=[ 117], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 8126, max= 8126, per=25.03%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=25.00%, 100=66.67%, 250=8.33% cpu : usr=0.12%, sys=17.27%, ctx=51, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.6MiB/s (23.8MB/s), 2444KiB/s-10.6MiB/s (2503kB/s-11.1MB/s), io=20.0MiB (20.0MB), run=663-883msec WRITE: bw=31.7MiB/s (33.2MB/s), 6958KiB/s-11.9MiB/s (7125kB/s-12.5MB/s), io=28.0MiB (29.4MB), run=663-883msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5498: Sat Feb 24 23:40:50 2024 read: IOPS=11, BW=11.2MiB/s (11.7MB/s)(7168KiB/626msec) clat (usec): min=455, max=84121, avg=44595.06, stdev=26426.26 lat (usec): min=456, max=84122, avg=44596.32, stdev=26426.33 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 457], 10.00th=[ 457], 20.00th=[30016], | 30.00th=[36439], 40.00th=[36439], 50.00th=[43779], 60.00th=[55313], | 70.00th=[55313], 80.00th=[61604], 90.00th=[84411], 95.00th=[84411], | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min=10240, max=10240, per=44.95%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=7, BW=8179KiB/s (8375kB/s)(5120KiB/626msec) clat (msec): min=27, max=105, avg=62.44, stdev=30.99 lat (msec): min=28, max=105, avg=62.50, stdev=30.97 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 28], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 70], 60.00th=[ 70], | 70.00th=[ 71], 80.00th=[ 71], 90.00th=[ 106], 95.00th=[ 106], | 99.00th=[ 106], 99.50th=[ 106], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 8192, max= 8192, per=25.69%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=41.67%, 100=41.67%, 250=8.33% cpu : usr=0.00%, sys=44.96%, ctx=95, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5499: Sat Feb 24 23:40:50 2024 read: IOPS=2, BW=2278KiB/s (2333kB/s)(2048KiB/899msec) clat (usec): min=39996, max=54026, avg=47011.47, stdev=9920.04 lat (usec): min=39998, max=54026, avg=47012.89, stdev=9919.43 clat percentiles (usec): | 1.00th=[40109], 5.00th=[40109], 10.00th=[40109], 20.00th=[40109], | 30.00th=[40109], 40.00th=[40109], 50.00th=[40109], 60.00th=[54264], | 70.00th=[54264], 80.00th=[54264], 90.00th=[54264], 95.00th=[54264], | 99.00th=[54264], 99.50th=[54264], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 2048, max= 2048, per=8.99%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.1MiB/s (11.7MB/s)(10.0MiB/899msec) clat (msec): min=26, max=175, avg=80.23, stdev=49.66 lat (msec): min=26, max=175, avg=80.30, stdev=49.66 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 28], | 30.00th=[ 33], 40.00th=[ 43], 50.00th=[ 83], 60.00th=[ 86], | 70.00th=[ 97], 80.00th=[ 97], 90.00th=[ 136], 95.00th=[ 176], | 99.00th=[ 176], 99.50th=[ 176], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 8192, max= 8192, per=25.69%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=41.67%, 250=16.67% cpu : usr=0.00%, sys=34.52%, ctx=111, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5500: Sat Feb 24 23:40:50 2024 read: IOPS=9, BW=9.90MiB/s (10.4MB/s)(6144KiB/606msec) clat (usec): min=478, max=64037, avg=25388.12, stdev=25540.20 lat (usec): min=480, max=64038, avg=25389.73, stdev=25539.85 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 478], 10.00th=[ 478], 20.00th=[ 4178], | 30.00th=[ 4178], 40.00th=[ 7439], 50.00th=[ 7439], 60.00th=[32900], | 70.00th=[43254], 80.00th=[43254], 90.00th=[64226], 95.00th=[64226], | 99.00th=[64226], 99.50th=[64226], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min=10098, max=10098, per=44.33%, avg=10098.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9.90MiB/s (10.4MB/s)(6144KiB/606msec) clat (msec): min=45, max=111, avg=75.18, stdev=28.57 lat (msec): min=45, max=111, avg=75.25, stdev=28.55 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 59], 50.00th=[ 59], 60.00th=[ 91], | 70.00th=[ 100], 80.00th=[ 100], 90.00th=[ 112], 95.00th=[ 112], | 99.00th=[ 112], 99.50th=[ 112], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min=10098, max=10098, per=31.66%, avg=10098.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 10=16.67%, 50=33.33%, 100=33.33%, 250=8.33% cpu : usr=0.17%, sys=37.19%, ctx=87, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5501: Sat Feb 24 23:40:50 2024 read: IOPS=6, BW=6360KiB/s (6513kB/s)(5120KiB/805msec) clat (usec): min=448, max=55525, avg=29666.15, stdev=27524.33 lat (usec): min=449, max=55527, avg=29667.79, stdev=27524.27 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 449], 10.00th=[ 449], 20.00th=[ 449], | 30.00th=[ 529], 40.00th=[ 529], 50.00th=[38011], 60.00th=[38011], | 70.00th=[54264], 80.00th=[54264], 90.00th=[55313], 95.00th=[55313], | 99.00th=[55313], 99.50th=[55313], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 2039, max= 2039, per=8.95%, avg=2039.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8904KiB/s (9118kB/s)(7168KiB/805msec) clat (msec): min=49, max=160, avg=93.03, stdev=48.53 lat (msec): min=49, max=160, avg=93.09, stdev=48.53 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 50], | 30.00th=[ 55], 40.00th=[ 55], 50.00th=[ 69], 60.00th=[ 122], | 70.00th=[ 122], 80.00th=[ 148], 90.00th=[ 161], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 8159, max= 8159, per=25.58%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=25.00%, 100=33.33%, 250=25.00% cpu : usr=0.00%, sys=34.08%, ctx=90, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.2MiB/s (23.3MB/s), 2278KiB/s-11.2MiB/s (2333kB/s-11.7MB/s), io=20.0MiB (20.0MB), run=606-899msec WRITE: bw=31.1MiB/s (32.7MB/s), 8179KiB/s-11.1MiB/s (8375kB/s-11.7MB/s), io=28.0MiB (29.4MB), run=606-899msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5507: Sat Feb 24 23:40:52 2024 read: IOPS=6, BW=24.0MiB/s (25.2MB/s)(12.0MiB/499msec) clat (msec): min=27, max=326, avg=153.39, stdev=155.29 lat (msec): min=27, max=326, avg=153.39, stdev=155.29 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 106], 50.00th=[ 106], 60.00th=[ 106], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] lat (msec) : 50=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=45.38%, ctx=270, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5508: Sat Feb 24 23:40:52 2024 read: IOPS=1, BW=5057KiB/s (5178kB/s)(4096KiB/810msec) clat (nsec): min=115970k, max=115970k, avg=115970398.00, stdev= 0.00 lat (nsec): min=115972k, max=115972k, avg=115971948.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 116], 10.00th=[ 116], 20.00th=[ 116], | 30.00th=[ 116], 40.00th=[ 116], 50.00th=[ 116], 60.00th=[ 116], | 70.00th=[ 116], 80.00th=[ 116], 90.00th=[ 116], 95.00th=[ 116], | 99.00th=[ 116], 99.50th=[ 116], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 8192, max= 8192, per=40.50%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9.88MiB/s (10.4MB/s)(8192KiB/810msec) clat (msec): min=297, max=367, avg=332.61, stdev=49.26 lat (msec): min=298, max=367, avg=332.93, stdev=49.28 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 296], 10.00th=[ 296], 20.00th=[ 296], | 30.00th=[ 296], 40.00th=[ 296], 50.00th=[ 296], 60.00th=[ 368], | 70.00th=[ 368], 80.00th=[ 368], 90.00th=[ 368], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=32.26%, ctx=175, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5509: Sat Feb 24 23:40:52 2024 write: IOPS=5, BW=20.3MiB/s (21.3MB/s)(12.0MiB/592msec) clat (msec): min=183, max=207, avg=191.39, stdev=14.04 lat (msec): min=183, max=207, avg=191.73, stdev=13.98 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 184], | 70.00th=[ 207], 80.00th=[ 207], 90.00th=[ 207], 95.00th=[ 207], | 99.00th=[ 207], 99.50th=[ 207], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min=16384, max=16384, per=40.50%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.34%, sys=49.07%, ctx=93, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5510: Sat Feb 24 23:40:52 2024 write: IOPS=4, BW=18.6MiB/s (19.5MB/s)(12.0MiB/646msec) clat (msec): min=104, max=402, avg=209.43, stdev=167.04 lat (msec): min=105, max=402, avg=209.79, stdev=167.02 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 106], 10.00th=[ 106], 20.00th=[ 106], | 30.00th=[ 106], 40.00th=[ 122], 50.00th=[ 122], 60.00th=[ 122], | 70.00th=[ 401], 80.00th=[ 401], 90.00th=[ 401], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min=16384, max=16384, per=40.50%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.16%, sys=49.15%, ctx=50, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.8MiB/s (20.7MB/s), 5057KiB/s-24.0MiB/s (5178kB/s-25.2MB/s), io=16.0MiB (16.8MB), run=499-810msec WRITE: bw=39.5MiB/s (41.4MB/s), 9.88MiB/s-20.3MiB/s (10.4MB/s-21.3MB/s), io=32.0MiB (33.6MB), run=592-810msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5511: Sat Feb 24 23:40:52 2024 read: IOPS=3, BW=14.5MiB/s (15.2MB/s)(12.0MiB/826msec) clat (msec): min=231, max=344, avg=270.93, stdev=63.83 lat (msec): min=231, max=344, avg=270.94, stdev=63.83 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 232], 10.00th=[ 232], 20.00th=[ 232], | 30.00th=[ 232], 40.00th=[ 236], 50.00th=[ 236], 60.00th=[ 236], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 8192, max= 8192, per=51.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=14.91%, ctx=64, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5512: Sat Feb 24 23:40:52 2024 read: IOPS=0, BW=4000KiB/s (4096kB/s)(4096KiB/1024msec) clat (nsec): min=480652k, max=480652k, avg=480651958.00, stdev= 0.00 lat (nsec): min=480654k, max=480654k, avg=480653648.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 481], 10.00th=[ 481], 20.00th=[ 481], | 30.00th=[ 481], 40.00th=[ 481], 50.00th=[ 481], 60.00th=[ 481], | 70.00th=[ 481], 80.00th=[ 481], 90.00th=[ 481], 95.00th=[ 481], | 99.00th=[ 481], 99.50th=[ 481], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 8192, max= 8192, per=51.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=8000KiB/s (8192kB/s)(8192KiB/1024msec) clat (msec): min=124, max=415, avg=269.98, stdev=205.91 lat (msec): min=124, max=415, avg=270.29, stdev=205.95 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 125], 10.00th=[ 125], 20.00th=[ 125], | 30.00th=[ 125], 40.00th=[ 125], 50.00th=[ 125], 60.00th=[ 418], | 70.00th=[ 418], 80.00th=[ 418], 90.00th=[ 418], 95.00th=[ 418], | 99.00th=[ 418], 99.50th=[ 418], 99.90th=[ 418], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 8175, max= 8175, per=25.55%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=18.08%, ctx=112, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5513: Sat Feb 24 23:40:52 2024 write: IOPS=3, BW=13.5MiB/s (14.1MB/s)(12.0MiB/892msec) clat (msec): min=226, max=401, avg=289.53, stdev=97.01 lat (msec): min=227, max=401, avg=290.06, stdev=96.85 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 226], 10.00th=[ 226], 20.00th=[ 226], | 30.00th=[ 226], 40.00th=[ 243], 50.00th=[ 243], 60.00th=[ 243], | 70.00th=[ 401], 80.00th=[ 401], 90.00th=[ 401], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 8192, max= 8192, per=25.60%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=13.58%, ctx=41, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5514: Sat Feb 24 23:40:52 2024 write: IOPS=3, BW=15.7MiB/s (16.5MB/s)(12.0MiB/763msec) clat (msec): min=197, max=313, avg=253.57, stdev=58.53 lat (msec): min=197, max=314, avg=253.83, stdev=58.46 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 199], 10.00th=[ 199], 20.00th=[ 199], | 30.00th=[ 199], 40.00th=[ 249], 50.00th=[ 249], 60.00th=[ 249], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 313], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 8175, max= 8175, per=25.55%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.13%, sys=33.46%, ctx=97, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.6MiB/s (16.4MB/s), 4000KiB/s-14.5MiB/s (4096kB/s-15.2MB/s), io=16.0MiB (16.8MB), run=826-1024msec WRITE: bw=31.2MiB/s (32.8MB/s), 8000KiB/s-15.7MiB/s (8192kB/s-16.5MB/s), io=32.0MiB (33.6MB), run=763-1024msec