== sanity test 398b: DIO and buffer IO race ============== 23:37:28 (1708835848) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.1639 s, 43.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8495: Sat Feb 24 23:37:44 2024 read: IOPS=113, BW=453KiB/s (464kB/s)(5992KiB/13219msec) clat (usec): min=39, max=153511, avg=4796.76, stdev=4845.83 lat (usec): min=39, max=153512, avg=4797.64, stdev=4845.85 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 92], 10.00th=[ 2057], 20.00th=[ 2638], | 30.00th=[ 3097], 40.00th=[ 3589], 50.00th=[ 4146], 60.00th=[ 4948], | 70.00th=[ 5735], 80.00th=[ 6783], 90.00th=[ 8094], 95.00th=[ 9634], | 99.00th=[ 15139], 99.50th=[ 17957], 99.90th=[ 24511], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 248, max= 632, per=25.26%, avg=450.58, stdev=73.45, samples=26 iops : min= 62, max= 158, avg=112.50, stdev=18.43, samples=26 write: IOPS=119, BW=476KiB/s (488kB/s)(6296KiB/13219msec) clat (usec): min=378, max=26154, avg=3787.37, stdev=3541.02 lat (usec): min=379, max=26155, avg=3789.39, stdev=3543.83 clat percentiles (usec): | 1.00th=[ 424], 5.00th=[ 523], 10.00th=[ 594], 20.00th=[ 668], | 30.00th=[ 791], 40.00th=[ 1500], 50.00th=[ 2999], 60.00th=[ 3785], | 70.00th=[ 4883], 80.00th=[ 6587], 90.00th=[ 8848], 95.00th=[10814], | 99.00th=[14484], 99.50th=[16450], 99.90th=[21365], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 296, max= 712, per=25.32%, avg=472.96, stdev=90.14, samples=26 iops : min= 74, max= 178, avg=118.12, stdev=22.54, samples=26 lat (usec) : 50=0.20%, 100=3.06%, 250=0.81%, 500=2.05%, 750=12.37% lat (usec) : 1000=3.55% lat (msec) : 2=4.36%, 4=29.10%, 10=39.23%, 20=5.08%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=21.98%, ctx=5271, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8497: Sat Feb 24 23:37:44 2024 read: IOPS=110, BW=441KiB/s (452kB/s)(5940KiB/13458msec) clat (usec): min=49, max=25188, avg=4796.19, stdev=2973.57 lat (usec): min=50, max=25189, avg=4797.40, stdev=2973.29 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 96], 10.00th=[ 2114], 20.00th=[ 2671], | 30.00th=[ 3130], 40.00th=[ 3490], 50.00th=[ 4228], 60.00th=[ 5014], | 70.00th=[ 5866], 80.00th=[ 6783], 90.00th=[ 8160], 95.00th=[ 9896], | 99.00th=[15401], 99.50th=[16712], 99.90th=[22152], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 256, max= 520, per=24.28%, avg=433.19, stdev=57.16, samples=26 iops : min= 64, max= 130, avg=108.27, stdev=14.31, samples=26 write: IOPS=117, BW=472KiB/s (483kB/s)(6348KiB/13458msec) clat (usec): min=395, max=164120, avg=3946.74, stdev=5344.90 lat (usec): min=395, max=164121, avg=3947.77, stdev=5344.90 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 545], 10.00th=[ 603], 20.00th=[ 668], | 30.00th=[ 807], 40.00th=[ 1811], 50.00th=[ 3261], 60.00th=[ 4015], | 70.00th=[ 5014], 80.00th=[ 6587], 90.00th=[ 8717], 95.00th=[ 10683], | 99.00th=[ 15795], 99.50th=[ 18220], 99.90th=[ 25035], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 288, max= 640, per=24.52%, avg=458.08, stdev=78.05, samples=26 iops : min= 72, max= 160, avg=114.50, stdev=19.49, samples=26 lat (usec) : 50=0.03%, 100=2.57%, 250=0.68%, 500=1.27%, 750=12.70% lat (usec) : 1000=3.81% lat (msec) : 2=4.10%, 4=28.42%, 10=40.98%, 20=5.24%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=21.88%, ctx=5507, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8499: Sat Feb 24 23:37:44 2024 read: IOPS=114, BW=458KiB/s (469kB/s)(6028KiB/13158msec) clat (usec): min=36, max=153460, avg=4842.10, stdev=4789.31 lat (usec): min=36, max=153461, avg=4843.15, stdev=4789.39 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 96], 10.00th=[ 2147], 20.00th=[ 2671], | 30.00th=[ 3097], 40.00th=[ 3523], 50.00th=[ 4228], 60.00th=[ 5014], | 70.00th=[ 5866], 80.00th=[ 6783], 90.00th=[ 8291], 95.00th=[ 9503], | 99.00th=[ 13435], 99.50th=[ 16581], 99.90th=[ 26346], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 328, max= 656, per=25.65%, avg=457.54, stdev=76.57, samples=26 iops : min= 82, max= 164, avg=114.38, stdev=19.14, samples=26 write: IOPS=118, BW=476KiB/s (487kB/s)(6260KiB/13158msec) clat (usec): min=392, max=26614, avg=3698.59, stdev=3467.31 lat (usec): min=393, max=26614, avg=3701.88, stdev=3471.54 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 537], 10.00th=[ 603], 20.00th=[ 676], | 30.00th=[ 791], 40.00th=[ 1401], 50.00th=[ 2999], 60.00th=[ 3785], | 70.00th=[ 4883], 80.00th=[ 6325], 90.00th=[ 8586], 95.00th=[10552], | 99.00th=[14484], 99.50th=[15926], 99.90th=[24773], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 320, max= 648, per=25.37%, avg=473.85, stdev=79.34, samples=26 iops : min= 80, max= 162, avg=118.46, stdev=19.83, samples=26 lat (usec) : 50=0.16%, 100=2.51%, 250=0.81%, 500=1.43%, 750=12.83% lat (usec) : 1000=4.04% lat (msec) : 2=4.23%, 4=29.33%, 10=39.62%, 20=4.82%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=21.94%, ctx=5335, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8501: Sat Feb 24 23:37:44 2024 read: IOPS=114, BW=459KiB/s (470kB/s)(6052KiB/13196msec) clat (usec): min=48, max=162348, avg=4807.05, stdev=5012.92 lat (usec): min=49, max=162349, avg=4807.88, stdev=5012.95 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 93], 10.00th=[ 2057], 20.00th=[ 2638], | 30.00th=[ 3032], 40.00th=[ 3458], 50.00th=[ 4146], 60.00th=[ 4948], | 70.00th=[ 5866], 80.00th=[ 6849], 90.00th=[ 8160], 95.00th=[ 10159], | 99.00th=[ 14615], 99.50th=[ 16450], 99.90th=[ 27395], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 264, max= 680, per=25.61%, avg=456.88, stdev=72.87, samples=26 iops : min= 66, max= 170, avg=114.19, stdev=18.21, samples=26 write: IOPS=118, BW=473KiB/s (484kB/s)(6236KiB/13196msec) clat (usec): min=361, max=22828, avg=3752.30, stdev=3454.94 lat (usec): min=362, max=22829, avg=3753.30, stdev=3454.99 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 537], 10.00th=[ 603], 20.00th=[ 685], | 30.00th=[ 799], 40.00th=[ 1450], 50.00th=[ 3064], 60.00th=[ 3851], | 70.00th=[ 4883], 80.00th=[ 6587], 90.00th=[ 8586], 95.00th=[10814], | 99.00th=[14615], 99.50th=[15926], 99.90th=[22152], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 336, max= 816, per=25.30%, avg=472.58, stdev=90.70, samples=26 iops : min= 84, max= 204, avg=118.12, stdev=22.69, samples=26 lat (usec) : 50=0.03%, 100=3.03%, 250=0.78%, 500=1.17%, 750=12.63% lat (usec) : 1000=4.20% lat (msec) : 2=4.46%, 4=28.87%, 10=39.16%, 20=5.50%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.30%, sys=21.95%, ctx=5527, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1784KiB/s (1827kB/s), 441KiB/s-459KiB/s (452kB/s-470kB/s), io=23.4MiB (24.6MB), run=13158-13458msec WRITE: bw=1868KiB/s (1913kB/s), 472KiB/s-476KiB/s (483kB/s-488kB/s), io=24.6MiB (25.7MB), run=13158-13458msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8496: Sat Feb 24 23:37:49 2024 read: IOPS=80, BW=323KiB/s (330kB/s)(5992KiB/18573msec) clat (usec): min=1713, max=155642, avg=4870.85, stdev=5354.09 lat (usec): min=1713, max=155643, avg=4871.74, stdev=5354.07 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2008], 10.00th=[ 2147], 20.00th=[ 2376], | 30.00th=[ 2606], 40.00th=[ 2868], 50.00th=[ 3228], 60.00th=[ 3687], | 70.00th=[ 4555], 80.00th=[ 6652], 90.00th=[ 10552], 95.00th=[ 13304], | 99.00th=[ 17695], 99.50th=[ 19006], 99.90th=[ 26346], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 160, max= 624, per=24.87%, avg=321.32, stdev=138.07, samples=37 iops : min= 40, max= 156, avg=80.22, stdev=34.54, samples=37 write: IOPS=84, BW=339KiB/s (347kB/s)(6296KiB/18573msec) clat (usec): min=2277, max=36354, avg=7121.56, stdev=4127.41 lat (usec): min=2278, max=36355, avg=7122.71, stdev=4127.41 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3392], 10.00th=[ 3687], 20.00th=[ 4146], | 30.00th=[ 4555], 40.00th=[ 5014], 50.00th=[ 5473], 60.00th=[ 6325], | 70.00th=[ 7832], 80.00th=[10028], 90.00th=[12780], 95.00th=[15139], | 99.00th=[20317], 99.50th=[28443], 99.90th=[33424], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 176, max= 632, per=24.95%, avg=337.54, stdev=141.39, samples=37 iops : min= 44, max= 158, avg=84.27, stdev=35.36, samples=37 lat (msec) : 2=2.34%, 4=37.01%, 10=44.92%, 20=14.88%, 50=0.81% lat (msec) : 250=0.03% cpu : usr=0.27%, sys=10.21%, ctx=3454, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8498: Sat Feb 24 23:37:49 2024 read: IOPS=80, BW=322KiB/s (330kB/s)(5940KiB/18447msec) clat (usec): min=1622, max=167309, avg=4818.71, stdev=5610.63 lat (usec): min=1622, max=167310, avg=4819.68, stdev=5610.58 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 2024], 10.00th=[ 2180], 20.00th=[ 2409], | 30.00th=[ 2606], 40.00th=[ 2933], 50.00th=[ 3294], 60.00th=[ 3720], | 70.00th=[ 4490], 80.00th=[ 6390], 90.00th=[ 9896], 95.00th=[ 13042], | 99.00th=[ 20055], 99.50th=[ 22676], 99.90th=[ 29754], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 128, max= 624, per=24.65%, avg=318.53, stdev=143.26, samples=36 iops : min= 32, max= 156, avg=79.56, stdev=35.82, samples=36 write: IOPS=86, BW=344KiB/s (352kB/s)(6348KiB/18447msec) clat (usec): min=2500, max=31489, avg=7074.16, stdev=3997.79 lat (usec): min=2501, max=31490, avg=7075.31, stdev=3997.80 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3425], 10.00th=[ 3785], 20.00th=[ 4178], | 30.00th=[ 4555], 40.00th=[ 4948], 50.00th=[ 5473], 60.00th=[ 6390], | 70.00th=[ 7963], 80.00th=[ 9896], 90.00th=[12649], 95.00th=[14746], | 99.00th=[21103], 99.50th=[26870], 99.90th=[29754], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 200, max= 624, per=24.81%, avg=335.64, stdev=131.13, samples=36 iops : min= 50, max= 156, avg=83.83, stdev=32.77, samples=36 lat (msec) : 2=2.08%, 4=37.04%, 10=46.13%, 20=13.64%, 50=1.07% lat (msec) : 250=0.03% cpu : usr=0.27%, sys=10.19%, ctx=3420, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8500: Sat Feb 24 23:37:49 2024 read: IOPS=81, BW=326KiB/s (334kB/s)(6028KiB/18478msec) clat (usec): min=1761, max=26551, avg=4788.47, stdev=3636.28 lat (usec): min=1762, max=26552, avg=4789.33, stdev=3636.27 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2057], 10.00th=[ 2212], 20.00th=[ 2442], | 30.00th=[ 2638], 40.00th=[ 2900], 50.00th=[ 3261], 60.00th=[ 3752], | 70.00th=[ 4686], 80.00th=[ 6652], 90.00th=[10159], 95.00th=[12780], | 99.00th=[18482], 99.50th=[20317], 99.90th=[25822], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 152, max= 600, per=24.82%, avg=320.67, stdev=126.07, samples=36 iops : min= 38, max= 150, avg=80.03, stdev=31.55, samples=36 write: IOPS=84, BW=339KiB/s (347kB/s)(6260KiB/18478msec) clat (msec): min=2, max=168, avg= 7.16, stdev= 5.70 lat (msec): min=2, max=168, avg= 7.16, stdev= 5.70 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 16], | 99.00th=[ 23], 99.50th=[ 26], 99.90th=[ 31], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 176, max= 600, per=24.49%, avg=331.31, stdev=135.99, samples=36 iops : min= 44, max= 150, avg=82.69, stdev=33.99, samples=36 lat (msec) : 2=1.46%, 4=36.56%, 10=47.75%, 20=13.02%, 50=1.17% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=10.43%, ctx=3411, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8502: Sat Feb 24 23:37:49 2024 read: IOPS=81, BW=326KiB/s (334kB/s)(6052KiB/18552msec) clat (usec): min=1707, max=28840, avg=4741.88, stdev=3680.15 lat (usec): min=1708, max=28841, avg=4742.76, stdev=3680.16 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2089], 10.00th=[ 2212], 20.00th=[ 2409], | 30.00th=[ 2606], 40.00th=[ 2868], 50.00th=[ 3195], 60.00th=[ 3687], | 70.00th=[ 4555], 80.00th=[ 6521], 90.00th=[ 9896], 95.00th=[12649], | 99.00th=[18482], 99.50th=[20841], 99.90th=[28181], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 152, max= 616, per=25.22%, avg=325.89, stdev=139.73, samples=37 iops : min= 38, max= 154, avg=81.38, stdev=34.92, samples=37 write: IOPS=84, BW=336KiB/s (344kB/s)(6236KiB/18552msec) clat (msec): min=2, max=169, avg= 7.26, stdev= 5.78 lat (msec): min=2, max=169, avg= 7.26, stdev= 5.78 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 13], 95.00th=[ 16], | 99.00th=[ 22], 99.50th=[ 27], 99.90th=[ 34], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 168, max= 616, per=24.76%, avg=334.95, stdev=136.70, samples=37 iops : min= 42, max= 154, avg=83.65, stdev=34.17, samples=37 lat (msec) : 2=1.53%, 4=37.73%, 10=45.83%, 20=13.87%, 50=1.01% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=10.29%, ctx=3423, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1293KiB/s (1324kB/s), 322KiB/s-326KiB/s (330kB/s-334kB/s), io=23.4MiB (24.6MB), run=18447-18573msec WRITE: bw=1354KiB/s (1386kB/s), 336KiB/s-344KiB/s (344kB/s-352kB/s), io=24.6MiB (25.7MB), run=18447-18573msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8539: Sat Feb 24 23:37:54 2024 read: IOPS=104, BW=1678KiB/s (1718kB/s)(5920KiB/3529msec) clat (usec): min=49, max=14324, avg=4943.73, stdev=2937.57 lat (usec): min=50, max=14324, avg=4944.50, stdev=2937.59 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 87], 10.00th=[ 116], 20.00th=[ 3032], | 30.00th=[ 3392], 40.00th=[ 3818], 50.00th=[ 4490], 60.00th=[ 5276], | 70.00th=[ 5997], 80.00th=[ 6980], 90.00th=[ 8586], 95.00th=[10683], | 99.00th=[13566], 99.50th=[14222], 99.90th=[14353], 99.95th=[14353], | 99.99th=[14353] bw ( KiB/s): min= 1312, max= 2144, per=26.25%, avg=1673.14, stdev=321.97, samples=7 iops : min= 82, max= 134, avg=104.57, stdev=20.12, samples=7 write: IOPS=112, BW=1804KiB/s (1848kB/s)(6368KiB/3529msec) clat (usec): min=566, max=20515, avg=4249.33, stdev=3714.26 lat (usec): min=567, max=20516, avg=4250.58, stdev=3714.20 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 816], 10.00th=[ 881], 20.00th=[ 1004], | 30.00th=[ 1172], 40.00th=[ 2089], 50.00th=[ 3294], 60.00th=[ 4146], | 70.00th=[ 5538], 80.00th=[ 7242], 90.00th=[ 9241], 95.00th=[11469], | 99.00th=[17957], 99.50th=[19268], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 1120, max= 2176, per=26.93%, avg=1805.71, stdev=428.47, samples=7 iops : min= 70, max= 136, avg=112.86, stdev=26.78, samples=7 lat (usec) : 50=0.13%, 100=4.17%, 250=0.91%, 750=0.91%, 1000=9.37% lat (msec) : 2=10.29%, 4=24.74%, 10=41.80%, 20=7.55%, 50=0.13% cpu : usr=0.17%, sys=25.99%, ctx=1632, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8541: Sat Feb 24 23:37:54 2024 read: IOPS=96, BW=1542KiB/s (1579kB/s)(5744KiB/3724msec) clat (usec): min=49, max=24755, avg=5348.97, stdev=3322.63 lat (usec): min=49, max=24755, avg=5349.78, stdev=3322.63 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 79], 10.00th=[ 102], 20.00th=[ 3294], | 30.00th=[ 3851], 40.00th=[ 4359], 50.00th=[ 5014], 60.00th=[ 5866], | 70.00th=[ 6587], 80.00th=[ 7439], 90.00th=[ 8717], 95.00th=[10814], | 99.00th=[18220], 99.50th=[19268], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 1152, max= 1920, per=23.81%, avg=1517.71, stdev=280.10, samples=7 iops : min= 72, max= 120, avg=94.86, stdev=17.51, samples=7 write: IOPS=109, BW=1757KiB/s (1799kB/s)(6544KiB/3724msec) clat (usec): min=625, max=21413, avg=4377.67, stdev=3586.76 lat (usec): min=625, max=21415, avg=4378.94, stdev=3586.84 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 775], 10.00th=[ 881], 20.00th=[ 1012], | 30.00th=[ 1450], 40.00th=[ 2868], 50.00th=[ 3589], 60.00th=[ 4490], | 70.00th=[ 5669], 80.00th=[ 6915], 90.00th=[ 9372], 95.00th=[11600], | 99.00th=[14877], 99.50th=[18220], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 1312, max= 2144, per=25.36%, avg=1700.57, stdev=273.68, samples=7 iops : min= 82, max= 134, avg=106.29, stdev=17.10, samples=7 lat (usec) : 50=0.13%, 100=4.43%, 250=1.04%, 750=2.08%, 1000=7.94% lat (msec) : 2=7.68%, 4=22.27%, 10=47.01%, 20=7.16%, 50=0.26% cpu : usr=0.30%, sys=24.44%, ctx=1825, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8543: Sat Feb 24 23:37:54 2024 read: IOPS=106, BW=1697KiB/s (1737kB/s)(6256KiB/3687msec) clat (usec): min=43, max=23924, avg=5326.14, stdev=3304.06 lat (usec): min=43, max=23925, avg=5327.31, stdev=3304.20 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 85], 10.00th=[ 108], 20.00th=[ 3163], | 30.00th=[ 3720], 40.00th=[ 4228], 50.00th=[ 4752], 60.00th=[ 5932], | 70.00th=[ 6587], 80.00th=[ 7504], 90.00th=[ 8848], 95.00th=[10683], | 99.00th=[16712], 99.50th=[17695], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 1280, max= 2528, per=26.54%, avg=1691.43, stdev=438.93, samples=7 iops : min= 80, max= 158, avg=105.71, stdev=27.43, samples=7 write: IOPS=102, BW=1636KiB/s (1675kB/s)(6032KiB/3687msec) clat (usec): min=630, max=23229, avg=4230.56, stdev=3556.83 lat (usec): min=631, max=23230, avg=4231.73, stdev=3556.78 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 766], 10.00th=[ 865], 20.00th=[ 988], | 30.00th=[ 1385], 40.00th=[ 2409], 50.00th=[ 3359], 60.00th=[ 4359], | 70.00th=[ 5473], 80.00th=[ 6849], 90.00th=[ 8979], 95.00th=[11076], | 99.00th=[15533], 99.50th=[16712], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 1120, max= 2112, per=23.66%, avg=1586.29, stdev=307.41, samples=7 iops : min= 70, max= 132, avg=99.14, stdev=19.21, samples=7 lat (usec) : 50=0.13%, 100=4.69%, 250=1.04%, 750=1.95%, 1000=8.59% lat (msec) : 2=7.81%, 4=21.35%, 10=46.74%, 20=7.42%, 50=0.26% cpu : usr=0.22%, sys=24.53%, ctx=1626, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8544: Sat Feb 24 23:37:54 2024 read: IOPS=100, BW=1605KiB/s (1644kB/s)(6032KiB/3758msec) clat (usec): min=45, max=23695, avg=5371.13, stdev=3482.26 lat (usec): min=45, max=23696, avg=5371.95, stdev=3482.38 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 84], 10.00th=[ 143], 20.00th=[ 3032], | 30.00th=[ 3687], 40.00th=[ 4359], 50.00th=[ 4883], 60.00th=[ 5669], | 70.00th=[ 6390], 80.00th=[ 7308], 90.00th=[ 8848], 95.00th=[11863], | 99.00th=[17695], 99.50th=[20841], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 894, max= 1920, per=24.22%, avg=1543.71, stdev=361.18, samples=7 iops : min= 55, max= 120, avg=96.14, stdev=22.68, samples=7 write: IOPS=104, BW=1665KiB/s (1705kB/s)(6256KiB/3758msec) clat (usec): min=519, max=20488, avg=4410.81, stdev=3833.01 lat (usec): min=520, max=20489, avg=4411.91, stdev=3833.06 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 783], 10.00th=[ 914], 20.00th=[ 1074], | 30.00th=[ 1369], 40.00th=[ 2376], 50.00th=[ 3425], 60.00th=[ 4359], | 70.00th=[ 5407], 80.00th=[ 7046], 90.00th=[10028], 95.00th=[12256], | 99.00th=[17695], 99.50th=[19268], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 1053, max= 1984, per=24.18%, avg=1621.29, stdev=345.55, samples=7 iops : min= 65, max= 124, avg=101.00, stdev=21.68, samples=7 lat (usec) : 50=0.13%, 100=4.43%, 250=0.39%, 500=0.13%, 750=2.08% lat (usec) : 1000=6.25% lat (msec) : 2=10.68%, 4=22.01%, 10=44.79%, 20=8.72%, 50=0.39% cpu : usr=0.11%, sys=24.59%, ctx=1736, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6374KiB/s (6527kB/s), 1542KiB/s-1697KiB/s (1579kB/s-1737kB/s), io=23.4MiB (24.5MB), run=3529-3758msec WRITE: bw=6706KiB/s (6867kB/s), 1636KiB/s-1804KiB/s (1675kB/s-1848kB/s), io=24.6MiB (25.8MB), run=3529-3758msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8537: Sat Feb 24 23:37:55 2024 read: IOPS=80, BW=1291KiB/s (1322kB/s)(5920KiB/4584msec) clat (usec): min=1432, max=21353, avg=4858.81, stdev=3778.48 lat (usec): min=1432, max=21354, avg=4859.64, stdev=3778.52 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1958], 10.00th=[ 2089], 20.00th=[ 2245], | 30.00th=[ 2474], 40.00th=[ 2900], 50.00th=[ 3359], 60.00th=[ 4047], | 70.00th=[ 4883], 80.00th=[ 6915], 90.00th=[10159], 95.00th=[13042], | 99.00th=[19530], 99.50th=[20841], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 800, max= 2528, per=23.95%, avg=1251.33, stdev=587.23, samples=9 iops : min= 50, max= 158, avg=78.11, stdev=36.76, samples=9 write: IOPS=86, BW=1389KiB/s (1423kB/s)(6368KiB/4584msec) clat (usec): min=2389, max=27216, avg=6978.99, stdev=4467.33 lat (usec): min=2390, max=27218, avg=6980.67, stdev=4467.35 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2769], 10.00th=[ 2999], 20.00th=[ 3359], | 30.00th=[ 3884], 40.00th=[ 4555], 50.00th=[ 5211], 60.00th=[ 6718], | 70.00th=[ 8356], 80.00th=[10290], 90.00th=[13042], 95.00th=[15533], | 99.00th=[24249], 99.50th=[26608], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 896, max= 3040, per=24.96%, avg=1372.11, stdev=695.59, samples=9 iops : min= 56, max= 190, avg=85.67, stdev=43.50, samples=9 lat (msec) : 2=2.99%, 4=41.80%, 10=39.06%, 20=14.84%, 50=1.30% cpu : usr=0.17%, sys=11.35%, ctx=883, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8538: Sat Feb 24 23:37:55 2024 read: IOPS=78, BW=1254KiB/s (1285kB/s)(5744KiB/4579msec) clat (usec): min=1523, max=25011, avg=4546.67, stdev=3610.19 lat (usec): min=1523, max=25012, avg=4548.23, stdev=3609.97 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1942], 10.00th=[ 2057], 20.00th=[ 2245], | 30.00th=[ 2474], 40.00th=[ 2737], 50.00th=[ 3130], 60.00th=[ 3621], | 70.00th=[ 4424], 80.00th=[ 5932], 90.00th=[ 9110], 95.00th=[13042], | 99.00th=[18744], 99.50th=[19792], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 832, max= 2592, per=23.81%, avg=1244.22, stdev=631.76, samples=9 iops : min= 52, max= 162, avg=77.67, stdev=39.54, samples=9 write: IOPS=89, BW=1429KiB/s (1463kB/s)(6544KiB/4579msec) clat (usec): min=2130, max=26837, avg=7184.12, stdev=4501.62 lat (usec): min=2130, max=26838, avg=7185.76, stdev=4501.71 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2933], 10.00th=[ 3032], 20.00th=[ 3425], | 30.00th=[ 3785], 40.00th=[ 4424], 50.00th=[ 5538], 60.00th=[ 6980], | 70.00th=[ 9110], 80.00th=[10814], 90.00th=[13304], 95.00th=[16581], | 99.00th=[21627], 99.50th=[24249], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 896, max= 3008, per=25.35%, avg=1393.44, stdev=684.95, samples=9 iops : min= 56, max= 188, avg=87.00, stdev=42.84, samples=9 lat (msec) : 2=3.39%, 4=44.79%, 10=35.03%, 20=15.89%, 50=0.91% cpu : usr=0.20%, sys=11.34%, ctx=905, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8540: Sat Feb 24 23:37:55 2024 read: IOPS=85, BW=1373KiB/s (1406kB/s)(6256KiB/4557msec) clat (usec): min=1657, max=22279, avg=4794.15, stdev=3811.68 lat (usec): min=1657, max=22280, avg=4795.09, stdev=3811.65 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1926], 10.00th=[ 2073], 20.00th=[ 2278], | 30.00th=[ 2540], 40.00th=[ 2868], 50.00th=[ 3326], 60.00th=[ 3884], | 70.00th=[ 4752], 80.00th=[ 6456], 90.00th=[10028], 95.00th=[12911], | 99.00th=[20579], 99.50th=[22152], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 768, max= 3104, per=26.19%, avg=1368.56, stdev=770.65, samples=9 iops : min= 48, max= 194, avg=85.44, stdev=48.17, samples=9 write: IOPS=82, BW=1324KiB/s (1355kB/s)(6032KiB/4557msec) clat (usec): min=2357, max=25508, avg=7095.43, stdev=4371.05 lat (usec): min=2358, max=25510, avg=7096.93, stdev=4371.20 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 2999], 10.00th=[ 3097], 20.00th=[ 3458], | 30.00th=[ 3785], 40.00th=[ 4359], 50.00th=[ 5538], 60.00th=[ 6980], | 70.00th=[ 7963], 80.00th=[10683], 90.00th=[13566], 95.00th=[15926], | 99.00th=[21103], 99.50th=[21365], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 798, max= 2624, per=23.60%, avg=1297.56, stdev=610.75, samples=9 iops : min= 49, max= 164, avg=81.00, stdev=38.26, samples=9 lat (msec) : 2=3.91%, 4=45.31%, 10=33.59%, 20=15.89%, 50=1.30% cpu : usr=0.18%, sys=11.50%, ctx=887, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8542: Sat Feb 24 23:37:55 2024 read: IOPS=83, BW=1330KiB/s (1362kB/s)(6032KiB/4534msec) clat (usec): min=1693, max=25045, avg=4733.28, stdev=3921.82 lat (usec): min=1693, max=25046, avg=4734.45, stdev=3922.35 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1958], 10.00th=[ 2057], 20.00th=[ 2212], | 30.00th=[ 2474], 40.00th=[ 2835], 50.00th=[ 3294], 60.00th=[ 3752], | 70.00th=[ 4293], 80.00th=[ 6587], 90.00th=[ 9634], 95.00th=[13435], | 99.00th=[20055], 99.50th=[22414], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 832, max= 2944, per=25.17%, avg=1315.22, stdev=682.84, samples=9 iops : min= 52, max= 184, avg=82.11, stdev=42.67, samples=9 write: IOPS=86, BW=1380KiB/s (1413kB/s)(6256KiB/4534msec) clat (usec): min=2454, max=32516, avg=7009.12, stdev=4545.38 lat (usec): min=2456, max=32517, avg=7010.48, stdev=4545.49 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2802], 10.00th=[ 3130], 20.00th=[ 3490], | 30.00th=[ 3818], 40.00th=[ 4555], 50.00th=[ 5407], 60.00th=[ 6456], | 70.00th=[ 8160], 80.00th=[10421], 90.00th=[13566], 95.00th=[15533], | 99.00th=[22414], 99.50th=[28705], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 832, max= 2752, per=24.96%, avg=1372.11, stdev=678.98, samples=9 iops : min= 52, max= 172, avg=85.67, stdev=42.46, samples=9 lat (msec) : 2=3.52%, 4=46.09%, 10=35.16%, 20=13.93%, 50=1.30% cpu : usr=0.13%, sys=11.47%, ctx=918, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5225KiB/s (5351kB/s), 1254KiB/s-1373KiB/s (1285kB/s-1406kB/s), io=23.4MiB (24.5MB), run=4534-4584msec WRITE: bw=5497KiB/s (5629kB/s), 1324KiB/s-1429KiB/s (1355kB/s-1463kB/s), io=24.6MiB (25.8MB), run=4534-4584msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8554: Sat Feb 24 23:37:56 2024 read: IOPS=19, BW=19.9MiB/s (20.9MB/s)(7168KiB/352msec) clat (usec): min=510, max=61572, avg=26043.83, stdev=24788.11 lat (usec): min=511, max=61573, avg=26045.48, stdev=24788.45 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 510], 10.00th=[ 510], 20.00th=[ 586], | 30.00th=[ 2212], 40.00th=[ 2212], 50.00th=[33817], 60.00th=[41157], | 70.00th=[41157], 80.00th=[42730], 90.00th=[61604], 95.00th=[61604], | 99.00th=[61604], 99.50th=[61604], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] write: IOPS=14, BW=14.2MiB/s (14.9MB/s)(5120KiB/352msec) clat (usec): min=18856, max=54823, avg=32366.28, stdev=14570.08 lat (usec): min=18873, max=54887, avg=32411.22, stdev=14589.66 clat percentiles (usec): | 1.00th=[18744], 5.00th=[18744], 10.00th=[18744], 20.00th=[18744], | 30.00th=[21365], 40.00th=[21365], 50.00th=[29230], 60.00th=[29230], | 70.00th=[37487], 80.00th=[37487], 90.00th=[54789], 95.00th=[54789], | 99.00th=[54789], 99.50th=[54789], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] lat (usec) : 750=16.67% lat (msec) : 4=8.33%, 20=8.33%, 50=50.00%, 100=16.67% cpu : usr=0.00%, sys=59.54%, ctx=88, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8555: Sat Feb 24 23:37:56 2024 read: IOPS=2, BW=2368KiB/s (2424kB/s)(2048KiB/865msec) clat (usec): min=37852, max=55252, avg=46552.38, stdev=12303.19 lat (usec): min=37854, max=55253, avg=46554.01, stdev=12303.29 clat percentiles (usec): | 1.00th=[38011], 5.00th=[38011], 10.00th=[38011], 20.00th=[38011], | 30.00th=[38011], 40.00th=[38011], 50.00th=[38011], 60.00th=[55313], | 70.00th=[55313], 80.00th=[55313], 90.00th=[55313], 95.00th=[55313], | 99.00th=[55313], 99.50th=[55313], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 4096, max= 4096, per=17.30%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.6MiB/s (12.1MB/s)(10.0MiB/865msec) clat (msec): min=23, max=156, avg=75.11, stdev=49.16 lat (msec): min=23, max=156, avg=75.18, stdev=49.16 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 28], | 30.00th=[ 31], 40.00th=[ 43], 50.00th=[ 62], 60.00th=[ 66], | 70.00th=[ 116], 80.00th=[ 116], 90.00th=[ 148], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min=10240, max=10240, per=30.89%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=33.33%, 250=25.00% cpu : usr=0.00%, sys=35.14%, ctx=81, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8556: Sat Feb 24 23:37:56 2024 read: IOPS=7, BW=7728KiB/s (7914kB/s)(6144KiB/795msec) clat (usec): min=27466, max=59875, avg=39577.04, stdev=11610.44 lat (usec): min=27467, max=59877, avg=39578.26, stdev=11610.65 clat percentiles (usec): | 1.00th=[27395], 5.00th=[27395], 10.00th=[27395], 20.00th=[31065], | 30.00th=[31065], 40.00th=[36963], 50.00th=[36963], 60.00th=[36963], | 70.00th=[44827], 80.00th=[44827], 90.00th=[60031], 95.00th=[60031], | 99.00th=[60031], 99.50th=[60031], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 4096, max= 4096, per=17.30%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7728KiB/s (7914kB/s)(6144KiB/795msec) clat (msec): min=46, max=143, avg=89.47, stdev=43.66 lat (msec): min=46, max=143, avg=89.53, stdev=43.67 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 54], 50.00th=[ 54], 60.00th=[ 112], | 70.00th=[ 130], 80.00th=[ 130], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 6144, max= 6144, per=18.54%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=25.00%, 250=25.00% cpu : usr=0.00%, sys=33.00%, ctx=87, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8557: Sat Feb 24 23:37:56 2024 read: IOPS=8, BW=9078KiB/s (9296kB/s)(5120KiB/564msec) clat (usec): min=369, max=58502, avg=18460.00, stdev=26332.05 lat (usec): min=370, max=58503, avg=18460.99, stdev=26331.93 clat percentiles (usec): | 1.00th=[ 371], 5.00th=[ 371], 10.00th=[ 371], 20.00th=[ 371], | 30.00th=[ 412], 40.00th=[ 412], 50.00th=[ 545], 60.00th=[ 545], | 70.00th=[32375], 80.00th=[32375], 90.00th=[58459], 95.00th=[58459], | 99.00th=[58459], 99.50th=[58459], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 8192, max= 8192, per=34.60%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.4MiB/s (13.0MB/s)(7168KiB/564msec) clat (msec): min=22, max=126, avg=64.44, stdev=40.30 lat (msec): min=22, max=126, avg=64.49, stdev=40.31 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 26], | 30.00th=[ 30], 40.00th=[ 30], 50.00th=[ 68], 60.00th=[ 81], | 70.00th=[ 81], 80.00th=[ 100], 90.00th=[ 127], 95.00th=[ 127], | 99.00th=[ 127], 99.50th=[ 127], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min=10240, max=10240, per=30.89%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 50=33.33%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=37.48%, ctx=70, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=23.1MiB/s (24.2MB/s), 2368KiB/s-19.9MiB/s (2424kB/s-20.9MB/s), io=20.0MiB (20.0MB), run=352-865msec WRITE: bw=32.4MiB/s (33.9MB/s), 7728KiB/s-14.2MiB/s (7914kB/s-14.9MB/s), io=28.0MiB (29.4MB), run=352-865msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8558: Sat Feb 24 23:37:57 2024 read: IOPS=13, BW=13.1MiB/s (13.8MB/s)(7168KiB/533msec) clat (usec): min=17965, max=58074, avg=34691.42, stdev=14576.99 lat (usec): min=17966, max=58075, avg=34692.66, stdev=14576.83 clat percentiles (usec): | 1.00th=[17957], 5.00th=[17957], 10.00th=[17957], 20.00th=[25035], | 30.00th=[27395], 40.00th=[27395], 50.00th=[29492], 60.00th=[33817], | 70.00th=[33817], 80.00th=[51119], 90.00th=[57934], 95.00th=[57934], | 99.00th=[57934], 99.50th=[57934], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min=12166, max=12166, per=55.72%, avg=12166.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9606KiB/s (9837kB/s)(5120KiB/533msec) clat (usec): min=23096, max=81115, avg=57165.29, stdev=24293.09 lat (usec): min=23170, max=81202, avg=57236.01, stdev=24294.62 clat percentiles (usec): | 1.00th=[23200], 5.00th=[23200], 10.00th=[23200], 20.00th=[23200], | 30.00th=[47973], 40.00th=[47973], 50.00th=[53740], 60.00th=[53740], | 70.00th=[80217], 80.00th=[80217], 90.00th=[81265], 95.00th=[81265], | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 8110, max= 8110, per=26.53%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=41.67% cpu : usr=0.19%, sys=24.06%, ctx=75, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8559: Sat Feb 24 23:37:57 2024 read: IOPS=2, BW=2183KiB/s (2236kB/s)(2048KiB/938msec) clat (usec): min=25405, max=38002, avg=31704.39, stdev=8907.32 lat (usec): min=25407, max=38004, avg=31706.14, stdev=8907.47 clat percentiles (usec): | 1.00th=[25297], 5.00th=[25297], 10.00th=[25297], 20.00th=[25297], | 30.00th=[25297], 40.00th=[25297], 50.00th=[25297], 60.00th=[38011], | 70.00th=[38011], 80.00th=[38011], 90.00th=[38011], 95.00th=[38011], | 99.00th=[38011], 99.50th=[38011], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 4096, max= 4096, per=18.76%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.7MiB/s (11.2MB/s)(10.0MiB/938msec) clat (msec): min=22, max=234, avg=87.20, stdev=68.80 lat (msec): min=22, max=234, avg=87.27, stdev=68.80 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 26], | 30.00th=[ 41], 40.00th=[ 45], 50.00th=[ 56], 60.00th=[ 61], | 70.00th=[ 142], 80.00th=[ 142], 90.00th=[ 153], 95.00th=[ 234], | 99.00th=[ 234], 99.50th=[ 234], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min=10240, max=10240, per=33.50%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=25.00%, 250=25.00% cpu : usr=0.00%, sys=16.86%, ctx=64, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8560: Sat Feb 24 23:37:57 2024 read: IOPS=7, BW=7651KiB/s (7835kB/s)(6144KiB/803msec) clat (usec): min=29103, max=76077, avg=53653.76, stdev=20540.47 lat (usec): min=29104, max=76078, avg=53654.80, stdev=20540.65 clat percentiles (usec): | 1.00th=[29230], 5.00th=[29230], 10.00th=[29230], 20.00th=[33424], | 30.00th=[33424], 40.00th=[46400], 50.00th=[46400], 60.00th=[61080], | 70.00th=[76022], 80.00th=[76022], 90.00th=[76022], 95.00th=[76022], | 99.00th=[76022], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 6144, max= 6144, per=28.14%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7651KiB/s (7835kB/s)(6144KiB/803msec) clat (msec): min=44, max=105, avg=79.24, stdev=25.84 lat (msec): min=44, max=105, avg=79.31, stdev=25.84 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 89], 50.00th=[ 89], 60.00th=[ 90], | 70.00th=[ 100], 80.00th=[ 100], 90.00th=[ 106], 95.00th=[ 106], | 99.00th=[ 106], 99.50th=[ 106], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 8192, max= 8192, per=26.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=50.00%, 250=8.33% cpu : usr=0.00%, sys=15.21%, ctx=49, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8561: Sat Feb 24 23:37:57 2024 read: IOPS=5, BW=5926KiB/s (6068kB/s)(5120KiB/864msec) clat (usec): min=20167, max=42576, avg=34100.37, stdev=9875.15 lat (usec): min=20169, max=42577, avg=34101.63, stdev=9874.78 clat percentiles (usec): | 1.00th=[20055], 5.00th=[20055], 10.00th=[20055], 20.00th=[20055], | 30.00th=[27132], 40.00th=[27132], 50.00th=[40109], 60.00th=[40109], | 70.00th=[40633], 80.00th=[40633], 90.00th=[42730], 95.00th=[42730], | 99.00th=[42730], 99.50th=[42730], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] write: IOPS=8, BW=8296KiB/s (8495kB/s)(7168KiB/864msec) clat (msec): min=22, max=178, avg=95.40, stdev=54.09 lat (msec): min=22, max=178, avg=95.49, stdev=54.11 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 53], | 30.00th=[ 56], 40.00th=[ 56], 50.00th=[ 112], 60.00th=[ 124], | 70.00th=[ 124], 80.00th=[ 124], 90.00th=[ 180], 95.00th=[ 180], | 99.00th=[ 180], 99.50th=[ 180], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 8094, max= 8094, per=26.48%, avg=8094.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=16.67%, 250=33.33% cpu : usr=0.35%, sys=17.96%, ctx=67, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=21.3MiB/s (22.4MB/s), 2183KiB/s-13.1MiB/s (2236kB/s-13.8MB/s), io=20.0MiB (20.0MB), run=533-938msec WRITE: bw=29.9MiB/s (31.3MB/s), 7651KiB/s-10.7MiB/s (7835kB/s-11.2MB/s), io=28.0MiB (29.4MB), run=533-938msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8571: Sat Feb 24 23:37:58 2024 read: IOPS=6, BW=26.2MiB/s (27.5MB/s)(12.0MiB/458msec) clat (msec): min=62, max=204, avg=152.29, stdev=78.16 lat (msec): min=62, max=204, avg=152.30, stdev=78.16 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 63], | 30.00th=[ 63], 40.00th=[ 190], 50.00th=[ 190], 60.00th=[ 190], | 70.00th=[ 205], 80.00th=[ 205], 90.00th=[ 205], 95.00th=[ 205], | 99.00th=[ 205], 99.50th=[ 205], 99.90th=[ 205], 99.95th=[ 205], | 99.99th=[ 205] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=64.33%, ctx=326, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8572: Sat Feb 24 23:37:58 2024 read: IOPS=1, BW=5285KiB/s (5412kB/s)(4096KiB/775msec) clat (nsec): min=170097k, max=170097k, avg=170096750.00, stdev= 0.00 lat (nsec): min=170099k, max=170099k, avg=170098845.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 171], 50.00th=[ 171], 60.00th=[ 171], | 70.00th=[ 171], 80.00th=[ 171], 90.00th=[ 171], 95.00th=[ 171], | 99.00th=[ 171], 99.50th=[ 171], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 8192, max= 8192, per=38.75%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.3MiB/s (10.8MB/s)(8192KiB/775msec) clat (msec): min=285, max=316, avg=300.91, stdev=22.00 lat (msec): min=285, max=316, avg=301.17, stdev=22.00 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 288], 10.00th=[ 288], 20.00th=[ 288], | 30.00th=[ 288], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 317], | 70.00th=[ 317], 80.00th=[ 317], 90.00th=[ 317], 95.00th=[ 317], | 99.00th=[ 317], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 8192, max= 8192, per=23.83%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=30.62%, ctx=133, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8573: Sat Feb 24 23:37:58 2024 write: IOPS=3, BW=12.6MiB/s (13.2MB/s)(12.0MiB/953msec) clat (msec): min=94, max=508, avg=313.74, stdev=208.03 lat (msec): min=95, max=509, avg=314.01, stdev=208.07 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 95], 10.00th=[ 95], 20.00th=[ 95], | 30.00th=[ 95], 40.00th=[ 338], 50.00th=[ 338], 60.00th=[ 338], | 70.00th=[ 510], 80.00th=[ 510], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] lat (msec) : 100=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.11%, sys=26.47%, ctx=15, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8574: Sat Feb 24 23:37:58 2024 write: IOPS=4, BW=16.3MiB/s (17.1MB/s)(12.0MiB/738msec) clat (msec): min=190, max=319, avg=245.16, stdev=66.50 lat (msec): min=190, max=319, avg=245.50, stdev=66.47 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 190], 10.00th=[ 190], 20.00th=[ 190], | 30.00th=[ 190], 40.00th=[ 226], 50.00th=[ 226], 60.00th=[ 226], | 70.00th=[ 321], 80.00th=[ 321], 90.00th=[ 321], 95.00th=[ 321], | 99.00th=[ 321], 99.50th=[ 321], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 8192, max= 8192, per=23.83%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=34.87%, ctx=53, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=20.6MiB/s (21.6MB/s), 5285KiB/s-26.2MiB/s (5412kB/s-27.5MB/s), io=16.0MiB (16.8MB), run=458-775msec WRITE: bw=33.6MiB/s (35.2MB/s), 10.3MiB/s-16.3MiB/s (10.8MB/s-17.1MB/s), io=32.0MiB (33.6MB), run=738-953msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8567: Sat Feb 24 23:37:58 2024 read: IOPS=3, BW=14.9MiB/s (15.7MB/s)(12.0MiB/804msec) clat (msec): min=114, max=387, avg=267.60, stdev=139.60 lat (msec): min=114, max=387, avg=267.60, stdev=139.60 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 114], 10.00th=[ 114], 20.00th=[ 114], | 30.00th=[ 114], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 8192, max= 8192, per=49.45%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=12.58%, ctx=59, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8568: Sat Feb 24 23:37:58 2024 read: IOPS=1, BW=4142KiB/s (4241kB/s)(4096KiB/989msec) clat (nsec): min=444348k, max=444348k, avg=444347889.00, stdev= 0.00 lat (nsec): min=444349k, max=444349k, avg=444348705.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 443], 10.00th=[ 443], 20.00th=[ 443], | 30.00th=[ 443], 40.00th=[ 443], 50.00th=[ 443], 60.00th=[ 443], | 70.00th=[ 443], 80.00th=[ 443], 90.00th=[ 443], 95.00th=[ 443], | 99.00th=[ 443], 99.50th=[ 443], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 8192, max= 8192, per=49.45%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=8283KiB/s (8482kB/s)(8192KiB/989msec) clat (msec): min=176, max=365, avg=271.32, stdev=133.70 lat (msec): min=177, max=366, avg=271.60, stdev=133.71 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 178], 10.00th=[ 178], 20.00th=[ 178], | 30.00th=[ 178], 40.00th=[ 178], 50.00th=[ 178], 60.00th=[ 368], | 70.00th=[ 368], 80.00th=[ 368], 90.00th=[ 368], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.10%, sys=14.98%, ctx=67, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8569: Sat Feb 24 23:37:58 2024 write: IOPS=4, BW=16.1MiB/s (16.9MB/s)(12.0MiB/745msec) clat (msec): min=188, max=284, avg=246.81, stdev=51.03 lat (msec): min=188, max=284, avg=247.04, stdev=51.06 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 188], 10.00th=[ 188], 20.00th=[ 188], | 30.00th=[ 188], 40.00th=[ 268], 50.00th=[ 268], 60.00th=[ 268], | 70.00th=[ 284], 80.00th=[ 284], 90.00th=[ 284], 95.00th=[ 284], | 99.00th=[ 284], 99.50th=[ 284], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min=16384, max=16384, per=49.45%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.13%, sys=15.19%, ctx=78, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8570: Sat Feb 24 23:37:58 2024 write: IOPS=4, BW=18.5MiB/s (19.4MB/s)(12.0MiB/649msec) clat (msec): min=64, max=485, avg=212.31, stdev=236.61 lat (msec): min=65, max=485, avg=212.61, stdev=236.62 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 65], 10.00th=[ 65], 20.00th=[ 65], | 30.00th=[ 65], 40.00th=[ 87], 50.00th=[ 87], 60.00th=[ 87], | 70.00th=[ 485], 80.00th=[ 485], 90.00th=[ 485], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min=16384, max=16384, per=49.45%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=66.67%, 500=33.33% cpu : usr=0.15%, sys=17.75%, ctx=60, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.2MiB/s (16.0MB/s), 4142KiB/s-14.9MiB/s (4241kB/s-15.7MB/s), io=16.0MiB (16.8MB), run=804-989msec WRITE: bw=32.4MiB/s (33.9MB/s), 8283KiB/s-18.5MiB/s (8482kB/s-19.4MB/s), io=32.0MiB (33.6MB), run=649-989msec