== sanity test 398b: DIO and buffer IO race ============== 18:43:41 (1746139421) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.34941 s, 21.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=3969: Thu May 1 18:44:35 2025 read: IOPS=30, BW=123KiB/s (126kB/s)(5992KiB/48706msec) clat (usec): min=52, max=128419, avg=14233.84, stdev=12508.96 lat (usec): min=53, max=128420, avg=14235.01, stdev=12508.95 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 98], 10.00th=[ 111], 20.00th=[ 293], | 30.00th=[ 7767], 40.00th=[ 10159], 50.00th=[ 12125], 60.00th=[ 14353], | 70.00th=[ 17957], 80.00th=[ 23462], 90.00th=[ 30016], 95.00th=[ 36963], | 99.00th=[ 53740], 99.50th=[ 58983], 99.90th=[109577], 99.95th=[128451], | 99.99th=[128451] bw ( KiB/s): min= 24, max= 256, per=25.10%, avg=121.98, stdev=44.03, samples=97 iops : min= 6, max= 64, avg=30.10, stdev=11.06, samples=97 write: IOPS=32, BW=129KiB/s (132kB/s)(6296KiB/48706msec) clat (usec): min=549, max=307924, avg=17168.51, stdev=20329.33 lat (usec): min=549, max=307925, avg=17172.58, stdev=20330.34 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 742], 10.00th=[ 783], 20.00th=[ 1074], | 30.00th=[ 2769], 40.00th=[ 6456], 50.00th=[ 12125], 60.00th=[ 17171], | 70.00th=[ 22152], 80.00th=[ 30278], 90.00th=[ 41681], 95.00th=[ 52691], | 99.00th=[ 73925], 99.50th=[ 88605], 99.90th=[258999], 99.95th=[308282], | 99.99th=[308282] bw ( KiB/s): min= 32, max= 264, per=25.44%, avg=129.49, stdev=42.65, samples=96 iops : min= 8, max= 66, avg=31.98, stdev=10.70, samples=96 lat (usec) : 100=3.06%, 250=6.61%, 500=0.29%, 750=3.16%, 1000=7.13% lat (msec) : 2=3.48%, 4=4.88%, 10=14.16%, 20=27.28%, 50=26.37% lat (msec) : 100=3.35%, 250=0.16%, 500=0.07% cpu : usr=0.08%, sys=9.83%, ctx=4882, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3972: Thu May 1 18:44:35 2025 read: IOPS=30, BW=120KiB/s (123kB/s)(5940KiB/49330msec) clat (usec): min=65, max=429040, avg=14736.56, stdev=16563.38 lat (usec): min=66, max=429042, avg=14738.36, stdev=16563.40 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 97], 10.00th=[ 110], 20.00th=[ 206], | 30.00th=[ 8160], 40.00th=[ 10290], 50.00th=[ 12649], 60.00th=[ 15008], | 70.00th=[ 19006], 80.00th=[ 23725], 90.00th=[ 30802], 95.00th=[ 35914], | 99.00th=[ 52167], 99.50th=[ 67634], 99.90th=[127402], 99.95th=[429917], | 99.99th=[429917] bw ( KiB/s): min= 8, max= 248, per=24.65%, avg=119.81, stdev=40.98, samples=98 iops : min= 2, max= 62, avg=29.67, stdev=10.32, samples=98 write: IOPS=32, BW=129KiB/s (132kB/s)(6348KiB/49330msec) clat (usec): min=498, max=166494, avg=17104.00, stdev=18443.05 lat (usec): min=499, max=166494, avg=17105.79, stdev=18442.92 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 1090], | 30.00th=[ 3228], 40.00th=[ 7504], 50.00th=[ 12780], 60.00th=[ 17171], | 70.00th=[ 22414], 80.00th=[ 29754], 90.00th=[ 40109], 95.00th=[ 50594], | 99.00th=[ 79168], 99.50th=[ 92799], 99.90th=[160433], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 40, max= 280, per=25.08%, avg=127.65, stdev=40.33, samples=98 iops : min= 10, max= 70, avg=31.63, stdev=10.14, samples=98 lat (usec) : 100=3.12%, 250=6.64%, 500=0.39%, 750=3.68%, 1000=6.41% lat (msec) : 2=2.64%, 4=5.08%, 10=14.10%, 20=26.63%, 50=27.99% lat (msec) : 100=3.03%, 250=0.26%, 500=0.03% cpu : usr=0.16%, sys=9.93%, ctx=4895, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3974: Thu May 1 18:44:35 2025 read: IOPS=30, BW=122KiB/s (125kB/s)(6028KiB/49323msec) clat (usec): min=71, max=89186, avg=13767.12, stdev=12086.84 lat (usec): min=71, max=89187, avg=13770.34, stdev=12087.34 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 94], 10.00th=[ 105], 20.00th=[ 139], | 30.00th=[ 7373], 40.00th=[ 9896], 50.00th=[12256], 60.00th=[14746], | 70.00th=[17957], 80.00th=[22152], 90.00th=[28967], 95.00th=[35914], | 99.00th=[50070], 99.50th=[60031], 99.90th=[77071], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 16, max= 254, per=24.91%, avg=121.06, stdev=44.62, samples=98 iops : min= 4, max= 63, avg=29.97, stdev=11.17, samples=98 write: IOPS=31, BW=127KiB/s (130kB/s)(6260KiB/49323msec) clat (usec): min=637, max=405811, avg=18032.06, stdev=23052.33 lat (usec): min=638, max=405813, avg=18034.42, stdev=23051.75 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 742], 10.00th=[ 783], 20.00th=[ 1020], | 30.00th=[ 3097], 40.00th=[ 7701], 50.00th=[ 12911], 60.00th=[ 17695], | 70.00th=[ 23462], 80.00th=[ 30278], 90.00th=[ 42206], 95.00th=[ 54264], | 99.00th=[ 83362], 99.50th=[101188], 99.90th=[346031], 99.95th=[404751], | 99.99th=[404751] bw ( KiB/s): min= 24, max= 216, per=24.63%, avg=125.36, stdev=40.61, samples=98 iops : min= 6, max= 54, avg=31.05, stdev=10.19, samples=98 lat (usec) : 100=3.74%, 250=7.29%, 500=0.46%, 750=3.32%, 1000=6.97% lat (msec) : 2=2.90%, 4=4.39%, 10=13.67%, 20=27.21%, 50=26.20% lat (msec) : 100=3.58%, 250=0.16%, 500=0.10% cpu : usr=0.15%, sys=9.46%, ctx=4799, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3975: Thu May 1 18:44:35 2025 read: IOPS=30, BW=123KiB/s (126kB/s)(6052KiB/49338msec) clat (usec): min=64, max=219475, avg=14060.51, stdev=13995.61 lat (usec): min=64, max=219477, avg=14063.85, stdev=13996.51 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 97], 10.00th=[ 110], 20.00th=[ 145], | 30.00th=[ 7373], 40.00th=[ 9765], 50.00th=[ 12125], 60.00th=[ 14615], | 70.00th=[ 17695], 80.00th=[ 22152], 90.00th=[ 30278], 95.00th=[ 37487], | 99.00th=[ 54264], 99.50th=[ 67634], 99.90th=[162530], 99.95th=[219153], | 99.99th=[219153] bw ( KiB/s): min= 31, max= 240, per=25.41%, avg=123.49, stdev=38.40, samples=96 iops : min= 7, max= 60, avg=30.54, stdev= 9.62, samples=96 write: IOPS=31, BW=126KiB/s (129kB/s)(6236KiB/49338msec) clat (usec): min=480, max=393703, avg=17779.84, stdev=20930.64 lat (usec): min=482, max=393706, avg=17781.51, stdev=20930.72 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 750], 10.00th=[ 791], 20.00th=[ 1074], | 30.00th=[ 3294], 40.00th=[ 7767], 50.00th=[ 12780], 60.00th=[ 17695], | 70.00th=[ 23200], 80.00th=[ 30540], 90.00th=[ 42206], 95.00th=[ 53740], | 99.00th=[ 80217], 99.50th=[ 90702], 99.90th=[158335], 99.95th=[392168], | 99.99th=[392168] bw ( KiB/s): min= 23, max= 232, per=24.78%, avg=126.14, stdev=39.82, samples=97 iops : min= 5, max= 58, avg=31.21, stdev= 9.98, samples=97 lat (usec) : 100=3.12%, 250=7.71%, 500=0.39%, 750=2.93%, 1000=7.06% lat (msec) : 2=3.81%, 4=3.71%, 10=13.83%, 20=27.73%, 50=25.98% lat (msec) : 100=3.45%, 250=0.23%, 500=0.03% cpu : usr=0.16%, sys=9.53%, ctx=4785, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=487KiB/s (498kB/s), 120KiB/s-123KiB/s (123kB/s-126kB/s), io=23.4MiB (24.6MB), run=48706-49338msec WRITE: bw=510KiB/s (522kB/s), 126KiB/s-129KiB/s (129kB/s-132kB/s), io=24.6MiB (25.7MB), run=48706-49338msec rand-rw: (groupid=0, jobs=1): err= 0: pid=3970: Thu May 1 18:46:33 2025 read: IOPS=8, BW=35.9KiB/s (36.8kB/s)(5992KiB/166886msec) clat (msec): min=2, max=457, avg=16.28, stdev=17.69 lat (msec): min=2, max=457, avg=16.29, stdev=17.69 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 8], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 17], 80.00th=[ 21], 90.00th=[ 29], 95.00th=[ 42], | 99.00th=[ 66], 99.50th=[ 89], 99.90th=[ 176], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 7, max= 151, per=26.11%, avg=37.34, stdev=24.94, samples=317 iops : min= 1, max= 37, avg= 9.06, stdev= 6.22, samples=317 write: IOPS=9, BW=37.7KiB/s (38.6kB/s)(6296KiB/166886msec) clat (msec): min=27, max=315, avg=90.30, stdev=39.99 lat (msec): min=27, max=315, avg=90.31, stdev=39.98 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 45], 10.00th=[ 50], 20.00th=[ 59], | 30.00th=[ 67], 40.00th=[ 73], 50.00th=[ 82], 60.00th=[ 90], | 70.00th=[ 101], 80.00th=[ 116], 90.00th=[ 146], 95.00th=[ 169], | 99.00th=[ 236], 99.50th=[ 251], 99.90th=[ 288], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 7, max= 80, per=24.95%, avg=37.42, stdev=13.28, samples=332 iops : min= 1, max= 20, avg= 9.08, stdev= 3.37, samples=332 lat (msec) : 4=0.52%, 10=16.83%, 20=21.06%, 50=14.42%, 100=31.48% lat (msec) : 250=15.36%, 500=0.33% cpu : usr=0.03%, sys=2.33%, ctx=4031, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3971: Thu May 1 18:46:33 2025 read: IOPS=8, BW=35.5KiB/s (36.4kB/s)(5940KiB/167258msec) clat (msec): min=2, max=357, avg=15.94, stdev=14.64 lat (msec): min=2, max=357, avg=15.95, stdev=14.64 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 21], 90.00th=[ 29], 95.00th=[ 40], | 99.00th=[ 62], 99.50th=[ 74], 99.90th=[ 138], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 7, max= 120, per=25.84%, avg=36.95, stdev=23.48, samples=317 iops : min= 1, max= 30, avg= 8.91, stdev= 5.93, samples=317 write: IOPS=9, BW=37.0KiB/s (38.9kB/s)(6348KiB/167258msec) clat (msec): min=27, max=355, avg=90.26, stdev=42.04 lat (msec): min=27, max=355, avg=90.27, stdev=42.04 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 44], 10.00th=[ 49], 20.00th=[ 57], | 30.00th=[ 65], 40.00th=[ 72], 50.00th=[ 80], 60.00th=[ 90], | 70.00th=[ 102], 80.00th=[ 117], 90.00th=[ 146], 95.00th=[ 178], | 99.00th=[ 234], 99.50th=[ 268], 99.90th=[ 355], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 7, max= 87, per=25.04%, avg=37.56, stdev=13.72, samples=333 iops : min= 1, max= 21, avg= 9.07, stdev= 3.49, samples=333 lat (msec) : 4=0.39%, 10=16.11%, 20=21.00%, 50=15.43%, 100=30.96% lat (msec) : 250=15.76%, 500=0.36% cpu : usr=0.03%, sys=2.37%, ctx=4066, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3973: Thu May 1 18:46:33 2025 read: IOPS=9, BW=36.1KiB/s (36.0kB/s)(6028KiB/167045msec) clat (msec): min=2, max=178, avg=16.67, stdev=13.97 lat (msec): min=2, max=178, avg=16.67, stdev=13.97 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 12], 50.00th=[ 13], 60.00th=[ 16], | 70.00th=[ 18], 80.00th=[ 22], 90.00th=[ 31], 95.00th=[ 42], | 99.00th=[ 64], 99.50th=[ 84], 99.90th=[ 176], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 7, max= 136, per=25.59%, avg=36.59, stdev=23.04, samples=324 iops : min= 1, max= 34, avg= 8.80, stdev= 5.84, samples=324 write: IOPS=9, BW=37.5KiB/s (38.4kB/s)(6260KiB/167045msec) clat (msec): min=26, max=406, avg=90.45, stdev=41.10 lat (msec): min=26, max=406, avg=90.45, stdev=41.10 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 44], 10.00th=[ 50], 20.00th=[ 59], | 30.00th=[ 66], 40.00th=[ 72], 50.00th=[ 82], 60.00th=[ 91], | 70.00th=[ 102], 80.00th=[ 117], 90.00th=[ 144], 95.00th=[ 174], | 99.00th=[ 236], 99.50th=[ 262], 99.90th=[ 284], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 7, max= 72, per=24.68%, avg=37.02, stdev=13.46, samples=332 iops : min= 1, max= 18, avg= 8.90, stdev= 3.43, samples=332 lat (msec) : 4=0.62%, 10=15.53%, 20=21.58%, 50=15.30%, 100=30.96% lat (msec) : 250=15.66%, 500=0.36% cpu : usr=0.03%, sys=2.38%, ctx=4077, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3976: Thu May 1 18:46:33 2025 read: IOPS=9, BW=36.4KiB/s (37.3kB/s)(6052KiB/166357msec) clat (msec): min=3, max=182, avg=15.67, stdev=12.65 lat (msec): min=3, max=182, avg=15.67, stdev=12.65 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 21], 90.00th=[ 29], 95.00th=[ 36], | 99.00th=[ 64], 99.50th=[ 88], 99.90th=[ 134], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 7, max= 128, per=26.82%, avg=38.35, stdev=22.70, samples=312 iops : min= 1, max= 32, avg= 9.30, stdev= 5.72, samples=312 write: IOPS=9, BW=37.5KiB/s (38.4kB/s)(6236KiB/166357msec) clat (msec): min=26, max=391, avg=91.28, stdev=43.55 lat (msec): min=26, max=391, avg=91.28, stdev=43.55 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 44], 10.00th=[ 50], 20.00th=[ 59], | 30.00th=[ 67], 40.00th=[ 73], 50.00th=[ 83], 60.00th=[ 90], | 70.00th=[ 100], 80.00th=[ 117], 90.00th=[ 146], 95.00th=[ 182], | 99.00th=[ 245], 99.50th=[ 262], 99.90th=[ 355], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 7, max= 72, per=24.79%, avg=37.19, stdev=13.54, samples=331 iops : min= 1, max= 18, avg= 9.00, stdev= 3.43, samples=331 lat (msec) : 4=0.29%, 10=17.55%, 20=20.93%, 50=15.33%, 100=30.70% lat (msec) : 250=14.78%, 500=0.42% cpu : usr=0.06%, sys=2.37%, ctx=4070, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=144KiB/s (147kB/s), 35.5KiB/s-36.4KiB/s (36.4kB/s-37.3kB/s), io=23.4MiB (24.6MB), run=166357-167258msec WRITE: bw=150KiB/s (154kB/s), 37.5KiB/s-37.0KiB/s (38.4kB/s-38.9kB/s), io=24.6MiB (25.7MB), run=166357-167258msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4122: Thu May 1 18:46:44 2025 read: IOPS=38, BW=624KiB/s (639kB/s)(5920KiB/9489msec) clat (usec): min=62, max=69313, avg=10692.52, stdev=11891.00 lat (usec): min=63, max=69315, avg=10696.48, stdev=11888.90 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 90], 10.00th=[ 100], 20.00th=[ 113], | 30.00th=[ 127], 40.00th=[ 302], 50.00th=[ 8717], 60.00th=[12125], | 70.00th=[15401], 80.00th=[20055], 90.00th=[27395], 95.00th=[33817], | 99.00th=[42730], 99.50th=[44303], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 160, max= 1245, per=25.74%, avg=625.94, stdev=268.79, samples=18 iops : min= 10, max= 77, avg=38.67, stdev=16.87, samples=18 write: IOPS=41, BW=671KiB/s (687kB/s)(6368KiB/9489msec) clat (usec): min=855, max=166887, avg=13563.37, stdev=18527.21 lat (usec): min=856, max=166889, avg=13565.16, stdev=18527.40 clat percentiles (usec): | 1.00th=[ 906], 5.00th=[ 1012], 10.00th=[ 1106], 20.00th=[ 1237], | 30.00th=[ 1762], 40.00th=[ 3261], 50.00th=[ 6259], 60.00th=[ 10159], | 70.00th=[ 16450], 80.00th=[ 22676], 90.00th=[ 34341], 95.00th=[ 47973], | 99.00th=[ 89654], 99.50th=[107480], 99.90th=[166724], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 96, max= 1088, per=26.75%, avg=684.50, stdev=265.07, samples=18 iops : min= 6, max= 68, avg=42.33, stdev=16.65, samples=18 lat (usec) : 100=5.08%, 250=14.06%, 500=0.91%, 750=0.26%, 1000=2.60% lat (msec) : 2=14.45%, 4=5.60%, 10=13.41%, 20=21.74%, 50=19.27% lat (msec) : 100=2.21%, 250=0.39% cpu : usr=0.17%, sys=11.50%, ctx=930, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4125: Thu May 1 18:46:44 2025 read: IOPS=37, BW=604KiB/s (619kB/s)(5744KiB/9504msec) clat (usec): min=67, max=64226, avg=11367.16, stdev=11917.14 lat (usec): min=67, max=64226, avg=11368.30, stdev=11917.20 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 93], 10.00th=[ 103], 20.00th=[ 117], | 30.00th=[ 141], 40.00th=[ 5669], 50.00th=[ 9634], 60.00th=[13304], | 70.00th=[16450], 80.00th=[20841], 90.00th=[27132], 95.00th=[34341], | 99.00th=[46924], 99.50th=[57934], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 64, max= 1024, per=24.81%, avg=603.37, stdev=250.49, samples=19 iops : min= 4, max= 64, avg=37.32, stdev=15.78, samples=19 write: IOPS=43, BW=689KiB/s (705kB/s)(6544KiB/9504msec) clat (usec): min=740, max=113137, avg=13093.12, stdev=17699.99 lat (usec): min=741, max=113139, avg=13097.02, stdev=17699.12 clat percentiles (usec): | 1.00th=[ 906], 5.00th=[ 996], 10.00th=[ 1057], 20.00th=[ 1221], | 30.00th=[ 1860], 40.00th=[ 3326], 50.00th=[ 5014], 60.00th=[ 8848], | 70.00th=[ 15008], 80.00th=[ 22938], 90.00th=[ 34866], 95.00th=[ 47973], | 99.00th=[ 80217], 99.50th=[108528], 99.90th=[112722], 99.95th=[112722], | 99.99th=[112722] bw ( KiB/s): min= 288, max= 1088, per=26.79%, avg=685.63, stdev=230.35, samples=19 iops : min= 18, max= 68, avg=42.47, stdev=14.58, samples=19 lat (usec) : 100=3.65%, 250=12.76%, 500=0.65%, 750=0.39%, 1000=3.26% lat (msec) : 2=14.06%, 4=6.25%, 10=16.15%, 20=20.18%, 50=19.79% lat (msec) : 100=2.47%, 250=0.39% cpu : usr=0.07%, sys=11.45%, ctx=998, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4127: Thu May 1 18:46:44 2025 read: IOPS=41, BW=658KiB/s (673kB/s)(6256KiB/9513msec) clat (usec): min=58, max=93989, avg=10328.03, stdev=13249.17 lat (usec): min=59, max=93990, avg=10329.10, stdev=13249.12 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 92], 10.00th=[ 99], 20.00th=[ 112], | 30.00th=[ 127], 40.00th=[ 165], 50.00th=[ 6456], 60.00th=[10945], | 70.00th=[14746], 80.00th=[20055], 90.00th=[26346], 95.00th=[32900], | 99.00th=[66323], 99.50th=[71828], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 95, max= 1147, per=27.01%, avg=656.95, stdev=267.18, samples=19 iops : min= 5, max= 71, avg=40.63, stdev=16.68, samples=19 write: IOPS=39, BW=634KiB/s (649kB/s)(6032KiB/9513msec) clat (usec): min=883, max=203456, avg=14200.99, stdev=21985.76 lat (usec): min=885, max=203457, avg=14202.62, stdev=21985.79 clat percentiles (usec): | 1.00th=[ 906], 5.00th=[ 1004], 10.00th=[ 1090], 20.00th=[ 1221], | 30.00th=[ 1713], 40.00th=[ 2933], 50.00th=[ 5800], 60.00th=[ 11338], | 70.00th=[ 15664], 80.00th=[ 22414], 90.00th=[ 35390], 95.00th=[ 47449], | 99.00th=[103285], 99.50th=[185598], 99.90th=[204473], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 126, max= 1024, per=24.63%, avg=630.16, stdev=210.42, samples=19 iops : min= 7, max= 64, avg=38.95, stdev=13.22, samples=19 lat (usec) : 100=5.34%, 250=15.76%, 500=0.78%, 750=0.39%, 1000=2.60% lat (msec) : 2=14.45%, 4=6.64%, 10=11.33%, 20=21.09%, 50=18.75% lat (msec) : 100=2.34%, 250=0.52% cpu : usr=0.13%, sys=10.86%, ctx=909, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4128: Thu May 1 18:46:44 2025 read: IOPS=38, BW=613KiB/s (627kB/s)(6032KiB/9846msec) clat (usec): min=62, max=63283, avg=10903.22, stdev=12511.50 lat (usec): min=62, max=63284, avg=10905.37, stdev=12511.69 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 92], 10.00th=[ 99], 20.00th=[ 113], | 30.00th=[ 128], 40.00th=[ 174], 50.00th=[ 8455], 60.00th=[12780], | 70.00th=[16057], 80.00th=[20841], 90.00th=[29492], 95.00th=[34866], | 99.00th=[50070], 99.50th=[57410], 99.90th=[63177], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 318, max= 1056, per=25.95%, avg=631.22, stdev=189.52, samples=18 iops : min= 19, max= 66, avg=38.94, stdev=12.07, samples=18 write: IOPS=39, BW=635KiB/s (651kB/s)(6256KiB/9846msec) clat (usec): min=848, max=126430, avg=14243.26, stdev=17897.61 lat (usec): min=849, max=126433, avg=14252.44, stdev=17895.71 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 1045], 10.00th=[ 1106], 20.00th=[ 1336], | 30.00th=[ 1827], 40.00th=[ 3130], 50.00th=[ 6456], 60.00th=[ 11338], | 70.00th=[ 18220], 80.00th=[ 26870], 90.00th=[ 35914], 95.00th=[ 48497], | 99.00th=[ 89654], 99.50th=[102237], 99.90th=[126354], 99.95th=[126354], | 99.99th=[126354] bw ( KiB/s): min= 96, max= 992, per=24.42%, avg=624.84, stdev=208.24, samples=19 iops : min= 6, max= 62, avg=38.58, stdev=13.01, samples=19 lat (usec) : 100=5.47%, 250=14.45%, 500=0.78%, 750=0.39%, 1000=1.82% lat (msec) : 2=15.10%, 4=6.51%, 10=11.59%, 20=19.14%, 50=22.14% lat (msec) : 100=2.34%, 250=0.26% cpu : usr=0.18%, sys=11.09%, ctx=1016, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2433KiB/s (2491kB/s), 604KiB/s-658KiB/s (619kB/s-673kB/s), io=23.4MiB (24.5MB), run=9489-9846msec WRITE: bw=2559KiB/s (2621kB/s), 634KiB/s-689KiB/s (649kB/s-705kB/s), io=24.6MiB (25.8MB), run=9489-9846msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4121: Thu May 1 18:47:13 2025 read: IOPS=9, BW=153KiB/s (156kB/s)(5920KiB/38752msec) clat (msec): min=3, max=104, avg=14.09, stdev=13.24 lat (msec): min=3, max=104, avg=14.09, stdev=13.24 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 11], 60.00th=[ 12], | 70.00th=[ 15], 80.00th=[ 19], 90.00th=[ 25], 95.00th=[ 33], | 99.00th=[ 94], 99.50th=[ 103], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 31, max= 416, per=26.15%, avg=159.51, stdev=80.09, samples=73 iops : min= 1, max= 26, avg= 9.58, stdev= 5.07, samples=73 write: IOPS=10, BW=164KiB/s (168kB/s)(6368KiB/38752msec) clat (msec): min=34, max=276, avg=84.12, stdev=38.25 lat (msec): min=34, max=276, avg=84.12, stdev=38.25 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 45], 10.00th=[ 49], 20.00th=[ 57], | 30.00th=[ 63], 40.00th=[ 70], 50.00th=[ 74], 60.00th=[ 83], | 70.00th=[ 89], 80.00th=[ 103], 90.00th=[ 134], 95.00th=[ 167], | 99.00th=[ 224], 99.50th=[ 259], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 63, max= 256, per=25.42%, avg=163.18, stdev=57.65, samples=77 iops : min= 3, max= 16, avg= 9.82, stdev= 3.66, samples=77 lat (msec) : 4=0.26%, 10=23.05%, 20=16.41%, 50=13.80%, 100=35.42% lat (msec) : 250=10.68%, 500=0.39% cpu : usr=0.03%, sys=2.81%, ctx=1012, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4123: Thu May 1 18:47:13 2025 read: IOPS=9, BW=146KiB/s (150kB/s)(5744KiB/39220msec) clat (msec): min=3, max=167, avg=13.87, stdev=14.32 lat (msec): min=3, max=167, avg=13.87, stdev=14.32 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 14], 80.00th=[ 19], 90.00th=[ 26], 95.00th=[ 41], | 99.00th=[ 68], 99.50th=[ 97], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 31, max= 476, per=25.09%, avg=153.07, stdev=90.57, samples=74 iops : min= 1, max= 29, avg= 9.23, stdev= 5.74, samples=74 write: IOPS=10, BW=167KiB/s (171kB/s)(6544KiB/39220msec) clat (msec): min=28, max=359, avg=83.61, stdev=38.74 lat (msec): min=28, max=359, avg=83.61, stdev=38.74 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 43], 10.00th=[ 47], 20.00th=[ 56], | 30.00th=[ 63], 40.00th=[ 70], 50.00th=[ 77], 60.00th=[ 83], | 70.00th=[ 92], 80.00th=[ 102], 90.00th=[ 117], 95.00th=[ 159], | 99.00th=[ 239], 99.50th=[ 255], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 31, max= 319, per=25.80%, avg=165.67, stdev=57.85, samples=78 iops : min= 1, max= 19, avg=10.00, stdev= 3.71, samples=78 lat (msec) : 4=0.39%, 10=25.78%, 20=12.37%, 50=13.67%, 100=35.81% lat (msec) : 250=11.59%, 500=0.39% cpu : usr=0.06%, sys=2.65%, ctx=952, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4124: Thu May 1 18:47:13 2025 read: IOPS=10, BW=166KiB/s (170kB/s)(6256KiB/37775msec) clat (msec): min=4, max=116, avg=14.46, stdev=13.49 lat (msec): min=4, max=116, avg=14.46, stdev=13.49 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 15], 80.00th=[ 20], 90.00th=[ 28], 95.00th=[ 36], | 99.00th=[ 79], 99.50th=[ 96], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 31, max= 512, per=27.64%, avg=168.58, stdev=112.43, samples=74 iops : min= 1, max= 32, avg=10.18, stdev= 7.13, samples=74 write: IOPS=9, BW=160KiB/s (164kB/s)(6032KiB/37775msec) clat (msec): min=36, max=269, avg=85.01, stdev=39.49 lat (msec): min=36, max=269, avg=85.01, stdev=39.49 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 47], 10.00th=[ 52], 20.00th=[ 56], | 30.00th=[ 64], 40.00th=[ 69], 50.00th=[ 74], 60.00th=[ 82], | 70.00th=[ 91], 80.00th=[ 105], 90.00th=[ 131], 95.00th=[ 176], | 99.00th=[ 249], 99.50th=[ 255], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 62, max= 256, per=24.71%, avg=158.65, stdev=53.80, samples=75 iops : min= 3, max= 16, avg= 9.56, stdev= 3.47, samples=75 lat (msec) : 10=27.60%, 20=13.41%, 50=12.76%, 100=35.55%, 250=10.29% lat (msec) : 500=0.39% cpu : usr=0.03%, sys=2.91%, ctx=1001, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4126: Thu May 1 18:47:13 2025 read: IOPS=9, BW=155KiB/s (159kB/s)(6032KiB/38948msec) clat (usec): min=3076, max=84347, avg=13839.63, stdev=10691.51 lat (usec): min=3077, max=84349, avg=13841.16, stdev=10691.52 clat percentiles (usec): | 1.00th=[ 4228], 5.00th=[ 5145], 10.00th=[ 5800], 20.00th=[ 6587], | 30.00th=[ 7504], 40.00th=[ 8717], 50.00th=[10028], 60.00th=[11994], | 70.00th=[15401], 80.00th=[20055], 90.00th=[25560], 95.00th=[31851], | 99.00th=[61604], 99.50th=[72877], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 31, max= 479, per=26.44%, avg=161.30, stdev=95.59, samples=73 iops : min= 1, max= 29, avg= 9.75, stdev= 5.92, samples=73 write: IOPS=10, BW=161KiB/s (164kB/s)(6256KiB/38948msec) clat (msec): min=31, max=281, avg=85.98, stdev=40.26 lat (msec): min=31, max=281, avg=85.99, stdev=40.27 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 44], 10.00th=[ 49], 20.00th=[ 58], | 30.00th=[ 64], 40.00th=[ 70], 50.00th=[ 77], 60.00th=[ 85], | 70.00th=[ 93], 80.00th=[ 104], 90.00th=[ 138], 95.00th=[ 169], | 99.00th=[ 253], 99.50th=[ 275], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 62, max= 256, per=24.85%, avg=159.53, stdev=56.79, samples=77 iops : min= 3, max= 16, avg= 9.64, stdev= 3.65, samples=77 lat (msec) : 4=0.26%, 10=23.96%, 20=14.97%, 50=15.36%, 100=33.59% lat (msec) : 250=11.33%, 500=0.52% cpu : usr=0.04%, sys=2.87%, ctx=1004, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=611KiB/s (625kB/s), 146KiB/s-166KiB/s (150kB/s-170kB/s), io=23.4MiB (24.5MB), run=37775-39220msec WRITE: bw=643KiB/s (658kB/s), 160KiB/s-167KiB/s (164kB/s-171kB/s), io=24.6MiB (25.8MB), run=37775-39220msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4166: Thu May 1 18:47:17 2025 read: IOPS=3, BW=3579KiB/s (3665kB/s)(7168KiB/2003msec) clat (usec): min=714, max=173023, avg=92869.23, stdev=52598.60 lat (usec): min=716, max=173025, avg=92870.85, stdev=52598.64 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 717], 10.00th=[ 717], 20.00th=[ 74974], | 30.00th=[ 87557], 40.00th=[ 87557], 50.00th=[ 90702], 60.00th=[ 93848], | 70.00th=[ 93848], 80.00th=[129500], 90.00th=[173016], 95.00th=[173016], | 99.00th=[173016], 99.50th=[173016], 99.90th=[173016], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 4031, max= 6131, per=57.96%, avg=5081.00, stdev=1484.92, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2556KiB/s (2618kB/s)(5120KiB/2003msec) clat (msec): min=178, max=302, avg=240.35, stdev=49.39 lat (msec): min=178, max=302, avg=240.42, stdev=49.39 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 180], 10.00th=[ 180], 20.00th=[ 180], | 30.00th=[ 213], 40.00th=[ 213], 50.00th=[ 232], 60.00th=[ 232], | 70.00th=[ 275], 80.00th=[ 275], 90.00th=[ 305], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 2015, max= 4096, per=24.90%, avg=3055.50, stdev=1471.49, samples=2 iops : min= 1, max= 4, avg= 2.50, stdev= 2.12, samples=2 lat (usec) : 750=8.33% lat (msec) : 100=33.33%, 250=41.67%, 500=16.67% cpu : usr=0.00%, sys=11.39%, ctx=319, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4167: Thu May 1 18:47:17 2025 read: IOPS=0, BW=877KiB/s (898kB/s)(2048KiB/2336msec) clat (msec): min=102, max=111, avg=106.85, stdev= 6.17 lat (msec): min=102, max=111, avg=106.85, stdev= 6.17 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 103], 10.00th=[ 103], 20.00th=[ 103], | 30.00th=[ 103], 40.00th=[ 103], 50.00th=[ 103], 60.00th=[ 112], | 70.00th=[ 112], 80.00th=[ 112], 90.00th=[ 112], 95.00th=[ 112], | 99.00th=[ 112], 99.50th=[ 112], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 2039, max= 2048, per=23.31%, avg=2043.50, stdev= 6.36, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4384KiB/s (4489kB/s)(10.0MiB/2336msec) clat (msec): min=60, max=664, avg=208.84, stdev=183.08 lat (msec): min=60, max=664, avg=208.92, stdev=183.08 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 62], 10.00th=[ 62], 20.00th=[ 77], | 30.00th=[ 111], 40.00th=[ 113], 50.00th=[ 113], 60.00th=[ 129], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 317], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 1992, max= 8159, per=28.98%, avg=3556.50, stdev=3068.42, samples=4 iops : min= 1, max= 7, avg= 2.75, stdev= 2.87, samples=4 lat (msec) : 100=16.67%, 250=58.33%, 500=16.67%, 750=8.33% cpu : usr=0.09%, sys=11.73%, ctx=125, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4168: Thu May 1 18:47:17 2025 read: IOPS=3, BW=3926KiB/s (4020kB/s)(6144KiB/1565msec) clat (usec): min=585, max=111764, avg=60326.11, stdev=49025.44 lat (usec): min=587, max=111766, avg=60327.98, stdev=49024.62 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 586], 10.00th=[ 586], 20.00th=[ 644], | 30.00th=[ 644], 40.00th=[ 62653], 50.00th=[ 62653], 60.00th=[ 87557], | 70.00th=[ 99091], 80.00th=[ 99091], 90.00th=[111674], 95.00th=[111674], | 99.00th=[111674], 99.50th=[111674], 99.90th=[111674], 99.95th=[111674], | 99.99th=[111674] bw ( KiB/s): min= 2027, max= 8192, per=58.28%, avg=5109.50, stdev=4359.31, samples=2 iops : min= 1, max= 8, avg= 4.50, stdev= 4.95, samples=2 write: IOPS=3, BW=3926KiB/s (4020kB/s)(6144KiB/1565msec) clat (msec): min=64, max=350, avg=188.31, stdev=127.48 lat (msec): min=64, max=350, avg=188.37, stdev=127.47 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 65], 10.00th=[ 65], 20.00th=[ 82], | 30.00th=[ 82], 40.00th=[ 132], 50.00th=[ 132], 60.00th=[ 159], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 351], 95.00th=[ 351], | 99.00th=[ 351], 99.50th=[ 351], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 4055, max= 4096, per=33.26%, avg=4082.33, stdev=23.67, samples=3 iops : min= 3, max= 4, avg= 3.67, stdev= 0.58, samples=3 lat (usec) : 750=16.67% lat (msec) : 100=41.67%, 250=25.00%, 500=16.67% cpu : usr=0.00%, sys=12.47%, ctx=214, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4169: Thu May 1 18:47:17 2025 read: IOPS=2, BW=2698KiB/s (2762kB/s)(5120KiB/1898msec) clat (usec): min=419, max=54710, avg=11354.82, stdev=24236.71 lat (usec): min=419, max=54712, avg=11356.02, stdev=24237.02 clat percentiles (usec): | 1.00th=[ 420], 5.00th=[ 420], 10.00th=[ 420], 20.00th=[ 420], | 30.00th=[ 490], 40.00th=[ 490], 50.00th=[ 498], 60.00th=[ 498], | 70.00th=[ 660], 80.00th=[ 660], 90.00th=[54789], 95.00th=[54789], | 99.00th=[54789], 99.50th=[54789], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 8175, max= 8175, per=93.25%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3777KiB/s (3867kB/s)(7168KiB/1898msec) clat (msec): min=91, max=439, avg=255.22, stdev=122.06 lat (msec): min=93, max=439, avg=255.60, stdev=121.58 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 92], 10.00th=[ 92], 20.00th=[ 128], | 30.00th=[ 199], 40.00th=[ 199], 50.00th=[ 300], 60.00th=[ 313], | 70.00th=[ 313], 80.00th=[ 317], 90.00th=[ 439], 95.00th=[ 439], | 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 2043, max= 6131, per=27.76%, avg=3407.33, stdev=2358.77, samples=3 iops : min= 1, max= 5, avg= 2.67, stdev= 2.08, samples=3 lat (usec) : 500=25.00%, 750=8.33% lat (msec) : 100=16.67%, 250=16.67%, 500=33.33% cpu : usr=0.00%, sys=8.91%, ctx=62, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8767KiB/s (8978kB/s), 877KiB/s-3926KiB/s (898kB/s-4020kB/s), io=20.0MiB (20.0MB), run=1565-2336msec WRITE: bw=11.0MiB/s (12.6MB/s), 2556KiB/s-4384KiB/s (2618kB/s-4489kB/s), io=28.0MiB (29.4MB), run=1565-2336msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4170: Thu May 1 18:47:18 2025 read: IOPS=3, BW=3633KiB/s (3720kB/s)(7168KiB/1973msec) clat (msec): min=61, max=257, avg=117.03, stdev=70.20 lat (msec): min=61, max=257, avg=117.04, stdev=70.19 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 62], 10.00th=[ 62], 20.00th=[ 74], | 30.00th=[ 83], 40.00th=[ 83], 50.00th=[ 88], 60.00th=[ 93], | 70.00th=[ 93], 80.00th=[ 163], 90.00th=[ 257], 95.00th=[ 257], | 99.00th=[ 257], 99.50th=[ 257], 99.90th=[ 257], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 4096, max= 6131, per=63.12%, avg=5113.50, stdev=1438.96, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2595KiB/s (2657kB/s)(5120KiB/1973msec) clat (msec): min=135, max=274, avg=220.58, stdev=52.54 lat (msec): min=135, max=274, avg=220.66, stdev=52.54 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 215], 40.00th=[ 215], 50.00th=[ 228], 60.00th=[ 228], | 70.00th=[ 251], 80.00th=[ 251], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 2048, max= 4087, per=27.05%, avg=3067.50, stdev=1441.79, samples=2 iops : min= 2, max= 3, avg= 2.50, stdev= 0.71, samples=2 lat (msec) : 100=41.67%, 250=41.67%, 500=16.67% cpu : usr=0.05%, sys=5.02%, ctx=46, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4171: Thu May 1 18:47:18 2025 read: IOPS=0, BW=810KiB/s (830kB/s)(2048KiB/2528msec) clat (msec): min=77, max=180, avg=129.01, stdev=73.45 lat (msec): min=77, max=180, avg=129.02, stdev=73.45 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 78], 10.00th=[ 78], 20.00th=[ 78], | 30.00th=[ 78], 40.00th=[ 78], 50.00th=[ 78], 60.00th=[ 182], | 70.00th=[ 182], 80.00th=[ 182], 90.00th=[ 182], 95.00th=[ 182], | 99.00th=[ 182], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 2039, max= 2048, per=25.23%, avg=2043.50, stdev= 6.36, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=4051KiB/s (4148kB/s)(10.0MiB/2528msec) clat (msec): min=77, max=552, avg=209.67, stdev=140.04 lat (msec): min=77, max=552, avg=209.74, stdev=140.04 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 78], 10.00th=[ 78], 20.00th=[ 129], | 30.00th=[ 138], 40.00th=[ 146], 50.00th=[ 148], 60.00th=[ 159], | 70.00th=[ 239], 80.00th=[ 239], 90.00th=[ 338], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 2048, max= 6144, per=40.36%, avg=4577.75, stdev=1929.91, samples=4 iops : min= 2, max= 6, avg= 4.25, stdev= 1.71, samples=4 lat (msec) : 100=16.67%, 250=66.67%, 500=8.33%, 750=8.33% cpu : usr=0.08%, sys=4.75%, ctx=54, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4172: Thu May 1 18:47:18 2025 read: IOPS=2, BW=2661KiB/s (2725kB/s)(6144KiB/2309msec) clat (msec): min=34, max=241, avg=146.57, stdev=75.68 lat (msec): min=34, max=241, avg=146.57, stdev=75.68 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 81], | 30.00th=[ 81], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 176], | 70.00th=[ 188], 80.00th=[ 188], 90.00th=[ 241], 95.00th=[ 241], | 99.00th=[ 241], 99.50th=[ 241], 99.90th=[ 241], 99.95th=[ 241], | 99.99th=[ 241] bw ( KiB/s): min= 2048, max= 6144, per=50.56%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=2, BW=2661KiB/s (2725kB/s)(6144KiB/2309msec) clat (msec): min=112, max=377, avg=210.87, stdev=100.43 lat (msec): min=112, max=377, avg=210.96, stdev=100.43 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 113], 10.00th=[ 113], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 169], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 380], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 2048, max= 4079, per=22.54%, avg=2555.75, stdev=1015.50, samples=4 iops : min= 2, max= 3, avg= 2.25, stdev= 0.50, samples=4 lat (msec) : 50=8.33%, 100=8.33%, 250=66.67%, 500=16.67% cpu : usr=0.09%, sys=4.64%, ctx=80, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4173: Thu May 1 18:47:18 2025 read: IOPS=2, BW=2171KiB/s (2223kB/s)(5120KiB/2358msec) clat (usec): min=57532, max=91556, avg=79669.35, stdev=13617.79 lat (usec): min=57533, max=91558, avg=79670.96, stdev=13617.68 clat percentiles (usec): | 1.00th=[57410], 5.00th=[57410], 10.00th=[57410], 20.00th=[57410], | 30.00th=[76022], 40.00th=[76022], 50.00th=[85459], 60.00th=[85459], | 70.00th=[87557], 80.00th=[87557], 90.00th=[91751], 95.00th=[91751], | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 2035, max= 6095, per=50.18%, avg=4065.00, stdev=2870.85, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 write: IOPS=2, BW=3040KiB/s (3113kB/s)(7168KiB/2358msec) clat (msec): min=167, max=372, avg=256.59, stdev=65.48 lat (msec): min=167, max=372, avg=256.66, stdev=65.49 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 213], | 30.00th=[ 236], 40.00th=[ 236], 50.00th=[ 245], 60.00th=[ 264], | 70.00th=[ 264], 80.00th=[ 300], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 2031, max= 4096, per=29.97%, avg=3399.33, stdev=1185.08, samples=3 iops : min= 1, max= 4, avg= 2.67, stdev= 1.53, samples=3 lat (msec) : 100=41.67%, 250=33.33%, 500=25.00% cpu : usr=0.08%, sys=4.58%, ctx=62, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8101KiB/s (8296kB/s), 810KiB/s-3633KiB/s (830kB/s-3720kB/s), io=20.0MiB (20.0MB), run=1973-2528msec WRITE: bw=11.1MiB/s (11.6MB/s), 2595KiB/s-4051KiB/s (2657kB/s-4148kB/s), io=28.0MiB (29.4MB), run=1973-2528msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=4185: Thu May 1 18:47:22 2025 read: IOPS=3, BW=12.8MiB/s (13.4MB/s)(12.0MiB/937msec) clat (msec): min=21, max=492, avg=287.90, stdev=241.60 lat (msec): min=21, max=492, avg=287.90, stdev=241.60 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 351], 50.00th=[ 351], 60.00th=[ 351], | 70.00th=[ 493], 80.00th=[ 493], 90.00th=[ 493], 95.00th=[ 493], | 99.00th=[ 493], 99.50th=[ 493], 99.90th=[ 493], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 8143, max= 8143, per=76.59%, avg=8143.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 500=66.67% cpu : usr=0.00%, sys=14.21%, ctx=541, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4186: Thu May 1 18:47:22 2025 read: IOPS=0, BW=2658KiB/s (2722kB/s)(4096KiB/1541msec) clat (nsec): min=321820k, max=321820k, avg=321819742.00, stdev= 0.00 lat (nsec): min=321822k, max=321822k, avg=321822071.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 321], 10.00th=[ 321], 20.00th=[ 321], | 30.00th=[ 321], 40.00th=[ 321], 50.00th=[ 321], 60.00th=[ 321], | 70.00th=[ 321], 80.00th=[ 321], 90.00th=[ 321], 95.00th=[ 321], | 99.00th=[ 321], 99.50th=[ 321], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 8192, max= 8192, per=77.05%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5316KiB/s (5444kB/s)(8192KiB/1541msec) clat (msec): min=505, max=626, avg=566.17, stdev=85.12 lat (msec): min=506, max=626, avg=566.48, stdev=85.13 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 506], 10.00th=[ 506], 20.00th=[ 506], | 30.00th=[ 506], 40.00th=[ 506], 50.00th=[ 506], 60.00th=[ 625], | 70.00th=[ 625], 80.00th=[ 625], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 8175, max= 8175, per=50.72%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.00%, sys=12.53%, ctx=327, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4187: Thu May 1 18:47:22 2025 write: IOPS=1, BW=6044KiB/s (6189kB/s)(12.0MiB/2033msec) clat (msec): min=233, max=1083, avg=655.66, stdev=425.03 lat (msec): min=234, max=1084, avg=656.02, stdev=424.96 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 234], 10.00th=[ 234], 20.00th=[ 234], | 30.00th=[ 234], 40.00th=[ 651], 50.00th=[ 651], 60.00th=[ 651], | 70.00th=[ 1083], 80.00th=[ 1083], 90.00th=[ 1083], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 8078, max= 8078, per=50.12%, avg=8078.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=12.16%, ctx=63, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4188: Thu May 1 18:47:22 2025 write: IOPS=1, BW=7327KiB/s (7503kB/s)(12.0MiB/1677msec) clat (msec): min=289, max=871, avg=511.17, stdev=315.00 lat (msec): min=290, max=884, avg=515.64, stdev=322.16 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 292], 10.00th=[ 292], 20.00th=[ 292], | 30.00th=[ 292], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 372], | 70.00th=[ 869], 80.00th=[ 869], 90.00th=[ 869], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 8159, max= 8192, per=50.72%, avg=8175.50, stdev=23.33, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67%, 1000=33.33% cpu : usr=0.00%, sys=16.23%, ctx=88, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.4MiB/s (10.9MB/s), 2658KiB/s-12.8MiB/s (2722kB/s-13.4MB/s), io=16.0MiB (16.8MB), run=937-1541msec WRITE: bw=15.7MiB/s (16.5MB/s), 5316KiB/s-7327KiB/s (5444kB/s-7503kB/s), io=32.0MiB (33.6MB), run=1541-2033msec rand-rw: (groupid=0, jobs=1): err= 0: pid=4180: Thu May 1 18:47:22 2025 read: IOPS=2, BW=8371KiB/s (8571kB/s)(12.0MiB/1468msec) clat (msec): min=423, max=522, avg=473.25, stdev=49.22 lat (msec): min=423, max=522, avg=473.26, stdev=49.22 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 426], 10.00th=[ 426], 20.00th=[ 426], | 30.00th=[ 426], 40.00th=[ 472], 50.00th=[ 472], 60.00th=[ 472], | 70.00th=[ 523], 80.00th=[ 523], 90.00th=[ 523], 95.00th=[ 523], | 99.00th=[ 523], 99.50th=[ 523], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 7816, max= 7816, per=99.62%, avg=7816.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.07%, sys=4.50%, ctx=39, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4181: Thu May 1 18:47:22 2025 read: IOPS=0, BW=1962KiB/s (2009kB/s)(4096KiB/2088msec) clat (nsec): min=626695k, max=626695k, avg=626695420.00, stdev= 0.00 lat (nsec): min=626698k, max=626698k, avg=626698194.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 625], 5.00th=[ 625], 10.00th=[ 625], 20.00th=[ 625], | 30.00th=[ 625], 40.00th=[ 625], 50.00th=[ 625], 60.00th=[ 625], | 70.00th=[ 625], 80.00th=[ 625], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 7816, max= 7816, per=99.62%, avg=7816.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3923KiB/s (4018kB/s)(8192KiB/2088msec) clat (msec): min=696, max=728, avg=712.19, stdev=22.68 lat (msec): min=696, max=728, avg=712.43, stdev=22.75 clat percentiles (msec): | 1.00th=[ 693], 5.00th=[ 693], 10.00th=[ 693], 20.00th=[ 693], | 30.00th=[ 693], 40.00th=[ 693], 50.00th=[ 693], 60.00th=[ 726], | 70.00th=[ 726], 80.00th=[ 726], 90.00th=[ 726], 95.00th=[ 726], | 99.00th=[ 726], 99.50th=[ 726], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 5520, max= 5520, per=35.17%, avg=5520.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 750=100.00% cpu : usr=0.00%, sys=6.03%, ctx=76, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4182: Thu May 1 18:47:22 2025 write: IOPS=3, BW=14.5MiB/s (15.2MB/s)(12.0MiB/827msec) clat (msec): min=233, max=297, avg=267.45, stdev=32.29 lat (msec): min=233, max=297, avg=267.75, stdev=32.24 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 234], 10.00th=[ 234], 20.00th=[ 234], | 30.00th=[ 234], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 296], 80.00th=[ 296], 90.00th=[ 296], 95.00th=[ 296], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 8175, max= 8175, per=52.09%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=10.29%, ctx=43, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=4183: Thu May 1 18:47:22 2025 write: IOPS=2, BW=8752KiB/s (8962kB/s)(12.0MiB/1404msec) clat (msec): min=186, max=641, avg=445.55, stdev=233.88 lat (msec): min=187, max=642, avg=445.82, stdev=233.87 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 188], 10.00th=[ 188], 20.00th=[ 188], | 30.00th=[ 188], 40.00th=[ 510], 50.00th=[ 510], 60.00th=[ 510], | 70.00th=[ 642], 80.00th=[ 642], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 8078, max= 8192, per=51.84%, avg=8135.00, stdev=80.61, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 750=66.67% cpu : usr=0.00%, sys=6.91%, ctx=25, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7847KiB/s (8035kB/s), 1962KiB/s-8371KiB/s (2009kB/s-8571kB/s), io=16.0MiB (16.8MB), run=1468-2088msec WRITE: bw=15.3MiB/s (16.1MB/s), 3923KiB/s-14.5MiB/s (4018kB/s-15.2MB/s), io=32.0MiB (33.6MB), run=827-2088msec