== sanity test 398b: DIO and buffer IO race ============== 19:51:53 (1746143513) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.83179 s, 27.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=3028: Thu May 1 19:52:33 2025 read: IOPS=41, BW=165KiB/s (169kB/s)(5992KiB/36340msec) clat (usec): min=57, max=188604, avg=10571.56, stdev=10304.71 lat (usec): min=58, max=188606, avg=10574.15, stdev=10304.17 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 99], 10.00th=[ 114], 20.00th=[ 184], | 30.00th=[ 5800], 40.00th=[ 7570], 50.00th=[ 8848], 60.00th=[ 10814], | 70.00th=[ 13173], 80.00th=[ 16450], 90.00th=[ 22414], 95.00th=[ 27657], | 99.00th=[ 41157], 99.50th=[ 45876], 99.90th=[ 82314], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 24, max= 320, per=25.10%, avg=164.42, stdev=57.19, samples=72 iops : min= 6, max= 80, avg=40.78, stdev=14.30, samples=72 write: IOPS=43, BW=173KiB/s (177kB/s)(6296KiB/36340msec) clat (usec): min=494, max=256035, avg=12869.74, stdev=15961.48 lat (usec): min=496, max=256037, avg=12871.43, stdev=15961.64 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 750], 10.00th=[ 799], 20.00th=[ 1037], | 30.00th=[ 2114], 40.00th=[ 4621], 50.00th=[ 8848], 60.00th=[ 12387], | 70.00th=[ 16188], 80.00th=[ 21103], 90.00th=[ 31327], 95.00th=[ 39584], | 99.00th=[ 62129], 99.50th=[ 76022], 99.90th=[166724], 99.95th=[256902], | 99.99th=[256902] bw ( KiB/s): min= 8, max= 424, per=25.20%, avg=172.65, stdev=65.47, samples=72 iops : min= 2, max= 106, avg=42.83, stdev=16.37, samples=72 lat (usec) : 100=2.77%, 250=7.19%, 500=0.62%, 750=3.03%, 1000=7.03% lat (msec) : 2=5.70%, 4=5.11%, 10=23.76%, 20=26.56%, 50=16.99% lat (msec) : 100=1.04%, 250=0.16%, 500=0.03% cpu : usr=0.17%, sys=12.77%, ctx=4925, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3029: Thu May 1 19:52:33 2025 read: IOPS=40, BW=163KiB/s (167kB/s)(5940KiB/36467msec) clat (usec): min=61, max=155069, avg=10768.70, stdev=10489.36 lat (usec): min=62, max=155071, avg=10770.24, stdev=10489.35 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 100], 10.00th=[ 117], 20.00th=[ 668], | 30.00th=[ 5866], 40.00th=[ 7439], 50.00th=[ 8979], 60.00th=[ 10945], | 70.00th=[ 13173], 80.00th=[ 16712], 90.00th=[ 22152], 95.00th=[ 27657], | 99.00th=[ 43779], 99.50th=[ 55313], 99.90th=[108528], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 7, max= 351, per=24.86%, avg=162.85, stdev=60.77, samples=72 iops : min= 1, max= 87, avg=40.46, stdev=15.15, samples=72 write: IOPS=43, BW=174KiB/s (178kB/s)(6348KiB/36467msec) clat (usec): min=467, max=281045, avg=12750.51, stdev=15324.76 lat (usec): min=468, max=281045, avg=12752.03, stdev=15324.79 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 750], 10.00th=[ 807], 20.00th=[ 1074], | 30.00th=[ 2114], 40.00th=[ 5407], 50.00th=[ 9503], 60.00th=[ 12518], | 70.00th=[ 16712], 80.00th=[ 21890], 90.00th=[ 28967], 95.00th=[ 37487], | 99.00th=[ 58983], 99.50th=[ 72877], 99.90th=[162530], 99.95th=[283116], | 99.99th=[283116] bw ( KiB/s): min= 15, max= 335, per=25.30%, avg=173.31, stdev=64.21, samples=72 iops : min= 3, max= 83, avg=43.07, stdev=16.01, samples=72 lat (usec) : 100=2.44%, 250=6.80%, 500=0.36%, 750=2.77%, 1000=7.06% lat (msec) : 2=5.92%, 4=4.95%, 10=22.98%, 20=29.17%, 50=16.34% lat (msec) : 100=1.01%, 250=0.16%, 500=0.03% cpu : usr=0.16%, sys=12.74%, ctx=4908, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3030: Thu May 1 19:52:33 2025 read: IOPS=41, BW=164KiB/s (168kB/s)(6028KiB/36657msec) clat (usec): min=51, max=182710, avg=10635.61, stdev=10033.80 lat (usec): min=51, max=182710, avg=10638.79, stdev=10033.81 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 98], 10.00th=[ 114], 20.00th=[ 161], | 30.00th=[ 5866], 40.00th=[ 7701], 50.00th=[ 9241], 60.00th=[ 11076], | 70.00th=[ 13435], 80.00th=[ 16581], 90.00th=[ 22152], 95.00th=[ 27132], | 99.00th=[ 36439], 99.50th=[ 46400], 99.90th=[ 78119], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 72, max= 326, per=25.37%, avg=166.18, stdev=57.48, samples=72 iops : min= 18, max= 81, avg=41.28, stdev=14.38, samples=72 write: IOPS=42, BW=171KiB/s (175kB/s)(6260KiB/36657msec) clat (usec): min=491, max=282949, avg=13025.08, stdev=15762.89 lat (usec): min=492, max=282950, avg=13026.72, stdev=15763.03 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 750], 10.00th=[ 832], 20.00th=[ 1090], | 30.00th=[ 2114], 40.00th=[ 5080], 50.00th=[ 9634], 60.00th=[ 13173], | 70.00th=[ 16712], 80.00th=[ 22152], 90.00th=[ 29492], 95.00th=[ 37487], | 99.00th=[ 67634], 99.50th=[ 84411], 99.90th=[162530], 99.95th=[283116], | 99.99th=[283116] bw ( KiB/s): min= 16, max= 319, per=24.78%, avg=169.71, stdev=54.84, samples=73 iops : min= 4, max= 79, avg=42.16, stdev=13.72, samples=73 lat (usec) : 100=3.03%, 250=7.16%, 500=0.46%, 750=2.70%, 1000=6.74% lat (msec) : 2=5.79%, 4=4.82%, 10=22.17%, 20=28.06%, 50=17.81% lat (msec) : 100=1.11%, 250=0.13%, 500=0.03% cpu : usr=0.22%, sys=12.61%, ctx=4899, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3032: Thu May 1 19:52:33 2025 read: IOPS=41, BW=166KiB/s (170kB/s)(6052KiB/36352msec) clat (usec): min=71, max=283001, avg=10598.72, stdev=11956.41 lat (usec): min=71, max=283003, avg=10600.52, stdev=11956.31 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 99], 10.00th=[ 116], 20.00th=[ 239], | 30.00th=[ 5800], 40.00th=[ 7570], 50.00th=[ 9241], 60.00th=[ 10945], | 70.00th=[ 13173], 80.00th=[ 16057], 90.00th=[ 21627], 95.00th=[ 26346], | 99.00th=[ 40633], 99.50th=[ 47449], 99.90th=[119014], 99.95th=[283116], | 99.99th=[283116] bw ( KiB/s): min= 32, max= 352, per=25.42%, avg=166.47, stdev=58.09, samples=72 iops : min= 8, max= 88, avg=41.32, stdev=14.59, samples=72 write: IOPS=42, BW=172KiB/s (176kB/s)(6236KiB/36352msec) clat (usec): min=481, max=127452, avg=12879.22, stdev=13936.44 lat (usec): min=482, max=127453, avg=12883.00, stdev=13935.42 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 758], 10.00th=[ 816], 20.00th=[ 1020], | 30.00th=[ 2245], 40.00th=[ 6128], 50.00th=[ 9634], 60.00th=[ 13304], | 70.00th=[ 16712], 80.00th=[ 21627], 90.00th=[ 30278], 95.00th=[ 38011], | 99.00th=[ 58983], 99.50th=[ 70779], 99.90th=[125305], 99.95th=[127402], | 99.99th=[127402] bw ( KiB/s): min= 48, max= 456, per=25.03%, avg=171.47, stdev=60.92, samples=72 iops : min= 12, max= 114, avg=42.57, stdev=15.24, samples=72 lat (usec) : 100=2.60%, 250=7.36%, 500=0.68%, 750=2.41%, 1000=7.71% lat (msec) : 2=5.18%, 4=4.52%, 10=22.40%, 20=29.49%, 50=16.47% lat (msec) : 100=0.91%, 250=0.23%, 500=0.03% cpu : usr=0.20%, sys=12.65%, ctx=4794, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=655KiB/s (671kB/s), 163KiB/s-166KiB/s (167kB/s-170kB/s), io=23.4MiB (24.6MB), run=36340-36657msec WRITE: bw=686KiB/s (702kB/s), 171KiB/s-174KiB/s (175kB/s-178kB/s), io=24.6MiB (25.7MB), run=36340-36657msec rand-rw: (groupid=0, jobs=1): err= 0: pid=3031: Thu May 1 19:54:05 2025 read: IOPS=11, BW=46.7KiB/s (47.9kB/s)(5992KiB/128216msec) clat (msec): min=3, max=336, avg=11.05, stdev=13.55 lat (msec): min=3, max=336, avg=11.05, stdev=13.55 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 13], 90.00th=[ 21], 95.00th=[ 29], | 99.00th=[ 50], 99.50th=[ 64], 99.90th=[ 165], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 7, max= 152, per=25.28%, avg=47.28, stdev=27.88, samples=251 iops : min= 1, max= 38, avg=11.61, stdev= 7.02, samples=251 write: IOPS=12, BW=49.1KiB/s (50.3kB/s)(6296KiB/128216msec) clat (msec): min=25, max=243, avg=70.79, stdev=27.26 lat (msec): min=25, max=243, avg=70.79, stdev=27.26 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 48], | 30.00th=[ 55], 40.00th=[ 61], 50.00th=[ 66], 60.00th=[ 72], | 70.00th=[ 80], 80.00th=[ 89], 90.00th=[ 106], 95.00th=[ 122], | 99.00th=[ 163], 99.50th=[ 192], 99.90th=[ 218], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min= 16, max= 88, per=24.91%, avg=48.82, stdev=14.50, samples=255 iops : min= 4, max= 22, avg=11.99, stdev= 3.71, samples=255 lat (msec) : 4=1.14%, 10=32.71%, 20=9.80%, 50=16.37%, 100=33.04% lat (msec) : 250=6.90%, 500=0.03% cpu : usr=0.07%, sys=2.73%, ctx=3426, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3033: Thu May 1 19:54:05 2025 read: IOPS=11, BW=46.5KiB/s (47.6kB/s)(5940KiB/127873msec) clat (msec): min=2, max=287, avg=10.85, stdev=12.99 lat (msec): min=3, max=287, avg=10.85, stdev=12.99 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 13], 90.00th=[ 19], 95.00th=[ 29], | 99.00th=[ 56], 99.50th=[ 85], 99.90th=[ 165], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 7, max= 168, per=25.23%, avg=47.18, stdev=26.78, samples=249 iops : min= 1, max= 42, avg=11.53, stdev= 6.76, samples=249 write: IOPS=12, BW=49.6KiB/s (50.8kB/s)(6348KiB/127873msec) clat (msec): min=21, max=290, avg=70.27, stdev=27.60 lat (msec): min=21, max=290, avg=70.27, stdev=27.60 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 38], 10.00th=[ 42], 20.00th=[ 48], | 30.00th=[ 54], 40.00th=[ 60], 50.00th=[ 66], 60.00th=[ 72], | 70.00th=[ 80], 80.00th=[ 89], 90.00th=[ 105], 95.00th=[ 122], | 99.00th=[ 161], 99.50th=[ 176], 99.90th=[ 262], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 15, max= 88, per=25.15%, avg=49.29, stdev=14.73, samples=254 iops : min= 3, max= 22, avg=12.06, stdev= 3.76, samples=254 lat (msec) : 4=1.17%, 10=32.78%, 20=9.99%, 50=16.34%, 100=33.17% lat (msec) : 250=6.45%, 500=0.10% cpu : usr=0.06%, sys=2.75%, ctx=3404, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3034: Thu May 1 19:54:05 2025 read: IOPS=11, BW=47.8KiB/s (48.0kB/s)(6028KiB/126061msec) clat (msec): min=2, max=165, avg=10.95, stdev=11.18 lat (msec): min=2, max=165, avg=10.95, stdev=11.18 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 13], 90.00th=[ 20], 95.00th=[ 30], | 99.00th=[ 53], 99.50th=[ 82], 99.90th=[ 140], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 7, max= 152, per=25.73%, avg=48.12, stdev=26.70, samples=249 iops : min= 1, max= 38, avg=11.81, stdev= 6.72, samples=249 write: IOPS=12, BW=49.7KiB/s (50.8kB/s)(6260KiB/126061msec) clat (msec): min=23, max=368, avg=69.85, stdev=28.38 lat (msec): min=23, max=368, avg=69.86, stdev=28.38 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 48], | 30.00th=[ 53], 40.00th=[ 58], 50.00th=[ 65], 60.00th=[ 71], | 70.00th=[ 78], 80.00th=[ 87], 90.00th=[ 105], 95.00th=[ 125], | 99.00th=[ 159], 99.50th=[ 184], 99.90th=[ 292], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 7, max= 88, per=25.26%, avg=49.51, stdev=15.52, samples=251 iops : min= 1, max= 22, avg=12.16, stdev= 3.95, samples=251 lat (msec) : 4=1.56%, 10=32.26%, 20=10.61%, 50=16.28%, 100=32.88% lat (msec) : 250=6.35%, 500=0.07% cpu : usr=0.06%, sys=2.77%, ctx=3417, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3035: Thu May 1 19:54:05 2025 read: IOPS=11, BW=47.0KiB/s (49.1kB/s)(6052KiB/126174msec) clat (msec): min=2, max=132, avg=10.55, stdev= 9.69 lat (msec): min=2, max=132, avg=10.56, stdev= 9.69 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 12], 90.00th=[ 21], 95.00th=[ 28], | 99.00th=[ 47], 99.50th=[ 64], 99.90th=[ 125], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 7, max= 144, per=26.37%, avg=49.30, stdev=26.26, samples=244 iops : min= 1, max= 36, avg=12.13, stdev= 6.61, samples=244 write: IOPS=12, BW=49.4KiB/s (50.6kB/s)(6236KiB/126174msec) clat (msec): min=25, max=326, avg=70.54, stdev=27.26 lat (msec): min=25, max=326, avg=70.54, stdev=27.25 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 38], 10.00th=[ 43], 20.00th=[ 50], | 30.00th=[ 54], 40.00th=[ 61], 50.00th=[ 66], 60.00th=[ 72], | 70.00th=[ 80], 80.00th=[ 88], 90.00th=[ 104], 95.00th=[ 122], | 99.00th=[ 163], 99.50th=[ 186], 99.90th=[ 211], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 7, max= 88, per=25.06%, avg=49.12, stdev=15.02, samples=252 iops : min= 1, max= 22, avg=12.08, stdev= 3.87, samples=252 lat (msec) : 4=1.01%, 10=33.69%, 20=9.64%, 50=15.62%, 100=33.92% lat (msec) : 250=6.09%, 500=0.03% cpu : usr=0.04%, sys=2.80%, ctx=3417, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=187KiB/s (192kB/s), 46.5KiB/s-47.0KiB/s (47.6kB/s-49.1kB/s), io=23.4MiB (24.6MB), run=126061-128216msec WRITE: bw=196KiB/s (201kB/s), 49.1KiB/s-49.7KiB/s (50.3kB/s-50.8kB/s), io=24.6MiB (25.7MB), run=126061-128216msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=3148: Thu May 1 19:54:13 2025 read: IOPS=52, BW=844KiB/s (864kB/s)(5920KiB/7014msec) clat (usec): min=72, max=109226, avg=8627.88, stdev=10862.60 lat (usec): min=82, max=109229, avg=8629.45, stdev=10862.44 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 103], 10.00th=[ 116], 20.00th=[ 133], | 30.00th=[ 151], 40.00th=[ 1713], 50.00th=[ 7504], 60.00th=[ 9110], | 70.00th=[ 11600], 80.00th=[ 14877], 90.00th=[ 20055], 95.00th=[ 23725], | 99.00th=[ 46400], 99.50th=[ 79168], 99.90th=[109577], 99.95th=[109577], | 99.99th=[109577] bw ( KiB/s): min= 640, max= 1152, per=24.73%, avg=844.21, stdev=143.21, samples=14 iops : min= 40, max= 72, avg=52.36, stdev= 9.07, samples=14 write: IOPS=56, BW=908KiB/s (930kB/s)(6368KiB/7014msec) clat (usec): min=785, max=111960, avg=9434.69, stdev=10763.53 lat (usec): min=787, max=111961, avg=9438.28, stdev=10762.94 clat percentiles (usec): | 1.00th=[ 922], 5.00th=[ 1074], 10.00th=[ 1156], 20.00th=[ 1401], | 30.00th=[ 1942], 40.00th=[ 2900], 50.00th=[ 4817], 60.00th=[ 8848], | 70.00th=[ 11863], 80.00th=[ 17433], 90.00th=[ 23725], 95.00th=[ 28967], | 99.00th=[ 38011], 99.50th=[ 49546], 99.90th=[111674], 99.95th=[111674], | 99.99th=[111674] bw ( KiB/s): min= 576, max= 1341, per=25.22%, avg=905.79, stdev=205.85, samples=14 iops : min= 36, max= 83, avg=56.21, stdev=12.80, samples=14 lat (usec) : 100=1.95%, 250=15.49%, 500=0.91%, 750=0.39%, 1000=0.91% lat (msec) : 2=15.89%, 4=8.33%, 10=19.27%, 20=23.96%, 50=12.37% lat (msec) : 100=0.26%, 250=0.26% cpu : usr=0.36%, sys=15.74%, ctx=1080, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3149: Thu May 1 19:54:13 2025 read: IOPS=53, BW=851KiB/s (871kB/s)(5744KiB/6752msec) clat (usec): min=73, max=134175, avg=8547.13, stdev=10418.72 lat (usec): min=74, max=134176, avg=8548.66, stdev=10418.72 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 100], 10.00th=[ 111], 20.00th=[ 127], | 30.00th=[ 159], 40.00th=[ 5407], 50.00th=[ 7635], 60.00th=[ 9241], | 70.00th=[ 11731], 80.00th=[ 15139], 90.00th=[ 19006], 95.00th=[ 22938], | 99.00th=[ 31065], 99.50th=[ 43779], 99.90th=[133694], 99.95th=[133694], | 99.99th=[133694] bw ( KiB/s): min= 544, max= 1466, per=25.19%, avg=860.00, stdev=228.37, samples=13 iops : min= 34, max= 91, avg=53.46, stdev=14.18, samples=13 write: IOPS=60, BW=969KiB/s (992kB/s)(6544KiB/6752msec) clat (usec): min=829, max=66315, avg=8928.82, stdev=10374.93 lat (usec): min=830, max=66318, avg=8930.85, stdev=10374.99 clat percentiles (usec): | 1.00th=[ 938], 5.00th=[ 1123], 10.00th=[ 1188], 20.00th=[ 1369], | 30.00th=[ 1926], 40.00th=[ 2573], 50.00th=[ 4228], 60.00th=[ 6652], | 70.00th=[11207], 80.00th=[16188], 90.00th=[24249], 95.00th=[30016], | 99.00th=[46400], 99.50th=[51643], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 736, max= 1216, per=27.02%, avg=970.62, stdev=158.83, samples=13 iops : min= 46, max= 76, avg=60.38, stdev= 9.95, samples=13 lat (usec) : 100=2.60%, 250=12.76%, 500=0.91%, 750=0.26%, 1000=1.43% lat (msec) : 2=16.28%, 4=9.77%, 10=21.74%, 20=22.79%, 50=10.94% lat (msec) : 100=0.39%, 250=0.13% cpu : usr=0.31%, sys=16.15%, ctx=1157, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3150: Thu May 1 19:54:13 2025 read: IOPS=59, BW=958KiB/s (981kB/s)(6256KiB/6533msec) clat (usec): min=53, max=34445, avg=7772.00, stdev=7877.23 lat (usec): min=53, max=34447, avg=7773.28, stdev=7877.17 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 94], 10.00th=[ 109], 20.00th=[ 127], | 30.00th=[ 147], 40.00th=[ 865], 50.00th=[ 7177], 60.00th=[ 9110], | 70.00th=[11207], 80.00th=[14746], 90.00th=[18744], 95.00th=[22414], | 99.00th=[31327], 99.50th=[34341], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 672, max= 1344, per=28.08%, avg=958.69, stdev=217.03, samples=13 iops : min= 42, max= 84, avg=59.54, stdev=13.68, samples=13 write: IOPS=57, BW=923KiB/s (945kB/s)(6032KiB/6533msec) clat (usec): min=770, max=92592, avg=9113.85, stdev=10093.80 lat (usec): min=771, max=92596, avg=9118.54, stdev=10093.94 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 1074], 10.00th=[ 1188], 20.00th=[ 1500], | 30.00th=[ 2147], 40.00th=[ 3163], 50.00th=[ 5342], 60.00th=[ 8029], | 70.00th=[11600], 80.00th=[15270], 90.00th=[23200], 95.00th=[28967], | 99.00th=[38536], 99.50th=[52167], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 638, max= 1632, per=25.53%, avg=916.92, stdev=265.21, samples=13 iops : min= 39, max= 102, avg=56.92, stdev=16.74, samples=13 lat (usec) : 100=3.39%, 250=14.97%, 500=1.43%, 750=0.39%, 1000=1.95% lat (msec) : 2=12.63%, 4=7.94%, 10=21.74%, 20=25.13%, 50=10.16% lat (msec) : 100=0.26% cpu : usr=0.20%, sys=16.99%, ctx=1114, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3151: Thu May 1 19:54:13 2025 read: IOPS=54, BW=868KiB/s (889kB/s)(6032KiB/6948msec) clat (usec): min=66, max=108826, avg=8921.57, stdev=11864.06 lat (usec): min=66, max=108827, avg=8922.77, stdev=11864.06 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 96], 10.00th=[ 106], 20.00th=[ 122], | 30.00th=[ 141], 40.00th=[ 758], 50.00th=[ 7504], 60.00th=[ 9110], | 70.00th=[ 11338], 80.00th=[ 15008], 90.00th=[ 21103], 95.00th=[ 26870], | 99.00th=[ 54789], 99.50th=[ 86508], 99.90th=[108528], 99.95th=[108528], | 99.99th=[108528] bw ( KiB/s): min= 576, max= 1184, per=25.21%, avg=860.62, stdev=176.14, samples=13 iops : min= 36, max= 74, avg=53.54, stdev=11.03, samples=13 write: IOPS=56, BW=900KiB/s (922kB/s)(6256KiB/6948msec) clat (usec): min=702, max=99748, avg=8991.48, stdev=10761.18 lat (usec): min=704, max=99750, avg=8997.27, stdev=10759.72 clat percentiles (usec): | 1.00th=[ 873], 5.00th=[ 1057], 10.00th=[ 1139], 20.00th=[ 1336], | 30.00th=[ 1778], 40.00th=[ 2540], 50.00th=[ 3556], 60.00th=[ 7439], | 70.00th=[ 10814], 80.00th=[ 17171], 90.00th=[ 22414], 95.00th=[ 30802], | 99.00th=[ 43779], 99.50th=[ 48497], 99.90th=[100140], 99.95th=[100140], | 99.99th=[100140] bw ( KiB/s): min= 608, max= 1184, per=25.19%, avg=904.92, stdev=184.43, samples=13 iops : min= 38, max= 74, avg=56.31, stdev=11.66, samples=13 lat (usec) : 100=4.17%, 250=14.32%, 500=0.52%, 750=0.78%, 1000=2.21% lat (msec) : 2=16.15%, 4=9.24%, 10=18.49%, 20=21.48%, 50=11.98% lat (msec) : 100=0.52%, 250=0.13% cpu : usr=0.06%, sys=15.56%, ctx=1073, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3415KiB/s (3497kB/s), 844KiB/s-958KiB/s (864kB/s-981kB/s), io=23.4MiB (24.5MB), run=6533-7014msec WRITE: bw=3593KiB/s (3679kB/s), 900KiB/s-969KiB/s (922kB/s-992kB/s), io=24.6MiB (25.8MB), run=6533-7014msec rand-rw: (groupid=0, jobs=1): err= 0: pid=3153: Thu May 1 19:54:38 2025 read: IOPS=11, BW=191KiB/s (196kB/s)(5920KiB/30975msec) clat (usec): min=2953, max=82371, avg=11720.33, stdev=9792.32 lat (usec): min=2955, max=82372, avg=11722.43, stdev=9792.26 clat percentiles (usec): | 1.00th=[ 3916], 5.00th=[ 4752], 10.00th=[ 5276], 20.00th=[ 6063], | 30.00th=[ 6587], 40.00th=[ 7439], 50.00th=[ 8455], 60.00th=[ 9372], | 70.00th=[11207], 80.00th=[15270], 90.00th=[21890], 95.00th=[30016], | 99.00th=[53740], 99.50th=[71828], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 31, max= 479, per=24.75%, avg=189.07, stdev=97.03, samples=61 iops : min= 1, max= 29, avg=11.56, stdev= 6.04, samples=61 write: IOPS=12, BW=206KiB/s (211kB/s)(6368KiB/30975msec) clat (msec): min=29, max=162, avg=66.86, stdev=23.10 lat (msec): min=29, max=162, avg=66.86, stdev=23.10 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 39], 10.00th=[ 43], 20.00th=[ 48], | 30.00th=[ 53], 40.00th=[ 58], 50.00th=[ 64], 60.00th=[ 68], | 70.00th=[ 73], 80.00th=[ 84], 90.00th=[ 97], 95.00th=[ 109], | 99.00th=[ 148], 99.50th=[ 153], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 63, max= 320, per=25.50%, avg=204.77, stdev=55.38, samples=61 iops : min= 3, max= 20, avg=12.54, stdev= 3.63, samples=61 lat (msec) : 4=0.52%, 10=30.73%, 20=10.68%, 50=18.75%, 100=34.64% lat (msec) : 250=4.69% cpu : usr=0.05%, sys=3.22%, ctx=935, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3155: Thu May 1 19:54:38 2025 read: IOPS=11, BW=183KiB/s (188kB/s)(5744KiB/31349msec) clat (usec): min=3402, max=68536, avg=11529.12, stdev=9210.08 lat (usec): min=3404, max=68539, avg=11532.33, stdev=9210.25 clat percentiles (usec): | 1.00th=[ 3720], 5.00th=[ 4621], 10.00th=[ 5145], 20.00th=[ 5866], | 30.00th=[ 6587], 40.00th=[ 7373], 50.00th=[ 8225], 60.00th=[ 9241], | 70.00th=[11994], 80.00th=[14615], 90.00th=[21890], 95.00th=[30540], | 99.00th=[49546], 99.50th=[54264], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 31, max= 448, per=23.78%, avg=181.71, stdev=103.02, samples=62 iops : min= 1, max= 28, avg=10.97, stdev= 6.54, samples=62 write: IOPS=13, BW=209KiB/s (214kB/s)(6544KiB/31349msec) clat (msec): min=21, max=173, avg=66.46, stdev=22.24 lat (msec): min=21, max=173, avg=66.46, stdev=22.24 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 40], 10.00th=[ 43], 20.00th=[ 48], | 30.00th=[ 53], 40.00th=[ 58], 50.00th=[ 64], 60.00th=[ 69], | 70.00th=[ 74], 80.00th=[ 82], 90.00th=[ 95], 95.00th=[ 109], | 99.00th=[ 136], 99.50th=[ 146], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 95, max= 320, per=25.84%, avg=207.47, stdev=50.90, samples=62 iops : min= 5, max= 20, avg=12.58, stdev= 3.38, samples=62 lat (msec) : 4=0.91%, 10=28.52%, 20=11.72%, 50=18.75%, 100=35.81% lat (msec) : 250=4.30% cpu : usr=0.04%, sys=3.15%, ctx=907, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3156: Thu May 1 19:54:38 2025 read: IOPS=13, BW=212KiB/s (218kB/s)(6256KiB/29445msec) clat (usec): min=3065, max=82067, avg=11390.69, stdev=10553.68 lat (usec): min=3067, max=82069, avg=11394.15, stdev=10552.84 clat percentiles (usec): | 1.00th=[ 3425], 5.00th=[ 4555], 10.00th=[ 5080], 20.00th=[ 5932], | 30.00th=[ 6783], 40.00th=[ 7308], 50.00th=[ 8029], 60.00th=[ 8848], | 70.00th=[10028], 80.00th=[12649], 90.00th=[21365], 95.00th=[30802], | 99.00th=[67634], 99.50th=[71828], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 31, max= 604, per=28.11%, avg=214.74, stdev=116.39, samples=58 iops : min= 1, max= 37, avg=13.09, stdev= 7.24, samples=58 write: IOPS=12, BW=205KiB/s (210kB/s)(6032KiB/29445msec) clat (msec): min=27, max=181, avg=66.22, stdev=21.90 lat (msec): min=27, max=181, avg=66.23, stdev=21.90 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 40], 10.00th=[ 43], 20.00th=[ 48], | 30.00th=[ 53], 40.00th=[ 57], 50.00th=[ 63], 60.00th=[ 69], | 70.00th=[ 73], 80.00th=[ 83], 90.00th=[ 96], 95.00th=[ 111], | 99.00th=[ 134], 99.50th=[ 140], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 63, max= 288, per=25.37%, avg=203.72, stdev=50.45, samples=58 iops : min= 3, max= 18, avg=12.40, stdev= 3.20, samples=58 lat (msec) : 4=0.91%, 10=34.64%, 20=9.51%, 50=16.93%, 100=33.85% lat (msec) : 250=4.17% cpu : usr=0.13%, sys=3.36%, ctx=903, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3157: Thu May 1 19:54:38 2025 read: IOPS=12, BW=195KiB/s (200kB/s)(6032KiB/30893msec) clat (usec): min=3340, max=51188, avg=11456.83, stdev=8693.29 lat (usec): min=3341, max=51189, avg=11459.02, stdev=8692.95 clat percentiles (usec): | 1.00th=[ 3589], 5.00th=[ 4555], 10.00th=[ 5014], 20.00th=[ 5800], | 30.00th=[ 6587], 40.00th=[ 7504], 50.00th=[ 8455], 60.00th=[ 9503], | 70.00th=[11338], 80.00th=[15795], 90.00th=[21627], 95.00th=[29230], | 99.00th=[49546], 99.50th=[50070], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 31, max= 480, per=25.78%, avg=196.93, stdev=98.01, samples=60 iops : min= 1, max= 30, avg=11.97, stdev= 6.24, samples=60 write: IOPS=12, BW=203KiB/s (207kB/s)(6256KiB/30893msec) clat (msec): min=27, max=161, avg=67.90, stdev=22.90 lat (msec): min=27, max=161, avg=67.90, stdev=22.90 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 40], 10.00th=[ 44], 20.00th=[ 49], | 30.00th=[ 56], 40.00th=[ 60], 50.00th=[ 65], 60.00th=[ 69], | 70.00th=[ 75], 80.00th=[ 83], 90.00th=[ 96], 95.00th=[ 110], | 99.00th=[ 153], 99.50th=[ 155], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 95, max= 288, per=25.17%, avg=202.10, stdev=48.92, samples=61 iops : min= 5, max= 18, avg=12.30, stdev= 3.23, samples=61 lat (msec) : 4=0.91%, 10=30.08%, 20=11.98%, 50=17.58%, 100=35.16% lat (msec) : 250=4.30% cpu : usr=0.06%, sys=3.21%, ctx=922, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=764KiB/s (782kB/s), 183KiB/s-212KiB/s (188kB/s-218kB/s), io=23.4MiB (24.5MB), run=29445-31349msec WRITE: bw=804KiB/s (823kB/s), 203KiB/s-209KiB/s (207kB/s-214kB/s), io=24.6MiB (25.8MB), run=29445-31349msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=3192: Thu May 1 19:54:40 2025 read: IOPS=12, BW=12.8MiB/s (13.5MB/s)(7168KiB/545msec) clat (usec): min=599, max=63851, avg=31885.86, stdev=29706.80 lat (usec): min=601, max=63853, avg=31887.61, stdev=29706.92 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 603], 10.00th=[ 603], 20.00th=[ 611], | 30.00th=[ 644], 40.00th=[ 644], 50.00th=[46400], 60.00th=[53216], | 70.00th=[53216], 80.00th=[57934], 90.00th=[63701], 95.00th=[63701], | 99.00th=[63701], 99.50th=[63701], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min=10240, max=10240, per=62.60%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9394KiB/s (9620kB/s)(5120KiB/545msec) clat (msec): min=30, max=101, avg=59.05, stdev=28.06 lat (msec): min=30, max=101, avg=59.10, stdev=28.08 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 58], 60.00th=[ 58], | 70.00th=[ 69], 80.00th=[ 69], 90.00th=[ 102], 95.00th=[ 102], | 99.00th=[ 102], 99.50th=[ 102], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 6144, max= 6144, per=26.83%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (usec) : 750=25.00% lat (msec) : 50=25.00%, 100=41.67%, 250=8.33% cpu : usr=0.00%, sys=33.46%, ctx=203, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3193: Thu May 1 19:54:40 2025 read: IOPS=1, BW=1636KiB/s (1675kB/s)(2048KiB/1252msec) clat (usec): min=61361, max=61538, avg=61450.26, stdev=125.46 lat (usec): min=61364, max=61541, avg=61452.82, stdev=125.40 clat percentiles (usec): | 1.00th=[61604], 5.00th=[61604], 10.00th=[61604], 20.00th=[61604], | 30.00th=[61604], 40.00th=[61604], 50.00th=[61604], 60.00th=[61604], | 70.00th=[61604], 80.00th=[61604], 90.00th=[61604], 95.00th=[61604], | 99.00th=[61604], 99.50th=[61604], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 2048, max= 2048, per=12.52%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=7, BW=8179KiB/s (8375kB/s)(10.0MiB/1252msec) clat (msec): min=48, max=192, avg=106.61, stdev=45.07 lat (msec): min=49, max=192, avg=106.69, stdev=45.07 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 50], | 30.00th=[ 77], 40.00th=[ 87], 50.00th=[ 104], 60.00th=[ 104], | 70.00th=[ 129], 80.00th=[ 129], 90.00th=[ 155], 95.00th=[ 192], | 99.00th=[ 192], 99.50th=[ 192], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 6144, max= 8192, per=31.30%, avg=7168.00, stdev=1448.15, samples=2 iops : min= 6, max= 8, avg= 7.00, stdev= 1.41, samples=2 lat (msec) : 50=16.67%, 100=33.33%, 250=50.00% cpu : usr=0.00%, sys=22.54%, ctx=96, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3194: Thu May 1 19:54:40 2025 read: IOPS=6, BW=7119KiB/s (7290kB/s)(6144KiB/863msec) clat (usec): min=725, max=65106, avg=40919.31, stdev=21575.59 lat (usec): min=727, max=65108, avg=40920.99, stdev=21575.50 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 725], 10.00th=[ 725], 20.00th=[39584], | 30.00th=[39584], 40.00th=[43254], 50.00th=[43254], 60.00th=[47449], | 70.00th=[49546], 80.00th=[49546], 90.00th=[65274], 95.00th=[65274], | 99.00th=[65274], 99.50th=[65274], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] write: IOPS=6, BW=7119KiB/s (7290kB/s)(6144KiB/863msec) clat (msec): min=33, max=170, avg=75.33, stdev=49.27 lat (msec): min=33, max=171, avg=75.41, stdev=49.29 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 46], | 30.00th=[ 46], 40.00th=[ 59], 50.00th=[ 59], 60.00th=[ 69], | 70.00th=[ 75], 80.00th=[ 75], 90.00th=[ 171], 95.00th=[ 171], | 99.00th=[ 171], 99.50th=[ 171], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 6131, max= 6131, per=26.77%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=50.00%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=27.61%, ctx=712, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3195: Thu May 1 19:54:40 2025 read: IOPS=4, BW=4270KiB/s (4373kB/s)(5120KiB/1199msec) clat (usec): min=580, max=75475, avg=41720.38, stdev=37736.44 lat (usec): min=581, max=75477, avg=41721.80, stdev=37736.20 clat percentiles (usec): | 1.00th=[ 578], 5.00th=[ 578], 10.00th=[ 578], 20.00th=[ 578], | 30.00th=[ 660], 40.00th=[ 660], 50.00th=[64226], 60.00th=[64226], | 70.00th=[67634], 80.00th=[67634], 90.00th=[74974], 95.00th=[74974], | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 8175, max= 8175, per=49.98%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5978KiB/s (6122kB/s)(7168KiB/1199msec) clat (msec): min=49, max=202, avg=125.29, stdev=57.99 lat (msec): min=49, max=202, avg=125.37, stdev=57.96 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 72], | 30.00th=[ 93], 40.00th=[ 93], 50.00th=[ 116], 60.00th=[ 165], | 70.00th=[ 165], 80.00th=[ 180], 90.00th=[ 203], 95.00th=[ 203], | 99.00th=[ 203], 99.50th=[ 203], 99.90th=[ 203], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 4087, max= 6144, per=22.34%, avg=5115.50, stdev=1454.52, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 lat (usec) : 750=16.67% lat (msec) : 50=8.33%, 100=41.67%, 250=33.33% cpu : usr=0.00%, sys=16.44%, ctx=532, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.0MiB/s (16.8MB/s), 1636KiB/s-12.8MiB/s (1675kB/s-13.5MB/s), io=20.0MiB (20.0MB), run=545-1252msec WRITE: bw=22.4MiB/s (23.5MB/s), 5978KiB/s-9394KiB/s (6122kB/s-9620kB/s), io=28.0MiB (29.4MB), run=545-1252msec rand-rw: (groupid=0, jobs=1): err= 0: pid=3188: Thu May 1 19:54:40 2025 read: IOPS=8, BW=8483KiB/s (8686kB/s)(7168KiB/845msec) clat (msec): min=14, max=136, avg=49.08, stdev=42.64 lat (msec): min=14, max=136, avg=49.08, stdev=42.64 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 20], | 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 31], 60.00th=[ 55], | 70.00th=[ 55], 80.00th=[ 64], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min=10199, max=10199, per=65.59%, avg=10199.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=5, BW=6059KiB/s (6205kB/s)(5120KiB/845msec) clat (msec): min=69, max=134, avg=94.54, stdev=24.25 lat (msec): min=69, max=134, avg=94.63, stdev=24.25 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 70], 10.00th=[ 70], 20.00th=[ 70], | 30.00th=[ 88], 40.00th=[ 88], 50.00th=[ 90], 60.00th=[ 90], | 70.00th=[ 92], 80.00th=[ 92], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 2039, max= 2039, per=9.37%, avg=2039.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=16.67%, 100=50.00%, 250=16.67% cpu : usr=0.00%, sys=6.64%, ctx=35, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3189: Thu May 1 19:54:40 2025 read: IOPS=1, BW=1766KiB/s (1808kB/s)(2048KiB/1160msec) clat (usec): min=15207, max=32705, avg=23956.51, stdev=12372.85 lat (usec): min=15210, max=32706, avg=23958.81, stdev=12371.71 clat percentiles (usec): | 1.00th=[15270], 5.00th=[15270], 10.00th=[15270], 20.00th=[15270], | 30.00th=[15270], 40.00th=[15270], 50.00th=[15270], 60.00th=[32637], | 70.00th=[32637], 80.00th=[32637], 90.00th=[32637], 95.00th=[32637], | 99.00th=[32637], 99.50th=[32637], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 2048, max= 2048, per=13.17%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=8, BW=8828KiB/s (9039kB/s)(10.0MiB/1160msec) clat (msec): min=55, max=167, avg=108.68, stdev=34.56 lat (msec): min=55, max=167, avg=108.89, stdev=34.51 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 56], 10.00th=[ 56], 20.00th=[ 79], | 30.00th=[ 84], 40.00th=[ 102], 50.00th=[ 105], 60.00th=[ 105], | 70.00th=[ 126], 80.00th=[ 126], 90.00th=[ 159], 95.00th=[ 167], | 99.00th=[ 167], 99.50th=[ 167], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 6144, max=10240, per=37.63%, avg=8192.00, stdev=2896.31, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (msec) : 20=8.33%, 50=8.33%, 100=25.00%, 250=58.33% cpu : usr=0.00%, sys=6.13%, ctx=33, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3190: Thu May 1 19:54:40 2025 read: IOPS=4, BW=4665KiB/s (4777kB/s)(6144KiB/1317msec) clat (msec): min=21, max=244, avg=85.94, stdev=83.85 lat (msec): min=21, max=244, avg=85.94, stdev=83.85 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 44], 50.00th=[ 44], 60.00th=[ 56], | 70.00th=[ 115], 80.00th=[ 115], 90.00th=[ 245], 95.00th=[ 245], | 99.00th=[ 245], 99.50th=[ 245], 99.90th=[ 245], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 8175, max= 8175, per=52.57%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4665KiB/s (4777kB/s)(6144KiB/1317msec) clat (msec): min=56, max=183, avg=103.16, stdev=46.01 lat (msec): min=56, max=183, avg=103.30, stdev=45.97 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 78], | 30.00th=[ 78], 40.00th=[ 80], 50.00th=[ 80], 60.00th=[ 95], | 70.00th=[ 129], 80.00th=[ 129], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 2043, max= 6144, per=18.80%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (msec) : 50=25.00%, 100=41.67%, 250=33.33% cpu : usr=0.08%, sys=4.26%, ctx=38, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3191: Thu May 1 19:54:40 2025 read: IOPS=3, BW=3948KiB/s (4042kB/s)(5120KiB/1297msec) clat (msec): min=15, max=180, avg=80.96, stdev=71.79 lat (msec): min=15, max=180, avg=80.96, stdev=71.79 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 66], 60.00th=[ 66], | 70.00th=[ 126], 80.00th=[ 126], 90.00th=[ 182], 95.00th=[ 182], | 99.00th=[ 182], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 8192, max= 8192, per=52.68%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5527KiB/s (5659kB/s)(7168KiB/1297msec) clat (msec): min=74, max=158, avg=118.22, stdev=33.74 lat (msec): min=74, max=158, avg=118.31, stdev=33.74 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 75], 10.00th=[ 75], 20.00th=[ 82], | 30.00th=[ 99], 40.00th=[ 99], 50.00th=[ 130], 60.00th=[ 136], | 70.00th=[ 136], 80.00th=[ 153], 90.00th=[ 159], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 4096, max= 6144, per=23.52%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 20=16.67%, 100=33.33%, 250=50.00% cpu : usr=0.00%, sys=5.55%, ctx=158, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.2MiB/s (15.9MB/s), 1766KiB/s-8483KiB/s (1808kB/s-8686kB/s), io=20.0MiB (20.0MB), run=845-1317msec WRITE: bw=21.3MiB/s (22.3MB/s), 4665KiB/s-8828KiB/s (4777kB/s-9039kB/s), io=28.0MiB (29.4MB), run=845-1317msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=3202: Thu May 1 19:54:42 2025 read: IOPS=6, BW=25.4MiB/s (26.7MB/s)(12.0MiB/472msec) clat (usec): min=1990, max=283758, avg=140232.73, stdev=140958.23 lat (usec): min=1992, max=283759, avg=140234.51, stdev=140957.63 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 1991], 10.00th=[ 1991], 20.00th=[ 1991], | 30.00th=[ 1991], 40.00th=[135267], 50.00th=[135267], 60.00th=[135267], | 70.00th=[283116], 80.00th=[283116], 90.00th=[283116], 95.00th=[283116], | 99.00th=[283116], 99.50th=[283116], 99.90th=[283116], 99.95th=[283116], | 99.99th=[283116] lat (msec) : 2=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=26.54%, ctx=1058, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3203: Thu May 1 19:54:42 2025 read: IOPS=1, BW=4262KiB/s (4365kB/s)(4096KiB/961msec) clat (nsec): min=171505k, max=171505k, avg=171505105.00, stdev= 0.00 lat (nsec): min=171508k, max=171508k, avg=171508361.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 171], 50.00th=[ 171], 60.00th=[ 171], | 70.00th=[ 171], 80.00th=[ 171], 90.00th=[ 171], 95.00th=[ 171], | 99.00th=[ 171], 99.50th=[ 171], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 8159, max= 8159, per=47.86%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=8524KiB/s (8729kB/s)(8192KiB/961msec) clat (msec): min=271, max=445, avg=358.41, stdev=122.46 lat (msec): min=272, max=445, avg=358.80, stdev=122.48 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 271], 10.00th=[ 271], 20.00th=[ 271], | 30.00th=[ 271], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 447], | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=19.38%, ctx=494, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3205: Thu May 1 19:54:42 2025 write: IOPS=4, BW=17.7MiB/s (18.6MB/s)(12.0MiB/677msec) clat (msec): min=101, max=275, avg=207.55, stdev=93.18 lat (msec): min=101, max=275, avg=207.93, stdev=93.27 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 102], 10.00th=[ 102], 20.00th=[ 102], | 30.00th=[ 102], 40.00th=[ 247], 50.00th=[ 247], 60.00th=[ 247], | 70.00th=[ 275], 80.00th=[ 275], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 8175, max= 8175, per=23.98%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.15%, sys=38.46%, ctx=77, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3207: Thu May 1 19:54:42 2025 write: IOPS=3, BW=13.3MiB/s (13.9MB/s)(12.0MiB/905msec) clat (msec): min=234, max=323, avg=279.10, stdev=44.33 lat (msec): min=235, max=323, avg=279.55, stdev=44.33 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 234], 10.00th=[ 234], 20.00th=[ 234], | 30.00th=[ 234], 40.00th=[ 279], 50.00th=[ 279], 60.00th=[ 279], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 8192, max= 8192, per=24.03%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=30.86%, ctx=73, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.6MiB/s (17.5MB/s), 4262KiB/s-25.4MiB/s (4365kB/s-26.7MB/s), io=16.0MiB (16.8MB), run=472-961msec WRITE: bw=33.3MiB/s (34.9MB/s), 8524KiB/s-17.7MiB/s (8729kB/s-18.6MB/s), io=32.0MiB (33.6MB), run=677-961msec rand-rw: (groupid=0, jobs=1): err= 0: pid=3204: Thu May 1 19:54:43 2025 read: IOPS=3, BW=14.9MiB/s (15.7MB/s)(12.0MiB/803msec) clat (msec): min=54, max=356, avg=252.63, stdev=171.47 lat (msec): min=54, max=356, avg=252.63, stdev=171.47 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 55], 10.00th=[ 55], 20.00th=[ 55], | 30.00th=[ 55], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 347], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 8192, max= 8192, per=53.40%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.12%, sys=2.49%, ctx=22, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3206: Thu May 1 19:54:43 2025 read: IOPS=0, BW=3835KiB/s (3927kB/s)(4096KiB/1068msec) clat (nsec): min=517845k, max=517845k, avg=517844907.00, stdev= 0.00 lat (nsec): min=517847k, max=517847k, avg=517847289.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 518], 5.00th=[ 518], 10.00th=[ 518], 20.00th=[ 518], | 30.00th=[ 518], 40.00th=[ 518], 50.00th=[ 518], 60.00th=[ 518], | 70.00th=[ 518], 80.00th=[ 518], 90.00th=[ 518], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] write: IOPS=1, BW=7670KiB/s (7855kB/s)(8192KiB/1068msec) clat (msec): min=155, max=336, avg=245.98, stdev=127.55 lat (msec): min=156, max=336, avg=246.34, stdev=127.56 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 157], 50.00th=[ 157], 60.00th=[ 338], | 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.19%, sys=3.75%, ctx=21, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3208: Thu May 1 19:54:43 2025 write: IOPS=2, BW=9555KiB/s (9785kB/s)(12.0MiB/1286msec) clat (msec): min=105, max=628, avg=404.95, stdev=269.34 lat (msec): min=106, max=628, avg=405.32, stdev=269.29 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 106], 10.00th=[ 106], 20.00th=[ 106], | 30.00th=[ 106], 40.00th=[ 481], 50.00th=[ 481], 60.00th=[ 481], | 70.00th=[ 625], 80.00th=[ 625], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min=11570, max=11570, per=45.41%, avg=11570.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.23%, sys=3.35%, ctx=25, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3209: Thu May 1 19:54:43 2025 write: IOPS=3, BW=15.0MiB/s (16.8MB/s)(12.0MiB/751msec) clat (msec): min=92, max=505, avg=240.13, stdev=230.48 lat (msec): min=93, max=506, avg=240.65, stdev=230.45 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 93], 10.00th=[ 93], 20.00th=[ 93], | 30.00th=[ 93], 40.00th=[ 123], 50.00th=[ 123], 60.00th=[ 123], | 70.00th=[ 506], 80.00th=[ 506], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min=16384, max=16384, per=64.30%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 750=33.33% cpu : usr=0.27%, sys=3.47%, ctx=20, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=14.0MiB/s (15.7MB/s), 3835KiB/s-14.9MiB/s (3927kB/s-15.7MB/s), io=16.0MiB (16.8MB), run=803-1068msec WRITE: bw=24.9MiB/s (26.1MB/s), 7670KiB/s-15.0MiB/s (7855kB/s-16.8MB/s), io=32.0MiB (33.6MB), run=751-1286msec