== sanity test 398b: DIO and buffer IO race ============== 04:07:21 (1752480441) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB, 48 MiB) copied, 5.07995 s, 9.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=157564: Mon Jul 14 04:09:15 2025 read: IOPS=15, BW=62.1KiB/s (63.6kB/s)(5992KiB/96416msec) clat (usec): min=90, max=831051, avg=32242.40, stdev=31092.51 lat (usec): min=90, max=831053, avg=32248.18, stdev=31093.73 clat percentiles (usec): | 1.00th=[ 141], 5.00th=[ 204], 10.00th=[ 545], 20.00th=[ 15664], | 30.00th=[ 20055], 40.00th=[ 23200], 50.00th=[ 27395], 60.00th=[ 32113], | 70.00th=[ 38536], 80.00th=[ 47449], 90.00th=[ 59507], 95.00th=[ 73925], | 99.00th=[109577], 99.50th=[149947], 99.90th=[219153], 99.95th=[834667], | 99.99th=[834667] bw ( KiB/s): min= 14, max= 127, per=25.40%, avg=62.22, stdev=21.64, samples=188 iops : min= 3, max= 31, avg=14.94, stdev= 5.44, samples=188 write: IOPS=16, BW=65.3KiB/s (66.9kB/s)(6296KiB/96416msec); 0 zone resets clat (usec): min=990, max=404772, avg=30297.39, stdev=31606.25 lat (usec): min=991, max=404775, avg=30300.64, stdev=31605.02 clat percentiles (usec): | 1.00th=[ 1106], 5.00th=[ 1287], 10.00th=[ 1516], 20.00th=[ 3032], | 30.00th=[ 6325], 40.00th=[ 13173], 50.00th=[ 22676], 60.00th=[ 31327], | 70.00th=[ 40633], 80.00th=[ 53216], 90.00th=[ 68682], 95.00th=[ 86508], | 99.00th=[126354], 99.50th=[145753], 99.90th=[283116], 99.95th=[404751], | 99.99th=[404751] bw ( KiB/s): min= 15, max= 127, per=25.48%, avg=65.48, stdev=20.92, samples=188 iops : min= 3, max= 31, avg=15.76, stdev= 5.27, samples=188 lat (usec) : 100=0.07%, 250=3.35%, 500=1.33%, 750=0.23%, 1000=0.10% lat (msec) : 2=7.55%, 4=5.11%, 10=6.32%, 20=14.49%, 50=41.47% lat (msec) : 100=17.94%, 250=1.95%, 500=0.07%, 1000=0.03% cpu : usr=0.13%, sys=8.73%, ctx=6063, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157566: Mon Jul 14 04:09:15 2025 read: IOPS=15, BW=61.3KiB/s (62.8kB/s)(5940KiB/96904msec) clat (usec): min=92, max=302402, avg=31091.94, stdev=23510.93 lat (usec): min=92, max=302403, avg=31095.55, stdev=23511.58 clat percentiles (usec): | 1.00th=[ 130], 5.00th=[ 192], 10.00th=[ 310], 20.00th=[ 15139], | 30.00th=[ 19792], 40.00th=[ 23200], 50.00th=[ 27132], 60.00th=[ 32900], | 70.00th=[ 38536], 80.00th=[ 46400], 90.00th=[ 57934], 95.00th=[ 69731], | 99.00th=[103285], 99.50th=[125305], 99.90th=[189793], 99.95th=[304088], | 99.99th=[304088] bw ( KiB/s): min= 7, max= 119, per=24.89%, avg=60.98, stdev=23.37, samples=190 iops : min= 1, max= 29, avg=14.62, stdev= 5.88, samples=190 write: IOPS=16, BW=65.5KiB/s (67.1kB/s)(6348KiB/96904msec); 0 zone resets clat (usec): min=975, max=945984, avg=31708.26, stdev=38388.61 lat (usec): min=975, max=945985, avg=31710.21, stdev=38388.68 clat percentiles (usec): | 1.00th=[ 1074], 5.00th=[ 1270], 10.00th=[ 1549], 20.00th=[ 3720], | 30.00th=[ 8160], 40.00th=[ 17171], 50.00th=[ 23987], 60.00th=[ 32113], | 70.00th=[ 40633], 80.00th=[ 52167], 90.00th=[ 69731], 95.00th=[ 90702], | 99.00th=[131597], 99.50th=[181404], 99.90th=[248513], 99.95th=[943719], | 99.99th=[943719] bw ( KiB/s): min= 7, max= 127, per=25.30%, avg=65.03, stdev=22.28, samples=190 iops : min= 1, max= 31, avg=15.65, stdev= 5.62, samples=190 lat (usec) : 100=0.10%, 250=3.97%, 500=1.69%, 750=0.13%, 1000=0.23% lat (msec) : 2=7.13%, 4=4.26%, 10=6.61%, 20=13.64%, 50=42.74% lat (msec) : 100=17.22%, 250=2.21%, 500=0.03%, 1000=0.03% cpu : usr=0.11%, sys=8.72%, ctx=6069, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157568: Mon Jul 14 04:09:15 2025 read: IOPS=15, BW=62.9KiB/s (64.4kB/s)(6028KiB/95815msec) clat (usec): min=106, max=227626, avg=29771.82, stdev=21662.61 lat (usec): min=106, max=227628, avg=29773.67, stdev=21662.57 clat percentiles (usec): | 1.00th=[ 133], 5.00th=[ 180], 10.00th=[ 265], 20.00th=[ 14746], | 30.00th=[ 19268], 40.00th=[ 22676], 50.00th=[ 26346], 60.00th=[ 30802], | 70.00th=[ 36963], 80.00th=[ 44827], 90.00th=[ 55313], 95.00th=[ 66847], | 99.00th=[ 95945], 99.50th=[116917], 99.90th=[170918], 99.95th=[227541], | 99.99th=[227541] bw ( KiB/s): min= 15, max= 151, per=25.87%, avg=63.39, stdev=23.72, samples=186 iops : min= 3, max= 37, avg=15.24, stdev= 5.98, samples=186 write: IOPS=16, BW=65.3KiB/s (66.9kB/s)(6260KiB/95815msec); 0 zone resets clat (usec): min=986, max=1252.5k, avg=32270.07, stdev=44347.02 lat (usec): min=987, max=1252.5k, avg=32272.87, stdev=44346.40 clat percentiles (usec): | 1.00th=[ 1106], 5.00th=[ 1319], 10.00th=[ 1565], | 20.00th=[ 3851], 30.00th=[ 8094], 40.00th=[ 17433], | 50.00th=[ 25035], 60.00th=[ 31065], 70.00th=[ 40109], | 80.00th=[ 52691], 90.00th=[ 70779], 95.00th=[ 88605], | 99.00th=[ 139461], 99.50th=[ 168821], 99.90th=[ 278922], | 99.95th=[1249903], 99.99th=[1249903] bw ( KiB/s): min= 7, max= 134, per=25.45%, avg=65.40, stdev=23.55, samples=187 iops : min= 1, max= 33, avg=15.73, stdev= 5.95, samples=187 lat (usec) : 250=4.65%, 500=1.33%, 750=0.20%, 1000=0.07% lat (msec) : 2=6.97%, 4=3.87%, 10=6.67%, 20=13.48%, 50=44.21% lat (msec) : 100=16.44%, 250=1.95%, 500=0.13%, 2000=0.03% cpu : usr=0.12%, sys=8.56%, ctx=5968, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157570: Mon Jul 14 04:09:15 2025 read: IOPS=15, BW=61.9KiB/s (63.4kB/s)(6052KiB/97802msec) clat (usec): min=100, max=238791, avg=30953.98, stdev=22933.63 lat (usec): min=100, max=238793, avg=30956.17, stdev=22933.30 clat percentiles (usec): | 1.00th=[ 133], 5.00th=[ 186], 10.00th=[ 293], 20.00th=[ 15401], | 30.00th=[ 19530], 40.00th=[ 23462], 50.00th=[ 27132], 60.00th=[ 31327], | 70.00th=[ 38536], 80.00th=[ 46400], 90.00th=[ 58459], 95.00th=[ 70779], | 99.00th=[ 96994], 99.50th=[122160], 99.90th=[235930], 99.95th=[238027], | 99.99th=[238027] bw ( KiB/s): min= 7, max= 142, per=25.53%, avg=62.55, stdev=23.20, samples=189 iops : min= 1, max= 35, avg=15.02, stdev= 5.86, samples=189 write: IOPS=15, BW=63.8KiB/s (65.3kB/s)(6236KiB/97802msec); 0 zone resets clat (usec): min=917, max=1110.1k, avg=32382.17, stdev=42648.33 lat (usec): min=919, max=1110.1k, avg=32384.26, stdev=42648.42 clat percentiles (usec): | 1.00th=[ 1074], 5.00th=[ 1287], 10.00th=[ 1565], | 20.00th=[ 3654], 30.00th=[ 7570], 40.00th=[ 16450], | 50.00th=[ 24511], 60.00th=[ 32375], 70.00th=[ 42206], | 80.00th=[ 53740], 90.00th=[ 71828], 95.00th=[ 85459], | 99.00th=[ 141558], 99.50th=[ 187696], 99.90th=[ 425722], | 99.95th=[1115685], 99.99th=[1115685] bw ( KiB/s): min= 7, max= 135, per=24.80%, avg=63.75, stdev=22.88, samples=191 iops : min= 1, max= 33, avg=15.33, stdev= 5.75, samples=191 lat (usec) : 250=4.23%, 500=1.66%, 750=0.23%, 1000=0.29% lat (msec) : 2=6.80%, 4=4.00%, 10=6.41%, 20=13.87%, 50=42.68% lat (msec) : 100=18.00%, 250=1.73%, 500=0.07%, 2000=0.03% cpu : usr=0.12%, sys=8.73%, ctx=6031, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=246KiB/s (251kB/s), 61.3KiB/s-62.9KiB/s (62.8kB/s-64.4kB/s), io=23.4MiB (24.6MB), run=95815-97802msec WRITE: bw=257KiB/s (263kB/s), 63.8KiB/s-65.5KiB/s (65.3kB/s-67.1kB/s), io=24.6MiB (25.7MB), run=95815-97802msec rand-rw: (groupid=0, jobs=1): err= 0: pid=157563: Mon Jul 14 04:11:36 2025 read: IOPS=6, BW=25.3KiB/s (25.9kB/s)(5992KiB/236950msec) clat (msec): min=5, max=848, avg=31.31, stdev=32.01 lat (msec): min=5, max=848, avg=31.31, stdev=32.01 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 19], 50.00th=[ 23], 60.00th=[ 28], | 70.00th=[ 37], 80.00th=[ 47], 90.00th=[ 61], 95.00th=[ 75], | 99.00th=[ 123], 99.50th=[ 136], 99.90th=[ 275], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 7, max= 96, per=27.43%, avg=27.43, stdev=17.12, samples=423 iops : min= 1, max= 24, avg= 6.21, stdev= 4.32, samples=423 write: IOPS=6, BW=26.6KiB/s (27.2kB/s)(6296KiB/236950msec); 0 zone resets clat (msec): min=37, max=489, avg=120.45, stdev=47.36 lat (msec): min=37, max=489, avg=120.45, stdev=47.36 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 65], 10.00th=[ 72], 20.00th=[ 85], | 30.00th=[ 93], 40.00th=[ 103], 50.00th=[ 112], 60.00th=[ 122], | 70.00th=[ 134], 80.00th=[ 148], 90.00th=[ 178], 95.00th=[ 207], | 99.00th=[ 288], 99.50th=[ 334], 99.90th=[ 405], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 7, max= 56, per=24.90%, avg=26.14, stdev= 9.03, samples=465 iops : min= 1, max= 14, avg= 5.89, stdev= 2.30, samples=465 lat (msec) : 10=2.93%, 20=18.59%, 50=19.30%, 100=26.37%, 250=31.48% lat (msec) : 500=1.30%, 1000=0.03% cpu : usr=0.06%, sys=1.85%, ctx=3943, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157565: Mon Jul 14 04:11:36 2025 read: IOPS=6, BW=24.9KiB/s (25.5kB/s)(5940KiB/238445msec) clat (msec): min=5, max=985, avg=31.22, stdev=33.72 lat (msec): min=5, max=985, avg=31.22, stdev=33.72 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 27], | 70.00th=[ 35], 80.00th=[ 46], 90.00th=[ 61], 95.00th=[ 74], | 99.00th=[ 120], 99.50th=[ 129], 99.90th=[ 226], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 7, max= 96, per=26.89%, avg=26.89, stdev=16.22, samples=427 iops : min= 1, max= 24, avg= 6.08, stdev= 4.11, samples=427 write: IOPS=6, BW=26.6KiB/s (27.3kB/s)(6348KiB/238445msec); 0 zone resets clat (msec): min=43, max=493, avg=120.78, stdev=47.56 lat (msec): min=43, max=493, avg=120.78, stdev=47.56 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 65], 10.00th=[ 72], 20.00th=[ 85], | 30.00th=[ 93], 40.00th=[ 103], 50.00th=[ 112], 60.00th=[ 122], | 70.00th=[ 133], 80.00th=[ 150], 90.00th=[ 182], 95.00th=[ 209], | 99.00th=[ 284], 99.50th=[ 313], 99.90th=[ 443], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 7, max= 55, per=24.96%, avg=26.21, stdev= 9.21, samples=467 iops : min= 1, max= 13, avg= 5.91, stdev= 2.34, samples=467 lat (msec) : 10=2.02%, 20=18.62%, 50=20.15%, 100=26.27%, 250=31.80% lat (msec) : 500=1.11%, 1000=0.03% cpu : usr=0.06%, sys=1.79%, ctx=3893, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157567: Mon Jul 14 04:11:36 2025 read: IOPS=6, BW=25.2KiB/s (25.8kB/s)(6028KiB/238880msec) clat (msec): min=5, max=368, avg=31.23, stdev=24.41 lat (msec): min=5, max=368, avg=31.23, stdev=24.41 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 29], | 70.00th=[ 36], 80.00th=[ 47], 90.00th=[ 64], 95.00th=[ 75], | 99.00th=[ 111], 99.50th=[ 140], 99.90th=[ 161], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 7, max= 79, per=26.61%, avg=26.61, stdev=16.96, samples=438 iops : min= 1, max= 19, avg= 6.01, stdev= 4.27, samples=438 write: IOPS=6, BW=26.2KiB/s (26.8kB/s)(6260KiB/238880msec); 0 zone resets clat (msec): min=40, max=1242, avg=122.31, stdev=55.19 lat (msec): min=40, max=1242, avg=122.31, stdev=55.19 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 66], 10.00th=[ 73], 20.00th=[ 85], | 30.00th=[ 94], 40.00th=[ 103], 50.00th=[ 112], 60.00th=[ 123], | 70.00th=[ 136], 80.00th=[ 153], 90.00th=[ 182], 95.00th=[ 211], | 99.00th=[ 288], 99.50th=[ 313], 99.90th=[ 502], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 7, max= 55, per=24.57%, avg=25.80, stdev= 9.64, samples=468 iops : min= 1, max= 13, avg= 5.81, stdev= 2.45, samples=468 lat (msec) : 10=2.83%, 20=18.42%, 50=19.27%, 100=26.76%, 250=31.87% lat (msec) : 500=0.78%, 750=0.03%, 2000=0.03% cpu : usr=0.06%, sys=1.82%, ctx=4020, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157569: Mon Jul 14 04:11:36 2025 read: IOPS=6, BW=25.5KiB/s (26.1kB/s)(6052KiB/237110msec) clat (msec): min=6, max=367, avg=30.55, stdev=26.20 lat (msec): min=6, max=367, avg=30.55, stdev=26.20 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 22], 60.00th=[ 27], | 70.00th=[ 34], 80.00th=[ 45], 90.00th=[ 61], 95.00th=[ 78], | 99.00th=[ 131], 99.50th=[ 157], 99.90th=[ 326], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 7, max= 118, per=27.19%, avg=27.19, stdev=17.56, samples=431 iops : min= 1, max= 29, avg= 6.16, stdev= 4.42, samples=431 write: IOPS=6, BW=26.3KiB/s (26.9kB/s)(6236KiB/237110msec); 0 zone resets clat (msec): min=34, max=1096, avg=122.19, stdev=52.44 lat (msec): min=34, max=1096, avg=122.19, stdev=52.44 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 66], 10.00th=[ 74], 20.00th=[ 85], | 30.00th=[ 94], 40.00th=[ 104], 50.00th=[ 113], 60.00th=[ 124], | 70.00th=[ 138], 80.00th=[ 153], 90.00th=[ 182], 95.00th=[ 209], | 99.00th=[ 279], 99.50th=[ 317], 99.90th=[ 393], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 7, max= 55, per=24.64%, avg=25.88, stdev= 9.45, samples=466 iops : min= 1, max= 13, avg= 5.82, stdev= 2.41, samples=466 lat (msec) : 10=2.93%, 20=19.56%, 50=19.50%, 100=24.84%, 250=32.13% lat (msec) : 500=1.01%, 2000=0.03% cpu : usr=0.06%, sys=1.84%, ctx=3959, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=101KiB/s (103kB/s), 24.9KiB/s-25.5KiB/s (25.5kB/s-26.1kB/s), io=23.4MiB (24.6MB), run=236950-238880msec WRITE: bw=105KiB/s (108kB/s), 26.2KiB/s-26.6KiB/s (26.8kB/s-27.3kB/s), io=24.6MiB (25.7MB), run=236950-238880msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=157621: Mon Jul 14 04:12:05 2025 read: IOPS=18, BW=290KiB/s (297kB/s)(5920KiB/20421msec) clat (usec): min=84, max=116096, avg=24705.63, stdev=20976.61 lat (usec): min=85, max=116098, avg=24706.90, stdev=20976.61 clat percentiles (usec): | 1.00th=[ 108], 5.00th=[ 151], 10.00th=[ 169], 20.00th=[ 285], | 30.00th=[ 12780], 40.00th=[ 18220], 50.00th=[ 22676], 60.00th=[ 27919], | 70.00th=[ 33817], 80.00th=[ 41157], 90.00th=[ 52691], 95.00th=[ 61604], | 99.00th=[ 88605], 99.50th=[104334], 99.90th=[115868], 99.95th=[115868], | 99.99th=[115868] bw ( KiB/s): min= 31, max= 536, per=25.14%, avg=294.69, stdev=120.81, samples=39 iops : min= 1, max= 33, avg=17.74, stdev= 7.64, samples=39 write: IOPS=19, BW=312KiB/s (319kB/s)(6368KiB/20421msec); 0 zone resets clat (usec): min=1207, max=350937, avg=27571.81, stdev=37022.31 lat (usec): min=1208, max=350939, avg=27574.27, stdev=37022.40 clat percentiles (usec): | 1.00th=[ 1336], 5.00th=[ 1614], 10.00th=[ 1860], 20.00th=[ 3458], | 30.00th=[ 5866], 40.00th=[ 9372], 50.00th=[ 14353], 60.00th=[ 22676], | 70.00th=[ 32900], 80.00th=[ 46924], 90.00th=[ 64226], 95.00th=[ 82314], | 99.00th=[212861], 99.50th=[287310], 99.90th=[350225], 99.95th=[350225], | 99.99th=[350225] bw ( KiB/s): min= 31, max= 568, per=25.77%, avg=318.05, stdev=112.26, samples=39 iops : min= 1, max= 35, avg=19.23, stdev= 7.06, samples=39 lat (usec) : 100=0.26%, 250=8.59%, 500=2.34%, 750=0.39%, 1000=0.13% lat (msec) : 2=6.38%, 4=6.64%, 10=10.03%, 20=15.10%, 50=35.42% lat (msec) : 100=12.50%, 250=1.95%, 500=0.26% cpu : usr=0.15%, sys=9.77%, ctx=1312, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157622: Mon Jul 14 04:12:05 2025 read: IOPS=18, BW=304KiB/s (311kB/s)(5744KiB/18905msec) clat (usec): min=101, max=162478, avg=26238.42, stdev=23209.65 lat (usec): min=102, max=162479, avg=26239.64, stdev=23209.66 clat percentiles (usec): | 1.00th=[ 117], 5.00th=[ 159], 10.00th=[ 192], 20.00th=[ 367], | 30.00th=[ 15008], 40.00th=[ 20055], 50.00th=[ 23462], 60.00th=[ 28705], | 70.00th=[ 33424], 80.00th=[ 40633], 90.00th=[ 54264], 95.00th=[ 63701], | 99.00th=[108528], 99.50th=[152044], 99.90th=[162530], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 128, max= 478, per=25.88%, avg=303.27, stdev=96.28, samples=37 iops : min= 8, max= 29, avg=18.30, stdev= 5.99, samples=37 write: IOPS=21, BW=346KiB/s (354kB/s)(6544KiB/18905msec); 0 zone resets clat (usec): min=1305, max=122654, avg=22692.37, stdev=23618.81 lat (usec): min=1306, max=122657, avg=22699.22, stdev=23615.67 clat percentiles (usec): | 1.00th=[ 1401], 5.00th=[ 1663], 10.00th=[ 1893], 20.00th=[ 2802], | 30.00th=[ 5669], 40.00th=[ 8586], 50.00th=[ 14091], 60.00th=[ 19792], | 70.00th=[ 29754], 80.00th=[ 39060], 90.00th=[ 58459], 95.00th=[ 68682], | 99.00th=[ 96994], 99.50th=[109577], 99.90th=[122160], 99.95th=[122160], | 99.99th=[122160] bw ( KiB/s): min= 91, max= 510, per=28.11%, avg=346.89, stdev=95.75, samples=37 iops : min= 5, max= 31, avg=21.03, stdev= 6.04, samples=37 lat (usec) : 250=7.81%, 500=2.47%, 750=0.52%, 1000=0.13% lat (msec) : 2=7.29%, 4=5.86%, 10=10.03%, 20=16.54%, 50=35.29% lat (msec) : 100=12.76%, 250=1.30% cpu : usr=0.10%, sys=10.79%, ctx=1393, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157624: Mon Jul 14 04:12:05 2025 read: IOPS=19, BW=319KiB/s (326kB/s)(6256KiB/19631msec) clat (usec): min=104, max=137686, avg=25523.93, stdev=22164.48 lat (usec): min=104, max=137688, avg=25529.63, stdev=22170.71 clat percentiles (usec): | 1.00th=[ 114], 5.00th=[ 149], 10.00th=[ 180], 20.00th=[ 359], | 30.00th=[ 13698], 40.00th=[ 19006], 50.00th=[ 23987], 60.00th=[ 28967], | 70.00th=[ 34341], 80.00th=[ 41157], 90.00th=[ 53216], 95.00th=[ 64226], | 99.00th=[106431], 99.50th=[120062], 99.90th=[137364], 99.95th=[137364], | 99.99th=[137364] bw ( KiB/s): min= 127, max= 638, per=27.62%, avg=323.76, stdev=98.77, samples=38 iops : min= 7, max= 39, avg=19.58, stdev= 6.20, samples=38 write: IOPS=19, BW=307KiB/s (315kB/s)(6032KiB/19631msec); 0 zone resets clat (usec): min=1310, max=152800, avg=25467.66, stdev=26040.84 lat (usec): min=1310, max=152804, avg=25471.22, stdev=26041.88 clat percentiles (usec): | 1.00th=[ 1467], 5.00th=[ 1631], 10.00th=[ 1795], 20.00th=[ 3589], | 30.00th=[ 5800], 40.00th=[ 10159], 50.00th=[ 16909], 60.00th=[ 25297], | 70.00th=[ 33817], 80.00th=[ 44827], 90.00th=[ 62129], 95.00th=[ 76022], | 99.00th=[127402], 99.50th=[145753], 99.90th=[152044], 99.95th=[152044], | 99.99th=[152044] bw ( KiB/s): min= 125, max= 504, per=24.93%, avg=307.63, stdev=87.46, samples=38 iops : min= 7, max= 31, avg=18.55, stdev= 5.56, samples=38 lat (usec) : 250=7.94%, 500=3.65%, 750=0.78%, 1000=0.13% lat (msec) : 2=6.51%, 4=5.86%, 10=8.72%, 20=14.58%, 50=37.37% lat (msec) : 100=13.28%, 250=1.17% cpu : usr=0.13%, sys=10.17%, ctx=1386, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157625: Mon Jul 14 04:12:05 2025 read: IOPS=18, BW=302KiB/s (309kB/s)(6032KiB/19997msec) clat (usec): min=93, max=152553, avg=26424.22, stdev=24878.14 lat (usec): min=94, max=152555, avg=26425.50, stdev=24878.11 clat percentiles (usec): | 1.00th=[ 113], 5.00th=[ 149], 10.00th=[ 159], 20.00th=[ 235], | 30.00th=[ 8455], 40.00th=[ 19530], 50.00th=[ 23987], 60.00th=[ 29754], | 70.00th=[ 34866], 80.00th=[ 43254], 90.00th=[ 56361], 95.00th=[ 67634], | 99.00th=[111674], 99.50th=[147850], 99.90th=[152044], 99.95th=[152044], | 99.99th=[152044] bw ( KiB/s): min= 31, max= 415, per=25.54%, avg=299.28, stdev=84.85, samples=39 iops : min= 1, max= 25, avg=18.03, stdev= 5.37, samples=39 write: IOPS=19, BW=313KiB/s (320kB/s)(6256KiB/19997msec); 0 zone resets clat (usec): min=1241, max=139785, avg=25213.56, stdev=25803.89 lat (usec): min=1243, max=139787, avg=25217.74, stdev=25808.61 clat percentiles (usec): | 1.00th=[ 1369], 5.00th=[ 1631], 10.00th=[ 1844], 20.00th=[ 3490], | 30.00th=[ 5932], 40.00th=[ 9503], 50.00th=[ 15795], 60.00th=[ 25035], | 70.00th=[ 32375], 80.00th=[ 44827], 90.00th=[ 61080], 95.00th=[ 77071], | 99.00th=[108528], 99.50th=[135267], 99.90th=[139461], 99.95th=[139461], | 99.99th=[139461] bw ( KiB/s): min= 127, max= 508, per=25.27%, avg=311.87, stdev=94.36, samples=39 iops : min= 7, max= 31, avg=18.85, stdev= 5.96, samples=39 lat (usec) : 100=0.13%, 250=9.90%, 500=2.34%, 750=0.91%, 1000=0.13% lat (msec) : 2=6.90%, 4=4.82%, 10=10.94%, 20=11.98%, 50=36.07% lat (msec) : 100=14.19%, 250=1.69% cpu : usr=0.10%, sys=9.80%, ctx=1360, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1173KiB/s (1201kB/s), 290KiB/s-319KiB/s (297kB/s-326kB/s), io=23.4MiB (24.5MB), run=18905-20421msec WRITE: bw=1234KiB/s (1264kB/s), 307KiB/s-346KiB/s (315kB/s-354kB/s), io=24.6MiB (25.8MB), run=18905-20421msec rand-rw: (groupid=0, jobs=1): err= 0: pid=157626: Mon Jul 14 04:12:39 2025 read: IOPS=6, BW=109KiB/s (112kB/s)(5920KiB/54292msec) clat (msec): min=6, max=214, avg=27.62, stdev=22.98 lat (msec): min=6, max=214, avg=27.62, stdev=22.97 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 23], | 70.00th=[ 29], 80.00th=[ 36], 90.00th=[ 55], 95.00th=[ 75], | 99.00th=[ 106], 99.50th=[ 157], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 31, max= 381, per=25.41%, avg=112.04, stdev=69.21, samples=103 iops : min= 1, max= 23, avg= 6.24, stdev= 4.36, samples=103 write: IOPS=7, BW=117KiB/s (120kB/s)(6368KiB/54292msec); 0 zone resets clat (msec): min=40, max=367, avg=110.48, stdev=48.17 lat (msec): min=40, max=367, avg=110.49, stdev=48.17 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 58], 10.00th=[ 65], 20.00th=[ 72], | 30.00th=[ 81], 40.00th=[ 89], 50.00th=[ 99], 60.00th=[ 108], | 70.00th=[ 122], 80.00th=[ 142], 90.00th=[ 178], 95.00th=[ 215], | 99.00th=[ 284], 99.50th=[ 288], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 30, max= 191, per=25.60%, avg=118.76, stdev=42.25, samples=105 iops : min= 1, max= 11, avg= 6.66, stdev= 2.72, samples=105 lat (msec) : 10=2.47%, 20=22.53%, 50=17.71%, 100=32.29%, 250=24.22% lat (msec) : 500=0.78% cpu : usr=0.08%, sys=2.15%, ctx=980, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157627: Mon Jul 14 04:12:39 2025 read: IOPS=6, BW=106KiB/s (109kB/s)(5744KiB/54060msec) clat (msec): min=6, max=168, avg=28.71, stdev=25.77 lat (msec): min=6, max=168, avg=28.71, stdev=25.77 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 21], 60.00th=[ 24], | 70.00th=[ 29], 80.00th=[ 37], 90.00th=[ 59], 95.00th=[ 88], | 99.00th=[ 140], 99.50th=[ 144], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 31, max= 382, per=26.24%, avg=115.73, stdev=76.08, samples=98 iops : min= 1, max= 23, avg= 6.47, stdev= 4.78, samples=98 write: IOPS=7, BW=121KiB/s (124kB/s)(6544KiB/54060msec); 0 zone resets clat (msec): min=35, max=324, avg=106.21, stdev=45.23 lat (msec): min=35, max=324, avg=106.21, stdev=45.22 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 56], 10.00th=[ 62], 20.00th=[ 71], | 30.00th=[ 78], 40.00th=[ 87], 50.00th=[ 93], 60.00th=[ 105], | 70.00th=[ 121], 80.00th=[ 138], 90.00th=[ 165], 95.00th=[ 199], | 99.00th=[ 255], 99.50th=[ 271], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 30, max= 222, per=26.19%, avg=121.53, stdev=45.61, samples=106 iops : min= 1, max= 13, avg= 6.83, stdev= 2.88, samples=106 lat (msec) : 10=3.26%, 20=20.05%, 50=18.75%, 100=33.33%, 250=23.70% lat (msec) : 500=0.91% cpu : usr=0.06%, sys=2.26%, ctx=995, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157628: Mon Jul 14 04:12:39 2025 read: IOPS=7, BW=121KiB/s (124kB/s)(6256KiB/51740msec) clat (msec): min=6, max=212, avg=26.42, stdev=22.20 lat (msec): min=6, max=212, avg=26.43, stdev=22.20 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 23], | 70.00th=[ 28], 80.00th=[ 36], 90.00th=[ 53], 95.00th=[ 71], | 99.00th=[ 120], 99.50th=[ 138], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 31, max= 352, per=28.34%, avg=125.00, stdev=82.54, samples=99 iops : min= 1, max= 22, avg= 7.05, stdev= 5.26, samples=99 write: IOPS=7, BW=117KiB/s (119kB/s)(6032KiB/51740msec); 0 zone resets clat (msec): min=41, max=335, avg=109.20, stdev=46.76 lat (msec): min=41, max=335, avg=109.20, stdev=46.76 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 56], 10.00th=[ 64], 20.00th=[ 73], | 30.00th=[ 82], 40.00th=[ 90], 50.00th=[ 99], 60.00th=[ 107], | 70.00th=[ 118], 80.00th=[ 138], 90.00th=[ 180], 95.00th=[ 203], | 99.00th=[ 262], 99.50th=[ 292], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 30, max= 192, per=25.23%, avg=117.07, stdev=44.47, samples=101 iops : min= 1, max= 12, avg= 6.55, stdev= 2.81, samples=101 lat (msec) : 10=4.17%, 20=23.70%, 50=18.62%, 100=29.43%, 250=23.18% lat (msec) : 500=0.91% cpu : usr=0.07%, sys=2.25%, ctx=954, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157629: Mon Jul 14 04:12:39 2025 read: IOPS=6, BW=111KiB/s (114kB/s)(6032KiB/54223msec) clat (msec): min=6, max=219, avg=28.62, stdev=26.21 lat (msec): min=6, max=219, avg=28.63, stdev=26.21 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 20], 60.00th=[ 23], | 70.00th=[ 27], 80.00th=[ 36], 90.00th=[ 61], 95.00th=[ 80], | 99.00th=[ 140], 99.50th=[ 144], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 31, max= 352, per=26.88%, avg=118.56, stdev=70.39, samples=99 iops : min= 1, max= 22, avg= 6.65, stdev= 4.48, samples=99 write: IOPS=7, BW=115KiB/s (118kB/s)(6256KiB/54223msec); 0 zone resets clat (msec): min=44, max=302, avg=110.52, stdev=45.88 lat (msec): min=44, max=302, avg=110.53, stdev=45.88 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 59], 10.00th=[ 65], 20.00th=[ 74], | 30.00th=[ 83], 40.00th=[ 90], 50.00th=[ 100], 60.00th=[ 108], | 70.00th=[ 125], 80.00th=[ 142], 90.00th=[ 174], 95.00th=[ 197], | 99.00th=[ 279], 99.50th=[ 300], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 31, max= 223, per=25.01%, avg=116.07, stdev=44.39, samples=106 iops : min= 1, max= 13, avg= 6.49, stdev= 2.84, samples=106 lat (msec) : 10=2.86%, 20=23.18%, 50=16.93%, 100=31.12%, 250=24.74% lat (msec) : 500=1.17% cpu : usr=0.06%, sys=2.20%, ctx=985, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=441KiB/s (452kB/s), 106KiB/s-121KiB/s (109kB/s-124kB/s), io=23.4MiB (24.5MB), run=51740-54292msec WRITE: bw=464KiB/s (475kB/s), 115KiB/s-121KiB/s (118kB/s-124kB/s), io=24.6MiB (25.8MB), run=51740-54292msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=157649: Mon Jul 14 04:12:53 2025 read: IOPS=1, BW=1256KiB/s (1286kB/s)(7168KiB/5707msec) clat (usec): min=1995, max=3021.7k, avg=586378.08, stdev=1079314.00 lat (usec): min=1998, max=3021.7k, avg=586380.32, stdev=1079315.11 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 1991], 10.00th=[ 1991], | 20.00th=[ 88605], 30.00th=[ 156238], 40.00th=[ 156238], | 50.00th=[ 261096], 60.00th=[ 278922], 70.00th=[ 278922], | 80.00th=[ 295699], 90.00th=[3036677], 95.00th=[3036677], | 99.00th=[3036677], 99.50th=[3036677], 99.90th=[3036677], | 99.95th=[3036677], 99.99th=[3036677] bw ( KiB/s): min= 1861, max= 6144, per=100.00%, avg=3022.75, stdev=2082.60, samples=4 iops : min= 1, max= 6, avg= 2.25, stdev= 2.50, samples=4 write: IOPS=0, BW=897KiB/s (919kB/s)(5120KiB/5707msec); 0 zone resets clat (msec): min=116, max=418, avg=306.74, stdev=122.49 lat (msec): min=116, max=420, avg=307.30, stdev=123.01 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 117], 10.00th=[ 117], 20.00th=[ 117], | 30.00th=[ 257], 40.00th=[ 257], 50.00th=[ 351], 60.00th=[ 351], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 418], 95.00th=[ 418], | 99.00th=[ 418], 99.50th=[ 418], 99.90th=[ 418], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 1861, max= 6119, per=97.36%, avg=3990.00, stdev=3010.86, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 lat (msec) : 2=8.33%, 100=8.33%, 250=16.67%, 500=58.33%, >=2000=8.33% cpu : usr=0.00%, sys=14.25%, ctx=601, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157650: Mon Jul 14 04:12:53 2025 read: IOPS=0, BW=293KiB/s (300kB/s)(2048KiB/6995msec) clat (usec): min=855, max=242488, avg=121672.05, stdev=170859.73 lat (usec): min=858, max=242490, avg=121674.50, stdev=170859.94 clat percentiles (usec): | 1.00th=[ 857], 5.00th=[ 857], 10.00th=[ 857], 20.00th=[ 857], | 30.00th=[ 857], 40.00th=[ 857], 50.00th=[ 857], 60.00th=[242222], | 70.00th=[242222], 80.00th=[242222], 90.00th=[242222], 95.00th=[242222], | 99.00th=[242222], 99.50th=[242222], 99.90th=[242222], 99.95th=[242222], | 99.99th=[242222] bw ( KiB/s): min= 1420, max= 2035, per=59.02%, avg=1727.50, stdev=434.87, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=1, BW=1464KiB/s (1499kB/s)(10.0MiB/6995msec); 0 zone resets clat (msec): min=195, max=3218, avg=656.01, stdev=919.15 lat (msec): min=195, max=3218, avg=656.19, stdev=919.11 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 197], 10.00th=[ 197], 20.00th=[ 234], | 30.00th=[ 275], 40.00th=[ 284], 50.00th=[ 317], 60.00th=[ 384], | 70.00th=[ 393], 80.00th=[ 405], 90.00th=[ 860], 95.00th=[ 3205], | 99.00th=[ 3205], 99.50th=[ 3205], 99.90th=[ 3205], 99.95th=[ 3205], | 99.99th=[ 3205] bw ( KiB/s): min= 1861, max= 4087, per=74.01%, avg=3032.83, stdev=1153.76, samples=6 iops : min= 1, max= 3, avg= 2.17, stdev= 0.98, samples=6 lat (usec) : 1000=8.33% lat (msec) : 250=25.00%, 500=50.00%, 1000=8.33%, >=2000=8.33% cpu : usr=0.01%, sys=11.84%, ctx=197, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157651: Mon Jul 14 04:12:53 2025 read: IOPS=1, BW=1043KiB/s (1069kB/s)(6144KiB/5888msec) clat (usec): min=796, max=640174, avg=191218.52, stdev=243915.21 lat (usec): min=798, max=640175, avg=191220.19, stdev=243915.09 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 799], 10.00th=[ 799], 20.00th=[ 25035], | 30.00th=[ 25035], 40.00th=[ 51643], 50.00th=[ 51643], 60.00th=[139461], | 70.00th=[291505], 80.00th=[291505], 90.00th=[641729], 95.00th=[641729], | 99.00th=[641729], 99.50th=[641729], 99.90th=[641729], 99.95th=[641729], | 99.99th=[641729] bw ( KiB/s): min= 4087, max= 6131, per=100.00%, avg=5109.00, stdev=1445.33, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 write: IOPS=1, BW=1043KiB/s (1069kB/s)(6144KiB/5888msec); 0 zone resets clat (msec): min=132, max=3344, avg=748.49, stdev=1273.21 lat (msec): min=132, max=3344, avg=748.66, stdev=1273.20 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 284], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 3339], 95.00th=[ 3339], | 99.00th=[ 3339], 99.50th=[ 3339], 99.90th=[ 3339], 99.95th=[ 3339], | 99.99th=[ 3339] bw ( KiB/s): min= 4039, max= 4087, per=99.34%, avg=4071.00, stdev=27.71, samples=3 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=3 lat (usec) : 1000=8.33% lat (msec) : 50=8.33%, 100=8.33%, 250=25.00%, 500=33.33%, 750=8.33% lat (msec) : >=2000=8.33% cpu : usr=0.02%, sys=9.10%, ctx=156, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157652: Mon Jul 14 04:12:53 2025 read: IOPS=0, BW=848KiB/s (869kB/s)(5120KiB/6036msec) clat (usec): min=716, max=299829, avg=176546.82, stdev=120184.13 lat (usec): min=718, max=299830, avg=176548.57, stdev=120183.78 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 717], 10.00th=[ 717], 20.00th=[ 717], | 30.00th=[137364], 40.00th=[137364], 50.00th=[168821], 60.00th=[168821], | 70.00th=[274727], 80.00th=[274727], 90.00th=[299893], 95.00th=[299893], | 99.00th=[299893], 99.50th=[299893], 99.90th=[299893], 99.95th=[299893], | 99.99th=[299893] bw ( KiB/s): min= 2043, max= 6107, per=100.00%, avg=4075.00, stdev=2873.68, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 write: IOPS=1, BW=1188KiB/s (1216kB/s)(7168KiB/6036msec); 0 zone resets clat (msec): min=179, max=3079, avg=703.70, stdev=1053.84 lat (msec): min=179, max=3079, avg=703.84, stdev=1053.85 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 180], 10.00th=[ 180], 20.00th=[ 222], | 30.00th=[ 236], 40.00th=[ 236], 50.00th=[ 279], 60.00th=[ 439], | 70.00th=[ 439], 80.00th=[ 493], 90.00th=[ 3071], 95.00th=[ 3071], | 99.00th=[ 3071], 99.50th=[ 3071], 99.90th=[ 3071], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min= 1354, max= 3771, per=54.06%, avg=2215.40, stdev=914.50, samples=5 iops : min= 1, max= 3, avg= 1.60, stdev= 0.89, samples=5 lat (usec) : 750=8.33% lat (msec) : 250=41.67%, 500=41.67%, >=2000=8.33% cpu : usr=0.02%, sys=10.06%, ctx=149, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2928KiB/s (2998kB/s), 293KiB/s-1256KiB/s (300kB/s-1286kB/s), io=20.0MiB (20.0MB), run=5707-6995msec WRITE: bw=4099KiB/s (4197kB/s), 897KiB/s-1464KiB/s (919kB/s-1499kB/s), io=28.0MiB (29.4MB), run=5707-6995msec rand-rw: (groupid=0, jobs=1): err= 0: pid=157644: Mon Jul 14 04:12:53 2025 read: IOPS=1, BW=1067KiB/s (1092kB/s)(7168KiB/6721msec) clat (msec): min=88, max=3013, avg=638.53, stdev=1064.16 lat (msec): min=88, max=3013, avg=638.53, stdev=1064.16 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 89], 10.00th=[ 89], 20.00th=[ 106], | 30.00th=[ 121], 40.00th=[ 121], 50.00th=[ 167], 60.00th=[ 363], | 70.00th=[ 363], 80.00th=[ 617], 90.00th=[ 3004], 95.00th=[ 3004], | 99.00th=[ 3004], 99.50th=[ 3004], 99.90th=[ 3004], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 1378, max= 2048, per=69.30%, avg=1884.20, stdev=287.72, samples=5 iops : min= 1, max= 2, avg= 1.20, stdev= 0.45, samples=5 write: IOPS=0, BW=762KiB/s (780kB/s)(5120KiB/6721msec); 0 zone resets clat (msec): min=206, max=733, avg=425.88, stdev=232.70 lat (msec): min=207, max=733, avg=426.08, stdev=232.63 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 207], 10.00th=[ 207], 20.00th=[ 207], | 30.00th=[ 271], 40.00th=[ 271], 50.00th=[ 305], 60.00th=[ 305], | 70.00th=[ 617], 80.00th=[ 617], 90.00th=[ 735], 95.00th=[ 735], | 99.00th=[ 735], 99.50th=[ 735], 99.90th=[ 735], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 2039, max= 3930, per=70.20%, avg=2672.33, stdev=1089.18, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 lat (msec) : 100=8.33%, 250=33.33%, 500=25.00%, 750=25.00%, >=2000=8.33% cpu : usr=0.00%, sys=3.81%, ctx=85, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157645: Mon Jul 14 04:12:53 2025 read: IOPS=0, BW=292KiB/s (299kB/s)(2048KiB/7003msec) clat (msec): min=111, max=169, avg=140.38, stdev=41.42 lat (msec): min=111, max=169, avg=140.39, stdev=41.41 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 111], 50.00th=[ 111], 60.00th=[ 169], | 70.00th=[ 169], 80.00th=[ 169], 90.00th=[ 169], 95.00th=[ 169], | 99.00th=[ 169], 99.50th=[ 169], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 1378, max= 1976, per=61.68%, avg=1677.00, stdev=422.85, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=1, BW=1462KiB/s (1497kB/s)(10.0MiB/7003msec); 0 zone resets clat (msec): min=145, max=3376, avg=661.52, stdev=967.83 lat (msec): min=145, max=3376, avg=661.66, stdev=967.85 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 146], 10.00th=[ 146], 20.00th=[ 224], | 30.00th=[ 232], 40.00th=[ 268], 50.00th=[ 330], 60.00th=[ 359], | 70.00th=[ 426], 80.00th=[ 625], 90.00th=[ 634], 95.00th=[ 3373], | 99.00th=[ 3373], 99.50th=[ 3373], 99.90th=[ 3373], 99.95th=[ 3373], | 99.99th=[ 3373] bw ( KiB/s): min= 1965, max= 6083, per=78.58%, avg=2991.50, stdev=1683.49, samples=6 iops : min= 1, max= 5, avg= 2.17, stdev= 1.60, samples=6 lat (msec) : 250=41.67%, 500=33.33%, 750=16.67%, >=2000=8.33% cpu : usr=0.04%, sys=4.08%, ctx=66, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157646: Mon Jul 14 04:12:53 2025 read: IOPS=0, BW=816KiB/s (835kB/s)(6144KiB/7531msec) clat (msec): min=62, max=889, avg=291.39, stdev=307.35 lat (msec): min=62, max=889, avg=291.39, stdev=307.35 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 85], | 30.00th=[ 85], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 230], | 70.00th=[ 317], 80.00th=[ 317], 90.00th=[ 885], 95.00th=[ 885], | 99.00th=[ 885], 99.50th=[ 885], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 2048, max= 4055, per=100.00%, avg=3352.00, stdev=1130.45, samples=3 iops : min= 2, max= 3, avg= 2.67, stdev= 0.58, samples=3 write: IOPS=0, BW=816KiB/s (835kB/s)(6144KiB/7531msec); 0 zone resets clat (msec): min=150, max=3243, avg=925.86, stdev=1181.90 lat (msec): min=150, max=3243, avg=926.31, stdev=1181.66 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 150], 10.00th=[ 150], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 426], 50.00th=[ 426], 60.00th=[ 531], | 70.00th=[ 1053], 80.00th=[ 1053], 90.00th=[ 3239], 95.00th=[ 3239], | 99.00th=[ 3239], 99.50th=[ 3239], 99.90th=[ 3239], 99.95th=[ 3239], | 99.99th=[ 3239] bw ( KiB/s): min= 1393, max= 2048, per=49.59%, avg=1888.00, stdev=281.65, samples=5 iops : min= 1, max= 2, avg= 1.20, stdev= 0.45, samples=5 lat (msec) : 100=16.67%, 250=33.33%, 500=16.67%, 750=8.33%, 1000=8.33% lat (msec) : 2000=8.33%, >=2000=8.33% cpu : usr=0.01%, sys=3.19%, ctx=85, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157647: Mon Jul 14 04:12:53 2025 read: IOPS=0, BW=688KiB/s (704kB/s)(5120KiB/7446msec) clat (msec): min=76, max=137, avg=109.68, stdev=23.10 lat (msec): min=76, max=137, avg=109.68, stdev=23.10 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 78], 10.00th=[ 78], 20.00th=[ 78], | 30.00th=[ 102], 40.00th=[ 102], 50.00th=[ 110], 60.00th=[ 110], | 70.00th=[ 124], 80.00th=[ 124], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 2048, max= 6059, per=100.00%, avg=4053.50, stdev=2836.21, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=0, BW=963KiB/s (986kB/s)(7168KiB/7446msec); 0 zone resets clat (msec): min=101, max=4474, avg=980.14, stdev=1564.54 lat (msec): min=102, max=4475, avg=980.35, stdev=1564.72 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 103], 10.00th=[ 103], 20.00th=[ 159], | 30.00th=[ 215], 40.00th=[ 215], 50.00th=[ 414], 60.00th=[ 659], | 70.00th=[ 659], 80.00th=[ 835], 90.00th=[ 4463], 95.00th=[ 4463], | 99.00th=[ 4463], 99.50th=[ 4463], 99.90th=[ 4463], 99.95th=[ 4463], | 99.99th=[ 4463] bw ( KiB/s): min= 1954, max= 2048, per=53.08%, avg=2020.80, stdev=39.20, samples=5 iops : min= 1, max= 2, avg= 1.40, stdev= 0.55, samples=5 lat (msec) : 100=8.33%, 250=58.33%, 500=8.33%, 750=8.33%, 1000=8.33% lat (msec) : >=2000=8.33% cpu : usr=0.00%, sys=4.20%, ctx=89, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2719KiB/s (2785kB/s), 292KiB/s-1067KiB/s (299kB/s-1092kB/s), io=20.0MiB (20.0MB), run=6721-7531msec WRITE: bw=3807KiB/s (3899kB/s), 762KiB/s-1462KiB/s (780kB/s-1497kB/s), io=28.0MiB (29.4MB), run=6721-7531msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=157658: Mon Jul 14 04:13:05 2025 read: IOPS=1, BW=5721KiB/s (5858kB/s)(12.0MiB/2148msec) clat (msec): min=533, max=867, avg=698.19, stdev=167.11 lat (msec): min=533, max=867, avg=698.19, stdev=167.11 clat percentiles (msec): | 1.00th=[ 535], 5.00th=[ 535], 10.00th=[ 535], 20.00th=[ 535], | 30.00th=[ 535], 40.00th=[ 693], 50.00th=[ 693], 60.00th=[ 693], | 70.00th=[ 869], 80.00th=[ 869], 90.00th=[ 869], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 7801, max= 8047, per=100.00%, avg=7924.00, stdev=173.95, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 750=66.67%, 1000=33.33% cpu : usr=0.00%, sys=19.24%, ctx=357, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157659: Mon Jul 14 04:13:05 2025 read: IOPS=0, BW=1064KiB/s (1090kB/s)(4096KiB/3849msec) clat (nsec): min=649891k, max=649891k, avg=649890573.00, stdev= 0.00 lat (nsec): min=649894k, max=649894k, avg=649893523.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 651], 5.00th=[ 651], 10.00th=[ 651], 20.00th=[ 651], | 30.00th=[ 651], 40.00th=[ 651], 50.00th=[ 651], 60.00th=[ 651], | 70.00th=[ 651], 80.00th=[ 651], 90.00th=[ 651], 95.00th=[ 651], | 99.00th=[ 651], 99.50th=[ 651], 99.90th=[ 651], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 7801, max= 7801, per=100.00%, avg=7801.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2128KiB/s (2179kB/s)(8192KiB/3849msec); 0 zone resets clat (msec): min=1391, max=1759, avg=1575.56, stdev=260.55 lat (msec): min=1391, max=1760, avg=1576.07, stdev=260.74 clat percentiles (msec): | 1.00th=[ 1385], 5.00th=[ 1385], 10.00th=[ 1385], 20.00th=[ 1385], | 30.00th=[ 1385], 40.00th=[ 1385], 50.00th=[ 1385], 60.00th=[ 1754], | 70.00th=[ 1754], 80.00th=[ 1754], 90.00th=[ 1754], 95.00th=[ 1754], | 99.00th=[ 1754], 99.50th=[ 1754], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 8110, max= 8110, per=96.80%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 750=33.33%, 2000=66.67% cpu : usr=0.03%, sys=15.77%, ctx=210, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157660: Mon Jul 14 04:13:05 2025 write: IOPS=0, BW=3142KiB/s (3217kB/s)(12.0MiB/3911msec); 0 zone resets clat (msec): min=1023, max=1478, avg=1282.99, stdev=234.49 lat (msec): min=1023, max=1479, avg=1283.56, stdev=234.64 clat percentiles (msec): | 1.00th=[ 1020], 5.00th=[ 1020], 10.00th=[ 1020], 20.00th=[ 1020], | 30.00th=[ 1020], 40.00th=[ 1351], 50.00th=[ 1351], 60.00th=[ 1351], | 70.00th=[ 1485], 80.00th=[ 1485], 90.00th=[ 1485], 95.00th=[ 1485], | 99.00th=[ 1485], 99.50th=[ 1485], 99.90th=[ 1485], 99.95th=[ 1485], | 99.99th=[ 1485] bw ( KiB/s): min= 7801, max= 8110, per=94.96%, avg=7955.50, stdev=218.50, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 2000=100.00% cpu : usr=0.03%, sys=22.53%, ctx=131, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157661: Mon Jul 14 04:13:05 2025 write: IOPS=1, BW=4316KiB/s (4420kB/s)(12.0MiB/2847msec); 0 zone resets clat (msec): min=858, max=1004, avg=940.57, stdev=74.45 lat (msec): min=859, max=1006, avg=941.24, stdev=74.97 clat percentiles (msec): | 1.00th=[ 860], 5.00th=[ 860], 10.00th=[ 860], 20.00th=[ 860], | 30.00th=[ 860], 40.00th=[ 961], 50.00th=[ 961], 60.00th=[ 961], | 70.00th=[ 1003], 80.00th=[ 1003], 90.00th=[ 1003], 95.00th=[ 1003], | 99.00th=[ 1003], 99.50th=[ 1003], 99.90th=[ 1003], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 7801, max= 8143, per=95.15%, avg=7972.00, stdev=241.83, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 1000=66.67%, 2000=33.33% cpu : usr=0.00%, sys=29.41%, ctx=105, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4257KiB/s (4359kB/s), 1064KiB/s-5721KiB/s (1090kB/s-5858kB/s), io=16.0MiB (16.8MB), run=2148-3849msec WRITE: bw=8378KiB/s (8580kB/s), 2128KiB/s-4316KiB/s (2179kB/s-4420kB/s), io=32.0MiB (33.6MB), run=2847-3911msec rand-rw: (groupid=0, jobs=1): err= 0: pid=157663: Mon Jul 14 04:13:06 2025 read: IOPS=0, BW=3229KiB/s (3306kB/s)(12.0MiB/3806msec) clat (msec): min=1041, max=1589, avg=1258.76, stdev=291.31 lat (msec): min=1041, max=1589, avg=1258.76, stdev=291.31 clat percentiles (msec): | 1.00th=[ 1045], 5.00th=[ 1045], 10.00th=[ 1045], 20.00th=[ 1045], | 30.00th=[ 1045], 40.00th=[ 1150], 50.00th=[ 1150], 60.00th=[ 1150], | 70.00th=[ 1586], 80.00th=[ 1586], 90.00th=[ 1586], 95.00th=[ 1586], | 99.00th=[ 1586], 99.50th=[ 1586], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 7907, max= 8159, per=100.00%, avg=8033.00, stdev=178.19, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 2000=100.00% cpu : usr=0.00%, sys=4.73%, ctx=76, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157664: Mon Jul 14 04:13:06 2025 read: IOPS=0, BW=886KiB/s (907kB/s)(4096KiB/4624msec) clat (nsec): min=1060.6M, max=1060.6M, avg=1060584984.00, stdev= 0.00 lat (nsec): min=1060.6M, max=1060.6M, avg=1060587441.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1062], 5.00th=[ 1062], 10.00th=[ 1062], 20.00th=[ 1062], | 30.00th=[ 1062], 40.00th=[ 1062], 50.00th=[ 1062], 60.00th=[ 1062], | 70.00th=[ 1062], 80.00th=[ 1062], 90.00th=[ 1062], 95.00th=[ 1062], | 99.00th=[ 1062], 99.50th=[ 1062], 99.90th=[ 1062], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 7968, max= 7968, per=100.00%, avg=7968.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1772KiB/s (1814kB/s)(8192KiB/4624msec); 0 zone resets clat (msec): min=1638, max=1811, avg=1724.84, stdev=122.18 lat (msec): min=1639, max=1811, avg=1725.50, stdev=122.16 clat percentiles (msec): | 1.00th=[ 1636], 5.00th=[ 1636], 10.00th=[ 1636], 20.00th=[ 1636], | 30.00th=[ 1636], 40.00th=[ 1636], 50.00th=[ 1636], 60.00th=[ 1804], | 70.00th=[ 1804], 80.00th=[ 1804], 90.00th=[ 1804], 95.00th=[ 1804], | 99.00th=[ 1804], 99.50th=[ 1804], 99.90th=[ 1804], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 8159, max= 8159, per=100.00%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 2000=100.00% cpu : usr=0.00%, sys=3.22%, ctx=42, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157665: Mon Jul 14 04:13:06 2025 write: IOPS=0, BW=2603KiB/s (2666kB/s)(12.0MiB/4720msec); 0 zone resets clat (msec): min=1120, max=2173, avg=1545.03, stdev=555.11 lat (msec): min=1121, max=2173, avg=1545.58, stdev=554.91 clat percentiles (msec): | 1.00th=[ 1116], 5.00th=[ 1116], 10.00th=[ 1116], 20.00th=[ 1116], | 30.00th=[ 1116], 40.00th=[ 1334], 50.00th=[ 1334], 60.00th=[ 1334], | 70.00th=[ 2165], 80.00th=[ 2165], 90.00th=[ 2165], 95.00th=[ 2165], | 99.00th=[ 2165], 99.50th=[ 2165], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 2000=66.67%, >=2000=33.33% cpu : usr=0.02%, sys=6.82%, ctx=92, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157666: Mon Jul 14 04:13:06 2025 write: IOPS=0, BW=2689KiB/s (2753kB/s)(12.0MiB/4570msec); 0 zone resets clat (msec): min=677, max=2448, avg=1514.14, stdev=889.53 lat (msec): min=680, max=2453, avg=1517.16, stdev=890.33 clat percentiles (msec): | 1.00th=[ 676], 5.00th=[ 676], 10.00th=[ 676], 20.00th=[ 676], | 30.00th=[ 676], 40.00th=[ 1418], 50.00th=[ 1418], 60.00th=[ 1418], | 70.00th=[ 2433], 80.00th=[ 2433], 90.00th=[ 2433], 95.00th=[ 2433], | 99.00th=[ 2433], 99.50th=[ 2433], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 750=33.33%, 2000=33.33%, >=2000=33.33% cpu : usr=0.02%, sys=5.80%, ctx=150, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3543KiB/s (3628kB/s), 886KiB/s-3229KiB/s (907kB/s-3306kB/s), io=16.0MiB (16.8MB), run=3806-4624msec WRITE: bw=6942KiB/s (7109kB/s), 1772KiB/s-2689KiB/s (1814kB/s-2753kB/s), io=32.0MiB (33.6MB), run=4570-4720msec