== sanity test 398b: DIO and buffer IO race ============== 12:42:41 (1752511361) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB, 48 MiB) copied, 3.54178 s, 14.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=157522: Mon Jul 14 12:43:56 2025 read: IOPS=23, BW=93.8KiB/s (96.1kB/s)(5992KiB/63864msec) clat (usec): min=110, max=250845, avg=20158.72, stdev=14975.86 lat (usec): min=110, max=250846, avg=20159.64, stdev=14975.88 clat percentiles (usec): | 1.00th=[ 133], 5.00th=[ 265], 10.00th=[ 5145], 20.00th=[ 10159], | 30.00th=[ 12780], 40.00th=[ 15008], 50.00th=[ 17695], 60.00th=[ 21103], | 70.00th=[ 24511], 80.00th=[ 29492], 90.00th=[ 37487], 95.00th=[ 42730], | 99.00th=[ 60031], 99.50th=[ 67634], 99.90th=[177210], 99.95th=[250610], | 99.99th=[250610] bw ( KiB/s): min= 31, max= 208, per=25.03%, avg=93.86, stdev=32.86, samples=125 iops : min= 7, max= 52, avg=22.98, stdev= 8.29, samples=125 write: IOPS=24, BW=98.6KiB/s (101kB/s)(6296KiB/63864msec); 0 zone resets clat (usec): min=687, max=730879, avg=21203.19, stdev=27345.59 lat (usec): min=689, max=730880, avg=21204.41, stdev=27345.55 clat percentiles (usec): | 1.00th=[ 1004], 5.00th=[ 1205], 10.00th=[ 2073], 20.00th=[ 4178], | 30.00th=[ 6587], 40.00th=[ 11338], 50.00th=[ 15139], 60.00th=[ 20055], | 70.00th=[ 26084], 80.00th=[ 32900], 90.00th=[ 45876], 95.00th=[ 59507], | 99.00th=[100140], 99.50th=[116917], 99.90th=[164627], 99.95th=[734004], | 99.99th=[734004] bw ( KiB/s): min= 7, max= 208, per=24.83%, avg=97.57, stdev=36.84, samples=126 iops : min= 1, max= 52, avg=23.90, stdev= 9.27, samples=126 lat (usec) : 250=2.31%, 500=1.27%, 750=0.55%, 1000=0.68% lat (msec) : 2=4.72%, 4=5.01%, 10=13.67%, 20=30.34%, 50=35.94% lat (msec) : 100=4.85%, 250=0.59%, 500=0.03%, 750=0.03% cpu : usr=0.10%, sys=18.59%, ctx=7316, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157523: Mon Jul 14 12:43:56 2025 read: IOPS=23, BW=93.1KiB/s (95.3kB/s)(5940KiB/63830msec) clat (usec): min=93, max=787762, avg=21731.26, stdev=25503.24 lat (usec): min=94, max=787762, avg=21732.50, stdev=25503.14 clat percentiles (usec): | 1.00th=[ 135], 5.00th=[ 322], 10.00th=[ 6718], 20.00th=[ 10552], | 30.00th=[ 13173], 40.00th=[ 15795], 50.00th=[ 18482], 60.00th=[ 21627], | 70.00th=[ 25297], 80.00th=[ 30278], 90.00th=[ 38536], 95.00th=[ 45876], | 99.00th=[ 73925], 99.50th=[106431], 99.90th=[214959], 99.95th=[784335], | 99.99th=[784335] bw ( KiB/s): min= 26, max= 183, per=24.89%, avg=93.34, stdev=31.51, samples=125 iops : min= 6, max= 45, avg=22.84, stdev= 7.95, samples=125 write: IOPS=24, BW=99.5KiB/s (102kB/s)(6348KiB/63830msec); 0 zone resets clat (usec): min=740, max=198966, avg=19698.60, stdev=19901.93 lat (usec): min=741, max=198967, avg=19699.75, stdev=19901.93 clat percentiles (usec): | 1.00th=[ 971], 5.00th=[ 1156], 10.00th=[ 2040], 20.00th=[ 4015], | 30.00th=[ 6259], 40.00th=[ 10028], 50.00th=[ 14746], 60.00th=[ 19006], | 70.00th=[ 24511], 80.00th=[ 32113], 90.00th=[ 43779], 95.00th=[ 57410], | 99.00th=[ 89654], 99.50th=[109577], 99.90th=[179307], 99.95th=[198181], | 99.99th=[198181] bw ( KiB/s): min= 31, max= 216, per=25.25%, avg=99.25, stdev=33.75, samples=125 iops : min= 7, max= 54, avg=24.34, stdev= 8.52, samples=125 lat (usec) : 100=0.03%, 250=1.63%, 500=1.50%, 750=0.49%, 1000=1.01% lat (msec) : 2=4.56%, 4=5.37%, 10=14.84%, 20=29.20%, 50=36.04% lat (msec) : 100=4.72%, 250=0.59%, 1000=0.03% cpu : usr=0.10%, sys=18.56%, ctx=7243, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157524: Mon Jul 14 12:43:56 2025 read: IOPS=23, BW=95.2KiB/s (97.4kB/s)(6028KiB/63347msec) clat (usec): min=100, max=179569, avg=20152.52, stdev=16067.78 lat (usec): min=100, max=179569, avg=20154.10, stdev=16068.51 clat percentiles (usec): | 1.00th=[ 129], 5.00th=[ 277], 10.00th=[ 5211], 20.00th=[ 10290], | 30.00th=[ 12387], 40.00th=[ 14615], 50.00th=[ 17171], 60.00th=[ 19530], | 70.00th=[ 23200], 80.00th=[ 28705], 90.00th=[ 36963], 95.00th=[ 46400], | 99.00th=[ 72877], 99.50th=[ 98042], 99.90th=[175113], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 39, max= 208, per=25.36%, avg=95.10, stdev=34.32, samples=124 iops : min= 9, max= 52, avg=23.31, stdev= 8.66, samples=124 write: IOPS=24, BW=98.8KiB/s (101kB/s)(6260KiB/63347msec); 0 zone resets clat (usec): min=718, max=729837, avg=20870.54, stdev=26185.56 lat (usec): min=719, max=729838, avg=20872.23, stdev=26185.90 clat percentiles (usec): | 1.00th=[ 971], 5.00th=[ 1156], 10.00th=[ 1991], 20.00th=[ 3884], | 30.00th=[ 6521], 40.00th=[ 11338], 50.00th=[ 16057], 60.00th=[ 20579], | 70.00th=[ 26870], 80.00th=[ 32900], 90.00th=[ 45351], 95.00th=[ 57410], | 99.00th=[ 87557], 99.50th=[103285], 99.90th=[149947], 99.95th=[734004], | 99.99th=[734004] bw ( KiB/s): min= 7, max= 191, per=24.86%, avg=97.71, stdev=33.02, samples=125 iops : min= 1, max= 47, avg=23.94, stdev= 8.35, samples=125 lat (usec) : 250=2.18%, 500=1.95%, 750=0.36%, 1000=0.85% lat (msec) : 2=4.49%, 4=5.57%, 10=12.66%, 20=31.97%, 50=34.41% lat (msec) : 100=4.98%, 250=0.55%, 750=0.03% cpu : usr=0.12%, sys=18.70%, ctx=7189, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157525: Mon Jul 14 12:43:56 2025 read: IOPS=24, BW=96.3KiB/s (98.6kB/s)(6052KiB/62876msec) clat (usec): min=87, max=739732, avg=20703.28, stdev=22991.31 lat (usec): min=87, max=739733, avg=20705.24, stdev=22991.81 clat percentiles (usec): | 1.00th=[ 135], 5.00th=[ 281], 10.00th=[ 6390], 20.00th=[ 10683], | 30.00th=[ 13042], 40.00th=[ 15270], 50.00th=[ 17695], 60.00th=[ 20579], | 70.00th=[ 24249], 80.00th=[ 28967], 90.00th=[ 36963], 95.00th=[ 44827], | 99.00th=[ 65799], 99.50th=[ 83362], 99.90th=[117965], 99.95th=[742392], | 99.99th=[742392] bw ( KiB/s): min= 7, max= 184, per=25.53%, avg=95.75, stdev=30.70, samples=124 iops : min= 1, max= 46, avg=23.45, stdev= 7.75, samples=124 write: IOPS=24, BW=99.2KiB/s (102kB/s)(6236KiB/62876msec); 0 zone resets clat (usec): min=650, max=273756, avg=20047.75, stdev=20793.89 lat (usec): min=651, max=273756, avg=20048.92, stdev=20793.92 clat percentiles (usec): | 1.00th=[ 971], 5.00th=[ 1172], 10.00th=[ 2008], 20.00th=[ 3785], | 30.00th=[ 6390], 40.00th=[ 10552], 50.00th=[ 15270], 60.00th=[ 19268], | 70.00th=[ 24773], 80.00th=[ 32113], 90.00th=[ 44827], 95.00th=[ 54789], | 99.00th=[ 95945], 99.50th=[122160], 99.90th=[164627], 99.95th=[274727], | 99.99th=[274727] bw ( KiB/s): min= 39, max= 184, per=25.37%, avg=99.69, stdev=34.01, samples=123 iops : min= 9, max= 46, avg=24.44, stdev= 8.57, samples=123 lat (usec) : 100=0.03%, 250=2.15%, 500=1.37%, 750=0.62%, 1000=0.94% lat (msec) : 2=4.33%, 4=5.86%, 10=13.09%, 20=31.09%, 50=35.42% lat (msec) : 100=4.56%, 250=0.49%, 500=0.03%, 750=0.03% cpu : usr=0.09%, sys=18.74%, ctx=7203, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=376KiB/s (385kB/s), 93.1KiB/s-96.3KiB/s (95.3kB/s-98.6kB/s), io=23.4MiB (24.6MB), run=62876-63864msec WRITE: bw=394KiB/s (403kB/s), 98.6KiB/s-99.5KiB/s (101kB/s-102kB/s), io=24.6MiB (25.7MB), run=62876-63864msec rand-rw: (groupid=0, jobs=1): err= 0: pid=157527: Mon Jul 14 12:44:40 2025 read: IOPS=13, BW=55.7KiB/s (57.0kB/s)(5992KiB/107581msec) clat (msec): min=2, max=786, avg=18.22, stdev=25.82 lat (msec): min=2, max=786, avg=18.22, stdev=25.82 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 10], 50.00th=[ 13], 60.00th=[ 16], | 70.00th=[ 19], 80.00th=[ 25], 90.00th=[ 40], 95.00th=[ 53], | 99.00th=[ 80], 99.50th=[ 103], 99.90th=[ 176], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 7, max= 168, per=25.56%, avg=57.00, stdev=37.30, samples=207 iops : min= 1, max= 42, avg=13.90, stdev= 9.45, samples=207 write: IOPS=14, BW=58.5KiB/s (59.9kB/s)(6296KiB/107581msec); 0 zone resets clat (msec): min=13, max=297, avg=50.97, stdev=35.89 lat (msec): min=13, max=297, avg=50.97, stdev=35.89 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 22], 10.00th=[ 24], 20.00th=[ 28], | 30.00th=[ 31], 40.00th=[ 34], 50.00th=[ 38], 60.00th=[ 44], | 70.00th=[ 53], 80.00th=[ 68], 90.00th=[ 102], 95.00th=[ 128], | 99.00th=[ 180], 99.50th=[ 218], 99.90th=[ 271], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 7, max= 152, per=25.09%, avg=58.47, stdev=31.87, samples=212 iops : min= 1, max= 38, avg=14.26, stdev= 8.12, samples=212 lat (msec) : 4=0.65%, 10=19.01%, 20=17.35%, 50=43.82%, 100=13.51% lat (msec) : 250=5.44%, 500=0.20%, 1000=0.03% cpu : usr=0.08%, sys=5.87%, ctx=4207, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157528: Mon Jul 14 12:44:40 2025 read: IOPS=13, BW=55.3KiB/s (56.6kB/s)(5940KiB/107399msec) clat (msec): min=2, max=750, avg=17.84, stdev=25.92 lat (msec): min=2, max=750, avg=17.84, stdev=25.92 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 10], 50.00th=[ 13], 60.00th=[ 16], | 70.00th=[ 19], 80.00th=[ 23], 90.00th=[ 37], 95.00th=[ 48], | 99.00th=[ 78], 99.50th=[ 129], 99.90th=[ 232], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 7, max= 176, per=25.48%, avg=56.81, stdev=38.05, samples=207 iops : min= 1, max= 44, avg=13.86, stdev= 9.65, samples=207 write: IOPS=14, BW=59.1KiB/s (60.5kB/s)(6348KiB/107399msec); 0 zone resets clat (msec): min=14, max=251, avg=50.93, stdev=35.00 lat (msec): min=14, max=251, avg=50.94, stdev=35.00 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 22], 10.00th=[ 24], 20.00th=[ 27], | 30.00th=[ 31], 40.00th=[ 34], 50.00th=[ 39], 60.00th=[ 44], | 70.00th=[ 52], 80.00th=[ 69], 90.00th=[ 102], 95.00th=[ 124], | 99.00th=[ 184], 99.50th=[ 203], 99.90th=[ 234], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 8, max= 136, per=25.37%, avg=59.11, stdev=31.75, samples=212 iops : min= 2, max= 34, avg=14.42, stdev= 8.09, samples=212 lat (msec) : 4=0.72%, 10=19.08%, 20=17.87%, 50=43.78%, 100=12.86% lat (msec) : 250=5.63%, 500=0.03%, 1000=0.03% cpu : usr=0.09%, sys=5.81%, ctx=4234, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157529: Mon Jul 14 12:44:40 2025 read: IOPS=14, BW=56.0KiB/s (57.4kB/s)(6028KiB/107593msec) clat (msec): min=2, max=205, avg=18.21, stdev=18.07 lat (msec): min=2, max=205, avg=18.21, stdev=18.07 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 11], 50.00th=[ 14], 60.00th=[ 16], | 70.00th=[ 20], 80.00th=[ 26], 90.00th=[ 39], 95.00th=[ 52], | 99.00th=[ 84], 99.50th=[ 122], 99.90th=[ 199], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 7, max= 152, per=25.57%, avg=57.02, stdev=34.28, samples=208 iops : min= 1, max= 38, avg=13.90, stdev= 8.71, samples=208 write: IOPS=14, BW=58.2KiB/s (59.6kB/s)(6260KiB/107593msec); 0 zone resets clat (msec): min=13, max=765, avg=51.17, stdev=38.57 lat (msec): min=13, max=765, avg=51.17, stdev=38.57 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 22], 10.00th=[ 24], 20.00th=[ 27], | 30.00th=[ 31], 40.00th=[ 34], 50.00th=[ 39], 60.00th=[ 45], | 70.00th=[ 53], 80.00th=[ 70], 90.00th=[ 101], 95.00th=[ 125], | 99.00th=[ 169], 99.50th=[ 205], 99.90th=[ 305], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 7, max= 136, per=24.91%, avg=58.05, stdev=32.31, samples=212 iops : min= 1, max= 34, avg=14.16, stdev= 8.23, samples=212 lat (msec) : 4=0.88%, 10=18.36%, 20=17.35%, 50=44.24%, 100=13.67% lat (msec) : 250=5.44%, 500=0.03%, 1000=0.03% cpu : usr=0.08%, sys=5.85%, ctx=4221, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157530: Mon Jul 14 12:44:40 2025 read: IOPS=14, BW=56.7KiB/s (58.0kB/s)(6052KiB/106797msec) clat (msec): min=2, max=223, avg=17.99, stdev=18.21 lat (msec): min=2, max=223, avg=18.00, stdev=18.21 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 16], | 70.00th=[ 19], 80.00th=[ 24], 90.00th=[ 40], 95.00th=[ 54], | 99.00th=[ 80], 99.50th=[ 97], 99.90th=[ 197], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 7, max= 160, per=25.97%, avg=57.91, stdev=36.45, samples=207 iops : min= 1, max= 40, avg=14.13, stdev= 9.22, samples=207 write: IOPS=14, BW=58.4KiB/s (59.8kB/s)(6236KiB/106797msec); 0 zone resets clat (msec): min=14, max=767, avg=51.00, stdev=37.69 lat (msec): min=14, max=767, avg=51.01, stdev=37.69 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 22], 10.00th=[ 24], 20.00th=[ 27], | 30.00th=[ 31], 40.00th=[ 35], 50.00th=[ 39], 60.00th=[ 45], | 70.00th=[ 53], 80.00th=[ 70], 90.00th=[ 99], 95.00th=[ 118], | 99.00th=[ 169], 99.50th=[ 188], 99.90th=[ 259], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 7, max= 128, per=25.08%, avg=58.44, stdev=31.23, samples=211 iops : min= 1, max= 32, avg=14.26, stdev= 7.96, samples=211 lat (msec) : 4=1.07%, 10=18.65%, 20=17.97%, 50=43.07%, 100=14.00% lat (msec) : 250=5.14%, 500=0.07%, 1000=0.03% cpu : usr=0.09%, sys=5.80%, ctx=4169, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=223KiB/s (229kB/s), 55.3KiB/s-56.7KiB/s (56.6kB/s-58.0kB/s), io=23.4MiB (24.6MB), run=106797-107593msec WRITE: bw=234KiB/s (239kB/s), 58.2KiB/s-59.1KiB/s (59.6kB/s-60.5kB/s), io=24.6MiB (25.7MB), run=106797-107593msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=157561: Mon Jul 14 12:44:58 2025 read: IOPS=31, BW=508KiB/s (520kB/s)(5920KiB/11665msec) clat (usec): min=88, max=59974, avg=15129.80, stdev=10320.97 lat (usec): min=89, max=59976, avg=15130.82, stdev=10321.03 clat percentiles (usec): | 1.00th=[ 176], 5.00th=[ 289], 10.00th=[ 355], 20.00th=[ 5997], | 30.00th=[10683], 40.00th=[13173], 50.00th=[15008], 60.00th=[16581], | 70.00th=[19530], 80.00th=[22414], 90.00th=[27657], 95.00th=[30802], | 99.00th=[53216], 99.50th=[56361], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 256, max= 795, per=25.25%, avg=509.09, stdev=148.17, samples=23 iops : min= 16, max= 49, avg=31.35, stdev= 9.21, samples=23 write: IOPS=34, BW=546KiB/s (559kB/s)(6368KiB/11665msec); 0 zone resets clat (usec): min=1579, max=82408, avg=15101.35, stdev=12287.12 lat (usec): min=1581, max=82411, avg=15103.17, stdev=12287.22 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2737], 10.00th=[ 3458], 20.00th=[ 4490], | 30.00th=[ 5473], 40.00th=[ 8225], 50.00th=[11600], 60.00th=[15664], | 70.00th=[19006], 80.00th=[25297], 90.00th=[33162], 95.00th=[40109], | 99.00th=[51643], 99.50th=[52167], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 224, max= 766, per=25.75%, avg=546.13, stdev=113.19, samples=23 iops : min= 14, max= 47, avg=33.65, stdev= 7.13, samples=23 lat (usec) : 100=0.13%, 250=1.30%, 500=6.51%, 750=0.65%, 1000=0.13% lat (msec) : 2=0.65%, 4=7.29%, 10=19.79%, 20=35.55%, 50=26.95% lat (msec) : 100=1.04% cpu : usr=0.10%, sys=29.98%, ctx=1895, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157562: Mon Jul 14 12:44:58 2025 read: IOPS=30, BW=484KiB/s (495kB/s)(5744KiB/11880msec) clat (usec): min=131, max=64201, avg=16309.30, stdev=10368.83 lat (usec): min=132, max=64202, avg=16310.31, stdev=10368.88 clat percentiles (usec): | 1.00th=[ 155], 5.00th=[ 255], 10.00th=[ 388], 20.00th=[ 9110], | 30.00th=[11469], 40.00th=[13435], 50.00th=[15533], 60.00th=[18220], | 70.00th=[21103], 80.00th=[25035], 90.00th=[28443], 95.00th=[33817], | 99.00th=[43779], 99.50th=[55313], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 254, max= 704, per=23.91%, avg=482.00, stdev=130.59, samples=23 iops : min= 15, max= 44, avg=29.65, stdev= 8.11, samples=23 write: IOPS=34, BW=551KiB/s (564kB/s)(6544KiB/11880msec); 0 zone resets clat (usec): min=1306, max=66873, avg=14656.73, stdev=12185.36 lat (usec): min=1308, max=66876, avg=14659.01, stdev=12185.19 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 3064], 10.00th=[ 3621], 20.00th=[ 4424], | 30.00th=[ 5407], 40.00th=[ 7373], 50.00th=[10945], 60.00th=[14877], | 70.00th=[18744], 80.00th=[23987], 90.00th=[31327], 95.00th=[38011], | 99.00th=[58459], 99.50th=[61604], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 319, max= 736, per=25.72%, avg=545.57, stdev=112.94, samples=23 iops : min= 19, max= 46, avg=33.61, stdev= 7.17, samples=23 lat (usec) : 250=1.95%, 500=4.17%, 750=0.39%, 1000=0.13% lat (msec) : 2=1.43%, 4=6.51%, 10=20.83%, 20=34.11%, 50=29.43% lat (msec) : 100=1.04% cpu : usr=0.18%, sys=29.35%, ctx=2071, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157563: Mon Jul 14 12:44:58 2025 read: IOPS=32, BW=527KiB/s (540kB/s)(6256KiB/11864msec) clat (usec): min=117, max=110804, avg=16172.27, stdev=12547.42 lat (usec): min=118, max=110805, avg=16173.35, stdev=12547.47 clat percentiles (usec): | 1.00th=[ 143], 5.00th=[ 277], 10.00th=[ 367], 20.00th=[ 824], | 30.00th=[ 10683], 40.00th=[ 12518], 50.00th=[ 15008], 60.00th=[ 17957], | 70.00th=[ 21103], 80.00th=[ 24773], 90.00th=[ 30802], 95.00th=[ 37487], | 99.00th=[ 54264], 99.50th=[ 64226], 99.90th=[110625], 99.95th=[110625], | 99.99th=[110625] bw ( KiB/s): min= 351, max= 736, per=26.38%, avg=531.83, stdev=101.59, samples=23 iops : min= 21, max= 46, avg=32.74, stdev= 6.54, samples=23 write: IOPS=31, BW=508KiB/s (521kB/s)(6032KiB/11864msec); 0 zone resets clat (usec): min=1202, max=65698, avg=14479.31, stdev=10726.76 lat (usec): min=1205, max=65699, avg=14481.11, stdev=10726.77 clat percentiles (usec): | 1.00th=[ 1500], 5.00th=[ 2540], 10.00th=[ 3523], 20.00th=[ 4621], | 30.00th=[ 6128], 40.00th=[ 9372], 50.00th=[12387], 60.00th=[15401], | 70.00th=[19268], 80.00th=[22152], 90.00th=[29492], 95.00th=[34866], | 99.00th=[46400], 99.50th=[61080], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 288, max= 704, per=23.78%, avg=504.43, stdev=113.25, samples=23 iops : min= 18, max= 44, avg=31.04, stdev= 7.10, samples=23 lat (usec) : 250=1.82%, 500=6.90%, 750=1.30%, 1000=0.39% lat (msec) : 2=1.30%, 4=5.86%, 10=16.54%, 20=36.07%, 50=28.91% lat (msec) : 100=0.78%, 250=0.13% cpu : usr=0.13%, sys=28.85%, ctx=2020, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157564: Mon Jul 14 12:44:58 2025 read: IOPS=32, BW=525KiB/s (538kB/s)(6032KiB/11487msec) clat (usec): min=106, max=62779, avg=15386.44, stdev=10781.53 lat (usec): min=106, max=62781, avg=15387.44, stdev=10781.60 clat percentiles (usec): | 1.00th=[ 178], 5.00th=[ 289], 10.00th=[ 367], 20.00th=[ 742], | 30.00th=[10683], 40.00th=[12911], 50.00th=[14877], 60.00th=[17433], | 70.00th=[20055], 80.00th=[23725], 90.00th=[28967], 95.00th=[34341], | 99.00th=[43779], 99.50th=[44827], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 190, max= 864, per=25.81%, avg=520.27, stdev=134.69, samples=22 iops : min= 11, max= 54, avg=32.05, stdev= 8.63, samples=22 write: IOPS=34, BW=545KiB/s (558kB/s)(6256KiB/11487msec); 0 zone resets clat (usec): min=1185, max=84747, avg=14350.81, stdev=12234.50 lat (usec): min=1187, max=84749, avg=14352.63, stdev=12234.54 clat percentiles (usec): | 1.00th=[ 1467], 5.00th=[ 2180], 10.00th=[ 3228], 20.00th=[ 4359], | 30.00th=[ 5604], 40.00th=[ 7504], 50.00th=[10552], 60.00th=[14222], | 70.00th=[17433], 80.00th=[22938], 90.00th=[32637], 95.00th=[36439], | 99.00th=[51119], 99.50th=[82314], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 413, max= 768, per=25.80%, avg=547.14, stdev=79.08, samples=22 iops : min= 25, max= 48, avg=33.73, stdev= 5.01, samples=22 lat (usec) : 250=1.56%, 500=5.86%, 750=2.47%, 1000=0.13% lat (msec) : 2=2.34%, 4=5.21%, 10=19.66%, 20=35.03%, 50=27.08% lat (msec) : 100=0.65% cpu : usr=0.15%, sys=29.43%, ctx=2069, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2016KiB/s (2065kB/s), 484KiB/s-527KiB/s (495kB/s-540kB/s), io=23.4MiB (24.5MB), run=11487-11880msec WRITE: bw=2121KiB/s (2172kB/s), 508KiB/s-551KiB/s (521kB/s-564kB/s), io=24.6MiB (25.8MB), run=11487-11880msec rand-rw: (groupid=0, jobs=1): err= 0: pid=157566: Mon Jul 14 12:45:09 2025 read: IOPS=16, BW=263KiB/s (270kB/s)(5920KiB/22489msec) clat (msec): min=3, max=109, avg=17.16, stdev=16.07 lat (msec): min=3, max=109, avg=17.16, stdev=16.07 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 10], 50.00th=[ 13], 60.00th=[ 16], | 70.00th=[ 19], 80.00th=[ 24], 90.00th=[ 34], 95.00th=[ 51], | 99.00th=[ 83], 99.50th=[ 110], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 63, max= 542, per=23.92%, avg=254.73, stdev=121.43, samples=44 iops : min= 3, max= 33, avg=15.52, stdev= 7.69, samples=44 write: IOPS=17, BW=283KiB/s (290kB/s)(6368KiB/22489msec); 0 zone resets clat (msec): min=13, max=142, avg=40.41, stdev=22.15 lat (msec): min=13, max=142, avg=40.41, stdev=22.15 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 21], 10.00th=[ 22], 20.00th=[ 25], | 30.00th=[ 28], 40.00th=[ 31], 50.00th=[ 34], 60.00th=[ 37], | 70.00th=[ 43], 80.00th=[ 52], 90.00th=[ 70], 95.00th=[ 88], | 99.00th=[ 130], 99.50th=[ 140], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 95, max= 544, per=24.85%, avg=278.32, stdev=136.87, samples=44 iops : min= 5, max= 34, avg=17.00, stdev= 8.78, samples=44 lat (msec) : 4=1.04%, 10=18.62%, 20=19.01%, 50=47.92%, 100=11.59% lat (msec) : 250=1.82% cpu : usr=0.10%, sys=8.60%, ctx=1140, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157567: Mon Jul 14 12:45:09 2025 read: IOPS=16, BW=257KiB/s (263kB/s)(5744KiB/22341msec) clat (msec): min=3, max=107, avg=16.96, stdev=16.01 lat (msec): min=3, max=107, avg=16.96, stdev=16.01 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 6], | 30.00th=[ 8], 40.00th=[ 10], 50.00th=[ 12], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 23], 90.00th=[ 39], 95.00th=[ 57], | 99.00th=[ 71], 99.50th=[ 90], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 26, max= 672, per=24.54%, avg=261.37, stdev=152.60, samples=43 iops : min= 1, max= 42, avg=15.95, stdev= 9.71, samples=43 write: IOPS=18, BW=293KiB/s (300kB/s)(6544KiB/22341msec); 0 zone resets clat (msec): min=16, max=144, avg=39.31, stdev=21.80 lat (msec): min=16, max=144, avg=39.31, stdev=21.80 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 20], 10.00th=[ 22], 20.00th=[ 24], | 30.00th=[ 27], 40.00th=[ 29], 50.00th=[ 32], 60.00th=[ 36], | 70.00th=[ 43], 80.00th=[ 51], 90.00th=[ 68], 95.00th=[ 88], | 99.00th=[ 117], 99.50th=[ 126], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 96, max= 576, per=26.00%, avg=291.25, stdev=136.94, samples=44 iops : min= 6, max= 36, avg=17.82, stdev= 8.76, samples=44 lat (msec) : 4=0.91%, 10=19.79%, 20=18.10%, 50=47.53%, 100=11.85% lat (msec) : 250=1.82% cpu : usr=0.09%, sys=8.86%, ctx=1186, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157568: Mon Jul 14 12:45:09 2025 read: IOPS=17, BW=283KiB/s (290kB/s)(6256KiB/22082msec) clat (usec): min=3742, max=96949, avg=15483.75, stdev=12508.85 lat (usec): min=3743, max=96950, avg=15484.97, stdev=12508.83 clat percentiles (usec): | 1.00th=[ 3884], 5.00th=[ 4621], 10.00th=[ 5211], 20.00th=[ 6325], | 30.00th=[ 7767], 40.00th=[ 9896], 50.00th=[12125], 60.00th=[14091], | 70.00th=[17433], 80.00th=[22152], 90.00th=[28705], 95.00th=[39584], | 99.00th=[64226], 99.50th=[82314], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 64, max= 768, per=26.80%, avg=285.44, stdev=160.79, samples=43 iops : min= 4, max= 48, avg=17.44, stdev=10.18, samples=43 write: IOPS=17, BW=273KiB/s (280kB/s)(6032KiB/22082msec); 0 zone resets clat (msec): min=16, max=147, avg=42.15, stdev=21.39 lat (msec): min=16, max=147, avg=42.15, stdev=21.39 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 21], 10.00th=[ 23], 20.00th=[ 26], | 30.00th=[ 29], 40.00th=[ 32], 50.00th=[ 35], 60.00th=[ 40], | 70.00th=[ 47], 80.00th=[ 57], 90.00th=[ 75], 95.00th=[ 87], | 99.00th=[ 112], 99.50th=[ 118], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 105, max= 544, per=24.17%, avg=270.65, stdev=120.11, samples=43 iops : min= 6, max= 34, avg=16.51, stdev= 7.72, samples=43 lat (msec) : 4=0.78%, 10=20.18%, 20=19.53%, 50=44.79%, 100=13.67% lat (msec) : 250=1.04% cpu : usr=0.08%, sys=8.70%, ctx=1154, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157569: Mon Jul 14 12:45:09 2025 read: IOPS=17, BW=273KiB/s (279kB/s)(6032KiB/22112msec) clat (msec): min=3, max=120, avg=15.65, stdev=13.85 lat (msec): min=3, max=120, avg=15.65, stdev=13.85 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 10], 50.00th=[ 12], 60.00th=[ 15], | 70.00th=[ 17], 80.00th=[ 22], 90.00th=[ 30], 95.00th=[ 45], | 99.00th=[ 70], 99.50th=[ 96], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 95, max= 608, per=25.27%, avg=269.12, stdev=135.56, samples=43 iops : min= 5, max= 38, avg=16.42, stdev= 8.69, samples=43 write: IOPS=17, BW=283KiB/s (290kB/s)(6256KiB/22112msec); 0 zone resets clat (msec): min=16, max=118, avg=41.12, stdev=20.60 lat (msec): min=16, max=118, avg=41.12, stdev=20.60 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 20], 10.00th=[ 22], 20.00th=[ 26], | 30.00th=[ 28], 40.00th=[ 32], 50.00th=[ 34], 60.00th=[ 39], | 70.00th=[ 46], 80.00th=[ 56], 90.00th=[ 70], 95.00th=[ 86], | 99.00th=[ 113], 99.50th=[ 120], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 105, max= 608, per=25.23%, avg=282.53, stdev=133.41, samples=43 iops : min= 6, max= 38, avg=17.26, stdev= 8.50, samples=43 lat (msec) : 4=0.39%, 10=20.18%, 20=20.44%, 50=44.66%, 100=13.15% lat (msec) : 250=1.17% cpu : usr=0.09%, sys=8.72%, ctx=1147, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1065KiB/s (1091kB/s), 257KiB/s-283KiB/s (263kB/s-290kB/s), io=23.4MiB (24.5MB), run=22082-22489msec WRITE: bw=1121KiB/s (1147kB/s), 273KiB/s-293KiB/s (280kB/s-300kB/s), io=24.6MiB (25.8MB), run=22082-22489msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=157578: Mon Jul 14 12:45:19 2025 read: IOPS=3, BW=3167KiB/s (3243kB/s)(7168KiB/2263msec) clat (usec): min=657, max=876224, avg=207843.82, stdev=298786.08 lat (usec): min=659, max=876226, avg=207845.25, stdev=298786.46 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 660], 10.00th=[ 660], 20.00th=[ 74974], | 30.00th=[ 94897], 40.00th=[ 94897], 50.00th=[131597], 60.00th=[132645], | 70.00th=[132645], 80.00th=[143655], 90.00th=[876610], 95.00th=[876610], | 99.00th=[876610], 99.50th=[876610], 99.90th=[876610], 99.95th=[876610], | 99.99th=[876610] bw ( KiB/s): min= 2000, max= 6144, per=87.55%, avg=4080.00, stdev=2072.05, samples=3 iops : min= 1, max= 6, avg= 3.67, stdev= 2.52, samples=3 write: IOPS=2, BW=2262KiB/s (2317kB/s)(5120KiB/2263msec); 0 zone resets clat (msec): min=102, max=227, avg=158.89, stdev=54.89 lat (msec): min=102, max=227, avg=158.95, stdev=54.87 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 103], 10.00th=[ 103], 20.00th=[ 103], | 30.00th=[ 111], 40.00th=[ 111], 50.00th=[ 153], 60.00th=[ 153], | 70.00th=[ 203], 80.00th=[ 203], 90.00th=[ 228], 95.00th=[ 228], | 99.00th=[ 228], 99.50th=[ 228], 99.90th=[ 228], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 2048, max= 6000, per=61.67%, avg=4024.00, stdev=2794.49, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (usec) : 750=8.33% lat (msec) : 100=16.67%, 250=66.67%, 1000=8.33% cpu : usr=0.04%, sys=47.08%, ctx=952, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157580: Mon Jul 14 12:45:19 2025 read: IOPS=0, BW=484KiB/s (495kB/s)(2048KiB/4235msec) clat (usec): min=847, max=93851, avg=47349.59, stdev=65763.49 lat (usec): min=849, max=93853, avg=47351.49, stdev=65763.77 clat percentiles (usec): | 1.00th=[ 848], 5.00th=[ 848], 10.00th=[ 848], 20.00th=[ 848], | 30.00th=[ 848], 40.00th=[ 848], 50.00th=[ 848], 60.00th=[93848], | 70.00th=[93848], 80.00th=[93848], 90.00th=[93848], 95.00th=[93848], | 99.00th=[93848], 99.50th=[93848], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 1984, max= 1996, per=42.70%, avg=1990.00, stdev= 8.49, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2418KiB/s (2476kB/s)(10.0MiB/4235msec); 0 zone resets clat (msec): min=128, max=1119, avg=409.16, stdev=359.41 lat (msec): min=129, max=1119, avg=409.27, stdev=359.39 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 129], 10.00th=[ 129], 20.00th=[ 142], | 30.00th=[ 184], 40.00th=[ 192], 50.00th=[ 192], 60.00th=[ 245], | 70.00th=[ 288], 80.00th=[ 743], 90.00th=[ 869], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 1984, max= 4096, per=56.01%, avg=3654.40, stdev=934.71, samples=5 iops : min= 1, max= 4, avg= 3.20, stdev= 1.30, samples=5 lat (usec) : 1000=8.33% lat (msec) : 100=8.33%, 250=50.00%, 500=8.33%, 750=8.33%, 1000=8.33% lat (msec) : 2000=8.33% cpu : usr=0.02%, sys=30.47%, ctx=446, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157582: Mon Jul 14 12:45:19 2025 read: IOPS=1, BW=1398KiB/s (1432kB/s)(6144KiB/4394msec) clat (msec): min=95, max=1238, avg=313.29, stdev=454.34 lat (msec): min=95, max=1238, avg=313.29, stdev=454.34 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 96], 10.00th=[ 96], 20.00th=[ 103], | 30.00th=[ 103], 40.00th=[ 126], 50.00th=[ 126], 60.00th=[ 142], | 70.00th=[ 176], 80.00th=[ 176], 90.00th=[ 1234], 95.00th=[ 1234], | 99.00th=[ 1234], 99.50th=[ 1234], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 3968, max= 4096, per=86.52%, avg=4032.00, stdev=90.51, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 write: IOPS=1, BW=1398KiB/s (1432kB/s)(6144KiB/4394msec); 0 zone resets clat (msec): min=135, max=1417, avg=399.54, stdev=500.29 lat (msec): min=135, max=1417, avg=399.62, stdev=500.28 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 180], | 30.00th=[ 180], 40.00th=[ 205], 50.00th=[ 205], 60.00th=[ 211], | 70.00th=[ 249], 80.00th=[ 249], 90.00th=[ 1418], 95.00th=[ 1418], | 99.00th=[ 1418], 99.50th=[ 1418], 99.90th=[ 1418], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 1984, max= 4031, per=38.56%, avg=2515.75, stdev=1010.53, samples=4 iops : min= 1, max= 3, avg= 1.75, stdev= 0.96, samples=4 lat (msec) : 100=8.33%, 250=75.00%, 2000=16.67% cpu : usr=0.00%, sys=29.27%, ctx=944, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157585: Mon Jul 14 12:45:19 2025 read: IOPS=1, BW=1945KiB/s (1991kB/s)(5120KiB/2633msec) clat (usec): min=732, max=210550, avg=70076.99, stdev=98351.30 lat (usec): min=733, max=210551, avg=70077.98, stdev=98351.19 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 734], 10.00th=[ 734], 20.00th=[ 734], | 30.00th=[ 758], 40.00th=[ 758], 50.00th=[ 824], 60.00th=[ 824], | 70.00th=[137364], 80.00th=[137364], 90.00th=[210764], 95.00th=[210764], | 99.00th=[210764], 99.50th=[210764], 99.90th=[210764], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 2048, max= 6000, per=86.35%, avg=4024.00, stdev=2794.49, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=2, BW=2722KiB/s (2788kB/s)(7168KiB/2633msec); 0 zone resets clat (msec): min=77, max=924, avg=315.83, stdev=287.48 lat (msec): min=77, max=924, avg=315.90, stdev=287.48 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 78], 10.00th=[ 78], 20.00th=[ 127], | 30.00th=[ 190], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 326], | 70.00th=[ 326], 80.00th=[ 368], 90.00th=[ 927], 95.00th=[ 927], | 99.00th=[ 927], 99.50th=[ 927], 99.90th=[ 927], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 2000, max= 6144, per=54.17%, avg=3534.75, stdev=1968.47, samples=4 iops : min= 1, max= 6, avg= 2.75, stdev= 2.36, samples=4 lat (usec) : 750=8.33%, 1000=16.67% lat (msec) : 100=8.33%, 250=41.67%, 500=16.67%, 1000=8.33% cpu : usr=0.00%, sys=32.94%, ctx=333, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4661KiB/s (4773kB/s), 484KiB/s-3167KiB/s (495kB/s-3243kB/s), io=20.0MiB (20.0MB), run=2263-4394msec WRITE: bw=6525KiB/s (6682kB/s), 1398KiB/s-2722KiB/s (1432kB/s-2788kB/s), io=28.0MiB (29.4MB), run=2263-4394msec rand-rw: (groupid=0, jobs=1): err= 0: pid=157579: Mon Jul 14 12:45:19 2025 read: IOPS=1, BW=1624KiB/s (1663kB/s)(7168KiB/4414msec) clat (msec): min=23, max=1259, avg=353.27, stdev=487.56 lat (msec): min=23, max=1259, avg=353.27, stdev=487.56 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 38], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 134], 60.00th=[ 144], | 70.00th=[ 144], 80.00th=[ 818], 90.00th=[ 1267], 95.00th=[ 1267], | 99.00th=[ 1267], 99.50th=[ 1267], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 2048, max= 4096, per=72.77%, avg=3376.00, stdev=1151.44, samples=3 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=3 write: IOPS=1, BW=1160KiB/s (1188kB/s)(5120KiB/4414msec); 0 zone resets clat (msec): min=88, max=1376, avg=364.36, stdev=566.30 lat (msec): min=88, max=1376, avg=364.44, stdev=566.31 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 88], 10.00th=[ 88], 20.00th=[ 88], | 30.00th=[ 93], 40.00th=[ 93], 50.00th=[ 124], 60.00th=[ 124], | 70.00th=[ 142], 80.00th=[ 142], 90.00th=[ 1385], 95.00th=[ 1385], | 99.00th=[ 1385], 99.50th=[ 1385], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 6144, max= 6144, per=94.60%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 50=16.67%, 100=25.00%, 250=33.33%, 1000=8.33%, 2000=16.67% cpu : usr=0.00%, sys=10.29%, ctx=101, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157581: Mon Jul 14 12:45:19 2025 read: IOPS=0, BW=507KiB/s (520kB/s)(2048KiB/4036msec) clat (msec): min=55, max=145, avg=100.42, stdev=63.44 lat (msec): min=55, max=145, avg=100.42, stdev=63.44 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 56], 10.00th=[ 56], 20.00th=[ 56], | 30.00th=[ 56], 40.00th=[ 56], 50.00th=[ 56], 60.00th=[ 146], | 70.00th=[ 146], 80.00th=[ 146], 90.00th=[ 146], 95.00th=[ 146], | 99.00th=[ 146], 99.50th=[ 146], 99.90th=[ 146], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 1992, max= 2007, per=43.10%, avg=1999.50, stdev=10.61, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2537KiB/s (2598kB/s)(10.0MiB/4036msec); 0 zone resets clat (msec): min=71, max=1033, avg=381.99, stdev=375.80 lat (msec): min=71, max=1033, avg=382.08, stdev=375.80 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 72], 10.00th=[ 72], 20.00th=[ 78], | 30.00th=[ 107], 40.00th=[ 117], 50.00th=[ 176], 60.00th=[ 264], | 70.00th=[ 300], 80.00th=[ 701], 90.00th=[ 969], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 1946, max= 5919, per=53.71%, avg=3488.25, stdev=1905.71, samples=4 iops : min= 1, max= 5, avg= 2.75, stdev= 2.06, samples=4 lat (msec) : 100=25.00%, 250=33.33%, 500=16.67%, 750=8.33%, 1000=8.33% lat (msec) : 2000=8.33% cpu : usr=0.02%, sys=13.93%, ctx=72, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157583: Mon Jul 14 12:45:19 2025 read: IOPS=1, BW=1670KiB/s (1710kB/s)(6144KiB/3679msec) clat (msec): min=74, max=605, avg=255.14, stdev=186.50 lat (msec): min=74, max=605, avg=255.14, stdev=186.50 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 74], 20.00th=[ 125], | 30.00th=[ 125], 40.00th=[ 228], 50.00th=[ 228], 60.00th=[ 239], | 70.00th=[ 262], 80.00th=[ 262], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 2048, max= 3984, per=71.32%, avg=3308.33, stdev=1092.43, samples=3 iops : min= 2, max= 3, avg= 2.67, stdev= 0.58, samples=3 write: IOPS=1, BW=1670KiB/s (1710kB/s)(6144KiB/3679msec); 0 zone resets clat (msec): min=110, max=1098, avg=340.66, stdev=377.57 lat (msec): min=110, max=1098, avg=340.76, stdev=377.57 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 180], 50.00th=[ 180], 60.00th=[ 218], | 70.00th=[ 305], 80.00th=[ 305], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 1973, max= 4031, per=37.23%, avg=2418.40, stdev=902.09, samples=5 iops : min= 1, max= 3, avg= 1.80, stdev= 0.84, samples=5 lat (msec) : 100=8.33%, 250=58.33%, 500=16.67%, 750=8.33%, 2000=8.33% cpu : usr=0.00%, sys=13.10%, ctx=136, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157584: Mon Jul 14 12:45:19 2025 read: IOPS=1, BW=1220KiB/s (1249kB/s)(5120KiB/4196msec) clat (msec): min=69, max=409, avg=188.96, stdev=137.38 lat (msec): min=69, max=409, avg=188.96, stdev=137.38 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 69], 10.00th=[ 69], 20.00th=[ 69], | 30.00th=[ 112], 40.00th=[ 112], 50.00th=[ 122], 60.00th=[ 122], | 70.00th=[ 234], 80.00th=[ 234], 90.00th=[ 409], 95.00th=[ 409], | 99.00th=[ 409], 99.50th=[ 409], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 3984, max= 4096, per=87.09%, avg=4040.00, stdev=79.20, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 write: IOPS=1, BW=1708KiB/s (1749kB/s)(7168KiB/4196msec); 0 zone resets clat (msec): min=44, max=1205, avg=449.83, stdev=441.89 lat (msec): min=44, max=1205, avg=449.92, stdev=441.88 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 82], | 30.00th=[ 192], 40.00th=[ 192], 50.00th=[ 317], 60.00th=[ 397], | 70.00th=[ 397], 80.00th=[ 911], 90.00th=[ 1200], 95.00th=[ 1200], | 99.00th=[ 1200], 99.50th=[ 1200], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 1992, max= 3893, per=36.96%, avg=2400.80, stdev=834.48, samples=5 iops : min= 1, max= 3, avg= 1.80, stdev= 0.84, samples=5 lat (msec) : 50=8.33%, 100=16.67%, 250=33.33%, 500=25.00%, 1000=8.33% lat (msec) : 2000=8.33% cpu : usr=0.00%, sys=11.32%, ctx=213, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4640KiB/s (4751kB/s), 507KiB/s-1670KiB/s (520kB/s-1710kB/s), io=20.0MiB (20.0MB), run=3679-4414msec WRITE: bw=6496KiB/s (6652kB/s), 1160KiB/s-2537KiB/s (1188kB/s-2598kB/s), io=28.0MiB (29.4MB), run=3679-4414msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=157591: Mon Jul 14 12:45:29 2025 read: IOPS=1, BW=6153KiB/s (6301kB/s)(12.0MiB/1997msec) clat (msec): min=374, max=842, avg=634.24, stdev=238.16 lat (msec): min=374, max=842, avg=634.24, stdev=238.16 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 376], 10.00th=[ 376], 20.00th=[ 376], | 30.00th=[ 376], 40.00th=[ 684], 50.00th=[ 684], 60.00th=[ 684], | 70.00th=[ 844], 80.00th=[ 844], 90.00th=[ 844], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 7876, max= 8062, per=100.00%, avg=7969.00, stdev=131.52, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=49.50%, ctx=1718, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157593: Mon Jul 14 12:45:29 2025 read: IOPS=0, BW=1111KiB/s (1137kB/s)(4096KiB/3688msec) clat (nsec): min=371690k, max=371690k, avg=371689706.00, stdev= 0.00 lat (nsec): min=371691k, max=371691k, avg=371690587.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 372], 10.00th=[ 372], 20.00th=[ 372], | 30.00th=[ 372], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 372], | 70.00th=[ 372], 80.00th=[ 372], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 7831, max= 7831, per=100.00%, avg=7831.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2221KiB/s (2275kB/s)(8192KiB/3688msec); 0 zone resets clat (msec): min=1222, max=1995, avg=1609.06, stdev=546.40 lat (msec): min=1222, max=1995, avg=1609.38, stdev=546.46 clat percentiles (msec): | 1.00th=[ 1217], 5.00th=[ 1217], 10.00th=[ 1217], 20.00th=[ 1217], | 30.00th=[ 1217], 40.00th=[ 1217], 50.00th=[ 1217], 60.00th=[ 1989], | 70.00th=[ 1989], 80.00th=[ 1989], 90.00th=[ 1989], 95.00th=[ 1989], | 99.00th=[ 1989], 99.50th=[ 1989], 99.90th=[ 1989], 99.95th=[ 1989], | 99.99th=[ 1989] bw ( KiB/s): min= 8192, max= 8192, per=92.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 2000=66.67% cpu : usr=0.00%, sys=29.08%, ctx=799, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157594: Mon Jul 14 12:45:29 2025 write: IOPS=0, BW=3793KiB/s (3884kB/s)(12.0MiB/3240msec); 0 zone resets clat (msec): min=616, max=1728, avg=1052.35, stdev=594.00 lat (msec): min=616, max=1729, avg=1052.72, stdev=594.17 clat percentiles (msec): | 1.00th=[ 617], 5.00th=[ 617], 10.00th=[ 617], 20.00th=[ 617], | 30.00th=[ 617], 40.00th=[ 810], 50.00th=[ 810], 60.00th=[ 810], | 70.00th=[ 1737], 80.00th=[ 1737], 90.00th=[ 1737], 95.00th=[ 1737], | 99.00th=[ 1737], 99.50th=[ 1737], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 8062, max= 8143, per=91.19%, avg=8102.50, stdev=57.28, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 750=33.33%, 1000=33.33%, 2000=33.33% cpu : usr=0.06%, sys=42.73%, ctx=137, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157595: Mon Jul 14 12:45:29 2025 write: IOPS=1, BW=4713KiB/s (4827kB/s)(12.0MiB/2607msec); 0 zone resets clat (msec): min=568, max=993, avg=836.96, stdev=233.83 lat (msec): min=568, max=994, avg=837.33, stdev=233.85 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 567], 10.00th=[ 567], 20.00th=[ 567], | 30.00th=[ 567], 40.00th=[ 953], 50.00th=[ 953], 60.00th=[ 953], | 70.00th=[ 995], 80.00th=[ 995], 90.00th=[ 995], 95.00th=[ 995], | 99.00th=[ 995], 99.50th=[ 995], 99.90th=[ 995], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 8062, max= 8192, per=91.47%, avg=8127.00, stdev=91.92, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 750=33.33%, 1000=66.67% cpu : usr=0.04%, sys=58.02%, ctx=212, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4443KiB/s (4549kB/s), 1111KiB/s-6153KiB/s (1137kB/s-6301kB/s), io=16.0MiB (16.8MB), run=1997-3688msec WRITE: bw=8885KiB/s (9098kB/s), 2221KiB/s-4713KiB/s (2275kB/s-4827kB/s), io=32.0MiB (33.6MB), run=2607-3688msec rand-rw: (groupid=0, jobs=1): err= 0: pid=157596: Mon Jul 14 12:45:29 2025 read: IOPS=0, BW=3452KiB/s (3535kB/s)(12.0MiB/3560msec) clat (msec): min=1108, max=1297, avg=1181.76, stdev=101.48 lat (msec): min=1108, max=1297, avg=1181.76, stdev=101.48 clat percentiles (msec): | 1.00th=[ 1116], 5.00th=[ 1116], 10.00th=[ 1116], 20.00th=[ 1116], | 30.00th=[ 1116], 40.00th=[ 1133], 50.00th=[ 1133], 60.00th=[ 1133], | 70.00th=[ 1301], 80.00th=[ 1301], 90.00th=[ 1301], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 7613, max= 8175, per=100.00%, avg=7894.00, stdev=397.39, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 2000=100.00% cpu : usr=0.00%, sys=9.53%, ctx=33, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157597: Mon Jul 14 12:45:29 2025 read: IOPS=0, BW=1055KiB/s (1080kB/s)(4096KiB/3882msec) clat (nsec): min=1323.8M, max=1323.8M, avg=1323772618.00, stdev= 0.00 lat (nsec): min=1323.8M, max=1323.8M, avg=1323775381.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1318], 5.00th=[ 1318], 10.00th=[ 1318], 20.00th=[ 1318], | 30.00th=[ 1318], 40.00th=[ 1318], 50.00th=[ 1318], 60.00th=[ 1318], | 70.00th=[ 1318], 80.00th=[ 1318], 90.00th=[ 1318], 95.00th=[ 1318], | 99.00th=[ 1318], 99.50th=[ 1318], 99.90th=[ 1318], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 7488, max= 7488, per=100.00%, avg=7488.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2110KiB/s (2161kB/s)(8192KiB/3882msec); 0 zone resets clat (msec): min=1236, max=1288, avg=1262.23, stdev=37.02 lat (msec): min=1236, max=1288, avg=1262.59, stdev=37.11 clat percentiles (msec): | 1.00th=[ 1234], 5.00th=[ 1234], 10.00th=[ 1234], 20.00th=[ 1234], | 30.00th=[ 1234], 40.00th=[ 1234], 50.00th=[ 1234], 60.00th=[ 1284], | 70.00th=[ 1284], 80.00th=[ 1284], 90.00th=[ 1284], 95.00th=[ 1284], | 99.00th=[ 1284], 99.50th=[ 1284], 99.90th=[ 1284], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 8175, max= 8175, per=96.85%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 2000=100.00% cpu : usr=0.00%, sys=11.26%, ctx=75, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157598: Mon Jul 14 12:45:29 2025 write: IOPS=0, BW=3298KiB/s (3377kB/s)(12.0MiB/3726msec); 0 zone resets clat (msec): min=589, max=1633, avg=1229.39, stdev=560.43 lat (msec): min=589, max=1633, avg=1229.86, stdev=560.48 clat percentiles (msec): | 1.00th=[ 592], 5.00th=[ 592], 10.00th=[ 592], 20.00th=[ 592], | 30.00th=[ 592], 40.00th=[ 1469], 50.00th=[ 1469], 60.00th=[ 1469], | 70.00th=[ 1636], 80.00th=[ 1636], 90.00th=[ 1636], 95.00th=[ 1636], | 99.00th=[ 1636], 99.50th=[ 1636], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 7488, max= 8175, per=92.78%, avg=7831.50, stdev=485.78, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 750=33.33%, 2000=66.67% cpu : usr=0.03%, sys=10.66%, ctx=89, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=157599: Mon Jul 14 12:45:29 2025 write: IOPS=0, BW=3944KiB/s (4038kB/s)(12.0MiB/3116msec); 0 zone resets clat (msec): min=496, max=1540, avg=1031.97, stdev=522.14 lat (msec): min=497, max=1540, avg=1032.46, stdev=522.13 clat percentiles (msec): | 1.00th=[ 498], 5.00th=[ 498], 10.00th=[ 498], 20.00th=[ 498], | 30.00th=[ 498], 40.00th=[ 1062], 50.00th=[ 1062], 60.00th=[ 1062], | 70.00th=[ 1536], 80.00th=[ 1536], 90.00th=[ 1536], 95.00th=[ 1536], | 99.00th=[ 1536], 99.50th=[ 1536], 99.90th=[ 1536], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 7488, max= 8175, per=92.78%, avg=7831.50, stdev=485.78, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 2000=66.67% cpu : usr=0.03%, sys=18.36%, ctx=120, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4221KiB/s (4322kB/s), 1055KiB/s-3452KiB/s (1080kB/s-3535kB/s), io=16.0MiB (16.8MB), run=3560-3882msec WRITE: bw=8441KiB/s (8644kB/s), 2110KiB/s-3944KiB/s (2161kB/s-4038kB/s), io=32.0MiB (33.6MB), run=3116-3882msec