== sanity test 398b: DIO and buffer IO race ============== 18:12:32 (1756678352) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB, 48 MiB) copied, 2.94252 s, 17.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=194153: Sun Aug 31 18:13:31 2025 read: IOPS=30, BW=121KiB/s (123kB/s)(5992KiB/49689msec) clat (usec): min=81, max=635972, avg=18442.50, stdev=18968.34 lat (usec): min=81, max=635972, avg=18443.00, stdev=18968.34 clat percentiles (usec): | 1.00th=[ 347], 5.00th=[ 6849], 10.00th=[ 8225], 20.00th=[ 10028], | 30.00th=[ 11600], 40.00th=[ 13698], 50.00th=[ 16057], 60.00th=[ 18220], | 70.00th=[ 21103], 80.00th=[ 24773], 90.00th=[ 32375], 95.00th=[ 38011], | 99.00th=[ 54264], 99.50th=[ 60031], 99.90th=[ 74974], 99.95th=[633340], | 99.99th=[633340] bw ( KiB/s): min= 13, max= 207, per=25.26%, avg=120.24, stdev=32.98, samples=98 iops : min= 3, max= 51, avg=29.72, stdev= 8.33, samples=98 write: IOPS=31, BW=127KiB/s (130kB/s)(6296KiB/49689msec); 0 zone resets clat (usec): min=628, max=83407, avg=13895.20, stdev=11227.63 lat (usec): min=628, max=83408, avg=13895.81, stdev=11227.69 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 2114], 10.00th=[ 2737], 20.00th=[ 3523], | 30.00th=[ 5800], 40.00th=[ 8979], 50.00th=[11338], 60.00th=[13698], | 70.00th=[17957], 80.00th=[22676], 90.00th=[28967], 95.00th=[35390], | 99.00th=[49021], 99.50th=[59507], 99.90th=[82314], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 55, max= 208, per=25.47%, avg=127.11, stdev=42.42, samples=97 iops : min= 13, max= 52, avg=31.44, stdev=10.65, samples=97 lat (usec) : 100=0.07%, 250=0.10%, 500=0.88%, 750=0.85%, 1000=0.65% lat (msec) : 2=1.24%, 4=9.83%, 10=18.72%, 20=38.22%, 50=28.45% lat (msec) : 100=0.98%, 750=0.03% cpu : usr=0.07%, sys=26.64%, ctx=8385, majf=0, minf=17 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194154: Sun Aug 31 18:13:31 2025 read: IOPS=29, BW=119KiB/s (122kB/s)(5940KiB/50046msec) clat (usec): min=82, max=107414, avg=18028.56, stdev=9617.26 lat (usec): min=82, max=107414, avg=18029.14, stdev=9617.32 clat percentiles (usec): | 1.00th=[ 404], 5.00th=[ 7373], 10.00th=[ 8586], 20.00th=[ 10290], | 30.00th=[ 12387], 40.00th=[ 14353], 50.00th=[ 16450], 60.00th=[ 18744], | 70.00th=[ 20841], 80.00th=[ 23987], 90.00th=[ 30016], 95.00th=[ 36439], | 99.00th=[ 49021], 99.50th=[ 53740], 99.90th=[ 74974], 99.95th=[107480], | 99.99th=[107480] bw ( KiB/s): min= 6, max= 184, per=24.87%, avg=118.39, stdev=34.33, samples=99 iops : min= 1, max= 46, avg=29.26, stdev= 8.68, samples=99 write: IOPS=31, BW=127KiB/s (130kB/s)(6348KiB/50046msec); 0 zone resets clat (usec): min=657, max=634096, avg=14544.91, stdev=19124.70 lat (usec): min=658, max=634096, avg=14545.54, stdev=19124.76 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 2278], 10.00th=[ 2802], 20.00th=[ 3589], | 30.00th=[ 6194], 40.00th=[ 9372], 50.00th=[ 11338], 60.00th=[ 14091], | 70.00th=[ 18220], 80.00th=[ 22938], 90.00th=[ 29492], 95.00th=[ 35914], | 99.00th=[ 49021], 99.50th=[ 53216], 99.90th=[ 81265], 99.95th=[633340], | 99.99th=[633340] bw ( KiB/s): min= 13, max= 240, per=25.25%, avg=125.98, stdev=41.13, samples=99 iops : min= 3, max= 60, avg=31.16, stdev=10.38, samples=99 lat (usec) : 100=0.07%, 250=0.03%, 500=0.55%, 750=0.72%, 1000=0.55% lat (msec) : 2=0.85%, 4=9.70%, 10=18.42%, 20=39.26%, 50=29.00% lat (msec) : 100=0.78%, 250=0.03%, 750=0.03% cpu : usr=0.08%, sys=26.59%, ctx=8455, majf=0, minf=18 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194155: Sun Aug 31 18:13:31 2025 read: IOPS=30, BW=121KiB/s (124kB/s)(6028KiB/49912msec) clat (usec): min=85, max=640961, avg=18231.31, stdev=18981.45 lat (usec): min=86, max=640962, avg=18231.89, stdev=18981.49 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 6980], 10.00th=[ 8356], 20.00th=[ 10028], | 30.00th=[ 11469], 40.00th=[ 13829], 50.00th=[ 16057], 60.00th=[ 18220], | 70.00th=[ 20579], 80.00th=[ 24249], 90.00th=[ 30016], 95.00th=[ 36963], | 99.00th=[ 57410], 99.50th=[ 68682], 99.90th=[ 80217], 99.95th=[641729], | 99.99th=[641729] bw ( KiB/s): min= 6, max= 223, per=25.38%, avg=120.83, stdev=38.80, samples=99 iops : min= 1, max= 55, avg=29.87, stdev= 9.78, samples=99 write: IOPS=31, BW=125KiB/s (128kB/s)(6260KiB/49912msec); 0 zone resets clat (usec): min=652, max=90247, avg=14214.67, stdev=11318.90 lat (usec): min=652, max=90248, avg=14215.39, stdev=11318.88 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 2409], 10.00th=[ 2868], 20.00th=[ 3654], | 30.00th=[ 6325], 40.00th=[ 9372], 50.00th=[11600], 60.00th=[14091], | 70.00th=[17957], 80.00th=[22938], 90.00th=[30278], 95.00th=[36439], | 99.00th=[50070], 99.50th=[53216], 99.90th=[74974], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 19, max= 216, per=25.11%, avg=125.30, stdev=38.43, samples=99 iops : min= 4, max= 54, avg=30.99, stdev= 9.68, samples=99 lat (usec) : 100=0.07%, 250=0.07%, 500=0.72%, 750=0.65%, 1000=0.49% lat (msec) : 2=0.65%, 4=9.64%, 10=19.50%, 20=39.45%, 50=27.64% lat (msec) : 100=1.11%, 750=0.03% cpu : usr=0.08%, sys=26.67%, ctx=8397, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194156: Sun Aug 31 18:13:31 2025 read: IOPS=30, BW=120KiB/s (123kB/s)(6052KiB/50352msec) clat (usec): min=98, max=676959, avg=18162.98, stdev=19517.19 lat (usec): min=98, max=676960, avg=18163.56, stdev=19517.19 clat percentiles (usec): | 1.00th=[ 404], 5.00th=[ 6783], 10.00th=[ 8291], 20.00th=[ 10028], | 30.00th=[ 11731], 40.00th=[ 13829], 50.00th=[ 15926], 60.00th=[ 17957], | 70.00th=[ 20317], 80.00th=[ 24249], 90.00th=[ 30540], 95.00th=[ 36439], | 99.00th=[ 48497], 99.50th=[ 56886], 99.90th=[ 63177], 99.95th=[675283], | 99.99th=[675283] bw ( KiB/s): min= 6, max= 231, per=25.24%, avg=120.15, stdev=39.37, samples=100 iops : min= 1, max= 57, avg=29.70, stdev= 9.89, samples=100 write: IOPS=30, BW=124KiB/s (127kB/s)(6236KiB/50352msec); 0 zone resets clat (usec): min=639, max=112401, avg=14546.38, stdev=11769.87 lat (usec): min=639, max=112402, avg=14547.00, stdev=11769.89 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 2376], 10.00th=[ 2835], 20.00th=[ 3687], | 30.00th=[ 6390], 40.00th=[ 9503], 50.00th=[ 11600], 60.00th=[ 14484], | 70.00th=[ 18220], 80.00th=[ 23725], 90.00th=[ 30540], 95.00th=[ 38011], | 99.00th=[ 52691], 99.50th=[ 57410], 99.90th=[ 67634], 99.95th=[112722], | 99.99th=[112722] bw ( KiB/s): min= 26, max= 247, per=24.79%, avg=123.71, stdev=40.25, samples=100 iops : min= 6, max= 61, avg=30.59, stdev=10.11, samples=100 lat (usec) : 100=0.03%, 250=0.03%, 500=0.88%, 750=0.75%, 1000=0.33% lat (msec) : 2=0.98%, 4=9.38%, 10=18.95%, 20=40.04%, 50=27.54% lat (msec) : 100=1.04%, 250=0.03%, 750=0.03% cpu : usr=0.06%, sys=26.37%, ctx=8437, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=477KiB/s (488kB/s), 119KiB/s-121KiB/s (122kB/s-124kB/s), io=23.4MiB (24.6MB), run=49689-50352msec WRITE: bw=499KiB/s (511kB/s), 124KiB/s-127KiB/s (127kB/s-130kB/s), io=24.6MiB (25.7MB), run=49689-50352msec rand-rw: (groupid=0, jobs=1): err= 0: pid=194158: Sun Aug 31 18:13:38 2025 read: IOPS=26, BW=104KiB/s (107kB/s)(5992KiB/57589msec) clat (msec): min=2, max=649, avg=16.55, stdev=20.83 lat (msec): min=2, max=649, avg=16.55, stdev=20.83 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 10], 50.00th=[ 12], 60.00th=[ 15], | 70.00th=[ 19], 80.00th=[ 24], 90.00th=[ 35], 95.00th=[ 42], | 99.00th=[ 61], 99.50th=[ 66], 99.90th=[ 130], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 31, max= 352, per=25.15%, avg=104.13, stdev=68.20, samples=113 iops : min= 7, max= 88, avg=25.71, stdev=17.19, samples=113 write: IOPS=27, BW=109KiB/s (112kB/s)(6296KiB/57589msec); 0 zone resets clat (msec): min=4, max=103, avg=20.76, stdev=14.54 lat (msec): min=4, max=103, avg=20.76, stdev=14.54 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 13], 50.00th=[ 17], 60.00th=[ 21], | 70.00th=[ 26], 80.00th=[ 32], 90.00th=[ 41], 95.00th=[ 49], | 99.00th=[ 67], 99.50th=[ 75], 99.90th=[ 97], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 23, max= 344, per=25.18%, avg=109.02, stdev=68.54, samples=113 iops : min= 5, max= 86, avg=26.93, stdev=17.27, samples=113 lat (msec) : 4=0.46%, 10=35.61%, 20=29.62%, 50=30.83%, 100=3.35% lat (msec) : 250=0.10%, 750=0.03% cpu : usr=0.07%, sys=13.28%, ctx=4109, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194159: Sun Aug 31 18:13:38 2025 read: IOPS=25, BW=103KiB/s (105kB/s)(5940KiB/57931msec) clat (msec): min=2, max=689, avg=16.74, stdev=22.03 lat (msec): min=2, max=689, avg=16.74, stdev=22.03 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 6], | 30.00th=[ 8], 40.00th=[ 10], 50.00th=[ 12], 60.00th=[ 15], | 70.00th=[ 19], 80.00th=[ 25], 90.00th=[ 36], 95.00th=[ 45], | 99.00th=[ 64], 99.50th=[ 68], 99.90th=[ 92], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 23, max= 352, per=24.85%, avg=102.87, stdev=72.83, samples=114 iops : min= 5, max= 88, avg=25.39, stdev=18.35, samples=114 write: IOPS=27, BW=110KiB/s (112kB/s)(6348KiB/57931msec); 0 zone resets clat (msec): min=3, max=114, avg=20.77, stdev=14.98 lat (msec): min=3, max=114, avg=20.77, stdev=14.98 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 13], 50.00th=[ 17], 60.00th=[ 22], | 70.00th=[ 26], 80.00th=[ 33], 90.00th=[ 42], 95.00th=[ 50], | 99.00th=[ 70], 99.50th=[ 80], 99.90th=[ 102], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 31, max= 352, per=25.31%, avg=109.61, stdev=71.64, samples=114 iops : min= 7, max= 88, avg=27.08, stdev=18.03, samples=114 lat (msec) : 4=1.14%, 10=36.75%, 20=27.64%, 50=30.40%, 100=3.97% lat (msec) : 250=0.07%, 750=0.03% cpu : usr=0.08%, sys=13.04%, ctx=3988, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194160: Sun Aug 31 18:13:38 2025 read: IOPS=26, BW=104KiB/s (107kB/s)(6028KiB/57847msec) clat (usec): min=2731, max=95966, avg=16722.70, stdev=13547.68 lat (usec): min=2731, max=95967, avg=16723.36, stdev=13547.73 clat percentiles (usec): | 1.00th=[ 3687], 5.00th=[ 4555], 10.00th=[ 5211], 20.00th=[ 6194], | 30.00th=[ 7373], 40.00th=[ 9634], 50.00th=[11994], 60.00th=[14746], | 70.00th=[19268], 80.00th=[26608], 90.00th=[35390], 95.00th=[44303], | 99.00th=[62653], 99.50th=[71828], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 32, max= 328, per=25.34%, avg=104.90, stdev=71.89, samples=114 iops : min= 8, max= 82, avg=25.90, stdev=18.11, samples=114 write: IOPS=27, BW=108KiB/s (111kB/s)(6260KiB/57847msec); 0 zone resets clat (msec): min=3, max=696, avg=20.78, stdev=22.60 lat (msec): min=3, max=696, avg=20.79, stdev=22.60 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 12], 50.00th=[ 16], 60.00th=[ 21], | 70.00th=[ 26], 80.00th=[ 32], 90.00th=[ 40], 95.00th=[ 48], | 99.00th=[ 69], 99.50th=[ 84], 99.90th=[ 146], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 39, max= 336, per=25.12%, avg=108.76, stdev=71.68, samples=114 iops : min= 9, max= 84, avg=26.87, stdev=18.05, samples=114 lat (msec) : 4=0.94%, 10=36.10%, 20=28.09%, 50=31.25%, 100=3.52% lat (msec) : 250=0.07%, 750=0.03% cpu : usr=0.08%, sys=13.11%, ctx=4086, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194161: Sun Aug 31 18:13:38 2025 read: IOPS=26, BW=104KiB/s (107kB/s)(6052KiB/57939msec) clat (usec): min=2664, max=89373, avg=16295.06, stdev=13204.57 lat (usec): min=2665, max=89373, avg=16295.58, stdev=13204.57 clat percentiles (usec): | 1.00th=[ 3359], 5.00th=[ 4359], 10.00th=[ 5145], 20.00th=[ 6128], | 30.00th=[ 7242], 40.00th=[ 9110], 50.00th=[11469], 60.00th=[14353], | 70.00th=[19006], 80.00th=[25297], 90.00th=[34866], 95.00th=[43779], | 99.00th=[59507], 99.50th=[66323], 99.90th=[88605], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 7, max= 320, per=25.19%, avg=104.27, stdev=69.98, samples=114 iops : min= 1, max= 80, avg=25.75, stdev=17.62, samples=114 write: IOPS=26, BW=108KiB/s (110kB/s)(6236KiB/57939msec); 0 zone resets clat (msec): min=3, max=691, avg=21.27, stdev=22.77 lat (msec): min=3, max=691, avg=21.27, stdev=22.77 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 12], 50.00th=[ 17], 60.00th=[ 22], | 70.00th=[ 27], 80.00th=[ 33], 90.00th=[ 41], 95.00th=[ 48], | 99.00th=[ 68], 99.50th=[ 84], 99.90th=[ 148], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 31, max= 352, per=24.91%, avg=107.85, stdev=73.42, samples=114 iops : min= 7, max= 88, avg=26.64, stdev=18.49, samples=114 lat (msec) : 4=1.43%, 10=36.59%, 20=26.46%, 50=31.74%, 100=3.65% lat (msec) : 250=0.10%, 750=0.03% cpu : usr=0.06%, sys=12.95%, ctx=4032, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=414KiB/s (424kB/s), 103KiB/s-104KiB/s (105kB/s-107kB/s), io=23.4MiB (24.6MB), run=57589-57939msec WRITE: bw=434KiB/s (444kB/s), 108KiB/s-110KiB/s (110kB/s-112kB/s), io=24.6MiB (25.7MB), run=57589-57939msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=194182: Sun Aug 31 18:13:57 2025 read: IOPS=27, BW=443KiB/s (454kB/s)(5920KiB/13367msec) clat (usec): min=103, max=72271, avg=18462.04, stdev=11020.05 lat (usec): min=103, max=72271, avg=18462.51, stdev=11020.06 clat percentiles (usec): | 1.00th=[ 273], 5.00th=[ 437], 10.00th=[ 8225], 20.00th=[11207], | 30.00th=[12649], 40.00th=[15139], 50.00th=[16909], 60.00th=[19006], | 70.00th=[21890], 80.00th=[25560], 90.00th=[31065], 95.00th=[39584], | 99.00th=[56886], 99.50th=[61604], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 287, max= 736, per=24.65%, avg=441.46, stdev=88.94, samples=26 iops : min= 17, max= 46, avg=27.08, stdev= 5.68, samples=26 write: IOPS=29, BW=476KiB/s (488kB/s)(6368KiB/13367msec); 0 zone resets clat (usec): min=1095, max=53700, avg=16228.15, stdev=11364.81 lat (usec): min=1096, max=53701, avg=16229.17, stdev=11365.01 clat percentiles (usec): | 1.00th=[ 1336], 5.00th=[ 3687], 10.00th=[ 4359], 20.00th=[ 5342], | 30.00th=[ 7373], 40.00th=[11338], 50.00th=[13173], 60.00th=[16188], | 70.00th=[21103], 80.00th=[27657], 90.00th=[32637], 95.00th=[37487], | 99.00th=[51119], 99.50th=[52167], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 192, max= 736, per=25.34%, avg=477.73, stdev=125.12, samples=26 iops : min= 12, max= 46, avg=29.35, stdev= 7.81, samples=26 lat (usec) : 250=0.39%, 500=2.73%, 750=0.52%, 1000=0.39% lat (msec) : 2=1.17%, 4=2.47%, 10=18.49%, 20=39.84%, 50=32.55% lat (msec) : 100=1.43% cpu : usr=0.04%, sys=28.69%, ctx=2556, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194183: Sun Aug 31 18:13:57 2025 read: IOPS=28, BW=448KiB/s (459kB/s)(5744KiB/12814msec) clat (usec): min=262, max=78998, avg=18034.01, stdev=10780.92 lat (usec): min=263, max=78998, avg=18034.47, stdev=10780.92 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 449], 10.00th=[ 8225], 20.00th=[10290], | 30.00th=[12780], 40.00th=[14091], 50.00th=[16712], 60.00th=[19006], | 70.00th=[22152], 80.00th=[24249], 90.00th=[29754], 95.00th=[35390], | 99.00th=[56361], 99.50th=[63177], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 159, max= 670, per=25.04%, avg=448.52, stdev=114.05, samples=25 iops : min= 9, max= 41, avg=27.52, stdev= 7.26, samples=25 write: IOPS=31, BW=511KiB/s (523kB/s)(6544KiB/12814msec); 0 zone resets clat (usec): min=1062, max=65781, avg=15322.15, stdev=10838.97 lat (usec): min=1062, max=65782, avg=15322.91, stdev=10838.98 clat percentiles (usec): | 1.00th=[ 1205], 5.00th=[ 3589], 10.00th=[ 4359], 20.00th=[ 5276], | 30.00th=[ 7308], 40.00th=[10159], 50.00th=[13042], 60.00th=[15270], | 70.00th=[18744], 80.00th=[24511], 90.00th=[30802], 95.00th=[37487], | 99.00th=[45351], 99.50th=[49021], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 316, max= 796, per=27.14%, avg=511.64, stdev=118.37, samples=25 iops : min= 19, max= 49, avg=31.48, stdev= 7.44, samples=25 lat (usec) : 500=2.99%, 750=0.52% lat (msec) : 2=1.17%, 4=2.73%, 10=21.74%, 20=38.54%, 50=31.12% lat (msec) : 100=1.17% cpu : usr=0.06%, sys=29.69%, ctx=2516, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194184: Sun Aug 31 18:13:57 2025 read: IOPS=29, BW=476KiB/s (487kB/s)(6256KiB/13151msec) clat (usec): min=165, max=82084, avg=17709.79, stdev=10817.33 lat (usec): min=165, max=82085, avg=17710.26, stdev=10817.34 clat percentiles (usec): | 1.00th=[ 255], 5.00th=[ 388], 10.00th=[ 8029], 20.00th=[10945], | 30.00th=[12649], 40.00th=[14484], 50.00th=[16057], 60.00th=[18482], | 70.00th=[20579], 80.00th=[22938], 90.00th=[28705], 95.00th=[39060], | 99.00th=[56361], 99.50th=[71828], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 254, max= 800, per=26.81%, avg=480.12, stdev=126.67, samples=25 iops : min= 15, max= 50, avg=29.52, stdev= 7.97, samples=25 write: IOPS=28, BW=459KiB/s (470kB/s)(6032KiB/13151msec); 0 zone resets clat (usec): min=1553, max=70357, avg=16390.37, stdev=11587.11 lat (usec): min=1554, max=70358, avg=16391.18, stdev=11587.11 clat percentiles (usec): | 1.00th=[ 2180], 5.00th=[ 3982], 10.00th=[ 4490], 20.00th=[ 5669], | 30.00th=[ 8848], 40.00th=[10945], 50.00th=[13698], 60.00th=[16057], | 70.00th=[20317], 80.00th=[25560], 90.00th=[33162], 95.00th=[40109], | 99.00th=[53216], 99.50th=[53216], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 223, max= 675, per=24.10%, avg=454.36, stdev=118.77, samples=25 iops : min= 13, max= 42, avg=27.92, stdev= 7.52, samples=25 lat (usec) : 250=0.39%, 500=3.26%, 750=0.39%, 1000=0.13% lat (msec) : 2=0.26%, 4=2.47%, 10=17.58%, 20=43.75%, 50=30.21% lat (msec) : 100=1.56% cpu : usr=0.08%, sys=28.62%, ctx=2521, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194185: Sun Aug 31 18:13:57 2025 read: IOPS=29, BW=475KiB/s (486kB/s)(6032KiB/12702msec) clat (usec): min=86, max=63645, avg=17371.07, stdev=10642.06 lat (usec): min=86, max=63645, avg=17371.55, stdev=10642.06 clat percentiles (usec): | 1.00th=[ 247], 5.00th=[ 383], 10.00th=[ 1844], 20.00th=[ 9765], | 30.00th=[11863], 40.00th=[13960], 50.00th=[15795], 60.00th=[17957], | 70.00th=[20841], 80.00th=[25297], 90.00th=[30278], 95.00th=[38011], | 99.00th=[49021], 99.50th=[52691], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 287, max= 637, per=26.39%, avg=472.64, stdev=94.82, samples=25 iops : min= 17, max= 39, avg=29.04, stdev= 5.91, samples=25 write: IOPS=30, BW=493KiB/s (504kB/s)(6256KiB/12702msec); 0 zone resets clat (usec): min=1016, max=55015, avg=15687.24, stdev=11100.14 lat (usec): min=1016, max=55016, avg=15688.06, stdev=11100.12 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 3228], 10.00th=[ 4555], 20.00th=[ 5735], | 30.00th=[ 8160], 40.00th=[10683], 50.00th=[12649], 60.00th=[15664], | 70.00th=[19530], 80.00th=[25035], 90.00th=[31327], 95.00th=[38011], | 99.00th=[52691], 99.50th=[54789], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 287, max= 858, per=26.16%, avg=493.08, stdev=120.61, samples=25 iops : min= 17, max= 53, avg=30.36, stdev= 7.55, samples=25 lat (usec) : 100=0.26%, 250=0.26%, 500=3.12%, 750=0.91%, 1000=0.13% lat (msec) : 2=1.56%, 4=2.21%, 10=20.96%, 20=39.32%, 50=30.08% lat (msec) : 100=1.17% cpu : usr=0.07%, sys=28.68%, ctx=2555, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1792KiB/s (1835kB/s), 443KiB/s-476KiB/s (454kB/s-487kB/s), io=23.4MiB (24.5MB), run=12702-13367msec WRITE: bw=1885KiB/s (1930kB/s), 459KiB/s-511KiB/s (470kB/s-523kB/s), io=24.6MiB (25.8MB), run=12702-13367msec rand-rw: (groupid=0, jobs=1): err= 0: pid=194187: Sun Aug 31 18:13:59 2025 read: IOPS=24, BW=385KiB/s (394kB/s)(5920KiB/15372msec) clat (usec): min=3399, max=93157, avg=17309.76, stdev=13721.63 lat (usec): min=3399, max=93158, avg=17310.27, stdev=13721.65 clat percentiles (usec): | 1.00th=[ 4228], 5.00th=[ 4948], 10.00th=[ 5538], 20.00th=[ 6652], | 30.00th=[ 8979], 40.00th=[11076], 50.00th=[12518], 60.00th=[15139], | 70.00th=[18744], 80.00th=[25035], 90.00th=[36963], 95.00th=[45876], | 99.00th=[66847], 99.50th=[74974], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 127, max= 1152, per=24.42%, avg=377.83, stdev=247.85, samples=30 iops : min= 7, max= 72, avg=23.20, stdev=15.59, samples=30 write: IOPS=25, BW=414KiB/s (424kB/s)(6368KiB/15372msec); 0 zone resets clat (usec): min=5132, max=91814, avg=22341.48, stdev=15529.24 lat (usec): min=5133, max=91817, avg=22342.94, stdev=15529.19 clat percentiles (usec): | 1.00th=[ 5276], 5.00th=[ 6521], 10.00th=[ 7177], 20.00th=[ 8586], | 30.00th=[11207], 40.00th=[13829], 50.00th=[17171], 60.00th=[22676], | 70.00th=[27919], 80.00th=[33817], 90.00th=[45351], 95.00th=[53216], | 99.00th=[68682], 99.50th=[84411], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 191, max= 1408, per=25.28%, avg=411.30, stdev=258.64, samples=30 iops : min= 11, max= 88, avg=25.30, stdev=16.25, samples=30 lat (msec) : 4=0.39%, 10=30.21%, 20=33.07%, 50=31.38%, 100=4.95% cpu : usr=0.07%, sys=13.00%, ctx=1123, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194188: Sun Aug 31 18:13:59 2025 read: IOPS=23, BW=373KiB/s (382kB/s)(5744KiB/15393msec) clat (usec): min=3539, max=80893, avg=17689.28, stdev=13901.59 lat (usec): min=3540, max=80894, avg=17689.92, stdev=13901.59 clat percentiles (usec): | 1.00th=[ 4178], 5.00th=[ 5080], 10.00th=[ 5669], 20.00th=[ 6783], | 30.00th=[ 8979], 40.00th=[11076], 50.00th=[12387], 60.00th=[15008], | 70.00th=[20055], 80.00th=[26608], 90.00th=[38011], 95.00th=[48497], | 99.00th=[65799], 99.50th=[71828], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 157, max= 992, per=24.02%, avg=371.53, stdev=243.41, samples=30 iops : min= 9, max= 62, avg=22.80, stdev=15.27, samples=30 write: IOPS=26, BW=425KiB/s (435kB/s)(6544KiB/15393msec); 0 zone resets clat (usec): min=4824, max=99614, avg=21955.58, stdev=15330.92 lat (usec): min=4825, max=99614, avg=21956.66, stdev=15330.93 clat percentiles (usec): | 1.00th=[ 5407], 5.00th=[ 6259], 10.00th=[ 6980], 20.00th=[ 8455], | 30.00th=[11469], 40.00th=[13566], 50.00th=[17171], 60.00th=[22414], | 70.00th=[27919], 80.00th=[33817], 90.00th=[41681], 95.00th=[53216], | 99.00th=[67634], 99.50th=[77071], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 128, max= 1472, per=25.49%, avg=414.70, stdev=273.36, samples=30 iops : min= 8, max= 92, avg=25.50, stdev=17.17, samples=30 lat (msec) : 4=0.39%, 10=28.91%, 20=33.20%, 50=32.29%, 100=5.21% cpu : usr=0.06%, sys=13.00%, ctx=1068, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194189: Sun Aug 31 18:13:59 2025 read: IOPS=25, BW=405KiB/s (414kB/s)(6256KiB/15461msec) clat (msec): min=3, max=107, avg=17.94, stdev=14.87 lat (msec): min=3, max=107, avg=17.94, stdev=14.87 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 16], | 70.00th=[ 20], 80.00th=[ 27], 90.00th=[ 39], 95.00th=[ 47], | 99.00th=[ 71], 99.50th=[ 83], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 160, max= 1376, per=25.93%, avg=401.13, stdev=278.83, samples=30 iops : min= 10, max= 86, avg=24.67, stdev=17.48, samples=30 write: IOPS=24, BW=390KiB/s (400kB/s)(6032KiB/15461msec); 0 zone resets clat (usec): min=4320, max=84162, avg=22055.35, stdev=15540.99 lat (usec): min=4321, max=84163, avg=22056.42, stdev=15541.01 clat percentiles (usec): | 1.00th=[ 4948], 5.00th=[ 6259], 10.00th=[ 7242], 20.00th=[ 9110], | 30.00th=[11338], 40.00th=[13435], 50.00th=[17433], 60.00th=[21365], | 70.00th=[27657], 80.00th=[34341], 90.00th=[43254], 95.00th=[53216], | 99.00th=[77071], 99.50th=[83362], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 159, max= 1120, per=23.19%, avg=377.37, stdev=231.50, samples=30 iops : min= 9, max= 70, avg=23.17, stdev=14.55, samples=30 lat (msec) : 4=0.65%, 10=29.17%, 20=33.85%, 50=31.12%, 100=5.08% lat (msec) : 250=0.13% cpu : usr=0.06%, sys=13.12%, ctx=1105, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194190: Sun Aug 31 18:13:59 2025 read: IOPS=24, BW=390KiB/s (399kB/s)(6032KiB/15482msec) clat (usec): min=3206, max=75296, avg=18103.88, stdev=14259.56 lat (usec): min=3207, max=75297, avg=18104.44, stdev=14259.53 clat percentiles (usec): | 1.00th=[ 3752], 5.00th=[ 4948], 10.00th=[ 5669], 20.00th=[ 6521], | 30.00th=[ 9110], 40.00th=[11207], 50.00th=[13042], 60.00th=[15139], | 70.00th=[20841], 80.00th=[27395], 90.00th=[39584], 95.00th=[48497], | 99.00th=[65274], 99.50th=[68682], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 159, max= 1408, per=24.42%, avg=377.77, stdev=253.09, samples=30 iops : min= 9, max= 88, avg=23.20, stdev=15.92, samples=30 write: IOPS=25, BW=404KiB/s (414kB/s)(6256KiB/15482msec); 0 zone resets clat (usec): min=4066, max=99616, avg=21885.05, stdev=15491.28 lat (usec): min=4067, max=99617, avg=21886.15, stdev=15491.35 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 12], 40.00th=[ 14], 50.00th=[ 17], 60.00th=[ 21], | 70.00th=[ 28], 80.00th=[ 34], 90.00th=[ 43], 95.00th=[ 53], | 99.00th=[ 71], 99.50th=[ 85], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 128, max= 1056, per=24.26%, avg=394.63, stdev=228.96, samples=30 iops : min= 8, max= 66, avg=24.27, stdev=14.40, samples=30 lat (msec) : 4=0.78%, 10=29.69%, 20=33.46%, 50=30.73%, 100=5.34% cpu : usr=0.08%, sys=12.71%, ctx=1096, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1547KiB/s (1584kB/s), 373KiB/s-405KiB/s (382kB/s-414kB/s), io=23.4MiB (24.5MB), run=15372-15482msec WRITE: bw=1628KiB/s (1667kB/s), 390KiB/s-425KiB/s (400kB/s-435kB/s), io=24.6MiB (25.8MB), run=15372-15482msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=194198: Sun Aug 31 18:14:08 2025 read: IOPS=3, BW=3500KiB/s (3584kB/s)(7168KiB/2048msec) clat (usec): min=849, max=458338, avg=151001.37, stdev=145955.80 lat (usec): min=851, max=458339, avg=151002.41, stdev=145955.68 clat percentiles (usec): | 1.00th=[ 848], 5.00th=[ 848], 10.00th=[ 848], 20.00th=[ 55313], | 30.00th=[112722], 40.00th=[112722], 50.00th=[135267], 60.00th=[145753], | 70.00th=[145753], 80.00th=[147850], 90.00th=[459277], 95.00th=[459277], | 99.00th=[459277], 99.50th=[459277], 99.90th=[459277], 99.95th=[459277], | 99.99th=[459277] bw ( KiB/s): min= 3984, max= 5128, per=61.20%, avg=4556.00, stdev=808.93, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2500KiB/s (2560kB/s)(5120KiB/2048msec); 0 zone resets clat (msec): min=130, max=253, avg=192.40, stdev=58.77 lat (msec): min=130, max=253, avg=192.45, stdev=58.76 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 131], 10.00th=[ 131], 20.00th=[ 131], | 30.00th=[ 142], 40.00th=[ 142], 50.00th=[ 184], 60.00th=[ 184], | 70.00th=[ 253], 80.00th=[ 253], 90.00th=[ 255], 95.00th=[ 255], | 99.00th=[ 255], 99.50th=[ 255], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 1709, max= 4000, per=27.39%, avg=2854.50, stdev=1619.98, samples=2 iops : min= 1, max= 3, avg= 2.00, stdev= 1.41, samples=2 lat (usec) : 1000=8.33% lat (msec) : 100=8.33%, 250=58.33%, 500=25.00% cpu : usr=0.00%, sys=55.15%, ctx=873, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194199: Sun Aug 31 18:14:08 2025 read: IOPS=0, BW=744KiB/s (762kB/s)(2048KiB/2751msec) clat (msec): min=57, max=201, avg=129.76, stdev=101.48 lat (msec): min=57, max=201, avg=129.76, stdev=101.48 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 58], 10.00th=[ 58], 20.00th=[ 58], | 30.00th=[ 58], 40.00th=[ 58], 50.00th=[ 58], 60.00th=[ 203], | 70.00th=[ 203], 80.00th=[ 203], 90.00th=[ 203], 95.00th=[ 203], | 99.00th=[ 203], 99.50th=[ 203], 99.90th=[ 203], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 1692, max= 2048, per=25.12%, avg=1870.00, stdev=251.73, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3722KiB/s (3812kB/s)(10.0MiB/2751msec); 0 zone resets clat (msec): min=73, max=490, avg=248.28, stdev=134.06 lat (msec): min=73, max=490, avg=248.33, stdev=134.05 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 73], 10.00th=[ 73], 20.00th=[ 115], | 30.00th=[ 153], 40.00th=[ 167], 50.00th=[ 215], 60.00th=[ 228], | 70.00th=[ 300], 80.00th=[ 330], 90.00th=[ 414], 95.00th=[ 489], | 99.00th=[ 489], 99.50th=[ 489], 99.90th=[ 489], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 1692, max= 5976, per=34.06%, avg=3549.60, stdev=1748.33, samples=5 iops : min= 1, max= 5, avg= 2.80, stdev= 1.79, samples=5 lat (msec) : 100=16.67%, 250=50.00%, 500=33.33% cpu : usr=0.00%, sys=50.91%, ctx=509, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194200: Sun Aug 31 18:14:08 2025 read: IOPS=2, BW=2626KiB/s (2689kB/s)(6144KiB/2340msec) clat (usec): min=679, max=326063, avg=101177.66, stdev=129285.74 lat (usec): min=679, max=326064, avg=101178.56, stdev=129285.88 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 676], 10.00th=[ 676], 20.00th=[ 775], | 30.00th=[ 775], 40.00th=[ 46400], 50.00th=[ 46400], 60.00th=[ 47973], | 70.00th=[185598], 80.00th=[185598], 90.00th=[325059], 95.00th=[325059], | 99.00th=[325059], 99.50th=[325059], 99.90th=[325059], 99.95th=[325059], | 99.99th=[325059] bw ( KiB/s): min= 2048, max= 4000, per=44.92%, avg=3344.00, stdev=1122.40, samples=3 iops : min= 2, max= 3, avg= 2.67, stdev= 0.58, samples=3 write: IOPS=2, BW=2626KiB/s (2689kB/s)(6144KiB/2340msec); 0 zone resets clat (msec): min=140, max=393, avg=285.49, stdev=101.98 lat (msec): min=140, max=393, avg=285.54, stdev=101.97 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 140], 10.00th=[ 140], 20.00th=[ 218], | 30.00th=[ 218], 40.00th=[ 236], 50.00th=[ 236], 60.00th=[ 359], | 70.00th=[ 372], 80.00th=[ 372], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 1992, max= 3407, per=22.66%, avg=2361.75, stdev=697.27, samples=4 iops : min= 1, max= 3, avg= 1.75, stdev= 0.96, samples=4 lat (usec) : 750=8.33%, 1000=8.33% lat (msec) : 50=16.67%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=45.75%, ctx=623, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194201: Sun Aug 31 18:14:08 2025 read: IOPS=1, BW=2013KiB/s (2061kB/s)(5120KiB/2544msec) clat (usec): min=621, max=175608, avg=117764.56, stdev=68129.26 lat (usec): min=622, max=175608, avg=117765.20, stdev=68129.36 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 619], 10.00th=[ 619], 20.00th=[ 619], | 30.00th=[124257], 40.00th=[124257], 50.00th=[139461], 60.00th=[139461], | 70.00th=[149947], 80.00th=[149947], 90.00th=[175113], 95.00th=[175113], | 99.00th=[175113], 99.50th=[175113], 99.90th=[175113], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 8000, max= 8000, per=100.00%, avg=8000.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2818KiB/s (2885kB/s)(7168KiB/2544msec); 0 zone resets clat (msec): min=152, max=405, avg=277.76, stdev=92.92 lat (msec): min=152, max=405, avg=277.81, stdev=92.91 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 153], 10.00th=[ 153], 20.00th=[ 203], | 30.00th=[ 232], 40.00th=[ 232], 50.00th=[ 259], 60.00th=[ 313], | 70.00th=[ 313], 80.00th=[ 380], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 1747, max= 5976, per=37.80%, avg=3939.67, stdev=2118.83, samples=3 iops : min= 1, max= 5, avg= 3.33, stdev= 2.08, samples=3 lat (usec) : 750=8.33% lat (msec) : 250=58.33%, 500=33.33% cpu : usr=0.00%, sys=50.85%, ctx=902, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7445KiB/s (7623kB/s), 744KiB/s-3500KiB/s (762kB/s-3584kB/s), io=20.0MiB (20.0MB), run=2048-2751msec WRITE: bw=10.2MiB/s (10.7MB/s), 2500KiB/s-3722KiB/s (2560kB/s-3812kB/s), io=28.0MiB (29.4MB), run=2048-2751msec rand-rw: (groupid=0, jobs=1): err= 0: pid=194203: Sun Aug 31 18:14:08 2025 read: IOPS=2, BW=2617KiB/s (2680kB/s)(7168KiB/2739msec) clat (msec): min=36, max=659, avg=227.62, stdev=239.29 lat (msec): min=36, max=659, avg=227.62, stdev=239.29 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 53], | 30.00th=[ 58], 40.00th=[ 58], 50.00th=[ 140], 60.00th=[ 192], | 70.00th=[ 192], 80.00th=[ 456], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 2048, max= 4096, per=39.84%, avg=2897.00, stdev=1067.92, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 write: IOPS=1, BW=1869KiB/s (1914kB/s)(5120KiB/2739msec); 0 zone resets clat (msec): min=76, max=386, avg=208.23, stdev=132.01 lat (msec): min=76, max=386, avg=208.29, stdev=132.00 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 77], 10.00th=[ 77], 20.00th=[ 77], | 30.00th=[ 108], 40.00th=[ 108], 50.00th=[ 169], 60.00th=[ 169], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 2015, max= 2048, per=19.95%, avg=2031.50, stdev=23.33, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 50=8.33%, 100=25.00%, 250=33.33%, 500=25.00%, 750=8.33% cpu : usr=0.00%, sys=19.39%, ctx=153, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194204: Sun Aug 31 18:14:08 2025 read: IOPS=0, BW=753KiB/s (771kB/s)(2048KiB/2719msec) clat (msec): min=59, max=257, avg=158.36, stdev=140.38 lat (msec): min=59, max=257, avg=158.36, stdev=140.38 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 59], | 30.00th=[ 59], 40.00th=[ 59], 50.00th=[ 59], 60.00th=[ 257], | 70.00th=[ 257], 80.00th=[ 257], 90.00th=[ 257], 95.00th=[ 257], | 99.00th=[ 257], 99.50th=[ 257], 99.90th=[ 257], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 1825, max= 2019, per=26.43%, avg=1922.00, stdev=137.18, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3766KiB/s (3856kB/s)(10.0MiB/2719msec); 0 zone resets clat (msec): min=63, max=444, avg=198.70, stdev=143.24 lat (msec): min=63, max=444, avg=198.76, stdev=143.24 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 74], | 30.00th=[ 104], 40.00th=[ 106], 50.00th=[ 111], 60.00th=[ 167], | 70.00th=[ 171], 80.00th=[ 342], 90.00th=[ 405], 95.00th=[ 443], | 99.00th=[ 443], 99.50th=[ 443], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 4096, max= 8078, per=53.27%, avg=5423.33, stdev=2299.01, samples=3 iops : min= 4, max= 7, avg= 5.00, stdev= 1.73, samples=3 lat (msec) : 100=25.00%, 250=41.67%, 500=33.33% cpu : usr=0.07%, sys=21.74%, ctx=172, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194205: Sun Aug 31 18:14:08 2025 read: IOPS=2, BW=2220KiB/s (2274kB/s)(6144KiB/2767msec) clat (msec): min=44, max=395, avg=149.75, stdev=133.78 lat (msec): min=44, max=395, avg=149.75, stdev=133.78 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 63], | 30.00th=[ 63], 40.00th=[ 68], 50.00th=[ 68], 60.00th=[ 125], | 70.00th=[ 205], 80.00th=[ 205], 90.00th=[ 397], 95.00th=[ 397], | 99.00th=[ 397], 99.50th=[ 397], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 4096, max= 4096, per=56.33%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2220KiB/s (2274kB/s)(6144KiB/2767msec); 0 zone resets clat (msec): min=58, max=662, avg=281.87, stdev=245.69 lat (msec): min=58, max=662, avg=281.93, stdev=245.70 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 63], | 30.00th=[ 63], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 300], | 70.00th=[ 477], 80.00th=[ 477], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 1268, max= 2048, per=18.12%, avg=1844.75, stdev=384.81, samples=4 iops : min= 1, max= 2, avg= 1.50, stdev= 0.58, samples=4 lat (msec) : 50=8.33%, 100=33.33%, 250=25.00%, 500=25.00%, 750=8.33% cpu : usr=0.00%, sys=16.34%, ctx=86, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194206: Sun Aug 31 18:14:08 2025 read: IOPS=1, BW=1818KiB/s (1862kB/s)(5120KiB/2816msec) clat (usec): min=21784, max=99879, avg=69148.88, stdev=31628.87 lat (usec): min=21784, max=99880, avg=69149.81, stdev=31628.80 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 62], 40.00th=[ 62], 50.00th=[ 67], 60.00th=[ 67], | 70.00th=[ 96], 80.00th=[ 96], 90.00th=[ 101], 95.00th=[ 101], | 99.00th=[ 101], 99.50th=[ 101], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 2048, max= 2048, per=28.16%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2545KiB/s (2607kB/s)(7168KiB/2816msec); 0 zone resets clat (msec): min=23, max=751, avg=326.65, stdev=292.49 lat (msec): min=23, max=751, avg=326.71, stdev=292.49 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 69], | 30.00th=[ 144], 40.00th=[ 144], 50.00th=[ 169], 60.00th=[ 558], | 70.00th=[ 558], 80.00th=[ 575], 90.00th=[ 751], 95.00th=[ 751], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 2048, max= 4031, per=26.61%, avg=2709.00, stdev=1144.89, samples=3 iops : min= 2, max= 3, avg= 2.33, stdev= 0.58, samples=3 lat (msec) : 50=16.67%, 100=41.67%, 250=16.67%, 750=16.67%, 1000=8.33% cpu : usr=0.00%, sys=15.24%, ctx=78, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7273KiB/s (7447kB/s), 753KiB/s-2617KiB/s (771kB/s-2680kB/s), io=20.0MiB (20.0MB), run=2719-2816msec WRITE: bw=9.94MiB/s (10.4MB/s), 1869KiB/s-3766KiB/s (1914kB/s-3856kB/s), io=28.0MiB (29.4MB), run=2719-2816msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=194212: Sun Aug 31 18:14:18 2025 read: IOPS=3, BW=14.1MiB/s (14.8MB/s)(12.0MiB/851msec) clat (usec): min=1774, max=458029, avg=279536.72, stdev=243789.08 lat (usec): min=1775, max=458030, avg=279537.84, stdev=243788.86 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1778], 10.00th=[ 1778], 20.00th=[ 1778], | 30.00th=[ 1778], 40.00th=[379585], 50.00th=[379585], 60.00th=[379585], | 70.00th=[459277], 80.00th=[459277], 90.00th=[459277], 95.00th=[459277], | 99.00th=[459277], 99.50th=[459277], 99.90th=[459277], 99.95th=[459277], | 99.99th=[459277] bw ( KiB/s): min= 7086, max= 7086, per=100.00%, avg=7086.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 2=33.33%, 500=66.67% cpu : usr=0.00%, sys=57.41%, ctx=1620, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194213: Sun Aug 31 18:14:18 2025 read: IOPS=0, BW=1093KiB/s (1120kB/s)(4096KiB/3746msec) clat (nsec): min=358337k, max=358337k, avg=358336563.00, stdev= 0.00 lat (nsec): min=358339k, max=358339k, avg=358338959.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 359], | 30.00th=[ 359], 40.00th=[ 359], 50.00th=[ 359], 60.00th=[ 359], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 7262, max= 7262, per=100.00%, avg=7262.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2187KiB/s (2239kB/s)(8192KiB/3746msec); 0 zone resets clat (msec): min=1247, max=2120, avg=1684.44, stdev=617.25 lat (msec): min=1248, max=2121, avg=1684.63, stdev=617.31 clat percentiles (msec): | 1.00th=[ 1250], 5.00th=[ 1250], 10.00th=[ 1250], 20.00th=[ 1250], | 30.00th=[ 1250], 40.00th=[ 1250], 50.00th=[ 1250], 60.00th=[ 2123], | 70.00th=[ 2123], 80.00th=[ 2123], 90.00th=[ 2123], 95.00th=[ 2123], | 99.00th=[ 2123], 99.50th=[ 2123], 99.90th=[ 2123], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 8175, max= 8175, per=93.46%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 2000=33.33%, >=2000=33.33% cpu : usr=0.00%, sys=34.50%, ctx=694, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194214: Sun Aug 31 18:14:18 2025 write: IOPS=0, BW=3706KiB/s (3795kB/s)(12.0MiB/3316msec); 0 zone resets clat (msec): min=675, max=1675, avg=1095.29, stdev=519.38 lat (msec): min=675, max=1676, avg=1095.58, stdev=519.36 clat percentiles (msec): | 1.00th=[ 676], 5.00th=[ 676], 10.00th=[ 676], 20.00th=[ 676], | 30.00th=[ 676], 40.00th=[ 936], 50.00th=[ 936], 60.00th=[ 936], | 70.00th=[ 1670], 80.00th=[ 1670], 90.00th=[ 1670], 95.00th=[ 1670], | 99.00th=[ 1670], 99.50th=[ 1670], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 8143, max= 8175, per=93.28%, avg=8159.00, stdev=22.63, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 750=33.33%, 1000=33.33%, 2000=33.33% cpu : usr=0.03%, sys=56.26%, ctx=75, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194215: Sun Aug 31 18:14:18 2025 write: IOPS=1, BW=4925KiB/s (5043kB/s)(12.0MiB/2495msec); 0 zone resets clat (msec): min=404, max=1450, avg=827.99, stdev=550.96 lat (msec): min=404, max=1451, avg=828.22, stdev=550.98 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 405], 10.00th=[ 405], 20.00th=[ 405], | 30.00th=[ 405], 40.00th=[ 625], 50.00th=[ 625], 60.00th=[ 625], | 70.00th=[ 1452], 80.00th=[ 1452], 90.00th=[ 1452], 95.00th=[ 1452], | 99.00th=[ 1452], 99.50th=[ 1452], 99.90th=[ 1452], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min=16286, max=16286, per=100.00%, avg=16286.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33%, 2000=33.33% cpu : usr=0.04%, sys=70.06%, ctx=33, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4374KiB/s (4479kB/s), 1093KiB/s-14.1MiB/s (1120kB/s-14.8MB/s), io=16.0MiB (16.8MB), run=851-3746msec WRITE: bw=8747KiB/s (8957kB/s), 2187KiB/s-4925KiB/s (2239kB/s-5043kB/s), io=32.0MiB (33.6MB), run=2495-3746msec rand-rw: (groupid=0, jobs=1): err= 0: pid=194217: Sun Aug 31 18:14:18 2025 read: IOPS=0, BW=3472KiB/s (3555kB/s)(12.0MiB/3539msec) clat (msec): min=723, max=1640, avg=1168.60, stdev=458.82 lat (msec): min=723, max=1640, avg=1168.60, stdev=458.82 clat percentiles (msec): | 1.00th=[ 726], 5.00th=[ 726], 10.00th=[ 726], 20.00th=[ 726], | 30.00th=[ 726], 40.00th=[ 1150], 50.00th=[ 1150], 60.00th=[ 1150], | 70.00th=[ 1636], 80.00th=[ 1636], 90.00th=[ 1636], 95.00th=[ 1636], | 99.00th=[ 1636], 99.50th=[ 1636], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 8159, max= 8192, per=100.00%, avg=8175.50, stdev=23.33, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 750=33.33%, 2000=66.67% cpu : usr=0.00%, sys=18.23%, ctx=72, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194218: Sun Aug 31 18:14:18 2025 read: IOPS=0, BW=1066KiB/s (1091kB/s)(4096KiB/3844msec) clat (nsec): min=1347.2M, max=1347.2M, avg=1347202761.00, stdev= 0.00 lat (nsec): min=1347.2M, max=1347.2M, avg=1347203527.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1351], 5.00th=[ 1351], 10.00th=[ 1351], 20.00th=[ 1351], | 30.00th=[ 1351], 40.00th=[ 1351], 50.00th=[ 1351], 60.00th=[ 1351], | 70.00th=[ 1351], 80.00th=[ 1351], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2131KiB/s (2182kB/s)(8192KiB/3844msec); 0 zone resets clat (msec): min=531, max=1942, avg=1237.19, stdev=998.08 lat (msec): min=531, max=1943, avg=1237.33, stdev=998.07 clat percentiles (msec): | 1.00th=[ 531], 5.00th=[ 531], 10.00th=[ 531], 20.00th=[ 531], | 30.00th=[ 531], 40.00th=[ 531], 50.00th=[ 531], 60.00th=[ 1938], | 70.00th=[ 1938], 80.00th=[ 1938], 90.00th=[ 1938], 95.00th=[ 1938], | 99.00th=[ 1938], 99.50th=[ 1938], 99.90th=[ 1938], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 7968, max= 7968, per=93.48%, avg=7968.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 750=33.33%, 2000=66.67% cpu : usr=0.00%, sys=10.46%, ctx=210, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194219: Sun Aug 31 18:14:18 2025 write: IOPS=0, BW=3388KiB/s (3469kB/s)(12.0MiB/3627msec); 0 zone resets clat (msec): min=302, max=1751, avg=1188.78, stdev=777.16 lat (msec): min=302, max=1751, avg=1189.08, stdev=777.16 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 305], 10.00th=[ 305], 20.00th=[ 305], | 30.00th=[ 305], 40.00th=[ 1519], 50.00th=[ 1519], 60.00th=[ 1519], | 70.00th=[ 1754], 80.00th=[ 1754], 90.00th=[ 1754], 95.00th=[ 1754], | 99.00th=[ 1754], 99.50th=[ 1754], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 7968, max= 8192, per=94.79%, avg=8080.00, stdev=158.39, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 2000=66.67% cpu : usr=0.00%, sys=7.31%, ctx=42, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=194220: Sun Aug 31 18:14:18 2025 write: IOPS=0, BW=3628KiB/s (3715kB/s)(12.0MiB/3387msec); 0 zone resets clat (msec): min=307, max=1551, avg=1111.78, stdev=697.62 lat (msec): min=307, max=1552, avg=1112.07, stdev=697.64 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 309], 10.00th=[ 309], 20.00th=[ 309], | 30.00th=[ 309], 40.00th=[ 1469], 50.00th=[ 1469], 60.00th=[ 1469], | 70.00th=[ 1552], 80.00th=[ 1552], 90.00th=[ 1552], 95.00th=[ 1552], | 99.00th=[ 1552], 99.50th=[ 1552], 99.90th=[ 1552], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 8192, max= 8192, per=96.11%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 2000=66.67% cpu : usr=0.00%, sys=13.56%, ctx=111, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4262KiB/s (4365kB/s), 1066KiB/s-3472KiB/s (1091kB/s-3555kB/s), io=16.0MiB (16.8MB), run=3539-3844msec WRITE: bw=8524KiB/s (8729kB/s), 2131KiB/s-3628KiB/s (2182kB/s-3715kB/s), io=32.0MiB (33.6MB), run=3387-3844msec