== sanity test 398b: DIO and buffer IO race ============== 12:59:08 (1752512348) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB, 48 MiB) copied, 2.48253 s, 20.3 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=190120: Mon Jul 14 13:00:13 2025 read: IOPS=26, BW=108KiB/s (111kB/s)(5992KiB/55501msec) clat (usec): min=101, max=701178, avg=20301.58, stdev=20346.16 lat (usec): min=101, max=701179, avg=20302.22, stdev=20346.21 clat percentiles (usec): | 1.00th=[ 408], 5.00th=[ 7111], 10.00th=[ 8979], 20.00th=[ 11469], | 30.00th=[ 13960], 40.00th=[ 16057], 50.00th=[ 17957], 60.00th=[ 20579], | 70.00th=[ 23462], 80.00th=[ 27395], 90.00th=[ 33162], 95.00th=[ 39060], | 99.00th=[ 53216], 99.50th=[ 61604], 99.90th=[ 74974], 99.95th=[700449], | 99.99th=[700449] bw ( KiB/s): min= 13, max= 208, per=25.19%, avg=107.31, stdev=33.92, samples=110 iops : min= 3, max= 52, avg=26.39, stdev= 8.55, samples=110 write: IOPS=28, BW=113KiB/s (116kB/s)(6296KiB/55501msec); 0 zone resets clat (usec): min=541, max=95060, avg=15812.69, stdev=12001.99 lat (usec): min=541, max=95061, avg=15813.49, stdev=12002.09 clat percentiles (usec): | 1.00th=[ 1029], 5.00th=[ 2671], 10.00th=[ 3261], 20.00th=[ 5145], | 30.00th=[ 7504], 40.00th=[10421], 50.00th=[13042], 60.00th=[16319], | 70.00th=[19792], 80.00th=[24511], 90.00th=[32375], 95.00th=[39584], | 99.00th=[51643], 99.50th=[56361], 99.90th=[71828], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 39, max= 255, per=25.49%, avg=113.68, stdev=37.64, samples=109 iops : min= 9, max= 63, avg=27.98, stdev= 9.48, samples=109 lat (usec) : 250=0.13%, 500=0.49%, 750=0.52%, 1000=0.39% lat (msec) : 2=0.81%, 4=6.18%, 10=18.00%, 20=38.31%, 50=33.76% lat (msec) : 100=1.37%, 750=0.03% cpu : usr=0.12%, sys=27.60%, ctx=9111, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190121: Mon Jul 14 13:00:13 2025 read: IOPS=26, BW=107KiB/s (109kB/s)(5940KiB/55771msec) clat (usec): min=350, max=88348, avg=20642.54, stdev=10704.47 lat (usec): min=350, max=88348, avg=20643.26, stdev=10704.63 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 7570], 10.00th=[ 9110], 20.00th=[11338], | 30.00th=[13829], 40.00th=[16057], 50.00th=[18744], 60.00th=[21627], | 70.00th=[25035], 80.00th=[29230], 90.00th=[34341], 95.00th=[39060], | 99.00th=[53740], 99.50th=[63701], 99.90th=[72877], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 6, max= 200, per=24.98%, avg=106.41, stdev=35.52, samples=111 iops : min= 1, max= 50, avg=26.16, stdev= 8.91, samples=111 write: IOPS=28, BW=114KiB/s (117kB/s)(6348KiB/55771msec); 0 zone resets clat (usec): min=658, max=621599, avg=15698.24, stdev=19279.69 lat (usec): min=658, max=621600, avg=15703.62, stdev=19277.91 clat percentiles (usec): | 1.00th=[ 1156], 5.00th=[ 2474], 10.00th=[ 2900], 20.00th=[ 4752], | 30.00th=[ 7111], 40.00th=[ 9634], 50.00th=[ 12387], 60.00th=[ 15795], | 70.00th=[ 19792], 80.00th=[ 24773], 90.00th=[ 30802], 95.00th=[ 37487], | 99.00th=[ 53216], 99.50th=[ 61080], 99.90th=[ 86508], 99.95th=[624952], | 99.99th=[624952] bw ( KiB/s): min= 6, max= 303, per=25.49%, avg=113.70, stdev=44.58, samples=111 iops : min= 1, max= 75, avg=27.98, stdev=11.22, samples=111 lat (usec) : 500=0.33%, 750=0.20%, 1000=0.23% lat (msec) : 2=1.01%, 4=7.42%, 10=18.85%, 20=34.67%, 50=36.07% lat (msec) : 100=1.20%, 750=0.03% cpu : usr=0.13%, sys=27.26%, ctx=9060, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190122: Mon Jul 14 13:00:13 2025 read: IOPS=27, BW=109KiB/s (111kB/s)(6028KiB/55447msec) clat (usec): min=230, max=70503, avg=19969.45, stdev=10363.57 lat (usec): min=230, max=70504, avg=19970.27, stdev=10363.67 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 7373], 10.00th=[ 9110], 20.00th=[11469], | 30.00th=[13960], 40.00th=[15664], 50.00th=[17957], 60.00th=[20579], | 70.00th=[23725], 80.00th=[27657], 90.00th=[33817], 95.00th=[39060], | 99.00th=[53740], 99.50th=[58459], 99.90th=[63701], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 39, max= 216, per=25.68%, avg=109.41, stdev=36.91, samples=109 iops : min= 9, max= 54, avg=26.92, stdev= 9.28, samples=109 write: IOPS=28, BW=113KiB/s (116kB/s)(6260KiB/55447msec); 0 zone resets clat (usec): min=571, max=642922, avg=16069.66, stdev=19639.62 lat (usec): min=571, max=642923, avg=16070.54, stdev=19639.66 clat percentiles (usec): | 1.00th=[ 1401], 5.00th=[ 2638], 10.00th=[ 3228], 20.00th=[ 5145], | 30.00th=[ 7570], 40.00th=[ 10159], 50.00th=[ 13304], 60.00th=[ 16057], | 70.00th=[ 19792], 80.00th=[ 25035], 90.00th=[ 31589], 95.00th=[ 38536], | 99.00th=[ 52167], 99.50th=[ 55837], 99.90th=[ 71828], 99.95th=[641729], | 99.99th=[641729] bw ( KiB/s): min= 47, max= 216, per=25.41%, avg=113.31, stdev=34.61, samples=109 iops : min= 11, max= 54, avg=27.89, stdev= 8.76, samples=109 lat (usec) : 250=0.03%, 500=0.52%, 750=0.46%, 1000=0.16% lat (msec) : 2=0.91%, 4=6.25%, 10=18.52%, 20=37.96%, 50=33.76% lat (msec) : 100=1.40%, 750=0.03% cpu : usr=0.11%, sys=27.23%, ctx=8896, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190123: Mon Jul 14 13:00:13 2025 read: IOPS=26, BW=107KiB/s (110kB/s)(6052KiB/56351msec) clat (usec): min=129, max=88302, avg=20413.06, stdev=10917.88 lat (usec): min=130, max=88302, avg=20414.15, stdev=10917.98 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 7373], 10.00th=[ 9110], 20.00th=[11600], | 30.00th=[13960], 40.00th=[16319], 50.00th=[18220], 60.00th=[20841], | 70.00th=[23725], 80.00th=[27919], 90.00th=[34341], 95.00th=[40633], | 99.00th=[57934], 99.50th=[64750], 99.90th=[83362], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 39, max= 223, per=25.42%, avg=108.31, stdev=34.10, samples=111 iops : min= 9, max= 55, avg=26.63, stdev= 8.60, samples=111 write: IOPS=27, BW=111KiB/s (113kB/s)(6236KiB/56351msec); 0 zone resets clat (usec): min=678, max=627940, avg=16201.64, stdev=19408.96 lat (usec): min=679, max=627941, avg=16202.48, stdev=19409.00 clat percentiles (usec): | 1.00th=[ 1221], 5.00th=[ 2606], 10.00th=[ 3064], 20.00th=[ 5276], | 30.00th=[ 7635], 40.00th=[ 10421], 50.00th=[ 13566], 60.00th=[ 16581], | 70.00th=[ 20055], 80.00th=[ 25297], 90.00th=[ 32113], 95.00th=[ 38536], | 99.00th=[ 52167], 99.50th=[ 58459], 99.90th=[ 72877], 99.95th=[624952], | 99.99th=[624952] bw ( KiB/s): min= 13, max= 215, per=24.75%, avg=110.38, stdev=37.11, samples=112 iops : min= 3, max= 53, avg=27.15, stdev= 9.31, samples=112 lat (usec) : 250=0.07%, 500=0.42%, 750=0.16%, 1000=0.26% lat (msec) : 2=0.98%, 4=6.15%, 10=18.23%, 20=37.37%, 50=34.93% lat (msec) : 100=1.40%, 750=0.03% cpu : usr=0.11%, sys=27.16%, ctx=9251, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=426KiB/s (436kB/s), 107KiB/s-109KiB/s (109kB/s-111kB/s), io=23.4MiB (24.6MB), run=55447-56351msec WRITE: bw=446KiB/s (457kB/s), 111KiB/s-114KiB/s (113kB/s-117kB/s), io=24.6MiB (25.7MB), run=55447-56351msec rand-rw: (groupid=0, jobs=1): err= 0: pid=190124: Mon Jul 14 13:00:18 2025 read: IOPS=24, BW=97.5KiB/s (99.8kB/s)(5992KiB/61457msec) clat (msec): min=2, max=705, avg=18.25, stdev=23.19 lat (msec): min=2, max=705, avg=18.25, stdev=23.19 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 8], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 17], | 70.00th=[ 21], 80.00th=[ 27], 90.00th=[ 40], 95.00th=[ 50], | 99.00th=[ 69], 99.50th=[ 82], 99.90th=[ 103], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 23, max= 464, per=25.14%, avg=97.81, stdev=80.20, samples=121 iops : min= 5, max= 116, avg=24.02, stdev=20.20, samples=121 write: IOPS=25, BW=102KiB/s (105kB/s)(6296KiB/61457msec); 0 zone resets clat (usec): min=3643, max=99781, avg=21569.59, stdev=15934.80 lat (usec): min=3643, max=99781, avg=21570.61, stdev=15935.06 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 10], 40.00th=[ 14], 50.00th=[ 18], 60.00th=[ 23], | 70.00th=[ 28], 80.00th=[ 34], 90.00th=[ 44], 95.00th=[ 53], | 99.00th=[ 74], 99.50th=[ 80], 99.90th=[ 93], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 31, max= 440, per=25.22%, avg=102.90, stdev=82.50, samples=121 iops : min= 7, max= 110, avg=25.30, stdev=20.76, samples=121 lat (msec) : 4=3.94%, 10=29.82%, 20=28.19%, 50=32.62%, 100=5.34% lat (msec) : 250=0.07%, 750=0.03% cpu : usr=0.08%, sys=13.94%, ctx=4402, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190125: Mon Jul 14 13:00:18 2025 read: IOPS=24, BW=96.5KiB/s (98.8kB/s)(5940KiB/61578msec) clat (msec): min=2, max=626, avg=18.00, stdev=21.42 lat (msec): min=2, max=626, avg=18.00, stdev=21.42 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 11], 50.00th=[ 14], 60.00th=[ 17], | 70.00th=[ 22], 80.00th=[ 28], 90.00th=[ 38], 95.00th=[ 47], | 99.00th=[ 65], 99.50th=[ 80], 99.90th=[ 101], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 24, max= 456, per=24.75%, avg=96.29, stdev=81.65, samples=121 iops : min= 6, max= 114, avg=23.64, stdev=20.55, samples=121 write: IOPS=25, BW=103KiB/s (106kB/s)(6348KiB/61578msec); 0 zone resets clat (msec): min=3, max=100, avg=21.85, stdev=16.22 lat (msec): min=3, max=100, avg=21.85, stdev=16.22 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 10], 40.00th=[ 14], 50.00th=[ 18], 60.00th=[ 23], | 70.00th=[ 28], 80.00th=[ 35], 90.00th=[ 45], 95.00th=[ 54], | 99.00th=[ 73], 99.50th=[ 80], 99.90th=[ 88], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 39, max= 448, per=25.11%, avg=102.44, stdev=80.52, samples=121 iops : min= 9, max= 112, avg=25.18, stdev=20.28, samples=121 lat (msec) : 4=3.84%, 10=30.70%, 20=25.81%, 50=34.34%, 100=5.21% lat (msec) : 250=0.07%, 750=0.03% cpu : usr=0.10%, sys=13.82%, ctx=4450, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190126: Mon Jul 14 13:00:18 2025 read: IOPS=24, BW=98.3KiB/s (101kB/s)(6028KiB/61346msec) clat (usec): min=2050, max=87675, avg=17123.87, stdev=14385.74 lat (usec): min=2051, max=87676, avg=17124.63, stdev=14385.84 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3785], 10.00th=[ 4178], 20.00th=[ 5014], | 30.00th=[ 7046], 40.00th=[10028], 50.00th=[13042], 60.00th=[16057], | 70.00th=[20055], 80.00th=[26346], 90.00th=[38011], 95.00th=[47449], | 99.00th=[67634], 99.50th=[70779], 99.90th=[84411], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 23, max= 448, per=25.43%, avg=98.93, stdev=85.24, samples=121 iops : min= 5, max= 112, avg=24.31, stdev=21.45, samples=121 write: IOPS=25, BW=102KiB/s (104kB/s)(6260KiB/61346msec); 0 zone resets clat (msec): min=4, max=633, avg=22.60, stdev=22.71 lat (msec): min=4, max=633, avg=22.60, stdev=22.71 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 10], 40.00th=[ 14], 50.00th=[ 19], 60.00th=[ 23], | 70.00th=[ 29], 80.00th=[ 35], 90.00th=[ 46], 95.00th=[ 56], | 99.00th=[ 77], 99.50th=[ 83], 99.90th=[ 105], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 31, max= 416, per=25.16%, avg=102.64, stdev=81.19, samples=121 iops : min= 7, max= 104, avg=25.23, stdev=20.43, samples=121 lat (msec) : 4=3.74%, 10=31.35%, 20=26.89%, 50=32.10%, 100=5.86% lat (msec) : 250=0.03%, 750=0.03% cpu : usr=0.09%, sys=13.91%, ctx=4448, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190127: Mon Jul 14 13:00:18 2025 read: IOPS=24, BW=98.4KiB/s (101kB/s)(6052KiB/61497msec) clat (msec): min=2, max=102, avg=17.77, stdev=15.46 lat (msec): min=2, max=102, avg=17.77, stdev=15.46 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 8], 40.00th=[ 11], 50.00th=[ 14], 60.00th=[ 17], | 70.00th=[ 21], 80.00th=[ 27], 90.00th=[ 40], 95.00th=[ 52], | 99.00th=[ 71], 99.50th=[ 79], 99.90th=[ 99], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min= 31, max= 424, per=25.30%, avg=98.40, stdev=79.56, samples=121 iops : min= 7, max= 106, avg=24.17, stdev=20.02, samples=121 write: IOPS=25, BW=101KiB/s (104kB/s)(6236KiB/61497msec); 0 zone resets clat (msec): min=3, max=638, avg=22.09, stdev=22.49 lat (msec): min=3, max=638, avg=22.09, stdev=22.49 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 10], 40.00th=[ 14], 50.00th=[ 18], 60.00th=[ 22], | 70.00th=[ 28], 80.00th=[ 35], 90.00th=[ 45], 95.00th=[ 53], | 99.00th=[ 73], 99.50th=[ 83], 99.90th=[ 90], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 31, max= 440, per=24.91%, avg=101.64, stdev=82.72, samples=121 iops : min= 7, max= 110, avg=24.98, stdev=20.82, samples=121 lat (msec) : 4=3.78%, 10=30.89%, 20=28.29%, 50=31.18%, 100=5.79% lat (msec) : 250=0.03%, 750=0.03% cpu : usr=0.09%, sys=13.87%, ctx=4409, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=390KiB/s (399kB/s), 96.5KiB/s-98.4KiB/s (98.8kB/s-101kB/s), io=23.4MiB (24.6MB), run=61346-61578msec WRITE: bw=408KiB/s (418kB/s), 101KiB/s-103KiB/s (104kB/s-106kB/s), io=24.6MiB (25.7MB), run=61346-61578msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=190151: Mon Jul 14 13:00:40 2025 read: IOPS=23, BW=382KiB/s (391kB/s)(5920KiB/15498msec) clat (usec): min=250, max=70121, avg=21210.59, stdev=10417.81 lat (usec): min=251, max=70121, avg=21211.17, stdev=10417.87 clat percentiles (usec): | 1.00th=[ 424], 5.00th=[ 7898], 10.00th=[10159], 20.00th=[12387], | 30.00th=[15533], 40.00th=[17957], 50.00th=[19792], 60.00th=[22676], | 70.00th=[25035], 80.00th=[28967], 90.00th=[33817], 95.00th=[39584], | 99.00th=[52691], 99.50th=[63701], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 192, max= 672, per=24.81%, avg=382.60, stdev=104.78, samples=30 iops : min= 12, max= 42, avg=23.30, stdev= 6.56, samples=30 write: IOPS=25, BW=411KiB/s (421kB/s)(6368KiB/15498msec); 0 zone resets clat (usec): min=1416, max=74228, avg=19012.36, stdev=12240.82 lat (usec): min=1417, max=74230, avg=19014.41, stdev=12240.13 clat percentiles (usec): | 1.00th=[ 3326], 5.00th=[ 4555], 10.00th=[ 5407], 20.00th=[ 8586], | 30.00th=[10421], 40.00th=[13829], 50.00th=[16319], 60.00th=[18744], | 70.00th=[23725], 80.00th=[28443], 90.00th=[35390], 95.00th=[41157], | 99.00th=[58459], 99.50th=[69731], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 95, max= 670, per=25.54%, avg=414.30, stdev=128.64, samples=30 iops : min= 5, max= 41, avg=25.30, stdev= 8.06, samples=30 lat (usec) : 500=1.30%, 750=0.39%, 1000=0.13% lat (msec) : 2=0.65%, 4=0.91%, 10=15.63%, 20=38.15%, 50=40.89% lat (msec) : 100=1.95% cpu : usr=0.09%, sys=30.20%, ctx=2828, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190152: Mon Jul 14 13:00:40 2025 read: IOPS=23, BW=370KiB/s (379kB/s)(5744KiB/15528msec) clat (usec): min=285, max=80064, avg=22468.40, stdev=10906.66 lat (usec): min=285, max=80065, avg=22469.02, stdev=10906.68 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 9110], 10.00th=[10683], 20.00th=[13698], | 30.00th=[15533], 40.00th=[18220], 50.00th=[21103], 60.00th=[24249], | 70.00th=[27395], 80.00th=[30540], 90.00th=[35914], 95.00th=[41157], | 99.00th=[52691], 99.50th=[63701], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 190, max= 608, per=24.11%, avg=371.73, stdev=100.77, samples=30 iops : min= 11, max= 38, avg=22.63, stdev= 6.34, samples=30 write: IOPS=26, BW=421KiB/s (432kB/s)(6544KiB/15528msec); 0 zone resets clat (usec): min=983, max=76992, avg=18174.12, stdev=11802.08 lat (usec): min=984, max=76994, avg=18175.29, stdev=11802.16 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 4555], 10.00th=[ 5407], 20.00th=[ 8029], | 30.00th=[10814], 40.00th=[13173], 50.00th=[15533], 60.00th=[18482], | 70.00th=[22414], 80.00th=[26084], 90.00th=[34866], 95.00th=[42206], | 99.00th=[54264], 99.50th=[57934], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 127, max= 766, per=25.99%, avg=421.50, stdev=129.91, samples=30 iops : min= 7, max= 47, avg=25.73, stdev= 8.18, samples=30 lat (usec) : 500=0.78%, 750=0.65%, 1000=0.13% lat (msec) : 2=0.26%, 4=1.04%, 10=13.67%, 20=39.71%, 50=42.06% lat (msec) : 100=1.69% cpu : usr=0.08%, sys=29.90%, ctx=2864, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190153: Mon Jul 14 13:00:40 2025 read: IOPS=25, BW=405KiB/s (414kB/s)(6256KiB/15458msec) clat (usec): min=109, max=83156, avg=21520.70, stdev=11395.93 lat (usec): min=110, max=83157, avg=21521.39, stdev=11395.96 clat percentiles (usec): | 1.00th=[ 363], 5.00th=[ 8356], 10.00th=[10683], 20.00th=[13304], | 30.00th=[15533], 40.00th=[17433], 50.00th=[19792], 60.00th=[21890], | 70.00th=[24773], 80.00th=[28967], 90.00th=[35390], 95.00th=[41157], | 99.00th=[60556], 99.50th=[70779], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 190, max= 574, per=26.63%, avg=410.63, stdev=107.86, samples=30 iops : min= 11, max= 35, avg=25.07, stdev= 6.70, samples=30 write: IOPS=24, BW=390KiB/s (400kB/s)(6032KiB/15458msec); 0 zone resets clat (usec): min=1134, max=73082, avg=18467.87, stdev=11841.70 lat (usec): min=1135, max=73085, avg=18469.16, stdev=11841.78 clat percentiles (usec): | 1.00th=[ 3064], 5.00th=[ 4817], 10.00th=[ 6325], 20.00th=[ 8356], | 30.00th=[10421], 40.00th=[13042], 50.00th=[15533], 60.00th=[18744], | 70.00th=[23200], 80.00th=[27395], 90.00th=[33424], 95.00th=[41157], | 99.00th=[60556], 99.50th=[66323], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 126, max= 608, per=23.97%, avg=388.80, stdev=119.20, samples=30 iops : min= 7, max= 38, avg=23.70, stdev= 7.47, samples=30 lat (usec) : 250=0.26%, 500=1.56%, 750=0.26%, 1000=0.26% lat (msec) : 2=0.26%, 4=0.91%, 10=13.80%, 20=39.97%, 50=40.49% lat (msec) : 100=2.21% cpu : usr=0.09%, sys=29.65%, ctx=2883, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190154: Mon Jul 14 13:00:40 2025 read: IOPS=24, BW=389KiB/s (398kB/s)(6032KiB/15515msec) clat (usec): min=291, max=78560, avg=21655.75, stdev=11439.98 lat (usec): min=291, max=78561, avg=21656.33, stdev=11440.03 clat percentiles (usec): | 1.00th=[ 338], 5.00th=[ 529], 10.00th=[10421], 20.00th=[13960], | 30.00th=[16057], 40.00th=[17695], 50.00th=[20317], 60.00th=[22414], | 70.00th=[25560], 80.00th=[28967], 90.00th=[35914], 95.00th=[41157], | 99.00th=[59507], 99.50th=[73925], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 222, max= 605, per=25.16%, avg=387.97, stdev=101.50, samples=30 iops : min= 13, max= 37, avg=23.63, stdev= 6.37, samples=30 write: IOPS=25, BW=403KiB/s (413kB/s)(6256KiB/15515msec); 0 zone resets clat (usec): min=1164, max=73580, avg=18555.91, stdev=11420.30 lat (usec): min=1165, max=73581, avg=18557.03, stdev=11420.37 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 4555], 10.00th=[ 5604], 20.00th=[ 8979], | 30.00th=[11207], 40.00th=[13566], 50.00th=[16188], 60.00th=[19268], | 70.00th=[22938], 80.00th=[27132], 90.00th=[34866], 95.00th=[40109], | 99.00th=[52691], 99.50th=[60556], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 254, max= 702, per=24.87%, avg=403.37, stdev=102.11, samples=30 iops : min= 15, max= 43, avg=24.60, stdev= 6.37, samples=30 lat (usec) : 500=2.34%, 750=0.91% lat (msec) : 2=0.65%, 4=1.30%, 10=11.46%, 20=38.67%, 50=42.84% lat (msec) : 100=1.82% cpu : usr=0.06%, sys=29.50%, ctx=2848, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1543KiB/s (1580kB/s), 370KiB/s-405KiB/s (379kB/s-414kB/s), io=23.4MiB (24.5MB), run=15458-15528msec WRITE: bw=1623KiB/s (1662kB/s), 390KiB/s-421KiB/s (400kB/s-432kB/s), io=24.6MiB (25.8MB), run=15458-15528msec rand-rw: (groupid=0, jobs=1): err= 0: pid=190146: Mon Jul 14 13:00:43 2025 read: IOPS=20, BW=330KiB/s (338kB/s)(5920KiB/17954msec) clat (msec): min=2, max=104, avg=21.22, stdev=15.97 lat (msec): min=2, max=104, avg=21.22, stdev=15.97 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 12], 40.00th=[ 13], 50.00th=[ 16], 60.00th=[ 20], | 70.00th=[ 23], 80.00th=[ 32], 90.00th=[ 44], 95.00th=[ 55], | 99.00th=[ 77], 99.50th=[ 90], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 63, max= 672, per=23.68%, avg=315.94, stdev=146.49, samples=35 iops : min= 3, max= 42, avg=19.11, stdev= 9.24, samples=35 write: IOPS=22, BW=355KiB/s (363kB/s)(6368KiB/17954msec); 0 zone resets clat (msec): min=3, max=107, avg=25.25, stdev=16.38 lat (msec): min=3, max=107, avg=25.25, stdev=16.38 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 12], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 22], 60.00th=[ 25], | 70.00th=[ 30], 80.00th=[ 37], 90.00th=[ 48], 95.00th=[ 58], | 99.00th=[ 82], 99.50th=[ 87], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 159, max= 864, per=24.57%, avg=344.69, stdev=161.12, samples=35 iops : min= 9, max= 54, avg=20.91, stdev=10.18, samples=35 lat (msec) : 4=1.56%, 10=14.97%, 20=38.41%, 50=37.24%, 100=7.55% lat (msec) : 250=0.26% cpu : usr=0.06%, sys=14.44%, ctx=1228, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190147: Mon Jul 14 13:00:43 2025 read: IOPS=20, BW=321KiB/s (329kB/s)(5744KiB/17876msec) clat (usec): min=3711, max=97359, avg=21421.65, stdev=14657.32 lat (usec): min=3711, max=97359, avg=21422.38, stdev=14657.33 clat percentiles (usec): | 1.00th=[ 4146], 5.00th=[ 7898], 10.00th=[ 9503], 20.00th=[10683], | 30.00th=[12256], 40.00th=[13698], 50.00th=[15664], 60.00th=[19006], | 70.00th=[23725], 80.00th=[31065], 90.00th=[42206], 95.00th=[52691], | 99.00th=[72877], 99.50th=[94897], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 127, max= 576, per=23.79%, avg=317.34, stdev=115.58, samples=35 iops : min= 7, max= 36, avg=19.20, stdev= 7.30, samples=35 write: IOPS=22, BW=366KiB/s (375kB/s)(6544KiB/17876msec); 0 zone resets clat (usec): min=5483, max=86166, avg=24794.51, stdev=14982.08 lat (usec): min=5485, max=86167, avg=24796.25, stdev=14982.17 clat percentiles (usec): | 1.00th=[ 6783], 5.00th=[ 8717], 10.00th=[10028], 20.00th=[12125], | 30.00th=[14353], 40.00th=[17695], 50.00th=[20841], 60.00th=[24773], | 70.00th=[28443], 80.00th=[34866], 90.00th=[45351], 95.00th=[54264], | 99.00th=[72877], 99.50th=[82314], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 158, max= 960, per=25.59%, avg=359.03, stdev=169.17, samples=35 iops : min= 9, max= 60, avg=21.80, stdev=10.70, samples=35 lat (msec) : 4=0.26%, 10=11.98%, 20=42.71%, 50=38.67%, 100=6.38% cpu : usr=0.08%, sys=15.14%, ctx=1244, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190148: Mon Jul 14 13:00:43 2025 read: IOPS=22, BW=352KiB/s (361kB/s)(6256KiB/17758msec) clat (usec): min=5388, max=95787, avg=20535.04, stdev=14254.71 lat (usec): min=5389, max=95789, avg=20535.99, stdev=14254.66 clat percentiles (usec): | 1.00th=[ 6325], 5.00th=[ 8094], 10.00th=[ 8979], 20.00th=[10290], | 30.00th=[11863], 40.00th=[13566], 50.00th=[15401], 60.00th=[17957], | 70.00th=[21103], 80.00th=[28705], 90.00th=[41681], 95.00th=[53216], | 99.00th=[67634], 99.50th=[73925], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 95, max= 832, per=26.48%, avg=353.20, stdev=178.42, samples=35 iops : min= 5, max= 52, avg=21.43, stdev=11.29, samples=35 write: IOPS=21, BW=340KiB/s (348kB/s)(6032KiB/17758msec); 0 zone resets clat (usec): min=5824, max=79909, avg=25749.29, stdev=15063.38 lat (usec): min=5827, max=79912, avg=25751.07, stdev=15063.43 clat percentiles (usec): | 1.00th=[ 7635], 5.00th=[ 9241], 10.00th=[10290], 20.00th=[12387], | 30.00th=[15139], 40.00th=[17957], 50.00th=[21890], 60.00th=[25822], | 70.00th=[31851], 80.00th=[36963], 90.00th=[47973], 95.00th=[55313], | 99.00th=[72877], 99.50th=[76022], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 191, max= 734, per=24.05%, avg=337.46, stdev=118.97, samples=35 iops : min= 11, max= 45, avg=20.46, stdev= 7.50, samples=35 lat (msec) : 10=12.89%, 20=41.80%, 50=38.02%, 100=7.29% cpu : usr=0.08%, sys=15.26%, ctx=1285, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190149: Mon Jul 14 13:00:43 2025 read: IOPS=21, BW=339KiB/s (347kB/s)(6032KiB/17777msec) clat (usec): min=3475, max=89217, avg=20059.71, stdev=13657.17 lat (usec): min=3475, max=89218, avg=20060.48, stdev=13657.19 clat percentiles (usec): | 1.00th=[ 6063], 5.00th=[ 7701], 10.00th=[ 8455], 20.00th=[ 9765], | 30.00th=[11600], 40.00th=[13042], 50.00th=[15401], 60.00th=[18220], | 70.00th=[21890], 80.00th=[28705], 90.00th=[39060], 95.00th=[50594], | 99.00th=[62653], 99.50th=[78119], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 63, max= 928, per=25.22%, avg=336.40, stdev=169.77, samples=35 iops : min= 3, max= 58, avg=20.40, stdev=10.71, samples=35 write: IOPS=21, BW=352KiB/s (360kB/s)(6256KiB/17777msec); 0 zone resets clat (usec): min=6758, max=97772, avg=26031.65, stdev=15292.19 lat (usec): min=6760, max=97774, avg=26033.42, stdev=15292.21 clat percentiles (usec): | 1.00th=[ 7767], 5.00th=[ 9372], 10.00th=[11076], 20.00th=[13566], | 30.00th=[15401], 40.00th=[18744], 50.00th=[22414], 60.00th=[25297], | 70.00th=[30802], 80.00th=[36963], 90.00th=[47973], 95.00th=[56361], | 99.00th=[79168], 99.50th=[91751], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 127, max= 704, per=25.13%, avg=352.57, stdev=126.05, samples=35 iops : min= 7, max= 44, avg=21.40, stdev= 7.99, samples=35 lat (msec) : 4=0.13%, 10=13.80%, 20=40.36%, 50=38.67%, 100=7.03% cpu : usr=0.08%, sys=15.26%, ctx=1308, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1334KiB/s (1366kB/s), 321KiB/s-352KiB/s (329kB/s-361kB/s), io=23.4MiB (24.5MB), run=17758-17954msec WRITE: bw=1404KiB/s (1437kB/s), 340KiB/s-366KiB/s (348kB/s-375kB/s), io=24.6MiB (25.8MB), run=17758-17954msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=190162: Mon Jul 14 13:00:51 2025 read: IOPS=3, BW=3100KiB/s (3175kB/s)(7168KiB/2312msec) clat (usec): min=779, max=278065, avg=127298.50, stdev=97881.86 lat (usec): min=780, max=278066, avg=127299.69, stdev=97881.99 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 783], 10.00th=[ 783], 20.00th=[ 807], | 30.00th=[147850], 40.00th=[147850], 50.00th=[149947], 60.00th=[152044], | 70.00th=[152044], 80.00th=[162530], 90.00th=[278922], 95.00th=[278922], | 99.00th=[278922], 99.50th=[278922], 99.90th=[278922], 99.95th=[278922], | 99.99th=[278922] bw ( KiB/s): min= 2000, max= 5215, per=39.32%, avg=2825.50, stdev=1593.14, samples=4 iops : min= 1, max= 5, avg= 2.25, stdev= 1.89, samples=4 write: IOPS=2, BW=2215KiB/s (2268kB/s)(5120KiB/2312msec); 0 zone resets clat (msec): min=170, max=380, avg=279.40, stdev=88.61 lat (msec): min=170, max=380, avg=279.47, stdev=88.59 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 230], 40.00th=[ 230], 50.00th=[ 259], 60.00th=[ 259], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 380], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 2039, max= 4000, per=26.80%, avg=2695.67, stdev=1129.59, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 lat (usec) : 1000=16.67% lat (msec) : 250=50.00%, 500=33.33% cpu : usr=0.00%, sys=43.27%, ctx=646, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190163: Mon Jul 14 13:00:51 2025 read: IOPS=0, BW=719KiB/s (736kB/s)(2048KiB/2850msec) clat (usec): min=899, max=117790, avg=59344.56, stdev=82654.49 lat (usec): min=900, max=117793, avg=59346.84, stdev=82655.43 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 898], 10.00th=[ 898], 20.00th=[ 898], | 30.00th=[ 898], 40.00th=[ 898], 50.00th=[ 898], 60.00th=[117965], | 70.00th=[117965], 80.00th=[117965], 90.00th=[117965], 95.00th=[117965], | 99.00th=[117965], 99.50th=[117965], 99.90th=[117965], 99.95th=[117965], | 99.99th=[117965] bw ( KiB/s): min= 1732, max= 2000, per=25.97%, avg=1866.00, stdev=189.50, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3593KiB/s (3679kB/s)(10.0MiB/2850msec); 0 zone resets clat (msec): min=80, max=562, avg=270.54, stdev=188.14 lat (msec): min=81, max=562, avg=270.61, stdev=188.13 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 82], 10.00th=[ 82], 20.00th=[ 99], | 30.00th=[ 140], 40.00th=[ 146], 50.00th=[ 171], 60.00th=[ 211], | 70.00th=[ 271], 80.00th=[ 472], 90.00th=[ 558], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 2000, max= 6144, per=40.48%, avg=4072.50, stdev=2370.70, samples=4 iops : min= 1, max= 6, avg= 3.25, stdev= 2.63, samples=4 lat (usec) : 1000=8.33% lat (msec) : 100=16.67%, 250=41.67%, 500=16.67%, 750=16.67% cpu : usr=0.00%, sys=45.88%, ctx=351, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190164: Mon Jul 14 13:00:51 2025 read: IOPS=2, BW=2920KiB/s (2990kB/s)(6144KiB/2104msec) clat (usec): min=646, max=453101, avg=121663.69, stdev=176008.98 lat (usec): min=647, max=453102, avg=121664.79, stdev=176008.89 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 644], 10.00th=[ 644], 20.00th=[ 750], | 30.00th=[ 750], 40.00th=[ 816], 50.00th=[ 816], 60.00th=[119014], | 70.00th=[156238], 80.00th=[156238], 90.00th=[455082], 95.00th=[455082], | 99.00th=[455082], 99.50th=[455082], 99.90th=[455082], 99.95th=[455082], | 99.99th=[455082] bw ( KiB/s): min= 8159, max= 8159, per=100.00%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2920KiB/s (2990kB/s)(6144KiB/2104msec); 0 zone resets clat (msec): min=145, max=401, avg=226.98, stdev=118.58 lat (msec): min=145, max=402, avg=227.06, stdev=118.58 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 146], 10.00th=[ 146], 20.00th=[ 153], | 30.00th=[ 153], 40.00th=[ 153], 50.00th=[ 153], 60.00th=[ 155], | 70.00th=[ 355], 80.00th=[ 355], 90.00th=[ 401], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 2039, max= 3453, per=24.98%, avg=2513.33, stdev=813.79, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 lat (usec) : 750=16.67%, 1000=8.33% lat (msec) : 250=50.00%, 500=25.00% cpu : usr=0.05%, sys=48.83%, ctx=262, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190165: Mon Jul 14 13:00:51 2025 read: IOPS=2, BW=2482KiB/s (2541kB/s)(5120KiB/2063msec) clat (usec): min=836, max=120524, avg=27784.43, stdev=52072.32 lat (usec): min=837, max=120525, avg=27785.58, stdev=52072.39 clat percentiles (usec): | 1.00th=[ 840], 5.00th=[ 840], 10.00th=[ 840], 20.00th=[ 840], | 30.00th=[ 1205], 40.00th=[ 1205], 50.00th=[ 3490], 60.00th=[ 3490], | 70.00th=[ 12911], 80.00th=[ 12911], 90.00th=[120062], 95.00th=[120062], | 99.00th=[120062], 99.50th=[120062], 99.90th=[120062], 99.95th=[120062], | 99.99th=[120062] bw ( KiB/s): min= 8159, max= 8159, per=100.00%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3475KiB/s (3558kB/s)(7168KiB/2063msec); 0 zone resets clat (msec): min=181, max=595, avg=272.98, stdev=146.88 lat (msec): min=181, max=595, avg=273.08, stdev=146.87 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 182], 10.00th=[ 182], 20.00th=[ 197], | 30.00th=[ 199], 40.00th=[ 199], 50.00th=[ 207], 60.00th=[ 243], | 70.00th=[ 243], 80.00th=[ 292], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 1780, max= 4079, per=26.20%, avg=2635.67, stdev=1257.13, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 lat (usec) : 1000=8.33% lat (msec) : 2=8.33%, 4=8.33%, 20=8.33%, 250=50.00%, 500=8.33% lat (msec) : 750=8.33% cpu : usr=0.00%, sys=46.31%, ctx=183, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7186KiB/s (7358kB/s), 719KiB/s-3100KiB/s (736kB/s-3175kB/s), io=20.0MiB (20.0MB), run=2063-2850msec WRITE: bw=9.82MiB/s (10.3MB/s), 2215KiB/s-3593KiB/s (2268kB/s-3679kB/s), io=28.0MiB (29.4MB), run=2063-2850msec rand-rw: (groupid=0, jobs=1): err= 0: pid=190167: Mon Jul 14 13:00:51 2025 read: IOPS=2, BW=2282KiB/s (2337kB/s)(7168KiB/3141msec) clat (msec): min=63, max=702, avg=281.28, stdev=280.37 lat (msec): min=63, max=702, avg=281.28, stdev=280.37 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 77], | 30.00th=[ 82], 40.00th=[ 82], 50.00th=[ 83], 60.00th=[ 334], | 70.00th=[ 334], 80.00th=[ 634], 90.00th=[ 701], 95.00th=[ 701], | 99.00th=[ 701], 99.50th=[ 701], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 1906, max= 4063, per=37.25%, avg=2403.80, stdev=929.55, samples=5 iops : min= 1, max= 3, avg= 1.80, stdev= 0.84, samples=5 write: IOPS=1, BW=1630KiB/s (1669kB/s)(5120KiB/3141msec); 0 zone resets clat (msec): min=66, max=380, avg=196.44, stdev=118.68 lat (msec): min=66, max=381, avg=196.53, stdev=118.70 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 67], | 30.00th=[ 125], 40.00th=[ 125], 50.00th=[ 197], 60.00th=[ 197], | 70.00th=[ 213], 80.00th=[ 213], 90.00th=[ 380], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 1921, max= 8192, per=55.96%, avg=5056.50, stdev=4434.27, samples=2 iops : min= 1, max= 8, avg= 4.50, stdev= 4.95, samples=2 lat (msec) : 100=41.67%, 250=25.00%, 500=16.67%, 750=16.67% cpu : usr=0.00%, sys=15.13%, ctx=86, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190168: Mon Jul 14 13:00:51 2025 read: IOPS=0, BW=665KiB/s (681kB/s)(2048KiB/3081msec) clat (msec): min=145, max=225, avg=185.43, stdev=56.80 lat (msec): min=145, max=225, avg=185.43, stdev=56.80 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 146], 10.00th=[ 146], 20.00th=[ 146], | 30.00th=[ 146], 40.00th=[ 146], 50.00th=[ 146], 60.00th=[ 226], | 70.00th=[ 226], 80.00th=[ 226], 90.00th=[ 226], 95.00th=[ 226], | 99.00th=[ 226], 99.50th=[ 226], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 1965, max= 2031, per=30.96%, avg=1998.00, stdev=46.67, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3324KiB/s (3403kB/s)(10.0MiB/3081msec); 0 zone resets clat (msec): min=81, max=417, avg=255.49, stdev=121.74 lat (msec): min=81, max=417, avg=255.59, stdev=121.71 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 82], 10.00th=[ 82], 20.00th=[ 110], | 30.00th=[ 127], 40.00th=[ 190], 50.00th=[ 271], 60.00th=[ 300], | 70.00th=[ 305], 80.00th=[ 376], 90.00th=[ 380], 95.00th=[ 418], | 99.00th=[ 418], 99.50th=[ 418], 99.90th=[ 418], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 2031, max= 3908, per=32.73%, avg=2957.25, stdev=1060.09, samples=4 iops : min= 1, max= 3, avg= 2.25, stdev= 0.96, samples=4 lat (msec) : 100=8.33%, 250=41.67%, 500=50.00% cpu : usr=0.06%, sys=21.79%, ctx=163, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190169: Mon Jul 14 13:00:51 2025 read: IOPS=2, BW=2153KiB/s (2204kB/s)(6144KiB/2854msec) clat (msec): min=57, max=426, avg=137.85, stdev=143.92 lat (msec): min=57, max=426, avg=137.85, stdev=143.92 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 60], | 30.00th=[ 60], 40.00th=[ 63], 50.00th=[ 63], 60.00th=[ 94], | 70.00th=[ 129], 80.00th=[ 129], 90.00th=[ 426], 95.00th=[ 426], | 99.00th=[ 426], 99.50th=[ 426], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 1921, max= 4063, per=41.48%, avg=2677.33, stdev=1201.70, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 write: IOPS=2, BW=2153KiB/s (2204kB/s)(6144KiB/2854msec); 0 zone resets clat (msec): min=51, max=554, avg=311.46, stdev=184.31 lat (msec): min=51, max=554, avg=311.55, stdev=184.28 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 255], 50.00th=[ 255], 60.00th=[ 372], | 70.00th=[ 456], 80.00th=[ 456], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 1921, max= 2048, per=21.81%, avg=1971.00, stdev=54.22, samples=4 iops : min= 1, max= 2, avg= 1.25, stdev= 0.50, samples=4 lat (msec) : 100=41.67%, 250=16.67%, 500=33.33%, 750=8.33% cpu : usr=0.00%, sys=18.65%, ctx=123, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190170: Mon Jul 14 13:00:51 2025 read: IOPS=1, BW=1614KiB/s (1652kB/s)(5120KiB/3173msec) clat (msec): min=36, max=367, avg=154.44, stdev=130.50 lat (msec): min=36, max=367, avg=154.44, stdev=130.50 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 37], | 30.00th=[ 63], 40.00th=[ 63], 50.00th=[ 148], 60.00th=[ 148], | 70.00th=[ 159], 80.00th=[ 159], 90.00th=[ 368], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 2048, max= 6095, per=63.08%, avg=4071.50, stdev=2861.66, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=2, BW=2259KiB/s (2313kB/s)(7168KiB/3173msec); 0 zone resets clat (msec): min=49, max=898, avg=333.83, stdev=292.67 lat (msec): min=49, max=898, avg=333.91, stdev=292.67 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 105], | 30.00th=[ 118], 40.00th=[ 118], 50.00th=[ 351], 60.00th=[ 368], | 70.00th=[ 368], 80.00th=[ 451], 90.00th=[ 902], 95.00th=[ 902], | 99.00th=[ 902], 99.50th=[ 902], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 1921, max= 4096, per=26.61%, avg=2404.80, stdev=947.18, samples=5 iops : min= 1, max= 4, avg= 1.80, stdev= 1.30, samples=5 lat (msec) : 50=16.67%, 100=8.33%, 250=33.33%, 500=33.33%, 1000=8.33% cpu : usr=0.00%, sys=17.18%, ctx=103, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6454KiB/s (6609kB/s), 665KiB/s-2282KiB/s (681kB/s-2337kB/s), io=20.0MiB (20.0MB), run=2854-3173msec WRITE: bw=9036KiB/s (9253kB/s), 1630KiB/s-3324KiB/s (1669kB/s-3403kB/s), io=28.0MiB (29.4MB), run=2854-3173msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=190175: Mon Jul 14 13:01:01 2025 read: IOPS=1, BW=6471KiB/s (6626kB/s)(12.0MiB/1899msec) clat (msec): min=8, max=1068, avg=619.29, stdev=547.83 lat (msec): min=8, max=1068, avg=619.29, stdev=547.83 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 9], 10.00th=[ 9], 20.00th=[ 9], | 30.00th=[ 9], 40.00th=[ 785], 50.00th=[ 785], 60.00th=[ 785], | 70.00th=[ 1070], 80.00th=[ 1070], 90.00th=[ 1070], 95.00th=[ 1070], | 99.00th=[ 1070], 99.50th=[ 1070], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 8159, max= 8159, per=100.00%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 10=33.33%, 1000=33.33%, 2000=33.33% cpu : usr=0.05%, sys=17.54%, ctx=719, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190176: Mon Jul 14 13:01:01 2025 read: IOPS=0, BW=1136KiB/s (1163kB/s)(4096KiB/3606msec) clat (nsec): min=673863k, max=673863k, avg=673862961.00, stdev= 0.00 lat (nsec): min=673866k, max=673866k, avg=673865563.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 676], 5.00th=[ 676], 10.00th=[ 676], 20.00th=[ 676], | 30.00th=[ 676], 40.00th=[ 676], 50.00th=[ 676], 60.00th=[ 676], | 70.00th=[ 676], 80.00th=[ 676], 90.00th=[ 676], 95.00th=[ 676], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 8159, max= 8159, per=100.00%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2272KiB/s (2326kB/s)(8192KiB/3606msec); 0 zone resets clat (msec): min=1095, max=1705, avg=1400.33, stdev=431.55 lat (msec): min=1095, max=1705, avg=1400.59, stdev=431.55 clat percentiles (msec): | 1.00th=[ 1099], 5.00th=[ 1099], 10.00th=[ 1099], 20.00th=[ 1099], | 30.00th=[ 1099], 40.00th=[ 1099], 50.00th=[ 1099], 60.00th=[ 1703], | 70.00th=[ 1703], 80.00th=[ 1703], 90.00th=[ 1703], 95.00th=[ 1703], | 99.00th=[ 1703], 99.50th=[ 1703], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 8015, max= 8015, per=88.20%, avg=8015.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 750=33.33%, 2000=66.67% cpu : usr=0.00%, sys=15.92%, ctx=224, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190177: Mon Jul 14 13:01:01 2025 write: IOPS=0, BW=4029KiB/s (4126kB/s)(12.0MiB/3050msec); 0 zone resets clat (msec): min=824, max=1111, avg=989.42, stdev=148.22 lat (msec): min=824, max=1111, avg=989.71, stdev=148.13 clat percentiles (msec): | 1.00th=[ 827], 5.00th=[ 827], 10.00th=[ 827], 20.00th=[ 827], | 30.00th=[ 827], 40.00th=[ 1036], 50.00th=[ 1036], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1116], 90.00th=[ 1116], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 8015, max= 8031, per=88.29%, avg=8023.00, stdev=11.31, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 1000=33.33%, 2000=66.67% cpu : usr=0.00%, sys=29.68%, ctx=119, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190179: Mon Jul 14 13:01:01 2025 write: IOPS=0, BW=3965KiB/s (4060kB/s)(12.0MiB/3099msec); 0 zone resets clat (msec): min=625, max=1315, avg=1010.84, stdev=352.10 lat (msec): min=625, max=1315, avg=1011.09, stdev=352.10 clat percentiles (msec): | 1.00th=[ 625], 5.00th=[ 625], 10.00th=[ 625], 20.00th=[ 625], | 30.00th=[ 625], 40.00th=[ 1099], 50.00th=[ 1099], 60.00th=[ 1099], | 70.00th=[ 1318], 80.00th=[ 1318], 90.00th=[ 1318], 95.00th=[ 1318], | 99.00th=[ 1318], 99.50th=[ 1318], 99.90th=[ 1318], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 7301, max= 8031, per=84.36%, avg=7666.00, stdev=516.19, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 750=33.33%, 2000=66.67% cpu : usr=0.00%, sys=29.47%, ctx=131, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4544KiB/s (4653kB/s), 1136KiB/s-6471KiB/s (1163kB/s-6626kB/s), io=16.0MiB (16.8MB), run=1899-3606msec WRITE: bw=9087KiB/s (9305kB/s), 2272KiB/s-4029KiB/s (2326kB/s-4126kB/s), io=32.0MiB (33.6MB), run=3050-3606msec rand-rw: (groupid=0, jobs=1): err= 0: pid=190180: Mon Jul 14 13:01:01 2025 read: IOPS=0, BW=3495KiB/s (3579kB/s)(12.0MiB/3516msec) clat (msec): min=427, max=1593, avg=1134.87, stdev=621.81 lat (msec): min=427, max=1593, avg=1134.87, stdev=621.81 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 426], 10.00th=[ 426], 20.00th=[ 426], | 30.00th=[ 426], 40.00th=[ 1385], 50.00th=[ 1385], 60.00th=[ 1385], | 70.00th=[ 1586], 80.00th=[ 1586], 90.00th=[ 1586], 95.00th=[ 1586], | 99.00th=[ 1586], 99.50th=[ 1586], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 8031, max= 8143, per=100.00%, avg=8087.00, stdev=79.20, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 2000=66.67% cpu : usr=0.03%, sys=9.08%, ctx=102, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190181: Mon Jul 14 13:01:01 2025 read: IOPS=0, BW=1028KiB/s (1052kB/s)(4096KiB/3986msec) clat (nsec): min=1430.1M, max=1430.1M, avg=1430072835.00, stdev= 0.00 lat (nsec): min=1430.1M, max=1430.1M, avg=1430075031.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1435], 5.00th=[ 1435], 10.00th=[ 1435], 20.00th=[ 1435], | 30.00th=[ 1435], 40.00th=[ 1435], 50.00th=[ 1435], 60.00th=[ 1435], | 70.00th=[ 1435], 80.00th=[ 1435], 90.00th=[ 1435], 95.00th=[ 1435], | 99.00th=[ 1435], 99.50th=[ 1435], 99.90th=[ 1435], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 8031, max= 8031, per=100.00%, avg=8031.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2055KiB/s (2105kB/s)(8192KiB/3986msec); 0 zone resets clat (msec): min=700, max=1669, avg=1185.33, stdev=685.43 lat (msec): min=700, max=1670, avg=1185.54, stdev=685.37 clat percentiles (msec): | 1.00th=[ 701], 5.00th=[ 701], 10.00th=[ 701], 20.00th=[ 701], | 30.00th=[ 701], 40.00th=[ 701], 50.00th=[ 701], 60.00th=[ 1670], | 70.00th=[ 1670], 80.00th=[ 1670], 90.00th=[ 1670], 95.00th=[ 1670], | 99.00th=[ 1670], 99.50th=[ 1670], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 750=33.33%, 2000=66.67% cpu : usr=0.05%, sys=7.53%, ctx=74, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190182: Mon Jul 14 13:01:01 2025 write: IOPS=0, BW=2957KiB/s (3028kB/s)(12.0MiB/4156msec); 0 zone resets clat (msec): min=133, max=2206, avg=1328.86, stdev=1072.33 lat (msec): min=133, max=2206, avg=1329.15, stdev=1072.31 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 134], 10.00th=[ 134], 20.00th=[ 134], | 30.00th=[ 134], 40.00th=[ 1653], 50.00th=[ 1653], 60.00th=[ 1653], | 70.00th=[ 2198], 80.00th=[ 2198], 90.00th=[ 2198], 95.00th=[ 2198], | 99.00th=[ 2198], 99.50th=[ 2198], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7937, max= 7937, per=100.00%, avg=7937.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 2000=33.33%, >=2000=33.33% cpu : usr=0.00%, sys=4.79%, ctx=48, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190183: Mon Jul 14 13:01:01 2025 write: IOPS=1, BW=6175KiB/s (6323kB/s)(12.0MiB/1990msec); 0 zone resets clat (msec): min=461, max=967, avg=646.11, stdev=279.78 lat (msec): min=461, max=968, avg=646.60, stdev=279.68 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 460], 10.00th=[ 460], 20.00th=[ 460], | 30.00th=[ 460], 40.00th=[ 510], 50.00th=[ 510], 60.00th=[ 510], | 70.00th=[ 969], 80.00th=[ 969], 90.00th=[ 969], 95.00th=[ 969], | 99.00th=[ 969], 99.50th=[ 969], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 6206, max= 8159, per=91.10%, avg=7182.50, stdev=1380.98, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=7.64%, ctx=49, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4110KiB/s (4209kB/s), 1028KiB/s-3495KiB/s (1052kB/s-3579kB/s), io=16.0MiB (16.8MB), run=3516-3986msec WRITE: bw=7885KiB/s (8074kB/s), 2055KiB/s-6175KiB/s (2105kB/s-6323kB/s), io=32.0MiB (33.6MB), run=1990-4156msec