== sanity test 398b: DIO and buffer IO race ============== 14:28:11 (1752690491) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB, 48 MiB) copied, 2.63615 s, 19.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=193899: Wed Jul 16 14:29:21 2025 read: IOPS=24, BW=99.0KiB/s (101kB/s)(5992KiB/60501msec) clat (usec): min=102, max=719051, avg=22113.41, stdev=21311.53 lat (usec): min=103, max=719052, avg=22113.91, stdev=21311.52 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 7635], 10.00th=[ 9765], 20.00th=[ 12649], | 30.00th=[ 15008], 40.00th=[ 17171], 50.00th=[ 19792], 60.00th=[ 22152], | 70.00th=[ 25297], 80.00th=[ 29754], 90.00th=[ 36439], 95.00th=[ 42730], | 99.00th=[ 59507], 99.50th=[ 68682], 99.90th=[ 94897], 99.95th=[717226], | 99.99th=[717226] bw ( KiB/s): min= 14, max= 192, per=25.20%, avg=98.78, stdev=30.50, samples=120 iops : min= 3, max= 48, avg=24.30, stdev= 7.69, samples=120 write: IOPS=26, BW=104KiB/s (107kB/s)(6296KiB/60501msec); 0 zone resets clat (usec): min=592, max=80481, avg=17261.64, stdev=12754.89 lat (usec): min=593, max=80483, avg=17262.28, stdev=12754.95 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 2507], 10.00th=[ 3425], 20.00th=[ 5473], | 30.00th=[ 8225], 40.00th=[11469], 50.00th=[14615], 60.00th=[17695], | 70.00th=[21890], 80.00th=[27919], 90.00th=[35390], 95.00th=[41681], | 99.00th=[54264], 99.50th=[59507], 99.90th=[74974], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 40, max= 207, per=25.48%, avg=104.73, stdev=32.13, samples=119 iops : min= 10, max= 51, avg=25.79, stdev= 8.09, samples=119 lat (usec) : 250=0.20%, 500=0.29%, 750=0.81%, 1000=0.52% lat (msec) : 2=0.72%, 4=5.01%, 10=15.82%, 20=35.48%, 50=39.16% lat (msec) : 100=1.95%, 750=0.03% cpu : usr=0.06%, sys=26.53%, ctx=8738, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193901: Wed Jul 16 14:29:21 2025 read: IOPS=24, BW=98.9KiB/s (101kB/s)(5940KiB/60038msec) clat (usec): min=119, max=102412, avg=21682.80, stdev=11684.52 lat (usec): min=120, max=102412, avg=21683.31, stdev=11684.53 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 7767], 10.00th=[ 9372], 20.00th=[ 11994], | 30.00th=[ 14877], 40.00th=[ 17433], 50.00th=[ 19268], 60.00th=[ 22152], | 70.00th=[ 25822], 80.00th=[ 30016], 90.00th=[ 36439], 95.00th=[ 42730], | 99.00th=[ 60031], 99.50th=[ 71828], 99.90th=[ 79168], 99.95th=[102237], | 99.99th=[102237] bw ( KiB/s): min= 7, max= 200, per=25.17%, avg=98.68, stdev=33.71, samples=119 iops : min= 1, max= 50, avg=24.27, stdev= 8.51, samples=119 write: IOPS=26, BW=106KiB/s (108kB/s)(6348KiB/60038msec); 0 zone resets clat (usec): min=632, max=685158, avg=17412.46, stdev=21440.87 lat (usec): min=632, max=685159, avg=17414.79, stdev=21440.30 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 2442], 10.00th=[ 3163], 20.00th=[ 5145], | 30.00th=[ 7767], 40.00th=[ 10683], 50.00th=[ 13698], 60.00th=[ 17171], | 70.00th=[ 21365], 80.00th=[ 27395], 90.00th=[ 35390], 95.00th=[ 42730], | 99.00th=[ 60556], 99.50th=[ 71828], 99.90th=[ 83362], 99.95th=[683672], | 99.99th=[683672] bw ( KiB/s): min= 7, max= 224, per=25.63%, avg=105.32, stdev=36.78, samples=119 iops : min= 1, max= 56, avg=25.93, stdev= 9.29, samples=119 lat (usec) : 250=0.03%, 500=0.29%, 750=0.91%, 1000=0.55% lat (msec) : 2=0.94%, 4=5.73%, 10=16.70%, 20=34.96%, 50=37.53% lat (msec) : 100=2.28%, 250=0.03%, 750=0.03% cpu : usr=0.07%, sys=26.37%, ctx=8710, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193902: Wed Jul 16 14:29:21 2025 read: IOPS=24, BW=98.6KiB/s (101kB/s)(6028KiB/61128msec) clat (usec): min=102, max=95595, avg=21900.67, stdev=12500.42 lat (usec): min=102, max=95595, avg=21901.21, stdev=12500.42 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 7504], 10.00th=[ 8979], 20.00th=[11731], | 30.00th=[14615], 40.00th=[16909], 50.00th=[19530], 60.00th=[22152], | 70.00th=[25822], 80.00th=[30016], 90.00th=[38011], 95.00th=[44827], | 99.00th=[64226], 99.50th=[70779], 99.90th=[81265], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 14, max= 184, per=24.99%, avg=97.97, stdev=31.64, samples=121 iops : min= 3, max= 46, avg=24.10, stdev= 7.98, samples=121 write: IOPS=25, BW=102KiB/s (105kB/s)(6260KiB/61128msec); 0 zone resets clat (usec): min=647, max=702545, avg=17841.25, stdev=21751.85 lat (usec): min=647, max=702545, avg=17842.93, stdev=21751.60 clat percentiles (usec): | 1.00th=[ 816], 5.00th=[ 2606], 10.00th=[ 3195], 20.00th=[ 5604], | 30.00th=[ 8455], 40.00th=[ 11863], 50.00th=[ 14484], 60.00th=[ 17957], | 70.00th=[ 22152], 80.00th=[ 27132], 90.00th=[ 35914], 95.00th=[ 43779], | 99.00th=[ 61604], 99.50th=[ 64226], 99.90th=[ 81265], 99.95th=[700449], | 99.99th=[700449] bw ( KiB/s): min= 21, max= 192, per=24.68%, avg=101.45, stdev=34.32, samples=121 iops : min= 5, max= 48, avg=24.98, stdev= 8.68, samples=121 lat (usec) : 250=0.10%, 500=0.33%, 750=0.88%, 1000=0.42% lat (msec) : 2=0.88%, 4=5.11%, 10=16.15%, 20=35.35%, 50=37.92% lat (msec) : 100=2.83%, 750=0.03% cpu : usr=0.10%, sys=26.02%, ctx=8779, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193904: Wed Jul 16 14:29:21 2025 read: IOPS=24, BW=99.5KiB/s (102kB/s)(6052KiB/60843msec) clat (usec): min=98, max=698875, avg=21994.13, stdev=20944.82 lat (usec): min=99, max=698876, avg=21994.66, stdev=20944.82 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 7701], 10.00th=[ 9372], 20.00th=[ 11863], | 30.00th=[ 14877], 40.00th=[ 17171], 50.00th=[ 19792], 60.00th=[ 22152], | 70.00th=[ 25560], 80.00th=[ 30016], 90.00th=[ 35914], 95.00th=[ 42206], | 99.00th=[ 61604], 99.50th=[ 72877], 99.90th=[117965], 99.95th=[700449], | 99.99th=[700449] bw ( KiB/s): min= 39, max= 184, per=25.53%, avg=100.06, stdev=32.68, samples=120 iops : min= 9, max= 46, avg=24.63, stdev= 8.27, samples=120 write: IOPS=25, BW=102KiB/s (105kB/s)(6236KiB/60843msec); 0 zone resets clat (usec): min=670, max=79342, avg=17544.38, stdev=13494.81 lat (usec): min=670, max=79343, avg=17545.01, stdev=13494.84 clat percentiles (usec): | 1.00th=[ 816], 5.00th=[ 2540], 10.00th=[ 3228], 20.00th=[ 5276], | 30.00th=[ 7963], 40.00th=[11338], 50.00th=[14353], 60.00th=[17957], | 70.00th=[22414], 80.00th=[28181], 90.00th=[36963], 95.00th=[44303], | 99.00th=[58983], 99.50th=[65799], 99.90th=[77071], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 29, max= 200, per=24.87%, avg=102.20, stdev=35.43, samples=121 iops : min= 7, max= 50, avg=25.16, stdev= 8.95, samples=121 lat (usec) : 100=0.03%, 250=0.03%, 500=0.46%, 750=0.78%, 1000=0.55% lat (msec) : 2=0.98%, 4=5.96%, 10=15.23%, 20=34.67%, 50=38.96% lat (msec) : 100=2.28%, 250=0.03%, 750=0.03% cpu : usr=0.06%, sys=26.24%, ctx=8944, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=393KiB/s (402kB/s), 98.6KiB/s-99.5KiB/s (101kB/s-102kB/s), io=23.4MiB (24.6MB), run=60038-61128msec WRITE: bw=411KiB/s (421kB/s), 102KiB/s-106KiB/s (105kB/s-108kB/s), io=24.6MiB (25.7MB), run=60038-61128msec rand-rw: (groupid=0, jobs=1): err= 0: pid=193900: Wed Jul 16 14:29:33 2025 read: IOPS=20, BW=81.8KiB/s (83.8kB/s)(5992KiB/73250msec) clat (msec): min=3, max=723, avg=21.41, stdev=23.39 lat (msec): min=3, max=723, avg=21.41, stdev=23.39 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 12], 40.00th=[ 14], 50.00th=[ 17], 60.00th=[ 21], | 70.00th=[ 24], 80.00th=[ 31], 90.00th=[ 43], 95.00th=[ 52], | 99.00th=[ 73], 99.50th=[ 84], 99.90th=[ 122], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 23, max= 272, per=25.11%, avg=81.85, stdev=48.46, samples=144 iops : min= 5, max= 68, avg=20.14, stdev=12.21, samples=144 write: IOPS=21, BW=85.0KiB/s (88.0kB/s)(6296KiB/73250msec); 0 zone resets clat (msec): min=3, max=156, avg=26.04, stdev=17.49 lat (msec): min=3, max=156, avg=26.04, stdev=17.49 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 12], | 30.00th=[ 15], 40.00th=[ 18], 50.00th=[ 22], 60.00th=[ 26], | 70.00th=[ 32], 80.00th=[ 39], 90.00th=[ 49], 95.00th=[ 61], | 99.00th=[ 83], 99.50th=[ 94], 99.90th=[ 129], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 16, max= 264, per=25.09%, avg=85.79, stdev=45.48, samples=144 iops : min= 4, max= 66, avg=21.13, stdev=11.42, samples=144 lat (msec) : 4=0.20%, 10=19.50%, 20=33.98%, 50=38.57%, 100=7.49% lat (msec) : 250=0.23%, 750=0.03% cpu : usr=0.06%, sys=14.49%, ctx=4888, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193903: Wed Jul 16 14:29:33 2025 read: IOPS=20, BW=80.9KiB/s (82.8kB/s)(5940KiB/73467msec) clat (msec): min=2, max=703, avg=21.63, stdev=23.48 lat (msec): min=2, max=703, avg=21.63, stdev=23.48 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 12], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 20], | 70.00th=[ 24], 80.00th=[ 31], 90.00th=[ 44], 95.00th=[ 55], | 99.00th=[ 73], 99.50th=[ 86], 99.90th=[ 111], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 16, max= 288, per=24.66%, avg=80.40, stdev=48.61, samples=145 iops : min= 4, max= 72, avg=19.78, stdev=12.24, samples=145 write: IOPS=21, BW=86.4KiB/s (88.5kB/s)(6348KiB/73467msec); 0 zone resets clat (msec): min=3, max=126, avg=25.93, stdev=17.42 lat (msec): min=3, max=126, avg=25.93, stdev=17.42 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 12], | 30.00th=[ 15], 40.00th=[ 18], 50.00th=[ 21], 60.00th=[ 26], | 70.00th=[ 33], 80.00th=[ 40], 90.00th=[ 51], 95.00th=[ 59], | 99.00th=[ 84], 99.50th=[ 89], 99.90th=[ 125], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 31, max= 272, per=25.04%, avg=85.63, stdev=47.21, samples=145 iops : min= 7, max= 68, avg=21.09, stdev=11.88, samples=145 lat (msec) : 4=0.75%, 10=20.25%, 20=33.50%, 50=37.04%, 100=8.27% lat (msec) : 250=0.16%, 750=0.03% cpu : usr=0.06%, sys=14.35%, ctx=4776, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193905: Wed Jul 16 14:29:33 2025 read: IOPS=20, BW=82.3KiB/s (84.3kB/s)(6028KiB/73220msec) clat (msec): min=3, max=116, avg=21.51, stdev=15.35 lat (msec): min=3, max=116, avg=21.51, stdev=15.35 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 12], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 20], | 70.00th=[ 24], 80.00th=[ 32], 90.00th=[ 44], 95.00th=[ 55], | 99.00th=[ 68], 99.50th=[ 81], 99.90th=[ 110], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 15, max= 304, per=25.26%, avg=82.35, stdev=46.96, samples=144 iops : min= 3, max= 76, avg=20.26, stdev=11.82, samples=144 write: IOPS=21, BW=85.5KiB/s (87.5kB/s)(6260KiB/73220msec); 0 zone resets clat (msec): min=5, max=708, avg=25.95, stdev=24.20 lat (msec): min=5, max=708, avg=25.95, stdev=24.20 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 12], | 30.00th=[ 15], 40.00th=[ 18], 50.00th=[ 21], 60.00th=[ 26], | 70.00th=[ 32], 80.00th=[ 38], 90.00th=[ 48], 95.00th=[ 59], | 99.00th=[ 83], 99.50th=[ 93], 99.90th=[ 155], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 23, max= 256, per=24.97%, avg=85.40, stdev=46.18, samples=144 iops : min= 5, max= 64, avg=21.03, stdev=11.62, samples=144 lat (msec) : 4=0.03%, 10=19.66%, 20=35.45%, 50=37.27%, 100=7.39% lat (msec) : 250=0.16%, 750=0.03% cpu : usr=0.08%, sys=14.46%, ctx=4850, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193906: Wed Jul 16 14:29:33 2025 read: IOPS=20, BW=82.7KiB/s (84.7kB/s)(6052KiB/73156msec) clat (msec): min=3, max=122, avg=21.72, stdev=15.96 lat (msec): min=3, max=122, avg=21.72, stdev=15.96 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 12], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 21], | 70.00th=[ 24], 80.00th=[ 32], 90.00th=[ 44], 95.00th=[ 55], | 99.00th=[ 81], 99.50th=[ 91], 99.90th=[ 118], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 15, max= 304, per=25.41%, avg=82.84, stdev=47.35, samples=144 iops : min= 3, max= 76, avg=20.39, stdev=11.92, samples=144 write: IOPS=21, BW=85.2KiB/s (87.3kB/s)(6236KiB/73156msec); 0 zone resets clat (msec): min=4, max=705, avg=25.72, stdev=23.70 lat (msec): min=4, max=705, avg=25.72, stdev=23.70 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 12], | 30.00th=[ 15], 40.00th=[ 18], 50.00th=[ 21], 60.00th=[ 26], | 70.00th=[ 32], 80.00th=[ 39], 90.00th=[ 47], 95.00th=[ 58], | 99.00th=[ 78], 99.50th=[ 89], 99.90th=[ 123], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 23, max= 280, per=25.00%, avg=85.51, stdev=46.42, samples=144 iops : min= 5, max= 70, avg=21.06, stdev=11.67, samples=144 lat (msec) : 4=0.10%, 10=18.82%, 20=35.06%, 50=38.96%, 100=6.84% lat (msec) : 250=0.20%, 750=0.03% cpu : usr=0.06%, sys=14.65%, ctx=4888, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=327KiB/s (335kB/s), 80.9KiB/s-82.7KiB/s (82.8kB/s-84.7kB/s), io=23.4MiB (24.6MB), run=73156-73467msec WRITE: bw=342KiB/s (350kB/s), 85.2KiB/s-86.4KiB/s (87.3kB/s-88.5kB/s), io=24.6MiB (25.7MB), run=73156-73467msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=193930: Wed Jul 16 14:30:03 2025 read: IOPS=16, BW=266KiB/s (273kB/s)(5920KiB/22233msec) clat (usec): min=511, max=84101, avg=30018.84, stdev=14956.74 lat (usec): min=512, max=84102, avg=30019.60, stdev=14956.70 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 1123], 10.00th=[14615], 20.00th=[19530], | 30.00th=[22152], 40.00th=[24773], 50.00th=[28443], 60.00th=[31589], | 70.00th=[35914], 80.00th=[40633], 90.00th=[47973], 95.00th=[56886], | 99.00th=[74974], 99.50th=[81265], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 95, max= 542, per=25.41%, avg=265.07, stdev=87.55, samples=44 iops : min= 5, max= 33, avg=15.93, stdev= 5.59, samples=44 write: IOPS=17, BW=286KiB/s (293kB/s)(6368KiB/22233msec); 0 zone resets clat (usec): min=1156, max=87148, avg=27773.94, stdev=16284.90 lat (usec): min=1157, max=87150, avg=27775.31, stdev=16284.91 clat percentiles (usec): | 1.00th=[ 5342], 5.00th=[ 8455], 10.00th=[ 9634], 20.00th=[12780], | 30.00th=[16057], 40.00th=[19792], 50.00th=[25560], 60.00th=[29230], | 70.00th=[33817], 80.00th=[41681], 90.00th=[49546], 95.00th=[60556], | 99.00th=[78119], 99.50th=[80217], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 63, max= 480, per=25.99%, avg=285.11, stdev=86.65, samples=44 iops : min= 3, max= 30, avg=17.18, stdev= 5.54, samples=44 lat (usec) : 750=1.30%, 1000=1.04% lat (msec) : 2=0.52%, 4=0.26%, 10=5.73%, 20=23.18%, 50=58.85% lat (msec) : 100=9.11% cpu : usr=0.06%, sys=31.40%, ctx=3293, majf=0, minf=17 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193931: Wed Jul 16 14:30:03 2025 read: IOPS=15, BW=251KiB/s (258kB/s)(5744KiB/22840msec) clat (usec): min=359, max=94796, avg=32653.20, stdev=14755.00 lat (usec): min=360, max=94797, avg=32653.93, stdev=14755.02 clat percentiles (usec): | 1.00th=[ 635], 5.00th=[14091], 10.00th=[16909], 20.00th=[21365], | 30.00th=[24511], 40.00th=[28181], 50.00th=[30802], 60.00th=[33817], | 70.00th=[36963], 80.00th=[42730], 90.00th=[52167], 95.00th=[60556], | 99.00th=[79168], 99.50th=[82314], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 96, max= 383, per=23.91%, avg=249.40, stdev=67.22, samples=45 iops : min= 6, max= 23, avg=14.93, stdev= 4.19, samples=45 write: IOPS=17, BW=287KiB/s (293kB/s)(6544KiB/22840msec); 0 zone resets clat (usec): min=1050, max=120975, avg=27106.06, stdev=16576.44 lat (usec): min=1051, max=120976, avg=27107.42, stdev=16576.45 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 17], 40.00th=[ 21], 50.00th=[ 24], 60.00th=[ 28], | 70.00th=[ 33], 80.00th=[ 39], 90.00th=[ 54], 95.00th=[ 60], | 99.00th=[ 72], 99.50th=[ 74], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 158, max= 546, per=26.02%, avg=285.42, stdev=93.93, samples=45 iops : min= 9, max= 34, avg=17.20, stdev= 5.92, samples=45 lat (usec) : 500=0.13%, 750=0.52%, 1000=0.26% lat (msec) : 2=0.26%, 4=0.13%, 10=5.21%, 20=22.27%, 50=59.51% lat (msec) : 100=11.46%, 250=0.26% cpu : usr=0.05%, sys=31.45%, ctx=3265, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193932: Wed Jul 16 14:30:03 2025 read: IOPS=17, BW=272KiB/s (279kB/s)(6256KiB/22963msec) clat (usec): min=536, max=98391, avg=31753.92, stdev=15225.95 lat (usec): min=537, max=98392, avg=31754.72, stdev=15225.98 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[12387], 10.00th=[15139], 20.00th=[19792], | 30.00th=[23725], 40.00th=[27132], 50.00th=[30016], 60.00th=[33817], | 70.00th=[36963], 80.00th=[42206], 90.00th=[51643], 95.00th=[56886], | 99.00th=[84411], 99.50th=[90702], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 126, max= 415, per=26.02%, avg=271.40, stdev=69.49, samples=45 iops : min= 7, max= 25, avg=16.31, stdev= 4.37, samples=45 write: IOPS=16, BW=263KiB/s (269kB/s)(6032KiB/22963msec); 0 zone resets clat (usec): min=1065, max=83306, avg=27819.01, stdev=15992.93 lat (usec): min=1066, max=83309, avg=27820.67, stdev=15992.98 clat percentiles (usec): | 1.00th=[ 4359], 5.00th=[ 8029], 10.00th=[ 9634], 20.00th=[13698], | 30.00th=[17957], 40.00th=[20579], 50.00th=[24249], 60.00th=[29230], | 70.00th=[33424], 80.00th=[41157], 90.00th=[52167], 95.00th=[58459], | 99.00th=[77071], 99.50th=[83362], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 126, max= 510, per=23.46%, avg=257.31, stdev=92.70, samples=45 iops : min= 7, max= 31, avg=15.42, stdev= 5.80, samples=45 lat (usec) : 750=0.91%, 1000=0.78% lat (msec) : 2=0.65%, 4=0.13%, 10=4.82%, 20=22.40%, 50=58.98% lat (msec) : 100=11.33% cpu : usr=0.07%, sys=30.60%, ctx=3290, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193933: Wed Jul 16 14:30:03 2025 read: IOPS=17, BW=274KiB/s (280kB/s)(6032KiB/22049msec) clat (usec): min=283, max=81674, avg=30523.92, stdev=14787.85 lat (usec): min=284, max=81675, avg=30524.82, stdev=14787.61 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 881], 10.00th=[14353], 20.00th=[19792], | 30.00th=[22938], 40.00th=[27132], 50.00th=[30278], 60.00th=[33424], | 70.00th=[36439], 80.00th=[40633], 90.00th=[48497], 95.00th=[58983], | 99.00th=[69731], 99.50th=[80217], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 63, max= 416, per=26.02%, avg=271.35, stdev=73.86, samples=43 iops : min= 3, max= 26, avg=16.30, stdev= 4.63, samples=43 write: IOPS=17, BW=284KiB/s (291kB/s)(6256KiB/22049msec); 0 zone resets clat (usec): min=5184, max=85678, avg=26827.73, stdev=16661.66 lat (usec): min=5186, max=85679, avg=26840.59, stdev=16652.22 clat percentiles (usec): | 1.00th=[ 6128], 5.00th=[ 8356], 10.00th=[ 9765], 20.00th=[12518], | 30.00th=[15795], 40.00th=[18744], 50.00th=[22938], 60.00th=[26608], | 70.00th=[31327], 80.00th=[38011], 90.00th=[52691], 95.00th=[62653], | 99.00th=[77071], 99.50th=[82314], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 96, max= 523, per=25.55%, avg=280.26, stdev=94.45, samples=43 iops : min= 6, max= 32, avg=16.86, stdev= 5.89, samples=43 lat (usec) : 500=0.39%, 750=1.82%, 1000=0.78% lat (msec) : 2=0.39%, 4=0.26%, 10=5.86%, 20=23.31%, 50=56.77% lat (msec) : 100=10.42% cpu : usr=0.07%, sys=31.66%, ctx=3122, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1043KiB/s (1068kB/s), 251KiB/s-274KiB/s (258kB/s-280kB/s), io=23.4MiB (24.5MB), run=22049-22963msec WRITE: bw=1097KiB/s (1124kB/s), 263KiB/s-287KiB/s (269kB/s-293kB/s), io=24.6MiB (25.8MB), run=22049-22963msec rand-rw: (groupid=0, jobs=1): err= 0: pid=193935: Wed Jul 16 14:30:05 2025 read: IOPS=15, BW=244KiB/s (250kB/s)(5920KiB/24227msec) clat (msec): min=2, max=109, avg=28.09, stdev=20.90 lat (msec): min=2, max=109, avg=28.09, stdev=20.90 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 12], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 22], 60.00th=[ 26], | 70.00th=[ 32], 80.00th=[ 43], 90.00th=[ 59], 95.00th=[ 73], | 99.00th=[ 96], 99.50th=[ 105], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 32, max= 896, per=24.70%, avg=244.06, stdev=172.11, samples=48 iops : min= 2, max= 56, avg=14.56, stdev=10.85, samples=48 write: IOPS=16, BW=263KiB/s (269kB/s)(6368KiB/24227msec); 0 zone resets clat (msec): min=3, max=119, avg=34.34, stdev=23.42 lat (msec): min=3, max=119, avg=34.34, stdev=23.42 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 14], | 30.00th=[ 19], 40.00th=[ 23], 50.00th=[ 29], 60.00th=[ 37], | 70.00th=[ 45], 80.00th=[ 54], 90.00th=[ 67], 95.00th=[ 79], | 99.00th=[ 109], 99.50th=[ 117], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 95, max= 894, per=25.28%, avg=262.88, stdev=162.13, samples=48 iops : min= 5, max= 55, avg=15.75, stdev=10.25, samples=48 lat (msec) : 4=0.91%, 10=13.80%, 20=26.04%, 50=38.67%, 100=19.53% lat (msec) : 250=1.04% cpu : usr=0.06%, sys=15.38%, ctx=1486, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193936: Wed Jul 16 14:30:05 2025 read: IOPS=14, BW=238KiB/s (243kB/s)(5744KiB/24176msec) clat (msec): min=4, max=105, avg=30.02, stdev=21.16 lat (msec): min=4, max=105, avg=30.02, stdev=21.16 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 21], 50.00th=[ 24], 60.00th=[ 28], | 70.00th=[ 36], 80.00th=[ 47], 90.00th=[ 59], 95.00th=[ 73], | 99.00th=[ 100], 99.50th=[ 105], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 52, max= 736, per=23.64%, avg=233.53, stdev=127.72, samples=47 iops : min= 3, max= 46, avg=13.89, stdev= 8.05, samples=47 write: IOPS=16, BW=271KiB/s (277kB/s)(6544KiB/24176msec); 0 zone resets clat (msec): min=6, max=122, avg=32.36, stdev=21.21 lat (msec): min=6, max=122, avg=32.36, stdev=21.21 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 11], 20.00th=[ 15], | 30.00th=[ 18], 40.00th=[ 23], 50.00th=[ 27], 60.00th=[ 32], | 70.00th=[ 42], 80.00th=[ 50], 90.00th=[ 64], 95.00th=[ 72], | 99.00th=[ 91], 99.50th=[ 112], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 95, max= 894, per=24.98%, avg=259.83, stdev=154.15, samples=47 iops : min= 5, max= 55, avg=15.53, stdev= 9.73, samples=47 lat (msec) : 10=11.85%, 20=25.91%, 50=43.62%, 100=17.71%, 250=0.91% cpu : usr=0.06%, sys=15.19%, ctx=1532, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193937: Wed Jul 16 14:30:05 2025 read: IOPS=16, BW=263KiB/s (269kB/s)(6256KiB/23808msec) clat (msec): min=3, max=116, avg=27.76, stdev=18.85 lat (msec): min=3, max=116, avg=27.76, stdev=18.85 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 13], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 27], | 70.00th=[ 32], 80.00th=[ 44], 90.00th=[ 55], 95.00th=[ 65], | 99.00th=[ 88], 99.50th=[ 89], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 95, max= 1245, per=26.66%, avg=263.43, stdev=185.20, samples=47 iops : min= 5, max= 77, avg=15.77, stdev=11.66, samples=47 write: IOPS=15, BW=253KiB/s (259kB/s)(6032KiB/23808msec); 0 zone resets clat (msec): min=5, max=118, avg=33.99, stdev=21.01 lat (msec): min=5, max=118, avg=33.99, stdev=21.01 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 17], | 30.00th=[ 21], 40.00th=[ 24], 50.00th=[ 30], 60.00th=[ 35], | 70.00th=[ 42], 80.00th=[ 51], 90.00th=[ 64], 95.00th=[ 75], | 99.00th=[ 107], 99.50th=[ 111], 99.90th=[ 118], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 95, max= 800, per=24.06%, avg=250.23, stdev=130.12, samples=47 iops : min= 5, max= 50, avg=14.94, stdev= 8.23, samples=47 lat (msec) : 4=0.13%, 10=11.85%, 20=23.96%, 50=47.27%, 100=15.89% lat (msec) : 250=0.91% cpu : usr=0.06%, sys=15.72%, ctx=1517, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193938: Wed Jul 16 14:30:05 2025 read: IOPS=15, BW=250KiB/s (256kB/s)(6032KiB/24150msec) clat (msec): min=4, max=112, avg=28.64, stdev=20.76 lat (msec): min=4, max=112, avg=28.64, stdev=20.76 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 13], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 23], 60.00th=[ 27], | 70.00th=[ 34], 80.00th=[ 44], 90.00th=[ 60], 95.00th=[ 71], | 99.00th=[ 99], 99.50th=[ 103], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 32, max= 862, per=24.37%, avg=240.74, stdev=153.82, samples=47 iops : min= 2, max= 53, avg=14.34, stdev= 9.66, samples=47 write: IOPS=16, BW=259KiB/s (265kB/s)(6256KiB/24150msec); 0 zone resets clat (msec): min=5, max=122, avg=33.78, stdev=22.53 lat (msec): min=5, max=122, avg=33.78, stdev=22.53 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 11], 20.00th=[ 15], | 30.00th=[ 18], 40.00th=[ 23], 50.00th=[ 28], 60.00th=[ 35], | 70.00th=[ 41], 80.00th=[ 53], 90.00th=[ 65], 95.00th=[ 79], | 99.00th=[ 102], 99.50th=[ 123], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 95, max= 894, per=24.09%, avg=250.55, stdev=127.25, samples=47 iops : min= 5, max= 55, avg=14.96, stdev= 8.05, samples=47 lat (msec) : 10=13.02%, 20=26.30%, 50=41.93%, 100=17.84%, 250=0.91% cpu : usr=0.08%, sys=15.03%, ctx=1512, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=989KiB/s (1012kB/s), 238KiB/s-263KiB/s (243kB/s-269kB/s), io=23.4MiB (24.5MB), run=23808-24227msec WRITE: bw=1040KiB/s (1065kB/s), 253KiB/s-271KiB/s (259kB/s-277kB/s), io=24.6MiB (25.8MB), run=23808-24227msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=193954: Wed Jul 16 14:30:17 2025 read: IOPS=1, BW=1847KiB/s (1891kB/s)(7168KiB/3881msec) clat (usec): min=1305, max=420396, avg=224012.98, stdev=131035.19 lat (usec): min=1306, max=420397, avg=224014.31, stdev=131035.26 clat percentiles (usec): | 1.00th=[ 1303], 5.00th=[ 1303], 10.00th=[ 1303], 20.00th=[179307], | 30.00th=[191890], 40.00th=[191890], 50.00th=[210764], 60.00th=[231736], | 70.00th=[231736], 80.00th=[333448], 90.00th=[421528], 95.00th=[421528], | 99.00th=[421528], 99.50th=[421528], 99.90th=[421528], 99.95th=[421528], | 99.99th=[421528] bw ( KiB/s): min= 2023, max= 3482, per=57.92%, avg=2328.80, stdev=644.74, samples=5 iops : min= 1, max= 3, avg= 1.80, stdev= 0.84, samples=5 write: IOPS=1, BW=1319KiB/s (1351kB/s)(5120KiB/3881msec); 0 zone resets clat (msec): min=346, max=682, avg=456.65, stdev=139.63 lat (msec): min=346, max=682, avg=456.70, stdev=139.65 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 347], 10.00th=[ 347], 20.00th=[ 347], | 30.00th=[ 372], 40.00th=[ 372], 50.00th=[ 380], 60.00th=[ 380], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 1957, max= 2048, per=35.64%, avg=2006.50, stdev=48.36, samples=4 iops : min= 1, max= 2, avg= 1.50, stdev= 0.58, samples=4 lat (msec) : 2=8.33%, 250=33.33%, 500=41.67%, 750=16.67% cpu : usr=0.00%, sys=58.53%, ctx=1325, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193955: Wed Jul 16 14:30:17 2025 read: IOPS=0, BW=402KiB/s (412kB/s)(2048KiB/5092msec) clat (usec): min=1221, max=251760, avg=126490.83, stdev=177158.03 lat (usec): min=1222, max=251761, avg=126491.97, stdev=177158.11 clat percentiles (usec): | 1.00th=[ 1221], 5.00th=[ 1221], 10.00th=[ 1221], 20.00th=[ 1221], | 30.00th=[ 1221], 40.00th=[ 1221], 50.00th=[ 1221], 60.00th=[252707], | 70.00th=[252707], 80.00th=[252707], 90.00th=[252707], 95.00th=[252707], | 99.00th=[252707], 99.50th=[252707], 99.90th=[252707], 99.95th=[252707], | 99.99th=[252707] bw ( KiB/s): min= 1750, max= 2023, per=46.92%, avg=1886.50, stdev=193.04, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=1, BW=2011KiB/s (2059kB/s)(10.0MiB/5092msec); 0 zone resets clat (msec): min=273, max=784, avg=469.63, stdev=162.52 lat (msec): min=273, max=784, avg=469.75, stdev=162.52 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 275], 10.00th=[ 275], 20.00th=[ 351], | 30.00th=[ 359], 40.00th=[ 393], 50.00th=[ 435], 60.00th=[ 435], | 70.00th=[ 460], 80.00th=[ 485], 90.00th=[ 726], 95.00th=[ 785], | 99.00th=[ 785], 99.50th=[ 785], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 1957, max= 4096, per=44.81%, avg=2522.57, stdev=893.80, samples=7 iops : min= 1, max= 4, avg= 1.86, stdev= 1.21, samples=7 lat (msec) : 2=8.33%, 500=75.00%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=48.08%, ctx=579, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193956: Wed Jul 16 14:30:17 2025 read: IOPS=1, BW=1448KiB/s (1483kB/s)(6144KiB/4242msec) clat (usec): min=1056, max=396817, avg=182537.03, stdev=172782.41 lat (usec): min=1058, max=396818, avg=182538.28, stdev=172782.10 clat percentiles (usec): | 1.00th=[ 1057], 5.00th=[ 1057], 10.00th=[ 1057], 20.00th=[ 1074], | 30.00th=[ 1074], 40.00th=[ 94897], 50.00th=[ 94897], 60.00th=[270533], | 70.00th=[333448], 80.00th=[333448], 90.00th=[396362], 95.00th=[396362], | 99.00th=[396362], 99.50th=[396362], 99.90th=[396362], 99.95th=[396362], | 99.99th=[396362] bw ( KiB/s): min= 1957, max= 4096, per=62.48%, avg=2512.25, stdev=1056.21, samples=4 iops : min= 1, max= 4, avg= 1.75, stdev= 1.50, samples=4 write: IOPS=1, BW=1448KiB/s (1483kB/s)(6144KiB/4242msec); 0 zone resets clat (msec): min=302, max=657, avg=503.29, stdev=134.91 lat (msec): min=302, max=657, avg=503.33, stdev=134.90 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 305], 10.00th=[ 305], 20.00th=[ 430], | 30.00th=[ 430], 40.00th=[ 456], 50.00th=[ 456], 60.00th=[ 535], | 70.00th=[ 642], 80.00th=[ 642], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 1735, max= 2048, per=35.14%, avg=1978.33, stdev=122.58, samples=6 iops : min= 1, max= 2, avg= 1.33, stdev= 0.52, samples=6 lat (msec) : 2=16.67%, 100=8.33%, 500=50.00%, 750=25.00% cpu : usr=0.00%, sys=45.60%, ctx=538, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193957: Wed Jul 16 14:30:17 2025 read: IOPS=1, BW=1098KiB/s (1124kB/s)(5120KiB/4664msec) clat (usec): min=950, max=290878, avg=152656.37, stdev=140393.58 lat (usec): min=950, max=290879, avg=152657.56, stdev=140394.19 clat percentiles (usec): | 1.00th=[ 947], 5.00th=[ 947], 10.00th=[ 947], 20.00th=[ 947], | 30.00th=[ 1029], 40.00th=[ 1029], 50.00th=[227541], 60.00th=[227541], | 70.00th=[242222], 80.00th=[242222], 90.00th=[291505], 95.00th=[291505], | 99.00th=[291505], 99.50th=[291505], 99.90th=[291505], 99.95th=[291505], | 99.99th=[291505] bw ( KiB/s): min= 1973, max= 5873, per=97.56%, avg=3923.00, stdev=2757.72, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 write: IOPS=1, BW=1537KiB/s (1574kB/s)(7168KiB/4664msec); 0 zone resets clat (msec): min=217, max=1488, avg=541.12, stdev=487.33 lat (msec): min=217, max=1488, avg=541.18, stdev=487.33 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 218], 10.00th=[ 218], 20.00th=[ 220], | 30.00th=[ 257], 40.00th=[ 257], 50.00th=[ 334], 60.00th=[ 338], | 70.00th=[ 338], 80.00th=[ 936], 90.00th=[ 1485], 95.00th=[ 1485], | 99.00th=[ 1485], 99.50th=[ 1485], 99.90th=[ 1485], 99.95th=[ 1485], | 99.99th=[ 1485] bw ( KiB/s): min= 1973, max= 6144, per=54.12%, avg=3047.00, stdev=2064.90, samples=4 iops : min= 1, max= 6, avg= 2.50, stdev= 2.38, samples=4 lat (usec) : 1000=8.33% lat (msec) : 2=8.33%, 250=33.33%, 500=33.33%, 1000=8.33%, 2000=8.33% cpu : usr=0.00%, sys=40.83%, ctx=282, majf=0, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4022KiB/s (4119kB/s), 402KiB/s-1847KiB/s (412kB/s-1891kB/s), io=20.0MiB (20.0MB), run=3881-5092msec WRITE: bw=5631KiB/s (5766kB/s), 1319KiB/s-2011KiB/s (1351kB/s-2059kB/s), io=28.0MiB (29.4MB), run=3881-5092msec rand-rw: (groupid=0, jobs=1): err= 0: pid=193949: Wed Jul 16 14:30:18 2025 read: IOPS=1, BW=1506KiB/s (1542kB/s)(7168KiB/4759msec) clat (msec): min=108, max=548, avg=212.71, stdev=153.56 lat (msec): min=108, max=548, avg=212.71, stdev=153.56 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 109], 10.00th=[ 109], 20.00th=[ 116], | 30.00th=[ 138], 40.00th=[ 138], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 184], 80.00th=[ 224], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 2043, max= 5086, per=73.65%, avg=2806.25, stdev=1519.84, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 write: IOPS=1, BW=1076KiB/s (1102kB/s)(5120KiB/4759msec); 0 zone resets clat (msec): min=170, max=1476, avg=645.70, stdev=533.58 lat (msec): min=170, max=1476, avg=645.79, stdev=533.57 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 192], 40.00th=[ 192], 50.00th=[ 617], 60.00th=[ 617], | 70.00th=[ 768], 80.00th=[ 768], 90.00th=[ 1485], 95.00th=[ 1485], | 99.00th=[ 1485], 99.50th=[ 1485], 99.90th=[ 1485], 99.95th=[ 1485], | 99.99th=[ 1485] bw ( KiB/s): min= 1943, max= 2048, per=37.88%, avg=2020.50, stdev=51.72, samples=4 iops : min= 1, max= 2, avg= 1.50, stdev= 0.58, samples=4 lat (msec) : 250=66.67%, 750=16.67%, 1000=8.33%, 2000=8.33% cpu : usr=0.00%, sys=21.48%, ctx=196, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193951: Wed Jul 16 14:30:18 2025 read: IOPS=0, BW=383KiB/s (393kB/s)(2048KiB/5343msec) clat (msec): min=111, max=681, avg=396.50, stdev=402.98 lat (msec): min=111, max=681, avg=396.50, stdev=402.98 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 112], 50.00th=[ 112], 60.00th=[ 684], | 70.00th=[ 684], 80.00th=[ 684], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 1759, max= 2043, per=49.90%, avg=1901.00, stdev=200.82, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=1, BW=1917KiB/s (1963kB/s)(10.0MiB/5343msec); 0 zone resets clat (msec): min=92, max=1691, avg=443.52, stdev=487.77 lat (msec): min=92, max=1691, avg=443.59, stdev=487.76 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 93], 10.00th=[ 93], 20.00th=[ 120], | 30.00th=[ 180], 40.00th=[ 199], 50.00th=[ 241], 60.00th=[ 243], | 70.00th=[ 409], 80.00th=[ 443], 90.00th=[ 818], 95.00th=[ 1687], | 99.00th=[ 1687], 99.50th=[ 1687], 99.90th=[ 1687], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 1759, max= 8192, per=67.54%, avg=3602.40, stdev=2736.92, samples=5 iops : min= 1, max= 8, avg= 3.20, stdev= 2.95, samples=5 lat (msec) : 100=8.33%, 250=50.00%, 500=16.67%, 750=8.33%, 1000=8.33% lat (msec) : 2000=8.33% cpu : usr=0.02%, sys=22.67%, ctx=199, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193952: Wed Jul 16 14:30:18 2025 read: IOPS=1, BW=1143KiB/s (1171kB/s)(6144KiB/5375msec) clat (msec): min=65, max=740, avg=361.02, stdev=260.86 lat (msec): min=65, max=740, avg=361.02, stdev=260.86 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 66], 10.00th=[ 66], 20.00th=[ 116], | 30.00th=[ 116], 40.00th=[ 253], 50.00th=[ 253], 60.00th=[ 489], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 743], 95.00th=[ 743], | 99.00th=[ 743], 99.50th=[ 743], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 1759, max= 4096, per=64.44%, avg=2455.00, stdev=1100.36, samples=4 iops : min= 1, max= 4, avg= 2.00, stdev= 1.41, samples=4 write: IOPS=1, BW=1143KiB/s (1171kB/s)(6144KiB/5375msec); 0 zone resets clat (msec): min=104, max=1095, avg=531.78, stdev=374.86 lat (msec): min=105, max=1095, avg=531.85, stdev=374.87 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 106], 10.00th=[ 106], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 472], 50.00th=[ 472], 60.00th=[ 684], | 70.00th=[ 693], 80.00th=[ 693], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 1706, max= 2048, per=36.60%, avg=1952.40, stdev=148.71, samples=5 iops : min= 1, max= 2, avg= 1.40, stdev= 0.55, samples=5 lat (msec) : 100=8.33%, 250=25.00%, 500=25.00%, 750=33.33%, 2000=8.33% cpu : usr=0.00%, sys=20.54%, ctx=189, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193953: Wed Jul 16 14:30:18 2025 read: IOPS=0, BW=971KiB/s (994kB/s)(5120KiB/5275msec) clat (msec): min=110, max=746, avg=371.25, stdev=268.56 lat (msec): min=110, max=746, avg=371.25, stdev=268.56 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 178], 40.00th=[ 178], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 751], 95.00th=[ 751], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 3835, max= 4039, per=100.00%, avg=3937.00, stdev=144.25, samples=2 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=2 write: IOPS=1, BW=1359KiB/s (1391kB/s)(7168KiB/5275msec); 0 zone resets clat (msec): min=225, max=669, avg=476.30, stdev=163.02 lat (msec): min=225, max=669, avg=476.38, stdev=163.00 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 226], 10.00th=[ 226], 20.00th=[ 342], | 30.00th=[ 405], 40.00th=[ 405], 50.00th=[ 502], 60.00th=[ 542], | 70.00th=[ 542], 80.00th=[ 659], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 1759, max= 4087, per=43.86%, avg=2339.67, stdev=863.79, samples=6 iops : min= 1, max= 3, avg= 2.00, stdev= 0.63, samples=6 lat (msec) : 250=25.00%, 500=25.00%, 750=50.00% cpu : usr=0.02%, sys=20.65%, ctx=358, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3810KiB/s (3902kB/s), 383KiB/s-1506KiB/s (393kB/s-1542kB/s), io=20.0MiB (20.0MB), run=4759-5375msec WRITE: bw=5334KiB/s (5462kB/s), 1076KiB/s-1917KiB/s (1102kB/s-1963kB/s), io=28.0MiB (29.4MB), run=4759-5375msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=193963: Wed Jul 16 14:30:30 2025 read: IOPS=0, BW=2205KiB/s (2258kB/s)(12.0MiB/5572msec) clat (msec): min=1659, max=2018, avg=1853.14, stdev=180.81 lat (msec): min=1659, max=2018, avg=1853.14, stdev=180.81 clat percentiles (msec): | 1.00th=[ 1653], 5.00th=[ 1653], 10.00th=[ 1653], 20.00th=[ 1653], | 30.00th=[ 1653], 40.00th=[ 1888], 50.00th=[ 1888], 60.00th=[ 1888], | 70.00th=[ 2022], 80.00th=[ 2022], 90.00th=[ 2022], 95.00th=[ 2022], | 99.00th=[ 2022], 99.50th=[ 2022], 99.90th=[ 2022], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 8047, max= 8192, per=100.00%, avg=8119.50, stdev=102.53, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 2000=66.67%, >=2000=33.33% cpu : usr=0.00%, sys=9.51%, ctx=43, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193964: Wed Jul 16 14:30:30 2025 read: IOPS=0, BW=679KiB/s (695kB/s)(4096KiB/6033msec) clat (nsec): min=847828k, max=847828k, avg=847827605.00, stdev= 0.00 lat (nsec): min=847830k, max=847830k, avg=847829943.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 852], 5.00th=[ 852], 10.00th=[ 852], 20.00th=[ 852], | 30.00th=[ 852], 40.00th=[ 852], 50.00th=[ 852], 60.00th=[ 852], | 70.00th=[ 852], 80.00th=[ 852], 90.00th=[ 852], 95.00th=[ 852], | 99.00th=[ 852], 99.50th=[ 852], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1358KiB/s (1390kB/s)(8192KiB/6033msec); 0 zone resets clat (msec): min=641, max=4514, avg=2577.65, stdev=2738.63 lat (msec): min=641, max=4514, avg=2577.90, stdev=2738.73 clat percentiles (msec): | 1.00th=[ 642], 5.00th=[ 642], 10.00th=[ 642], 20.00th=[ 642], | 30.00th=[ 642], 40.00th=[ 642], 50.00th=[ 642], 60.00th=[ 4530], | 70.00th=[ 4530], 80.00th=[ 4530], 90.00th=[ 4530], 95.00th=[ 4530], | 99.00th=[ 4530], 99.50th=[ 4530], 99.90th=[ 4530], 99.95th=[ 4530], | 99.99th=[ 4530] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 750=33.33%, 1000=33.33%, >=2000=33.33% cpu : usr=0.00%, sys=22.25%, ctx=109, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193965: Wed Jul 16 14:30:30 2025 write: IOPS=0, BW=3688KiB/s (3776kB/s)(12.0MiB/3332msec); 0 zone resets clat (msec): min=189, max=2824, avg=1107.95, stdev=1487.69 lat (msec): min=189, max=2824, avg=1108.20, stdev=1487.74 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 190], 10.00th=[ 190], 20.00th=[ 190], | 30.00th=[ 190], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 2836], 80.00th=[ 2836], 90.00th=[ 2836], 95.00th=[ 2836], | 99.00th=[ 2836], 99.50th=[ 2836], 99.90th=[ 2836], 99.95th=[ 2836], | 99.99th=[ 2836] bw ( KiB/s): min=15058, max=15058, per=100.00%, avg=15058.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, >=2000=33.33% cpu : usr=0.00%, sys=26.12%, ctx=154, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193966: Wed Jul 16 14:30:30 2025 write: IOPS=0, BW=2371KiB/s (2428kB/s)(12.0MiB/5182msec); 0 zone resets clat (msec): min=330, max=4176, avg=1726.12, stdev=2128.83 lat (msec): min=331, max=4176, avg=1726.43, stdev=2128.95 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 330], | 30.00th=[ 330], 40.00th=[ 676], 50.00th=[ 676], 60.00th=[ 676], | 70.00th=[ 4178], 80.00th=[ 4178], 90.00th=[ 4178], 95.00th=[ 4178], | 99.00th=[ 4178], 99.50th=[ 4178], 99.90th=[ 4178], 99.95th=[ 4178], | 99.99th=[ 4178] bw ( KiB/s): min= 7984, max= 8175, per=100.00%, avg=8079.50, stdev=135.06, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=33.33%, >=2000=33.33% cpu : usr=0.02%, sys=14.40%, ctx=320, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2716KiB/s (2781kB/s), 679KiB/s-2205KiB/s (695kB/s-2258kB/s), io=16.0MiB (16.8MB), run=5572-6033msec WRITE: bw=5431KiB/s (5562kB/s), 1358KiB/s-3688KiB/s (1390kB/s-3776kB/s), io=32.0MiB (33.6MB), run=3332-6033msec rand-rw: (groupid=0, jobs=1): err= 0: pid=193968: Wed Jul 16 14:30:30 2025 read: IOPS=0, BW=4010KiB/s (4107kB/s)(12.0MiB/3064msec) clat (msec): min=873, max=1176, avg=1020.75, stdev=151.69 lat (msec): min=873, max=1176, avg=1020.75, stdev=151.69 clat percentiles (msec): | 1.00th=[ 877], 5.00th=[ 877], 10.00th=[ 877], 20.00th=[ 877], | 30.00th=[ 877], 40.00th=[ 1011], 50.00th=[ 1011], 60.00th=[ 1011], | 70.00th=[ 1183], 80.00th=[ 1183], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 7098, max= 7846, per=100.00%, avg=7472.00, stdev=528.92, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 1000=33.33%, 2000=66.67% cpu : usr=0.00%, sys=54.95%, ctx=1709, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193969: Wed Jul 16 14:30:30 2025 read: IOPS=0, BW=825KiB/s (845kB/s)(4096KiB/4965msec) clat (nsec): min=876238k, max=876238k, avg=876237966.00, stdev= 0.00 lat (nsec): min=876239k, max=876239k, avg=876239378.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 877], 5.00th=[ 877], 10.00th=[ 877], 20.00th=[ 877], | 30.00th=[ 877], 40.00th=[ 877], 50.00th=[ 877], 60.00th=[ 877], | 70.00th=[ 877], 80.00th=[ 877], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 8094, max= 8094, per=100.00%, avg=8094.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1650KiB/s (1690kB/s)(8192KiB/4965msec); 0 zone resets clat (msec): min=1002, max=3084, avg=2043.38, stdev=1471.92 lat (msec): min=1002, max=3084, avg=2043.64, stdev=1472.08 clat percentiles (msec): | 1.00th=[ 1003], 5.00th=[ 1003], 10.00th=[ 1003], 20.00th=[ 1003], | 30.00th=[ 1003], 40.00th=[ 1003], 50.00th=[ 1003], 60.00th=[ 3071], | 70.00th=[ 3071], 80.00th=[ 3071], 90.00th=[ 3071], 95.00th=[ 3071], | 99.00th=[ 3071], 99.50th=[ 3071], 99.90th=[ 3071], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 1000=33.33%, 2000=33.33%, >=2000=33.33% cpu : usr=0.02%, sys=39.63%, ctx=974, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193970: Wed Jul 16 14:30:30 2025 write: IOPS=0, BW=2165KiB/s (2217kB/s)(12.0MiB/5675msec); 0 zone resets clat (msec): min=1321, max=2535, avg=1891.09, stdev=610.18 lat (msec): min=1321, max=2535, avg=1891.37, stdev=610.14 clat percentiles (msec): | 1.00th=[ 1318], 5.00th=[ 1318], 10.00th=[ 1318], 20.00th=[ 1318], | 30.00th=[ 1318], 40.00th=[ 1821], 50.00th=[ 1821], 60.00th=[ 1821], | 70.00th=[ 2534], 80.00th=[ 2534], 90.00th=[ 2534], 95.00th=[ 2534], | 99.00th=[ 2534], 99.50th=[ 2534], 99.90th=[ 2534], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 7984, max= 8192, per=100.00%, avg=8088.00, stdev=147.08, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 2000=66.67%, >=2000=33.33% cpu : usr=0.00%, sys=42.77%, ctx=121, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=193971: Wed Jul 16 14:30:30 2025 write: IOPS=0, BW=2957KiB/s (3028kB/s)(12.0MiB/4155msec); 0 zone resets clat (msec): min=1100, max=1849, avg=1384.04, stdev=406.12 lat (msec): min=1100, max=1849, avg=1384.36, stdev=406.12 clat percentiles (msec): | 1.00th=[ 1099], 5.00th=[ 1099], 10.00th=[ 1099], 20.00th=[ 1099], | 30.00th=[ 1099], 40.00th=[ 1200], 50.00th=[ 1200], 60.00th=[ 1200], | 70.00th=[ 1854], 80.00th=[ 1854], 90.00th=[ 1854], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 7846, max= 7984, per=100.00%, avg=7915.00, stdev=97.58, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 2000=100.00% cpu : usr=0.00%, sys=60.35%, ctx=366, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3300KiB/s (3379kB/s), 825KiB/s-4010KiB/s (845kB/s-4107kB/s), io=16.0MiB (16.8MB), run=3064-4965msec WRITE: bw=5774KiB/s (5913kB/s), 1650KiB/s-2957KiB/s (1690kB/s-3028kB/s), io=32.0MiB (33.6MB), run=4155-5675msec